From greg.ewing at canterbury.ac.nz Wed Sep 1 00:23:42 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 01 Sep 2010 10:23:42 +1200 Subject: [Python-Dev] PEP 384 status In-Reply-To: References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> Message-ID: <4C7D80EE.8090705@canterbury.ac.nz> Daniel Stutzbach wrote: > Likewise, a FILE * isn't safe to pass around, unless I can guarantee > that the application really is one big happy family compiled against the > same version of the C library. Given that, it seems to me that it's a mistake for Python to provide any APIs that take a FILE* directly in the first place. Maybe PyObject_Print should be deprecated in favour of something that takes a Python I/O object instead. -- Greg From ncoghlan at gmail.com Wed Sep 1 00:31:42 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 1 Sep 2010 08:31:42 +1000 Subject: [Python-Dev] PEP 384 status In-Reply-To: <4C7D78F4.9040900@egenix.com> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D78F4.9040900@egenix.com> Message-ID: On Wed, Sep 1, 2010 at 7:49 AM, M.-A. Lemburg wrote: >> Yes, and it's a pretty common situation. ? The fopen() that I call within a >> DLL may not be the same fopen() called by another DLL. ?When writing a DLL >> for Windows, the API must be designed with the assumption that anything >> returned by the C library cannot be passed a different C library. ?For >> example, suppose I a expose a function in my DLL that allocates some memory, >> populates it with useful information, and returns a pointer. ?I must also >> supply a function to free the memory. ?I cannot ask the caller to simply >> call free(), because their free() may not be using the same heap as my >> malloc(). > > But isn't exactly that a major problem for Python ? > > An extension written for a different MS CRT version would > not be able to safely free a buffer allocated by the Python > DLL. Correct, but we generally don't allow raw pointers across the API boundary like that. Instead, users are told to release memory with PyMem_Free, PyObject_Delete, reference decrementing, etc. That means the memory release happens in the runtime linked with the Python DLL and everything is fine. > >> Likewise, a FILE * isn't safe to pass around, unless I can guarantee that >> the application really is one big happy family compiled against the same >> version of the C library. > > According to the MS document I found this also applies to the OS > environment and handles. > > Taking all these things together makes it sound like there's > a rather small chance of actually getting PEP 384 working > across Windows compiler upgrades. Not really - we just need to keep an eye out for the places where those things can leak through and ensure they're instead abstracted so the extension module only talks to Python rather than being given direct access to the underlying C runtime details. FILE* is one, errno is another, locale we may just have to live with (alternatively, we could add a callback API to allow extension modules to be notified when the locale changes, which may be useful for other reasons... see issue9727). Antoine pointed out file descriptors as another possible problem, but I'm not sure we can prevent that at the API level (there are too many ways for Python code to get at file descriptors - it is likely better to just say that if you get a file descriptor from Python, only use it with Python APIs, not directly with the C runtime). > Then again, I haven't heard of many people actually running into > those possible issues. Even today you can use a different C runtime in a Python extension so long as you avoid the APIs that would cause problems (e.g. I have legacy SWIG-wrapped extensions built with VC6 that run quite happily even when linked against Python 2.4). The idea with the stable ABI is to explicitly exclude those functions so that you *know* you aren't going to run into those problems. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Sep 1 00:33:45 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 1 Sep 2010 08:33:45 +1000 Subject: [Python-Dev] PEP 384 status In-Reply-To: <4C7C81EE.8050303@gmail.com> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7C81EE.8050303@gmail.com> Message-ID: On Tue, Aug 31, 2010 at 2:15 PM, Mark Hammond wrote: > It would be interesting to know how, in practice, these FILE pointers come > to life. ?In my experience they are generally obtained via fopen. If that is > broadly true, then a middle-ground may be for Python to expose something > like Py_fopen, Py_fclose and a PyFILE opaque "handle". ?API elements which > currently take a FILE * could be exposed using a PyFILE * in the ABI. > ?People who didn't care about this level of portability could continue to > use the non-ABI FILE * functions, but people who do could use > Py_fopen/Py_fclose in place of fopen/fclose but otherwise work as now. No need for that - the Python-specific equivalent is our own I/O stack (and things like os.fdopen). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Wed Sep 1 00:42:38 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 1 Sep 2010 00:42:38 +0200 Subject: [Python-Dev] PEP 384 status References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D80EE.8090705@canterbury.ac.nz> Message-ID: <20100901004238.72d050ac@pitrou.net> On Wed, 01 Sep 2010 10:23:42 +1200 Greg Ewing wrote: > Daniel Stutzbach wrote: > > > Likewise, a FILE * isn't safe to pass around, unless I can guarantee > > that the application really is one big happy family compiled against the > > same version of the C library. > > Given that, it seems to me that it's a mistake for Python > to provide any APIs that take a FILE* directly in the > first place. What I think would be a mistake would be to define the API in terms of Windows-specific quirks. All this discussion seems bent on satisfying the needs of Windows developers at the expense of non-Windows developers. "FILE*" is a perfectly standard C feature (and a widely-used one at that). If Windows doesn't handle it correctly then it's a Windows issue and we shouldn't annoy other people by refusing access to that feature. Again, putting a warning in the documentation should be enough for those people who want to have perfect Windows support. And if the Microsoft compiler/linker doesn't bother warning the user when passing FILE* between various CRTs, then it's not Python's job to supplement it. After all, we don't usually try to workaround platform-specific bugs (not as a low level such as the C API level); at worse, we mention their existence in the docs. Regards Antoine. From greg.ewing at canterbury.ac.nz Wed Sep 1 00:48:37 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 01 Sep 2010 10:48:37 +1200 Subject: [Python-Dev] PEP 384 status In-Reply-To: <4C7D78F4.9040900@egenix.com> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D78F4.9040900@egenix.com> Message-ID: <4C7D86C5.7000402@canterbury.ac.nz> M.-A. Lemburg wrote: > But isn't exactly that a major problem for Python ? > > An extension written for a different MS CRT version would > not be able to safely free a buffer allocated by the Python > DLL. Python provides its own set of memory alloc/free functions to deal with this exact problem. For every Python function that allocates something, there's a corresponding function for freeing it, and you do it any other way at your peril. -- Greg From adal.chiriliuc at gmail.com Wed Sep 1 02:28:39 2010 From: adal.chiriliuc at gmail.com (Adal Chiriliuc) Date: Wed, 1 Sep 2010 03:28:39 +0300 Subject: [Python-Dev] PEP 384 status In-Reply-To: <20100901004238.72d050ac@pitrou.net> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D80EE.8090705@canterbury.ac.nz> <20100901004238.72d050ac@pitrou.net> Message-ID: On Wed, Sep 1, 2010 at 1:42 AM, Antoine Pitrou wrote: > What I think would be a mistake would be to define the API in terms of > Windows-specific quirks. All this discussion seems bent on satisfying > the needs of Windows developers at the expense of non-Windows > developers. "FILE*" is a perfectly standard C feature (and a > widely-used one at that). If Windows doesn't handle it correctly then > it's a Windows issue and we shouldn't annoy other people by refusing > access to that feature. As an alternative the stable ABI could use the native system file handle instead of FILE*. HANDLE (returned by CreateFile) on Windows and int (returned by open) on Linux (or even FILE*). The extension writer would have to write it's own abstraction over this. Python could provide a default wrapper (as a sample). Unfortunately you would also lose the buffering provided by fopen (unless the sample also does this). As a side note, I wrote a plugin for a big Windows application (Mastercam) once. There were at least 5 different CRTs running in that process - 3 different release versions and 2 different debug versions. Besides a few other statically linked into other plugin DLLs. This is quite common, especially since you can have foreign CRTs injected into your process by various shell extensions like TortoiseSVN. From eliben at gmail.com Wed Sep 1 05:41:00 2010 From: eliben at gmail.com (Eli Bendersky) Date: Wed, 1 Sep 2010 06:41:00 +0300 Subject: [Python-Dev] Working on the Python code-base with a VIM-based setup Message-ID: Hello pydev, This is a meta-question which I hope is appropriate in this list (**). Recently I've switched to to VIM as my main development platform (in terms of code editing and navigation). Working on the Python code-base is both a concrete requirement and a yardstick for me - I want to be as effective as possible at it. Therefore I would like to ask those of you working on Python's code with VIM about your setups - the special tweaks to VIM & plugins you use to make working with the code as simple and effective as possible. Myself, since I'm still a VIM newbie, my setup is quite spartan. I created tags with: ctags -R Grammar Include Modules/ Objects/ Parser/ Python/ And now happily browse around the source code with Ctrl-], Ctrl-I/O, Ctrl-T and so on. I've also installed the Taglist plugin to show all functions/macros in open buffers - it's sometimes helpful. Other plugins I've found useful ar NERD-commenter (for commenting out chunks of code) and highlight_current_line (for a visual cue of where I am in a file). Besides, I've taken the suggested settings from the .vimrc file in Misc/ to help enforcing PEP-8. I heard that there are more sophisticated tags plugins that also allow one to check where a function is called for, and other intellisens-y stuff, though I'm not sure whether anyone uses it for Python's code. Thanks in advance, Eli (**) Note that it deals with the source code *of Python* (the stuff you download from Python's official SVN), not Python source code. -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phd.pp.ru Wed Sep 1 10:15:02 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Wed, 1 Sep 2010 12:15:02 +0400 Subject: [Python-Dev] Working on the Python code-base with a VIM-based setup In-Reply-To: References: Message-ID: <20100901081502.GB14927@phd.pp.ru> On Wed, Sep 01, 2010 at 06:41:00AM +0300, Eli Bendersky wrote: > I would like to ask those of you working on > Python's code with VIM about your setups - the special tweaks to VIM & > plugins you use to make working with the code as simple and effective as > possible. You have already seen Misc/Vim directory in Python distribution, have you? Shameless plug: look at my .vimrc and plugins: http://phd.pp.ru/Software/dotfiles/vimrc.html http://phd.pp.ru/Software/dotfiles/vim/ Borrow whatever you find useful. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From ncoghlan at gmail.com Wed Sep 1 14:43:34 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 1 Sep 2010 22:43:34 +1000 Subject: [Python-Dev] PEP 384 status In-Reply-To: <20100901004238.72d050ac@pitrou.net> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D80EE.8090705@canterbury.ac.nz> <20100901004238.72d050ac@pitrou.net> Message-ID: On Wed, Sep 1, 2010 at 8:42 AM, Antoine Pitrou wrote: > After all, we don't usually try to workaround platform-specific > bugs (not as a low level such as the C API level); at worse, we mention > their existence in the docs. You persist in viewing the allowance of multiple C runtimes in a single process as a bug instead of a feature. This comes from the monolithic approach to software management in Unix systems (with the associated web of dependencies and implications for backwards compatibility management). The Windows heritage is different, allowing each application to be largely independent of everything other than the OS itself, so the execution model is also different. Windows is a supported platform for Python and to meet the promise of a stable ABI, we need to support multiple C runtimes in a single process. Since we aren't in the habit of passing low level objects around, the number of APIs affected is relatively tiny (the current tally is less than half a dozen that I know of - the two mentioned in this thread, plus a couple of of the PyRun_* APIs). There will be some cases (like file descriptors) where we just say "don't do that" but those should be fairly rare. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Wed Sep 1 14:54:51 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 01 Sep 2010 14:54:51 +0200 Subject: [Python-Dev] PEP 384 status In-Reply-To: References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D80EE.8090705@canterbury.ac.nz> <20100901004238.72d050ac@pitrou.net> Message-ID: <1283345691.3228.11.camel@localhost.localdomain> Le mercredi 01 septembre 2010 ? 22:43 +1000, Nick Coghlan a ?crit : > On Wed, Sep 1, 2010 at 8:42 AM, Antoine Pitrou wrote: > > After all, we don't usually try to workaround platform-specific > > bugs (not as a low level such as the C API level); at worse, we mention > > their existence in the docs. > > You persist in viewing the allowance of multiple C runtimes in a > single process as a bug instead of a feature. No, I view the fact that you can't share FILE structures as a bug. I'm sure there would be ways to have multiple C runtimes loaded in memory with compatible FILE structures (for example, by versioning the FILE structure itself, or by embedding inside the FILE structure a set of function pointers, so that fread(), fwrite() and friends always get redirected to the proper implementation). Please consider this: even without relying on PEP 384, using FILE* is /already/ dangerous; because you might compile an extension with a different compiler version than Python was compiled with. So, if we were following you, we should rip out PyObject_Print() of the whole C API, not only the limited subset which is defined by PEP 384. (now I have nothing against completely ripping out PyObject_Print() if we find out that it's not really useful...) Regards Antoine. From fuzzyman at voidspace.org.uk Wed Sep 1 15:04:12 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 01 Sep 2010 14:04:12 +0100 Subject: [Python-Dev] PEP 384 status In-Reply-To: <1283345691.3228.11.camel@localhost.localdomain> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D80EE.8090705@canterbury.ac.nz> <20100901004238.72d050ac@pitrou.net> <1283345691.3228.11.camel@localhost.localdomain> Message-ID: <4C7E4F4C.6050102@voidspace.org.uk> On 01/09/2010 13:54, Antoine Pitrou wrote: > [snip...] > Please consider this: even without relying on PEP 384, using FILE* > is /already/ dangerous; because you might compile an extension with a > different compiler version than Python was compiled with. So, if we were > following you, we should rip out PyObject_Print() of the whole C API, > not only the limited subset which is defined by PEP 384. Definitely. On Windows it is relatively common to configure distutils to compile extensions with Mingw, which is likely to have problems with FILE* from the Microsoft C runtime [1]. Michael [1] I say *likely* because I have vague memories of Mingw being able to compile against msvcrt, but I'm not sure about this. This is actually less of an issue at the moment as we are now using a Microsoft compiler for which a free version is available. This wasn't the case for a while (and still isn't for those using Python 2.5 and earlier). > (now I have nothing against completely ripping out PyObject_Print() if > we find out that it's not really useful...) > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From fuzzyman at voidspace.org.uk Wed Sep 1 15:26:40 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 01 Sep 2010 14:26:40 +0100 Subject: [Python-Dev] 'hasattr' is broken by design In-Reply-To: <4C737755.30509@voidspace.org.uk> References: <2C284C09-3819-4858-A75A-34597DD668B7@gmail.com> <4C72D552.2030306@voidspace.org.uk> <65FC8A3B-F9AF-4ED9-B63C-41335EF49E1A@gmail.com> <4C72E1F2.2020209@voidspace.org.uk> <4C72E372.5060201@voidspace.org.uk> <4C737755.30509@voidspace.org.uk> Message-ID: <4C7E5490.5000703@voidspace.org.uk> On 24/08/2010 08:40, Michael Foord wrote: > On 24/08/2010 01:25, Nick Coghlan wrote: >> On Tue, Aug 24, 2010 at 8:15 AM, Nick Coghlan wrote: >>> Now, it may be worth considering an addition to the inspect module >>> that was basically: >>> >>> def getattr_static(obj, attr): >>> """Retrieve attributes without triggering dynamic lookup via the >>> descriptor protocol, >>> __getattr__ or __getattribute__. >>> >>> Note: this function may not be able to retrieve all attributes >>> reported by dir(obj) >>> """ >>> try: >>> instance_dict = object.__getattribute__(obj, "__dict__") >>> except AttributeError: >>> pass >>> else: >>> if attr in instance_dict: >>> return instance_dict[attr] >>> for entry in getmro(obj.__class__): >>> try: >>> return entry.__dict__[attr] >>> except AttributeError: >>> pass >> Second attempt with a default value parameter and correctly raising >> AttributeError if not found: >> >> _sentinel = object() >> def getattr_static(obj, attr, default=_sentinel): >> """Retrieve attributes without triggering dynamic lookup via the >> descriptor protocol, __getattr__ or __getattribute__. >> >> Note: this function may not be able to retrieve all attributes >> reported by dir(obj) >> """ >> try: >> instance_dict = object.__getattribute__(obj, "__dict__") >> except AttributeError: >> pass >> else: >> if attr in instance_dict: >> return instance_dict[attr] >> for entry in getmro(obj.__class__): >> try: >> return entry.__dict__[attr] >> except AttributeError: >> pass >> if default is not _sentinel: >> return default >> raise AttributeError(attr) >> > > This doesn't correctly handle the case where obj is a type object or > obj uses __slots__. > > If I have time (currently on vacation with only intermittent internet > access) I'll provide an update. > I managed an updated version (with tests) during the flight home last night. I've attached it to issue 9732, along with a discussion of the caveats and ways of breaking it: http://bugs.python.org/issue9732 Michael > Michael > > >> Cheers, >> Nick. >> > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From fuzzyman at voidspace.org.uk Wed Sep 1 17:34:59 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 1 Sep 2010 16:34:59 +0100 Subject: [Python-Dev] 'hasattr' is broken by design In-Reply-To: <20100825195722.506873A411C@sparrow.telecommunity.com> References: <201008240956.27584.steve@pearwood.info> <201008242151.21940.steve@pearwood.info> <4C745F5A.4030909@canterbury.ac.nz> <20100825162746.078483A40A4@sparrow.telecommunity.com> <4C7559BC.9020807@voidspace.org.uk> <20100825195722.506873A411C@sparrow.telecommunity.com> Message-ID: On 25 August 2010 20:57, P.J. Eby wrote: > [snip...] > As you can see, the __call__ attribute in each case is whatever the proxied > object's __call__ attribute is, even though the proxy itself has a __call__ > method, that is invoked when the proxy is called. > > This is actually pretty straightforward stuff since the introduction of > __getattribute__. > > (The code is at http://pypi.python.org/pypi/ProxyTypes, btw.) > For what it's worth this code is useful enough (and simple enough) that I would support its inclusion in the standard library. I've written proxy objects several times (for various different purposes) and this would have saved me the effort. :-) All the best, Michael Foord -- http://www.voidspace.org.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Sep 1 23:04:31 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 2 Sep 2010 07:04:31 +1000 Subject: [Python-Dev] PEP 384 status In-Reply-To: <1283345691.3228.11.camel@localhost.localdomain> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D80EE.8090705@canterbury.ac.nz> <20100901004238.72d050ac@pitrou.net> <1283345691.3228.11.camel@localhost.localdomain> Message-ID: On Wed, Sep 1, 2010 at 10:54 PM, Antoine Pitrou wrote: > Please consider this: even without relying on PEP 384, using FILE* > is /already/ dangerous; because you might compile an extension with a > different compiler version than Python was compiled with. So, if we were > following you, we should rip out PyObject_Print() of the whole C API, > not only the limited subset which is defined by PEP 384. > > (now I have nothing against completely ripping out PyObject_Print() if > we find out that it's not really useful...) I think it would be better if everything dealing with FILE* was a macro rather than a function, yes. The definition of the limited API is a chance to fix that without incurring the cost in backwards incompatibility that would otherwise arise. Since we have that opportunity, why not take it? Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Wed Sep 1 23:19:22 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 1 Sep 2010 23:19:22 +0200 Subject: [Python-Dev] PEP 384 status In-Reply-To: References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D80EE.8090705@canterbury.ac.nz> <20100901004238.72d050ac@pitrou.net> <1283345691.3228.11.camel@localhost.localdomain> Message-ID: <20100901231922.02e44c24@pitrou.net> On Thu, 2 Sep 2010 07:04:31 +1000 Nick Coghlan wrote: > On Wed, Sep 1, 2010 at 10:54 PM, Antoine Pitrou wrote: > > Please consider this: even without relying on PEP 384, using FILE* > > is /already/ dangerous; because you might compile an extension with a > > different compiler version than Python was compiled with. So, if we were > > following you, we should rip out PyObject_Print() of the whole C API, > > not only the limited subset which is defined by PEP 384. > > > > (now I have nothing against completely ripping out PyObject_Print() if > > we find out that it's not really useful...) > > I think it would be better if everything dealing with FILE* was a > macro rather than a function, yes. The definition of the limited API > is a chance to fix that without incurring the cost in backwards > incompatibility that would otherwise arise. Since we have that > opportunity, why not take it? Maybe I've missed your answer, but what would prevent the "inline" solution from working? (a macro with the result-as-a-pointer is quite ugly) Regards Antoine. From g.brandl at gmx.net Wed Sep 1 23:25:07 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 01 Sep 2010 23:25:07 +0200 Subject: [Python-Dev] r84388 - in python/branches/py3k/Doc: conf.py using/index.rst In-Reply-To: <20100901085716.683C2EE9E8@mail.python.org> References: <20100901085716.683C2EE9E8@mail.python.org> Message-ID: That title isn't better though, since it doesn't cover the "using/cmdline" document which deals with command line options, environment variables and the like. I agree that "Using Python" is not very descriptive though. Georg Am 01.09.2010 10:57, schrieb raymond.hettinger: > Author: raymond.hettinger > Date: Wed Sep 1 10:57:16 2010 > New Revision: 84388 > > Log: > More descriptive title. > > Modified: > python/branches/py3k/Doc/conf.py > python/branches/py3k/Doc/using/index.rst > > Modified: python/branches/py3k/Doc/conf.py > ============================================================================== > --- python/branches/py3k/Doc/conf.py (original) > +++ python/branches/py3k/Doc/conf.py Wed Sep 1 10:57:16 2010 > @@ -131,7 +131,7 @@ > ('tutorial/index', 'tutorial.tex', > 'Python Tutorial', _stdauthor, 'manual'), > ('using/index', 'using.tex', > - 'Using Python', _stdauthor, 'manual'), > + 'Python Setup', _stdauthor, 'manual'), > ('whatsnew/' + version, 'whatsnew.tex', > 'What\'s New in Python', 'A. M. Kuchling', 'howto'), > ] > > Modified: python/branches/py3k/Doc/using/index.rst > ============================================================================== > --- python/branches/py3k/Doc/using/index.rst (original) > +++ python/branches/py3k/Doc/using/index.rst Wed Sep 1 10:57:16 2010 > @@ -1,7 +1,7 @@ > .. _using-index: > > ################ > - Using Python > + Python Setup > ################ -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From ncoghlan at gmail.com Wed Sep 1 23:31:10 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 2 Sep 2010 07:31:10 +1000 Subject: [Python-Dev] PEP 384 status In-Reply-To: <20100901231922.02e44c24@pitrou.net> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D80EE.8090705@canterbury.ac.nz> <20100901004238.72d050ac@pitrou.net> <1283345691.3228.11.camel@localhost.localdomain> <20100901231922.02e44c24@pitrou.net> Message-ID: On Thu, Sep 2, 2010 at 7:19 AM, Antoine Pitrou wrote: > Maybe I've missed your answer, but what would prevent the "inline" > solution from working? Only PEP 7 (since standard support for inline is a C99 feature) On the other hand, if gcc (including cygwin/mingw) and MSVC support it (even with non-standard spellings), then that objection probably doesn't matter. > (a macro with the result-as-a-pointer is quite ugly) Agreed. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Sep 1 23:42:34 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 2 Sep 2010 07:42:34 +1000 Subject: [Python-Dev] [Python-checkins] r84391 - in python/branches/py3k: Include/abstract.h Misc/NEWS Objects/abstract.c Objects/memoryobject.c In-Reply-To: <20100901125825.55BB5EE98A@mail.python.org> References: <20100901125825.55BB5EE98A@mail.python.org> Message-ID: On Wed, Sep 1, 2010 at 10:58 PM, antoine.pitrou wrote: > Modified: python/branches/py3k/Misc/NEWS > ============================================================================== > --- python/branches/py3k/Misc/NEWS ? ? ?(original) > +++ python/branches/py3k/Misc/NEWS ? ? ?Wed Sep ?1 14:58:21 2010 > @@ -354,6 +354,9 @@ > ?Build > ?----- > > +- Issue #3101: Helper functions _add_one_to_C() and _add_one_to_F() become > + ?_Py_add_one_to_C() and _Py_add_one_to_F(), respectively. > + Minor point: the names are actually *_index_C and *_index_F. (the patch is right, just NEWS and the checkin comment are off) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Sep 1 23:43:24 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 2 Sep 2010 07:43:24 +1000 Subject: [Python-Dev] r84388 - in python/branches/py3k/Doc: conf.py using/index.rst In-Reply-To: References: <20100901085716.683C2EE9E8@mail.python.org> Message-ID: On Thu, Sep 2, 2010 at 7:25 AM, Georg Brandl wrote: > That title isn't better though, since it doesn't cover the "using/cmdline" > document which deals with command line options, environment variables > and the like. > > I agree that "Using Python" is not very descriptive though. Python Setup and Usage? Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From g.brandl at gmx.net Thu Sep 2 10:47:45 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 02 Sep 2010 10:47:45 +0200 Subject: [Python-Dev] r84388 - in python/branches/py3k/Doc: conf.py using/index.rst In-Reply-To: References: <20100901085716.683C2EE9E8@mail.python.org> Message-ID: Am 01.09.2010 23:43, schrieb Nick Coghlan: > On Thu, Sep 2, 2010 at 7:25 AM, Georg Brandl wrote: >> That title isn't better though, since it doesn't cover the "using/cmdline" >> document which deals with command line options, environment variables >> and the like. >> >> I agree that "Using Python" is not very descriptive though. > > Python Setup and Usage? I'll change to that unless something better comes up :) Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From g.brandl at gmx.net Thu Sep 2 10:51:15 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 02 Sep 2010 10:51:15 +0200 Subject: [Python-Dev] r84430 - in python/branches/py3k: Include/unicodeobject.h Objects/unicodeobject.c In-Reply-To: <20100901234353.468A5EEAAC@mail.python.org> References: <20100901234353.468A5EEAAC@mail.python.org> Message-ID: Hi Victor, 1. This function and PyUnicode_strcat are missing documentation. 2. Are you sure they need to be public APIs? What are they going to be used for? (I'm not sure myself, but I think we usually have a short notice here when new C APIs are added.) Georg Am 02.09.2010 01:43, schrieb victor.stinner: > Author: victor.stinner > Date: Thu Sep 2 01:43:53 2010 > New Revision: 84430 > > Log: > Create PyUnicode_strdup() function > > Modified: > python/branches/py3k/Include/unicodeobject.h > python/branches/py3k/Objects/unicodeobject.c > > Modified: python/branches/py3k/Include/unicodeobject.h > ============================================================================== > --- python/branches/py3k/Include/unicodeobject.h (original) > +++ python/branches/py3k/Include/unicodeobject.h Thu Sep 2 01:43:53 2010 > @@ -220,6 +220,7 @@ > # define _PyUnicode_AsDefaultEncodedString _PyUnicodeUCS2_AsDefaultEncodedString > # define _PyUnicode_Fini _PyUnicodeUCS2_Fini > # define _PyUnicode_Init _PyUnicodeUCS2_Init > +# define PyUnicode_strdup PyUnicodeUCS2_strdup > > #else > > @@ -302,7 +303,7 @@ > # define _PyUnicode_AsDefaultEncodedString _PyUnicodeUCS4_AsDefaultEncodedString > # define _PyUnicode_Fini _PyUnicodeUCS4_Fini > # define _PyUnicode_Init _PyUnicodeUCS4_Init > - > +# define PyUnicode_strdup PyUnicodeUCS4_strdup > > #endif > > @@ -1602,6 +1603,14 @@ > Py_UNICODE c > ); > > +/* Create a copy of a unicode string ending with a nul character. Return NULL > + and raise a MemoryError exception on memory allocation failure, otherwise > + return a new allocated buffer (use PyMem_Free() to free the buffer). */ > + > +PyAPI_FUNC(Py_UNICODE*) PyUnicode_strdup( > + PyObject *unicode > + ); > + > #ifdef __cplusplus > } > #endif > > Modified: python/branches/py3k/Objects/unicodeobject.c > ============================================================================== > --- python/branches/py3k/Objects/unicodeobject.c (original) > +++ python/branches/py3k/Objects/unicodeobject.c Thu Sep 2 01:43:53 2010 > @@ -10014,6 +10014,28 @@ > return NULL; > } > > +Py_UNICODE* > +PyUnicode_strdup(PyObject *object) > +{ > + PyUnicodeObject *unicode = (PyUnicodeObject *)object; > + Py_UNICODE *copy; > + Py_ssize_t size; > + > + /* Ensure we won't overflow the size. */ > + if (PyUnicode_GET_SIZE(unicode) > ((PY_SSIZE_T_MAX / sizeof(Py_UNICODE)) - 1)) { > + PyErr_NoMemory(); > + return NULL; > + } > + size = PyUnicode_GET_SIZE(unicode) + 1; /* copy the nul character */ > + size *= sizeof(Py_UNICODE); > + copy = PyMem_Malloc(size); > + if (copy == NULL) { > + PyErr_NoMemory(); > + return NULL; > + } > + memcpy(copy, PyUnicode_AS_UNICODE(unicode), size); > + return copy; > +} > > #ifdef __cplusplus > } -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From mal at egenix.com Thu Sep 2 11:13:22 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 02 Sep 2010 11:13:22 +0200 Subject: [Python-Dev] r84430 - in python/branches/py3k: Include/unicodeobject.h Objects/unicodeobject.c In-Reply-To: References: <20100901234353.468A5EEAAC@mail.python.org> Message-ID: <4C7F6AB2.5090606@egenix.com> Georg Brandl wrote: > Hi Victor, > > 1. This function and PyUnicode_strcat are missing documentation. > > 2. Are you sure they need to be public APIs? What are they going to > be used for? (I'm not sure myself, but I think we usually have a > short notice here when new C APIs are added.) If you want to make this a public API function, it also needs to be renamed to fit the rest of the API. I'd suggest PyUnicode_AsUnicodeCopy() which then corresponds to PyUnicode_FromUnicode(), but I'm not sure whether we should have such a public API in the first place. > Georg > > Am 02.09.2010 01:43, schrieb victor.stinner: >> Author: victor.stinner >> Date: Thu Sep 2 01:43:53 2010 >> New Revision: 84430 >> >> Log: >> Create PyUnicode_strdup() function >> >> Modified: >> python/branches/py3k/Include/unicodeobject.h >> python/branches/py3k/Objects/unicodeobject.c >> >> Modified: python/branches/py3k/Include/unicodeobject.h >> ============================================================================== >> --- python/branches/py3k/Include/unicodeobject.h (original) >> +++ python/branches/py3k/Include/unicodeobject.h Thu Sep 2 01:43:53 2010 >> @@ -220,6 +220,7 @@ >> # define _PyUnicode_AsDefaultEncodedString _PyUnicodeUCS2_AsDefaultEncodedString >> # define _PyUnicode_Fini _PyUnicodeUCS2_Fini >> # define _PyUnicode_Init _PyUnicodeUCS2_Init >> +# define PyUnicode_strdup PyUnicodeUCS2_strdup >> >> #else >> >> @@ -302,7 +303,7 @@ >> # define _PyUnicode_AsDefaultEncodedString _PyUnicodeUCS4_AsDefaultEncodedString >> # define _PyUnicode_Fini _PyUnicodeUCS4_Fini >> # define _PyUnicode_Init _PyUnicodeUCS4_Init >> - >> +# define PyUnicode_strdup PyUnicodeUCS4_strdup >> >> #endif >> >> @@ -1602,6 +1603,14 @@ >> Py_UNICODE c >> ); >> >> +/* Create a copy of a unicode string ending with a nul character. Return NULL >> + and raise a MemoryError exception on memory allocation failure, otherwise >> + return a new allocated buffer (use PyMem_Free() to free the buffer). */ >> + >> +PyAPI_FUNC(Py_UNICODE*) PyUnicode_strdup( >> + PyObject *unicode >> + ); >> + >> #ifdef __cplusplus >> } >> #endif >> >> Modified: python/branches/py3k/Objects/unicodeobject.c >> ============================================================================== >> --- python/branches/py3k/Objects/unicodeobject.c (original) >> +++ python/branches/py3k/Objects/unicodeobject.c Thu Sep 2 01:43:53 2010 >> @@ -10014,6 +10014,28 @@ >> return NULL; >> } >> >> +Py_UNICODE* >> +PyUnicode_strdup(PyObject *object) >> +{ >> + PyUnicodeObject *unicode = (PyUnicodeObject *)object; >> + Py_UNICODE *copy; >> + Py_ssize_t size; >> + >> + /* Ensure we won't overflow the size. */ >> + if (PyUnicode_GET_SIZE(unicode) > ((PY_SSIZE_T_MAX / sizeof(Py_UNICODE)) - 1)) { >> + PyErr_NoMemory(); >> + return NULL; >> + } >> + size = PyUnicode_GET_SIZE(unicode) + 1; /* copy the nul character */ >> + size *= sizeof(Py_UNICODE); >> + copy = PyMem_Malloc(size); >> + if (copy == NULL) { >> + PyErr_NoMemory(); >> + return NULL; >> + } >> + memcpy(copy, PyUnicode_AS_UNICODE(unicode), size); >> + return copy; >> +} >> >> #ifdef __cplusplus >> } > > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 02 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2010-08-19: Released mxODBC 3.1.0 http://python.egenix.com/ 2010-09-15: DZUG Tagung, Dresden, Germany 12 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Thu Sep 2 22:35:09 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 2 Sep 2010 22:35:09 +0200 Subject: [Python-Dev] Buffer protocol for io.BytesIO? Message-ID: <20100902223509.64e62db3@pitrou.net> Hello all, In issue #5506, I originally proposed that io.BytesIO objects support the buffer protocol, to make it possible to access the internal buffer without intermediate copies. Then it came to me then perhaps it would be too automatic. So I'm currently floating between: - add implicit buffer protocol support to BytesIO objects - add explicit buffer protocol support through the call of a getbuffer() method, which would return a small intermediate object supporting the buffer protocol on behalf of the original BytesIO object What do you think would be better? Thank you Antoine. From ncoghlan at gmail.com Thu Sep 2 23:19:04 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 Sep 2010 07:19:04 +1000 Subject: [Python-Dev] r84430 - in python/branches/py3k: Include/unicodeobject.h Objects/unicodeobject.c In-Reply-To: References: <20100901234353.468A5EEAAC@mail.python.org> Message-ID: On Thu, Sep 2, 2010 at 6:51 PM, Georg Brandl wrote: > Hi Victor, > > 1. This function and PyUnicode_strcat are missing documentation. > > 2. Are you sure they need to be public APIs? ?What are they going to > ? be used for? ?(I'm not sure myself, but I think we usually have a > ? short notice here when new C APIs are added.) I believe I first saw them on Victor's branch for improved Unicode support (especially in the context of import). I agree that private _Py prefixes are probably a better idea here. The public API should focus on Py_Unicode, with only a minimal collection of Py_UNICODE interfaces. Regarding the naming scheme, if these are what I recall them as (drop in replacements for the 8-bit C stdlib equivalents to simplify the relevant import.c updates), I'm fine with the current naming for private API functions, but agree with MAL they should be made more explicit if they're part of the published API. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Thu Sep 2 23:22:31 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 Sep 2010 07:22:31 +1000 Subject: [Python-Dev] Buffer protocol for io.BytesIO? In-Reply-To: <20100902223509.64e62db3@pitrou.net> References: <20100902223509.64e62db3@pitrou.net> Message-ID: On Fri, Sep 3, 2010 at 6:35 AM, Antoine Pitrou wrote: > > Hello all, > > In issue #5506, I originally proposed that io.BytesIO objects support > the buffer protocol, to make it possible to access the internal buffer > without intermediate copies. > > Then it came to me then perhaps it would be too automatic. So I'm > currently floating between: > - add implicit buffer protocol support to BytesIO objects > - add explicit buffer protocol support through the call of a > ?getbuffer() method, which would return a small intermediate object > ?supporting the buffer protocol on behalf of the original BytesIO > ?object > > What do you think would be better? The latter offers an easy future upgrade path if we decide that implicit would be better (i.e. change getbuffer() to just return self). If we go straight to implicit there's no easy way back to the explicit version. I think getbuffer() also parallels getvalue() quite well. Call it +1 for explicit and -0 for implicit. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Thu Sep 2 23:48:35 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 Sep 2010 07:48:35 +1000 Subject: [Python-Dev] PEP 384 status In-Reply-To: <1283345691.3228.11.camel@localhost.localdomain> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D80EE.8090705@canterbury.ac.nz> <20100901004238.72d050ac@pitrou.net> <1283345691.3228.11.camel@localhost.localdomain> Message-ID: On Wed, Sep 1, 2010 at 10:54 PM, Antoine Pitrou wrote: > Please consider this: even without relying on PEP 384, using FILE* > is /already/ dangerous; because you might compile an extension with a > different compiler version than Python was compiled with. So, if we were > following you, we should rip out PyObject_Print() of the whole C API, > not only the limited subset which is defined by PEP 384. > > (now I have nothing against completely ripping out PyObject_Print() if > we find out that it's not really useful...) I finally took the obvious step of grepping the include directory to see what APIs were even affected by this question. Turns out there are more APIs than I thought, but most extension modules aren't going to need most of them (since they're related to code parsing, compilation and execution directly from files). For the remainder, I propose as a starting point that users of the stable ABI be directed to the Python equivalents, with the possibility of later adding inline functions for more commonly used API elements. PyObject_Print is the only one I would bother providing as part of the initial stable ABI implementation. Cheers, Nick. Search criteria: grep "FILE \?\*" * object.h:typedef int (*printfunc)(PyObject *, FILE *, int); object.h:PyAPI_FUNC(int) PyObject_Print(PyObject *, FILE *, int); - should provide inline equivalent fileobject.h:PyAPI_FUNC(char *) Py_UniversalNewlineFgets(char *, int, FILE*, PyObject *); marshal.h:PyAPI_FUNC(void) PyMarshal_WriteLongToFile(long, FILE *, int); marshal.h:PyAPI_FUNC(void) PyMarshal_WriteObjectToFile(PyObject *, FILE *, int); marshal.h:PyAPI_FUNC(long) PyMarshal_ReadLongFromFile(FILE *); marshal.h:PyAPI_FUNC(int) PyMarshal_ReadShortFromFile(FILE *); marshal.h:PyAPI_FUNC(PyObject *) PyMarshal_ReadObjectFromFile(FILE *); marshal.h:PyAPI_FUNC(PyObject *) PyMarshal_ReadLastObjectFromFile(FILE *); parsetok.h:PyAPI_FUNC(node *) PyParser_ParseFile (FILE *, const char *, grammar *, int, parsetok.h:PyAPI_FUNC(node *) PyParser_ParseFileFlags(FILE *, const char *, parsetok.h:PyAPI_FUNC(node *) PyParser_ParseFileFlagsEx(FILE *, const char *, pythonrun.h:PyAPI_FUNC(int) PyRun_AnyFileFlags(FILE *, const char *, PyCompilerFlags *); pythonrun.h:PyAPI_FUNC(int) PyRun_AnyFileExFlags(FILE *, const char *, int, PyCompilerFlags *); pythonrun.h:PyAPI_FUNC(int) PyRun_SimpleFileExFlags(FILE *, const char *, int, PyCompilerFlags *); pythonrun.h:PyAPI_FUNC(int) PyRun_InteractiveOneFlags(FILE *, const char *, PyCompilerFlags *); pythonrun.h:PyAPI_FUNC(int) PyRun_InteractiveLoopFlags(FILE *, const char *, PyCompilerFlags *); pythonrun.h:PyAPI_FUNC(struct _mod *) PyParser_ASTFromFile(FILE *, const char *, pythonrun.h:PyAPI_FUNC(struct _node *) PyParser_SimpleParseFileFlags(FILE *, const char *, pythonrun.h:PyAPI_FUNC(PyObject *) PyRun_FileExFlags(FILE *, const char *, int, pythonrun.h:PyAPI_FUNC(int) Py_FdIsInteractive(FILE *, const char *); pythonrun.h:PyAPI_FUNC(char *) PyOS_Readline(FILE *, FILE *, char *); pythonrun.h:PyAPI_DATA(char) *(*PyOS_ReadlineFunctionPointer)(FILE *, FILE *, char *); - omit from stable ABI, recommend equivalent Python calls (may add inline versions to stable ABI over time if requested by developers) abstract.h: int PyObject_Print(PyObject *o, FILE *fp, int flags); - comment, not declaration (actual declaration is in object.h) grammar.h:void printgrammar(grammar *g, FILE *fp); grammar.h:void printnonterminals(grammar *g, FILE *fp); object.h:PyAPI_FUNC(void) _Py_PrintReferences(FILE *); object.h:PyAPI_FUNC(void) _Py_PrintReferenceAddresses(FILE *); Python.h:PyAPI_FUNC(FILE *) _Py_wfopen(const wchar_t *path, const wchar_t *mode); Python.h:PyAPI_FUNC(FILE*) _Py_fopen(PyObject *unicode, const char *mode); - private, omit from stable ABI -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From victor.stinner at haypocalc.com Fri Sep 3 01:26:32 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 3 Sep 2010 01:26:32 +0200 Subject: [Python-Dev] =?iso-8859-1?q?r84430_-_in_python/branches/py3k=3A?= =?iso-8859-1?q?=09Include/unicodeobject=2Eh_Objects/unicodeobject=2Ec?= In-Reply-To: <4C7F6AB2.5090606@egenix.com> References: <20100901234353.468A5EEAAC@mail.python.org> <4C7F6AB2.5090606@egenix.com> Message-ID: <201009030126.32665.victor.stinner@haypocalc.com> Hi, Le jeudi 02 septembre 2010 11:13:22, vous avez ?crit : > > 1. This function and PyUnicode_strcat are missing documentation. It's Py_UNICODE_strcat(), not PyUnicode_strcat(). But yes, Py_UNICODE_strcat() is not documented. But I didn't found any doc for other Py_UNICODE_str*() functions in Doc/c-api/*.rst. Does it mean that we should write a document for all these functions? They mimic the str*() C API. If PyUnicode_strdup() is kept public, it should be documented, yes. -- > If you want to make this a public API function, it also needs to be > renamed to fit the rest of the API. PyUnicode_strdup(): I don't like this name. In my first patch, it was called Py_UNICODE_strdup() but all Py_UNICODE_*() functions take Py_UNICODE* arguments, no PyUnicodeObject* argument. > I'd suggest PyUnicode_AsUnicodeCopy() which then corresponds > to PyUnicode_FromUnicode(), but I'm not sure whether we should > have such a public API in the first place. PyUnicode_AsUnicodeCopy() is maybe a better name, I like it. -- > What are they going to be used for? I am not able to merge my huge work on unicode import (see #8611, #9425 and the "import_unicode" svn branch) at once, so I'm splitting it into atomic commits. PyUnicode_strdup() and Py_UNICODE_strcat() are not used yet, but I plan to use them for the import machinery (Python/import.c) to replace char* by Py_UNICODE*. > > 2. Are you sure they need to be public APIs? About Py_UNICODE_strcat(): yes, why not? Other Py_UNICODE_str*() functions are pratical but also required to manipulate Py_UNICODE* strings. About PyUnicode_strdup() (PyUnicode_AsUnicodeCopy): I don't know. It is possible to rewrite it in few lines. Why don't you want to add them to the public API? For my work, it doesn't matter if it's public or not. This function uses PyMem_xxx API, I don't know if a third part library would like to rely on PyMem_xxx. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Fri Sep 3 01:31:13 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 3 Sep 2010 01:31:13 +0200 Subject: [Python-Dev] r84430 - in python/branches/py3k: Include/unicodeobject.h Objects/unicodeobject.c In-Reply-To: References: <20100901234353.468A5EEAAC@mail.python.org> Message-ID: <201009030131.13806.victor.stinner@haypocalc.com> Le jeudi 02 septembre 2010 23:19:04, Nick Coghlan a ?crit : > On Thu, Sep 2, 2010 at 6:51 PM, Georg Brandl wrote: > > Hi Victor, > > > > 1. This function and PyUnicode_strcat are missing documentation. > > > > 2. Are you sure they need to be public APIs? What are they going to > > be used for? (I'm not sure myself, but I think we usually have a > > short notice here when new C APIs are added.) > > I believe I first saw them on Victor's branch for improved Unicode > support (especially in the context of import). Yes, it is used in my branch. Py_UNICODE_strcat() is used in make_compiled_pathname(). Py_UNICODE_strdup() is used in make_source_pathname() and make_compiled_pathname(). > I agree that private _Py prefixes are probably a better idea here. The > public API should focus on Py_Unicode, with only a minimal collection > of Py_UNICODE interfaces. Are you talking about Py_UNICODE_strcat() or Py_UNICODE_strdup() (called PyUnicode_strdup in py3k branch)? -- Victor Stinner http://www.haypocalc.com/ From greg.ewing at canterbury.ac.nz Fri Sep 3 04:44:36 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 03 Sep 2010 14:44:36 +1200 Subject: [Python-Dev] PEP 384 status In-Reply-To: References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D80EE.8090705@canterbury.ac.nz> <20100901004238.72d050ac@pitrou.net> <1283345691.3228.11.camel@localhost.localdomain> Message-ID: <4C806114.5030004@canterbury.ac.nz> On 02/09/10 09:04, Nick Coghlan wrote: > I think it would be better if everything dealing with FILE* was a > macro rather than a function, yes. How would that help? -- Greg From raymond.hettinger at gmail.com Fri Sep 3 06:08:58 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Thu, 2 Sep 2010 21:08:58 -0700 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <4BDB34CE.9080804@v.loewis.de> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> Message-ID: <37D6B5E2-2DA4-47C4-B773-11CB6F98162E@gmail.com> On Apr 30, 2010, at 12:51 PM, Martin v. L?wis wrote: > Without a BDFL, I think we need a committee to make decisions, e.g. by > majority vote amongst committers. I like Guido's idea. Just appoint have one of the experienced developers who is independent of the proposal and have them be the final arbiter. For example, Guido had earlier suggested that I decide the fate of the "yield from" proposal because I had experience in the topic but was not not personally involved in the proposal. Guido has set a good example for others to follow: * let a conversation evolve until an outcome is self-evident * or kill it early if it has no chance * or if discussion teases out all of the meaningful thinking but doesn't reach a clear conclusion, just make a choice based on instinct * have biases toward real-world use cases, towards ideas proven in other languages (category killers), towards slow rates of language evolution, and think about the long-term. It is better to have one experienced developer decide than to have a committee. Raymond From guido at python.org Fri Sep 3 06:23:15 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 2 Sep 2010 21:23:15 -0700 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <37D6B5E2-2DA4-47C4-B773-11CB6F98162E@gmail.com> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <37D6B5E2-2DA4-47C4-B773-11CB6F98162E@gmail.com> Message-ID: On Thu, Sep 2, 2010 at 9:08 PM, Raymond Hettinger wrote: > > On Apr 30, 2010, at 12:51 PM, Martin v. L?wis wrote: >> Without a BDFL, I think we need a committee to make decisions, e.g. by >> majority vote amongst committers. > > I like Guido's idea. ?Just appoint have one of the experienced developers > who is independent of the proposal and have them be the final arbiter. > For example, Guido had earlier suggested that I decide the fate of the > "yield from" proposal because I had experience in the topic but was not > not personally involved in the proposal. > > Guido has set a good example for others to follow: > * let a conversation evolve until an outcome is self-evident > * or kill it early if it has no chance > * or if discussion teases out all of the meaningful thinking > ? but doesn't reach a clear conclusion, just make a choice > ? based on instinct > * have biases toward real-world use cases, towards ideas proven in > ? other languages (category killers), ?towards slow rates of language > ? evolution, and think about the long-term. > > It is better to have one experienced developer decide than to have > a committee. +1. BTW, Barry just asked me about PEP 3149 and we decided to leave the ultimate decision to Georg and Benjamin. That's about as large a committee I'd be comfortable to appoint for any specific PEP. -- --Guido van Rossum (python.org/~guido) From chef at ghum.de Fri Sep 3 07:28:31 2010 From: chef at ghum.de (Massa, Harald Armin) Date: Fri, 3 Sep 2010 07:28:31 +0200 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <37D6B5E2-2DA4-47C4-B773-11CB6F98162E@gmail.com> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <37D6B5E2-2DA4-47C4-B773-11CB6F98162E@gmail.com> Message-ID: >> Without a BDFL, I think we need a committee to make decisions, e.g. by >> majority vote amongst committers. > > It is better to have one experienced developer decide than to have > a committee. I feel that the concept of a BDFM (benevolent dictator for the moment) has the advantage of a clear visible responsibility, which most times leads to better decisions than if nobody is sure which ammunituion is the real one while shooting in a committee. Harald (tldr:+1) -- GHUM Harald Massa persuadere et programmare Harald Armin Massa Spielberger Stra?e 49 70435 Stuttgart 0173/9409607 no fx, no carrier pigeon - Using PostgreSQL is mostly about sleeping well at night. From g.brandl at gmx.net Fri Sep 3 09:32:10 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 03 Sep 2010 09:32:10 +0200 Subject: [Python-Dev] r84430 - in python/branches/py3k: Include/unicodeobject.h Objects/unicodeobject.c In-Reply-To: <201009030126.32665.victor.stinner@haypocalc.com> References: <20100901234353.468A5EEAAC@mail.python.org> <4C7F6AB2.5090606@egenix.com> <201009030126.32665.victor.stinner@haypocalc.com> Message-ID: Am 03.09.2010 01:26, schrieb Victor Stinner: > Hi, > > Le jeudi 02 septembre 2010 11:13:22, vous avez ?crit : >> > 1. This function and PyUnicode_strcat are missing documentation. > > It's Py_UNICODE_strcat(), not PyUnicode_strcat(). But yes, Py_UNICODE_strcat() > is not documented. But I didn't found any doc for other Py_UNICODE_str*() > functions in Doc/c-api/*.rst. Does it mean that we should write a document for > all these functions? They mimic the str*() C API. Yes. All public functions should be documented. I know that some existing functions are not; that should not be a guideline for new functions though. > If PyUnicode_strdup() is kept public, it should be documented, yes. > >> If you want to make this a public API function, it also needs to be >> renamed to fit the rest of the API. > > PyUnicode_strdup(): I don't like this name. In my first patch, it was called > Py_UNICODE_strdup() but all Py_UNICODE_*() functions take Py_UNICODE* > arguments, no PyUnicodeObject* argument. I'll leave the naming decisions to Marc-Andre. >> What are they going to be used for? > > I am not able to merge my huge work on unicode import (see #8611, #9425 and > the "import_unicode" svn branch) at once, so I'm splitting it into atomic > commits. PyUnicode_strdup() and Py_UNICODE_strcat() are not used yet, but I > plan to use them for the import machinery (Python/import.c) to replace char* > by Py_UNICODE*. > >> > 2. Are you sure they need to be public APIs? > > About Py_UNICODE_strcat(): yes, why not? Other Py_UNICODE_str*() functions are > pratical but also required to manipulate Py_UNICODE* strings. For public APIs, the question is not "why not", but "why". If they are needed for the import machinery in Unicode, sure they are useful, but that's no reason to make them public. I'd recommend to start out with _Py prefixes, and avoid any new public APIs unless they are asked for. > About PyUnicode_strdup() (PyUnicode_AsUnicodeCopy): I don't know. It is > possible to rewrite it in few lines. Why don't you want to add them to the > public API? For my work, it doesn't matter if it's public or not. This > function uses PyMem_xxx API, I don't know if a third part library would like > to rely on PyMem_xxx. cheers, Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From g.brandl at gmx.net Fri Sep 3 09:36:22 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 03 Sep 2010 09:36:22 +0200 Subject: [Python-Dev] PEP 3149 accepted Message-ID: This is to let you all know that PEP 3149 is accepted. Benjamin and I decided that on the basis that * strong precedent is set with PEP 3147 * it is not mutually exclusive with PEP 384; should PEP 384 become widely accepted, its use can fade out again * it is a strictly optional feature Barry promised the code will go in before this weekend's alpha2, so that distributors can subject the new code to proper testing, should they choose to use PEP 3149 features. cheers, Georg From hrvoje.niksic at avl.com Fri Sep 3 09:51:11 2010 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Fri, 03 Sep 2010 09:51:11 +0200 Subject: [Python-Dev] Buffer protocol for io.BytesIO? In-Reply-To: <20100902223509.64e62db3@pitrou.net> References: <20100902223509.64e62db3@pitrou.net> Message-ID: <4C80A8EF.5080204@avl.com> On 09/02/2010 10:35 PM, Antoine Pitrou wrote: > Then it came to me then perhaps it would be too automatic. So I'm > currently floating between: > - add implicit buffer protocol support to BytesIO objects > - add explicit buffer protocol support through the call of a > getbuffer() method, which would return a small intermediate object > supporting the buffer protocol on behalf of the original BytesIO > object > > What do you think would be better? getbuffer() sounds better. Accessing buffer contents is a nice feature, but one shouldn't be able to do it by accident. From mal at egenix.com Fri Sep 3 10:01:12 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 03 Sep 2010 10:01:12 +0200 Subject: [Python-Dev] r84430 - in python/branches/py3k: Include/unicodeobject.h Objects/unicodeobject.c In-Reply-To: <201009030126.32665.victor.stinner@haypocalc.com> References: <20100901234353.468A5EEAAC@mail.python.org> <4C7F6AB2.5090606@egenix.com> <201009030126.32665.victor.stinner@haypocalc.com> Message-ID: <4C80AB48.7040100@egenix.com> Victor Stinner wrote: > Hi, > > Le jeudi 02 septembre 2010 11:13:22, vous avez ?crit : >>> 1. This function and PyUnicode_strcat are missing documentation. > > It's Py_UNICODE_strcat(), not PyUnicode_strcat(). But yes, Py_UNICODE_strcat() > is not documented. But I didn't found any doc for other Py_UNICODE_str*() > functions in Doc/c-api/*.rst. Does it mean that we should write a document for > all these functions? They mimic the str*() C API. > > If PyUnicode_strdup() is kept public, it should be documented, yes. As for naming, I think Py_UNICODE_strdup() is fine, since these functions work directly on Py_UNICODE buffers and they mimic the C functions. They should be documented, though, even if the documentation is terse and refers to the C API of a similar name. > -- > >> If you want to make this a public API function, it also needs to be >> renamed to fit the rest of the API. > > PyUnicode_strdup(): I don't like this name. In my first patch, it was called > Py_UNICODE_strdup() but all Py_UNICODE_*() functions take Py_UNICODE* > arguments, no PyUnicodeObject* argument. See above: the current naming is fine. >> I'd suggest PyUnicode_AsUnicodeCopy() which then corresponds >> to PyUnicode_FromUnicode(), but I'm not sure whether we should >> have such a public API in the first place. > > PyUnicode_AsUnicodeCopy() is maybe a better name, I like it. For APIs that work on PyUnicodeObjects, we should stick to the naming convention used by the Unicode and string/bytes API. It's a little more verbose, but also provides more clarity than the C API function names. > -- > >> What are they going to be used for? > > I am not able to merge my huge work on unicode import (see #8611, #9425 and > the "import_unicode" svn branch) at once, so I'm splitting it into atomic > commits. PyUnicode_strdup() and Py_UNICODE_strcat() are not used yet, but I > plan to use them for the import machinery (Python/import.c) to replace char* > by Py_UNICODE*. > >>> 2. Are you sure they need to be public APIs? > > About Py_UNICODE_strcat(): yes, why not? Other Py_UNICODE_str*() functions are > pratical but also required to manipulate Py_UNICODE* strings. > > About PyUnicode_strdup() (PyUnicode_AsUnicodeCopy): I don't know. It is > possible to rewrite it in few lines. Why don't you want to add them to the > public API? For my work, it doesn't matter if it's public or not. This > function uses PyMem_xxx API, I don't know if a third part library would like > to rely on PyMem_xxx. This will have to be documented (see PEP 384 for some reasoning on malloc() use in DLLs). Other than that, ok, let's have them. They do provide some use to other tools working on Py_UNICODE, just like the macros we have for these. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 03 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2010-08-19: Released mxODBC 3.1.0 http://python.egenix.com/ 2010-09-15: DZUG Tagung, Dresden, Germany 11 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From stefan_ml at behnel.de Fri Sep 3 10:04:12 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Sep 2010 10:04:12 +0200 Subject: [Python-Dev] Buffer protocol for io.BytesIO? In-Reply-To: <20100902223509.64e62db3@pitrou.net> References: <20100902223509.64e62db3@pitrou.net> Message-ID: Antoine Pitrou, 02.09.2010 22:35: > In issue #5506, I originally proposed that io.BytesIO objects support > the buffer protocol, to make it possible to access the internal buffer > without intermediate copies. > > Then it came to me then perhaps it would be too automatic. So I'm > currently floating between: > - add implicit buffer protocol support to BytesIO objects Hmm, given that you actually have to *explicitly* call the getbuffer operation on the BytesIO object, I don't have the impression that this would be *implicit* support. What would be the advantage of having to do two operations? Is it just that you want to make the internal buffer joining step more visible? (thinking in terms of good-old Py2 StringIO here, not sure how io.BytesIO works these days) Also, note that the BytesIO object must not be modified while someone reads the buffer. That would be a less obvious side effect if the returned buffer object was different from the BytesIO object itself. Stefan From solipsis at pitrou.net Fri Sep 3 10:54:23 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 3 Sep 2010 10:54:23 +0200 Subject: [Python-Dev] Buffer protocol for io.BytesIO? References: <20100902223509.64e62db3@pitrou.net> Message-ID: <20100903105423.471d7382@pitrou.net> On Fri, 03 Sep 2010 10:04:12 +0200 Stefan Behnel wrote: > Antoine Pitrou, 02.09.2010 22:35: > > In issue #5506, I originally proposed that io.BytesIO objects support > > the buffer protocol, to make it possible to access the internal buffer > > without intermediate copies. > > > > Then it came to me then perhaps it would be too automatic. So I'm > > currently floating between: > > - add implicit buffer protocol support to BytesIO objects > > Hmm, given that you actually have to *explicitly* call the getbuffer > operation on the BytesIO object, I don't have the impression that this > would be *implicit* support. That's the other alternative, precisely. > What would be the advantage of having to do two operations? Is it just that > you want to make the internal buffer joining step more visible? I was just wondering if it's a good thing to let people, e.g., concatenate a bytes object and a BytesIO by using '+'. > Also, note that the BytesIO object must not be modified while someone reads > the buffer. That would be a less obvious side effect if the returned buffer > object was different from the BytesIO object itself. It could not be resized, but it could be modified (same as what happens with bytearrays today). Actually, the buffer itself would be writable, and allow modifying the BytesIO contents. Regards Antoine. From daniel at stutzbachenterprises.com Fri Sep 3 11:20:04 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Fri, 3 Sep 2010 04:20:04 -0500 Subject: [Python-Dev] r84430 - in python/branches/py3k: Include/unicodeobject.h Objects/unicodeobject.c In-Reply-To: <201009030126.32665.victor.stinner@haypocalc.com> References: <20100901234353.468A5EEAAC@mail.python.org> <4C7F6AB2.5090606@egenix.com> <201009030126.32665.victor.stinner@haypocalc.com> Message-ID: On Thu, Sep 2, 2010 at 6:26 PM, Victor Stinner wrote: > But I didn't found any doc for other Py_UNICODE_str*() > functions in Doc/c-api/*.rst. > http://bugs.python.org/issue8649 - Py_UNICODE_* functions are undocumented -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Fri Sep 3 11:37:03 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 3 Sep 2010 12:37:03 +0300 Subject: [Python-Dev] Internal counter to debug leaking file descriptors In-Reply-To: References: Message-ID: On Tue, Aug 31, 2010 at 5:03 PM, Guido van Rossum wrote: > If you wanted to do something like this in the Python stdlib, you'd > have to monkey-patch (with a proxy/wrapper) all places that can open > or close a filedescriptor -- os.open, os.popen, os.close, file > open/close, socket open/close, and probably a bunch more that I've > forgotten. Also some extension modules may open file descriptors > directly through the C interfaces. Actually I monkey-patched fdopen and open, but it appeared that it is not enough. Extensions cause the biggest problem. How can I monkey-patch opening of file descriptor deep inside mmap module? How can I know that a file descriptor is opened there at all? I thought that maybe Python has internal API for opening file descriptors and it is possible to intercept the operation on this level. Is it feasible to route all file descriptor open operations through such API that allows to audit open/close operations and filenames through callback? > On Linux you can look somewhere in /proc, but I don't know that it > would help you find where a file was opened. If I can query FD counter - I can automate the process of walking through the code line by line to find places where this descriptor incremented or decremented. Of course it would be nice to get access to FD stack so that a full filename can also be retrieved in this case. It would be nice if at least Linux implementation provided a way to detect leaking descriptors, thanks for suggestions, but my expertise and available resources are limited to Windows machines, so for now I won't be able to try anything more complicated than an unpack-and-launch Linux solution. -- anatoly t. From g.rodola at gmail.com Fri Sep 3 12:09:50 2010 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Fri, 3 Sep 2010 12:09:50 +0200 Subject: [Python-Dev] Internal counter to debug leaking file descriptors In-Reply-To: References: Message-ID: > Of course it would be nice to get access to FD stack so that a > full filename can also be retrieved in this case. On Linux, this can be easily achieved by using /proc. You can take a look at how this is done in the current development version of psutil: http://code.google.com/p/psutil/source/browse/trunk/psutil/_pslinux.py?spec=svn633&r=630#266 Usage: >>> import psutil, os >>> this_process = psutil.Process(os.getpid()) >>> f = open('file.ext', 'w') >>> this_process.get_open_files() ['/home/giampaolo/svn/psutil/file.ext'] Same for sockets, a bunch of lines later: http://code.google.com/p/psutil/source/browse/trunk/psutil/_pslinux.py?spec=svn633&r=630#284 >>> import socket >>> s = socket.create_connection(('google.com', 80)) >>> this_process.get_connections() [connection(family=2, type=1, local_address=('192.168.1.43', 38067), remote_address=('72.14.234.104', 80), status='ESTABLISHED')] >>> Hope this helps --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ From victor.stinner at haypocalc.com Fri Sep 3 12:14:03 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 3 Sep 2010 12:14:03 +0200 Subject: [Python-Dev] r84430 - in python/branches/py3k: Include/unicodeobject.h Objects/unicodeobject.c In-Reply-To: References: <20100901234353.468A5EEAAC@mail.python.org> <201009030126.32665.victor.stinner@haypocalc.com> Message-ID: <201009031214.03601.victor.stinner@haypocalc.com> Le vendredi 03 septembre 2010 11:20:04, vous avez ?crit : > > But I didn't found any doc for other Py_UNICODE_str*() > > functions in Doc/c-api/*.rst. > > http://bugs.python.org/issue8649 - Py_UNICODE_* functions are undocumented Opened since 3 months. It looks like programmers don't like writing documentation :-) -- Victor Stinner http://www.haypocalc.com/ From ncoghlan at gmail.com Fri Sep 3 12:24:21 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 Sep 2010 20:24:21 +1000 Subject: [Python-Dev] PEP 384 status In-Reply-To: <4C806114.5030004@canterbury.ac.nz> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D80EE.8090705@canterbury.ac.nz> <20100901004238.72d050ac@pitrou.net> <1283345691.3228.11.camel@localhost.localdomain> <4C806114.5030004@canterbury.ac.nz> Message-ID: On Fri, Sep 3, 2010 at 12:44 PM, Greg Ewing wrote: > On 02/09/10 09:04, Nick Coghlan wrote: > >> I think it would be better if everything dealing with FILE* was a >> macro rather than a function, yes. > > How would that help? Macros (and, as Antoine pointed out, inline functions) will never cross C runtime boundaries, even on Windows. They'll get compiled in when the extension module is built and linked to a C runtime library from there, so it won't matter if that differs from the one that is linked to the Python binary. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Fri Sep 3 12:24:21 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 Sep 2010 20:24:21 +1000 Subject: [Python-Dev] PEP 384 status In-Reply-To: <4C806114.5030004@canterbury.ac.nz> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D80EE.8090705@canterbury.ac.nz> <20100901004238.72d050ac@pitrou.net> <1283345691.3228.11.camel@localhost.localdomain> <4C806114.5030004@canterbury.ac.nz> Message-ID: On Fri, Sep 3, 2010 at 12:44 PM, Greg Ewing wrote: > On 02/09/10 09:04, Nick Coghlan wrote: > >> I think it would be better if everything dealing with FILE* was a >> macro rather than a function, yes. > > How would that help? Macros (and, as Antoine pointed out, inline functions) will never cross C runtime boundaries, even on Windows. They'll get compiled in when the extension module is built and linked to a C runtime library from there, so it won't matter if that differs from the one that is linked to the Python binary. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Fri Sep 3 12:24:21 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 Sep 2010 20:24:21 +1000 Subject: [Python-Dev] PEP 384 status In-Reply-To: <4C806114.5030004@canterbury.ac.nz> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D80EE.8090705@canterbury.ac.nz> <20100901004238.72d050ac@pitrou.net> <1283345691.3228.11.camel@localhost.localdomain> <4C806114.5030004@canterbury.ac.nz> Message-ID: On Fri, Sep 3, 2010 at 12:44 PM, Greg Ewing wrote: > On 02/09/10 09:04, Nick Coghlan wrote: > >> I think it would be better if everything dealing with FILE* was a >> macro rather than a function, yes. > > How would that help? Macros (and, as Antoine pointed out, inline functions) will never cross C runtime boundaries, even on Windows. They'll get compiled in when the extension module is built and linked to a C runtime library from there, so it won't matter if that differs from the one that is linked to the Python binary. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Fri Sep 3 12:28:47 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 3 Sep 2010 12:28:47 +0200 Subject: [Python-Dev] r84447 - in python/branches/py3k: Lib/test/test_set.py Objects/setobject.c References: <20100903100051.16CA7EEA4D@mail.python.org> Message-ID: <20100903122847.05e03c22@pitrou.net> On Fri, 3 Sep 2010 12:00:51 +0200 (CEST) raymond.hettinger wrote: > rv = set_discard_entry(so, &an_entry); > - if (rv == -1) > + if (rv == -1) { > + Py_DECREF(key); > return NULL; > + } > if (rv == DISCARD_NOTFOUND) { > - if (set_add_entry(so, &an_entry) == -1) > + if (set_add_entry(so, &an_entry) == -1) { > + Py_DECREF(key); > return NULL; > + } > } > + Py_DECREF(key); > } You have some misindented code here. Regards Antoine. From ncoghlan at gmail.com Fri Sep 3 12:44:01 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 Sep 2010 20:44:01 +1000 Subject: [Python-Dev] Buffer protocol for io.BytesIO? In-Reply-To: <20100903105423.471d7382@pitrou.net> References: <20100902223509.64e62db3@pitrou.net> <20100903105423.471d7382@pitrou.net> Message-ID: On Fri, Sep 3, 2010 at 6:54 PM, Antoine Pitrou wrote: >> What would be the advantage of having to do two operations? Is it just that >> you want to make the internal buffer joining step more visible? > > I was just wondering if it's a good thing to let people, e.g., > concatenate a bytes object and a BytesIO by using '+'. It actually strikes me as a fairly bad thing, so I think you're right to distrust it. Better to follow the precedent set with getvalue() and require an explicit call to getbuffer(). The fact there are two options (immutable copy via getvalue() or mutable accessor via getbuffer()) is all the more reason not to make either conversion implicit. >> Also, note that the BytesIO object must not be modified while someone reads >> the buffer. That would be a less obvious side effect if the returned buffer >> object was different from the BytesIO object itself. > > It could not be resized, but it could be modified (same as what happens > with bytearrays today). Actually, the buffer itself would be writable, > and allow modifying the BytesIO contents. You may need to be careful with reads and writes while the buffer is exposed (e.g. throwing an exception until the buffer is released again). Perhaps the buffer accessor should be a context manager so it is easy to enforce prompt release? That is: with datastream.getbuffer() as buf: ... # any resizing operations on datastream, including read/write raise until the buffer is released Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Fri Sep 3 12:44:01 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 Sep 2010 20:44:01 +1000 Subject: [Python-Dev] Buffer protocol for io.BytesIO? In-Reply-To: <20100903105423.471d7382@pitrou.net> References: <20100902223509.64e62db3@pitrou.net> <20100903105423.471d7382@pitrou.net> Message-ID: On Fri, Sep 3, 2010 at 6:54 PM, Antoine Pitrou wrote: >> What would be the advantage of having to do two operations? Is it just that >> you want to make the internal buffer joining step more visible? > > I was just wondering if it's a good thing to let people, e.g., > concatenate a bytes object and a BytesIO by using '+'. It actually strikes me as a fairly bad thing, so I think you're right to distrust it. Better to follow the precedent set with getvalue() and require an explicit call to getbuffer(). The fact there are two options (immutable copy via getvalue() or mutable accessor via getbuffer()) is all the more reason not to make either conversion implicit. >> Also, note that the BytesIO object must not be modified while someone reads >> the buffer. That would be a less obvious side effect if the returned buffer >> object was different from the BytesIO object itself. > > It could not be resized, but it could be modified (same as what happens > with bytearrays today). Actually, the buffer itself would be writable, > and allow modifying the BytesIO contents. You may need to be careful with reads and writes while the buffer is exposed (e.g. throwing an exception until the buffer is released again). Perhaps the buffer accessor should be a context manager so it is easy to enforce prompt release? That is: with datastream.getbuffer() as buf: ... # any resizing operations on datastream, including read/write raise until the buffer is released Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Fri Sep 3 12:52:51 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 Sep 2010 20:52:51 +1000 Subject: [Python-Dev] r84430 - in python/branches/py3k: Include/unicodeobject.h Objects/unicodeobject.c In-Reply-To: <201009031214.03601.victor.stinner@haypocalc.com> References: <20100901234353.468A5EEAAC@mail.python.org> <201009030126.32665.victor.stinner@haypocalc.com> <201009031214.03601.victor.stinner@haypocalc.com> Message-ID: On Fri, Sep 3, 2010 at 8:14 PM, Victor Stinner wrote: > Le vendredi 03 septembre 2010 11:20:04, vous avez ?crit : >> > But I didn't found any doc for other Py_UNICODE_str*() >> > functions in Doc/c-api/*.rst. >> >> http://bugs.python.org/issue8649 - Py_UNICODE_* functions are undocumented > > Opened since 3 months. It looks like programmers don't like writing > documentation :-) In this case, "C programmers think C stdlib function names are intuitive" may be closer to the mark ;) Cheers, Nick. P.S. More seriously, these particular functions really are just like the standard C function of the same name, only operating on Py_UNICODE* rather than char*, so I can understand documentation being skipped when they were added. However, they may as well be mentioned in http://docs.python.org/dev/c-api/unicode.html#plain-py-unicode -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Fri Sep 3 12:56:24 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 3 Sep 2010 12:56:24 +0200 Subject: [Python-Dev] Buffer protocol for io.BytesIO? In-Reply-To: References: <20100902223509.64e62db3@pitrou.net> <20100903105423.471d7382@pitrou.net> Message-ID: <20100903125624.646acbb4@pitrou.net> On Fri, 3 Sep 2010 20:44:01 +1000 Nick Coghlan wrote: > > It actually strikes me as a fairly bad thing, so I think you're right > to distrust it. Better to follow the precedent set with getvalue() and > require an explicit call to getbuffer(). The fact there are two > options (immutable copy via getvalue() or mutable accessor via > getbuffer()) is all the more reason not to make either conversion > implicit. Thank you for the advice. I think getbuffer() it will be :) > > It could not be resized, but it could be modified (same as what happens > > with bytearrays today). Actually, the buffer itself would be writable, > > and allow modifying the BytesIO contents. > > You may need to be careful with reads and writes while the buffer is > exposed (e.g. throwing an exception until the buffer is released > again). Perhaps the buffer accessor should be a context manager so it > is easy to enforce prompt release? That's an interesting idea. I was planning to return a memoryview object (in order to hide the intermediate object, and make it really minimal), so perhaps the context protocol should be enabled on memoryviews? (__enter__ would be a no-op, and __exit__ would release the internal buffer and render it invalid, a bit like closing a file) Regards Antoine. From ncoghlan at gmail.com Fri Sep 3 13:02:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 Sep 2010 21:02:50 +1000 Subject: [Python-Dev] r84430 - in python/branches/py3k: Include/unicodeobject.h Objects/unicodeobject.c In-Reply-To: <4C80AB48.7040100@egenix.com> References: <20100901234353.468A5EEAAC@mail.python.org> <4C7F6AB2.5090606@egenix.com> <201009030126.32665.victor.stinner@haypocalc.com> <4C80AB48.7040100@egenix.com> Message-ID: On Fri, Sep 3, 2010 at 6:01 PM, M.-A. Lemburg wrote: >> About PyUnicode_strdup() (PyUnicode_AsUnicodeCopy): I don't know. It is >> possible to rewrite it in few lines. Why don't you want to add them to the >> public API? For my work, it doesn't matter if it's public or not. This >> function uses PyMem_xxx API, I don't know if a third part library would like >> to rely on PyMem_xxx. > > This will have to be documented (see PEP 384 for some reasoning on > malloc() use in DLLs). PyUnicode_AsUnicodeCopy is indeed a better name. As far as the need to call PyMem_Free goes, a similar requirement already exists for argument parsing via es and es# (see http://docs.python.org/dev/c-api/arg.html#strings-and-buffers), so just documenting it is fine. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From stefan_ml at behnel.de Fri Sep 3 13:10:22 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Sep 2010 13:10:22 +0200 Subject: [Python-Dev] Buffer protocol for io.BytesIO? In-Reply-To: <20100903125624.646acbb4@pitrou.net> References: <20100902223509.64e62db3@pitrou.net> <20100903105423.471d7382@pitrou.net> <20100903125624.646acbb4@pitrou.net> Message-ID: Antoine Pitrou, 03.09.2010 12:56: > On Fri, 3 Sep 2010 20:44:01 +1000 > Nick Coghlan wrote: >> >> It actually strikes me as a fairly bad thing, so I think you're right >> to distrust it. +1 >>> It could not be resized, but it could be modified (same as what happens >>> with bytearrays today). Actually, the buffer itself would be writable, >>> and allow modifying the BytesIO contents. >> >> You may need to be careful with reads and writes while the buffer is >> exposed (e.g. throwing an exception until the buffer is released >> again). Perhaps the buffer accessor should be a context manager so it >> is easy to enforce prompt release? > > That's an interesting idea. I was planning to return a memoryview > object (in order to hide the intermediate object, and make it really > minimal), so perhaps the context protocol should be enabled on > memoryviews? > > (__enter__ would be a no-op, and __exit__ would release the internal > buffer and render it invalid, a bit like closing a file) I can't see a reason not to support that. Sounds a lot simpler than requiring to set the memory view reference to None in order to trigger a cleanup at an undefined point in time. Stefan From ncoghlan at gmail.com Fri Sep 3 13:13:07 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 Sep 2010 21:13:07 +1000 Subject: [Python-Dev] C API doc question Message-ID: Due to the Unicode API discussion, I happened to be looking at the C API docs at http://docs.python.org/dev/c-api/unicode.html#plain-py-unicode and noticed that out of: PyUnicode_FromUnicode PyUnicode_FromStringAndSize PyUnicode_FromString PyUnicode_FromFormat PyUnicode_FromFormatV PyUnicode_FromEncodedObject PyUnicode_FromObject only the first and the last two are flagged in the online docs as returning a new reference. The other 4 are not (but probably should be). However, I can't see anything in the markup which is even causing those "Return value: New reference" markings to appear in the first place, nor any clues in the Documenting Python info. What am I missing? Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Fri Sep 3 13:16:13 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 Sep 2010 21:16:13 +1000 Subject: [Python-Dev] Buffer protocol for io.BytesIO? In-Reply-To: <20100903125624.646acbb4@pitrou.net> References: <20100902223509.64e62db3@pitrou.net> <20100903105423.471d7382@pitrou.net> <20100903125624.646acbb4@pitrou.net> Message-ID: On Fri, Sep 3, 2010 at 8:56 PM, Antoine Pitrou wrote: > On Fri, 3 Sep 2010 20:44:01 +1000 Nick Coghlan wrote: > That's an interesting idea. I was planning to return a memoryview > object (in order to hide the intermediate object, and make it really > minimal), so perhaps the context protocol should be enabled on > memoryviews? > > (__enter__ would be a no-op, and __exit__ would release the internal > buffer and render it invalid, a bit like closing a file) Yep, sounds good (with "no-op" in this case meaning "just return self"). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From amauryfa at gmail.com Fri Sep 3 13:19:47 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Fri, 3 Sep 2010 13:19:47 +0200 Subject: [Python-Dev] C API doc question In-Reply-To: References: Message-ID: 2010/9/3 Nick Coghlan : > Due to the Unicode API discussion, I happened to be looking at the C > API docs at http://docs.python.org/dev/c-api/unicode.html#plain-py-unicode > and noticed that out of: > > PyUnicode_FromUnicode > PyUnicode_FromStringAndSize > PyUnicode_FromString > PyUnicode_FromFormat > PyUnicode_FromFormatV > PyUnicode_FromEncodedObject > PyUnicode_FromObject > > only the first and the last two are flagged in the online docs as > returning a new reference. The other 4 are not (but probably should > be). > > However, I can't see anything in the markup which is even causing > those "Return value: New reference" markings to appear in the first > place, nor any clues in the Documenting Python info. What am I > missing? This information is in the file: Doc/data/refcounts.dat There is a extension to sphinx that reads this file and generates the annotation in the documentation. This file is not very well known, even by core developers... -- Amaury Forgeot d'Arc From ncoghlan at gmail.com Fri Sep 3 13:44:51 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 Sep 2010 21:44:51 +1000 Subject: [Python-Dev] C API doc question In-Reply-To: References: Message-ID: On Fri, Sep 3, 2010 at 9:19 PM, Amaury Forgeot d'Arc wrote: > 2010/9/3 Nick Coghlan : >> Due to the Unicode API discussion, I happened to be looking at the C >> API docs at http://docs.python.org/dev/c-api/unicode.html#plain-py-unicode >> and noticed that out of: >> >> PyUnicode_FromUnicode >> PyUnicode_FromStringAndSize >> PyUnicode_FromString >> PyUnicode_FromFormat >> PyUnicode_FromFormatV >> PyUnicode_FromEncodedObject >> PyUnicode_FromObject >> >> only the first and the last two are flagged in the online docs as >> returning a new reference. The other 4 are not (but probably should >> be). >> >> However, I can't see anything in the markup which is even causing >> those "Return value: New reference" markings to appear in the first >> place, nor any clues in the Documenting Python info. What am I >> missing? > > This information is in the file: > ? ?Doc/data/refcounts.dat > There is a extension to sphinx that reads this file and generates > the annotation in the documentation. > This file is not very well known, even by core developers... Nope, never knew it existed until just now :) Since it does exist, how hard would it be to get the extension to emit a warning if a C API function isn't listed in that file, or if a function listed in that file doesn't appear anywhere in the docs? Currently it appears to default to reporting no effect on references if a function isn't mentioned, which is potentially misleading (as the above examples show, this file completely missed out on the PyString->PyUnicode rename). >From a more wish-list point of view, how hard would it be to modify the extension to support having this markup inline with the function documentation rather than having it off in a separate file? Or addressing Skip's comment by allowing "-0" to indicate cases where item references are stolen? I created http://bugs.python.org/issue9755 to cover these questions. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From barry at python.org Fri Sep 3 16:47:22 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 3 Sep 2010 10:47:22 -0400 Subject: [Python-Dev] PEP 3149 accepted In-Reply-To: References: Message-ID: <20100903104722.33c1508a@mission> On Sep 03, 2010, at 09:36 AM, Georg Brandl wrote: >This is to let you all know that PEP 3149 is accepted. > >Benjamin and I decided that on the basis that > >* strong precedent is set with PEP 3147 >* it is not mutually exclusive with PEP 384; should PEP 384 become > widely accepted, its use can fade out again >* it is a strictly optional feature > >Barry promised the code will go in before this weekend's alpha2, so >that distributors can subject the new code to proper testing, should >they choose to use PEP 3149 features. Thanks very much Georg and Benjamin. I am going to merge the code in the next hour or so. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From solipsis at pitrou.net Fri Sep 3 17:10:19 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 3 Sep 2010 17:10:19 +0200 Subject: [Python-Dev] Buffer protocol for io.BytesIO? In-Reply-To: References: <20100902223509.64e62db3@pitrou.net> <20100903105423.471d7382@pitrou.net> <20100903125624.646acbb4@pitrou.net> Message-ID: <20100903171019.793a55d7@pitrou.net> On Fri, 3 Sep 2010 21:16:13 +1000 Nick Coghlan wrote: > On Fri, Sep 3, 2010 at 8:56 PM, Antoine Pitrou wrote: > > On Fri, 3 Sep 2010 20:44:01 +1000 Nick Coghlan wrote: > > That's an interesting idea. I was planning to return a memoryview > > object (in order to hide the intermediate object, and make it really > > minimal), so perhaps the context protocol should be enabled on > > memoryviews? > > > > (__enter__ would be a no-op, and __exit__ would release the internal > > buffer and render it invalid, a bit like closing a file) > > Yep, sounds good (with "no-op" in this case meaning "just return self"). It just occurred to me that it could be interpreted as a breach of the moratorium. On the other hand, according to PEP 3003: ?The case for adding a method to a built-in object can be made.? This case seems even lighter than adding a new method. Regards Antoine. From barry at python.org Fri Sep 3 17:15:51 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 3 Sep 2010 11:15:51 -0400 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <37D6B5E2-2DA4-47C4-B773-11CB6F98162E@gmail.com> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <37D6B5E2-2DA4-47C4-B773-11CB6F98162E@gmail.com> Message-ID: <20100903111551.660fb0e3@mission> On Sep 02, 2010, at 09:08 PM, Raymond Hettinger wrote: >On Apr 30, 2010, at 12:51 PM, Martin v. L?wis wrote: >> Without a BDFL, I think we need a committee to make decisions, e.g. >> by majority vote amongst committers. > >I like Guido's idea. Just appoint have one of the experienced >developers who is independent of the proposal and have them be the >final arbiter. For example, Guido had earlier suggested that I decide >the fate of the "yield from" proposal because I had experience in the >topic but was not not personally involved in the proposal. > >Guido has set a good example for others to follow: >* let a conversation evolve until an outcome is self-evident >* or kill it early if it has no chance >* or if discussion teases out all of the meaningful thinking > but doesn't reach a clear conclusion, just make a choice > based on instinct >* have biases toward real-world use cases, towards ideas proven in > other languages (category killers), towards slow rates of language > evolution, and think about the long-term. > >It is better to have one experienced developer decide than to have >a committee. I completely agree that it's good we're beginning to give Guido a break so that he doesn't have to follow every thread, think hard about every PEP and make every decision. Having just gone through this as a PEP author with 3149, I have a couple of thoughts on how we might begin to formalize this process. Of course, we want to keep it lightweight, but also effective and as always in the best interest of Python. One thing that would help would be for Guido to let us know early on when he'd prefer to delegate the decision. This goes along with identifying the ultimate PEP arbiter (UPA? :) as early as possible. As Raymond says, it should be someone independent of the proposal, but with the interest, time, and experience necessary to make an informed decision. We might even want to capture the arbiter selection in a PEP header (similar to the new Resolution header for capturing the final decision reference). While I agree that we don't want decision by committee, I think we should consider a preference for paired arbiters. I have the highest respect for all the senior developers who would likely make up the pool of PEP deciders, but it gave me great confidence to have both Benjamin and Georg decide the fate of PEP 3149. As someone who might serve a similar role in the future, I would value a second person to sanity check my own thoughts on the matter and to identify any holes in my understanding (or missed emails :). I'd say, let's state a preference (not a requirement) for two arbiters for any PEP that's not decided by Guido. We'd talked before about allowing the RM for the target version to make the decision. Maybe the RM can serve as that second arbiter when no other obvious candidate is available. Raymond, you identified a great set of criteria that the arbiters should use to guide them to a decision. I'm willing to write up an informational PEP that codifies this and any other guidelines we come up with for non-BDFL PEP decisions. Finally a reminder to PEP authors that it is your responsibility to shepherd your PEP through the process. Don't be a pest, but do keep an eye on the release calendar so that you're not scrambling for a snap decision at the last minute. 18 months can go by quickly. :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From guido at python.org Fri Sep 3 17:45:42 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 3 Sep 2010 08:45:42 -0700 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <20100903111551.660fb0e3@mission> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <37D6B5E2-2DA4-47C4-B773-11CB6F98162E@gmail.com> <20100903111551.660fb0e3@mission> Message-ID: On Fri, Sep 3, 2010 at 8:15 AM, Barry Warsaw wrote: > One thing that would help would be for Guido to let us know early on when he'd > prefer to delegate the decision. Hey! I'm still here! :-) More to the point, you can assume that I'm happy to have every PEP decision made by someone else *except* if you see me participate in the thread. If I don't show up in the thread, you can assume that either (a) I don't care or know enough about the topic, or (b) I am confident that it's going in the right direction. > This goes along with identifying the > ultimate PEP arbiter (UPA? :) as early as possible. I prefer acronyms derived from BDFL, like BDFM ("BD for the moment") or perhaps BDFP ("BD for the PEP"). (BDFM sounds better but BDFP is more accurate. I'll leave it up to the FLUFL to pick one.) >?As Raymond says, it > should be someone independent of the proposal, but with the interest, time, > and experience necessary to make an informed decision. ?We might even want to > capture the arbiter selection in a PEP header (similar to the new Resolution > header for capturing the final decision reference). > > While I agree that we don't want decision by committee, I think we should > consider a preference for paired arbiters. ?I have the highest respect for all > the senior developers who would likely make up the pool of PEP deciders, but > it gave me great confidence to have both Benjamin and Georg decide the fate of > PEP 3149. ?As someone who might serve a similar role in the future, I would > value a second person to sanity check my own thoughts on the matter and to > identify any holes in my understanding (or missed emails :). ?I'd > say, let's state a preference (not a requirement) for two arbiters for any PEP > that's not decided by Guido. Hm... As long as we can make sure not to pick the same pair all the time this makes sense. (Not that I have any objections to how the Georg+Benjamin pair decided PEP 3149 -- to the contrary -- but I think it would be good to spread the power.) But if a pair can't be found I think a single BDFM/BDFP will work too. > We'd talked before about allowing the RM for the target version to make the > decision. ?Maybe the RM can serve as that second arbiter when no other obvious > candidate is available. Good fallback plan. > Raymond, you identified a great set of criteria that the arbiters should use > to guide them to a decision. ?I'm willing to write up an informational PEP > that codifies this and any other guidelines we come up with for non-BDFL PEP > decisions. > > Finally a reminder to PEP authors that it is your responsibility to shepherd > your PEP through the process. ?Don't be a pest, but do keep an eye on the > release calendar so that you're not scrambling for a snap decision at the last > minute. ?18 months can go by quickly. :) Well, realistically, there's only so much grief that anyone PEP author can be expected to put up with. I expect that a lot of PEPs won't be written or will be withdrawn in the face of prolonged discussion. Early selection of a BDFM (maybe the M can also refer to mentorship) ought to help in encouraging where encouragement would help -- and of course sometimes the best thing to do is to encourage the PEP author to drop the idea, if no consensus is in view (or if the author is particularly hard-headed). -- --Guido van Rossum (python.org/~guido) From merwok at netwok.org Fri Sep 3 17:49:50 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 03 Sep 2010 17:49:50 +0200 Subject: [Python-Dev] Buffer protocol for io.BytesIO? In-Reply-To: <20100903171019.793a55d7@pitrou.net> References: <20100902223509.64e62db3@pitrou.net> <20100903105423.471d7382@pitrou.net> <20100903125624.646acbb4@pitrou.net> <20100903171019.793a55d7@pitrou.net> Message-ID: <4C81191E.2040100@netwok.org> > It just occurred to me that it could be interpreted as a breach of the > moratorium. On the other hand, according to PEP 3003: > > ?The case for adding a method to a built-in object can be made.? > > This case seems even lighter than adding a new method. I?ve been satisfied to see other proposals breaking the moratorium being refused. Respecting rules set by ourselves for the benefit of other VMs sends a good signal. In this case, the PEP leaves an opening (quoted line above), the change should be very easy to implement in other VMs, and it?s an elegant way of adding a useful feature. +1 from me. Regards From status at bugs.python.org Fri Sep 3 18:07:45 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 3 Sep 2010 18:07:45 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100903160745.99A3578130@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-08-27 - 2010-09-03) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues stats: open 2569 (+41) closed 18970 (+40) total 21539 (+55) Open issues with patches: 1091 Issues opened (41) ================== #5217: testExtractDir (test.test_zipfile.TestWithDirectory) fails whe http://bugs.python.org/issue5217 reopened by eric.araujo #9706: ssl errors checking http://bugs.python.org/issue9706 opened by giampaolo.rodola #9707: Reworked threading.local reference implementation http://bugs.python.org/issue9707 opened by pitrou #9708: cElementTree iterparse does not support "parser" argument http://bugs.python.org/issue9708 opened by adrian_nye #9709: test_distutils warning: initfunc exported twice on Windows http://bugs.python.org/issue9709 opened by skrah #9713: Py_CompileString fails on non decode-able paths. http://bugs.python.org/issue9713 opened by ideasman42 #9714: urllib2 digest authentication doesn't work when connecting to http://bugs.python.org/issue9714 opened by warpr #9716: The inittab modules cannot be packages http://bugs.python.org/issue9716 opened by kayhayen #9717: operator module - "in place" operators documentation http://bugs.python.org/issue9717 opened by arno #9719: build_ssl.py: cannot find 'asm64/*.*' http://bugs.python.org/issue9719 opened by srid #9720: zipfile writes incorrect local file header for large files in http://bugs.python.org/issue9720 opened by craigds #9721: urlparse.urljoin() cuts off last base character with semicolon http://bugs.python.org/issue9721 opened by calvin #9722: PyObject_Print with Visual Studio 2010 http://bugs.python.org/issue9722 opened by Krauzi #9723: pipes.quote() needs to be documented http://bugs.python.org/issue9723 opened by brandon-rhodes #9725: urllib.request.FancyURLopener won't connect to pages requiring http://bugs.python.org/issue9725 opened by petr6.6 #9727: Add callbacks to be invoked when locale changes http://bugs.python.org/issue9727 opened by ncoghlan #9728: Docs point to FAQ Section 3, but FAQs are not numbered http://bugs.python.org/issue9728 opened by stutzbach #9729: Unconnected SSLSocket.{send,recv} raises TypeError: 'member_de http://bugs.python.org/issue9729 opened by spiv #9730: base64 docs refers to strings instead of bytes http://bugs.python.org/issue9730 opened by JingCheng.LIU #9731: Add ABCMeta.has_methods and tests that use it http://bugs.python.org/issue9731 opened by stutzbach #9732: Addition of getattr_static for inspect module http://bugs.python.org/issue9732 opened by michael.foord #9733: Can't iterate over multiprocessing.managers.DictProxy http://bugs.python.org/issue9733 opened by jjconti #9736: doctest.DocTestSuite doesn't handle test globs correctly http://bugs.python.org/issue9736 opened by j1m #9738: Document the encoding of functions bytes arguments of the C AP http://bugs.python.org/issue9738 opened by haypo #9739: Output of help(...) is wider than 80 characters http://bugs.python.org/issue9739 opened by casevh #9740: Support for HTTP 1.1 persistent connections throughout the sta http://bugs.python.org/issue9740 opened by ipatrol #9741: msgfmt.py generates invalid mo because msgfmt.make() does not http://bugs.python.org/issue9741 opened by timothy.ty.lee #9742: Python 2.7: math module fails to build on Solaris 9 http://bugs.python.org/issue9742 opened by mhenriq #9743: __call__.__call__ chain cause crash when long enough http://bugs.python.org/issue9743 opened by modchan #9745: MSVC .pdb files not created by python 2.7 distutils http://bugs.python.org/issue9745 opened by jpe #9746: All sequence types support .index and .count http://bugs.python.org/issue9746 opened by stutzbach #9747: os.getresgid() documentation mentions "user ids", not "group i http://bugs.python.org/issue9747 opened by Red HamsterX #9748: .inputrc magic-space breaks interactive mode http://bugs.python.org/issue9748 opened by stephenk #9750: sqlite3 iterdump fails on column with reserved name http://bugs.python.org/issue9750 opened by Marko.Kohtala #9751: _PyInstance_Lookup() defeats its purpose http://bugs.python.org/issue9751 opened by arigo #9752: MSVC Compiler warning in Python\import.c http://bugs.python.org/issue9752 opened by stutzbach #9754: assertWarns and assertWarnsRegexp http://bugs.python.org/issue9754 opened by pitrou #9755: Fix refcounting details in Py3k C API documentation http://bugs.python.org/issue9755 opened by ncoghlan #9756: Crash with custom __getattribute__ http://bugs.python.org/issue9756 opened by flox #9757: Add context manager protocol to memoryviews http://bugs.python.org/issue9757 opened by pitrou #1367631: maximum length not enforced in cgi.parse() http://bugs.python.org/issue1367631 reopened by r.david.murray Most recent 15 issues with no replies (15) ========================================== #9757: Add context manager protocol to memoryviews http://bugs.python.org/issue9757 #9755: Fix refcounting details in Py3k C API documentation http://bugs.python.org/issue9755 #9752: MSVC Compiler warning in Python\import.c http://bugs.python.org/issue9752 #9748: .inputrc magic-space breaks interactive mode http://bugs.python.org/issue9748 #9747: os.getresgid() documentation mentions "user ids", not "group i http://bugs.python.org/issue9747 #9746: All sequence types support .index and .count http://bugs.python.org/issue9746 #9745: MSVC .pdb files not created by python 2.7 distutils http://bugs.python.org/issue9745 #9741: msgfmt.py generates invalid mo because msgfmt.make() does not http://bugs.python.org/issue9741 #9736: doctest.DocTestSuite doesn't handle test globs correctly http://bugs.python.org/issue9736 #9733: Can't iterate over multiprocessing.managers.DictProxy http://bugs.python.org/issue9733 #9728: Docs point to FAQ Section 3, but FAQs are not numbered http://bugs.python.org/issue9728 #9727: Add callbacks to be invoked when locale changes http://bugs.python.org/issue9727 #9725: urllib.request.FancyURLopener won't connect to pages requiring http://bugs.python.org/issue9725 #9720: zipfile writes incorrect local file header for large files in http://bugs.python.org/issue9720 #9717: operator module - "in place" operators documentation http://bugs.python.org/issue9717 Most recent 15 issues waiting for review (15) ============================================= #9757: Add context manager protocol to memoryviews http://bugs.python.org/issue9757 #9754: assertWarns and assertWarnsRegexp http://bugs.python.org/issue9754 #9750: sqlite3 iterdump fails on column with reserved name http://bugs.python.org/issue9750 #9741: msgfmt.py generates invalid mo because msgfmt.make() does not http://bugs.python.org/issue9741 #9739: Output of help(...) is wider than 80 characters http://bugs.python.org/issue9739 #9738: Document the encoding of functions bytes arguments of the C AP http://bugs.python.org/issue9738 #9732: Addition of getattr_static for inspect module http://bugs.python.org/issue9732 #9721: urlparse.urljoin() cuts off last base character with semicolon http://bugs.python.org/issue9721 #9720: zipfile writes incorrect local file header for large files in http://bugs.python.org/issue9720 #9717: operator module - "in place" operators documentation http://bugs.python.org/issue9717 #9707: Reworked threading.local reference implementation http://bugs.python.org/issue9707 #9706: ssl errors checking http://bugs.python.org/issue9706 #9701: Update ZSH profile on Mac OS installation http://bugs.python.org/issue9701 #9693: asynchat push_callable() patch http://bugs.python.org/issue9693 #9687: dbmmodule.c:dbm_contains fails on 64bit big-endian (test_dbm.p http://bugs.python.org/issue9687 Top 10 most discussed issues (10) ================================= #9706: ssl errors checking http://bugs.python.org/issue9706 10 msgs #9754: assertWarns and assertWarnsRegexp http://bugs.python.org/issue9754 7 msgs #9042: Gettext cache and classes http://bugs.python.org/issue9042 6 msgs #9212: dict_keys purports to implement the Set ABC, but is missing th http://bugs.python.org/issue9212 6 msgs #9732: Addition of getattr_static for inspect module http://bugs.python.org/issue9732 6 msgs #9205: Parent process hanging in multiprocessing if children terminat http://bugs.python.org/issue9205 5 msgs #9377: socket, PEP 383: Mishandling of non-ASCII bytes in host/domain http://bugs.python.org/issue9377 5 msgs #9731: Add ABCMeta.has_methods and tests that use it http://bugs.python.org/issue9731 5 msgs #9742: Python 2.7: math module fails to build on Solaris 9 http://bugs.python.org/issue9742 5 msgs #7231: Windows installer does not add \Scripts folder to the path http://bugs.python.org/issue7231 4 msgs Issues closed (40) ================== #1512791: module wave does no rounding http://bugs.python.org/issue1512791 closed by mark.dickinson #1652: subprocess should have an option to restore SIGPIPE to default http://bugs.python.org/issue1652 closed by gregory.p.smith #1868: threading.local doesn't free attrs when assigning thread exits http://bugs.python.org/issue1868 closed by pitrou #3101: global function _add_one_to_index_C http://bugs.python.org/issue3101 closed by pitrou #3985: removed string module use in distutils http://bugs.python.org/issue3985 closed by eric.araujo #4319: optparse and non-ascii help strings http://bugs.python.org/issue4319 closed by r.david.murray #4835: SIZEOF_SOCKET_T not defined http://bugs.python.org/issue4835 closed by pitrou #5362: Add configure option to disable Py3k warnings http://bugs.python.org/issue5362 closed by brett.cannon #5553: Py_LOCAL_INLINE(type) doesn't actually inline except using MSC http://bugs.python.org/issue5553 closed by stutzbach #7141: 2to3 should add from __future__ import print_statement http://bugs.python.org/issue7141 closed by benjamin.peterson #7415: PyUnicode_FromEncodedObject() uses PyObject_AsCharBuffer() http://bugs.python.org/issue7415 closed by pitrou #8420: Objects/setobject.c contains unsafe code http://bugs.python.org/issue8420 closed by rhettinger #8990: array constructor and array.fromstring should accept bytearray http://bugs.python.org/issue8990 closed by pitrou #9053: distutils compiles extensions so that Python.h cannot be found http://bugs.python.org/issue9053 closed by exarkun #9155: Reserve COMPARE_OP for rich comparisons http://bugs.python.org/issue9155 closed by rhettinger #9549: Remove sys.setdefaultencoding() http://bugs.python.org/issue9549 closed by pitrou #9634: Add timeout parameter to Queue.join() http://bugs.python.org/issue9634 closed by rhettinger #9664: Make gzip module not require that underlying file object suppo http://bugs.python.org/issue9664 closed by terry.reedy #9677: "Global Module Index" link dead http://bugs.python.org/issue9677 closed by georg.brandl #9695: Return from generators in Python 3.2 http://bugs.python.org/issue9695 closed by georg.brandl #9700: semaphore errors on AIX 6.1 http://bugs.python.org/issue9700 closed by pitrou #9703: default param values http://bugs.python.org/issue9703 closed by georg.brandl #9704: 3.2 - zlib.pc.in is missing in source tree http://bugs.python.org/issue9704 closed by loewis #9705: limit dict.update() to a given list of keys http://bugs.python.org/issue9705 closed by benjamin.peterson #9710: 2to3 could remove "-*- coding: utf-8 -*-" http://bugs.python.org/issue9710 closed by georg.brandl #9711: ssl.SSLSocket's keyfile argument seems to be ignored if specif http://bugs.python.org/issue9711 closed by giampaolo.rodola #9712: tokenize yield an ERRORTOKEN if the identifier starts with a n http://bugs.python.org/issue9712 closed by benjamin.peterson #9715: io doc improvements http://bugs.python.org/issue9715 closed by pitrou #9718: Python Interpreter Crash http://bugs.python.org/issue9718 closed by pitrou #9724: help('nonlocal') missing http://bugs.python.org/issue9724 closed by benjamin.peterson #9726: logging.handlers.RotatingFileHandler: implement "preserve log http://bugs.python.org/issue9726 closed by vinay.sajip #9734: ABC issubclass/isinstance leaks in Python 2 http://bugs.python.org/issue9734 closed by benjamin.peterson #9735: cheatsheet outdated http://bugs.python.org/issue9735 closed by eric.araujo #9737: Del on memoryview crashes CPython http://bugs.python.org/issue9737 closed by pitrou #9744: calling __getattribute__ with wrong instance causes hang up http://bugs.python.org/issue9744 closed by benjamin.peterson #9749: tuple-to-list conversion http://bugs.python.org/issue9749 closed by rhettinger #9753: test_socket.testDup, testFromFd fail on Windows http://bugs.python.org/issue9753 closed by stutzbach #1372650: Cookie and multiple names http://bugs.python.org/issue1372650 closed by BreamoreBoy #1111130: tkSimpleDialog broken on MacOS X (Aqua Tk) http://bugs.python.org/issue1111130 closed by r.david.murray #808164: socket.close() doesn't play well with __del__ http://bugs.python.org/issue808164 closed by stutzbach From victor.stinner at haypocalc.com Fri Sep 3 18:24:56 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 3 Sep 2010 18:24:56 +0200 Subject: [Python-Dev] r84430 - in python/branches/py3k: Include/unicodeobject.h Objects/unicodeobject.c In-Reply-To: <4C80AB48.7040100@egenix.com> References: <20100901234353.468A5EEAAC@mail.python.org> <201009030126.32665.victor.stinner@haypocalc.com> <4C80AB48.7040100@egenix.com> Message-ID: <201009031824.56399.victor.stinner@haypocalc.com> Le vendredi 03 septembre 2010 10:01:12, vous avez ?crit : > > (...) > > About PyUnicode_strdup() (PyUnicode_AsUnicodeCopy): I don't know. It is > > possible to rewrite it in few lines. Why don't you want to add them to > > the public API? For my work, it doesn't matter if it's public or not. > > This function uses PyMem_xxx API, I don't know if a third part library > > would like to rely on PyMem_xxx. > > This will have to be documented (see PEP 384 for some reasoning on > malloc() use in DLLs). It is already documented in unicodeobject.h. > Other than that, ok, let's have them. Ok. r84455 renames PyUnicode_strdup() to PyUnicode_AsUnicodeCopy(), and r84456 document it: ---------- .. cfunction:: Py_UNICODE* PyUnicode_AsUnicodeCopy(PyObject *unicode) Create a copy of a unicode string ending with a nul character. Return *NULL* and raise a :exc:`MemoryError` exception on memory allocation failure, otherwise return a new allocated buffer (use :cfunc:`PyMem_Free` to free the buffer). ---------- Thanks all for your review and advices ;-) -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Fri Sep 3 18:30:32 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 3 Sep 2010 18:30:32 +0200 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: <20100903160745.99A3578130@psf.upfronthosting.co.za> References: <20100903160745.99A3578130@psf.upfronthosting.co.za> Message-ID: <201009031830.32456.victor.stinner@haypocalc.com> Le vendredi 03 septembre 2010 18:07:45, Python tracker a ?crit : > ACTIVITY SUMMARY (2010-08-27 - 2010-09-03) > Python tracker at http://bugs.python.org/ Remember also the buildbot report: http://code.google.com/p/bbreport/wiki/PythonBuildbotReport Eg. there are some "no space left on device" on "x86 XP-5 *" build slaves. -- Victor Stinner http://www.haypocalc.com/ From guido at python.org Fri Sep 3 18:32:22 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 3 Sep 2010 09:32:22 -0700 Subject: [Python-Dev] Buffer protocol for io.BytesIO? In-Reply-To: <20100903125624.646acbb4@pitrou.net> References: <20100902223509.64e62db3@pitrou.net> <20100903105423.471d7382@pitrou.net> <20100903125624.646acbb4@pitrou.net> Message-ID: On Fri, Sep 3, 2010 at 3:56 AM, Antoine Pitrou wrote: > On Fri, 3 Sep 2010 20:44:01 +1000 > Nick Coghlan wrote: >> >> It actually strikes me as a fairly bad thing, so I think you're right >> to distrust it. Better to follow the precedent set with getvalue() and >> require an explicit call to getbuffer(). The fact there are two >> options (immutable copy via getvalue() or mutable accessor via >> getbuffer()) is all the more reason not to make either conversion >> implicit. > > Thank you for the advice. I think getbuffer() it will be :) +1 >> > It could not be resized, but it could be modified (same as what happens >> > with bytearrays today). Actually, the buffer itself would be writable, >> > and allow modifying the BytesIO contents. >> >> You may need to be careful with reads and writes while the buffer is >> exposed (e.g. throwing an exception until the buffer is released >> again). Perhaps the buffer accessor should be a context manager so it >> is easy to enforce prompt release? > > That's an interesting idea. I was planning to return a memoryview > object (in order to hide the intermediate object, and make it really > minimal), so perhaps the context protocol should be enabled on > memoryviews? > > (__enter__ would be a no-op, and __exit__ would release the internal > buffer and render it invalid, a bit like closing a file) So far I'm -0 on this. I'm not sure if it solves a real problem, and I think that if we were to add a way to define the scope of a buffer operation using a with-statement, it should work for all objects that support the buffer protocol (which IIUC means more than just memoryview). I'd be more enthusiastic about a separate context manager to wrap any such object. -- --Guido van Rossum (python.org/~guido) From p.f.moore at gmail.com Fri Sep 3 18:41:26 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 3 Sep 2010 17:41:26 +0100 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: <201009031830.32456.victor.stinner@haypocalc.com> References: <20100903160745.99A3578130@psf.upfronthosting.co.za> <201009031830.32456.victor.stinner@haypocalc.com> Message-ID: On 3 September 2010 17:30, Victor Stinner wrote: > Remember also the buildbot report: > http://code.google.com/p/bbreport/wiki/PythonBuildbotReport > > Eg. there are some "no space left on device" on "x86 XP-5 *" build slaves. Thanks, I wasn't aware of that. I'll look into those issues. Paul. From solipsis at pitrou.net Fri Sep 3 19:03:09 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 3 Sep 2010 19:03:09 +0200 Subject: [Python-Dev] Buffer protocol for io.BytesIO? In-Reply-To: References: <20100902223509.64e62db3@pitrou.net> <20100903105423.471d7382@pitrou.net> <20100903125624.646acbb4@pitrou.net> Message-ID: <20100903190309.14fa9f5f@pitrou.net> On Fri, 3 Sep 2010 09:32:22 -0700 Guido van Rossum wrote: > >> > It could not be resized, but it could be modified (same as what happens > >> > with bytearrays today). Actually, the buffer itself would be writable, > >> > and allow modifying the BytesIO contents. > >> > >> You may need to be careful with reads and writes while the buffer is > >> exposed (e.g. throwing an exception until the buffer is released > >> again). Perhaps the buffer accessor should be a context manager so it > >> is easy to enforce prompt release? > > > > That's an interesting idea. I was planning to return a memoryview > > object (in order to hide the intermediate object, and make it really > > minimal), so perhaps the context protocol should be enabled on > > memoryviews? > > > > (__enter__ would be a no-op, and __exit__ would release the internal > > buffer and render it invalid, a bit like closing a file) > > So far I'm -0 on this. I'm not sure if it solves a real problem, and > I think that if we were to add a way to define the scope of a buffer > operation using a with-statement, it should work for all objects that > support the buffer protocol (which IIUC means more than just > memoryview). I'd be more enthusiastic about a separate context manager > to wrap any such object. Most objects don't have a dedicated Python method to return a buffer, because they implement the buffer API implicitly at the C level, using stack-allocated Py_buffer structs. Therefore, there would be no point in a context manager for these objects (the buffer is released timely by the consuming function). It's only when creating a persistent Python-level wrapper for the Py_buffer struct (that is, a memoryview) that heap memory management issues could come into play. You are right that it's probably not a very important issue. Mutating an object's buffer from several threads isn't recommended anyway. As for resource consumption, what matters is when the original object (which owns the memory area) gets deallocated, and there's little chance that creating an additional memoryview makes a significant difference. Regards Antoine. From guido at python.org Fri Sep 3 19:13:43 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 3 Sep 2010 10:13:43 -0700 Subject: [Python-Dev] Buffer protocol for io.BytesIO? In-Reply-To: <20100903190309.14fa9f5f@pitrou.net> References: <20100902223509.64e62db3@pitrou.net> <20100903105423.471d7382@pitrou.net> <20100903125624.646acbb4@pitrou.net> <20100903190309.14fa9f5f@pitrou.net> Message-ID: On Fri, Sep 3, 2010 at 10:03 AM, Antoine Pitrou wrote: > On Fri, 3 Sep 2010 09:32:22 -0700 > Guido van Rossum wrote: >> >> > It could not be resized, but it could be modified (same as what happens >> >> > with bytearrays today). Actually, the buffer itself would be writable, >> >> > and allow modifying the BytesIO contents. >> >> >> >> You may need to be careful with reads and writes while the buffer is >> >> exposed (e.g. throwing an exception until the buffer is released >> >> again). Perhaps the buffer accessor should be a context manager so it >> >> is easy to enforce prompt release? >> > >> > That's an interesting idea. I was planning to return a memoryview >> > object (in order to hide the intermediate object, and make it really >> > minimal), so perhaps the context protocol should be enabled on >> > memoryviews? >> > >> > (__enter__ would be a no-op, and __exit__ would release the internal >> > buffer and render it invalid, a bit like closing a file) >> >> So far I'm ?-0 on this. I'm not sure if it solves a real problem, and >> I think that if we were to add a way to define the scope of a buffer >> operation using a with-statement, it should work for all objects that >> support the buffer protocol (which IIUC means more than just >> memoryview). I'd be more enthusiastic about a separate context manager >> to wrap any such object. > > Most objects don't have a dedicated Python method to return a buffer, > because they implement the buffer API implicitly at the C level, using > stack-allocated Py_buffer structs. Therefore, there would be no point in > a context manager for these objects (the buffer is released timely by > the consuming function). It's only when creating a persistent > Python-level wrapper for the Py_buffer struct (that is, a memoryview) > that heap memory management issues could come into play. > > You are right that it's probably not a very important issue. Mutating > an object's buffer from several threads isn't recommended anyway. As > for resource consumption, what matters is when the original object > (which owns the memory area) gets deallocated, and there's little > chance that creating an additional memoryview makes a significant > difference. Thanks for the explanation. Still -0. -- --Guido van Rossum (python.org/~guido) From tjreedy at udel.edu Fri Sep 3 19:14:38 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 03 Sep 2010 13:14:38 -0400 Subject: [Python-Dev] Internal counter to debug leaking file descriptors In-Reply-To: References: Message-ID: On 9/3/2010 6:09 AM, Giampaolo Rodol? wrote: >> Of course it would be nice to get access to FD stack so that a >> full filename can also be retrieved in this case. > > On Linux, this can be easily achieved by using /proc. > You can take a look at how this is done in the current development > version of psutil: > http://code.google.com/p/psutil/source/browse/trunk/psutil/_pslinux.py?spec=svn633&r=630#266 > Usage: > >>>> import psutil, os >>>> this_process = psutil.Process(os.getpid()) >>>> f = open('file.ext', 'w') >>>> this_process.get_open_files() > ['/home/giampaolo/svn/psutil/file.ext'] > > Same for sockets, a bunch of lines later: > http://code.google.com/p/psutil/source/browse/trunk/psutil/_pslinux.py?spec=svn633&r=630#284 > >>>> import socket >>>> s = socket.create_connection(('google.com', 80)) >>>> this_process.get_connections() > [connection(family=2, type=1, local_address=('192.168.1.43', 38067), > remote_address=('72.14.234.104', 80), status='ESTABLISHED')] If you can use psutil itself, it has compiled Windows versions for 2.7 and 3.1 https://code.google.com/p/psutil/ -- Terry Jan Reedy From g.rodola at gmail.com Fri Sep 3 20:10:19 2010 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Fri, 3 Sep 2010 20:10:19 +0200 Subject: [Python-Dev] Internal counter to debug leaking file descriptors In-Reply-To: References: Message-ID: The Windows part slipped under my radar. =) Unfortunately the Windows binaries still refer to the current version which doesn't include open files and open connections functionalities. To have those he'll have to get the latest code from svn and compile it with mingw32. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ 2010/9/3 Terry Reedy : > On 9/3/2010 6:09 AM, Giampaolo Rodol? wrote: >>> >>> Of course it would be nice to get access to FD stack so that a >>> full filename can also be retrieved in this case. >> >> On Linux, this can be easily achieved by using /proc. >> You can take a look at how this is done in the current development >> version of psutil: >> >> http://code.google.com/p/psutil/source/browse/trunk/psutil/_pslinux.py?spec=svn633&r=630#266 >> Usage: >> >>>>> import psutil, os >>>>> this_process = psutil.Process(os.getpid()) >>>>> f = open('file.ext', 'w') >>>>> this_process.get_open_files() >> >> ['/home/giampaolo/svn/psutil/file.ext'] >> >> Same for sockets, a bunch of lines later: >> >> http://code.google.com/p/psutil/source/browse/trunk/psutil/_pslinux.py?spec=svn633&r=630#284 >> >>>>> import socket >>>>> s = socket.create_connection(('google.com', 80)) >>>>> this_process.get_connections() >> >> [connection(family=2, type=1, local_address=('192.168.1.43', 38067), >> remote_address=('72.14.234.104', 80), status='ESTABLISHED')] > > If you can use psutil itself, it has compiled Windows versions for 2.7 and > 3.1 > https://code.google.com/p/psutil/ > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com > From g.brandl at gmx.net Fri Sep 3 20:07:54 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 03 Sep 2010 20:07:54 +0200 Subject: [Python-Dev] r84456 - python/branches/py3k/Doc/c-api/unicode.rst In-Reply-To: <20100903162329.A386CEE9BC@mail.python.org> References: <20100903162329.A386CEE9BC@mail.python.org> Message-ID: Please add a "versionadded" tag. Georg Am 03.09.2010 18:23, schrieb victor.stinner: > Author: victor.stinner > Date: Fri Sep 3 18:23:29 2010 > New Revision: 84456 > > Log: > Document PyUnicode_AsUnicodeCopy() > > > Modified: > python/branches/py3k/Doc/c-api/unicode.rst > > Modified: python/branches/py3k/Doc/c-api/unicode.rst > ============================================================================== > --- python/branches/py3k/Doc/c-api/unicode.rst (original) > +++ python/branches/py3k/Doc/c-api/unicode.rst Fri Sep 3 18:23:29 2010 > @@ -335,6 +335,14 @@ > buffer, *NULL* if *unicode* is not a Unicode object. > > > +.. cfunction:: Py_UNICODE* PyUnicode_AsUnicodeCopy(PyObject *unicode) > + > + Create a copy of a unicode string ending with a nul character. Return *NULL* > + and raise a :exc:`MemoryError` exception on memory allocation failure, > + otherwise return a new allocated buffer (use :cfunc:`PyMem_Free` to free the > + buffer). > + > + > .. cfunction:: Py_ssize_t PyUnicode_GetSize(PyObject *unicode) > > Return the length of the Unicode object. -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From barry at python.org Fri Sep 3 20:37:42 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 3 Sep 2010 14:37:42 -0400 Subject: [Python-Dev] PEP 3149 accepted In-Reply-To: References: Message-ID: <20100903143742.63b59013@mission> On Sep 03, 2010, at 09:36 AM, Georg Brandl wrote: >This is to let you all know that PEP 3149 is accepted. > >Benjamin and I decided that on the basis that > >* strong precedent is set with PEP 3147 >* it is not mutually exclusive with PEP 384; should PEP 384 become > widely accepted, its use can fade out again >* it is a strictly optional feature > >Barry promised the code will go in before this weekend's alpha2, so >that distributors can subject the new code to proper testing, should >they choose to use PEP 3149 features. Once again, thanks. Committed in r84458. You'll want to 'make distclean' and reconfigure your py3k tree. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From brett at python.org Fri Sep 3 21:50:14 2010 From: brett at python.org (Brett Cannon) Date: Fri, 3 Sep 2010 12:50:14 -0700 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <37D6B5E2-2DA4-47C4-B773-11CB6F98162E@gmail.com> <20100903111551.660fb0e3@mission> Message-ID: On Fri, Sep 3, 2010 at 08:45, Guido van Rossum wrote: > On Fri, Sep 3, 2010 at 8:15 AM, Barry Warsaw wrote: >> One thing that would help would be for Guido to let us know early on when he'd >> prefer to delegate the decision. > > Hey! I'm still here! :-) > > More to the point, you can assume that I'm happy to have every PEP > decision made by someone else *except* if you see me participate in > the thread. If I don't show up in the thread, you can assume that > either (a) I don't care or know enough about the topic, or (b) I am > confident that it's going in the right direction. I guess the real question comes down to whether you want us to bug you to select the temp dictator or just make a call amongst ourselves? > >> This goes along with identifying the >> ultimate PEP arbiter (UPA? :) as early as possible. > > I prefer acronyms derived from BDFL, like BDFM ("BD for the moment") > or perhaps BDFP ("BD for the PEP"). (BDFM sounds better but BDFP is > more accurate. I'll leave it up to the FLUFL to pick one.) I personally like BDFN ("BD for now") or BDAGW ("BD at Guido's whim"), but this will bikeshed into eternity, so I am happy with the FLUFL being the dictator for the new acronym choice. =) > >>?As Raymond says, it >> should be someone independent of the proposal, but with the interest, time, >> and experience necessary to make an informed decision. ?We might even want to >> capture the arbiter selection in a PEP header (similar to the new Resolution >> header for capturing the final decision reference). >> >> While I agree that we don't want decision by committee, I think we should >> consider a preference for paired arbiters. ?I have the highest respect for all >> the senior developers who would likely make up the pool of PEP deciders, but >> it gave me great confidence to have both Benjamin and Georg decide the fate of >> PEP 3149. ?As someone who might serve a similar role in the future, I would >> value a second person to sanity check my own thoughts on the matter and to >> identify any holes in my understanding (or missed emails :). ?I'd >> say, let's state a preference (not a requirement) for two arbiters for any PEP >> that's not decided by Guido. > > Hm... As long as we can make sure not to pick the same pair all the > time this makes sense. (Not that I have any objections to how the > Georg+Benjamin pair decided PEP 3149 -- to the contrary -- but I think > it would be good to spread the power.) I agree, and a RM as the backup/sanity check would not spread it out. Considering the position is held for 1.5 years (or more) and has in the past been held sequentially by the same person, that wouldn't exactly spread it about. It also limits trying to bring new people into the process as RMs tend to be old-hands and not new blood. Plus we are not about to make the lead on a PEP decision be a new person either. > > But if a pair can't be found I think a single BDFM/BDFP will work too. I agree. I would trust anyone who is given the ability to make a call to listen to reason enough to not necessarily need the sanity check. But a duopoly is not a bad thing overall either. > >> We'd talked before about allowing the RM for the target version to make the >> decision. ?Maybe the RM can serve as that second arbiter when no other obvious >> candidate is available. > > Good fallback plan. As long as it gets spread around and the fallback is not the default, I agree. > >> Raymond, you identified a great set of criteria that the arbiters should use >> to guide them to a decision. ?I'm willing to write up an informational PEP >> that codifies this and any other guidelines we come up with for non-BDFL PEP >> decisions. >> >> Finally a reminder to PEP authors that it is your responsibility to shepherd >> your PEP through the process. ?Don't be a pest, but do keep an eye on the >> release calendar so that you're not scrambling for a snap decision at the last >> minute. ?18 months can go by quickly. :) > > Well, realistically, there's only so much grief that anyone PEP author > can be expected to put up with. I expect that a lot of PEPs won't be > written or will be withdrawn in the face of prolonged discussion. > Early selection of a BDFM (maybe the M can also refer to mentorship) > ought to help in encouraging where encouragement would help -- and of > course sometimes the best thing to do is to encourage the PEP author > to drop the idea, if no consensus is in view (or if the author is > particularly hard-headed). Hopefully PEPs like this will get stopped before they even get checked in. The PEP editors have been sending PEPs back to their authors to share on python-ideas first for a little while now and that seems to have helped make sure the PEPs that do reach us are of sufficient quality. From guido at python.org Fri Sep 3 23:34:26 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 3 Sep 2010 14:34:26 -0700 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <37D6B5E2-2DA4-47C4-B773-11CB6F98162E@gmail.com> <20100903111551.660fb0e3@mission> Message-ID: On Fri, Sep 3, 2010 at 12:50 PM, Brett Cannon wrote: > I guess the real question comes down to whether you want us to bug you > to select the temp dictator or just make a call amongst ourselves? It's okay to bug me only if you can't find or agree on a temp dictator. -- --Guido van Rossum (python.org/~guido) From g.brandl at gmx.net Sat Sep 4 00:02:59 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 04 Sep 2010 00:02:59 +0200 Subject: [Python-Dev] r84430 - in python/branches/py3k: Include/unicodeobject.h Objects/unicodeobject.c In-Reply-To: <201009031824.56399.victor.stinner@haypocalc.com> References: <20100901234353.468A5EEAAC@mail.python.org> <201009030126.32665.victor.stinner@haypocalc.com> <4C80AB48.7040100@egenix.com> <201009031824.56399.victor.stinner@haypocalc.com> Message-ID: Am 03.09.2010 18:24, schrieb Victor Stinner: >> Other than that, ok, let's have them. > > Ok. > > r84455 renames PyUnicode_strdup() to PyUnicode_AsUnicodeCopy(), and r84456 > document it: > > ---------- > ... cfunction:: Py_UNICODE* PyUnicode_AsUnicodeCopy(PyObject *unicode) > > Create a copy of a unicode string ending with a nul character. Return > *NULL* > and raise a :exc:`MemoryError` exception on memory allocation failure, > otherwise return a new allocated buffer (use :cfunc:`PyMem_Free` to free the > buffer). > ---------- > > Thanks all for your review and advices ;-) What about Py_UNICODE_strcat? If it remains, it needs to be documented as well. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From victor.stinner at haypocalc.com Sat Sep 4 00:14:03 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 4 Sep 2010 00:14:03 +0200 Subject: [Python-Dev] r84456 - python/branches/py3k/Doc/c-api/unicode.rst In-Reply-To: References: <20100903162329.A386CEE9BC@mail.python.org> Message-ID: <201009040014.03439.victor.stinner@haypocalc.com> Le vendredi 03 septembre 2010 20:07:54, Georg Brandl a ?crit : > Please add a "versionadded" tag. Oh no, I always forget this one. Done in r84475. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Sep 4 00:14:41 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 4 Sep 2010 00:14:41 +0200 Subject: [Python-Dev] r84430 - in python/branches/py3k: Include/unicodeobject.h Objects/unicodeobject.c In-Reply-To: References: <20100901234353.468A5EEAAC@mail.python.org> <201009031824.56399.victor.stinner@haypocalc.com> Message-ID: <201009040014.41899.victor.stinner@haypocalc.com> Le samedi 04 septembre 2010 00:02:59, Georg Brandl a ?crit : > What about Py_UNICODE_strcat? If it remains, it needs to be documented as > well. There is already an issue for that: http://bugs.python.org/issue8649 -- Victor Stinner http://www.haypocalc.com/ From g.brandl at gmx.net Sat Sep 4 00:28:05 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 04 Sep 2010 00:28:05 +0200 Subject: [Python-Dev] C API doc question In-Reply-To: References: Message-ID: Am 03.09.2010 13:44, schrieb Nick Coghlan: > On Fri, Sep 3, 2010 at 9:19 PM, Amaury Forgeot d'Arc wrote: >> 2010/9/3 Nick Coghlan : >>> Due to the Unicode API discussion, I happened to be looking at the C >>> API docs at http://docs.python.org/dev/c-api/unicode.html#plain-py-unicode >>> and noticed that out of: >>> >>> PyUnicode_FromUnicode >>> PyUnicode_FromStringAndSize >>> PyUnicode_FromString >>> PyUnicode_FromFormat >>> PyUnicode_FromFormatV >>> PyUnicode_FromEncodedObject >>> PyUnicode_FromObject >>> >>> only the first and the last two are flagged in the online docs as >>> returning a new reference. The other 4 are not (but probably should >>> be). >>> >>> However, I can't see anything in the markup which is even causing >>> those "Return value: New reference" markings to appear in the first >>> place, nor any clues in the Documenting Python info. What am I >>> missing? >> >> This information is in the file: >> Doc/data/refcounts.dat >> There is a extension to sphinx that reads this file and generates >> the annotation in the documentation. >> This file is not very well known, even by core developers... > > Nope, never knew it existed until just now :) > > Since it does exist, how hard would it be to get the extension to emit > a warning if a C API function isn't listed in that file, or if a > function listed in that file doesn't appear anywhere in the docs? > Currently it appears to default to reporting no effect on references > if a function isn't mentioned, which is potentially misleading (as the > above examples show, this file completely missed out on the > PyString->PyUnicode rename). > >>From a more wish-list point of view, how hard would it be to modify > the extension to support having this markup inline with the function > documentation rather than having it off in a separate file? Or > addressing Skip's comment by allowing "-0" to indicate cases where > item references are stolen? > > I created http://bugs.python.org/issue9755 to cover these questions. Thanks, I will handle that at some point; I intended to overhaul the refcounting extension anyway, and add data for the missing functions. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From g.brandl at gmx.net Sat Sep 4 00:52:38 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 04 Sep 2010 00:52:38 +0200 Subject: [Python-Dev] 3.2 "What's New" Message-ID: Hi, I'm not sure what the arrangement for the What's New in Python 3.2 document is; especially if either Andrew or Raymond still feel in charge for it. For this weekend's 3.2a2, it would be rather nice to have some more coverage of changes in the document, since it is the main thing people will look at when determining whether to download and test the alpha. You'll have about 1.4 days to commit changes yourself; I'll freeze the branch on early Sunday morning UTC. Thanks, Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From raymond.hettinger at gmail.com Sat Sep 4 01:26:18 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 3 Sep 2010 16:26:18 -0700 Subject: [Python-Dev] 3.2 "What's New" In-Reply-To: References: Message-ID: <8A154EC5-96E0-4459-B953-B49FCE365B61@gmail.com> On Sep 3, 2010, at 3:52 PM, Georg Brandl wrote: > Hi, > > I'm not sure what the arrangement for the What's New in Python 3.2 document > is; especially if either Andrew or Raymond still feel in charge for it. I'm already working on it. Raymond From victor.stinner at haypocalc.com Sat Sep 4 01:40:54 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 4 Sep 2010 01:40:54 +0200 Subject: [Python-Dev] 3.2 "What's New" In-Reply-To: References: Message-ID: <201009040140.54766.victor.stinner@haypocalc.com> Le samedi 04 septembre 2010 00:52:38, Georg Brandl a ?crit : > For this weekend's 3.2a2, it would be rather nice to have some more > coverage of changes in the document, since it is the main thing people > will look at when determining whether to download and test the alpha. About unicode, Python 3.2 has a better support of undecodable strings, especially filenames (PEP 383): - bytes version of os.environ: os.environb and os.getenvb() - better support of undecodable data, eg. in tarfile, pickle, bz2 and subprocess - os.exec*() and subprocess.Popen() accept bytes arguments - mbcs encoding (default Windows encoding) is not more strict, eg. don't encode ? as L in codepage 1252 I have a list of the related issues, do you need it? -- Victor Stinner http://www.haypocalc.com/ From g.brandl at gmx.net Sat Sep 4 08:45:15 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 04 Sep 2010 08:45:15 +0200 Subject: [Python-Dev] 3.2 "What's New" In-Reply-To: <201009040140.54766.victor.stinner@haypocalc.com> References: <201009040140.54766.victor.stinner@haypocalc.com> Message-ID: Am 04.09.2010 01:40, schrieb Victor Stinner: > Le samedi 04 septembre 2010 00:52:38, Georg Brandl a ?crit : >> For this weekend's 3.2a2, it would be rather nice to have some more >> coverage of changes in the document, since it is the main thing people >> will look at when determining whether to download and test the alpha. > > About unicode, Python 3.2 has a better support of undecodable strings, > especially filenames (PEP 383): > > - bytes version of os.environ: os.environb and os.getenvb() > - better support of undecodable data, eg. in tarfile, pickle, bz2 and > subprocess > - os.exec*() and subprocess.Popen() accept bytes arguments > - mbcs encoding (default Windows encoding) is not more strict, eg. don't > encode ? as L in codepage 1252 > > I have a list of the related issues, do you need it? Best send it to Raymond then :) Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From g.brandl at gmx.net Sat Sep 4 08:45:29 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 04 Sep 2010 08:45:29 +0200 Subject: [Python-Dev] 3.2 "What's New" In-Reply-To: <8A154EC5-96E0-4459-B953-B49FCE365B61@gmail.com> References: <8A154EC5-96E0-4459-B953-B49FCE365B61@gmail.com> Message-ID: Am 04.09.2010 01:26, schrieb Raymond Hettinger: > > On Sep 3, 2010, at 3:52 PM, Georg Brandl wrote: > >> Hi, >> >> I'm not sure what the arrangement for the What's New in Python 3.2 document >> is; especially if either Andrew or Raymond still feel in charge for it. > > I'm already working on it. Thanks! Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From stefan_ml at behnel.de Sat Sep 4 10:28:45 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 04 Sep 2010 10:28:45 +0200 Subject: [Python-Dev] new LRU cache API in Py3.2 Message-ID: Hi, reading the description of the new LRU cache in the "What's new in 3.2" document now, I got the impression that the hits/misses attributes and the .clear() method aren't really well namespaced. When I read get_phone_number.clear() it's not very obvious to me what happens, unless I know that there actually *is* a cache involved, which simply has the same name as the function. So this will likely encourage users to add a half-way redundant comment like "clear the cache" to their code. What about adding an intermediate namespace called "cache", so that the new operations are available like this: print get_phone_number.cache.hits get_phone_number.cache.clear() ? It's just a little more overhead, but I think it reads quite a bit better. Stefan From merwok at netwok.org Sat Sep 4 11:42:08 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sat, 04 Sep 2010 11:42:08 +0200 Subject: [Python-Dev] new LRU cache API in Py3.2 In-Reply-To: References: Message-ID: <4C821470.5060603@netwok.org> > What about adding an intermediate namespace called "cache", so that the new > operations are available like this: > > print get_phone_number.cache.hits > get_phone_number.cache.clear() > > ? > > It's just a little more overhead, but I think it reads quite a bit better. Or we could use pseudo-namespacing: get_phone_number.cache_hits, get_phone_number.clear_cache() Regards From solipsis at pitrou.net Sat Sep 4 12:06:11 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 4 Sep 2010 12:06:11 +0200 Subject: [Python-Dev] new LRU cache API in Py3.2 References: <4C821470.5060603@netwok.org> Message-ID: <20100904120611.77fe678e@pitrou.net> On Sat, 04 Sep 2010 11:42:08 +0200 ?ric Araujo wrote: > > What about adding an intermediate namespace called "cache", so that the new > > operations are available like this: > > > > print get_phone_number.cache.hits > > get_phone_number.cache.clear() > > > > ? > > > > It's just a little more overhead, but I think it reads quite a bit better. > > Or we could use pseudo-namespacing: get_phone_number.cache_hits, > get_phone_number.clear_cache() It looks better indeed. Regards Antoine. From g.brandl at gmx.net Sat Sep 4 12:15:07 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 04 Sep 2010 12:15:07 +0200 Subject: [Python-Dev] new LRU cache API in Py3.2 In-Reply-To: <20100904120611.77fe678e@pitrou.net> References: <4C821470.5060603@netwok.org> <20100904120611.77fe678e@pitrou.net> Message-ID: Am 04.09.2010 12:06, schrieb Antoine Pitrou: > On Sat, 04 Sep 2010 11:42:08 +0200 > ?ric Araujo wrote: >> > What about adding an intermediate namespace called "cache", so that the new >> > operations are available like this: >> > >> > print get_phone_number.cache.hits >> > get_phone_number.cache.clear() >> > >> > ? >> > >> > It's just a little more overhead, but I think it reads quite a bit better. >> >> Or we could use pseudo-namespacing: get_phone_number.cache_hits, >> get_phone_number.clear_cache() > > It looks better indeed. +1. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From martin at v.loewis.de Sat Sep 4 15:06:46 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Sep 2010 15:06:46 +0200 Subject: [Python-Dev] PEP 384 status In-Reply-To: <4C7C81EE.8050303@gmail.com> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7C81EE.8050303@gmail.com> Message-ID: <4C824466.2020902@v.loewis.de> > It would be interesting to know how, in practice, these FILE pointers > come to life. In my experience they are generally obtained via fopen. I think that experience can't be generalized. I personally guess that in most cases, the FILE* being passed across CRT boundaries is stdout. > If that is broadly true, then a middle-ground may be for Python to > expose something like Py_fopen, Py_fclose and a PyFILE opaque "handle". > API elements which currently take a FILE * could be exposed using a > PyFILE * in the ABI. People who didn't care about this level of > portability could continue to use the non-ABI FILE * functions, but > people who do could use Py_fopen/Py_fclose in place of fopen/fclose but > otherwise work as now. I don't think this would solve the problem; see above. Regards, Martin From martin at v.loewis.de Sat Sep 4 15:08:21 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Sep 2010 15:08:21 +0200 Subject: [Python-Dev] PEP 384 status In-Reply-To: <4C7D798A.3040200@egenix.com> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D798A.3040200@egenix.com> Message-ID: <4C8244C5.9020401@v.loewis.de> > This sounds like the issues such a mix can cause are mostly > theoretical and don't really bother much in practice, so > PEP 384 on Windows does have a chance :-) Actually, the CRT issues (FILE* in particular) have been causing real crashes for many years, for many people. Regards, Martin From martin at v.loewis.de Sat Sep 4 15:11:31 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Sep 2010 15:11:31 +0200 Subject: [Python-Dev] PEP 384 status In-Reply-To: <20100901004238.72d050ac@pitrou.net> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D80EE.8090705@canterbury.ac.nz> <20100901004238.72d050ac@pitrou.net> Message-ID: <4C824583.5050202@v.loewis.de> > What I think would be a mistake would be to define the API in terms of > Windows-specific quirks. All this discussion seems bent on satisfying > the needs of Windows developers at the expense of non-Windows > developers. "FILE*" is a perfectly standard C feature (and a > widely-used one at that). If Windows doesn't handle it correctly then > it's a Windows issue and we shouldn't annoy other people by refusing > access to that feature. The point of the PEP is specifically to solve Python versioning problems *ON WINDOWS*. It would be pointless if it didn't have that effect. If you think it doesn't affect you, then you can just ignore the stable ABI. > After all, we don't usually try to workaround platform-specific > bugs (not as a low level such as the C API level); at worse, we mention > their existence in the docs. Yes, and that policy is an ongoing annoyance, one that PEP 384 tries to remedy. Only after I wrote the PEP, I learned that it can have uses for Linux distributions, too. Regards, Martin From martin at v.loewis.de Sat Sep 4 15:13:55 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 04 Sep 2010 15:13:55 +0200 Subject: [Python-Dev] PEP 384 status In-Reply-To: <1283345691.3228.11.camel@localhost.localdomain> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D80EE.8090705@canterbury.ac.nz> <20100901004238.72d050ac@pitrou.net> <1283345691.3228.11.camel@localhost.localdomain> Message-ID: <4C824613.3020607@v.loewis.de> > Please consider this: even without relying on PEP 384, using FILE* > is /already/ dangerous; because you might compile an extension with a > different compiler version than Python was compiled with. It's only dangerous *if* you compile with a different compiler. That's why we take serious precautions that you never ever do. For example, distutils will, by default, refuse to use a different compiler. Regards, Martin From martin at v.loewis.de Sat Sep 4 16:00:44 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Sep 2010 16:00:44 +0200 Subject: [Python-Dev] versioned .so files for Python 3.2 In-Reply-To: <4C7CC9ED.9090405@egenix.com> References: <20100624115048.4fd152e3@heresy> <20100714195955.4815f980@heresy> <4C40372C.9030101@ubuntu.com> <20100722164036.7a80d27c@snowdog> <20100724002509.0ad8d359@snowdog> <4C78E4F0.9090306@v.loewis.de> <7BA45755-EF5F-4A4E-9E47-21C9A80C29B4@mac.com> <4C7CC9ED.9090405@egenix.com> Message-ID: <4C82510C.9010405@v.loewis.de> > -1 on always using wchar_t as well. Python's default is UCS2 and the > stable ABI should not change that. It's not really Python's default. It is what configure.in does by default. Python's default, on Linux, is UCS-4. > I also think that this information is not relevant for the stable > ABI: Extensions that want to stick to the stable ABI should really > not have to know whether Py_UNICODE maps to wchar_t or not. If they > want to interface to a local whcar_t type they can use the conversion > APIs we have for that in the Unicode API: PyUnicode_FromWideChar() > and PyUnicode_AsWideChar(). Ok. I'm fine with excluding Py_UNICODE from the stable ABI. However, I think in the long run, I guess more support for wchar_t will then be needed in the API, e.g. more convenient argument parsing. Regards, Martin From solipsis at pitrou.net Sat Sep 4 16:19:57 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 4 Sep 2010 16:19:57 +0200 Subject: [Python-Dev] PEP 384 status In-Reply-To: <4C824613.3020607@v.loewis.de> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D80EE.8090705@canterbury.ac.nz> <20100901004238.72d050ac@pitrou.net> <1283345691.3228.11.camel@localhost.localdomain> <4C824613.3020607@v.loewis.de> Message-ID: <20100904161957.0e6ea628@pitrou.net> On Sat, 04 Sep 2010 15:13:55 +0200 "Martin v. L?wis" wrote: > > Please consider this: even without relying on PEP 384, using FILE* > > is /already/ dangerous; because you might compile an extension with a > > different compiler version than Python was compiled with. > > It's only dangerous *if* you compile with a different compiler. > That's why we take serious precautions that you never ever do. > For example, distutils will, by default, refuse to use a different > compiler. Thanks for the explanations. Regards Antoine. From daniel at stutzbachenterprises.com Sat Sep 4 16:21:25 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Sat, 4 Sep 2010 09:21:25 -0500 Subject: [Python-Dev] new LRU cache API in Py3.2 In-Reply-To: References: Message-ID: On Sat, Sep 4, 2010 at 3:28 AM, Stefan Behnel wrote: > What about adding an intermediate namespace called "cache", so that the new > operations are available like this: > I had been thinking that the lru_cache should be a class (with a dict-like interface), so it can be used explicitly and not just as a decorator. It could provide a wrap() method to be used as a decorator (or implement __call__ to keep the current semantics, but explicit is better than implicit) widget_cache = lru_cache() widget_cache[name] = widget @lru_cache().wrap def get_thingy(name): return something(name) # get_thingy.cache is an lru_cache instance print(get_thingy.cache.hits) I have been using a similar LRU cache class to store items retrieved from a database. In my case, a decorator-paradigm wouldn't work well because I only want to cache a few of the columns from a much larger query, plus there are multiple functions that want to talk to the cache. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Sep 4 20:20:34 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 4 Sep 2010 20:20:34 +0200 Subject: [Python-Dev] new LRU cache API in Py3.2 References: Message-ID: <20100904202034.4df2b475@pitrou.net> On Sat, 4 Sep 2010 09:21:25 -0500 Daniel Stutzbach wrote: > > I have been using a similar LRU cache class to store items retrieved from a > database. In my case, a decorator-paradigm wouldn't work well because I > only want to cache a few of the columns from a much larger query, plus there > are multiple functions that want to talk to the cache. Well, perhaps lru_cache() would have deserved a review before committing? Regards Antoine. From raymond.hettinger at gmail.com Sat Sep 4 23:07:34 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 4 Sep 2010 14:07:34 -0700 Subject: [Python-Dev] new LRU cache API in Py3.2 In-Reply-To: <20100904202034.4df2b475@pitrou.net> References: <20100904202034.4df2b475@pitrou.net> Message-ID: On Sep 4, 2010, at 11:20 AM, Antoine Pitrou wrote: > Well, perhaps lru_cache() would have deserved a review before > committing? Not everything needs to be designed by committee. This API is based on one that was published as a recipe several years ago and has been used in a number of companies. Its design reflects feedback from a variety of advanced python users (the keyword argument support from Miki Tebeka, the concurrency support from Jim Baker, the clearing option and introspectability from Nick Coghlan, etc). Aside from the compact API, the actual implementation is so dirt simple that it will be trivial for folks to roll their own variants if they have more exotic needs. After I'm done with other work for the alpha, I'll take a further look at the suggestions here. Raymond From raymond.hettinger at gmail.com Sat Sep 4 23:11:26 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 4 Sep 2010 14:11:26 -0700 Subject: [Python-Dev] new LRU cache API in Py3.2 In-Reply-To: References: <4C821470.5060603@netwok.org> <20100904120611.77fe678e@pitrou.net> Message-ID: On Sep 4, 2010, at 3:15 AM, Georg Brandl wrote: > Am 04.09.2010 12:06, schrieb Antoine Pitrou: >> On Sat, 04 Sep 2010 11:42:08 +0200 >> ?ric Araujo wrote: >>>> >>>> It's just a little more overhead, but I think it reads quite a bit better. >>> >>> Or we could use pseudo-namespacing: get_phone_number.cache_hits, >>> get_phone_number.clear_cache() >> >> It looks better indeed. > > +1. Those seem like reasonable names. Raymond From solipsis at pitrou.net Sun Sep 5 13:18:30 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 5 Sep 2010 13:18:30 +0200 Subject: [Python-Dev] r84513 - python/branches/py3k/Doc/whatsnew/3.2.rst References: <20100905002725.6F458FBF2@mail.python.org> Message-ID: <20100905131830.6f133375@pitrou.net> On Sun, 5 Sep 2010 02:27:25 +0200 (CEST) raymond.hettinger wrote: > + > +The common directory is "pyshared" and the file names are made distinct by > +identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the > +major and minor version numbers, and optional build flags (such as "d" for > +debug, "m" for pymalloc, "u" for wide-unicode). For an arbtrary package, "foo", > +you may see these files when the distribution package is installed:: > + > + /usr/share/pyshared/foo.cpython-32m.so > + /usr/share/pyshared/foo.cpython-33md.so Does it happen by default? I can't see any trace of "pyshared" in the Python source tree. Antoine. From g.brandl at gmx.net Sun Sep 5 17:40:28 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 05 Sep 2010 17:40:28 +0200 Subject: [Python-Dev] r84513 - python/branches/py3k/Doc/whatsnew/3.2.rst In-Reply-To: <20100905131830.6f133375@pitrou.net> References: <20100905002725.6F458FBF2@mail.python.org> <20100905131830.6f133375@pitrou.net> Message-ID: Am 05.09.2010 13:18, schrieb Antoine Pitrou: > On Sun, 5 Sep 2010 02:27:25 +0200 (CEST) > raymond.hettinger wrote: >> + >> +The common directory is "pyshared" and the file names are made distinct by >> +identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the >> +major and minor version numbers, and optional build flags (such as "d" for >> +debug, "m" for pymalloc, "u" for wide-unicode). For an arbtrary package, "foo", >> +you may see these files when the distribution package is installed:: >> + >> + /usr/share/pyshared/foo.cpython-32m.so >> + /usr/share/pyshared/foo.cpython-33md.so > > Does it happen by default? I can't see any trace of "pyshared" in the > Python source tree. No; it has to be implemented this way by distributors. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From martin at v.loewis.de Sun Sep 5 19:22:02 2010 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 05 Sep 2010 19:22:02 +0200 Subject: [Python-Dev] PEP 3149 thoughts Message-ID: <4C83D1BA.4050505@v.loewis.de> I know the PEP is accepted, but I would still like to see some changes/clarifications. 1. What is the effect of this PEP on Windows? Is this a Linux-only feature? If not, who is going to provide the changes for Windows? (More specifically: if this is indeed meant for Windows, and if no Windows implementation arrives before 3.2b1, I'd ask that the changes be rolled back, and integration is deferred until there is Windows support) 2. Why does the PEP recommend installing stuff into /usr/share/pyshared? According to the Linux FHS, /usr/share is for Architecture- independent data, see http://www.pathname.com/fhs/pub/fhs-2.3.html#USRSHAREARCHITECTUREINDEPENDENTDATA In particular, it's objective is that you can NFS-share it across, say, both SPARC Linux and x86 Linux. I believe the PEP would break this, as SPARC and x86 executables would override each other. 3. When the PEP recommends that stuff gets installed into pyshared, why does the patch then not implement this recommendation, but continues installing files into lib-dynload? Regards, Martin From g.brandl at gmx.net Sun Sep 5 20:16:57 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 05 Sep 2010 20:16:57 +0200 Subject: [Python-Dev] PEP 3149 thoughts In-Reply-To: <4C83D1BA.4050505@v.loewis.de> References: <4C83D1BA.4050505@v.loewis.de> Message-ID: Am 05.09.2010 19:22, schrieb "Martin v. L?wis": > I know the PEP is accepted, but I would still like to see some > changes/clarifications. > > 1. What is the effect of this PEP on Windows? Is this a Linux-only > feature? If not, who is going to provide the changes for Windows? > (More specifically: if this is indeed meant for Windows, and > if no Windows implementation arrives before 3.2b1, I'd ask that > the changes be rolled back, and integration is deferred until there > is Windows support) I don't think Windows support is planned or necessary; after all, isn't the default installation mode on Windows to install every Python version into its own root direction (C:\PythonXY)? > 2. Why does the PEP recommend installing stuff into /usr/share/pyshared? > According to the Linux FHS, /usr/share is for Architecture- > independent data, see > > http://www.pathname.com/fhs/pub/fhs-2.3.html#USRSHAREARCHITECTUREINDEPENDENTDATA > In particular, it's objective is that you can NFS-share it across, > say, both SPARC Linux and x86 Linux. I believe the PEP would break > this, as SPARC and x86 executables would override each other. Indeed. I think this is probably just an oversight and should be corrected in the PEP. However, it's the distributions' call anyway. > 3. When the PEP recommends that stuff gets installed into pyshared, > why does the patch then not implement this recommendation, but > continues installing files into lib-dynload? That section is talking about files installed by distributions, which need to take special steps to get everything into /usr/{lib,share}/pyshared; a standard out-of-the-tarball install will not change the way it is installed. It may well be that Barry had his Ubuntu hat on a bit too firmly when writing that PEP :) Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From p.f.moore at gmail.com Sun Sep 5 21:21:31 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 5 Sep 2010 20:21:31 +0100 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: References: <20100903160745.99A3578130@psf.upfronthosting.co.za> <201009031830.32456.victor.stinner@haypocalc.com> Message-ID: On 3 September 2010 17:41, Paul Moore wrote: > On 3 September 2010 17:30, Victor Stinner wrote: >> Remember also the buildbot report: >> http://code.google.com/p/bbreport/wiki/PythonBuildbotReport >> >> Eg. there are some "no space left on device" on "x86 XP-5 *" build slaves. > > Thanks, I wasn't aware of that. I'll look into those issues. Odd. There's 2GB free on the disk (not a lot, but it's only a 19GB disk and it's had the same amount of free space for ages). I suspect that it's something that has changed triggering an error which is getting reported as space but isn't actually (IIRC, you can get that error from things like broken pipes on Windows). I'll keep digging when I get the chance, but I've not got much free time just now, so it may be a while... Paul. From martin at v.loewis.de Sun Sep 5 23:39:35 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 05 Sep 2010 23:39:35 +0200 Subject: [Python-Dev] PEP 3149 thoughts In-Reply-To: References: <4C83D1BA.4050505@v.loewis.de> Message-ID: <4C840E17.4030105@v.loewis.de> >> 1. What is the effect of this PEP on Windows? Is this a Linux-only >> feature? If not, who is going to provide the changes for Windows? >> (More specifically: if this is indeed meant for Windows, and >> if no Windows implementation arrives before 3.2b1, I'd ask that >> the changes be rolled back, and integration is deferred until there >> is Windows support) > > I don't think Windows support is planned or necessary; after all, isn't the > default installation mode on Windows to install every Python version into > its own root direction (C:\PythonXY)? Yes. People are asking to change that to Program Files\Pythonxy, but that wouldn't change anything wrt. the PEP. Notice, however, that the PEP also talks about creating different names for different compilation options of the same Python binary. This applies to Windows as well (as the PEP actually points out: there is _d.pyd and .pyd). In any case, if the PEP is specific to Unix (or even to Linux?), it should say so (either as a positive list, or a negative one; if negative, it should also say whether it applies to OSX). > That section is talking about files installed by distributions, which need > to take special steps to get everything into /usr/{lib,share}/pyshared; a > standard out-of-the-tarball install will not change the way it is installed. > > It may well be that Barry had his Ubuntu hat on a bit too firmly when writing > that PEP :) I think then that the PEP should better separate what is actually being specified (and apparently, that's just the name of the shared objects), from possible use case scenarios. Regards, Martin From cournape at gmail.com Mon Sep 6 08:40:25 2010 From: cournape at gmail.com (David Cournapeau) Date: Mon, 6 Sep 2010 15:40:25 +0900 Subject: [Python-Dev] PEP 3149 thoughts In-Reply-To: References: <4C83D1BA.4050505@v.loewis.de> Message-ID: On Mon, Sep 6, 2010 at 3:16 AM, Georg Brandl wrote: > Am 05.09.2010 19:22, schrieb "Martin v. L?wis": >> I know the PEP is accepted, but I would still like to see some >> changes/clarifications. >> >> 1. What is the effect of this PEP on Windows? Is this a Linux-only >> ? ?feature? If not, who is going to provide the changes for Windows? >> ? ?(More specifically: if this is indeed meant for Windows, and >> ? ?if no Windows implementation arrives before 3.2b1, I'd ask that >> ? ?the changes be rolled back, and integration is deferred until there >> ? ?is Windows support) > > I don't think Windows support is planned or necessary; after all, isn't the > default installation mode on Windows to install every Python version into > its own root direction (C:\PythonXY)? > >> 2. Why does the PEP recommend installing stuff into /usr/share/pyshared? >> ? ?According to the Linux FHS, /usr/share is for Architecture- >> ? ?independent data, see >> >> http://www.pathname.com/fhs/pub/fhs-2.3.html#USRSHAREARCHITECTUREINDEPENDENTDATA >> ? ?In particular, it's objective is that you can NFS-share it across, >> ? ?say, both SPARC Linux and x86 Linux. I believe the PEP would break >> ? ?this, as SPARC and x86 executables would override each other. > > Indeed. ?I think this is probably just an oversight and should be corrected > in the PEP. ?However, it's the distributions' call anyway. Reading the related paragraph in the PEP, it seems to me that the use of package as in "these distributions install third party (i.e. non-standard library) packages ..." is too vague. On Ubuntu at least, the package content is spread out over different paths, and only *some* files of the package are put into ...pyshared (namely, the ones that can indeed be shared across different versions, that is onlythe .py files in general, with the .so and the .pyc in /usr/lib/...). I guess this is obvious for Barry and other people accustomed with packaging on debian-like systems, but not so much for others. Maybe the PEP would benefit from a stronger example (for example how is a simple package with a C extension actually installed on the system), but OTOH, this keeps changing between debian/ubuntu versions, so a complete example may be more confusing. cheers, David From georg at python.org Mon Sep 6 10:22:17 2010 From: georg at python.org (Georg Brandl) Date: Mon, 06 Sep 2010 10:22:17 +0200 Subject: [Python-Dev] [RELEASED] Python 3.2 alpha 2 Message-ID: <4C84A4B9.2040406@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team, I'm happy to announce the second alpha preview release of Python 3.2. Python 3.2 is a continuation of the efforts to improve and stabilize the Python 3.x line. Since the final release of Python 2.7, the 2.x line will only receive bugfixes, and new features are developed for 3.x only. Since PEP 3003, the Moratorium on Language Changes, is in effect, there are no changes in Python's syntax and built-in types in Python 3.2. Development efforts concentrated on the standard library and support for porting code to Python 3. Highlights are: * numerous improvements to the unittest module * PEP 3147, support for .pyc repository directories * PEP 3149, support for version tagged dynamic libraries * an overhauled GIL implementation that reduces contention * many consistency and behavior fixes for numeric operations * countless fixes regarding string/unicode issues; among them full support for a bytes environment (filenames, environment variables) * a sysconfig module to access configuration information * a pure-Python implementation of the datetime module * additions to the shutil module, among them archive file support * improvements to pdb, the Python debugger For an extensive list of changes in 3.2, see Misc/NEWS in the Python distribution. To download Python 3.2 visit: http://www.python.org/download/releases/3.2/ 3.2 documentation can be found at: http://docs.python.org/3.2/ Please consider trying Python 3.2 with your code and reporting any bugs you may notice to: http://bugs.python.org/ Enjoy! - -- Georg Brandl, Release Manager georg at python.org (on behalf of the entire python-dev team and 3.2's contributors) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) iEYEARECAAYFAkyEpLkACgkQN9GcIYhpnLCzSwCdFyPz1dPEehJZmeW8wDltqkqe /ogAnim1J99qDpeLmcUDTf0YBh1W95vf =x+ee -----END PGP SIGNATURE----- From piotr at debian.org Mon Sep 6 12:08:36 2010 From: piotr at debian.org (Piotr =?utf-8?Q?O=C5=BCarowski?=) Date: Mon, 6 Sep 2010 12:08:36 +0200 Subject: [Python-Dev] PEP 3149 thoughts In-Reply-To: References: <4C83D1BA.4050505@v.loewis.de> Message-ID: <20100906100836.GE27684@piotro.eu> [Georg Brandl, 2010-09-05] > Am 05.09.2010 19:22, schrieb "Martin v. L?wis": > > 2. Why does the PEP recommend installing stuff into /usr/share/pyshared? > > According to the Linux FHS, /usr/share is for Architecture- > > independent data, see > > > > http://www.pathname.com/fhs/pub/fhs-2.3.html#USRSHAREARCHITECTUREINDEPENDENTDATA > > In particular, it's objective is that you can NFS-share it across, > > say, both SPARC Linux and x86 Linux. I believe the PEP would break > > this, as SPARC and x86 executables would override each other. > > Indeed. I think this is probably just an oversight and should be corrected > in the PEP. However, it's the distributions' call anyway. FYI: in Debian we will use /usr/lib/python3/ or even /usr/lib/python3/dist-packages/, as there are many modules with things like: os.path.join(os.path.dirname(__file__), '..', '..', '..', '..', 'share', 'foo') -- Piotr O?arowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 From solipsis at pitrou.net Mon Sep 6 13:45:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 6 Sep 2010 13:45:33 +0200 Subject: [Python-Dev] PEP 384 status References: <4C78DF1A.1060601@v.loewis.de> Message-ID: <20100906134533.57b3712b@pitrou.net> Hello Martin, On Sat, 28 Aug 2010 12:04:10 +0200 "Martin v. L?wis" wrote: > I have now started an initial patch for PEP 384, in the pep-0384 branch. [...] On http://bugs.python.org/issue9778 you elaborated on what the PEP would entail in its current state: ?No, vice versa. The PEP promises that the ABI won't change until Python 4. For any change that might break the ABI, either a backwards-compatible solution needs to be found, or the change be deferred to Python 4.? This sounds like it could be detrimental by blocking desired improvements (the aforementioned issue is a potential example of this). Do you think it would complicate things a lot to version the ABI itself? Actually, PYTHON_API_VERSION is already used as some kind of ABI tag (since it's checked at module load time rather than at compile time). Regards Antoine. From trigves at yahoo.com Mon Sep 6 14:21:17 2010 From: trigves at yahoo.com (Trigve) Date: Mon, 6 Sep 2010 12:21:17 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?=5FPyUnicode=5FNew_crash?= Message-ID: Hi, I've found something strange while using unicode objects in embedded python. Here is a situation. I'm creating a couple of unicode objects and then some exception is thrown. While handling exception I create 2 unicode object. The 1. one is created sucessfully but the second one not. The problem is in _PyUnicode_New() on line (1): if (free_list) { unicode = free_list; /*<- (1)*/ free_list = *(PyUnicodeObject **)unicode; In statement "unicode = free_list;" "unicode" pointer is bad pointer, in fact it's address is 5 bytes "lower" (i.e. should be 0x030bc3ff but is 0x030bc3fa). It looks like this only happens when exception is thrown. Anyone know where could be the problem? I'm using python 3.2.1 on Vista with MSVS 2010 Trigve From mal at egenix.com Mon Sep 6 14:34:39 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 06 Sep 2010 14:34:39 +0200 Subject: [Python-Dev] _PyUnicode_New crash In-Reply-To: References: Message-ID: <4C84DFDF.4050208@egenix.com> Trigve wrote: > Hi, > I've found something strange while using unicode objects in embedded python. > Here is a situation. I'm creating a couple of unicode objects and then some > exception is thrown. While handling exception I create 2 unicode object. The 1. > one is created sucessfully but the second one not. The problem is in > _PyUnicode_New() > on line (1): > > if (free_list) { > unicode = free_list; /*<- (1)*/ > free_list = *(PyUnicodeObject **)unicode; > > In statement "unicode = free_list;" "unicode" pointer is bad pointer, in fact > it's address is 5 bytes "lower" (i.e. should be 0x030bc3ff but is 0x030bc3fa). > It looks like this only happens when exception is thrown. Anyone know where > could be the problem? I'm using python 3.2.1 on Vista with MSVS 2010 Please file a bug report for this. We can then discuss this on the tracker. Thanks -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 06 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2010-08-19: Released mxODBC 3.1.0 http://python.egenix.com/ 2010-09-15: DZUG Tagung, Dresden, Germany 8 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From trigves at yahoo.com Mon Sep 6 15:26:35 2010 From: trigves at yahoo.com (Trigve) Date: Mon, 6 Sep 2010 13:26:35 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?=5FPyUnicode=5FNew_crash?= References: <4C84DFDF.4050208@egenix.com> Message-ID: M.-A. Lemburg egenix.com> writes: > > > Please file a bug report for this. We can then discuss this on the > tracker. Done as http://bugs.python.org/issue9785 . > Thanks Thanks Trigve From techtonik at gmail.com Mon Sep 6 20:06:27 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 6 Sep 2010 21:06:27 +0300 Subject: [Python-Dev] r84388 - in python/branches/py3k/Doc: conf.py using/index.rst In-Reply-To: References: <20100901085716.683C2EE9E8@mail.python.org> Message-ID: On Thu, Sep 2, 2010 at 12:43 AM, Nick Coghlan wrote: >> That title isn't better though, since it doesn't cover the "using/cmdline" >> document which deals with command line options, environment variables >> and the like. >> >> I agree that "Using Python" is not very descriptive though. > > Python Setup and Usage? Using Python Interpreter? -- anatoly t. From solipsis at pitrou.net Tue Sep 7 01:04:02 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Sep 2010 01:04:02 +0200 Subject: [Python-Dev] Write access to hg.python.org Message-ID: <20100907010402.7c7a3001@pitrou.net> Hello, What is needed in order to have write (i.e. push) access to the hg.python.org repositories? What are the URLs (for example for the "benchmarks" repository)? Regards Antoine. From dirkjan at ochtman.nl Tue Sep 7 09:21:38 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Tue, 7 Sep 2010 09:21:38 +0200 Subject: [Python-Dev] Write access to hg.python.org In-Reply-To: <20100907010402.7c7a3001@pitrou.net> References: <20100907010402.7c7a3001@pitrou.net> Message-ID: On Tue, Sep 7, 2010 at 01:04, Antoine Pitrou wrote: > What is needed in order to have write (i.e. push) access to the > hg.python.org repositories? > What are the URLs (for example for the "benchmarks" repository)? IIRC you just need to have your public key on there, and you'll have commit access for everything on hg.p.o. Cheers, Dirkjan From g.brandl at gmx.net Tue Sep 7 10:11:41 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 07 Sep 2010 10:11:41 +0200 Subject: [Python-Dev] Write access to hg.python.org In-Reply-To: References: <20100907010402.7c7a3001@pitrou.net> Message-ID: Am 07.09.2010 09:21, schrieb Dirkjan Ochtman: > On Tue, Sep 7, 2010 at 01:04, Antoine Pitrou wrote: >> What is needed in order to have write (i.e. push) access to the >> hg.python.org repositories? >> What are the URLs (for example for the "benchmarks" repository)? > > IIRC you just need to have your public key on there, and you'll have > commit access for everything on hg.p.o. To be a bit more precise, having a public key on file for SVN commits is enough, and your URI is ssh://hg at hg.python.org/repos/benchmarks (That may change depending on the final setup, of course.) cheers, Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From solipsis at pitrou.net Tue Sep 7 10:29:34 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Sep 2010 10:29:34 +0200 Subject: [Python-Dev] Write access to hg.python.org References: <20100907010402.7c7a3001@pitrou.net> Message-ID: <20100907102934.0fdf96cf@pitrou.net> On Tue, 07 Sep 2010 10:11:41 +0200 Georg Brandl wrote: > Am 07.09.2010 09:21, schrieb Dirkjan Ochtman: > > On Tue, Sep 7, 2010 at 01:04, Antoine Pitrou wrote: > >> What is needed in order to have write (i.e. push) access to the > >> hg.python.org repositories? > >> What are the URLs (for example for the "benchmarks" repository)? > > > > IIRC you just need to have your public key on there, and you'll have > > commit access for everything on hg.p.o. > > To be a bit more precise, having a public key on file for SVN commits is enough, > and your URI is > > ssh://hg at hg.python.org/repos/benchmarks > > (That may change depending on the final setup, of course.) Thank you! cheers Antoine. From dirkjan at ochtman.nl Tue Sep 7 10:29:48 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Tue, 7 Sep 2010 10:29:48 +0200 Subject: [Python-Dev] Write access to hg.python.org In-Reply-To: References: <20100907010402.7c7a3001@pitrou.net> Message-ID: On Tue, Sep 7, 2010 at 10:11, Georg Brandl wrote: > To be a bit more precise, having a public key on file for SVN commits is enough, Not exactly, hg uses a separate keystore, so it might not have some of the newer keys. > and your URI is > > ssh://hg at hg.python.org/repos/benchmarks > > (That may change depending on the final setup, of course.) Yes, I think I'd prefer to just get rid of the /repos/ for the URLs (which makes http and ssh more similar, for one thing). Cheers, Dirkjan From g.brandl at gmx.net Tue Sep 7 11:57:03 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 07 Sep 2010 11:57:03 +0200 Subject: [Python-Dev] Write access to hg.python.org In-Reply-To: References: <20100907010402.7c7a3001@pitrou.net> Message-ID: Am 07.09.2010 10:29, schrieb Dirkjan Ochtman: > On Tue, Sep 7, 2010 at 10:11, Georg Brandl wrote: >> To be a bit more precise, having a public key on file for SVN commits is enough, > > Not exactly, hg uses a separate keystore, so it might not have some of > the newer keys. Oh ok, I thought they were both generated from the keys repo. >> and your URI is >> >> ssh://hg at hg.python.org/repos/benchmarks >> >> (That may change depending on the final setup, of course.) > > Yes, I think I'd prefer to just get rid of the /repos/ for the URLs > (which makes http and ssh more similar, for one thing). Makes sense. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From ncoghlan at gmail.com Tue Sep 7 14:34:18 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 7 Sep 2010 22:34:18 +1000 Subject: [Python-Dev] [Python-checkins] r84559 - python/branches/py3k/Lib/subprocess.py In-Reply-To: <20100906162929.8439CEE9C7@mail.python.org> References: <20100906162929.8439CEE9C7@mail.python.org> Message-ID: On Tue, Sep 7, 2010 at 2:29 AM, brian.curtin wrote: > Author: brian.curtin > Date: Mon Sep ?6 18:29:29 2010 > New Revision: 84559 > > Log: > Fix #8956. ValueError message was only mentioning one signal. > > Rather than list out the three signals (or more over time), the message was > made less specific but still descriptive. > > > > Modified: > ? python/branches/py3k/Lib/subprocess.py > > Modified: python/branches/py3k/Lib/subprocess.py > ============================================================================== > --- python/branches/py3k/Lib/subprocess.py ? ? ?(original) > +++ python/branches/py3k/Lib/subprocess.py ? ? ?Mon Sep ?6 18:29:29 2010 > @@ -983,7 +983,7 @@ > ? ? ? ? ? ? elif sig == signal.CTRL_BREAK_EVENT: > ? ? ? ? ? ? ? ? os.kill(self.pid, signal.CTRL_BREAK_EVENT) > ? ? ? ? ? ? else: > - ? ? ? ? ? ? ? ?raise ValueError("Only SIGTERM is supported on Windows") > + ? ? ? ? ? ? ? ?raise ValueError("Unsupported signal") Would it be worth including the signal number here, to at least give some hint as to exactly which signal was received? Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Tue Sep 7 15:01:17 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 7 Sep 2010 23:01:17 +1000 Subject: [Python-Dev] [Python-checkins] r84562 - in python/branches/py3k: Doc/library/io.rst Lib/_pyio.py Lib/test/test_memoryio.py Misc/NEWS Modules/_io/_iomodule.c Modules/_io/_iomodule.h Modules/_io/bytesio.c In-Reply-To: <20100906184822.26A92FA6F@mail.python.org> References: <20100906184822.26A92FA6F@mail.python.org> Message-ID: On Tue, Sep 7, 2010 at 4:48 AM, antoine.pitrou wrote: > Modified: python/branches/py3k/Lib/test/test_memoryio.py > ============================================================================== > --- python/branches/py3k/Lib/test/test_memoryio.py ? ? ?(original) > +++ python/branches/py3k/Lib/test/test_memoryio.py ? ? ?Mon Sep ?6 20:48:21 2010 > @@ -384,7 +384,31 @@ > ? ? ? ? del __main__.PickleTestMemIO > > > -class PyBytesIOTest(MemoryTestMixin, MemorySeekTestMixin, unittest.TestCase): > +class BytesIOMixin: > + > + ? ?def test_getbuffer(self): > + ? ? ? ?memio = self.ioclass(b"1234567890") > + ? ? ? ?buf = memio.getbuffer() > + ? ? ? ?self.assertEqual(bytes(buf), b"1234567890") > + ? ? ? ?memio.seek(5) > + ? ? ? ?buf = memio.getbuffer() > + ? ? ? ?self.assertEqual(bytes(buf), b"1234567890") > + ? ? ? ?# Trying to change the size of the BytesIO while a buffer is exported > + ? ? ? ?# raises a BufferError. > + ? ? ? ?self.assertRaises(BufferError, memio.write, b'x' * 100) > + ? ? ? ?self.assertRaises(BufferError, memio.truncate) > + ? ? ? ?# Mutating the buffer updates the BytesIO > + ? ? ? ?buf[3:6] = b"abc" > + ? ? ? ?self.assertEqual(bytes(buf), b"123abc7890") > + ? ? ? ?self.assertEqual(memio.getvalue(), b"123abc7890") > + ? ? ? ?# After the buffer gets released, we can resize the BytesIO again > + ? ? ? ?del buf > + ? ? ? ?support.gc_collect() > + ? ? ? ?memio.truncate() I've raised an RFE (http://bugs.python.org/issue9789) to point out that the need for that GC collect call in there to make the test portable to other implementations is rather ugly and supporting an explicit "buf.release()" call may be a nicer option. (And added Guido to the nosy list, since he wasn't keen on supporting the context management protocol idea, but I don't believe he said anything one way or the other about an ordinary method). > +class PyBytesIOTest(MemoryTestMixin, MemorySeekTestMixin, > + ? ? ? ? ? ? ? ? ? ?BytesIOMixin, unittest.TestCase): I was going to ask why CBytesIOTest wasn't affected, but checking the full source of the test file made everything clear :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From brian.curtin at gmail.com Tue Sep 7 15:05:21 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 7 Sep 2010 08:05:21 -0500 Subject: [Python-Dev] [Python-checkins] r84559 - python/branches/py3k/Lib/subprocess.py In-Reply-To: References: <20100906162929.8439CEE9C7@mail.python.org> Message-ID: On Tue, Sep 7, 2010 at 07:34, Nick Coghlan wrote: > On Tue, Sep 7, 2010 at 2:29 AM, brian.curtin > wrote: > > Author: brian.curtin > > Date: Mon Sep 6 18:29:29 2010 > > New Revision: 84559 > > > > Log: > > Fix #8956. ValueError message was only mentioning one signal. > > > > Rather than list out the three signals (or more over time), the message > was > > made less specific but still descriptive. > > > > > > > > Modified: > > python/branches/py3k/Lib/subprocess.py > > > > Modified: python/branches/py3k/Lib/subprocess.py > > > ============================================================================== > > --- python/branches/py3k/Lib/subprocess.py (original) > > +++ python/branches/py3k/Lib/subprocess.py Mon Sep 6 18:29:29 2010 > > @@ -983,7 +983,7 @@ > > elif sig == signal.CTRL_BREAK_EVENT: > > os.kill(self.pid, signal.CTRL_BREAK_EVENT) > > else: > > - raise ValueError("Only SIGTERM is supported on Windows") > > + raise ValueError("Unsupported signal") > > Would it be worth including the signal number here, to at least give > some hint as to exactly which signal was received? > > Cheers, > Nick. Sure, seems reasonable to me. Does """raise ValueError("Unsupported signal: {}".format(sig))""" look fine, or is there a more preferred format when displaying bad values in exception messages? -------------- next part -------------- An HTML attachment was scrubbed... URL: From contactprashantat at gmail.com Tue Sep 7 15:06:10 2010 From: contactprashantat at gmail.com (Prashant Kumar) Date: Tue, 7 Sep 2010 18:36:10 +0530 Subject: [Python-Dev] Volunteer help with porting Message-ID: Hi everyone, My name is Prashant Kumar and I wish to contribute to the Python development process by helping convert certain existing python over to python3k. Is there anyway I could obtain a list of libraries which need to be ported over to python3k, sorted by importance(by importance i mean packages which serve as a dependency for larger number of packages being more important). Thanks, Prashant Kumar -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Sep 7 15:09:34 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Sep 2010 15:09:34 +0200 Subject: [Python-Dev] r84562 - in python/branches/py3k: Doc/library/io.rst Lib/_pyio.py Lib/test/test_memoryio.py Misc/NEWS Modules/_io/_iomodule.c Modules/_io/_iomodule.h Modules/_io/bytesio.c References: <20100906184822.26A92FA6F@mail.python.org> Message-ID: <20100907150934.7a29043c@pitrou.net> On Tue, 7 Sep 2010 23:01:17 +1000 Nick Coghlan wrote: > > + ? ? ? ?# After the buffer gets released, we can resize the BytesIO again > > + ? ? ? ?del buf > > + ? ? ? ?support.gc_collect() > > + ? ? ? ?memio.truncate() > > I've raised an RFE (http://bugs.python.org/issue9789) to point out > that the need for that GC collect call in there to make the test > portable to other implementations is rather ugly and supporting an > explicit "buf.release()" call may be a nicer option. There was already an issue open for the context management option: http://bugs.python.org/issue9757 Regards Antoine. From ncoghlan at gmail.com Tue Sep 7 15:12:33 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 7 Sep 2010 23:12:33 +1000 Subject: [Python-Dev] [Python-checkins] r84564 - in python/branches/py3k/Lib: ntpath.py test/test_ntpath.py In-Reply-To: <20100906194617.4ECACEE9B1@mail.python.org> References: <20100906194617.4ECACEE9B1@mail.python.org> Message-ID: On Tue, Sep 7, 2010 at 5:46 AM, brian.curtin wrote: > Modified: python/branches/py3k/Lib/ntpath.py > ============================================================================== > --- python/branches/py3k/Lib/ntpath.py ?(original) > +++ python/branches/py3k/Lib/ntpath.py ?Mon Sep ?6 21:46:17 2010 > @@ -10,7 +10,6 @@ > ?import stat > ?import genericpath > ?from genericpath import * > -from nt import _getfileinformation > > ?__all__ = ["normcase","isabs","join","splitdrive","split","splitext", > ? ? ? ? ? ?"basename","dirname","commonprefix","getsize","getmtime", > @@ -656,4 +655,10 @@ > > ?def sameopenfile(f1, f2): > ? ? """Test whether two file objects reference the same file""" > - ? ?return _getfileinformation(f1) == _getfileinformation(f2) > + ? ?try: > + ? ? ? ?from nt import _getfileinformation > + ? ? ? ?return _getfileinformation(f1) == _getfileinformation(f2) > + ? ?except ImportError: > + ? ? ? ?# On other operating systems, return True if the file descriptors > + ? ? ? ?# are the same. > + ? ? ? ?return f1 == f2 Given the potential deadlock problems with imports inside functions, I'd prefer to see this written as either: try: from nt import _getfileinformation def sameopenfile(f1, f2): return _getfileinformation(f1) == _getfileinformation(f2) except ImportError: # On other operating systems, return True if the file descriptors # are the same. def sameopenfile(f1, f2): return f1 == f2 sameopenfile.__doc__ = "Test whether two file objects reference the same file" or as: try: from nt import _getfileinformation except ImportError: # On other operating systems, file comparison is by file descriptor anyway # so a separate file information object is unnecessary def _getfileinformation(f): return f def sameopenfile(f1, f2): """Test whether two file objects reference the same file""" return _getfileinformation(f1) == _getfileinformation(f2) The latter is cleaner code, while the former is likely an unnecessary micro-optimisation. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Tue Sep 7 15:19:47 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 7 Sep 2010 23:19:47 +1000 Subject: [Python-Dev] [Python-checkins] r84559 - python/branches/py3k/Lib/subprocess.py In-Reply-To: References: <20100906162929.8439CEE9C7@mail.python.org> Message-ID: On Tue, Sep 7, 2010 at 11:05 PM, Brian Curtin wrote: > Sure, seems reasonable to me. > Does """raise ValueError("Unsupported signal: {}".format(sig))""" look fine, > or is there a more preferred format when displaying bad values in exception > messages? No, that's about what I was thinking as well. When all we have is an error code (or similar number), it's difficult to make the associated error message particularly pretty. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From brian.curtin at gmail.com Tue Sep 7 15:20:21 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 7 Sep 2010 08:20:21 -0500 Subject: [Python-Dev] [Python-checkins] r84564 - in python/branches/py3k/Lib: ntpath.py test/test_ntpath.py In-Reply-To: References: <20100906194617.4ECACEE9B1@mail.python.org> Message-ID: On Tue, Sep 7, 2010 at 08:12, Nick Coghlan wrote: > On Tue, Sep 7, 2010 at 5:46 AM, brian.curtin > wrote: > > Modified: python/branches/py3k/Lib/ntpath.py > > > ============================================================================== > > --- python/branches/py3k/Lib/ntpath.py (original) > > +++ python/branches/py3k/Lib/ntpath.py Mon Sep 6 21:46:17 2010 > > @@ -10,7 +10,6 @@ > > import stat > > import genericpath > > from genericpath import * > > -from nt import _getfileinformation > > > > __all__ = ["normcase","isabs","join","splitdrive","split","splitext", > > "basename","dirname","commonprefix","getsize","getmtime", > > @@ -656,4 +655,10 @@ > > > > def sameopenfile(f1, f2): > > """Test whether two file objects reference the same file""" > > - return _getfileinformation(f1) == _getfileinformation(f2) > > + try: > > + from nt import _getfileinformation > > + return _getfileinformation(f1) == _getfileinformation(f2) > > + except ImportError: > > + # On other operating systems, return True if the file > descriptors > > + # are the same. > > + return f1 == f2 > > Given the potential deadlock problems with imports inside functions, > I'd prefer to see this written as either: > > try: > from nt import _getfileinformation > def sameopenfile(f1, f2): > return _getfileinformation(f1) == _getfileinformation(f2) > except ImportError: > # On other operating systems, return True if the file descriptors > # are the same. > def sameopenfile(f1, f2): > return f1 == f2 > sameopenfile.__doc__ = "Test whether two file objects reference the same > file" > > or as: > > try: > from nt import _getfileinformation > except ImportError: > # On other operating systems, file comparison is by file descriptor anyway > # so a separate file information object is unnecessary > def _getfileinformation(f): return f > > def sameopenfile(f1, f2): > """Test whether two file objects reference the same file""" > return _getfileinformation(f1) == _getfileinformation(f2) > > The latter is cleaner code, while the former is likely an unnecessary > micro-optimisation. > > Cheers, > Nick. Similar idea(s) would also apply to the function right above that, samefile. I'll create a new issue for cleaning this up. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.curtin at gmail.com Tue Sep 7 15:28:40 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 7 Sep 2010 08:28:40 -0500 Subject: [Python-Dev] [Python-checkins] r84559 - python/branches/py3k/Lib/subprocess.py In-Reply-To: References: <20100906162929.8439CEE9C7@mail.python.org> Message-ID: On Tue, Sep 7, 2010 at 08:19, Nick Coghlan wrote: > On Tue, Sep 7, 2010 at 11:05 PM, Brian Curtin > wrote: > > Sure, seems reasonable to me. > > Does """raise ValueError("Unsupported signal: {}".format(sig))""" look > fine, > > or is there a more preferred format when displaying bad values in > exception > > messages? > > No, that's about what I was thinking as well. When all we have is an > error code (or similar number), it's difficult to make the associated > error message particularly pretty. > > Cheers, > Nick. Made the adjustment in r84582 (py3k) and r84583 (release27-maint). Thanks for the suggestion. -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phd.pp.ru Tue Sep 7 15:33:30 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Tue, 7 Sep 2010 17:33:30 +0400 Subject: [Python-Dev] Volunteer help with porting In-Reply-To: References: Message-ID: <20100907133330.GA32192@phd.pp.ru> Hello. Thank you for the offer! On Tue, Sep 07, 2010 at 06:36:10PM +0530, Prashant Kumar wrote: > My name is Prashant Kumar and I wish to contribute to the Python development > process by helping convert certain existing python > over to python3k. > > Is there anyway I could obtain a list of libraries which need to be ported > over to python3k, sorted by importance(by importance i mean > packages which serve as a dependency for larger number of packages being > more important). As there is already Python 3.2 alpha, the core of Python has already been ported - and this mailing list is for discussion of development of the very Python, not about working on 3rd-party libraries. I don't know if there are "near core" unported libraries that the core team would want to be ported. Hence the question belongs rather to python generic news groups/mailing lists/fora. See http://www.python.org/community/ . As for my personal preferences - I would like to see these ported: -- database drivers, especially MySQL-python and psycopg; -- GUI frameworks, especially wxPython. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From solipsis at pitrou.net Tue Sep 7 15:40:19 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Sep 2010 15:40:19 +0200 Subject: [Python-Dev] random guarantees References: <20100907003815.6BB04EE98A@mail.python.org> Message-ID: <20100907154019.515c1dc4@pitrou.net> On Tue, 7 Sep 2010 02:38:15 +0200 (CEST) raymond.hettinger wrote: > Author: raymond.hettinger > Date: Tue Sep 7 02:38:15 2010 > New Revision: 84574 > > Log: > Document which part of the random module module are guaranteed. test_random fails here: ====================================================================== FAIL: test_guaranteed_stable (test.test_random.MersenneTwister_TestBasicOps) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/py3k/Lib/test/test_random.py", line 235, in test_guaranteed_stable '0x1.85ff833f71576p-1', '0x1.87efb37462927p-1']) AssertionError: Lists differ: ['0x1.7fafd5169cc8fp-1', '0x1.... != ['0x1.9ee265c177cdep-2', '0x1.... First differing element 0: 0x1.7fafd5169cc8fp-1 0x1.9ee265c177cdep-2 - ['0x1.7fafd5169cc8fp-1', - '0x1.df66035cce42ap-2', - '0x1.87773588f902cp-3', - '0x1.0b32eb7b2fd14p-3'] + ['0x1.9ee265c177cdep-2', + '0x1.bad51092e3c25p-1', + '0x1.85ff833f71576p-1', + '0x1.87efb37462927p-1'] It looks like the guarantees might be too ambitious. Regards Antoine. From ncoghlan at gmail.com Tue Sep 7 15:41:40 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 7 Sep 2010 23:41:40 +1000 Subject: [Python-Dev] r84562 - in python/branches/py3k: Doc/library/io.rst Lib/_pyio.py Lib/test/test_memoryio.py Misc/NEWS Modules/_io/_iomodule.c Modules/_io/_iomodule.h Modules/_io/bytesio.c In-Reply-To: <20100907150934.7a29043c@pitrou.net> References: <20100906184822.26A92FA6F@mail.python.org> <20100907150934.7a29043c@pitrou.net> Message-ID: On Tue, Sep 7, 2010 at 11:09 PM, Antoine Pitrou wrote: > On Tue, 7 Sep 2010 23:01:17 +1000 > Nick Coghlan wrote: >> > + ? ? ? ?# After the buffer gets released, we can resize the BytesIO again >> > + ? ? ? ?del buf >> > + ? ? ? ?support.gc_collect() >> > + ? ? ? ?memio.truncate() >> >> I've raised an RFE (http://bugs.python.org/issue9789) to point out >> that the need for that GC collect call in there to make the test >> portable to other implementations is rather ugly and supporting an >> explicit "buf.release()" call may be a nicer option. > > There was already an issue open for the context management option: > http://bugs.python.org/issue9757 Yeah, I didn't even think to check if you'd already put that up there. I've now closed 9789 as a duplicate of that issue. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From exarkun at twistedmatrix.com Tue Sep 7 16:02:59 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Tue, 07 Sep 2010 14:02:59 -0000 Subject: [Python-Dev] Volunteer help with porting In-Reply-To: <20100907133330.GA32192@phd.pp.ru> References: <20100907133330.GA32192@phd.pp.ru> Message-ID: <20100907140259.2058.1140393887.divmod.xquotient.453@localhost.localdomain> On 01:33 pm, phd at phd.pp.ru wrote: >Hello. Thank you for the offer! > >On Tue, Sep 07, 2010 at 06:36:10PM +0530, Prashant Kumar wrote: >>My name is Prashant Kumar and I wish to contribute to the Python >>development >>process by helping convert certain existing python >>over to python3k. >> >>Is there anyway I could obtain a list of libraries which need to be >>ported >>over to python3k, sorted by importance(by importance i mean >>packages which serve as a dependency for larger number of packages >>being >>more important). > > As there is already Python 3.2 alpha, the core of Python has already >been ported - and this mailing list is for discussion of development of >the >very Python, not about working on 3rd-party libraries. How about the email package? Jean-Paul From solipsis at pitrou.net Tue Sep 7 16:07:48 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Sep 2010 16:07:48 +0200 Subject: [Python-Dev] r84576 - in python/branches/py3k: Doc/library/random.rst Lib/random.py Lib/test/test_random.py Misc/NEWS References: <20100907044452.5B0EEFAAC@mail.python.org> Message-ID: <20100907160748.7c14ee34@pitrou.net> On Tue, 7 Sep 2010 06:44:52 +0200 (CEST) raymond.hettinger wrote: > Author: raymond.hettinger > Date: Tue Sep 7 06:44:52 2010 > New Revision: 84576 > > Log: > Issues #7889, #9025 and #9379: Improvements to the random module. This broke test_generators here: [1/1] test_generators ********************************************************************** File "/home/antoine/py3k/py3k/Lib/test/test_generators.py", line ?, in test.test_generators.__test__.email Failed example: while 1: for s in sets: print(" %s->%s" % (s, s.find()), end='') print() if len(roots) > 1: s1 = gen.choice(roots) roots.remove(s1) s2 = gen.choice(roots) s1.union(s2) print("merged", s1, "into", s2) else: break Expected: A->A B->B C->C D->D E->E F->F G->G H->H I->I J->J K->K L->L M->M merged I into A A->A B->B C->C D->D E->E F->F G->G H->H I->A J->J K->K L->L M->M merged D into C A->A B->B C->C D->C E->E F->F G->G H->H I->A J->J K->K L->L M->M merged K into H A->A B->B C->C D->C E->E F->F G->G H->H I->A J->J K->H L->L M->M merged L into A A->A B->B C->C D->C E->E F->F G->G H->H I->A J->J K->H L->A M->M merged E into A A->A B->B C->C D->C E->A F->F G->G H->H I->A J->J K->H L->A M->M merged B into G A->A B->G C->C D->C E->A F->F G->G H->H I->A J->J K->H L->A M->M merged A into F A->F B->G C->C D->C E->F F->F G->G H->H I->F J->J K->H L->F M->M merged H into G A->F B->G C->C D->C E->F F->F G->G H->G I->F J->J K->G L->F M->M merged F into J A->J B->G C->C D->C E->J F->J G->G H->G I->J J->J K->G L->J M->M merged M into C A->J B->G C->C D->C E->J F->J G->G H->G I->J J->J K->G L->J M->C merged J into G A->G B->G C->C D->C E->G F->G G->G H->G I->G J->G K->G L->G M->C merged C into G A->G B->G C->G D->G E->G F->G G->G H->G I->G J->G K->G L->G M->G Got: A->A B->B C->C D->D E->E F->F G->G H->H I->I J->J K->K L->L M->M merged K into B A->A B->B C->C D->D E->E F->F G->G H->H I->I J->J K->B L->L M->M merged A into F A->F B->B C->C D->D E->E F->F G->G H->H I->I J->J K->B L->L M->M merged E into F A->F B->B C->C D->D E->F F->F G->G H->H I->I J->J K->B L->L M->M merged D into C A->F B->B C->C D->C E->F F->F G->G H->H I->I J->J K->B L->L M->M merged M into C A->F B->B C->C D->C E->F F->F G->G H->H I->I J->J K->B L->L M->C merged J into B A->F B->B C->C D->C E->F F->F G->G H->H I->I J->B K->B L->L M->C merged B into C A->F B->C C->C D->C E->F F->F G->G H->H I->I J->C K->C L->L M->C merged F into G A->G B->C C->C D->C E->G F->G G->G H->H I->I J->C K->C L->L M->C merged L into C A->G B->C C->C D->C E->G F->G G->G H->H I->I J->C K->C L->C M->C merged G into I A->I B->C C->C D->C E->I F->I G->I H->H I->I J->C K->C L->C M->C merged I into H A->H B->C C->C D->C E->H F->H G->H H->H I->H J->C K->C L->C M->C merged C into H A->H B->H C->H D->H E->H F->H G->H H->H I->H J->H K->H L->H M->H ********************************************************************** 1 items had failures: 1 of 31 in test.test_generators.__test__.email ***Test Failed*** 1 failures. test test_generators failed -- 1 of 286 doctests failed From ncoghlan at gmail.com Tue Sep 7 16:09:31 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Sep 2010 00:09:31 +1000 Subject: [Python-Dev] Volunteer help with porting In-Reply-To: <20100907133330.GA32192@phd.pp.ru> References: <20100907133330.GA32192@phd.pp.ru> Message-ID: On Tue, Sep 7, 2010 at 11:33 PM, Oleg Broytman wrote: > -- GUI frameworks, especially wxPython. That would be very cool, but the practicality of it will depend on how current the version of SWIG used in wxPython's build process happens to be. A version of wxPython built around SWIG's -py3 option could be quite interesting. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From fuzzyman at voidspace.org.uk Tue Sep 7 16:15:58 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 07 Sep 2010 15:15:58 +0100 Subject: [Python-Dev] Volunteer help with porting In-Reply-To: <20100907140259.2058.1140393887.divmod.xquotient.453@localhost.localdomain> References: <20100907133330.GA32192@phd.pp.ru> <20100907140259.2058.1140393887.divmod.xquotient.453@localhost.localdomain> Message-ID: <4C86491E.3090000@voidspace.org.uk> On 07/09/2010 15:02, exarkun at twistedmatrix.com wrote: > On 01:33 pm, phd at phd.pp.ru wrote: >> Hello. Thank you for the offer! >> >> On Tue, Sep 07, 2010 at 06:36:10PM +0530, Prashant Kumar wrote: >>> My name is Prashant Kumar and I wish to contribute to the Python >>> development >>> process by helping convert certain existing python >>> over to python3k. >>> >>> Is there anyway I could obtain a list of libraries which need to be >>> ported >>> over to python3k, sorted by importance(by importance i mean >>> packages which serve as a dependency for larger number of packages >>> being >>> more important). >> >> As there is already Python 3.2 alpha, the core of Python has already >> been ported - and this mailing list is for discussion of development >> of the >> very Python, not about working on 3rd-party libraries. > > How about the email package? Right, and there are other standard library modules (cgi, ftplib, nntplib, etc) that either need fixing or auditing as to how they handle bytes / strings. Michael > > Jean-Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From solipsis at pitrou.net Tue Sep 7 16:28:54 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Sep 2010 16:28:54 +0200 Subject: [Python-Dev] Volunteer help with porting References: Message-ID: <20100907162854.40db454c@pitrou.net> Hello, > My name is Prashant Kumar and I wish to contribute to the Python development > process by helping convert certain existing python > over to python3k. > > Is there anyway I could obtain a list of libraries which need to be ported > over to python3k, sorted by importance(by importance i mean > packages which serve as a dependency for larger number of packages being > more important). I think you should ask on comp.lang.python for people's suggestions. Also, be aware that some libraries might already be in the process of being ported. If you consider working on a library, it is probably good practice to first ask its authors about the status of py3k porting. I hope you'll find a place where you can contribute. Regards Antoine. From phd at phd.pp.ru Tue Sep 7 16:34:49 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Tue, 7 Sep 2010 18:34:49 +0400 Subject: [Python-Dev] Volunteer help with porting In-Reply-To: <20100907140259.2058.1140393887.divmod.xquotient.453@localhost.localdomain> References: <20100907133330.GA32192@phd.pp.ru> <20100907140259.2058.1140393887.divmod.xquotient.453@localhost.localdomain> Message-ID: <20100907143449.GA9568@phd.pp.ru> On Tue, Sep 07, 2010 at 02:02:59PM -0000, exarkun at twistedmatrix.com wrote: > On 01:33 pm, phd at phd.pp.ru wrote: >> As there is already Python 3.2 alpha, the core of Python has already >> been ported > > How about the email package? What about email? It is a core library, right? It has been ported AFAIK. Or have I missed something? Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From solipsis at pitrou.net Tue Sep 7 16:38:46 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Sep 2010 16:38:46 +0200 Subject: [Python-Dev] Push notification References: <20100907010402.7c7a3001@pitrou.net> Message-ID: <20100907163846.4bdf3cc1@pitrou.net> On Tue, 7 Sep 2010 10:29:48 +0200 Dirkjan Ochtman wrote: > > and your URI is > > > > ssh://hg at hg.python.org/repos/benchmarks > > > > (That may change depending on the final setup, of course.) > > Yes, I think I'd prefer to just get rid of the /repos/ for the URLs > (which makes http and ssh more similar, for one thing). Could push notification be added for the benchmarks repo? I think the python-checkins list would be an appropriate recipient for the e-mails (the repo has a low activity). Thanks Antoine. From contactprashantat at gmail.com Tue Sep 7 16:42:26 2010 From: contactprashantat at gmail.com (Prashant Kumar) Date: Tue, 7 Sep 2010 20:12:26 +0530 Subject: [Python-Dev] Volunteer help with porting In-Reply-To: <4C86491E.3090000@voidspace.org.uk> References: <20100907133330.GA32192@phd.pp.ru> <20100907140259.2058.1140393887.divmod.xquotient.453@localhost.localdomain> <4C86491E.3090000@voidspace.org.uk> Message-ID: On Tue, Sep 7, 2010 at 7:45 PM, Michael Foord wrote: > On 07/09/2010 15:02, exarkun at twistedmatrix.com wrote: > >> On 01:33 pm, phd at phd.pp.ru wrote: >> >>> Hello. Thank you for the offer! >>> >>> On Tue, Sep 07, 2010 at 06:36:10PM +0530, Prashant Kumar wrote: >>> >>>> My name is Prashant Kumar and I wish to contribute to the Python >>>> development >>>> process by helping convert certain existing python >>>> over to python3k. >>>> >>>> Is there anyway I could obtain a list of libraries which need to be >>>> ported >>>> over to python3k, sorted by importance(by importance i mean >>>> packages which serve as a dependency for larger number of packages being >>>> more important). >>>> >>> >>> As there is already Python 3.2 alpha, the core of Python has already >>> been ported - and this mailing list is for discussion of development of >>> the >>> very Python, not about working on 3rd-party libraries. >>> >> >> How about the email package? >> > > Right, and there are other standard library modules (cgi, ftplib, nntplib, > etc) that either need fixing or auditing as to how they handle bytes / > strings. > Sure I will look into this. Could you please point me towards the repository(I'd love it if I could use mercurial for the development process rather than svn)?. pk > > Michael > > >> Jean-Paul >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk >> > > > -- > http://www.ironpythoninaction.com/ > http://www.voidspace.org.uk/blog > > READ CAREFULLY. By accepting and reading this email you agree, on behalf of > your employer, to release me from all obligations and waivers arising from > any and all NON-NEGOTIATED agreements, licenses, terms-of-service, > shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, > non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have > entered into with your employer, its partners, licensors, agents and > assigns, in perpetuity, without prejudice to my ongoing rights and > privileges. You further represent that you have the authority to release me > from any BOGUS AGREEMENTS on behalf of your employer. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/contactprashantat%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From exarkun at twistedmatrix.com Tue Sep 7 16:58:41 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Tue, 07 Sep 2010 14:58:41 -0000 Subject: [Python-Dev] Volunteer help with porting In-Reply-To: <20100907143449.GA9568@phd.pp.ru> References: <20100907133330.GA32192@phd.pp.ru> <20100907140259.2058.1140393887.divmod.xquotient.453@localhost.localdomain> <20100907143449.GA9568@phd.pp.ru> Message-ID: <20100907145841.2058.1687067386.divmod.xquotient.458@localhost.localdomain> On 02:34 pm, phd at phd.pp.ru wrote: >On Tue, Sep 07, 2010 at 02:02:59PM -0000, exarkun at twistedmatrix.com >wrote: >>On 01:33 pm, phd at phd.pp.ru wrote: >>> As there is already Python 3.2 alpha, the core of Python has >>>already >>>been ported >> >>How about the email package? > > What about email? It is a core library, right? It has been ported >AFAIK. >Or have I missed something? Are you just assuming that because there have been 3.x releases, everything in the standard library works properly in 3.x? Or do you know something about the email package specifically? Last I checked, there were a number of outstanding issues with email in 3.x. And as Michael Foord pointed out, there are other standard library modules in the same state. Jean-Paul From merwok at netwok.org Tue Sep 7 17:13:07 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 07 Sep 2010 17:13:07 +0200 Subject: [Python-Dev] Volunteer help with porting In-Reply-To: References: <20100907133330.GA32192@phd.pp.ru> <20100907140259.2058.1140393887.divmod.xquotient.453@localhost.localdomain> <4C86491E.3090000@voidspace.org.uk> Message-ID: <4C865683.3040403@netwok.org> > Sure I will look into this. Could you please point me towards the > repository(I'd love it if I could use mercurial for the development process > rather than svn)?. Core developers still use Subversion (we?re in the process of switching), but you can clone the mirror at http://code.python.org/hg/branches/py3k/ and use named branches for your work. You can look for bugs needing patches on bugs.python.org or open reports and then post patches. The email package is a high-priority task, but it?s not trivial. R. David Murray and Shashwat Anand will maybe comment about the status of the port. Other interesting bugs: http://bugs.python.org/issue7962 http://bugs.python.org/issue8077 Maybe you can post a call on python-list and form a small group of py3k porters. Happy hacking From solipsis at pitrou.net Tue Sep 7 17:17:19 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Sep 2010 17:17:19 +0200 Subject: [Python-Dev] Volunteer help with porting References: <20100907133330.GA32192@phd.pp.ru> <20100907140259.2058.1140393887.divmod.xquotient.453@localhost.localdomain> <4C86491E.3090000@voidspace.org.uk> Message-ID: <20100907171719.43b060a9@pitrou.net> On Tue, 7 Sep 2010 20:12:26 +0530 Prashant Kumar wrote: > > > > Right, and there are other standard library modules (cgi, ftplib, nntplib, > > etc) that either need fixing or auditing as to how they handle bytes / > > strings. > > > > Sure I will look into this. Could you please point me towards the > repository(I'd love it if I could use mercurial for the development process > rather than svn)?. http://code.python.org/hg/branches/py3k/ Also don't forget to check for relevant issues on http://bugs.python.org For example, NNTP-related issues (either bugs or feature wishes): http://bugs.python.org/issue1926 http://bugs.python.org/issue7644 Regards Antoine. From rdmurray at bitdance.com Tue Sep 7 17:37:06 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 07 Sep 2010 11:37:06 -0400 Subject: [Python-Dev] Volunteer help with porting In-Reply-To: <20100907143449.GA9568@phd.pp.ru> References: <20100907133330.GA32192@phd.pp.ru> <20100907140259.2058.1140393887.divmod.xquotient.453@localhost.localdomain> <20100907143449.GA9568@phd.pp.ru> Message-ID: <20100907153706.DE96F2221ED@kimball.webabinitio.net> On Tue, 07 Sep 2010 18:34:49 +0400, Oleg Broytman wrote: > On Tue, Sep 07, 2010 at 02:02:59PM -0000, exarkun at twistedmatrix.com wrote: > > On 01:33 pm, phd at phd.pp.ru wrote: > >> As there is already Python 3.2 alpha, the core of Python has already > >> been ported > > > > How about the email package? > > What about email? It is a core library, right? It has been ported AFAIK. > Or have I missed something? Some bug reports, perhaps? Email has been "ported" in the sense that all the existing tests pass under py3k. It is not, however, fully functional: it cannot handle input that contains binary data with the high bit set. (To put it another way, the python3 email package currently handles only 7bit clean data.) This has various unfortunate consequences for other standard library packages (cgi, nntp, etc) that depend directly or indirectly on the email package. To fix this the email package API and a chunk of the internals needs to be redesigned so that it can handle bytes while still providing a string-based interface to the textual data. This is no small task, since email in python2 has been getting away with all kinds of dirty tricks because of the lack of a clean bytes/string separation in python2. See http://launchpad.net/python-email6 for current status of the project. Prashant Kumar, if you would like to help with the email6 project, please sign up to the email-sig mailing list and introduce yourself there. All help is welcome! It is not a "porting" project strictly speaking, but it certainly is interesting :) -- R. David Murray www.bitdance.com From tjreedy at udel.edu Tue Sep 7 17:48:19 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 07 Sep 2010 11:48:19 -0400 Subject: [Python-Dev] Volunteer help with porting In-Reply-To: <4C86491E.3090000@voidspace.org.uk> References: <20100907133330.GA32192@phd.pp.ru> <20100907140259.2058.1140393887.divmod.xquotient.453@localhost.localdomain> <4C86491E.3090000@voidspace.org.uk> Message-ID: On 9/7/2010 10:15 AM, Michael Foord wrote: > Right, and there are other standard library modules (cgi, ftplib, > nntplib, etc) that either need fixing or auditing as to how they handle > bytes / strings. If you wanted to help with the documentation of the stdlib, one thing that needs to be done is run doctest over the examples. See http://bugs.python.org/issue9730 for an example where the examples were not properly ported (two 'b' prefixes need to be added). Of course, there are other errors to be found. More contributions from additional contributors are welcome! -- Terry Jan Reedy From phd at phd.pp.ru Tue Sep 7 17:54:42 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Tue, 7 Sep 2010 19:54:42 +0400 Subject: [Python-Dev] Volunteer help with porting In-Reply-To: <20100907145841.2058.1687067386.divmod.xquotient.458@localhost.localdomain> References: <20100907133330.GA32192@phd.pp.ru> <20100907140259.2058.1140393887.divmod.xquotient.453@localhost.localdomain> <20100907143449.GA9568@phd.pp.ru> <20100907145841.2058.1687067386.divmod.xquotient.458@localhost.localdomain> Message-ID: <20100907155442.GA16079@phd.pp.ru> On Tue, Sep 07, 2010 at 02:58:41PM -0000, exarkun at twistedmatrix.com wrote: > On 02:34 pm, phd at phd.pp.ru wrote: >> On Tue, Sep 07, 2010 at 02:02:59PM -0000, exarkun at twistedmatrix.com >> wrote: >>> On 01:33 pm, phd at phd.pp.ru wrote: >>>> As there is already Python 3.2 alpha, the core of Python has >>>> already >>>> been ported >>> >>> How about the email package? >> >> What about email? It is a core library, right? It has been ported >> AFAIK. >> Or have I missed something? > > Are you just assuming that because there have been 3.x releases, > everything in the standard library works properly in 3.x? Or do you > know something about the email package specifically? Yes, I assumed that because the core team planned a beta release of 3.2 for October. "3.2" sounds more stable and complete than "3.0". > Last I checked, there were a number of outstanding issues with email in > 3.x. And as Michael Foord pointed out, there are other standard library > modules in the same state. I see. Thank you for the clarification. I am sorry for being so haste. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From contactprashantat at gmail.com Tue Sep 7 18:11:12 2010 From: contactprashantat at gmail.com (Prashant Kumar) Date: Tue, 7 Sep 2010 21:41:12 +0530 Subject: [Python-Dev] Volunteer help with porting In-Reply-To: <4C865683.3040403@netwok.org> References: <20100907133330.GA32192@phd.pp.ru> <20100907140259.2058.1140393887.divmod.xquotient.453@localhost.localdomain> <4C86491E.3090000@voidspace.org.uk> <4C865683.3040403@netwok.org> Message-ID: > Other interesting bugs: > http://bugs.python.org/issue7962 > http://bugs.python.org/issue8077 Thanks, I will look into these bugs and see if I can fix them. > > Maybe you can post a call on python-list and form a small group of py3k > porters. > I have already mailed the python-list ML and am expecting a reply. pk -------------- next part -------------- An HTML attachment was scrubbed... URL: From dirkjan at ochtman.nl Tue Sep 7 18:08:54 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Tue, 7 Sep 2010 18:08:54 +0200 Subject: [Python-Dev] Push notification In-Reply-To: <20100907163846.4bdf3cc1@pitrou.net> References: <20100907010402.7c7a3001@pitrou.net> <20100907163846.4bdf3cc1@pitrou.net> Message-ID: On Tue, Sep 7, 2010 at 16:38, Antoine Pitrou wrote: > Could push notification be added for the benchmarks repo? > I think the python-checkins list would be an appropriate recipient for > the e-mails (the repo has a low activity). Fine with me, if the list agrees. Cheers, Dirkjan From fdrake at acm.org Tue Sep 7 18:56:04 2010 From: fdrake at acm.org (Fred Drake) Date: Tue, 7 Sep 2010 12:56:04 -0400 Subject: [Python-Dev] Push notification In-Reply-To: <20100907163846.4bdf3cc1@pitrou.net> References: <20100907010402.7c7a3001@pitrou.net> <20100907163846.4bdf3cc1@pitrou.net> Message-ID: On Tue, Sep 7, 2010 at 10:38 AM, Antoine Pitrou wrote: > Could push notification be added for the benchmarks repo? > I think the python-checkins list would be an appropriate recipient for > the e-mails (the repo has a low activity). +1 ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From benjamin at python.org Tue Sep 7 19:39:54 2010 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 7 Sep 2010 12:39:54 -0500 Subject: [Python-Dev] [Python-checkins] r84562 - in python/branches/py3k: Doc/library/io.rst Lib/_pyio.py Lib/test/test_memoryio.py Misc/NEWS Modules/_io/_iomodule.c Modules/_io/_iomodule.h Modules/_io/bytesio.c In-Reply-To: References: <20100906184822.26A92FA6F@mail.python.org> Message-ID: 2010/9/7 Nick Coghlan : > I've raised an RFE (http://bugs.python.org/issue9789) to point out > that the need for that GC collect call in there to make the test > portable to other implementations is rather ugly Why? You're testing garbage collection, so you should call garbage collection. -- Regards, Benjamin From mal at egenix.com Tue Sep 7 19:46:25 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 07 Sep 2010 19:46:25 +0200 Subject: [Python-Dev] versioned .so files for Python 3.2 In-Reply-To: <4C82510C.9010405@v.loewis.de> References: <20100624115048.4fd152e3@heresy> <20100714195955.4815f980@heresy> <4C40372C.9030101@ubuntu.com> <20100722164036.7a80d27c@snowdog> <20100724002509.0ad8d359@snowdog> <4C78E4F0.9090306@v.loewis.de> <7BA45755-EF5F-4A4E-9E47-21C9A80C29B4@mac.com> <4C7CC9ED.9090405@egenix.com> <4C82510C.9010405@v.loewis.de> Message-ID: <4C867A71.1000105@egenix.com> "Martin v. L?wis" wrote: >> -1 on always using wchar_t as well. Python's default is UCS2 and the >> stable ABI should not change that. > > It's not really Python's default. It is what configure.in does by > default. Python's default, on Linux, is UCS-4. No, the default is UCS2 on all platforms and in configure.in. configure.in only uses UCS4 if it finds a TCL installation that happens to use UCS4 - for some reason I don't know :-) However, most Linux distros and more recently also some BSDs have switched over to using UCS4 for their distribution versions of Python. >> I also think that this information is not relevant for the stable >> ABI: Extensions that want to stick to the stable ABI should really >> not have to know whether Py_UNICODE maps to wchar_t or not. If they >> want to interface to a local whcar_t type they can use the conversion >> APIs we have for that in the Unicode API: PyUnicode_FromWideChar() >> and PyUnicode_AsWideChar(). > > Ok. I'm fine with excluding Py_UNICODE from the stable ABI. However, > I think in the long run, I guess more support for wchar_t will then > be needed in the API, e.g. more convenient argument parsing. Sure, we could add that. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 07 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2010-08-19: Released mxODBC 3.1.0 http://python.egenix.com/ 2010-09-15: DZUG Tagung, Dresden, Germany 7 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Tue Sep 7 19:48:49 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 07 Sep 2010 19:48:49 +0200 Subject: [Python-Dev] PEP 384 status In-Reply-To: <4C8244C5.9020401@v.loewis.de> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D798A.3040200@egenix.com> <4C8244C5.9020401@v.loewis.de> Message-ID: <4C867B01.6070609@egenix.com> "Martin v. L?wis" wrote: > >> This sounds like the issues such a mix can cause are mostly >> theoretical and don't really bother much in practice, so >> PEP 384 on Windows does have a chance :-) > > Actually, the CRT issues (FILE* in particular) have been > causing real crashes for many years, for many people. Do you have some pointers ? I don't remember this being a real practical issue, at least not for different versions of the MS CRTs. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 07 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2010-08-19: Released mxODBC 3.1.0 http://python.egenix.com/ 2010-09-15: DZUG Tagung, Dresden, Germany 7 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From amauryfa at gmail.com Tue Sep 7 20:00:25 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 7 Sep 2010 20:00:25 +0200 Subject: [Python-Dev] versioned .so files for Python 3.2 In-Reply-To: <4C867A71.1000105@egenix.com> References: <20100624115048.4fd152e3@heresy> <20100714195955.4815f980@heresy> <4C40372C.9030101@ubuntu.com> <20100722164036.7a80d27c@snowdog> <20100724002509.0ad8d359@snowdog> <4C78E4F0.9090306@v.loewis.de> <7BA45755-EF5F-4A4E-9E47-21C9A80C29B4@mac.com> <4C7CC9ED.9090405@egenix.com> <4C82510C.9010405@v.loewis.de> <4C867A71.1000105@egenix.com> Message-ID: Hi, 2010/9/7 M.-A. Lemburg : >> Ok. I'm fine with excluding Py_UNICODE from the stable ABI. However, >> I think in the long run, I guess more support for wchar_t will then >> be needed in the API, e.g. more convenient argument parsing. > > Sure, we could add that. Just to be clear: does this mean that PyUnicode_FromUnicode() and PyUnicode_AsUnicode() won't belong to the stable ABI? PyUnicode_AsWideChar() is not as fast, because it needs to copy the data. -- Amaury Forgeot d'Arc From matthew at woodcraft.me.uk Tue Sep 7 21:34:17 2010 From: matthew at woodcraft.me.uk (Matthew Woodcraft) Date: Tue, 7 Sep 2010 19:34:17 +0000 (UTC) Subject: [Python-Dev] Behaviour of max() and min() with equal keys Message-ID: In CPython, the builtin max() and min() have the property that if there are items with equal keys, the first item is returned. From a quick look at their source, I think this is true for Jython and IronPython too. However, this isn't currently a documented guarantee. Could it be made so? (As with the decision to declare sort() stable, it seems likely that by now there's code out there relying on it anyway.) -M- From mal at egenix.com Tue Sep 7 22:18:16 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 07 Sep 2010 22:18:16 +0200 Subject: [Python-Dev] versioned .so files for Python 3.2 In-Reply-To: References: <20100624115048.4fd152e3@heresy> <20100714195955.4815f980@heresy> <4C40372C.9030101@ubuntu.com> <20100722164036.7a80d27c@snowdog> <20100724002509.0ad8d359@snowdog> <4C78E4F0.9090306@v.loewis.de> <7BA45755-EF5F-4A4E-9E47-21C9A80C29B4@mac.com> <4C7CC9ED.9090405@egenix.com> <4C82510C.9010405@v.loewis.de> <4C867A71.1000105@egenix.com> Message-ID: <4C869E08.9080908@egenix.com> Amaury Forgeot d'Arc wrote: > Hi, > > 2010/9/7 M.-A. Lemburg : >>> Ok. I'm fine with excluding Py_UNICODE from the stable ABI. However, >>> I think in the long run, I guess more support for wchar_t will then >>> be needed in the API, e.g. more convenient argument parsing. >> >> Sure, we could add that. > > Just to be clear: does this mean that PyUnicode_FromUnicode() and > PyUnicode_AsUnicode() won't belong to the stable ABI? As I understood Martin's comment Py_UNICODE would not be part of the ABI in the sense that you can access the Py_UNICODE data from within the extension module. It should still be fine, passing around opaque Py_UNICODE buffers. > PyUnicode_AsWideChar() is not as fast, because it needs to copy the data. True. Also see this patch which tries to address the issue: http://bugs.python.org/issue8654 With the terminology used there, the stable ABI would implicitly have Py_UNICODE_AGNOSTIC set - and then prevent exposing the structure of Py_UNICODE* buffers while still allowing to pass them around. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 07 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2010-08-19: Released mxODBC 3.1.0 http://python.egenix.com/ 2010-09-15: DZUG Tagung, Dresden, Germany 7 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dickinsm at gmail.com Tue Sep 7 22:44:03 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 7 Sep 2010 21:44:03 +0100 Subject: [Python-Dev] Behaviour of max() and min() with equal keys In-Reply-To: References: Message-ID: On Tue, Sep 7, 2010 at 8:34 PM, Matthew Woodcraft wrote: > In CPython, the builtin max() and min() have the property that if there > are items with equal keys, the first item is returned. From a quick look > at their source, I think this is true for Jython and IronPython too. It's actually not clear to me that this behaviour is ideal; it might make more sense to have max() return the first item among equal largest elements, and min() return the last item. That way, the special case of two operands has the nice property that (max(x, y), min(x, y)) is always either (x, y) or (y, x), rather than being possibly (x, x) or (y, y). (That is, id(max(x, y)) and id(min(x, y)) are id(x) and id(y) in one order or the other.) The max and min methods for the Decimal module take care to work this way, for example: >>> x, y = Decimal(2), Decimal('2.0') >>> x.max(y), x.min(y) (Decimal('2'), Decimal('2.0')) But: >>> max(x, y), min(x, y) (Decimal('2'), Decimal('2')) Can you give examples of code that relies on max and min returning the first among equals? Mark From lvh at laurensvh.be Tue Sep 7 23:05:20 2010 From: lvh at laurensvh.be (Laurens Van Houtven) Date: Tue, 7 Sep 2010 23:05:20 +0200 Subject: [Python-Dev] Behaviour of max() and min() with equal keys In-Reply-To: References: Message-ID: FWIW: I think Mark is right. I never quite understood why that was, but never cared enough to complain. lvh -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew at woodcraft.me.uk Tue Sep 7 23:07:35 2010 From: matthew at woodcraft.me.uk (Matthew Woodcraft) Date: Tue, 7 Sep 2010 21:07:35 +0000 (UTC) Subject: [Python-Dev] Behaviour of max() and min() with equal keys References: Message-ID: Mark Dickinson wrote: > Matthew Woodcraft wrote: >> In CPython, the builtin max() and min() have the property that if there >> are items with equal keys, the first item is returned. From a quick look >> at their source, I think this is true for Jython and IronPython too. > It's actually not clear to me that this behaviour is ideal; it might > make more sense to have max() return the first item among equal > largest elements, and min() return the last item. I don't care a great deal what the behaviour is; I would like it to be consistent across Python versions, and I think the pragmatic way to achieve that is to document the current behaviour. > Can you give examples of code that relies on max and min returning the > first among equals? An existing regression test whose output depends on which choice is made. (I was writing some code today which had to duplicate the behaviour of a non-Python program which uses first-among-equals, which is what brought this question up. In that case I wouldn't think it unreasonable to have to hand-code the loop rather than using max(), but if in practice Python is always going to be first-among-equals, it seems to me we might as well be 'allowed' to take advantage of it.) -M- From jyasskin at gmail.com Tue Sep 7 23:40:51 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Tue, 7 Sep 2010 14:40:51 -0700 Subject: [Python-Dev] Behaviour of max() and min() with equal keys In-Reply-To: References: Message-ID: On Tue, Sep 7, 2010 at 1:44 PM, Mark Dickinson wrote: > On Tue, Sep 7, 2010 at 8:34 PM, Matthew Woodcraft > wrote: >> In CPython, the builtin max() and min() have the property that if there >> are items with equal keys, the first item is returned. From a quick look >> at their source, I think this is true for Jython and IronPython too. > > It's actually not clear to me that this behaviour is ideal; ?it might > make more sense to have max() return the first item among equal > largest elements, and min() return the last item. ?That way, the > special case of two operands has the nice property that (max(x, y), > min(x, y)) is always either (x, y) or (y, x), rather than being > possibly (x, x) or (y, y). ?(That is, id(max(x, y)) and id(min(x, y)) > are id(x) and id(y) in one order or the other.) > > The max and min methods for the Decimal module take care to work this > way, for example: > >>>> x, y = Decimal(2), Decimal('2.0') >>>> x.max(y), x.min(y) > (Decimal('2'), Decimal('2.0')) > > But: > >>>> max(x, y), min(x, y) > (Decimal('2'), Decimal('2')) > Decimal may actually have this backwards. The idea would be that min(*lst) == sorted(lst)[0], and max(*lst) == sorted(lst)[-1]. Given a stable sort, then, max of equivalent elements would return the last element, and min the first. According to Alex Stepanov, this helps some with composing these small order statistics into larger stably-ordered functions. I do think Decimal.max's answer is better than the builtin's answer, and that the incremental benefit from returning the last instead of first is pretty small. From jyasskin at gmail.com Tue Sep 7 23:47:25 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Tue, 7 Sep 2010 14:47:25 -0700 Subject: [Python-Dev] Behaviour of max() and min() with equal keys In-Reply-To: References: Message-ID: On Tue, Sep 7, 2010 at 2:40 PM, Jeffrey Yasskin wrote: > On Tue, Sep 7, 2010 at 1:44 PM, Mark Dickinson wrote: >> On Tue, Sep 7, 2010 at 8:34 PM, Matthew Woodcraft >> wrote: >>> In CPython, the builtin max() and min() have the property that if there >>> are items with equal keys, the first item is returned. From a quick look >>> at their source, I think this is true for Jython and IronPython too. >> >> It's actually not clear to me that this behaviour is ideal; ?it might >> make more sense to have max() return the first item among equal >> largest elements, and min() return the last item. ?That way, the >> special case of two operands has the nice property that (max(x, y), >> min(x, y)) is always either (x, y) or (y, x), rather than being >> possibly (x, x) or (y, y). ?(That is, id(max(x, y)) and id(min(x, y)) >> are id(x) and id(y) in one order or the other.) >> >> The max and min methods for the Decimal module take care to work this >> way, for example: >> >>>>> x, y = Decimal(2), Decimal('2.0') >>>>> x.max(y), x.min(y) >> (Decimal('2'), Decimal('2.0')) >> >> But: >> >>>>> max(x, y), min(x, y) >> (Decimal('2'), Decimal('2')) >> > > Decimal may actually have this backwards. The idea would be that > min(*lst) == sorted(lst)[0], and max(*lst) == sorted(lst)[-1]. Given a > stable sort, then, max of equivalent elements would return the last > element, and min the first. According to Alex Stepanov, this helps > some with composing these small order statistics into larger > stably-ordered functions. > > I do think Decimal.max's answer is better than the builtin's answer, > and that the incremental benefit from returning the last instead of > first is pretty small. Actually, Decimal isn't doing anything along these lines. At least in Python 2.6, I get: >>> Decimal('2').max(Decimal('2.0')) Decimal('2') >>> Decimal('2.0').max(Decimal('2')) Decimal('2') >>> Decimal('2.0').min(Decimal('2')) Decimal('2.0') >>> Decimal('2').min(Decimal('2.0')) Decimal('2.0') indicating that '2' is considered larger than '2.0' in the min/max comparison. It's ignoring the order of the arguments. It also creates a new Decimal object for the return value, so I can't use id() to check which one of identical elements it returns. From dickinsm at gmail.com Tue Sep 7 23:51:26 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 7 Sep 2010 22:51:26 +0100 Subject: [Python-Dev] Behaviour of max() and min() with equal keys In-Reply-To: References: Message-ID: On Tue, Sep 7, 2010 at 10:47 PM, Jeffrey Yasskin wrote: > Actually, Decimal isn't doing anything along these lines. At least in > Python 2.6, I get: > >>>> Decimal('2').max(Decimal('2.0')) > Decimal('2') >>>> Decimal('2.0').max(Decimal('2')) > Decimal('2') >>>> Decimal('2.0').min(Decimal('2')) > Decimal('2.0') >>>> Decimal('2').min(Decimal('2.0')) > Decimal('2.0') > > indicating that '2' is considered larger than '2.0' in the min/max > comparison. Right; there's a strict specification about how equal values with different exponents (or different signs in the case of comparing zeros) should be treated. > It's ignoring the order of the arguments. It also creates > a new Decimal object for the return value, so I can't use id() to > check which one of identical elements it returns. This bit surprises me. I honestly thought I'd fixed it up so that max(x, y) actually returned one of x and y (and min(x, y) returned the other). Oh well. Mark From dickinsm at gmail.com Tue Sep 7 23:54:27 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 7 Sep 2010 22:54:27 +0100 Subject: [Python-Dev] Behaviour of max() and min() with equal keys In-Reply-To: References: Message-ID: On Tue, Sep 7, 2010 at 10:40 PM, Jeffrey Yasskin wrote: > Decimal may actually have this backwards. The idea would be that > min(*lst) == sorted(lst)[0], and max(*lst) == sorted(lst)[-1]. Given a > stable sort, then, max of equivalent elements would return the last > element, and min the first. Yes, you're right; that would make more sense than the other way around. Mark From barry at python.org Tue Sep 7 23:58:04 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 7 Sep 2010 17:58:04 -0400 Subject: [Python-Dev] [Python-checkins] r84536 - sandbox/trunk/release/release.py In-Reply-To: <20100905182846.4E3DAEEA02@mail.python.org> References: <20100905182846.4E3DAEEA02@mail.python.org> Message-ID: <20100907175804.1eba70d7@mission> On Sep 05, 2010, at 08:28 PM, georg.brandl wrote: >Author: georg.brandl >Date: Sun Sep 5 20:28:46 2010 >New Revision: 84536 > >Log: >Fix after changing NEWS layout. > >Modified: > sandbox/trunk/release/release.py > >Modified: sandbox/trunk/release/release.py >============================================================================== >--- sandbox/trunk/release/release.py (original) >+++ sandbox/trunk/release/release.py Sun Sep 5 20:28:46 2010 >@@ -396,13 +396,13 @@ > with open('Misc/NEWS', encoding="utf-8") as fp: > lines = fp.readlines() > for i, line in enumerate(lines): >- if line.startswith("(editors"): >+ if line.startswith("Python News"): > start = i > if line.startswith("What's"): > end = i > break > with open('Misc/NEWS', 'w', encoding="utf-8") as fp: >- fp.writelines(lines[:start+1]) >+ fp.writelines(lines[:start+2]) > fp.write(NEWS_TEMPLATE) > fp.writelines(lines[end-1:]) > print("Please fill in the the name of the next version.") Will this still work with the Python 2.7 NEWS file? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From dickinsm at gmail.com Wed Sep 8 00:00:21 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 7 Sep 2010 23:00:21 +0100 Subject: [Python-Dev] Behaviour of max() and min() with equal keys In-Reply-To: References: Message-ID: On Tue, Sep 7, 2010 at 10:51 PM, Mark Dickinson wrote: > On Tue, Sep 7, 2010 at 10:47 PM, Jeffrey Yasskin wrote: >> It's ignoring the order of the arguments. It also creates >> a new Decimal object for the return value, so I can't use id() to >> check which one of identical elements it returns. > > This bit surprises me. ?I honestly thought I'd fixed it up so that > max(x, y) actually returned one of x and y (and min(x, y) returned the > other). ?Oh well. Ah. I'd forgotten that the Decimal max and min methods are context aware, so that max(x, y) is rounded to the current context, and hence can actually be different from both x and y. So that was a bad example from me. Sorry. >>> from decimal import * >>> getcontext().Emin = -500 >>> x, y = Decimal('-1e-100'), Decimal('-1e-1000') >>> x.max(y) Decimal('-0E-527') Mark From dickinsm at gmail.com Wed Sep 8 00:03:04 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 7 Sep 2010 23:03:04 +0100 Subject: [Python-Dev] Behaviour of max() and min() with equal keys In-Reply-To: References: Message-ID: On Tue, Sep 7, 2010 at 11:00 PM, Mark Dickinson wrote: > On Tue, Sep 7, 2010 at 10:51 PM, Mark Dickinson wrote: >> On Tue, Sep 7, 2010 at 10:47 PM, Jeffrey Yasskin wrote: >>> It's ignoring the order of the arguments. It also creates >>> a new Decimal object for the return value, so I can't use id() to >>> check which one of identical elements it returns. >> >> This bit surprises me. ?I honestly thought I'd fixed it up so that >> max(x, y) actually returned one of x and y (and min(x, y) returned the >> other). ?Oh well. > > Ah. ?I'd forgotten that the Decimal max and min methods are context > aware, so that max(x, y) is rounded to the current context, and hence > can actually be different from both x and y. ?So that was a bad > example from me. ?Sorry. Grr. s/max(x, y)/x.max(y)/g Mark From ncoghlan at gmail.com Wed Sep 8 00:09:13 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Sep 2010 08:09:13 +1000 Subject: [Python-Dev] [Python-checkins] r84562 - in python/branches/py3k: Doc/library/io.rst Lib/_pyio.py Lib/test/test_memoryio.py Misc/NEWS Modules/_io/_iomodule.c Modules/_io/_iomodule.h Modules/_io/bytesio.c In-Reply-To: References: <20100906184822.26A92FA6F@mail.python.org> Message-ID: On Wed, Sep 8, 2010 at 3:39 AM, Benjamin Peterson wrote: > 2010/9/7 Nick Coghlan : >> I've raised an RFE (http://bugs.python.org/issue9789) to point out >> that the need for that GC collect call in there to make the test >> portable to other implementations is rather ugly > > Why? You're testing garbage collection, so you should call garbage collection. Having that test *as well* as a specific release test (to check implicit as well as explicit release) is a different story (we do that for generators, files and assorted other things). Having forced GC invocation be the only way to do explicit release is the part I found ugly. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From cournape at gmail.com Wed Sep 8 00:34:06 2010 From: cournape at gmail.com (David Cournapeau) Date: Wed, 8 Sep 2010 07:34:06 +0900 Subject: [Python-Dev] PEP 384 status In-Reply-To: <4C867B01.6070609@egenix.com> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D798A.3040200@egenix.com> <4C8244C5.9020401@v.loewis.de> <4C867B01.6070609@egenix.com> Message-ID: On Wed, Sep 8, 2010 at 2:48 AM, M.-A. Lemburg wrote: > "Martin v. L?wis" wrote: >> >>> This sounds like the issues such a mix can cause are mostly >>> theoretical and don't really bother much in practice, so >>> PEP 384 on Windows does have a chance :-) >> >> Actually, the CRT issues (FILE* in particular) have been >> causing real crashes for many years, for many people. > > Do you have some pointers ? I don't have a bug report, but I have had issues in my audiolab package, where the package itself is built with visual studio, but the library it is linked against is built with mingw (libsndfile, which cannot be built with visual studio). The problem happens when you want to use the mingw-built library which accepts a file descriptor that you create with mkstemp inside code built with Visual Studio. This issue is actually quite easy to reproduce. > I don't remember this being a real practical issue, at least > not for different versions of the MS CRTs. I would turn the question around: what are the cases where you manage to mix CRT and not getting any issues ? This has never worked in my own experience in the context of python extensions, cheers, David From barry at python.org Wed Sep 8 00:45:41 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 7 Sep 2010 18:45:41 -0400 Subject: [Python-Dev] PEP 3149 thoughts In-Reply-To: <4C83D1BA.4050505@v.loewis.de> References: <4C83D1BA.4050505@v.loewis.de> Message-ID: <20100907184541.3dc917b7@mission> On Sep 05, 2010, at 07:22 PM, Martin v. L?wis wrote: >I know the PEP is accepted, but I would still like to see some >changes/clarifications. > >1. What is the effect of this PEP on Windows? Is this a Linux-only > feature? If not, who is going to provide the changes for Windows? > (More specifically: if this is indeed meant for Windows, and > if no Windows implementation arrives before 3.2b1, I'd ask that > the changes be rolled back, and integration is deferred until there > is Windows support) I only mandated ./configure-based builds to be PEP 3149 conforming. I have no objection to expanding the PEP to include Windows builds, but I'm not sure it's necessary and it would take a Windows build expert to make and test those changes. Does PEP 3149 have any advantage for Windows installations? >2. Why does the PEP recommend installing stuff >into /usr/share/pyshared? It's just a suggestion, but as it turns out, probably an incorrect one. I'll rephrase this to make it clear that it's up to the distribution as to where exactly these files get installed. Nothing about this PEP changes the default from-source installation directory. > According to the Linux FHS, /usr/share is for Architecture- > independent data, see > >http://www.pathname.com/fhs/pub/fhs-2.3.html#USRSHAREARCHITECTUREINDEPENDENTDATA > In particular, it's objective is that you can NFS-share it across, > say, both SPARC Linux and x86 Linux. I believe the PEP would break > this, as SPARC and x86 executables would override each other. > >3. When the PEP recommends that stuff gets installed into pyshared, > why does the patch then not implement this recommendation, but > continues installing files into lib-dynload? See above. It is not the intent of this PEP to change the installation directories. Distributions have all the other tools they need to get the files into the right place for their layouts. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Sep 8 01:48:25 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 7 Sep 2010 19:48:25 -0400 Subject: [Python-Dev] PEP 3149 thoughts In-Reply-To: References: <4C83D1BA.4050505@v.loewis.de> Message-ID: <20100907194825.2921cbb3@mission> On Sep 06, 2010, at 03:40 PM, David Cournapeau wrote: >Reading the related paragraph in the PEP, it seems to me that the use >of package as in "these distributions install third party (i.e. >non-standard library) packages ..." is too vague. Rephrased as: [...]these distributions install third party package modules (``.pyc`` and ``.so`` files) into `/usr/share/pyshared`[...] -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Sep 8 01:50:38 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 7 Sep 2010 19:50:38 -0400 Subject: [Python-Dev] PEP 3149 thoughts In-Reply-To: <4C840E17.4030105@v.loewis.de> References: <4C83D1BA.4050505@v.loewis.de> <4C840E17.4030105@v.loewis.de> Message-ID: <20100907195038.1105f5f3@mission> On Sep 05, 2010, at 11:39 PM, Martin v. L?wis wrote: >Notice, however, that the PEP also talks about creating different names >for different compilation options of the same Python binary. This >applies to Windows as well (as the PEP actually points out: there is >_d.pyd and .pyd). > >In any case, if the PEP is specific to Unix (or even to Linux?), it >should say so (either as a positive list, or a negative one; if >negative, it should also say whether it applies to OSX). Section added: Windows ======= This PEP only addresses build issues on POSIX systems that use the ``configure`` script. While Windows or other platform support is not explicitly disallowed under this PEP, platform expertise is needed in order to evaluate, describe, and implement support on such platforms. It is not currently clear that the facilities in this PEP are even useful for Windows. >> That section is talking about files installed by distributions, >> which need to take special steps to get everything >> into /usr/{lib,share}/pyshared; a standard out-of-the-tarball >> install will not change the way it is installed. >> >> It may well be that Barry had his Ubuntu hat on a bit too firmly >> when writing that PEP :) > >I think then that the PEP should better separate what is actually being >specified (and apparently, that's just the name of the shared objects), >from possible use case scenarios. I'm not really sure what to do about this. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From glenn at nevcal.com Wed Sep 8 07:00:15 2010 From: glenn at nevcal.com (Glenn Linderman) Date: Tue, 07 Sep 2010 22:00:15 -0700 Subject: [Python-Dev] Thoughts fresh after EuroPython In-Reply-To: References: <4C4D5CA5.7020809@voidspace.org.uk> Message-ID: <4C87185F.6000408@nevcal.com> On 7/26/2010 7:36 AM, Guido van Rossum wrote: > According to CSP advicates, this approach will break down when you > need more than 8-16 cores since cache coherence breaks down at 16 > cores. Then you would have to figure out a message-passing approach > (but the messages would have to be very fast). Catching up on Python-Dev after 3 months of travel (lucky me!), so apologies for a "blast from the past" as I'm 6 weeks late in replying here. Think of the hardware implementation of cache coherence as a MIL - memory interleave lock, or a micro interpreter lock (the hardware is interpreting what the compiled software is doing). That is not so different than Python's GIL, just at a lower level. I didn't read the CSP advocacy papers, but experience in early parallel system at CMU, Tandem Computers, and Teradata strongly imply that multiprocessing of some sort will always be able to scale larger than memory coherent cores -- if the application can be made parallel at all. It is interesting to note that all the parallel systems mentioned above implemented fast message passing hardware of various sorts (affected by available technologies of their times). It is interesting to note the similarities between some of the extreme multi-way cache coherence approaches and the various message passing hardware, also... some of the papers that talk about exceeding 16 cores were going down a message passing road to achieve it. Maybe something new has been discovered in the last 8 years since I've not been following the research... the only thing I've read about that in the last 8 years is the loss of Jim Gray at sea... but the IEEE paper you posted later seems to confirm my suspicions that there has not yet been a breakthrough. The point of the scalability remark, though, is that while lots of problems can be solved on a multi-core system, problems also grow bigger, and there will likely always be problems that cannot be solved on a multi-core (single cache coherent memory) system. Those problems will require message passing solutions. Experience with the systems above has shown that switching from a multi-core (semaphore based) design to a message passing design is usually a rewrite. Perhaps the existence of the GIL, forcing a message passing solution to be created early, is a blessing in disguise for the design of large scale applications. I've been hearing about problems for which the data is too large to share, and the calculation is too complex to parallelize for years, but once the available hardware is exhausted as the problem grows, the only path to larger scale is message passing parallelism... forcing a redesign of applications that outgrew the available hardware. That said, applications that do fit in available hardware generally can run a little faster with some sort of shared memory approach: message passing does have overhead. -- Glenn ------------------------------------------------------------------------ I have CDO..It's like OCD, but in alphabetical order..The way it should be! (a facebook group is named this, except for a misspelling.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Wed Sep 8 08:59:21 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Tue, 7 Sep 2010 23:59:21 -0700 Subject: [Python-Dev] Behaviour of max() and min() with equal keys In-Reply-To: References: Message-ID: On Sep 7, 2010, at 12:34 PM, Matthew Woodcraft wrote: > In CPython, the builtin max() and min() have the property that if there > are items with equal keys, the first item is returned. From a quick look > at their source, I think this is true for Jython and IronPython too. > > However, this isn't currently a documented guarantee. Could it be made > so? (As with the decision to declare sort() stable, it seems likely that > by now there's code out there relying on it anyway.) That seems like a reasonable request. This behavior has been around for a very long time is unlikely to change. Elsewhere, we've made efforts to document sort stability (i.e. sorted(), heapq.nlargest(), heapq.nsmallest, merge(), etc). It is nice that min(it) parallels sorted(it)[0] and nsmallest(1, it). The same goes for max(it) paralleling sorted(it,reverse=True)[0] and nlargest(1, it). Raymond From ralf at brainbot.com Wed Sep 8 09:42:44 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 08 Sep 2010 09:42:44 +0200 Subject: [Python-Dev] PEP 3149 thoughts In-Reply-To: <20100907195038.1105f5f3@mission> (Barry Warsaw's message of "Tue, 7 Sep 2010 19:50:38 -0400") References: <4C83D1BA.4050505@v.loewis.de> <4C840E17.4030105@v.loewis.de> <20100907195038.1105f5f3@mission> Message-ID: <87sk1kfzpn.fsf@muni.brainbot.com> Barry Warsaw writes: > > Section added: > > Windows > ======= > > This PEP only addresses build issues on POSIX systems that use the > ``configure`` script. While Windows or other platform support is not > explicitly disallowed under this PEP, platform expertise is needed in > order to evaluate, describe, and implement support on such platforms. > It is not currently clear that the facilities in this PEP are even > useful for Windows. > If it's useful on unix like systems, why shouldn't it be useful on windows? Multiple versions of python can be installed on windows too. And people might also like to share their packages between installations. - Ralf From ncoghlan at gmail.com Wed Sep 8 10:19:26 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Sep 2010 18:19:26 +1000 Subject: [Python-Dev] PEP 384 status In-Reply-To: References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D798A.3040200@egenix.com> <4C8244C5.9020401@v.loewis.de> <4C867B01.6070609@egenix.com> Message-ID: On Wed, Sep 8, 2010 at 8:34 AM, David Cournapeau wrote: > I would turn the question around: what are the cases where you manage > to mix CRT and not getting any issues ? This has never worked in my > own experience in the context of python extensions, I've done it quite a bit over the years with SWIG-wrapped libraries where all I did was recompile my extension for new versions of Python. It works fine, so long as you avoid the very APIs that PEP 384 is blocking from the stable ABI. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Sep 8 10:25:14 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Sep 2010 18:25:14 +1000 Subject: [Python-Dev] PEP 3149 thoughts In-Reply-To: <87sk1kfzpn.fsf@muni.brainbot.com> References: <4C83D1BA.4050505@v.loewis.de> <4C840E17.4030105@v.loewis.de> <20100907195038.1105f5f3@mission> <87sk1kfzpn.fsf@muni.brainbot.com> Message-ID: On Wed, Sep 8, 2010 at 5:42 PM, Ralf Schmitt wrote: > Barry Warsaw writes: > >> >> Section added: >> >> Windows >> ======= >> >> This PEP only addresses build issues on POSIX systems that use the >> ``configure`` script. ?While Windows or other platform support is not >> explicitly disallowed under this PEP, platform expertise is needed in >> order to evaluate, describe, and implement support on such platforms. >> It is not currently clear that the facilities in this PEP are even >> useful for Windows. >> > > If it's useful on unix like systems, why shouldn't it be useful on > windows? Multiple versions of python can be installed on windows > too. And people might also like to share their packages between > installations. Platform mindset. Windows has a history of duplication of files to avoid dependency management (even the C runtime is typically duplicated on disk for each application that uses it). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From cournape at gmail.com Wed Sep 8 10:34:43 2010 From: cournape at gmail.com (David Cournapeau) Date: Wed, 8 Sep 2010 17:34:43 +0900 Subject: [Python-Dev] PEP 384 status In-Reply-To: References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D798A.3040200@egenix.com> <4C8244C5.9020401@v.loewis.de> <4C867B01.6070609@egenix.com> Message-ID: On Wed, Sep 8, 2010 at 5:19 PM, Nick Coghlan wrote: > On Wed, Sep 8, 2010 at 8:34 AM, David Cournapeau wrote: >> I would turn the question around: what are the cases where you manage >> to mix CRT and not getting any issues ? This has never worked in my >> own experience in the context of python extensions, > > I've done it quite a bit over the years with SWIG-wrapped libraries > where all I did was recompile my extension for new versions of Python. > It works fine, so long as you avoid the very APIs that PEP 384 is > blocking from the stable ABI. What do you mean by recompiling ? Using Mingw ? Normally, if you just recompile your extensions without using special distutils options, you use the same compiler as the one used by python, which generally avoids mixing runtimes. In other words, the problem mainly arises when you need to integrate libraries which you can not recompile with the compiler used by python, because the code is not visual-studio compatible, or because the library is only available in binary form. cheers, David From hrvoje.niksic at avl.com Wed Sep 8 10:48:34 2010 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Wed, 08 Sep 2010 10:48:34 +0200 Subject: [Python-Dev] Behaviour of max() and min() with equal keys In-Reply-To: References: Message-ID: <4C874DE2.2030907@avl.com> On 09/07/2010 11:40 PM, Jeffrey Yasskin wrote: > Decimal may actually have this backwards. The idea would be that > min(*lst) == sorted(lst)[0], and max(*lst) == sorted(lst)[-1]. Here you mean "is" rather than "==", right? The relations you spelled are guaranteed regardless of stability. (This doesn't apply to Decimal.max and Decimal.min, which return new objects.) From ncoghlan at gmail.com Wed Sep 8 12:59:22 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Sep 2010 20:59:22 +1000 Subject: [Python-Dev] PEP 384 status In-Reply-To: References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D798A.3040200@egenix.com> <4C8244C5.9020401@v.loewis.de> <4C867B01.6070609@egenix.com> Message-ID: On Wed, Sep 8, 2010 at 6:34 PM, David Cournapeau wrote: > In other words, the problem mainly arises when you need to integrate > libraries which you can not recompile with the compiler used by > python, because the code is not visual-studio compatible, or because > the library is only available in binary form. In my case, I was building on an old dev system which only has VC6, but needed to link against Python 2.4 (which was compiled with MSVC 2005). The build process didn't use distutils, so that didn't affect anything. It works, you just have to know what APIs you have to avoid. The C runtime independence aspects of PEP 384 are just about enlisting the compiler in the task of making sure you *have* avoided all those APIs. The rest of the PEP is about hiding the details of some of Python's own data structures to allow them to change without affecting the stable ABI. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From cournape at gmail.com Wed Sep 8 17:43:51 2010 From: cournape at gmail.com (David Cournapeau) Date: Thu, 9 Sep 2010 00:43:51 +0900 Subject: [Python-Dev] PEP 384 status In-Reply-To: References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D798A.3040200@egenix.com> <4C8244C5.9020401@v.loewis.de> <4C867B01.6070609@egenix.com> Message-ID: On Wed, Sep 8, 2010 at 7:59 PM, Nick Coghlan wrote: > On Wed, Sep 8, 2010 at 6:34 PM, David Cournapeau wrote: >> In other words, the problem mainly arises when you need to integrate >> libraries which you can not recompile with the compiler used by >> python, because the code is not visual-studio compatible, or because >> the library is only available in binary form. > > In my case, I was building on an old dev system which only has VC6, > but needed to link against Python 2.4 (which was compiled with MSVC > 2005). The build process didn't use distutils, so that didn't affect > anything. ok, I was confused by "I just recompiled". > > It works, you just have to know what APIs you have to avoid. The critical point is that you cannot always do that. Retaking my example of mkstemp: I have a C library which has a fdopen-like function (built with one C runtime, not the same as python), there is no way that I know of to use this API with a file descriptor obtained from tempfile.mkstemp function. The only solution is to build my own C extension with C mkstemp, built with the same runtime as my library, and make that available to python. cheers, David From kevin.watters at gmail.com Wed Sep 8 19:01:16 2010 From: kevin.watters at gmail.com (Kevin Watters) Date: Wed, 08 Sep 2010 13:01:16 -0400 Subject: [Python-Dev] Working on the Python code-base with a VIM-based setup In-Reply-To: References: Message-ID: Another shameless plug: My pyflakes.vim plugin will highlight errors as you type: http://www.vim.org/scripts/script.php?script_id=2441 It's really helpful for catching those typos that you usually don't notice until the runtime NameError. Eli Bendersky wrote: > Hello pydev, > > This is a meta-question which I hope is appropriate in this list (**). > Recently I've switched to to VIM as my main development platform (in > terms of code editing and navigation). Working on the Python code-base > is both a concrete requirement and a yardstick for me - I want to be as > effective as possible at it. Therefore I would like to ask those of you > working on Python's code with VIM about your setups - the special tweaks > to VIM & plugins you use to make working with the code as simple and > effective as possible. > > Myself, since I'm still a VIM newbie, my setup is quite spartan. I > created tags with: > > ctags -R Grammar Include Modules/ Objects/ Parser/ Python/ > > And now happily browse around the source code with Ctrl-], Ctrl-I/O, > Ctrl-T and so on. I've also installed the Taglist plugin to show all > functions/macros in open buffers - it's sometimes helpful. Other plugins > I've found useful ar NERD-commenter (for commenting out chunks of code) > and highlight_current_line (for a visual cue of where I am in a file). > Besides, I've taken the suggested settings from the .vimrc file in Misc/ > to help enforcing PEP-8. > > I heard that there are more sophisticated tags plugins that also allow > one to check where a function is called for, and other intellisens-y > stuff, though I'm not sure whether anyone uses it for Python's code. > > Thanks in advance, > Eli > > > (**) Note that it deals with the source code *of Python* (the stuff you > download from Python's official SVN), not Python source code. > From barry at python.org Wed Sep 8 20:40:35 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 8 Sep 2010 14:40:35 -0400 Subject: [Python-Dev] PEP 3149 thoughts In-Reply-To: <87sk1kfzpn.fsf@muni.brainbot.com> References: <4C83D1BA.4050505@v.loewis.de> <4C840E17.4030105@v.loewis.de> <20100907195038.1105f5f3@mission> <87sk1kfzpn.fsf@muni.brainbot.com> Message-ID: <20100908144035.68015dc5@mission> On Sep 08, 2010, at 09:42 AM, Ralf Schmitt wrote: >If it's useful on unix like systems, why shouldn't it be useful on >windows? Multiple versions of python can be installed on windows >too. And people might also like to share their packages between >installations. I guess the point is *I* don't know! I'm not a Windows developer, and I'm still unable to even build Python on Windows 7 with VS2010. In the absence of a Windows developer explicitly saying that it's useful, and offering to help implement it, then we'll just leave it for configure-based builds. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From matthew at woodcraft.me.uk Wed Sep 8 21:32:48 2010 From: matthew at woodcraft.me.uk (Matthew Woodcraft) Date: Wed, 8 Sep 2010 19:32:48 +0000 (UTC) Subject: [Python-Dev] Behaviour of max() and min() with equal keys References: Message-ID: Raymond Hettinger wrote: >Matthew Woodcraft wrote: >> In CPython, the builtin max() and min() have the property that if there >> are items with equal keys, the first item is returned. From a quick look >> at their source, I think this is true for Jython and IronPython too. >> However, this isn't currently a documented guarantee. Could it be made >> so? (As with the decision to declare sort() stable, it seems likely that >> by now there's code out there relying on it anyway.) > That seems like a reasonable request. This behavior has been around > for a very long time is unlikely to change. Elsewhere, we've made > efforts to document sort stability (i.e. sorted(), heapq.nlargest(), > heapq.nsmallest, merge(), etc). I've submitted issue 9802 as a feature request with a concrete suggestion for the docs. -M- From grosbedo at gmail.com Wed Sep 8 22:56:05 2010 From: grosbedo at gmail.com (GrosBedo) Date: Wed, 8 Sep 2010 22:56:05 +0200 Subject: [Python-Dev] cProfile and threads Message-ID: Hello, Ive just stumbled accross your changes Krisvale, and from your last reply, I can see that you invalidated your changes : I just realized that this is probably a redundant change. > We have C apis to get all the Thread states in an interpreter state (I didn't even know there was such a thing as multiple interpreter states, but there!) > This is the PyInterpreterState_ThreadHead() api et al. > From C, all that is missing is a SetTrace api that takes a thread state. > > From python, the threading module provides access to all Thread objects, and each of those has a settrace/setprofile method. > > To turn on global tracing from cProfile, all that is needed is to iterate over all the Thread objects. > > Setting this to invalid, since there already are APIs to do this, at least from .py code. > > Could you please provide more explanations, or even an example ? Because it seems that you're the only one on earth to finally find a way to multithread the cProfiler... -------------- next part -------------- An HTML attachment was scrubbed... URL: From grosbedo at gmail.com Wed Sep 8 23:09:43 2010 From: grosbedo at gmail.com (GrosBedo) Date: Wed, 8 Sep 2010 23:09:43 +0200 Subject: [Python-Dev] cProfile and threads In-Reply-To: References: Message-ID: Hello, Ive just stumbled accross your changes Krisvale, and from your last reply, I can see that you invalidated your changes : I just realized that this is probably a redundant change. > We have C apis to get all the Thread states in an interpreter state (I didn't even know there was such a thing as multiple interpreter states, but there!) > This is the PyInterpreterState_ThreadHead() api et al. > From C, all that is missing is a SetTrace api that takes a thread state. > > From python, the threading module provides access to all Thread objects, and each of those has a settrace/setprofile method. > > To turn on global tracing from cProfile, all that is needed is to iterate over all the Thread objects. > > Setting this to invalid, since there already are APIs to do this, at least from .py code. > > Could you please provide more explanations, or even an example ? Because it seems that you're the only one on earth to finally find a way to multithread the cProfiler... -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristjan at ccpgames.com Thu Sep 9 06:22:00 2010 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 9 Sep 2010 12:22:00 +0800 Subject: [Python-Dev] cProfile and threads In-Reply-To: References: Message-ID: <2E034B571A5CE44E949B9FCC3B6D24EE5398DDFB@exchcn.ccp.ad.local> Ok, I've been busy with other stuff, but I'm almost finished with a patch to submit as an alternative. cProfile.enable() will then take an "allthreads" argument, which when enabled, will set the profiler state on all threads ?_currently_ running. Now, I only realized this late, but _currently_ is really not good enough. In order to do this properly, you want to also profile threads that are created during the session. For this reason, Maybe I was a bit rash invalidating this. Anyway, keep your eyes trained on issue 9609 where I'll hopefully put my patch in real soon now. K From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of GrosBedo Sent: Thursday, September 09, 2010 5:10 To: python-dev at python.org Subject: Re: [Python-Dev] cProfile and threads Hello, Ive just stumbled accross your changes Krisvale, and from your last reply, I can see that you invalidated your changes : I just realized that this is probably a redundant change. We have C apis to get all the Thread states in an interpreter state (I didn't even know there was such a thing as multiple interpreter states, but there!) This is the PyInterpreterState_ThreadHead() api et al. >From C, all that is missing is a SetTrace api that takes a thread state. >From python, the threading module provides access to all Thread objects, and each of those has a settrace/setprofile method. To turn on global tracing from cProfile, all that is needed is to iterate over all the Thread objects. Setting this to invalid, since there already are APIs to do this, at least from .py code. Could you please provide more explanations, or even an example ? Because it seems that you're the only one on earth to finally find a way to multithread the cProfiler... -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Thu Sep 9 18:43:54 2010 From: eliben at gmail.com (Eli Bendersky) Date: Thu, 9 Sep 2010 19:43:54 +0300 Subject: [Python-Dev] terminology for "free variables" in Python Message-ID: The execution model section of the Python reference manual defines free variables as follows: "If a variable is used in a code block but not defined there, it is a free variable" This makes sense and fits the academic definition. The documentation of the symtable module supports this definition - it says about is_free(): "return True if the symbol is referenced in its block but not assigned to". However, it appears that in the CPython front-end source code (in particular the parts dealing with the symbol table), a free variables has a somewhat stricter meaning. For example, in this chunk of code: def some_func(myparam): def internalfunc(): return cc * myparam CPython infers that in 'internalfunc', while 'myparam' is free, 'cc' is global because 'cc' isn't bound in the enclosing scope, although according to the definitions stated above, both should be considered free. The bytecode generated for loading cc and myparam is different, of course. Is there a (however slight) inconsistency of terms here, or is it my misunderstanding? Thanks in advance, Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Sep 9 18:55:23 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 9 Sep 2010 09:55:23 -0700 Subject: [Python-Dev] terminology for "free variables" in Python In-Reply-To: References: Message-ID: On Thu, Sep 9, 2010 at 9:43 AM, Eli Bendersky wrote: > The execution model section of the Python reference manual defines free > variables as follows: > > ??? "If a variable is used in a code block but not defined there, it is > a?free variable" > > This makes sense and fits the academic definition. The documentation of the > symtable module supports this definition - it says about is_free(): "return > True if the symbol is referenced in its block but not assigned to". > > However, it appears that in the CPython front-end source code (in particular > the parts dealing with the symbol table), a free variables has a somewhat > stricter meaning. For example, in this chunk of code: > > def some_func(myparam): > ??? def internalfunc(): > ??????? return cc * myparam > > CPython infers that in 'internalfunc', while 'myparam' is free, 'cc' is What exactly do you mean by "infers" ? How do you know that it infers that? How does it matter for your understanding of the code? > global because 'cc' isn't bound in the enclosing scope, although according > to the definitions stated above, both should be considered free. The > bytecode generated for loading cc and myparam is different, of course. > > Is there a (however slight) inconsistency of terms here, or is it my > misunderstanding? That remains to be seen (please answer the questions above for a better understanding of your question). Maybe this helps though: global variables are a subset of free variables, and they are treated different for various reasons (some historic, some having to do with optimizations in the code -- I think you saw the latter in the bytecode). -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Fri Sep 10 00:08:44 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Sep 2010 08:08:44 +1000 Subject: [Python-Dev] terminology for "free variables" in Python In-Reply-To: References: Message-ID: On Fri, Sep 10, 2010 at 2:43 AM, Eli Bendersky wrote: > def some_func(myparam): > ??? def internalfunc(): > ??????? return cc * myparam > > CPython infers that in 'internalfunc', while 'myparam' is free, 'cc' is > global because 'cc' isn't bound in the enclosing scope, although according > to the definitions stated above, both should be considered free. The > bytecode generated for loading cc and myparam is different, of course. > > Is there a (however slight) inconsistency of terms here, or is it my > misunderstanding? There's a slight inconsistency. The names a code object explicitly calls out as free variables (i.e. references to cells in outer scopes) are only a subset of the full set of free variables (every referenced name that isn't a local variable or an attribute). >>> from dis import show_code >>> def outer(): ... x, y = 1, 2 ... def inner(): ... print (x, y, a, b, c.e) ... return inner ... >>> f = outer() >>> show_code(f) Name: inner Filename: Argument count: 0 Kw-only arguments: 0 Number of locals: 0 Stack size: 6 Flags: OPTIMIZED, NEWLOCALS, NESTED Constants: 0: None Names: 0: print 1: a 2: b 3: c 4: e Free variables: 0: y 1: x a, b, and c are also free variables in the more general sense, but the code object doesn't explicitly flag them as such since it doesn't need to do anything special with them. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From eliben at gmail.com Fri Sep 10 09:00:03 2010 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 10 Sep 2010 10:00:03 +0300 Subject: [Python-Dev] terminology for "free variables" in Python In-Reply-To: References: Message-ID: >> def some_func(myparam): > > def internalfunc(): > > return cc * myparam > > > > CPython infers that in 'internalfunc', while 'myparam' is free, 'cc' is > > What exactly do you mean by "infers" ? How do you know that it infers > that? How does it matter for your understanding of the code? > The easiest way I found to see what CPython thinks is use the 'symtable' module. With its help, it's clear that in the function above, myparam is considered free while cc is considered global. When querying symtable about the symbol myparam, the is_free method returns True while the is_global method returns False, and vice versa for cc. Of course it can also be seen in the code of symtable.c in function analyze_name, and as Nick showed in his message it also affects the way bytecode is generated for the two symbols. My intention in this post was to clarify whether I'm misunderstanding something or the term 'free' is indeed used for different things in different places. If this is the latter, IMHO it's an inconsistency, even if a small one. When I read the code I saw 'free' I went to the docs only to read that 'free' is something else. This was somewhat confusing. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Fri Sep 10 09:06:04 2010 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 10 Sep 2010 10:06:04 +0300 Subject: [Python-Dev] terminology for "free variables" in Python In-Reply-To: References: Message-ID: > There's a slight inconsistency. The names a code object explicitly > calls out as free variables (i.e. references to cells in outer scopes) > are only a subset of the full set of free variables (every referenced > name that isn't a local variable or an attribute). > > >>> from dis import show_code > >>> def outer(): > ... x, y = 1, 2 > ... def inner(): > ... print (x, y, a, b, c.e) > ... return inner > ... > >>> f = outer() > >>> show_code(f) > Nick, did you know that dis.show_code is neither exported by default from the dis module, nor it's documented in its help() or .rst documentation? Neither is code_info(), which is used by show_code(). I wonder if this is intentional. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Sep 10 14:41:59 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Sep 2010 22:41:59 +1000 Subject: [Python-Dev] terminology for "free variables" in Python In-Reply-To: References: Message-ID: On Fri, Sep 10, 2010 at 5:06 PM, Eli Bendersky wrote: > Nick, did you know that dis.show_code is neither exported by default from > the dis module, nor it's documented in its help() or .rst documentation? > Neither is code_info(), which is used by show_code(). I wonder if this is > intentional. code_info is in the normal documentation. I even remembered the versionadded tag without Georg reminding me ;) The omission from __all__ (and hence the module help text) was accidental and is now fixed. The omission of show_code from the documentation was deliberate, and I've now added a comment to that effect (the history is that dis.show_code has been around, but undocumented, for a while. The fact that it printed directly to stdout rather than producing a formatted string was mildly irritating, so I refactored the formatting part out into code_info, leaving just a single print call in show_code. Since I only kept show_code around for backwards compatibility reasons, I don't see any point in advertising its existence - better for people to just call code_info and print the result themselves. Although it *is* somewhat handy for quick introspection at the interpreter prompt... maybe I should document it after all. Thoughts? Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From eliben at gmail.com Fri Sep 10 15:23:57 2010 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 10 Sep 2010 16:23:57 +0300 Subject: [Python-Dev] terminology for "free variables" in Python In-Reply-To: References: Message-ID: On Fri, Sep 10, 2010 at 15:41, Nick Coghlan wrote: > On Fri, Sep 10, 2010 at 5:06 PM, Eli Bendersky wrote: > > Nick, did you know that dis.show_code is neither exported by default from > > the dis module, nor it's documented in its help() or .rst documentation? > > Neither is code_info(), which is used by show_code(). I wonder if this is > > intentional. > > code_info is in the normal documentation. I even remembered the > versionadded tag without Georg reminding me ;) > When you say "is in the normal documentation", do you mean you added it recently ? Although I see it here: http://docs.python.org/dev/py3k/library/dis.html, it's neither in the docs of 3.1.2 (http://docs.python.org/py3k/library/dis.html), nor in 2.7, nor in a build of 3.2 I have lying around from a couple of weeks ago. Although it *is* somewhat handy for quick introspection at the > interpreter prompt... maybe I should document it after all. Thoughts? > > I mostly use the dis module for quick-n-dirty exploration of the results of compilation into bytecode, and I'm sure many people use for the same effect. Thus show_code seems like a convenient shortcut, although not a necessary one. The string returned by code_info isn't interactive-shell friendly, and show_code saves the print(...). Personally I think that if it's there, it should be documented. If it's better not to use it, it should be removed or at least marked deprecated in the documentation/docstring. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Sep 10 16:16:01 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Sep 2010 00:16:01 +1000 Subject: [Python-Dev] terminology for "free variables" in Python In-Reply-To: References: Message-ID: On Fri, Sep 10, 2010 at 11:23 PM, Eli Bendersky wrote: > When you say "is in the normal documentation", do you mean you added it > recently ? Although I see it here: > http://docs.python.org/dev/py3k/library/dis.html, it's neither in the docs > of 3.1.2 (http://docs.python.org/py3k/library/dis.html), nor in 2.7, nor in > a build of 3.2 I have lying around from a couple of weeks ago. The module and docs changes both went in on August 17 as part of the same commit (r84133), so I'm not sure how you could have a local checkout with the module changes but not the doc changes. A checkout from early August wouldn't have either, of course. > I mostly use the dis module for quick-n-dirty exploration of the results of > compilation into bytecode, and I'm sure many people use for the same effect. > Thus show_code seems like a convenient shortcut, although not a necessary > one. The string returned by code_info isn't interactive-shell friendly, and > show_code saves the print(...). > > Personally I think that if it's there, it should be documented. If it's > better not to use it, it should be removed or at least marked deprecated in > the documentation/docstring. Yeah, I changed my mind and have now documented it properly. The 3.2 versionadded tag on show_code is currently a little questionable though. Guido actually checked in the original (undocumented) version of show_code before 3.0 was released. The only thing new about it in 3.2 is it being mentioned in the documentation. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From eliben at gmail.com Fri Sep 10 17:13:42 2010 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 10 Sep 2010 18:13:42 +0300 Subject: [Python-Dev] terminology for "free variables" in Python In-Reply-To: References: Message-ID: > > I mostly use the dis module for quick-n-dirty exploration of the results > of > > compilation into bytecode, and I'm sure many people use for the same > effect. > > Thus show_code seems like a convenient shortcut, although not a necessary > > one. The string returned by code_info isn't interactive-shell friendly, > and > > show_code saves the print(...). > > > > Personally I think that if it's there, it should be documented. If it's > > better not to use it, it should be removed or at least marked deprecated > in > > the documentation/docstring. > > Yeah, I changed my mind and have now documented it properly. The 3.2 > versionadded tag on show_code is currently a little questionable > though. Guido actually checked in the original (undocumented) version > of show_code before 3.0 was released. The only thing new about it in > 3.2 is it being mentioned in the documentation. > Looks good to me. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Sep 10 17:17:31 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 10 Sep 2010 08:17:31 -0700 Subject: [Python-Dev] terminology for "free variables" in Python In-Reply-To: References: Message-ID: On Fri, Sep 10, 2010 at 12:00 AM, Eli Bendersky wrote: >>> def some_func(myparam): >> >> > ??? def internalfunc(): >> > ??????? return cc * myparam >> > >> > CPython infers that in 'internalfunc', while 'myparam' is free, 'cc' is >> >> What exactly do you mean by "infers" ? How do you know that it infers >> that? How does it matter for your understanding of the code? > > The easiest way I found to see what CPython thinks is use the 'symtable' > module. With its help, it's clear that in the function above, myparam is > considered free while cc is considered global. When querying symtable about > the symbol myparam, the is_free method returns True while the is_global > method returns False, and vice versa for cc. > > Of course it can also be seen in the code of symtable.c in function > analyze_name, and as Nick showed in his message it also affects the way > bytecode is generated for the two symbols. > > My intention in this post was to clarify whether I'm misunderstanding > something or the term 'free' is indeed used for different things in > different places. If this is the latter, IMHO it's an inconsistency, even if > a small one. When I read the code I saw? 'free' I went to the docs only to > read that 'free' is something else. This was somewhat confusing. I'm still not clear if my explanation that globals are a subset of free variables got rid of the confusion. The full name for what CPython marks as "free" would be "free but not global" but that's too much of a mouthful. Also you're digging awfully deep into the implementation here -- AFAIC CPython could have called them "type A" and "type B" and there would not have been any problem for compliance with the langage reference. -- --Guido van Rossum (python.org/~guido) From eliben at gmail.com Fri Sep 10 17:30:35 2010 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 10 Sep 2010 18:30:35 +0300 Subject: [Python-Dev] terminology for "free variables" in Python In-Reply-To: References: Message-ID: > > > My intention in this post was to clarify whether I'm misunderstanding > > something or the term 'free' is indeed used for different things in > > different places. If this is the latter, IMHO it's an inconsistency, even > if > > a small one. When I read the code I saw 'free' I went to the docs only > to > > read that 'free' is something else. This was somewhat confusing. > > I'm still not clear if my explanation that globals are a subset of > free variables got rid of the confusion. The full name for what > CPython marks as "free" would be "free but not global" but that's too > much of a mouthful. > Yes, I understand it now. The source code of symtable.c has a long comment above the SET_SCOPE macro which says, among other things: "An implicit global is a free variable for which the compiler has found no binding in an enclosing function scope", which is in tune with what you said. > Also you're digging awfully deep into the implementation here -- > Indeed, it all started when I set to understand how symbol tables are implemented in CPython. The inconsistency in the usage of "free" confused me, so I consulted pydev for clarification. I'm no longer confused :-) Regards, Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From merwok at netwok.org Fri Sep 10 17:31:45 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 10 Sep 2010 17:31:45 +0200 Subject: [Python-Dev] terminology for "free variables" in Python In-Reply-To: References: Message-ID: <4C8A4F61.7030105@netwok.org> > Yeah, I changed my mind and have now documented it properly. The 3.2 > versionadded tag on show_code is currently a little questionable > though. Guido actually checked in the original (undocumented) version > of show_code before 3.0 was released. The only thing new about it in > 3.2 is it being mentioned in the documentation. versionadded marks the addition of a feature (see docs.python.org/documenting), so it should be removed here. Regards From status at bugs.python.org Fri Sep 10 18:08:35 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 10 Sep 2010 18:08:35 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100910160836.05E107818A@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-09-03 - 2010-09-10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues stats: open 2554 (+46) closed 19050 (+69) total 21604 (+65) Open issues with patches: 1078 Issues opened (46) ================== #8420: Objects/setobject.c contains unsafe code http://bugs.python.org/issue8420 reopened by eric.araujo #9129: DoS smtpd module vulnerability http://bugs.python.org/issue9129 reopened by giampaolo.rodola #9261: include higher (../../) dirs fails http://bugs.python.org/issue9261 reopened by eric.araujo #9759: GzipFile object should raise ValueError on .read() after .clos http://bugs.python.org/issue9759 opened by ack #9761: stale .pyc files aren't cleaned out http://bugs.python.org/issue9761 opened by Steve.Thompson #9762: build failures http://bugs.python.org/issue9762 opened by pitrou #9763: Crashes upon run after syntax error encountered in OSX 10.5.8 http://bugs.python.org/issue9763 opened by william.barr #9764: Tools/buildbot/external.bat should download and built tix http://bugs.python.org/issue9764 opened by stutzbach #9765: tcl-8 vs tcl8 http://bugs.python.org/issue9765 opened by stutzbach #9769: PyUnicode_FromFormatV() doesn't handle non-ascii text correctl http://bugs.python.org/issue9769 opened by haypo #9770: curses.isblank function doesn't match ctype.h http://bugs.python.org/issue9770 opened by kevinpt #9771: add an optional "default" argument to tokenize.detect_encoding http://bugs.python.org/issue9771 opened by flox #9772: test_pep277 failure on AMD64 debian parallel buildbot http://bugs.python.org/issue9772 opened by flox #9773: test_tarfile fails because of inaccurate mtime on AMD64 debian http://bugs.python.org/issue9773 opened by flox #9774: test_smtpnet fails with "[110] Connection timed out" on AMD64 http://bugs.python.org/issue9774 opened by flox #9775: Multiprocessing, logging and atexit play not well together http://bugs.python.org/issue9775 opened by aronacher #9778: Make hash values the same width as a pointer (or Py_ssize_t) http://bugs.python.org/issue9778 opened by pitrou #9779: argparse.ArgumentParser not support unicode in print help http://bugs.python.org/issue9779 opened by gkraser #9782: _multiprocessing.c warnings under 64-bit Windows http://bugs.python.org/issue9782 opened by pitrou #9783: _elementtree.c warnings under 64-bit Windows http://bugs.python.org/issue9783 opened by pitrou #9784: _msi.c warnings under 64-bit Windows http://bugs.python.org/issue9784 opened by pitrou #9786: Native TLS support for pthreads http://bugs.python.org/issue9786 opened by krisvale #9787: Release the TLS lock during allocations http://bugs.python.org/issue9787 opened by krisvale #9788: atexit and execution order http://bugs.python.org/issue9788 opened by giampaolo.rodola #9790: ntpath contains imports inside functions http://bugs.python.org/issue9790 opened by brian.curtin #9791: nntplib.py lacks a test suite http://bugs.python.org/issue9791 opened by giampaolo.rodola #9795: nntplib.NNTP should support the context protocol http://bugs.python.org/issue9795 opened by pitrou #9796: Add summary tables for unittest API http://bugs.python.org/issue9796 opened by eric.araujo #9800: Fast path for small int-indexing of lists and tuples http://bugs.python.org/issue9800 opened by pitrou #9801: Can not use append/extend to lists in a multiprocessing manage http://bugs.python.org/issue9801 opened by Jimbofbx #9802: Document 'stability' of builtin min() and max() http://bugs.python.org/issue9802 opened by mattheww #9803: IDLE closes with save while breakpoint open http://bugs.python.org/issue9803 opened by Jimbofbx #9807: deriving configuration information for different builds with t http://bugs.python.org/issue9807 opened by doko #9808: Implement os.getlogin on Windows http://bugs.python.org/issue9808 opened by janglin #9809: Wrong Registery Entries on win64 http://bugs.python.org/issue9809 opened by GreYFoX #9810: bzip2 build sometimes fails (VS8.0) http://bugs.python.org/issue9810 opened by ocean-city #9811: strftime strips '%' from unknown format codes on OS X http://bugs.python.org/issue9811 opened by sverrejoh #9812: cPickle segfault with nested dicts in threaded env http://bugs.python.org/issue9812 opened by kdombrowski #9814: subprocess isn't friendly to other Python implementations with http://bugs.python.org/issue9814 opened by dino.viehland #9815: test_tarfile sometimes ends with error "Cannot remoe dir" http://bugs.python.org/issue9815 opened by ocean-city #9817: expat copyright/license file is missing http://bugs.python.org/issue9817 opened by doko #9818: build files to build Lib/distutils/command/wininst-9.0* are mi http://bugs.python.org/issue9818 opened by doko #9820: Windows : os.listdir(b'.') doesn't raise an error for unencoda http://bugs.python.org/issue9820 opened by haypo #9821: Support PEP 383 on Windows: mbcs support of surrogateescape er http://bugs.python.org/issue9821 opened by haypo #9822: windows batch files are dependent on cmd current directory http://bugs.python.org/issue9822 opened by sorin #1058305: HTMLParser.locatestartagend regex too stringent http://bugs.python.org/issue1058305 reopened by r.david.murray Most recent 15 issues with no replies (15) ========================================== #9818: build files to build Lib/distutils/command/wininst-9.0* are mi http://bugs.python.org/issue9818 #9817: expat copyright/license file is missing http://bugs.python.org/issue9817 #9815: test_tarfile sometimes ends with error "Cannot remoe dir" http://bugs.python.org/issue9815 #9814: subprocess isn't friendly to other Python implementations with http://bugs.python.org/issue9814 #9810: bzip2 build sometimes fails (VS8.0) http://bugs.python.org/issue9810 #9803: IDLE closes with save while breakpoint open http://bugs.python.org/issue9803 #9800: Fast path for small int-indexing of lists and tuples http://bugs.python.org/issue9800 #9790: ntpath contains imports inside functions http://bugs.python.org/issue9790 #9783: _elementtree.c warnings under 64-bit Windows http://bugs.python.org/issue9783 #9782: _multiprocessing.c warnings under 64-bit Windows http://bugs.python.org/issue9782 #9779: argparse.ArgumentParser not support unicode in print help http://bugs.python.org/issue9779 #9774: test_smtpnet fails with "[110] Connection timed out" on AMD64 http://bugs.python.org/issue9774 #9773: test_tarfile fails because of inaccurate mtime on AMD64 debian http://bugs.python.org/issue9773 #9772: test_pep277 failure on AMD64 debian parallel buildbot http://bugs.python.org/issue9772 #9771: add an optional "default" argument to tokenize.detect_encoding http://bugs.python.org/issue9771 Most recent 15 issues waiting for review (15) ============================================= #9822: windows batch files are dependent on cmd current directory http://bugs.python.org/issue9822 #9815: test_tarfile sometimes ends with error "Cannot remoe dir" http://bugs.python.org/issue9815 #9812: cPickle segfault with nested dicts in threaded env http://bugs.python.org/issue9812 #9808: Implement os.getlogin on Windows http://bugs.python.org/issue9808 #9807: deriving configuration information for different builds with t http://bugs.python.org/issue9807 #9802: Document 'stability' of builtin min() and max() http://bugs.python.org/issue9802 #9800: Fast path for small int-indexing of lists and tuples http://bugs.python.org/issue9800 #9787: Release the TLS lock during allocations http://bugs.python.org/issue9787 #9786: Native TLS support for pthreads http://bugs.python.org/issue9786 #9775: Multiprocessing, logging and atexit play not well together http://bugs.python.org/issue9775 #9771: add an optional "default" argument to tokenize.detect_encoding http://bugs.python.org/issue9771 #9769: PyUnicode_FromFormatV() doesn't handle non-ascii text correctl http://bugs.python.org/issue9769 #9765: tcl-8 vs tcl8 http://bugs.python.org/issue9765 #9750: sqlite3 iterdump fails on column with reserved name http://bugs.python.org/issue9750 #9741: msgfmt.py generates invalid mo because msgfmt.make() does not http://bugs.python.org/issue9741 Top 10 most discussed issues (10) ================================= #4026: fcntl extension fails to build on AIX 6.1 http://bugs.python.org/issue4026 14 msgs #9598: untabify.py fails on files that contain non-ascii characters http://bugs.python.org/issue9598 12 msgs #9730: base64 docs refers to strings instead of bytes http://bugs.python.org/issue9730 11 msgs #941346: AIX shared library fix http://bugs.python.org/issue941346 10 msgs #9807: deriving configuration information for different builds with t http://bugs.python.org/issue9807 9 msgs #9662: ctypes not building under OS X because of ffi_closure_free not http://bugs.python.org/issue9662 8 msgs #9761: stale .pyc files aren't cleaned out http://bugs.python.org/issue9761 7 msgs #9778: Make hash values the same width as a pointer (or Py_ssize_t) http://bugs.python.org/issue9778 7 msgs #9786: Native TLS support for pthreads http://bugs.python.org/issue9786 7 msgs #8372: socket: Buffer overrun while reading unterminated AF_UNIX addr http://bugs.python.org/issue8372 6 msgs Issues closed (69) ================== #3805: sslobj.read py3k takes odd arguments http://bugs.python.org/issue3805 closed by pitrou #3899: test_ssl.py doesn't properly test ssl integration with asyncor http://bugs.python.org/issue3899 closed by pitrou #3978: ZipFileExt.read() can be incredibly slow; patch included http://bugs.python.org/issue3978 closed by pitrou #4194: default subprocess.Popen buffer size http://bugs.python.org/issue4194 closed by pitrou #4941: Tell GCC Py_DECREF is unlikely to call the destructor http://bugs.python.org/issue4941 closed by pitrou #4947: sys.stdout fails to use default encoding as advertised http://bugs.python.org/issue4947 closed by haypo #5506: io.BytesIO doesn't support the buffer protocol http://bugs.python.org/issue5506 closed by pitrou #5985: Implement os.path.samefile and os.path.sameopenfile on Windows http://bugs.python.org/issue5985 closed by ocean-city #6394: getppid support in os module on Windows http://bugs.python.org/issue6394 closed by amaury.forgeotdarc #6656: locale.format_string fails on escaped percentage http://bugs.python.org/issue6656 closed by r.david.murray #6768: asyncore file_wrapper leaking file descriptors? http://bugs.python.org/issue6768 closed by giampaolo.rodola #7451: improve json decoding performance http://bugs.python.org/issue7451 closed by pitrou #7566: Add ntpath.sameopenfile support for Windows http://bugs.python.org/issue7566 closed by brian.curtin #7736: os.listdir hangs since opendir() and closedir() do not release http://bugs.python.org/issue7736 closed by pitrou #7862: fileio.c: ValueError vs. IOError with impossible operations http://bugs.python.org/issue7862 closed by pitrou #7889: random produces different output on different architectures http://bugs.python.org/issue7889 closed by rhettinger #8312: Add post/pre hooks for distutils commands http://bugs.python.org/issue8312 closed by eric.araujo #8324: add a distutils test command http://bugs.python.org/issue8324 closed by eric.araujo #8574: transient_internet() (test_support): use socket.setdefaulttime http://bugs.python.org/issue8574 closed by pitrou #8597: build out-of-line asm on Windows http://bugs.python.org/issue8597 closed by skrah #8734: msvcrt get_osfhandle crash on bad FD http://bugs.python.org/issue8734 closed by pitrou #8956: Incorrect ValueError message for subprocess.Popen.send_signal( http://bugs.python.org/issue8956 closed by brian.curtin #9025: Non-uniformity in randrange for large arguments. http://bugs.python.org/issue9025 closed by rhettinger #9188: test_gdb fails for UCS2 builds with UCS2 gdb http://bugs.python.org/issue9188 closed by pitrou #9193: PEP 3149 (versioned .so files) reference implementation http://bugs.python.org/issue9193 closed by barry #9199: distutils upload command crashes when displaying server respon http://bugs.python.org/issue9199 closed by eric.araujo #9225: Replace DUP_TOPX with DUP_TOP_TWO http://bugs.python.org/issue9225 closed by pitrou #9270: distutils.util.getplatform and sysconfig.getplatform differ http://bugs.python.org/issue9270 closed by eric.araujo #9293: Unsupported IO operations should raise UnsupportedOperation http://bugs.python.org/issue9293 closed by pitrou #9324: signal.signal(bogus_signal, handler) segfaults on Windows http://bugs.python.org/issue9324 closed by brian.curtin #9379: random.randrange behaviour problems http://bugs.python.org/issue9379 closed by rhettinger #9410: Add Unladen Swallow's optimizations to Python 3's pickle. http://bugs.python.org/issue9410 closed by pitrou #9503: print statement hangs Windows service http://bugs.python.org/issue9503 closed by amaury.forgeotdarc #9581: PosixGroupsTester fails as root http://bugs.python.org/issue9581 closed by pitrou #9707: Reworked threading.local reference implementation http://bugs.python.org/issue9707 closed by pitrou #9719: build_ssl.py: cannot find 'asm64/*.*' http://bugs.python.org/issue9719 closed by loewis #9747: os.getresgid() documentation mentions "user ids", not "group i http://bugs.python.org/issue9747 closed by georg.brandl #9748: .inputrc magic-space breaks interactive mode http://bugs.python.org/issue9748 closed by amaury.forgeotdarc #9752: MSVC Compiler warning in Python\import.c http://bugs.python.org/issue9752 closed by stutzbach #9754: assertWarns and assertWarnsRegexp http://bugs.python.org/issue9754 closed by pitrou #9757: Add context manager protocol to memoryviews http://bugs.python.org/issue9757 closed by pitrou #9758: ioctl mutable buffer copying problem http://bugs.python.org/issue9758 closed by pitrou #9760: Suggestion for improving with documentation http://bugs.python.org/issue9760 closed by georg.brandl #9766: warnings has "onceregistry" and "once_registry" http://bugs.python.org/issue9766 closed by brett.cannon #9767: Failures in json doc examples http://bugs.python.org/issue9767 closed by georg.brandl #9768: IDLE / Black frame in active window http://bugs.python.org/issue9768 closed by ned.deily #9776: Inconsistent spacing in fcntl.fcntl docstring http://bugs.python.org/issue9776 closed by georg.brandl #9777: test_socket.GeneralModuleTests.test_idna should require the "n http://bugs.python.org/issue9777 closed by pitrou #9780: fill character cannot be '{' http://bugs.python.org/issue9780 closed by georg.brandl #9781: Dead link in Italian Tutorial http://bugs.python.org/issue9781 closed by benjamin.peterson #9785: _PyUnicode_New(), throw and memory problem http://bugs.python.org/issue9785 closed by Trigve.Siver #9789: Explicit buffer release for memoryview objects http://bugs.python.org/issue9789 closed by ncoghlan #9792: create_connection() recasts timeout errors http://bugs.python.org/issue9792 closed by pitrou #9793: Typo fix in What's New for 3.2: dynmaic -> dynamic http://bugs.python.org/issue9793 closed by eric.araujo #9794: socket.create_connection context manager http://bugs.python.org/issue9794 closed by giampaolo.rodola #9797: wrong assumption in pystate.c http://bugs.python.org/issue9797 closed by pitrou #9798: time.tzset() doesn't properly reset the time.timezone variable http://bugs.python.org/issue9798 closed by eric.araujo #9799: Compilation error for branch py3k on AIX 6 http://bugs.python.org/issue9799 closed by pitrou #9804: ascii() does not always join surrogate pairs http://bugs.python.org/issue9804 closed by pitrou #9805: Documentation on old-style formatting of dicts is overly restr http://bugs.python.org/issue9805 closed by r.david.murray #9806: no need to try loading posix extensions without SOABI http://bugs.python.org/issue9806 closed by benjamin.peterson #9813: Module Name Changed http://bugs.python.org/issue9813 closed by brett.cannon #9816: random.jumpahead and PRNG sequence independence http://bugs.python.org/issue9816 closed by rhettinger #9819: TESTFN_UNICODE and TESTFN_UNDECODABLE http://bugs.python.org/issue9819 closed by haypo #1552880: [Python2] Use utf-8 in the import machinery on Windows to supp http://bugs.python.org/issue1552880 closed by loewis #1718574: build_clib --build-clib/--build-temp option bugs http://bugs.python.org/issue1718574 closed by eric.araujo #1100562: deepcopying listlike and dictlike objects http://bugs.python.org/issue1100562 closed by pitrou #1053365: nntplib: add support for NNTP over SSL http://bugs.python.org/issue1053365 closed by pitrou #1479626: Uninstall does not clean registry http://bugs.python.org/issue1479626 closed by brian.curtin From g.brandl at gmx.net Fri Sep 10 22:32:25 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 10 Sep 2010 22:32:25 +0200 Subject: [Python-Dev] Garbage announcement printed on interpreter shutdown In-Reply-To: <20100808221846.CFD80EEA3F@mail.python.org> References: <20100808221846.CFD80EEA3F@mail.python.org> Message-ID: Hey #python-dev, I'd like to ask your opinion on this change; I think it should be reverted or at least made silent by default. Basically, it prints a warning like gc: 2 uncollectable objects at shutdown: Use gc.set_debug(gc.DEBUG_UNCOLLECTABLE) to list them. at interpreter shutdown if gc.garbage is nonempty. IMO this runs contrary to the decision we made when DeprecationWarnings were made silent by default: it spews messages not only at developers, but also at users, who don't need it and probably are going to be quite confused by it, assuming it came from their console application (imagine Mercurial printing this). Opinions? Georg Am 09.08.2010 00:18, schrieb antoine.pitrou: > Author: antoine.pitrou > Date: Mon Aug 9 00:18:46 2010 > New Revision: 83861 > > Log: > Issue #477863: Print a warning at shutdown if gc.garbage is not empty. > > > > Modified: > python/branches/py3k/Doc/library/gc.rst > python/branches/py3k/Doc/whatsnew/3.2.rst > python/branches/py3k/Include/pythonrun.h > python/branches/py3k/Lib/test/test_gc.py > python/branches/py3k/Misc/NEWS > python/branches/py3k/Modules/gcmodule.c > python/branches/py3k/Python/pythonrun.c > > Modified: python/branches/py3k/Doc/library/gc.rst > ============================================================================== > --- python/branches/py3k/Doc/library/gc.rst (original) > +++ python/branches/py3k/Doc/library/gc.rst Mon Aug 9 00:18:46 2010 > @@ -177,6 +177,15 @@ > If :const:`DEBUG_SAVEALL` is set, then all unreachable objects will be added to > this list rather than freed. > > + .. versionchanged:: 3.2 > + If this list is non-empty at interpreter shutdown, a warning message > + gets printed: > + > + :: > + > + gc: 2 uncollectable objects at shutdown: > + Use gc.set_debug(gc.DEBUG_UNCOLLECTABLE) to list them. > + > The following constants are provided for use with :func:`set_debug`: > > > @@ -197,6 +206,9 @@ > reachable but cannot be freed by the collector). These objects will be added to > the ``garbage`` list. > > + .. versionchanged:: 3.2 > + Also print the contents of the :data:`garbage` list at interpreter > + shutdown (rather than just its length), if it isn't empty. > > .. data:: DEBUG_SAVEALL > > > Modified: python/branches/py3k/Doc/whatsnew/3.2.rst > ============================================================================== > --- python/branches/py3k/Doc/whatsnew/3.2.rst (original) > +++ python/branches/py3k/Doc/whatsnew/3.2.rst Mon Aug 9 00:18:46 2010 > @@ -119,6 +119,11 @@ > * The :class:`ftplib.FTP` class now supports the context manager protocol > (Contributed by Tarek Ziad? and Giampaolo Rodol?; :issue:`4972`.) > > +* A warning message will now get printed at interpreter shutdown if > + the :data:`gc.garbage` list isn't empty. This is meant to make the > + programmer aware that his code contains object finalization issues. > + (Added by Antoine Pitrou; :issue:`477863`.) > + > * The :func:`shutil.copytree` function has two new options: > > * *ignore_dangling_symlinks*: when ``symlinks=False`` (meaning that the > > Modified: python/branches/py3k/Include/pythonrun.h > ============================================================================== > --- python/branches/py3k/Include/pythonrun.h (original) > +++ python/branches/py3k/Include/pythonrun.h Mon Aug 9 00:18:46 2010 > @@ -148,6 +148,7 @@ > PyAPI_FUNC(void) PyByteArray_Fini(void); > PyAPI_FUNC(void) PyFloat_Fini(void); > PyAPI_FUNC(void) PyOS_FiniInterrupts(void); > +PyAPI_FUNC(void) _PyGC_Fini(void); > > /* Stuff with no proper home (yet) */ > PyAPI_FUNC(char *) PyOS_Readline(FILE *, FILE *, char *); > > Modified: python/branches/py3k/Lib/test/test_gc.py > ============================================================================== > --- python/branches/py3k/Lib/test/test_gc.py (original) > +++ python/branches/py3k/Lib/test/test_gc.py Mon Aug 9 00:18:46 2010 > @@ -1,5 +1,5 @@ > import unittest > -from test.support import verbose, run_unittest > +from test.support import verbose, run_unittest, strip_python_stderr > import sys > import gc > import weakref > @@ -466,6 +466,42 @@ > # would be damaged, with an empty __dict__. > self.assertEqual(x, None) > > + def test_garbage_at_shutdown(self): > + import subprocess > + code = """if 1: > + import gc > + class X: > + def __init__(self, name): > + self.name = name > + def __repr__(self): > + return "" %% self.name > + def __del__(self): > + pass > + > + x = X('first') > + x.x = x > + x.y = X('second') > + del x > + if %d: > + gc.set_debug(gc.DEBUG_UNCOLLECTABLE) > + """ > + def run_command(code): > + p = subprocess.Popen([sys.executable, "-c", code], > + stdout=subprocess.PIPE, > + stderr=subprocess.PIPE) > + stdout, stderr = p.communicate() > + self.assertEqual(p.returncode, 0) > + self.assertEqual(stdout.strip(), b"") > + return strip_python_stderr(stderr) > + > + stderr = run_command(code % 0) > + self.assertIn(b"gc: 2 uncollectable objects at shutdown", stderr) > + self.assertNotIn(b"[, ]", stderr) > + # With DEBUG_UNCOLLECTABLE, the garbage list gets printed > + stderr = run_command(code % 1) > + self.assertIn(b"gc: 2 uncollectable objects at shutdown", stderr) > + self.assertIn(b"[, ]", stderr) > + > class GCTogglingTests(unittest.TestCase): > def setUp(self): > gc.enable() > > Modified: python/branches/py3k/Misc/NEWS > ============================================================================== > --- python/branches/py3k/Misc/NEWS (original) > +++ python/branches/py3k/Misc/NEWS Mon Aug 9 00:18:46 2010 > @@ -30,6 +30,8 @@ > Extensions > ---------- > > +- Issue #477863: Print a warning at shutdown if gc.garbage is not empty. > + > - Issue #6869: Fix a refcount problem in the _ctypes extension. > > - Issue #5504: ctypes should now work with systems where mmap can't > > Modified: python/branches/py3k/Modules/gcmodule.c > ============================================================================== > --- python/branches/py3k/Modules/gcmodule.c (original) > +++ python/branches/py3k/Modules/gcmodule.c Mon Aug 9 00:18:46 2010 > @@ -1295,17 +1295,16 @@ > > static struct PyModuleDef gcmodule = { > PyModuleDef_HEAD_INIT, > - "gc", > - gc__doc__, > - -1, > - GcMethods, > - NULL, > - NULL, > - NULL, > - NULL > + "gc", /* m_name */ > + gc__doc__, /* m_doc */ > + -1, /* m_size */ > + GcMethods, /* m_methods */ > + NULL, /* m_reload */ > + NULL, /* m_traverse */ > + NULL, /* m_clear */ > + NULL /* m_free */ > }; > > - > PyMODINIT_FUNC > PyInit_gc(void) > { > @@ -1364,6 +1363,37 @@ > return n; > } > > +void > +_PyGC_Fini(void) > +{ > + if (garbage != NULL && PyList_GET_SIZE(garbage) > 0) { > + PySys_WriteStderr( > + "gc: " > + "%" PY_FORMAT_SIZE_T "d uncollectable objects at shutdown:\n", > + PyList_GET_SIZE(garbage) > + ); > + if (debug & DEBUG_UNCOLLECTABLE) { > + PyObject *repr = NULL, *bytes = NULL; > + repr = PyObject_Repr(garbage); > + if (!repr || !(bytes = PyUnicode_EncodeFSDefault(repr))) > + PyErr_WriteUnraisable(garbage); > + else { > + PySys_WriteStderr( > + " %s\n", > + PyBytes_AS_STRING(bytes) > + ); > + } > + Py_XDECREF(repr); > + Py_XDECREF(bytes); > + } > + else { > + PySys_WriteStderr( > + " Use gc.set_debug(gc.DEBUG_UNCOLLECTABLE) to list them.\n" > + ); > + } > + } > +} > + > /* for debugging */ > void > _PyGC_Dump(PyGC_Head *g) > > Modified: python/branches/py3k/Python/pythonrun.c > ============================================================================== > --- python/branches/py3k/Python/pythonrun.c (original) > +++ python/branches/py3k/Python/pythonrun.c Mon Aug 9 00:18:46 2010 > @@ -404,6 +404,9 @@ > while (PyGC_Collect() > 0) > /* nothing */; > #endif > + /* We run this while most interpreter state is still alive, so that > + debug information can be printed out */ > + _PyGC_Fini(); > > /* Destroy all modules */ > PyImport_Cleanup(); > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From g.brandl at gmx.net Fri Sep 10 22:46:12 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 10 Sep 2010 22:46:12 +0200 Subject: [Python-Dev] terminology for "free variables" in Python In-Reply-To: References: Message-ID: Am 10.09.2010 14:41, schrieb Nick Coghlan: > On Fri, Sep 10, 2010 at 5:06 PM, Eli Bendersky wrote: >> Nick, did you know that dis.show_code is neither exported by default from >> the dis module, nor it's documented in its help() or .rst documentation? >> Neither is code_info(), which is used by show_code(). I wonder if this is >> intentional. > > code_info is in the normal documentation. I even remembered the > versionadded tag without Georg reminding me ;) > > The omission from __all__ (and hence the module help text) was > accidental and is now fixed. > > The omission of show_code from the documentation was deliberate, and > I've now added a comment to that effect (the history is that > dis.show_code has been around, but undocumented, for a while. The fact > that it printed directly to stdout rather than producing a formatted > string was mildly irritating, so I refactored the formatting part out > into code_info, leaving just a single print call in show_code. Since I > only kept show_code around for backwards compatibility reasons, I > don't see any point in advertising its existence - better for people > to just call code_info and print the result themselves. > > Although it *is* somewhat handy for quick introspection at the > interpreter prompt... maybe I should document it after all. Thoughts? IMO show_code() is not a good name, because the only thing it doesn't do is to -- show the code. I'd rather call it "codeinfo" (which also is more in line with current dis module function names). Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From fdrake at acm.org Fri Sep 10 22:57:47 2010 From: fdrake at acm.org (Fred Drake) Date: Fri, 10 Sep 2010 16:57:47 -0400 Subject: [Python-Dev] Garbage announcement printed on interpreter shutdown In-Reply-To: References: <20100808221846.CFD80EEA3F@mail.python.org> Message-ID: On Fri, Sep 10, 2010 at 4:32 PM, Georg Brandl wrote: > IMO this runs contrary to the decision we made when DeprecationWarnings were > made silent by default: it spews messages not only at developers, but also at > users, who don't need it and probably are going to be quite confused by it, Agreed; this should be silent by default. ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From amauryfa at gmail.com Fri Sep 10 23:10:44 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Fri, 10 Sep 2010 23:10:44 +0200 Subject: [Python-Dev] Garbage announcement printed on interpreter shutdown In-Reply-To: References: <20100808221846.CFD80EEA3F@mail.python.org> Message-ID: 2010/9/10 Fred Drake : > On Fri, Sep 10, 2010 at 4:32 PM, Georg Brandl wrote: >> IMO this runs contrary to the decision we made when DeprecationWarnings were >> made silent by default: it spews messages not only at developers, but also at >> users, who don't need it and probably are going to be quite confused by it, > > Agreed; this should be silent by default. +1. I suggest to enable it only when Py_DEBUG (or Py_TRACE_REFS or Py_REF_DEBUG?) is defined. -- Amaury Forgeot d'Arc From lukasz at langa.pl Fri Sep 10 22:55:52 2010 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Fri, 10 Sep 2010 22:55:52 +0200 Subject: [Python-Dev] Garbage announcement printed on interpreter shutdown In-Reply-To: References: <20100808221846.CFD80EEA3F@mail.python.org> Message-ID: Georg Brandl writes: > it prints a warning (...) at interpreter shutdown if gc.garbage is nonempty. > > IMO this runs contrary to the decision we made when DeprecationWarnings were > made silent by default > > Opinions? Agreed, this should be reverted for the reasons you give but DO LEAVE THIS on by default for regrtest (or maybe unittest in general) :) It has already proved useful for me. Is that doable? -- Best regards, ?ukasz Langa tel. +48 791 080 144 WWW http://lukasz.langa.pl/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel at stutzbachenterprises.com Fri Sep 10 23:55:46 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Fri, 10 Sep 2010 16:55:46 -0500 Subject: [Python-Dev] Garbage announcement printed on interpreter shutdown In-Reply-To: References: <20100808221846.CFD80EEA3F@mail.python.org> Message-ID: On Fri, Sep 10, 2010 at 3:32 PM, Georg Brandl wrote: > IMO this runs contrary to the decision we made when DeprecationWarnings > were > made silent by default: it spews messages not only at developers, but also > at > users, who don't need it and probably are going to be quite confused by it, > assuming it came from their console application (imagine Mercurial printing > this). > A non-empty gc.garbage often indicates that there is a bug in the program and that it is probably leaking memory [1]. That's a little different from a DeprecationWarning which doesn't indicate a bug; it just indicates that the program might not run correctly using a future version of Python. I think a better comparison would be with exceptions throw from a __del__, which (as far as I know) are still printed to the console. +1 on adding a way to enable/disable the feature. -1 on removing the feature -0 on making it disabled by default [1] I know that some large, long-running programs periodically check gc.garbage and carefully choose where to break cycles, but those are the exception and not the rule. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at openvpn.net Sat Sep 11 01:00:04 2010 From: james at openvpn.net (James Yonan) Date: Fri, 10 Sep 2010 17:00:04 -0600 Subject: [Python-Dev] Python needs a standard asynchronous return object Message-ID: <4C8AB874.9010703@openvpn.net> I'd like to propose that the Python community standardize on a "deferred" object for asynchronous return values, modeled after the well-thought-out Twisted Deferred class. With more and more Python libraries implementing asynchronicity (for example Futures -- PEP 3148), it's crucial to have a standard deferred object in place so that code using a single asynchronous reactor can interoperate with different asynchronous libraries. I think a lot of people don't realize how much cooler and more elegant it is to return a deferred object from an asynchronous function rather than using a generic callback approach (where you pass a function argument to the asynchronous function telling it where to call when the asynchronous operation completes). While asynchronous systems have been shown to have excellent scalability properties, the callback-based programming style often used in asynchronous programming has been criticized for breaking up the sequential readability of program logic. This problem is elegantly addressed by using Deferred Generators. Since Python 2.5 added enhanced generators (i.e. the capability for "yield" to return a value), the infrastructure is now in place to allow an asynchronous function to be written in a sequential style, without the use of explicit callbacks. See the following blog article for a nice write-up on the capability: http://blog.mekk.waw.pl/archives/14-Twisted-inlineCallbacks-and-deferredGenerator.html Mekk's Twisted Deferred example: @defer.inlineCallbacks def someFunction(): a = 1 b = yield deferredReturningFunction(a) c = yield anotherDeferredReturningFunction(a, b) defer.returnValue(c) What's cool about this is that between the two yield statements, the Twisted reactor is in control meaning that other pending asynchronous tasks can be attended to or the thread's remaining time slice can be yielded to the kernel, yet this is all accomplished without the use of multi-threading. Another interesting aspect of this approach is that since it leverages on Python's enhanced generators, an exception thrown inside either of the deferred-returning functions will be propagated through to someFunction() where it can be handled with try/except. Think about what this means -- this sort of emulates the "stackless" design pattern you would expect in Erlang or Stackless Python without leaving standard Python. And it's made possible under the hood by Python Enhanced Generators. Needless to say, it would be great to see this coolness be part of the standard Python library, instead of having every Python asynchronous library implement its own ad-hoc callback system. James Yonan From glyph at twistedmatrix.com Sat Sep 11 01:42:10 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 10 Sep 2010 19:42:10 -0400 Subject: [Python-Dev] Garbage announcement printed on interpreter shutdown In-Reply-To: References: <20100808221846.CFD80EEA3F@mail.python.org> Message-ID: <3A95C977-4C22-4DF0-9BB9-DC325932CED5@twistedmatrix.com> On Sep 10, 2010, at 5:10 PM, Amaury Forgeot d'Arc wrote: > 2010/9/10 Fred Drake : >> On Fri, Sep 10, 2010 at 4:32 PM, Georg Brandl wrote: >>> IMO this runs contrary to the decision we made when DeprecationWarnings were >>> made silent by default: it spews messages not only at developers, but also at >>> users, who don't need it and probably are going to be quite confused by it, >> >> Agreed; this should be silent by default. > > +1. I suggest to enable it only when Py_DEBUG (or Py_TRACE_REFS or > Py_REF_DEBUG?) is defined. Would it be possible to treat it the same way as a deprecation warning, and show it under the same conditions? It would be nice to know if my Python program is leaking uncollectable objects without rebuilding the interpreter. From ncoghlan at gmail.com Sat Sep 11 01:59:44 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Sep 2010 09:59:44 +1000 Subject: [Python-Dev] terminology for "free variables" in Python In-Reply-To: References: Message-ID: On Sat, Sep 11, 2010 at 6:46 AM, Georg Brandl wrote: > [me] >> Although it *is* somewhat handy for quick introspection at the >> interpreter prompt... maybe I should document it after all. Thoughts? > > IMO show_code() is not a good name, because the only thing it doesn't > do is to -- show the code. > > I'd rather call it "codeinfo" (which also is more in line with current > dis module function names). And, indeed, the variant I added that just returns the formatted string instead of printing it directly to stdout is called dis.code_info. dis.show_code is the existing helper that Guido added way back in 2007. As the checkin comment from back then put it, it shows you everything the interpreter knows about the code object except the details of the bytecode (which is already covered by dis.dis). So while I agree the name isn't great, I also don't think it is wrong enough to bother changing. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From guido at python.org Sat Sep 11 01:25:04 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 10 Sep 2010 16:25:04 -0700 Subject: [Python-Dev] Python needs a standard asynchronous return object In-Reply-To: <4C8AB874.9010703@openvpn.net> References: <4C8AB874.9010703@openvpn.net> Message-ID: Moving to python-ideas. Have you seen http://www.python.org/dev/peps/pep-3148/ ? That seems exactly what you want. --Guido On Fri, Sep 10, 2010 at 4:00 PM, James Yonan wrote: > I'd like to propose that the Python community standardize on a "deferred" > object for asynchronous return values, modeled after the well-thought-out > Twisted Deferred class. > > With more and more Python libraries implementing asynchronicity (for example > Futures -- PEP 3148), it's crucial to have a standard deferred object in > place so that code using a single asynchronous reactor can interoperate with > different asynchronous libraries. > > I think a lot of people don't realize how much cooler and more elegant it is > to return a deferred object from an asynchronous function rather than using > a generic callback approach (where you pass a function argument to the > asynchronous function telling it where to call when the asynchronous > operation completes). > > While asynchronous systems have been shown to have excellent scalability > properties, the callback-based programming style often used in asynchronous > programming has been criticized for breaking up the sequential readability > of program logic. > > This problem is elegantly addressed by using Deferred Generators. ?Since > Python 2.5 added enhanced generators (i.e. the capability for "yield" to > return a value), the infrastructure is now in place to allow an asynchronous > function to be written in a sequential style, without the use of explicit > callbacks. > > See the following blog article for a nice write-up on the capability: > > http://blog.mekk.waw.pl/archives/14-Twisted-inlineCallbacks-and-deferredGenerator.html > > Mekk's Twisted Deferred example: > > @defer.inlineCallbacks > def someFunction(): > ? ?a = 1 > ? ?b = yield deferredReturningFunction(a) > ? ?c = yield anotherDeferredReturningFunction(a, b) > ? ?defer.returnValue(c) > > What's cool about this is that between the two yield statements, the Twisted > reactor is in control meaning that other pending asynchronous tasks can be > attended to or the thread's remaining time slice can be yielded to the > kernel, yet this is all accomplished without the use of multi-threading. > ?Another interesting aspect of this approach is that since it leverages on > Python's enhanced generators, an exception thrown inside either of the > deferred-returning functions will be propagated through to someFunction() > where it can be handled with try/except. > > Think about what this means -- this sort of emulates the "stackless" design > pattern you would expect in Erlang or Stackless Python without leaving > standard Python. ?And it's made possible under the hood by Python Enhanced > Generators. > > Needless to say, it would be great to see this coolness be part of the > standard Python library, instead of having every Python asynchronous library > implement its own ad-hoc callback system. > > James Yonan > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Sat Sep 11 02:17:18 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Sep 2010 10:17:18 +1000 Subject: [Python-Dev] Garbage announcement printed on interpreter shutdown In-Reply-To: <3A95C977-4C22-4DF0-9BB9-DC325932CED5@twistedmatrix.com> References: <20100808221846.CFD80EEA3F@mail.python.org> <3A95C977-4C22-4DF0-9BB9-DC325932CED5@twistedmatrix.com> Message-ID: On Sat, Sep 11, 2010 at 9:42 AM, Glyph Lefkowitz wrote: > > On Sep 10, 2010, at 5:10 PM, Amaury Forgeot d'Arc wrote: > >> 2010/9/10 Fred Drake : >>> On Fri, Sep 10, 2010 at 4:32 PM, Georg Brandl wrote: >>>> IMO this runs contrary to the decision we made when DeprecationWarnings were >>>> made silent by default: it spews messages not only at developers, but also at >>>> users, who don't need it and probably are going to be quite confused by it, >>> >>> Agreed; this should be silent by default. >> >> +1. I suggest to enable it only when Py_DEBUG (or Py_TRACE_REFS or >> Py_REF_DEBUG?) is defined. > > Would it be possible to treat it the same way as a deprecation warning, and show it under the same conditions? ?It would be nice to know if my Python program is leaking uncollectable objects without rebuilding the interpreter. My suggestion: 1. Add a gc.WARN_UNCOLLECTABLE flag on gc.set_debug that enables the warning message. 2. Have regrtest explicitly set this for our own test suite As far as automatically turning it on for third party test suites goes, we could either: - require them to turn it on explicitly via gc.set_debug - have gc.WARN_UNCOLLECTABLE default to true for non-optimised runs (__debug__ == True) and false for runs with -O or -OO (__debug__ == False) - set it by looking at the -W arguments passed in at interpreter startup (e.g. enable it when all warnings are enabled, leave it disabled by default otherwise) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Sat Sep 11 02:28:11 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Sep 2010 10:28:11 +1000 Subject: [Python-Dev] [Python-checkins] r84685 - in python/branches/py3k: Doc/library/dis.rst Doc/reference/simple_stmts.rst Doc/whatsnew/3.2.rst Include/opcode.h Lib/opcode.py Lib/test/test_exceptions.py Lib/test/test_scope.py Lib/test/test_syntax.py Misc/NEWS Python Message-ID: On Sat, Sep 11, 2010 at 7:39 AM, amaury.forgeotdarc wrote: > There is no need to bump the PYC magic number: the new opcode is used > for code that did not compile before. If the magic number doesn't change for 3.2, how will 3.1 know it can't run pyc and pyo files containing these opcodes? The magic number needs a bump or this change may cause SystemErrors when older versions attempt to run affected pyc files. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From benjamin at python.org Sat Sep 11 02:33:10 2010 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 10 Sep 2010 19:33:10 -0500 Subject: [Python-Dev] [Python-checkins] r84685 - in python/branches/py3k: Doc/library/dis.rst Doc/reference/simple_stmts.rst Doc/whatsnew/3.2.rst Include/opcode.h Lib/opcode.py Lib/test/test_exceptions.py Lib/test/test_scope.py Lib/test/test_syntax.py Misc/NEWS Python In-Reply-To: References: Message-ID: 2010/9/10 Nick Coghlan : > On Sat, Sep 11, 2010 at 7:39 AM, amaury.forgeotdarc > wrote: >> There is no need to bump the PYC magic number: the new opcode is used >> for code that did not compile before. > > If the magic number doesn't change for 3.2, how will 3.1 know it can't > run pyc and pyo files containing these opcodes? The magic number is already bumped since 3.1. However, it's true that the number should be bumped anyway for good measure. -- Regards, Benjamin From ncoghlan at gmail.com Sat Sep 11 02:43:14 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Sep 2010 10:43:14 +1000 Subject: [Python-Dev] [Python-checkins] r84685 - in python/branches/py3k: Doc/library/dis.rst Doc/reference/simple_stmts.rst Doc/whatsnew/3.2.rst Include/opcode.h Lib/opcode.py Lib/test/test_exceptions.py Lib/test/test_scope.py Lib/test/test_syntax.py Misc/NEWS Python In-Reply-To: References: Message-ID: On Sat, Sep 11, 2010 at 10:33 AM, Benjamin Peterson wrote: > 2010/9/10 Nick Coghlan : >> On Sat, Sep 11, 2010 at 7:39 AM, amaury.forgeotdarc >> wrote: >>> There is no need to bump the PYC magic number: the new opcode is used >>> for code that did not compile before. >> >> If the magic number doesn't change for 3.2, how will 3.1 know it can't >> run pyc and pyo files containing these opcodes? > > The magic number is already bumped since 3.1. However, it's true that > the number should be bumped anyway for good measure. Yeah, I saw your subsequent checkin. I've updated the comment just above MAGIC and TAG to make it clearer when they should be changed. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From merwok at netwok.org Sat Sep 11 18:29:23 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sat, 11 Sep 2010 18:29:23 +0200 Subject: [Python-Dev] [Python-checkins] r84619 - python/branches/py3k/Misc/developers.txt In-Reply-To: <20100908104345.BA23DEE981@mail.python.org> References: <20100908104345.BA23DEE981@mail.python.org> Message-ID: <4C8BAE63.8010708@netwok.org> Author: georg.brandl Date: Wed Sep 8 12:43:45 2010 New Revision: 84619 Log: Add Lukasz. I guess you had to asciify ?ukasz? name because developers.txt is in Latin-1, which cannot encode the first character. I think the file should be recoded to UTF-8 so that we have no artificial restrictions on people?s names. I?ll wait some days and if nobody disagrees I?ll recode the file and fix the name. Regards From tjreedy at udel.edu Sat Sep 11 19:22:25 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 11 Sep 2010 13:22:25 -0400 Subject: [Python-Dev] [Python-checkins] r84619 - python/branches/py3k/Misc/developers.txt In-Reply-To: <4C8BAE63.8010708@netwok.org> References: <20100908104345.BA23DEE981@mail.python.org> <4C8BAE63.8010708@netwok.org> Message-ID: On 9/11/2010 12:29 PM, ?ric Araujo wrote: > Log: > Add Lukasz. > > I guess you had to asciify ?ukasz? name because developers.txt is in > Latin-1, which cannot encode the first character. I think the file > should be recoded to UTF-8 so that we have no artificial restrictions on > people?s names. I?ll wait some days and if nobody disagrees I?ll recode > the file and fix the name. I would agree, but also suggest a latin transcription for non-latin-alphabet names. -- Terry Jan Reedy From brett at python.org Sat Sep 11 20:57:15 2010 From: brett at python.org (Brett Cannon) Date: Sat, 11 Sep 2010 11:57:15 -0700 Subject: [Python-Dev] Garbage announcement printed on interpreter shutdown In-Reply-To: References: <20100808221846.CFD80EEA3F@mail.python.org> Message-ID: On Fri, Sep 10, 2010 at 14:55, Daniel Stutzbach wrote: > On Fri, Sep 10, 2010 at 3:32 PM, Georg Brandl wrote: >> >> IMO this runs contrary to the decision we made when DeprecationWarnings >> were >> made silent by default: it spews messages not only at developers, but also >> at >> users, who don't need it and probably are going to be quite confused by >> it, >> assuming it came from their console application (imagine Mercurial >> printing >> this). > > A non-empty gc.garbage often indicates that there is a bug in the program > and that it is probably leaking memory [1].? That's a little different from > a DeprecationWarning which doesn't indicate a bug; it just indicates that > the program might not run correctly using a future version of Python.? I > think a better comparison would be with exceptions throw from a __del__, > which (as far as I know) are still printed to the console. > Sure, but exceptions printed out by a __del__ method during interpreter shutdown are not explicitly done as part of the shutdown process, they just happen to typically be triggered by a shutdown. This gc info, OTOH, is explicitly debugging information that is only printed out (typically) at shutdown and thus at a point where it will no longer effect semantics or the performance of the application. So I view this as entirely debugging information and thus in no way critical for a user to know about. -Brett > +1 on adding a way to enable/disable the feature. > -1 on removing the feature > -0 on making it disabled by default > > [1] I know that some large, long-running programs periodically check > gc.garbage and carefully choose where to break cycles, but those are the > exception and not the rule. > -- > Daniel Stutzbach, Ph.D. > President, Stutzbach Enterprises, LLC > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > From merwok at netwok.org Sat Sep 11 21:05:08 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sat, 11 Sep 2010 21:05:08 +0200 Subject: [Python-Dev] [Python-checkins] r84619 - python/branches/py3k/Misc/developers.txt In-Reply-To: References: <20100908104345.BA23DEE981@mail.python.org> <4C8BAE63.8010708@netwok.org> Message-ID: <4C8BD2E4.2040207@netwok.org> > I would agree, but also suggest a latin transcription for > non-latin-alphabet names. I think that people whose alphabet is for example Cyrillic already use a Latin transliteration in those files, so it?s good. Are you also including names using extended Latin alphabets (like ?ukasz) in your suggestion? Regards From tjreedy at udel.edu Sun Sep 12 08:27:17 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 12 Sep 2010 02:27:17 -0400 Subject: [Python-Dev] [Python-checkins] r84619 - python/branches/py3k/Misc/developers.txt In-Reply-To: <4C8BD2E4.2040207@netwok.org> References: <20100908104345.BA23DEE981@mail.python.org> <4C8BAE63.8010708@netwok.org> <4C8BD2E4.2040207@netwok.org> Message-ID: On 9/11/2010 3:05 PM, ?ric Araujo wrote: >> I would agree, but also suggest a latin transcription for >> non-latin-alphabet names. > > I think that people whose alphabet is for example Cyrillic already use a > Latin transliteration in those files, so it?s good. At present, such people have no choice ;-). > Are you also > including names using extended Latin alphabets (like ?ukasz) in your > suggestion? No. -- Terry Jan Reedy From merwok at netwok.org Sun Sep 12 10:36:32 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sun, 12 Sep 2010 10:36:32 +0200 Subject: [Python-Dev] [Python-checkins] r84619 - python/branches/py3k/Misc/developers.txt In-Reply-To: References: <20100908104345.BA23DEE981@mail.python.org> <4C8BAE63.8010708@netwok.org> <4C8BD2E4.2040207@netwok.org> Message-ID: <4C8C9110.1060203@netwok.org> >> I think that people whose alphabet is for example Cyrillic already use a >> Latin transliteration in those files, so it?s good. > At present, such people have no choice ;-). Right :) So the new policy of real name thanks to UTF-8 + ASCII transliteration is a superset of the existing conditions, which should be fine with everyone (i.e. names can finally be written with whatever characters they require, and people in the old world can read it with non-UTF 8-capable tools). >> Are you also including names using extended Latin alphabets (like >> ?ukasz) in your suggestion? > No. Then it?s fine. Thanks for the feedback. Regards From solipsis at pitrou.net Sun Sep 12 12:38:24 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 12 Sep 2010 12:38:24 +0200 Subject: [Python-Dev] r84727 - in python/branches/py3k: Lib/collections.py Misc/NEWS References: <20100912041242.C927AFA7A@mail.python.org> Message-ID: <20100912123824.4f0baf2f@pitrou.net> On Sun, 12 Sep 2010 06:12:42 +0200 (CEST) raymond.hettinger wrote: > > - # Each link is stored as a list of length three: [PREV, NEXT, KEY]. > + # The back links are weakref proxies (to prevent circular references). Are you sure this prevents anything? Since your list is circular, forward links are enough to build a reference cycle. Actually, this is exemplified here: > + self.__root = root = _Link() # sentinel node for the doubly linked list > + root.prev = root.next = root `root.next = root` is enough to create a cycle, even if you make root.prev a weakref. What you need is to make both prev and next weakrefs. All list nodes all held by the __map dict anyway. If you are bothered about speed, an alternative would be a classic finalization scheme, using a class-wide set of OrderedDict weakrefs that remove themselves and clean up the doubly-linked list when the OrderedDict becomes dead. From martin at v.loewis.de Sun Sep 12 15:51:43 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 12 Sep 2010 15:51:43 +0200 Subject: [Python-Dev] versioned .so files for Python 3.2 In-Reply-To: <4C867A71.1000105@egenix.com> References: <20100624115048.4fd152e3@heresy> <20100714195955.4815f980@heresy> <4C40372C.9030101@ubuntu.com> <20100722164036.7a80d27c@snowdog> <20100724002509.0ad8d359@snowdog> <4C78E4F0.9090306@v.loewis.de> <7BA45755-EF5F-4A4E-9E47-21C9A80C29B4@mac.com> <4C7CC9ED.9090405@egenix.com> <4C82510C.9010405@v.loewis.de> <4C867A71.1000105@egenix.com> Message-ID: <4C8CDAEF.3070607@v.loewis.de> Am 07.09.2010 19:46, schrieb M.-A. Lemburg: > "Martin v. L?wis" wrote: >>> -1 on always using wchar_t as well. Python's default is UCS2 and the >>> stable ABI should not change that. >> >> It's not really Python's default. It is what configure.in does by >> default. Python's default, on Linux, is UCS-4. > > No, the default is UCS2 on all platforms and in configure.in. > > configure.in only uses UCS4 if it finds a TCL installation that > happens to use UCS4 - for some reason I don't know :-) > > However, most Linux distros and more recently also some BSDs > have switched over to using UCS4 for their distribution versions > of Python. Hmm. So UCS4 *is* the default for Linux. The default on the system is not what Python's configure makes it, but what the system vendors make it - they are the ones making the system, after all. Regards, Martin From martin at v.loewis.de Sun Sep 12 16:10:40 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 12 Sep 2010 16:10:40 +0200 Subject: [Python-Dev] PEP 3149 thoughts In-Reply-To: <20100907184541.3dc917b7@mission> References: <4C83D1BA.4050505@v.loewis.de> <20100907184541.3dc917b7@mission> Message-ID: <4C8CDF60.1010702@v.loewis.de> > I only mandated ./configure-based builds to be PEP 3149 conforming. I have no > objection to expanding the PEP to include Windows builds, but I'm not sure > it's necessary and it would take a Windows build expert to make and test those > changes. > > Does PEP 3149 have any advantage for Windows installations? I think this is resolved by now, responding anyway: I don't see any immediate advantage. The problem that the PEP addresses typically doesn't exist on Windows. >> 2. Why does the PEP recommend installing stuff >> into /usr/share/pyshared? > > It's just a suggestion, but as it turns out, probably an incorrect one. I'll > rephrase this to make it clear that it's up to the distribution as to where > exactly these files get installed. Nothing about this PEP changes the default > from-source installation directory. Is the rephrasing done already? I still wonder why you suggest /usr/share should be used, when the FHS says it should be /usr/lib. This keeps confusing me, despite not being part of the specification. Regards, Martin From martin at v.loewis.de Sun Sep 12 16:22:08 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 12 Sep 2010 16:22:08 +0200 Subject: [Python-Dev] PEP 384 status In-Reply-To: <4C867B01.6070609@egenix.com> References: <4C78DF1A.1060601@v.loewis.de> <4C795B06.1010903@v.loewis.de> <20100829102447.115448f2@pitrou.net> <20100829111043.4d786fe6@pitrou.net> <20100829174314.7129953a@pitrou.net> <20100829234349.50cdcabf@pitrou.net> <20100830103525.71039548@heresy> <4C7BC469.2030109@voidspace.org.uk> <4C7CD16B.1000604@egenix.com> <4C7D798A.3040200@egenix.com> <4C8244C5.9020401@v.loewis.de> <4C867B01.6070609@egenix.com> Message-ID: <4C8CE210.6000305@v.loewis.de> Am 07.09.2010 19:48, schrieb M.-A. Lemburg: > "Martin v. L?wis" wrote: >> >>> This sounds like the issues such a mix can cause are mostly >>> theoretical and don't really bother much in practice, so >>> PEP 384 on Windows does have a chance :-) >> >> Actually, the CRT issues (FILE* in particular) have been >> causing real crashes for many years, for many people. > > Do you have some pointers ? You mean, other than FILE* :-? Unfortunately, I don't have time to search for reports, but they had been several on comp.lang.python over the years. As others say, this is *really* easy to reproduce. > I don't remember this being a real practical issue, at least > not for different versions of the MS CRTs. You probably had not been passing FILE* across CRT boundaries, then. Notice that this is an uncommon thing to do, except that the Python API has some functions that expect FILE*. As others have mentioned, CRT file handles are a problem, too, but less so: a) they aren't explicitly passed in any of the Python APIs (except for the obvious os.* functions), b) they don't cause crashes, just outputs to the wrong files, and c) if they are 0/1/2, they actually do the right thing. Regards, Martin From steve at holdenweb.com Sun Sep 12 18:49:06 2010 From: steve at holdenweb.com (Steve Holden) Date: Sun, 12 Sep 2010 12:49:06 -0400 Subject: [Python-Dev] Volunteer help with porting In-Reply-To: <4C86491E.3090000@voidspace.org.uk> References: <20100907133330.GA32192@phd.pp.ru> <20100907140259.2058.1140393887.divmod.xquotient.453@localhost.localdomain> <4C86491E.3090000@voidspace.org.uk> Message-ID: On 9/7/2010 10:15 AM, Michael Foord wrote: > On 07/09/2010 15:02, exarkun at twistedmatrix.com wrote: >> On 01:33 pm, phd at phd.pp.ru wrote: >>> Hello. Thank you for the offer! >>> >>> On Tue, Sep 07, 2010 at 06:36:10PM +0530, Prashant Kumar wrote: >>>> My name is Prashant Kumar and I wish to contribute to the Python >>>> development >>>> process by helping convert certain existing python >>>> over to python3k. >>>> >>>> Is there anyway I could obtain a list of libraries which need to be >>>> ported >>>> over to python3k, sorted by importance(by importance i mean >>>> packages which serve as a dependency for larger number of packages >>>> being >>>> more important). >>> >>> As there is already Python 3.2 alpha, the core of Python has already >>> been ported - and this mailing list is for discussion of development >>> of the >>> very Python, not about working on 3rd-party libraries. >> >> How about the email package? > > Right, and there are other standard library modules (cgi, ftplib, > nntplib, etc) that either need fixing or auditing as to how they handle > bytes / strings. > Including, to my certain knowledge, the mailbox handling code, though writing code to read them sequentially is fairly easy. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From g.brandl at gmx.net Sun Sep 12 19:21:11 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 12 Sep 2010 19:21:11 +0200 Subject: [Python-Dev] r84744 - python/branches/py3k/Objects/unicodeobject.c In-Reply-To: <20100912164053.CB6F9EBDE@mail.python.org> References: <20100912164053.CB6F9EBDE@mail.python.org> Message-ID: Victor changed this from "return NULL" to "goto fail" in r84730, claiming that it would fix a reference leak. Is the leak somewhere else then? Georg Am 12.09.2010 18:40, schrieb benjamin.peterson: > Author: benjamin.peterson > Date: Sun Sep 12 18:40:53 2010 > New Revision: 84744 > > Log: > use return NULL; it's just as correct > > Modified: > python/branches/py3k/Objects/unicodeobject.c > > Modified: python/branches/py3k/Objects/unicodeobject.c > ============================================================================== > --- python/branches/py3k/Objects/unicodeobject.c (original) > +++ python/branches/py3k/Objects/unicodeobject.c Sun Sep 12 18:40:53 2010 > @@ -769,7 +769,7 @@ > "PyUnicode_FromFormatV() expects an ASCII-encoded format " > "string, got a non-ASCII byte: 0x%02x", > (unsigned char)*f); > - goto fail; > + return NULL; > } > } > /* step 2: allocate memory for the results of -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From martin at v.loewis.de Sun Sep 12 19:31:36 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 12 Sep 2010 19:31:36 +0200 Subject: [Python-Dev] PEP 3149 thoughts In-Reply-To: <87sk1kfzpn.fsf@muni.brainbot.com> References: <4C83D1BA.4050505@v.loewis.de> <4C840E17.4030105@v.loewis.de> <20100907195038.1105f5f3@mission> <87sk1kfzpn.fsf@muni.brainbot.com> Message-ID: <4C8D0E78.40601@v.loewis.de> > If it's useful on unix like systems, why shouldn't it be useful on > windows? Multiple versions of python can be installed on windows > too. And people might also like to share their packages between > installations. Multiple versions of Python install into distinct directories, so extension modules don't conflict. There is currently no provision for a central directory where Python packages get installed on Windows across Python versions, so the main motivation for the PEP doesn't exist. The secondary motivation, to have both debug and non-debug versions of an extension module in the same folder, is already provided by the _d.pyd mechanism on Windows (which was introduced before the PEP process was established). Of course, somebody could come along and propose that extension modules should install into "\Program Files\Common\Python3" (say), and that Python versions starting from 3.3 (or so) should include that directory in their search path by default. IMO, that would be a separate PEP, and I would hope that *if* such a location is established, the PEP 3149 motivation is not relevant anymore, since extension modules installed into such a location might use the stable ABI, anyway. Regards, Martin From benjamin at python.org Sun Sep 12 19:37:53 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 12 Sep 2010 12:37:53 -0500 Subject: [Python-Dev] r84744 - python/branches/py3k/Objects/unicodeobject.c In-Reply-To: References: <20100912164053.CB6F9EBDE@mail.python.org> Message-ID: 2010/9/12 Georg Brandl : > Victor changed this from "return NULL" to "goto fail" in r84730, claiming > that it would fix a reference leak. ?Is the leak somewhere else then? Indeed. He missed my earlier fix, though. :) -- Regards, Benjamin From martin at v.loewis.de Sun Sep 12 19:38:33 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 12 Sep 2010 19:38:33 +0200 Subject: [Python-Dev] PEP 384 status In-Reply-To: <20100906134533.57b3712b@pitrou.net> References: <4C78DF1A.1060601@v.loewis.de> <20100906134533.57b3712b@pitrou.net> Message-ID: <4C8D1019.1050201@v.loewis.de> > On http://bugs.python.org/issue9778 you elaborated on what the PEP would > entail in its current state: > > ?No, vice versa. The PEP promises that the ABI won't change until > Python 4. For any change that might break the ABI, either a > backwards-compatible solution needs to be found, or the change be > deferred to Python 4.? > > This sounds like it could be detrimental by blocking desired > improvements (the aforementioned issue is a potential example of this). Notice that it's only potential: in the specific case, there would be an ABI-compatible way of introducing wide hashes, using a second type slot. > Do you think it would complicate things a lot to version the ABI itself? It would miss the point. The ABI deliberately restricts itself to features that really have been stable, and we should be committed to keeping that subset stable. If you think this is too restrictive, please point out specific aspects that you think might need to change in the mid-term future. They should then be excluded from the ABI. > Actually, PYTHON_API_VERSION is already used as some kind of ABI tag > (since it's checked at module load time rather than at compile time). It's too late, though: when the module is being created, the code using the ABI is already running. As I wrote in the issue: the *real* ABI version is "python3.dll". Regards, Martin From solipsis at pitrou.net Sun Sep 12 20:00:00 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 12 Sep 2010 20:00:00 +0200 Subject: [Python-Dev] PEP 384 status In-Reply-To: <4C8D1019.1050201@v.loewis.de> References: <4C78DF1A.1060601@v.loewis.de> <20100906134533.57b3712b@pitrou.net> <4C8D1019.1050201@v.loewis.de> Message-ID: <20100912200000.6494432e@pitrou.net> On Sun, 12 Sep 2010 19:38:33 +0200 "Martin v. L?wis" wrote: > > On http://bugs.python.org/issue9778 you elaborated on what the PEP would > > entail in its current state: > > > > ?No, vice versa. The PEP promises that the ABI won't change until > > Python 4. For any change that might break the ABI, either a > > backwards-compatible solution needs to be found, or the change be > > deferred to Python 4.? > > > > This sounds like it could be detrimental by blocking desired > > improvements (the aforementioned issue is a potential example of this). > > Notice that it's only potential: in the specific case, there would be > an ABI-compatible way of introducing wide hashes, using a second type > slot. Yes, but it would add complication, and be potentially detrimental to performance. > If you think this is too restrictive, please point out specific aspects > that you think might need to change in the mid-term future. They should > then be excluded from the ABI. I have no a priori knowledge of what might happen in the future :) That said, looking at the PEP, I was wondering whether fields such as ob_type, ob_refcnt, ob_size have to be directly accessible, rather than through a macro-turned-into-a-function such as Py_REFCNT(). From steve at holdenweb.com Sun Sep 12 21:44:21 2010 From: steve at holdenweb.com (Steve Holden) Date: Sun, 12 Sep 2010 15:44:21 -0400 Subject: [Python-Dev] Volunteer help with porting In-Reply-To: <20100907153706.DE96F2221ED@kimball.webabinitio.net> References: <20100907133330.GA32192@phd.pp.ru> <20100907140259.2058.1140393887.divmod.xquotient.453@localhost.localdomain> <20100907143449.GA9568@phd.pp.ru> <20100907153706.DE96F2221ED@kimball.webabinitio.net> Message-ID: On 9/7/2010 11:37 AM, R. David Murray wrote: > On Tue, 07 Sep 2010 18:34:49 +0400, Oleg Broytman wrote: >> On Tue, Sep 07, 2010 at 02:02:59PM -0000, exarkun at twistedmatrix.com wrote: >>> On 01:33 pm, phd at phd.pp.ru wrote: >>>> As there is already Python 3.2 alpha, the core of Python has already >>>> been ported >>> >>> How about the email package? >> >> What about email? It is a core library, right? It has been ported AFAIK. >> Or have I missed something? > > Some bug reports, perhaps? > > Email has been "ported" in the sense that all the existing tests pass > under py3k. It is not, however, fully functional: it cannot handle > input that contains binary data with the high bit set. (To put it another > way, the python3 email package currently handles only 7bit clean data.) > This has various unfortunate consequences for other standard library > packages (cgi, nntp, etc) that depend directly or indirectly on the > email package. > > To fix this the email package API and a chunk of the internals needs > to be redesigned so that it can handle bytes while still providing a > string-based interface to the textual data. This is no small task, > since email in python2 has been getting away with all kinds of dirty > tricks because of the lack of a clean bytes/string separation in python2. > > See http://launchpad.net/python-email6 for current status of the project. > > Prashant Kumar, if you would like to help with the email6 project, please > sign up to the email-sig mailing list and introduce yourself there. > All help is welcome! It is not a "porting" project strictly > speaking, but it certainly is interesting :) > And if anyone knows people who would help with *funding* this effort the PSF very much wants to talk to them. This ought logically to include everyone using "Mailman". I would imagine if we had $10 from 1% of its users we would be able to fund the effort comfortably. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From martin at v.loewis.de Sun Sep 12 22:26:54 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 12 Sep 2010 22:26:54 +0200 Subject: [Python-Dev] PEP 384 status In-Reply-To: <20100912200000.6494432e@pitrou.net> References: <4C78DF1A.1060601@v.loewis.de> <20100906134533.57b3712b@pitrou.net> <4C8D1019.1050201@v.loewis.de> <20100912200000.6494432e@pitrou.net> Message-ID: <4C8D378E.9010507@v.loewis.de> > That said, looking at the PEP, I was wondering whether fields such as > ob_type, ob_refcnt, ob_size have to be directly accessible, rather than > through a macro-turned-into-a-function such as Py_REFCNT(). That they are macros still is primarily for performance reasons. For ob_type, that may be questionable; for Py_INCREF, I hope you agree that it's really desirable for it to be inline. However, more importantly, user-defined objects need to start with the standard object header. I really see no way to let extensions define types without them also being able to access their own struct fields, which in turn requires compile-time (ABI) knowledge of PyObject_HEAD. This is also the reason why Py_TRACE_REFS can't be supported in the ABI: it increases sizeof(PyObject), and thus shifts all user fields in the object. I actually *do* have a priori knowledge of what might happen in the future :-): If we were to change the layout of the object header in a significant way, then the majority of extension modules would break - not only on the binary level, but also on the source level. So any change to this likely justifies calling it Python 4. (I would be that even Python 4 keeps that structure) Regards, Martin From merwok at netwok.org Mon Sep 13 00:53:39 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Mon, 13 Sep 2010 00:53:39 +0200 Subject: [Python-Dev] [Python-checkins] r84743 - in python/branches/py3k: Doc/license.rst Misc/NEWS Modules/expat/COPYING In-Reply-To: <20100912163158.7902EF11D@mail.python.org> References: <20100912163158.7902EF11D@mail.python.org> Message-ID: <4C8D59F3.9020101@netwok.org> +zlib +---- + +The :mod:`zlib` extension is built using an included copy of the zlib +sources unless the zlib version found on the system is too old to be +used for the build:: Unless or if? Building with an included copy *if* the system one is too old makes sense to me, not the contrary. Am I not seeing something? Regards From barry at python.org Sat Sep 11 00:23:43 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 10 Sep 2010 18:23:43 -0400 Subject: [Python-Dev] PEP 3149 thoughts In-Reply-To: <4C8CDF60.1010702@v.loewis.de> References: <4C83D1BA.4050505@v.loewis.de> <20100907184541.3dc917b7@mission> <4C8CDF60.1010702@v.loewis.de> Message-ID: <20100910182343.35e18b8b@mission> On Sep 12, 2010, at 04:10 PM, Martin v. L?wis wrote: >> Does PEP 3149 have any advantage for Windows installations? > >I don't see any immediate advantage. The problem that the PEP addresses >typically doesn't exist on Windows. Cool. >>> 2. Why does the PEP recommend installing stuff >>> into /usr/share/pyshared? >> >> It's just a suggestion, but as it turns out, probably an incorrect >> one. I'll rephrase this to make it clear that it's up to the >> distribution as to where exactly these files get installed. Nothing >> about this PEP changes the default from-source installation >> directory. > >Is the rephrasing done already? Yes, but I forgot to commit the change. Done now. >I still wonder why you suggest /usr/share should be used, when the FHS >says it should be /usr/lib. This keeps confusing me, despite not being >part of the specification. I don't, as you should now see. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From solipsis at pitrou.net Mon Sep 13 16:36:19 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 13 Sep 2010 16:36:19 +0200 Subject: [Python-Dev] r84775 - peps/trunk/pep-3149.txt References: <20100913141844.28C78FB1E@mail.python.org> Message-ID: <20100913163619.64c8d50a@pitrou.net> On Mon, 13 Sep 2010 16:18:44 +0200 (CEST) barry.warsaw wrote: > > -For an arbitrary package `foo`, you would see these files when the > +For an arbitrary package `foo`, you might see these files when the > distribution package was installed:: > > - /usr/share/pyshared/foo.cpython-32m.so > - /usr/share/pyshared/foo.cpython-33m.so > + /usr/lib/python/foo.cpython-32m.so > + /usr/lib/python/foo.cpython-33m.so Are these the default paths? The PEP doesn't say how a distribution is supposed to choose its PEP 3149 filesystem layout (instead of /usr/lib/python3.2/site-packages). Thanks Antoine. From vinay_sajip at yahoo.co.uk Mon Sep 13 16:37:25 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 13 Sep 2010 14:37:25 +0000 (UTC) Subject: [Python-Dev] Python needs a standard asynchronous return object References: <4C8AB874.9010703@openvpn.net> Message-ID: James Yonan openvpn.net> writes: > I'd like to propose that the Python community standardize on a > "deferred" object for asynchronous return values Have a look at PEP 3148: http://www.python.org/dev/peps/pep-3148/ From vinay_sajip at yahoo.co.uk Mon Sep 13 16:39:18 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 13 Sep 2010 14:39:18 +0000 (UTC) Subject: [Python-Dev] Python needs a standard asynchronous return object References: <4C8AB874.9010703@openvpn.net> Message-ID: James Yonan openvpn.net> writes: > > I'd like to propose that the Python community standardize on a > "deferred" object for asynchronous return values, modeled after the > well-thought-out Twisted Deferred class. Ummm, sorry, I seem to need new glasses :-( ignore my last post. From barry at python.org Mon Sep 13 16:55:16 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 13 Sep 2010 10:55:16 -0400 Subject: [Python-Dev] r84775 - peps/trunk/pep-3149.txt In-Reply-To: <20100913163619.64c8d50a@pitrou.net> References: <20100913141844.28C78FB1E@mail.python.org> <20100913163619.64c8d50a@pitrou.net> Message-ID: <20100913105516.2a237510@mission> On Sep 13, 2010, at 04:36 PM, Antoine Pitrou wrote: >On Mon, 13 Sep 2010 16:18:44 +0200 (CEST) >barry.warsaw wrote: >> >> -For an arbitrary package `foo`, you would see these files when the >> +For an arbitrary package `foo`, you might see these files when the >> distribution package was installed:: >> >> - /usr/share/pyshared/foo.cpython-32m.so >> - /usr/share/pyshared/foo.cpython-33m.so >> + /usr/lib/python/foo.cpython-32m.so >> + /usr/lib/python/foo.cpython-33m.so > >Are these the default paths? The PEP doesn't say how a distribution >is supposed to choose its PEP 3149 filesystem layout (instead of >/usr/lib/python3.2/site-packages). Why should it? Distributions are going to make their own decisions independent of the PEP. That's why s/would/might/ in the above change. I'm open to suggestions for better ways to explain it. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From solipsis at pitrou.net Mon Sep 13 17:04:52 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 13 Sep 2010 17:04:52 +0200 Subject: [Python-Dev] r84775 - peps/trunk/pep-3149.txt References: <20100913141844.28C78FB1E@mail.python.org> <20100913163619.64c8d50a@pitrou.net> <20100913105516.2a237510@mission> Message-ID: <20100913170452.747c2890@pitrou.net> On Mon, 13 Sep 2010 10:55:16 -0400 Barry Warsaw wrote: > On Sep 13, 2010, at 04:36 PM, Antoine Pitrou wrote: > > >On Mon, 13 Sep 2010 16:18:44 +0200 (CEST) > >barry.warsaw wrote: > >> > >> -For an arbitrary package `foo`, you would see these files when the > >> +For an arbitrary package `foo`, you might see these files when the > >> distribution package was installed:: > >> > >> - /usr/share/pyshared/foo.cpython-32m.so > >> - /usr/share/pyshared/foo.cpython-33m.so > >> + /usr/lib/python/foo.cpython-32m.so > >> + /usr/lib/python/foo.cpython-33m.so > > > >Are these the default paths? The PEP doesn't say how a distribution > >is supposed to choose its PEP 3149 filesystem layout (instead of > >/usr/lib/python3.2/site-packages). > > Why should it? Distributions are going to make their own decisions > independent of the PEP. That's why s/would/might/ in the above change. I'm > open to suggestions for better ways to explain it. I meant how these decisions are implemented. Is there a configure switch (there doesn't seem to be)? Does it require patching Python? Thanks Antoine. From barry at python.org Mon Sep 13 17:30:10 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 13 Sep 2010 11:30:10 -0400 Subject: [Python-Dev] r84775 - peps/trunk/pep-3149.txt In-Reply-To: <20100913170452.747c2890@pitrou.net> References: <20100913141844.28C78FB1E@mail.python.org> <20100913163619.64c8d50a@pitrou.net> <20100913105516.2a237510@mission> <20100913170452.747c2890@pitrou.net> Message-ID: <20100913113010.4b756152@mission> On Sep 13, 2010, at 05:04 PM, Antoine Pitrou wrote: >On Mon, 13 Sep 2010 10:55:16 -0400 >Barry Warsaw wrote: >> On Sep 13, 2010, at 04:36 PM, Antoine Pitrou wrote: >> >> - /usr/share/pyshared/foo.cpython-32m.so >> >> - /usr/share/pyshared/foo.cpython-33m.so >> >> + /usr/lib/python/foo.cpython-32m.so >> >> + /usr/lib/python/foo.cpython-33m.so >> > >> >Are these the default paths? The PEP doesn't say how a distribution >> >is supposed to choose its PEP 3149 filesystem layout (instead of >> >/usr/lib/python3.2/site-packages). >> >> Why should it? Distributions are going to make their own decisions >> independent of the PEP. That's why s/would/might/ in the above >> change. I'm open to suggestions for better ways to explain it. > >I meant how these decisions are implemented. Is there a configure >switch (there doesn't seem to be)? Does it require patching Python? Ah, no. Standard configure switches are used. Debian (inherited by Ubuntu) has a post-installation script for Python packages which create the .py symlinks and do the byte-compilation. The big win here is that much of this can go away now (and in fact there are modifications to this post-installation script already). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From solipsis at pitrou.net Mon Sep 13 17:36:24 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 13 Sep 2010 17:36:24 +0200 Subject: [Python-Dev] r84775 - peps/trunk/pep-3149.txt In-Reply-To: <20100913113010.4b756152@mission> References: <20100913141844.28C78FB1E@mail.python.org> <20100913163619.64c8d50a@pitrou.net> <20100913105516.2a237510@mission> <20100913170452.747c2890@pitrou.net> <20100913113010.4b756152@mission> Message-ID: <1284392184.3225.4.camel@localhost.localdomain> > >I meant how these decisions are implemented. Is there a configure > >switch (there doesn't seem to be)? Does it require patching Python? > > Ah, no. Standard configure switches are used. Debian (inherited by Ubuntu) > has a post-installation script for Python packages which create the .py > symlinks and do the byte-compilation. The big win here is that much of this > can go away now (and in fact there are modifications to this post-installation > script already). Ok, so can you explain how the new thing will work (on Debian)? :) Does it mean that e.g. /usr/lib/python3.2/site-packages will get symlinked to /usr/lib/python? Regards Antoine. From martin at v.loewis.de Mon Sep 13 22:29:32 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 13 Sep 2010 22:29:32 +0200 Subject: [Python-Dev] r84775 - peps/trunk/pep-3149.txt In-Reply-To: <1284392184.3225.4.camel@localhost.localdomain> References: <20100913141844.28C78FB1E@mail.python.org> <20100913163619.64c8d50a@pitrou.net> <20100913105516.2a237510@mission> <20100913170452.747c2890@pitrou.net> <20100913113010.4b756152@mission> <1284392184.3225.4.camel@localhost.localdomain> Message-ID: <4C8E89AC.6090703@v.loewis.de> Am 13.09.2010 17:36, schrieb Antoine Pitrou: > >>> I meant how these decisions are implemented. Is there a configure >>> switch (there doesn't seem to be)? Does it require patching Python? >> >> Ah, no. Standard configure switches are used. Debian (inherited by Ubuntu) >> has a post-installation script for Python packages which create the .py >> symlinks and do the byte-compilation. The big win here is that much of this >> can go away now (and in fact there are modifications to this post-installation >> script already). > > Ok, so can you explain how the new thing will work (on Debian)? :) > Does it mean that e.g. /usr/lib/python3.2/site-packages will get > symlinked to /usr/lib/python? They currently get /usr/lib/pyshared/ onto sys.path, by providing a .pth file (python-support.pth). There are many other ways: you can edit site.py, provide a sitecustomize.py, or edit Modules/Setup (providing a Modules/Setup.local may also work). If I was them, I wouldn't relocate the standard extension modules, but keep them in lib-dynload; /usr/lib/pyshared would only be there for additional packages (which then don't need to bring /usr/lib/python3.3 into existance even though python 3.3 isn't installed). Regards, Martin From ncoghlan at gmail.com Mon Sep 13 23:50:34 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Sep 2010 07:50:34 +1000 Subject: [Python-Dev] [Python-checkins] r84792 - in python/branches/py3k: Doc/library/reprlib.rst Lib/collections.py Lib/reprlib.py Lib/test/test_reprlib.py Misc/NEWS In-Reply-To: <20100913213600.4FC73DD9A@mail.python.org> References: <20100913213600.4FC73DD9A@mail.python.org> Message-ID: On Tue, Sep 14, 2010 at 7:36 AM, raymond.hettinger wrote: > + ? ? ? ?# Can't use functools.wraps() here because of bootstrap issues > + ? ? ? ?wrapper.__module__ = getattr(user_function, '__module__') > + ? ? ? ?wrapper.__doc__ = getattr(user_function, '__doc__') > + ? ? ? ?wrapper.__name__ = getattr(user_function, '__name__') > + ? ? ? ?return wrapper Perhaps add __wrapped__ as well? (I assume that, similar to _collections before it was made a builtin, the bootstrap issue is that _functools is an extension module rather than builtin, but reprlib is needed when building the extension modules?) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Tue Sep 14 12:17:44 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 14 Sep 2010 12:17:44 +0200 Subject: [Python-Dev] Rework nntlib? Message-ID: <20100914121744.31ef5aaf@pitrou.net> Hello, Like the email package, nntplib in py3k is broken (because of various bytes/str mismatches; I suppose the lack of a test suite didn't help when porting). I would like to take the opportunity to improve the API a bit; no heavy re-architecting, but simply a bunch of changes to make it higher-level. Is it acceptable? (and, yes, I would add a test suite) Regards Antoine. From fuzzyman at voidspace.org.uk Tue Sep 14 12:28:38 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 14 Sep 2010 11:28:38 +0100 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <20100914121744.31ef5aaf@pitrou.net> References: <20100914121744.31ef5aaf@pitrou.net> Message-ID: <4C8F4E56.3080804@voidspace.org.uk> On 14/09/2010 11:17, Antoine Pitrou wrote: > Hello, > > Like the email package, nntplib in py3k is broken (because of > various bytes/str mismatches; I suppose the lack of a test suite didn't > help when porting). > > I would like to take the opportunity to improve the API a bit; no heavy > re-architecting, but simply a bunch of changes to make it higher-level. > Is it acceptable? > > (and, yes, I would add a test suite) > Given that backwards incompatible changes are likely to be unavoidable due to the bytes / str issue, taking the opportunity to cleanup and improve the API sounds great. Just a shame we didn't get to it for 3.0, but thank you for picking this up. Michael > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ncoghlan at gmail.com Tue Sep 14 12:30:14 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Sep 2010 20:30:14 +1000 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <20100914121744.31ef5aaf@pitrou.net> References: <20100914121744.31ef5aaf@pitrou.net> Message-ID: On Tue, Sep 14, 2010 at 8:17 PM, Antoine Pitrou wrote: > > Hello, > > Like the email package, nntplib in py3k is broken (because of > various bytes/str mismatches; I suppose the lack of a test suite didn't > help when porting). > > I would like to take the opportunity to improve the API a bit; no heavy > re-architecting, but simply a bunch of changes to make it higher-level. > Is it acceptable? > > (and, yes, I would add a test suite) This is the kind of question that's hard to answer in the abstract. In principle it sounds like a reasonable idea, but in practice, if the old API works correctly for 7-bit ASCII, it should probably remain available (even if it's nominally broken for Unicode and 8-bit ASCII). Adding a higher level API is almost certainly fine. Removing the old lower level APIs is more questionable, and will likely depend on precisely how broken they are. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From skip at pobox.com Tue Sep 14 12:31:16 2010 From: skip at pobox.com (skip at pobox.com) Date: Tue, 14 Sep 2010 05:31:16 -0500 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <20100914121744.31ef5aaf@pitrou.net> References: <20100914121744.31ef5aaf@pitrou.net> Message-ID: <19599.20212.241866.713348@montanaro.dyndns.org> Antoine> Like the email package, nntplib in py3k is broken (because of Antoine> various bytes/str mismatches; I suppose the lack of a test Antoine> suite didn't help when porting). How heavily used is nntp these days (unless you're looking for spam)? Would it make more sense to find someone willing to maintain it outside the Python core and just remove it altogether? Skip From solipsis at pitrou.net Tue Sep 14 12:40:29 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 14 Sep 2010 12:40:29 +0200 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> Message-ID: <20100914124029.5c569d2c@pitrou.net> On Tue, 14 Sep 2010 20:30:14 +1000 Nick Coghlan wrote: > On Tue, Sep 14, 2010 at 8:17 PM, Antoine Pitrou wrote: > > > > Hello, > > > > Like the email package, nntplib in py3k is broken (because of > > various bytes/str mismatches; I suppose the lack of a test suite didn't > > help when porting). > > > > I would like to take the opportunity to improve the API a bit; no heavy > > re-architecting, but simply a bunch of changes to make it higher-level. > > Is it acceptable? > > > > (and, yes, I would add a test suite) > > This is the kind of question that's hard to answer in the abstract. In > principle it sounds like a reasonable idea, but in practice, if the > old API works correctly for 7-bit ASCII, it should probably remain > available (even if it's nominally broken for Unicode and 8-bit ASCII). The current module is broken in py3k. I suppose it is still usable for things like fetching group descriptions, but you can't e.g. read articles: http://bugs.python.org/issue7644 Also, http://bugs.python.org/issue9360 mentions that API changes were discussed by Brett and the OP at EuroPython (I wasn't there): ?Change API methods to return strings instead of bytes. This breaks API compatibility, but given that the parameters need to be passed as strings and many of the returned values would need to be passed to other API methods, I consider the current API to be broken. I've discussed this with Brett at the EuroPython sprint, and he agrees.? My changes build on the patch proposed on this issue and add further enhancements. Yes, the new API could be checked in as "nntplib2" instead. I am a bit uneasy with that, though, since 1) the original nntplib would still be broken 2) the changes I have in mind don't seem enough to warrant such a grandiloquent naming :-) Regards Antoine. From baptiste13z at free.fr Tue Sep 14 12:44:30 2010 From: baptiste13z at free.fr (Baptiste Carvello) Date: Tue, 14 Sep 2010 12:44:30 +0200 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <19599.20212.241866.713348@montanaro.dyndns.org> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> Message-ID: skip at pobox.com a ?crit : > Antoine> Like the email package, nntplib in py3k is broken (because of > Antoine> various bytes/str mismatches; I suppose the lack of a test > Antoine> suite didn't help when porting). > > How heavily used is nntp these days (unless you're looking for spam)? Would > it make more sense to find someone willing to maintain it outside the Python > core and just remove it altogether? > > Skip Reading this from GMANE ;-) From solipsis at pitrou.net Tue Sep 14 12:45:25 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 14 Sep 2010 12:45:25 +0200 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <19599.20212.241866.713348@montanaro.dyndns.org> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> Message-ID: <20100914124525.1ceb5e55@pitrou.net> On Tue, 14 Sep 2010 05:31:16 -0500 skip at pobox.com wrote: > > Antoine> Like the email package, nntplib in py3k is broken (because of > Antoine> various bytes/str mismatches; I suppose the lack of a test > Antoine> suite didn't help when porting). > > How heavily used is nntp these days (unless you're looking for spam)? Routinely, for example I'm posting this through NNTP on news.gmane.org :) Of course the need for a working NNTP library is much less than for a working email package. I'm doing this because it scratches an itch, not because I need it for myself. > Would > it make more sense to find someone willing to maintain it outside the Python > core and just remove it altogether? That someone remains to be found (it would not be me). Also, since the NNTP protocol itself doesn't seem to see massive changes, it makes sense to keep an implementation in the stdlib. Regards Antoine. From ncoghlan at gmail.com Tue Sep 14 12:58:23 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Sep 2010 20:58:23 +1000 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <20100914124029.5c569d2c@pitrou.net> References: <20100914121744.31ef5aaf@pitrou.net> <20100914124029.5c569d2c@pitrou.net> Message-ID: On Tue, Sep 14, 2010 at 8:40 PM, Antoine Pitrou wrote: > The current module is broken in py3k. I suppose it is still usable for > things like fetching group descriptions, but you can't e.g. read > articles: > http://bugs.python.org/issue7644 > > Also, http://bugs.python.org/issue9360 mentions that API changes were > discussed by Brett and the OP at EuroPython (I wasn't there): > > ?Change API methods to return strings instead of bytes. This breaks API > compatibility, but given that the parameters need to be passed as > strings and many of the returned values would need to be passed to > other API methods, I consider the current API to be broken. I've > discussed this with Brett at the EuroPython sprint, and he agrees.? > > My changes build on the patch proposed on this issue and add further > enhancements. Given the additional explanation, tidying up the API sounds like a reasonable way forward to me. Some notes either in the docs or on the wiki regarding how to port from the 2.x API to the 3.x API will be useful though. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From orsenthil at gmail.com Tue Sep 14 13:04:33 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Tue, 14 Sep 2010 16:34:33 +0530 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> Message-ID: <20100914110433.GA17037@remy> On Tue, Sep 14, 2010 at 12:44:30PM +0200, Baptiste Carvello wrote: > > Antoine> Like the email package, nntplib in py3k is broken (because of > > Antoine> various bytes/str mismatches; I suppose the lack of a test > > Antoine> suite didn't help when porting). > > > >How heavily used is nntp these days (unless you're looking for spam)? Would > >it make more sense to find someone willing to maintain it outside the Python > >core and just remove it altogether? > > > Reading this from GMANE ;-) I guess, Skip's question or intention was, how often nntplib as a module is being used these days to write scripts/tools or clients? Very rarely. It would definitely be interesting to know, if there are python applications out there which are using nntplib at the moment. -- Senthil From fuzzyman at voidspace.org.uk Tue Sep 14 13:10:54 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 14 Sep 2010 12:10:54 +0100 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <20100914110433.GA17037@remy> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> Message-ID: <4C8F583E.4020105@voidspace.org.uk> On 14/09/2010 12:04, Senthil Kumaran wrote: > On Tue, Sep 14, 2010 at 12:44:30PM +0200, Baptiste Carvello wrote: >>> Antoine> Like the email package, nntplib in py3k is broken (because of >>> Antoine> various bytes/str mismatches; I suppose the lack of a test >>> Antoine> suite didn't help when porting). >>> >>> How heavily used is nntp these days (unless you're looking for spam)? Would >>> it make more sense to find someone willing to maintain it outside the Python >>> core and just remove it altogether? >>> >> Reading this from GMANE ;-) > I guess, Skip's question or intention was, how often nntplib as a > module is being used these days to write scripts/tools or clients? > Very rarely. > > It would definitely be interesting to know, if there are python > applications out there which are using nntplib at the moment. > Google code search shows a *few* uses. Most occurences are projects that include Python sources, but there are a handful that use it. e.g. sinntp http://sinntp.googlecode.com/hg/ Search url: http://www.google.com/codesearch?as_q=nntplib&btnG=Search+Code&hl=&as_package=&as_lang=python&as_filename=&as_class=&as_function=&as_license=&as_case= Of course *every* standard library module will have *some* users. The question is whether or not a handful of users justifies something being in the standard library. If it was proposed as a new package then we probably wouldn't want it, but as we already have it then making it *work* is a different matter... :-) All the best, Michael Foord -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ncoghlan at gmail.com Tue Sep 14 13:14:06 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Sep 2010 21:14:06 +1000 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <20100914110433.GA17037@remy> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> Message-ID: On Tue, Sep 14, 2010 at 9:04 PM, Senthil Kumaran wrote: > I guess, Skip's question or intention was, how often nntplib as a > module is being used these days to write scripts/tools or clients? > Very rarely. > > It would definitely be interesting to know, if there are python > applications out there which are using nntplib at the moment. It isn't a huge number, but not all of the results here are copies of Python standard library: http://www.google.com/codesearch?hl=en&start=10&sa=N&q=nntplib+lang:python Not comprehensive of course, and it doesn't tell us how active these projects are, but it does indicate the real world use is greater than zero. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From fuzzyman at voidspace.org.uk Tue Sep 14 13:17:55 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 14 Sep 2010 12:17:55 +0100 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> Message-ID: <4C8F59E3.9010804@voidspace.org.uk> On 14/09/2010 12:14, Nick Coghlan wrote: > On Tue, Sep 14, 2010 at 9:04 PM, Senthil Kumaran wrote: >> I guess, Skip's question or intention was, how often nntplib as a >> module is being used these days to write scripts/tools or clients? >> Very rarely. >> >> It would definitely be interesting to know, if there are python >> applications out there which are using nntplib at the moment. > It isn't a huge number, but not all of the results here are copies of > Python standard library: > http://www.google.com/codesearch?hl=en&start=10&sa=N&q=nntplib+lang:python > > Not comprehensive of course, and it doesn't tell us how active these > projects are, but it does indicate the real world use is greater than > zero. Interestingly one of the uses is mailman, which uses it for its nntp gateway maintenance. Michael > Cheers, > Nick. > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From orsenthil at gmail.com Tue Sep 14 13:20:54 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Tue, 14 Sep 2010 16:50:54 +0530 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <4C8F583E.4020105@voidspace.org.uk> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <4C8F583E.4020105@voidspace.org.uk> Message-ID: <20100914112054.GB17037@remy> On Tue, Sep 14, 2010 at 12:10:54PM +0100, Michael Foord wrote: > But as we already have it then making it *work* is a different matter... :-) Of course, I buy this argument. :) I am +1 on improving the nntplib in py3k, but if we have real world users raising bug reports and asking particular changes/improvements, it would be all the more useful. I see that Antoine referenced some bug reports.. -- Senthil From steve at holdenweb.com Tue Sep 14 13:47:16 2010 From: steve at holdenweb.com (Steve Holden) Date: Tue, 14 Sep 2010 07:47:16 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <4C8F583E.4020105@voidspace.org.uk> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <4C8F583E.4020105@voidspace.org.uk> Message-ID: <4C8F60C4.8010802@holdenweb.com> On 9/14/2010 7:10 AM, Michael Foord wrote: > On 14/09/2010 12:04, Senthil Kumaran wrote: >> On Tue, Sep 14, 2010 at 12:44:30PM +0200, Baptiste Carvello wrote: >>>> Antoine> Like the email package, nntplib in py3k is broken >>>> (because of >>>> Antoine> various bytes/str mismatches; I suppose the lack of a >>>> test >>>> Antoine> suite didn't help when porting). >>>> >>>> How heavily used is nntp these days (unless you're looking for >>>> spam)? Would >>>> it make more sense to find someone willing to maintain it outside >>>> the Python >>>> core and just remove it altogether? >>>> >>> Reading this from GMANE ;-) >> I guess, Skip's question or intention was, how often nntplib as a >> module is being used these days to write scripts/tools or clients? >> Very rarely. >> >> It would definitely be interesting to know, if there are python >> applications out there which are using nntplib at the moment. >> > Google code search shows a *few* uses. Most occurences are projects that > include Python sources, but there are a handful that use it. e.g. sinntp > > http://sinntp.googlecode.com/hg/ > > Search url: > > http://www.google.com/codesearch?as_q=nntplib&btnG=Search+Code&hl=&as_package=&as_lang=python&as_filename=&as_class=&as_function=&as_license=&as_case= > > > Of course *every* standard library module will have *some* users. The > question is whether or not a handful of users justifies something being > in the standard library. If it was proposed as a new package then we > probably wouldn't want it, but as we already have it then making it > *work* is a different matter... :-) > > All the best, > > Michael Foord > How many of those uses are in Python 3? How many would break if ported to Python 3? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From steve at holdenweb.com Tue Sep 14 13:47:16 2010 From: steve at holdenweb.com (Steve Holden) Date: Tue, 14 Sep 2010 07:47:16 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <4C8F583E.4020105@voidspace.org.uk> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <4C8F583E.4020105@voidspace.org.uk> Message-ID: <4C8F60C4.8010802@holdenweb.com> On 9/14/2010 7:10 AM, Michael Foord wrote: > On 14/09/2010 12:04, Senthil Kumaran wrote: >> On Tue, Sep 14, 2010 at 12:44:30PM +0200, Baptiste Carvello wrote: >>>> Antoine> Like the email package, nntplib in py3k is broken >>>> (because of >>>> Antoine> various bytes/str mismatches; I suppose the lack of a >>>> test >>>> Antoine> suite didn't help when porting). >>>> >>>> How heavily used is nntp these days (unless you're looking for >>>> spam)? Would >>>> it make more sense to find someone willing to maintain it outside >>>> the Python >>>> core and just remove it altogether? >>>> >>> Reading this from GMANE ;-) >> I guess, Skip's question or intention was, how often nntplib as a >> module is being used these days to write scripts/tools or clients? >> Very rarely. >> >> It would definitely be interesting to know, if there are python >> applications out there which are using nntplib at the moment. >> > Google code search shows a *few* uses. Most occurences are projects that > include Python sources, but there are a handful that use it. e.g. sinntp > > http://sinntp.googlecode.com/hg/ > > Search url: > > http://www.google.com/codesearch?as_q=nntplib&btnG=Search+Code&hl=&as_package=&as_lang=python&as_filename=&as_class=&as_function=&as_license=&as_case= > > > Of course *every* standard library module will have *some* users. The > question is whether or not a handful of users justifies something being > in the standard library. If it was proposed as a new package then we > probably wouldn't want it, but as we already have it then making it > *work* is a different matter... :-) > > All the best, > > Michael Foord > How many of those uses are in Python 3? How many would break if ported to Python 3? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From piotr at debian.org Tue Sep 14 13:56:42 2010 From: piotr at debian.org (Piotr =?utf-8?Q?O=C5=BCarowski?=) Date: Tue, 14 Sep 2010 13:56:42 +0200 Subject: [Python-Dev] r84775 - peps/trunk/pep-3149.txt In-Reply-To: <1284392184.3225.4.camel@localhost.localdomain> References: <20100913141844.28C78FB1E@mail.python.org> <20100913163619.64c8d50a@pitrou.net> <20100913105516.2a237510@mission> <20100913170452.747c2890@pitrou.net> <20100913113010.4b756152@mission> <1284392184.3225.4.camel@localhost.localdomain> Message-ID: <20100914115642.GV15608@piotro.eu> [Antoine Pitrou, 2010-09-13] > > > >I meant how these decisions are implemented. Is there a configure > > >switch (there doesn't seem to be)? Does it require patching Python? > > > > Ah, no. Standard configure switches are used. Debian (inherited by Ubuntu) > > has a post-installation script for Python packages which create the .py > > symlinks and do the byte-compilation. The big win here is that much of this > > can go away now (and in fact there are modifications to this post-installation > > script already). > > Ok, so can you explain how the new thing will work (on Debian)? :) we have /usr/lib/python3/dist-packages in sys.path (via patched Lib/site.py). Our python3.1 will use the same directory as well (version in experimental is modified to use tagged extensions). distutils has additional --install-layout command which when set to "deb" uses Debian's locations, if distutils is not used (or --install-layout=deb not set), dh_python3 will move files to the right location at (package's) build time (and rename .so files) > Does it mean that e.g. /usr/lib/python3.2/site-packages will get > symlinked to /usr/lib/python? no, /usr/lib/python3.2/site-packages is not used at all (we don't use "site-packages" anymore to avoid conflicts with local installations of Python. /usr/lib/python3.2/dist-packages on the other hand is still in sys.path, but I'm not sure what we'll do with it (we still have to figure out what to do with modules that work with 3.2 only and cannot be patched due to f.e. from __future__ imports) -- Piotr O?arowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 From michael at voidspace.org.uk Tue Sep 14 15:36:56 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Tue, 14 Sep 2010 16:36:56 +0300 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <4C8F60C4.8010802@holdenweb.com> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <4C8F583E.4020105@voidspace.org.uk> <4C8F60C4.8010802@holdenweb.com> Message-ID: <4C8F7A78.5050901@voidspace.org.uk> On 14/09/2010 12:47, Steve Holden wrote: > On 9/14/2010 7:10 AM, Michael Foord wrote: >> On 14/09/2010 12:04, Senthil Kumaran wrote: >>> On Tue, Sep 14, 2010 at 12:44:30PM +0200, Baptiste Carvello wrote: >>>>> Antoine> Like the email package, nntplib in py3k is broken >>>>> (because of >>>>> Antoine> various bytes/str mismatches; I suppose the lack of a >>>>> test >>>>> Antoine> suite didn't help when porting). >>>>> >>>>> How heavily used is nntp these days (unless you're looking for >>>>> spam)? Would >>>>> it make more sense to find someone willing to maintain it outside >>>>> the Python >>>>> core and just remove it altogether? >>>>> >>>> Reading this from GMANE ;-) >>> I guess, Skip's question or intention was, how often nntplib as a >>> module is being used these days to write scripts/tools or clients? >>> Very rarely. >>> >>> It would definitely be interesting to know, if there are python >>> applications out there which are using nntplib at the moment. >>> >> Google code search shows a *few* uses. Most occurences are projects that >> include Python sources, but there are a handful that use it. e.g. sinntp >> >> http://sinntp.googlecode.com/hg/ >> >> Search url: >> >> http://www.google.com/codesearch?as_q=nntplib&btnG=Search+Code&hl=&as_package=&as_lang=python&as_filename=&as_class=&as_function=&as_license=&as_case= >> >> >> Of course *every* standard library module will have *some* users. The >> question is whether or not a handful of users justifies something being >> in the standard library. If it was proposed as a new package then we >> probably wouldn't want it, but as we already have it then making it >> *work* is a different matter... :-) >> >> All the best, >> >> Michael Foord >> > How many of those uses are in Python 3? How many would break if ported > to Python 3? Given that nntplib *doesn't work* with Python 3, I would guess none. :-) Michael > regards > Steve -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From steve at holdenweb.com Tue Sep 14 15:50:16 2010 From: steve at holdenweb.com (Steve Holden) Date: Tue, 14 Sep 2010 09:50:16 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <4C8F7A78.5050901@voidspace.org.uk> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <4C8F583E.4020105@voidspace.org.uk> <4C8F60C4.8010802@holdenweb.com> <4C8F7A78.5050901@voidspace.org.uk> Message-ID: On 9/14/2010 9:36 AM, Michael Foord wrote: > On 14/09/2010 12:47, Steve Holden wrote: >> On 9/14/2010 7:10 AM, Michael Foord wrote: >>> On 14/09/2010 12:04, Senthil Kumaran wrote: >>>> On Tue, Sep 14, 2010 at 12:44:30PM +0200, Baptiste Carvello wrote: >>>>>> Antoine> Like the email package, nntplib in py3k is broken >>>>>> (because of >>>>>> Antoine> various bytes/str mismatches; I suppose the lack of a >>>>>> test >>>>>> Antoine> suite didn't help when porting). >>>>>> >>>>>> How heavily used is nntp these days (unless you're looking for >>>>>> spam)? Would >>>>>> it make more sense to find someone willing to maintain it outside >>>>>> the Python >>>>>> core and just remove it altogether? >>>>>> >>>>> Reading this from GMANE ;-) >>>> I guess, Skip's question or intention was, how often nntplib as a >>>> module is being used these days to write scripts/tools or clients? >>>> Very rarely. >>>> >>>> It would definitely be interesting to know, if there are python >>>> applications out there which are using nntplib at the moment. >>>> >>> Google code search shows a *few* uses. Most occurences are projects that >>> include Python sources, but there are a handful that use it. e.g. sinntp >>> >>> http://sinntp.googlecode.com/hg/ >>> >>> Search url: >>> >>> http://www.google.com/codesearch?as_q=nntplib&btnG=Search+Code&hl=&as_package=&as_lang=python&as_filename=&as_class=&as_function=&as_license=&as_case= >>> >>> >>> >>> Of course *every* standard library module will have *some* users. The >>> question is whether or not a handful of users justifies something being >>> in the standard library. If it was proposed as a new package then we >>> probably wouldn't want it, but as we already have it then making it >>> *work* is a different matter... :-) >>> >>> All the best, >>> >>> Michael Foord >>> >> How many of those uses are in Python 3? How many would break if ported >> to Python 3? > > Given that nntplib *doesn't work* with Python 3, I would guess none. :-) > Quite. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From steve at holdenweb.com Tue Sep 14 15:51:23 2010 From: steve at holdenweb.com (Steve Holden) Date: Tue, 14 Sep 2010 09:51:23 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <4C8F7A78.5050901@voidspace.org.uk> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <4C8F583E.4020105@voidspace.org.uk> <4C8F60C4.8010802@holdenweb.com> <4C8F7A78.5050901@voidspace.org.uk> Message-ID: On 9/14/2010 9:36 AM, Michael Foord wrote: > On 14/09/2010 12:47, Steve Holden wrote: >> On 9/14/2010 7:10 AM, Michael Foord wrote: >>> On 14/09/2010 12:04, Senthil Kumaran wrote: >>>> On Tue, Sep 14, 2010 at 12:44:30PM +0200, Baptiste Carvello wrote: >>>>>> Antoine> Like the email package, nntplib in py3k is broken >>>>>> (because of >>>>>> Antoine> various bytes/str mismatches; I suppose the lack of a >>>>>> test >>>>>> Antoine> suite didn't help when porting). >>>>>> >>>>>> How heavily used is nntp these days (unless you're looking for >>>>>> spam)? Would >>>>>> it make more sense to find someone willing to maintain it outside >>>>>> the Python >>>>>> core and just remove it altogether? >>>>>> >>>>> Reading this from GMANE ;-) >>>> I guess, Skip's question or intention was, how often nntplib as a >>>> module is being used these days to write scripts/tools or clients? >>>> Very rarely. >>>> >>>> It would definitely be interesting to know, if there are python >>>> applications out there which are using nntplib at the moment. >>>> >>> Google code search shows a *few* uses. Most occurences are projects that >>> include Python sources, but there are a handful that use it. e.g. sinntp >>> >>> http://sinntp.googlecode.com/hg/ >>> >>> Search url: >>> >>> http://www.google.com/codesearch?as_q=nntplib&btnG=Search+Code&hl=&as_package=&as_lang=python&as_filename=&as_class=&as_function=&as_license=&as_case= >>> >>> >>> >>> Of course *every* standard library module will have *some* users. The >>> question is whether or not a handful of users justifies something being >>> in the standard library. If it was proposed as a new package then we >>> probably wouldn't want it, but as we already have it then making it >>> *work* is a different matter... :-) >>> >>> All the best, >>> >>> Michael Foord >>> >> How many of those uses are in Python 3? How many would break if ported >> to Python 3? > > Given that nntplib *doesn't work* with Python 3, I would guess none. :-) > Sorry - premature send. I imagine a 2to3 component could help with the transition to a new API (this component would normally be disabled). regards Stefve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From barry at python.org Tue Sep 14 00:51:20 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 13 Sep 2010 18:51:20 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <20100914121744.31ef5aaf@pitrou.net> References: <20100914121744.31ef5aaf@pitrou.net> Message-ID: <20100913185120.1cc57be9@mission> On Sep 14, 2010, at 12:17 PM, Antoine Pitrou wrote: >Like the email package, nntplib in py3k is broken (because of >various bytes/str mismatches; I suppose the lack of a test suite didn't >help when porting). > >I would like to take the opportunity to improve the API a bit; no heavy >re-architecting, but simply a bunch of changes to make it higher-level. >Is it acceptable? > >(and, yes, I would add a test suite) Heartily +1. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Tue Sep 14 00:56:01 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 13 Sep 2010 18:56:01 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <4C8F60C4.8010802@holdenweb.com> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <4C8F583E.4020105@voidspace.org.uk> <4C8F60C4.8010802@holdenweb.com> Message-ID: <20100913185601.49fc06d1@mission> On Sep 14, 2010, at 07:47 AM, Steve Holden wrote: >How many of those uses are in Python 3? How many would break if ported >to Python 3? How many Python 3 applications in general are there now? It would certainly bum me out if nntplib were removed (rather than improved as Antoine is offering to do) from the stdlib when it comes time to port Mailman to py3k. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Tue Sep 14 00:58:31 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 13 Sep 2010 18:58:31 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <4C8F4E56.3080804@voidspace.org.uk> References: <20100914121744.31ef5aaf@pitrou.net> <4C8F4E56.3080804@voidspace.org.uk> Message-ID: <20100913185831.1fd9876a@mission> On Sep 14, 2010, at 11:28 AM, Michael Foord wrote: > On 14/09/2010 11:17, Antoine Pitrou wrote: >> Hello, >> >> Like the email package, nntplib in py3k is broken (because of >> various bytes/str mismatches; I suppose the lack of a test suite >> didn't help when porting). >> >> I would like to take the opportunity to improve the API a bit; no >> heavy re-architecting, but simply a bunch of changes to make it >> higher-level. Is it acceptable? >> >> (and, yes, I would add a test suite) >> > >Given that backwards incompatible changes are likely to be unavoidable >due to the bytes / str issue, taking the opportunity to cleanup and >improve the API sounds great. Just a shame we didn't get to it for >3.0, but thank you for picking this up. A side benefit may be that Antoine finds some clever, useful, and more generally applicable techniques for dealing with byte/strings in these contexts. That would be time well spent, and may transfer to work on the email package too for example. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From solipsis at pitrou.net Tue Sep 14 16:23:42 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 14 Sep 2010 16:23:42 +0200 Subject: [Python-Dev] Rework nntlib? References: <20100914121744.31ef5aaf@pitrou.net> <4C8F4E56.3080804@voidspace.org.uk> <20100913185831.1fd9876a@mission> Message-ID: <20100914162342.1a070b20@pitrou.net> On Mon, 13 Sep 2010 18:58:31 -0400 Barry Warsaw wrote: > On Sep 14, 2010, at 11:28 AM, Michael Foord wrote: > > > On 14/09/2010 11:17, Antoine Pitrou wrote: > >> Hello, > >> > >> Like the email package, nntplib in py3k is broken (because of > >> various bytes/str mismatches; I suppose the lack of a test suite > >> didn't help when porting). > >> > >> I would like to take the opportunity to improve the API a bit; no > >> heavy re-architecting, but simply a bunch of changes to make it > >> higher-level. Is it acceptable? > >> > >> (and, yes, I would add a test suite) > >> > > > >Given that backwards incompatible changes are likely to be unavoidable > >due to the bytes / str issue, taking the opportunity to cleanup and > >improve the API sounds great. Just a shame we didn't get to it for > >3.0, but thank you for picking this up. > > A side benefit may be that Antoine finds some clever, useful, and more > generally applicable techniques for dealing with byte/strings in these > contexts. Well... One clever, useful and generally applicable technique I'm currently using is "surrogateescape" combined with RFC 3977's standardization on UTF-8 for non-message data. (thanks Martin!) But I don't think nntplib should implement any message parsing or decoding. Users will have to rely on the email module (or their own code) instead. Regards Antoine. From skip at pobox.com Tue Sep 14 18:43:46 2010 From: skip at pobox.com (skip at pobox.com) Date: Tue, 14 Sep 2010 11:43:46 -0500 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> Message-ID: <19599.42562.239559.453455@montanaro.dyndns.org> Baptiste> Reading this from GMANE ;-) I use GMANE too on occasion, but only via the web, as I suppose most people do. I haven't actually used an NNTP-based app like xrn or gnus in probably two decades. Do the GMANE folks publish HTTP v. NNTP statistics for their service? I question if the nntplib module is used enough to warrant inclusion with the rest of the batteries. Not to mention dedication of scarce people resources. Even though Antoine has kindly volunteered to make the necessary changes, I seem to recall that the email package transition took awhile to get right (and already had a reasonable test suite I suspect). There are probably more critical parts of Python he could work on instead (though who's to dispute that Antoine has an NNTP itch?) Finally, is there an application out in the wild using nntplib which is desirous of better NNTP support than Python currently has? We got rid of gopherlib a few years ago (deprecated in 2.5, presumably gone in 2.6). I suspect the NNTP protocol has a greatly diminished user base as well, GMANE's presence notwithstanding. Skip From steve at holdenweb.com Tue Sep 14 20:01:44 2010 From: steve at holdenweb.com (Steve Holden) Date: Tue, 14 Sep 2010 14:01:44 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <19599.42562.239559.453455@montanaro.dyndns.org> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <19599.42562.239559.453455@montanaro.dyndns.org> Message-ID: On 9/14/2010 12:43 PM, skip at pobox.com wrote: > > Baptiste> Reading this from GMANE ;-) > > I use GMANE too on occasion, but only via the web, as I suppose most people > do. I haven't actually used an NNTP-based app like xrn or gnus in probably > two decades. Do the GMANE folks publish HTTP v. NNTP statistics for their > service? > > I question if the nntplib module is used enough to warrant inclusion with > the rest of the batteries. Not to mention dedication of scarce people > resources. Even though Antoine has kindly volunteered to make the necessary > changes, I seem to recall that the email package transition took awhile to > get right (and already had a reasonable test suite I suspect). There are > probably more critical parts of Python he could work on instead (though > who's to dispute that Antoine has an NNTP itch?) Finally, is there an > application out in the wild using nntplib which is desirous of better NNTP > support than Python currently has? > > We got rid of gopherlib a few years ago (deprecated in 2.5, presumably gone > in 2.6). I suspect the NNTP protocol has a greatly diminished user base as > well, GMANE's presence notwithstanding. > > Skip The fact that Mailman will need it would alone outweigh all those considerations in my mind. regards Steve PS: I read c.l.py-dev using NNTP from Thunderbird. Why anyone bothers with these web interfaces is beyond me ... PPS: What makes you think the email package transition is complete? -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From barry at python.org Tue Sep 14 20:24:20 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 14 Sep 2010 14:24:20 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <19599.42562.239559.453455@montanaro.dyndns.org> Message-ID: <20100914142420.50e29b87@mission> On Sep 14, 2010, at 02:01 PM, Steve Holden wrote: >The fact that Mailman will need it would alone outweigh all those >considerations in my mind. /me makes the check out to Mr. Holden. >PS: I read c.l.py-dev using NNTP from Thunderbird. Why anyone bothers >with these web interfaces is beyond me ... I'll note one other thing. Mailman 3 intends[*] to provide NNTP access directly to its archives, exactly because IMO it's so much better to read old mail that way. >PPS: What makes you think the email package transition is complete? Heh. -Barry [0] 'round tuits willing -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From brett at python.org Tue Sep 14 20:34:26 2010 From: brett at python.org (Brett Cannon) Date: Tue, 14 Sep 2010 11:34:26 -0700 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <19599.42562.239559.453455@montanaro.dyndns.org> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <19599.42562.239559.453455@montanaro.dyndns.org> Message-ID: I'm +1 for Antoine to go ahead and do what he wants; the module is busted as-is and I trust him to make a good judgement call. But I wanted to specifically reply to Skip about removal. On Tue, Sep 14, 2010 at 09:43, wrote: > > ? ?Baptiste> Reading this from GMANE ;-) > > I use GMANE too on occasion, but only via the web, as I suppose most people > do. ?I haven't actually used an NNTP-based app like xrn or gnus in probably > two decades. ?Do the GMANE folks publish HTTP v. NNTP statistics for their > service? > > I question if the nntplib module is used enough to warrant inclusion with > the rest of the batteries. ?Not to mention dedication of scarce people > resources. ?Even though Antoine has kindly volunteered to make the necessary > changes, I seem to recall that the email package transition took awhile to > get right (and already had a reasonable test suite I suspect). ?There are > probably more critical parts of Python he could work on instead (though > who's to dispute that Antoine has an NNTP itch?) ?Finally, is there an > application out in the wild using nntplib which is desirous of better NNTP > support than Python currently has? Because it is Antoine and not some random person offering to clean this up I see no need to remove the module. For other modules that turn out to be severely str/bytes broken at this point and does not have a proven core developer to maintain it, then I would agree that removing the module, creating a Mercurial repository for the module, and then having a wiki page listing forks of the code so people can find it more maintained would make sense. But I don't think this is such a case. As an aside, I also think that having the module in pure Python makes it also an easier argument to keep around. If this was an extension module I probably would be more for removing it (e.g., avoiding security issues like we have had with audioop, etc.). > > We got rid of gopherlib a few years ago (deprecated in 2.5, presumably gone > in 2.6). ?I suspect the NNTP protocol has a greatly diminished user base as > well, GMANE's presence notwithstanding. gopherlib was dropped thanks to lack of developer interest. In this case we have interest so that comparison is not 1:1. From skip at pobox.com Tue Sep 14 22:40:51 2010 From: skip at pobox.com (skip at pobox.com) Date: Tue, 14 Sep 2010 15:40:51 -0500 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <19599.42562.239559.453455@montanaro.dyndns.org> Message-ID: <19599.56787.970612.373138@montanaro.dyndns.org> Steve> PS: I read c.l.py-dev using NNTP from Thunderbird. Why anyone Steve> bothers with these web interfaces is beyond me ... Thunderbird is just a special-purpose web browser. ;-) Steve> PPS: What makes you think the email package transition is complete? I didn't mean to suggest that it was complete. I sent my second reply before seeing Barry's reply about Mailman. That's certainly a reasonable use case. Skip From jon+python-dev at unequivocal.co.uk Tue Sep 14 22:40:57 2010 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Tue, 14 Sep 2010 21:40:57 +0100 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <19599.42562.239559.453455@montanaro.dyndns.org> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <19599.42562.239559.453455@montanaro.dyndns.org> Message-ID: <20100914204057.GC23421@snowy.squish.net> On Tue, Sep 14, 2010 at 11:43:46AM -0500, skip at pobox.com wrote: > We got rid of gopherlib a few years ago (deprecated in 2.5, presumably gone > in 2.6). I suspect the NNTP protocol has a greatly diminished user base as > well, GMANE's presence notwithstanding. NNTP is *very* considerably less dead than gopher. From steve at holdenweb.com Tue Sep 14 23:22:59 2010 From: steve at holdenweb.com (Steve Holden) Date: Tue, 14 Sep 2010 17:22:59 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <20100914204057.GC23421@snowy.squish.net> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <19599.42562.239559.453455@montanaro.dyndns.org> <20100914204057.GC23421@snowy.squish.net> Message-ID: On 9/14/2010 4:40 PM, Jon Ribbens wrote: > On Tue, Sep 14, 2010 at 11:43:46AM -0500, skip at pobox.com wrote: >> We got rid of gopherlib a few years ago (deprecated in 2.5, presumably gone >> in 2.6). I suspect the NNTP protocol has a greatly diminished user base as >> well, GMANE's presence notwithstanding. > > NNTP is *very* considerably less dead than gopher. That's an interesting metric. Would you like to list the extant libraries implementing protocols that are *not* "*very* considerably less dead than gopher"? ;-) regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From chris at simplistix.co.uk Tue Sep 14 23:34:35 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 14 Sep 2010 22:34:35 +0100 Subject: [Python-Dev] 2.6.6 is a release candidate? Message-ID: <4C8FEA6B.5090506@simplistix.co.uk> Hi All, http://www.python.org/download/releases/2.6.6/ ..includes the text: "This is a release candidate; we currently support these formats:" Is that text meant to be there? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From jon+python-dev at unequivocal.co.uk Wed Sep 15 00:41:13 2010 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Tue, 14 Sep 2010 23:41:13 +0100 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <19599.42562.239559.453455@montanaro.dyndns.org> <20100914204057.GC23421@snowy.squish.net> Message-ID: <20100914224113.GD23421@snowy.squish.net> On Tue, Sep 14, 2010 at 05:22:59PM -0400, Steve Holden wrote: > On 9/14/2010 4:40 PM, Jon Ribbens wrote: > > NNTP is *very* considerably less dead than gopher. > > That's an interesting metric. Would you like to list the extant > libraries implementing protocols that are *not* "*very* considerably > less dead than gopher"? ;-) Hmm, interesting question. "uu" perhaps? ;-) From rdmurray at bitdance.com Wed Sep 15 00:45:24 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 14 Sep 2010 18:45:24 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <20100914110433.GA17037@remy> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> Message-ID: <20100914224525.0489C159FBD@kimball.webabinitio.net> On Tue, 14 Sep 2010 16:34:33 +0530, Senthil Kumaran wrote: > On Tue, Sep 14, 2010 at 12:44:30PM +0200, Baptiste Carvello wrote: > > > Antoine> Like the email package, nntplib in py3k is broken (because of > > > Antoine> various bytes/str mismatches; I suppose the lack of a test > > > Antoine> suite didn't help when porting). > > > > > >How heavily used is nntp these days (unless you're looking for spam)? Would > > >it make more sense to find someone willing to maintain it outside the Python > > >core and just remove it altogether? > > > > > Reading this from GMANE ;-) > > I guess, Skip's question or intention was, how often nntplib as a > module is being used these days to write scripts/tools or clients? > Very rarely. > > It would definitely be interesting to know, if there are python > applications out there which are using nntplib at the moment. You all might find it interesting to know that I'm now maintaining email and working on email6 as a direct consequence of nntplib. I was using it to read mailing lists through gmane, and when I tried to port my nntp tool to Python3 I found that decode_header (among other things) was broken, and as a consequence of talking to Barry about that walked in to the email minefield.... I'm currently not using my nttp reader, but it is because I couldn't stand working on my client in Python2, I wanted to be using Python3. So I volunteered to help with email...but I figure I'll come back around and help Antoine with nttplib by and by :) --David From steve at holdenweb.com Wed Sep 15 01:57:24 2010 From: steve at holdenweb.com (Steve Holden) Date: Tue, 14 Sep 2010 19:57:24 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <20100914224525.0489C159FBD@kimball.webabinitio.net> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> Message-ID: <4C900BE4.2000609@holdenweb.com> On 9/14/2010 6:45 PM, R. David Murray wrote: > On Tue, 14 Sep 2010 16:34:33 +0530, Senthil Kumaran wrote: >> On Tue, Sep 14, 2010 at 12:44:30PM +0200, Baptiste Carvello wrote: >>>> Antoine> Like the email package, nntplib in py3k is broken (because of >>>> Antoine> various bytes/str mismatches; I suppose the lack of a test >>>> Antoine> suite didn't help when porting). >>>> >>>> How heavily used is nntp these days (unless you're looking for spam)? Would >>>> it make more sense to find someone willing to maintain it outside the Python >>>> core and just remove it altogether? >>>> >>> Reading this from GMANE ;-) >> >> I guess, Skip's question or intention was, how often nntplib as a >> module is being used these days to write scripts/tools or clients? >> Very rarely. >> >> It would definitely be interesting to know, if there are python >> applications out there which are using nntplib at the moment. > > You all might find it interesting to know that I'm now maintaining > email and working on email6 as a direct consequence of nntplib. I was > using it to read mailing lists through gmane, and when I tried to > port my nntp tool to Python3 I found that decode_header (among > other things) was broken, and as a consequence of talking to Barry > about that walked in to the email minefield.... > > I'm currently not using my nttp reader, but it is because I couldn't > stand working on my client in Python2, I wanted to be using Python3. > So I volunteered to help with email...but I figure I'll come back around > and help Antoine with nttplib by and by :) > And again I say, if anyone knows of any budgets to which this work is important, the PSF will be happy to try and tap these people for money that can help the development effort. Frankly I am a little embarrassed by the poor quality of some library code. I think it shows that the "rush to release" which might not have been in Python's best short-term interests, even though actually getting it out the door was a significant occurrence for the long term.. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From steve at holdenweb.com Wed Sep 15 01:57:24 2010 From: steve at holdenweb.com (Steve Holden) Date: Tue, 14 Sep 2010 19:57:24 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <20100914224525.0489C159FBD@kimball.webabinitio.net> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> Message-ID: <4C900BE4.2000609@holdenweb.com> On 9/14/2010 6:45 PM, R. David Murray wrote: > On Tue, 14 Sep 2010 16:34:33 +0530, Senthil Kumaran wrote: >> On Tue, Sep 14, 2010 at 12:44:30PM +0200, Baptiste Carvello wrote: >>>> Antoine> Like the email package, nntplib in py3k is broken (because of >>>> Antoine> various bytes/str mismatches; I suppose the lack of a test >>>> Antoine> suite didn't help when porting). >>>> >>>> How heavily used is nntp these days (unless you're looking for spam)? Would >>>> it make more sense to find someone willing to maintain it outside the Python >>>> core and just remove it altogether? >>>> >>> Reading this from GMANE ;-) >> >> I guess, Skip's question or intention was, how often nntplib as a >> module is being used these days to write scripts/tools or clients? >> Very rarely. >> >> It would definitely be interesting to know, if there are python >> applications out there which are using nntplib at the moment. > > You all might find it interesting to know that I'm now maintaining > email and working on email6 as a direct consequence of nntplib. I was > using it to read mailing lists through gmane, and when I tried to > port my nntp tool to Python3 I found that decode_header (among > other things) was broken, and as a consequence of talking to Barry > about that walked in to the email minefield.... > > I'm currently not using my nttp reader, but it is because I couldn't > stand working on my client in Python2, I wanted to be using Python3. > So I volunteered to help with email...but I figure I'll come back around > and help Antoine with nttplib by and by :) > And again I say, if anyone knows of any budgets to which this work is important, the PSF will be happy to try and tap these people for money that can help the development effort. Frankly I am a little embarrassed by the poor quality of some library code. I think it shows that the "rush to release" which might not have been in Python's best short-term interests, even though actually getting it out the door was a significant occurrence for the long term.. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From debatem1 at gmail.com Wed Sep 15 05:06:29 2010 From: debatem1 at gmail.com (geremy condra) Date: Tue, 14 Sep 2010 20:06:29 -0700 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <19599.42562.239559.453455@montanaro.dyndns.org> <20100914204057.GC23421@snowy.squish.net> Message-ID: On Tue, Sep 14, 2010 at 2:22 PM, Steve Holden wrote: > On 9/14/2010 4:40 PM, Jon Ribbens wrote: >> On Tue, Sep 14, 2010 at 11:43:46AM -0500, skip at pobox.com wrote: >>> We got rid of gopherlib a few years ago (deprecated in 2.5, presumably gone >>> in 2.6). ?I suspect the NNTP protocol has a greatly diminished user base as >>> well, GMANE's presence notwithstanding. >> >> NNTP is *very* considerably less dead than gopher. > > That's an interesting metric. Would you like to list the extant > libraries implementing protocols that are *not* "*very* considerably > less dead than gopher"? ;-) > > regards > ?Steve I ran some statistics on the number of times modules out of the stdlib got imported a few months ago and came up with a reasonably comprehensive list of the least-used things in the stdlib. For the record, since I wound up parsing import statements and know some garbage data got in, its reasonable to assume that a few otherwise valid imports aren't recorded here. But enough with the disclaimers. I'm not sure what the name of the library was originally, but the word 'gopher' does not appear in any of the imports that I was able to parse in pypi. By contrast, nntplib and poplib are tied at 8, and as would be expected there are only a few recognizable names below that- aepack, aetypes, and posixfile are each stuck at 0; fractions, Bastion, and xdrlib have three, etc. The top five are os, sys, unittest, re, and time (in that order) with 27468, 18334, 14714, 13019, and 9906 imports respectively. If it doesn't annoy I can post the whole list, or email it privately to the interested. Geremy Condra From steve at holdenweb.com Wed Sep 15 05:07:27 2010 From: steve at holdenweb.com (Steve Holden) Date: Tue, 14 Sep 2010 23:07:27 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <19599.42562.239559.453455@montanaro.dyndns.org> <20100914204057.GC23421@snowy.squish.net> Message-ID: <4C90386F.50609@holdenweb.com> On 9/14/2010 11:06 PM, geremy condra wrote: > On Tue, Sep 14, 2010 at 2:22 PM, Steve Holden wrote: >> On 9/14/2010 4:40 PM, Jon Ribbens wrote: >>> On Tue, Sep 14, 2010 at 11:43:46AM -0500, skip at pobox.com wrote: >>>> We got rid of gopherlib a few years ago (deprecated in 2.5, presumably gone >>>> in 2.6). I suspect the NNTP protocol has a greatly diminished user base as >>>> well, GMANE's presence notwithstanding. >>> >>> NNTP is *very* considerably less dead than gopher. >> >> That's an interesting metric. Would you like to list the extant >> libraries implementing protocols that are *not* "*very* considerably >> less dead than gopher"? ;-) >> >> regards >> Steve > > I ran some statistics on the number of times modules out of the stdlib > got imported a few months ago and came up with a reasonably > comprehensive list of the least-used things in the stdlib. For the > record, since I wound up parsing import statements and know some > garbage data got in, its reasonable to assume that a few otherwise > valid imports aren't recorded here. But enough with the disclaimers. > > I'm not sure what the name of the library was originally, but the word > 'gopher' does not appear in any of the imports that I was able to > parse in pypi. By contrast, nntplib and poplib are tied at 8, and as > would be expected there are only a few recognizable names below that- > aepack, aetypes, and posixfile are each stuck at 0; fractions, > Bastion, and xdrlib have three, etc. > > The top five are os, sys, unittest, re, and time (in that order) with > 27468, 18334, 14714, 13019, and 9906 imports respectively. > > If it doesn't annoy I can post the whole list, or email it privately > to the interested. > > Geremy Condra > Post it to the list. If people complain, they can complain at me. Thanks very much. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From debatem1 at gmail.com Wed Sep 15 05:18:33 2010 From: debatem1 at gmail.com (geremy condra) Date: Tue, 14 Sep 2010 20:18:33 -0700 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <4C90386F.50609@holdenweb.com> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <19599.42562.239559.453455@montanaro.dyndns.org> <20100914204057.GC23421@snowy.squish.net> <4C90386F.50609@holdenweb.com> Message-ID: On Tue, Sep 14, 2010 at 8:07 PM, Steve Holden wrote: > On 9/14/2010 11:06 PM, geremy condra wrote: >> On Tue, Sep 14, 2010 at 2:22 PM, Steve Holden wrote: >>> On 9/14/2010 4:40 PM, Jon Ribbens wrote: >>>> On Tue, Sep 14, 2010 at 11:43:46AM -0500, skip at pobox.com wrote: >>>>> We got rid of gopherlib a few years ago (deprecated in 2.5, presumably gone >>>>> in 2.6). ?I suspect the NNTP protocol has a greatly diminished user base as >>>>> well, GMANE's presence notwithstanding. >>>> >>>> NNTP is *very* considerably less dead than gopher. >>> >>> That's an interesting metric. Would you like to list the extant >>> libraries implementing protocols that are *not* "*very* considerably >>> less dead than gopher"? ;-) >>> >>> regards >>> ?Steve >> >> I ran some statistics on the number of times modules out of the stdlib >> got imported a few months ago and came up with a reasonably >> comprehensive list of the least-used things in the stdlib. For the >> record, since I wound up parsing import statements and know some >> garbage data got in, its reasonable to assume that a few otherwise >> valid imports aren't recorded here. But enough with the disclaimers. >> >> I'm not sure what the name of the library was originally, but the word >> 'gopher' does not appear in any of the imports that I was able to >> parse in pypi. By contrast, nntplib and poplib are tied at 8, and as >> would be expected there are only a few recognizable names below that- >> aepack, aetypes, and posixfile are each stuck at 0; fractions, >> Bastion, and xdrlib have three, etc. >> >> The top five are os, sys, unittest, re, and time (in that order) with >> 27468, 18334, 14714, 13019, and 9906 imports respectively. >> >> If it doesn't annoy I can post the whole list, or email it privately >> to the interested. >> >> Geremy Condra >> > Post it to the list. If people complain, they can complain at me. > > Thanks very much. Ok then, on your head be it ;) 0 AL 0 ColorPicker 0 DEVICE 0 FL 0 FrameWork 0 Nav 0 PixMapWrapper 0 SUNAUDIODEV 0 aepack 0 aetypes 0 al 0 applesingle 0 autoGIL 0 buildtools 0 cd 0 cfmfile 0 dbhash 0 dl 0 dummy_threading 0 findertools 0 flp 0 fm 0 fpectl 0 gensuitemodule 0 icopen 0 imageop 0 imgfile 0 jpeg 0 macerrors 0 macostools 0 macresource 0 nis 0 posixfile 0 spwd 0 sunaudiodev 0 symtable 0 videoreader 0 winsound 1 Tix 1 audioop 2 ic 3 Bastion 3 binhex 3 dumbdbm 3 dummy_thread 3 fractions 3 future_builtins 3 mailcap 3 ossaudiodev 3 tabnanny 3 xdrlib 4 ScrolledText 4 macpath 4 stringprep 5 DocXMLRPCServer 5 GL 5 aifc 5 mimify 5 sunau 6 fl 6 pickletools 6 statvfs 6 turtle 7 W 8 codeop 8 multifile 8 nntplib 8 poplib 8 sndhdr 9 EasyDialogs 9 pipes 9 pyclbr 10 dbm 10 gdbm 10 imputil 11 MiniAEFrame 11 fpformat 11 numbers 14 CGIHTTPServer 14 pty 16 rexec 18 netrc 19 msvcrt 19 uu 20 rlcompleter 21 compileall 22 tty 24 lib2to3 24 mutex 25 chunk 25 mhlib 27 whichdb 28 robotparser 29 ssl 30 dircache 32 gl 33 runpy 34 posix 36 aetools 36 wave 37 termios 42 bdb 44 imaplib 46 ast 47 bsddb 47 imghdr 50 crypt 50 smtpd 53 Carbon 57 MimeWriter 57 msilib 60 cmath 66 filecmp 67 syslog 68 MacOS 73 cProfile 74 asynchat 74 repr 75 ftplib 76 htmllib 83 abc 91 quopri 93 pkgutil 98 anydbm 98 telnetlib 99 trace 102 formatter 104 __main__ 104 readline 105 colorsys 110 _winreg 111 curses 113 plistlib 115 modulefinder 116 UserString 121 cookielib 125 mailbox 126 cgitb 128 bz2 128 sched 134 io 146 mimetools 147 pydoc 148 SimpleXMLRPCServer 154 mmap 155 user 156 site 157 symbol 159 zipimport 166 pstats 172 fileinput 173 encodings 179 py_compile 180 SimpleHTTPServer 181 profile 183 cmd 198 Tkinter 200 fcntl 206 copy_reg 225 linecache 226 hotshot 234 multiprocessing 262 dis 273 UserList 273 resource 287 SocketServer 289 shelve 297 sqlite3 317 grp 322 asyncore 335 timeit 339 keyword 345 sgmllib 363 token 367 test 383 parser 386 shlex 421 wsgiref 451 contextlib 458 unicodedata 471 tokenize 472 pwd 487 webbrowser 526 hmac 529 heapq 542 platform 573 gettext 594 pdb 597 popen2 607 json 608 marshal 619 smtplib 621 bisect 637 difflib 647 commands 657 BaseHTTPServer 677 Cookie 688 locale 695 zlib 708 HTMLParser 710 code 721 rfc822 748 compiler 759 gzip 759 select 771 ctypes 788 gc 796 binascii 812 getpass 822 __builtin__ 854 htmlentitydefs 857 tarfile 869 decimal 872 xmlrpclib 903 csv 933 atexit 943 functools 946 exceptions 976 array 979 sha 1044 thread 1056 calendar 1064 zipfile 1070 UserDict 1078 new 1102 uuid 1148 Queue 1159 sets 1172 signal 1213 hashlib 1242 getopt 1276 email 1310 imp 1321 fnmatch 1328 mimetypes 1348 collections 1442 httplib 1469 cPickle 1505 md5 1614 weakref 1618 textwrap 1654 pickle 1722 errno 1729 stat 2020 pprint 2060 struct 2389 codecs 2391 ConfigParser 2406 operator 2578 math 2626 base64 2925 inspect 3013 cgi 3105 itertools 3250 xml 3318 glob 3402 __future__ 3505 warnings 3549 socket 3722 urlparse 4014 traceback 4142 subprocess 4194 threading 4198 cStringIO 4224 string 4501 copy 4696 random 5088 shutil 5392 tempfile 5426 doctest 5642 optparse 5913 types 6185 StringIO 6522 urllib 7346 distutils 7930 datetime 8416 urllib2 9567 logging 9906 time 13019 re 14714 unittest 18334 sys 27468 os Enjoy! Geremy Condra From guido at python.org Wed Sep 15 05:33:49 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 14 Sep 2010 20:33:49 -0700 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <19599.42562.239559.453455@montanaro.dyndns.org> <20100914204057.GC23421@snowy.squish.net> Message-ID: On Tue, Sep 14, 2010 at 8:06 PM, geremy condra wrote: > I ran some statistics on the number of times modules out of the stdlib > got imported a few months ago and came up with a reasonably > comprehensive list of the least-used things in the stdlib. For the > record, since I wound up parsing import statements and know some > garbage data got in, its reasonable to assume that a few otherwise > valid imports aren't recorded here. But enough with the disclaimers. Neat! > I'm not sure what the name of the library was originally, but the word > 'gopher' does not appear in any of the imports that I was able to > parse in pypi. By contrast, nntplib and poplib are tied at 8, and as > would be expected there are only a few recognizable names below that- > aepack, aetypes, and posixfile are each stuck at 0; fractions, > Bastion, and xdrlib have three, etc. > > The top five are os, sys, unittest, re, and time (in that order) with > 27468, 18334, 14714, 13019, and 9906 imports respectively. Looks like we did a poor job naming unittest. The Huffman-encoding seems to have worked well for the others though. > If it doesn't annoy I can post the whole list, or email it privately > to the interested. Do post. -- --Guido van Rossum (python.org/~guido) From debatem1 at gmail.com Wed Sep 15 08:52:43 2010 From: debatem1 at gmail.com (geremy condra) Date: Tue, 14 Sep 2010 23:52:43 -0700 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <19599.42562.239559.453455@montanaro.dyndns.org> <20100914204057.GC23421@snowy.squish.net> Message-ID: On Tue, Sep 14, 2010 at 8:33 PM, Guido van Rossum wrote: > On Tue, Sep 14, 2010 at 8:06 PM, geremy condra wrote: >> I ran some statistics on the number of times modules out of the stdlib >> got imported a few months ago and came up with a reasonably >> comprehensive list of the least-used things in the stdlib. For the >> record, since I wound up parsing import statements and know some >> garbage data got in, its reasonable to assume that a few otherwise >> valid imports aren't recorded here. But enough with the disclaimers. > > Neat! > >> I'm not sure what the name of the library was originally, but the word >> 'gopher' does not appear in any of the imports that I was able to >> parse in pypi. By contrast, nntplib and poplib are tied at 8, and as >> would be expected there are only a few recognizable names below that- >> aepack, aetypes, and posixfile are each stuck at 0; fractions, >> Bastion, and xdrlib have three, etc. >> >> The top five are os, sys, unittest, re, and time (in that order) with >> 27468, 18334, 14714, 13019, and 9906 imports respectively. > > Looks like we did a poor job naming unittest. The Huffman-encoding > seems to have worked well for the others though. > >> If it doesn't annoy I can post the whole list, or email it privately >> to the interested. > > Do post. Looks like this didn't come through last time. Here you go- 0 AL 0 ColorPicker 0 DEVICE 0 FL 0 FrameWork 0 Nav 0 PixMapWrapper 0 SUNAUDIODEV 0 aepack 0 aetypes 0 al 0 applesingle 0 autoGIL 0 buildtools 0 cd 0 cfmfile 0 dbhash 0 dl 0 dummy_threading 0 findertools 0 flp 0 fm 0 fpectl 0 gensuitemodule 0 icopen 0 imageop 0 imgfile 0 jpeg 0 macerrors 0 macostools 0 macresource 0 nis 0 posixfile 0 spwd 0 sunaudiodev 0 symtable 0 videoreader 0 winsound 1 Tix 1 audioop 2 ic 3 Bastion 3 binhex 3 dumbdbm 3 dummy_thread 3 fractions 3 future_builtins 3 mailcap 3 ossaudiodev 3 tabnanny 3 xdrlib 4 ScrolledText 4 macpath 4 stringprep 5 DocXMLRPCServer 5 GL 5 aifc 5 mimify 5 sunau 6 fl 6 pickletools 6 statvfs 6 turtle 7 W 8 codeop 8 multifile 8 nntplib 8 poplib 8 sndhdr 9 EasyDialogs 9 pipes 9 pyclbr 10 dbm 10 gdbm 10 imputil 11 MiniAEFrame 11 fpformat 11 numbers 14 CGIHTTPServer 14 pty 16 rexec 18 netrc 19 msvcrt 19 uu 20 rlcompleter 21 compileall 22 tty 24 lib2to3 24 mutex 25 chunk 25 mhlib 27 whichdb 28 robotparser 29 ssl 30 dircache 32 gl 33 runpy 34 posix 36 aetools 36 wave 37 termios 42 bdb 44 imaplib 46 ast 47 bsddb 47 imghdr 50 crypt 50 smtpd 53 Carbon 57 MimeWriter 57 msilib 60 cmath 66 filecmp 67 syslog 68 MacOS 73 cProfile 74 asynchat 74 repr 75 ftplib 76 htmllib 83 abc 91 quopri 93 pkgutil 98 anydbm 98 telnetlib 99 trace 102 formatter 104 __main__ 104 readline 105 colorsys 110 _winreg 111 curses 113 plistlib 115 modulefinder 116 UserString 121 cookielib 125 mailbox 126 cgitb 128 bz2 128 sched 134 io 146 mimetools 147 pydoc 148 SimpleXMLRPCServer 154 mmap 155 user 156 site 157 symbol 159 zipimport 166 pstats 172 fileinput 173 encodings 179 py_compile 180 SimpleHTTPServer 181 profile 183 cmd 198 Tkinter 200 fcntl 206 copy_reg 225 linecache 226 hotshot 234 multiprocessing 262 dis 273 UserList 273 resource 287 SocketServer 289 shelve 297 sqlite3 317 grp 322 asyncore 335 timeit 339 keyword 345 sgmllib 363 token 367 test 383 parser 386 shlex 421 wsgiref 451 contextlib 458 unicodedata 471 tokenize 472 pwd 487 webbrowser 526 hmac 529 heapq 542 platform 573 gettext 594 pdb 597 popen2 607 json 608 marshal 619 smtplib 621 bisect 637 difflib 647 commands 657 BaseHTTPServer 677 Cookie 688 locale 695 zlib 708 HTMLParser 710 code 721 rfc822 748 compiler 759 gzip 759 select 771 ctypes 788 gc 796 binascii 812 getpass 822 __builtin__ 854 htmlentitydefs 857 tarfile 869 decimal 872 xmlrpclib 903 csv 933 atexit 943 functools 946 exceptions 976 array 979 sha 1044 thread 1056 calendar 1064 zipfile 1070 UserDict 1078 new 1102 uuid 1148 Queue 1159 sets 1172 signal 1213 hashlib 1242 getopt 1276 email 1310 imp 1321 fnmatch 1328 mimetypes 1348 collections 1442 httplib 1469 cPickle 1505 md5 1614 weakref 1618 textwrap 1654 pickle 1722 errno 1729 stat 2020 pprint 2060 struct 2389 codecs 2391 ConfigParser 2406 operator 2578 math 2626 base64 2925 inspect 3013 cgi 3105 itertools 3250 xml 3318 glob 3402 __future__ 3505 warnings 3549 socket 3722 urlparse 4014 traceback 4142 subprocess 4194 threading 4198 cStringIO 4224 string 4501 copy 4696 random 5088 shutil 5392 tempfile 5426 doctest 5642 optparse 5913 types 6185 StringIO 6522 urllib 7346 distutils 7930 datetime 8416 urllib2 9567 logging 9906 time 13019 re 14714 unittest 18334 sys 27468 os Enjoy! Geremy Condra From steve at pearwood.info Wed Sep 15 13:18:19 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 15 Sep 2010 21:18:19 +1000 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> Message-ID: <201009152118.19591.steve@pearwood.info> On Wed, 15 Sep 2010 01:06:29 pm geremy condra wrote: > I ran some statistics on the number of times modules out of the > stdlib got imported a few months ago What do those statistics measure? The number of individual import statements with a given module name? The number of times those imports are actually executed? Something else? > and came up with a reasonably > comprehensive list of the least-used things in the stdlib. Least used by whom? > I'm not sure what the name of the library was originally, but the > word 'gopher' does not appear in any of the imports that I was able > to parse in pypi. By contrast, nntplib and poplib are tied at 8 [...] I don't know how to interpret that. Does that mean that there are eight modules in the whole of PyPi which import nntplib or poplib? If so, what does that tell us? Those eight modules could have three users between them, or they could be critical infrastructure for a quarter of the Internet. I'm not trying to belittle the stats you have gathered, but without the context of *what* the numbers represent, it's impossible to put any meaning to them. -- Steven D'Aprano From orsenthil at gmail.com Wed Sep 15 13:54:11 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Wed, 15 Sep 2010 17:24:11 +0530 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <201009152118.19591.steve@pearwood.info> References: <20100914121744.31ef5aaf@pitrou.net> <201009152118.19591.steve@pearwood.info> Message-ID: <20100915115411.GA16903@remy> On Wed, Sep 15, 2010 at 09:18:19PM +1000, Steven D'Aprano wrote: > I'm not trying to belittle the stats you have gathered, but without the > context of *what* the numbers represent, it's impossible to put any > meaning to them. I thought Geremy mentioned somewhere that he collected those metrics as number of times the stdlib modules were imported by the packages in the pypi infrastructure. -- Senthil From debatem1 at gmail.com Wed Sep 15 14:06:37 2010 From: debatem1 at gmail.com (geremy condra) Date: Wed, 15 Sep 2010 05:06:37 -0700 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <201009152118.19591.steve@pearwood.info> References: <20100914121744.31ef5aaf@pitrou.net> <201009152118.19591.steve@pearwood.info> Message-ID: On Wed, Sep 15, 2010 at 4:18 AM, Steven D'Aprano wrote: > On Wed, 15 Sep 2010 01:06:29 pm geremy condra wrote: > >> I ran some statistics on the number of times modules out of the >> stdlib got imported a few months ago > > What do those statistics measure? > > The number of individual import statements with a given module name? The > number of times those imports are actually executed? Something else? Not to put too fine a point on it, but I already told you that. It's the number of packages on pypi that imported each of those, and I measured it by parsing for import statements. >> and came up with a reasonably >> comprehensive list of the least-used things in the stdlib. > > Least used by whom? Not to be pedantic, but there isn't a good answer to a question as vague as this. If you want a piece of information, ask for it and I'll tell you if I can get it or not. >> I'm not sure what the name of the library was originally, but the >> word 'gopher' does not appear in any of the imports that I was able >> to parse in pypi. By contrast, nntplib and poplib are tied at 8 > [...] > > > I don't know how to interpret that. Does that mean that there are eight > modules in the whole of PyPi which import nntplib or poplib? If so, > what does that tell us? Those eight modules could have three users > between them, or they could be critical infrastructure for a quarter of > the Internet. Yup. The closest I could come to representing that would be by the number of downloads, and IIRC I have that information laying around here somewhere. It would probably be several days before I could get ahold of it though. > I'm not trying to belittle the stats you have gathered, but without the > context of *what* the numbers represent, it's impossible to put any > meaning to them. Not really. It's a metric of use, and as such a starting point in a discussion. After that I wouldn't take it too seriously no matter what metric is applied- looking at the number of downloads on pypi it's a fairly small sample anyway. Geremy Condra From ncoghlan at gmail.com Wed Sep 15 14:36:29 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Sep 2010 22:36:29 +1000 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <201009152118.19591.steve@pearwood.info> Message-ID: On Wed, Sep 15, 2010 at 10:06 PM, geremy condra wrote: > Not really. It's a metric of use, and as such a starting point in a > discussion. After that I wouldn't take it too seriously no matter what > metric is applied- looking at the number of downloads on pypi it's a > fairly small sample anyway. Agreed. Still an interesting snapshot though. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Sep 15 15:28:20 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Sep 2010 23:28:20 +1000 Subject: [Python-Dev] [Python-checkins] r84823 - python/branches/py3k/Doc/reference/expressions.rst In-Reply-To: <20100915000926.8AFF3EE9B1@mail.python.org> References: <20100915000926.8AFF3EE9B1@mail.python.org> Message-ID: On Wed, Sep 15, 2010 at 10:09 AM, raymond.hettinger wrote: > +The formal syntax makes no special provision for negative indices in > +sequences; however, built-in sequences all provide a :meth:`__getitem__` > +method that interprets negative indices by adding the length of the sequence > +to the index (so that ``x[-1]`` selects the last item of ``x``). ?The > +resulting value must be a nonnegative integer less than the number of items in > +the sequence, and the subscription selects the item whose index is that value > +(counting from zero). Since the support for negative indices and slicing > +occurs in the object's :meth:`__getitem__` method, subclasses overriding > +this method will need to explicitly add that support. Perhaps mention the slice.indices(len) helper for performing the standard conversion from negative indices to positive ones when dealing with negative indices *in* slices? Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Sep 15 15:52:05 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Sep 2010 23:52:05 +1000 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <4C900BE4.2000609@holdenweb.com> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> Message-ID: On Wed, Sep 15, 2010 at 9:57 AM, Steve Holden wrote: > And again I say, if anyone knows of any budgets to which this work is > important, the PSF will be happy to try and tap these people for money > that can help the development effort. Frankly I am a little embarrassed > by the poor quality of some library code. > > I think it shows that the "rush to release" which might not have been in > Python's best short-term interests, even though actually getting it out > the door was a significant occurrence for the long term.. I'm not sure we would ever have realised the full implications of the bytes/str split without pushing 3.0 out the door when we did. And it was the early feedback on 3.0 that showed 3.1 really was necessary to deal with the I/O performance issue. I agree the current state of things is not ideal, but I don't think it's a given waiting longer to release 3.x would have actually helped anything (and there is already code out there showing that, so long as you don't need the modules with bytes/unicode issues, Python 3 is very usable). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From jnoller at gmail.com Wed Sep 15 16:02:59 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 15 Sep 2010 10:02:59 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <4C900BE4.2000609@holdenweb.com> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> Message-ID: On Tue, Sep 14, 2010 at 7:57 PM, Steve Holden wrote: > On 9/14/2010 6:45 PM, R. David Murray wrote: >> On Tue, 14 Sep 2010 16:34:33 +0530, Senthil Kumaran wrote: >>> On Tue, Sep 14, 2010 at 12:44:30PM +0200, Baptiste Carvello wrote: >>>>> ? ?Antoine> Like the email package, nntplib in py3k is broken (because of >>>>> ? ?Antoine> various bytes/str mismatches; I suppose the lack of a test >>>>> ? ?Antoine> suite didn't help when porting). >>>>> >>>>> How heavily used is nntp these days (unless you're looking for spam)? ?Would >>>>> it make more sense to find someone willing to maintain it outside the Python >>>>> core and just remove it altogether? >>>>> >>>> Reading this from GMANE ;-) >>> >>> I guess, Skip's question or intention was, how often nntplib as a >>> module is being used these days to write scripts/tools or clients? >>> Very rarely. >>> >>> It would definitely be interesting to know, if there are python >>> applications out there which are using nntplib at the moment. >> >> You all might find it interesting to know that I'm now maintaining >> email and working on email6 as a direct consequence of nntplib. ?I was >> using it to read mailing lists through gmane, and when I tried to >> port my nntp tool to Python3 I found that decode_header (among >> other things) was broken, and as a consequence of talking to Barry >> about that walked in to the email minefield.... >> >> I'm currently not using my nttp reader, but it is because I couldn't >> stand working on my client in Python2, I wanted to be using Python3. >> So I volunteered to help with email...but I figure I'll come back around >> and help Antoine with nttplib by and by :) >> > And again I say, if anyone knows of any budgets to which this work is > important, the PSF will be happy to try and tap these people for money > that can help the development effort. Frankly I am a little embarrassed > by the poor quality of some library code. > > I think it shows that the "rush to release" which might not have been in > Python's best short-term interests, even though actually getting it out > the door was a significant occurrence for the long term.. > > regards > ?Steve Without the release we probably would not have found out about these issues; no one seems to take the betas or alphas for serious test drives (to be expected) with real code, so yeah, in hindsight, there are issues - but then again, they would have been fixed if everyone had really known about them in advance. No one wants to ship something with horrible bugs in it. From steve at holdenweb.com Wed Sep 15 16:21:11 2010 From: steve at holdenweb.com (Steve Holden) Date: Wed, 15 Sep 2010 10:21:11 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> Message-ID: <4C90D657.8070105@holdenweb.com> On 9/15/2010 10:02 AM, Jesse Noller wrote: > On Tue, Sep 14, 2010 at 7:57 PM, Steve Holden wrote: >> On 9/14/2010 6:45 PM, R. David Murray wrote: >>> On Tue, 14 Sep 2010 16:34:33 +0530, Senthil Kumaran wrote: >>>> On Tue, Sep 14, 2010 at 12:44:30PM +0200, Baptiste Carvello wrote: >>>>>> Antoine> Like the email package, nntplib in py3k is broken (because of >>>>>> Antoine> various bytes/str mismatches; I suppose the lack of a test >>>>>> Antoine> suite didn't help when porting). >>>>>> >>>>>> How heavily used is nntp these days (unless you're looking for spam)? Would >>>>>> it make more sense to find someone willing to maintain it outside the Python >>>>>> core and just remove it altogether? >>>>>> >>>>> Reading this from GMANE ;-) >>>> >>>> I guess, Skip's question or intention was, how often nntplib as a >>>> module is being used these days to write scripts/tools or clients? >>>> Very rarely. >>>> >>>> It would definitely be interesting to know, if there are python >>>> applications out there which are using nntplib at the moment. >>> >>> You all might find it interesting to know that I'm now maintaining >>> email and working on email6 as a direct consequence of nntplib. I was >>> using it to read mailing lists through gmane, and when I tried to >>> port my nntp tool to Python3 I found that decode_header (among >>> other things) was broken, and as a consequence of talking to Barry >>> about that walked in to the email minefield.... >>> >>> I'm currently not using my nttp reader, but it is because I couldn't >>> stand working on my client in Python2, I wanted to be using Python3. >>> So I volunteered to help with email...but I figure I'll come back around >>> and help Antoine with nttplib by and by :) >>> >> And again I say, if anyone knows of any budgets to which this work is >> important, the PSF will be happy to try and tap these people for money >> that can help the development effort. Frankly I am a little embarrassed >> by the poor quality of some library code. >> >> I think it shows that the "rush to release" which might not have been in >> Python's best short-term interests, even though actually getting it out >> the door was a significant occurrence for the long term.. >> >> regards >> Steve > > Without the release we probably would not have found out about these > issues; no one seems to take the betas or alphas for serious test > drives (to be expected) with real code, so yeah, in hindsight, there > are issues - but then again, they would have been fixed if everyone > had really known about them in advance. No one wants to ship something > with horrible bugs in it. > Well, consider my remarks in a historical rather than a critical context. I have no beef with 3.0 having come out when it did, and see no reason why 3.x shouldn't continue to yield lessons for all Python implementers. The question of when to declare 3.x the "official" release is interesting. I am inclined to say "when there's at least one other implementation at 3.2" - even if CPython is then at 3.3 or 3.4. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From jnoller at gmail.com Wed Sep 15 16:38:52 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 15 Sep 2010 10:38:52 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <4C90D657.8070105@holdenweb.com> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> Message-ID: On Wed, Sep 15, 2010 at 10:21 AM, Steve Holden wrote: > On 9/15/2010 10:02 AM, Jesse Noller wrote: >> On Tue, Sep 14, 2010 at 7:57 PM, Steve Holden wrote: >>> On 9/14/2010 6:45 PM, R. David Murray wrote: >>>> On Tue, 14 Sep 2010 16:34:33 +0530, Senthil Kumaran wrote: >>>>> On Tue, Sep 14, 2010 at 12:44:30PM +0200, Baptiste Carvello wrote: >>>>>>> ? ?Antoine> Like the email package, nntplib in py3k is broken (because of >>>>>>> ? ?Antoine> various bytes/str mismatches; I suppose the lack of a test >>>>>>> ? ?Antoine> suite didn't help when porting). >>>>>>> >>>>>>> How heavily used is nntp these days (unless you're looking for spam)? ?Would >>>>>>> it make more sense to find someone willing to maintain it outside the Python >>>>>>> core and just remove it altogether? >>>>>>> >>>>>> Reading this from GMANE ;-) >>>>> >>>>> I guess, Skip's question or intention was, how often nntplib as a >>>>> module is being used these days to write scripts/tools or clients? >>>>> Very rarely. >>>>> >>>>> It would definitely be interesting to know, if there are python >>>>> applications out there which are using nntplib at the moment. >>>> >>>> You all might find it interesting to know that I'm now maintaining >>>> email and working on email6 as a direct consequence of nntplib. ?I was >>>> using it to read mailing lists through gmane, and when I tried to >>>> port my nntp tool to Python3 I found that decode_header (among >>>> other things) was broken, and as a consequence of talking to Barry >>>> about that walked in to the email minefield.... >>>> >>>> I'm currently not using my nttp reader, but it is because I couldn't >>>> stand working on my client in Python2, I wanted to be using Python3. >>>> So I volunteered to help with email...but I figure I'll come back around >>>> and help Antoine with nttplib by and by :) >>>> >>> And again I say, if anyone knows of any budgets to which this work is >>> important, the PSF will be happy to try and tap these people for money >>> that can help the development effort. Frankly I am a little embarrassed >>> by the poor quality of some library code. >>> >>> I think it shows that the "rush to release" which might not have been in >>> Python's best short-term interests, even though actually getting it out >>> the door was a significant occurrence for the long term.. >>> >>> regards >>> ?Steve >> >> Without the release we probably would not have found out about these >> issues; no one seems to take the betas or alphas for serious test >> drives (to be expected) with real code, so yeah, in hindsight, there >> are issues - but then again, they would have been fixed if everyone >> had really known about them in advance. No one wants to ship something >> with horrible bugs in it. >> > Well, consider my remarks in a historical rather than a critical > context. I have no beef with 3.0 having come out when it did, and see no > reason why 3.x shouldn't continue to yield lessons for all Python > implementers. > > The question of when to declare 3.x the "official" release is > interesting. I am inclined to say "when there's at least one other > implementation at 3.2" - even if CPython is then at 3.3 or 3.4. The moratorium is in place to allow this catch-up to occur (in theory) as cpython is not as large a moving target. However, I don't see how waiting for another implementation, which would probably inherit the same standard library code (and therefore, the same issues) is going to help with issues in the 3.x standard library. Besides; we have two official releases of python - 2.7 and 3.1, I see us as already having declared or decided to mark it as official, even if it's not entirely bug free. From solipsis at pitrou.net Wed Sep 15 16:43:08 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 15 Sep 2010 16:43:08 +0200 Subject: [Python-Dev] 3.x as the official release References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> Message-ID: <20100915164308.6ef1ef63@pitrou.net> On Wed, 15 Sep 2010 10:21:11 -0400 Steve Holden wrote: > > The question of when to declare 3.x the "official" release is > interesting. I am inclined to say "when there's at least one other > implementation at 3.2" - even if CPython is then at 3.3 or 3.4. I don't think that's a good criterion. 95% of Python users (my guesstimate) are on CPython, so whether or not alternative implementations are up-to-date isn't critically important. 3.1 had some warts left (*), but 3.2 should really be a high-quality release. Many bugs have been squashed, small improvements done (including additional features in the stdlib, or the new GIL), and unicode support has been polished again thanks to Martin's and Victor's efforts. Not only will it be as robust as any 2.x release (**), but it's also more pleasant to use, and there's upwards compatibility for many years to come. (*) some of them fixed in the 3.1 maintenance branch (**) with a couple of lacking areas such as the email module, I suppose Regards Antoine. From jnoller at gmail.com Wed Sep 15 16:50:18 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 15 Sep 2010 10:50:18 -0400 Subject: [Python-Dev] 3.x as the official release In-Reply-To: <20100915164308.6ef1ef63@pitrou.net> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> Message-ID: On Wed, Sep 15, 2010 at 10:43 AM, Antoine Pitrou wrote: > On Wed, 15 Sep 2010 10:21:11 -0400 > Steve Holden wrote: >> >> The question of when to declare 3.x the "official" release is >> interesting. I am inclined to say "when there's at least one other >> implementation at 3.2" - even if CPython is then at 3.3 or 3.4. > > I don't think that's a good criterion. 95% of Python users (my > guesstimate) are on CPython, so whether or not alternative > implementations are up-to-date isn't critically important. > > 3.1 had some warts left (*), but 3.2 should really be a high-quality > release. Many bugs have been squashed, small improvements done > (including additional features in the stdlib, or the new GIL), and > unicode support has been polished again thanks to Martin's and Victor's > efforts. Not only will it be as robust as any 2.x release (**), but it's > also more pleasant to use, and there's upwards compatibility for many > years to come. > > (*) some of them fixed in the 3.1 maintenance branch > > (**) with a couple of lacking areas such as the email module, I suppose > > Regards > > Antoine. +0.5 The one area I have concerns about is the state of WSGI and other web-oriented modules. These issues have been brought up by Armin and others, but given a lack of a clear path forward (bugs, peps, etc), I don't think it's fair to use it as a measurement of overall quality. jesse From barry at python.org Wed Sep 15 16:54:03 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 15 Sep 2010 10:54:03 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <4C90D657.8070105@holdenweb.com> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> Message-ID: <20100915105403.1be65573@mission> On Sep 15, 2010, at 10:21 AM, Steve Holden wrote: >The question of when to declare 3.x the "official" release is >interesting. I am inclined to say "when there's at least one other >implementation at 3.2" - even if CPython is then at 3.3 or 3.4. The fun starts now. We've just seen the last official release of Python 2, and while it will be maintained for a long time (and used for longer I'm sure), we've put out the clear message that the future is Python 3. So I think you'll start to see momentum around Python 3 pick up, and that includes improvements to the stdlib. In fact, with the moratorium in place, I'm hoping *most* of the incredible Pythonic brain power will be devoted to Python 3 stdlib improvements. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Sep 15 17:01:47 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 15 Sep 2010 11:01:47 -0400 Subject: [Python-Dev] 2.6.6 is a release candidate? In-Reply-To: <4C8FEA6B.5090506@simplistix.co.uk> References: <4C8FEA6B.5090506@simplistix.co.uk> Message-ID: <20100915110147.4959df0a@mission> On Sep 14, 2010, at 10:34 PM, Chris Withers wrote: >http://www.python.org/download/releases/2.6.6/ > >..includes the text: > >"This is a release candidate; we currently support these formats:" > >Is that text meant to be there? Nope, and fixed. Thanks for noticing this typo. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From janssen at parc.com Wed Sep 15 18:14:05 2010 From: janssen at parc.com (Bill Janssen) Date: Wed, 15 Sep 2010 09:14:05 PDT Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> Message-ID: <66528.1284567245@parc.com> Jesse Noller wrote: > no one seems to take the betas or alphas for serious test drives (to > be expected) with real code I wonder if there's some way to improve that situation -- perhaps by some engineering of the Python packaging, or some such? Bill From janssen at parc.com Wed Sep 15 18:11:33 2010 From: janssen at parc.com (Bill Janssen) Date: Wed, 15 Sep 2010 09:11:33 PDT Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <19599.42562.239559.453455@montanaro.dyndns.org> <20100914204057.GC23421@snowy.squish.net> <4C90386F.50609@holdenweb.com> Message-ID: <66384.1284567093@parc.com> Interesting. I personally use nis and poplib quite a bit, but I can see how that would be very location-dependent. Bill From brett at python.org Wed Sep 15 18:22:12 2010 From: brett at python.org (Brett Cannon) Date: Wed, 15 Sep 2010 09:22:12 -0700 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> Message-ID: On Wed, Sep 15, 2010 at 07:50, Jesse Noller wrote: > On Wed, Sep 15, 2010 at 10:43 AM, Antoine Pitrou wrote: >> On Wed, 15 Sep 2010 10:21:11 -0400 >> Steve Holden wrote: >>> >>> The question of when to declare 3.x the "official" release is >>> interesting. I am inclined to say "when there's at least one other >>> implementation at 3.2" - even if CPython is then at 3.3 or 3.4. >> >> I don't think that's a good criterion. 95% of Python users (my >> guesstimate) are on CPython, so whether or not alternative >> implementations are up-to-date isn't critically important. >> >> 3.1 had some warts left (*), but 3.2 should really be a high-quality >> release. Many bugs have been squashed, small improvements done >> (including additional features in the stdlib, or the new GIL), and >> unicode support has been polished again thanks to Martin's and Victor's >> efforts. Not only will it be as robust as any 2.x release (**), but it's >> also more pleasant to use, and there's upwards compatibility for many >> years to come. >> >> (*) some of them fixed in the 3.1 maintenance branch >> >> (**) with a couple of lacking areas such as the email module, I suppose >> >> Regards >> >> Antoine. > > +0.5 > > The one area I have concerns about is the state of WSGI and other > web-oriented modules. These issues have been brought up by Armin and > others, but given a lack of a clear path forward (bugs, peps, etc), I > don't think it's fair to use it as a measurement of overall quality. The whole WSGI situation is not going to get cleared up (from my understanding) until someone flat-out declares a winner in the whole str/bytes argument that keeps coming up. I think it might be time to have a PEP or two on this and use our new PEP dictator procedure to settle this so it stops dragging on (unless it has been miraculously settled and I am just unaware of it). From jnoller at gmail.com Wed Sep 15 18:35:14 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 15 Sep 2010 12:35:14 -0400 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> Message-ID: On Wed, Sep 15, 2010 at 12:22 PM, Brett Cannon wrote: ...snip... >> The one area I have concerns about is the state of WSGI and other >> web-oriented modules. These issues have been brought up by Armin and >> others, but given a lack of a clear path forward (bugs, peps, etc), I >> don't think it's fair to use it as a measurement of overall quality. > > The whole WSGI situation is not going to get cleared up (from my > understanding) until someone flat-out declares a winner in the whole > str/bytes argument that keeps coming up. I think it might be time to > have a PEP or two on this and use our new PEP dictator procedure to > settle this so it stops dragging on (unless it has been miraculously > settled and I am just unaware of it). > Yup, and I spoke with some people with horses in that race at Djangocon. The important thing is that the PEP(s) and suggestion come from the people with the most experience in that domain. That's why I said we (in the "committer" sense) need a clear path of things we need to change or fix - without it we're just stabbing in the dark. jesse From jnoller at gmail.com Wed Sep 15 18:37:06 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 15 Sep 2010 12:37:06 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <66528.1284567245@parc.com> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <66528.1284567245@parc.com> Message-ID: On Wed, Sep 15, 2010 at 12:14 PM, Bill Janssen wrote: > Jesse Noller wrote: > >> no one seems to take the betas or alphas for serious test drives (to >> be expected) with real code > > I wonder if there's some way to improve that situation -- perhaps by > some engineering of the Python packaging, or some such? > > Bill > You need people with the time and willingness to download, install and run production code on the releases. I don't know that fudging with the packaging would help as python versions can and do live side by side in most cases. From fuzzyman at voidspace.org.uk Wed Sep 15 18:44:34 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 15 Sep 2010 17:44:34 +0100 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> Message-ID: <4C90F7F2.3060200@voidspace.org.uk> On 15/09/2010 17:35, Jesse Noller wrote: > On Wed, Sep 15, 2010 at 12:22 PM, Brett Cannon wrote: > ...snip... >>> The one area I have concerns about is the state of WSGI and other >>> web-oriented modules. These issues have been brought up by Armin and >>> others, but given a lack of a clear path forward (bugs, peps, etc), I >>> don't think it's fair to use it as a measurement of overall quality. >> The whole WSGI situation is not going to get cleared up (from my >> understanding) until someone flat-out declares a winner in the whole >> str/bytes argument that keeps coming up. I think it might be time to >> have a PEP or two on this and use our new PEP dictator procedure to >> settle this so it stops dragging on (unless it has been miraculously >> settled and I am just unaware of it). >> > Yup, and I spoke with some people with horses in that race at > Djangocon. The important thing is that the PEP(s) and suggestion come > from the people with the most experience in that domain. That's why I > said we (in the "committer" sense) need a clear path of things we need > to change or fix - without it we're just stabbing in the dark. I agree. wsgi is a different kettle of fish to the other "web related modules" in the standard library though. (wsgiref is the only directly wsgi related standard library module IIUC.) email, cgi, nntplib, ftplib and friends all still need to work correctly with both bytes and strings and that shouldn't need a great deal of discussion (well perhaps except email) just people willing to do the work. Unfortunately in some cases will need backwards incompatible changes. For example at the moment cgi reads from stdin in text mode and so is broken for file uploads. We have also heard recently from Antoine about backwards incompatible changes required in nntplib. All the best, Michael > jesse > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From brett at python.org Wed Sep 15 18:55:19 2010 From: brett at python.org (Brett Cannon) Date: Wed, 15 Sep 2010 09:55:19 -0700 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> Message-ID: On Wed, Sep 15, 2010 at 09:35, Jesse Noller wrote: > On Wed, Sep 15, 2010 at 12:22 PM, Brett Cannon wrote: > ...snip... >>> The one area I have concerns about is the state of WSGI and other >>> web-oriented modules. These issues have been brought up by Armin and >>> others, but given a lack of a clear path forward (bugs, peps, etc), I >>> don't think it's fair to use it as a measurement of overall quality. >> >> The whole WSGI situation is not going to get cleared up (from my >> understanding) until someone flat-out declares a winner in the whole >> str/bytes argument that keeps coming up. I think it might be time to >> have a PEP or two on this and use our new PEP dictator procedure to >> settle this so it stops dragging on (unless it has been miraculously >> settled and I am just unaware of it). >> > > Yup, and I spoke with some people with horses in that race at > Djangocon. The important thing is that the PEP(s) and suggestion come > from the people with the most experience in that domain. Yes. They have to be people who are not only stakeholders but people who actively use and develop large applications using WSGI. > That's why I > said we (in the "committer" sense) need a clear path of things we need > to change or fix - without it we're just stabbing in the dark. So, who do we get to write the PEP(s)? Should we ask the web-sig to choose a person or two and then once we have the PEPs we designate PEP dictators? Either way we should probably set a deadline to get the PEPs in else the SIG might argue too long where to go look at paint samples. From jnoller at gmail.com Wed Sep 15 19:09:58 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 15 Sep 2010 13:09:58 -0400 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> Message-ID: On Wed, Sep 15, 2010 at 12:55 PM, Brett Cannon wrote: > On Wed, Sep 15, 2010 at 09:35, Jesse Noller wrote: >> On Wed, Sep 15, 2010 at 12:22 PM, Brett Cannon wrote: >> ...snip... >>>> The one area I have concerns about is the state of WSGI and other >>>> web-oriented modules. These issues have been brought up by Armin and >>>> others, but given a lack of a clear path forward (bugs, peps, etc), I >>>> don't think it's fair to use it as a measurement of overall quality. >>> >>> The whole WSGI situation is not going to get cleared up (from my >>> understanding) until someone flat-out declares a winner in the whole >>> str/bytes argument that keeps coming up. I think it might be time to >>> have a PEP or two on this and use our new PEP dictator procedure to >>> settle this so it stops dragging on (unless it has been miraculously >>> settled and I am just unaware of it). >>> >> >> Yup, and I spoke with some people with horses in that race at >> Djangocon. The important thing is that the PEP(s) and suggestion come >> from the people with the most experience in that domain. > > Yes. They have to be people who are not only stakeholders but people > who actively use and develop large applications using WSGI. > >> That's why I >> said we (in the "committer" sense) need a clear path of things we need >> to change or fix - without it we're just stabbing in the dark. > > So, who do we get to write the PEP(s)? Should we ask the web-sig to > choose a person or two and then once we have the PEPs we designate PEP > dictators? Either way we should probably set a deadline to get the > PEPs in else the SIG might argue too long where to go look at paint > samples. > At Djangocon, I was told this is being discussed amongst people on web-sig, and I encouraged a few people to get involved. I don't think this is something we can set a deadline for (and I don't know that it's *our* place), especially given a lack of people to actually write the code in some cases. In at least one case, I've encouraged them to contact the PSF with a proposal in case funding is needed (such as your own, or Jean-Paul's work). Fundamentally; I would gladly hold up 3.2 (just my opinion) for the needed fixes to the standard lib I've heard discussed (once we have bugs and/or patches) but that requires the decisions to be made, and I don't think the people here are the ones to make the decisions - so we can only state the release date of 3.2 and the subsequent releases and let the people who know infinitely more about the nuances then us decide on it. jesse From jacob at jacobian.org Wed Sep 15 19:36:38 2010 From: jacob at jacobian.org (Jacob Kaplan-Moss) Date: Wed, 15 Sep 2010 12:36:38 -0500 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> Message-ID: On Wed, Sep 15, 2010 at 12:09 PM, Jesse Noller wrote: > Fundamentally; I would gladly hold up 3.2 (just my opinion) for the > needed fixes to the standard lib [...] I think I should share a little anecdote at this point: Earlier in the year I worked for a while on Django/Py3. It's actually not that hard of a task (because I'm building on the work by MvL and some of Greg Wilson's students!) and I quickly got a simple app working locally. So the next step for me was to see about putting the app into production... and that's where the wheels fell off. So that's where I stopped. As far as I'm concerned, I'm not willing to expend the effort to get Django ported if I can't put it into production. Most of us working on Django are going to feel the same way, I suspect. Further, I can say with some confidence that until the WSGI issue is sorted the Python web world isn't going to have much enthusiasm for Python 3. I'm trying not to sound overly negative here -- really, I can't *WAIT* to be able to switch to Py3! But until I have a bunch of interoperable, robust WSGI servers like I do on Python 2 -- modwsgi, uwsgi, cherrypy, gunicorn, ... -- Python 3 is going to remain a pipe dream. Jacob From devin.c.cook at gmail.com Wed Sep 15 19:48:36 2010 From: devin.c.cook at gmail.com (Devin Cook) Date: Wed, 15 Sep 2010 12:48:36 -0500 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <66528.1284567245@parc.com> Message-ID: On Wed, Sep 15, 2010 at 11:37 AM, Jesse Noller wrote: > You need people with the time and willingness to download, install and > run production code on the releases. This might be getting off-topic, but maybe not as many people as you think. How many projects in pypi provide unittests? That's at least more tests to add to the ones already being run in the stdlib. -Devin From debatem1 at gmail.com Wed Sep 15 19:57:58 2010 From: debatem1 at gmail.com (geremy condra) Date: Wed, 15 Sep 2010 10:57:58 -0700 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <66528.1284567245@parc.com> Message-ID: On Wed, Sep 15, 2010 at 10:48 AM, Devin Cook wrote: > On Wed, Sep 15, 2010 at 11:37 AM, Jesse Noller wrote: >> You need people with the time and willingness to download, install and >> run production code on the releases. > > This might be getting off-topic, but maybe not as many people as you > think. How many projects in pypi provide unittests? That's at least > more tests to add to the ones already being run in the stdlib. > > -Devin Make it automatic and I can probably get some machines put on the job. I don't have a lot of time to devote, though. Geremy Condra From jnoller at gmail.com Wed Sep 15 20:02:40 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 15 Sep 2010 14:02:40 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <66528.1284567245@parc.com> Message-ID: On Wed, Sep 15, 2010 at 1:48 PM, Devin Cook wrote: > On Wed, Sep 15, 2010 at 11:37 AM, Jesse Noller wrote: >> You need people with the time and willingness to download, install and >> run production code on the releases. > > This might be getting off-topic, but maybe not as many people as you > think. How many projects in pypi provide unittests? That's at least > more tests to add to the ones already being run in the stdlib. > > -Devin > And who do you get to maintain all the new tests and buildbots you spawn from running hundreds of community projects unittests? How do you know those tests are real, and actually work? You quickly outstrip the ability of the core team to stay on top of it, and you run into issues in the past akin to the community build bots project. jesse From brett at python.org Wed Sep 15 20:02:28 2010 From: brett at python.org (Brett Cannon) Date: Wed, 15 Sep 2010 11:02:28 -0700 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> Message-ID: On Wed, Sep 15, 2010 at 10:36, Jacob Kaplan-Moss wrote: > On Wed, Sep 15, 2010 at 12:09 PM, Jesse Noller wrote: >> Fundamentally; I would gladly hold up 3.2 (just my opinion) for the >> needed fixes to the standard lib [...] > > I think I should share a little anecdote at this point: > > Earlier in the year I worked for a while on Django/Py3. It's actually > not that hard of a task (because I'm building on the work by MvL and > some of Greg Wilson's students!) and I quickly got a simple app > working locally. So the next step for me was to see about putting the > app into production... and that's where the wheels fell off. > > So that's where I stopped. As far as I'm concerned, I'm not willing to > expend the effort to get Django ported if I can't put it into > production. Most of us working on Django are going to feel the same > way, I suspect. > > Further, I can say with some confidence that until the WSGI issue is > sorted the Python web world isn't going to have much enthusiasm for > Python 3. > > I'm trying not to sound overly negative here -- really, I can't *WAIT* > to be able to switch to Py3! But until I have a bunch of > interoperable, robust WSGI servers like I do on Python 2 -- modwsgi, > uwsgi, cherrypy, gunicorn, ... -- Python 3 is going to remain a pipe > dream. Which is why I would like to see this settled *now* rather than later. It's Georg's call, but I'm also fine with holding up Python 3.2 *if* we set a goal date to get this settled. If we release 3.2 without these fixes we won't have a chance for wsgiref to get updated until roughly June 2012 for Python 3.3 which will be 3.5 years since Python 3.0 was released. The Python web development community is a big (and friendly) part of the overall Python community. I think they deserve to have us do what we can as the harbingers of the language (and by extension, the technical aspect of the community as what we decide the community takes queues from) to solve this issue to allow forward movement towards using Python 3. From guido at python.org Wed Sep 15 20:11:14 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 15 Sep 2010 11:11:14 -0700 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> Message-ID: Given that wsgiref is in the stdlib, I think we should hold up the 3.2 release (and even the first beta) until this is resolved, unless we can convince ourselves that it's okay to delete wsgiref from the stdlib (which sounds unlikely but may not be any more incompatible than making it work properly :-). I want to emphasize that I am *not* a stakeholder so my preference for bytes or Unicode shouldn't matter; that said, given WSGI's traditional emphasis on using the lowest-level, vanilla standard datatypes (e.g. you can't even subclass dict let alone provide another kind of mapping -- it has to be a real dict) it makes sense to me that the values should be bytes, os.environ notwithstanding. The keys probably could be Unicode (HTTP headers are required to use only 7-bit ASCII characters anyways right?). But I'd be happy to be shown the error of my ways (or given a link showing prior discussion of the matter -- preferably with a conclusion :-). --Guido On Wed, Sep 15, 2010 at 10:36 AM, Jacob Kaplan-Moss wrote: > On Wed, Sep 15, 2010 at 12:09 PM, Jesse Noller wrote: >> Fundamentally; I would gladly hold up 3.2 (just my opinion) for the >> needed fixes to the standard lib [...] > > I think I should share a little anecdote at this point: > > Earlier in the year I worked for a while on Django/Py3. It's actually > not that hard of a task (because I'm building on the work by MvL and > some of Greg Wilson's students!) and I quickly got a simple app > working locally. So the next step for me was to see about putting the > app into production... and that's where the wheels fell off. > > So that's where I stopped. As far as I'm concerned, I'm not willing to > expend the effort to get Django ported if I can't put it into > production. Most of us working on Django are going to feel the same > way, I suspect. > > Further, I can say with some confidence that until the WSGI issue is > sorted the Python web world isn't going to have much enthusiasm for > Python 3. > > I'm trying not to sound overly negative here -- really, I can't *WAIT* > to be able to switch to Py3! But until I have a bunch of > interoperable, robust WSGI servers like I do on Python 2 -- modwsgi, > uwsgi, cherrypy, gunicorn, ... -- Python 3 is going to remain a pipe > dream. > > Jacob > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From barry at python.org Wed Sep 15 20:29:32 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 15 Sep 2010 14:29:32 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <66528.1284567245@parc.com> Message-ID: <20100915142932.191c153d@mission> On Sep 15, 2010, at 02:02 PM, Jesse Noller wrote: >And who do you get to maintain all the new tests and buildbots you >spawn from running hundreds of community projects unittests? How do >you know those tests are real, and actually work? You quickly outstrip >the ability of the core team to stay on top of it, and you run into >issues in the past akin to the community build bots project. This was something we were thinking about as part of the snakebite project. I don't know if that's still alive though. I would love to have *some* kind of health/QA metric show up next to packages on the Cheeseshop for example. It's also something I've been mulling over as part of QA for the large number of Python packages available in Debian/Ubuntu. This was in the context of trying to judge the health of those packages for Python 2.7. At our last UDS I spoke to a number of people and thought that it actually wouldn't be infeasible to set up some kind of automated Hudson-based farm to run test suites for all the packages we make available. I think all the basic pieces are there, it's mostly a matter of finding the time to put it all together. I of course did not find the time :/ so it hasn't happened yet. Of course, the set of platforms, Python versions, and packages we care about is much narrower than the full Python community, but it would still be an interesting exercise. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From debatem1 at gmail.com Wed Sep 15 20:30:43 2010 From: debatem1 at gmail.com (geremy condra) Date: Wed, 15 Sep 2010 11:30:43 -0700 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <66528.1284567245@parc.com> Message-ID: On Wed, Sep 15, 2010 at 11:02 AM, Jesse Noller wrote: > On Wed, Sep 15, 2010 at 1:48 PM, Devin Cook wrote: >> On Wed, Sep 15, 2010 at 11:37 AM, Jesse Noller wrote: >>> You need people with the time and willingness to download, install and >>> run production code on the releases. >> >> This might be getting off-topic, but maybe not as many people as you >> think. How many projects in pypi provide unittests? That's at least >> more tests to add to the ones already being run in the stdlib. >> >> -Devin >> > > And who do you get to maintain all the new tests and buildbots you > spawn from running hundreds of community projects unittests? How do > you know those tests are real, and actually work? You quickly outstrip > the ability of the core team to stay on top of it, and you run into > issues in the past akin to the community build bots project. > > jesse Obviously information overload can be an issue, but ISTM that having projects opt-in for testing with new builds would give the core team some potentially useful feedback in the event that they needed it. Since pypi also has download statistics, it might help to figure out relative importance of different problems. Having said that, I'm only volunteering firepower, so don't interpret this as trying to push this on you guys if you don't want it. Geremy Condra From barry at python.org Wed Sep 15 20:32:20 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 15 Sep 2010 14:32:20 -0400 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> Message-ID: <20100915143220.5f09618e@mission> On Sep 15, 2010, at 11:11 AM, Guido van Rossum wrote: >Given that wsgiref is in the stdlib, I think we should hold up the 3.2 >release (and even the first beta) until this is resolved, unless we >can convince ourselves that it's okay to delete wsgiref from the >stdlib (which sounds unlikely but may not be any more incompatible >than making it work properly :-). I would much prefer holding up the release to fix wsgiref rather than remove it. I think it's an important module worthy of being in the stdlib. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From jnoller at gmail.com Wed Sep 15 20:39:43 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 15 Sep 2010 14:39:43 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: <20100915142932.191c153d@mission> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <66528.1284567245@parc.com> <20100915142932.191c153d@mission> Message-ID: On Wed, Sep 15, 2010 at 2:29 PM, Barry Warsaw wrote: > On Sep 15, 2010, at 02:02 PM, Jesse Noller wrote: > >>And who do you get to maintain all the new tests and buildbots you >>spawn from running hundreds of community projects unittests? How do >>you know those tests are real, and actually work? You quickly outstrip >>the ability of the core team to stay on top of it, and you run into >>issues in the past akin to the community build bots project. > > This was something we were thinking about as part of the snakebite project. ?I > don't know if that's still alive though. ?I would love to have *some* kind of > health/QA metric show up next to packages on the Cheeseshop for example. My GSOC student this past year worked on a testing backend for PyPI, I think there's a want and strong desire for this, but a lack of person-resources. Also, the onus has to be on the maintainers of the package; not on core. > It's also something I've been mulling over as part of QA for the large number > of Python packages available in Debian/Ubuntu. ?This was in the context of > trying to judge the health of those packages for Python 2.7. > > At our last UDS I spoke to a number of people and thought that it actually > wouldn't be infeasible to set up some kind of automated Hudson-based farm to > run test suites for all the packages we make available. ?I think all the basic > pieces are there, it's mostly a matter of finding the time to put it all > together. ?I of course did not find the time :/ so it hasn't happened yet. Yeah; we have a plethora of options - hudson, pony-build, buildbot, pyti (http://bitbucket.org/mouad/pypi-testing-infrastructure-pyti) and many more. We also have the isolation tools (such as virtualenv) and awesome little utilities like tox (http://pypi.python.org/pypi/tox) for doing all of this now. Manpower and time prohibit it > Of course, the set of platforms, Python versions, and packages we care about > is much narrower than the full Python community, but it would still be an > interesting exercise. If we had an existing back end for this - say "python 2.6, 2.7, 3.1" and package maintainers could use that infrastructure, on pypi, to run their tests and we could see Green Dots (pass) for those packages, I think it's a short jump from that, to having a "dev" section where we can take tests that pass 3.1 and run it on a new experimental interpreter. If we know in advance that it passes on the old interpreter, we can at least assume that we may have broken something. From raymond.hettinger at gmail.com Wed Sep 15 20:43:15 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 15 Sep 2010 11:43:15 -0700 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> Message-ID: <6C2EA59A-E14D-4484-9716-83DA22D78DDC@gmail.com> On Sep 15, 2010, at 7:50 AM, Jesse Noller wrote: > > +0.5 > > The one area I have concerns about is the state of WSGI and other > web-oriented modules. These issues have been brought up by Armin and > others, but given a lack of a clear path forward (bugs, peps, etc), I > don't think it's fair to use it as a measurement of overall quality. Any chance you're going to have time to do work on multiprocessing? There are a huge number of bugs reports open for that module. Raymond From jnoller at gmail.com Wed Sep 15 20:47:06 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 15 Sep 2010 14:47:06 -0400 Subject: [Python-Dev] 3.x as the official release In-Reply-To: <6C2EA59A-E14D-4484-9716-83DA22D78DDC@gmail.com> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <6C2EA59A-E14D-4484-9716-83DA22D78DDC@gmail.com> Message-ID: On Wed, Sep 15, 2010 at 2:43 PM, Raymond Hettinger wrote: > > On Sep 15, 2010, at 7:50 AM, Jesse Noller wrote: >> >> +0.5 >> >> The one area I have concerns about is the state of WSGI and other >> web-oriented modules. These issues have been brought up by Armin and >> others, but given a lack of a clear path forward (bugs, peps, etc), I >> don't think it's fair to use it as a measurement of overall quality. > > Any chance you're going to have time to do work on multiprocessing? > There are a huge number of bugs reports open for that module. > Trying to get that time; and I've recently brought on Ask Solem to help me there, I concur that the current situation is sub optimal. From steve at holdenweb.com Wed Sep 15 20:58:44 2010 From: steve at holdenweb.com (Steve Holden) Date: Wed, 15 Sep 2010 14:58:44 -0400 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <6C2EA59A-E14D-4484-9716-83DA22D78DDC@gmail.com> Message-ID: On 9/15/2010 2:47 PM, Jesse Noller wrote: > On Wed, Sep 15, 2010 at 2:43 PM, Raymond Hettinger > wrote: >> >> On Sep 15, 2010, at 7:50 AM, Jesse Noller wrote: >>> >>> +0.5 >>> >>> The one area I have concerns about is the state of WSGI and other >>> web-oriented modules. These issues have been brought up by Armin and >>> others, but given a lack of a clear path forward (bugs, peps, etc), I >>> don't think it's fair to use it as a measurement of overall quality. >> >> Any chance you're going to have time to do work on multiprocessing? >> There are a huge number of bugs reports open for that module. >> > > Trying to get that time; and I've recently brought on Ask Solem to > help me there, I concur that the current situation is sub optimal. Great that the "bus number" for multiprocessing has gone up by one! regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From barry at python.org Wed Sep 15 21:12:19 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 15 Sep 2010 15:12:19 -0400 Subject: [Python-Dev] Rework nntlib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <66528.1284567245@parc.com> <20100915142932.191c153d@mission> Message-ID: <20100915151219.0d12b30d@mission> On Sep 15, 2010, at 02:39 PM, Jesse Noller wrote: >Yeah; we have a plethora of options - hudson, pony-build, buildbot, >pyti (http://bitbucket.org/mouad/pypi-testing-infrastructure-pyti) and >many more. We also have the isolation tools (such as virtualenv) and >awesome little utilities like tox (http://pypi.python.org/pypi/tox) >for doing all of this now. > >Manpower and time prohibit it This might make for a fun sprint at Pycon 2011. It's closer than you think. (Well, maybe not closer than Jesse thinks :). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From jnoller at gmail.com Wed Sep 15 21:15:15 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 15 Sep 2010 15:15:15 -0400 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <6C2EA59A-E14D-4484-9716-83DA22D78DDC@gmail.com> Message-ID: On Wed, Sep 15, 2010 at 2:58 PM, Steve Holden wrote: > On 9/15/2010 2:47 PM, Jesse Noller wrote: >> On Wed, Sep 15, 2010 at 2:43 PM, Raymond Hettinger >> wrote: >>> >>> On Sep 15, 2010, at 7:50 AM, Jesse Noller wrote: >>>> >>>> +0.5 >>>> >>>> The one area I have concerns about is the state of WSGI and other >>>> web-oriented modules. These issues have been brought up by Armin and >>>> others, but given a lack of a clear path forward (bugs, peps, etc), I >>>> don't think it's fair to use it as a measurement of overall quality. >>> >>> Any chance you're going to have time to do work on multiprocessing? >>> There are a huge number of bugs reports open for that module. >>> >> >> Trying to get that time; and I've recently brought on Ask Solem to >> help me there, I concur that the current situation is sub optimal. > > Great that the "bus number" for multiprocessing has gone up by one! > > regards > ?Steve No one is happier about this then me. From pje at telecommunity.com Wed Sep 15 21:18:14 2010 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 15 Sep 2010 15:18:14 -0400 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> Message-ID: <20100915191812.5D0663A4116@sparrow.telecommunity.com> At 11:11 AM 9/15/2010 -0700, Guido van Rossum wrote: >Given that wsgiref is in the stdlib, I think we should hold up the 3.2 >release (and even the first beta) until this is resolved, unless we >can convince ourselves that it's okay to delete wsgiref from the >stdlib (which sounds unlikely but may not be any more incompatible >than making it work properly :-). FWIW, I'd be fine with that option. >I want to emphasize that I am *not* a stakeholder so my preference for >bytes or Unicode shouldn't matter; that said, given WSGI's traditional >emphasis on using the lowest-level, vanilla standard datatypes (e.g. >you can't even subclass dict let alone provide another kind of mapping >-- it has to be a real dict) it makes sense to me that the values >should be bytes, os.environ notwithstanding. The keys probably could >be Unicode (HTTP headers are required to use only 7-bit ASCII >characters anyways right?). But I'd be happy to be shown the error of >my ways (or given a link showing prior discussion of the matter -- >preferably with a conclusion :-). There isn't a conclusion yet, but the proposals under discussion are summarized here: http://www.wsgi.org/wsgi/Python_3#Proposals The primary points of consensus are bytes for wsgi.input, and native strings (i.e. Unicode on Python 3) for environment keys. If I were to offer a suggestion to a PEP author or dictator wanting to get something out ASAP, it would probably be to create a compromise between the "flat" model (my personal favorite) and the mod_wsgi model, as an addendum to PEP 333. Specifically: * leave start_response/write in play (ala mod_wsgi) * use the required types from the "flat" proposal (i.e. status, headers, and output stream MUST be bytes) * add a decorator to wsgiref that supports using native strings as output instead of bytes, for ease-of-porting (combine mod_wsgi's ease-of-porting w/"flat"'s simple verifiability) This would probably allow us to get by with the least changes to existing code, the stdlib, the standard itself, and wsgiref. (wsgiref itself would still need a thorough code review, especially wsgiref.validate, but it'd be unlikely to change much.) From g.brandl at gmx.net Wed Sep 15 21:21:40 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 15 Sep 2010 21:21:40 +0200 Subject: [Python-Dev] 3.x as the official release In-Reply-To: <20100915143220.5f09618e@mission> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100915143220.5f09618e@mission> Message-ID: Am 15.09.2010 20:32, schrieb Barry Warsaw: > On Sep 15, 2010, at 11:11 AM, Guido van Rossum wrote: > >>Given that wsgiref is in the stdlib, I think we should hold up the 3.2 >>release (and even the first beta) until this is resolved, unless we >>can convince ourselves that it's okay to delete wsgiref from the >>stdlib (which sounds unlikely but may not be any more incompatible >>than making it work properly :-). I also think this must be resolved before 3.2 can be released, and especially in the case of fixing it (vs removing), it should happen before beta 1. > I would much prefer holding up the release to fix wsgiref rather than remove > it. I think it's an important module worthy of being in the stdlib. Really? I'd like to hear from some of its users first. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From chrism at plope.com Wed Sep 15 21:03:34 2010 From: chrism at plope.com (Chris McDonough) Date: Wed, 15 Sep 2010 15:03:34 -0400 Subject: [Python-Dev] 3.x as the official release In-Reply-To: <1284577191.14651.25.camel@thinko> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <1284577191.14651.25.camel@thinko> Message-ID: <1284577414.14651.27.camel@thinko> On Wed, 2010-09-15 at 14:59 -0400, Chris McDonough wrote: > For reference, I have developed a spec and an (untested) reference > implementation of a WSGI successor I've given the name "Web3". Ian is > not hot on this spec (he prefers native strings as environ keys). That should read "as environ values" sorry. - C From chrism at plope.com Wed Sep 15 20:59:51 2010 From: chrism at plope.com (Chris McDonough) Date: Wed, 15 Sep 2010 14:59:51 -0400 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> Message-ID: <1284577191.14651.25.camel@thinko> For reference, I have developed a spec and an (untested) reference implementation of a WSGI successor I've given the name "Web3". Ian is not hot on this spec (he prefers native strings as environ keys). I'm definitely not going to write a WebOb analogue, so I'd more or less given up trying to promote it. But it's here for consideration. Given that nobody else has written this all out in spec form, it may be useful: http://github.com/mcdonc/web3 - C On Wed, 2010-09-15 at 11:11 -0700, Guido van Rossum wrote: > Given that wsgiref is in the stdlib, I think we should hold up the 3.2 > release (and even the first beta) until this is resolved, unless we > can convince ourselves that it's okay to delete wsgiref from the > stdlib (which sounds unlikely but may not be any more incompatible > than making it work properly :-). > > I want to emphasize that I am *not* a stakeholder so my preference for > bytes or Unicode shouldn't matter; that said, given WSGI's traditional > emphasis on using the lowest-level, vanilla standard datatypes (e.g. > you can't even subclass dict let alone provide another kind of mapping > -- it has to be a real dict) it makes sense to me that the values > should be bytes, os.environ notwithstanding. The keys probably could > be Unicode (HTTP headers are required to use only 7-bit ASCII > characters anyways right?). But I'd be happy to be shown the error of > my ways (or given a link showing prior discussion of the matter -- > preferably with a conclusion :-). > > --Guido > > On Wed, Sep 15, 2010 at 10:36 AM, Jacob Kaplan-Moss wrote: > > On Wed, Sep 15, 2010 at 12:09 PM, Jesse Noller wrote: > >> Fundamentally; I would gladly hold up 3.2 (just my opinion) for the > >> needed fixes to the standard lib [...] > > > > I think I should share a little anecdote at this point: > > > > Earlier in the year I worked for a while on Django/Py3. It's actually > > not that hard of a task (because I'm building on the work by MvL and > > some of Greg Wilson's students!) and I quickly got a simple app > > working locally. So the next step for me was to see about putting the > > app into production... and that's where the wheels fell off. > > > > So that's where I stopped. As far as I'm concerned, I'm not willing to > > expend the effort to get Django ported if I can't put it into > > production. Most of us working on Django are going to feel the same > > way, I suspect. > > > > Further, I can say with some confidence that until the WSGI issue is > > sorted the Python web world isn't going to have much enthusiasm for > > Python 3. > > > > I'm trying not to sound overly negative here -- really, I can't *WAIT* > > to be able to switch to Py3! But until I have a bunch of > > interoperable, robust WSGI servers like I do on Python 2 -- modwsgi, > > uwsgi, cherrypy, gunicorn, ... -- Python 3 is going to remain a pipe > > dream. > > > > Jacob > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > From barry at python.org Wed Sep 15 21:38:13 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 15 Sep 2010 15:38:13 -0400 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100915143220.5f09618e@mission> Message-ID: <20100915153813.7a21f1d4@mission> On Sep 15, 2010, at 09:21 PM, Georg Brandl wrote: >Am 15.09.2010 20:32, schrieb Barry Warsaw: >> I would much prefer holding up the release to fix wsgiref rather >> than remove it. I think it's an important module worthy of being in >> the stdlib. > >Really? I'd like to hear from some of its users first. I've used it. When combined with something like restish it's about 20 lines of code to publish a database-backed REST service. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ziade.tarek at gmail.com Wed Sep 15 21:45:26 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 15 Sep 2010 21:45:26 +0200 Subject: [Python-Dev] 3.x as the official release In-Reply-To: <20100915191812.5D0663A4116@sparrow.telecommunity.com> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100915191812.5D0663A4116@sparrow.telecommunity.com> Message-ID: On Wed, Sep 15, 2010 at 9:18 PM, P.J. Eby wrote: > At 11:11 AM 9/15/2010 -0700, Guido van Rossum wrote: >> >> Given that wsgiref is in the stdlib, I think we should hold up the 3.2 >> release (and even the first beta) until this is resolved, unless we >> can convince ourselves that it's okay to delete wsgiref from the >> stdlib (which sounds unlikely but may not be any more incompatible >> than making it work properly :-). > > FWIW, I'd be fine with that option. Could we remove in any case the wsgiref.egg-info file ? Since we've been working on a new format for that (PEP 376), that should be starting to get used in the coming years, it'll be a bit of a non-sense to have that metadata file in the sdtlib shipped with 3,2 I am not really sure what was the intent to have it there in the first place though. Regards Tarek -- Tarek Ziad? | http://ziade.org From brett at python.org Wed Sep 15 22:32:27 2010 From: brett at python.org (Brett Cannon) Date: Wed, 15 Sep 2010 13:32:27 -0700 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100915191812.5D0663A4116@sparrow.telecommunity.com> Message-ID: On Wed, Sep 15, 2010 at 12:45, Tarek Ziad? wrote: > On Wed, Sep 15, 2010 at 9:18 PM, P.J. Eby wrote: >> At 11:11 AM 9/15/2010 -0700, Guido van Rossum wrote: >>> >>> Given that wsgiref is in the stdlib, I think we should hold up the 3.2 >>> release (and even the first beta) until this is resolved, unless we >>> can convince ourselves that it's okay to delete wsgiref from the >>> stdlib (which sounds unlikely but may not be any more incompatible >>> than making it work properly :-). >> >> FWIW, I'd be fine with that option. I'm also fine with removing the module if that is what it takes to move all of this forward. That would remove Python 3.2's release cycle from this picture while also allowing a finalization of WSGI specs to continue in parallel. But this probably shouldn't happen as Geremy's Cheeseshop #s put wsgiref squarely in the middle of most used stdlib modules with 421 projects. So there are users of the module. > > Could we remove in any case the wsgiref.egg-info file ? Since we've > been working on a new format for that (PEP 376), that should be > starting to get used in the coming years, it'll be a bit of a > non-sense to have that metadata file in the sdtlib shipped with 3,2 > > I am not really sure what was the intent to have it there in the first > place though. I say go ahead and remove it. From brett at python.org Wed Sep 15 22:46:06 2010 From: brett at python.org (Brett Cannon) Date: Wed, 15 Sep 2010 13:46:06 -0700 Subject: [Python-Dev] how to decide on a Python 3 design for wsgiref Message-ID: Both the RM and BDFL agree that Python 3.2b1 should be held up until we settle this wsgi matter. That makes it a question of how to settle it. Thinking out loud here to keep this discussion focused, I say we give a deadline for PEPs to be submitted by October 15th. We then choose two PEP dictators to make a call by November 1, get wsgiref cleaned up ASAP, and get Python 3.2b1 out the door immediately thereafter. If web-sig manages to come to an agreement as a whole before then we can skip the PEPs, but if they have not managed to do this already then it probably is not going to suddenly happen now under threat of python-dev making the call for them by blessing a new wsgiref implementation (happy to be proven wrong, though). From jnoller at gmail.com Wed Sep 15 22:56:22 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 15 Sep 2010 16:56:22 -0400 Subject: [Python-Dev] how to decide on a Python 3 design for wsgiref In-Reply-To: References: Message-ID: On Wed, Sep 15, 2010 at 4:46 PM, Brett Cannon wrote: > Both the RM and BDFL agree that Python 3.2b1 should be held up until > we settle this wsgi matter. That makes it a question of how to settle > it. > > Thinking out loud here to keep this discussion focused, I say we give > a deadline for PEPs to be submitted by October 15th. We then choose > two PEP dictators to make a call by November 1, get wsgiref cleaned up > ASAP, and get Python 3.2b1 out the door immediately thereafter. If > web-sig manages to come to an agreement as a whole before then we can > skip the PEPs, but if they have not managed to do this already then it > probably is not going to suddenly happen now under threat of > python-dev making the call for them by blessing a new wsgiref > implementation (happy to be proven wrong, though). There's not just wsgiref; there's the possibility that other core and standard library changes may be needed. For example, the discussion from back in june: http://mail.python.org/pipermail/python-dev/2010-June/100733.html (that thread goes on for awhile) I'd personally like to see a plan which provides a path for both wsgiref, WSGI itself in python3 and the related changes to core which might be predicated, that way we can have some level of confidence we're not making the same misstep we made in the first place, making it so painful for the web frameworks/gateways/etc. jesse From merwok at netwok.org Wed Sep 15 23:12:11 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Wed, 15 Sep 2010 23:12:11 +0200 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100915191812.5D0663A4116@sparrow.telecommunity.com> Message-ID: <4C9136AB.8030803@netwok.org> Le 15/09/2010 21:45, Tarek Ziad? a ?crit : > Could we remove in any case the wsgiref.egg-info file ? Since we've > been working on a new format for that (PEP 376), that should be > starting to get used in the coming years, it'll be a bit of a > non-sense to have that metadata file in the sdtlib shipped with 3,2 +1 on removing wsgiref.egg-info in the same version that ships with distutils2, or with 3.2 (in case they?re not the same version). > I am not really sure what was the intent to have it there in the first > place though. Unless I remember wrong, the intent was not to break code that used pkg_resources.require('wsgiref'). Regards From merwok at netwok.org Wed Sep 15 23:19:29 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Wed, 15 Sep 2010 23:19:29 +0200 Subject: [Python-Dev] 3.x as the official release In-Reply-To: <20100915191812.5D0663A4116@sparrow.telecommunity.com> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100915191812.5D0663A4116@sparrow.telecommunity.com> Message-ID: <4C913861.2010607@netwok.org> > * add a decorator to wsgiref that supports using native strings as > output instead of bytes, for ease-of-porting (combine mod_wsgi's > ease-of-porting w/"flat"'s simple verifiability) Ah, thanks, I?ve been reading web-sig and was totally at a loss to understand what a ?native string? was. Now I get it?s a character string / string / str object / former unicode object. FWIW, I?m glad to see concern for web-sig in python-dev, and messages from Web people in python-dev. Some threads going ?they broke strings, we can?t use the stdlib? were a bit alarming to me. Regards From dirkjan at ochtman.nl Wed Sep 15 23:41:59 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 15 Sep 2010 23:41:59 +0200 Subject: [Python-Dev] how to decide on a Python 3 design for wsgiref In-Reply-To: References: Message-ID: On Wed, Sep 15, 2010 at 22:46, Brett Cannon wrote: > Both the RM and BDFL agree that Python 3.2b1 should be held up until > we settle this wsgi matter. That makes it a question of how to settle > it. I think that's a very good goal. Given all the times it's come up on the Web-SIG list (I even tried my hand at it once) and a consensus has failed to form, it seems pretty important that we force some kind of breakthrough. One problem: Graham Dumpleton, who is certainly one of the more knowledgeable people on this subject, will be on vacation "from late September". It looks like he's gone at least until half October. It would be a pity if we force a decision without his input. Cheers, Dirkjan From brett at python.org Wed Sep 15 23:49:01 2010 From: brett at python.org (Brett Cannon) Date: Wed, 15 Sep 2010 14:49:01 -0700 Subject: [Python-Dev] how to decide on a Python 3 design for wsgiref In-Reply-To: References: Message-ID: On Wed, Sep 15, 2010 at 14:41, Dirkjan Ochtman wrote: > On Wed, Sep 15, 2010 at 22:46, Brett Cannon wrote: >> Both the RM and BDFL agree that Python 3.2b1 should be held up until >> we settle this wsgi matter. That makes it a question of how to settle >> it. > > I think that's a very good goal. Given all the times it's come up on > the Web-SIG list (I even tried my hand at it once) and a consensus has > failed to form, it seems pretty important that we force some kind of > breakthrough. > > One problem: Graham Dumpleton, who is certainly one of the more > knowledgeable people on this subject, will be on vacation "from late > September". It looks like he's gone at least until half October. It > would be a pity if we force a decision without his input. It can get shifted to November 1 if needed. I just don't want it to drag on forever as it will keep postponing Python 3.2. From dirkjan at ochtman.nl Wed Sep 15 23:50:58 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 15 Sep 2010 23:50:58 +0200 Subject: [Python-Dev] 3.x as the official release In-Reply-To: <20100915191812.5D0663A4116@sparrow.telecommunity.com> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100915191812.5D0663A4116@sparrow.telecommunity.com> Message-ID: On Wed, Sep 15, 2010 at 21:18, P.J. Eby wrote: > If I were to offer a suggestion to a PEP author or dictator wanting to get > something out ASAP, it would probably be to create a compromise between the > "flat" model (my personal favorite) and the mod_wsgi model, as an addendum > to PEP 333. ?Specifically: > > * leave start_response/write in play (ala mod_wsgi) The alternative is returning a three-tuple status, headers, content-iterable, right? I would definitely prefer just returning a three-tuple instead of the crappy start_response callback that returns a write callable. It makes applications easier to write, and the unified model should also make server implemation easier. It also combines nicely with yield from in some cases. Cheers, Dirkjan From solipsis at pitrou.net Thu Sep 16 00:09:28 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 16 Sep 2010 00:09:28 +0200 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> Message-ID: <20100916000928.0ada59d1@pitrou.net> On Wed, 15 Sep 2010 11:11:14 -0700 Guido van Rossum wrote: > Given that wsgiref is in the stdlib, I think we should hold up the 3.2 > release (and even the first beta) until this is resolved, unless we > can convince ourselves that it's okay to delete wsgiref from the > stdlib (which sounds unlikely but may not be any more incompatible > than making it work properly :-). I'm not sure I agree with this. Considering that: - a PEP can be written at any time, and third-party implementations made compatible independently of our release schedule, and - a wsgi3ref package can be uploaded to the Cheese Shop as soon as the PEP is finalized, I don't see what we gain by holding up the 3.2 release. Some writing a Web application will need third-party modules anyway, so downloading wsgi3ref shouldn't be too painful. On the other hand, if 3.2 is delayed by a month or more, I might try to push for rapid acceptance of PEP 3151, in the hope that the implementation is ready before the first beta! Regards Antoine. From chrism at plope.com Thu Sep 16 00:16:23 2010 From: chrism at plope.com (Chris McDonough) Date: Wed, 15 Sep 2010 18:16:23 -0400 Subject: [Python-Dev] how to decide on a Python 3 design for wsgiref In-Reply-To: References: Message-ID: <1284588983.14651.33.camel@thinko> On Wed, 2010-09-15 at 13:46 -0700, Brett Cannon wrote: > Both the RM and BDFL agree that Python 3.2b1 should be held up until > we settle this wsgi matter. That makes it a question of how to settle > it. > > Thinking out loud here to keep this discussion focused, I say we give > a deadline for PEPs to be submitted by October 15th. We then choose > two PEP dictators to make a call by November 1, get wsgiref cleaned up > ASAP, and get Python 3.2b1 out the door immediately thereafter. If > web-sig manages to come to an agreement as a whole before then we can > skip the PEPs, but if they have not managed to do this already then it > probably is not going to suddenly happen now under threat of > python-dev making the call for them by blessing a new wsgiref > implementation (happy to be proven wrong, though). FWIW, I've submitted http://github.com/mcdonc/web3/blob/master/web3.rst as a PEP (by sending a mail to peps at python.org). - C From pje at telecommunity.com Thu Sep 16 00:31:02 2010 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 15 Sep 2010 18:31:02 -0400 Subject: [Python-Dev] 3.x as the official release In-Reply-To: <4C9136AB.8030803@netwok.org> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100915191812.5D0663A4116@sparrow.telecommunity.com> <4C9136AB.8030803@netwok.org> Message-ID: <20100915223101.662CB3A403D@sparrow.telecommunity.com> At 11:12 PM 9/15/2010 +0200, ??ric Araujo wrote: >Unless I remember wrong, the intent was not to break code that used >pkg_resources.require('wsgiref') More precisely, at the time it was done, setuptools was slated for inclusion in Python 2.5, and the idea was that when modules moved from PyPI to the stdlib, they would include the metadata so that projects requiring the module on an older version of Python would not need to use Python-version-dependent dependencies. So, for example, if a package was written on 2.4 using a requirement of wsgiref, then that code would run unchanged on 2.5 using the stdlib-supplied copy. In practice, this didn't work out in 2.x, and it's meaningless on 3.x where nothing has migrated yet from PyPI to stdlib AFAIK. ;-) From pje at telecommunity.com Thu Sep 16 00:35:30 2010 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 15 Sep 2010 18:35:30 -0400 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100915191812.5D0663A4116@sparrow.telecommunity.com> Message-ID: <20100915223529.370C83A403D@sparrow.telecommunity.com> At 11:50 PM 9/15/2010 +0200, Dirkjan Ochtman wrote: >On Wed, Sep 15, 2010 at 21:18, P.J. Eby wrote: > > If I were to offer a suggestion to a PEP author or dictator wanting to get > > something out ASAP, it would probably be to create a compromise between the > > "flat" model (my personal favorite) and the mod_wsgi model, as an addendum > > to PEP 333. ? Specifically: > > > > * leave start_response/write in play (ala mod_wsgi) > >The alternative is returning a three-tuple status, headers, >content-iterable, right? > >I would definitely prefer just returning a three-tuple instead of the >crappy start_response callback that returns a write callable. It makes >applications easier to write, and the unified model should also make >server implemation easier. It also combines nicely with yield from in >some cases. I would prefer it too (which is why the "flat" model is my favorite), but I think it would be easier to get a quick consensus for something that allows apps to be more mechanically ported from 2.x to 3.x. That's why I said, "offer a suggestion to ... get something out ASAP". ;-) From prologic at shortcircuit.net.au Thu Sep 16 00:44:05 2010 From: prologic at shortcircuit.net.au (James Mills) Date: Thu, 16 Sep 2010 08:44:05 +1000 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: <20100916000928.0ada59d1@pitrou.net> References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> Message-ID: On Thu, Sep 16, 2010 at 8:09 AM, Antoine Pitrou wrote: > I don't see what we gain by holding up the 3.2 release. ?Some writing a > Web application will need third-party modules anyway, so downloading > wsgi3ref shouldn't be too painful. I agree with you. Further, is wsgiref actually heavily used by web developers and or web framework developers at all ? I would tend to think that web developers might be more interested in using some of the larger more popular web frameworks such as: TurboGears, Django, Pylons, Cherrypy, etc. i don't think a Python 3.2 release should be held up because of wsgiref. cheers James -- -- James Mills -- -- "Problems are solved by method" From chrism at plope.com Thu Sep 16 01:06:10 2010 From: chrism at plope.com (Chris McDonough) Date: Wed, 15 Sep 2010 19:06:10 -0400 Subject: [Python-Dev] PEP 444 aka "Web3" (was Re: how to decide on a Python 3 design for wsgiref) In-Reply-To: <1284588983.14651.33.camel@thinko> References: <1284588983.14651.33.camel@thinko> Message-ID: <1284591970.14651.38.camel@thinko> This spec, a WSGI-like spec for Python2 and Python3 named Web3 was accepted today as PEP 444. http://python.org/dev/peps/pep-0444/ Comments and competing specs would be useful. - C On Wed, 2010-09-15 at 18:16 -0400, Chris McDonough wrote: > On Wed, 2010-09-15 at 13:46 -0700, Brett Cannon wrote: > > Both the RM and BDFL agree that Python 3.2b1 should be held up until > > we settle this wsgi matter. That makes it a question of how to settle > > it. > > > > Thinking out loud here to keep this discussion focused, I say we give > > a deadline for PEPs to be submitted by October 15th. We then choose > > two PEP dictators to make a call by November 1, get wsgiref cleaned up > > ASAP, and get Python 3.2b1 out the door immediately thereafter. If > > web-sig manages to come to an agreement as a whole before then we can > > skip the PEPs, but if they have not managed to do this already then it > > probably is not going to suddenly happen now under threat of > > python-dev making the call for them by blessing a new wsgiref > > implementation (happy to be proven wrong, though). > > FWIW, I've submitted http://github.com/mcdonc/web3/blob/master/web3.rst > as a PEP (by sending a mail to peps at python.org). > > - C > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40plope.com > From benjamin at python.org Thu Sep 16 01:17:21 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 15 Sep 2010 18:17:21 -0500 Subject: [Python-Dev] [Python-checkins] r84842 - peps/trunk/pep-0444.txt In-Reply-To: References: <20100915224038.95AECC5D5@mail.python.org> Message-ID: 2010/9/15 Brett Cannon : > Can I just ask why 444 since 392 was the last assigned Python 2 number? Likely because WSGI 1.0 is PEP 333. -- Regards, Benjamin From prologic at shortcircuit.net.au Thu Sep 16 01:22:04 2010 From: prologic at shortcircuit.net.au (James Mills) Date: Thu, 16 Sep 2010 09:22:04 +1000 Subject: [Python-Dev] PEP 444 aka "Web3" (was Re: how to decide on a Python 3 design for wsgiref) In-Reply-To: <1284591970.14651.38.camel@thinko> References: <1284588983.14651.33.camel@thinko> <1284591970.14651.38.camel@thinko> Message-ID: On Thu, Sep 16, 2010 at 9:06 AM, Chris McDonough wrote: > Comments and competing specs would be useful. Can I post comments here ? :) I have one initial comment (at first glance). I'm not sure I quite like the idea of changing the keys from wsgi to web3 for the simple reason that you will already have to port your application to python3 (bytes vs. str) anyway. Changing the keys is just more unnecessary work (although one could just use a search/replace). My 2c (or pence) cheers james -- -- James Mills -- -- "Problems are solved by method" From jnoller at gmail.com Thu Sep 16 01:31:02 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 15 Sep 2010 19:31:02 -0400 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> Message-ID: On Sep 15, 2010, at 6:44 PM, James Mills wrote: > On Thu, Sep 16, 2010 at 8:09 AM, Antoine Pitrou wrote: >> I don't see what we gain by holding up the 3.2 release. Some writing a >> Web application will need third-party modules anyway, so downloading >> wsgi3ref shouldn't be too painful. > > I agree with you. Further, is wsgiref actually heavily used by web developers > and or web framework developers at all ? I would tend to think that > web developers > might be more interested in using some of the larger more popular web frameworks > such as: TurboGears, Django, Pylons, Cherrypy, etc. > > i don't think a Python 3.2 release should be held up because of wsgiref. > > cheers > James > It's not just wsgiref. We need to talk with the web-sig participants to work out any other issues in addition to wsgiref which have been contentious (str vs byte methods, bytearrays, stdlib issues) or at least hear from the group that these issues are resolved. My goal (personally) is to make sure python 3.2 is perfectly good for use in web applications, and is therefore a much more interesting porting target for web projects/libraries and frameworks. If it was just wsgiref, it would be one thing, but based on conversations with members of web-sig in the past, it's not. I can't clearly communicate everything they've mentioned in the past for them. jesse From prologic at shortcircuit.net.au Thu Sep 16 01:35:03 2010 From: prologic at shortcircuit.net.au (James Mills) Date: Thu, 16 Sep 2010 09:35:03 +1000 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> Message-ID: On Thu, Sep 16, 2010 at 9:31 AM, Jesse Noller wrote: > My goal (personally) is to make sure python 3.2 is perfectly good for use in web applications, and is therefore a much more interesting porting target for web projects/libraries and frameworks. Python 3 is already quite well done and very usable by web frameworks (at least my tiny one). (It's just my opinion of course). I think the biggest problem is the uptake of Python 3 itself and the general attitude towards Python 3 as a whole. That's what's stopping me from focusing on and even maintaining a Python 3 branch for my project(s). cheers James -- -- James Mills -- -- "Problems are solved by method" From brett at python.org Thu Sep 16 01:15:33 2010 From: brett at python.org (Brett Cannon) Date: Wed, 15 Sep 2010 16:15:33 -0700 Subject: [Python-Dev] [Python-checkins] r84842 - peps/trunk/pep-0444.txt In-Reply-To: <20100915224038.95AECC5D5@mail.python.org> References: <20100915224038.95AECC5D5@mail.python.org> Message-ID: Can I just ask why 444 since 392 was the last assigned Python 2 number? On Wed, Sep 15, 2010 at 15:40, georg.brandl wrote: > Author: georg.brandl > Date: Thu Sep 16 00:40:38 2010 > New Revision: 84842 > > Log: > Add PEP 444, Python Web3 Interface. > > Added: > ? peps/trunk/pep-0444.txt ? (contents, props changed) > > Added: peps/trunk/pep-0444.txt > ============================================================================== > --- (empty file) > +++ peps/trunk/pep-0444.txt ? ? Thu Sep 16 00:40:38 2010 > @@ -0,0 +1,1570 @@ > +PEP: 444 > +Title: Python Web3 Interface > +Version: $Revision$ > +Last-Modified: $Date$ > +Author: Chris McDonough , > + ? ? ? ?Armin Ronacher > +Discussions-To: Python Web-SIG > +Status: Draft > +Type: Informational > +Content-Type: text/x-rst > +Created: 19-Jul-2010 > + > + > +Abstract > +======== > + > +This document specifies a proposed second-generation standard > +interface between web servers and Python web applications or > +frameworks. > + > + > +Rationale and Goals > +=================== > + > +This protocol and specification is influenced heavily by the Web > +Services Gateway Interface (WSGI) 1.0 standard described in PEP 333 > +[1]_ . ?The high-level rationale for having any standard that allows > +Python-based web servers and applications to interoperate is outlined > +in PEP 333. ?This document essentially uses PEP 333 as a template, and > +changes its wording in various places for the purpose of forming a > +different standard. > + > +Python currently boasts a wide variety of web application frameworks > +which use the WSGI 1.0 protocol. ?However, due to changes in the > +language, the WSGI 1.0 protocol is not compatible with Python 3. ?This > +specification describes a standardized WSGI-like protocol that lets > +Python 2.6, 2.7 and 3.1+ applications communicate with web servers. > +Web3 is clearly a WSGI derivative; it only uses a different name than > +"WSGI" in order to indicate that it is not in any way backwards > +compatible. > + > +Applications and servers which are written to this specification are > +meant to work properly under Python 2.6.X, Python 2.7.X and Python > +3.1+. ?Neither an application nor a server that implements the Web3 > +specification can be easily written which will work under Python 2 > +versions earlier than 2.6 nor Python 3 versions earlier than 3.1. > + > +.. note:: > + > + ? Whatever Python 3 version fixed http://bugs.python.org/issue4006 so > + ? ``os.environ['foo']`` returns surrogates (ala PEP 383) when the > + ? value of 'foo' cannot be decoded using the current locale instead > + ? of failing with a KeyError is the *true* minimum Python 3 version. > + ? In particular, however, Python 3.0 is not supported. > + > +.. note:: > + > + ? Python 2.6 is the first Python version that supported an alias for > + ? ``bytes`` and the ``b"foo"`` literal syntax. ?This is why it is the > + ? minimum version supported by Web3. > + > +Explicability and documentability are the main technical drivers for > +the decisions made within the standard. > + > + > +Differences from WSGI > +===================== > + > +- All protocol-specific environment names are prefixed with ``web3.`` > + ?rather than ``wsgi.``, eg. ``web3.input`` rather than > + ?``wsgi.input``. > + > +- All values present as environment dictionary *values* are explicitly > + ?*bytes* instances instead of native strings. ?(Environment *keys* > + ?however are native strings, always ``str`` regardless of > + ?platform). > + > +- All values returned by an application must be bytes instances, > + ?including status code, header names and values, and the body. > + > +- Wherever WSGI 1.0 referred to an ``app_iter``, this specification > + ?refers to a ``body``. > + > +- No ``start_response()`` callback (and therefore no ``write()`` > + ?callable nor ``exc_info`` data). > + > +- The ``readline()`` function of ``web3.input`` must support a size > + ?hint parameter. > + > +- The ``read()`` function of ``web3.input`` must be length delimited. > + ?A call without a size argument must not read more than the content > + ?length header specifies. ?In case a content length header is absent > + ?the stream must not return anything on read. ?It must never request > + ?more data than specified from the client. > + > +- No requirement for middleware to yield an empty string if it needs > + ?more information from an application to produce output (e.g. no > + ?"Middleware Handling of Block Boundaries"). > + > +- Filelike objects passed to a "file_wrapper" must have an > + ?``__iter__`` which returns bytes (never text). > + > +- ``wsgi.file_wrapper`` is not supported. > + > +- ``QUERY_STRING``, ``SCRIPT_NAME``, ``PATH_INFO`` values required to > + ?be placed in environ by server (each as the empty bytes instance if > + ?no associated value is received in the HTTP request). > + > +- ``web3.path_info`` and ``web3.script_name`` should be put into the > + ?Web3 environment, if possible, by the origin Web3 server. ?When > + ?available, each is the original, plain 7-bit ASCII, URL-encoded > + ?variant of its CGI equivalent derived directly from the request URI > + ?(with %2F segment markers and other meta-characters intact). ?If the > + ?server cannot provide one (or both) of these values, it must omit > + ?the value(s) it cannot provide from the environment. > + > +- This requirement was removed: "middleware components **must not** > + ?block iteration waiting for multiple values from an application > + ?iterable. ?If the middleware needs to accumulate more data from the > + ?application before it can produce any output, it **must** yield an > + ?empty string." > + > +- ``SERVER_PORT`` must be a bytes instance (not an integer). > + > +- The server must not inject an additional ``Content-Length`` header > + ?by guessing the length from the response iterable. ?This must be set > + ?by the application itself in all situations. > + > +- If the origin server advertises that it has the ``web3.async`` > + ?capability, a Web3 application callable used by the server is > + ?permitted to return a callable that accepts no arguments. ?When it > + ?does so, this callable is to be called periodically by the origin > + ?server until it returns a non-``None`` response, which must be a > + ?normal Web3 response tuple. > + > + ?.. XXX (chrism) Needs a section of its own for explanation. > + > + > +Specification Overview > +====================== > + > +The Web3 interface has two sides: the "server" or "gateway" side, and > +the "application" or "framework" side. ?The server side invokes a > +callable object that is provided by the application side. ?The > +specifics of how that object is provided are up to the server or > +gateway. ?It is assumed that some servers or gateways will require an > +application's deployer to write a short script to create an instance > +of the server or gateway, and supply it with the application object. > +Other servers and gateways may use configuration files or other > +mechanisms to specify where an application object should be imported > +from, or otherwise obtained. > + > +In addition to "pure" servers/gateways and applications/frameworks, it > +is also possible to create "middleware" components that implement both > +sides of this specification. ?Such components act as an application to > +their containing server, and as a server to a contained application, > +and can be used to provide extended APIs, content transformation, > +navigation, and other useful functions. > + > +Throughout this specification, we will use the term "application > +callable" to mean "a function, a method, or an instance with a > +``__call__`` method". ?It is up to the server, gateway, or application > +implementing the application callable to choose the appropriate > +implementation technique for their needs. ?Conversely, a server, > +gateway, or application that is invoking a callable **must not** have > +any dependency on what kind of callable was provided to it. > +Application callables are only to be called, not introspected upon. > + > + > +The Application/Framework Side > +------------------------------ > + > +The application object is simply a callable object that accepts one > +argument. ?The term "object" should not be misconstrued as requiring > +an actual object instance: a function, method, or instance with a > +``__call__`` method are all acceptable for use as an application > +object. ?Application objects must be able to be invoked more than > +once, as virtually all servers/gateways (other than CGI) will make > +such repeated requests. ?It this cannot be guaranteed by the > +implementation of the actual application, it has to be wrapped in a > +function that creates a new instance on each call. > + > +.. note:: > + > + ? Although we refer to it as an "application" object, this should not > + ? be construed to mean that application developers will use Web3 as a > + ? web programming API. ?It is assumed that application developers > + ? will continue to use existing, high-level framework services to > + ? develop their applications. ?Web3 is a tool for framework and > + ? server developers, and is not intended to directly support > + ? application developers.) > + > +An example of an application which is a function (``simple_app``):: > + > + ? ?def simple_app(environ): > + ? ? ? ?"""Simplest possible application object""" > + ? ? ? ?status = b'200 OK' > + ? ? ? ?headers = [(b'Content-type', b'text/plain')] > + ? ? ? ?body = [b'Hello world!\n'] > + ? ? ? ?return body, status, headers > + > +An example of an application which is an instance (``simple_app``):: > + > + ? ?class AppClass(object): > + > + ? ? ? ?"""Produce the same output, but using an instance. ?An > + ? ? ? ?instance of this class must be instantiated before it is > + ? ? ? ?passed to the server. ?""" > + > + ? ? ?def __call__(self, environ): > + ? ? ? ? ? ?status = b'200 OK' > + ? ? ? ? ? ?headers = [(b'Content-type', b'text/plain')] > + ? ? ? ? ? ?body = [b'Hello world!\n'] > + ? ? ? ? ? ?return body, status, headers > + > + ? ?simple_app = AppClass() > + > +Alternately, an application callable may return a callable instead of > +the tuple if the server supports asynchronous execution. ?See > +information concerning ``web3.async`` for more information. > + > + > +The Server/Gateway Side > +----------------------- > + > +The server or gateway invokes the application callable once for each > +request it receives from an HTTP client, that is directed at the > +application. ?To illustrate, here is a simple CGI gateway, implemented > +as a function taking an application object. ?Note that this simple > +example has limited error handling, because by default an uncaught > +exception will be dumped to ``sys.stderr`` and logged by the web > +server. > + > +:: > + > + ? ?import locale > + ? ?import os > + ? ?import sys > + > + ? ?encoding = locale.getpreferredencoding() > + > + ? ?stdout = sys.stdout > + > + ? ?if hasattr(sys.stdout, 'buffer'): > + ? ? ? ?# Python 3 compatibility; we need to be able to push bytes out > + ? ? ? ?stdout = sys.stdout.buffer > + > + ? ?def get_environ(): > + ? ? ? ?d = {} > + ? ? ? ?for k, v in os.environ.items(): > + ? ? ? ? ? ?# Python 3 compatibility > + ? ? ? ? ? ?if not isinstance(v, bytes): > + ? ? ? ? ? ? ? ?# We must explicitly encode the string to bytes under > + ? ? ? ? ? ? ? ?# Python 3.1+ > + ? ? ? ? ? ? ? ?v = v.encode(encoding, 'surrogateescape') > + ? ? ? ? ? ?d[k] = v > + ? ? ? ?return d > + > + ? ?def run_with_cgi(application): > + > + ? ? ? ?environ = get_environ() > + ? ? ? ?environ['web3.input'] ? ? ? ?= sys.stdin > + ? ? ? ?environ['web3.errors'] ? ? ? = sys.stderr > + ? ? ? ?environ['web3.version'] ? ? ?= (1, 0) > + ? ? ? ?environ['web3.multithread'] ?= False > + ? ? ? ?environ['web3.multiprocess'] = True > + ? ? ? ?environ['web3.run_once'] ? ? = True > + ? ? ? ?environ['web3.async'] ? ? ? ?= False > + > + ? ? ? ?if environ.get('HTTPS', b'off') in (b'on', b'1'): > + ? ? ? ? ? ?environ['web3.url_scheme'] = b'https' > + ? ? ? ?else: > + ? ? ? ? ? ?environ['web3.url_scheme'] = b'http' > + > + ? ? ? ?rv = application(environ) > + ? ? ? ?if hasattr(rv, '__call__'): > + ? ? ? ? ? ?raise TypeError('This webserver does not support asynchronous ' > + ? ? ? ? ? ? ? ? ? ? ? ? ? ?'responses.') > + ? ? ? ?body, status, headers = rv > + > + ? ? ? ?CLRF = b'\r\n' > + > + ? ? ? ?try: > + ? ? ? ? ? ?stdout.write(b'Status: ' + status + CRLF) > + ? ? ? ? ? ?for header_name, header_val in headers: > + ? ? ? ? ? ? ? ?stdout.write(header_name + b': ' + header_val + CRLF) > + ? ? ? ? ? ?stdout.write(CRLF) > + ? ? ? ? ? ?for chunk in body: > + ? ? ? ? ? ? ? ?stdout.write(chunk) > + ? ? ? ? ? ? ? ?stdout.flush() > + ? ? ? ?finally: > + ? ? ? ? ? ?if hasattr(body, 'close'): > + ? ? ? ? ? ? ? ?body.close() > + > + > +Middleware: Components that Play Both Sides > +------------------------------------------- > + > +A single object may play the role of a server with respect to some > +application(s), while also acting as an application with respect to > +some server(s). ?Such "middleware" components can perform such > +functions as: > + > +* Routing a request to different application objects based on the > + ?target URL, after rewriting the ``environ`` accordingly. > + > +* Allowing multiple applications or frameworks to run side-by-side in > + ?the same process. > + > +* Load balancing and remote processing, by forwarding requests and > + ?responses over a network. > + > +* Perform content postprocessing, such as applying XSL stylesheets. > + > +The presence of middleware in general is transparent to both the > +"server/gateway" and the "application/framework" sides of the > +interface, and should require no special support. ?A user who desires > +to incorporate middleware into an application simply provides the > +middleware component to the server, as if it were an application, and > +configures the middleware component to invoke the application, as if > +the middleware component were a server. ?Of course, the "application" > +that the middleware wraps may in fact be another middleware component > +wrapping another application, and so on, creating what is referred to > +as a "middleware stack". > + > +A middleware must support asychronous execution if possible or fall > +back to disabling itself. > + > +Here a middleware that changes the ``HTTP_HOST`` key if an ``X-Host`` > +header exists and adds a comment to all html responses:: > + > + ? ?import time > + > + ? ?def apply_filter(app, environ, filter_func): > + ? ? ? ?"""Helper function that passes the return value from an > + ? ? ? ?application to a filter function when the results are > + ? ? ? ?ready. > + ? ? ? ?""" > + ? ? ? ?app_response = app(environ) > + > + ? ? ? ?# synchronous response, filter now > + ? ? ? ?if not hasattr(app_response, '__call__'): > + ? ? ? ? ? ?return filter_func(*app_response) > + > + ? ? ? ?# asychronous response. ?filter when results are ready > + ? ? ? ?def polling_function(): > + ? ? ? ? ? ?rv = app_response() > + ? ? ? ? ? ?if rv is not None: > + ? ? ? ? ? ? ? ?return filter_func(*rv) > + ? ? ? ?return polling_function > + > + ? ?def proxy_and_timing_support(app): > + ? ? ? ?def new_application(environ): > + ? ? ? ? ? ?def filter_func(body, status, headers): > + ? ? ? ? ? ? ? ?now = time.time() > + ? ? ? ? ? ? ? ?for key, value in headers: > + ? ? ? ? ? ? ? ? ? ?if key.lower() == b'content-type' and \ > + ? ? ? ? ? ? ? ? ? ? ? value.split(b';')[0] == b'text/html': > + ? ? ? ? ? ? ? ? ? ? ? ?# assumes ascii compatible encoding in body, > + ? ? ? ? ? ? ? ? ? ? ? ?# but the middleware should actually parse the > + ? ? ? ? ? ? ? ? ? ? ? ?# content type header and figure out the > + ? ? ? ? ? ? ? ? ? ? ? ?# encoding when doing that. > + ? ? ? ? ? ? ? ? ? ? ? ?body += ('' % > + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? (now - then)).encode('ascii') > + ? ? ? ? ? ? ? ? ? ? ? ?break > + ? ? ? ? ? ? ? ?return body, status, headers > + ? ? ? ? ? ?then = time.time() > + ? ? ? ? ? ?host = environ.get('HTTP_X_HOST') > + ? ? ? ? ? ?if host is not None: > + ? ? ? ? ? ? ? ?environ['HTTP_HOST'] = host > + > + ? ? ? ? ? ?# use the apply_filter function that applies a given filter > + ? ? ? ? ? ?# function for both async and sync responses. > + ? ? ? ? ? ?return apply_filter(app, environ, filter_func) > + ? ? ? ?return new_application > + > + ? ?app = proxy_and_timing_support(app) > + > + > +Specification Details > +===================== > + > +The application callable must accept one positional argument. ?For the > +sake of illustration, we have named it ``environ``, but it is not > +required to have this name. ?A server or gateway **must** invoke the > +application object using a positional (not keyword) argument. > +(E.g. by calling ``status, headers, body = application(environ)`` as > +shown above.) > + > +The ``environ`` parameter is a dictionary object, containing CGI-style > +environment variables. ?This object **must** be a builtin Python > +dictionary (*not* a subclass, ``UserDict`` or other dictionary > +emulation), and the application is allowed to modify the dictionary in > +any way it desires. ?The dictionary must also include certain > +Web3-required variables (described in a later section), and may also > +include server-specific extension variables, named according to a > +convention that will be described below. > + > +When called by the server, the application object must return a tuple > +yielding three elements: ``status``, ``headers`` and ``body``, or, if > +supported by an async server, an argumentless callable which either > +returns ``None`` or a tuple of those three elements. > + > +The ``status`` element is a status in bytes of the form ``b'999 > +Message here'``. > + > +``headers`` is a Python list of ``(header_name, header_value)`` pairs > +describing the HTTP response header. ?The ``headers`` structure must > +be a literal Python list; it must yield two-tuples. ?Both > +``header_name`` and ``header_value`` must be bytes values. > + > +The ``body`` is an iterable yielding zero or more bytes instances. > +This can be accomplished in a variety of ways, such as by returning a > +list containing bytes instances as ``body``, or by returning a > +generator function as ``body`` that yields bytes instances, or by the > +``body`` being an instance of a class which is iterable. ?Regardless > +of how it is accomplished, the application object must always return a > +``body`` iterable yielding zero or more bytes instances. > + > +The server or gateway must transmit the yielded bytes to the client in > +an unbuffered fashion, completing the transmission of each set of > +bytes before requesting another one. ?(In other words, applications > +**should** perform their own buffering. ?See the `Buffering and > +Streaming`_ section below for more on how application output must be > +handled.) > + > +The server or gateway should treat the yielded bytes as binary byte > +sequences: in particular, it should ensure that line endings are not > +altered. ?The application is responsible for ensuring that the > +string(s) to be written are in a format suitable for the client. ?(The > +server or gateway **may** apply HTTP transfer encodings, or perform > +other transformations for the purpose of implementing HTTP features > +such as byte-range transmission. ?See `Other HTTP Features`_, below, > +for more details.) > + > +If the ``body`` iterable returned by the application has a ``close()`` > +method, the server or gateway **must** call that method upon > +completion of the current request, whether the request was completed > +normally, or terminated early due to an error. ?This is to support > +resource release by the application amd is intended to complement PEP > +325's generator support, and other common iterables with ``close()`` > +methods. > + > +Finally, servers and gateways **must not** directly use any other > +attributes of the ``body`` iterable returned by the application. > + > + > +``environ`` Variables > +--------------------- > + > +The ``environ`` dictionary is required to contain various CGI > +environment variables, as defined by the Common Gateway Interface > +specification [2]_. > + > +The following CGI variables **must** be present. ?Each key is a native > +string. ?Each value is a bytes instance. > + > +.. note:: > + > + ? In Python 3.1+, a "native string" is a ``str`` type decoded using > + ? the ``surrogateescape`` error handler, as done by > + ? ``os.environ.__getitem__``. ?In Python 2.6 and 2.7, a "native > + ? string" is a ``str`` types representing a set of bytes. > + > +``REQUEST_METHOD`` > + ?The HTTP request method, such as ``"GET"`` or ``"POST"``. > + > +``SCRIPT_NAME`` > + ?The initial portion of the request URL's "path" that corresponds to > + ?the application object, so that the application knows its virtual > + ?"location". ?This may be the empty bytes instance if the application > + ?corresponds to the "root" of the server. ?SCRIPT_NAME will be a > + ?bytes instance representing a sequence of URL-encoded segments > + ?separated by the slash character (``/``). ?It is assumed that > + ?``%2F`` characters will be decoded into literal slash characters > + ?within ``PATH_INFO`` , as per CGI. > + > +``PATH_INFO`` > + ?The remainder of the request URL's "path", designating the virtual > + ?"location" of the request's target within the application. ?This > + ?**may** be a bytes instance if the request URL targets the > + ?application root and does not have a trailing slash. ?PATH_INFO will > + ?be a bytes instance representing a sequence of URL-encoded segments > + ?separated by the slash character (``/``). ?It is assumed that > + ?``%2F`` characters will be decoded into literal slash characters > + ?within ``PATH_INFO`` , as per CGI. > + > +``QUERY_STRING`` > + ?The portion of the request URL (in bytes) that follows the ``"?"``, > + ?if any, or the empty bytes instance. > + > +``SERVER_NAME``, ``SERVER_PORT`` > + ?When combined with ``SCRIPT_NAME`` and ``PATH_INFO`` (or their raw > + ?equivalents)`, these variables can be used to complete the URL. > + ?Note, however, that ``HTTP_HOST``, if present, should be used in > + ?preference to ``SERVER_NAME`` for reconstructing the request URL. > + ?See the `URL Reconstruction`_ section below for more detail. > + ?``SERVER_PORT`` should be a bytes instance, not an integer. > + > +``SERVER_PROTOCOL`` > + ?The version of the protocol the client used to send the request. > + ?Typically this will be something like ``"HTTP/1.0"`` or > + ?``"HTTP/1.1"`` and may be used by the application to determine how > + ?to treat any HTTP request headers. ?(This variable should probably > + ?be called ``REQUEST_PROTOCOL``, since it denotes the protocol used > + ?in the request, and is not necessarily the protocol that will be > + ?used in the server's response. ?However, for compatibility with CGI > + ?we have to keep the existing name.) > + > +The following CGI values **may** present be in the Web3 environment. > +Each key is a native string. ?Each value is a bytes instances. > + > +``CONTENT_TYPE`` > + ?The contents of any ``Content-Type`` fields in the HTTP request. > + > +``CONTENT_LENGTH`` > + ?The contents of any ``Content-Length`` fields in the HTTP request. > + > +``HTTP_`` Variables > + ?Variables corresponding to the client-supplied HTTP request headers > + ?(i.e., variables whose names begin with ``"HTTP_"``). ?The presence > + ?or absence of these variables should correspond with the presence or > + ?absence of the appropriate HTTP header in the request. > + > +A server or gateway **should** attempt to provide as many other CGI > +variables as are applicable, each with a string for its key and a > +bytes instance for its value. ?In addition, if SSL is in use, the > +server or gateway **should** also provide as many of the Apache SSL > +environment variables [5]_ as are applicable, such as ``HTTPS=on`` and > +``SSL_PROTOCOL``. ?Note, however, that an application that uses any > +CGI variables other than the ones listed above are necessarily > +non-portable to web servers that do not support the relevant > +extensions. ?(For example, web servers that do not publish files will > +not be able to provide a meaningful ``DOCUMENT_ROOT`` or > +``PATH_TRANSLATED``.) > + > +A Web3-compliant server or gateway **should** document what variables > +it provides, along with their definitions as appropriate. > +Applications **should** check for the presence of any variables they > +require, and have a fallback plan in the event such a variable is > +absent. > + > +Note that CGI variable *values* must be bytes instances, if they are > +present at all. ?It is a violation of this specification for a CGI > +variable's value to be of any type other than ``bytes``. ?On Python 2, > +this means they will be of type ``str``. ?On Python 3, this means they > +will be of type ``bytes``. > + > +They *keys* of all CGI and non-CGI variables in the environ, however, > +must be "native strings" (on both Python 2 and Python 3, they will be > +of type ``str``). > + > +In addition to the CGI-defined variables, the ``environ`` dictionary > +**may** also contain arbitrary operating-system "environment > +variables", and **must** contain the following Web3-defined variables. > + > +===================== ?=============================================== > +Variable ? ? ? ? ? ? ? Value > +===================== ?=============================================== > +``web3.version`` ? ? ? The tuple ``(1, 0)``, representing Web3 > + ? ? ? ? ? ? ? ? ? ? ? version 1.0. > + > +``web3.url_scheme`` ? ?A bytes value representing the "scheme" portion of > + ? ? ? ? ? ? ? ? ? ? ? the URL at which the application is being > + ? ? ? ? ? ? ? ? ? ? ? invoked. ?Normally, this will have the value > + ? ? ? ? ? ? ? ? ? ? ? ``b"http"`` or ``b"https"``, as appropriate. > + > +``web3.input`` ? ? ? ? An input stream (file-like object) from which bytes > + ? ? ? ? ? ? ? ? ? ? ? constituting the HTTP request body can be read. > + ? ? ? ? ? ? ? ? ? ? ? (The server or gateway may perform reads > + ? ? ? ? ? ? ? ? ? ? ? on-demand as requested by the application, or > + ? ? ? ? ? ? ? ? ? ? ? it may pre- read the client's request body and > + ? ? ? ? ? ? ? ? ? ? ? buffer it in-memory or on disk, or use any > + ? ? ? ? ? ? ? ? ? ? ? other technique for providing such an input > + ? ? ? ? ? ? ? ? ? ? ? stream, according to its preference.) > + > +``web3.errors`` ? ? ? ?An output stream (file-like object) to which error > + ? ? ? ? ? ? ? ? ? ? ? output text can be written, for the purpose of > + ? ? ? ? ? ? ? ? ? ? ? recording program or other errors in a > + ? ? ? ? ? ? ? ? ? ? ? standardized and possibly centralized location. > + ? ? ? ? ? ? ? ? ? ? ? This should be a "text mode" stream; i.e., > + ? ? ? ? ? ? ? ? ? ? ? applications should use ``"\n"`` as a line > + ? ? ? ? ? ? ? ? ? ? ? ending, and assume that it will be converted to > + ? ? ? ? ? ? ? ? ? ? ? the correct line ending by the server/gateway. > + ? ? ? ? ? ? ? ? ? ? ? Applications may *not* send bytes to the > + ? ? ? ? ? ? ? ? ? ? ? 'write' method of this stream; they may only > + ? ? ? ? ? ? ? ? ? ? ? send text. > + > + ? ? ? ? ? ? ? ? ? ? ? For many servers, ``web3.errors`` will be the > + ? ? ? ? ? ? ? ? ? ? ? server's main error log. Alternatively, this > + ? ? ? ? ? ? ? ? ? ? ? may be ``sys.stderr``, or a log file of some > + ? ? ? ? ? ? ? ? ? ? ? sort. ?The server's documentation should > + ? ? ? ? ? ? ? ? ? ? ? include an explanation of how to configure this > + ? ? ? ? ? ? ? ? ? ? ? or where to find the recorded output. ?A server > + ? ? ? ? ? ? ? ? ? ? ? or gateway may supply different error streams > + ? ? ? ? ? ? ? ? ? ? ? to different applications, if this is desired. > + > +``web3.multithread`` ? This value should evaluate true if the > + ? ? ? ? ? ? ? ? ? ? ? application object may be simultaneously > + ? ? ? ? ? ? ? ? ? ? ? invoked by another thread in the same process, > + ? ? ? ? ? ? ? ? ? ? ? and should evaluate false otherwise. > + > +``web3.multiprocess`` ?This value should evaluate true if an > + ? ? ? ? ? ? ? ? ? ? ? equivalent application object may be > + ? ? ? ? ? ? ? ? ? ? ? simultaneously invoked by another process, and > + ? ? ? ? ? ? ? ? ? ? ? should evaluate false otherwise. > + > +``web3.run_once`` ? ? ?This value should evaluate true if the server > + ? ? ? ? ? ? ? ? ? ? ? or gateway expects (but does not guarantee!) > + ? ? ? ? ? ? ? ? ? ? ? that the application will only be invoked this > + ? ? ? ? ? ? ? ? ? ? ? one time during the life of its containing > + ? ? ? ? ? ? ? ? ? ? ? process. ?Normally, this will only be true for > + ? ? ? ? ? ? ? ? ? ? ? a gateway based on CGI (or something similar). > + > +``web3.script_name`` ? The non-URL-decoded ``SCRIPT_NAME`` value. > + ? ? ? ? ? ? ? ? ? ? ? Through a historical inequity, by virtue of the > + ? ? ? ? ? ? ? ? ? ? ? CGI specification, ``SCRIPT_NAME`` is present > + ? ? ? ? ? ? ? ? ? ? ? within the environment as an already > + ? ? ? ? ? ? ? ? ? ? ? URL-decoded string. ?This is the original > + ? ? ? ? ? ? ? ? ? ? ? URL-encoded value derived from the request URI. > + ? ? ? ? ? ? ? ? ? ? ? If the server cannot provide this value, it > + ? ? ? ? ? ? ? ? ? ? ? must omit it from the environ. > + > +``web3.path_info`` ? ? The non-URL-decoded ``PATH_INFO`` value. > + ? ? ? ? ? ? ? ? ? ? ? Through a historical inequity, by virtue of the > + ? ? ? ? ? ? ? ? ? ? ? CGI specification, ``PATH_INFO`` is present > + ? ? ? ? ? ? ? ? ? ? ? within the environment as an already > + ? ? ? ? ? ? ? ? ? ? ? URL-decoded string. ?This is the original > + ? ? ? ? ? ? ? ? ? ? ? URL-encoded value derived from the request URI. > + ? ? ? ? ? ? ? ? ? ? ? If the server cannot provide this value, it > + ? ? ? ? ? ? ? ? ? ? ? must omit it from the environ. > + > +``web3.async`` ? ? ? ? This is ``True`` if the webserver supports > + ? ? ? ? ? ? ? ? ? ? ? async invocation. ?In that case an application > + ? ? ? ? ? ? ? ? ? ? ? is allowed to return a callable instead of a > + ? ? ? ? ? ? ? ? ? ? ? tuple with the response. ?The exact semantics > + ? ? ? ? ? ? ? ? ? ? ? are not specified by this specification. > + > +===================== ?=============================================== > + > +Finally, the ``environ`` dictionary may also contain server-defined > +variables. ?These variables should have names which are native > +strings, composed of only lower-case letters, numbers, dots, and > +underscores, and should be prefixed with a name that is unique to the > +defining server or gateway. ?For example, ``mod_web3`` might define > +variables with names like ``mod_web3.some_variable``. > + > + > +Input Stream > +~~~~~~~~~~~~ > + > +The input stream (``web3.input``) provided by the server must support > +the following methods: > + > +===================== ?======== > +Method ? ? ? ? ? ? ? ? Notes > +===================== ?======== > +``read(size)`` ? ? ? ? 1,4 > +``readline([size])`` ? 1,2,4 > +``readlines([size])`` ?1,3,4 > +``__iter__()`` ? ? ? ? 4 > +===================== ?======== > + > +The semantics of each method are as documented in the Python Library > +Reference, except for these notes as listed in the table above: > + > +1. The server is not required to read past the client's specified > + ? ``Content-Length``, and is allowed to simulate an end-of-file > + ? condition if the application attempts to read past that point. ?The > + ? application **should not** attempt to read more data than is > + ? specified by the ``CONTENT_LENGTH`` variable. > + > +2. The implementation must support the optional ``size`` argument to > + ? ``readline()``. > + > +3. The application is free to not supply a ``size`` argument to > + ? ``readlines()``, and the server or gateway is free to ignore the > + ? value of any supplied ``size`` argument. > + > +4. The ``read``, ``readline`` and ``__iter__`` methods must return a > + ? bytes instance. ?The ``readlines`` method must return a sequence > + ? which contains instances of bytes. > + > +The methods listed in the table above **must** be supported by all > +servers conforming to this specification. ?Applications conforming to > +this specification **must not** use any other methods or attributes of > +the ``input`` object. ?In particular, applications **must not** > +attempt to close this stream, even if it possesses a ``close()`` > +method. > + > +The input stream should silently ignore attempts to read more than the > +content length of the request. ?If no content length is specified the > +stream must be a dummy stream that does not return anything. > + > + > +Error Stream > +~~~~~~~~~~~~ > + > +The error stream (``web3.errors``) provided by the server must support > +the following methods: > + > +=================== ? ========== ?======== > +Method ? ? ? ? ? ? ? ?Stream ? ? ?Notes > +=================== ? ========== ?======== > +``flush()`` ? ? ? ? ? ``errors`` ?1 > +``write(str)`` ? ? ? ?``errors`` ?2 > +``writelines(seq)`` ? ``errors`` ?2 > +=================== ? ========== ?======== > + > +The semantics of each method are as documented in the Python Library > +Reference, except for these notes as listed in the table above: > + > +1. Since the ``errors`` stream may not be rewound, servers and > + ? gateways are free to forward write operations immediately, without > + ? buffering. ?In this case, the ``flush()`` method may be a no-op. > + ? Portable applications, however, cannot assume that output is > + ? unbuffered or that ``flush()`` is a no-op. ?They must call > + ? ``flush()`` if they need to ensure that output has in fact been > + ? written. ?(For example, to minimize intermingling of data from > + ? multiple processes writing to the same error log.) > + > +2. The ``write()`` method must accept a string argument, but needn't > + ? necessarily accept a bytes argument. ?The ``writelines()`` method > + ? must accept a sequence argument that consists entirely of strings, > + ? but needn't necessarily accept any bytes instance as a member of > + ? the sequence. > + > +The methods listed in the table above **must** be supported by all > +servers conforming to this specification. ?Applications conforming to > +this specification **must not** use any other methods or attributes of > +the ``errors`` object. ?In particular, applications **must not** > +attempt to close this stream, even if it possesses a ``close()`` > +method. > + > + > +Values Returned by A Web3 Application > +------------------------------------- > + > +Web3 applications return an iterable in the form (``status``, > +``headers``, ``body``). ?The return value can be any iterable type > +that returns exactly three values. ?If the server supports > +asynchronous applications (``web3.async``), the response may be a > +callable object (which accepts no arguments). > + > +The ``status`` value is assumed by a gateway or server to be an HTTP > +"status" bytes instance like ``b'200 OK'`` or ``b'404 Not Found'``. > +That is, it is a string consisting of a Status-Code and a > +Reason-Phrase, in that order and separated by a single space, with no > +surrounding whitespace or other characters. ?(See RFC 2616, Section > +6.1.1 for more information.) ?The string **must not** contain control > +characters, and must not be terminated with a carriage return, > +linefeed, or combination thereof. > + > +The ``headers`` value is assumed by a gateway or server to be a > +literal Python list of ``(header_name, header_value)`` tuples. ?Each > +``header_name`` must be a bytes instance representing a valid HTTP > +header field-name (as defined by RFC 2616, Section 4.2), without a > +trailing colon or other punctuation. ?Each ``header_value`` must be a > +bytes instance and **must not** include any control characters, > +including carriage returns or linefeeds, either embedded or at the > +end. ?(These requirements are to minimize the complexity of any > +parsing that must be performed by servers, gateways, and intermediate > +response processors that need to inspect or modify response headers.) > + > +In general, the server or gateway is responsible for ensuring that > +correct headers are sent to the client: if the application omits a > +header required by HTTP (or other relevant specifications that are in > +effect), the server or gateway **must** add it. ?For example, the HTTP > +``Date:`` and ``Server:`` headers would normally be supplied by the > +server or gateway. ?The gateway must however not override values with > +the same name if they are emitted by the application. > + > +(A reminder for server/gateway authors: HTTP header names are > +case-insensitive, so be sure to take that into consideration when > +examining application-supplied headers!) > + > +Applications and middleware are forbidden from using HTTP/1.1 > +"hop-by-hop" features or headers, any equivalent features in HTTP/1.0, > +or any headers that would affect the persistence of the client's > +connection to the web server. ?These features are the exclusive > +province of the actual web server, and a server or gateway **should** > +consider it a fatal error for an application to attempt sending them, > +and raise an error if they are supplied as return values from an > +application in the ``headers`` structure. ?(For more specifics on > +"hop-by-hop" features and headers, please see the `Other HTTP > +Features`_ section below.) > + > + > +Dealing with Compatibility Across Python Versions > +------------------------------------------------- > + > +Creating Web3 code that runs under both Python 2.6/2.7 and Python 3.1+ > +requires some care on the part of the developer. ?In general, the Web3 > +specification assumes a certain level of equivalence between the > +Python 2 ``str`` type and the Python 3 ``bytes`` type. ?For example, > +under Python 2, the values present in the Web3 ``environ`` will be > +instances of the ``str`` type; in Python 3, these will be instances of > +the ``bytes`` type. ?The Python 3 ``bytes`` type does not possess all > +the methods of the Python 2 ``str`` type, and some methods which it > +does possess behave differently than the Python 2 ``str`` type. > +Effectively, to ensure that Web3 middleware and applications work > +across Python versions, developers must do these things: > + > +#) Do not assume comparison equivalence between text values and bytes > + ? values. ?If you do so, your code may work under Python 2, but it > + ? will not work properly under Python 3. ?For example, don't write > + ? ``somebytes == 'abc'``. ?This will sometimes be true on Python 2 > + ? but it will never be true on Python 3, because a sequence of bytes > + ? never compares equal to a string under Python 3. ?Instead, always > + ? compare a bytes value with a bytes value, e.g. "somebytes == > + ? b'abc'". ?Code which does this is compatible with and works the > + ? same in Python 2.6, 2.7, and 3.1. ?The ``b`` in front of ``'abc'`` > + ? signals to Python 3 that the value is a literal bytes instance; > + ? under Python 2 it's a forward compatibility placebo. > + > +#) Don't use the ``__contains__`` method (directly or indirectly) of > + ? items that are meant to be byteslike without ensuring that its > + ? argument is also a bytes instance. ?If you do so, your code may > + ? work under Python 2, but it will not work properly under Python 3. > + ? For example, ``'abc' in somebytes'`` will raise a ``TypeError`` > + ? under Python 3, but it will return ``True`` under Python 2.6 and > + ? 2.7. ?However, ``b'abc' in somebytes`` will work the same on both > + ? versions. ?In Python 3.2, this restriction may be partially > + ? removed, as it's rumored that bytes types may obtain a ``__mod__`` > + ? implementation. > + > +#) ``__getitem__`` should not be used. > + > + ? .. XXX > + > +#) Dont try to use the ``format`` method or the ``__mod__`` method of > + ? instances of bytes (directly or indirectly). ?In Python 2, the > + ? ``str`` type which we treat equivalently to Python 3's ``bytes`` > + ? supports these method but actual Python 3's ``bytes`` instances > + ? don't support these methods. ?If you use these methods, your code > + ? will work under Python 2, but not under Python 3. > + > +#) Do not try to concatenate a bytes value with a string value. ?This > + ? may work under Python 2, but it will not work under Python 3. ?For > + ? example, doing ``'abc' + somebytes`` will work under Python 2, but > + ? it will result in a ``TypeError`` under Python 3. ?Instead, always > + ? make sure you're concatenating two items of the same type, > + ? e.g. ``b'abc' + somebytes``. > + > +Web3 expects byte values in other places, such as in all the values > +returned by an application. > + > +In short, to ensure compatibility of Web3 application code between > +Python 2 and Python 3, in Python 2, treat CGI and server variable > +values in the environment as if they had the Python 3 ``bytes`` API > +even though they actually have a more capable API. ?Likewise for all > +stringlike values returned by a Web3 application. > + > + > +Buffering and Streaming > +----------------------- > + > +Generally speaking, applications will achieve the best throughput by > +buffering their (modestly-sized) output and sending it all at once. > +This is a common approach in existing frameworks: the output is > +buffered in a StringIO or similar object, then transmitted all at > +once, along with the response headers. > + > +The corresponding approach in Web3 is for the application to simply > +return a single-element ``body`` iterable (such as a list) containing > +the response body as a single string. ?This is the recommended > +approach for the vast majority of application functions, that render > +HTML pages whose text easily fits in memory. > + > +For large files, however, or for specialized uses of HTTP streaming > +(such as multipart "server push"), an application may need to provide > +output in smaller blocks (e.g. to avoid loading a large file into > +memory). ?It's also sometimes the case that part of a response may be > +time-consuming to produce, but it would be useful to send ahead the > +portion of the response that precedes it. > + > +In these cases, applications will usually return a ``body`` iterator > +(often a generator-iterator) that produces the output in a > +block-by-block fashion. ?These blocks may be broken to coincide with > +mulitpart boundaries (for "server push"), or just before > +time-consuming tasks (such as reading another block of an on-disk > +file). > + > +Web3 servers, gateways, and middleware **must not** delay the > +transmission of any block; they **must** either fully transmit the > +block to the client, or guarantee that they will continue transmission > +even while the application is producing its next block. ?A > +server/gateway or middleware may provide this guarantee in one of > +three ways: > + > +1. Send the entire block to the operating system (and request that any > + ? O/S buffers be flushed) before returning control to the > + ? application, OR > + > +2. Use a different thread to ensure that the block continues to be > + ? transmitted while the application produces the next block. > + > +3. (Middleware only) send the entire block to its parent > + ? gateway/server. > + > +By providing this guarantee, Web3 allows applications to ensure that > +transmission will not become stalled at an arbitrary point in their > +output data. ?This is critical for proper functioning of > +e.g. multipart "server push" streaming, where data between multipart > +boundaries should be transmitted in full to the client. > + > + > +Unicode Issues > +-------------- > + > +HTTP does not directly support Unicode, and neither does this > +interface. ?All encoding/decoding must be handled by the > +**application**; all values passed to or from the server must be of > +the Python 3 type ``bytes`` or instances of the Python 2 type ``str``, > +not Python 2 ``unicode`` or Python 3 ``str`` objects. > + > +All "bytes instances" referred to in this specification **must**: > + > +- On Python 2, be of type ``str``. > + > +- On Python 3, be of type ``bytes``. > + > +All "bytes instances" **must not** : > + > +- On Python 2, ?be of type ``unicode``. > + > +- On Python 3, be of type ``str``. > + > +The result of using a textlike object where a byteslike object is > +required is undefined. > + > +Values returned from a Web3 app as a status or as response headers > +**must** follow RFC 2616 with respect to encoding. ?That is, the bytes > +returned must contain a character stream of ISO-8859-1 characters, or > +the character stream should use RFC 2047 MIME encoding. > + > +On Python platforms which do not have a native bytes-like type > +(e.g. IronPython, etc.), but instead which generally use textlike > +strings to represent bytes data, the definition of "bytes instance" > +can be changed: their "bytes instances" must be native strings that > +contain only code points representable in ISO-8859-1 encoding > +(``\u0000`` through ``\u00FF``, inclusive). ?It is a fatal error for > +an application on such a platform to supply strings containing any > +other Unicode character or code point. ?Similarly, servers and > +gateways on those platforms **must not** supply strings to an > +application containing any other Unicode characters. > + > +.. XXX (armin: Jython now has a bytes type, we might remove this > + ? section after seeing about IronPython) > + > + > +HTTP 1.1 Expect/Continue > +------------------------ > + > +Servers and gateways that implement HTTP 1.1 **must** provide > +transparent support for HTTP 1.1's "expect/continue" mechanism. ?This > +may be done in any of several ways: > + > +1. Respond to requests containing an ``Expect: 100-continue`` request > + ? with an immediate "100 Continue" response, and proceed normally. > + > +2. Proceed with the request normally, but provide the application with > + ? a ``web3.input`` stream that will send the "100 Continue" response > + ? if/when the application first attempts to read from the input > + ? stream. ?The read request must then remain blocked until the client > + ? responds. > + > +3. Wait until the client decides that the server does not support > + ? expect/continue, and sends the request body on its own. ?(This is > + ? suboptimal, and is not recommended.) > + > +Note that these behavior restrictions do not apply for HTTP 1.0 > +requests, or for requests that are not directed to an application > +object. ?For more information on HTTP 1.1 Expect/Continue, see RFC > +2616, sections 8.2.3 and 10.1.1. > + > + > +Other HTTP Features > +------------------- > + > +In general, servers and gateways should "play dumb" and allow the > +application complete control over its output. ?They should only make > +changes that do not alter the effective semantics of the application's > +response. ?It is always possible for the application developer to add > +middleware components to supply additional features, so server/gateway > +developers should be conservative in their implementation. ?In a > +sense, a server should consider itself to be like an HTTP "gateway > +server", with the application being an HTTP "origin server". ?(See RFC > +2616, section 1.3, for the definition of these terms.) > + > +However, because Web3 servers and applications do not communicate via > +HTTP, what RFC 2616 calls "hop-by-hop" headers do not apply to Web3 > +internal communications. ?Web3 applications **must not** generate any > +"hop-by-hop" headers [4]_, attempt to use HTTP features that would > +require them to generate such headers, or rely on the content of any > +incoming "hop-by-hop" headers in the ``environ`` dictionary. ?Web3 > +servers **must** handle any supported inbound "hop-by-hop" headers on > +their own, such as by decoding any inbound ``Transfer-Encoding``, > +including chunked encoding if applicable. > + > +Applying these principles to a variety of HTTP features, it should be > +clear that a server **may** handle cache validation via the > +``If-None-Match`` and ``If-Modified-Since`` request headers and the > +``Last-Modified`` and ``ETag`` response headers. ?However, it is not > +required to do this, and the application **should** perform its own > +cache validation if it wants to support that feature, since the > +server/gateway is not required to do such validation. > + > +Similarly, a server **may** re-encode or transport-encode an > +application's response, but the application **should** use a suitable > +content encoding on its own, and **must not** apply a transport > +encoding. ?A server **may** transmit byte ranges of the application's > +response if requested by the client, and the application doesn't > +natively support byte ranges. ?Again, however, the application > +**should** perform this function on its own if desired. > + > +Note that these restrictions on applications do not necessarily mean > +that every application must reimplement every HTTP feature; many HTTP > +features can be partially or fully implemented by middleware > +components, thus freeing both server and application authors from > +implementing the same features over and over again. > + > + > +Thread Support > +-------------- > + > +Thread support, or lack thereof, is also server-dependent. ?Servers > +that can run multiple requests in parallel, **should** also provide > +the option of running an application in a single-threaded fashion, so > +that applications or frameworks that are not thread-safe may still be > +used with that server. > + > + > +Implementation/Application Notes > +================================ > + > +Server Extension APIs > +--------------------- > + > +Some server authors may wish to expose more advanced APIs, that > +application or framework authors can use for specialized purposes. > +For example, a gateway based on ``mod_python`` might wish to expose > +part of the Apache API as a Web3 extension. > + > +In the simplest case, this requires nothing more than defining an > +``environ`` variable, such as ``mod_python.some_api``. ?But, in many > +cases, the possible presence of middleware can make this difficult. > +For example, an API that offers access to the same HTTP headers that > +are found in ``environ`` variables, might return different data if > +``environ`` has been modified by middleware. > + > +In general, any extension API that duplicates, supplants, or bypasses > +some portion of Web3 functionality runs the risk of being incompatible > +with middleware components. ?Server/gateway developers should *not* > +assume that nobody will use middleware, because some framework > +developers specifically organize their frameworks to function almost > +entirely as middleware of various kinds. > + > +So, to provide maximum compatibility, servers and gateways that > +provide extension APIs that replace some Web3 functionality, **must** > +design those APIs so that they are invoked using the portion of the > +API that they replace. ?For example, an extension API to access HTTP > +request headers must require the application to pass in its current > +``environ``, so that the server/gateway may verify that HTTP headers > +accessible via the API have not been altered by middleware. ?If the > +extension API cannot guarantee that it will always agree with > +``environ`` about the contents of HTTP headers, it must refuse service > +to the application, e.g. by raising an error, returning ``None`` > +instead of a header collection, or whatever is appropriate to the API. > + > +These guidelines also apply to middleware that adds information such > +as parsed cookies, form variables, sessions, and the like to > +``environ``. ?Specifically, such middleware should provide these > +features as functions which operate on ``environ``, rather than simply > +stuffing values into ``environ``. ?This helps ensure that information > +is calculated from ``environ`` *after* any middleware has done any URL > +rewrites or other ``environ`` modifications. > + > +It is very important that these "safe extension" rules be followed by > +both server/gateway and middleware developers, in order to avoid a > +future in which middleware developers are forced to delete any and all > +extension APIs from ``environ`` to ensure that their mediation isn't > +being bypassed by applications using those extensions! > + > + > +Application Configuration > +------------------------- > + > +This specification does not define how a server selects or obtains an > +application to invoke. ?These and other configuration options are > +highly server-specific matters. ?It is expected that server/gateway > +authors will document how to configure the server to execute a > +particular application object, and with what options (such as > +threading options). > + > +Framework authors, on the other hand, should document how to create an > +application object that wraps their framework's functionality. ?The > +user, who has chosen both the server and the application framework, > +must connect the two together. ?However, since both the framework and > +the server have a common interface, this should be merely a mechanical > +matter, rather than a significant engineering effort for each new > +server/framework pair. > + > +Finally, some applications, frameworks, and middleware may wish to use > +the ``environ`` dictionary to receive simple string configuration > +options. ?Servers and gateways **should** support this by allowing an > +application's deployer to specify name-value pairs to be placed in > +``environ``. ?In the simplest case, this support can consist merely of > +copying all operating system-supplied environment variables from > +``os.environ`` into the ``environ`` dictionary, since the deployer in > +principle can configure these externally to the server, or in the CGI > +case they may be able to be set via the server's configuration files. > + > +Applications **should** try to keep such required variables to a > +minimum, since not all servers will support easy configuration of > +them. ?Of course, even in the worst case, persons deploying an > +application can create a script to supply the necessary configuration > +values:: > + > + ? from the_app import application > + > + ? def new_app(environ): > + ? ? ? environ['the_app.configval1'] = b'something' > + ? ? ? return application(environ) > + > +But, most existing applications and frameworks will probably only need > +a single configuration value from ``environ``, to indicate the > +location of their application or framework-specific configuration > +file(s). ?(Of course, applications should cache such configuration, to > +avoid having to re-read it upon each invocation.) > + > + > +URL Reconstruction > +------------------ > + > +If an application wishes to reconstruct a request's complete URL (as a > +bytes object), it may do so using the following algorithm:: > + > + ? ?host = environ.get('HTTP_HOST') > + > + ? ?scheme = environ['web3.url_scheme'] > + ? ?port = environ['SERVER_PORT'] > + ? ?query = environ['QUERY_STRING'] > + > + ? ?url = scheme + b'://' > + > + ? ?if host: > + ? ? ? ?url += host > + ? ?else: > + ? ? ? ?url += environ['SERVER_NAME'] > + > + ? ? ? ?if scheme == b'https': > + ? ? ? ? ? ?if port != b'443': > + ? ? ? ? ? ? ? url += b':' + port > + ? ? ? ?else: > + ? ? ? ? ? ?if port != b'80': > + ? ? ? ? ? ? ? url += b':' + port > + > + ? ?if 'web3.script_name' in url: > + ? ? ? ?url += url_quote(environ['web3.script_name']) > + ? ?else: > + ? ? ? ?url += environ['SCRIPT_NAME'] > + ? ?if 'web3.path_info' in environ: > + ? ? ? ?url += url_quote(environ['web3.path_info']) > + ? ?else: > + ? ? ? ?url += environ['PATH_INFO'] > + ? ?if query: > + ? ? ? ?url += b'?' + query > + > +Note that such a reconstructed URL may not be precisely the same URI > +as requested by the client. ?Server rewrite rules, for example, may > +have modified the client's originally requested URL to place it in a > +canonical form. > + > + > +Open Questions > +============== > + > +- ``file_wrapper`` replacement. ?Currently nothing is specified here > + ?but it's clear that the old system of in-band signalling is broken > + ?if it does not provide a way to figure out as a middleware in the > + ?process if the response is a file wrapper. > + > + > +Points of Contention > +==================== > + > +Outlined below are potential points of contention regarding this > +specification. > + > + > +WSGI 1.0 Compatibility > +---------------------- > + > +Components written using the WSGI 1.0 specification will not > +transparently interoperate with components written using this > +specification. ?That's because the goals of this proposal and the > +goals of WSGI 1.0 are not directly aligned. > + > +WSGI 1.0 is obliged to provide specification-level backwards > +compatibility with versions of Python between 2.2 and 2.7. ?This > +specification, however, ditches Python 2.5 and lower compatibility in > +order to provide compatibility between relatively recent versions of > +Python 2 (2.6 and 2.7) as well as relatively recent versions of Python > +3 (3.1). > + > +It is currently impossible to write components which work reliably > +under both Python 2 and Python 3 using the WSGI 1.0 specification, > +because the specification implicitly posits that CGI and server > +variable values in the environ and values returned via > +``start_response`` represent a sequence of bytes that can be addressed > +using the Python 2 string API. ?It posits such a thing because that > +sort of data type was the sensible way to represent bytes in all > +Python 2 versions, and WSGI 1.0 was conceived before Python 3 existed. > + > +Python 3's ``str`` type supports the full API provided by the Python 2 > +``str`` type, but Python 3's ``str`` type does not represent a > +sequence of bytes, it instead represents text. ?Therefore, using it to > +represent environ values also requires that the environ byte sequence > +be decoded to text via some encoding. ?We cannot decode these bytes to > +text (at least in any way where the decoding has any meaning other > +than as a tunnelling mechanism) without widening the scope of WSGI to > +include server and gateway knowledge of decoding policies and > +mechanics. ?WSGI 1.0 never concerned itself with encoding and > +decoding. ?It made statements about allowable transport values, and > +suggested that various values might be best decoded as one encoding or > +another, but it never required a server to *perform* any decoding > +before > + > +Python 3 does not have a stringlike type that can be used instead to > +represent bytes: it has a ``bytes`` type. ?A bytes type operates quite > +a bit like a Python 2 ``str`` in Python 3.1+, but it lacks behavior > +equivalent to ``str.__mod__`` and its iteration protocol, and > +containment, sequence treatment, and equivalence comparisons are > +different. > + > +In either case, there is no type in Python 3 that behaves just like > +the Python 2 ``str`` type, and a way to create such a type doesn't > +exist because there is no such thing as a "String ABC" which would > +allow a suitable type to be built. ?Due to this design > +incompatibility, existing WSGI 1.0 servers, middleware, and > +applications will not work under Python 3, even after they are run > +through ``2to3``. > + > +Existing Web-SIG discussions about updating the WSGI specification so > +that it is possible to write a WSGI application that runs in both > +Python 2 and Python 3 tend to revolve around creating a > +specification-level equivalence between the Python 2 ``str`` type > +(which represents a sequence of bytes) and the Python 3 ``str`` type > +(which represents text). ?Such an equivalence becomes strained in > +various areas, given the different roles of these types. ?An arguably > +more straightforward equivalence exists between the Python 3 ``bytes`` > +type API and a subset of the Python 2 ``str`` type API. ?This > +specification exploits this subset equivalence. > + > +In the meantime, aside from any Python 2 vs. Python 3 compatibility > +issue, as various discussions on Web-SIG have pointed out, the WSGI > +1.0 specification is too general, providing support (via ``.write``) > +for asynchronous applications at the expense of implementation > +complexity. ?This specification uses the fundamental incompatibility > +between WSGI 1.0 and Python 3 as a natural divergence point to create > +a specification with reduced complexity by changing specialized > +support for asynchronous applications. > + > +To provide backwards compatibility for older WSGI 1.0 applications, so > +that they may run on a Web3 stack, it is presumed that Web3 middleware > +will be created which can be used "in front" of existing WSGI 1.0 > +applications, allowing those existing WSGI 1.0 applications to run > +under a Web3 stack. ?This middleware will require, when under Python > +3, an equivalence to be drawn between Python 3 ``str`` types and the > +bytes values represented by the HTTP request and all the attendant > +encoding-guessing (or configuration) it implies. > + > +.. note:: > + > + ? Such middleware *might* in the future, instead of drawing an > + ? equivalence between Python 3 ``str`` and HTTP byte values, make use > + ? of a yet-to-be-created "ebytes" type (aka "bytes-with-benefits"), > + ? particularly if a String ABC proposal is accepted into the Python > + ? core and implemented. > + > +Conversely, it is presumed that WSGI 1.0 middleware will be created > +which will allow a Web3 application to run behind a WSGI 1.0 stack on > +the Python 2 platform. > + > + > +Environ and Response Values as Bytes > +------------------------------------ > + > +Casual middleware and application writers may consider the use of > +bytes as environment values and response values inconvenient. ?In > +particular, they won't be able to use common string formatting > +functions such as ``('%s' % bytes_val)`` or > +``bytes_val.format('123')`` because bytes don't have the same API as > +strings on platforms such as Python 3 where the two types differ. > +Likewise, on such platforms, stdlib HTTP-related API support for using > +bytes interchangeably with text can be spotty. ?In places where bytes > +are inconvenient or incompatible with library APIs, middleware and > +application writers will have to decode such bytes to text explicitly. > +This is particularly inconvenient for middleware writers: to work with > +environment values as strings, they'll have to decode them from an > +implied encoding and if they need to mutate an environ value, they'll > +then need to encode the value into a byte stream before placing it > +into the environ. ?While the use of bytes by the specification as > +environ values might be inconvenient for casual developers, it > +provides several benefits. > + > +Using bytes types to represent HTTP and server values to an > +application most closely matches reality because HTTP is fundamentally > +a bytes-oriented protocol. ?If the environ values are mandated to be > +strings, each server will need to use heuristics to guess about the > +encoding of various values provided by the HTTP environment. ?Using > +all strings might increase casual middleware writer convenience, but > +will also lead to ambiguity and confusion when a value cannot be > +decoded to a meaningful non-surrogate string. > + > +Use of bytes as environ values avoids any potential for the need for > +the specification to mandate that a participating server be informed > +of encoding configuration parameters. ?If environ values are treated > +as strings, and so must be decoded from bytes, configuration > +parameters may eventually become necessary as policy clues from the > +application deployer. ?Such a policy would be used to guess an > +appropriate decoding strategy in various circumstances, effectively > +placing the burden for enforcing a particular application encoding > +policy upon the server. ?If the server must serve more than one > +application, such configuration would quickly become complex. ?Many > +policies would also be impossible to express declaratively. > + > +In reality, HTTP is a complicated and legacy-fraught protocol which > +requires a complex set of heuristics to make sense of. It would be > +nice if we could allow this protocol to protect us from this > +complexity, but we cannot do so reliably while still providing to > +application writers a level of control commensurate with reality. > +Python applications must often deal with data embedded in the > +environment which not only must be parsed by legacy heuristics, but > +*does not conform even to any existing HTTP specification*. ?While > +these eventualities are unpleasant, they crop up with regularity, > +making it impossible and undesirable to hide them from application > +developers, as application developers are the only people who are able > +to decide upon an appropriate action when an HTTP specification > +violation is detected. > + > +Some have argued for mixed use of bytes and string values as environ > +*values*. ?This proposal avoids that strategy. ?Sole use of bytes as > +environ values makes it possible to fit this specification entirely in > +one's head; you won't need to guess about which values are strings and > +which are bytes. > + > +This protocol would also fit in a developer's head if all environ > +values were strings, but this specification doesn't use that strategy. > +This will likely be the point of greatest contention regarding the use > +of bytes. ?In defense of bytes: developers often prefer protocols with > +consistent contracts, even if the contracts themselves are suboptimal. > +If we hide encoding issues from a developer until a value that > +contains surrogates causes problems after it has already reached > +beyond the I/O boundary of their application, they will need to do a > +lot more work to fix assumptions made by their application than if we > +were to just present the problem much earlier in terms of "here's some > +bytes, you decode them". ?This is also a counter-argument to the > +"bytes are inconvenient" assumption: while presenting bytes to an > +application developer may be inconvenient for a casual application > +developer who doesn't care about edge cases, they are extremely > +convenient for the application developer who needs to deal with > +complex, dirty eventualities, because use of bytes allows him the > +appropriate level of control with a clear separation of > +responsibility. > + > +If the protocol uses bytes, it is presumed that libraries will be > +created to make working with bytes-only in the environ and within > +return values more pleasant; for example, analogues of the WSGI 1.0 > +libraries named "WebOb" and "Werkzeug". ?Such libraries will fill the > +gap between convenience and control, allowing the spec to remain > +simple and regular while still allowing casual authors a convenient > +way to create Web3 middleware and application components. ?This seems > +to be a reasonable alternative to baking encoding policy into the > +protocol, because many such libraries can be created independently > +from the protocol, and application developers can choose the one that > +provides them the appropriate levels of control and convenience for a > +particular job. > + > +Here are some alternatives to using all bytes: > + > +- Have the server decode all values representing CGI and server > + ?environ values into strings using the ``latin-1`` encoding, which is > + ?lossless. ?Smuggle any undecodable bytes within the resulting > + ?string. > + > +- Encode all CGI and server environ values to strings using the > + ?``utf-8`` encoding with the ``surrogateescape`` error handler. ?This > + ?does not work under any existing Python 2. > + > +- Encode some values into bytes and other values into strings, as > + ?decided by their typical usages. > + > + > +Applications Should be Allowed to Read ``web3.input`` Past ``CONTENT_LENGTH`` > +----------------------------------------------------------------------------- > + > +At [6]_, Graham Dumpleton makes the assertion that ``wsgi.input`` > +should be required to return the empty string as a signifier of > +out-of-data, and that applications should be allowed to read past the > +number of bytes specified in ``CONTENT_LENGTH``, depending only upon > +the empty string as an EOF marker. ?WSGI relies on an application > +"being well behaved and once all data specified by ``CONTENT_LENGTH`` > +is read, that it processes the data and returns any response. That > +same socket connection could then be used for a subsequent request." > +Graham would like WSGI adapters to be required to wrap raw socket > +connections: "this wrapper object will need to count how much data has > +been read, and when the amount of data reaches that as defined by > +``CONTENT_LENGTH``, any subsequent reads should return an empty string > +instead." ?This may be useful to support chunked encoding and input > +filters. > + > + > +``web3.input`` Unknown Length > +----------------------------- > + > +There's no documented way to indicate that there is content in > +``environ['web3.input']``, but the content length is unknown. > + > + > +``read()`` of ``web3.input`` Should Support No-Size Calling Convention > +---------------------------------------------------------------------- > + > +At [6]_, Graham Dumpleton makes the assertion that the ``read()`` > +method of ``wsgi.input`` should be callable without arguments, and > +that the result should be "all available request content". ?Needs > +discussion. > + > +Comment Armin: I changed the spec to require that from an > +implementation. ?I had too much pain with that in the past already. > +Open for discussions though. > + > + > +Input Filters should set environ ``CONTENT_LENGTH`` to -1 > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +At [6]_, Graham Dumpleton suggests that an input filter might set > +``environ['CONTENT_LENGTH']`` to -1 to indicate that it mutated the > +input. > + > + > +``headers`` as Literal List of Two-Tuples > +----------------------------------------- > + > +Why do we make applications return a ``headers`` structure that is a > +literal list of two-tuples? ?I think the iterability of ``headers`` > +needs to be maintained while it moves up the stack, but I don't think > +we need to be able to mutate it in place at all times. ?Could we > +loosen that requirement? > + > +Comment Armin: Strong yes > + > + > +Removed Requirement that Middleware Not Block > +--------------------------------------------- > + > +This requirement was removed: "middleware components **must not** > +block iteration waiting for multiple values from an application > +iterable. ?If the middleware needs to accumulate more data from the > +application before it can produce any output, it **must** yield an > +empty string." ?This requirement existed to support asynchronous > +applications and servers (see PEP 333's "Middleware Handling of Block > +Boundaries"). ?Asynchronous applications are now serviced explicitly > +by ``web3.async`` capable protocol (a Web3 application callable may > +itself return a callable). > + > + > +``web3.script_name`` and ``web3.path_info`` > +------------------------------------------- > + > +These values are required to be placed into the environment by an > +origin server under this specification. ?Unlike ``SCRIPT_NAME`` and > +``PATH_INFO``, these must be the original *URL-encoded* variants > +derived from the request URI. ?We probably need to figure out how > +these should be computed originally, and what their values should be > +if the server performs URL rewriting. > + > + > +Long Response Headers > +--------------------- > + > +Bob Brewer notes on Web-SIG [7]_: > + > + ? ?Each header_value must not include any control characters, > + ? ?including carriage returns or linefeeds, either embedded or at the > + ? ?end. ?(These requirements are to minimize the complexity of any > + ? ?parsing that must be performed by servers, gateways, and > + ? ?intermediate response processors that need to inspect or modify > + ? ?response headers.) [1]_ > + > +That's understandable, but HTTP headers are defined as (mostly) > +\*TEXT, and "words of \*TEXT MAY contain characters from character > +sets other than ISO-8859-1 only when encoded according to the rules of > +RFC 2047." ?[2]_ And RFC 2047 specifies that "an 'encoded-word' may > +not be more than 75 characters long... ?If it is desirable to encode > +more text than will fit in an 'encoded-word' of 75 characters, > +multiple 'encoded-word's (separated by CRLF SPACE) may be used." [3]_ > +This satisfies HTTP header folding rules, as well: "Header fields can > +be extended over multiple lines by preceding each extra line with at > +least one SP or HT." [1]_ > + > +So in my reading of HTTP, some code somewhere should introduce > +newlines in longish, encoded response header values. ?I see three > +options: > + > +1. Keep things as they are and disallow response header values if they > + ? contain words over 75 chars that are outside the ISO-8859-1 > + ? character set. > + > +2. Allow newline characters in WSGI response headers. > + > +3. Require/strongly suggest WSGI servers to do the encoding and > + ? folding before sending the value over HTTP. > + > + > +Request Trailers and Chunked Transfer Encoding > +---------------------------------------------- > + > +When using chunked transfer encoding on request content, the RFCs > +allow there to be request trailers. ?These are like request headers > +but come after the final null data chunk. ?These trailers are only > +available when the chunked data stream is finite length and when it > +has all been read in. ?Neither WSGI nor Web3 currently supports them. > + > +.. XXX (armin) yield from application iterator should be specify write > + ? plus flush by server. > + > +.. XXX (armin) websocket API. > + > + > +References > +========== > + > +.. [1] PEP 333: Python Web Services Gateway Interface > + ? (http://www.python.org/dev/peps/pep-0333/) > + > +.. [2] The Common Gateway Interface Specification, v 1.1, 3rd Draft > + ? (http://cgi-spec.golux.com/draft-coar-cgi-v11-03.txt) > + > +.. [3] "Chunked Transfer Coding" -- HTTP/1.1, section 3.6.1 > + ? (http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1) > + > +.. [4] "End-to-end and Hop-by-hop Headers" -- HTTP/1.1, Section 13.5.1 > + ? (http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.5.1) > + > +.. [5] mod_ssl Reference, "Environment Variables" > + ? (http://www.modssl.org/docs/2.8/ssl_reference.html#ToC25) > + > +.. [6] Details on WSGI 1.0 amendments/clarifications. > + ? (http://blog.dscpl.com.au/2009/10/details-on-wsgi-10-amendmentsclarificat.html) > + > +.. [7] [Web-SIG] WSGI and long response header values > + ? http://mail.python.org/pipermail/web-sig/2006-September/002244.html > + > +Copyright > +========= > + > +This document has been placed in the public domain. > + > + > + > +.. > + ? Local Variables: > + ? mode: indented-text > + ? indent-tabs-mode: nil > + ? sentence-end-double-space: t > + ? fill-column: 70 > + ? coding: utf-8 > + ? End: > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > From pje at telecommunity.com Thu Sep 16 01:48:06 2010 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 15 Sep 2010 19:48:06 -0400 Subject: [Python-Dev] PEP 444 aka "Web3" (was Re: how to decide on a Python 3 design for wsgiref) In-Reply-To: References: <1284588983.14651.33.camel@thinko> <1284591970.14651.38.camel@thinko> Message-ID: <20100915234805.BCDCD3A403D@sparrow.telecommunity.com> At 09:22 AM 9/16/2010 +1000, James Mills wrote: >On Thu, Sep 16, 2010 at 9:06 AM, Chris McDonough wrote: > > Comments and competing specs would be useful. > >Can I post comments here ? :) Please, let's put any spec-detail commentary on the Web-SIG instead (commenting here on process issues related to the 3.x releases is of course fine). From nagle at animats.com Thu Sep 16 02:02:44 2010 From: nagle at animats.com (John Nagle) Date: Wed, 15 Sep 2010 17:02:44 -0700 Subject: [Python-Dev] Add PEP 444, Python Web3 Interface. In-Reply-To: References: Message-ID: <4C915EA4.5040300@animats.com> On 9/15/2010 4:44 PM, python-dev-request at python.org wrote: > ``SERVER_PORT`` must be a bytes instance (not an integer). What's that supposed to mean? What goes in the "bytes instance"? A character string in some format? A long binary number? If the latter, with which byte ordering? What problem does this\ solve? John Nagle From johnvmo at hotmail.com Thu Sep 16 02:17:39 2010 From: johnvmo at hotmail.com (=?iso-8859-1?B?Sm/jbyBWaXRvcg==?=) Date: Thu, 16 Sep 2010 03:17:39 +0300 Subject: [Python-Dev] problem with python 3.1 Message-ID: I made a program that, according to my teacher, is correct but is not running properly.The program is really simple: import mathx = input ("Coloque o valor do primeiro cateto:")y = input ("Coloque o valor do segundo cateto:")z = x**2w = y**2soma = z + wh = math.sqrt (soma)print = "O valor da hipotenusa ?:", h But after I put the value of x and y this error appears: Traceback (most recent call last): File "C:/lista03.py", line 4, in z = x**2TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int' My teacher said the program ran normally in his computer, but in my it doesn't!why? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jacob at jacobian.org Thu Sep 16 02:55:16 2010 From: jacob at jacobian.org (Jacob Kaplan-Moss) Date: Wed, 15 Sep 2010 19:55:16 -0500 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> Message-ID: On Wed, Sep 15, 2010 at 6:31 PM, Jesse Noller wrote: > My goal (personally) is to make sure python 3.2 is perfectly good for use in web applications, and is therefore a much more interesting porting target for web projects/libraries and frameworks. To try (again) to make things concrete here: I didn't work to get Django running on Python 3.0 because it was just too slow. I'm not working to get Django running on Python 3.1 because I don't feel confident I'll be able to put any apps I write into production. If Python 3.2 is the same, I won't feel any motivation to target it and I'll get to be lazy and wait for Python 3.3. If, on the other hand, Python 3.2 "is perfectly good for use in web applications" then I'll be inspired to work on porting to it. And if I don't Jesse will yell out me, louder and louder, until I find some inspiration. Now, I'm really only speaking for myself here -- not making any statements on behalf of the Django core team, or community, or Python web community, or whatever. So if you think I'm alone (or a minority) in feeling this way then by all means hurry up and push out Python 3.2. If, on the other hand, you think that others in the Python web community might feel similarly, well, then maybe the delay would be worth it. Jacob From benjamin at python.org Thu Sep 16 02:58:01 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 15 Sep 2010 19:58:01 -0500 Subject: [Python-Dev] problem with python 3.1 In-Reply-To: References: Message-ID: This list is for the development of Python, not questions about its use. You should ask python-list. None the less, it's because input() changed between Python 2 and 3. Look at the docs for both. 2010/9/15 Jo?o Vitor : > I made a program that, according to my teacher, is correct but is not > running properly. > The program is really simple: > import math > x = input ("Coloque o valor do primeiro cateto:") > y = input ("Coloque o valor do segundo cateto:") > z = x**2 > w = y**2 > soma = z + w > h = math.sqrt (soma) > print = "O valor da hipotenusa ?:", h > But after I put the value of x and y this error appears: > Traceback (most recent call last): > ??File "C:/lista03.py", line 4, in > ?? ?z = x**2 > TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int' > My teacher said the program ran normally in his computer, but in my it > doesn't! > why? > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/benjamin%40python.org > > -- Regards, Benjamin From tjreedy at udel.edu Thu Sep 16 08:16:47 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 16 Sep 2010 02:16:47 -0400 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> Message-ID: On 9/15/2010 8:55 PM, Jacob Kaplan-Moss wrote: > To try (again) to make things concrete here: > > I didn't work to get Django running on Python 3.0 because it was just too slow. Soon after 3.0 was released, it was discovered and acknowledged thay the new I/O has some speed problems. (Why not discovered sooner? Because the people who had the use cases that exposed it did not run them on the alpha or beta ;-). It was quickly fixed and 3.1 was released earlier than planned it order to get the fix out. Otherwise, 3.x should be about the same speed as 2.x. If you know of other, specific, unreported problems, clue us in. > > I'm not working to get Django running on Python 3.1 because I don't > feel confident I'll be able to put any apps I write into production. Why not? Since the I/O speed problem is fixed, I have no idea what you are referring to. Please do be concrete. -- Terry Jan Reedy From hagen at zhuliguan.net Thu Sep 16 10:16:08 2010 From: hagen at zhuliguan.net (=?ISO-8859-1?Q?Hagen_F=FCrstenau?=) Date: Thu, 16 Sep 2010 10:16:08 +0200 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> Message-ID: <4C91D248.1010503@zhuliguan.net> > Why not? Since the I/O speed problem is fixed, I have no idea what you > are referring to. Please do be concrete. There's still a performance issue with pickling, but if issue 3873 could be resolved, Python 3 would actually be faster there. - Hagen -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 554 bytes Desc: OpenPGP digital signature URL: From chris at simplistix.co.uk Thu Sep 16 13:08:59 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 16 Sep 2010 12:08:59 +0100 Subject: [Python-Dev] standards for distribution names Message-ID: <4C91FACB.3010300@simplistix.co.uk> Hi All, Following on from this question: http://twistedmatrix.com/pipermail/twisted-python/2010-September/022877.html ...I'd thought that the "correct names" for distributions would have been documented in one of: http://www.python.org/dev/peps/pep-0345 http://www.python.org/dev/peps/pep-0376 http://www.python.org/dev/peps/pep-0386 ...but having read them, I drew a blank. Where are the standards for this or is it still a case of "whatever setuptools does"? Chris From solipsis at pitrou.net Thu Sep 16 14:26:10 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 16 Sep 2010 14:26:10 +0200 Subject: [Python-Dev] (Not) delaying the 3.2 release References: <20100914121744.31ef5aaf@pitrou.net> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> Message-ID: <20100916142610.6b9bca7b@pitrou.net> On Wed, 15 Sep 2010 19:55:16 -0500 Jacob Kaplan-Moss wrote: > On Wed, Sep 15, 2010 at 6:31 PM, Jesse Noller wrote: > > My goal (personally) is to make sure python 3.2 is perfectly good for use in web applications, and is therefore a much more interesting porting target for web projects/libraries and frameworks. > > To try (again) to make things concrete here: > > I didn't work to get Django running on Python 3.0 because it was just too slow. > > I'm not working to get Django running on Python 3.1 because I don't > feel confident I'll be able to put any apps I write into production. > > If Python 3.2 is the same, I won't feel any motivation to target it > and I'll get to be lazy and wait for Python 3.3. Why won't you feel confident? Are there any specific issues (apart from the lack of a WSGI PEP)? If they are technical problems, they should be reported on the bug tracker. If they are representational, cultural or psychological issues, I'm not sure what we can do. But delaying the release won't solve them. Regards Antoine. From jnoller at gmail.com Thu Sep 16 15:26:23 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 16 Sep 2010 09:26:23 -0400 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: <20100916142610.6b9bca7b@pitrou.net> References: <20100914121744.31ef5aaf@pitrou.net> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> Message-ID: On Thu, Sep 16, 2010 at 8:26 AM, Antoine Pitrou wrote: > On Wed, 15 Sep 2010 19:55:16 -0500 > Jacob Kaplan-Moss wrote: >> On Wed, Sep 15, 2010 at 6:31 PM, Jesse Noller wrote: >> > My goal (personally) is to make sure python 3.2 is perfectly good for use in web applications, and is therefore a much more interesting porting target for web projects/libraries and frameworks. >> >> To try (again) to make things concrete here: >> >> I didn't work to get Django running on Python 3.0 because it was just too slow. >> >> I'm not working to get Django running on Python 3.1 because I don't >> feel confident I'll be able to put any apps I write into production. >> >> If Python 3.2 is the same, I won't feel any motivation to target it >> and I'll get to be lazy and wait for Python 3.3. > > Why won't you feel confident? Are there any specific issues (apart from > the lack of a WSGI PEP)? > If they are technical problems, they should be reported on the bug > tracker. > If they are representational, cultural or psychological issues, I'm > not sure what we can do. But delaying the release won't solve them. > > Regards > > Antoine. Can we please give it a little bit of time to hear from the WSGI / Web-Sig folks? I've encouraged bugs to be filed, and discussions to happen here so we know if things (and what those things are), should be fixed. Is there any need, other then our current schedule to push 3.2 out until we can at least get some feedback from the interested parties? From ncoghlan at gmail.com Thu Sep 16 15:28:22 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 16 Sep 2010 23:28:22 +1000 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: <20100916142610.6b9bca7b@pitrou.net> References: <20100914121744.31ef5aaf@pitrou.net> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> Message-ID: On Thu, Sep 16, 2010 at 10:26 PM, Antoine Pitrou wrote: > Why won't you feel confident? Are there any specific issues (apart from > the lack of a WSGI PEP)? > If they are technical problems, they should be reported on the bug > tracker. > If they are representational, cultural or psychological issues, I'm > not sure what we can do. But delaying the release won't solve them. There are some APIs that should be able to handle bytes *or* strings, but the current use of string literals in their implementation means that bytes don't work. This turns out to be a PITA for some networking related code which really wants to be working with raw bytes (e.g. URLs coming off the wire). For example: >>> import urllib.parse as parse >>> parse.urlsplit("http://www.ubuntu.com") SplitResult(scheme='http', netloc='www.ubuntu.com', path='', query='', fragment='') >>> parse.urlsplit(b"http://www.ubuntu.com") Traceback (most recent call last): File "", line 1, in File "/home/ncoghlan/devel/py3k/Lib/urllib/parse.py", line 178, in urlsplit i = url.find(':') TypeError: expected an object with the buffer interface There's no real reason urlsplit (and similar urllib.parse APIs) shouldn't support bytes, but the internal use of string literals currently prevents it. We don't seem to have created a tracker issue from the discussion back in June where this came up, so I went ahead and created one just now: http://bugs.python.org/issue9873 I think there were other APIs mentioned back then beyond the urllib.parse ones, but I didn't find them when I went trawling through the list archives yesterday. If anyone else thinks of any APIs that should allow bytes as well as strings (or vice-versa) feel free to add them to that issue. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From barry at python.org Thu Sep 16 15:52:48 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 16 Sep 2010 09:52:48 -0400 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> Message-ID: <20100916095248.432bd12a@mission> On Sep 16, 2010, at 11:28 PM, Nick Coghlan wrote: >There are some APIs that should be able to handle bytes *or* strings, >but the current use of string literals in their implementation means >that bytes don't work. This turns out to be a PITA for some networking >related code which really wants to be working with raw bytes (e.g. >URLs coming off the wire). Note that email has exactly the same problem. A general solution -- even if embodied in *well documented* best-practices and convention -- would really help make the stdlib work consistently, and I bet third party libraries too. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From p.f.moore at gmail.com Thu Sep 16 16:59:47 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 16 Sep 2010 15:59:47 +0100 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> Message-ID: On 16 September 2010 07:16, Terry Reedy wrote: >> I'm not working to get Django running on Python 3.1 because I don't >> feel confident I'll be able to put any apps I write into production. > > Why not? Since the I/O speed problem is fixed, I have no idea what you are > referring to. ?Please do be concrete. At the risk of putting words into Jacob's mouth, I understood him to mean that "production quality" WSGI servers either do not exist, or do not implement a consistently defined spec (i.e., everyone is doing their own thing to adapt WSGI to Python 3). There is something of a chicken and egg situation here as with everywhere else (scientific users weren't moving until scipy did, lots of projects based round Twisted can't go until Twisted does, ...) but in the case of web/WSGI, there's a standard, defined in a PEP, with a reference implementation (wsgiref) in the stdlib. So the core has a greater interest. Personally, I don't write web applications (not even in Python :-)) so my interest is minimal. But I think the issue is real, and it's valid for the core team to be concerned. Whether I'd want to delay 3.2, I'm not so sure - certainly not indefinitely, there should be a "put up or shut up" deadline. But I'd be sad if Python 3 saw a reversion to the days of "Python isn't a good web development language because there's no standard infrastructure" comments that was the situation before WSGI existed... Paul. From rdmurray at bitdance.com Thu Sep 16 17:30:12 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 16 Sep 2010 11:30:12 -0400 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <20100916095248.432bd12a@mission> References: <20100914121744.31ef5aaf@pitrou.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> Message-ID: <20100916153012.8FCA11BF412@kimball.webabinitio.net> On Thu, 16 Sep 2010 09:52:48 -0400, Barry Warsaw wrote: > On Sep 16, 2010, at 11:28 PM, Nick Coghlan wrote: > >There are some APIs that should be able to handle bytes *or* strings, > >but the current use of string literals in their implementation means > >that bytes don't work. This turns out to be a PITA for some networking > >related code which really wants to be working with raw bytes (e.g. > >URLs coming off the wire). > > Note that email has exactly the same problem. A general solution -- even if > embodied in *well documented* best-practices and convention -- would really > help make the stdlib work consistently, and I bet third party libraries too. Allowing bytes-in -> bytes-out where possible would definitely be a help (and Guido has endorsed this, IIUC), but some care has to be taken to understand the API contract of the method in question before blindly applying it. Are you "merely" allowing bytes to be processed as ASCII strings, or does processing the bytes *correctly* imply that you are converting from an ASCII encoding of text in order to process it? In Python2, the latter might not generate unicode yet still produce a correct result most of the time, but a big point of Python3 is to eliminate that "most of the time", so we need to be careful not to reintroduce it. This was all covered in the thread Nick refers to; I just want to emphasize that one needs to look at the API contract carefully before making it polymorphic (in Guido's sense of the term). If the way to do this is well documented best practices, we first have to figure out what those best practices are. To do that we have to write some real-world code. I'm trying one approach in email6: Bytes and String subclasses, where the subclasses have an attribute named 'literals' derived from a utility module that does this: literals = dict( empty = '', colon = ':', newline = '\n', space = ' ', tab = '\t', fws = ' \t', headersep = ': ', ) class _string_literals: pass class _bytes_literals: pass for name, value in literals.items(): setattr(_string_literals, name, value) setattr(_bytes_literals, name, bytes(value, 'ASCII')) del literals, name, value And the subclasses do: class BytesHeader(BaseHeader): lit = email.utils._bytes_literals class StringHeader(BaseHeader): lit = email.utils._string_literals And then BaseHeader uses self.lit.colon, etc, when manipulating strings. It also has to use slice notation rather than indexing when looking at individual characters, which is a PITA but not terrible. I'm not saying this is the best approach, since this is all experimental code at the moment, but it is *an* approach.... -- R. David Murray www.bitdance.com From g.brandl at gmx.net Thu Sep 16 17:27:50 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 16 Sep 2010 17:27:50 +0200 Subject: [Python-Dev] r84771 - python/branches/release27-maint/Lib/test/test_io.py In-Reply-To: <20100913082019.DDF9CF525@mail.python.org> References: <20100913082019.DDF9CF525@mail.python.org> Message-ID: Maybe you want to mention *who* warns? Georg Am 13.09.2010 10:20, schrieb florent.xicluna: > Author: florent.xicluna > Date: Mon Sep 13 10:20:19 2010 > New Revision: 84771 > > Log: > Silence warning about 1/0 > > Modified: > python/branches/release27-maint/Lib/test/test_io.py > > Modified: python/branches/release27-maint/Lib/test/test_io.py > ============================================================================== > --- python/branches/release27-maint/Lib/test/test_io.py (original) > +++ python/branches/release27-maint/Lib/test/test_io.py Mon Sep 13 10:20:19 2010 > @@ -2484,7 +2484,7 @@ > signal.signal(signal.SIGALRM, self.oldalrm) > > def alarm_interrupt(self, sig, frame): > - 1/0 > + 1 // 0 > > @unittest.skipUnless(threading, 'Threading required for this test.') > def check_interrupted_write(self, item, bytes, **fdopen_kwargs): -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From solipsis at pitrou.net Thu Sep 16 17:38:50 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 16 Sep 2010 17:38:50 +0200 Subject: [Python-Dev] r84771 - python/branches/release27-maint/Lib/test/test_io.py References: <20100913082019.DDF9CF525@mail.python.org> Message-ID: <20100916173850.4a6f7b7d@pitrou.net> On Thu, 16 Sep 2010 17:27:50 +0200 Georg Brandl wrote: > Maybe you want to mention *who* warns? I suppose it's the -3 flag: $ ~/cpython/27/python -3 -c "1/0" -c:1: DeprecationWarning: classic int division Traceback (most recent call last): File "", line 1, in ZeroDivisionError: integer division or modulo by zero From g.brandl at gmx.net Thu Sep 16 17:37:43 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 16 Sep 2010 17:37:43 +0200 Subject: [Python-Dev] r84847 - python/branches/py3k/Doc/library/re.rst In-Reply-To: <20100916120217.8C93FFD78@mail.python.org> References: <20100916120217.8C93FFD78@mail.python.org> Message-ID: That reminds me of the undocumented re.Scanner -- which is meant to do exactly this. Wouldn't it be about time to document or remove it? Georg Am 16.09.2010 14:02, schrieb raymond.hettinger: > Author: raymond.hettinger > Date: Thu Sep 16 14:02:17 2010 > New Revision: 84847 > > Log: > Add tokenizer example to regex docs. > > Modified: > python/branches/py3k/Doc/library/re.rst > > Modified: python/branches/py3k/Doc/library/re.rst > ============================================================================== > --- python/branches/py3k/Doc/library/re.rst (original) > +++ python/branches/py3k/Doc/library/re.rst Thu Sep 16 14:02:17 2010 > @@ -1282,3 +1282,66 @@ > <_sre.SRE_Match object at ...> > >>> re.match("\\\\", r"\\") > <_sre.SRE_Match object at ...> > + > + > +Writing a Tokenizer > +^^^^^^^^^^^^^^^^^^^ > + > +A `tokenizer or scanner `_ > +analyzes a string to categorize groups of characters. This is a useful first > +step in writing a compiler or interpreter. > + > +The text categories are specified with regular expressions. The technique is > +to combine those into a single master regular expression and to loop over > +successive matches:: > + > + Token = collections.namedtuple('Token', 'typ value line column') > + > + def tokenize(s): > + tok_spec = [ > + ('NUMBER', r'\d+(.\d+)?'), # Integer or decimal number > + ('ASSIGN', r':='), # Assignment operator > + ('END', ';'), # Statement terminator > + ('ID', r'[A-Za-z]+'), # Identifiers > + ('OP', r'[+*\/\-]'), # Arithmetic operators > + ('NEWLINE', r'\n'), # Line endings > + ('SKIP', r'[ \t]'), # Skip over spaces and tabs > + ] > + tok_re = '|'.join('(?P<%s>%s)' % pair for pair in tok_spec) > + gettok = re.compile(tok_re).match > + line = 1 > + pos = line_start = 0 > + mo = gettok(s) > + while mo is not None: > + typ = mo.lastgroup > + if typ == 'NEWLINE': > + line_start = pos > + line += 1 > + elif typ != 'SKIP': > + yield Token(typ, mo.group(typ), line, mo.start()-line_start) > + pos = mo.end() > + mo = gettok(s, pos) > + if pos != len(s): > + raise RuntimeError('Unexpected character %r on line %d' %(s[pos], line)) > + > + >>> statements = '''\ > + total := total + price * quantity; > + tax := price * 0.05; > + ''' > + >>> for token in tokenize(statements): > + ... print(token) > + ... > + Token(typ='ID', value='total', line=1, column=8) > + Token(typ='ASSIGN', value=':=', line=1, column=14) > + Token(typ='ID', value='total', line=1, column=17) > + Token(typ='OP', value='+', line=1, column=23) > + Token(typ='ID', value='price', line=1, column=25) > + Token(typ='OP', value='*', line=1, column=31) > + Token(typ='ID', value='quantity', line=1, column=33) > + Token(typ='END', value=';', line=1, column=41) > + Token(typ='ID', value='tax', line=2, column=9) > + Token(typ='ASSIGN', value=':=', line=2, column=13) > + Token(typ='ID', value='price', line=2, column=16) > + Token(typ='OP', value='*', line=2, column=22) > + Token(typ='NUMBER', value='0.05', line=2, column=24) > + Token(typ='END', value=';', line=2, column=28) -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From a.badger at gmail.com Thu Sep 16 17:42:07 2010 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 16 Sep 2010 11:42:07 -0400 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: <20100916095248.432bd12a@mission> References: <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd12a@mission> Message-ID: <20100916154207.GS3086@unaka.lan> On Thu, Sep 16, 2010 at 09:52:48AM -0400, Barry Warsaw wrote: > On Sep 16, 2010, at 11:28 PM, Nick Coghlan wrote: > > >There are some APIs that should be able to handle bytes *or* strings, > >but the current use of string literals in their implementation means > >that bytes don't work. This turns out to be a PITA for some networking > >related code which really wants to be working with raw bytes (e.g. > >URLs coming off the wire). > > Note that email has exactly the same problem. A general solution -- even if > embodied in *well documented* best-practices and convention -- would really > help make the stdlib work consistently, and I bet third party libraries too. > I too await a solution with abated breath :-) I've been working on documenting best practices for APIs and Unicode and for this type of function (take bytes or unicode and output the same type), knowing the encoding is seems like a requirement in most cases: http://packages.python.org/kitchen/designing-unicode-apis.html#take-either-bytes-or-unicode-output-the-same-type I'd love to add another strategy there that shows how you can robustly operate on bytes without knowing the encoding but from writing that, I think that anytime you simplify your API you have to accept limitations on the data you can take in. (For instance, some simplifications can handle anything except ASCII-incompatible encodings). -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From solipsis at pitrou.net Thu Sep 16 17:40:53 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 16 Sep 2010 17:40:53 +0200 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] References: <20100914121744.31ef5aaf@pitrou.net> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> Message-ID: <20100916174053.007c783f@pitrou.net> On Thu, 16 Sep 2010 11:30:12 -0400 "R. David Murray" wrote: > > And then BaseHeader uses self.lit.colon, etc, when manipulating strings. > It also has to use slice notation rather than indexing when looking at > individual characters, which is a PITA but not terrible. > > I'm not saying this is the best approach, since this is all experimental > code at the moment, but it is *an* approach.... Out of curiousity, can you explain why polymorphism is needed for e-mail? I would assume that headers are bytes until they are parsed, at which point they become a pair of unicode strings (one for the header name and one for its value). Regards Antoine. From fuzzyman at voidspace.org.uk Thu Sep 16 17:49:49 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 16 Sep 2010 16:49:49 +0100 Subject: [Python-Dev] r84847 - python/branches/py3k/Doc/library/re.rst In-Reply-To: References: <20100916120217.8C93FFD78@mail.python.org> Message-ID: <4C923C9D.3080204@voidspace.org.uk> On 16/09/2010 16:37, Georg Brandl wrote: > That reminds me of the undocumented re.Scanner -- which is meant to do > exactly this. Wouldn't it be about time to document or remove it? > There was a long discussion about this on the bug tracker (the suggestion to document it was rejected at the time). http://bugs.python.org/issue5337 Michael Foord > Georg > > Am 16.09.2010 14:02, schrieb raymond.hettinger: >> Author: raymond.hettinger >> Date: Thu Sep 16 14:02:17 2010 >> New Revision: 84847 >> >> Log: >> Add tokenizer example to regex docs. >> >> Modified: >> python/branches/py3k/Doc/library/re.rst >> >> Modified: python/branches/py3k/Doc/library/re.rst >> ============================================================================== >> --- python/branches/py3k/Doc/library/re.rst (original) >> +++ python/branches/py3k/Doc/library/re.rst Thu Sep 16 14:02:17 2010 >> @@ -1282,3 +1282,66 @@ >> <_sre.SRE_Match object at ...> >> >>> re.match("\\\\", r"\\") >> <_sre.SRE_Match object at ...> >> + >> + >> +Writing a Tokenizer >> +^^^^^^^^^^^^^^^^^^^ >> + >> +A `tokenizer or scanner`_ >> +analyzes a string to categorize groups of characters. This is a useful first >> +step in writing a compiler or interpreter. >> + >> +The text categories are specified with regular expressions. The technique is >> +to combine those into a single master regular expression and to loop over >> +successive matches:: >> + >> + Token = collections.namedtuple('Token', 'typ value line column') >> + >> + def tokenize(s): >> + tok_spec = [ >> + ('NUMBER', r'\d+(.\d+)?'), # Integer or decimal number >> + ('ASSIGN', r':='), # Assignment operator >> + ('END', ';'), # Statement terminator >> + ('ID', r'[A-Za-z]+'), # Identifiers >> + ('OP', r'[+*\/\-]'), # Arithmetic operators >> + ('NEWLINE', r'\n'), # Line endings >> + ('SKIP', r'[ \t]'), # Skip over spaces and tabs >> + ] >> + tok_re = '|'.join('(?P<%s>%s)' % pair for pair in tok_spec) >> + gettok = re.compile(tok_re).match >> + line = 1 >> + pos = line_start = 0 >> + mo = gettok(s) >> + while mo is not None: >> + typ = mo.lastgroup >> + if typ == 'NEWLINE': >> + line_start = pos >> + line += 1 >> + elif typ != 'SKIP': >> + yield Token(typ, mo.group(typ), line, mo.start()-line_start) >> + pos = mo.end() >> + mo = gettok(s, pos) >> + if pos != len(s): >> + raise RuntimeError('Unexpected character %r on line %d' %(s[pos], line)) >> + >> +>>> statements = '''\ >> + total := total + price * quantity; >> + tax := price * 0.05; >> + ''' >> +>>> for token in tokenize(statements): >> + ... print(token) >> + ... >> + Token(typ='ID', value='total', line=1, column=8) >> + Token(typ='ASSIGN', value=':=', line=1, column=14) >> + Token(typ='ID', value='total', line=1, column=17) >> + Token(typ='OP', value='+', line=1, column=23) >> + Token(typ='ID', value='price', line=1, column=25) >> + Token(typ='OP', value='*', line=1, column=31) >> + Token(typ='ID', value='quantity', line=1, column=33) >> + Token(typ='END', value=';', line=1, column=41) >> + Token(typ='ID', value='tax', line=2, column=9) >> + Token(typ='ASSIGN', value=':=', line=2, column=13) >> + Token(typ='ID', value='price', line=2, column=16) >> + Token(typ='OP', value='*', line=2, column=22) >> + Token(typ='NUMBER', value='0.05', line=2, column=24) >> + Token(typ='END', value=';', line=2, column=28) > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From guido at python.org Thu Sep 16 19:21:33 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 16 Sep 2010 10:21:33 -0700 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: <20100916154207.GS3086@unaka.lan> References: <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd12a@mission> <20100916154207.GS3086@unaka.lan> Message-ID: On Thu, Sep 16, 2010 at 8:42 AM, Toshio Kuratomi wrote: > On Thu, Sep 16, 2010 at 09:52:48AM -0400, Barry Warsaw wrote: >> On Sep 16, 2010, at 11:28 PM, Nick Coghlan wrote: >> >> >There are some APIs that should be able to handle bytes *or* strings, >> >but the current use of string literals in their implementation means >> >that bytes don't work. This turns out to be a PITA for some networking >> >related code which really wants to be working with raw bytes (e.g. >> >URLs coming off the wire). >> >> Note that email has exactly the same problem. ?A general solution -- even if >> embodied in *well documented* best-practices and convention -- would really >> help make the stdlib work consistently, and I bet third party libraries too. >> > I too await a solution with abated breath :-) I've been working on > documenting best practices for APIs and Unicode and for this type of > function (take bytes or unicode and output the same type), knowing the > encoding is seems like a requirement in most cases: > > http://packages.python.org/kitchen/designing-unicode-apis.html#take-either-bytes-or-unicode-output-the-same-type > > I'd love to add another strategy there that shows how you can robustly > operate on bytes without knowing the encoding but from writing that, I think > that anytime you simplify your API you have to accept limitations on the > data you can take in. ?(For instance, some simplifications can handle > anything except ASCII-incompatible encodings). In all cases I can imagine where such polymorphic functions make sense, the necessary and sufficient assumption should be that the encoding is a superset of 7-bit(*) ASCII. This includes UTF-8, all Latin-N variant, and AFAIK also the popular CJK encodings other than UTF-16. This is the same assumption made by Python's byte type when you use "character-based" methods like lower(). --Guido __________ (*) In my mind ASCII and 7-bit are synonymous, but unfortunately there are droves of naive users who believe that ASCII includes all 256 possible 8-bit bytes using some encoding -- typically the default encoding of their DOS or Windows box. :-( -- --Guido van Rossum (python.org/~guido) From gzlist at googlemail.com Thu Sep 16 19:46:22 2010 From: gzlist at googlemail.com (Martin (gzlist)) Date: Thu, 16 Sep 2010 18:46:22 +0100 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd12a@mission> <20100916154207.GS3086@unaka.lan> Message-ID: On 16/09/2010, Guido van Rossum wrote: > > In all cases I can imagine where such polymorphic functions make > sense, the necessary and sufficient assumption should be that the > encoding is a superset of 7-bit(*) ASCII. This includes UTF-8, all > Latin-N variant, and AFAIK also the popular CJK encodings other than > UTF-16. This is the same assumption made by Python's byte type when > you use "character-based" methods like lower(). Well, depends on what exactly you're doing, it's pretty easy to go wrong: Python 3.2a2+ (py3k, Sep 16 2010, 18:43:45) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import os, sys >>> os.path.split("C:\\?") ('C:\\', '?') >>> os.path.split("C:\\?".encode(sys.getfilesystemencoding())) (b'C:\\\x8f', b'') Similar things can catch out web developers once they step outside the percent encoding. Martin From guido at python.org Thu Sep 16 19:56:56 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 16 Sep 2010 10:56:56 -0700 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd12a@mission> <20100916154207.GS3086@unaka.lan> Message-ID: On Thu, Sep 16, 2010 at 10:46 AM, Martin (gzlist) wrote: > On 16/09/2010, Guido van Rossum wrote: >> >> In all cases I can imagine where such polymorphic functions make >> sense, the necessary and sufficient assumption should be that the >> encoding is a superset of 7-bit(*) ASCII. This includes UTF-8, all >> Latin-N variant, and AFAIK also the popular CJK encodings other than >> UTF-16. This is the same assumption made by Python's byte type when >> you use "character-based" methods like lower(). > > Well, depends on what exactly you're doing, it's pretty easy to go wrong: > > Python 3.2a2+ (py3k, Sep 16 2010, 18:43:45) [MSC v.1500 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. >>>> import os, sys >>>> os.path.split("C:\\?") > ('C:\\', '?') >>>> os.path.split("C:\\?".encode(sys.getfilesystemencoding())) > (b'C:\\\x8f', b'') > > Similar things can catch out web developers once they step outside the > percent encoding. Well, that character is not 7-bit ASCII. Of course things will go wrong there. That's the whole point of what I said, isn't it? -- --Guido van Rossum (python.org/~guido) From pje at telecommunity.com Thu Sep 16 20:11:37 2010 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 16 Sep 2010 14:11:37 -0400 Subject: [Python-Dev] standards for distribution names In-Reply-To: <4C91FACB.3010300@simplistix.co.uk> References: <4C91FACB.3010300@simplistix.co.uk> Message-ID: <20100916181137.563DB3A403D@sparrow.telecommunity.com> At 12:08 PM 9/16/2010 +0100, Chris Withers wrote: >Following on from this question: > >http://twistedmatrix.com/pipermail/twisted-python/2010-September/022877.html > >...I'd thought that the "correct names" for distributions would have >been documented in one of: > >... > >Where are the standards for this or is it still a case of "whatever >setuptools does"? Actually, in this case, it's "whatever distutils does". If you don't build your .exe's with Distutils, or if you rename them after the fact, then setuptools won't recognize them as things it can consume. FYI, Twisted has a long history of releasing distribution files that are either built using non-distutils tools or else renamed after being built. Note, too, that if the Windows exe's they're providing aren't built by the distutils bdist_wininst command, then setuptools is probably not going to be able to consume them, no matter what they're called. From a.badger at gmail.com Thu Sep 16 20:16:53 2010 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 16 Sep 2010 14:16:53 -0400 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd12a@mission> <20100916154207.GS3086@unaka.lan> Message-ID: <20100916181653.GT3086@unaka.lan> On Thu, Sep 16, 2010 at 10:56:56AM -0700, Guido van Rossum wrote: > On Thu, Sep 16, 2010 at 10:46 AM, Martin (gzlist) wrote: > > On 16/09/2010, Guido van Rossum wrote: > >> > >> In all cases I can imagine where such polymorphic functions make > >> sense, the necessary and sufficient assumption should be that the > >> encoding is a superset of 7-bit(*) ASCII. This includes UTF-8, all > >> Latin-N variant, and AFAIK also the popular CJK encodings other than > >> UTF-16. This is the same assumption made by Python's byte type when > >> you use "character-based" methods like lower(). > > > > Well, depends on what exactly you're doing, it's pretty easy to go wrong: > > > > Python 3.2a2+ (py3k, Sep 16 2010, 18:43:45) [MSC v.1500 32 bit (Intel)] on win32 > > Type "help", "copyright", "credits" or "license" for more information. > >>>> import os, sys > >>>> os.path.split("C:\\?") > > ('C:\\', '?') > >>>> os.path.split("C:\\?".encode(sys.getfilesystemencoding())) > > (b'C:\\\x8f', b'') > > > > Similar things can catch out web developers once they step outside the > > percent encoding. > > Well, that character is not 7-bit ASCII. Of course things will go > wrong there. That's the whole point of what I said, isn't it? > You were talking about encodings that were supersets of 7-bit ASCII. I think Martin was demonstrating a byte string that was a superset of 7-bit ASCII being fed to a stdlib function which went wrong. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From pje at telecommunity.com Thu Sep 16 20:22:12 2010 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 16 Sep 2010 14:22:12 -0400 Subject: [Python-Dev] standards for distribution names In-Reply-To: <4C91FACB.3010300@simplistix.co.uk> References: <4C91FACB.3010300@simplistix.co.uk> Message-ID: <20100916182212.D97173A403D@sparrow.telecommunity.com> At 12:08 PM 9/16/2010 +0100, Chris Withers wrote: >...I'd thought that the "correct names" for distributions would have >been documented in one of: > >http://www.python.org/dev/peps/pep-0345 >http://www.python.org/dev/peps/pep-0376 >http://www.python.org/dev/peps/pep-0386 > >...but having read them, I drew a blank. Forgot to mention: see distinfo_dirname() in PEP 376 for an explanation of distribution-name normalization. (Case-insensitivity and os-specific case handling is not addressed in the PEPs, though, AFAICT.) From tom.browder at gmail.com Thu Sep 16 20:41:32 2010 From: tom.browder at gmail.com (Tom Browder) Date: Thu, 16 Sep 2010 13:41:32 -0500 Subject: [Python-Dev] Python 2.7 Won't Build Message-ID: I am trying to rebujild the 2.7 maintenance branch and get this error on Ubuntu 10.04.1 LTS: XXX lineno: 743, opcode: 0 Traceback (most recent call last): File "/usr/local/src/python-2.7-maint-svn/Lib/site.py", line 62, in import os File "/usr/local/src/python-2.7-maint-svn/Lib/os.py", line 743, in def urandom(n): SystemError: unknown opcode I installed it successfully once so I may be getting conflicts, but I can't figure out why. There were some similar bugs reported in previous versions but I didn't see a clear solution. I have done "make distclean" and "./configure". I have unset my PYTHONPATH and LD_LIBRARY_PATH, but python2.7 is my default python. I guess my next step will be to manually remove the installed python 2.7 unless I hear some other suggestions. And I will file a bug report soon unless that is inappropriate. Thanks, -Tom Thomas M. Browder, Jr. Niceville, Florida USA From brett at python.org Thu Sep 16 20:45:42 2010 From: brett at python.org (Brett Cannon) Date: Thu, 16 Sep 2010 11:45:42 -0700 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> Message-ID: On Thu, Sep 16, 2010 at 06:28, Nick Coghlan wrote: > On Thu, Sep 16, 2010 at 10:26 PM, Antoine Pitrou wrote: >> Why won't you feel confident? Are there any specific issues (apart from >> the lack of a WSGI PEP)? >> If they are technical problems, they should be reported on the bug >> tracker. >> If they are representational, cultural or psychological issues, I'm >> not sure what we can do. But delaying the release won't solve them. > > There are some APIs that should be able to handle bytes *or* strings, > but the current use of string literals in their implementation means > that bytes don't work. This turns out to be a PITA for some networking > related code which really wants to be working with raw bytes (e.g. > URLs coming off the wire). > > For example: > >>>> import urllib.parse as parse >>>> parse.urlsplit("http://www.ubuntu.com") > SplitResult(scheme='http', netloc='www.ubuntu.com', path='', query='', > fragment='') >>>> parse.urlsplit(b"http://www.ubuntu.com") > Traceback (most recent call last): > ?File "", line 1, in > ?File "/home/ncoghlan/devel/py3k/Lib/urllib/parse.py", line 178, in urlsplit > ? ?i = url.find(':') > TypeError: expected an object with the buffer interface > > There's no real reason urlsplit (and similar urllib.parse APIs) > shouldn't support bytes, but the internal use of string literals > currently prevents it. > > We don't seem to have created a tracker issue from the discussion back > in June where this came up, so I went ahead and created one just now: > http://bugs.python.org/issue9873 When I do my two months of PSF-sponsored core work (expected to be Jan/Feb) I was planning on (finally) redoing the dev docs, writing a HOWTO for maintaining a Python 2/3 code base, and cleaning up the test suite. But I am starting to think I should change the last one to solving this polymorphism problem in a way that can be applied across the board in the stdlib. > > I think there were other APIs mentioned back then beyond the > urllib.parse ones, but I didn't find them when I went trawling through > the list archives yesterday. If anyone else thinks of any APIs that > should allow bytes as well as strings (or vice-versa) feel free to add > them to that issue. Or create separate issues and make them dependencies for issue9873. From brett at python.org Thu Sep 16 20:48:23 2010 From: brett at python.org (Brett Cannon) Date: Thu, 16 Sep 2010 11:48:23 -0700 Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: References: Message-ID: Go ahead and file the bug, but chances are that some other installed Python is executing the code and picking up the .pyc files which have bytecode new to Python 2.7. On Thu, Sep 16, 2010 at 11:41, Tom Browder wrote: > I am trying to rebujild the 2.7 maintenance branch and get this error > on Ubuntu 10.04.1 LTS: > > XXX lineno: 743, opcode: 0 > Traceback (most recent call last): > ?File "/usr/local/src/python-2.7-maint-svn/Lib/site.py", line 62, in > ? import os > ?File "/usr/local/src/python-2.7-maint-svn/Lib/os.py", line 743, in > ? def urandom(n): > SystemError: unknown opcode > > I installed it successfully once so I may be getting conflicts, but I > can't figure out why. ?There were some similar bugs reported in > previous versions but I didn't see a clear solution. > > I have done "make distclean" and "./configure". ?I have unset my > PYTHONPATH and LD_LIBRARY_PATH, but python2.7 is my default python. > > I guess my next step will be to manually remove the installed python > 2.7 unless I hear some other suggestions. > > And I will file a bug report soon unless that is inappropriate. > > Thanks, > > -Tom > > Thomas M. Browder, Jr. > Niceville, Florida > USA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From tom.browder at gmail.com Thu Sep 16 21:05:38 2010 From: tom.browder at gmail.com (Tom Browder) Date: Thu, 16 Sep 2010 14:05:38 -0500 Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: References: Message-ID: On Thu, Sep 16, 2010 at 13:48, Brett Cannon wrote: > Go ahead and file the bug, but chances are that some other installed > Python is executing the code and picking up the .pyc files which have > bytecode new to Python 2.7. But isn't that a problem with the build system? It seems to me it should be using all modules from within the build, thus there should be no such error. Regards, -Tom From jacob at jacobian.org Thu Sep 16 21:07:01 2010 From: jacob at jacobian.org (Jacob Kaplan-Moss) Date: Thu, 16 Sep 2010 14:07:01 -0500 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> Message-ID: On Thu, Sep 16, 2010 at 9:59 AM, Paul Moore wrote: > On 16 September 2010 07:16, Terry Reedy wrote: >>> I'm not working to get Django running on Python 3.1 because I don't >>> feel confident I'll be able to put any apps I write into production. >> >> Why not? Since the I/O speed problem is fixed, I have no idea what you are >> referring to. ?Please do be concrete. > > At the risk of putting words into Jacob's mouth, I understood him to > mean that "production quality" WSGI servers either do not exist, or do > not implement a consistently defined spec (i.e., everyone is doing > their own thing to adapt WSGI to Python 3). Yup, exactly. Deploying web apps under Python 2 right now is actually pretty awesome. There's a clear leader in mod_wsgi that's fast, stable, easy to use, and under active development. There's a few great lightweight pure-Python servers, some new-hotness (Gunicorn) and some tried-and-true (CherryPy). There's a fast-as-hell bleeding-edge option (nginx + uwsgi). And those are just the ones I've successfully put into production -- there're still *more* options if one of those won't cut it. The key here is that switching between all of these deployment situations is *incredibly* easy. Actually, this very afternoon I'm planning to experiment with a switch from mod_wsgi to gunicon. I'm confident enough with the inter-op that I'm going to make the switch on a production web server, monitor it for a bit, then switch back. I've budgeted an hour for this, and I'll probably end up spending half that time playing Minecraft while I gather statistics. Python 3 offers me none of this. I don't have a wide variety of tools to choose from. Worse, I don't even have a guarantee of interoperability between the tools that *do* exist. --- I'm sorry if I'm coming across as a complainer here. It's a frustrating situation for me: I want to start using Python 3, but until there's a working web stack waiting for me I just can't justify the time. And unfortunately I'm just not familiar enough with the problem(s) to have any real shot at working towards a solution, and I'm *certainly* not enough of an expert to work on a PEP or spec. So all I can really do is agitate. Jacob From guido at python.org Thu Sep 16 21:21:13 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 16 Sep 2010 12:21:13 -0700 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: <20100916181653.GT3086@unaka.lan> References: <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd12a@mission> <20100916154207.GS3086@unaka.lan> <20100916181653.GT3086@unaka.lan> Message-ID: On Thu, Sep 16, 2010 at 11:16 AM, Toshio Kuratomi wrote: > On Thu, Sep 16, 2010 at 10:56:56AM -0700, Guido van Rossum wrote: >> On Thu, Sep 16, 2010 at 10:46 AM, Martin (gzlist) wrote: >> > On 16/09/2010, Guido van Rossum wrote: >> >> >> >> In all cases I can imagine where such polymorphic functions make >> >> sense, the necessary and sufficient assumption should be that the >> >> encoding is a superset of 7-bit(*) ASCII. This includes UTF-8, all >> >> Latin-N variant, and AFAIK also the popular CJK encodings other than >> >> UTF-16. This is the same assumption made by Python's byte type when >> >> you use "character-based" methods like lower(). >> > >> > Well, depends on what exactly you're doing, it's pretty easy to go wrong: >> > >> > Python 3.2a2+ (py3k, Sep 16 2010, 18:43:45) [MSC v.1500 32 bit (Intel)] on win32 >> > Type "help", "copyright", "credits" or "license" for more information. >> >>>> import os, sys >> >>>> os.path.split("C:\\?") >> > ('C:\\', '?') >> >>>> os.path.split("C:\\?".encode(sys.getfilesystemencoding())) >> > (b'C:\\\x8f', b'') >> > >> > Similar things can catch out web developers once they step outside the >> > percent encoding. >> >> Well, that character is not 7-bit ASCII. Of course things will go >> wrong there. That's the whole point of what I said, isn't it? >> > You were talking about encodings that were supersets of 7-bit ASCII. > I think Martin was demonstrating a byte string that was a superset of 7-bit > ASCII being fed to a stdlib function which went wrong. Whoops, sorry. I don't have access to Windows so I can't reproduce this though. I also don't understand it. What is the Unicode codepoint for that ? character? What is sys.getfilesystemencoding()? What is the value of "C:\\?".encode(sys.getfilesystemencoding())? -- --Guido van Rossum (python.org/~guido) From brett at python.org Thu Sep 16 21:28:37 2010 From: brett at python.org (Brett Cannon) Date: Thu, 16 Sep 2010 12:28:37 -0700 Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: References: Message-ID: Please file the bug and it can be discussed further there. On Thu, Sep 16, 2010 at 12:05, Tom Browder wrote: > On Thu, Sep 16, 2010 at 13:48, Brett Cannon wrote: >> Go ahead and file the bug, but chances are that some other installed >> Python is executing the code and picking up the .pyc files which have >> bytecode new to Python 2.7. > > But isn't that a problem with the build system? ?It seems to me it > should be using all modules from within the build, thus there should > be no such error. > > Regards, > > -Tom > From barry at python.org Thu Sep 16 21:36:24 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 16 Sep 2010 15:36:24 -0400 Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: References: Message-ID: <20100916153624.52d98c4e@mission> On Sep 16, 2010, at 01:41 PM, Tom Browder wrote: >I am trying to rebujild the 2.7 maintenance branch and get this error >on Ubuntu 10.04.1 LTS: I just tried this on my vanilla 10.04.1 system. I checked out release27-maint ran configure && make. It built without problem. >XXX lineno: 743, opcode: 0 >Traceback (most recent call last): > File "/usr/local/src/python-2.7-maint-svn/Lib/site.py", line 62, in > import os > File "/usr/local/src/python-2.7-maint-svn/Lib/os.py", line 743, in > def urandom(n): >SystemError: unknown opcode > >I installed it successfully once so I may be getting conflicts, but I >can't figure out why. There were some similar bugs reported in >previous versions but I didn't see a clear solution. I installed Python 2.7 to /usr/local, then did a make distclean, configure, make. Again, successfully. >I have done "make distclean" and "./configure". I have unset my >PYTHONPATH and LD_LIBRARY_PATH, but python2.7 is my default python. > >I guess my next step will be to manually remove the installed python >2.7 unless I hear some other suggestions. When you say "installed python 2.7" do you mean the one you installed to /usr/local from a from-source build, or something else (e.g. a Python 2.7 package perhaps)? >And I will file a bug report soon unless that is inappropriate. Sure. Please +nosy me. But I think something else is going on. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From gzlist at googlemail.com Thu Sep 16 21:43:25 2010 From: gzlist at googlemail.com (Martin (gzlist)) Date: Thu, 16 Sep 2010 20:43:25 +0100 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd12a@mission> <20100916154207.GS3086@unaka.lan> <20100916181653.GT3086@unaka.lan> Message-ID: On 16/09/2010, Guido van Rossum wrote: > On Thu, Sep 16, 2010 at 11:16 AM, Toshio Kuratomi > wrote: >> You were talking about encodings that were supersets of 7-bit ASCII. >> I think Martin was demonstrating a byte string that was a superset of >> 7-bit >> ASCII being fed to a stdlib function which went wrong. > > Whoops, sorry. I don't have access to Windows so I can't reproduce > this though. I also don't understand it. What is the Unicode codepoint > for that ? character? What is sys.getfilesystemencoding()? What is the > value of "C:\\?".encode(sys.getfilesystemencoding())? My fault, should have been clearer. I was trying to demonstrate that there's a difference between the unix-friendly encodings like UTF-8 and the EUC codecs which only use high-bit characters for non-ascii text, and the ISO-2022 codecs and Shift JIS. In the example I gave, ? encodes in CP932 as '\x8f\\', and the function gets confused by the second byte. Obviously the right answer there is just to use unicode, rather than write a function that works with weird multibyte codecs. Martin From tom.browder at gmail.com Thu Sep 16 21:56:25 2010 From: tom.browder at gmail.com (Tom Browder) Date: Thu, 16 Sep 2010 14:56:25 -0500 Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: <20100916153624.52d98c4e@mission> References: <20100916153624.52d98c4e@mission> Message-ID: On Thu, Sep 16, 2010 at 14:36, Barry Warsaw wrote: > On Sep 16, 2010, at 01:41 PM, Tom Browder wrote: > >>I am trying to rebujild the 2.7 maintenance branch and get this error >>on Ubuntu 10.04.1 LTS: > > I just tried this on my vanilla 10.04.1 system. ?I checked out release27-maint > ran configure && make. ?It built without problem. > >>XXX lineno: 743, opcode: 0 >>Traceback (most recent call last): >> File "/usr/local/src/python-2.7-maint-svn/Lib/site.py", line 62, in >> import os >> File "/usr/local/src/python-2.7-maint-svn/Lib/os.py", line 743, in >> def urandom(n): >>SystemError: unknown opcode ... > When you say "installed python 2.7" do you mean the one you installed to > /usr/local from a from-source build, or something else (e.g. a Python 2.7 > package perhaps)? It was the released source tarball for 2.7, and I get the same error when I try it from that directory. -Tom Thomas M. Browder, Jr. Niceville, Florida USA From steve at holdenweb.com Thu Sep 16 22:12:15 2010 From: steve at holdenweb.com (Steve Holden) Date: Thu, 16 Sep 2010 16:12:15 -0400 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> Message-ID: <4C927A1F.6020304@holdenweb.com> On 9/16/2010 3:07 PM, Jacob Kaplan-Moss wrote: > On Thu, Sep 16, 2010 at 9:59 AM, Paul Moore wrote: >> On 16 September 2010 07:16, Terry Reedy wrote: >>>> I'm not working to get Django running on Python 3.1 because I don't >>>> feel confident I'll be able to put any apps I write into production. >>> >>> Why not? Since the I/O speed problem is fixed, I have no idea what you are >>> referring to. Please do be concrete. >> >> At the risk of putting words into Jacob's mouth, I understood him to >> mean that "production quality" WSGI servers either do not exist, or do >> not implement a consistently defined spec (i.e., everyone is doing >> their own thing to adapt WSGI to Python 3). > > Yup, exactly. > > Deploying web apps under Python 2 right now is actually pretty > awesome. There's a clear leader in mod_wsgi that's fast, stable, easy > to use, and under active development. There's a few great lightweight > pure-Python servers, some new-hotness (Gunicorn) and some > tried-and-true (CherryPy). There's a fast-as-hell bleeding-edge option > (nginx + uwsgi). And those are just the ones I've successfully put > into production -- there're still *more* options if one of those won't > cut it. > > The key here is that switching between all of these deployment > situations is *incredibly* easy. Actually, this very afternoon I'm > planning to experiment with a switch from mod_wsgi to gunicon. I'm > confident enough with the inter-op that I'm going to make the switch > on a production web server, monitor it for a bit, then switch back. > > I've budgeted an hour for this, and I'll probably end up spending half > that time playing Minecraft while I gather statistics. > > Python 3 offers me none of this. I don't have a wide variety of tools > to choose from. Worse, I don't even have a guarantee of > interoperability between the tools that *do* exist. > > --- > > I'm sorry if I'm coming across as a complainer here. It's a > frustrating situation for me: I want to start using Python 3, but > until there's a working web stack waiting for me I just can't justify > the time. And unfortunately I'm just not familiar enough with the > problem(s) to have any real shot at working towards a solution, and > I'm *certainly* not enough of an expert to work on a PEP or spec. So > all I can really do is agitate. > I think you are entitled to describe real-world use cases that Python 3 needs to start solving to be accepted as production-ready. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From steve at holdenweb.com Thu Sep 16 22:12:15 2010 From: steve at holdenweb.com (Steve Holden) Date: Thu, 16 Sep 2010 16:12:15 -0400 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> Message-ID: <4C927A1F.6020304@holdenweb.com> On 9/16/2010 3:07 PM, Jacob Kaplan-Moss wrote: > On Thu, Sep 16, 2010 at 9:59 AM, Paul Moore wrote: >> On 16 September 2010 07:16, Terry Reedy wrote: >>>> I'm not working to get Django running on Python 3.1 because I don't >>>> feel confident I'll be able to put any apps I write into production. >>> >>> Why not? Since the I/O speed problem is fixed, I have no idea what you are >>> referring to. Please do be concrete. >> >> At the risk of putting words into Jacob's mouth, I understood him to >> mean that "production quality" WSGI servers either do not exist, or do >> not implement a consistently defined spec (i.e., everyone is doing >> their own thing to adapt WSGI to Python 3). > > Yup, exactly. > > Deploying web apps under Python 2 right now is actually pretty > awesome. There's a clear leader in mod_wsgi that's fast, stable, easy > to use, and under active development. There's a few great lightweight > pure-Python servers, some new-hotness (Gunicorn) and some > tried-and-true (CherryPy). There's a fast-as-hell bleeding-edge option > (nginx + uwsgi). And those are just the ones I've successfully put > into production -- there're still *more* options if one of those won't > cut it. > > The key here is that switching between all of these deployment > situations is *incredibly* easy. Actually, this very afternoon I'm > planning to experiment with a switch from mod_wsgi to gunicon. I'm > confident enough with the inter-op that I'm going to make the switch > on a production web server, monitor it for a bit, then switch back. > > I've budgeted an hour for this, and I'll probably end up spending half > that time playing Minecraft while I gather statistics. > > Python 3 offers me none of this. I don't have a wide variety of tools > to choose from. Worse, I don't even have a guarantee of > interoperability between the tools that *do* exist. > > --- > > I'm sorry if I'm coming across as a complainer here. It's a > frustrating situation for me: I want to start using Python 3, but > until there's a working web stack waiting for me I just can't justify > the time. And unfortunately I'm just not familiar enough with the > problem(s) to have any real shot at working towards a solution, and > I'm *certainly* not enough of an expert to work on a PEP or spec. So > all I can really do is agitate. > I think you are entitled to describe real-world use cases that Python 3 needs to start solving to be accepted as production-ready. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From merwok at netwok.org Thu Sep 16 22:18:30 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 16 Sep 2010 22:18:30 +0200 Subject: [Python-Dev] 3.x as the official release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <19599.20212.241866.713348@montanaro.dyndns.org> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100915191812.5D0663A4116@sparrow.telecommunity.com> Message-ID: <4C927B96.6090903@netwok.org> Le 15/09/2010 21:45, Tarek Ziad? a ?crit : > Could we remove in any case the wsgiref.egg-info file ? Since we've > been working on a new format for that (PEP 376), that should be > starting to get used in the coming years, it'll be a bit of a > non-sense to have that metadata file in the sdtlib shipped with 3,2 On a related subject: Would it make sense not to run install_egg_info from install anymore? We probably can?t remove the command because of backward compat, but we could stop running it (thus creating egg-info files) by default. Regards From barry at python.org Thu Sep 16 22:26:26 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 16 Sep 2010 16:26:26 -0400 Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: References: <20100916153624.52d98c4e@mission> Message-ID: <20100916162626.42c39b60@mission> On Sep 16, 2010, at 02:56 PM, Tom Browder wrote: >On Thu, Sep 16, 2010 at 14:36, Barry Warsaw wrote: >> When you say "installed python 2.7" do you mean the one you >> installed to /usr/local from a from-source build, or something else >> (e.g. a Python 2.7 package perhaps)? > >It was the released source tarball for 2.7, and I get the same error >when I try it from that directory. Yep, sorry, I still cannot reproduce it. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From pje at telecommunity.com Thu Sep 16 22:38:56 2010 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 16 Sep 2010 16:38:56 -0400 Subject: [Python-Dev] 3.x as the official release Message-ID: <20100916203856.81D073A403D@sparrow.telecommunity.com> At 10:18 PM 9/16/2010 +0200, ??ric Araujo wrote: >Le 15/09/2010 21:45, Tarek Ziad?? a ??crit : > Could we remove in >any case the wsgiref.egg-info file ? Since we've > been working on a >new format for that (PEP 376), that should be > starting to get used >in the coming years, it'll be a bit of a > non-sense to have that >metadata file in the sdtlib shipped with 3,2 On a related subject: >Would it make sense not to run install_egg_info from install >anymore? We probably can???t remove the command because of backward >compat, but we could stop running it (thus creating egg-info files) by default. If you're talking about distutils2 on Python 3, then of course anything goes: backward compatibility isn't an issue. For 2.x, not writing the files would indeed produce backward compatibility problems. From rdmurray at bitdance.com Thu Sep 16 22:51:58 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 16 Sep 2010 16:51:58 -0400 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <20100916174053.007c783f@pitrou.net> References: <20100914121744.31ef5aaf@pitrou.net> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> <20100916174053.007c783f@pitro u.net> Message-ID: <20100916205158.5098B1BF4E1@kimball.webabinitio.net> On Thu, 16 Sep 2010 17:40:53 +0200, Antoine Pitrou wrote: > On Thu, 16 Sep 2010 11:30:12 -0400 > "R. David Murray" wrote: > > > > And then BaseHeader uses self.lit.colon, etc, when manipulating strings. > > It also has to use slice notation rather than indexing when looking at > > individual characters, which is a PITA but not terrible. > > > > I'm not saying this is the best approach, since this is all experimental > > code at the moment, but it is *an* approach.... > > Out of curiousity, can you explain why polymorphism is needed for > e-mail? I would assume that headers are bytes until they are parsed, at > which point they become a pair of unicode strings (one for the header > name and one for its value). Currently email accepts strings as input, and produces strings as output. It needs to also accept bytes as input, and emit bytes as output, because unicode can only be used as a 7-bit clean data transmission channel, and that's too restrictive for many email applications (many of which need to deal with "dirty" (non-RFC conformant) 8bit data. [1] Backward compatibility says "case closed". If we were designing from scratch, we could insist that input to the parser is always bytes, and when the model is serialized it always produces bytes. It is possible that one could live with that, but I don't think it is optimal. Given a message, there are many times you want to serialize it as text (for example, for presentation in a UI). You could provide alternate serialization methods to get text out on demand....but then what if someone wants to push that text representation back in to email to rebuild a model of the message? So now we have both a bytes parser and a string parser. What do we store in the model? We could say that the model is always text. But then we lose information about the original bytes message, and we can't reproduce it. For various reasons (mailman being a big one), this is not acceptable. So we could say that the model is always bytes. But we want access to (for example) the header values as text, so header lookup should take string keys and return string values[2]. But for certain types of processing, particularly examination of "dirty", non-RFC conforming input data, you need to be able to access the raw bytes data. What about email files on disk? They could be bytes, or they could be, effectively, text (for example, utf-8 encoded). On disk, using utf-8, one might store the text representation of the message, rather than the wire-format (ASCII encoded) version. We might want to write such messages from scratch. As I said above, we could insist that files on disk be in wire-format, and for many applications that would work fine, but I think people would get mad at us if didn't support text files[3]. So, after much discussion, what we arrived at (so far!) is a model that mimics the Python3 split between bytes and strings. If you start with bytes input, you end up with a BytesMessage object. If you start with string input to the parser, you end up with a StringMessage. If you have a BytesMessage and you want to do something with the text version of the message, you decode it: print(mymsg.decode()) If the message is RFC conformant, the message contains all the information needed to decode it correctly. If its not conformant, email does the best it can and registers defects for the non-conformant bits (or, optionally, email6 will raise errors when the policy is set to strict). If you have a StringMessage and you want to use it where wire-format is needed, you encode it: outmsg = mymsg.encode() smtpserver.sendmail( bytes(outmsg['from']), [bytes(x) for x in itertools.chain( outmsg['to'], outmsg['cc'], outmsg['bcc'])], outmsg.serialize(policy=email.policy.SMTP)) Encoding uses the utf-8 character set by default, but this can be modified by changing the policy. The trick for gathering the list of addresses is how I *think* that part of the API is going to work: iterating the object that models an address header gives you a list of address objects, and converting one of those to a bytes string gives you the wire-format byte string representing a single address. Also note that this is the new API; in compatibility mode (which is controlled by the policy) you'd get the old behavior of just getting the string representation of the whole header back (but then you'd have to parse it to turn it into a list of addresses). The point here is that because we've encoded the message to a BytesMessage, what we get when we turn the pieces into a bytes string are the wire-format byte strings that are required for transmission; for example, non-ASCII characters will be encoded according to the policy and then RFC2047 transfer encoded as needed. At this point you may notice there's a problem with the example above. We actually need to decode each of those byte strings using the ASCII codec before passing them as arguments to smtplib, since smtplib in Python3 expects string arguments. If smtplib were polymorphic we could pass in the bytes strings directly. In that case if a string were passed in instead, smtplib could call some utility routines from email to encode the text into bytes using the RFC2047 conventions. As it stands now, there's no *easy* way for a user program to construct a list of addresses that require RFC2047 encoding and pass it to smtplib. (This last item is just as much a problem in Python2, by the way.) This is probably not the right thing to do, though, because that isn't the kind of polymorphism we're talking about. When accepting input to sendmail, smtplib is always bytes out, so having it accept both bytes and strings as input is probably wrong[4]. Especially since the message body needs to be passed in in wire-format, because smtplib should not have to know how to convert text into wire-format...that's the email module's job. Instead smtplib could take a Message object as input, and do that serialize call itself. In which case it could also figure out the addresses by itself, and/or accept email address objects for the from and to parameters. You can see what a can of worms this stuff is :) This is what I meant about carefully examining the API contract before blindly providing polymorphism. For email, a wire-format bytes string *contains* encoding information, and you have to stay aware of that as you redesign the bytes/string interface. Anyway, what polymorphism means in email is that if you put in bytes, you get a BytesMessage, if you put in strings you get a StringMessage, and if you want the other one you convert. I'm giving consideration to additional polymorphism, such as having the use of a key of a particular type return a value of that type. That is, looking up the subject by the key 'subject' would get you a StringHeader regardless of whether you were looking it up in a BytesMessagge or a StringMessage. But I'm still thinking about whether or not that is a good idea, I need to write up some more example code to convince myself one way or another. The sendmail example above is an example on the "no" side: you'll note that in that example the natural thing to do was to use string keys, but get bytes out. Well, that was probably more than you wanted to know or read, but hopefully it will give some perspective on what's involved here. Feedback on any of this is welcome. I've got a hole in my schedule next week that I'm planning on filling with email6 work, so any feedback will all be grist for the mill. Anyone interested should also sign up for the email-sig mailing list and provide feedback when I start posting there again (which, as I said, should be next week). -- R. David Murray www.bitdance.com [1] Now that surrogateesscape exists, one might suppose that strings could be used as an 8bit channel, but that only works if you don't need to *parse* the non-ASCII data, just transmit it. email needs to parse it. In theory email6 could decode to bytes using surrougateescape and then process, but the infrastructure to handle that still looks like what is described above, so it makes more sense to accept bytes directly. [2] actually they the return StringHeader objects, but the principle is the same. [3] note that you can also have 7bit clean wire format messages stored on disk as text. These can be read as text, but my current thought is that you must give them to email as bytes (that is, encode them using the ASCII codec). For email6 in the current design, bytes means wire format, and text/string means fully decoded. [4] unless the strings consist only of 7bit clean wire-format ASCII characters, as is required now. So what we will probably end up with is smtplib.sendmail accepting both bytes and strings for backward compatibility, but string input must continue to be (the equivalent of) the ASCII decode of 7bit clean wire-format data. From tom.browder at gmail.com Thu Sep 16 23:10:22 2010 From: tom.browder at gmail.com (Tom Browder) Date: Thu, 16 Sep 2010 16:10:22 -0500 Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: <20100916162626.42c39b60@mission> References: <20100916153624.52d98c4e@mission> <20100916162626.42c39b60@mission> Message-ID: I'm attempting to file a bug but keep getting: An error has occurred A problem was encountered processing your request. The tracker maintainers have been notified of the problem. -Tom Thomas M. Browder, Jr. Niceville, Florida USA From victor.stinner at haypocalc.com Thu Sep 16 23:36:06 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Thu, 16 Sep 2010 23:36:06 +0200 Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: References: <20100916162626.42c39b60@mission> Message-ID: <201009162336.06353.victor.stinner@haypocalc.com> Le jeudi 16 septembre 2010 23:10:22, Tom Browder a ?crit : > I'm attempting to file a bug but keep getting: File another bug about this bug! -- Victor Stinner http://www.haypocalc.com/ From winston at netwok.org Thu Sep 16 22:42:19 2010 From: winston at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 16 Sep 2010 22:42:19 +0200 Subject: [Python-Dev] 3.x as the official release In-Reply-To: <20100916203831.78C823A403D@sparrow.telecommunity.com> References: <20100914121744.31ef5aaf@pitrou.net> <20100914110433.GA17037@remy> <20100914224525.0489C159FBD@kimball.webabinitio.net> <4C900BE4.2000609@holdenweb.com> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100915191812.5D0663A4116@sparrow.telecommunity.com> <4C927B96.6090903@netwok.org> <20100916203831.78C823A403D@sparrow.telecommunity.com> Message-ID: <4C92812B.3060304@netwok.org> > If you're talking about distutils2 on Python 3, then of course > anything goes: backward compatibility isn't an issue. For 2.x, not > writing the files would indeed produce backward compatibility problems. I was talking about distutils in 3.2 (or in the release where wsgiref.egg-info goes away). install_egg_info.py has already been turned into install_distinfo.py in distutils2, following PEP 376. Thank you for your reply, I withdraw my suggestion. Regards From solipsis at pitrou.net Fri Sep 17 00:05:12 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 17 Sep 2010 00:05:12 +0200 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <20100916205158.5098B1BF4E1@kimball.webabinitio.net> References: <20100914121744.31ef5aaf@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> <20100916174053.007c783f@pitro u.net> <20100916205158.5098B1BF4E1@kimball.webabinitio.net> Message-ID: <20100917000512.66183149@pitrou.net> On Thu, 16 Sep 2010 16:51:58 -0400 "R. David Murray" wrote: > > What do we store in the model? We could say that the model is always > text. But then we lose information about the original bytes message, > and we can't reproduce it. For various reasons (mailman being a big one), > this is not acceptable. So we could say that the model is always bytes. > But we want access to (for example) the header values as text, so header > lookup should take string keys and return string values[2]. Why can't you have both in a single class? If you create the class using a bytes source (a raw message sent by SMTP, for example), the class automatically parses and decodes it to unicode strings; if you create the class using an unicode source (the text body of the e-mail message and the list of recipients, for example), the class automatically creates the bytes representation. (of course all processing can be done lazily for performance reasons) > What about email files on disk? They could be bytes, or they could be, > effectively, text (for example, utf-8 encoded). Such a file can be two things: - the raw encoding of a whole message (including headers, etc.), then it should be fed as a bytes object - the single text body of a hypothetical message, then it should be fed as a unicode object I don't see any possible middle-ground. > On disk, using utf-8, > one might store the text representation of the message, rather than > the wire-format (ASCII encoded) version. We might want to write such > messages from scratch. But then the user knows the encoding (by "user" I mean what/whoever calls the email API) and mentions it to the email package. What I'm having an issue with is that you are talking about a bytes representation and an unicode representation of a message. But they aren't representations of the same things: - if it's a bytes representation, it will be the whole, raw message including envelope / headers (also, MIME sections etc.) - if it's an unicode representation, it will only be a section of the message decodable as such (a text/plain MIME section, for example; or a decoded header value; or even a single e-mail address part of a decoded header) So, there doesn't seem to be any reason for having both a BytesMessage and an UnicodeMessage at the same abstraction level. They are both representing different things at different abstraction levels. I don't see any potential for confusion: raw assembled e-mail message = bytes; decoded text section of a message = unicode. As for the problem of potential "bogus" raw e-mail data (e.g., undecodable headers), well, I guess the library has to make a choice between purity and practicality, or perhaps let the user choose themselves. For example, through a `strict` flag. If `strict` is true, raise an error as soon as a non-decodable byte appears in a header, if `strict` is false, decode it through a default (encoding, errors) convention which can be overriden by the user (a sensible possibility being "utf-8, surrogateescape" to allow for lossless round-tripping). > As I said above, we could insist that files on > disk be in wire-format, and for many applications that would work fine, > but I think people would get mad at us if didn't support text files[3]. Again, this simply seems to be two different abstraction levels: pre-generated raw email messages including headers, or a single text waiting to be embedded in an actual e-mail. > Anyway, what polymorphism means in email is that if you put in bytes, > you get a BytesMessage, if you put in strings you get a StringMessage, > and if you want the other one you convert. And then you have two separate worlds while ultimately the same concepts are underlying. A library accepting BytesMessage will crash when a program wants to give a StringMessage and vice-versa. That doesn't sound very practical. > [1] Now that surrogateesscape exists, one might suppose that strings > could be used as an 8bit channel, but that only works if you don't need > to *parse* the non-ASCII data, just transmit it. Well, you can parse it, precisely. Not only, but it round-trips if you unparse it again: >>> header_bytes = b"From: bogus\xFFname " >>> name, value = header_bytes.decode("utf-8", "surrogateescape").split(":") >>> name 'From' >>> value ' bogus\udcffname ' >>> "{0}:{1}".format(name, value).encode("utf-8", "surrogateescape") b'From: bogus\xffname ' In the end, what I would call a polymorphic best practice is "try to avoid bytes/str polymorphism if your domain is well-defined enough" (which I admit URLs aren't necessarily; but there's no question a single text/XXX e-mail section is text, and a whole assembled e-mail message is bytes). Regards Antoine. From tom.browder at gmail.com Fri Sep 17 00:09:09 2010 From: tom.browder at gmail.com (Tom Browder) Date: Thu, 16 Sep 2010 17:09:09 -0500 Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: <201009162336.06353.victor.stinner@haypocalc.com> References: <20100916162626.42c39b60@mission> <201009162336.06353.victor.stinner@haypocalc.com> Message-ID: USAOn Thu, Sep 16, 2010 at 16:36, Victor Stinner wrote: > Le jeudi 16 septembre 2010 23:10:22, Tom Browder a ?crit : >> I'm attempting to file a bug but keep getting: > > File another bug about this bug! I did, and eventually discovered the problem: I tried to "nosy" Barry as requested by adding his e-mail address, but that causes an error in the tracker. After I finally figured that out, I successfully entered the original bug (and reported it on the "tracker bug"). -Tom Thomas M. Browder, Jr. Niceville, Florida From glyph at twistedmatrix.com Fri Sep 17 00:11:30 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 16 Sep 2010 18:11:30 -0400 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <20100916205158.5098B1BF4E1@kimball.webabinitio.net> References: <20100914121744.31ef5aaf@pitrou.net> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> <20100916174053.007c783f@pitro u.net> <20100916205158.5098B1BF4E1@kimball.webabinitio.net> Message-ID: <4FE9B10E-B45B-4920-9C7A-16FB00CD1C43@twistedmatrix.com> On Sep 16, 2010, at 4:51 PM, R. David Murray wrote: > Given a message, there are many times you want to serialize it as text > (for example, for presentation in a UI). You could provide alternate > serialization methods to get text out on demand....but then what if > someone wants to push that text representation back in to email to > rebuild a model of the message? You tell them "too bad, make some bytes out of that text." Leave it up to the application. Period, the end, it's not the library's job. If you pushed the text out to a 'view message source' UI representation, then the vicissitudes of the system clipboard and other encoding and decoding things may corrupt it in inscrutable ways. You can't fix it. Don't try. > So now we have both a bytes parser and a string parser. Why do so many messages on this subject take this for granted? It's wrong for the email module just like it's wrong for every other package. There are plenty of other (better) ways to deal with this problem. Let the application decide how to fudge the encoding of the characters back into bytes that can be parsed. "In the face of ambiguity, refuse the temptation to guess" and all that. The application has more of an idea of what's going on than the library here, so let it make encoding decisions. Put another way, there's nothing wrong with having a text parser, as long as it just encodes the text according to some known encoding and then parses the bytes :). > So, after much discussion, what we arrived at (so far!) is a model > that mimics the Python3 split between bytes and strings. If you > start with bytes input, you end up with a BytesMessage object. > If you start with string input to the parser, you end up with a > StringMessage. That may be a handy way to deal with some grotty internal implementation details, but having a 'decode()' method is broken. The thing I care about, as a consumer of this API, is that there is a clearly defined "Message" interface, which gives me a uniform-looking place where I can ask for either characters (if I'm displaying them to the user) or bytes (if I'm putting them on the wire). I don't particularly care where those bytes came from. I don't care what decoding tricks were necessary to produce the characters. Now, it may be worthwhile to have specific normalization / debrokenifying methods which deal with specific types of corrupt data from the wire; encoding-guessing, replacement-character insertion or whatever else are fine things to try. It may also be helpful to keep around a list of errors in the message, for inspection. But as we know, there are lots of ways that MIME data can go bad other than encoding, so that's just one variety of error that we might want to keep around. (Looking at later messages as I'm about to post this, I think this all sounds pretty similar to Antoine's suggestions, with respect to keeping the implementation within a single class, and not having BytesMessage/UnicodeMessage at the same abstraction level.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Sep 17 01:34:35 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 16 Sep 2010 19:34:35 -0400 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <4FE9B10E-B45B-4920-9C7A-16FB00CD1C43@twistedmatrix.com> References: <20100914121744.31ef5aaf@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> <20100916174053.007c783f@pitro u.net> <20100916205158.5098B1BF4E1@kimball.webabinitio.net> <4FE9B10E-B45B-4920-9C7A-16FB00CD1C43@twistedmatrix.com> Message-ID: <20100916193435.303de54d@mission> On Sep 16, 2010, at 06:11 PM, Glyph Lefkowitz wrote: >That may be a handy way to deal with some grotty internal >implementation details, but having a 'decode()' method is broken. The >thing I care about, as a consumer of this API, is that there is a >clearly defined "Message" interface, which gives me a uniform-looking >place where I can ask for either characters (if I'm displaying them to >the user) or bytes (if I'm putting them on the wire). I don't >particularly care where those bytes came from. I don't care what >decoding tricks were necessary to produce the characters. But first you have to get to that Message interface. This is why the current email package separates parsing and generating from the representation model. You could conceivably have a parser that rot13's all the payload, or just parses the headers and leaves the payload as a blob of bytes. But the parser tries to be lenient in what it accepts, so that one bad header doesn't cause it to just punt on everything that follows. Instead, it parses what it can and registers a defect on that header, which the application can then reason about, because it has a Message object. If it were to just throw up its hands (i.e. raise an exception), you'd basically be left with a blob of useless crap that will just get /dev/null'd. >Now, it may be worthwhile to have specific normalization / >debrokenifying methods which deal with specific types of corrupt data >from the wire; encoding-guessing, replacement-character insertion or >whatever else are fine things to try. It may also be helpful to keep >around a list of errors in the message, for inspection. But as we >know, there are lots of ways that MIME data can go bad other than >encoding, so that's just one variety of error that we might want to >keep around. Right. The middle ground IMO is what the current parser does. It recognizes the problem, registers a defect, and tries to recover, but it doesn't fix the corrupt data. So for example, if you had a valid RFC 2047 encoded Subject but a broken X-Foo header, you'd at least still end up with a Message object. The value of the good headers would be things from which you can get the unicode value, the raw bytes value, parse its parameters, munge it, etc. while the bad header might be something you can only get the raw bytes from. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From glyph at twistedmatrix.com Fri Sep 17 02:10:29 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 16 Sep 2010 20:10:29 -0400 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <20100916193435.303de54d@mission> References: <20100914121744.31ef5aaf@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> <20100916174053.007c783f@pitro u.net> <20100916205158.5098B1BF4E1@kimball.webabinitio.net> <4FE9B10E-B45B-4920-9C7A-16FB00CD1C43@twistedmatrix.com> <20100916193435.303de54d@mission> Message-ID: <3AC6C986-6C7F-4135-9627-9B95D3325042@twistedmatrix.com> On Sep 16, 2010, at 7:34 PM, Barry Warsaw wrote: > On Sep 16, 2010, at 06:11 PM, Glyph Lefkowitz wrote: > >> That may be a handy way to deal with some grotty internal >> implementation details, but having a 'decode()' method is broken. The >> thing I care about, as a consumer of this API, is that there is a >> clearly defined "Message" interface, which gives me a uniform-looking >> place where I can ask for either characters (if I'm displaying them to >> the user) or bytes (if I'm putting them on the wire). I don't >> particularly care where those bytes came from. I don't care what >> decoding tricks were necessary to produce the characters. > > But first you have to get to that Message interface. This is why the current > email package separates parsing and generating from the representation model. > You could conceivably have a parser that rot13's all the payload, or just > parses the headers and leaves the payload as a blob of bytes. But the parser > tries to be lenient in what it accepts, so that one bad header doesn't cause > it to just punt on everything that follows. Instead, it parses what it can > and registers a defect on that header, which the application can then reason > about, because it has a Message object. If it were to just throw up its hands > (i.e. raise an exception), you'd basically be left with a blob of useless crap > that will just get /dev/null'd. Oh, absolutely. Please don't interpret anything I say as meaning that the email API should not handle broken data. I'm just saying that you should not expect broken data to round-trip through translation to characters and back, any more than you should expect a broken PNG to round-trip through a translation to a 2d array of pixels and back. >> Now, it may be worthwhile to have specific normalization / >> debrokenifying methods which deal with specific types of corrupt data >> from the wire; encoding-guessing, replacement-character insertion or >> whatever else are fine things to try. It may also be helpful to keep >> around a list of errors in the message, for inspection. But as we >> know, there are lots of ways that MIME data can go bad other than >> encoding, so that's just one variety of error that we might want to >> keep around. > > Right. The middle ground IMO is what the current parser does. It recognizes > the problem, registers a defect, and tries to recover, but it doesn't fix the > corrupt data. So for example, if you had a valid RFC 2047 encoded Subject but > a broken X-Foo header, you'd at least still end up with a Message object. The > value of the good headers would be things from which you can get the unicode > value, the raw bytes value, parse its parameters, munge it, etc. while the bad > header might be something you can only get the raw bytes from. My take on this would be that you should always be able to get bytes or characters, but characters are always suspect, in that once you've decoded, if you had invalid bytes, then they're replacement characters (or your choice of encoding fix). -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Fri Sep 17 03:34:26 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 16 Sep 2010 21:34:26 -0400 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <4FE9B10E-B45B-4920-9C7A-16FB00CD1C43@twistedmatrix.com> References: <20100914121744.31ef5aaf@pitrou.net> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> <20100916174053.007c783f@pitro u.net> <20100916205158.5098B1BF4E1@kimball.webabinitio.net> <4FE9B10E-B45B-4920-9C7A-16FB00CD1C43@twistedmatrix.com> Message-ID: <20100917013426.1FE0F1BF445@kimball.webabinitio.net> On Thu, 16 Sep 2010 18:11:30 -0400, Glyph Lefkowitz wrote: > On Sep 16, 2010, at 4:51 PM, R. David Murray wrote: > > > Given a message, there are many times you want to serialize it as text > > (for example, for presentation in a UI). You could provide alternate > > serialization methods to get text out on demand....but then what if > > someone wants to push that text representation back in to email to > > rebuild a model of the message? > > You tell them "too bad, make some bytes out of that text." Leave it up > to the application. Period, the end, it's not the library's job. If > you pushed the text out to a 'view message source' UI representation, > then the vicissitudes of the system clipboard and other encoding and > decoding things may corrupt it in inscrutable ways. You can't fix it. > Don't try. Say we start with this bytes input: To: Glyph Lefkowitz From: "R. David Murray" Subject: =?utf-8?q?p=F6stal?= A simple message. Part of the responsibility of the email module is to provide that in text form on demand, so the application gets: To: Glyph Lefkowitz From: "R. David Murray" Subject: p??stal A simple message. Now the application allows the user to do some manipulation of that, and we have: To: "R. David Murray" From: Glyph Lefkowitz Subject: Re: p??stal A simple reply. How does the application "make some bytes out of that text" before passing it back to email? The application shouldn't have to know how to do RFC2047 encoding, certainly, that's one of the jobs of the email module. If the application just encodes the above as UTF8, then it also has to be calling an email API that knows it is getting bytes input that has not been transfer-encoded, and needs to be told the encoding, so that it can do the correct transfer encoding. In that case why not have the API be pass in the text, with an optional override for the default utf-8 encoding that email will otherwise use? Perhaps some of the disconnect here with Antoine (and Jean-Paul, on IRC) is that the email-sig feels that the format of data handled by the email module (rfcx822-style headers, perhaps with a body, perhaps including MIME attachments) is of much wider utility than just handling email, and that since the email module already has to be very liberal in what it accepts, it isn't much of a stretch to have it handle those use cases as well (and in Python2 it does, in the same 'most of the time' way it handles other non-ASCII byte issues). In that context, it seems perfectly reasonable to expect it to parse string (unicode) headers containing non-ascii data. In such use cases there might be no reason to encode to email RFC wire-format, and thus an encode-to-bytes-and-tell-me-the-encoding interface wouldn't serve the use case particularly well because the application wouldn't want the RFC2047 encoding in the file version of the data. We could conceivably drop those use cases if it simplified the API and implementation, but right now it doesn't feel like it does. Further, Python2 serves these use cases, because you can read the non-ascii data and process it as binary data and it would all just work (most of the time). So such use cases probably do exist out in the wild (but no, we don't have any specific pointers, though I myself was working on such an ap once that never got to production). If Python3 email parses only bytes, then it could serve the use case in somewhat the same way as Python2: the application would encode the data as, say, utf8 and pass it to the 'wire format bytes' input interface, which would then register a defect but otherwise pass the data along to the 'wire' (the file in this case). On read it would again register a defect, and the application could pull the data out using the 'give me the wire-bytes' interface and decode it itself. But this feels yucky to me, like a regression to Python2's conflation of bytes and text. This type of application really wants to work with unicode, not to have to futz with bytes. > > So now we have both a bytes parser and a string parser. > > Why do so many messages on this subject take this for granted? It's > wrong for the email module just like it's wrong for every other package. > > There are plenty of other (better) ways to deal with this problem. Let > the application decide how to fudge the encoding of the characters back > into bytes that can be parsed. "In the face of ambiguity, refuse the > temptation to guess" and all that. The application has more of an idea > of what's going on than the library here, so let it make encoding > decisions. > > Put another way, there's nothing wrong with having a text parser, as > long as it just encodes the text according to some known encoding and > then parses the bytes :). See above for why I don't think that serves all the use cases for text parsing. Perhaps another difference is that in my mind *as an application developer*, the "real" email message consists of unicode headers and message bodies, with attachments that are sometimes binary, and that the wire-format is this formalized encoding we have to use to be able to send it from place to place. In that mental model it seems to make perfect sense to have a StringMessage that I have encode to transmit, and a BytesMessage that I receive and have to decode to work with. Just like I decode generic bytes strings that I get from outside my program and encode my text strings to emit them. In this email design, I'm just doing the encode/decode at a higher level of abstraction. So, forget about the implementation. What's a better object model/API for the email package to use? Keep in mind that at all levels of the model there are applications that need to access the bytes representation, and applications that need to access the string representation. I came up with the two-class API because it seemed simplest from a user point of view: you take in bytes input and get a BytesMessage, which you either manipulate or convert to a StringMessage and then manipulate, depending on your application, or vice versa. The alternative seems to be have two methods for almost every API call, one that accepts or returns string and another that accepts or returns bytes. Perhaps others think that the latter is better, but the email-sig liked my idea, so that's what the current code base implements :) > > So, after much discussion, what we arrived at (so far!) is a model > > that mimics the Python3 split between bytes and strings. If you > > start with bytes input, you end up with a BytesMessage object. > > If you start with string input to the parser, you end up with a > > StringMessage. > > That may be a handy way to deal with some grotty internal implementation > details, but having a 'decode()' method is broken. The thing I care Why is having a decode method broken? > about, as a consumer of this API, is that there is a clearly defined > "Message" interface, which gives me a uniform-looking place where I can > ask for either characters (if I'm displaying them to the user) or bytes > (if I'm putting them on the wire). I don't particularly care where > those bytes came from. I don't care what decoding tricks were necessary > to produce the characters. Exactly. But how does having Bytes and String message objects not provide this? decode and encode hide all those grotty details from the higher level application. If you are worried that at some point in your application you might not know if you have a StringMessage or a BytesMessage, well, that is equivalent to having a point in your application where you might have a string object or you might have a bytes object. Which is to say, if you end up there, then there is something wrong with your design. > Now, it may be worthwhile to have specific normalization / > debrokenifying methods which deal with specific types of corrupt data > from the wire; encoding-guessing, replacement-character insertion or > whatever else are fine things to try. It may also be helpful to keep > around a list of errors in the message, for inspection. But as we know, > there are lots of ways that MIME data can go bad other than encoding, so > that's just one variety of error that we might want to keep around. Yes. email6 intends to extend the already existing error recovery and diagnostics that the email module currently provides. > (Looking at later messages as I'm about to post this, I think this all > sounds pretty similar to Antoine's suggestions, with respect to keeping > the implementation within a single class, and not having > BytesMessage/UnicodeMessage at the same abstraction level.) Forget about the implementation, let's just talk about the API. The two class design came out of *API* thoughts, the implementation came second. If I'm understanding you correctly, you'd prefer to have only one type of Message object and one type of Header object visible at the API level. Then, if you want to present the message to the user 'cat' fashion you'd do: for line in mymsg.serialize_as_string(): print(line, end=None) while when writing it to smtplib.SMTP.sendmail you'd do: smtpserver.sendmail( mymsg['from'].addresses[0].as_bytes(), [x.as_bytes() for x in itertools.chaim( mymsg['to'], mymsg['cc'], mymsg['bcc'])], mymsg.serialize_as_bytes(policy=email.policy.SMTP)) (I'm again ignoring the deficiencies of the current smtplib API.) I can see the appeal of that in that you don't have to think about whether the object is bytes or string based at that point in your code. You just put your data type desire into the method name. But it strikes me as mostly being extra typing. Kind of like having all strings in Python represented internally as a bytes/encoding tuple, and doing print(mytext.as_string) and mybinfile.write(mytext.as_bytes+'\n'.as_bytes) The two cases are not exactly parallel, yet I think they are parallel enough that we're not completely crazy in what we are proposing. But I *am* open to being convinced otherwise. If everyone hates the BytesMessage/StringMessage API design, then that should certainly not be what we implement in email. -- R. David Murray www.bitdance.com From barry at python.org Fri Sep 17 03:53:17 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 16 Sep 2010 21:53:17 -0400 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <20100917013426.1FE0F1BF445@kimball.webabinitio.net> References: <20100914121744.31ef5aaf@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> <20100916174053.007c783f@pitro u.net> <20100916205158.5098B1BF4E1@kimball.webabinitio.net> <4FE9B10E-B45B-4920-9C7A-16FB00CD1C43@twistedmatrix.com> <20100917013426.1FE0F1BF445@kimball.webabinitio.net> Message-ID: <20100916215317.1ea0c38c@mission> On Sep 16, 2010, at 09:34 PM, R. David Murray wrote: >Say we start with this bytes input: > > To: Glyph Lefkowitz > From: "R. David Murray" > Subject: =?utf-8?q?p=F6stal?= > > A simple message. > >Part of the responsibility of the email module is to provide that >in text form on demand, so the application gets: > > To: Glyph Lefkowitz > From: "R. David Murray" > Subject: p??stal > > A simple message. > >Now the application allows the user to do some manipulation of that, >and we have: > > To: "R. David Murray" > From: Glyph Lefkowitz > Subject: Re: p??stal > > A simple reply. And of course, what happens if the original subject is in one charset and the prefix is in an incompatible one? Then you end up with a wire format of two RFC 2047 encoded words separated by whitespace. You have to keep those chunks separate all the way through to do that properly. (I know RDM knows this. :) >But I *am* open to being convinced otherwise. If everyone hates the >BytesMessage/StringMessage API design, then that should certainly not >be what we implement in email. Just as a point of order, to the extent that we're discussing generic approaches to similar problems across multiple modules, it's okay that we're having this discussion on python-dev. But the email-sig has put in a lot of work on specific API and implementation designs for the email package, so any deviation really needs to be debated, discussed, and decided there. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From steve at pearwood.info Fri Sep 17 03:55:35 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 17 Sep 2010 11:55:35 +1000 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <20100917013426.1FE0F1BF445@kimball.webabinitio.net> References: <20100914121744.31ef5aaf@pitrou.net> <4FE9B10E-B45B-4920-9C7A-16FB00CD1C43@twistedmatrix.com> <20100917013426.1FE0F1BF445@kimball.webabinitio.net> Message-ID: <201009171155.35276.steve@pearwood.info> On Fri, 17 Sep 2010 11:34:26 am R. David Murray wrote: > Perhaps another difference is that in my mind *as an application > developer*, the "real" email message consists of unicode headers and > message bodies, with attachments that are sometimes binary, and that > the wire-format is this formalized encoding we have to use to be able > to send it from place to place. ?In that mental model it seems to > make perfect sense to have a StringMessage that I have encode to > transmit, and a BytesMessage that I receive and have to decode to > work with. +1 -- Steven D'Aprano From tjreedy at udel.edu Fri Sep 17 04:30:24 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 16 Sep 2010 22:30:24 -0400 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> Message-ID: On 9/16/2010 3:07 PM, Jacob Kaplan-Moss wrote: >> On 16 September 2010 07:16, Terry Reedy wrote: >>>> I'm not working to get Django running on Python 3.1 because I don't >>>> feel confident I'll be able to put any apps I write into production. >>> >>> Why not? Since the I/O speed problem is fixed, I have no idea what you are >>> referring to. Please do be concrete. > Deploying web apps under Python 2 right now is actually pretty > awesome. ... And will remain so for years. > The key here is that switching between all of these deployment > situations is *incredibly* easy. ... > Python 3 offers me none of this. I don't have a wide variety of tools > to choose from. Worse, I don't even have a guarantee of > interoperability between the tools that *do* exist. That last needs an updated standard, which may require a bit of nudging to get agreement on *something*, along with an updated reference implementation. I would expect a usable variety of production implementations to gradually follow thereafter, as they have for 2.x. > I'm sorry if I'm coming across as a complainer here. No. You answered my question quite well. -- Terry Jan Reedy From rdmurray at bitdance.com Fri Sep 17 04:51:04 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 16 Sep 2010 22:51:04 -0400 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <20100917000512.66183149@pitrou.net> References: <20100914121744.31ef5aaf@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> <20100916174053.007c783f@pitro u.net> <20100916205158.5098B1BF4E1@kimball.webabinitio.net> <20100917000512.66183149@pitrou.net> Message-ID: <20100917025104.BA8661D6A76@kimball.webabinitio.net> On Fri, 17 Sep 2010 00:05:12 +0200, Antoine Pitrou wrote: > On Thu, 16 Sep 2010 16:51:58 -0400 > "R. David Murray" wrote: > > > > What do we store in the model? We could say that the model is always > > text. But then we lose information about the original bytes message, > > and we can't reproduce it. For various reasons (mailman being a big one), > > this is not acceptable. So we could say that the model is always bytes. > > But we want access to (for example) the header values as text, so header > > lookup should take string keys and return string values[2]. > > Why can't you have both in a single class? If you create the class > using a bytes source (a raw message sent by SMTP, for example), the > class automatically parses and decodes it to unicode strings; if you > create the class using an unicode source (the text body of the e-mail > message and the list of recipients, for example), the class > automatically creates the bytes representation. > > (of course all processing can be done lazily for performance reasons) Certainly we could do that. There are methods, though, whose implementation is the same except for the detail of whether they are processing bytes or string, so the dual class structure allows that implementation to be shared. So even if we changed the API to be single class, I might well retain the dual class implementation under the hood. I'd have to explore which looked better when the time came. > > What about email files on disk? They could be bytes, or they could be, > > effectively, text (for example, utf-8 encoded). > > Such a file can be two things: > - the raw encoding of a whole message (including headers, etc.), then > it should be fed as a bytes object > - the single text body of a hypothetical message, then it should be fed > as a unicode object > I don't see any possible middle-ground. It's not a middle ground, but as I discussed in my response to Glyph, it could be a series of headers and a body in, say, utf-8 where the application wants to treat them as unicode, not bytes (ie: *not* an email). Python2 supports this use case, albeit with the same "works most of the time" as it does with other non-ascii edge cases. > > On disk, using utf-8, > > one might store the text representation of the message, rather than > > the wire-format (ASCII encoded) version. We might want to write such > > messages from scratch. > > But then the user knows the encoding (by "user" I mean what/whoever > calls the email API) and mentions it to the email package. Yes? And then? The email package still has to parse the file, and it can't use its normal parse-the-RFC-data parser because the file could contain *legitimate* non-ASCII header data. So there has to be a separate parser for this case that will convert the non-ASCII data into RFC2047 encoded data. At that point you have two parsers that share a bunch of code...and my current implementation lets the input to the second parser be text, which is the natural representation of that data, the one the user or application writer is going to expect. I *could* implement it as a variant bytes parser, and have the application call the variant parser with encoded bytes, but why? What's the benefit? If the API takes text, it is *obvious* that non-ascii data is allowed and is going to get wire-encoded. If it takes bytes....there is more mental overhead in figuring out which bytes-parser interface one should call, depending on whether one has 'wire format" data or encoded non-ascii data. I can just imagine someone using the bytes-that-need-transfer-encoding to try to parse a file containing RFC encoded data that he knows is stored in a utf-8 encoded file, because that's the interface that accepts an encoding paramter. And then the RFC2047 encoded words wouldn't get decoded. Overall it seems simpler to me that text file == pass text to the text parser, RFC-encoded bytes data == pass bytes data to the bytes parser. This also separates opening the file correctly (specify the encoding on open) from encoding the data as you prefer (encoding specified to the email package when telling it to encode to wire format). > What I'm having an issue with is that you are talking about a bytes > representation and an unicode representation of a message. But they > aren't representations of the same things: > - if it's a bytes representation, it will be the whole, raw message > including envelope / headers (also, MIME sections etc.) > - if it's an unicode representation, it will only be a section of the > message decodable as such (a text/plain MIME section, for example; > or a decoded header value; or even a single e-mail address part of a > decoded header) Conceptually, a BytesMessage is a model of the entire message with all the parts encoded in RFC wire-format. When you access pieces of it, you get the RFC encoded byte strings. Conceptually a StringMessage is a model of the entire message with all the parts decoded as far as possible. This means that header values are unicode, and jpeg images are...jpeg images. When you access pieces of it, you get the most useful kind of object for manipulating that piece in a program. (So perhaps StringMessage is a bad label). This split is about making the API simple, in my mind. But as I said to Glyph, perhaps I am wrong about this being the simpler/easier to understand API. > So, there doesn't seem to be any reason for having both a BytesMessage > and an UnicodeMessage at the same abstraction level. They are both > representing different things at different abstraction levels. I don't > see any potential for confusion: raw assembled e-mail message = bytes; > decoded text section of a message = unicode. Perhaps my explanation above helps clarify this? They are only at the same level of abstraction in the sense that encoded byte strings and decoded unicode strings are at the same abstraction level. That is, they aren't. > As for the problem of potential "bogus" raw e-mail data > (e.g., undecodable headers), well, I guess the library has to make a > choice between purity and practicality, or perhaps let the user choose > themselves. For example, through a `strict` flag. If `strict` is true, > raise an error as soon as a non-decodable byte appears in a header, if > `strict` is false, decode it through a default (encoding, errors) > convention which can be overriden by the user (a sensible possibility > being "utf-8, surrogateescape" to allow for lossless round-tripping). Yes, there will be a stict setting, and error reporting, and optional control over the default encoding and error handler. > > As I said above, we could insist that files on > > disk be in wire-format, and for many applications that would work fine, > > but I think people would get mad at us if didn't support text files[3]. > > Again, this simply seems to be two different abstraction levels: > pre-generated raw email messages including headers, or a single text > waiting to be embedded in an actual e-mail. But why not a text (unicode/utf8) representation of a message on disk, including headers? Why should that not be supported? (If it were a lot of extra work to support it, I'd drop it. But it isn't.) > > Anyway, what polymorphism means in email is that if you put in bytes, > > you get a BytesMessage, if you put in strings you get a StringMessage, > > and if you want the other one you convert. > > And then you have two separate worlds while ultimately the same > concepts are underlying. A library accepting BytesMessage will crash > when a program wants to give a StringMessage and vice-versa. That > doesn't sound very practical. Yes, and a library accepting bytes will crash when a program wants to give it a string. So? That's how Python3 works. Unless, of course, the application decides to be polymorphic :) > > [1] Now that surrogateesscape exists, one might suppose that strings > > could be used as an 8bit channel, but that only works if you don't need > > to *parse* the non-ASCII data, just transmit it. > > Well, you can parse it, precisely. Not only, but it round-trips if you > unparse it again: > > >>> header_bytes = b"From: bogus\xFFname " > >>> name, value = header_bytes.decode("utf-8", "surrogateescape").split(":") > >>> name > 'From' > >>> value > ' bogus\udcffname ' > >>> "{0}:{1}".format(name, value).encode("utf-8", "surrogateescape") > b'From: bogus\xffname ' Well, yes, the email module could accept surrogateescape decoded 8bit input, and encode it to bytes internally (remember that non-header 8bit data can be valid data). But that only changes how the bytes get in to the email module, it doesn't change anything else. I don't see it as an improvement over just accepting bytes. On the other hand, that might be a way to make the current API work at least a little bit better with 8bit input data. I'll have to think about that... > In the end, what I would call a polymorphic best practice is "try to > avoid bytes/str polymorphism if your domain is well-defined > enough" (which I admit URLs aren't necessarily; but there's no > question a single text/XXX e-mail section is text, and a whole > assembled e-mail message is bytes). Hmm. Yes, we have strayed quite far from the original question into the broader motivations behind the current email6 API. Having gone through this discussion, I realize now that the design isn't really polymorphic in the strict sense. Instead it is about handling both text and bytes use cases with the simplest API I could come up with. Rather than polymorphism in the email6 interface, what we really have is a higher-abstraction-level equivalent of the bytes/string split made by Python3. And that makes sense to me as a direct outcome of that Python3 split, and I suspect it is something that may need to be replicated elsewhere in the stdlib (if I'm right!), such as in an as-yet-nonexistent IRI module. --David From tjreedy at udel.edu Fri Sep 17 05:45:12 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 16 Sep 2010 23:45:12 -0400 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <20100917025104.BA8661D6A76@kimball.webabinitio.net> References: <20100914121744.31ef5aaf@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> <20100916174053.007c783f@pitro u.net> <20100916205158.5098B1BF4E1@kimball.webabinitio.net> <20100917000512.66183149@pitrou.net> <20100917025104.BA8661D6A76@kimball.webabinitio.net> Message-ID: Based on the discussion so far, I think you should go ahead and implement the API agreed on by the mail sig both because is *has* been agreed on (and thinking about the wsgi discussion, that seems to be a major achievement) and because it seems sensible to me also, as far as I understand it. The proof of the API will be in the testing. As long as you *think* it covers all intended use cases, I am not sure that abstract discussion can go much further. I do have a thought about space and duplication. For normal messages, it is not an issue. For megabyte (or in the future, gigabyte?) attachments, it is. So if possible, there should only be one extracted blob for both bytes and string versions of parsed messages. Or even make the extraction from the raw stream lazy, when specifically requested. -- Terry Jan Reedy From rdmurray at bitdance.com Fri Sep 17 06:26:52 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 17 Sep 2010 00:26:52 -0400 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <20100916215317.1ea0c38c@mission> References: <20100914121744.31ef5aaf@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> <20100916174053.007c783f@pitro u.net> <20100916205158.5098B1BF4E1@kimball.webabinitio.net> <4FE9B10E-B45B-4920-9C7A-16FB00CD1C43@twistedmatrix.com> <20100917013426.1FE0F1BF445@kimball.webabinitio.net> <20100916215317.1ea0c38c@ mission> Message-ID: <20100917042652.C49111ABB1C@kimball.webabinitio.net> On Thu, 16 Sep 2010 21:53:17 -0400, Barry Warsaw wrote: > And of course, what happens if the original subject is in one charset and the > prefix is in an incompatible one? Then you end up with a wire format of two > RFC 2047 encoded words separated by whitespace. You have to keep those chunks > separate all the way through to do that properly. (I know RDM knows this. :) Heh, my example got messed up because my current mailer didn't MIME encode it properly. That is, I emitted a non-RFC-compliant email :( This is actually a pretty interesting issue in a number of ways, though I'm not sure it relates to any other part of the stdlib. A header can contain encoded words encoded with different charsets. An MUA that sorts by subject and takes prefixes ('Re:') into account, for example, might be decoding the header entirely before doing header matching/sorting, or it might be matching against the RFC2047 encoded header. Hopefully the former, these days, but don't count on it. So when emitting a reply, a careful MUA would want to *only* attach the 'Re:' to the front, and not otherwise change the header. If it is going to do that, though, it is going to have to (a) make sure it preserves the original bytes version of the header and (b) refold the line if necessary. This means knowing lots of stuff about header encoding. So, really, that job should be done by the email package, or at least the email package should provide tools to do this. The naive way (decode the header to unicode, attach the prefix, re-encode using your favorite charset) is going to work most of the time, and that's what it will be easiest to do with email6. Tacking the Re: on the front of the bytes version of the header and having email6 refold it will probably work about as well as it currently does in the old email package, which is to say that sometimes the unfolded header is otherwise unchanged, and sometimes it isn't. > >But I *am* open to being convinced otherwise. If everyone hates the > >BytesMessage/StringMessage API design, then that should certainly not > >be what we implement in email. > > Just as a point of order, to the extent that we're discussing generic > approaches to similar problems across multiple modules, it's okay that we're > having this discussion on python-dev. But the email-sig has put in a lot of > work on specific API and implementation designs for the email package, so any > deviation really needs to be debated, discussed, and decided there. I am also finding it useful to have the API exposed to a wider audience for feedback, but I agree, any substantive change would need to be discussed on the email-sig, not here. --David From rdmurray at bitdance.com Fri Sep 17 06:41:49 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 17 Sep 2010 00:41:49 -0400 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> <20100916174053.007c783f@pitro u.net> <20100916205158.5098B1BF4E1@kimball.webabinitio.net> <20100917000512.66183149@pitrou.net> <20100917025104.BA8661D6A76@kimball.webabinitio.net> Message-ID: <20100917044149.8AFD41D6AC5@kimball.webabinitio.net> On Thu, 16 Sep 2010 23:45:12 -0400, Terry Reedy wrote: > Based on the discussion so far, I think you should go ahead and > implement the API agreed on by the mail sig both because is *has* been > agreed on (and thinking about the wsgi discussion, that seems to be a > major achievement) and because it seems sensible to me also, as far as I > understand it. The proof of the API will be in the testing. As long as > you *think* it covers all intended use cases, I am not sure that > abstract discussion can go much further. > > I do have a thought about space and duplication. For normal messages, it > is not an issue. For megabyte (or in the future, gigabyte?) attachments, > it is. So if possible, there should only be one extracted blob for both > bytes and string versions of parsed messages. Or even make the > extraction from the raw stream lazy, when specifically requested. Our intent is to have conversions be as lazy as possible. There will doubtless be some interesting heuristics to develop as to what to convert when and what to cache when, and consequent problems to solve when it comes to garbage collection... There's also slated to be a back-end API for storing parts of messages elsewhere than in memory, though I haven't worked out what that is going to look like yet. But we are definitely getting off topic now :) --David From victor.stinner at haypocalc.com Fri Sep 17 09:21:41 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 17 Sep 2010 09:21:41 +0200 Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: References: <201009162336.06353.victor.stinner@haypocalc.com> Message-ID: <201009170921.41486.victor.stinner@haypocalc.com> Le vendredi 17 septembre 2010 00:09:09, Tom Browder a ?crit : > I did, and eventually discovered the problem: I tried to "nosy" Barry > as requested by adding his e-mail address, but that causes an error in > the tracker. After I finally figured that out, I successfully entered > the original bug (and reported it on the "tracker bug"). http://bugs.python.org/issue9880 Ah, yes, you have to add nicknames, not emails. Barry nickname is "barry", and he's already on the nosy list (because he answered to your issue). -- Victor Stinner http://www.haypocalc.com/ From baptiste13z at free.fr Fri Sep 17 11:25:20 2010 From: baptiste13z at free.fr (Baptiste Carvello) Date: Fri, 17 Sep 2010 11:25:20 +0200 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <20100916153012.8FCA11BF412@kimball.webabinitio.net> References: <20100914121744.31ef5aaf@pitrou.net> <4C90D657.8070105@holdenweb.com> <20100915164308.6ef1ef63@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> Message-ID: R. David Murray a ?crit : > I'm trying one approach in email6: > Bytes and String subclasses, where the subclasses have an attribute > named 'literals' derived from a utility module that does this: > > literals = dict( > empty = '', > colon = ':', > newline = '\n', > space = ' ', > tab = '\t', > fws = ' \t', > headersep = ': ', > ) > > class _string_literals: > pass > class _bytes_literals: > pass > > for name, value in literals.items(): > setattr(_string_literals, name, value) > setattr(_bytes_literals, name, bytes(value, 'ASCII')) > del literals, name, value > > And the subclasses do: > > class BytesHeader(BaseHeader): > lit = email.utils._bytes_literals > > class StringHeader(BaseHeader): > lit = email.utils._string_literals > I've just written a decorator which applies a similar strategy for insulated functions, by passing them an appropriate namespace as an argument. It could be useful in cases where only a few functions are polymorphic, not a full class or module. http://code.activestate.com/recipes/577393-decorator-for-writing-polymorphic-functions/ Cheers, B. From sable at users.sourceforge.net Fri Sep 17 11:40:12 2010 From: sable at users.sourceforge.net (=?UTF-8?B?U8OpYmFzdGllbiBTYWJsw6k=?=) Date: Fri, 17 Sep 2010 11:40:12 +0200 Subject: [Python-Dev] [issue1633863] AIX: configure ignores $CC In-Reply-To: <1284586127.79.0.767793857364.issue1633863@psf.upfronthosting.co.za> References: <1284586127.79.0.767793857364.issue1633863@psf.upfronthosting.co.za> Message-ID: <4C93377C.4040300@users.sourceforge.net> Hi Martin, I have started to correct quite a lot of issues I have with Python on AIX, and since I had to test quite a lot of patchs, I though it would be more convenient to setup a buildbot for that platform. So I now have a buildbot environment with 2 slaves (AIX 5.3 and 6.1) that builds and tests Python (branch py3k) with both gcc and xlc (the native AIX compiler) (I have 4 builders ("py3k-aix6-xlc", "py3k-aix5-xlc", "py3k-aix6-gcc", "py3k-aix5-gcc"). I expect to add 4 more builders for branch 2.7 in coming days. I would like to share the results of this buildbot to the Python community so that issues with AIX could be addressed more easily. R. David Murray pointed me to the page on the python wiki concerning buildbot. It is stated there that is is possible to connect some slaves to some official Python buildbot master. Unfortunately, I don't think this solution is possible for me: I don't think the security team in my company would appreciate that a server inside our network runs some arbitrary shell commands provided by some external source. Neither can I expose the buildbot master web interface. Also I had to customize the buildbot rules in order to work with some specificities of AIX (see attached master.cfg), and I can't guarantee that this buildbot will run 24 hours a day; I may have to schedule it only once at night for example if it consumes too much resources. (And the results are very unstable at the moment, mostly because of issue 9862). On the other hand, I could upload the build results with rsync or scp somewhere or setup some MailNotifier if that can help. How do you think I could share those results? regards -- S?bastien Sabl? Le 15/09/2010 23:28, R. David Murray a ?crit : > > R. David Murray added the comment: > > S?bastien, you could email Martin (tracker id loewis) about adding your buildbot to our unstable fleet (or even to stable if it is stable; that is, the tests normally pass and don't randomly fail). As long as you are around to help fix bugs it would be great to have an aix buildbot in our buildbot fleet. > > (NB: see also http://wiki.python.org/moin/BuildBot, which unfortunately is a bit out of date...) > > ---------- > nosy: +r.david.murray > > _______________________________________ > Python tracker > > _______________________________________ -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: master.cfg URL: From solipsis at pitrou.net Fri Sep 17 12:10:26 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 17 Sep 2010 12:10:26 +0200 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <20100917025104.BA8661D6A76@kimball.webabinitio.net> References: <20100914121744.31ef5aaf@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> <20100916174053.007c783f@pitro u.net> <20100916205158.5098B1BF4E1@kimball.webabinitio.net> <20100917000512.66183149@pitrou.net> <20100917025104.BA8661D6A76@kimball.webabinitio.net> Message-ID: <1284718226.3169.17.camel@localhost.localdomain> Le jeudi 16 septembre 2010 ? 22:51 -0400, R. David Murray a ?crit : > > > On disk, using utf-8, > > > one might store the text representation of the message, rather than > > > the wire-format (ASCII encoded) version. We might want to write such > > > messages from scratch. > > > > But then the user knows the encoding (by "user" I mean what/whoever > > calls the email API) and mentions it to the email package. > > Yes? And then? The email package still has to parse the file, and it > can't use its normal parse-the-RFC-data parser because the file could > contain *legitimate* non-ASCII header data. So there has to be a separate > parser for this case that will convert the non-ASCII data into RFC2047 > encoded data. At that point you have two parsers that share a bunch of > code...and my current implementation lets the input to the second parser > be text, which is the natural representation of that data, the one the > user or application writer is going to expect. But you said it yourself: that "e-mail-like data" data is not an email. You could have a separate converter class for these special cases. Also, I don't understand why an application would want to assemble an e-mail by itself if it doesn't know how to do so, and produces wrong data. Why not simply let the application do: m = Message() m.add_header("From", "Accented B?rry ") m.add_body("Hello Barry") > > And then you have two separate worlds while ultimately the same > > concepts are underlying. A library accepting BytesMessage will crash > > when a program wants to give a StringMessage and vice-versa. That > > doesn't sound very practical. > > Yes, and a library accepting bytes will crash when a program wants > to give it a string. So? That's how Python3 works. Unless, of > course, the application decides to be polymorphic :) Well, the application wants to handle abstracted e-mail messages. I'm sure people would rather not deal with the difference(s) between BytesMessages and StringMessages. That's like saying we should have BytesConfigParser for bytes configuration files and StringConfigParser for string configuration files, with incompatible APIs. ("surrogateescape") > On the other hand, that might be a way to make the current API work > at least a little bit better with 8bit input data. I'll have to think > about that... Yes, that's what I was talking about. You can even choose ("ascii", "surrogateescape") if you don't want to wrongly choose an 8-bit encoding such as utf-8 or latin-1. (I'm deliberately ignoring the case where people would use a non-ASCII compatible encoding such as utf-16; I hope you don't want to support that :-)) Regards Antoine. From dan at meatballhat.com Fri Sep 17 13:54:03 2010 From: dan at meatballhat.com (Dan Buch) Date: Fri, 17 Sep 2010 07:54:03 -0400 Subject: [Python-Dev] standards for distribution names In-Reply-To: <4C91FACB.3010300@simplistix.co.uk> References: <4C91FACB.3010300@simplistix.co.uk> Message-ID: <20100917115401.GI26366@gmail.com> On Thu, Sep 16, 2010 at 12:08:59PM +0100, Chris Withers wrote: > Hi All, > > Following on from this question: > > http://twistedmatrix.com/pipermail/twisted-python/2010-September/022877.html > > ...I'd thought that the "correct names" for distributions would have > been documented in one of: > > http://www.python.org/dev/peps/pep-0345 > http://www.python.org/dev/peps/pep-0376 > http://www.python.org/dev/peps/pep-0386 > > ...but having read them, I drew a blank. > > Where are the standards for this or is it still a case of "whatever > setuptools does"? > > Chris > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/daniel.buch%40gmail.com You may also find this thread from the packaging google group useful, although it may not be quite what you're looking for: http://bit.ly/96SMuM Cheers, -- ~Dan -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 230 bytes Desc: not available URL: From martin at v.loewis.de Fri Sep 17 14:42:13 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 17 Sep 2010 14:42:13 +0200 Subject: [Python-Dev] [issue1633863] AIX: configure ignores $CC In-Reply-To: <4C93377C.4040300@users.sourceforge.net> References: <1284586127.79.0.767793857364.issue1633863@psf.upfronthosting.co.za> <4C93377C.4040300@users.sourceforge.net> Message-ID: <4C936225.1000500@v.loewis.de> Hi Sebastien, > Unfortunately, I don't think this solution is possible for me: I don't > think the security team in my company would appreciate that a server > inside our network runs some arbitrary shell commands provided by some > external source. I still think this would be the best thing, and I feel that from a security point of view, it doesn't really differ from what you are doing now already - see below. > Neither can I expose the buildbot master web interface. That shouldn't be necessary. > Also I had to customize the buildbot rules in order to work with some > specificities of AIX (see attached master.cfg), and I can't guarantee > that this buildbot will run 24 hours a day; I may have to schedule it > only once at night for example if it consumes too much resources. > > (And the results are very unstable at the moment, mostly because of > issue 9862). If you are having the build slave compile Python, I'd like to point out that you *already* run arbitrary shell commands provided by some external source: if somebody would check some commands into Python's configure.in, you would unconditionally execute them. So if it's ok that you run the Python build process at all, it should (IMO) also be acceptable to run a build slave. If there are concerns that running it under your Unix account gives it too much power, you should create a separate, locked-down account. > On the other hand, I could upload the build results with rsync or scp > somewhere or setup some MailNotifier if that can help. > > How do you think I could share those results? I'd be hesitant to support this as a special case. If the results are not in the standard locations, people won't look at them, anyway. Given that one often also needs access to the hardware in order to fix problems, it might be sufficient if only you look at the buildslave results, and create bug reports whenever you notice a problem. Regards, Martin From solipsis at pitrou.net Fri Sep 17 15:05:51 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 17 Sep 2010 15:05:51 +0200 Subject: [Python-Dev] [issue1633863] AIX: configure ignores $CC References: <1284586127.79.0.767793857364.issue1633863@psf.upfronthosting.co.za> <4C93377C.4040300@users.sourceforge.net> Message-ID: <20100917150551.099113e4@pitrou.net> On Fri, 17 Sep 2010 11:40:12 +0200 S?bastien Sabl? wrote: > Hi Martin, > > I have started to correct quite a lot of issues I have with Python on > AIX, and since I had to test quite a lot of patchs, I though it would be > more convenient to setup a buildbot for that platform. > > So I now have a buildbot environment with 2 slaves (AIX 5.3 and 6.1) > that builds and tests Python (branch py3k) with both gcc and xlc (the > native AIX compiler) (I have 4 builders ("py3k-aix6-xlc", > "py3k-aix5-xlc", "py3k-aix6-gcc", "py3k-aix5-gcc"). Following on Martin's comments, you might also want to share things with the ActiveState guys who, AFAIK, maintain an AIX version of Python (but you have been the most active AIX user on the bug tracker lately; perhaps they are keeping their patches to themselves). (see http://www.activestate.com/activepython ) Regards Antoine. From martin at v.loewis.de Fri Sep 17 15:18:39 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 17 Sep 2010 15:18:39 +0200 Subject: [Python-Dev] Add PEP 444, Python Web3 Interface. In-Reply-To: <4C915EA4.5040300@animats.com> References: <4C915EA4.5040300@animats.com> Message-ID: <4C936AAF.2000504@v.loewis.de> Am 16.09.10 02:02, schrieb John Nagle: > On 9/15/2010 4:44 PM, python-dev-request at python.org wrote: >> ``SERVER_PORT`` must be a bytes instance (not an integer). > > What's that supposed to mean? What goes in the "bytes > instance"? A character string in some format? A long binary > number? If the latter, with which byte ordering? What > problem does this\ solve? Just interpreting (i.e. not having participated in the specification): Given the CGI background of all this, SERVER_PORT is an ASCII-encoded decimal rendering of the port number. As to what problem this solves: I guess it allows for easy pass-through from the web server. Regards, Martin From ncoghlan at gmail.com Fri Sep 17 15:40:15 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 17 Sep 2010 23:40:15 +1000 Subject: [Python-Dev] (Not) delaying the 3.2 release In-Reply-To: References: <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd12a@mission> <20100916154207.GS3086@unaka.lan> <20100916181653.GT3086@unaka.lan> Message-ID: On Fri, Sep 17, 2010 at 5:43 AM, Martin (gzlist) wrote: > In the example I gave, ? encodes in CP932 as '\x8f\\', and the > function gets confused by the second byte. Obviously the right answer > there is just to use unicode, rather than write a function that works > with weird multibyte codecs. That does make it clear that "ASCII superset" is an inaccurate term - a better phrase is "ASCII compatible", since that correctly includes multibyte codecs like UTF-8 which explicitly ensure that the byte values in multibyte characters are all outside the 0x00 to 0x7F range of ASCII. So the domain of any polymorphic text manipulation functions we define would be: - Unicode strings - byte sequences where the encoding is either: - a single byte ASCII superset (e.g. iso-8859-*, cp1252, koi8*, mac*) - an ASCII compatible multibyte encoding (e.g. UTF-8, EUC-JP) Passing in byte sequences that are encoded using an ASCII incompatible multibyte encoding (e.g. CP932, UTF-7, UTF-16, UTF-32, shift-JIS, big5, iso-2022-*, EUC-CN/KR/TW) or a single byte encoding that is not an ASCII superset (e.g. EBCDIC) will have undefined results. I think that's still a big enough win to be worth doing, particularly as more and more of the other variable width multibyte encodings are phased out in favour of UTF-8. Cheers, Nick. P.S. Hey Barry, is there anyone at Canonical you can poke about https://bugs.launchpad.net/xorg-server/+bug/531208? Tinkering with this stuff on Kubuntu would be significantly less annoying if I could easily type arbitrary Unicode characters into Konsole ;) -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From barry at python.org Fri Sep 17 16:56:02 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 17 Sep 2010 10:56:02 -0400 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <1284718226.3169.17.camel@localhost.localdomain> References: <20100914121744.31ef5aaf@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> <20100916174053.007c783f@pitro u.net> <20100916205158.5098B1BF4E1@kimball.webabinitio.net> <20100917000512.66183149@pitrou.net> <20100917025104.BA8661D6A76@kimball.webabinitio.net> <1284718226.3169.17.camel@localhost.localdomain> Message-ID: <20100917105602.39e82ec5@mission> On Sep 17, 2010, at 12:10 PM, Antoine Pitrou wrote: >Also, I don't understand why an application would want to assemble an >e-mail by itself if it doesn't know how to do so, and produces wrong >data. Why not simply let the application do: > >m = Message() >m.add_header("From", "Accented B?rry ") >m.add_body("Hello Barry") Very often you'll start with a template of a message your application wants to send. Then you'll interpolate a few values into it, and you'd like to easily convert the result into an RFC valid email. Is that template bytes or text (or either)? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Fri Sep 17 17:00:49 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 17 Sep 2010 11:00:49 -0400 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> <20100916174053.007c783f@pitro u.net> <20100916205158.5098B1BF4E1@kimball.webabinitio.net> <20100917000512.66183149@pitrou.net> <20100917025104.BA8661D6A76@kimball.webabinitio.net> Message-ID: <20100917110049.40f4bdb0@mission> On Sep 16, 2010, at 11:45 PM, Terry Reedy wrote: >Based on the discussion so far, I think you should go ahead and >implement the API agreed on by the mail sig both because is *has* been >agreed on (and thinking about the wsgi discussion, that seems to be a >major achievement) and because it seems sensible to me also, as far as >I understand it. The proof of the API will be in the testing. As long >as you *think* it covers all intended use cases, I am not sure that >abstract discussion can go much further. +1 >I do have a thought about space and duplication. For normal messages, >it is not an issue. For megabyte (or in the future, gigabyte?) >attachments, it is. So if possible, there should only be one extracted >blob for both bytes and string versions of parsed messages. Or even >make the extraction from the raw stream lazy, when specifically >requested. This has been discussed in the email-sig. Many people have asked for an API where message payloads can be stored on-disk instead of in-memory. Headers, I don't think will every practically be so big as to not be storable in-memory. But if your message has a huge mp3, the parser should have the option to leave the bytes of that payload in a disk cache and transparently load it when necessary. I think we should keep that in mind, but it's way down on the list of "gotta haves" for email6. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From status at bugs.python.org Fri Sep 17 18:09:07 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 17 Sep 2010 18:09:07 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100917160907.635E2781C0@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-09-10 - 2010-09-17) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues stats: open 2541 (+42) closed 19128 (+69) total 21669 (+65) Open issues with patches: 1060 Issues opened (42) ================== #9824: SimpleCookie should escape commas and semi-colons http://bugs.python.org/issue9824 opened by spookylukey #9831: test_distutils fails on MacOSX 10.6 http://bugs.python.org/issue9831 opened by cartman #9838: Inadequate C-API to Python 3 I/O objects http://bugs.python.org/issue9838 opened by pv #9841: sysconfig and distutils.sysconfig differ in subtle ways http://bugs.python.org/issue9841 opened by eric.araujo #9842: Document ... used in recursive repr of containers http://bugs.python.org/issue9842 opened by eric.araujo #9844: calling nonexisting function under __INSURE__ http://bugs.python.org/issue9844 opened by eli.bendersky #9845: Allow changing the method in urllib.request.Request http://bugs.python.org/issue9845 opened by tarek #9846: ZipExtFile provides no mechanism for closing the underlying fi http://bugs.python.org/issue9846 opened by john.admanski #9849: Argparse needs better error handling for nargs http://bugs.python.org/issue9849 opened by Jason.Baker #9850: obsolete macpath module dangerously broken and should be remov http://bugs.python.org/issue9850 opened by ned.deily #9851: multiprocessing socket timeout will break client http://bugs.python.org/issue9851 opened by hume #9852: test_ctypes fail with clang http://bugs.python.org/issue9852 opened by cartman #9854: SocketIO should return None on EWOULDBLOCK http://bugs.python.org/issue9854 opened by pitrou #9856: Change object.__format__(s) where s is non-empty to a Deprecat http://bugs.python.org/issue9856 opened by eric.smith #9857: SkipTest in tearDown is reported an as an error http://bugs.python.org/issue9857 opened by pitrou #9858: Python and C implementations of io are out of sync http://bugs.python.org/issue9858 opened by pitrou #9859: Add tests to verify API match of modules with 2 implementation http://bugs.python.org/issue9859 opened by stutzbach #9860: Building python outside of source directory fails http://bugs.python.org/issue9860 opened by belopolsky #9861: subprocess module changed exposed attributes http://bugs.python.org/issue9861 opened by pclinch #9862: test_subprocess hangs on AIX http://bugs.python.org/issue9862 opened by sable #9864: email.utils.{parsedate,parsedate_tz} should have better return http://bugs.python.org/issue9864 opened by pitrou #9865: OrderedDict doesn't implement __sizeof__ http://bugs.python.org/issue9865 opened by pitrou #9866: Inconsistencies in tracing list comprehensions http://bugs.python.org/issue9866 opened by belopolsky #9867: Interrupted system calls are not retried http://bugs.python.org/issue9867 opened by aronacher #9868: test_locale leaves locale changed http://bugs.python.org/issue9868 opened by ocean-city #9869: long_subtype_new segfault in pure-Python code http://bugs.python.org/issue9869 opened by cwitty #9871: IDLE dies when using some regex http://bugs.python.org/issue9871 opened by Popa.Claudiu #9873: Allow bytes in some APIs that use string literals internally http://bugs.python.org/issue9873 opened by ncoghlan #9874: Message.attach() loses empty attachments http://bugs.python.org/issue9874 opened by raph at mankin.org.uk #9875: Garbage output when running setup.py on Windows http://bugs.python.org/issue9875 opened by exarkun #9876: ConfigParser can't interpolate values from other sections http://bugs.python.org/issue9876 opened by asolovyov #9877: Expose sysconfig._get_makefile_filename() in public API http://bugs.python.org/issue9877 opened by barry #9878: Avoid parsing pyconfig.h and Makefile by autogenerating extens http://bugs.python.org/issue9878 opened by barry #9880: Python 2.7 Won't Build: SystemError: unknown opcode http://bugs.python.org/issue9880 opened by Tom.Browder #9882: abspath from directory http://bugs.python.org/issue9882 opened by ipatrol #9883: minidom: AttributeError: DocumentFragment instance has no attr http://bugs.python.org/issue9883 opened by Aubrey.Barnard #9884: The 4th parameter of method always None or 0 on x64 Windows. http://bugs.python.org/issue9884 opened by J2.NETe #9886: Make operator.itemgetter/attrgetter/methodcaller easier to dis http://bugs.python.org/issue9886 opened by ncoghlan #9887: distutil's build_scripts doesn't read utf-8 in all locales http://bugs.python.org/issue9887 opened by hagen #460474: codecs.StreamWriter: reset() on close() http://bugs.python.org/issue460474 reopened by r.david.murray #767645: incorrect os.path.supports_unicode_filenames http://bugs.python.org/issue767645 reopened by haypo #1076515: shutil.move clobbers read-only files. http://bugs.python.org/issue1076515 reopened by brian.curtin Most recent 15 issues with no replies (15) ========================================== #9887: distutil's build_scripts doesn't read utf-8 in all locales http://bugs.python.org/issue9887 #9886: Make operator.itemgetter/attrgetter/methodcaller easier to dis http://bugs.python.org/issue9886 #9884: The 4th parameter of method always None or 0 on x64 Windows. http://bugs.python.org/issue9884 #9883: minidom: AttributeError: DocumentFragment instance has no attr http://bugs.python.org/issue9883 #9882: abspath from directory http://bugs.python.org/issue9882 #9871: IDLE dies when using some regex http://bugs.python.org/issue9871 #9869: long_subtype_new segfault in pure-Python code http://bugs.python.org/issue9869 #9862: test_subprocess hangs on AIX http://bugs.python.org/issue9862 #9861: subprocess module changed exposed attributes http://bugs.python.org/issue9861 #9856: Change object.__format__(s) where s is non-empty to a Deprecat http://bugs.python.org/issue9856 #9852: test_ctypes fail with clang http://bugs.python.org/issue9852 #9803: IDLE closes with save while breakpoint open http://bugs.python.org/issue9803 #9800: Fast path for small int-indexing of lists and tuples http://bugs.python.org/issue9800 #9790: ntpath contains imports inside functions http://bugs.python.org/issue9790 #9774: test_smtpnet fails with "[110] Connection timed out" on AMD64 http://bugs.python.org/issue9774 Most recent 15 issues waiting for review (15) ============================================= #9877: Expose sysconfig._get_makefile_filename() in public API http://bugs.python.org/issue9877 #9865: OrderedDict doesn't implement __sizeof__ http://bugs.python.org/issue9865 #9854: SocketIO should return None on EWOULDBLOCK http://bugs.python.org/issue9854 #9845: Allow changing the method in urllib.request.Request http://bugs.python.org/issue9845 #9824: SimpleCookie should escape commas and semi-colons http://bugs.python.org/issue9824 #9822: windows batch files are dependent on cmd current directory http://bugs.python.org/issue9822 #9815: test_tarfile sometimes ends with error "Cannot remove dir" http://bugs.python.org/issue9815 #9812: cPickle segfault with nested dicts in threaded env http://bugs.python.org/issue9812 #9810: bzip2 build sometimes fails (VS8.0) http://bugs.python.org/issue9810 #9808: Implement os.getlogin on Windows http://bugs.python.org/issue9808 #9807: deriving configuration information for different builds with t http://bugs.python.org/issue9807 #9802: Document 'stability' of builtin min() and max() http://bugs.python.org/issue9802 #9800: Fast path for small int-indexing of lists and tuples http://bugs.python.org/issue9800 #9787: Release the TLS lock during allocations http://bugs.python.org/issue9787 #9786: Native TLS support for pthreads http://bugs.python.org/issue9786 Top 10 most discussed issues (10) ================================= #9867: Interrupted system calls are not retried http://bugs.python.org/issue9867 18 msgs #9807: deriving configuration information for different builds with t http://bugs.python.org/issue9807 14 msgs #9858: Python and C implementations of io are out of sync http://bugs.python.org/issue9858 12 msgs #767645: incorrect os.path.supports_unicode_filenames http://bugs.python.org/issue767645 12 msgs #2636: Regexp 2.7 (modifications to current re 2.2.2) http://bugs.python.org/issue2636 10 msgs #8746: os.chflags() and os.lchflags() are not built when they should http://bugs.python.org/issue8746 9 msgs #9786: Native TLS support for pthreads http://bugs.python.org/issue9786 9 msgs #9880: Python 2.7 Won't Build: SystemError: unknown opcode http://bugs.python.org/issue9880 9 msgs #9815: test_tarfile sometimes ends with error "Cannot remove dir" http://bugs.python.org/issue9815 8 msgs #9810: bzip2 build sometimes fails (VS8.0) http://bugs.python.org/issue9810 7 msgs Issues closed (69) ================== #1178: IDLE - add "paste code" functionality http://bugs.python.org/issue1178 closed by BreamoreBoy #1552: fromfd() and socketpair() should return wrapped sockets http://bugs.python.org/issue1552 closed by pitrou #3030: compiler warning on HP-UX http://bugs.python.org/issue3030 closed by BreamoreBoy #3053: test_shutil fails under AIX http://bugs.python.org/issue3053 closed by pitrou #4617: SyntaxError when free variable name is also an exception targe http://bugs.python.org/issue4617 closed by amaury.forgeotdarc #4837: Omits MACHINE and DEBUG when building tix8.4.3 http://bugs.python.org/issue4837 closed by BreamoreBoy #5016: FileIO.seekable() can return False http://bugs.python.org/issue5016 closed by haypo #5629: PEP 0 date and revision not being set http://bugs.python.org/issue5629 closed by brett.cannon #5934: fix gcc warnings: explicit type conversion for uid/gid in posi http://bugs.python.org/issue5934 closed by haypo #6627: threading.local() does not work with C-created threads http://bugs.python.org/issue6627 closed by ncoghlan #6893: defaultdict example in py3 doc should mention the new Counter http://bugs.python.org/issue6893 closed by rhettinger #7093: xmlrpclib.ServerProxy() doesn't support unicode uri http://bugs.python.org/issue7093 closed by haypo #7994: object.__format__ should reject format strings http://bugs.python.org/issue7994 closed by eric.smith #8197: Fatal error on thread creation in low memory condition: local http://bugs.python.org/issue8197 closed by haypo #8796: Deprecate codecs.open() http://bugs.python.org/issue8796 closed by lemburg #8835: buildbot: support.transient_internet() doesn't catch DNS socke http://bugs.python.org/issue8835 closed by pitrou #8921: 2.7rc1: test_ttk failures on OSX 10.4 http://bugs.python.org/issue8921 closed by ned.deily #9210: remove --with-wctype-functions configure option http://bugs.python.org/issue9210 closed by amaury.forgeotdarc #9212: dict_keys purports to implement the Set ABC, but is missing th http://bugs.python.org/issue9212 closed by stutzbach #9213: range purports to implement the Sequence ABC, but is missing i http://bugs.python.org/issue9213 closed by stutzbach #9313: distutils error on MSVC older than 8 http://bugs.python.org/issue9313 closed by ocean-city #9315: The trace module lacks unit tests http://bugs.python.org/issue9315 closed by belopolsky #9318: Py3k compilation on old MSVC http://bugs.python.org/issue9318 closed by ocean-city #9402: pyexpat: replace PyObject_DEL() by Py_DECREF() to fix a crash http://bugs.python.org/issue9402 closed by haypo #9579: In 3.x, os.confstr() returns garbage if value is longer than 2 http://bugs.python.org/issue9579 closed by haypo #9580: os.confstr() doesn't decode result according to PEP 383 http://bugs.python.org/issue9580 closed by haypo #9608: Re-phrase best way of using exceptions in doanddont.rst http://bugs.python.org/issue9608 closed by r.david.murray #9632: Remove sys.setfilesystemencoding() http://bugs.python.org/issue9632 closed by haypo #9662: ctypes not building under OS X because of ffi_closure_free not http://bugs.python.org/issue9662 closed by ronaldoussoren #9728: Docs point to FAQ Section 3, but FAQs are not numbered http://bugs.python.org/issue9728 closed by stutzbach #9729: Unconnected SSLSocket.{send,recv} raises TypeError: 'member_de http://bugs.python.org/issue9729 closed by pitrou #9769: PyUnicode_FromFormatV() doesn't handle non-ascii text correctl http://bugs.python.org/issue9769 closed by haypo #9809: Wrong Registery Entries on win64 http://bugs.python.org/issue9809 closed by loewis #9817: expat copyright/license file is missing http://bugs.python.org/issue9817 closed by doko #9818: build files to build Lib/distutils/command/wininst-9.0* are mi http://bugs.python.org/issue9818 closed by doko #9820: Windows : os.listdir(b'.') doesn't raise an errorfor unencodab http://bugs.python.org/issue9820 closed by haypo #9821: Support PEP 383 on Windows: mbcs support of surrogateescape er http://bugs.python.org/issue9821 closed by haypo #9823: OrderedDict is comparable to dict http://bugs.python.org/issue9823 closed by benjamin.peterson #9825: OrderedDict ref cycles cause memory leak http://bugs.python.org/issue9825 closed by rhettinger #9826: OrderedDict ref cycles break repr() http://bugs.python.org/issue9826 closed by rhettinger #9827: Clarify LogRecord documentation http://bugs.python.org/issue9827 closed by vinay.sajip #9828: Repeated Py_Initialize and Py_Finalize usage brings fatal erro http://bugs.python.org/issue9828 closed by pitrou #9829: Unexpected Fraction.from_float() Behavior http://bugs.python.org/issue9829 closed by r.david.murray #9830: Python 2.7 x64 not properly installed on Windows 7 (registry) http://bugs.python.org/issue9830 closed by r.david.murray #9832: test_unicodedata fails on MacOSX 10.6 http://bugs.python.org/issue9832 closed by ned.deily #9833: test_ttk_guionly fails on MacOSX 10.6 http://bugs.python.org/issue9833 closed by flox #9834: PySequence_GetSlice() lacks a NULL check http://bugs.python.org/issue9834 closed by benjamin.peterson #9835: ZipFile unix external attributes incorrect for entry written w http://bugs.python.org/issue9835 closed by amaury.forgeotdarc #9836: Refleak in PyUnicode_FormatV http://bugs.python.org/issue9836 closed by haypo #9837: ZipFileExt.read() reads more data than requested http://bugs.python.org/issue9837 closed by pitrou #9839: Test issue http://bugs.python.org/issue9839 closed by loewis #9840: Recursive Repr http://bugs.python.org/issue9840 closed by rhettinger #9843: imp documentation redundancy with acquire_lock and release_loc http://bugs.python.org/issue9843 closed by benjamin.peterson #9847: Binary strings never compare equal to raw/normal strings http://bugs.python.org/issue9847 closed by flox #9848: setup.py contains needless references to built-in _weakref mod http://bugs.python.org/issue9848 closed by brett.cannon #9853: Wrong signature for SSLSocket.recvfrom and SSLSocket.sendto http://bugs.python.org/issue9853 closed by pitrou #9855: Complex number slicing neither works nor causes an error on im http://bugs.python.org/issue9855 closed by eric.smith #9863: threading, signals, atexit: different execution with different http://bugs.python.org/issue9863 closed by r.david.murray #9870: compile and nested scopes http://bugs.python.org/issue9870 closed by georg.brandl #9872: `a.b.my_function is not b.my_function` when `a` and `b` are bo http://bugs.python.org/issue9872 closed by benjamin.peterson #9879: Tracker Won't Accept New Bugs http://bugs.python.org/issue9879 closed by ned.deily #9881: PySSL_SSLread loops until data is available, even in non-block http://bugs.python.org/issue9881 closed by pitrou #9885: Function Round does not work properly in some conditions http://bugs.python.org/issue9885 closed by r.david.murray #1651427: readline needs termcap on my FC6 http://bugs.python.org/issue1651427 closed by BreamoreBoy #1659705: Python extension problems after re-install http://bugs.python.org/issue1659705 closed by elfin24fw #416670: MatchObjects not deepcopy()able http://bugs.python.org/issue416670 closed by BreamoreBoy #941346: AIX shared library fix http://bugs.python.org/issue941346 closed by pitrou #1538778: pyo's are not overwritten by different optimization levels http://bugs.python.org/issue1538778 closed by barry #1648923: HP-UX: -lcurses missing for readline.so http://bugs.python.org/issue1648923 closed by BreamoreBoy From fuzzyman at voidspace.org.uk Fri Sep 17 21:25:39 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 17 Sep 2010 20:25:39 +0100 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <20100917000512.66183149@pitrou.net> References: <20100914121744.31ef5aaf@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916095248.432bd1 2a@mission> <20100916153012.8FCA11BF412@kimball.webabinitio.net> <20100916174053.007c783f@pitro u.net> <20100916205158.5098B1BF4E1@kimball.webabinitio.net> <20100917000512.66183149@pitrou.net> Message-ID: <4C93C0B3.2090905@voidspace.org.uk> On 16/09/2010 23:05, Antoine Pitrou wrote: > On Thu, 16 Sep 2010 16:51:58 -0400 > "R. David Murray" wrote: >> What do we store in the model? We could say that the model is always >> text. But then we lose information about the original bytes message, >> and we can't reproduce it. For various reasons (mailman being a big one), >> this is not acceptable. So we could say that the model is always bytes. >> But we want access to (for example) the header values as text, so header >> lookup should take string keys and return string values[2]. > Why can't you have both in a single class? If you create the class > using a bytes source (a raw message sent by SMTP, for example), the > class automatically parses and decodes it to unicode strings; if you > create the class using an unicode source (the text body of the e-mail > message and the list of recipients, for example), the class > automatically creates the bytes representation. > I think something like this would be great for WSGI. Rather than focus on whether bytes *or* text should be used, use a higher level object that provides a bytes view, and (where possible/appropriate) a unicode view too. Michael > (of course all processing can be done lazily for performance reasons) > >> What about email files on disk? They could be bytes, or they could be, >> effectively, text (for example, utf-8 encoded). > Such a file can be two things: > - the raw encoding of a whole message (including headers, etc.), then > it should be fed as a bytes object > - the single text body of a hypothetical message, then it should be fed > as a unicode object > > I don't see any possible middle-ground. > >> On disk, using utf-8, >> one might store the text representation of the message, rather than >> the wire-format (ASCII encoded) version. We might want to write such >> messages from scratch. > But then the user knows the encoding (by "user" I mean what/whoever > calls the email API) and mentions it to the email package. > > What I'm having an issue with is that you are talking about a bytes > representation and an unicode representation of a message. But they > aren't representations of the same things: > - if it's a bytes representation, it will be the whole, raw message > including envelope / headers (also, MIME sections etc.) > - if it's an unicode representation, it will only be a section of the > message decodable as such (a text/plain MIME section, for example; > or a decoded header value; or even a single e-mail address part of a > decoded header) > > So, there doesn't seem to be any reason for having both a BytesMessage > and an UnicodeMessage at the same abstraction level. They are both > representing different things at different abstraction levels. I don't > see any potential for confusion: raw assembled e-mail message = bytes; > decoded text section of a message = unicode. > > As for the problem of potential "bogus" raw e-mail data > (e.g., undecodable headers), well, I guess the library has to make a > choice between purity and practicality, or perhaps let the user choose > themselves. For example, through a `strict` flag. If `strict` is true, > raise an error as soon as a non-decodable byte appears in a header, if > `strict` is false, decode it through a default (encoding, errors) > convention which can be overriden by the user (a sensible possibility > being "utf-8, surrogateescape" to allow for lossless round-tripping). > >> As I said above, we could insist that files on >> disk be in wire-format, and for many applications that would work fine, >> but I think people would get mad at us if didn't support text files[3]. > Again, this simply seems to be two different abstraction levels: > pre-generated raw email messages including headers, or a single text > waiting to be embedded in an actual e-mail. > >> Anyway, what polymorphism means in email is that if you put in bytes, >> you get a BytesMessage, if you put in strings you get a StringMessage, >> and if you want the other one you convert. > And then you have two separate worlds while ultimately the same > concepts are underlying. A library accepting BytesMessage will crash > when a program wants to give a StringMessage and vice-versa. That > doesn't sound very practical. > >> [1] Now that surrogateesscape exists, one might suppose that strings >> could be used as an 8bit channel, but that only works if you don't need >> to *parse* the non-ASCII data, just transmit it. > Well, you can parse it, precisely. Not only, but it round-trips if you > unparse it again: > >>>> header_bytes = b"From: bogus\xFFname" >>>> name, value = header_bytes.decode("utf-8", "surrogateescape").split(":") >>>> name > 'From' >>>> value > ' bogus\udcffname' >>>> "{0}:{1}".format(name, value).encode("utf-8", "surrogateescape") > b'From: bogus\xffname' > > > In the end, what I would call a polymorphic best practice is "try to > avoid bytes/str polymorphism if your domain is well-defined > enough" (which I admit URLs aren't necessarily; but there's no > question a single text/XXX e-mail section is text, and a whole > assembled e-mail message is bytes). > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ From ianb at colorstudy.com Fri Sep 17 21:44:54 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 17 Sep 2010 15:44:54 -0400 Subject: [Python-Dev] Polymorphic best practices [was: (Not) delaying the 3.2 release] In-Reply-To: <4C93C0B3.2090905@voidspace.org.uk> References: <20100914121744.31ef5aaf@pitrou.net> <20100916000928.0ada59d1@pitrou.net> <20100916142610.6b9bca7b@pitrou.net> <20100916153012.8FCA11BF412@kimball.webabinitio.net> <20100916205158.5098B1BF4E1@kimball.webabinitio.net> <20100917000512.66183149@pitrou.net> <4C93C0B3.2090905@voidspace.org.uk> Message-ID: On Fri, Sep 17, 2010 at 3:25 PM, Michael Foord wrote: > On 16/09/2010 23:05, Antoine Pitrou wrote: > >> On Thu, 16 Sep 2010 16:51:58 -0400 >> "R. David Murray" wrote: >> >>> What do we store in the model? We could say that the model is always >>> text. But then we lose information about the original bytes message, >>> and we can't reproduce it. For various reasons (mailman being a big >>> one), >>> this is not acceptable. So we could say that the model is always bytes. >>> But we want access to (for example) the header values as text, so header >>> lookup should take string keys and return string values[2]. >>> >> Why can't you have both in a single class? If you create the class >> using a bytes source (a raw message sent by SMTP, for example), the >> class automatically parses and decodes it to unicode strings; if you >> create the class using an unicode source (the text body of the e-mail >> message and the list of recipients, for example), the class >> automatically creates the bytes representation. >> >> I think something like this would be great for WSGI. Rather than focus on > whether bytes *or* text should be used, use a higher level object that > provides a bytes view, and (where possible/appropriate) a unicode view too. > This is what WebOb does; e.g., there is only bytes version of a POST body, and a view on that body that does decoding and encoding. If you don't touch something, it is never decoded or encoded. I only vaguely understand the specifics here, and I suspect the specifics matter, but this seems applicable in this case too -- if you have an incoming email with a smattering of bytes, inline (2047) encoding, other encoding declarations, and then orthogonal systems like quoted-printable, you don't want to touch that stuff if you don't need to as handling unicode objects implies you are normalizing the content, and that might have subtle impacts you don't know about, or don't want to know about, or maybe just don't fit into the unicode model (like a string with two character sets). Note that WebOb does not have two views, it has only one view -- unicode viewing bytes. I'm not sure I could keep two views straight. I *think* Antoine is describing two possible canonical data types (unicode or bytes) and two views. That sounds hard. -- Ian Bicking | http://blog.ianbicking.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From ziade.tarek at gmail.com Sat Sep 18 01:04:18 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 18 Sep 2010 01:04:18 +0200 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> Message-ID: On Fri, Sep 17, 2010 at 10:02 PM, Jannis Leidel wrote: > On 17.09.2010, at 20:43, Martin v. L?wis wrote: > >> Here at the DZUG conference, we are planning to integrate explicit access to setuptools metadata to the package index. >> >> The current idea is to store the contents of the egg_info directory, >> giving remote access to specific files. By default, PyPI will extract, >> per release, data from the egg that may get uploaded (using the first >> one if multiple eggs get uploaded). If no egg gets uploaded, a VM >> based build service will generate it from a source distributions. >> Tools like setuptools or distribute could also directly upload this >> information, e.g. as part of the register command. >> >> Any opinions? > > I'm confused, wouldn't that basically be a slap in the face for the people having worked on PEP345 and distutils2, especially during the Summer of Code? > > Also, and I understand enthusiasm tends to build up during conferences, but wouldn't supporting setuptools' egg-info directory again be a step backwards after all those months of discussion about the direction of Python packaging? Yeah, we worked on a new standard that was accepted - PEP 345 PyPI is currently publishing pep 345 info as a matter of fact - I did the patch and there's one package that already uses it http://pypi.python.org/pypi/Distutils2. (no deps on this one, but other stuff like links..) I am about to release the work we did during GSOC in distutils2, a first beta that includes all the work we done. Now you want to publish another metadata format at PyPI ? If PyPI takes that direction and adopts, promotes and publishes a standard that is not the one we worked on in the past year, this will increase our difficulty to push the new format so its adopted by the tools then the community. People will just get confused because they will find two competing metadata formats That's exactly the situation where we were at, and that's exactly where I don't want to go back. I am not even understanding what's the benefit of doing this since an egg_info directory is obtained at *build* time and can differ from a machine to another, so it seems pretty useless for me to publish this. The whole point of PEP 345 is to extend our metadata to statically provide dependencies at PyPI, thanks to a micro-language that allows you to describe dependencies for any platform. We worked hard to build some standards, but if PyPI doesn't help us here, everything we did and are doing is useless. Tarek -- Tarek Ziad? | http://ziade.org From victor.stinner at haypocalc.com Sat Sep 18 01:27:28 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 18 Sep 2010 01:27:28 +0200 Subject: [Python-Dev] Some news from my sandbox project Message-ID: <201009180127.28419.victor.stinner@haypocalc.com> Hi, I'm still developing irregulary my sandbox project since last june. pysandbox is a sandbox to execute untrusted Python code. It is able to execute unmodified Python code with a low overhead. I consider it as stable and secure. http://github.com/haypo/pysandbox/ Today, the biggest problem is the creation of a read only view of the __builtins__ dictionary. I tried to create my own object with the dict API, but I got quickly a segfault. I realized that ceval.c is hardcoded to use PyDict functions on __builtins__ (LOAD_GLOBAL instruction). So I created a subclass of dict and replaced modify function (__setitem__, update, clear, ...). I would like to know if you will agree to modify ceval.c (and maybe some other functions) to support __builtins__ of another type than dict. I mean add a fast check (PyDict_CheckExact) on the type. If you agree, I can will an issue with a patch. The two last vulnerabilities came from this problem: it was possible to use dict methods on __builtins__, eg. dict.update(__builtins__, {...}) and dict.__init__(__builtins__, {...}). Because of that, pysandbox removes all dict methods able to modify a dict. And so "d={...}; d.update(...)" raises an error (d has no update attribute) :-/ --- If you would like to test pysandbox, just join ##fschfsch channel of the Freenode IRC server and talk to fschfsch. It's an IRC bot using pysandbox to evaluate Python expressions. It is also on #python-fr and #python channels, but please use ##fschfsch for tests. http://github.com/haypo/pysandbox/wiki/fschfsch Or you can pysandbox on your computer. Download the last git version (github provides tarballs if you don't have git program), install it and run: python interpreter.py. You have to compile _sandbox, a C module required to modify some Python internals. The last git version is compatible with Python 2.5, 2.6 and 2.7. It works on 3.1 and 3.2 after a conversion with 2to3 and a small hack on sandbox/proxy.py: replace "elif isinstance(value, OBJECT_TYPES):" by "else:" (and remove the existing else statement). I'm not sure that this hack is safe, and so I didn't commited it yet. -- Victor Stinner http://www.haypocalc.com/ From sridharr at activestate.com Sat Sep 18 02:23:01 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Fri, 17 Sep 2010 17:23:01 -0700 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> Message-ID: <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> On 2010-09-17, at 4:04 PM, Tarek Ziad? wrote: > I am not even understanding what's the benefit of doing this since an > egg_info directory is obtained at *build* time and can differ from a > machine to another, so it seems pretty useless for me to publish this. I am in full agreement with Tarek here. At ActiveState, we maintain our own index that differs from PyPI in two ways (among others): - use setuptools.package_index to scrap sdists for packages that don't upload them to pypi - PKG-INFO and requires.txt are extracted (if doesn't exist, generate using egg_info cmd) from each of the sdist (and then our index provides the full metadata - with internal links to sdists - as a sqlite db for the builder processes on each platform) The problem with extracting PKG-INFO and requires.txt on the index server is that, the contents of requires.txt sometimes differ based on which platform and Python version on which the egg_info command was run. For eg., the "tox" project depends[1] on "virtualenv" package if it is run using Python2, but not on Python3. > The whole point of PEP 345 is to extend our metadata to statically > provide dependencies at PyPI, thanks to a micro-language that allows > you to describe dependencies for any platform. Static metadata would allow packages like "tox" to configure Python version / platform specific dependencies without resorting to runtime checks. > We worked hard to build some standards, but if PyPI doesn't help us > here, everything we did and are doing is useless. Ideally, in future - I should be able to query static metadata (with environment markers[2] and such) for *any* package from PyPI. And this static metadata is simply a DIST-INFO file (instead of being a directory with a bunch of files in it). I don't really see a point in providing access to, say, the list of entry points of each package. As for as package managers is concerned, the only things that matter are a) list of package names and versions, b) source tarball for each release c) and the corresponding metadata with dependency information. -srid [1] http://code.google.com/p/pytox/source/browse/setup.py#30 [2] http://www.python.org/dev/peps/pep-0345/#environment-markers From martin at v.loewis.de Sat Sep 18 09:52:01 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 18 Sep 2010 09:52:01 +0200 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> Message-ID: <4C946FA1.2030705@v.loewis.de> > I am in full agreement with Tarek here. At ActiveState, we maintain > our own index that differs from PyPI in two ways (among others): I think you are saying something very different from what Tarek says. IIUC, you are saying that egg-info is ill-defined and may cause subtle problems. So you are saying there is a problem for the users of the data. I could live with that - it's the user's choice to use these data, after all. Tarek is saying that it will be evil and bad for the community to unpack some zip files. I find that statement itself counter-productive. > Ideally, in future - I should be able to query static metadata (with > environment markers[2] and such) for *any* package from PyPI. And > this static metadata is simply a DIST-INFO file (instead of being a > directory with a bunch of files in it). I don't really see a point in > providing access to, say, the list of entry points of each package. Again, that is completely different from what Tarek is saying. You said (just now, and literally): "I don't really see a point". Converting this to the -1/0/+1 system, it sounds like +0 to me: you are unlikely to ever use this data. This is perfectly fine with me: I won't use the data myself, either. However, I fail to see why this should stop me from providing the data, when there are people requesting this feature over and over again. I'd like to see some of Python's "consenting adults" attitude here. Regards, Martin From greg.ewing at canterbury.ac.nz Sat Sep 18 10:24:49 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 18 Sep 2010 20:24:49 +1200 Subject: [Python-Dev] Some news from my sandbox project In-Reply-To: <201009180127.28419.victor.stinner@haypocalc.com> References: <201009180127.28419.victor.stinner@haypocalc.com> Message-ID: <4C947751.6050605@canterbury.ac.nz> Victor Stinner wrote: > I'm still developing irregulary my sandbox project since last june. > Today, the biggest problem is the creation of a read only view of the > __builtins__ dictionary. Why do you think you need to do this? What form of attack would a writable __builtins__ expose you to that would be prevented by making it read-only? Seems to me that the only way you could exploit a writable __builtins__ would be to put a function in there that does something bad. But if you can create or obtain such a function in the first place, you can just call it directly. -- Greg From robertc at robertcollins.net Sat Sep 18 10:39:58 2010 From: robertc at robertcollins.net (Robert Collins) Date: Sat, 18 Sep 2010 20:39:58 +1200 Subject: [Python-Dev] Some news from my sandbox project In-Reply-To: <4C947751.6050605@canterbury.ac.nz> References: <201009180127.28419.victor.stinner@haypocalc.com> <4C947751.6050605@canterbury.ac.nz> Message-ID: On Sat, Sep 18, 2010 at 8:24 PM, Greg Ewing wrote: > Victor Stinner wrote: > >> I'm still developing irregulary my sandbox project since last june. > >> Today, the biggest problem is the creation of a read only view of the >> __builtins__ dictionary. > > Why do you think you need to do this? What form of attack > would a writable __builtins__ expose you to that would be > prevented by making it read-only? > > Seems to me that the only way you could exploit a writable > __builtins__ would be to put a function in there that does > something bad. But if you can create or obtain such a > function in the first place, you can just call it directly. __builtins__ is in everyone's global namespace, so if it can be mutated, different python programs running in the same sandbox can affect each other. Ditto sys.modules and os environ, but I guess that those are already addressed. -Rob From thomas at thomas-lotze.de Sat Sep 18 11:29:32 2010 From: thomas at thomas-lotze.de (Thomas Lotze) Date: Sat, 18 Sep 2010 11:29:32 +0200 Subject: [Python-Dev] egg_info in PyPI References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> Message-ID: Hi there, I'm going to add my own 2 cents to the discussion as I'm involved in the matter here at the DZUG conference. Tarek Ziad? wrote: > Now you want to publish another metadata format at PyPI ? If PyPI takes > that direction and adopts, promotes and publishes a standard that is not > the one we worked on in the past year, this will increase our difficulty > to push the new format so its adopted by the tools then the community. Actually, I see it being the other way around: People use what used to work for them and most people won't switch to using the new standard just for the fun of it. In order for the new standard to acquire some momentum, people need to be interested in the subject the standard is about and see some concrete advantage in switching. Exposing existing metadata will make it possible to build new applications that use metadata information; it's those applications' responsibility then to promote the new standard. Let me take the tl.eggdeps package as a concrete example. The package currently analyses dependencies between installed packages. I'd like to expand it to analyse dependencies between any packages on PyPI but I can't as long as dependency information is not available without actually installing things. Exposing PEP 345 metadata on PyPI won't be of any practical use until a critical number of packages actually specify that information. I simply won't bother implementing the feature either until some point far in the future or unless I can use legacy metadata as a fall-back. Having the legacy metadata available enables me to build the feature now; this could raise interest in metadata as such, and I might even make visually apparent which dependencies have been specified by PEP 345 means and which I could only guess from legacy information. > People will just get confused because they will find two competing > metadata formats That's exactly the situation where we were at, and > that's exactly where I don't want to go back. That's simply a matter of documentation. It has to be clearly communicated that the legacy information is provided as a transitional service and that users of that metadata use it only until the newly specified information is available for a critical number of packages (the exact value of which may of course depend on the application in question). > I am not even understanding what's the benefit of doing this since an > egg_info directory is obtained at *build* time and can differ from a > machine to another, so it seems pretty useless for me to publish this. We do understand that legacy metadata has its issues, but taken with the appropriate amount of salt, it's better than nothing in most cases, enough so at least to solve some chicken-and-egg problems. > We worked hard to build some standards, but if PyPI doesn't help us here, > everything we did and are doing is useless. Nothing that we're talking about in this thread will contradict what you want to achieve and what work you've done. By the reasoning I tried to explain above, what PyPI is going to do will actually help you. OTOH, I personally don't think it would be a constructive way of helping you if PyPI tried to enforce the new standard by keeping people from using information they may at the moment only get through legacy means. -- Thomas From fuzzyman at voidspace.org.uk Sat Sep 18 11:48:28 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 18 Sep 2010 10:48:28 +0100 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C946FA1.2030705@v.loewis.de> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> Message-ID: <4C948AEC.90607@voidspace.org.uk> On 18/09/2010 08:52, "Martin v. L?wis" wrote: >> I am in full agreement with Tarek here. At ActiveState, we maintain >> our own index that differs from PyPI in two ways (among others): > > I think you are saying something very different from what Tarek > says. IIUC, you are saying that egg-info is ill-defined and may > cause subtle problems. So you are saying there is a problem for > the users of the data. I could live with that - it's the user's > choice to use these data, after all. > > Tarek is saying that it will be evil and bad for the community > to unpack some zip files. I find that statement itself counter-productive. With the distutils2 work very close to landing in the standard library, providing infrastructure that will cause tools to *depend* on the old formats is a very bad idea. If tool use this metadata then it could well prevent packages that want to be compatible with these tools from using distutils2. What PyPI does effectively becomes "the standard" for a large chunk of the Python world (which is why changing the format PyPI provides data in can be so hard). Now seems a really dumb time to bless the setuptools metadata format as the defacto standard after so much work has gone into replacing it and that effort is so close to completion. So - I agree with Tarek. Exposing this information on PyPI would be problematic for the Python community. Not only does the data have the problems that Tarek and Sridhar point out, but it would actively hinder adoption of the better replacement. All the best, Michael Foord > > >> Ideally, in future - I should be able to query static metadata (with >> environment markers[2] and such) for *any* package from PyPI. And >> this static metadata is simply a DIST-INFO file (instead of being a >> directory with a bunch of files in it). I don't really see a point in >> providing access to, say, the list of entry points of each package. > > Again, that is completely different from what Tarek is saying. > > You said (just now, and literally): "I don't really see a point". > Converting this to the -1/0/+1 system, it sounds like +0 to me: > you are unlikely to ever use this data. > > This is perfectly fine with me: I won't use the data myself, either. > However, I fail to see why this should stop me from providing the > data, when there are people requesting this feature over and over > again. I'd like to see some of Python's "consenting adults" attitude > here. > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ From victor.stinner at haypocalc.com Sat Sep 18 11:58:55 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 18 Sep 2010 11:58:55 +0200 Subject: [Python-Dev] Some news from my sandbox project In-Reply-To: <4C947751.6050605@canterbury.ac.nz> References: <201009180127.28419.victor.stinner@haypocalc.com> <4C947751.6050605@canterbury.ac.nz> Message-ID: <201009181158.55359.victor.stinner@haypocalc.com> Le samedi 18 septembre 2010 10:24:49, Greg Ewing a ?crit : > Victor Stinner wrote: > > I'm still developing irregulary my sandbox project since last june. > > > > Today, the biggest problem is the creation of a read only view of the > > __builtins__ dictionary. > > Why do you think you need to do this? What form of attack > would a writable __builtins__ expose you to that would be > prevented by making it read-only? Replace builtin functions make it possible to modify (indirectly) functions outside the sandbox. Eg. one of the most important function of pysandbox is proxy() (a function to create a read only view of a object outside the sandbox, especially on an import), if you replace isinstance() by a function which always return True: you can get unmodified objects (read+write access). So it's possible to get the real file type from sys.stdout and so open a new file. The example on a modified version of pysandbox (to get the vulnerability): sandbox>>> __builtins__['isinstance']=lambda obj, cls: True sandbox>>> import sys sandbox>>> type(sys.stdout)('/etc/issue').read() 'Debian GNU/Linux squeeze/sid \\n \\l\n\n' --- It would be possible to create a local copy of each builtin in all pysandbox functions, but it doesn't protect functions outside pysandbox. -- Victor Stinner http://www.haypocalc.com/ From martin at v.loewis.de Sat Sep 18 12:03:58 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 18 Sep 2010 12:03:58 +0200 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C948AEC.90607@voidspace.org.uk> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> Message-ID: <4C948E8E.7080403@v.loewis.de> > With the distutils2 work very close to landing in the standard library, > providing infrastructure that will cause tools to *depend* on the old > formats is a very bad idea. I think you are misunderstanding. The infrastructure will *not* depend on the old formats. Instead, packaging that have that information will provide it, packages that don't will not. The infrastructure is entirely agnostic on whether the data is available or not. In particular, it will not try to interpret the data in any way. > If tool use this metadata then it could well > prevent packages that want to be compatible with these tools from using > distutils2. I don't think this can well happen. In most known use cases, the tools could support both forms of metadata. It may be that tools want to use information that is not provided by PEP 345. However, the tool will then likely fall back to just not having the information, as even existing packages only provide that information occasionally. > What PyPI does effectively becomes "the standard" for a large chunk of > the Python world (which is why changing the format PyPI provides data in > can be so hard). Now seems a really dumb time to bless the setuptools > metadata format as the defacto standard after so much work has gone into > replacing it and that effort is so close to completion. Please understand that the information is not being "blessed" (in any sense of the word that comes out of the dictionary). It's just being made available slightly more conveniently - please understand that it was available all the years already. > So - I agree with Tarek. Exposing this information on PyPI would be > problematic for the Python community. Not only does the data have the > problems that Tarek and Sridhar point out, but it would actively hinder > adoption of the better replacement. That's really sad. So people will have to wait a few years to efficiently implement tools that they could implement today. Regards, Martin From victor.stinner at haypocalc.com Sat Sep 18 12:05:26 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 18 Sep 2010 12:05:26 +0200 Subject: [Python-Dev] Some news from my sandbox project In-Reply-To: References: <201009180127.28419.victor.stinner@haypocalc.com> <4C947751.6050605@canterbury.ac.nz> Message-ID: <201009181205.26875.victor.stinner@haypocalc.com> Le samedi 18 septembre 2010 10:39:58, Robert Collins a ?crit : > __builtins__ is in everyone's global namespace, so if it can be > mutated, different python programs running in the same sandbox can > affect each other. > > Ditto sys.modules and os environ, but I guess that those are already > addressed. sys.modules and os.environ are not accessible in pysandbox :-) If you create a rule in the security policy to allow them, you will get read only views. Example with sys.modules: sandbox>>> from sys import modules sandbox>>> modules['sys'] TypeError: Unable to proxy a value of type sandbox>>> modules['sys']=1 SandboxError: Read only object Builtin module type is blocked because it is unsafe. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Sep 18 12:18:10 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 18 Sep 2010 12:18:10 +0200 Subject: [Python-Dev] [Python-checkins] r84877 - python/branches/release27-maint/Lib/posixpath.py In-Reply-To: <20100918054044.DF551EE9C9@mail.python.org> References: <20100918054044.DF551EE9C9@mail.python.org> Message-ID: <201009181218.10351.victor.stinner@haypocalc.com> Le samedi 18 septembre 2010 07:40:44, hirokazu.yamamoto a ?crit : > Author: hirokazu.yamamoto > Date: Sat Sep 18 07:40:44 2010 > New Revision: 84877 > > Log: > Added missing import. > > Modified: python/branches/release27-maint/Lib/posixpath.py > =========================================================================== > === --- python/branches/release27-maint/Lib/posixpath.py (original) > +++ python/branches/release27-maint/Lib/posixpath.py Sat Sep 18 07:40:44 > 2010 @@ -11,6 +11,7 @@ > """ > > import os > +import sys Woops, thanks :-) -- Victor Stinner http://www.haypocalc.com/ From fuzzyman at voidspace.org.uk Sat Sep 18 12:39:36 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 18 Sep 2010 11:39:36 +0100 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C948E8E.7080403@v.loewis.de> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> Message-ID: <4C9496E8.1020405@voidspace.org.uk> On 18/09/2010 11:03, "Martin v. L?wis" wrote: >> With the distutils2 work very close to landing in the standard library, >> providing infrastructure that will cause tools to *depend* on the old >> formats is a very bad idea. > > I think you are misunderstanding. The infrastructure will *not* depend > on the old formats. Instead, packaging that have that information will > provide it, packages that don't will not. The infrastructure is entirely > agnostic on whether the data is available or not. In particular, it will > not try to interpret the data in any way. > No, not at all. If tools *use* the information (and if they don't then what is the point?) then packages that want to be compatible with those tools will have to provide this information. That will require those packages to use setuptools and prevent them using distutils2. By providing information in this format PyPI will (like it or not) be blessing this format as the 'standard' way of providing this information. That I think is a very bad idea at this point in the evolution of Python packaging tools. >> If tool use this metadata then it could well >> prevent packages that want to be compatible with these tools from using >> distutils2. > > I don't think this can well happen. In most known use cases, the tools > could support both forms of metadata. Well, a) I would like to see that demonstrated and b) having one standard is *far* preferable and having the distutils2 format be that standard is also far preferable. Please wait a bit (or start on supporting the distutils2 metadata format now). > It may be that tools want to use > information that is not provided by PEP 345. However, the tool will then > likely fall back to just not having the information, as even existing > packages only provide that information occasionally. > >> What PyPI does effectively becomes "the standard" for a large chunk of >> the Python world (which is why changing the format PyPI provides data in >> can be so hard). Now seems a really dumb time to bless the setuptools >> metadata format as the defacto standard after so much work has gone into >> replacing it and that effort is so close to completion. > > Please understand that the information is not being "blessed" > (in any sense of the word that comes out of the dictionary). It's just > being made available slightly more conveniently - please understand that > it was available all the years already. > I'm sorry but this is wrong. By providing this information about packages in the *standard* Python package repository it is very much (whether intentionally or not) blessing it as a standard. >> So - I agree with Tarek. Exposing this information on PyPI would be >> problematic for the Python community. Not only does the data have the >> problems that Tarek and Sridhar point out, but it would actively hinder >> adoption of the better replacement. > > That's really sad. So people will have to wait a few years to > efficiently implement tools that they could implement today. Why a few years? All the best, Michael > > Regards, > Martin -- http://www.ironpythoninaction.com/ From cournape at gmail.com Sat Sep 18 12:48:46 2010 From: cournape at gmail.com (David Cournapeau) Date: Sat, 18 Sep 2010 19:48:46 +0900 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C9496E8.1020405@voidspace.org.uk> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> Message-ID: On Sat, Sep 18, 2010 at 7:39 PM, Michael Foord wrote: > ?On 18/09/2010 11:03, "Martin v. L?wis" wrote: >> That's really sad. So people will have to wait a few years to efficiently >> implement tools that they could implement today. > > Why a few years? That's the time it will take for all packages to support distutils2 ? cheers, David From fuzzyman at voidspace.org.uk Sat Sep 18 12:50:04 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 18 Sep 2010 11:50:04 +0100 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> Message-ID: <4C94995C.5090200@voidspace.org.uk> On 18/09/2010 11:48, David Cournapeau wrote: > On Sat, Sep 18, 2010 at 7:39 PM, Michael Foord > wrote: >> On 18/09/2010 11:03, "Martin v. L?wis" wrote: >>> That's really sad. So people will have to wait a few years to efficiently >>> implement tools that they could implement today. >> Why a few years? > That's the time it will take for all packages to support distutils2 ? Not "all packages" support setuptools. Michael > cheers, > > David -- http://www.ironpythoninaction.com/ From cournape at gmail.com Sat Sep 18 13:01:07 2010 From: cournape at gmail.com (David Cournapeau) Date: Sat, 18 Sep 2010 20:01:07 +0900 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C94995C.5090200@voidspace.org.uk> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94995C.5090200@voidspace.org.uk> Message-ID: On Sat, Sep 18, 2010 at 7:50 PM, Michael Foord wrote: > ?On 18/09/2010 11:48, David Cournapeau wrote: >> >> On Sat, Sep 18, 2010 at 7:39 PM, Michael Foord >> ?wrote: >>> >>> ?On 18/09/2010 11:03, "Martin v. L?wis" wrote: >>>> >>>> That's really sad. So people will have to wait a few years to >>>> efficiently >>>> implement tools that they could implement today. >>> >>> Why a few years? >> >> That's the time it will take for all packages to support distutils2 ? > > Not "all packages" support setuptools. Sure, but supporting setuptools was kind of possible for packages relying heavily on distutils, even if it was not simple and fragile. Distutils2 being incompatible API-wise with distutils, I am not sure it will be as "easy" as with setuptools. It may be, but the only way to know is to do it, and the incentive rather unclear. It means that anyway, a lot of infrastructure will have to support both "standards" for the time being. cheers, David From martin at v.loewis.de Sat Sep 18 13:25:34 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 18 Sep 2010 13:25:34 +0200 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C9496E8.1020405@voidspace.org.uk> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> Message-ID: <4C94A1AE.7010202@v.loewis.de> >> I think you are misunderstanding. The infrastructure will *not* depend >> on the old formats. Instead, packaging that have that information will >> provide it, packages that don't will not. The infrastructure is entirely >> agnostic on whether the data is available or not. In particular, it will >> not try to interpret the data in any way. >> > No, not at all. If tools *use* the information (and if they don't then > what is the point?) then packages that want to be compatible with those > tools will have to provide this information. Not true. Tools could/should also support PEP 345 data, and then they can support either kind of package. > By providing information in this format PyPI will (like it or not) be > blessing this format as the 'standard' way of providing this > information. By that definition, *both* formats are "blessed". The PEP 345 data is already blessed. Depending on the definition of "providing", the egg-info data are also already "provided", ever since PyPI started accepting egg files. >> I don't think this can well happen. In most known use cases, the tools >> could support both forms of metadata. > Well, a) I would like to see that demonstrated The tool in question is tl.eggdepend. It can easily support both kinds of metadata. > and b) having one > standard is *far* preferable and having the distutils2 format be that > standard is also far preferable. Please wait a bit (or start on > supporting the distutils2 metadata format now). The latter is already the case: the distutils2 metadata *is* supported *now*. It's just that no package is using it (except for pep345demo). As for a bit: how long exactly? >> That's really sad. So people will have to wait a few years to >> efficiently implement tools that they could implement today. > > Why a few years? Because it will take that long until a significant number of packages will use distutils 2. People still use very old versions of packaging tools (e.g. the ones that come with Debian); it will just take time to get the new tools and API adopted. Regards, Martin From fuzzyman at voidspace.org.uk Sat Sep 18 14:24:03 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 18 Sep 2010 13:24:03 +0100 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C94A1AE.7010202@v.loewis.de> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> Message-ID: <4C94AF63.8010502@voidspace.org.uk> On 18/09/2010 12:25, "Martin v. L?wis" wrote: >>> I think you are misunderstanding. The infrastructure will *not* depend >>> on the old formats. Instead, packaging that have that information will >>> provide it, packages that don't will not. The infrastructure is >>> entirely >>> agnostic on whether the data is available or not. In particular, it >>> will >>> not try to interpret the data in any way. >>> >> No, not at all. If tools *use* the information (and if they don't then >> what is the point?) then packages that want to be compatible with those >> tools will have to provide this information. > > Not true. Tools could/should also support PEP 345 data, and then they > can support either kind of package. > But you are proposing that PyPI will provide an interface / API for exposing the setuptools information. So tools that get metadata from PyPI in this way (and depend on it - as they will if you expose it) will have functionality that only works for packages that provide this information in this form. Saying tools "should" work with other formats too is empty words. >> By providing information in this format PyPI will (like it or not) be >> blessing this format as the 'standard' way of providing this >> information. > > By that definition, *both* formats are "blessed". The PEP 345 data > is already blessed. Depending on the definition of "providing", the > egg-info data are also already "provided", ever since PyPI started > accepting egg files. > No. See above comment. If exposing this information has no value then don't do it. If it does have value, then we are blessing it - and therefore blessing it *over* other formats. I accept this is not your intention. It *will* be the effect. >>> I don't think this can well happen. In most known use cases, the tools >>> could support both forms of metadata. >> Well, a) I would like to see that demonstrated > > The tool in question is tl.eggdepend. It can easily support both kinds > of metadata. > I couldn't find any references "tl.eggdepend" on the web. It is a minor point though. >> and b) having one >> standard is *far* preferable and having the distutils2 format be that >> standard is also far preferable. Please wait a bit (or start on >> supporting the distutils2 metadata format now). > > The latter is already the case: the distutils2 metadata *is* supported > *now*. As in exported by PyPI though an API / interface? If not then again, irrelevant. Tools that use the new interface you are proposing *won't* use that information. Saying that they *could* is more empty words if our *actions* promote the use of another format. > It's just that no package is using it (except for pep345demo). > > As for a bit: how long exactly? Distutils2 1a2 will be released in the next few days. > >>> That's really sad. So people will have to wait a few years to >>> efficiently implement tools that they could implement today. >> >> Why a few years? > > Because it will take that long until a significant number of > packages will use distutils 2. People still use very old versions > of packaging tools (e.g. the ones that come with Debian); it will > just take time to get the new tools and API adopted. Promoting another format in preference to distutils2 will very much prolong that. Michael > > Regards, > Martin -- http://www.ironpythoninaction.com/ From martin at v.loewis.de Sat Sep 18 15:21:49 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 18 Sep 2010 15:21:49 +0200 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C94AF63.8010502@voidspace.org.uk> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> Message-ID: <4C94BCED.8010405@v.loewis.de> > No. See above comment. If exposing this information has no value then > don't do it. If it does have value, then we are blessing it - and > therefore blessing it *over* other formats. No: not *over*. Only over formats that don't get exposed. However, the PEP 345 data are *already* exposed, via HTML, JSON, XML-RPC. So they are much more prominently presented than what I'm proposing to do. I fail to see why just extracting the egg-info would be exposing it *over* the PEP 345 data. >> The tool in question is tl.eggdepend. It can easily support both kinds >> of metadata. >> > I couldn't find any references "tl.eggdepend" on the web. It is a minor > point though. Oops, see http://pypi.python.org/pypi/tl.eggdeps > As in exported by PyPI though an API / interface? Sure. See, for example, http://pypi.python.org/pypi/pep345demo/json It's also available through the XML-RPC release_data. > Saying that they *could* is more empty words if > our *actions* promote the use of another format. But they do. Please stop saying that they might not, when they actually do (and have been for a while). >> It's just that no package is using it (except for pep345demo). >> >> As for a bit: how long exactly? > Distutils2 1a2 will be released in the next few days. Sure. But when can tools computing dependencies for widely used packages actually expect that these metadata will be available? > Promoting another format in preference to distutils2 will very much > prolong that. IT WILL BE NOT IN PREFERENCE TO DISTUTILS2. Regards, Martin From steve at holdenweb.com Sat Sep 18 15:27:12 2010 From: steve at holdenweb.com (Steve Holden) Date: Sat, 18 Sep 2010 09:27:12 -0400 Subject: [Python-Dev] egg_info in PyPI In-Reply-To: <4C94BCED.8010405@v.loewis.de> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> <4C94BCED.8010405@v.loewis.de> Message-ID: On 9/18/2010 9:21 AM, "Martin v. L?wis" wrote: > IT WILL BE NOT IN PREFERENCE TO DISTUTILS2. No need to shout. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From fuzzyman at voidspace.org.uk Sat Sep 18 16:01:32 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 18 Sep 2010 15:01:32 +0100 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C94BCED.8010405@v.loewis.de> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> <4C94BCED.8010405@v.loewis.de> Message-ID: <4C94C63C.1060505@voidspace.org.uk> Ok, I'm sorry - PEP 345 information is available via the PyPI API. (So exposing egg_info would not be promoting it *over* distutils2 but it would still be promoting and blessing it). Tarek's main point still stands though. The dependency information in the egg_info is tied to the platform and Python version that the package was *built* on. Neither pip nor easy_install use the egg_info to determine this; instead they re-generate it to get the correct information for the target platform. So if the use case is to provide dependency information exposing egg_info is not the right way to do it - and tools that use it will be using potentially (and frequently) inaccurate information. I stand by the point that once we start providing this information tools will start using it, and they shouldn't be. (So your question "when can tools computing dependencies for widely used packages actually expect that these metadata will be available?" is not answered by providing access to egg_info.) This problem (static availability of dependency information) is *exactly* the problem PEP 345 solves, so we should be pushing for progress on this front (distutils2a1 is already out and distutils2a2 will be out soon). All the best, Michael Foord On 18/09/2010 14:21, "Martin v. L?wis" wrote: >> No. See above comment. If exposing this information has no value then >> don't do it. If it does have value, then we are blessing it - and >> therefore blessing it *over* other formats. > > No: not *over*. Only over formats that don't get exposed. However, > the PEP 345 data are *already* exposed, via HTML, JSON, XML-RPC. > So they are much more prominently presented than what I'm proposing > to do. I fail to see why just extracting the egg-info would be > exposing it *over* the PEP 345 data. > >>> The tool in question is tl.eggdepend. It can easily support both kinds >>> of metadata. >>> >> I couldn't find any references "tl.eggdepend" on the web. It is a minor >> point though. > > Oops, see http://pypi.python.org/pypi/tl.eggdeps > >> As in exported by PyPI though an API / interface? > > Sure. See, for example, > > http://pypi.python.org/pypi/pep345demo/json > > It's also available through the XML-RPC release_data. > >> Saying that they *could* is more empty words if >> our *actions* promote the use of another format. > > But they do. Please stop saying that they might not, when they > actually do (and have been for a while). > >>> It's just that no package is using it (except for pep345demo). >>> >>> As for a bit: how long exactly? >> Distutils2 1a2 will be released in the next few days. > > Sure. But when can tools computing dependencies for widely used packages > actually expect that these metadata will be available? > >> Promoting another format in preference to distutils2 will very much >> prolong that. > > IT WILL BE NOT IN PREFERENCE TO DISTUTILS2. > > Regards, > Martin -- http://www.ironpythoninaction.com/ From ziade.tarek at gmail.com Sat Sep 18 16:58:06 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 18 Sep 2010 16:58:06 +0200 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C94C63C.1060505@voidspace.org.uk> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> <4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk> Message-ID: On Sat, Sep 18, 2010 at 4:01 PM, Michael Foord wrote: > ?Ok, I'm sorry - PEP 345 information is available via the PyPI API. (So > exposing egg_info would not be promoting it *over* distutils2 but it would > still be promoting and blessing it). > > Tarek's main point still stands though. The dependency information in the > egg_info is tied to the platform and Python version that the package was > *built* on. Neither pip nor easy_install use the egg_info to determine this; > instead they re-generate it to get the correct information for the target > platform. > > So if the use case is to provide dependency information exposing egg_info is > not the right way to do it - and tools that use it will be using potentially > (and frequently) inaccurate information. I stand by the point that once we > start providing this information tools will start using it, and they > shouldn't be. (So your question "when can tools computing dependencies for > widely used packages actually expect that these metadata will be available?" > is not answered by providing access to egg_info.) > > This problem (static availability of dependency information) is *exactly* > the problem PEP 345 solves, so we should be pushing for progress on this > front (distutils2a1 is already out and distutils2a2 will be out soon). Sorry to take back the discussion late I was away for a while. My main concern, like Michael says, is to see PyPI publish some information that are known to be accurate only on one platform, the one that is used to create the egg_info. Granted, most of the time the information is the same no matter what the platform is, but still: you don't know if this is the case when you read those metadata . The only way to do it properly is to re-run egg_info. So, right now Pip and easy_install are running egg_info to re-build those data before they start the install process. They re-generate the egg_info every time, and then browse the list of dependencies to get them. If you upload the egg_info data at PyPI so people can browse the dependencies, you basically make the assumption that the egg_info produced will work no matter what the platform is, unless you associate with the egg_info the platform that was used. So, I don't understand what is the benefit here, since a serious installer will re-run egg_info every time. Coming from the Plone/buildout community, I would be concerned if buildout would rely on this. json, ldap, mysql you name it --I have tons of other examples-- all those dependencies will not be accurate unless you re-run setup.py egg_info. "Good enough metadata" sounds completely wrong to me. Cheers Tarek -- Tarek Ziad? | http://ziade.org From martin at v.loewis.de Sat Sep 18 17:10:46 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 18 Sep 2010 17:10:46 +0200 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C94C63C.1060505@voidspace.org.uk> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> <4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk> Message-ID: <4C94D676.8080703@v.loewis.de> > So if the use case is to provide dependency information exposing > egg_info is not the right way to do it - and tools that use it will be > using potentially (and frequently) inaccurate information. I stand by > the point that once we start providing this information tools will start > using it, and they shouldn't be. I think this is precisely the flaw in this line of reasoning: that we refuse a service because we think the data is useless. Why can't the users decide for themselves whether the data is useless? As Thomas points out, people running into this will actually have a reason to migrate to PEP 345. So exposing this presumed-incorrect information will actually help promoting PEP 345. > This problem (static availability of dependency information) is > *exactly* the problem PEP 345 solves, so we should be pushing for > progress on this front (distutils2a1 is already out and distutils2a2 > will be out soon). And I don't mind pushing it, unless that means to deny users service now based on the promise of offering better service tomorrow. Regards, Martin From martin at v.loewis.de Sat Sep 18 17:19:46 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 18 Sep 2010 17:19:46 +0200 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> <4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk> Message-ID: <4C94D892.4070708@v.loewis.de> > So, I don't understand what is the benefit here, since a serious > installer will re-run egg_info every time. I think the main applications that people are after are not builds. They want the dependency information without downloading the packages, and dependency information for packages they have no plans to install. In the specific case of tl.eggdeps, the dependency information is only used to create printable graphs. If this turns out to be slightly incorrect, people would notice if they try to use the packages in question. > Coming from the Plone/buildout community, I would be concerned if > buildout would rely on this. json, ldap, mysql you name it --I have > tons of other examples-- all those dependencies will not be accurate > unless you re-run setup.py egg_info. "Good enough metadata" sounds > completely wrong to me. Still, people ask for it. I'm fine with telling them that the data is flawed for various reasons. I object to denying them the data, and I really dislike having to discard the patch that I already wrote to implement it. Regards, Martin From ziade.tarek at gmail.com Sat Sep 18 17:27:33 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 18 Sep 2010 17:27:33 +0200 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C94D892.4070708@v.loewis.de> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> <4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk> <4C94D892.4070708@v.loewis.de> Message-ID: 2010/9/18 "Martin v. L?wis" : >> So, I don't understand what is the benefit here, since a serious >> installer will re-run egg_info every time. > > I think the main applications that people are after are not builds. > They want the dependency information without downloading the packages, > and dependency information for packages they have no plans to install. Yes they want to build the graph of dependencies, which will be potentially false, as I explained. > > In the specific case of tl.eggdeps, the dependency information is only > used to create printable graphs. If this turns out to be slightly incorrect, > people would notice if they try to use the packages in > question. So you are fine with publishing "slightly incorrect" metadata at PyPI ? I am not. > >> Coming from the Plone/buildout community, I would be concerned if >> buildout would rely on this. json, ldap, mysql you name it --I have >> tons of other examples-- ?all those dependencies will not be accurate >> unless you re-run setup.py egg_info. "Good enough metadata" sounds >> completely wrong to me. > > Still, people ask for it. I'm fine with telling them that the data > is flawed for various reasons. I object to denying them the data, > and I really dislike having to discard the patch that I already > wrote to implement it. The can have those data, by downloading the tarball of the project and running the egg_info command. They will in this case have accurate data in fact. So right now you don't deny them the data, it's one command away from them. I don't understand the rational behind providing flawed data at PyPI. I am -1 on that. Regards Tarek > > Regards, > Martin > -- Tarek Ziad? | http://ziade.org From pje at telecommunity.com Sat Sep 18 17:49:44 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sat, 18 Sep 2010 11:49:44 -0400 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C94D892.4070708@v.loewis.de> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> <4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk> <4C94D892.4070708@v.loewis.de> Message-ID: <20100918154948.96C763A403D@sparrow.telecommunity.com> At 05:19 PM 9/18/2010 +0200, Martin v. L?wis wrote: >In the specific case of tl.eggdeps, the dependency information is only >used to create printable graphs. If this turns out to be slightly >incorrect, people would notice if they try to use the packages in >question. By the way, just providing this information for .egg files and *not* for sdists would ensure accuracy of the metadata for that platform/python version. From stephen at xemacs.org Sat Sep 18 17:46:45 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 19 Sep 2010 00:46:45 +0900 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> <4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk> Message-ID: <8762y3xdei.fsf@uwakimon.sk.tsukuba.ac.jp> Tarek Ziad? writes: > "Good enough metadata" sounds completely wrong to me. I hate to break it to you, but experience shows that the XEmacs package system, whose dependency tracking is in theory a pile of braindamaged rubbish, an abomination in the sight of She Who Created The World With 4-Space Indentation, is definitely good enough to be an big improvement over nothing at all. I don't know why people would choose to implement their tools based on egg_info rather than PEP 345, but if there's some momentum in that direction already, those developers and users are not going to be pleased to hear "we're sorry, but what's good enough for you is not allowed on PyPi." If PEP 345 is as much better than previous frameworks as it sounds to be, you really shouldn't have much trouble building momentum. In the meantime, it's better to let people using a competing "standard" (even if it's neither very good nor a "real" standard) do their thing until they see the light. If you can't pass that test, then the question "what was all our work on PEP 345 for, anyway" becomes quite acute, but not in the way you're using it. :-/ From martin at v.loewis.de Sat Sep 18 18:00:58 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 18 Sep 2010 18:00:58 +0200 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> <4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk> <4C94D892.4070708@v.loewis.de> Message-ID: <4C94E23A.7010709@v.loewis.de> > So you are fine with publishing "slightly incorrect" metadata at PyPI > ? I am not. I really have no intuition for in how many cases the data will be incorrect. However, if users find that the data is incorrect for specific package, they ought to complain to the maintainer. > I don't understand the rational behind providing flawed data at PyPI. > I am -1 on that. -1 sounds much better than vetoing it :-) Taking the votes together, I currently arrive at Tarek -1 Michael -1 (?) Jannis -1 (?) Jim +1 Thomas +1 Any other opinions? Regards, Martin From martin at v.loewis.de Sat Sep 18 18:06:40 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 18 Sep 2010 18:06:40 +0200 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <20100918154948.96C763A403D@sparrow.telecommunity.com> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> <4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk> <4C94D892.4070708@v.loewis.de> <20100918154948.96C763A403D@sparrow.telecommunity.com> Message-ID: <4C94E390.7050109@v.loewis.de> Am 18.09.10 17:49, schrieb P.J. Eby: > At 05:19 PM 9/18/2010 +0200, Martin v. L?wis wrote: >> In the specific case of tl.eggdeps, the dependency information is only >> used to create printable graphs. If this turns out to be slightly >> incorrect, people would notice if they try to use the packages in >> question. > > By the way, just providing this information for .egg files and *not* for > sdists would ensure accuracy of the metadata for that platform/python > version. True (I presume - unless there are also dependencies on the specific OS version or system installation that may affect the metadata). OTOH, I do think that the users asking for that prefer per-release information, despite the limitations that this may have. OTTH, if the concerns could be relieved if egg-info would provided for all files that have it, I could provide that as well/instead. Regards, Martin From thomas at thomas-lotze.de Sat Sep 18 18:39:02 2010 From: thomas at thomas-lotze.de (Thomas Lotze) Date: Sat, 18 Sep 2010 18:39:02 +0200 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> <4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk> <8762y3xdei.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Stephen J. Turnbull wrote: > In the meantime, it's better to let people using a competing "standard" > (even if it's neither very good nor a "real" standard) do their thing > until they see the light. It's not even about the people who consume egg-info data seeing the light, it's about PEP-345 data actually being available. It's package authors who have to see the light, if you want to put it that way. In the case of the dependency graphs, "good enough" is actually good enough as long as "100% correct" is not available. I am actually quite aware of the existence of PEP 345 and its advantages over legacy egg-info data and I wouldn't build an installer based on egg-info data straight from PyPI, but I feel it's good enough for my application that aims to visualise dependencies that people "know anyway". I have the following possibilities: - build graphs from PEP-345 info exclusively, which currently gives me a single graph with a single node (not useful now or for a long time), - find a volunteer who fills in PEP-345 information for all existing packages (not likely to happen), - give up and have a beer instead of building the application (not satisfying in the long run), - use PEP-345 information wherever possible and fall back to egg-info data otherwise, acknowledging and advertising the fact that the fall-back data may not be 100% correct and having the package author specify PEP-345 information would be the way to go (good enough for my use case). -- Thomas From pje at telecommunity.com Sat Sep 18 18:47:33 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sat, 18 Sep 2010 12:47:33 -0400 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C94E390.7050109@v.loewis.de> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> <4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk> <4C94D892.4070708@v.loewis.de> <20100918154948.96C763A403D@sparrow.telecommunity.com> <4C94E390.7050109@v.loewis.de> Message-ID: <20100918164736.DAC723A403D@sparrow.telecommunity.com> At 06:06 PM 9/18/2010 +0200, Martin v. L?wis wrote: >Am 18.09.10 17:49, schrieb P.J. Eby: >>At 05:19 PM 9/18/2010 +0200, Martin v. L?wis wrote: >>>In the specific case of tl.eggdeps, the dependency information is only >>>used to create printable graphs. If this turns out to be slightly >>>incorrect, people would notice if they try to use the packages in >>>question. >> >>By the way, just providing this information for .egg files and *not* for >>sdists would ensure accuracy of the metadata for that platform/python >>version. > >True (I presume - unless there are also dependencies on the specific >OS version or system installation that may affect the metadata). No, because an egg's egg-info is what it is. easy_install doesn't rebuild that information, so it is correct by definition. ;-) (Certainly, it is what it will be used for dependency information.) >OTOH, I do think that the users asking for that prefer per-release >information, despite the limitations that this may have. > >OTTH, if the concerns could be relieved if egg-info would provided >for all files that have it, I could provide that as well/instead. I am +0 on the idea myself, as I don't think the plan is quite enough to be able to provide a user-experience upgrade for use cases besides "make me a dependency graph without downloading the distributions themselves". It certainly would be nice to be able to say to the user, "here are the things I will need to download in order to fulfill your request," but if you have to download individual files to get at that information, I'm not sure how much it helps vs. just downloading the files. From ncoghlan at gmail.com Sat Sep 18 19:27:13 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 19 Sep 2010 03:27:13 +1000 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C94AF63.8010502@voidspace.org.uk> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> Message-ID: On Sat, Sep 18, 2010 at 10:24 PM, Michael Foord wrote: > ?On 18/09/2010 12:25, "Martin v. L?wis" wrote: >>>> >>>> I think you are misunderstanding. The infrastructure will *not* depend >>>> on the old formats. Instead, packaging that have that information will >>>> provide it, packages that don't will not. The infrastructure is entirely >>>> agnostic on whether the data is available or not. In particular, it will >>>> not try to interpret the data in any way. >>>> >>> No, not at all. If tools *use* the information (and if they don't then >>> what is the point?) then packages that want to be compatible with those >>> tools will have to provide this information. >> >> Not true. Tools could/should also support PEP 345 data, and then they can >> support either kind of package. >> > But you are proposing that PyPI will provide an interface / API for exposing > the setuptools information. So tools that get metadata from PyPI in this way > (and depend on it - as they will if you expose it) will have functionality > that only works for packages that provide this information in this form. > > Saying tools "should" work with other formats too is empty words. If the idea is solely to use legacy setuptools data as a fallback if the actual PEP 345 data is unavailable, it sounds like it would be far more robust to have *PyPI* do the fallback, implementing it correctly *once*, rather than simply exposing the raw setuptools data (which *will* lead to client applications that *only* understand the setuptools data and can't understand packages that only provide PEP 345 compliant metadata). Throwing the raw data at client applications and saying "here, you figure it out" when they can already do that by downloading and interrogating the packages directly seems likely to cause more confusion than anything else. So +1 to a "allow_fallback_metadata" flag in appropriate PyPI APIs, -1 on exposing the legacy data directly. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From nagle at animats.com Sat Sep 18 20:18:19 2010 From: nagle at animats.com (John Nagle) Date: Sat, 18 Sep 2010 11:18:19 -0700 Subject: [Python-Dev] Polymorphic best practices In-Reply-To: References: Message-ID: <4C95026B.6090207@animats.com> On 9/18/2010 2:29 AM, python-dev-request at python.org wrote: > Polymorphic best practices [was: (Not) delaying the 3.2 release] If you're hung up on this, try writing the user-level documentation first. Your target audience is a working-level Web programmer, not someone who knows six programming languages and has a CS degree. If the explanation is too complex, so is the design. Coding in this area is quite hard to do right. There are issues with character set, HTML encoding, URL encoding, and internationalized domain names. It's often done wrong; I recently found a Google service which botched it. Python libraries should strive to deliver textual data to the programmer in clean Unicode. If someone needs the underlying wire representation it should be available, but not the default. John Nagle From fuzzyman at voidspace.org.uk Sat Sep 18 20:47:08 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 18 Sep 2010 19:47:08 +0100 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> Message-ID: <4C95092C.9020103@voidspace.org.uk> On 18/09/2010 18:27, Nick Coghlan wrote: > On Sat, Sep 18, 2010 at 10:24 PM, Michael Foord > wrote: >> On 18/09/2010 12:25, "Martin v. L?wis" wrote: >>>>> I think you are misunderstanding. The infrastructure will *not* depend >>>>> on the old formats. Instead, packaging that have that information will >>>>> provide it, packages that don't will not. The infrastructure is entirely >>>>> agnostic on whether the data is available or not. In particular, it will >>>>> not try to interpret the data in any way. >>>>> >>>> No, not at all. If tools *use* the information (and if they don't then >>>> what is the point?) then packages that want to be compatible with those >>>> tools will have to provide this information. >>> Not true. Tools could/should also support PEP 345 data, and then they can >>> support either kind of package. >>> >> But you are proposing that PyPI will provide an interface / API for exposing >> the setuptools information. So tools that get metadata from PyPI in this way >> (and depend on it - as they will if you expose it) will have functionality >> that only works for packages that provide this information in this form. >> >> Saying tools "should" work with other formats too is empty words. > If the idea is solely to use legacy setuptools data as a fallback if > the actual PEP 345 data is unavailable, it sounds like it would be far > more robust to have *PyPI* do the fallback, implementing it correctly > *once*, rather than simply exposing the raw setuptools data (which > *will* lead to client applications that *only* understand the > setuptools data and can't understand packages that only provide PEP > 345 compliant metadata). > > Throwing the raw data at client applications and saying "here, you > figure it out" when they can already do that by downloading and > interrogating the packages directly seems likely to cause more > confusion than anything else. > > So +1 to a "allow_fallback_metadata" flag in appropriate PyPI APIs, -1 > on exposing the legacy data directly. > If the two different data formats can be exposed in a compatible way then this sounds good to me. Michael > Cheers, > Nick. > -- http://www.ironpythoninaction.com/ From fdrake at acm.org Sat Sep 18 21:47:46 2010 From: fdrake at acm.org (Fred Drake) Date: Sat, 18 Sep 2010 15:47:46 -0400 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C94E23A.7010709@v.loewis.de> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> <4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk> <4C94D892.4070708@v.loewis.de> <4C94E23A.7010709@v.loewis.de> Message-ID: 2010/9/18 "Martin v. L?wis" : > Any other opinions? -1 from me as well; I see no reason to encourage use of bad metadata, given mechanisms to get correct metadata exist (running "setup.py egg_info", as others have pointed out). I understand there are perceived uses for such data, but it's just as available for those uses as for tools like pip and buildout. ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From mal at egenix.com Sat Sep 18 23:09:35 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 18 Sep 2010 23:09:35 +0200 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: <4C94E23A.7010709@v.loewis.de> References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> <4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk> <4C94D892.4070708@v.loewis.de> <4C94E23A.7010709@v.loewis.de> Message-ID: <4C952A8F.9020202@egenix.com> "Martin v. L?wis" wrote: >> So you are fine with publishing "slightly incorrect" metadata at PyPI >> ? I am not. > > I really have no intuition for in how many cases the data will be > incorrect. However, if users find that the data is incorrect for > specific package, they ought to complain to the maintainer. > >> I don't understand the rational behind providing flawed data at PyPI. >> I am -1 on that. > > -1 sounds much better than vetoing it :-) Taking the votes together, > I currently arrive at > > Tarek -1 > Michael -1 (?) > Jannis -1 (?) > Jim +1 > Thomas +1 > > Any other opinions? -1 as well. The formats and file-names are just a complete mess and not all packages on PyPI are available as eggs or compatible to setuptools. I think the information from PEP 345 which is already made available via the XML-RPC .release_data() interface is sufficient for most cases. In those cases where it is not, we should extend the meta data. http://www.python.org/dev/peps/pep-0345/ http://wiki.python.org/moin/PyPiXmlRpc It may be worthwhile providing the same sort of information as static PKG-INFO file alongside the /simple index entries as well (e.g. via a link on the package page), to make it possible to extract the meta information without having to use the RPC mechanism and making use of the mirror infrastructure. http://www.python.org/dev/peps/pep-0314/ http://www.python.org/dev/peps/pep-0390/ PS: Could you put your DZUG talk up online somewhere. It includes some very valuable information on the various available APIs that is difficult to find elswhere. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 18 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From martin at v.loewis.de Sat Sep 18 23:15:04 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 18 Sep 2010 23:15:04 +0200 Subject: [Python-Dev] egg_info in PyPI In-Reply-To: References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com> <4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk> <4C94BCED.8010405@v.loewis.de> Message-ID: <4C952BD8.6020701@v.loewis.de> Am 18.09.2010 15:27, schrieb Steve Holden: > On 9/18/2010 9:21 AM, "Martin v. L?wis" wrote: >> IT WILL BE NOT IN PREFERENCE TO DISTUTILS2. > > No need to shout. I really felt that this otherwise wouldn't be heard - I tried to say it a number of times before, and it was ignored. Regards, Martin From chris at simplistix.co.uk Sat Sep 18 23:39:51 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Sat, 18 Sep 2010 22:39:51 +0100 Subject: [Python-Dev] os.path.normcase rationale? Message-ID: <4C9531A7.10405@simplistix.co.uk> Hi All, I'm curious as to why, with a file called "Foo.txt" on a case descriminating but case insensitive filesystem, os.path.normcase('FoO.txt') will return "foo.txt" rather than "Foo.txt"? Yes, I know the behaviour is documented, but I'm wondering if anyone can remember the rationale for that behaviour? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From guido at python.org Sun Sep 19 00:36:58 2010 From: guido at python.org (Guido van Rossum) Date: Sat, 18 Sep 2010 15:36:58 -0700 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: <4C9531A7.10405@simplistix.co.uk> References: <4C9531A7.10405@simplistix.co.uk> Message-ID: On Sat, Sep 18, 2010 at 2:39 PM, Chris Withers wrote: > I'm curious as to why, with a file called "Foo.txt" on a case descriminating > but case insensitive filesystem, os.path.normcase('FoO.txt') will return > "foo.txt" rather than "Foo.txt"? > > Yes, I know the behaviour is documented, but I'm wondering if anyone can > remember the rationale for that behaviour? Because normcase() and friends never look at the filesystem. Of course, exists() and isdir() etc. do, and so does realpath(), but the pure parsing functions don't. They can be used without a working filesystem even. (E.g. you can import ntpath on a Unix box and happily parse Windows paths.) -- --Guido van Rossum (python.org/~guido) From greg.ewing at canterbury.ac.nz Sun Sep 19 00:59:30 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 19 Sep 2010 10:59:30 +1200 Subject: [Python-Dev] Some news from my sandbox project In-Reply-To: References: <201009180127.28419.victor.stinner@haypocalc.com> <4C947751.6050605@canterbury.ac.nz> Message-ID: <4C954452.7000305@canterbury.ac.nz> Robert Collins wrote: > __builtins__ is in everyone's global namespace, so if it can be > mutated, different python programs running in the same sandbox can > affect each other. So give each program its own copy of __builtins__. -- Greg From greg.ewing at canterbury.ac.nz Sun Sep 19 01:05:45 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 19 Sep 2010 11:05:45 +1200 Subject: [Python-Dev] Some news from my sandbox project In-Reply-To: <201009181158.55359.victor.stinner@haypocalc.com> References: <201009180127.28419.victor.stinner@haypocalc.com> <4C947751.6050605@canterbury.ac.nz> <201009181158.55359.victor.stinner@haypocalc.com> Message-ID: <4C9545C9.6080804@canterbury.ac.nz> Victor Stinner wrote: > Eg. one of the most important function of pysandbox is > proxy() (a function to create a read only view of a object outside the > sandbox, especially on an import), if you replace isinstance() by a function > which always return True: you can get unmodified objects I don't follow. Trusted functions such as proxy() shouldn't be sharing a __builtins__ dict with sandboxed code. -- Greg From sridharr at activestate.com Sun Sep 19 03:40:42 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Sat, 18 Sep 2010 18:40:42 -0700 Subject: [Python-Dev] [Catalog-sig] egg_info in PyPI In-Reply-To: References: <4C93B6D8.1090700@v.loewis.de> <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info> Message-ID: <7171CCFF-5EF7-41E1-AD08-12BA0B0896FB@activestate.com> On 2010-09-18, at 2:29 AM, Thomas Lotze wrote: > I'd like to expand [tl.eggdeps] > to analyse dependencies between any packages on PyPI but I can't > as long as dependency information is not available without actually > installing things. [...] On 2010-09-18, at 2:29 AM, Thomas Lotze wrote: >> I am not even understanding what's the benefit of doing this since an >> egg_info directory is obtained at *build* time and can differ from a >> machine to another, so it seems pretty useless for me to publish this. > > We do understand that legacy metadata has its issues, but taken with the > appropriate amount of salt, it's better than nothing in most cases, enough > so at least to solve some chicken-and-egg problems. Just curious: aside from the static metadata issue which you recognize, how would you make tl.eggdeps analyze dependencies between any packages registered on PyPI when not all of them upload their sdists (hence no egg_info) to PyPI? Also, if I may ask, what is the intended use for tl.eggdeps? For use in package managers (buildout, pip)? If yes, then ... would providing egg_info in PyPI obviate the need for package managers such as pip to download sources before resolving dependencies (keeping in mind that even packages that do upload their sdists to PyPI sometimes end up not doing the same - perhaps out of forgetfulness - for theirs latest releases)? -srid From ncoghlan at gmail.com Sun Sep 19 04:03:03 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 19 Sep 2010 12:03:03 +1000 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) Message-ID: On Sun, Sep 19, 2010 at 4:18 AM, John Nagle wrote: > On 9/18/2010 2:29 AM, python-dev-request at python.org wrote: >> >> Polymorphic best practices [was: (Not) delaying the 3.2 release] > > ? If you're hung up on this, try writing the user-level documentation > first. ?Your target audience is a working-level Web programmer, not > someone who knows six programming languages and has a CS degree. > If the explanation is too complex, so is the design. > > ? Coding in this area is quite hard to do right. ?There are > issues with character set, HTML encoding, URL encoding, and > internationalized domain names. ?It's often done wrong; > I recently found a Google service which botched it. > Python libraries should strive to deliver textual data to the programmer > in clean Unicode. ?If someone needs the underlying wire representation > it should be available, but not the default. Even though URL byte sequences are defined as using only an ASCII subset, I'm currently inclined to add raw bytes supports to urlib.parse by providing parallel APIs (i.e. urlib.parse.urlsplitb, etc) rather than doing it implicitly in the normal functions. My rationale is as follows: - while URLs are *meant* to be encoded correctly as an ASCII subset, the real world isn't always quite so tidy (i.e. applications treat as URLs things that technically are not because the encoding is wrong) - separating the APIs forces the programmer to declare that they know they're working with the raw bytes off the wire to avoid the decode/encode overhead that comes with working in the Unicode domain - easier to change our minds later. Adding implicit bytes support to the normal names can be done any time, but removing it would require an extensive deprecation period Essentially, while I can see strong use cases for wanting to manipulate URLs in wire format, I *don't* see strong use cases for manipulating URLs without *knowing* whether they're in wire format (encoded bytes) or display format (Unicode text). For some APIs that work for arbitrary encodings (e.g. os.listdir) switching based on argument type seems like a reasonable idea. For those that may silently produce incorrect output for ASCII-incompatible encodings, the os.environ/os.environb seems like a better approach. I could probably be persuaded to merge the APIs, but the email6 precedent suggests to me that separating the APIs better reflects the mental model we're trying to encourage in programmers manipulating text (i.e. the difference between the raw octet sequence and the text character sequence/parsed data). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From steve at holdenweb.com Sun Sep 19 04:18:58 2010 From: steve at holdenweb.com (Steve Holden) Date: Sat, 18 Sep 2010 22:18:58 -0400 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: References: Message-ID: On 9/18/2010 10:03 PM, Nick Coghlan wrote: > On Sun, Sep 19, 2010 at 4:18 AM, John Nagle wrote: >> On 9/18/2010 2:29 AM, python-dev-request at python.org wrote: >>> >>> Polymorphic best practices [was: (Not) delaying the 3.2 release] >> >> If you're hung up on this, try writing the user-level documentation >> first. Your target audience is a working-level Web programmer, not >> someone who knows six programming languages and has a CS degree. >> If the explanation is too complex, so is the design. >> >> Coding in this area is quite hard to do right. There are >> issues with character set, HTML encoding, URL encoding, and >> internationalized domain names. It's often done wrong; >> I recently found a Google service which botched it. >> Python libraries should strive to deliver textual data to the programmer >> in clean Unicode. If someone needs the underlying wire representation >> it should be available, but not the default. > > Even though URL byte sequences are defined as using only an ASCII > subset, I'm currently inclined to add raw bytes supports to > urlib.parse by providing parallel APIs (i.e. urlib.parse.urlsplitb, > etc) rather than doing it implicitly in the normal functions. > > My rationale is as follows: > - while URLs are *meant* to be encoded correctly as an ASCII subset, > the real world isn't always quite so tidy (i.e. applications treat as > URLs things that technically are not because the encoding is wrong) > - separating the APIs forces the programmer to declare that they know > they're working with the raw bytes off the wire to avoid the > decode/encode overhead that comes with working in the Unicode domain > - easier to change our minds later. Adding implicit bytes support to > the normal names can be done any time, but removing it would require > an extensive deprecation period > > Essentially, while I can see strong use cases for wanting to > manipulate URLs in wire format, I *don't* see strong use cases for > manipulating URLs without *knowing* whether they're in wire format > (encoded bytes) or display format (Unicode text). For some APIs that > work for arbitrary encodings (e.g. os.listdir) switching based on > argument type seems like a reasonable idea. For those that may > silently produce incorrect output for ASCII-incompatible encodings, > the os.environ/os.environb seems like a better approach. > > I could probably be persuaded to merge the APIs, but the email6 > precedent suggests to me that separating the APIs better reflects the > mental model we're trying to encourage in programmers manipulating > text (i.e. the difference between the raw octet sequence and the text > character sequence/parsed data). > That sounds pretty sane and coherent to me. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From victor.stinner at haypocalc.com Sun Sep 19 12:19:44 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 19 Sep 2010 12:19:44 +0200 Subject: [Python-Dev] Some news from my sandbox project In-Reply-To: <4C9545C9.6080804@canterbury.ac.nz> References: <201009180127.28419.victor.stinner@haypocalc.com> <201009181158.55359.victor.stinner@haypocalc.com> <4C9545C9.6080804@canterbury.ac.nz> Message-ID: <201009191219.45024.victor.stinner@haypocalc.com> Le dimanche 19 septembre 2010 01:05:45, Greg Ewing a ?crit : > I don't follow. Trusted functions such as proxy() shouldn't > be sharing a __builtins__ dict with sandboxed code. > (...) > So give each program its own copy of __builtins__. By "program" you mean a "process"? proxy() and untrusted functions are executed in the same process and the same interpreter. Untrusted code calls (indrectly) proxy(): should I create a new copy of __builtins__ for each frame? I don't know how to do that in Python (without modify the Python interpreter) and I suppose that it will make Python slower. The frame mechanism is already slow (create a new frame to call a Python function is much slower than calling a function in C). pysandbox creates a new separated namespace for untrusted functions, but __builtins__ "namespace" (dict) is shared between Python and pysandbox namespaces. -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Sun Sep 19 14:08:04 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 19 Sep 2010 14:08:04 +0200 Subject: [Python-Dev] Some news from my sandbox project References: <201009180127.28419.victor.stinner@haypocalc.com> <201009181158.55359.victor.stinner@haypocalc.com> <4C9545C9.6080804@canterbury.ac.nz> <201009191219.45024.victor.stinner@haypocalc.com> Message-ID: <20100919140804.4a0ff650@pitrou.net> On Sun, 19 Sep 2010 12:19:44 +0200 Victor Stinner wrote: > Le dimanche 19 septembre 2010 01:05:45, Greg Ewing a ?crit : > > I don't follow. Trusted functions such as proxy() shouldn't > > be sharing a __builtins__ dict with sandboxed code. > > (...) > > So give each program its own copy of __builtins__. > > By "program" you mean a "process"? proxy() and untrusted functions are > executed in the same process and the same interpreter. Untrusted code calls > (indrectly) proxy(): should I create a new copy of __builtins__ for each > frame? I don't know how to do that in Python (without modify the Python > interpreter) and I suppose that it will make Python slower. >>> def f(): return oct ... >>> f() >>> import types >>> m = types.ModuleType("my builtins") >>> m.__dict__.update(__builtins__.__dict__) >>> m.oct = 3 >>> f.__globals__['__builtins__'] = m >>> f() 3 Regards Antoine. From solipsis at pitrou.net Sun Sep 19 19:27:08 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 19 Sep 2010 19:27:08 +0200 Subject: [Python-Dev] Rework nntplib? References: <20100914121744.31ef5aaf@pitrou.net> Message-ID: <20100919192708.1baed133@pitrou.net> For the record, the code is pretty much done now: http://bugs.python.org/issue9360 Regards Antoine. On Tue, 14 Sep 2010 12:17:44 +0200 Antoine Pitrou wrote: > > Hello, > > Like the email package, nntplib in py3k is broken (because of > various bytes/str mismatches; I suppose the lack of a test suite didn't > help when porting). > > I would like to take the opportunity to improve the API a bit; no heavy > re-architecting, but simply a bunch of changes to make it higher-level. > Is it acceptable? > > (and, yes, I would add a test suite) > > Regards > > Antoine. > > From ncoghlan at gmail.com Sun Sep 19 22:55:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 20 Sep 2010 06:55:50 +1000 Subject: [Python-Dev] Rework nntplib? In-Reply-To: <20100919192708.1baed133@pitrou.net> References: <20100914121744.31ef5aaf@pitrou.net> <20100919192708.1baed133@pitrou.net> Message-ID: On Mon, Sep 20, 2010 at 3:27 AM, Antoine Pitrou wrote: > > For the record, the code is pretty much done now: > http://bugs.python.org/issue9360 Generally looks pretty reasonable. As I noted on the issue, my one concern is that the current API seems to rely on the programmer remembering which methods return strings and which return bytes without any consistent mnemonic as to which is which. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From rdmurray at bitdance.com Mon Sep 20 04:25:36 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 19 Sep 2010 22:25:36 -0400 Subject: [Python-Dev] Rework nntplib? In-Reply-To: References: <20100914121744.31ef5aaf@pitrou.net> <20100919192708.1baed133@pitrou.net> Message-ID: <20100920022536.934071D6B9D@kimball.webabinitio.net> On Mon, 20 Sep 2010 06:55:50 +1000, Nick Coghlan wrote: > On Mon, Sep 20, 2010 at 3:27 AM, Antoine Pitrou wrote: > > > > For the record, the code is pretty much done now: > > http://bugs.python.org/issue9360 > > Generally looks pretty reasonable. As I noted on the issue, my one > concern is that the current API seems to rely on the programmer > remembering which methods return strings and which return bytes > without any consistent mnemonic as to which is which. The mnemonic is: *raw message data* is bytes, everything else is unicode. That is, the *content* of head, body, article, post, and ihave commands is bytes, otherwise you are dealing with strings. I think it is a very clear and obvious distinction, myself. --David From greg.ewing at canterbury.ac.nz Mon Sep 20 00:55:55 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 20 Sep 2010 10:55:55 +1200 Subject: [Python-Dev] Some news from my sandbox project In-Reply-To: <201009191219.45024.victor.stinner@haypocalc.com> References: <201009180127.28419.victor.stinner@haypocalc.com> <201009181158.55359.victor.stinner@haypocalc.com> <4C9545C9.6080804@canterbury.ac.nz> <201009191219.45024.victor.stinner@haypocalc.com> Message-ID: <4C9694FB.40307@canterbury.ac.nz> Victor Stinner wrote: > By "program" you mean a "process"? No, I mean whatever *you* meant by "program" when you said that different programs could otherwise interfere with each other. If you have conceptually separate programs running in the same interpreter that need to be isolated, each one should run in its own sandbox with its own __builtins__ dict. > should I create a new copy of __builtins__ for each > frame? No, not for each frame, just for each computation that needs to be isolated. -- Greg From glyph at twistedmatrix.com Mon Sep 20 06:12:44 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 20 Sep 2010 00:12:44 -0400 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: References: Message-ID: <5368A821-908E-42EF-BCE2-B9B5B19DCFE8@twistedmatrix.com> On Sep 18, 2010, at 10:18 PM, Steve Holden wrote: >> I could probably be persuaded to merge the APIs, but the email6 >> precedent suggests to me that separating the APIs better reflects the >> mental model we're trying to encourage in programmers manipulating >> text (i.e. the difference between the raw octet sequence and the text >> character sequence/parsed data). >> > That sounds pretty sane and coherent to me. While I don't like the email6 precedent as such (that there would be different parsed objects, based on whether you started parsing with bytes or with strings), the idea that when you are working directly with bytes or text, you should have to know which one you have, is a good one. +1 for keeping the APIs separate with 'urlsplitb' etc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Sep 20 13:26:28 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 20 Sep 2010 21:26:28 +1000 Subject: [Python-Dev] Rework nntplib? In-Reply-To: <20100920022536.934071D6B9D@kimball.webabinitio.net> References: <20100914121744.31ef5aaf@pitrou.net> <20100919192708.1baed133@pitrou.net> <20100920022536.934071D6B9D@kimball.webabinitio.net> Message-ID: On Mon, Sep 20, 2010 at 12:25 PM, R. David Murray wrote: > On Mon, 20 Sep 2010 06:55:50 +1000, Nick Coghlan wrote: >> On Mon, Sep 20, 2010 at 3:27 AM, Antoine Pitrou wrote: >> > >> > For the record, the code is pretty much done now: >> > http://bugs.python.org/issue9360 >> >> Generally looks pretty reasonable. As I noted on the issue, my one >> concern is that the current API seems to rely on the programmer >> remembering which methods return strings and which return bytes >> without any consistent mnemonic as to which is which. > > The mnemonic is: *raw message data* is bytes, everything else is unicode. > That is, the *content* of head, body, article, post, and ihave commands > is bytes, otherwise you are dealing with strings. ?I think it is a > very clear and obvious distinction, myself. Yeah, after Antoine's response, I was happy that was a problem with my understanding of the API, rather than the API itself. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Mon Sep 20 13:34:37 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 20 Sep 2010 21:34:37 +1000 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: <5368A821-908E-42EF-BCE2-B9B5B19DCFE8@twistedmatrix.com> References: <5368A821-908E-42EF-BCE2-B9B5B19DCFE8@twistedmatrix.com> Message-ID: On Mon, Sep 20, 2010 at 2:12 PM, Glyph Lefkowitz wrote: > While I don't like the email6 precedent as such (that there would be > different parsed objects, based on whether you started parsing with bytes or > with strings), the idea that when you are working directly?with bytes or > text, you should have to know which one you have, is a good one. ?+1 for > keeping the APIs separate with 'urlsplitb' etc. It's primarily the "am I dealing with bytes" or "am I dealing with text" precedent that I'm copying. That said, I'm not personally opposed to that distinction propagating to higher level data structures when it makes sense (e.g., to some degree it will propagate naturally in urllib.parse, since the tuples of URL fragments can be seen as higher level data structures analogous to the data structures in email6). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From chrism at plope.com Mon Sep 20 14:12:18 2010 From: chrism at plope.com (Chris McDonough) Date: Mon, 20 Sep 2010 08:12:18 -0400 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: References: Message-ID: <1284984738.3022.533.camel@thinko> On Sun, 2010-09-19 at 12:03 +1000, Nick Coghlan wrote: > On Sun, Sep 19, 2010 at 4:18 AM, John Nagle wrote: > > On 9/18/2010 2:29 AM, python-dev-request at python.org wrote: > >> > >> Polymorphic best practices [was: (Not) delaying the 3.2 release] > > > > If you're hung up on this, try writing the user-level documentation > > first. Your target audience is a working-level Web programmer, not > > someone who knows six programming languages and has a CS degree. > > If the explanation is too complex, so is the design. > > > > Coding in this area is quite hard to do right. There are > > issues with character set, HTML encoding, URL encoding, and > > internationalized domain names. It's often done wrong; > > I recently found a Google service which botched it. > > Python libraries should strive to deliver textual data to the programmer > > in clean Unicode. If someone needs the underlying wire representation > > it should be available, but not the default. > > Even though URL byte sequences are defined as using only an ASCII > subset, I'm currently inclined to add raw bytes supports to > urlib.parse by providing parallel APIs (i.e. urlib.parse.urlsplitb, > etc) rather than doing it implicitly in the normal functions. > > My rationale is as follows: > - while URLs are *meant* to be encoded correctly as an ASCII subset, > the real world isn't always quite so tidy (i.e. applications treat as > URLs things that technically are not because the encoding is wrong) > - separating the APIs forces the programmer to declare that they know > they're working with the raw bytes off the wire to avoid the > decode/encode overhead that comes with working in the Unicode domain > - easier to change our minds later. Adding implicit bytes support to > the normal names can be done any time, but removing it would require > an extensive deprecation period > > Essentially, while I can see strong use cases for wanting to > manipulate URLs in wire format, I *don't* see strong use cases for > manipulating URLs without *knowing* whether they're in wire format > (encoded bytes) or display format (Unicode text). For some APIs that > work for arbitrary encodings (e.g. os.listdir) switching based on > argument type seems like a reasonable idea. For those that may > silently produce incorrect output for ASCII-incompatible encodings, > the os.environ/os.environb seems like a better approach. urllib.parse.urlparse/urllib.parse.urlsplit will never need to decode anything when passed bytes input. Both could just put the bytes comprising the hex-encoded components (the path and query string) into its respective place in the parse results, just like it does now for string input. As far as I can tell, the only thing preventing it from working against bytes right now is the use of string literals in the source instead of input-type-dictated-literals. There should not really be any need to create a "urllib.parse.urlsplitb" unless the goal is to continue down the (not great IMO) precedent already set by the shadow bytes API in urllib.parse (*_to_bytes, *_from_bytes) or if we just want to make it deliberately harder to parse URLs. The only decoding that needs to be done to potential bytes input by APIs in urllib.parse will be in the face of percent encodings in the path and query components (handled entirely by "unquote" and "unquote_plus", which already deal in bytes under the hood). The only encoding that needs to be done by urllib.parse is in the face of input to the "urlencode" and "quote" APIs. "quote" already deals with bytes as input under the hood. "urlencode" does not, but it might be changed use the same strategy that "quote" does now (by using a "urlencode_to_bytes" under the hood). However, I think any thought about "adding raw bytes support" is largely moot at this point. This pool has already been peed in.There's effectively already a "shadow" bytes-only API in the urlparse module in the form of the *_to_bytes and *_from_bytes functions in most places where it counts. So as I see it, the options are: 1) continue the *_to_bytes and *_from_bytes pattern as necessary. 2) create a new module (urllib.parse2) that has only polymorphic functions. #1 is not very pleasant to think about as a web developer if I need to maintain a both-2-and-3-compatible codebase. Neither is #2, really, if I had to support Python 3.1 and 3.2. From my (obviously limited) perspective, a more attractive third option is backwards incompatibility in a later Python 3 version, where encoding-aware functions like quote, urlencode, and unquote_plus were polymorphic, accepting both bytes and string objects and returning same-typed data. - C From ncoghlan at gmail.com Mon Sep 20 15:23:20 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 20 Sep 2010 23:23:20 +1000 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: <1284984738.3022.533.camel@thinko> References: <1284984738.3022.533.camel@thinko> Message-ID: On Mon, Sep 20, 2010 at 10:12 PM, Chris McDonough wrote: > urllib.parse.urlparse/urllib.parse.urlsplit will never need to decode > anything when passed bytes input. Correct. Supporting manipulation of bytes directly is primarily a speed hack for when an application wants to avoid the decoding/encoding overhead needed to perform the operations in the text domain when the fragments being manipulated are all already correctly encoded ASCII text. However, supporting direct manipulation of bytes *implicitly* in the current functions is problematic, since it means that the function may fail silently when given bytes that are encoded with an ASCII incompatible codec (or which there are many, especially when it comes to multibyte codecs and other stateful codecs). Even ASCII compatible codecs are a potential source of hard to detect bugs, since using different encodings for different fragments will lead directly to mojibake. Moving the raw bytes support out to separate APIs allows their constraints to be spelled out clearly and for programmers to make a conscious decision that that is what they want to do. The onus is then on the programmer to get their encodings correct. If we decide to add implicit support later, that's pretty easy (just have urllib.parse.* delegate to urllib.parse.*b when given bytes). Taking implicit support *away* after providing it, however, means going through the whole deprecation song and dance. Given the choice, I prefer the API design that allows me to more easily change my mind later if I decide I made the wrong call. > There's effectively already a "shadow" bytes-only API in the urlparse module in the form of the *_to_bytes and *_from_bytes functions in most places where it counts. If by "most places where it counts" you mean "quote" and "unquote", then sure. However, those two functions predate most of the work on fixing the bytes/unicode issues in the OS facing libraries, so they blur the lines more than may be desirable (although reading http://bugs.python.org/issue3300 shows that there were a few other constraints in play when it comes to those two operations, especially those related to the encoding of the original URL *prior* to percent-encoding for transmission over the wire). Regardless, quoteb and unquoteb will both be strictly bytes->bytes functions, whereas the existing quoting APIs attempt to deal with both text encoding and URL quoting all at the same time (and become a fair bit more complicated as a result). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tom.browder at gmail.com Mon Sep 20 16:31:45 2010 From: tom.browder at gmail.com (Tom Browder) Date: Mon, 20 Sep 2010 09:31:45 -0500 Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: <201009170921.41486.victor.stinner@haypocalc.com> References: <201009162336.06353.victor.stinner@haypocalc.com> <201009170921.41486.victor.stinner@haypocalc.com> Message-ID: Continuing on with investigating Python 2.7 build problems, one problem I just discovered is a different installation on one 64-bit system (Debian Lenny) versus another (Ubuntu 10.04.1 LTS). I used gcc-4.5.1 on both systems, with no *PY* environment variables set. On Debian I got two directories: /usr/local/lib64/python2.7 /usr/local/lib/python2.7 and only the first had the "config" subdirectory. On Ubuntu I got only /usr/local/lib/python2.7 I see that the configure file has some architecture choices (--with-universal-archs=ARCH) but no explanation about the consequences. Can anyone explain the two different "default" installations I got? It seems to me I should force the Ubuntu-style installation by the "--with-universal-archs=64-bit" configure option, and I will try that on Debian while I await expert help. Thanks. -Tom From sable at users.sourceforge.net Mon Sep 20 16:41:42 2010 From: sable at users.sourceforge.net (=?ISO-8859-1?Q?S=E9bastien_Sabl=E9?=) Date: Mon, 20 Sep 2010 16:41:42 +0200 Subject: [Python-Dev] Buildbot for AIX In-Reply-To: <20100917150551.099113e4@pitrou.net> References: <1284586127.79.0.767793857364.issue1633863@psf.upfronthosting.co.za> <4C93377C.4040300@users.sourceforge.net> <20100917150551.099113e4@pitrou.net> Message-ID: <4C9772A6.2070908@users.sourceforge.net> Le 17/09/2010 15:05, Antoine Pitrou a ?crit : > Following on Martin's comments, you might also want to share things > with the ActiveState guys who, AFAIK, maintain an AIX version of Python > (but you have been the most active AIX user on the bug tracker lately; > perhaps they are keeping their patches to themselves). I tried to find some source package or subversion repository on their web site, but they only distribute binary versions. Also it is only the "Business Edition" that supports AIX and it is clearly sold with a proprietary license. So I doubt they would share their patches related to AIX, but I can always ask. Regards -- S?bastien Sabl? From sable at users.sourceforge.net Mon Sep 20 16:34:35 2010 From: sable at users.sourceforge.net (=?UTF-8?B?U8OpYmFzdGllbiBTYWJsw6k=?=) Date: Mon, 20 Sep 2010 16:34:35 +0200 Subject: [Python-Dev] Buildbot for AIX In-Reply-To: <4C936225.1000500@v.loewis.de> References: <1284586127.79.0.767793857364.issue1633863@psf.upfronthosting.co.za> <4C93377C.4040300@users.sourceforge.net> <4C936225.1000500@v.loewis.de> Message-ID: <4C9770FB.1010103@users.sourceforge.net> Hi Martin, Le 17/09/2010 14:42, "Martin v. L?wis" a ?crit : > If you are having the build slave compile Python, I'd like to point > out that you *already* run arbitrary shell commands provided by > some external source: if somebody would check some commands into > Python's configure.in, you would unconditionally execute them. > So if it's ok that you run the Python build process at all, it should > (IMO) also be acceptable to run a build slave. > > If there are concerns that running it under your Unix account gives it > too much power, you should create a separate, locked-down account. Someone messing with the configure script in python svn would probably get noticed very quickly, but I agree this is also a security risk, and the buildbot slave runs with a user with limited privileges. I will try to convince the IT Team that this is an acceptable risk and setup a chroot or something like that for the buildbot slave. That may take some time. Also could you provide me the master.cfg file (with obfuscated passwords) that is used by the Python buildbot master or tell me if it is in subversion somewhere? I would like to make my script as close as possible to yours, in order to propose a patch for the AIX specific flags that have to be used for compilation on this platform when everything will be stable enough. Regards -- S?bastien Sabl? From ocean-city at m2.ccsnet.ne.jp Mon Sep 20 17:07:44 2010 From: ocean-city at m2.ccsnet.ne.jp (Hirokazu Yamamoto) Date: Tue, 21 Sep 2010 00:07:44 +0900 Subject: [Python-Dev] Fwd: Builder: x86 Windows7 3.x OpenSSL compile error Message-ID: <4C9778C0.8010005@m2.ccsnet.ne.jp> Hello. I've sent following mail to buildbot manager, but I found that buildbot page saids the problem of unsable bot should be sent to python-dev at python.org. So I'll do it. -------- Original Message -------- Subject: Builder: x86 Windows7 3.x OpenSSL compile error Date: Mon, 20 Sep 2010 22:53:04 +0900 From: Hirokazu Yamamoto To: db3l.net at gmail.com Hello, David Bolen. I'm Hirokazu Yamamoto, Python committer. I've noticed your buildbot x86 Windows7 3.x fails to compile OpenSSL, I'm very sorry because this buildbot is very fast and looks useful. (Running on Windows7 which I don't have) Builder: x86 Windows7 3.x http://www.python.org/dev/buildbot/all/builders/x86%20Windows7%203.x I confirmed this happens when nasmw's version is 2.06. I'm using 2.07 on my machine, so I didn't notice this. I checked Windows buildbot's nasmw versions with Tools/buildbot/external_common.bat hack, the versions were.... x86 Windows7 3.x Buildbot (FAILURE) NASM version 2.06 compiled on Jul 1 2009 x86 XP-5 3.x Buildbot (OK) NASM version 2.07 compiled on Jul 19 2009 x86 XP-4 3.x Buildbot (OK) NASM version 2.02 compiled on Feb 23 2008 I didn't check 2.02 but probably it dosen't have this issue. Could you upgrade nasmw on Windows7 buildbot? Thank you. Regards, Yamamoto. From tom.browder at gmail.com Mon Sep 20 18:56:52 2010 From: tom.browder at gmail.com (Tom Browder) Date: Mon, 20 Sep 2010 11:56:52 -0500 Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: <8f2cb732-7a29-ce62-c0f3-bb83d50def9a@me.com> References: <8f2cb732-7a29-ce62-c0f3-bb83d50def9a@me.com> Message-ID: On Mon, Sep 20, 2010 at 10:12, Ronald Oussoren wrote: > > > On 20 Sep, 2010,at 04:31 PM, Tom Browder wrote: > > Continuing on with investigating Python 2.7 build problems, one > problem I just discovered is a different installation on one 64-bit > system (Debian Lenny) versus another (Ubuntu 10.04.1 LTS). > > I used gcc-4.5.1 on both systems, with no *PY* environment variables set. > > On Debian I got two directories: > > /usr/local/lib64/python2.7 > /usr/local/lib/python2.7 > > and only the first had the "config" subdirectory. > > On Ubuntu I got only > > /usr/local/lib/python2.7 Okay, that leads to an install problem with bazaar on the Debian beast: Could not find platform dependent libraries Consider setting $PYTHONHOME to [:] Traceback (most recent call last): File "/usr/local/lib/python2.7/site.py", line 549, in main() File "/usr/local/lib/python2.7/site.py", line 531, in main known_paths = addusersitepackages(known_paths) File "/usr/local/lib/python2.7/site.py", line 264, in addusersitepackages user_site = getusersitepackages() File "/usr/local/lib/python2.7/site.py", line 239, in getusersitepackages user_base = getuserbase() # this will also set USER_BASE File "/usr/local/lib/python2.7/site.py", line 229, in getuserbase USER_BASE = get_config_var('userbase') File "/usr/local/lib/python2.7/sysconfig.py", line 518, in get_config_var return get_config_vars().get(name) File "/usr/local/lib/python2.7/sysconfig.py", line 421, in get_config_vars _init_posix(_CONFIG_VARS) File "/usr/local/lib/python2.7/sysconfig.py", line 275, in _init_posix raise IOError(msg) IOError: invalid Python installation: unable to open /usr/local/lib/python2.7/config/Makefile (No such file or directory) I tried setting PYTHONHOME to various things but none worked. The problem is that the config directory is under /usr/local/lib64 and none of the documents I have found describes how to handle that situation. All the environment variables seem to be for libraries, modules, and executables, and there is no reference to any lib64. The pkg-config program looks as if it could be used by such packages as bazaar but it doesn't look like it's one of the usual installation tricks. I'll go to the bazaar mailing list while awaiting an answer here. Thanks, -Tom From tjreedy at udel.edu Mon Sep 20 20:05:27 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 20 Sep 2010 14:05:27 -0400 Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: References: <201009162336.06353.victor.stinner@haypocalc.com> <201009170921.41486.victor.stinner@haypocalc.com> Message-ID: On 9/20/2010 10:31 AM, Tom Browder wrote: ... > I see that the configure file has some architecture choices > (--with-universal-archs=ARCH) but no explanation about the > consequences. > > Can anyone explain the two different "default" installations I got? At the moment, this appears to be question about "how do I use (build) a current release" rather than a bug report or "how do we develop (improve) future releases" question. As such, it seems a bit more appropriate for python-list. In any case, there are expert gcc/unix builders and users there who might be able to answer your question. -- Terry Jan Reedy From ronaldoussoren at mac.com Mon Sep 20 17:12:27 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 20 Sep 2010 15:12:27 +0000 (GMT) Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: Message-ID: <8f2cb732-7a29-ce62-c0f3-bb83d50def9a@me.com> On 20 Sep, 2010,at 04:31 PM, Tom Browder wrote: Continuing on with investigating Python 2.7 build problems, one problem I just discovered is a different installation on one 64-bit system (Debian Lenny) versus another (Ubuntu 10.04.1 LTS). I used gcc-4.51 on both systems, with no *PY* environment variables set. On Debian I got two directories: /usr/local/lib64/python2.7 /usr/local/lib/python2.7 and only the first had the "config" subdirectory. On Ubuntu I got only /usr/local/lib/python2.7 I see that the configure file has some architecture choices (--with-universal-archs=ARCH) but no explanation about the consequences. Can anyone explain the two different "default" installations I got? It seems to me I should force the Ubuntu-style installation by the "--with-universal-archs=64-bit" configure option, and I will try that on Debian while I await expert help. ? --with-universal-archs only does something on OSX, and then only when --with-universalsdk is specified. It is used to select which processor architectures are included in a "universal binary" build of Python.? Ronald Thanks. -Tom _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From chrism at plope.com Mon Sep 20 20:30:35 2010 From: chrism at plope.com (Chris McDonough) Date: Mon, 20 Sep 2010 14:30:35 -0400 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: References: <1284984738.3022.533.camel@thinko> Message-ID: <1285007435.3022.616.camel@thinko> On Mon, 2010-09-20 at 23:23 +1000, Nick Coghlan wrote: > On Mon, Sep 20, 2010 at 10:12 PM, Chris McDonough wrote: > > urllib.parse.urlparse/urllib.parse.urlsplit will never need to decode > > anything when passed bytes input. > > Correct. Supporting manipulation of bytes directly is primarily a > speed hack for when an application wants to avoid the > decoding/encoding overhead needed to perform the operations in the > text domain when the fragments being manipulated are all already > correctly encoded ASCII text. The urllib.parse.urlparse/urlsplit functions should never need to know or care whether the input they're passed is correctly encoded. They actually don't care right now: both will happily consume non-ASCII characters and spit out nonsense in the parse results. If passed garbage, they'll return garbage: >>> urlparse('http://www.cwi.nl:80/%7Eguido/LaPe?a.html') ParseResult(scheme='http', netloc='www.cwi.nl:80', path='/%7Eguido/LaPe?a.html', params='', query='', fragment='') > However, supporting direct manipulation of bytes *implicitly* in the > current functions is problematic, since it means that the function may > fail silently when given bytes that are encoded with an ASCII > incompatible codec (or which there are many, especially when it comes > to multibyte codecs and other stateful codecs). Even ASCII compatible > codecs are a potential source of hard to detect bugs, since using > different encodings for different fragments will lead directly to > mojibake. The "path" component result above is potentially useless and broken, and if it is inserted into a web page as a link, it may cause mojibake, but urlparse doesn't (can't) complain. As far as I can tell, there would be no more and no less potential for mojibake if the same API were able to be fed a bytes object. The result can already be nonsense and allowing for bytes as input doesn't add any greater potential to receive nonsense back. Most APIs in urllib.parse exhibit the same behavior today, e.g. urljoin: >>> urljoin('http://goo?le.com', '%7Eguido/LaPe?a.html') 'http://goo?le.com/%7Eguido/LaPe?a.html' The resulting URL is total nonsense. > Moving the raw bytes support out to separate APIs allows their > constraints to be spelled out clearly and for programmers to make a > conscious decision that that is what they want to do. The onus is then > on the programmer to get their encodings correct. I guess my argument is that the onus already *is* on the programmer to get their encodings right. They can just as easily screw up while using str inputs. > If we decide to add implicit support later, that's pretty easy (just > have urllib.parse.* delegate to urllib.parse.*b when given bytes). > Taking implicit support *away* after providing it, however, means > going through the whole deprecation song and dance. Given the choice, > I prefer the API design that allows me to more easily change my mind > later if I decide I made the wrong call. Existing APIs save for "quote" don't really need to deal with charset encodings at all, at least on any level that Python needs to care about. The potential already exists to emit garbage which will turn into mojibake from almost all existing APIs. The only remaining issue seems to be fear of making a design mistake while designing APIs. IMO, having a separate module for all urllib.parse APIs, each designed for only bytes input is a design mistake greater than any mistake that could be made by allowing for both bytes and str input to existing APIs and returning whatever type was passed. The existence of such a module will make it more difficult to maintain a codebase which straddles Python 2 and Python 3. - C From tom.browder at gmail.com Mon Sep 20 20:41:45 2010 From: tom.browder at gmail.com (Tom Browder) Date: Mon, 20 Sep 2010 13:41:45 -0500 Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: References: <201009162336.06353.victor.stinner@haypocalc.com> <201009170921.41486.victor.stinner@haypocalc.com> Message-ID: On Mon, Sep 20, 2010 at 13:05, Terry Reedy wrote: > On 9/20/2010 10:31 AM, Tom Browder wrote: > ... >> >> I see that the configure file has some architecture choices >> (--with-universal-archs=ARCH) but no explanation about the >> consequences. >> >> Can anyone explain the two different "default" installations I got? > > At the moment, this appears to be question about "how do I use (build) a > current release" rather than a bug report or "how do we develop (improve) > future releases" question. As such, it seems a bit more appropriate for > python-list. In any case, there are expert gcc/unix builders and users there > who might be able to answer your question. Thanks, I'll do that--docs are pretty sketchy about building from source. -Tom Thomas M. Browder, Jr. Niceville, Florida USA From tom.browder at gmail.com Mon Sep 20 21:20:08 2010 From: tom.browder at gmail.com (Tom Browder) Date: Mon, 20 Sep 2010 14:20:08 -0500 Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: References: <201009162336.06353.victor.stinner@haypocalc.com> <201009170921.41486.victor.stinner@haypocalc.com> Message-ID: On Mon, Sep 20, 2010 at 13:05, Terry Reedy wrote: > On 9/20/2010 10:31 AM, Tom Browder wrote: > ... >> >> I see that the configure file has some architecture choices >> (--with-universal-archs=ARCH) but no explanation about the >> consequences. >> >> Can anyone explain the two different "default" installations I got? > > At the moment, this appears to be question about "how do I use (build) a > current release" rather than a bug report or "how do we develop (improve) > future releases" question. As such, it seems a bit more appropriate for > python-list. In any case, there are expert gcc/unix builders and users there > who might be able to answer your question. The more I look into it, though, it seems to be a legitimate developer question. The python installation scripts know how to install on a mixed 32/64 bit system but missed fixing it to look like a more universal system (if that's the proper word) for programs wanting to read its installation configuration. Respectfully, -Tom From martin at v.loewis.de Mon Sep 20 21:21:06 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 20 Sep 2010 21:21:06 +0200 Subject: [Python-Dev] Buildbot for AIX In-Reply-To: <4C9770FB.1010103@users.sourceforge.net> References: <1284586127.79.0.767793857364.issue1633863@psf.upfronthosting.co.za> <4C93377C.4040300@users.sourceforge.net> <4C936225.1000500@v.loewis.de> <4C9770FB.1010103@users.sourceforge.net> Message-ID: <4C97B422.4040606@v.loewis.de> > Also could you provide me the master.cfg file (with obfuscated > passwords) that is used by the Python buildbot master or tell me if it > is in subversion somewhere? Attached! Regards, Martin -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: master.cfg URL: From solipsis at pitrou.net Mon Sep 20 21:28:52 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 20 Sep 2010 21:28:52 +0200 Subject: [Python-Dev] Python 2.7 Won't Build References: <201009162336.06353.victor.stinner@haypocalc.com> <201009170921.41486.victor.stinner@haypocalc.com> Message-ID: <20100920212852.2ab5ec15@pitrou.net> On Mon, 20 Sep 2010 09:31:45 -0500 Tom Browder wrote: > > Can anyone explain the two different "default" installations I got? > > It seems to me I should force the Ubuntu-style installation by the > "--with-universal-archs=64-bit" configure option, and I will try that > on Debian while I await expert help. I think "universal arch" builds only apply under OS X where they produce fat binaries. Under 64-bit Linux, you can compile either a 64-bit executable (the default) or a 32-bit executable (by specifying e.g. CC="gcc -m32" to the configure script). However, the /usr/local/lib{,64}/python2.7 issue is a bit different, since those directories can host architecture independent files (such as .py and .pyc files). For example, on my Mandriva install, the 64-bit Python executable can import packages from both /usr/lib/python2.6/site-packages/ and /usr/lib64/python2.6/site-packages/. Regards Antoine. From tom.browder at gmail.com Mon Sep 20 22:01:24 2010 From: tom.browder at gmail.com (Tom Browder) Date: Mon, 20 Sep 2010 15:01:24 -0500 Subject: [Python-Dev] Python 2.7 Won't Build In-Reply-To: <20100920212852.2ab5ec15@pitrou.net> References: <201009162336.06353.victor.stinner@haypocalc.com> <201009170921.41486.victor.stinner@haypocalc.com> <20100920212852.2ab5ec15@pitrou.net> Message-ID: On Mon, Sep 20, 2010 at 14:28, Antoine Pitrou wrote: > On Mon, 20 Sep 2010 09:31:45 -0500 > Tom Browder wrote: >> >> Can anyone explain the two different "default" installations I got? >> >> It seems to me I should force the Ubuntu-style installation by ?the >> "--with-universal-archs=64-bit" configure option, and I will try that >> on Debian while I await expert help. > > I think "universal arch" builds only apply under OS X where they > produce fat binaries. > > Under 64-bit Linux, you can compile either a 64-bit executable (the > default) or a 32-bit executable (by specifying e.g. CC="gcc -m32" to > the configure script). > > > However, the /usr/local/lib{,64}/python2.7 issue is a bit different, > since those directories can host architecture independent files > (such as .py and .pyc files). For example, on my Mandriva > install, the 64-bit Python executable can import packages from > both /usr/lib/python2.6/site-packages/ > and /usr/lib64/python2.6/site-packages/. Thanks, Antoine. And I think I just found the problem with the installation (it may be worth a note): I had a special configuration file (/usr/local/share/config.site) for autoconf to force the primary local libraries to be installed in /usr/local/lib64 (left over from early 64-bit days of this old system). Removing that file, removing the /usr/local/lib64/python2.7 directory, and rebuilding and reinstalling seems to have solved the immediate problem. Moral of the story: watch out for old cruft in /usr/local when installing a new distribution. I apologize for the noise. However, I'm still investigating the original build problem (gcc trunk and corrupted byte compile), but it may be related--I'll see. Regards, -Tom Thomas M. Browder, Jr. Niceville, Florida USA From db3l.net at gmail.com Mon Sep 20 22:42:10 2010 From: db3l.net at gmail.com (David Bolen) Date: Mon, 20 Sep 2010 16:42:10 -0400 Subject: [Python-Dev] Fwd: Builder: x86 Windows7 3.x OpenSSL compile error References: <4C9778C0.8010005@m2.ccsnet.ne.jp> Message-ID: Hirokazu Yamamoto writes: > Hello. I've sent following mail to buildbot manager, > but I found that buildbot page saids the problem of unsable > bot should be sent to python-dev at python.org. So I'll do it. (I'm fine with direct email for any of my build slaves, and it's probably the quickest way to get me to pay attention, but this issue has been around a bit and likely also of general interest) Nice catch on this - the Win7 3.x branch has failed to build OpenSSL for a bit now and I hadn't figured out why. Flushing out the OpenSSL tree hadn't fixed it, and it was weird that it didn't happen on XP, nor did the pre-1.0 OpenSSL releases have a problem on Win7. So I was thinking it might just be an issue with OpenSSL. I hadn't even thought of the possibility of it being the assembler. I've bumped the slave up to the latest nasm (2.09.02) and restarted the most recent build and it's gotten past the OpenSSL stage, so it looks like 2.06 may be the specific bad version. I guess I just happened to construct this builder in the relative small interval when 2.06 was current. Thanks again for figuring this out. -- David From ncoghlan at gmail.com Mon Sep 20 23:12:13 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 21 Sep 2010 07:12:13 +1000 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: <1285007435.3022.616.camel@thinko> References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> Message-ID: On Tue, Sep 21, 2010 at 4:30 AM, Chris McDonough wrote: > Existing APIs save for "quote" don't really need to deal with charset > encodings at all, at least on any level that Python needs to care about. > The potential already exists to emit garbage which will turn into > mojibake from almost all existing APIs. ?The only remaining issue seems > to be fear of making a design mistake while designing APIs. > > IMO, having a separate module for all urllib.parse APIs, each designed > for only bytes input is a design mistake greater than any mistake that > could be made by allowing for both bytes and str input to existing APIs > and returning whatever type was passed. ?The existence of such a module > will make it more difficult to maintain a codebase which straddles > Python 2 and Python 3. Failure to use quote/unquote correctly is a completely different problem from using bytes with an ASCII incompatible encoding, or mixing bytes with different encodings. Yes, if you don't quote your URLs you may end up with mojibake. That's not a justification for creating a *new* way to accidentally create mojibake. Separating the APIs means that application programmers will be expected to know whether they are working with data formatted for display to the user (i.e. Unicode text) or transfer over the wire (i.e. ASCII compatible bytes). Can you give me a concrete use case where the application programmer won't *know* which format they're working with? Py3k made the conscious decision to stop allowing careless mixing of encoded and unencoded text. This is just taking that philosophy and propagating it further up the API stack (as has already been done with several OS facing APIs for 3.2). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From chrism at plope.com Mon Sep 20 23:39:13 2010 From: chrism at plope.com (Chris McDonough) Date: Mon, 20 Sep 2010 17:39:13 -0400 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> Message-ID: <1285018753.2130.36.camel@thinko> On Tue, 2010-09-21 at 07:12 +1000, Nick Coghlan wrote: > On Tue, Sep 21, 2010 at 4:30 AM, Chris McDonough wrote: > > Existing APIs save for "quote" don't really need to deal with charset > > encodings at all, at least on any level that Python needs to care about. > > The potential already exists to emit garbage which will turn into > > mojibake from almost all existing APIs. The only remaining issue seems > > to be fear of making a design mistake while designing APIs. > > > > IMO, having a separate module for all urllib.parse APIs, each designed > > for only bytes input is a design mistake greater than any mistake that > > could be made by allowing for both bytes and str input to existing APIs > > and returning whatever type was passed. The existence of such a module > > will make it more difficult to maintain a codebase which straddles > > Python 2 and Python 3. > > Failure to use quote/unquote correctly is a completely different > problem from using bytes with an ASCII incompatible encoding, or > mixing bytes with different encodings. Yes, if you don't quote your > URLs you may end up with mojibake. That's not a justification for > creating a *new* way to accidentally create mojibake. There's no new way to accidentally create new mojibake here by allowing bytes input, as far as I can tell. - If a user passes something that has character data outside the range 0-127 to an API that expects a URL or a "component" (in the definition that urllib.parse.urlparse uses for "component") of a URI, he can keep both pieces when it breaks. Whether that data is represented via bytes or text is not relevant. He provided bad input, he is going to lose one way or another. - If a user passes a bytestring to ``quote``, because ``quote`` is implemented in terms of ``quote_to_bytes`` the case is *already* handled by quote_to_bytes implicitly failing to convert nonascii characters. What are the cases you believe will cause new mojibake? > Separating the APIs means that application programmers will be > expected to know whether they are working with data formatted for > display to the user (i.e. Unicode text) or transfer over the wire > (i.e. ASCII compatible bytes). > > Can you give me a concrete use case where the application programmer > won't *know* which format they're working with? Py3k made the > conscious decision to stop allowing careless mixing of encoded and > unencoded text. This is just taking that philosophy and propagating it > further up the API stack (as has already been done with several OS > facing APIs for 3.2). Yes. Code which must explicitly deal with bytes input and output meant to straddle both Python 2 and Python 3. Please try to write some code which 1) uses the same codebase to straddle Python 2.6 and Python 3.2 and 2) which uses bytes input, and expects bytes output from, say, urlsplit. It becomes complex very quickly. A proposal to create yet another bytes-only API only makes it more complex, AFAICT. - C From chrism at plope.com Tue Sep 21 00:28:44 2010 From: chrism at plope.com (Chris McDonough) Date: Mon, 20 Sep 2010 18:28:44 -0400 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> <1285018753.2130.36.camel@thinko> Message-ID: <1285021724.2130.48.camel@thinko> On Tue, 2010-09-21 at 08:19 +1000, Nick Coghlan wrote: > On Tue, Sep 21, 2010 at 7:39 AM, Chris McDonough wrote: > > On Tue, 2010-09-21 at 07:12 +1000, Nick Coghlan wrote: > >> On Tue, Sep 21, 2010 at 4:30 AM, Chris McDonough wrote: > >> > Existing APIs save for "quote" don't really need to deal with charset > >> > encodings at all, at least on any level that Python needs to care about. > >> > The potential already exists to emit garbage which will turn into > >> > mojibake from almost all existing APIs. The only remaining issue seems > >> > to be fear of making a design mistake while designing APIs. > >> > > >> > IMO, having a separate module for all urllib.parse APIs, each designed > >> > for only bytes input is a design mistake greater than any mistake that > >> > could be made by allowing for both bytes and str input to existing APIs > >> > and returning whatever type was passed. The existence of such a module > >> > will make it more difficult to maintain a codebase which straddles > >> > Python 2 and Python 3. > >> > >> Failure to use quote/unquote correctly is a completely different > >> problem from using bytes with an ASCII incompatible encoding, or > >> mixing bytes with different encodings. Yes, if you don't quote your > >> URLs you may end up with mojibake. That's not a justification for > >> creating a *new* way to accidentally create mojibake. > > > > There's no new way to accidentally create new mojibake here by allowing > > bytes input, as far as I can tell. > > > > - If a user passes something that has character data outside the range > > 0-127 to an API that expects a URL or a "component" (in the > > definition that urllib.parse.urlparse uses for "component") of a URI, > > he can keep both pieces when it breaks. Whether that data is > > represented via bytes or text is not relevant. He provided > > bad input, he is going to lose one way or another. > > > > - If a user passes a bytestring to ``quote``, because ``quote`` is > > implemented in terms of ``quote_to_bytes`` the case is *already* > > handled by quote_to_bytes implicitly failing to convert nonascii > > characters. > > > > What are the cases you believe will cause new mojibake? > > Calling operations like urlsplit on byte sequences in non-ASCII > compatible encodings and operations like urljoin on byte sequences > that are encoded with different encodings. These errors differ from > the URL escaping errors you cite, since they can produce true mojibake > (i.e. a byte sequence without a single consistent encoding), rather > than merely non-compliant URLs. However, if someone has let their > encodings get that badly out of whack in URL manipulation they're > probably doomed anyway... Right, the bytes issue here is really a red herring in both the urlsplit and urljoin cases, I think. > It's certainly possible I hadn't given enough weight to the practical > issues associated with migration of existing code from 2.x to 3.x > (particularly with the precedent of some degree of polymorphism being > set back when Issue 3300 was dealt with). > > Given that a separate API still places the onus on the developer to > manage their encodings correctly, I'm beginning to lean back towards > the idea of a polymorphic API rather than separate functions. (the > quote/unquote legacy becomes somewhat unfortunate in that situation, > as they always returns str objects rather than allowing the type of > the result to be determined by the type of the argument. Something > like quotep/unquotep may prove necessary in order to work around that > situation and provide a bytes->bytes, str->str API) Yay, sounds much, much better! - C From solipsis at pitrou.net Tue Sep 21 00:32:14 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 21 Sep 2010 00:32:14 +0200 Subject: [Python-Dev] Builder: x86 Windows7 3.x OpenSSL compile error In-Reply-To: References: <4C9778C0.8010005@m2.ccsnet.ne.jp> Message-ID: <20100921003214.2face420@pitrou.net> Hello David, On Mon, 20 Sep 2010 16:42:10 -0400 David Bolen wrote: > Hirokazu Yamamoto writes: > > > Hello. I've sent following mail to buildbot manager, > > but I found that buildbot page saids the problem of unsable > > bot should be sent to python-dev at python.org. So I'll do it. > > (I'm fine with direct email for any of my build slaves, and it's > probably the quickest way to get me to pay attention, but this issue > has been around a bit and likely also of general interest) > > Nice catch on this - the Win7 3.x branch has failed to build OpenSSL > for a bit now and I hadn't figured out why. [...] There's a remaining problem on the Win7 3.x buildslave. test_ssl currently fails with the following problem: Re-running test 'test_ssl' in verbose mode test_ssl: testing with 'OpenSSL 1.0.0a 1 Jun 2010' (1, 0, 0, 1, 15) under Windows ('7', '6.1.7600', '', 'Multiprocessor Free') test test_ssl failed -- Can't read certificate file b'D:\\cygwin\\home\\db3l\\buildarea\\3.x.bolen-windows7\\build\\lib\\test\\keycert.pem' But the aforementioned certificate file is in the SVN tree and other buildslaves have no problem reading it. Can you try to check if there's something wrong on your buildslave? Regards Antoine. From ncoghlan at gmail.com Tue Sep 21 00:19:12 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 21 Sep 2010 08:19:12 +1000 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: <1285018753.2130.36.camel@thinko> References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> <1285018753.2130.36.camel@thinko> Message-ID: On Tue, Sep 21, 2010 at 7:39 AM, Chris McDonough wrote: > On Tue, 2010-09-21 at 07:12 +1000, Nick Coghlan wrote: >> On Tue, Sep 21, 2010 at 4:30 AM, Chris McDonough wrote: >> > Existing APIs save for "quote" don't really need to deal with charset >> > encodings at all, at least on any level that Python needs to care about. >> > The potential already exists to emit garbage which will turn into >> > mojibake from almost all existing APIs. ?The only remaining issue seems >> > to be fear of making a design mistake while designing APIs. >> > >> > IMO, having a separate module for all urllib.parse APIs, each designed >> > for only bytes input is a design mistake greater than any mistake that >> > could be made by allowing for both bytes and str input to existing APIs >> > and returning whatever type was passed. ?The existence of such a module >> > will make it more difficult to maintain a codebase which straddles >> > Python 2 and Python 3. >> >> Failure to use quote/unquote correctly is a completely different >> problem from using bytes with an ASCII incompatible encoding, or >> mixing bytes with different encodings. Yes, if you don't quote your >> URLs you may end up with mojibake. That's not a justification for >> creating a *new* way to accidentally create mojibake. > > There's no new way to accidentally create new mojibake here by allowing > bytes input, as far as I can tell. > > - If a user passes something that has character data outside the range > ?0-127 to an API that expects a URL or a "component" (in the > ?definition that urllib.parse.urlparse uses for "component") of a URI, > ?he can keep both pieces when it breaks. ?Whether that data is > ?represented via bytes or text is not relevant. ?He provided > ?bad input, he is going to lose one way or another. > > - If a user passes a bytestring to ``quote``, because ``quote`` is > ?implemented in terms of ``quote_to_bytes`` the case is *already* > ?handled by quote_to_bytes implicitly failing to convert nonascii > ?characters. > > What are the cases you believe will cause new mojibake? Calling operations like urlsplit on byte sequences in non-ASCII compatible encodings and operations like urljoin on byte sequences that are encoded with different encodings. These errors differ from the URL escaping errors you cite, since they can produce true mojibake (i.e. a byte sequence without a single consistent encoding), rather than merely non-compliant URLs. However, if someone has let their encodings get that badly out of whack in URL manipulation they're probably doomed anyway... It's certainly possible I hadn't given enough weight to the practical issues associated with migration of existing code from 2.x to 3.x (particularly with the precedent of some degree of polymorphism being set back when Issue 3300 was dealt with). Given that a separate API still places the onus on the developer to manage their encodings correctly, I'm beginning to lean back towards the idea of a polymorphic API rather than separate functions. (the quote/unquote legacy becomes somewhat unfortunate in that situation, as they always returns str objects rather than allowing the type of the result to be determined by the type of the argument. Something like quotep/unquotep may prove necessary in order to work around that situation and provide a bytes->bytes, str->str API) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From db3l.net at gmail.com Tue Sep 21 01:02:33 2010 From: db3l.net at gmail.com (David Bolen) Date: Mon, 20 Sep 2010 19:02:33 -0400 Subject: [Python-Dev] Builder: x86 Windows7 3.x OpenSSL compile error References: <4C9778C0.8010005@m2.ccsnet.ne.jp> <20100921003214.2face420@pitrou.net> Message-ID: Antoine Pitrou writes: > (...) > > test test_ssl failed -- Can't read certificate file > b'D:\\cygwin\\home\\db3l\\buildarea\\3.x.bolen-windows7\\build\\lib\\test\\keycert.pem' > > But the aforementioned certificate file is in the SVN tree and other > buildslaves have no problem reading it. Can you try to check if there's > something wrong on your buildslave? I suspect it's an actual Python bug, though perhaps Win7-specific. The file is in fact in the tree, and I can read it just fine from the command line. If I'm reading the test right, and regardless of the actual error message, it's really failing an existence test (os.path.exists) rather than any failure to open or read the file. Looks like some issue with using bytes for os.path.exists - I "borrowed" a Python 3.x build currently in the tree as part of build 1611 and tried: db3l at buildbot-win7 ~ $ buildarea/3.x.bolen-windows7/build/pcbuild/python_d Python 3.2a2+ (py3k, Sep 20 2010, 18:49:03) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import os [50683 refs] >>> os.path.exists('d:\\cygwin\\home\\db3l\\buildarea\\3.x.bolen-windows7\\build\\lib\\test\\keycert.pem') True [50689 refs] >>> os.path.exists(b'd:\\cygwin\\home\\db3l\\buildarea\\3.x.bolen-windows7\\buil d\\lib\\test\\keycert.pem') False [50718 refs] So the bug appears to be why those two calls return a different result. -- David From solipsis at pitrou.net Tue Sep 21 01:12:10 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 21 Sep 2010 01:12:10 +0200 Subject: [Python-Dev] Builder: x86 Windows7 3.x OpenSSL compile error In-Reply-To: References: <4C9778C0.8010005@m2.ccsnet.ne.jp> <20100921003214.2face420@pitrou.net> Message-ID: <20100921011210.24367bbe@pitrou.net> On Mon, 20 Sep 2010 19:02:33 -0400 David Bolen wrote: > > Looks like some issue with using bytes for os.path.exists - I > "borrowed" a Python 3.x build currently in the tree as part of build > 1611 and tried: > > db3l at buildbot-win7 ~ > $ buildarea/3.x.bolen-windows7/build/pcbuild/python_d > Python 3.2a2+ (py3k, Sep 20 2010, 18:49:03) [MSC v.1500 32 bit (Intel)] on win32 > > Type "help", "copyright", "credits" or "license" for more information. > >>> import os > [50683 refs] > >>> os.path.exists('d:\\cygwin\\home\\db3l\\buildarea\\3.x.bolen-windows7\\build\\lib\\test\\keycert.pem') > True > [50689 refs] > >>> os.path.exists(b'd:\\cygwin\\home\\db3l\\buildarea\\3.x.bolen-windows7\\buil > d\\lib\\test\\keycert.pem') > False > [50718 refs] Ah, indeed. I can confirm the above issue on a Windows 7 VM as well. I'm gonna try to investigate. Regards Antoine. From merwok at netwok.org Tue Sep 21 01:42:14 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 21 Sep 2010 01:42:14 +0200 Subject: [Python-Dev] [Python-checkins] r84858 - in python/branches: py3k/Doc/library/logging.rst release27-maint/Doc/library/logging.rst In-Reply-To: <20100917100904.748EDEE998@mail.python.org> References: <20100917100904.748EDEE998@mail.python.org> Message-ID: <4C97F156.4040505@netwok.org> Hello + NOTE: If you are thinking of defining your own levels, please see the section + on :ref:`custom-levels`. I think those instances of upper-case-as-markup should either be real reST note/warning/etc. directives or plain English (that is, integrating ?NOTE:? into the text flow, for example ?Note that...?/?Pay attention to...?). Regards From solipsis at pitrou.net Tue Sep 21 02:05:55 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 21 Sep 2010 02:05:55 +0200 Subject: [Python-Dev] Builder: x86 Windows7 3.x OpenSSL compile error In-Reply-To: References: <4C9778C0.8010005@m2.ccsnet.ne.jp> <20100921003214.2face420@pitrou.net> Message-ID: <20100921020555.24d5b334@pitrou.net> On Mon, 20 Sep 2010 19:02:33 -0400 David Bolen wrote: > > db3l at buildbot-win7 ~ > $ buildarea/3.x.bolen-windows7/build/pcbuild/python_d > Python 3.2a2+ (py3k, Sep 20 2010, 18:49:03) [MSC v.1500 32 bit (Intel)] on win32 > > Type "help", "copyright", "credits" or "license" for more information. > >>> import os > [50683 refs] > >>> os.path.exists('d:\\cygwin\\home\\db3l\\buildarea\\3.x.bolen-windows7\\build\\lib\\test\\keycert.pem') > True > [50689 refs] > >>> os.path.exists(b'd:\\cygwin\\home\\db3l\\buildarea\\3.x.bolen-windows7\\buil > d\\lib\\test\\keycert.pem') > False > [50718 refs] > > So the bug appears to be why those two calls return a different result. Ok, see http://bugs.python.org/issue9908 for the bug report and a patch. Regards Antoine. From merwok at netwok.org Tue Sep 21 02:12:28 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 21 Sep 2010 02:12:28 +0200 Subject: [Python-Dev] [Python-checkins] r84871 + committing distutils fixes In-Reply-To: <20100918025503.E27D6EE9D6@mail.python.org> References: <20100918025503.E27D6EE9D6@mail.python.org> Message-ID: <4C97F86C.2010105@netwok.org> >+ @unittest.skipIf(sys.platform.startswith('win'), >+ "This test is only appropriate for POSIX-like systems.") Is win32 the only non-POSIX supported platform now that OS 9 support is gone? I find those values in the docs for sys.platform: win32, cygwin, darwin, os2, os2emx. Is OS/2 a POSIX system? Isn?t it more robust not to ask this question and just use unittest.skipUnless(os.name == 'posix', msg)? On a related note, I?m very grateful that other developers take time to reply to bugs and commit simple fixes that don?t require knowledge of distutils dark corners. Recent changes have tended to turn buildbots red though, so I?d like to respectfully ask/remind you to make sure to run tests before committing in each branch (3.2, 3.1, 2.7), if possible on POSIX and Windows. (Those are Tarek?s rules, not mine; I hope don?t sound bossy or ungrateful for your help.) I apologize in advance if you did run the test and the buildbots just revealed an unforeseen platform-specific quirk; I?m not accusing anyone of anything?I have my share of embarrassing commits, so be sure I?m not judging here, I just want to improve the process. :) It would also be very helpful to always add the Distutils2 component so that synchronizing the fixes is not overlooked. (Maybe I should add a tracker admin to automate that.) mport-unittest-ly yours From merwok at netwok.org Tue Sep 21 02:34:46 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 21 Sep 2010 02:34:46 +0200 Subject: [Python-Dev] [Python-checkins] r84906 - python/branches/py3k/Doc/library/ssl.rst In-Reply-To: <20100919133107.17F59EE98D@mail.python.org> References: <20100919133107.17F59EE98D@mail.python.org> Message-ID: <4C97FDA6.5050703@netwok.org> > Log: > Remove references to read() and write() methods, which are useless > synonyms of recv() and send() Unless I?m mistaken, ssl.SSLSocket.write is still useful for use with print, pprint and maybe other functions, so I think the method should be listed somewhere. Regards From g.brandl at gmx.net Tue Sep 21 09:25:55 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 21 Sep 2010 09:25:55 +0200 Subject: [Python-Dev] [Python-checkins] r84858 - in python/branches: py3k/Doc/library/logging.rst release27-maint/Doc/library/logging.rst In-Reply-To: <4C97F156.4040505@netwok.org> References: <20100917100904.748EDEE998@mail.python.org> <4C97F156.4040505@netwok.org> Message-ID: Am 21.09.2010 01:42, schrieb ?ric Araujo: > Hello > > + NOTE: If you are thinking of defining your own levels, please see > the section > + on :ref:`custom-levels`. > > I think those instances of upper-case-as-markup should either be real > reST note/warning/etc. directives or plain English (that is, integrating > ?NOTE:? into the text flow, for example ?Note that...?/?Pay attention > to...?). I've been told that "Note that" should be removed altogether, as it's quite redundant :) Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From solipsis at pitrou.net Tue Sep 21 10:47:58 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 21 Sep 2010 10:47:58 +0200 Subject: [Python-Dev] [Python-checkins] r84906 - python/branches/py3k/Doc/library/ssl.rst References: <20100919133107.17F59EE98D@mail.python.org> <4C97FDA6.5050703@netwok.org> Message-ID: <20100921104758.2d249fa5@pitrou.net> On Tue, 21 Sep 2010 02:34:46 +0200 ?ric Araujo wrote: > > Log: > > Remove references to read() and write() methods, which are useless > > synonyms of recv() and send() > > Unless I?m mistaken, ssl.SSLSocket.write is still useful for use with > print, pprint and maybe other functions, Hmm, sorry? I don't get what you're talking about. You don't use print() with plain sockets, so you shouldn't use it with SSL sockets either. Regards Antoine. From g.brandl at gmx.net Tue Sep 21 13:38:40 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 21 Sep 2010 13:38:40 +0200 Subject: [Python-Dev] r84931 - in python/branches/py3k: Include/symtable.h Misc/NEWS Python/ast.c Python/compile.c Python/future.c Python/symtable.c In-Reply-To: <20100920230211.30B76EE9B3@mail.python.org> References: <20100920230211.30B76EE9B3@mail.python.org> Message-ID: Am 21.09.2010 01:02, schrieb benjamin.peterson: > Author: benjamin.peterson > Date: Tue Sep 21 01:02:10 2010 > New Revision: 84931 > > Log: > add column offset to all syntax errors > Modified: python/branches/py3k/Misc/NEWS > ============================================================================== > --- python/branches/py3k/Misc/NEWS (original) > +++ python/branches/py3k/Misc/NEWS Tue Sep 21 01:02:10 2010 > @@ -10,9 +10,8 @@ > Core and Builtins > ----------------- > > -- Issue #9901: Destroying the GIL in Py_Finalize() can fail if some other > - threads are still running. Instead, reinitialize the GIL on a second > - call to Py_Initialize(). > +- All SyntaxErrors now have a column offset and therefore a caret when the error > + is printed. > > - Issue #9252: PyImport_Import no longer uses a fromlist hack to return the > module that was imported, but instead gets the module from sys.modules. > @@ -59,10 +58,6 @@ > Library > ------- > > -- Issue #9877: Expose sysconfig.get_makefile_filename() > - > -- logging: Added hasHandlers() method to Logger and LoggerAdapter. > - > - Issue #1686: Fix string.Template when overriding the pattern attribute. > > - Issue #9854: SocketIO objects now observe the RawIOBase interface in That change looks like and accident. BTW, PyErr_SyntaxLocationEx needs a versionadded. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From ncoghlan at gmail.com Tue Sep 21 15:38:08 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 21 Sep 2010 23:38:08 +1000 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: <87vd5zwuvk.fsf@uwakimon.sk.tsukuba.ac.jp> References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> <1285018753.2130.36.camel@thinko> <87vd5zwuvk.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Tue, Sep 21, 2010 at 3:03 PM, Stephen J. Turnbull wrote: > On the other hand, it is dangerous to provide a polymorphic API which > does that more extensive parsing, because a less than paranoid > programmer will have very likely allowed the parsed components to > escape from the context where their encodings can be reliably > determined. ?Remember, *it is unlikely that they will ever be punished > for their own lack of caution.* ?The person who is doomed is somebody > who tries to take that code and reuse it in a different context. Yeah, that's the original reasoning that had me leaning towards the parallel API approach. If I seem to be changing my mind a lot in this thread it's because I'm genuinely torn between the desire to make it easier to port existing 2.x code to 3.x by making the current API polymorphic and the fear that doing so will reintroduce some of the exact same bytes/text confusion that the bytes/str split is trying to get rid of. There's no real way for 2to3 to help with the porting issue either, since it has no way to determine the original intent of the 2.x code. I *think* avoiding the quote/unquote precedent and applying the rule "bytes in -> bytes out" will help with avoiding the worst of any potential encoding confusion problems though. At some point the programmer is going to have to invoke decode() if they want a string to pass to display functions and the like (or vice versa with encode()) so there are still limits to how far any poorly handled code will get before blowing up. (Basically, while the issue of programmers assuming 'latin-1' or 'utf-8' or similar ASCII friendly encodings when they shouldn't is real, I don't believe a polymorphic API here will make things any *worse* than what would happen with a parallel API) And if this turns out to be a disaster in practice: a) on my head be it; and b) we still have the option of the DeprecationWarning dance for bytes inputs to the existing functions and moving to a parallel API Still-trying-to-figure-out-what-moment-of-insanity-prompted-me-to-volunteer-to-tackle-this'ly, Nick. From ncoghlan at gmail.com Tue Sep 21 15:42:37 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 21 Sep 2010 23:42:37 +1000 Subject: [Python-Dev] [Python-checkins] r84858 - in python/branches: py3k/Doc/library/logging.rst release27-maint/Doc/library/logging.rst In-Reply-To: References: <20100917100904.748EDEE998@mail.python.org> <4C97F156.4040505@netwok.org> Message-ID: On Tue, Sep 21, 2010 at 5:25 PM, Georg Brandl wrote: > I've been told that "Note that" should be removed altogether, as it's > quite redundant :) I still find starting a paragraph with "Note that" to be useful as a mild attention getter that isn't as shouty as an actual ReST note. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From fuzzyman at voidspace.org.uk Tue Sep 21 15:46:36 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 21 Sep 2010 14:46:36 +0100 Subject: [Python-Dev] [Python-checkins] r84858 - in python/branches: py3k/Doc/library/logging.rst release27-maint/Doc/library/logging.rst In-Reply-To: References: <20100917100904.748EDEE998@mail.python.org> <4C97F156.4040505@netwok.org> Message-ID: <4C98B73C.4080909@voidspace.org.uk> On 21/09/2010 14:42, Nick Coghlan wrote: > On Tue, Sep 21, 2010 at 5:25 PM, Georg Brandl wrote: >> I've been told that "Note that" should be removed altogether, as it's >> quite redundant :) > I still find starting a paragraph with "Note that" to be useful as a > mild attention getter that isn't as shouty as an actual ReST note. > I agree. Don't feel strongly about it though. (I'm sure Strunk and White would disapprove.) Michael > Cheers, > Nick. > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From benjamin at python.org Tue Sep 21 15:55:34 2010 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 21 Sep 2010 08:55:34 -0500 Subject: [Python-Dev] [Python-checkins] r84858 - in python/branches: py3k/Doc/library/logging.rst release27-maint/Doc/library/logging.rst In-Reply-To: References: <20100917100904.748EDEE998@mail.python.org> <4C97F156.4040505@netwok.org> Message-ID: 2010/9/21 Nick Coghlan : > On Tue, Sep 21, 2010 at 5:25 PM, Georg Brandl wrote: >> I've been told that "Note that" should be removed altogether, as it's >> quite redundant :) > > I still find starting a paragraph with "Note that" to be useful as a > mild attention getter that isn't as shouty as an actual ReST note. I think the preference should be to avoid phases like "Note that". For example the sentence in consideration could be phrased nicely as "See :ref:`custom-levels` for documentation on defining your levels." -- Regards, Benjamin From steven.bethard at gmail.com Tue Sep 21 16:03:18 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Tue, 21 Sep 2010 16:03:18 +0200 Subject: [Python-Dev] [Python-checkins] r84858 - in python/branches: py3k/Doc/library/logging.rst release27-maint/Doc/library/logging.rst In-Reply-To: <4C98B73C.4080909@voidspace.org.uk> References: <20100917100904.748EDEE998@mail.python.org> <4C97F156.4040505@netwok.org> <4C98B73C.4080909@voidspace.org.uk> Message-ID: On Tue, Sep 21, 2010 at 3:46 PM, Michael Foord wrote: > I agree. Don't feel strongly about it though. (I'm sure Strunk and White > would disapprove.) No doubt. http://chronicle.com/article/50-Years-of-Stupid-Grammar/25497 ;-) Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? ? ? ? ? --- The Hiphopopotamus From p.f.moore at gmail.com Tue Sep 21 17:01:54 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 21 Sep 2010 16:01:54 +0100 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> <1285018753.2130.36.camel@thinko> <87vd5zwuvk.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 21 September 2010 14:38, Nick Coghlan wrote: > On Tue, Sep 21, 2010 at 3:03 PM, Stephen J. Turnbull wrote: >> On the other hand, it is dangerous to provide a polymorphic API which [...] Sorry if this is off-topic, but I don't believe I ever saw Stephen's email. I have a feeling that's happened a couple of times recently. Before I go off trying to work out why gmail is dumping list mails on me, did anyone else see Stephen's mail via the list (as opposed to being a direct recipient)? Thanks, and sorry for the interruption. Paul. From stephen at xemacs.org Tue Sep 21 17:10:01 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 22 Sep 2010 00:10:01 +0900 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> <1285018753.2130.36.camel@thinko> <87vd5zwuvk.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <874odj3zg6.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > (Basically, while the issue of programmers assuming 'latin-1' or > 'utf-8' or similar ASCII friendly encodings when they shouldn't is > real, I don't believe a polymorphic API here will make things any > *worse* than what would happen with a parallel API) That depends on how far the polymorphic API goes. As long as the polymorphic API *never ever* does anything that involves decoding wire format (and I include URL-quoting here), the programmer will have to explicitly do some decoding to get into much trouble, and at that point it's really their problem; you can't stop them. But I don't know whether the web apps programmers will be satisfied with such a minimal API. If not, you're going to have to make some delicate judgments about what to provide and what not, and whether/how to provide a safety net of some kind. I don't envy you that task. > And if this turns out to be a disaster in practice: I would say be conservative about which APIs you make polymorphic. And there are a lot of APIs that probably should be considered candidates for polymorphic versions (regexp matching and searching, for example). So any experience or generic functionality you develop here is likely to benefit somebody down the road. From barry at python.org Tue Sep 21 17:23:06 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 21 Sep 2010 11:23:06 -0400 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> <1285018753.2130.36.camel@thinko> <87vd5zwuvk.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20100921112306.2ac5d66c@mission> On Sep 21, 2010, at 04:01 PM, Paul Moore wrote: >Sorry if this is off-topic, but I don't believe I ever saw Stephen's >email. I have a feeling that's happened a couple of times recently. >Before I go off trying to work out why gmail is dumping list mails on >me, did anyone else see Stephen's mail via the list (as opposed to >being a direct recipient)? I remember seeing a message from Stephen on this topic, but I didn't read it though and it's deleted now ;). I don't use gmail regularly. See also: http://wiki.list.org/x/2IA9 -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From solipsis at pitrou.net Tue Sep 21 17:26:51 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 21 Sep 2010 17:26:51 +0200 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> <1285018753.2130.36.camel@thinko> <87vd5zwuvk.fsf@uwakimon.sk.tsukuba.ac.jp> <874odj3zg6.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20100921172651.3abbfaea@pitrou.net> On Wed, 22 Sep 2010 00:10:01 +0900 "Stephen J. Turnbull" wrote: > > But I don't know whether the web apps programmers will be satisfied > with such a minimal API. Web app programmers will generally go through a framework, which handles encoding/decoding for them (already so in 2.x). > And there are a lot of APIs that probably should be considered > candidates for polymorphic versions (regexp matching and searching, > for example). As a matter of fact, the re module APIs are already polymorphic, all the while disallowing any mixing of bytes and unicode. Regards Antoine. From ianb at colorstudy.com Tue Sep 21 17:57:11 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 21 Sep 2010 11:57:11 -0400 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> <1285018753.2130.36.camel@thinko> Message-ID: On Mon, Sep 20, 2010 at 6:19 PM, Nick Coghlan wrote: > > What are the cases you believe will cause new mojibake? > > Calling operations like urlsplit on byte sequences in non-ASCII > compatible encodings and operations like urljoin on byte sequences > that are encoded with different encodings. These errors differ from > the URL escaping errors you cite, since they can produce true mojibake > (i.e. a byte sequence without a single consistent encoding), rather > than merely non-compliant URLs. However, if someone has let their > encodings get that badly out of whack in URL manipulation they're > probably doomed anyway... > FWIW, while I understand the problems non-ASCII-compatible encodings can create, I've never encountered them, perhaps because ASCII-compatible encodings are so dominant. There are ways you can get a URL (HTTP specifically) where there is no notion of Unicode. I think the use case everyone has in mind here is where you get a URL from one of these sources, and you want to handle it. I have a hard time imagining the sequence of events that would lead to mojibake. Naive parsing of a document in bytes couldn't do it, because if you have a non-ASCII-compatible document your ASCII-based parsing will also fail (e.g., looking for b'href="(.*?)"'). I suppose if you did urlparse.urlsplit(user_input.encode(sys.getdefaultencoding())) you could end up with the problem. All this is unrelated to the question, though -- a separate byte-oriented function won't help any case I can think of. If the programmer is implementing something like urlparse.urlsplit(user_input.encode(sys.getdefaultencoding())), it's because they *want* to get bytes out. So if it's named urlparse.urlsplit_bytes() they'll just use that, with the same corruption. Since bytes and text don't interact well, the choice of bytes in and bytes out will be a deliberate one. *Or*, bytes will unintentionally come through, but that will just delay the error a while when the bytes out don't work (e.g., urlparse.urljoin(text_url, urlparse.urlsplit(byte_url).path). Delaying the error is a little annoying, but a delayed error doesn't lead to mojibake. Mojibake is caused by allowing bytes and text to intermix, and the polymorphic functions as proposed don't add new dangers in that regard. -- Ian Bicking | http://blog.ianbicking.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Sep 21 18:05:35 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 21 Sep 2010 17:05:35 +0100 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: <20100921112306.2ac5d66c@mission> References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> <1285018753.2130.36.camel@thinko> <87vd5zwuvk.fsf@uwakimon.sk.tsukuba.ac.jp> <20100921112306.2ac5d66c@mission> Message-ID: On 21 September 2010 16:23, Barry Warsaw wrote: > On Sep 21, 2010, at 04:01 PM, Paul Moore wrote: > >>Sorry if this is off-topic, but I don't believe I ever saw Stephen's >>email. I have a feeling that's happened a couple of times recently. >>Before I go off trying to work out why gmail is dumping list mails on >>me, did anyone else see Stephen's mail via the list (as opposed to >>being a direct recipient)? > > I remember seeing a message from Stephen on this topic, but I didn't read it > though and it's deleted now ;). ?I don't use gmail regularly. > > See also: http://wiki.list.org/x/2IA9 Ta. I'm seeing some other messages now, it may be just that Stephen's were getting delayed. Paul From pje at telecommunity.com Tue Sep 21 18:09:44 2010 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 21 Sep 2010 12:09:44 -0400 Subject: [Python-Dev] Backup plan: WSGI 1 Addenda and wsgiref update for Py3 Message-ID: <20100921160937.4C2D23A4079@sparrow.telecommunity.com> While the Web-SIG is trying to hash out PEP 444, I thought it would be a good idea to have a backup plan that would allow the Python 3 stdlib to move forward, without needing a major new spec to settle out implementation questions. After all, even if PEP 333 is ultimately replaced by PEP 444, it's probably a good idea to have *some* sort of WSGI 1-ish thing available on Python 3, with bytes/unicode and other matters settled. In the past, I was waiting for some consensuses (consensi?) on Web-SIG about different approaches to Python 3, looking for some sort of definite, "yes, we all like this" response. However, I can see now that this just means it's my fault we don't have a spec yet. :-( So, unless any last-minute showstopper rebuttals show up this week, I've decided to go ahead officially bless nearly all of what Graham Dumpleton (who's not only the mod_wsgi author, but has put huge amounts of work into shepherding WSGI-on-Python3 proposals, WSGI amendments, etc.) has proposed, with a few minor exceptions. In other words: almost none of the following is my own original work; it's like 90% Graham's. Any praise for this belongs to him; the only thing that belongs to me is the blame for not doing this sooner! (Sorry Graham. You asked me to do this ages ago, and you were right.) Anyway, I'm posting this for comment to both Python-Dev and the Web-SIG. If you are commenting on the technical details of the amendments, please reply to the Web-SIG only. If you are commenting on the development agenda for wsgiref or other Python 3 library issues, please reply to Python-Dev only. That way, neither list will see off-topic discussions. Thanks! The Plan ======== I plan to update the proposal below per comments and feedback during this week, then update PEP 333 itself over the weekend or early next week, followed by a code review of Python 3's wsgiref, and implementation of needed changes (such as recoding os.environ to latin1-captured bytes in the CGI handler). To complete the changes, it is possible that I may need assistance from one or more developers who have more Python 3 experience. If after reading the proposed changes to the spec, you would like to volunteer to help with updating wsgiref to match, please let me know! The Proposal ============ Overview -------- 1. The primary purpose of this update is to provide a uniform porting pattern for moving Python 2 WSGI code to Python 3, meaning a pattern of changes that can be mechanically applied to as little code as practical, while still keeping the WSGI spec easy to programmatically validate (e.g. via ``wsgiref.validate``). The Python 3 specific changes are to use: * ``bytes`` for I/O streams in both directions * ``str`` for environ keys and values * ``bytes`` for arguments to start_response() and write() * text stream for wsgi.errors In other words, "strings in, bytes out" for headers, bytes for bodies. In general, only changes that don't break Python 2 WSGI implementations are allowed. The changes should also not break mod_wsgi on Python 3, but may make some Python 3 wsgi applications non-compliant, despite continuing to function on mod_wsgi. This is because mod_wsgi allows applications to output string headers and bodies, but I am ruling that option out because it forces every piece of middleware to have to be tested with arbitrary combinations of strings and bytes in order to test compliance. If you want your application to output strings rather than bytes, you can always use a decorator to do that. (And a sample one could be provided in wsgiref.) 2. The secondary purpose of the update is to address some long-standing open issues documented here: http://www.wsgi.org/wsgi/Amendments_1.0 As with the Python 3 changes, only changes that don't retroactively invalidate existing implementations are allowed. 3. There is no tertiary purpose. ;-) (By which I mean, all other kinds of changes are out-of-scope for this update.) 4. The section below labeled "A Note On String Types" is proposed for verbatim addition to the "Specification Overview" section in the PEP; the other sections below describe changes to be made inline at the appropriate part of the spec, and changes that were proposed but are rejected for inclusion in this amendment. A Note On String Types ---------------------- In general, HTTP deals with bytes, which means that this specification is mostly about handling bytes. However, the content of those bytes often has some kind of textual interpretation, and in Python, strings are the most convenient way to handle text. But in many Python versions and implementations, strings are Unicode, rather than bytes. This requires a careful balance between a usable API and correct translations between bytes and text in the context of HTTP... especially to support porting code between Python implementations with different ``str`` types. WSGI therefore defines two kinds of "string": * "Native" strings (which are always implemented using the type named ``str``) * "Bytestrings" (which are implemented using the ``bytes`` type in Python 3, and ``str`` elsewhere) So, even though HTTP is in some sense "really just bytes", there are many API conveniences to be had by using whatever Python's default ``str`` type is. Do not be confused however: even if Python's ``str`` is actually Unicode under the hood, the *content* of a native string is still restricted to bytes! See the section on `Unicode Issues`_ later in this document. In short: where you see the word "string" in this document, it refers to a "native" string, i.e., an object of type ``str``, whether it is internally implemented as bytes or unicode. Where you see references to "bytestring", this should be read as "an object of type ``bytes`` under Python 3, or type ``str`` under Python 2". Clarifications (To be made in-line) ----------------------------------- The following amendments are clarifications to parts of the existing spec that proved over the years to be ambiguous or insufficiently specified, as well as some attempts to correct practical errors. (Note: many of these issues cannot be completely fixed in WSGI 1 without breaking existing implementations, and so the text below has notations such as "(MUST in WSGI 2)" to indicate where any replacement spec for WSGI 1 should strengthen them.) * If an application returns a body iterator, a server (or middleware) MAY stop iterating over it and discard the remainder of the output, as long as it calls any close() method provided by the iterator. Applications returning a generator or other custom iterator SHOULD NOT assume that the entire iterator will be consumed. (This change makes it explicit that caching middleware or HEAD-processing servers can throw away the response body.) * start_response() SHOULD (MUST in WSGI 2) check for errors in the status or headers at the time it's called, so that an error can be raised as close to the problem as possible * If start_response() raises an error when called normally (i.e. without exc_info), it SHOULD be an error to call it a second time without passing exc_info * The SERVER_PORT variable is of type str, just like any other CGI environ variable. (According to the WSGI wiki, "some implementations" expect it to be an integer, even though there is nothing in the WSGI spec that allows a CGI variable to be anything but a str.) * A server SHOULD (MUST in WSGI 2) support the size hint argument to readline() on its wsgi.input stream. * A server SHOULD (MUST in WSGI 2) return an empty bytestring from read() on wsgi.input to indicate an end-of-file condition. (In WSGI 2, language should be clarified to allow the input stream length and CONTENT_LENGTH to be out of sync, for reasons explained in Graham's blog post.) * A server SHOULD (MUST in WSGI 2) allow read() to be called without an argument, and return the entire remaining contents of the stream * If an application provides a Content-Length header, the server SHOULD NOT (MUST NOT in WSGI 2) send more data to the client than was specified in that header, whether via write(), yielded body bytestrings, or via a wsgi.file_wrapper. (This rule applies to middleware as well.) * wsgi.errors is a text stream accepting "native strings" Rejected Amendments ------------------- * Manlio Perillo's suggestion to allow header specification to be delayed until the response iterator is producing non-empty output. This would've been a possible win for async WSGI, but could require substantial changes to existing servers. From chrism at plope.com Tue Sep 21 18:13:38 2010 From: chrism at plope.com (Chris McDonough) Date: Tue, 21 Sep 2010 12:13:38 -0400 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> <1285018753.2130.36.camel@thinko> <87vd5zwuvk.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1285085618.2130.63.camel@thinko> On Tue, 2010-09-21 at 23:38 +1000, Nick Coghlan wrote: > And if this turns out to be a disaster in practice: > a) on my head be it; and > b) we still have the option of the DeprecationWarning dance for bytes > inputs to the existing functions and moving to a parallel API In the case of urllib.parse, it's entirely safe. If someone beats you up over it later, you can tell them to bother "straddlers" in Web-SIG, as we're the folks who most want the polymorphism in that particular API. - C From chrism at plope.com Tue Sep 21 18:47:00 2010 From: chrism at plope.com (Chris McDonough) Date: Tue, 21 Sep 2010 12:47:00 -0400 Subject: [Python-Dev] [Web-SIG] Backup plan: WSGI 1 Addenda and wsgiref update for Py3 In-Reply-To: <20100921160937.4C2D23A4079@sparrow.telecommunity.com> References: <20100921160937.4C2D23A4079@sparrow.telecommunity.com> Message-ID: <1285087620.2130.104.camel@thinko> On Tue, 2010-09-21 at 12:09 -0400, P.J. Eby wrote: > While the Web-SIG is trying to hash out PEP 444, I thought it would > be a good idea to have a backup plan that would allow the Python 3 > stdlib to move forward, without needing a major new spec to settle > out implementation questions. If a WSGI-1-compatible protocol seems more sensible to folks, I'm personally happy to defer discussion on PEP 444 or any other backwards-incompatible proposal. - C From solipsis at pitrou.net Tue Sep 21 18:52:54 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 21 Sep 2010 18:52:54 +0200 Subject: [Python-Dev] Backup plan: WSGI 1 Addenda and wsgiref update for Py3 References: <20100921160937.4C2D23A4079@sparrow.telecommunity.com> Message-ID: <20100921185254.10becbcc@pitrou.net> On Tue, 21 Sep 2010 12:09:44 -0400 "P.J. Eby" wrote: > While the Web-SIG is trying to hash out PEP 444, I thought it would > be a good idea to have a backup plan that would allow the Python 3 > stdlib to move forward, without needing a major new spec to settle > out implementation questions. If this allows the Web situation in Python 3 to be improved faster and with less hassle then all the better. There's something strange in your proposal: it mentions WSGI 2 at several places while there's no guarantee about what WSGI 2 will be (is there?). Regards Antoine. From ianb at colorstudy.com Tue Sep 21 18:55:15 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 21 Sep 2010 12:55:15 -0400 Subject: [Python-Dev] [Web-SIG] Backup plan: WSGI 1 Addenda and wsgiref update for Py3 In-Reply-To: <1285087620.2130.104.camel@thinko> References: <20100921160937.4C2D23A4079@sparrow.telecommunity.com> <1285087620.2130.104.camel@thinko> Message-ID: On Tue, Sep 21, 2010 at 12:47 PM, Chris McDonough wrote: > On Tue, 2010-09-21 at 12:09 -0400, P.J. Eby wrote: > > While the Web-SIG is trying to hash out PEP 444, I thought it would > > be a good idea to have a backup plan that would allow the Python 3 > > stdlib to move forward, without needing a major new spec to settle > > out implementation questions. > > If a WSGI-1-compatible protocol seems more sensible to folks, I'm > personally happy to defer discussion on PEP 444 or any other > backwards-incompatible proposal. > I think both make sense, making WSGI 1 sensible for Python 3 (as well as other small errata like the size hint) doesn't detract from PEP 444 at all, IMHO. -- Ian Bicking | http://blog.ianbicking.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From ianb at colorstudy.com Tue Sep 21 18:57:44 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 21 Sep 2010 12:57:44 -0400 Subject: [Python-Dev] [Web-SIG] Backup plan: WSGI 1 Addenda and wsgiref update for Py3 In-Reply-To: <20100921160937.4C2D23A4079@sparrow.telecommunity.com> References: <20100921160937.4C2D23A4079@sparrow.telecommunity.com> Message-ID: On Tue, Sep 21, 2010 at 12:09 PM, P.J. Eby wrote: > The Python 3 specific changes are to use: > > * ``bytes`` for I/O streams in both directions > * ``str`` for environ keys and values > * ``bytes`` for arguments to start_response() and write() > This is the only thing that seems odd to me -- it seems like the response should be symmetric with the request, and the request in this case uses str for headers (status being header-like), and bytes for the body. Otherwise this seems good to me, the only other major errata I can think of are all listed in the links you included. * text stream for wsgi.errors > > In other words, "strings in, bytes out" for headers, bytes for bodies. > > In general, only changes that don't break Python 2 WSGI implementations are > allowed. The changes should also not break mod_wsgi on Python 3, but may > make some Python 3 wsgi applications non-compliant, despite continuing to > function on mod_wsgi. > > This is because mod_wsgi allows applications to output string headers and > bodies, but I am ruling that option out because it forces every piece of > middleware to have to be tested with arbitrary combinations of strings and > bytes in order to test compliance. If you want your application to output > strings rather than bytes, you can always use a decorator to do that. (And > a sample one could be provided in wsgiref.) > I agree allowing both is not ideal. -- Ian Bicking | http://blog.ianbicking.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Tue Sep 21 19:10:10 2010 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 21 Sep 2010 13:10:10 -0400 Subject: [Python-Dev] [Web-SIG] Backup plan: WSGI 1 Addenda and wsgiref update for Py3 In-Reply-To: References: <20100921160937.4C2D23A4079@sparrow.telecommunity.com> <1285087620.2130.104.camel@thinko> Message-ID: <20100921171005.0064F3A4079@sparrow.telecommunity.com> At 12:55 PM 9/21/2010 -0400, Ian Bicking wrote: >On Tue, Sep 21, 2010 at 12:47 PM, Chris McDonough ><chrism at plope.com> wrote: >On Tue, 2010-09-21 at 12:09 -0400, P.J. Eby wrote: > > While the Web-SIG is trying to hash out PEP 444, I thought it would > > be a good idea to have a backup plan that would allow the Python 3 > > stdlib to move forward, without needing a major new spec to settle > > out implementation questions. > >If a WSGI-1-compatible protocol seems more sensible to folks, I'm >personally happy to defer discussion on PEP 444 or any other >backwards-incompatible proposal. > > >I think both make sense, making WSGI 1 sensible for Python 3 (as >well as other small errata like the size hint) doesn't detract from >PEP 444 at all, IMHO. Yep. I agree. I do, however, want to get these amendments settled and make sure they get carried over to whatever spec is the successor to PEP 333. I've had a lot of trouble following exactly what was changed in 444, and I'm a tad worried that several new ambiguities may be being introduced. So, solidifying 333 a bit might be helpful if it gives a good baseline against which to diff 444 (or whatever). From jacob at jacobian.org Tue Sep 21 19:10:27 2010 From: jacob at jacobian.org (Jacob Kaplan-Moss) Date: Tue, 21 Sep 2010 12:10:27 -0500 Subject: [Python-Dev] Backup plan: WSGI 1 Addenda and wsgiref update for Py3 In-Reply-To: <20100921160937.4C2D23A4079@sparrow.telecommunity.com> References: <20100921160937.4C2D23A4079@sparrow.telecommunity.com> Message-ID: On Tue, Sep 21, 2010 at 11:09 AM, P.J. Eby wrote: > After all, even if PEP 333 is ultimately replaced by PEP 444, it's probably > a good idea to have *some* sort of WSGI 1-ish thing available on Python 3, > with bytes/unicode and other matters settled. Indeed. Though I generally like the direction that PEP 444 is going in, I know that writing specs is *HARD*. I think having something that works on Python 3 in time for the 3.2 release is a much bigger deal than having an WSGI2 (or whatever) done. This is classic "the perfect is the enemy of the good" territory. Let's get "the good" done and *then* spend time working on "the perfect". Jacob From pje at telecommunity.com Tue Sep 21 19:11:50 2010 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 21 Sep 2010 13:11:50 -0400 Subject: [Python-Dev] Backup plan: WSGI 1 Addenda and wsgiref update for Py3 In-Reply-To: <20100921185254.10becbcc@pitrou.net> References: <20100921160937.4C2D23A4079@sparrow.telecommunity.com> <20100921185254.10becbcc@pitrou.net> Message-ID: <20100921171143.1A8663A4079@sparrow.telecommunity.com> At 06:52 PM 9/21/2010 +0200, Antoine Pitrou wrote: >On Tue, 21 Sep 2010 12:09:44 -0400 >"P.J. Eby" wrote: > > While the Web-SIG is trying to hash out PEP 444, I thought it would > > be a good idea to have a backup plan that would allow the Python 3 > > stdlib to move forward, without needing a major new spec to settle > > out implementation questions. > >If this allows the Web situation in Python 3 to be improved faster >and with less hassle then all the better. >There's something strange in your proposal: it mentions WSGI 2 at >several places while there's no guarantee about what WSGI 2 will be (is >there?). Sorry - "WSGI 2" should be read as shorthand for, "whatever new spec succeeds PEP 333", whether that's PEP 444 or something else. It just means that any new spec that doesn't have to be backward-compatible can (and should) more thoroughly address the issue in question. From ncoghlan at gmail.com Tue Sep 21 23:33:19 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 22 Sep 2010 07:33:19 +1000 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: <874odj3zg6.fsf@uwakimon.sk.tsukuba.ac.jp> References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> <1285018753.2130.36.camel@thinko> <87vd5zwuvk.fsf@uwakimon.sk.tsukuba.ac.jp> <874odj3zg6.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Wed, Sep 22, 2010 at 1:10 AM, Stephen J. Turnbull wrote: > Nick Coghlan writes: > > ?> (Basically, while the issue of programmers assuming 'latin-1' or > ?> 'utf-8' or similar ASCII friendly encodings when they shouldn't is > ?> real, I don't believe a polymorphic API here will make things any > ?> *worse* than what would happen with a parallel API) > > That depends on how far the polymorphic API goes. ?As long as the > polymorphic API *never ever* does anything that involves decoding wire > format (and I include URL-quoting here), the programmer will have to > explicitly do some decoding to get into much trouble, and at that > point it's really their problem; you can't stop them. > > But I don't know whether the web apps programmers will be satisfied > with such a minimal API. ?If not, you're going to have to make some > delicate judgments about what to provide and what not, and whether/how > to provide a safety net of some kind. ?I don't envy you that task. As Chris pointed out, Issue 3300 means that particular boat has already sailed where quote/unquote are concerned. Those are the only APIs which ever need to do any encoding or decoding, as they deal with percent-encoding of Unicode characters. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Tue Sep 21 23:35:40 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 22 Sep 2010 07:35:40 +1000 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> <1285018753.2130.36.camel@thinko> Message-ID: On Wed, Sep 22, 2010 at 1:57 AM, Ian Bicking wrote: > All this is unrelated to the question, though -- a separate byte-oriented > function won't help any case I can think of.? If the programmer is > implementing something like > urlparse.urlsplit(user_input.encode(sys.getdefaultencoding())), it's because > they *want* to get bytes out.? So if it's named urlparse.urlsplit_bytes() > they'll just use that, with the same corruption.? Since bytes and text don't > interact well, the choice of bytes in and bytes out will be a deliberate > one.? *Or*, bytes will unintentionally come through, but that will just > delay the error a while when the bytes out don't work (e.g., > urlparse.urljoin(text_url, urlparse.urlsplit(byte_url).path).? Delaying the > error is a little annoying, but a delayed error doesn't lead to mojibake. Indeed, this line of thinking is what brought me back around to the polymorphic point of view. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From nyamatongwe at gmail.com Wed Sep 22 01:15:13 2010 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Wed, 22 Sep 2010 09:15:13 +1000 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> <1285018753.2130.36.camel@thinko> Message-ID: Ian Bicking: > I think the use case everyone has in mind here is where > you get a URL from one of these sources, and you want to handle it.? I have > a hard time imagining the sequence of events that would lead to mojibake. > Naive parsing of a document in bytes couldn't do it, because if you have a > non-ASCII-compatible document your ASCII-based parsing will also fail (e.g., > looking for b'href="(.*?)"'). It depends on what the particular ASCII-based parsing is doing. For example, the set of trail bytes in Shift-JIS includes the same bytes as some of the punctuation characters in ASCII as well as all the letters. A search or split on '@' or '|' may find the trail byte in a two-byte character rather than a true occurrence of that character so the operation 'succeeds' but produces an incorrect result. Over time, the set of trail bytes used has expanded - in GB18030 digits are possible although many of the most important characters for parsing such as ''' "#%&.?/''' are still safe as they may not be trail bytes in the common double-byte character sets. Neil From andrewm at object-craft.com.au Wed Sep 22 01:37:55 2010 From: andrewm at object-craft.com.au (Andrew McNamara) Date: Wed, 22 Sep 2010 09:37:55 +1000 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: Your message of "Tue, 21 Sep 2010 23:38:08 +1000." Message-ID: <20100921233756.1FD5220571@longblack.object-craft.com.au> >> On the other hand, it is dangerous to provide a polymorphic API which >> does that more extensive parsing, because a less than paranoid >> programmer will have very likely allowed the parsed components to >> escape from the context where their encodings can be reliably >> determined. =A0Remember, *it is unlikely that they will ever be punished >> for their own lack of caution.* =A0The person who is doomed is somebody >> who tries to take that code and reuse it in a different context. > >Yeah, that's the original reasoning that had me leaning towards the >parallel API approach. If I seem to be changing my mind a lot in this >thread it's because I'm genuinely torn between the desire to make it >easier to port existing 2.x code to 3.x by making the current API >polymorphic and the fear that doing so will reintroduce some of the >exact same bytes/text confusion that the bytes/str split is trying to >get rid of. I don't think polymorphic API's do anyone any favours in the long run. My experience of the Py2 email API was that it would give the developer false comfort, only to blow up when the app was in the hands of users, and it didn't seem to matter how careful I was. Py3 has gone the pure/strict route in the core, and I think libs should be consistent with that choice. Developers will have work a little harder, but there will be less surprises. -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/ From breamoreboy at yahoo.co.uk Wed Sep 22 01:58:54 2010 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Wed, 22 Sep 2010 00:58:54 +0100 Subject: [Python-Dev] Goodbye Message-ID: I'm rather sad to have been sacked, but such is life. I won't be doing any more work on the bug tracker for obvious reasons, but hope that you who have managed to keep your voluntary jobs manage to keep Python going. Kindest regards. Mark Lawrence. From jackdied at gmail.com Wed Sep 22 02:16:52 2010 From: jackdied at gmail.com (Jack Diederich) Date: Tue, 21 Sep 2010 20:16:52 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: References: Message-ID: On Tue, Sep 21, 2010 at 7:58 PM, Mark Lawrence wrote: > I'm rather sad to have been sacked, but such is life. ?I won't be doing any > more work on the bug tracker for obvious reasons, but hope that you who have > managed to keep your voluntary jobs manage to keep Python going. Umm, what? You mean http://bugs.python.org/issue2180 ? """ Mark, please stop closing these based on age. The needs to be a determination whether this is a valid bug. If so, then a patch is needed. If not, it can be closed.""" Am I missing something? -Jack From stephen at xemacs.org Wed Sep 22 04:59:23 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 22 Sep 2010 11:59:23 +0900 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> <1285018753.2130.36.camel@thinko> Message-ID: <87vd5y32lw.fsf@uwakimon.sk.tsukuba.ac.jp> Neil Hodgson writes: > Over time, the set of trail bytes used has expanded - in GB18030 > digits are possible although many of the most important characters > for parsing such as ''' "#%&.?/''' are still safe as they may not > be trail bytes in the common double-byte character sets. That's just not true. Many double-byte character sets in use are based on ISO-2022, which allows the whole GL repertoire to be used. Perhaps you're thinking about variable-width encodings like Shift JIS and Big5, where I believe that restriction on trailing bytes for double-byte characters holds. However, 7-bit encodings with control sequences remain common in several contexts, at least in Japan and Korea. In particular, I can't say how frequent it is, especially nowadays, but I have seen ISO-2022-JP in URLs "on the wire". What really saves the day here is not that "common encodings just don't do that". It's that even in the case where only syntactically significant bytes in the representation are URL-encoded, they *are* URL-encoded. As long as the parsing library restricts itself to treating only wire-format input, you're OK.[1] But once you start doing things that involve decoding URL-encoding, you can run into trouble. Footnotes: [1] With conforming input. I assume that the libraries know how to defend themselves from non-conforming input, which could be any kind of bug or attack, not just mojibake. From solipsis at pitrou.net Wed Sep 22 12:47:00 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 22 Sep 2010 12:47:00 +0200 Subject: [Python-Dev] Goodbye References: Message-ID: <20100922124700.000ac012@pitrou.net> On Tue, 21 Sep 2010 20:16:52 -0400 Jack Diederich wrote: > On Tue, Sep 21, 2010 at 7:58 PM, Mark Lawrence wrote: > > I'm rather sad to have been sacked, but such is life. ?I won't be doing any > > more work on the bug tracker for obvious reasons, but hope that you who have > > managed to keep your voluntary jobs manage to keep Python going. > > Umm, what? You mean http://bugs.python.org/issue2180 ? > > """ Mark, please stop closing these based on age. > The needs to be a determination whether this > is a valid bug. If so, then a patch is needed. > If not, it can be closed.""" > > Am I missing something? Simply, situations like the above (Mark closing a bug just because nobody would answer his message on a short delay) have happened multiple times - despite people opposing, obviously -, and we decided that it was better to remove his tracker privileges since his contribution has not really been productive for us. There was a whole python-dev thread some time (weeks? months?) ago where several of us already tried to suggest more fruitful ways of contributing, suggestions which weren't received very welcomingly AFAIR. Now I understand that opinions over this may vary and involve multiple factors, but I would suggest that at least a bit of mentoring is needed if we want to give privileges early on. (and the amount of mentoring needed can vary wildly from one person to another) Regards Antoine. From ncoghlan at gmail.com Wed Sep 22 12:48:21 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 22 Sep 2010 20:48:21 +1000 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: <20100921233756.1FD5220571@longblack.object-craft.com.au> References: <20100921233756.1FD5220571@longblack.object-craft.com.au> Message-ID: On Wed, Sep 22, 2010 at 9:37 AM, Andrew McNamara wrote: >>Yeah, that's the original reasoning that had me leaning towards the >>parallel API approach. If I seem to be changing my mind a lot in this >>thread it's because I'm genuinely torn between the desire to make it >>easier to port existing 2.x code to 3.x by making the current API >>polymorphic and the fear that doing so will reintroduce some of the >>exact same bytes/text confusion that the bytes/str split is trying to >>get rid of. > > I don't think polymorphic API's do anyone any favours in the long > run. My experience of the Py2 email API was that it would give the > developer false comfort, only to blow up when the app was in the hands > of users, and it didn't seem to matter how careful I was. Py3 has gone > the pure/strict route in the core, and I think libs should be consistent > with that choice. ?Developers will have work a little harder, but there > will be less surprises. There's an important distinction here though. Either change I could make to urllib.parse will still result in two distinct APIs. The only question is whether the new bytes->bytes APIs need to have a different spelling or not. Python 2.x is close to impossible to reliably test in this area because there's no programmatic way to tell the difference between encoded bytes and decoded text. In Python 3, while you can still get yourself in trouble by mixing encodings at the bytes level, you're almost never going to mistake bytes for text unless you go out of your way to support working that way. The structure of quote/unquote (which already contain implicit decode/encode steps to allow them to consume both bytes and strings with relative abandon and have done since 3.0) may cause us problems in the long run, but polymorphic APIs where the type of the input is the same as the type of the output shouldn't be any more dangerous than if those same APIs used a different spelling to operate on bytes. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Sep 22 13:07:22 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 22 Sep 2010 21:07:22 +1000 Subject: [Python-Dev] Goodbye In-Reply-To: <20100922124700.000ac012@pitrou.net> References: <20100922124700.000ac012@pitrou.net> Message-ID: On Wed, Sep 22, 2010 at 8:47 PM, Antoine Pitrou wrote: > Simply, situations like the above (Mark closing a bug just because > nobody would answer his message on a short delay) have happened > multiple times - despite people opposing, obviously -, and we decided > that it was better to remove his tracker privileges since his > contribution has not really been productive for us. > > There was a whole python-dev thread some time (weeks? months?) ago where > several of us already tried to suggest more fruitful ways of > contributing, suggestions which weren't received very welcomingly AFAIR. > > Now I understand that opinions over this may vary and involve multiple > factors, but I would suggest that at least a bit of mentoring is needed > if we want to give privileges early on. > (and the amount of mentoring needed can vary wildly from one person to > another) I still prefer the "trust but monitor" approach over excessively high barriers to entry, but we do need to recognise that one consequence of that approach is that we *will* get into situations where we need to tell people "thank you for your contributions, but we think, on balance, we will be better off if you don't contribute in this way any more". Mark *did* do quite a bit of good in his time with tracker privileges. A number of lingering issues that would have otherwise continued lingering did indeed get closed. That work is still appreciated, even if it was ultimately deemed by the other tracker admins not to be sufficient to balance out the hassles created by his aggressive stance towards closing older issues (which, while unloved, are not automatically invalid). If this had happened *without* the prior discussion regarding more appropriate handling of tracker issues, then I would have an issue with it. However, given that the first reaction was to provide additional mentoring, with revocation of privileges only happening when the problems continued, that seems to me like the way this process is *meant* to work. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Sep 22 14:07:47 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 22 Sep 2010 22:07:47 +1000 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: <87vd5y32lw.fsf@uwakimon.sk.tsukuba.ac.jp> References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> <1285018753.2130.36.camel@thinko> <87vd5y32lw.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Wed, Sep 22, 2010 at 12:59 PM, Stephen J. Turnbull wrote: > Neil Hodgson writes: > > ?> ? ?Over time, the set of trail bytes used has expanded - in GB18030 > ?> digits are possible although many of the most important characters > ?> for parsing such as ''' "#%&.?/''' are still safe as they may not > ?> be trail bytes in the common double-byte character sets. > > That's just not true. ?Many double-byte character sets in use are > based on ISO-2022, which allows the whole GL repertoire to be used. > > Perhaps you're thinking about variable-width encodings like Shift JIS > and Big5, where I believe that restriction on trailing bytes for > double-byte characters holds. ?However, 7-bit encodings with control > sequences remain common in several contexts, at least in Japan and > Korea. ?In particular, I can't say how frequent it is, especially > nowadays, but I have seen ISO-2022-JP in URLs "on the wire". Notably, utf-16 and utf-32 make no promises regarding avoidance of ASCII character codes in trail bytes - only utf-8 is guaranteed to be compatible with parsing as if it were ASCII (and even then, you need to be careful only to split the string at known ASCII characters rather than at arbitrary points). The known-ASCII-incompatible multibyte encodings I came up with when I reviewed the list in the codecs module docs the other day were: CP932 (the example posted here that prompted me to embark on this check in the first place) UTF-7 UTF-16 UTF-32 shift-JIS big5 iso-2022-* EUC-CN/KR/TW The only known-ASCII-compatible multibyte encodings I found were UTF-8 and EUC-JP (all of the non-EBCDIC single byte encodings appeared to be ASCII compatible though) I didn't check any of the other CP* encodings though, since I already had plenty of examples to show that the assumption of ASCII compatibility isn't likely to be valid in general unless there is some other constraint (such as the RFCs for safely encoding URLs to an octet-sequence). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From steve at holdenweb.com Wed Sep 22 15:58:18 2010 From: steve at holdenweb.com (Steve Holden) Date: Wed, 22 Sep 2010 09:58:18 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: References: Message-ID: On 9/21/2010 7:58 PM, Mark Lawrence wrote: > I'm rather sad to have been sacked, but such is life. I won't be doing > any more work on the bug tracker for obvious reasons, but hope that you > who have managed to keep your voluntary jobs manage to keep Python going. > > Kindest regards. > > Mark Lawrence. > Mark: Whatever the situation vis a vis the bug tracker, thank you for caring enough about Python to put in the time that you have. If you didn't care you would not have done all that hard work, and I hope you can find other ways to contribute further to Python's success. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From guido at python.org Wed Sep 22 16:17:09 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Sep 2010 07:17:09 -0700 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> Message-ID: On Wed, Sep 22, 2010 at 4:07 AM, Nick Coghlan wrote: > On Wed, Sep 22, 2010 at 8:47 PM, Antoine Pitrou wrote: >> Simply, situations like the above (Mark closing a bug just because >> nobody would answer his message on a short delay) have happened >> multiple times - despite people opposing, obviously -, and we decided >> that it was better to remove his tracker privileges since his >> contribution has not really been productive for us. >> >> There was a whole python-dev thread some time (weeks? months?) ago where I think it was the thread "No response to posts" started (by Mark) on July 31. >> several of us already tried to suggest more fruitful ways of >> contributing, suggestions which weren't received very welcomingly AFAIR. Yup. In that thread (and others) I see lots of evidence where Mark responded very negatively (from "I disagree entirely" to "I find this response quite pathetic") when people explained how we treat the tracker, and stuck to his guns no matter how many people tried to explain that he should stop. His attitude can be summarized by his "Fly back at me if you like. I don't care about me. I don't care about you. I do care about Python." Which to me sounds defiant and passive-aggressive. I don't want to go into analyzing, but I expect that Mark has issues that are beyond what this community can deal with. >> Now I understand that opinions over this may vary and involve multiple >> factors, but I would suggest that at least a bit of mentoring is needed >> if we want to give privileges early on. >> (and the amount of mentoring needed can vary wildly from one person to >> another) > > I still prefer the "trust but monitor" approach over excessively high > barriers to entry, but we do need to recognise that one consequence of > that approach is that we *will* get into situations where we need to > tell people "thank you for your contributions, but we think, on > balance, we will be better off if you don't contribute in this way any > more". > > Mark *did* do quite a bit of good in his time with tracker privileges. Right, that was my impression from the issues he touched on which I happened to be subscribed. > A number of lingering issues that would have otherwise continued > lingering did indeed get closed. That work is still appreciated, even > if it was ultimately deemed by the other tracker admins not to be > sufficient to balance out the hassles created by his aggressive stance > towards closing older issues (which, while unloved, are not > automatically invalid). How and how often was Mark reminded about this? > If this had happened *without* the prior discussion regarding more > appropriate handling of tracker issues, then I would have an issue > with it. However, given that the first reaction was to provide > additional mentoring, with revocation of privileges only happening > when the problems continued, that seems to me like the way this > process is *meant* to work. Where was the decision to revoke privileges discussed? Not on any mailing list that I am subscribed to. Was Mark given an ultimatum? Given that this came out rather unfortunately (even if the end result is the best that could have happened) I would recommend that in the future more attention is paid to "documenting" publicly that someone's being booted out was inevitable, by an exchange of messages on python-dev (or python-committers if we want to limit distribution). And no, I don't think that IRC (where I suspect this happened) is sufficient. -- --Guido van Rossum (python.org/~guido) From darren at ontrenet.com Wed Sep 22 16:33:39 2010 From: darren at ontrenet.com (darren at ontrenet.com) Date: Wed, 22 Sep 2010 10:33:39 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> Message-ID: If you guys continue to make a public jury of this, no one else will want to step into that role.... > On Wed, Sep 22, 2010 at 4:07 AM, Nick Coghlan wrote: >> On Wed, Sep 22, 2010 at 8:47 PM, Antoine Pitrou >> wrote: >>> Simply, situations like the above (Mark closing a bug just because >>> nobody would answer his message on a short delay) have happened >>> multiple times - despite people opposing, obviously -, and we decided >>> that it was better to remove his tracker privileges since his >>> contribution has not really been productive for us. >>> >>> There was a whole python-dev thread some time (weeks? months?) ago >>> where > > I think it was the thread "No response to posts" started (by Mark) on July > 31. > >>> several of us already tried to suggest more fruitful ways of >>> contributing, suggestions which weren't received very welcomingly >>> AFAIR. > > Yup. In that thread (and others) I see lots of evidence where Mark > responded very negatively (from "I disagree entirely" to "I find this > response quite pathetic") when people explained how we treat the > tracker, and stuck to his guns no matter how many people tried to > explain that he should stop. > > His attitude can be summarized by his "Fly back at me if you like. I > don't care about me. I don't care about you. I do care about > Python." > > Which to me sounds defiant and passive-aggressive. I don't want to go > into analyzing, but I expect that Mark has issues that are beyond what > this community can deal with. > >>> Now I understand that opinions over this may vary and involve multiple >>> factors, but I would suggest that at least a bit of mentoring is needed >>> if we want to give privileges early on. >>> (and the amount of mentoring needed can vary wildly from one person to >>> another) >> >> I still prefer the "trust but monitor" approach over excessively high >> barriers to entry, but we do need to recognise that one consequence of >> that approach is that we *will* get into situations where we need to >> tell people "thank you for your contributions, but we think, on >> balance, we will be better off if you don't contribute in this way any >> more". >> >> Mark *did* do quite a bit of good in his time with tracker privileges. > > Right, that was my impression from the issues he touched on which I > happened to be subscribed. > >> A number of lingering issues that would have otherwise continued >> lingering did indeed get closed. That work is still appreciated, even >> if it was ultimately deemed by the other tracker admins not to be >> sufficient to balance out the hassles created by his aggressive stance >> towards closing older issues (which, while unloved, are not >> automatically invalid). > > How and how often was Mark reminded about this? > >> If this had happened *without* the prior discussion regarding more >> appropriate handling of tracker issues, then I would have an issue >> with it. However, given that the first reaction was to provide >> additional mentoring, with revocation of privileges only happening >> when the problems continued, that seems to me like the way this >> process is *meant* to work. > > Where was the decision to revoke privileges discussed? Not on any > mailing list that I am subscribed to. Was Mark given an ultimatum? > > Given that this came out rather unfortunately (even if the end result > is the best that could have happened) I would recommend that in the > future more attention is paid to "documenting" publicly that someone's > being booted out was inevitable, by an exchange of messages on > python-dev (or python-committers if we want to limit distribution). > And no, I don't think that IRC (where I suspect this happened) is > sufficient. > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/darren%40ontrenet.com > From benjamin at python.org Wed Sep 22 17:06:06 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 22 Sep 2010 10:06:06 -0500 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> Message-ID: 2010/9/22 Guido van Rossum : > On Wed, Sep 22, 2010 at 4:07 AM, Nick Coghlan wrote: >> A number of lingering issues that would have otherwise continued >> lingering did indeed get closed. That work is still appreciated, even >> if it was ultimately deemed by the other tracker admins not to be >> sufficient to balance out the hassles created by his aggressive stance >> towards closing older issues (which, while unloved, are not >> automatically invalid). > > How and how often was Mark reminded about this? I believe that mailing list thread was the main thrust. However, many issues which he closed were reopened with a message saying why they shouldn't be closed. > >> If this had happened *without* the prior discussion regarding more >> appropriate handling of tracker issues, then I would have an issue >> with it. However, given that the first reaction was to provide >> additional mentoring, with revocation of privileges only happening >> when the problems continued, that seems to me like the way this >> process is *meant* to work. > > Where was the decision to revoke privileges discussed? Not on any > mailing list that I am subscribed to. Was Mark given an ultimatum? Indeed, it was on IRC. > > Given that this came out rather unfortunately (even if the end result > is the best that could have happened) I would recommend that in the > future more attention is paid to "documenting" publicly that someone's > being booted out was inevitable, by an exchange of messages on > python-dev (or python-committers if we want to limit distribution). > And no, I don't think that IRC (where I suspect this happened) is > sufficient. We'll note that for the future. -- Regards, Benjamin From fuzzyman at voidspace.org.uk Wed Sep 22 17:16:15 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 22 Sep 2010 16:16:15 +0100 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> Message-ID: <4C9A1DBF.3030409@voidspace.org.uk> On 22/09/2010 15:33, darren at ontrenet.com wrote: > If you guys continue to make a public jury of this, no one else will want > to step into that role.... > One of the perhaps-downsides of projects with an open community and open development processes is that any dirty-laundry there might be tends to get washed in public. Difficult decisions will always be accompanied by a measure of soul-searching and disagreement. I guess this is what you mean by "public jury". I think reaching decisions like this in private, without public discussion, would be worse (decisions could only be made by a 'secret cabal' with much less accountability and opportunity to improve). I don't think this kind of process can ever be easy (unless we eliminate the involvement of humans in Python development altogether), but we do have a process. Particularly bearing in mind the comments of Guido on the topic we can further improve the process. I too found Mark's contributions to issues I'm involved in helpful, but I understand the decision entirely. We all need to be able to work together and despite best efforts that just wasn't working out. I also wish Mark the best for the future and hope that he is still able to find some way to contribute to Python. All the best, Michael Foord >> On Wed, Sep 22, 2010 at 4:07 AM, Nick Coghlan wrote: >>> On Wed, Sep 22, 2010 at 8:47 PM, Antoine Pitrou >>> wrote: >>>> Simply, situations like the above (Mark closing a bug just because >>>> nobody would answer his message on a short delay) have happened >>>> multiple times - despite people opposing, obviously -, and we decided >>>> that it was better to remove his tracker privileges since his >>>> contribution has not really been productive for us. >>>> >>>> There was a whole python-dev thread some time (weeks? months?) ago >>>> where >> I think it was the thread "No response to posts" started (by Mark) on July >> 31. >> >>>> several of us already tried to suggest more fruitful ways of >>>> contributing, suggestions which weren't received very welcomingly >>>> AFAIR. >> Yup. In that thread (and others) I see lots of evidence where Mark >> responded very negatively (from "I disagree entirely" to "I find this >> response quite pathetic") when people explained how we treat the >> tracker, and stuck to his guns no matter how many people tried to >> explain that he should stop. >> >> His attitude can be summarized by his "Fly back at me if you like. I >> don't care about me. I don't care about you. I do care about >> Python." >> >> Which to me sounds defiant and passive-aggressive. I don't want to go >> into analyzing, but I expect that Mark has issues that are beyond what >> this community can deal with. >> >>>> Now I understand that opinions over this may vary and involve multiple >>>> factors, but I would suggest that at least a bit of mentoring is needed >>>> if we want to give privileges early on. >>>> (and the amount of mentoring needed can vary wildly from one person to >>>> another) >>> I still prefer the "trust but monitor" approach over excessively high >>> barriers to entry, but we do need to recognise that one consequence of >>> that approach is that we *will* get into situations where we need to >>> tell people "thank you for your contributions, but we think, on >>> balance, we will be better off if you don't contribute in this way any >>> more". >>> >>> Mark *did* do quite a bit of good in his time with tracker privileges. >> Right, that was my impression from the issues he touched on which I >> happened to be subscribed. >> >>> A number of lingering issues that would have otherwise continued >>> lingering did indeed get closed. That work is still appreciated, even >>> if it was ultimately deemed by the other tracker admins not to be >>> sufficient to balance out the hassles created by his aggressive stance >>> towards closing older issues (which, while unloved, are not >>> automatically invalid). >> How and how often was Mark reminded about this? >> >>> If this had happened *without* the prior discussion regarding more >>> appropriate handling of tracker issues, then I would have an issue >>> with it. However, given that the first reaction was to provide >>> additional mentoring, with revocation of privileges only happening >>> when the problems continued, that seems to me like the way this >>> process is *meant* to work. >> Where was the decision to revoke privileges discussed? Not on any >> mailing list that I am subscribed to. Was Mark given an ultimatum? >> >> Given that this came out rather unfortunately (even if the end result >> is the best that could have happened) I would recommend that in the >> future more attention is paid to "documenting" publicly that someone's >> being booted out was inevitable, by an exchange of messages on >> python-dev (or python-committers if we want to limit distribution). >> And no, I don't think that IRC (where I suspect this happened) is >> sufficient. >> >> -- >> --Guido van Rossum (python.org/~guido) >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/darren%40ontrenet.com >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From raymond.hettinger at gmail.com Wed Sep 22 17:21:08 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 22 Sep 2010 08:21:08 -0700 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> Message-ID: <0E6E4ABF-8361-4749-BAD2-B63E114ADAA7@gmail.com> On Sep 22, 2010, at 7:17 AM, Guido van Rossum wrote: > > Where was the decision to revoke privileges discussed? Not on any > mailing list that I am subscribed to. It was on IRC. > Was Mark given an ultimatum? I sent him a nicely worded email. The tracker privs were set back to the normal level which most participants have (can comment, submit patches, etc. but not close or reprioritize). That would have allowed him to contribute and participate. If that had worked out, it would have been easy to add back close/reprioritize capabilities No one needed to be hurt. Unfortunately, he responded with drama and another dev shut-off his tracker access entirely. Raymond From stephen at xemacs.org Wed Sep 22 17:29:23 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 23 Sep 2010 00:29:23 +0900 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> Message-ID: <87pqw53igc.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > I would recommend that in the future more attention is paid to > "documenting" publicly that someone's being booted out was > inevitable, by an exchange of messages on python-dev (or > python-committers if we want to limit distribution). And no, I > don't think that IRC (where I suspect this happened) is sufficient. +1 on explaining "what" and "why" where the committers can see it, and +1 on limiting distribution. The one time I lifted someone's privileges that's the way I did it (by luck, mostly). In hindsight, the fact that it was all done in plain sight of the committers made it easy for us to put the incident behind us. The fact that it was only visible to the committers made it easier mend the relationship later. From solipsis at pitrou.net Wed Sep 22 17:42:13 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 22 Sep 2010 17:42:13 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: <87pqw53igc.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20100922124700.000ac012@pitrou.net> <87pqw53igc.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20100922174213.0d5ff6ba@pitrou.net> On Thu, 23 Sep 2010 00:29:23 +0900 "Stephen J. Turnbull" wrote: > Guido van Rossum writes: > > > I would recommend that in the future more attention is paid to > > "documenting" publicly that someone's being booted out was > > inevitable, by an exchange of messages on python-dev (or > > python-committers if we want to limit distribution). And no, I > > don't think that IRC (where I suspect this happened) is sufficient. > > +1 on explaining "what" and "why" where the committers can see it, and > +1 on limiting distribution. > > The one time I lifted someone's privileges that's the way I did it (by > luck, mostly). In hindsight, the fact that it was all done in plain > sight of the committers made it easy for us to put the incident behind > us. The fact that it was only visible to the committers made it > easier mend the relationship later. I guess python-committers would have been the best recipient indeed. Sorry for that. Regards Antoine. From guido at python.org Wed Sep 22 17:44:12 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Sep 2010 08:44:12 -0700 Subject: [Python-Dev] Goodbye In-Reply-To: <87pqw53igc.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20100922124700.000ac012@pitrou.net> <87pqw53igc.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Wed, Sep 22, 2010 at 8:29 AM, Stephen J. Turnbull wrote: > Guido van Rossum writes: > > ?> I would recommend that in the future more attention is paid to > ?> "documenting" publicly that someone's being booted out was > ?> inevitable, by an exchange of messages on python-dev (or > ?> python-committers if we want to limit distribution). ?And no, I > ?> don't think that IRC (where I suspect this happened) is sufficient. > > +1 on explaining "what" and "why" where the committers can see it, and > +1 on limiting distribution. Agreed on both counts. > The one time I lifted someone's privileges that's the way I did it (by > luck, mostly). ?In hindsight, the fact that it was all done in plain > sight of the committers made it easy for us to put the incident behind > us. ?The fact that it was only visible to the committers made it > easier mend the relationship later. I understand the desire to keep dirty laundry in. I would like to keep it in too. Unfortunately the offending person in this case chose not to; I will not speculate about his motivation. This is not unusual; I can recall several incidents over the past few years (all completely different in every detail of course) where someone blew up publicly and there wasn't much of a chance to keep the incident under wraps. I see it as the risk of doing business in public -- which to me still beats the risk of doing business in back rooms many times over. -- --Guido van Rossum (python.org/~guido) From vinay_sajip at yahoo.co.uk Wed Sep 22 17:54:53 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 22 Sep 2010 15:54:53 +0000 (UTC) Subject: [Python-Dev] Some changes to logging for 3.2 Message-ID: Hi all, I'm planning to make some smallish changes to logging in Python 3.2, please see http://plumberjack.blogspot.com/2010/09/improved-queuehandler-queuelistener.html If you're interested, I'd be grateful for any feedback you can give. Regards, Vinay Sajip From martin at v.loewis.de Wed Sep 22 19:53:51 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 22 Sep 2010 19:53:51 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> Message-ID: <4C9A42AF.6030906@v.loewis.de> > Given that this came out rather unfortunately (even if the end result > is the best that could have happened) I would recommend that in the > future more attention is paid to "documenting" publicly that someone's > being booted out was inevitable, by an exchange of messages on > python-dev (or python-committers if we want to limit distribution). > And no, I don't think that IRC (where I suspect this happened) is > sufficient. While the community should be informed in public, I think that the actual revocation of privileges should happen in private. People getting such a "blue letter" often react overly negative, and then regret that their response is recorded in the public. The only downside of this approach is that people may feel that this is a unilateral decision by one committer, then appeal to you (I'm sure you recall cases that went that way :-). Also, I think it would be better to ask the contributor to step back "on his own" (acknowledging that this isn't really voluntarily), instead of revoking privileges and then informing about that decision. Regards, Martin From s_sourceforge at nedprod.com Wed Sep 22 22:12:59 2010 From: s_sourceforge at nedprod.com (Niall Douglas) Date: Wed, 22 Sep 2010 21:12:59 +0100 Subject: [Python-Dev] Advice sought on memory allocation latency reduction C1X standard proposal Message-ID: <4C9A634B.3927.174C9B7@s_sourceforge.nedprod.com> Dear Python Devs, I am hoping to gain feedback on an ISO C1X/C++ standard library proposal I hope to submit. It consists of a rationale (http://mallocv2.wordpress.com/) which shows how growth in RAM capacity is exponentially outgrowing the growth in RAM access speed. The consequences are profound: computer software which has always been written under the assumption of scarcity of RAM capacity will need to be retargeted to assume the scarcity of RAM access speed instead. The C1X proposal (http://mallocv2.wordpress.com/the-c-proposal-text/) enables four things of interest to Python: (i) aligned block resizing (ii) speculative in-place block resizing (iii) batch block allocation and (iv) the ability to reserve address space, thus avoiding the need to overallocate array storage. Aligned block resizing is especially useful to numpy. Where one has an array of aligned SSE vector quantities one cannot currently resize that block and guarantee that alignment will not be destroyed. With the new feature of non-relocating realloc() and being able to specify an alignment to realloc() one may avoid memory copying, and therefore reduce memory bandwidth utilisation and therefore overall memory access latencies. The ability to reserve address space and speculative in-place block resizing can be combined to allow Python to reserve an arbitrary amount of address space after the storage for an array object. Should the array then become extended, the speculative in-place block resizing can attempt to expand storage into that reserved space without having to relocate the contents of the storage. This again translates into much reduced memory copying as well as memory consumption, and once again reduces overall memory access latencies. Lastly, the batch allocation mechanism allows a sequence of allocations to be performed at once. I don't know of any attempts to have Python make use of similar functionality in Linux's system allocator, however Perl saw a 18% reduction in startup time (http://groups.google.com/group/perl-compiler/msg/31bca5297764002b). I am not familiar with Python's implementation outside working extensively with Boost.Python, so I was hoping that this list could advise me on what I might be forgetting, what problems there could be for Python with this design and/or any other general concerns and thoughts. I thank the list in advance for your time and consideration. Niall Douglas -- Technology & Consulting Services - ned Productions Limited. http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Company no: 472909. From cs at zip.com.au Wed Sep 22 22:59:21 2010 From: cs at zip.com.au (Cameron Simpson) Date: Thu, 23 Sep 2010 06:59:21 +1000 Subject: [Python-Dev] Goodbye In-Reply-To: References: Message-ID: <20100922205921.GA9692@cskk.homeip.net> Usual disclaimer: I am not a python-dev, just a lurker who sticks his 2c in sometimes... On 22Sep2010 07:17, Guido van Rossum wrote: | On Wed, Sep 22, 2010 at 4:07 AM, Nick Coghlan wrote: | > On Wed, Sep 22, 2010 at 8:47 PM, Antoine Pitrou wrote: | >> Simply, situations like the above (Mark closing a bug just because | >> nobody would answer his message on a short delay) have happened | >> multiple times - despite people opposing, obviously -, and we decided | >> that it was better to remove his tracker privileges since his | >> contribution has not really been productive for us. Which sounds like a genuine problem, but ... | >> There was a whole python-dev thread some time (weeks? months?) ago where | | I think it was the thread "No response to posts" started (by Mark) on July 31. | | >> several of us already tried to suggest more fruitful ways of | >> contributing, suggestions which weren't received very welcomingly AFAIR. | | Yup. In that thread (and others) I see lots of evidence where Mark | responded very negatively (from "I disagree entirely" to "I find this | response quite pathetic") when people explained how we treat the | tracker, and stuck to his guns no matter how many people tried to | explain that he should stop. | | His attitude can be summarized by his "Fly back at me if you like. I | don't care about me. I don't care about you. I do care about | Python." | | Which to me sounds defiant and passive-aggressive. I don't want to go | into analyzing, but I expect that Mark has issues that are beyond what | this community can deal with. I've just read that thread. Mark doesn't sound that way to me. "I disagree entirely" is an entirely valid response, when backed up with argument, such as his immediately following sentence: Perhaps we should simply agree to disagree, but I can't see how having issues that have been sitting for years without any response at all ties in with "the current set up works reasonably well" "I find this response quite pathetic" does seem an overreaction to a single point clarification-type post. The "Fly back at me if you like. I don't care about me. I don't care about you. I do care about Python." quote I actually think this quite laudable in its way; he's expressing a commitment to getting things done, and a determination to focus on the core issue (response workflow, from the sounds of it) in the face of the emotional responses the disagreement will inevitably produce in the discussions. Again, a follow on statement from that same post: Further, issues don't have to be closed, but what has to happen is that any issue that get raised *MUST* have a response. That's a concrete objection to the status quo for certain old bugs, and one that applies just as well to me own personal email practices (I have a bunch of unreplied-to threads in my supposedly "priority" inbox; at least these days a mark the things needing a real reply so i can find them later even if I often don't get back to them). I'm not disputing the decision to revoke his priviledges; if he really was closing a lot of bugs that most devs don't think should be closed and could not be dissuaded from doing so I can't see an alternative. I'm just saying I think the tone if his responses in the thread cited isn't as negative to my eye as people are making out. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ 'Soup: This is the one that Kawasaki sent out pictures, that looks so beautiful. Yanagawa: Yes, everybody says it's beautiful - but many problems! 'Soup: But you are not part of the design team, you're just a test rider. Yanagawa: Yes. I just complain. - _Akira Yanagawa Sounds Off_ @ www.amasuperbike.com From ncoghlan at gmail.com Thu Sep 23 00:00:01 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 23 Sep 2010 08:00:01 +1000 Subject: [Python-Dev] Some changes to logging for 3.2 In-Reply-To: References: Message-ID: On Thu, Sep 23, 2010 at 1:54 AM, Vinay Sajip wrote: > Hi all, > > I'm planning to make some smallish changes to logging in Python 3.2, please see > > http://plumberjack.blogspot.com/2010/09/improved-queuehandler-queuelistener.html > > If you're interested, I'd be grateful for any feedback you can give. Looks like a good idea to me - I have a (C++) logging system at work that pushes some of the I/O bound tasks out to a separate thread for similar reasons. To further reduce overhead, would it make sense for the signature of the QueueListener constructor to be (queue, *handlers)? (Providing the ability to add and remove handlers post-construction seems unnecessary, since you can add or remove new listeners to the original queue to get a similar effect without worrying about synchronisation of access to the list of handlers) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From breamoreboy at yahoo.co.uk Thu Sep 23 01:16:18 2010 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Thu, 23 Sep 2010 00:16:18 +0100 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <87pqw53igc.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 22/09/2010 16:44, Guido van Rossum wrote: > On Wed, Sep 22, 2010 at 8:29 AM, Stephen J. Turnbull wrote: >> Guido van Rossum writes: >> >> > I would recommend that in the future more attention is paid to >> > "documenting" publicly that someone's being booted out was >> > inevitable, by an exchange of messages on python-dev (or >> > python-committers if we want to limit distribution). And no, I >> > don't think that IRC (where I suspect this happened) is sufficient. >> >> +1 on explaining "what" and "why" where the committers can see it, and >> +1 on limiting distribution. > > Agreed on both counts. > >> The one time I lifted someone's privileges that's the way I did it (by >> luck, mostly). In hindsight, the fact that it was all done in plain >> sight of the committers made it easy for us to put the incident behind >> us. The fact that it was only visible to the committers made it >> easier mend the relationship later. > > I understand the desire to keep dirty laundry in. I would like to keep > it in too. Unfortunately the offending person in this case chose not > to; I will not speculate about his motivation. This is not unusual; I > can recall several incidents over the past few years (all completely > different in every detail of course) where someone blew up publicly > and there wasn't much of a chance to keep the incident under wraps. I > see it as the risk of doing business in public -- which to me still > beats the risk of doing business in back rooms many times over. > If you're referring to me I'm extremely offended. Yes or no? Kindest regards. Mark Lawrence. From tjreedy at udel.edu Thu Sep 23 01:18:35 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 22 Sep 2010 19:18:35 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: <20100922124700.000ac012@pitrou.net> References: <20100922124700.000ac012@pitrou.net> Message-ID: On 9/22/2010 6:47 AM, Antoine Pitrou wrote: > There was a whole python-dev thread some time (weeks? months?) ago where > several of us already tried to suggest more fruitful ways of > contributing, suggestions which weren't received very welcomingly AFAIR. There were two types of criticisms and suggestions, somewhat contradictory. The first type was something like 'Your tracker work is fine, but your pydev post berating others for not doing the same thing as you is not. Relax, do what you think is worthwhile, and let others decide what they will do." When I reiterated that in private email, he agreed with me and as far as I know he changed his behavior. The second type, which apparently included your response, began "Your tracker work is not all good. Change what your are doing ...". If the continuing disagreement and upset with his tracker (versus posting) behavior, to the point of possibly canceling admin privileges, had been brought up for instance on the committers list, I would have suggested to him that the low-hanging fruit had been picked (or discarded) and that a more careful approach was needed for the future. Would that have had any effect? I do not know. > Now I understand that opinions over this may vary and involve multiple > factors, but I would suggest that at least a bit of mentoring is needed > if we want to give privileges early on. > (and the amount of mentoring needed can vary wildly from one person to > another) Mentoring would be easier if there were clearer and more complete written guidelines. I was under the impression that this was being worked on. It would also be easier if it were clearer who is/are the deputed tracker authority/ies. Not everyone has the same idea about how to handle the various fields and processes. Who decides in cases of disagreement? -- Terry Jan Reedy From tim.peters at gmail.com Thu Sep 23 02:18:39 2010 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 22 Sep 2010 20:18:39 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <87pqw53igc.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Yikes - Mark has done terrific work in some very demanding areas, & I'd hate to see him feel unwelcome. So that's my advice: find a way to smooth this over. You're welcome ;-) [Guido] >> ... >> I understand the desire to keep dirty laundry in. I would like to keep >> it in too. Unfortunately the offending person in this case chose not >> to; I will not speculate about his motivation. This is not unusual; I >> can recall several incidents over the past few years (all completely >> different in every detail of course) where someone blew up publicly >> and there wasn't much of a chance to keep the incident under wraps. I >> see it as the risk of doing business in public -- which to me still >> beats the risk of doing business in back rooms many times over. [Mark Lawrence] > If you're referring to me I'm extremely offended. ?Yes or no? Have to confess I can't see what's offensive in what Guido wrote there. If you're inclined to feel offended, how about going back to Guido's: Which to me sounds defiant and passive-aggressive. I don't want to go into analyzing, but I expect that Mark has issues that are beyond what this community can deal with. Even I felt a bit offended by that one ;-) speaking-as-one-who-has-issues-no-community-can-deal-with-ly y'rs - tim From guido at python.org Thu Sep 23 02:31:18 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Sep 2010 17:31:18 -0700 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <87pqw53igc.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Wed, Sep 22, 2010 at 5:18 PM, Tim Peters wrote: > Yikes - Mark has done terrific work in some very demanding areas, & > I'd hate to see him feel unwelcome. ?So that's my advice: ?find a way > to smooth this over. ?You're welcome ;-) > > [Guido] >>> ... >>> I understand the desire to keep dirty laundry in. I would like to keep >>> it in too. Unfortunately the offending person in this case chose not >>> to; I will not speculate about his motivation. This is not unusual; I >>> can recall several incidents over the past few years (all completely >>> different in every detail of course) where someone blew up publicly >>> and there wasn't much of a chance to keep the incident under wraps. I >>> see it as the risk of doing business in public -- which to me still >>> beats the risk of doing business in back rooms many times over. > > [Mark Lawrence] >> If you're referring to me I'm extremely offended. ?Yes or no? > > Have to confess I can't see what's offensive in what Guido wrote > there. ?If you're inclined to feel offended, how about going back to > Guido's: > > ? ?Which to me sounds defiant and passive-aggressive. I don't > ? ?want to go into analyzing, but I expect that Mark has issues > ? ?that are beyond what this community can deal with. > > Even I felt a bit offended by that one ;-) That was not one of my finer moments, and I apologize. -- --Guido van Rossum (python.org/~guido) From rdmurray at bitdance.com Thu Sep 23 03:24:49 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 22 Sep 2010 21:24:49 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> Message-ID: <20100923012449.63B1022904D@kimball.webabinitio.net> On Wed, 22 Sep 2010 19:18:35 -0400, Terry Reedy wrote: > On 9/22/2010 6:47 AM, Antoine Pitrou wrote: > > Now I understand that opinions over this may vary and involve multiple > > factors, but I would suggest that at least a bit of mentoring is needed > > if we want to give privileges early on. > > (and the amount of mentoring needed can vary wildly from one person to > > another) > > Mentoring would be easier if there were clearer and more complete > written guidelines. I was under the impression that this was being > worked on. It would also be easier if it were clearer who is/are the Brett is planning to work on it. Because he is, I suspect everyone else is waiting for that, instead of working on it now. Just one of those things. (I believe his target is January-ish?) > deputed tracker authority/ies. Not everyone has the same idea about how > to handle the various fields and processes. Who decides in cases of > disagreement? We discussed this a while back and I don't think we really have a tracker BD. Brett and Martin come closest, but mostly we just sort of evolve a rough consensus. I think once Brett reduces that operating consensus to a written document things will be clearer. --David From steve at holdenweb.com Thu Sep 23 04:32:57 2010 From: steve at holdenweb.com (Steve Holden) Date: Wed, 22 Sep 2010 22:32:57 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <87pqw53igc.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4C9ABC59.7060703@holdenweb.com> On 9/22/2010 8:31 PM, Guido van Rossum wrote: [...] >> > >> > Which to me sounds defiant and passive-aggressive. I don't >> > want to go into analyzing, but I expect that Mark has issues >> > that are beyond what this community can deal with. >> > >> > Even I felt a bit offended by that one ;-) > That was not one of my finer moments, and I apologize. So even after losing his tracker privileges Mark is still managing to find ways to improve the Python community ;-) regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From steve at holdenweb.com Thu Sep 23 04:32:57 2010 From: steve at holdenweb.com (Steve Holden) Date: Wed, 22 Sep 2010 22:32:57 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <87pqw53igc.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4C9ABC59.7060703@holdenweb.com> On 9/22/2010 8:31 PM, Guido van Rossum wrote: [...] >> > >> > Which to me sounds defiant and passive-aggressive. I don't >> > want to go into analyzing, but I expect that Mark has issues >> > that are beyond what this community can deal with. >> > >> > Even I felt a bit offended by that one ;-) > That was not one of my finer moments, and I apologize. So even after losing his tracker privileges Mark is still managing to find ways to improve the Python community ;-) regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From brett at python.org Thu Sep 23 04:38:28 2010 From: brett at python.org (Brett Cannon) Date: Wed, 22 Sep 2010 19:38:28 -0700 Subject: [Python-Dev] Goodbye In-Reply-To: <20100923012449.63B1022904D@kimball.webabinitio.net> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> Message-ID: On Wed, Sep 22, 2010 at 18:24, R. David Murray wrote: > On Wed, 22 Sep 2010 19:18:35 -0400, Terry Reedy wrote: >> On 9/22/2010 6:47 AM, Antoine Pitrou wrote: >> ?> Now I understand that opinions over this may vary and involve multiple >> > factors, but I would suggest that at least a bit of mentoring is needed >> > if we want to give privileges early on. >> > (and the amount of mentoring needed can vary wildly from one person to >> > another) >> >> Mentoring would be easier if there were clearer and more complete >> written guidelines. I was under the impression that this was being >> worked on. It would also be easier if it were clearer who is/are the > > Brett is planning to work on it. ?Because he is, I suspect everyone else > is waiting for that, instead of working on it now. ?Just one of those > things. ?(I believe his target is January-ish?) Yep. I am planning on starting my two month PSF grant in January and the first thing on the agenda is a complete rewrite of the developer docs and moving them into the Doc/ directory (after that is managing code in Python 2/3 HOWTO and then after that most likely testing stuff, but maybe Python 3 stdlib fixes instead if that is deemed more important). -Brett > >> deputed tracker authority/ies. Not everyone has the same idea about how >> to handle the various fields and processes. Who decides in cases of >> disagreement? > > We discussed this a while back and I don't think we really have a tracker > BD. ?Brett and Martin come closest, but mostly we just sort of evolve > a rough consensus. ?I think once Brett reduces that operating consensus > to a written document things will be clearer. > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From steve at pearwood.info Thu Sep 23 04:35:13 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 23 Sep 2010 12:35:13 +1000 Subject: [Python-Dev] Goodbye In-Reply-To: References: Message-ID: <201009231235.13819.steve@pearwood.info> On Thu, 23 Sep 2010 10:18:39 am Tim Peters wrote: > Yikes - Mark has done terrific work in some very demanding areas, & > I'd hate to see him feel unwelcome. So that's my advice: find a > way to smooth this over. You're welcome ;-) I'd like to second that. Mark has been pushy, annoying and demanding, although he has his bad points too *grins*. Seriously, Mark has brought a remarkable amount of energy to the tracker, for good and bad. The good shouldn't give him a Get Out Of Jail Free card forever, but in the absence of any knowledge of what lead to Mark's tracker privileges being revoked, I have no objections in principle to giving him a second chance if the devs decide that is acceptable and Mark is willing. (Not that my objections carry much weight.) Either way, I would like to publicly thank Mark for his efforts and wish him the best for the future. [...] > Have to confess I can't see what's offensive in what Guido wrote > there. If you're inclined to feel offended, how about going back > to Guido's: > > Which to me sounds defiant and passive-aggressive. I don't > want to go into analyzing, but I expect that Mark has issues > that are beyond what this community can deal with. > > Even I felt a bit offended by that one ;-) I don't see why. Mark's emails often do sound defiant and passive-aggressive. Is that something we're supposed to ignore? And as for issues, Mark himself has explicitly and publicly mentioned them on this very list, and we *can't* deal with them. Nor should we be expected to. Mark has done good work, dealing with many issues in the tracker during a remarkably short time. But he's also managed to annoy and upset a number of people in a remarkably short time too. The long term health of Python depends more on the community than the existence of a few bugs more or less. -- Steven D'Aprano From raymond.hettinger at gmail.com Thu Sep 23 05:46:00 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 22 Sep 2010 20:46:00 -0700 Subject: [Python-Dev] Tracker BD Was:Goodbye In-Reply-To: <20100923012449.63B1022904D@kimball.webabinitio.net> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> Message-ID: On Sep 22, 2010, at 6:24 PM, R. David Murray wrote: > On Wed, 22 Sep 2010 19:18:35 -0400, Terry Reedy wrote: > >> deputed tracker authority/ies. Not everyone has the same idea about how >> to handle the various fields and processes. Who decides in cases of >> disagreement? > > We discussed this a while back and I don't think we really have a tracker > BD. Brett and Martin come closest, but mostly we just sort of evolve > a rough consensus. IMO, Benjamin and Antoine are the closest. They devote a substantial portion of their lives to Python and have been our most active contributors in the last year. They read almost every tracker post, read every check-in, and continuously monitor the IRC channel. Raymond From fdrake at acm.org Thu Sep 23 06:29:51 2010 From: fdrake at acm.org (Fred Drake) Date: Thu, 23 Sep 2010 00:29:51 -0400 Subject: [Python-Dev] Moving the developer docs? Message-ID: On Wed, Sep 22, 2010 at 10:38 PM, Brett Cannon wrote: > the first thing on the agenda is a complete rewrite of the developer > docs and moving them into the Doc/ directory I'd like to know why you think moving the developer docs into the CPython tree makes sense. My own thought here is that they're not specific to the version of Python, though some of the documentation deals with the group of specific branches being maintained. For me, keeping them in a separate space (like www.python.org/dev/) makes sense. ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From stephen at xemacs.org Thu Sep 23 06:32:07 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 23 Sep 2010 13:32:07 +0900 Subject: [Python-Dev] Goodbye In-Reply-To: <20100922205921.GA9692@cskk.homeip.net> References: <20100922205921.GA9692@cskk.homeip.net> Message-ID: <87mxr92i7s.fsf@uwakimon.sk.tsukuba.ac.jp> Cameron Simpson writes: > I've just read that thread. Mark doesn't sound that way to me. "I > disagree entirely" is an entirely valid response, when backed up > with argument, such as his immediately following sentence: > > Perhaps we should simply agree to disagree, Agreeing to disagree *on actions* does not work with shared resources, and the tracker is a shared resource. It's not a valid response here. According to reports, his disagreement *did* extend to action. > "I find this response quite pathetic" does seem an overreaction to > a single point clarification-type post. The "Fly back at me if you > like. I don't care about me. I don't care about you. I do care > about Python." quote I actually think this quite laudable in its > way; he's expressing a commitment to getting things done, and a > determination to focus on the core issue (response workflow, from > the sounds of it) in the face of the emotional responses the > disagreement will inevitably produce in the discussions. Expressing a commitment and emotional discussion are not problems worthy of even thinking about changing privileges. The problem is that (according to reports) he *imposed* his opinion on all the other tracker workers by making changes to the public database (ie, closing bugs), after being told by several people that the consensus was otherwise. And did this several times. While this was not clearly expressed in several of the key posts in this thread, I suspect that this is what was meant by "other ways are more fruitful" and Guido's now retracted psychoanalytic comment. There is a delicate balance to be kept between "he who does the work makes the decisions" and "polluting the common pool." In this case, the balance was clearly upset. Triaging and closing bug reports are not the only functions of the tracker, and in fact they are subsidiary to actual bug-fixing work. It's pretty clear to me that if a triager disagrees with the priorities of the bug fixers, his only recourse is public discussion, and to do what he personally can to respond to issues. From brett at python.org Thu Sep 23 06:45:32 2010 From: brett at python.org (Brett Cannon) Date: Wed, 22 Sep 2010 21:45:32 -0700 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: References: Message-ID: A discussion occurred (w/o me) on #python-dev where moving it to Doc/ would allow it to show up at docs.python.org to perhaps get more people involved. It also allows developers to contribute to the docs w/o having to get pydotorg commit rights. On Wed, Sep 22, 2010 at 21:29, Fred Drake wrote: > On Wed, Sep 22, 2010 at 10:38 PM, Brett Cannon wrote: >> the first thing on the agenda is a complete rewrite of the developer >> docs and moving them into the Doc/ directory > > I'd like to know why you think moving the developer docs into the > CPython tree makes sense. > > My own thought here is that they're not specific to the version of > Python, though some of the documentation deals with the group of > specific branches being maintained. ?For me, keeping them in a > separate space (like www.python.org/dev/) makes sense. > > > ? -Fred > > -- > Fred L. Drake, Jr.? ? > "A storm broke loose in my mind."? --Albert Einstein > From jackdied at gmail.com Thu Sep 23 07:32:23 2010 From: jackdied at gmail.com (Jack Diederich) Date: Thu, 23 Sep 2010 01:32:23 -0400 Subject: [Python-Dev] Tracker BD Was:Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> Message-ID: On Wed, Sep 22, 2010 at 11:46 PM, Raymond Hettinger wrote: > > On Sep 22, 2010, at 6:24 PM, R. David Murray wrote: > >> On Wed, 22 Sep 2010 19:18:35 -0400, Terry Reedy wrote: >> >>> deputed tracker authority/ies. Not everyone has the same idea about how >>> to handle the various fields and processes. Who decides in cases of >>> disagreement? >> >> We discussed this a while back and I don't think we really have a tracker >> BD. ?Brett and Martin come closest, but mostly we just sort of evolve >> a rough consensus. > > IMO, Benjamin and Antoine are the closest. ?They devote a substantial > portion of their lives to Python and have been our most active > contributors in the last year. ? They read almost every tracker post, > read every check-in, and continuously monitor the IRC channel. Off topic-er. Does anyone have scripts that pull data on how many committers commit or how many trac admins admin? I'm not asking for punitive reasons - I'd be the first against the wall - but I wouldn't mind graphing it. Power law, methinks. With big, confounding, and jumbley spikes in the Spring for PyCon. Likewise for mailing list subscriptions. Personally I've gone back and forth between subscribing to everything (-list -dev -commits -bugs -ideas, et al) and subscribing to almost nothing. -Jack From vinay_sajip at yahoo.co.uk Thu Sep 23 07:55:27 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 23 Sep 2010 05:55:27 +0000 (UTC) Subject: [Python-Dev] Some changes to logging for 3.2 References: Message-ID: Nick Coghlan gmail.com> writes: > To further reduce overhead, would it make sense for the signature of > the QueueListener constructor to be (queue, *handlers)? Good suggestion - thanks. Regards, Vinay From g.brandl at gmx.net Thu Sep 23 08:36:47 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 23 Sep 2010 08:36:47 +0200 Subject: [Python-Dev] Tracker BD Was:Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> Message-ID: Am 23.09.2010 07:32, schrieb Jack Diederich: > On Wed, Sep 22, 2010 at 11:46 PM, Raymond Hettinger > wrote: >> >> On Sep 22, 2010, at 6:24 PM, R. David Murray wrote: >> >>> On Wed, 22 Sep 2010 19:18:35 -0400, Terry Reedy wrote: >>> >>>> deputed tracker authority/ies. Not everyone has the same idea about how >>>> to handle the various fields and processes. Who decides in cases of >>>> disagreement? >>> >>> We discussed this a while back and I don't think we really have a tracker >>> BD. Brett and Martin come closest, but mostly we just sort of evolve >>> a rough consensus. >> >> IMO, Benjamin and Antoine are the closest. They devote a substantial >> portion of their lives to Python and have been our most active >> contributors in the last year. They read almost every tracker post, >> read every check-in, and continuously monitor the IRC channel. > > Off topic-er. Does anyone have scripts that pull data on how many > committers commit or how many trac admins admin? I'm not asking for > punitive reasons - I'd be the first against the wall - but I wouldn't > mind graphing it. Power law, methinks. With big, confounding, and > jumbley spikes in the Spring for PyCon. This should be easy to do with a hg repo such as the test conversion one on hg.python.org -- the "activity" extension already does the graphing, and I'm sure it can easily be tweaked to your liking. http://www.freehackers.org/thomas/2008/10/31/activity-extension-for-mercurial/ cheers, Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From g.brandl at gmx.net Thu Sep 23 08:40:21 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 23 Sep 2010 08:40:21 +0200 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: References: Message-ID: That's right. It is true that it isn't branch-specific information, and that does cause a little bit of irritation for me too, but neither is Misc/developers.txt or Misc/maintainers.rst. Of course, we might consider a separate HG repository (I'm all in favor of many small repos, instead of a gigantic sandbox one). The downside is that I really like the developer docs at docs.python.org, and it would complicate the build process a bit. Georg Am 23.09.2010 06:45, schrieb Brett Cannon: > A discussion occurred (w/o me) on #python-dev where moving it to Doc/ > would allow it to show up at docs.python.org to perhaps get more > people involved. It also allows developers to contribute to the docs > w/o having to get pydotorg commit rights. > > On Wed, Sep 22, 2010 at 21:29, Fred Drake wrote: >> On Wed, Sep 22, 2010 at 10:38 PM, Brett Cannon wrote: >>> the first thing on the agenda is a complete rewrite of the developer >>> docs and moving them into the Doc/ directory >> >> I'd like to know why you think moving the developer docs into the >> CPython tree makes sense. >> >> My own thought here is that they're not specific to the version of >> Python, though some of the documentation deals with the group of >> specific branches being maintained. For me, keeping them in a >> separate space (like www.python.org/dev/) makes sense. >> >> >> -Fred >> >> -- >> Fred L. Drake, Jr. >> "A storm broke loose in my mind." --Albert Einstein >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From g.brandl at gmx.net Thu Sep 23 09:04:30 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 23 Sep 2010 09:04:30 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: <201009231235.13819.steve@pearwood.info> References: <201009231235.13819.steve@pearwood.info> Message-ID: Am 23.09.2010 04:35, schrieb Steven D'Aprano: > On Thu, 23 Sep 2010 10:18:39 am Tim Peters wrote: >> Yikes - Mark has done terrific work in some very demanding areas, & >> I'd hate to see him feel unwelcome. So that's my advice: find a >> way to smooth this over. You're welcome ;-) > > I'd like to second that. Mark has been pushy, annoying and demanding, > although he has his bad points too *grins*. > > Seriously, Mark has brought a remarkable amount of energy to the > tracker, for good and bad. The good shouldn't give him a Get Out Of > Jail Free card forever, but in the absence of any knowledge of what > lead to Mark's tracker privileges being revoked, I have no objections > in principle to giving him a second chance if the devs decide that is > acceptable and Mark is willing. (Not that my objections carry much > weight.) I'd like to point out that only the Developer tracker privileges had been revoked at first, which meant that Mark could continue commenting and stirring up old issues, which is what made his work so valuable. He could not continue prematurely closing issues, which is what many devs objected to, and Raymond explained that to him in a private email (which I can only believe was polite and not at all offending). However, as a response to that, messages like this one started to be posted to several issues: http://bugs.python.org/msg117108 and since they were reposted even after being removed (unlinked) from the tracker item, the devs active on IRC at that moment could only suspect that this would continue for every issue that had Mark in its nosy list. That is a disruption that is not acceptable, and led to the removal of his User privilege as well, which I would say is wholly justified. Of course, this is *not* a ban! The privileges can be restored any time that Mark asks for it, but I would at least like to see an apology. > Either way, I would like to publicly thank Mark for his efforts and > wish him the best for the future. Same here. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From nad at acm.org Thu Sep 23 09:24:46 2010 From: nad at acm.org (Ned Deily) Date: Thu, 23 Sep 2010 00:24:46 -0700 Subject: [Python-Dev] Tracker BD Was:Goodbye References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> Message-ID: In article , Jack Diederich wrote: > Likewise for mailing list subscriptions. Personally I've gone back > and forth between subscribing to everything (-list -dev -commits -bugs > -ideas, et al) and subscribing to almost nothing. Mailing list subscription data could be very misleading as people follow this list (and other Python-related lists) through other interfaces, such as those provided by gmane.org (NNTP, RSS, web). Helps to keep the mailbox clutter down. -- Ned Deily, nad at acm.org From dirkjan at ochtman.nl Thu Sep 23 09:27:19 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Thu, 23 Sep 2010 09:27:19 +0200 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: References: Message-ID: On Thu, Sep 23, 2010 at 08:40, Georg Brandl wrote: > Of course, we might consider a separate HG repository (I'm all in favor > of many small repos, instead of a gigantic sandbox one). +1. Cheers, Dirkjan From martin at v.loewis.de Thu Sep 23 09:18:49 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 23 Sep 2010 09:18:49 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: <20100923012449.63B1022904D@kimball.webabinitio.net> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> Message-ID: <4C9AFF59.9020400@v.loewis.de> >> deputed tracker authority/ies. Not everyone has the same idea about how >> to handle the various fields and processes. Who decides in cases of >> disagreement? > > We discussed this a while back and I don't think we really have a tracker > BD. Brett and Martin come closest, but mostly we just sort of evolve > a rough consensus. I think once Brett reduces that operating consensus > to a written document things will be clearer. I personally think that the tracker fields and how they should be set is of minor importance. If there is a bug in Python, the most useful contribution is to submit a fix (or provide a rationale why this is not a bug). Asking every now and then "is this still an issue", or setting the version number, doesn't really advance the issue. Regards, Martin From g.brandl at gmx.net Thu Sep 23 10:46:24 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 23 Sep 2010 10:46:24 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: <4C9AFF59.9020400@v.loewis.de> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> Message-ID: Am 23.09.2010 09:18, schrieb "Martin v. L?wis": >>> deputed tracker authority/ies. Not everyone has the same idea about how >>> to handle the various fields and processes. Who decides in cases of >>> disagreement? >> >> We discussed this a while back and I don't think we really have a tracker >> BD. Brett and Martin come closest, but mostly we just sort of evolve >> a rough consensus. I think once Brett reduces that operating consensus >> to a written document things will be clearer. > > I personally think that the tracker fields and how they should be set is > of minor importance. If there is a bug in Python, the most useful > contribution is to submit a fix (or provide a rationale why this is not > a bug). Asking every now and then "is this still an issue", or setting > the version number, doesn't really advance the issue. It does however attract attention from developers who either weren't around when the original issue was submitted, or didn't feel competent enough to fix it then. It is also helpful to try reproducing the bug with a current version, in case the issue has been fixed already -- whether because of a duplicate bug report or by "chance". Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From baptiste13z at free.fr Thu Sep 23 10:56:22 2010 From: baptiste13z at free.fr (Baptiste Carvello) Date: Thu, 23 Sep 2010 10:56:22 +0200 Subject: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices) In-Reply-To: <87vd5y32lw.fsf@uwakimon.sk.tsukuba.ac.jp> References: <1284984738.3022.533.camel@thinko> <1285007435.3022.616.camel@thinko> <1285018753.2130.36.camel@thinko> <87vd5y32lw.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Stephen J. Turnbull a ?crit : > What really saves the day here is not that "common encodings just > don't do that". It's that even in the case where only syntactically > significant bytes in the representation are URL-encoded, they *are* > URL-encoded. As long as the parsing library restricts itself to > treating only wire-format input, you're OK.[1] But once you start > doing things that involve decoding URL-encoding, you can run into > trouble. > If I understand you well, any processing of unquoted bytes is dangerous per se. If this is true, then perhaps 'unquote' doesn't disserve the criticism it received in this thread for always returning str. This would be in fact quite fortunate, as it forces url processing to either happen on quoted bytes (before calling 'unqote'), or on unquoted str (on the result of 'unquote'), both of which are safe. From mail at timgolden.me.uk Thu Sep 23 11:00:26 2010 From: mail at timgolden.me.uk (Tim Golden) Date: Thu, 23 Sep 2010 10:00:26 +0100 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> Message-ID: <4C9B172A.2060504@timgolden.me.uk> On 23/09/2010 09:46, Georg Brandl wrote: > Am 23.09.2010 09:18, schrieb "Martin v. L?wis": >> I personally think that the tracker fields and how they should be set is >> of minor importance. If there is a bug in Python, the most useful >> contribution is to submit a fix (or provide a rationale why this is not >> a bug). Asking every now and then "is this still an issue", or setting >> the version number, doesn't really advance the issue. > > It does however attract attention from developers who either weren't around > when the original issue was submitted, or didn't feel competent enough to > fix it then. > > It is also helpful to try reproducing the bug with a current version, in > case the issue has been fixed already -- whether because of a duplicate > bug report or by "chance". I think my experience is that of several others. The work done by Mark and other tracker-trawlers has been useful: to dust off issues, attempt to assess their current validity, add suitable people to nosy lists, and where possible to try to reproduce or to encourage an OP to provide patches or other useful input. I've addressed, closed, or at least taken note of several issues in this way which I might not otherwise have done. The two less useful overspills of this generally useful activity have been: simple noise of the "Is anyone interested in this?" variety -- although even that might be useful, as Georg says, in highlighting older issues to newer developers; and the over-eager closure of calls on the basis of lack of response, and it seems to be an excess of this last which has brought the matter to a head. Let me ask a question which I don't think has been asked in this thread: are there guidelines for tracker-trawlers? I'm never sure where to look for this kind of thing myself. If there's nothing, I'm happy to pen a dos-and-donts (which I might do anyway, simply as a blog entry...) TJG From wiggin15 at gmail.com Thu Sep 23 11:31:31 2010 From: wiggin15 at gmail.com (Ender Wiggin) Date: Thu, 23 Sep 2010 11:31:31 +0200 Subject: [Python-Dev] Problems with hex-conversion functions In-Reply-To: <4AA331A9.70209@gmail.com> References: <8449046c0909051426g2eaa3f94s4c5842c25af17abf@mail.gmail.com> <4AA331A9.70209@gmail.com> Message-ID: Sorry for the late reply. I would really like to see this fixed. >> Or we [...] deprecate binascii.(un)hexlify(). ... >> binascii is the legacy approach here, so if anything was to go, those functions would be it ... I'm not entirely convinced binascii is the legacy approach. What makes this module "legacy"? On the contrary, I'm pretty sure modularity is better than sticking all the functionality in the core. As was written in this issue: http://psf.upfronthosting.co.za/roundup/tracker/issue3532 "If you wanted to produce base-85 (say), then you can extend the functionality of bytes by providing a function that does that, whereas you can't extend the existing bytes type." This example shows that "hex" is actually getting a special treatment by having builtin methods associated with the bytes type. Why don't we add ".base64" methods? Or even ".zlib"? After all, these options were present in Python 2.x using the "encode" method of string. In my opinion, having modules to deal with these types of conversions is better, and this is why I suggested sticking to binascii. In any case, seeing as both this discussion and the one linked above were abandoned, I would like to hear about what needs to be done to actually fix these issues. If no one else is willing to do it (that would be a little disappoiting), I think I have the skills to learn and fix the code itself, but I don't have the time and I am unfamiliar with the process of submitting patches and getting them approved. For example, who gets to decide about the correct approach? Is there a better place to discuss this? Thanks for the responses. -- Arnon On Sun, Sep 6, 2009 at 5:51 AM, Nick Coghlan wrote: > Brett Cannon wrote: > >> To fix these issues, three changes should be applied: > >> 1. Deprecate bytes.fromhex. This fixes the following problems: > >> #4 (go with option B and remove the function that does not allow > bytes > >> input) > >> #2 (the binascii functions will be the only way to "do it") > >> #1 (bytes.hex should not be implemented) > >> 2. In order to keep the functionality that bytes.fromhex has over > unhexlify, > >> the latter function should be able to handle spaces in its input (fix > #3) > >> 3. binascii.hexlify should return string as its return type (fix #5) > > > > Or we fix bytes.fromhex(), add bytes.hex() and deprecate > binascii.(un)hexlify(). > > binascii is the legacy approach here, so if anything was to go, those > functions would be it. I'm not sure getting rid of them is worth the > hassle though (especially in 2.x). > > Regarding bytes.hex(), it may be better to modify the builtin hex() > function to accept bytes as an input type. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > --------------------------------------------------------------- > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Thu Sep 23 11:38:17 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 23 Sep 2010 11:38:17 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: <4C9B172A.2060504@timgolden.me.uk> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9B172A.2060504@timgolden.me.uk> Message-ID: <4C9B2009.4080803@v.loewis.de> > Let me ask a question which I don't think has been asked in this > thread: are there guidelines for tracker-trawlers? I'm never sure > where to look for this kind of thing myself. If there's nothing, > I'm happy to pen a dos-and-donts (which I might do anyway, simply > as a blog entry...) Can you please rephrase the question? What's a "tracker-trawler"? Regards, Martin From mail at timgolden.me.uk Thu Sep 23 11:43:20 2010 From: mail at timgolden.me.uk (Tim Golden) Date: Thu, 23 Sep 2010 10:43:20 +0100 Subject: [Python-Dev] Goodbye In-Reply-To: <4C9B2009.4080803@v.loewis.de> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9B172A.2060504@timgolden.me.uk> <4C9B2009.4080803@v.loewis.de> Message-ID: <4C9B2138.8060305@timgolden.me.uk> On 23/09/2010 10:38, "Martin v. L?wis" wrote: >> Let me ask a question which I don't think has been asked in this >> thread: are there guidelines for tracker-trawlers? I'm never sure >> where to look for this kind of thing myself. If there's nothing, >> I'm happy to pen a dos-and-donts (which I might do anyway, simply >> as a blog entry...) > > Can you please rephrase the question? What's a "tracker-trawler"? My invented terminology for someone -- like Mark -- who invests time in going through issues in the tracker with a view to assessing them, prioritising them, de-duplicating, etc. As opposed to someone who's looking through issues with a view to finding things to fix within a particular area of competence. TJG From solipsis at pitrou.net Thu Sep 23 11:59:28 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 23 Sep 2010 11:59:28 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: <87mxr92i7s.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20100922205921.GA9692@cskk.homeip.net> <87mxr92i7s.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20100923115928.1f988cc7@pitrou.net> On Thu, 23 Sep 2010 13:32:07 +0900 "Stephen J. Turnbull" wrote: > > Triaging and closing bug reports are > not the only functions of the tracker, and in fact they are subsidiary > to actual bug-fixing work. +1. What we really need is people analyzing issues, posting patches or reviewing existing ones. Pushing for resolution of issues by threatening to close them isn't effective; it doesn't magically create more free time for us to deal with them. Regards Antoine. From solipsis at pitrou.net Thu Sep 23 12:07:57 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 23 Sep 2010 12:07:57 +0200 Subject: [Python-Dev] Goodbye References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> Message-ID: <20100923120757.53611829@pitrou.net> On Wed, 22 Sep 2010 21:24:49 -0400 "R. David Murray" wrote: > > > deputed tracker authority/ies. Not everyone has the same idea about how > > to handle the various fields and processes. Who decides in cases of > > disagreement? > > We discussed this a while back and I don't think we really have a tracker > BD. Brett and Martin come closest, but mostly we just sort of evolve > a rough consensus. If BD means Benevolent Dictator, then I certainly hope we don't need a tracker BD. I think the best until some written guideline exists is to watch how developers use the tracker today. There isn't much of a workflow apart from the initial version and priority setting. Everything is pretty much informal although we usually tend to follow the same steps. Regards Antoine. From fdrake at acm.org Thu Sep 23 12:12:17 2010 From: fdrake at acm.org (Fred Drake) Date: Thu, 23 Sep 2010 06:12:17 -0400 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: References: Message-ID: On Thu, Sep 23, 2010 at 2:40 AM, Georg Brandl wrote: > That's right. ?It is true that it isn't branch-specific information, > and that does cause a little bit of irritation for me too, but neither > is Misc/developers.txt or Misc/maintainers.rst. Agreed. I'd rather those were elsewhere as well, but I was paying less attention at the time. > Of course, we might consider a separate HG repository (I'm all in favor > of many small repos, instead of a gigantic sandbox one). ?The downside > is that I really like the developer docs at docs.python.org, and it > would complicate the build process a bit. Perhaps someone here knows enough about our documentation toolchain to figure out a way to generate a link from the docs to developer docs on the website. :-) I expect only a very small part of the audience for the general Python documentation & CPython docs are particularly interested in the development process we use, and sending them to the website from a convenient link is not a bad thing. We won't even need a new repository to do that. ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From solipsis at pitrou.net Thu Sep 23 12:11:44 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 23 Sep 2010 12:11:44 +0200 Subject: [Python-Dev] Moving the developer docs? References: Message-ID: <20100923121144.2006aeaa@pitrou.net> On Thu, 23 Sep 2010 00:29:51 -0400 Fred Drake wrote: > On Wed, Sep 22, 2010 at 10:38 PM, Brett Cannon wrote: > > the first thing on the agenda is a complete rewrite of the developer > > docs and moving them into the Doc/ directory > > I'd like to know why you think moving the developer docs into the > CPython tree makes sense. > > My own thought here is that they're not specific to the version of > Python, though some of the documentation deals with the group of > specific branches being maintained. Many parts of the library docs aren't version-specific either :) The dev docs may differ slightly from one version to another, for example if a version introduces some new possibilities for tooling, or far-reaching implementation changes (think Unladen Swallow). The practicality argument of being able to edit those docs without having to master a separate (pydotorg) workflow sounds quite strong to me. Regards Antoine. From martin at v.loewis.de Thu Sep 23 12:30:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 23 Sep 2010 12:30:07 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: <4C9B2138.8060305@timgolden.me.uk> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9B172A.2060504@timgolden.me.uk> <4C9B2009.4080803@v.loewis.de> <4C9B2138.8060305@timgolden.me.uk> Message-ID: <4C9B2C2F.60502@v.loewis.de> Am 23.09.2010 11:43, schrieb Tim Golden: > On 23/09/2010 10:38, "Martin v. L?wis" wrote: >>> Let me ask a question which I don't think has been asked in this >>> thread: are there guidelines for tracker-trawlers? I'm never sure >>> where to look for this kind of thing myself. If there's nothing, >>> I'm happy to pen a dos-and-donts (which I might do anyway, simply >>> as a blog entry...) >> >> Can you please rephrase the question? What's a "tracker-trawler"? > > My invented terminology for someone -- like Mark -- who invests time > in going through issues in the tracker with a view to assessing them, > prioritising them, de-duplicating, etc. As opposed to someone who's > looking through issues with a view to finding things to fix within > a particular area of competence. Ah. I think this goes to the core of the dispute: My recommendation is not to trawl at all. Instead, if you *really* want to contribute to Python, pick some area that you think needs most attention, and go through the tracker, and acquire competence in that area. The question is how much time you want to spend per issue. If it's only a few minutes per issue, I question whether this is a useful activity. If the issue has been long-standing, most likely, a few minutes will not be enough. There may, occasionally, be an issue that has been forgotten about, but overall, I'd expect that that the amount of wasted time becomes considerable - you can spend hours and hours looking through issues just to find out that they are all really tricky and would require a lot of expertise to resolve, which you then are not willing to acquire. Also, for me, as somebody on the nosy list, this activity doesn't help: *I* would have to spend much more time than I have at hands. So any "is this still valid?" message gets deleted immediately, especially if there are ten of them in my inbox. Regards, Martin From fuzzyman at voidspace.org.uk Thu Sep 23 13:44:44 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 23 Sep 2010 12:44:44 +0100 Subject: [Python-Dev] Goodbye In-Reply-To: <20100923115928.1f988cc7@pitrou.net> References: <20100922205921.GA9692@cskk.homeip.net> <87mxr92i7s.fsf@uwakimon.sk.tsukuba.ac.jp> <20100923115928.1f988cc7@pitrou.net> Message-ID: <4C9B3DAC.9000404@voidspace.org.uk> On 23/09/2010 10:59, Antoine Pitrou wrote: > On Thu, 23 Sep 2010 13:32:07 +0900 > "Stephen J. Turnbull" wrote: >> Triaging and closing bug reports are >> not the only functions of the tracker, and in fact they are subsidiary >> to actual bug-fixing work. > +1. What we really need is people analyzing issues, posting patches or > reviewing existing ones. Pushing for resolution of issues by > threatening to close them isn't effective; it doesn't magically > create more free time for us to deal with them. Well, useful things that can still be done in triaging on old issues: checking that a bug still exists checking that a patch still applies and that tests still pass pointing out if a patch lacks tests seeing if the issue is duplicated elsewhere If it is a *feature request* (as opposed to a bug report - and I appreciate that the difference is often unclear) and the original poster has 'lost interest' then closing the issue may be a reasonable response. As others have mentioned, bumping old issues can bring it to the attention of developers monitoring on IRC or the bug mailing list who may not have seen the original report. On all issues, new or old, additional potentially useful things to do: make sure the issue is assigned to the right person if appropriate add relevant keywords to make it easier to find in searches ensure other fields in the tracker are filled in correctly closing the issue if it is invalid or expired (for example it only applies to an out of maintenance version of Python) Some developers (Martin and Antoine at least) have complained about the 'noise' of receiving extra emails on issues they are nosy on. Several developers have stated that they have found it useful (myself, Guido and Tim Golden at least) - so there is certainly no consensus that this work is pointless. All the best, Michael Foord > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From fuzzyman at voidspace.org.uk Thu Sep 23 13:45:04 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 23 Sep 2010 12:45:04 +0100 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923121144.2006aeaa@pitrou.net> References: <20100923121144.2006aeaa@pitrou.net> Message-ID: <4C9B3DC0.1070100@voidspace.org.uk> On 23/09/2010 11:11, Antoine Pitrou wrote: > On Thu, 23 Sep 2010 00:29:51 -0400 > Fred Drake wrote: > >> On Wed, Sep 22, 2010 at 10:38 PM, Brett Cannon wrote: >>> the first thing on the agenda is a complete rewrite of the developer >>> docs and moving them into the Doc/ directory >> I'd like to know why you think moving the developer docs into the >> CPython tree makes sense. >> >> My own thought here is that they're not specific to the version of >> Python, though some of the documentation deals with the group of >> specific branches being maintained. > Many parts of the library docs aren't version-specific either :) > The dev docs may differ slightly from one version to another, for > example if a version introduces some new possibilities for tooling, or > far-reaching implementation changes (think Unladen Swallow). > > The practicality argument of being able to edit those docs without > having to master a separate (pydotorg) workflow sounds quite strong to > me. +1 Michael > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ketan.vijayvargiya at gmail.com Thu Sep 23 13:56:21 2010 From: ketan.vijayvargiya at gmail.com (Ketan Vijayvargiya) Date: Thu, 23 Sep 2010 17:26:21 +0530 Subject: [Python-Dev] (no subject) In-Reply-To: References: Message-ID: Hi. I have an issue which has been annoying me from quite sometime and any help would be greatly appreciated: I just installed Python 2.6 on my centOS 5 system and then I installed nltk. Now I am running a certain python script and it gives me this error- ImportError: No module named _sqlite3 Searching the internet tells me that sqlite should be installed on my system which is confirmed when I try to install it using yum. Yum tells me that all of the following are installed on my system- sqlite-devel.i386 sqlite.i386 python-sqlite.i386 Can anyone tell me where I am going wrong. -- Regards, Ketan Vijayvargiya -------------- next part -------------- An HTML attachment was scrubbed... URL: From lvh at laurensvh.be Thu Sep 23 14:02:00 2010 From: lvh at laurensvh.be (Laurens Van Houtven) Date: Thu, 23 Sep 2010 14:02:00 +0200 Subject: [Python-Dev] (no subject) In-Reply-To: References: Message-ID: Hi! This mailing list is about developing Python itself, not about developing *in* Python or debugging Python installations. Try seeing help elsewhere, such as on the comp.lang.python newsgroup, #python IRC channel on freenode, or other sources (check the wiki if you need any others). Thanks, lvh -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Thu Sep 23 14:11:18 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 23 Sep 2010 14:11:18 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: <4C9B3DAC.9000404@voidspace.org.uk> References: <20100922205921.GA9692@cskk.homeip.net> <87mxr92i7s.fsf@uwakimon.sk.tsukuba.ac.jp> <20100923115928.1f988cc7@pitrou.net> <4C9B3DAC.9000404@voidspace.org.uk> Message-ID: <4C9B43E6.8080308@v.loewis.de> > make sure the issue is assigned to the right person if appropriate -1. We typically don't assign issues to others. > add relevant keywords to make it easier to find in searches -0. Going through old issues just to make sure the keywords are right seems wasteful. > ensure other fields in the tracker are filled in correctly Likewise. Regards, Martin From fuzzyman at voidspace.org.uk Thu Sep 23 14:13:37 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 23 Sep 2010 13:13:37 +0100 Subject: [Python-Dev] Goodbye In-Reply-To: <4C9B43E6.8080308@v.loewis.de> References: <20100922205921.GA9692@cskk.homeip.net> <87mxr92i7s.fsf@uwakimon.sk.tsukuba.ac.jp> <20100923115928.1f988cc7@pitrou.net> <4C9B3DAC.9000404@voidspace.org.uk> <4C9B43E6.8080308@v.loewis.de> Message-ID: <4C9B4471.8060100@voidspace.org.uk> On 23/09/2010 13:11, "Martin v. L?wis" wrote: >> make sure the issue is assigned to the right person if appropriate > -1. We typically don't assign issues to others. Some people have requested to be assigned to issues. (Myself for unittest, Ronald for Mac OS X issues etc.) >> add relevant keywords to make it easier to find in searches > -0. Going through old issues just to make sure the keywords are right > seems wasteful. > Hard as it may be to believe, we do have (and are trying to cultivate) people who want to contribute to Python and start by searching for issues at the bug tracker. All the best, Michael >> ensure other fields in the tracker are filled in correctly > Likewise. > > Regards, > Martin -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From jnoller at gmail.com Thu Sep 23 14:21:36 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 23 Sep 2010 08:21:36 -0400 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923121144.2006aeaa@pitrou.net> References: <20100923121144.2006aeaa@pitrou.net> Message-ID: On Thu, Sep 23, 2010 at 6:11 AM, Antoine Pitrou wrote: > On Thu, 23 Sep 2010 00:29:51 -0400 > Fred Drake wrote: > >> On Wed, Sep 22, 2010 at 10:38 PM, Brett Cannon wrote: >> > the first thing on the agenda is a complete rewrite of the developer >> > docs and moving them into the Doc/ directory >> >> I'd like to know why you think moving the developer docs into the >> CPython tree makes sense. >> >> My own thought here is that they're not specific to the version of >> Python, though some of the documentation deals with the group of >> specific branches being maintained. > > Many parts of the library docs aren't version-specific either :) > The dev docs may differ slightly from one version to another, for > example if a version introduces some new possibilities for tooling, or > far-reaching implementation changes (think Unladen Swallow). > > The practicality argument of being able to edit those docs without > having to master a separate (pydotorg) workflow sounds quite strong to > me. Agreed with Antoine here the additional workflow/repo/build process/etc sucks Besides - who cares if only a subset of users would be interested in our workflow? If it's more than 0, and it helps bring on new contributors, who cares? If we can make it easier to maintain information, and find that information, why not do it? jesse From phd at phd.pp.ru Thu Sep 23 14:05:44 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Thu, 23 Sep 2010 16:05:44 +0400 Subject: [Python-Dev] (no subject) In-Reply-To: References: Message-ID: <20100923120544.GA32098@phd.pp.ru> Hello. We are sorry but we cannot help you. This mailing list is to work on developing Python (adding new features to Python itself and fixing bugs); if you're having problems learning, understanding or using Python, please find another forum. Probably python-list/comp.lang.python mailing list/news group is the best place; there are Python developers who participate in it; you may get a faster, and probably more complete, answer there. See http://www.python.org/community/ for other lists/news groups/fora. Thank you for understanding. On Thu, Sep 23, 2010 at 05:26:21PM +0530, Ketan Vijayvargiya wrote: > Hi. > I have an issue which has been annoying me from quite sometime and any help > would be greatly appreciated: > > I just installed Python 2.6 on my centOS 5 system and then I installed nltk. > Now I am running a certain python script and it gives me this error- > ImportError: No module named _sqlite3 > Searching the internet tells me that sqlite should be installed on my system > which is confirmed when I try to install it using yum. Yum tells me that all > of the following are installed on my system- > sqlite-devel.i386 > sqlite.i386 > python-sqlite.i386 > Can anyone tell me where I am going wrong. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From martin at v.loewis.de Thu Sep 23 14:28:22 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 23 Sep 2010 14:28:22 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: <4C9B4471.8060100@voidspace.org.uk> References: <20100922205921.GA9692@cskk.homeip.net> <87mxr92i7s.fsf@uwakimon.sk.tsukuba.ac.jp> <20100923115928.1f988cc7@pitrou.net> <4C9B3DAC.9000404@voidspace.org.uk> <4C9B43E6.8080308@v.loewis.de> <4C9B4471.8060100@voidspace.org.uk> Message-ID: <4C9B47E6.70601@v.loewis.de> >>> add relevant keywords to make it easier to find in searches >> -0. Going through old issues just to make sure the keywords are right >> seems wasteful. >> > > Hard as it may be to believe, we do have (and are trying to cultivate) > people who want to contribute to Python and start by searching for > issues at the bug tracker. Sure. However, on any specific search, many issues come up already. So people searching for stuff to do will easily find tasks already. Regards, Martin From mail at timgolden.me.uk Thu Sep 23 14:39:10 2010 From: mail at timgolden.me.uk (Tim Golden) Date: Thu, 23 Sep 2010 13:39:10 +0100 Subject: [Python-Dev] Goodbye In-Reply-To: <4C9B47E6.70601@v.loewis.de> References: <20100922205921.GA9692@cskk.homeip.net> <87mxr92i7s.fsf@uwakimon.sk.tsukuba.ac.jp> <20100923115928.1f988cc7@pitrou.net> <4C9B3DAC.9000404@voidspace.org.uk> <4C9B43E6.8080308@v.loewis.de> <4C9B4471.8060100@voidspace.org.uk> <4C9B47E6.70601@v.loewis.de> Message-ID: <4C9B4A6E.5070309@timgolden.me.uk> On 23/09/2010 13:28, "Martin v. L?wis" wrote: >>>> add relevant keywords to make it easier to find in searches >>> -0. Going through old issues just to make sure the keywords are right >>> seems wasteful. >>> >> >> Hard as it may be to believe, we do have (and are trying to cultivate) >> people who want to contribute to Python and start by searching for >> issues at the bug tracker. > > Sure. However, on any specific search, many issues come up already. > So people searching for stuff to do will easily find tasks already. Seems to me that the distinction to be made here is between activity which might, to some, appear a waste of time (eg setting flags and versions on existing calls) but which at worst provides no benefit and in fact might help someone narrow down a search more easily; and activity which is simply flatus vocis and which actually distracts or irritates. Individuals' thresholds clearly vary. TJG From ncoghlan at gmail.com Thu Sep 23 15:35:02 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 23 Sep 2010 23:35:02 +1000 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923121144.2006aeaa@pitrou.net> References: <20100923121144.2006aeaa@pitrou.net> Message-ID: On Thu, Sep 23, 2010 at 8:11 PM, Antoine Pitrou wrote: > The practicality argument of being able to edit those docs without > having to master a separate (pydotorg) workflow sounds quite strong to > me. This is the key point for me. For developer controlled stuff, the easiest place to have it if we want it kept up to date is in the source tree. Second best is the wiki. Having it off in a separately managed repository (that exists for perfectly valid reasons, since a lot of the content *isn't* developer controlled) is annoying. That said, in this case, what's the advantage of the source tree over the wiki? To include it in the main docs, so people reading them offline can still see the contribution workflow? Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From orzel at freehackers.org Thu Sep 23 15:34:01 2010 From: orzel at freehackers.org (Thomas Capricelli) Date: Thu, 23 Sep 2010 15:34:01 +0200 Subject: [Python-Dev] Tracker BD Was:Goodbye In-Reply-To: References: Message-ID: <201009231534.02555.orzel@freehackers.org> On Thursday 23 September 2010 08:36:47 Georg Brandl wrote: > This should be easy to do with a hg repo such as the test conversion one > on hg.python.org -- the "activity" extension already does the graphing, > and I'm sure it can easily be tweaked to your liking. > > http://www.freehackers.org/thomas/2008/10/31/activity-extension-for-mercurial/ Hi all, This blog entry is old, and I have improved 'hg activity' since then, for example the extension can now display several graphs, splitting the activity by authors, branches, files, ... And, of course, I would be very happy to help 'tweaking' the script for the python community. best wishes, Thomas ps: i'm not subscribed to python-dev. -- Thomas Capricelli http://www.freehackers.org/thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Thu Sep 23 16:01:57 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 23 Sep 2010 10:01:57 -0400 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: References: Message-ID: <20100923100157.288a72a8@mission> On Sep 23, 2010, at 08:40 AM, Georg Brandl wrote: >That's right. It is true that it isn't branch-specific information, >and that does cause a little bit of irritation for me too, but neither >is Misc/developers.txt or Misc/maintainers.rst. > >Of course, we might consider a separate HG repository (I'm all in favor >of many small repos, instead of a gigantic sandbox one). The downside >is that I really like the developer docs at docs.python.org, and it >would complicate the build process a bit. Ideally, I would really like to see the developer docs live outside the CPython source repository. There's no reason to tie the dev docs to CPython's svn merge policies, write acls, or release schedules. Given the way docs.python.org is stitched together, and the fact that we (still ;) haven't moved to a dvcs, this may not be feasible. These docs are better off in the wiki than in the source tree. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From benjamin at python.org Thu Sep 23 16:06:05 2010 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 23 Sep 2010 09:06:05 -0500 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923100157.288a72a8@mission> References: <20100923100157.288a72a8@mission> Message-ID: 2010/9/23 Barry Warsaw : > On Sep 23, 2010, at 08:40 AM, Georg Brandl wrote: > >>That's right. ?It is true that it isn't branch-specific information, >>and that does cause a little bit of irritation for me too, but neither >>is Misc/developers.txt or Misc/maintainers.rst. >> >>Of course, we might consider a separate HG repository (I'm all in favor >>of many small repos, instead of a gigantic sandbox one). ?The downside >>is that I really like the developer docs at docs.python.org, and it >>would complicate the build process a bit. > > Ideally, I would really like to see the developer docs live outside the > CPython source repository. ?There's no reason to tie the dev docs to CPython's > svn merge policies, write acls, or release schedules. ?Given the way > docs.python.org is stitched together, and the fact that we (still ;) haven't > moved to a dvcs, this may not be feasible. Are any of our docs subject to release schedules? > > These docs are better off in the wiki than in the source tree. -- Regards, Benjamin From jnoller at gmail.com Thu Sep 23 16:06:31 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 23 Sep 2010 10:06:31 -0400 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923100157.288a72a8@mission> References: <20100923100157.288a72a8@mission> Message-ID: On Thu, Sep 23, 2010 at 10:01 AM, Barry Warsaw wrote: > On Sep 23, 2010, at 08:40 AM, Georg Brandl wrote: > >>That's right. ?It is true that it isn't branch-specific information, >>and that does cause a little bit of irritation for me too, but neither >>is Misc/developers.txt or Misc/maintainers.rst. >> >>Of course, we might consider a separate HG repository (I'm all in favor >>of many small repos, instead of a gigantic sandbox one). ?The downside >>is that I really like the developer docs at docs.python.org, and it >>would complicate the build process a bit. > > Ideally, I would really like to see the developer docs live outside the > CPython source repository. ?There's no reason to tie the dev docs to CPython's > svn merge policies, write acls, or release schedules. ?Given the way > docs.python.org is stitched together, and the fact that we (still ;) haven't > moved to a dvcs, this may not be feasible. > > These docs are better off in the wiki than in the source tree. > > -Barry -1 on wiki; wikis are where good information goes off to die. From rdmurray at bitdance.com Thu Sep 23 16:16:01 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 23 Sep 2010 10:16:01 -0400 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: References: <20100923121144.2006aeaa@pitrou.net> Message-ID: <20100923141601.D148D1FB3B2@kimball.webabinitio.net> On Thu, 23 Sep 2010 23:35:02 +1000, Nick Coghlan wrote: > On Thu, Sep 23, 2010 at 8:11 PM, Antoine Pitrou wrote: > > The practicality argument of being able to edit those docs without > > having to master a separate (pydotorg) workflow sounds quite strong to > > me. > > This is the key point for me. For developer controlled stuff, the > easiest place to have it if we want it kept up to date is in the > source tree. Second best is the wiki. Having it off in a separately > managed repository (that exists for perfectly valid reasons, since a > lot of the content *isn't* developer controlled) is annoying. > > That said, in this case, what's the advantage of the source tree over > the wiki? To include it in the main docs, so people reading them > offline can still see the contribution workflow? I'd *much* rather edit rst files than futz with a web interface when editing docs. The wiki also somehow feels "less official". I do think exposing our development process to the wider user community by including them in the main docs could foster additional community involvement, but I don't have a strong opinion on that aspect of it. For me the change is about making it easier for the dev community (who are using/creating the development infrastructure) to update the relevant documentation. -- R. David Murray www.bitdance.com From fuzzyman at voidspace.org.uk Thu Sep 23 16:18:42 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 23 Sep 2010 15:18:42 +0100 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923141601.D148D1FB3B2@kimball.webabinitio.net> References: <20100923121144.2006aeaa@pitrou.net> <20100923141601.D148D1FB3B2@kimball.webabinitio.net> Message-ID: <4C9B61C2.8030204@voidspace.org.uk> On 23/09/2010 15:16, R. David Murray wrote: > On Thu, 23 Sep 2010 23:35:02 +1000, Nick Coghlan wrote: >> On Thu, Sep 23, 2010 at 8:11 PM, Antoine Pitrou wrote: >>> The practicality argument of being able to edit those docs without >>> having to master a separate (pydotorg) workflow sounds quite strong to >>> me. >> This is the key point for me. For developer controlled stuff, the >> easiest place to have it if we want it kept up to date is in the >> source tree. Second best is the wiki. Having it off in a separately >> managed repository (that exists for perfectly valid reasons, since a >> lot of the content *isn't* developer controlled) is annoying. >> >> That said, in this case, what's the advantage of the source tree over >> the wiki? To include it in the main docs, so people reading them >> offline can still see the contribution workflow? > I'd *much* rather edit rst files than futz with a web interface when > editing docs. The wiki also somehow feels "less official". > > I do think exposing our development process to the wider user community > by including them in the main docs could foster additional community > involvement, but I don't have a strong opinion on that aspect of it. > For me the change is about making it easier for the dev community > (who are using/creating the development infrastructure) to update the > relevant documentation. > +1 Keeping the dev docs in the development tree sounds good to me (however they are deployed to the web - but preferably automagically). Michael > -- > R. David Murray www.bitdance.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From solipsis at pitrou.net Thu Sep 23 16:31:47 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 23 Sep 2010 16:31:47 +0200 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923141601.D148D1FB3B2@kimball.webabinitio.net> References: <20100923121144.2006aeaa@pitrou.net> <20100923141601.D148D1FB3B2@kimball.webabinitio.net> Message-ID: <20100923163147.7f1a2ce9@pitrou.net> On Thu, 23 Sep 2010 10:16:01 -0400 "R. David Murray" wrote: > On Thu, 23 Sep 2010 23:35:02 +1000, Nick Coghlan wrote: > > On Thu, Sep 23, 2010 at 8:11 PM, Antoine Pitrou wrote: > > > The practicality argument of being able to edit those docs without > > > having to master a separate (pydotorg) workflow sounds quite strong to > > > me. > > > > This is the key point for me. For developer controlled stuff, the > > easiest place to have it if we want it kept up to date is in the > > source tree. Second best is the wiki. Having it off in a separately > > managed repository (that exists for perfectly valid reasons, since a > > lot of the content *isn't* developer controlled) is annoying. > > > > That said, in this case, what's the advantage of the source tree over > > the wiki? To include it in the main docs, so people reading them > > offline can still see the contribution workflow? > > I'd *much* rather edit rst files than futz with a web interface when > editing docs. The wiki also somehow feels "less official". I agree with the "less official" point. If our developer docs feel authoritative, people will be more encouraged to contribute. Also, the wiki in its current state looks much less polished than the Sphinx-generated docs. Regards Antoine. From barry at python.org Thu Sep 23 16:33:40 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 23 Sep 2010 10:33:40 -0400 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923141601.D148D1FB3B2@kimball.webabinitio.net> References: <20100923121144.2006aeaa@pitrou.net> <20100923141601.D148D1FB3B2@kimball.webabinitio.net> Message-ID: <20100923103340.5ff5e6f6@mission> On Sep 23, 2010, at 10:16 AM, R. David Murray wrote: >I'd *much* rather edit rst files than futz with a web interface when >editing docs. The wiki also somehow feels "less official". There are dvcs-backed wikis, for example: https://launchpad.net/wikkid :) I don't agree that the wiki feels less official, or perhaps that it *should* feel any less official. It's an important source of Pythonic information, and to me it feels much more inclusive and open. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Thu Sep 23 16:35:06 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 23 Sep 2010 10:35:06 -0400 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: References: <20100923100157.288a72a8@mission> Message-ID: <20100923103506.77307c5e@mission> On Sep 23, 2010, at 10:06 AM, Jesse Noller wrote: >-1 on wiki; wikis are where good information goes off to die. Well, *all* documentation requires vigilance to remain relevant and current. I'm sure you don't think the Python wiki is useless, right? ;) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Thu Sep 23 16:41:35 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 23 Sep 2010 10:41:35 -0400 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: References: <20100923100157.288a72a8@mission> Message-ID: <20100923104135.343e85bf@mission> On Sep 23, 2010, at 09:06 AM, Benjamin Peterson wrote: >Are any of our docs subject to release schedules? I guess what I'm concerned about is this scenario: You're a developer who has the source code to Python 3.1. You read the in-tree docs to get a sense of how our development process works. Now, you're a diligent and eager new contributor so you follow those instructions to the letter. Unfortunately, Python 3.5 is the current version and we've changed key parts of the process. There's no possible way that your 3.1 in-tree docs can be updated to reflect the new process. Okay, we can tell you to get the Python 3.5 code, or probably better yet, the Python 3.6 in-development trunk, but now we've got another dilemma. If we change the process in 3.6, there will be pressure to update the docs in 3.5 and previous versions that are still officially maintained. And what about security-only versions of Python? Our development processes are *primarily* independent of Python version, so I don't think they should be tied to our source tree, and our CPython source tree at that. I suspect the version-dependent instructions will be minimal and can be handled by the rare footnotes or whatever. IMHO of course, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From jnoller at gmail.com Thu Sep 23 16:43:10 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 23 Sep 2010 10:43:10 -0400 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923103506.77307c5e@mission> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> Message-ID: On Thu, Sep 23, 2010 at 10:35 AM, Barry Warsaw wrote: > On Sep 23, 2010, at 10:06 AM, Jesse Noller wrote: > >>-1 on wiki; wikis are where good information goes off to die. > > Well, *all* documentation requires vigilance to remain relevant and current. > I'm sure you don't think the Python wiki is useless, right? ;) I do. I visit it as little as possible. :( From martin at v.loewis.de Thu Sep 23 16:47:20 2010 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 23 Sep 2010 16:47:20 +0200 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923103340.5ff5e6f6@mission> References: <20100923121144.2006aeaa@pitrou.net> <20100923141601.D148D1FB3B2@kimball.webabinitio.net> <20100923103340.5ff5e6f6@mission> Message-ID: <4C9B6878.2090306@v.loewis.de> Am 23.09.2010 16:33, schrieb Barry Warsaw: > On Sep 23, 2010, at 10:16 AM, R. David Murray wrote: > >> I'd *much* rather edit rst files than futz with a web interface when >> editing docs. The wiki also somehow feels "less official". > > There are dvcs-backed wikis, for example: > > https://launchpad.net/wikkid > > :) > > I don't agree that the wiki feels less official, or perhaps that it *should* > feel any less official. It's an important source of Pythonic information, and > to me it feels much more inclusive and open. This impression comes along with the authority of potential authors. If only the release manager can write a document, it is very official. If any committer can write, but nobody else, it feels less officical. If anybody could modify the document, it's even less official. Since anybody can write to the Python wiki, it feels not very official. It's the same reason why people often trust Wikipedia less than a printed encyclopedia. Regards, Martin From guido at python.org Thu Sep 23 16:47:34 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Sep 2010 07:47:34 -0700 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923103506.77307c5e@mission> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> Message-ID: On Thu, Sep 23, 2010 at 7:35 AM, Barry Warsaw wrote: > On Sep 23, 2010, at 10:06 AM, Jesse Noller wrote: > >>-1 on wiki; wikis are where good information goes off to die. > > Well, *all* documentation requires vigilance to remain relevant and current. > I'm sure you don't think the Python wiki is useless, right? ;) I have to agree with Jesse. We have too many wiki pages that are so out of date they're dangerous. They serve a purpose, and I think we should have a wiki in addition to the "official" documentation. This could be aggressively linked from it so people can comment on that documentation -- a commenting system like the PHP docs have would be even better, but that's been an unimplemented idea for so long that I'm not holding my hopes up. But when one person (or a small group) sits down to write the "official" guidelines for doing something, I think using proper revision control and so on can only help improve the docs and keep them up to date. -- --Guido van Rossum (python.org/~guido) From barry at python.org Thu Sep 23 16:53:55 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 23 Sep 2010 10:53:55 -0400 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> Message-ID: <20100923105355.69068235@mission> On Sep 23, 2010, at 10:43 AM, Jesse Noller wrote: >On Thu, Sep 23, 2010 at 10:35 AM, Barry Warsaw >wrote: >> On Sep 23, 2010, at 10:06 AM, Jesse Noller wrote: >> >>>-1 on wiki; wikis are where good information goes off to die. >> >> Well, *all* documentation requires vigilance to remain relevant and >> current. I'm sure you don't think the Python wiki is useless, >> right? ;) > >I do. I visit it as little as possible. :( Bummer. There's no reason it *has* to be useless though. The Moin developer now has shell access, so if there are technical problems with wiki, like its theme, performance, or lack of features, we can get those fixed. If it's the content or organization that needs improvement, then we can recruit from the much larger Python community than those that have write access to the core svn. Let's honor and encourage folks who are really good at tending to wikis and give them the tools they need to make the wiki excellent. Of course, if the consensus is that wikis are just a waste of time and do more harm than good, then we should shut ours down. (I don't agree it is though.) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From guido at python.org Thu Sep 23 16:56:19 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Sep 2010 07:56:19 -0700 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <4C9B6878.2090306@v.loewis.de> References: <20100923121144.2006aeaa@pitrou.net> <20100923141601.D148D1FB3B2@kimball.webabinitio.net> <20100923103340.5ff5e6f6@mission> <4C9B6878.2090306@v.loewis.de> Message-ID: On Thu, Sep 23, 2010 at 7:47 AM, "Martin v. L?wis" wrote: > Am 23.09.2010 16:33, schrieb Barry Warsaw: >> On Sep 23, 2010, at 10:16 AM, R. David Murray wrote: >> >>> I'd *much* rather edit rst files than futz with a web interface when >>> editing docs. ?The wiki also somehow feels "less official". >> >> There are dvcs-backed wikis, for example: >> >> https://launchpad.net/wikkid >> >> :) >> >> I don't agree that the wiki feels less official, or perhaps that it *should* >> feel any less official. ?It's an important source of Pythonic information, and >> to me it feels much more inclusive and open. > > This impression comes along with the authority of potential authors. > > If only the release manager can write a document, it is very official. > If any committer can write, but nobody else, it feels less officical. > If anybody could modify the document, it's even less official. > > Since anybody can write to the Python wiki, it feels not very official. > It's the same reason why people often trust Wikipedia less than a > printed encyclopedia. I want to believe your theory (since I also have a feeling that some wiki pages feel less trustworthy than others) but my own use of Wikipedia makes me skeptical that this is all there is -- on many pages on important topics you can clearly tell that a lot of effort went into the article, and then I trust it more. On other places you can tell that almost nobody cared. But I never look at the names of the authors. -- --Guido van Rossum (python.org/~guido) From jnoller at gmail.com Thu Sep 23 17:01:33 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 23 Sep 2010 11:01:33 -0400 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923105355.69068235@mission> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> Message-ID: On Thu, Sep 23, 2010 at 10:53 AM, Barry Warsaw wrote: > On Sep 23, 2010, at 10:43 AM, Jesse Noller wrote: > >>On Thu, Sep 23, 2010 at 10:35 AM, Barry Warsaw >>wrote: >>> On Sep 23, 2010, at 10:06 AM, Jesse Noller wrote: >>> >>>>-1 on wiki; wikis are where good information goes off to die. >>> >>> Well, *all* documentation requires vigilance to remain relevant and >>> current. I'm sure you don't think the Python wiki is useless, >>> right? ;) >> >>I do. I visit it as little as possible. :( > > Bummer. > > There's no reason it *has* to be useless though. ?The Moin developer now has > shell access, so if there are technical problems with wiki, like its theme, > performance, or lack of features, we can get those fixed. ?If it's the content > or organization that needs improvement, then we can recruit from the much > larger Python community than those that have write access to the core svn. > Let's honor and encourage folks who are really good at tending to wikis and > give them the tools they need to make the wiki excellent. > > Of course, if the consensus is that wikis are just a waste of time and do more > harm than good, then we should shut ours down. ?(I don't agree it is though.) > > -Barry > To be honest; while I have a strong dislike for them - I think they work fine for unofficial sources of information, I don't think they work well for official "we stand by this" style information. So, no, I don't think it's totally useless, but I do think it's an information sinkhole, and I would never seriously publish anything I had to stand by to a "completely public" wiki personally. The larger community, however, probably finds it useful to have it as a resource, even as scattered and spottily curated as it can be - I just don't think it's a good location for official developer/development docs. I don't think we have the needed curation resources to keep on top of the willy-nilly editing wikis incur. jesse From dirkjan at ochtman.nl Thu Sep 23 17:06:39 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Thu, 23 Sep 2010 17:06:39 +0200 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: References: <20100923121144.2006aeaa@pitrou.net> <20100923141601.D148D1FB3B2@kimball.webabinitio.net> <20100923103340.5ff5e6f6@mission> <4C9B6878.2090306@v.loewis.de> Message-ID: On Thu, Sep 23, 2010 at 16:56, Guido van Rossum wrote: > I want to believe your theory (since I also have a feeling that some > wiki pages feel less trustworthy than others) but my own use of > Wikipedia makes me skeptical that this is all there is -- on many > pages on important topics you can clearly tell that a lot of effort > went into the article, and then I trust it more. On other places you > can tell that almost nobody cared. But I never look at the names of > the authors. Right -- I feel like wiki quality varies with the amount of attention spent on maintaining it. Wikis that get a lot of maintenance (or have someone devoted to "wiki gardening") will be good (consistent and up to date), while wikis that are only occasionally updated, or updated without much consistency or added to without editing get to feel bad. Seems like a variation of the broken window theory. So what we really need is a way to make editing the developer docs more rewarding (or less hard) for potential authors (i.e. python committers). If putting it in a proper VCS so they can use their editor of choice would help that, that seems like a good solution. Cheers, Dirkjan From solipsis at pitrou.net Thu Sep 23 17:32:55 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 23 Sep 2010 17:32:55 +0200 Subject: [Python-Dev] Python wiki References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> Message-ID: <20100923173255.282841a1@pitrou.net> On Thu, 23 Sep 2010 10:53:55 -0400 Barry Warsaw wrote: > > Of course, if the consensus is that wikis are just a waste of time and do more > harm than good, then we should shut ours down. (I don't agree it is though.) I don't think they are a waste of time. However, as you and Dirkjan pointed out, a wiki needs some "gardening" to take care of its structure and its presentation. The present Python wiki isn't very inviting graphically, and its structure doesn't look very thought out. Regards Antoine. From martin at v.loewis.de Thu Sep 23 17:42:03 2010 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 23 Sep 2010 17:42:03 +0200 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923105355.69068235@mission> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> Message-ID: <4C9B754B.5020700@v.loewis.de> > There's no reason it *has* to be useless though. The Moin developer now has > shell access, so if there are technical problems with wiki, like its theme, > performance, or lack of features, we can get those fixed. If it's the content > or organization that needs improvement, then we can recruit from the much > larger Python community than those that have write access to the core svn. > Let's honor and encourage folks who are really good at tending to wikis and > give them the tools they need to make the wiki excellent. > > Of course, if the consensus is that wikis are just a waste of time and do more > harm than good, then we should shut ours down. (I don't agree it is though.) I don't think there is (or can be) consensus about that. However, Jesse's objection is fairly widespread also, and it is not specific for wiki.python.org, or MoinMoin, but opposing Wikis as general. By nature (quick-quick), information is unorganized in a Wiki. This is what wiki advocates cite as its main feature, and wiki opponents as its main flaw. Regards, Martin From stevenrelliottjr1 at gmail.com Thu Sep 23 17:50:34 2010 From: stevenrelliottjr1 at gmail.com (Steven Elliott Jr) Date: Thu, 23 Sep 2010 11:50:34 -0400 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: References: <20100923121144.2006aeaa@pitrou.net> <20100923141601.D148D1FB3B2@kimball.webabinitio.net> <20100923103340.5ff5e6f6@mission> <4C9B6878.2090306@v.loewis.de> Message-ID: Hello All, I am new to this list, but I have been lurking around getting a feel for the environment and processes. I had some discussion yesterday about the developer documentation as well, since it?s what I do professionally. I am a technical writer but also work in the web development arena (using Django). In fact one of my projects now is to develop a comprehensive platform for distributing online help, user documentation, etc. which I am just about to put up on BitBucket (winter ?10). Anyway, that said, with regard to Wikis. I have worked in several organizations where almost all of the development documentation was maintained on a wiki. This can be great for getting up and running with something quickly, but over time it becomes very unmanageable and confusing. What I have done in various organizations has been to create a system where an official repository is kept with all of the *official* documentation and a way for users (developers) to submit their proposals as to what they would like to add and change. These proposals are kept in a tracker where they are read and evaluated. Generally, some discussion ensues and the choices are made as to what stays published or changed. This is what the system I am writing is all about as well. It maintains the documentation, and allows for users to comment on various parts of that documentation and submit requests to change or add. The admins can then change or deny the documentation based on community response. Anyway, I am not pitching my idea or trying to hump my system but I will be releasing it before winter on BitBucket for anyone to try and distribute freely. I do however, discourage the use of wikis at all costs. It has been said that they feel loose and unofficial, and although that my not be the intent, over time this becomes reality. Anyway, thank you for your time. Warmest Regards, Steve On Thu, Sep 23, 2010 at 11:06 AM, Dirkjan Ochtman wrote: > On Thu, Sep 23, 2010 at 16:56, Guido van Rossum wrote: > > I want to believe your theory (since I also have a feeling that some > > wiki pages feel less trustworthy than others) but my own use of > > Wikipedia makes me skeptical that this is all there is -- on many > > pages on important topics you can clearly tell that a lot of effort > > went into the article, and then I trust it more. On other places you > > can tell that almost nobody cared. But I never look at the names of > > the authors. > > Right -- I feel like wiki quality varies with the amount of attention > spent on maintaining it. Wikis that get a lot of maintenance (or have > someone devoted to "wiki gardening") will be good (consistent and up > to date), while wikis that are only occasionally updated, or updated > without much consistency or added to without editing get to feel bad. > Seems like a variation of the broken window theory. > > So what we really need is a way to make editing the developer docs > more rewarding (or less hard) for potential authors (i.e. python > committers). If putting it in a proper VCS so they can use their > editor of choice would help that, that seems like a good solution. > > Cheers, > > Dirkjan > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/stevenrelliottjr1%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Thu Sep 23 17:52:15 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 23 Sep 2010 17:52:15 +0200 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> Message-ID: <4C9B77AF.3010508@v.loewis.de> > I have to agree with Jesse. We have too many wiki pages that are so > out of date they're dangerous. They serve a purpose, and I think we > should have a wiki in addition to the "official" documentation. This > could be aggressively linked from it so people can comment on that > documentation -- a commenting system like the PHP docs have would be > even better, but that's been an unimplemented idea for so long that > I'm not holding my hopes up. You must have forgotten that you lent the time machine keys to Georg, though :-) http://gsoc.jacobmason.us/blog/?p=35 http://bitbucket.org/jacobmason/sphinx-web-support http://gsoc.jacobmason.us/demo/contents Regards, Martin From rdmurray at bitdance.com Thu Sep 23 17:49:36 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 23 Sep 2010 11:49:36 -0400 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923104135.343e85bf@mission> References: <20100923100157.288a72a8@mission> <20100923104135.343e85bf@mission> Message-ID: <20100923154936.AC8E422903C@kimball.webabinitio.net> On Thu, 23 Sep 2010 10:41:35 -0400, Barry Warsaw wrote: > On Sep 23, 2010, at 09:06 AM, Benjamin Peterson wrote: > >Are any of our docs subject to release schedules? > > I guess what I'm concerned about is this scenario: > > You're a developer who has the source code to Python 3.1. You read the > in-tree docs to get a sense of how our development process works. Now, you're > a diligent and eager new contributor so you follow those instructions to the > letter. Unfortunately, Python 3.5 is the current version and we've changed > key parts of the process. There's no possible way that your 3.1 in-tree docs > can be updated to reflect the new process. Except for major changes like the transition to hg, the dev process is no more likely to change than the code base (probably less!). That is, the eager developers with the 3.1 source code are just as likely to produce a patch that won't be useful because it doesn't apply to the current maintained versions as they are to encounter a piece of the dev process that has changed enough to break what they tried to do. (In this context, the switch to hg is analogous to the switch to Python3...) Also, the existence of the docs in the repository is (IMO) for *editing* convenience. The real place a new developer will be looking at the docs is on the web site, just as the place most people (even developers, unless I miss my guess; I know I do) look for Python documentation is on the web site. And that version will be up to date. > Okay, we can tell you to get the Python 3.5 code, or probably better yet, the > Python 3.6 in-development trunk, but now we've got another dilemma. If we > change the process in 3.6, there will be pressure to update the docs in 3.5 > and previous versions that are still officially maintained. And what about > security-only versions of Python? Yes, and? We update the docs of the maintained Python versions all the time. Doc backports are standard (even if Georg does most of them in batches) unless the documentation is about a new feature. The fact that even 'new features' of the dev process would also get backported is merely a detail. We don't update docs for security releases as far as I know, so I would expect we wouldn't update the dev docs either. > Our development processes are *primarily* independent of Python version, so I > don't think they should be tied to our source tree, and our CPython source > tree at that. I suspect the version-dependent instructions will be minimal > and can be handled by the rare footnotes or whatever. I don't think our development process applies to anything other than the CPython source. (At least at the moment...if we break out the stdlib that will change, but at that point the stdlib should have its own distinct development process, even if that process shares most of its features with the CPython one.) Our documentation is *primarily* independent of Python version, too, if you go by the ratio of the word count of the substantive changes from version to version to the word count of the docs as a whole :) True, the dev docs are even more independent, but I don't see that as trumping the convenience to the developers of having them in the source tree. A separate repository would also be fine, IMO. If someone can find or write the code to publish that repository to the appropriate location automatically, we could presumably do this even before the rest of the hg transition. --David From barry at python.org Thu Sep 23 17:57:19 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 23 Sep 2010 11:57:19 -0400 Subject: [Python-Dev] Python wiki In-Reply-To: <20100923173255.282841a1@pitrou.net> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> Message-ID: <20100923115719.2250e776@mission> On Sep 23, 2010, at 05:32 PM, Antoine Pitrou wrote: >I don't think they are a waste of time. However, as you and Dirkjan >pointed out, a wiki needs some "gardening" to take care of its >structure and its presentation. The present Python wiki isn't very >inviting graphically, and its structure doesn't look very thought out. I certainly agree with that. So, how can we solve those problems? Radomir has shell access now so perhaps we can ask him to make the Python wiki theme more visually appealing. What roadblocks do people encounter when they want to help garden or reorganize the wiki? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From rdmurray at bitdance.com Thu Sep 23 17:58:50 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 23 Sep 2010 11:58:50 -0400 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: References: <20100923121144.2006aeaa@pitrou.net> <20100923141601.D148D1FB3B2@kimball.webabinitio.net> <20100923103340.5ff5e6f6@mission> <4C9B6878.2090306@v.loewis.de> Message-ID: <20100923155850.B6E411FB3B2@kimball.webabinitio.net> On Thu, 23 Sep 2010 07:56:19 -0700, Guido van Rossum wrote: > On Thu, Sep 23, 2010 at 7:47 AM, wrote: > > This impression comes along with the authority of potential authors. > > > > If only the release manager can write a document, it is very official. > > If any committer can write, but nobody else, it feels less officical. > > If anybody could modify the document, it's even less official. > > > > Since anybody can write to the Python wiki, it feels not very official. > > It's the same reason why people often trust Wikipedia less than a > > printed encyclopedia. > > I want to believe your theory (since I also have a feeling that some > wiki pages feel less trustworthy than others) but my own use of > Wikipedia makes me skeptical that this is all there is -- on many > pages on important topics you can clearly tell that a lot of effort > went into the article, and then I trust it more. On other places you > can tell that almost nobody cared. But I never look at the names of > the authors. I think you've hit the nail on the head. The Python wiki pages mostly feel like nobody cares. At least that's the case for the ones I've stumbled across. And I'd include my own contributions in that (the email-sig wiki), because I was using them as a work area and have not updated them in some time, since development is now in a code repository. If we can recruit a bunch of somebodies who *do* care, then the wiki would be much more useful. But I still don't want to edit the dev docs there, if I have a choice :) There's a reason I stopped updating the wiki as soon as I moved to a code repository. -- R. David Murray www.bitdance.com From barry at python.org Thu Sep 23 18:05:13 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 23 Sep 2010 12:05:13 -0400 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923154936.AC8E422903C@kimball.webabinitio.net> References: <20100923100157.288a72a8@mission> <20100923104135.343e85bf@mission> <20100923154936.AC8E422903C@kimball.webabinitio.net> Message-ID: <20100923120513.2fe8ccef@mission> On Sep 23, 2010, at 11:49 AM, R. David Murray wrote: >A separate repository would also be fine, IMO. If someone can find or >write the code to publish that repository to the appropriate location >automatically, we could presumably do this even before the rest of the >hg transition. I'm not necessarily opposed to that either. I do think the switch to hg will cause lots of churn in the dev process, ultimately for the better, but there will be experiment and change at least for the code contribution bits. I'm also not as worried about the authority of the wiki. If we get good contributors and the rest of the community starts linking to wiki urls, it will feel (more) official. Anyway, it's all kind of secondary to actually writing stuff down. If Brett's going to do the work, then he gets to decide. :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From guido at python.org Thu Sep 23 18:16:49 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Sep 2010 09:16:49 -0700 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <4C9B77AF.3010508@v.loewis.de> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <4C9B77AF.3010508@v.loewis.de> Message-ID: On Thu, Sep 23, 2010 at 8:52 AM, "Martin v. L?wis" wrote: >> I have to agree with Jesse. We have too many wiki pages that are so >> out of date they're dangerous. They serve a purpose, and I think we >> should have a wiki in addition to the "official" documentation. This >> could be aggressively linked from it so people can comment on that >> documentation -- a commenting system like the PHP docs have would be >> even better, but that's been an unimplemented idea for so long that >> I'm not holding my hopes up. > > You must have forgotten that you lent the time machine keys to Georg, > though :-) > > http://gsoc.jacobmason.us/blog/?p=35 > http://bitbucket.org/jacobmason/sphinx-web-support > http://gsoc.jacobmason.us/demo/contents But before Georg returns the keys, he should make sure to install this on docs.python.org. :-) (I like it, but it needs some work. The login page needs instructions for people who've forgotten how to use OpenID. There needs to be an introduction on how to use the comment system at the root of the site. It would be nice to allow anonymous comments (with a way for site managers to turn this off on a per-page basis). And it would be nice if there was a pop-up with snippets of comments when you mouse over a comment bubble.) -- --Guido van Rossum (python.org/~guido) From tjreedy at udel.edu Thu Sep 23 18:19:54 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 23 Sep 2010 12:19:54 -0400 Subject: [Python-Dev] Problems with hex-conversion functions In-Reply-To: References: <8449046c0909051426g2eaa3f94s4c5842c25af17abf@mail.gmail.com> <4AA331A9.70209@gmail.com> Message-ID: On 9/23/2010 5:31 AM, Ender Wiggin wrote: > I think I have the skills to learn and fix the > code itself, but I don't have the time > and I am unfamiliar with the process of submitting patches and getting Anyone can submit a patch at bugs.python.org. The process of getting one approved includes responding to questions, suggestions, and criticisms. Beyond that, the process may be short if the patch is simple and non-controversial. Others may take extensive discussion on pydev or other forums. Some are ignored or rejected. One can also participate by commenting on issues started by others. See http://wiki.python.org/moin/TrackerDocs/ for more. > them approved. For example, who gets > to decide about the correct approach? This particular issue would probably require more discussion than less. However, submission of a patch using one approach would tend to push the discussion to happen. -- Terry Jan Reedy From g.brandl at gmx.net Thu Sep 23 18:20:34 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 23 Sep 2010 18:20:34 +0200 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923103506.77307c5e@mission> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> Message-ID: Am 23.09.2010 16:35, schrieb Barry Warsaw: > On Sep 23, 2010, at 10:06 AM, Jesse Noller wrote: > >>-1 on wiki; wikis are where good information goes off to die. > > Well, *all* documentation requires vigilance to remain relevant and current. > I'm sure you don't think the Python wiki is useless, right? ;) Don't worry, as soon as my thesis is gone for good, I will have time to finally make good use of the new features in Sphinx trunk, among them the often request commenting and patching feature. The result -- I dare say -- will be the best of both worlds: no unsupervised changes in content, but the possibility of instant feedback for readers. We'll require some more people wrangling the amount of information we get, but I've got quite a few requests from the community asking for things to help the docs; now I have to refer them to the tracker, which can be less than satisfying, then I can recruit them into the comment-handling team. cheers, Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From g.brandl at gmx.net Thu Sep 23 18:21:51 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 23 Sep 2010 18:21:51 +0200 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> Message-ID: Am 23.09.2010 16:47, schrieb Guido van Rossum: > On Thu, Sep 23, 2010 at 7:35 AM, Barry Warsaw wrote: >> On Sep 23, 2010, at 10:06 AM, Jesse Noller wrote: >> >>>-1 on wiki; wikis are where good information goes off to die. >> >> Well, *all* documentation requires vigilance to remain relevant and current. >> I'm sure you don't think the Python wiki is useless, right? ;) > > I have to agree with Jesse. We have too many wiki pages that are so > out of date they're dangerous. They serve a purpose, and I think we > should have a wiki in addition to the "official" documentation. This > could be aggressively linked from it so people can comment on that > documentation -- a commenting system like the PHP docs have would be > even better, but that's been an unimplemented idea for so long that > I'm not holding my hopes up. You should read my tweets more often :) Yes, I know I promised this for last year, but this time the code is already merged, and I "just" need to polish and set it up on docs.python.org. cheers, Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From g.brandl at gmx.net Thu Sep 23 18:25:38 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 23 Sep 2010 18:25:38 +0200 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923104135.343e85bf@mission> References: <20100923100157.288a72a8@mission> <20100923104135.343e85bf@mission> Message-ID: Am 23.09.2010 16:41, schrieb Barry Warsaw: > On Sep 23, 2010, at 09:06 AM, Benjamin Peterson wrote: > >>Are any of our docs subject to release schedules? > > I guess what I'm concerned about is this scenario: > > You're a developer who has the source code to Python 3.1. You read the > in-tree docs to get a sense of how our development process works. Now, you're > a diligent and eager new contributor so you follow those instructions to the > letter. Unfortunately, Python 3.5 is the current version and we've changed > key parts of the process. There's no possible way that your 3.1 in-tree docs > can be updated to reflect the new process. That's a pity, of course; however the small amount of bug reports we get that reflects content in old (= unsupported) library documentation suggests that it would not be a problem in practice: Most people look at docs.python.org anyway. > Okay, we can tell you to get the Python 3.5 code, or probably better yet, the > Python 3.6 in-development trunk, but now we've got another dilemma. If we > change the process in 3.6, there will be pressure to update the docs in 3.5 > and previous versions that are still officially maintained. And what about > security-only versions of Python? Well, with Mercurial we're supposed to check in all changes to the oldest branch they apply to. If everyone changing the dev docs keeps to that, all supported versions will have up-to-date docs. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From solipsis at pitrou.net Thu Sep 23 18:41:59 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 23 Sep 2010 18:41:59 +0200 Subject: [Python-Dev] Python wiki References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> Message-ID: <20100923184159.4ce2048e@pitrou.net> On Thu, 23 Sep 2010 11:57:19 -0400 Barry Warsaw wrote: > On Sep 23, 2010, at 05:32 PM, Antoine Pitrou wrote: > > >I don't think they are a waste of time. However, as you and Dirkjan > >pointed out, a wiki needs some "gardening" to take care of its > >structure and its presentation. The present Python wiki isn't very > >inviting graphically, and its structure doesn't look very thought out. > > I certainly agree with that. So, how can we solve those problems? Radomir > has shell access now so perhaps we can ask him to make the Python wiki theme > more visually appealing. What roadblocks do people encounter when they want > to help garden or reorganize the wiki? Given that few or none of us seem to (want to) actively contribute to the wiki, I'm afraid python-dev is not the place to ask. Perhaps a call should be issued on c.l.py asking interested people to subscribe to pydotorg (or any other list) and start there? By the way, I've just looked at the wiki: there are 2 or 3 edits per day. Compared to the size and vitality of the Python user community, this is a ridiculously low number. Regards Antoine. From stevenrelliottjr1 at gmail.com Thu Sep 23 18:59:37 2010 From: stevenrelliottjr1 at gmail.com (Steven Elliott Jr) Date: Thu, 23 Sep 2010 12:59:37 -0400 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923155850.B6E411FB3B2@kimball.webabinitio.net> References: <20100923121144.2006aeaa@pitrou.net> <20100923141601.D148D1FB3B2@kimball.webabinitio.net> <20100923103340.5ff5e6f6@mission> <4C9B6878.2090306@v.loewis.de> <20100923155850.B6E411FB3B2@kimball.webabinitio.net> Message-ID: If we can recruit a bunch of somebodies who *do* care, then the wiki > would be much more useful. But I still don't want to edit the > dev docs there, if I have a choice :) There's a reason I stopped > updating the wiki as soon as I moved to a code repository. > > I think that there are plenty that do care; I for one would be more than happy to work on whatever documentation needs might arise for this group. I am a bit of a documentation nut, since its what I do, also I come from the Django camp where people are obsessive over documentation. I still think that wikis are not the best solution but if that is something that needs to be tightened up then it would be something that I personally would have no problem working on. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Sep 23 19:22:14 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 23 Sep 2010 13:22:14 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: <4C9AFF59.9020400@v.loewis.de> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> Message-ID: On 9/23/2010 3:18 AM, "Martin v. L?wis" wrote: > I personally think that the tracker fields and how they should be set is > of minor importance. As of just now, if you were to wonder "What (security) bugs are open for 2.5" and search on open 2.5 issues, you would get a list of 44 issues. It is only 44 instead of hundreds because of the work I and Mark have done in the last 4 months. It it 44 instead of perhaps 5 because Tarek and Eric insist on marking all disutils2 issues for all versions even though though these issues have nothing to do with maintenance releases. It is a real nuisance when trying to do tracker cleanup. Trying to do searches in databases with inaccurate key data is a pain. > If there is a bug in Python, the most useful > contribution is to submit a fix (or provide a rationale why this is not > a bug). Agreed,at least abstractly, with applying a fix a close second. That does not mean that other activities are useless. However, there are currently 1034 issues with the patch keyword set and perhaps others with pacthes. So I think one can legitimately ask whether adding more *new* patches, to possibly sit unreviewed for years, is the most useful contribution at the moment. In any case, asking whether a patch submitted for 2.5 (and now 2.6) is relevant to future maintenance releases amounts to suggesting that is may not be a bug for current purposes. Certainly, anyone fixing a bug for 2.7 should also know whether or not it is also a 3.x bug. > Asking every now and then "is this still an issue", or setting > the version number, doesn't really advance the issue. Numerous issues have been advanced by the questions I and Mark have asked. Some were legitimately closed as out of date (the bug reported for 2.4/5/6 had already been fixed). Others were closed as fixed when someone committed something. The fact that Mark got over-zealous in closing issues too soon does not negate this. Some of our questions were more specific, and asking questions was not the only things we did. I tested some old reports against 3.1 and I believe Mark also did some testing himself. Setting Versions properly helps anyone searching for issues relevant to a particular version. If having a field set properly does not matter, then is should not be there. Are you suggesting that Versions be deleted? -- Terry Jan Reedy From tjreedy at udel.edu Thu Sep 23 19:44:47 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 23 Sep 2010 13:44:47 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: <4C9B43E6.8080308@v.loewis.de> References: <20100922205921.GA9692@cskk.homeip.net> <87mxr92i7s.fsf@uwakimon.sk.tsukuba.ac.jp> <20100923115928.1f988cc7@pitrou.net> <4C9B3DAC.9000404@voidspace.org.uk> <4C9B43E6.8080308@v.loewis.de> Message-ID: On 9/23/2010 8:11 AM, "Martin v. L?wis" wrote: >> make sure the issue is assigned to the right person if appropriate > > -1. We typically don't assign issues to others. What I and Mark (that I know of) did in that respect was to assign doc issues, including old issues reclassified as doc issues, to docs at python. > >> add relevant keywords to make it easier to find in searches > > -0. Going through old issues just to make sure the keywords are right > seems wasteful. >> ensure other fields in the tracker are filled in correctly > > Likewise. I am guessing that you have never done professional database management. And I suspect you would not be temperamentally suited to such. To repeat what I said in another post, if it does not matter how a field is set, it should not be there. You seem really antagonistic to, or at least dismissive of, people who contribute in ways other than how you do. That strikes me as unhelpful. -- Terry Jan Reedy From zooko at zooko.com Thu Sep 23 19:50:26 2010 From: zooko at zooko.com (Zooko O'Whielacronx) Date: Thu, 23 Sep 2010 11:50:26 -0600 Subject: [Python-Dev] Goodbye In-Reply-To: <4C9B4A6E.5070309@timgolden.me.uk> References: <20100922205921.GA9692@cskk.homeip.net> <87mxr92i7s.fsf@uwakimon.sk.tsukuba.ac.jp> <20100923115928.1f988cc7@pitrou.net> <4C9B3DAC.9000404@voidspace.org.uk> <4C9B43E6.8080308@v.loewis.de> <4C9B4471.8060100@voidspace.org.uk> <4C9B47E6.70601@v.loewis.de> <4C9B4A6E.5070309@timgolden.me.uk> Message-ID: Speaking as a frequent contributor of bug reports and an occasional contributor of patches, I must say that I feel like status quo of the tracker before Mark's work was discouraging. If there is a vast collection of abandoned tickets, it gives me the strong impression that my attempted contributions are likely to end up in that pile. The messages I got from the tracker due to Mark's work saying things like "This ticket is closed due to inactivity." or "Would you be interested in refreshing this patch?" started to get me interested in contributing again. Also, I would like to point out that, not having read the other traffic that this thread alludes to, either from earlier mailing list threads or from IRC, I don't really understand what exactly Mark did wrong. The complaints about his behavior on this thread seem to be a little... non-specific. Did he continue to close tickets after he was asked not to do so? I didn't see any quotes or timestamps showing what happened or when. Regards, Zooko From v+python at g.nevcal.com Thu Sep 23 19:42:59 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 23 Sep 2010 10:42:59 -0700 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923104135.343e85bf@mission> References: <20100923100157.288a72a8@mission> <20100923104135.343e85bf@mission> Message-ID: <4C9B91A3.2060506@g.nevcal.com> On 9/23/2010 7:41 AM, Barry Warsaw wrote: > Our development processes are*primarily* independent of Python version, so I > don't think they should be tied to our source tree, and our CPython source > tree at that. I suspect the version-dependent instructions will be minimal > and can be handled by the rare footnotes or whatever. +1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at holdenweb.com Thu Sep 23 20:12:08 2010 From: steve at holdenweb.com (Steve Holden) Date: Thu, 23 Sep 2010 14:12:08 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922205921.GA9692@cskk.homeip.net> <87mxr92i7s.fsf@uwakimon.sk.tsukuba.ac.jp> <20100923115928.1f988cc7@pitrou.net> <4C9B3DAC.9000404@voidspace.org.uk> <4C9B43E6.8080308@v.loewis.de> Message-ID: On 9/23/2010 1:44 PM, Terry Reedy wrote: > On 9/23/2010 8:11 AM, "Martin v. L?wis" wrote: >>> make sure the issue is assigned to the right person if appropriate >> >> -1. We typically don't assign issues to others. > > What I and Mark (that I know of) did in that respect was to assign doc > issues, including old issues reclassified as doc issues, to docs at python. > >> >>> add relevant keywords to make it easier to find in searches >> >> -0. Going through old issues just to make sure the keywords are right >> seems wasteful. > >>> ensure other fields in the tracker are filled in correctly >> >> Likewise. > > I am guessing that you have never done professional database management. > And I suspect you would not be temperamentally suited to such. To repeat > what I said in another post, if it does not matter how a field is set, > it should not be there. > > You seem really antagonistic to, or at least dismissive of, people who > contribute in ways other than how you do. That strikes me as unhelpful. > And before this descends into a general discussion of various people's approach to collaboration, let me remind us all that it does indeed "take all sorts to make a world". >From what I have seen on this list, nobody has suggested that Mark's work was *all* unhelpful. The fact that he can be difficult to work with doesn't remove the value of much of the work he was doing, and I am happy to note that there doesn't seem to be any blanket verdict that Mark should never have tracker privileges again. Mark courageously admitted on this list to having personal problems to overcome. That made me, at least, more sympathetic to his case, and more tolerant of his foibles and style of communication. I appreciate that a disruptive team member might eventually have to be isolated for the good of the team, but I am sorry to note that it came to that in Mark's case, and I hope that eventually he can return to the fold somehow. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From rdmurray at bitdance.com Thu Sep 23 20:43:04 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 23 Sep 2010 14:43:04 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922205921.GA9692@cskk.homeip.net> <87mxr92i7s.fsf@uwakimon.sk.tsukuba.ac.jp> <20100923115928.1f988cc7@pitrou.net> <4C9B3DAC.9000404@voidspace.org.uk> <4C9B43E6.8080308@v.loewis.de> <4C9B4471.8060100@voidspace.org.uk> <4C9B47E6.70601@v.loewis.de> <4C9B4A6E.5070309@timgolden.me.uk> Message-ID: <20100923184304.904041FFD4D@kimball.webabinitio.net> On Thu, 23 Sep 2010 11:50:26 -0600, "Zooko O'Whielacronx" wrote: > Also, I would like to point out that, not having read the other > traffic that this thread alludes to, either from earlier mailing list > threads or from IRC, I don't really understand what exactly Mark did > wrong. The complaints about his behavior on this thread seem to be a > little... non-specific. Did he continue to close tickets after he was > asked not to do so? I didn't see any quotes or timestamps showing what > happened or when. Yes. It was necessary for the devs to monitor his work and re-open tickets he closed inappropriately. We always explained why the closure was inappropriate. The number of tickets closed inappropriately was growing rather than shrinking. It was this latter fact that, I believe, led to what happened. I think this could have been handled better, but thankfully this is not a situation we often find ourselves in, so we don't have much practice at dealing with such. Good ideas for procedures have been aired here as a result, so if the next incident is not too far in the future hopefully we'll handle the next one more smoothly. (But you'll excuse me for hoping it is so far in the future that we forget :) Again, there was nothing preventing Mark from continuing to comment on issues, test things, ask questions, and even suggest that issues be closed. However, his inappropriate response to losing triage privileges was causing significant problems on the tracker, and so we were forced to disable his account. This disabling does not need to be permanent. -- R. David Murray www.bitdance.com From skip at pobox.com Thu Sep 23 20:50:06 2010 From: skip at pobox.com (skip at pobox.com) Date: Thu, 23 Sep 2010 13:50:06 -0500 Subject: [Python-Dev] Python wiki In-Reply-To: <20100923173255.282841a1@pitrou.net> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> Message-ID: <19611.41310.55203.746123@montanaro.dyndns.org> Antoine> The present Python wiki isn't very inviting graphically, and Antoine> its structure doesn't look very thought out. I imagine it can be made to look more like the rest of python.org without a lot of trouble. As to the structure, like most wikis it quickly resembles a bag-o-pages after awhile. Thank goodness for Google. Skip From skip at pobox.com Thu Sep 23 20:52:25 2010 From: skip at pobox.com (skip at pobox.com) Date: Thu, 23 Sep 2010 13:52:25 -0500 Subject: [Python-Dev] Python wiki In-Reply-To: <20100923184159.4ce2048e@pitrou.net> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> Message-ID: <19611.41449.475652.757621@montanaro.dyndns.org> Antoine> Given that few or none of us seem to (want to) actively Antoine> contribute to the wiki, I'm afraid python-dev is not the place Antoine> to ask. Perhaps a call should be issued on c.l.py ... It would be nice if you could actually send messages to the people who do actually update wiki content. Unfortunately, without donning my cape and blue tights, then digging into the users files on the wiki there's no real way to do that. Skip From g.brandl at gmx.net Thu Sep 23 21:50:50 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 23 Sep 2010 21:50:50 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> Message-ID: Am 23.09.2010 19:22, schrieb Terry Reedy: >> Asking every now and then "is this still an issue", or setting >> the version number, doesn't really advance the issue. > > Numerous issues have been advanced by the questions I and Mark have > asked. Some were legitimately closed as out of date (the bug reported > for 2.4/5/6 had already been fixed). Others were closed as fixed when > someone committed something. The fact that Mark got over-zealous in > closing issues too soon does not negate this. Some of our questions were > more specific, and asking questions was not the only things we did. I > tested some old reports against 3.1 and I believe Mark also did some > testing himself. > > Setting Versions properly helps anyone searching for issues relevant to > a particular version. If having a field set properly does not matter, > then is should not be there. Are you suggesting that Versions be deleted? ISTM that the "versions" field is not very useful if the other fields are filled accurately. For example, feature requests almost always only belong to the current trunk. Yes, for features that fall under the moratorium, the "versions" field would be different; however, we already have an "after moratorium" keyword that signifies this. Other than this, we currently don't assign feature requests to specific milestones, and I don't ever see us doing that. Bug reports by nature almost always belong to all branches in maintenance; if the bug in question is a regression, the reporter will usually report that; if it is in a new feature, a) the reporter might not know that anyway, but b) the committer fixing it will without looking at "versions". Security bugs are the last category, but they are also better served with a "security" keyword than with specific "versions" settings. The only use I see in the field is for the reporter to indicate which version he used when finding a bug; however this is next to useless since you have to reproduce it anyway, and ask for more detail when you can't. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From techtonik at gmail.com Thu Sep 23 22:25:31 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 23 Sep 2010 23:25:31 +0300 Subject: [Python-Dev] Python wiki In-Reply-To: <20100923115719.2250e776@mission> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> Message-ID: On Thu, Sep 23, 2010 at 6:57 PM, Barry Warsaw wrote: > > I certainly agree with that. ?So, how can we solve those problems? ?Radomir > has shell access now so perhaps we can ask him to make the Python wiki theme > more visually appealing. ?What roadblocks do people encounter when they want > to help garden or reorganize the wiki? First of all Wiki is outdated. Correct me, but python.org specific customizations are not modules, but patches that makes it hard to update the Wiki to latest version or add new customizations. Second - ugly Creole syntax. I am for inter-cooperation between wikis, and I understand that for non-developer communities [] symbols imposing problems, but as an open source developer I am addicted to Trac and Google Code syntax and never had problems with those. Third - there is not starting point to help with wiki. No instructions how to checkout wiki code, how to get python.org modification, how to get sample data and how to get started. This place should be easily editable by anyone (premoderated for disrespectful members), so anyone can share their experience. Take a look at Trac. Fourth. GPL license. I personally don't have interest to waste my time for the code I won't be able to use in some projects, unless the project is either exceptional (Mercurial) or interesting. To make python.org MoinMoin interesting - there should be an inviting entrypoint (see point three above) and the list of tasks to choose form. Something that is better than http://wiki.python.org/moin/SiteImprovements Fifth. Credits and motivation for all python.org works. I still convinced that there should be one primary dedicated list and it should be public. All sensitive issues can be discussed with webmasters@ privately, but the primary list should be run by community. Not the volunteers who are better than others. If there will be a feeling that site is run by community, then you can expect contributions. Otherwise expect the community to think "they're doing the stuff here, so they will fix it". -- anatoly t. From merwok at netwok.org Thu Sep 23 22:51:57 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 23 Sep 2010 22:51:57 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> Message-ID: <4C9BBDED.5010701@netwok.org> Le 23/09/2010 19:22, Terry Reedy a ?crit : > As of just now, if you were to wonder "What (security) bugs are open for > 2.5" and search on open 2.5 issues, you would get a list of 44 issues. > It is only 44 instead of hundreds because of the work I and Mark have > done in the last 4 months. It it 44 instead of perhaps 5 because Tarek > and Eric insist on marking all disutils2 issues for all versions even > though though these issues have nothing to do with maintenance releases. > It is a real nuisance when trying to do tracker cleanup. Let?s fix this. Two days ago, I said this in http://bugs.python.org/issue2200#msg117003 : distutils2 has to work with 2.4-2.7 and (soon) 3.1-3.2, so Tarek told me to set all available versions for those bugs (2.5-py3k), even if I think ?3rd party? is more appropriate and useful (since a distutils2 bug would not for example block a CPython 3.2 release). I?ve been following these rules since before GSoC, when I had less confidence and no will to conflict with Tarek on such a small thing . Now I argue that the versions field is not really useful for d2, since it lacks 2.4 which we support and chiefly because it does not actually help our workflow: We don?t have separate branches for backporting fixes, we apply patches and run tests for all supported versions before committing on a single branch. There is no use case of setting a version to remind that a port needs to be done. For bug purposes, I actually see distutils2 as a value for the versions field of distutils bugs. In short, setting versions other than ?3rd party? for distutils2 bugs does not help distutils2 and raises unhelpful results for people querying the status of CPython releases. +1 on changing that. respect-my-non-ASCII-diversity-write-my-acute-accent-thanks?ly yours, ?ric From brett at python.org Thu Sep 23 23:04:46 2010 From: brett at python.org (Brett Cannon) Date: Thu, 23 Sep 2010 14:04:46 -0700 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <20100923120513.2fe8ccef@mission> References: <20100923100157.288a72a8@mission> <20100923104135.343e85bf@mission> <20100923154936.AC8E422903C@kimball.webabinitio.net> <20100923120513.2fe8ccef@mission> Message-ID: On Thu, Sep 23, 2010 at 09:05, Barry Warsaw wrote: > On Sep 23, 2010, at 11:49 AM, R. David Murray wrote: > >>A separate repository would also be fine, IMO. ?If someone can find or >>write the code to publish that repository to the appropriate location >>automatically, we could presumably do this even before the rest of the >>hg transition. > > I'm not necessarily opposed to that either. > > I do think the switch to hg will cause lots of churn in the dev process, > ultimately for the better, but there will be experiment and change at least > for the code contribution bits. > > I'm also not as worried about the authority of the wiki. ?If we get good > contributors and the rest of the community starts linking to wiki urls, it > will feel (more) official. > > Anyway, it's all kind of secondary to actually writing stuff down. > If Brett's going to do the work, then he gets to decide. :) Whether it is in Doc/ or a separate Hg repo, I don't care. But I am not doing it in the wiki. While I am totally fine with wikis as a general community thing where community input and editing is good, this is not one of those cases. Our development process belongs to python-dev and thus should be influenced by its members. I do not want to have to police the dev docs after I write them because someone either disagreed or misunderstood what was expected. For something this formal and official I want pro-active instead of reactive editorial oversight. From martin at v.loewis.de Thu Sep 23 23:41:09 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 23 Sep 2010 23:41:09 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> Message-ID: <4C9BC975.5090801@v.loewis.de> >> Asking every now and then "is this still an issue", or setting >> the version number, doesn't really advance the issue. > > > Setting Versions properly helps anyone searching for issues relevant to > a particular version. If having a field set properly does not matter, > then is should not be there. Are you suggesting that Versions be deleted? I didn't say the field does not matter. I said adjusting it doesn't advance the issue. Anybody *really* working on the issue might choose to update it, or might choose to leave it incorrect when the issue is going to be closed, anyway. I do believe that much of the information on closed issues is irrelevant - yet I would oppose to deleting them entirely from the database. Regards, Martin From ncoghlan at gmail.com Fri Sep 24 00:04:21 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 24 Sep 2010 08:04:21 +1000 Subject: [Python-Dev] Problems with hex-conversion functions In-Reply-To: References: <8449046c0909051426g2eaa3f94s4c5842c25af17abf@mail.gmail.com> <4AA331A9.70209@gmail.com> Message-ID: On Thu, Sep 23, 2010 at 7:31 PM, Ender Wiggin wrote: > Sorry for the late reply. I would really like to see this fixed. > >>> Or we [...] deprecate binascii.(un)hexlify(). > ... >>> binascii is the legacy approach here, so if anything was to go, those >>> functions would be it > ... > > I'm not entirely convinced binascii is the legacy approach. What makes this > module "legacy"? Because the binascii functions predate the bytes type, and we added the bytes methods knowing full well that the hexlify/unhexlify functions already existed. > On the contrary, I'm pretty sure modularity is better than sticking all the > functionality in the core. > As was written in this issue: > http://psf.upfronthosting.co.za/roundup/tracker/issue3532 > "If you wanted to produce base-85 (say), then you can extend the > functionality of bytes by providing a > function that does that, whereas you can't extend the existing bytes type." > This example shows that "hex" is actually getting a special treatment by > having builtin methods associated > with the bytes type. Why don't we add ".base64" methods? Or even ".zlib"? > After all, these options were present > in Python 2.x using the "encode" method of string. In my opinion, having > modules to deal with these types of > conversions is better, and this is why I suggested sticking?to binascii. This *is* a matter of opinion, but python-dev's collective opinion was already expressed in the decision to include these methods in the bytes API. Base 16 *is* given special treatment by many parts of Python, precisely because it *is* special: it's the most convenient way to express binary numbers in a vaguely human readable format. No other coding even comes close to that level of importance in computer science. > If no one else is willing to do it (that would be a > little disappoiting) Why would it be disappointing? While it's untidy, nothing's actually broken and there are ways for programmers to do everything they want to do. I (and many others here) already have a pretty long list of "things I'd like to improve/fix but haven't got around to yet", so it isn't uncommon for things to have to wait awhile before someone looks at them. As Terry said though, there *are* ways to expedite that process (In this case, providing a patch that adds a .hex method in accordance with PEP 358, or, as a more ambitious, extensible alternative, consider updating the hex builtin to support the PEP 3118 API, which would allow it to automatically provide a hex dump of any object that exposes a view of a contiguous sequence of data bytes). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tjreedy at udel.edu Fri Sep 24 00:13:48 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 23 Sep 2010 18:13:48 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> Message-ID: On 9/23/2010 3:50 PM, Georg Brandl wrote: > ISTM that the "versions" field is not very useful if the other fields are > filled accurately. > > For example, feature requests almost always only belong to the current trunk. > Yes, for features that fall under the moratorium, the "versions" field would > be different; however, we already have an "after moratorium" keyword that > signifies this. Other than this, we currently don't assign feature requests > to specific milestones, and I don't ever see us doing that. > > Bug reports by nature almost always belong to all branches in maintenance; For various reasons, there bugs that are 2.x or 3.x specific. Those that are not may require sufficiently different fixes that a fix is applied for one branch while the issue is left open for the other. -- Terry Jan Reedy From ncoghlan at gmail.com Fri Sep 24 00:17:45 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 24 Sep 2010 08:17:45 +1000 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> Message-ID: On Fri, Sep 24, 2010 at 5:50 AM, Georg Brandl wrote: >> Setting Versions properly helps anyone searching for issues relevant to >> a particular version. If having a field set properly does not matter, >> then is should not be there. Are you suggesting that Versions be deleted? > > ISTM that the "versions" field is not very useful if the other fields are > filled accurately. It's useful for bug reports to flag affected versions. We should probably just *not set it* for feature requests, though. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Fri Sep 24 00:19:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 24 Sep 2010 08:19:50 +1000 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922205921.GA9692@cskk.homeip.net> <87mxr92i7s.fsf@uwakimon.sk.tsukuba.ac.jp> <20100923115928.1f988cc7@pitrou.net> <4C9B3DAC.9000404@voidspace.org.uk> <4C9B43E6.8080308@v.loewis.de> Message-ID: On Fri, Sep 24, 2010 at 3:44 AM, Terry Reedy wrote: > On 9/23/2010 8:11 AM, "Martin v. L?wis" wrote: >>> >>> make sure the issue is assigned to the right person if appropriate >> >> -1. We typically don't assign issues to others. > > What I and Mark (that I know of) did in that respect was to assign doc > issues, including old issues reclassified as doc issues, to docs at python. The other thing is that the maintainers file is there specifically to help the triage team decide when a direct assignment is appropriate, or just adding people to the nosy list. Since I don't follow the bugs list myself, the latter is *very* helpful to me. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Fri Sep 24 00:27:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 24 Sep 2010 08:27:39 +1000 Subject: [Python-Dev] Python wiki In-Reply-To: <19611.41449.475652.757621@montanaro.dyndns.org> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> Message-ID: On Fri, Sep 24, 2010 at 4:52 AM, wrote: > > ? ?Antoine> Given that few or none of us seem to (want to) actively > ? ?Antoine> contribute to the wiki, I'm afraid python-dev is not the place > ? ?Antoine> to ask. ?Perhaps a call should be issued on c.l.py ... > > It would be nice if you could actually send messages to the people who do > actually update wiki content. ?Unfortunately, without donning my cape and > blue tights, then digging into the users files on the wiki there's no real > way to do that. That's a good point actually... why *isn't* there a pydotorg-wiki-sig? (Aside from the obvious point of nobody ever asking for one). I must admit, that the various things I've thrown on there myself have been pretty much fire-and-forget. Without anyone that feels like they collectively "own" the wiki, the much needed pruning never happens. With an admin team behind it, you can also make more use of ACLs to flag certain parts of the wiki as "official" by making them only editable by certain people (e.g. only devs, only the triage team, only the wiki admins). But keeping those user lists up to date is itself something that requires a strong wiki admin team. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Fri Sep 24 00:30:44 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 24 Sep 2010 08:30:44 +1000 Subject: [Python-Dev] [Python-checkins] r84988 - in python/branches/py3k: Lib/ntpath.py Misc/NEWS In-Reply-To: <20100923203814.E6403EE983@mail.python.org> References: <20100923203814.E6403EE983@mail.python.org> Message-ID: On Fri, Sep 24, 2010 at 6:38 AM, brian.curtin wrote: > Modified: python/branches/py3k/Lib/ntpath.py > ============================================================================== > --- python/branches/py3k/Lib/ntpath.py ?(original) > +++ python/branches/py3k/Lib/ntpath.py ?Thu Sep 23 22:38:14 2010 > @@ -641,24 +641,29 @@ > > > ?# determine if two files are in fact the same file > +try: > + ? ?from nt import _getfinalpathname > +except (NotImplementedError, ImportError): > + ? ?# On Windows XP and earlier, two files are the same if their absolute > + ? ?# pathnames are the same. > + ? ?# Also, on other operating systems, fake this method with a > + ? ?# Windows-XP approximation. > + ? ?def _getfinalpathname(f): > + ? ? ? ?return abspath(f) This only needs to catch ImportError now. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From martin at v.loewis.de Fri Sep 24 00:35:59 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 24 Sep 2010 00:35:59 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> Message-ID: <4C9BD64F.5020908@v.loewis.de> > With an admin team behind it, you can also make more use of ACLs to > flag certain parts of the wiki as "official" by making them only > editable by certain people (e.g. only devs, only the triage team, only > the wiki admins). But keeping those user lists up to date is itself > something that requires a strong wiki admin team. There actually is an admin team, and they actually do set ACLs. IIUC, this is primarily for spam protection, though. Regards, Martin From guido at python.org Fri Sep 24 00:39:00 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Sep 2010 15:39:00 -0700 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9BD64F.5020908@v.loewis.de> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> Message-ID: On Thu, Sep 23, 2010 at 3:35 PM, "Martin v. L?wis" wrote: >> With an admin team behind it, you can also make more use of ACLs to >> flag certain parts of the wiki as "official" by making them only >> editable by certain people (e.g. only devs, only the triage team, only >> the wiki admins). But keeping those user lists up to date is itself >> something that requires a strong wiki admin team. > > There actually is an admin team, and they actually do set ACLs. Who are they? > IIUC, this is primarily for spam protection, though. So would they object against additions to the team? -- --Guido van Rossum (python.org/~guido) From tjreedy at udel.edu Fri Sep 24 01:06:29 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 23 Sep 2010 19:06:29 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> Message-ID: On 9/23/2010 6:17 PM, Nick Coghlan wrote: > On Fri, Sep 24, 2010 at 5:50 AM, Georg Brandl wrote: >>> Setting Versions properly helps anyone searching for issues relevant to >>> a particular version. If having a field set properly does not matter, >>> then is should not be there. Are you suggesting that Versions be deleted? >> >> ISTM that the "versions" field is not very useful if the other fields are >> filled accurately. > > It's useful for bug reports to flag affected versions. We should > probably just *not set it* for feature requests, though. Now that 2.7 is out, so that feature requests can only be for a future 3.x, I would actually like the tracker to restrict the allowed values for non-doc feature requests either to 3.2/3.3 or to Not Applicable or whatever. It is a nuisance that people can still file such for 2.7, for instance. -- Terry Jan Reedy From darren at ontrenet.com Fri Sep 24 01:12:47 2010 From: darren at ontrenet.com (darren at ontrenet.com) Date: Thu, 23 Sep 2010 19:12:47 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> Message-ID: So if there turns out to be a major security hole or sever bug in 2.7, then it shouldn't be filed against 2.7? and fixed in a 2.7.x sort of branch? In that case, would you just suggest everyone using 2.7 to jump to 3.x? As long as a 2.x version is supported, filing bugs, branching and even releasing critical updates is, although rare, a fact of life. On Thu, 23 Sep 2010 19:06:29 -0400, Terry Reedy wrote: > On 9/23/2010 6:17 PM, Nick Coghlan wrote: >> On Fri, Sep 24, 2010 at 5:50 AM, Georg Brandl wrote: >>>> Setting Versions properly helps anyone searching for issues relevant to >>>> a particular version. If having a field set properly does not matter, >>>> then is should not be there. Are you suggesting that Versions be >>>> deleted? >>> >>> ISTM that the "versions" field is not very useful if the other fields >>> are >>> filled accurately. >> >> It's useful for bug reports to flag affected versions. We should >> probably just *not set it* for feature requests, though. > > Now that 2.7 is out, so that feature requests can only be for a future > 3.x, I would actually like the tracker to restrict the allowed values > for non-doc feature requests either to 3.2/3.3 or to Not Applicable or > whatever. It is a nuisance that people can still file such for 2.7, for > instance. From brian.curtin at gmail.com Fri Sep 24 01:53:15 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Thu, 23 Sep 2010 18:53:15 -0500 Subject: [Python-Dev] [Python-checkins] r84988 - in python/branches/py3k: Lib/ntpath.py Misc/NEWS In-Reply-To: References: <20100923203814.E6403EE983@mail.python.org> Message-ID: On Thu, Sep 23, 2010 at 17:30, Nick Coghlan wrote: > On Fri, Sep 24, 2010 at 6:38 AM, brian.curtin > wrote: > > Modified: python/branches/py3k/Lib/ntpath.py > > > ============================================================================== > > --- python/branches/py3k/Lib/ntpath.py (original) > > +++ python/branches/py3k/Lib/ntpath.py Thu Sep 23 22:38:14 2010 > > @@ -641,24 +641,29 @@ > > > > > > # determine if two files are in fact the same file > > +try: > > + from nt import _getfinalpathname > > +except (NotImplementedError, ImportError): > > + # On Windows XP and earlier, two files are the same if their > absolute > > + # pathnames are the same. > > + # Also, on other operating systems, fake this method with a > > + # Windows-XP approximation. > > + def _getfinalpathname(f): > > + return abspath(f) > > This only needs to catch ImportError now. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > Good catch. I put up a patch on http://bugs.python.org/issue9790 to rework this yet again. The NotImplementedError had to do with the underlying Win32 call only existing on Vista and above, and it was loaded at runtime. I added another condition to the import dance which should have us covered. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Sep 24 02:31:29 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 23 Sep 2010 20:31:29 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> Message-ID: On 9/23/2010 7:12 PM, darren at ontrenet.com wrote: > > So if there turns out to be a major security hole or sever bug in 2.7, > then it shouldn't be filed against 2.7? and fixed in a 2.7.x sort of > branch? > In that case, would you just suggest everyone using 2.7 to jump to 3.x? > > As long as a 2.x version is supported, filing bugs, branching and even > releasing critical updates is, although rare, a fact of life. I am not sure who or what you are responding to. The below is based on the fact the 2.7 is now closed to *new features* and will be as long as the CPython pydev group maintains it. In another post I said that the Versions field is needed for *bugs* as long as we are maintaining 2.7, which will be for several years, because there are and will continue to be 2.7 and 3.x specific bugs. If you want *new features*, then yes, you need to jump to 3.x. Otherwise you can relax, and perhaps contribute to 2.7 bug fixes if you want. >> Now that 2.7 is out, so that feature requests can only be for a future >> 3.x, I would actually like the tracker to restrict the allowed values >> for non-doc feature requests either to 3.2/3.3 or to Not Applicable or >> whatever. It is a nuisance that people can still file such for 2.7, for >> instance. -- Terry Jan Reedy From jnoller at gmail.com Fri Sep 24 03:32:45 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 23 Sep 2010 21:32:45 -0400 Subject: [Python-Dev] Call for proposals -- PyCon 2011 Message-ID: [Forwarding to Python-Dev, as it's of some interest - jesse] Call for proposals -- PyCon 2011 -- =============================================================== Proposal Due date: November 1st, 2010 PyCon is back! With a rocking new website, a great location and more Python hackers and luminaries under one roof than you could possibly shake a stick at. We've also added an "Extreme" talk track this year - no introduction, no fluff - only the pure technical meat! PyCon 2011 will be held March 9th through the 17th, 2011 in Atlanta, Georgia. (Home of some of the best southern food you can possibly find on Earth!) The PyCon conference days will be March 11-13, preceded by two tutorial days (March 9-10), and followed by four days of development sprints (March 14-17). PyCon 2011 is looking for proposals for the formal presentation tracks (this includes "extreme talks"). A request for proposals for poster sessions and tutorials will come separately. Want to showcase your skills as a Python Hacker? Want to have hundreds of people see your talk on the subject of your choice? Have some hot button issue you think the community needs to address, or have some package, code or project you simply love talking about? Want to launch your master plan to take over the world with Python? PyCon is your platform for getting the word out and teaching something new to hundreds of people, face to face. In the past, PyCon has had a broad range of presentations, from reports on academic and commercial projects, tutorials on a broad range of subjects, and case studies. All conference speakers are volunteers and come from a myriad of backgrounds: some are new speakers, some have been speaking for years. Everyone is welcome, so bring your passion and your code! We've had some incredible past PyCons, and we're looking to you to help us top them! Online proposal submission is open now! Proposals will be accepted through November 10th, with acceptance notifications coming out by January 20th. To get started, please see: For videos of talks from previous years - check out: For more information on "Extreme Talks" see: We look forward to seeing you in Atlanta! Please also note - registration for PyCon 2011 will also be capped at a maximum of 1,500 delegates, including speakers. When registration opens (soon), you're going to want to make sure you register early! Speakers with accepted talks will have a guaranteed slot. Important Dates: * November 1st, 2010: Talk proposals due. * December 15th, 2010: Acceptance emails sent. * January 19th, 2010: Early bird registration closes. * March 9-10th, 2011: Tutorial days at PyCon. * March 11-13th, 2011: PyCon main conference. * March 14-17th, 2011: PyCon sprints days. Contact Emails: Van Lindberg (Conference Chair) - van at python.org Jesse Noller (Co-Chair) - jnoller at python.org PyCon Organizers list: pycon-organizers at python.org From steve at pearwood.info Fri Sep 24 04:35:42 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 24 Sep 2010 12:35:42 +1000 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: References: Message-ID: <201009241235.43464.steve@pearwood.info> On Fri, 24 Sep 2010 01:50:34 am Steven Elliott Jr wrote: > What I have done in various organizations has been to create a system > where an official repository is kept with all of the *official* > documentation and a way for users (developers) to submit their > proposals as to what they would like to add and change. [...] > I do however, discourage the use of wikis at all costs. It has been > said that they feel loose and unofficial, and although that my not be > the intent, over time this becomes reality. Surely that depends on how widely you give write-privileges to the wiki? If you wouldn't give arbitrary people write-access to your documentation repository, why would you give them write-access to your wiki? If the wiki doesn't allow you control who has read and write access, then use a different wiki. I'm not familiar with any wiki that doesn't allow you to track and review history of the documents. Some of them are just web interfaces to standard VCSes like Mercurial. Wikipedia is now experimenting with "pending changes" and having stable and unstable versions of pages. I've known people to work themselves into a tizz at the thought of their developers making "unauthorized" changes to the documentation, while not even tracking changes to the source code *at all*, let alone reviewing the commits. This makes no sense to me at all -- if you (generic you, not you specifically) trust your developers to make changes to the source code, why not trust them to make changes to the documentation? The real problem, it seems to me, is the difficulty in getting developers to write and update documentation, not in preventing them from writing it. -- Steven D'Aprano From steve at pearwood.info Fri Sep 24 05:02:17 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 24 Sep 2010 13:02:17 +1000 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: <4C9B754B.5020700@v.loewis.de> References: <20100923105355.69068235@mission> <4C9B754B.5020700@v.loewis.de> Message-ID: <201009241302.18201.steve@pearwood.info> On Fri, 24 Sep 2010 01:42:03 am Martin v. L?wis wrote: > By nature (quick-quick), information is unorganized in a Wiki. This > is what wiki advocates cite as its main feature, and wiki opponents > as its main flaw. I've never heard wiki advocates say that, and even a cursory glace at wikis like Wikipedia disprove the idea that wikis are necessarily disorganised. "Quick" does not mean unstructured, disorganised or unorganised. Do you mean that the *contributors* to the wiki are disorganised, rather than the wiki itself? If so, perhaps, but again Wikipedia demonstrates that this is not necessarily the case. Wikipedia has a hierarchy of contributors. In reverse order: - anonymous editors - editors with accounts - administrators - Wikimedia Foundation, which is responsible for policy - BDFL Jimmy Wales who is ultimately responsible for setting policy One of the criticisms of Wikipedia is that it has diverged from its official aim to be the encyclopedia that anyone can edit to one where contributions from insiders, particularly "the Cabal", are preferred to those of new anonymous editors. Other wikis have other policies: Citizendium and Scholarpedia are notable examples that attempt to increase the (real or perceived) reliability and accountability of their articles by prohibiting anonymous edits altogether. Despite the influence of Wikipedia, "wiki" does not mean "open to everyone to edit without supervision". -- Steven D'Aprano From steve at pearwood.info Fri Sep 24 05:40:17 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 24 Sep 2010 13:40:17 +1000 Subject: [Python-Dev] Python wiki In-Reply-To: <20100923115719.2250e776@mission> References: <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> Message-ID: <201009241340.18018.steve@pearwood.info> On Fri, 24 Sep 2010 01:57:19 am Barry Warsaw wrote: > I certainly agree with that. So, how can we solve those problems? > Radomir has shell access now so perhaps we can ask him to make the > Python wiki theme more visually appealing. What roadblocks do people > encounter when they want to help garden or reorganize the wiki? For me, the number one roadblock is unfamiliarity -- I always forget that there is a Python wiki. I think it would be helpful if the wiki could be integrated with the docs in some fashion, perhaps by having each page link to a sister page in the wiki. It seems to me that the wiki won't be useful until people regularly use it, and people won't regularly use it until it is useful. It will take a conscious effort by people (including myself) to break this vicious circle by improving articles and regularly linking to them. Number two, I just went to the wiki to see how I might get started. I randomly choose this page: http://wiki.python.org/moin/BeginnersGuide/Download and looked for an edit button or something. Unlike Wikipedia, there is no obvious Edit button. Eventually I noticed in tiny type at the bottom of the page: "Immutable page". Hmmm, that's not promising. Then I discovered a *tiny* icon that looks like a speech bubble that apparently means "edit". Clicking it gives me a message: "You are not allowed to edit this page." Not friendly, nor helpful. A better message might be something like "This page is locked. You must log in to edit this page." or similar. How does one get an account? Can I edit anonymously? Once I found a page I could edit, it was relatively straightforward. So that's a plus :) -- Steven D'Aprano From stephen at xemacs.org Fri Sep 24 06:59:42 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 24 Sep 2010 13:59:42 +0900 Subject: [Python-Dev] Moving the developer docs? In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> Message-ID: <87eicj3fep.fsf@uwakimon.sk.tsukuba.ac.jp> Georg Brandl writes: > You should read my tweets more often :) Now *there* is an innovation that never should have happened! At least you're bringing up the average quality with every twit I mean tweet. From martin at v.loewis.de Fri Sep 24 07:46:44 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 24 Sep 2010 07:46:44 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> Message-ID: <4C9C3B44.3050401@v.loewis.de> Am 24.09.2010 00:39, schrieb Guido van Rossum: > On Thu, Sep 23, 2010 at 3:35 PM, "Martin v. L?wis" wrote: >>> With an admin team behind it, you can also make more use of ACLs to >>> flag certain parts of the wiki as "official" by making them only >>> editable by certain people (e.g. only devs, only the triage team, only >>> the wiki admins). But keeping those user lists up to date is itself >>> something that requires a strong wiki admin team. >> >> There actually is an admin team, and they actually do set ACLs. > > Who are they? I don't actually know entirely; at a minimum, Skip Montanaro. >> IIUC, this is primarily for spam protection, though. > > So would they object against additions to the team? I don't think they would. Regards, Martin From stephen at xemacs.org Fri Sep 24 07:41:14 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 24 Sep 2010 14:41:14 +0900 Subject: [Python-Dev] Goodbye In-Reply-To: <4C9BC975.5090801@v.loewis.de> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9BC975.5090801@v.loewis.de> Message-ID: <87d3s33dhh.fsf@uwakimon.sk.tsukuba.ac.jp> "Martin v. L?wis" writes: > I didn't say the field does not matter. I said adjusting it doesn't > advance the issue. Anybody *really* working on the issue might > choose to update it, or might choose to leave it incorrect when the > issue is going to be closed, anyway. Yes, and we'd all like more people to do more "real" work. But not everybody has the time or skills. I think this is a case where "agreeing to disagree" is the best we can do. Specifically, Terry has made a strong case that "a few minutes per issue" *can* improve the tracker in ways that *are* noticable to some of the developers (and some of them have acknowledged that). It would be nice if the "tracker trimmers"[1] could assemble 60 of those into a few hours, and do some "real work", but that's often just not possible (especially for people with minimal programming expertise as yet). Footnotes: [1] Trawlers take fish out of the ocean: not really the best metaphor. Gardening is a better metaphor. From g.brandl at gmx.net Fri Sep 24 09:20:53 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 24 Sep 2010 09:20:53 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: <4C9BBDED.5010701@netwok.org> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9BBDED.5010701@netwok.org> Message-ID: Am 23.09.2010 22:51, schrieb ?ric Araujo: > Le 23/09/2010 19:22, Terry Reedy a ?crit : >> As of just now, if you were to wonder "What (security) bugs are open for >> 2.5" and search on open 2.5 issues, you would get a list of 44 issues. >> It is only 44 instead of hundreds because of the work I and Mark have >> done in the last 4 months. It it 44 instead of perhaps 5 because Tarek >> and Eric insist on marking all disutils2 issues for all versions even >> though though these issues have nothing to do with maintenance releases. >> It is a real nuisance when trying to do tracker cleanup. > > Let?s fix this. Two days ago, I said this in > http://bugs.python.org/issue2200#msg117003 : > > distutils2 has to work with 2.4-2.7 and (soon) 3.1-3.2, so Tarek told > me to set all available versions for those bugs (2.5-py3k), even if I > think ?3rd party? is more appropriate and useful (since a distutils2 > bug would not for example block a CPython 3.2 release). > > I?ve been following these rules since before GSoC, when I had less > confidence and no will to conflict with Tarek on such a small thing > . Now I argue that the versions field is not really useful for > d2, since it lacks 2.4 which we support and chiefly because it does not > actually help our workflow: We don?t have separate branches for > backporting fixes, we apply patches and run tests for all supported > versions before committing on a single branch. There is no use case of > setting a version to remind that a port needs to be done. For bug > purposes, I actually see distutils2 as a value for the versions field of > distutils bugs. Thanks Eric, that sounds good. > respect-my-non-ASCII-diversity-write-my-acute-accent-thanks?ly yours, Oops. My bad. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From g.brandl at gmx.net Fri Sep 24 09:24:35 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 24 Sep 2010 09:24:35 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> Message-ID: Am 23.09.2010 22:25, schrieb anatoly techtonik: > On Thu, Sep 23, 2010 at 6:57 PM, Barry Warsaw wrote: >> >> I certainly agree with that. So, how can we solve those problems? Radomir >> has shell access now so perhaps we can ask him to make the Python wiki theme >> more visually appealing. What roadblocks do people encounter when they want >> to help garden or reorganize the wiki? > > First of all Wiki is outdated. Correct me, but python.org specific > customizations are not modules, but patches that makes it hard to > update the Wiki to latest version or add new customizations. That's why we have a Moin core developer on the team. ISTM that Moin 1.x is notoriously hard to extend -- once you go beyond plugins adding new wiki macros -- which is part of what the team wants to fix in 2.x. > Second - ugly Creole syntax. I am for inter-cooperation between wikis, > and I understand that for non-developer communities [] symbols > imposing problems, but as an open source developer I am addicted to > Trac and Google Code syntax and never had problems with those. This isn't Creole syntax, it's Moin wiki syntax. And face it, it's not going to change. It's also not so much different from Trac wiki syntax. > Fourth. GPL license. I personally don't have interest to waste my time > for the code I won't be able to use in some projects, unless the > project is either exceptional (Mercurial) or interesting. To make > python.org MoinMoin interesting - there should be an inviting > entrypoint (see point three above) and the list of tasks to choose > form. Something that is better than > http://wiki.python.org/moin/SiteImprovements Yes, that needs to be updated indeed. > Fifth. Credits and motivation for all python.org works. I still > convinced that there should be one primary dedicated list and it > should be public. All sensitive issues can be discussed with > webmasters@ privately, but the primary list should be run by > community. Not the volunteers who are better than others. If there > will be a feeling that site is run by community, then you can expect > contributions. Otherwise expect the community to think "they're doing > the stuff here, so they will fix it". I'm puzzled what you expect in addition to the pydotorg-www list. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From grflanagan at gmail.com Fri Sep 24 11:04:48 2010 From: grflanagan at gmail.com (Gerard Flanagan) Date: Fri, 24 Sep 2010 10:04:48 +0100 Subject: [Python-Dev] wiki/docs enhancement idea Message-ID: I didn't know who to reply to in the previous thread. (Moving the Developer docs/ Python wiki). scraperwiki.org is a 'site scraper automater'. I threw together a script just now which scrapes certain specified pages from the python wiki and converts to something rest-like. It runs every 24hrs and the idea would be that the result could be included alongside the official docs. This may increase people's motivation to contribute to the wiki. Script here: http://scraperwiki.com/scrapers/pywiki2rest/edit/ Example api call: http://api.scraperwiki.com/api/1.0/datastore/getdata?format=json&name=pywiki2rest Regards From merwok at netwok.org Fri Sep 24 11:07:59 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 24 Sep 2010 11:07:59 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> Message-ID: <4C9C6A6F.6010202@netwok.org> How about revamping the type/versions fields? Issue type () Feature request (blocked by moratorium: () yes () no) () Bug (found in: [] 2.7 [] 3.1 [] py3k) () Security bug (found in: [] 2.5 [] 2.6 [] 2.7 [] 3.1 [] py3k) I?m getting tired of explaining the meaning of the versions field again and again, let?s put this information directly under the eyes of the bug reporter. Regards From chris at simplistix.co.uk Fri Sep 24 11:50:02 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 24 Sep 2010 10:50:02 +0100 Subject: [Python-Dev] url shortening services (was Re: standards for distribution names) In-Reply-To: <20100917115401.GI26366@gmail.com> References: <4C91FACB.3010300@simplistix.co.uk> <20100917115401.GI26366@gmail.com> Message-ID: <4C9C744A.2080202@simplistix.co.uk> On 17/09/2010 12:54, Dan Buch wrote: > You may also find this thread from the packaging google group useful, > although it may not be quite what you're looking for: > > http://bit.ly/96SMuM To echo a please from the main python list, please can we all stop using url shortening services? This isn't twitter, we have no character limits, and as a result the cons by far outweigh any pros... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Fri Sep 24 12:13:46 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 24 Sep 2010 11:13:46 +0100 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: References: <4C9531A7.10405@simplistix.co.uk> Message-ID: <4C9C79DA.7000506@simplistix.co.uk> On 18/09/2010 23:36, Guido van Rossum wrote: > course, exists() and isdir() etc. do, and so does realpath(), but the > pure parsing functions don't. Yes, but: H:\>echo foo > TeSt.txt ...>>> import os.path >>> os.path.realpath('test.txt') 'H:\\test.txt' >>> os.path.normcase('TeSt.txt') 'test.txt' Both feel unsatisfying to me :-S How can I get 'TeSt.txt' from 'test.txt' (which feels like the contract normcase *should* have...) > They can be used without a working > filesystem even. (E.g. you can import ntpath on a Unix box and happily > parse Windows paths.) But what value does that add over just doing a .lower() on the path? Chris From chris at simplistix.co.uk Fri Sep 24 12:19:43 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 24 Sep 2010 11:19:43 +0100 Subject: [Python-Dev] Some changes to logging for 3.2 In-Reply-To: References: Message-ID: <4C9C7B3F.6060807@simplistix.co.uk> On 22/09/2010 16:54, Vinay Sajip wrote: > I'm planning to make some smallish changes to logging in Python 3.2, please see > > http://plumberjack.blogspot.com/2010/09/improved-queuehandler-queuelistener.html > > If you're interested, I'd be grateful for any feedback you can give. Cool, how can I use it in Python 2.6? :-) Chris From fuzzyman at voidspace.org.uk Fri Sep 24 12:31:13 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 24 Sep 2010 11:31:13 +0100 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9C3B44.3050401@v.loewis.de> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> Message-ID: <4C9C7DF1.5040500@voidspace.org.uk> On 24/09/2010 06:46, "Martin v. L?wis" wrote: > Am 24.09.2010 00:39, schrieb Guido van Rossum: >> On Thu, Sep 23, 2010 at 3:35 PM, "Martin v. L?wis" wrote: >>>> With an admin team behind it, you can also make more use of ACLs to >>>> flag certain parts of the wiki as "official" by making them only >>>> editable by certain people (e.g. only devs, only the triage team, only >>>> the wiki admins). But keeping those user lists up to date is itself >>>> something that requires a strong wiki admin team. >>> There actually is an admin team, and they actually do set ACLs. >> Who are they? > I don't actually know entirely; at a minimum, Skip Montanaro. > Wiki maintenance is discussed, along with other python.org maintenance topics, on the pydotorg-www mailing list: http://mail.python.org/mailman/listinfo/pydotorg-www More wiki and website maintainers needed! All the best, Michael Foord >>> IIUC, this is primarily for spam protection, though. >> So would they object against additions to the team? > I don't think they would. > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From nad at acm.org Fri Sep 24 12:40:10 2010 From: nad at acm.org (Ned Deily) Date: Fri, 24 Sep 2010 03:40:10 -0700 Subject: [Python-Dev] Version fields [was Re: Goodbye] References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9C6A6F.6010202@netwok.org> Message-ID: In article <4C9C6A6F.6010202 at netwok.org>, ?ric Araujo wrote: > How about revamping the type/versions fields? > > Issue type > () Feature request (blocked by moratorium: () yes () no) > () Bug (found in: [] 2.7 [] 3.1 [] py3k) > () Security bug (found in: [] 2.5 [] 2.6 [] 2.7 [] 3.1 [] py3k) > > I?m getting tired of explaining the meaning of the versions field again > and again, let?s put this information directly under the eyes of the bug > reporter. I believe there is another separate use case for the versions field that hasn't been mentioned yet: for issues with "release blocker" priority, the versions field is often used to identify the upcoming release for which a resolution is deemed mandatory. However, having a single versions field is not totally satisfactory for this, particularly when - as has happened in the recent past - two different release cycles overlap (say, Python 2.7.x and 3.2.y). A given issue may or may not apply to both, it may be a "release blocker" for one or both, and, if both, a fix may be checked-in for one branch but not the other. The release managers for the two releases may end up using the one priority field and the one set of version fields for conflicting purposes. It certainly makes it more difficult to automate a tracking report of exactly what are the release blocker issues for a specific release. Besides adding fields to the database for an issue, there are other ways to handle the ambiguity, of course. The simplest might be to just open a separate duplicate issue for each additional release blocked. Presumably that level of detail might only be needed in the endgame of a release, beta or rc stages. It still places a restriction on the use of the version field and possibly other fields. In issue workflow documentation, there should be some description of how "release blocker" should work, perhaps including something along the lines of "once a release enters stage , 'release blocker' priority should only be changed with the approval of the release manager". -- Ned Deily, nad at acm.org From solipsis at pitrou.net Fri Sep 24 13:32:03 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 24 Sep 2010 13:32:03 +0200 Subject: [Python-Dev] r84983 - in python/branches/py3k: Doc/library/os.rst Lib/test/test_os.py Misc/NEWS Modules/posixmodule.c References: <20100923200414.A594CEE983@mail.python.org> Message-ID: <20100924133203.28b67ab6@pitrou.net> The getlogin test fails on many Unix buildbots, either with errno 2 (ENOENT) or 22 (EINVAL) or "OSError: unable to determine login name": ====================================================================== ERROR: test_getlogin (test.test_os.LoginTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/python-trunk/3.x.norwitz-x86/build/Lib/test/test_os.py", line 1208, in test_getlogin user_name = os.getlogin() OSError: [Errno 2] No such file or directory ====================================================================== ERROR: test_getlogin (test.test_os.LoginTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home2/buildbot2/slave/3.x.loewis-parallel/build/Lib/test/test_os.py", line 1208, in test_getlogin user_name = os.getlogin() OSError: [Errno 22] Invalid argument ====================================================================== ERROR: test_getlogin (test.test_os.LoginTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home2/buildbot/slave/3.x.loewis-sun/build/Lib/test/test_os.py", line 1208, in test_getlogin user_name = os.getlogin() OSError: unable to determine login name On Thu, 23 Sep 2010 22:04:14 +0200 (CEST) brian.curtin wrote: > Author: brian.curtin > Date: Thu Sep 23 22:04:14 2010 > New Revision: 84983 > > Log: > #9808. Implement os.getlogin for Windows, completed by Jon Anglin. > > The test is semi-dumb, it just makes sure something comes back since we > don't have a solid source to validate the returned login. We can't be 100% > sure that the USERNAME env var will always match what os.getlogin() returns, > so we don't make any specific assertion there. From amauryfa at gmail.com Fri Sep 24 13:38:44 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Fri, 24 Sep 2010 13:38:44 +0200 Subject: [Python-Dev] r84983 - in python/branches/py3k: Doc/library/os.rst Lib/test/test_os.py Misc/NEWS Modules/posixmodule.c In-Reply-To: <20100924133203.28b67ab6@pitrou.net> References: <20100923200414.A594CEE983@mail.python.org> <20100924133203.28b67ab6@pitrou.net> Message-ID: 2010/9/24 Antoine Pitrou : > > The getlogin test fails on many Unix buildbots, either with errno 2 > (ENOENT) or 22 (EINVAL) or "OSError: unable to determine login name": Do these buildbots run in a Windows service, i.e. with no user logged in? -- Amaury Forgeot d'Arc From solipsis at pitrou.net Fri Sep 24 13:49:49 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 24 Sep 2010 13:49:49 +0200 Subject: [Python-Dev] r84983 - in python/branches/py3k: Doc/library/os.rst Lib/test/test_os.py Misc/NEWS Modules/posixmodule.c In-Reply-To: References: <20100923200414.A594CEE983@mail.python.org> <20100924133203.28b67ab6@pitrou.net> Message-ID: <20100924134949.7d105630@pitrou.net> On Fri, 24 Sep 2010 13:38:44 +0200 "Amaury Forgeot d'Arc" wrote: > 2010/9/24 Antoine Pitrou : > > > > The getlogin test fails on many Unix buildbots, either with errno 2 > > (ENOENT) or 22 (EINVAL) or "OSError: unable to determine login name": > > Do these buildbots run in a Windows service, i.e. with no user logged in? I don't think any of our Unix buildbots runs in our Windows service :) The diversity of errors we get is a bit disturbing: in the Linux man pages, ENOENT is mentioned as a glibc extension (?There was no corresponding entry in the utmp-file?) but EINVAL is not mentioned at all; also, returning NULL without setting errno is not a possibility in the POSIX spec. Regards Antoine. From ned at nedbatchelder.com Fri Sep 24 13:58:21 2010 From: ned at nedbatchelder.com (Ned Batchelder) Date: Fri, 24 Sep 2010 07:58:21 -0400 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: <4C9C79DA.7000506@simplistix.co.uk> References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> Message-ID: <4C9C925D.9020506@nedbatchelder.com> On 9/24/2010 6:13 AM, Chris Withers wrote: > On 18/09/2010 23:36, Guido van Rossum wrote: >> course, exists() and isdir() etc. do, and so does realpath(), but the >> pure parsing functions don't. > > Yes, but: > > H:\>echo foo > TeSt.txt > ...>>> import os.path > >>> os.path.realpath('test.txt') > 'H:\\test.txt' > >>> os.path.normcase('TeSt.txt') > 'test.txt' > > Both feel unsatisfying to me :-S > > How can I get 'TeSt.txt' from 'test.txt' (which feels like the > contract normcase *should* have...) > http://stackoverflow.com/questions/3692261/in-python-how-can-i-get-the-correctly-cased-path-for-a-file >> They can be used without a working >> filesystem even. (E.g. you can import ntpath on a Unix box and happily >> parse Windows paths.) > > But what value does that add over just doing a .lower() on the path? > > Chris > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com > From rdmurray at bitdance.com Fri Sep 24 14:17:37 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 24 Sep 2010 08:17:37 -0400 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: <4C9C79DA.7000506@simplistix.co.uk> References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> Message-ID: <20100924121737.309071FA5C2@kimball.webabinitio.net> On Fri, 24 Sep 2010 11:13:46 +0100, Chris Withers wrote: > On 18/09/2010 23:36, Guido van Rossum wrote: > > course, exists() and isdir() etc. do, and so does realpath(), but the > > pure parsing functions don't. > > Yes, but: > > H:\>echo foo > TeSt.txt > ...>>> import os.path > >>> os.path.realpath('test.txt') > 'H:\\test.txt' > >>> os.path.normcase('TeSt.txt') > 'test.txt' > > Both feel unsatisfying to me :-S > > How can I get 'TeSt.txt' from 'test.txt' (which feels like the contract > normcase *should* have...) You can't, and you shouldn't be able to. "normalization" is something that happens without reference to existing objects, the whole point is to put the thing into "standard form" so that you can compare strings obtained from different sources and know that they will represent the same object on that filesystem. > > They can be used without a working > > filesystem even. (E.g. you can import ntpath on a Unix box and happily > > parse Windows paths.) > > But what value does that add over just doing a .lower() on the path? It does what is appropriate for that....oh, yeah. For that OS, not "for that filesystem". (e.g. on Unix normcase does nothing since files with different cases but the same letters are different files.) Being os specific rather than file system type specific is the usability bug. But to fix it we'll need to introduce a 'filesystems' module enumerating the different file systems we support, with tools for figuring out what filesystem your program is talking to. But normacase still, wouldn't (shouldn't) do what you want. -- R. David Murray www.bitdance.com From solipsis at pitrou.net Fri Sep 24 14:26:39 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 24 Sep 2010 14:26:39 +0200 Subject: [Python-Dev] Goodbye References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9C6A6F.6010202@netwok.org> Message-ID: <20100924142639.0e212df3@pitrou.net> On Fri, 24 Sep 2010 11:07:59 +0200 ?ric Araujo wrote: > How about revamping the type/versions fields? > > Issue type > () Feature request (blocked by moratorium: () yes () no) > () Bug (found in: [] 2.7 [] 3.1 [] py3k) > () Security bug (found in: [] 2.5 [] 2.6 [] 2.7 [] 3.1 [] py3k) > > I?m getting tired of explaining the meaning of the versions field again > and again, let?s put this information directly under the eyes of the bug > reporter. But we also have "performance", "crash", "resource usage"... Are we suggesting we devise a separate list box for each of these issue types? Regards Antoine. From brian.curtin at gmail.com Fri Sep 24 15:32:39 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Fri, 24 Sep 2010 08:32:39 -0500 Subject: [Python-Dev] r84983 - in python/branches/py3k: Doc/library/os.rst Lib/test/test_os.py Misc/NEWS Modules/posixmodule.c In-Reply-To: <20100924134949.7d105630@pitrou.net> References: <20100923200414.A594CEE983@mail.python.org> <20100924133203.28b67ab6@pitrou.net> <20100924134949.7d105630@pitrou.net> Message-ID: On Fri, Sep 24, 2010 at 06:49, Antoine Pitrou wrote: > On Fri, 24 Sep 2010 13:38:44 +0200 > "Amaury Forgeot d'Arc" wrote: > > 2010/9/24 Antoine Pitrou : > > > > > > The getlogin test fails on many Unix buildbots, either with errno 2 > > > (ENOENT) or 22 (EINVAL) or "OSError: unable to determine login name": > > > > Do these buildbots run in a Windows service, i.e. with no user logged in? > > I don't think any of our Unix buildbots runs in our Windows service :) > > The diversity of errors we get is a bit disturbing: in the Linux man > pages, ENOENT is mentioned as a glibc extension (?There was no > corresponding entry in the utmp-file?) but EINVAL is not mentioned at > all; also, returning NULL without setting errno is not a possibility in > the POSIX spec. > > Regards > > Antoine. Now, it makes sense why there was no os.getlogin() test in the past. I'll disable the test for the time being. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Sep 24 16:19:03 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 25 Sep 2010 00:19:03 +1000 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9C7DF1.5040500@voidspace.org.uk> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> Message-ID: On Fri, Sep 24, 2010 at 8:31 PM, Michael Foord wrote: > Wiki maintenance is discussed, along with other python.org maintenance > topics, on the pydotorg-www mailing list: > > http://mail.python.org/mailman/listinfo/pydotorg-www > > More wiki and website maintainers needed! We could probably advertise helping with pydotorg itself a bit more in the "How can I help?" sections of the docs and website. I did just notice there is actually a "Help Maintain Website" on the left of the main page which is a good start, but (for example) the general Python FAQ in the docs points people to http://python.org/dev/, which in turn has a "Suggested Tasks" link to http://python.org/dev/contributing/. That page could probably do with a section in the main text that links to the website maintenance info page. That page itself could also mention assisting in wiki maintenance as a way to demonstrate interest (similar to the way patches and triage assistance show interest in contributing to the official docs and source code). For the wiki itself, I would suggest a "Help Maintain the Wiki" link in the left navbar of each (with appropriately updated text, that could just link to the general website maintenance page). So, given that we don't actually *ask* anywhere for people to contribute to the wiki, I guess it isn't that surprising that it is underutilised. Something else the wiki can be *very* useful for is to provide information that is potentially useful to Python users, but that we don't want to be seen as unduly "blessed" by either python-dev or the PSF. Lists of libraries supporting particular tasks can fit into that category, since an entry on an open access wiki page is (justifiably) going to be given far less weight than a reference from the official documentation. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Fri Sep 24 16:25:40 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 25 Sep 2010 00:25:40 +1000 Subject: [Python-Dev] Goodbye In-Reply-To: <20100924142639.0e212df3@pitrou.net> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9C6A6F.6010202@netwok.org> <20100924142639.0e212df3@pitrou.net> Message-ID: On Fri, Sep 24, 2010 at 10:26 PM, Antoine Pitrou wrote: > On Fri, 24 Sep 2010 11:07:59 +0200 > ?ric Araujo wrote: >> How about revamping the type/versions fields? >> >> Issue type >> () Feature request (blocked by moratorium: () yes () no) >> () Bug (found in: [] 2.7 [] 3.1 [] py3k) >> () Security bug (found in: [] 2.5 [] 2.6 [] 2.7 [] 3.1 [] py3k) >> >> I?m getting tired of explaining the meaning of the versions field again >> and again, let?s put this information directly under the eyes of the bug >> reporter. > > But we also have "performance", "crash", "resource usage"... Are we > suggesting we devise a separate list box for each of these issue types? I must admit, I've never actually found much use for those additional options. If I'm flagging a bug I'll nearly always mark it "behaviour", otherwise I'll mark the issue "feature request". The characterisation of "what *kind* of bug is it?" is something that can usually be left until later in the process. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From guido at python.org Fri Sep 24 16:29:40 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 24 Sep 2010 07:29:40 -0700 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: <20100924121737.309071FA5C2@kimball.webabinitio.net> References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> Message-ID: On Fri, Sep 24, 2010 at 5:17 AM, R. David Murray wrote: > On Fri, 24 Sep 2010 11:13:46 +0100, Chris Withers wrote: >> On 18/09/2010 23:36, Guido van Rossum wrote: >> > course, exists() and isdir() etc. do, and so does realpath(), but the >> > pure parsing functions don't. >> >> Yes, but: >> >> H:\>echo foo > TeSt.txt >> ...>>> import os.path >> ?>>> os.path.realpath('test.txt') >> 'H:\\test.txt' >> ?>>> os.path.normcase('TeSt.txt') >> 'test.txt' >> >> Both feel unsatisfying to me :-S >> >> How can I get 'TeSt.txt' from 'test.txt' (which feels like the contract >> normcase *should* have...) > > You can't, and you shouldn't be able to. ?"normalization" is something > that happens without reference to existing objects, the whole point > is to put the thing into "standard form" so that you can compare > strings obtained from different sources and know that they will > represent the same object on that filesystem. Clearly there is another use case where people want to display the filename back to the user with the correct case. This is a reasonable request and I think it makes sense for us to add another API to os.path that does this by looking up the path on the filesystem, or making an OS-specific call. >> > They can be used without a working >> > filesystem even. (E.g. you can import ntpath on a Unix box and happily >> > parse Windows paths.) >> >> But what value does that add over just doing a .lower() on the path? > > It does what is appropriate for that....oh, yeah. ?For that OS, not > "for that filesystem". ?(e.g. on Unix normcase does nothing since files > with different cases but the same letters are different files.) Yeah, which is wrong on Mac OS X -- that's Unix but the default filesystem is case-preserving (though apparently it's possible to mount case-sensitive filesystems too). I've heard that on Windows there are also case-sensitive filesystems (part of a POSIX compliance package?). And on Linux you can mount FAT32 filesystems which are case-preserving. > Being os specific rather than file system type specific is the usability bug. Agreed. > But to fix it we'll need to introduce a 'filesystems' module enumerating > the different file systems we support, with tools for figuring out > what filesystem your program is talking to. ?But normacase still, > wouldn't (shouldn't) do what you want. I don't think we should try to reimplement what the filesystem does. I think we should just ask the filesystem (how exactly I haven't figured out yet but I expect it will be more OS-specific than filesystem-specific). It will have to be a new API -- normcase() at least is *intended* to return a case-flattened name on OSes where case-preserving filesystems are the default, and changing it to look at the filesystem would break too much code. For a new use case we need a new API. -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Fri Sep 24 16:31:00 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 24 Sep 2010 16:31:00 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9C6A6F.6010202@netwok.org> <20100924142639.0e212df3@pitrou.net> Message-ID: <1285338660.3243.1.camel@localhost.localdomain> > > But we also have "performance", "crash", "resource usage"... Are we > > suggesting we devise a separate list box for each of these issue types? > > I must admit, I've never actually found much use for those additional > options. If I'm flagging a bug I'll nearly always mark it "behaviour", > otherwise I'll mark the issue "feature request". The characterisation > of "what *kind* of bug is it?" is something that can usually be left > until later in the process. I have often used searches on "performance" or "resource usage" to find what was needing a review or a patch. I think it would be a mistake to remove those two categories. Regards Antoine. From guido at python.org Fri Sep 24 16:34:51 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 24 Sep 2010 07:34:51 -0700 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9C7DF1.5040500@voidspace.org.uk> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> Message-ID: >>>> There actually is an admin team, and they actually do set ACLs. >>> >>> Who are they? >> >> I don't actually know entirely; at a minimum, Skip Montanaro. On Fri, Sep 24, 2010 at 3:31 AM, Michael Foord wrote: > Wiki maintenance is discussed, along with other python.org maintenance > topics, on the pydotorg-www mailing list: > > http://mail.python.org/mailman/listinfo/pydotorg-www Which has hidden its membership (even to members). Does it really need to appear that secretive? At least the message archive is open and has all the usual suspects. > More wiki and website maintainers needed! Maybe a prominent wiki page with info about how to join the list and the responsibilities / needs would help? Also, I gotta say that the wiki login process is awkward. -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Fri Sep 24 16:35:43 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 24 Sep 2010 16:35:43 +0200 Subject: [Python-Dev] os.path.normcase rationale? References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> Message-ID: <20100924163543.76185666@pitrou.net> On Fri, 24 Sep 2010 07:29:40 -0700 Guido van Rossum wrote: > It will have to be a new API -- normcase() at > least is *intended* to return a case-flattened name on OSes where > case-preserving filesystems are the default, and changing it to look > at the filesystem would break too much code. For a new use case we > need a new API. realpath() sounds like the proper API for that. It just needs to have a better implementation :) Regards Antoine. From ncoghlan at gmail.com Fri Sep 24 16:42:56 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 25 Sep 2010 00:42:56 +1000 Subject: [Python-Dev] Goodbye In-Reply-To: <1285338660.3243.1.camel@localhost.localdomain> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9C6A6F.6010202@netwok.org> <20100924142639.0e212df3@pitrou.net> <1285338660.3243.1.camel@localhost.localdomain> Message-ID: On Sat, Sep 25, 2010 at 12:31 AM, Antoine Pitrou wrote: > >> > But we also have "performance", "crash", "resource usage"... Are we >> > suggesting we devise a separate list box for each of these issue types? >> >> I must admit, I've never actually found much use for those additional >> options. If I'm flagging a bug I'll nearly always mark it "behaviour", >> otherwise I'll mark the issue "feature request". The characterisation >> of "what *kind* of bug is it?" is something that can usually be left >> until later in the process. > > I have often used searches on "performance" or "resource usage" to find > what was needing a review or a patch. I think it would be a mistake to > remove those two categories. That purpose would be served just as well by keywords though (particularly since those attributes aren't mutually exclusive - resource usage problems will usually *cause* performance problems, and you may notice the latter first). A generic "bug" classification would also better suit documentation bugs. The simpler we can make the more common fields, while still providing the essential information, the better. When someone like me is looking at a field pondering what to set it to for a comparatively simple issue report, what hope does someone submitted their first issue have? Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Fri Sep 24 16:50:58 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 24 Sep 2010 16:50:58 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9C6A6F.6010202@netwok.org> <20100924142639.0e212df3@pitrou.net> <1285338660.3243.1.camel@localhost.localdomain> Message-ID: <1285339858.3243.4.camel@localhost.localdomain> Le samedi 25 septembre 2010 ? 00:42 +1000, Nick Coghlan a ?crit : > > > > I have often used searches on "performance" or "resource usage" to find > > what was needing a review or a patch. I think it would be a mistake to > > remove those two categories. > > That purpose would be served just as well by keywords though > (particularly since those attributes aren't mutually exclusive - > resource usage problems will usually *cause* performance problems, and > you may notice the latter first). > > A generic "bug" classification would also better suit documentation > bugs. The simpler we can make the more common fields, while still > providing the essential information, the better. But how should a performance improvement be filed? Bug? Feature request? Or should "feature request" be renamed "improvement"? cheers Antoine. From p.f.moore at gmail.com Fri Sep 24 16:53:56 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 24 Sep 2010 15:53:56 +0100 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> Message-ID: On 24 September 2010 15:29, Guido van Rossum wrote: > I don't think we should try to reimplement what the filesystem does. I > think we should just ask the filesystem (how exactly I haven't figured > out yet but I expect it will be more OS-specific than > filesystem-specific). It will have to be a new API -- normcase() at > least is *intended* to return a case-flattened name on OSes where > case-preserving filesystems are the default, and changing it to look > at the filesystem would break too much code. For a new use case we > need a new API. I dug into this once, and as far as I could tell, it's possible to get the information on Windows, but there's no way on Linux to "ask the filesystem". From my researches, the standard interfaces a filesystem has to implement on Linux don't offer any means of asking this question. Of course, (a) I'm no Linux expert so what do I know, and (b) it may well be possible to come up with a "good enough" solution by ignoring pathologically annoying theoretical cases. I'm happy to provide Windows code if someone needs it. Paul PS There were some places I'd have been glad of this feature (and from what I recall, Mercurial could have used it too) so I'm +1 on the idea. From ncoghlan at gmail.com Fri Sep 24 17:01:35 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 25 Sep 2010 01:01:35 +1000 Subject: [Python-Dev] Goodbye In-Reply-To: <1285339858.3243.4.camel@localhost.localdomain> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9C6A6F.6010202@netwok.org> <20100924142639.0e212df3@pitrou.net> <1285338660.3243.1.camel@localhost.localdomain> <1285339858.3243.4.camel@localhost.localdomain> Message-ID: On Sat, Sep 25, 2010 at 12:50 AM, Antoine Pitrou wrote: > Le samedi 25 septembre 2010 ? 00:42 +1000, Nick Coghlan a ?crit : >> > >> > I have often used searches on "performance" or "resource usage" to find >> > what was needing a review or a patch. I think it would be a mistake to >> > remove those two categories. >> >> That purpose would be served just as well by keywords though >> (particularly since those attributes aren't mutually exclusive - >> resource usage problems will usually *cause* performance problems, and >> you may notice the latter first). >> >> A generic "bug" classification would also better suit documentation >> bugs. The simpler we can make the more common fields, while still >> providing the essential information, the better. > > But how should a performance improvement be filed? Bug? Feature request? > Or should "feature request" be renamed "improvement"? It's a feature request (since we won't backport it unless there is a genuine performance problem being addressed as a bug fix). Whether that warrants changing the name, I don't know. A third option for "other improvement" may also work (since that would also cover things like clarifying doc wording, fixing comments, adjusting code to be more readable/obviously correct, etc). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Fri Sep 24 17:12:11 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 24 Sep 2010 17:12:11 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9C6A6F.6010202@netwok.org> <20100924142639.0e212df3@pitrou.net> <1285338660.3243.1.camel@localhost.localdomain> <1285339858.3243.4.camel@localhost.localdomain> Message-ID: <1285341131.3243.7.camel@localhost.localdomain> > > But how should a performance improvement be filed? Bug? Feature request? > > Or should "feature request" be renamed "improvement"? > > It's a feature request (since we won't backport it unless there is a > genuine performance problem being addressed as a bug fix). Whether > that warrants changing the name, I don't know. I think most people won't intuitively file performance issues as "feature requests", since it doesn't sound like a feature. > A third option for > "other improvement" may also work (since that would also cover things > like clarifying doc wording, fixing comments, adjusting code to be > more readable/obviously correct, etc). But then why not keep a clear categorization of these "other improvements"? By the way, doc wording fixes and cosmetic code changes often get backported. :) cheers Antoine. From status at bugs.python.org Fri Sep 24 18:14:27 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 24 Sep 2010 18:14:27 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-09-17 - 2010-09-24) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues stats: open 2533 (+42) closed 19189 (+57) total 21722 (+53) Open issues with patches: 1061 Issues opened (42) ================== #1763: Winpath module - easy access to Windows directories like My Do http://bugs.python.org/issue1763 reopened by r.david.murray #6627: threading.local() does not work with C-created threads http://bugs.python.org/issue6627 reopened by Nikratio #9441: increase logging handlers test coverage http://bugs.python.org/issue9441 reopened by r.david.murray #9552: ssl build under Windows always rebuilds OpenSSL http://bugs.python.org/issue9552 reopened by ocean-city #9877: Expose sysconfig._get_makefile_filename() in public API http://bugs.python.org/issue9877 reopened by eric.araujo #9889: PyUnicode_FormatV and Py_UNICODE*? http://bugs.python.org/issue9889 opened by mkleehammer #9890: Visual C++ Runtime Library Error is there a fix? http://bugs.python.org/issue9890 opened by Hydro56 #9891: Minor doc typo at datamodel.rst: "list" -> "alist" http://bugs.python.org/issue9891 opened by rbp #9893: Usefulness of the Misc/Vim/ files? http://bugs.python.org/issue9893 opened by pitrou #9896: Introspectable range objects http://bugs.python.org/issue9896 opened by durban #9897: multiprocessing problems http://bugs.python.org/issue9897 opened by hume #9903: test_concurrent_futures writes on stderr http://bugs.python.org/issue9903 opened by pitrou #9904: Cosmetic issues that may warrant a fix in symtable.h/c http://bugs.python.org/issue9904 opened by eli.bendersky #9905: subprocess.Popen fails with stdout=PIPE, stderr=PIPE if standa http://bugs.python.org/issue9905 opened by Thomas.Claveirole #9907: interactive mode TAB does not insert on OS X built with editli http://bugs.python.org/issue9907 opened by ned.deily #9909: request for calendar.dayofyear() function http://bugs.python.org/issue9909 opened by jfinkels #9910: Add Py_SetPath API for embeddint python http://bugs.python.org/issue9910 opened by krisvale #9912: Fail when vsvarsall.bat produces stderr http://bugs.python.org/issue9912 opened by flub #9913: Misc/SpecialBuilds.txt is out of date http://bugs.python.org/issue9913 opened by belopolsky #9914: trace/profile conflict with the use of sys.modules[__name__] http://bugs.python.org/issue9914 opened by belopolsky #9915: speeding up sorting with a key http://bugs.python.org/issue9915 opened by stutzbach #9917: resource max value represented as signed when should be unsign http://bugs.python.org/issue9917 opened by r.david.murray #9919: gdbinit lineno result is one line in excess http://bugs.python.org/issue9919 opened by qpatata #9920: test_cmath on atan fails on AIX http://bugs.python.org/issue9920 opened by sable #9921: os.path.join('x','') behavior http://bugs.python.org/issue9921 opened by rgrig #9922: subprocess.getstatusoutput can fail with utf8 UnicodeDecodeErr http://bugs.python.org/issue9922 opened by debatem1 #9923: mailcap module may not work on non-POSIX platforms if MAILCAPS http://bugs.python.org/issue9923 opened by gnofi #9924: sqlite3 SELECT does not BEGIN a transaction, but should accord http://bugs.python.org/issue9924 opened by zzzeek #9925: Idle doesn't launch http://bugs.python.org/issue9925 opened by Stephan.Bellegy #9926: Wrapped TestSuite subclass does not get __call__ executed http://bugs.python.org/issue9926 opened by loewis #9927: Leak around GetFinalPathNameByHandle (Windows) http://bugs.python.org/issue9927 opened by ocean-city #9929: subprocess.Popen unbuffered not work http://bugs.python.org/issue9929 opened by ocean-city #9931: test_ttk_guionly hangs on XP5 http://bugs.python.org/issue9931 opened by ocean-city #9934: Python Docs Typo http://bugs.python.org/issue9934 opened by Retro #9935: Faster pickling of instances http://bugs.python.org/issue9935 opened by pitrou #9936: trace misreports "missing" lines http://bugs.python.org/issue9936 opened by belopolsky #9937: _winreg.EnumValue causes MemoryError http://bugs.python.org/issue9937 opened by kzmi #9938: Documentation for argparse interactive use http://bugs.python.org/issue9938 opened by jayt #9939: Add a pipe type (FIFO) to the io module http://bugs.python.org/issue9939 opened by pitrou #9940: Strange error reporting with "with" statement http://bugs.python.org/issue9940 opened by pitrou #1724822: provide a shlex.split alternative for Windows shell syntax http://bugs.python.org/issue1724822 reopened by r.david.murray #678250: test_mmap failling on AIX http://bugs.python.org/issue678250 reopened by r.david.murray Most recent 15 issues with no replies (15) ========================================== #9940: Strange error reporting with "with" statement http://bugs.python.org/issue9940 #9938: Documentation for argparse interactive use http://bugs.python.org/issue9938 #9935: Faster pickling of instances http://bugs.python.org/issue9935 #9934: Python Docs Typo http://bugs.python.org/issue9934 #9931: test_ttk_guionly hangs on XP5 http://bugs.python.org/issue9931 #9914: trace/profile conflict with the use of sys.modules[__name__] http://bugs.python.org/issue9914 #9913: Misc/SpecialBuilds.txt is out of date http://bugs.python.org/issue9913 #9909: request for calendar.dayofyear() function http://bugs.python.org/issue9909 #9904: Cosmetic issues that may warrant a fix in symtable.h/c http://bugs.python.org/issue9904 #9903: test_concurrent_futures writes on stderr http://bugs.python.org/issue9903 #9891: Minor doc typo at datamodel.rst: "list" -> "alist" http://bugs.python.org/issue9891 #9889: PyUnicode_FormatV and Py_UNICODE*? http://bugs.python.org/issue9889 #9886: Make operator.itemgetter/attrgetter/methodcaller easier to dis http://bugs.python.org/issue9886 #9883: minidom: AttributeError: DocumentFragment instance has no attr http://bugs.python.org/issue9883 #9861: subprocess module changed exposed attributes http://bugs.python.org/issue9861 Most recent 15 issues waiting for review (15) ============================================= #9936: trace misreports "missing" lines http://bugs.python.org/issue9936 #9935: Faster pickling of instances http://bugs.python.org/issue9935 #9929: subprocess.Popen unbuffered not work http://bugs.python.org/issue9929 #9927: Leak around GetFinalPathNameByHandle (Windows) http://bugs.python.org/issue9927 #9922: subprocess.getstatusoutput can fail with utf8 UnicodeDecodeErr http://bugs.python.org/issue9922 #9919: gdbinit lineno result is one line in excess http://bugs.python.org/issue9919 #9915: speeding up sorting with a key http://bugs.python.org/issue9915 #9912: Fail when vsvarsall.bat produces stderr http://bugs.python.org/issue9912 #9910: Add Py_SetPath API for embeddint python http://bugs.python.org/issue9910 #9909: request for calendar.dayofyear() function http://bugs.python.org/issue9909 #9907: interactive mode TAB does not insert on OS X built with editli http://bugs.python.org/issue9907 #9896: Introspectable range objects http://bugs.python.org/issue9896 #9891: Minor doc typo at datamodel.rst: "list" -> "alist" http://bugs.python.org/issue9891 #9877: Expose sysconfig._get_makefile_filename() in public API http://bugs.python.org/issue9877 #9869: long_subtype_new segfault in pure-Python code http://bugs.python.org/issue9869 Top 10 most discussed issues (10) ================================= #8998: add crypto routines to stdlib http://bugs.python.org/issue8998 39 msgs #9552: ssl build under Windows always rebuilds OpenSSL http://bugs.python.org/issue9552 17 msgs #9630: Reencode filenames when setting the filesystem encoding http://bugs.python.org/issue9630 10 msgs #6627: threading.local() does not work with C-created threads http://bugs.python.org/issue6627 9 msgs #9360: nntplib cleanup http://bugs.python.org/issue9360 8 msgs #9864: email.utils.{parsedate,parsedate_tz} should have better return http://bugs.python.org/issue9864 8 msgs #9939: Add a pipe type (FIFO) to the io module http://bugs.python.org/issue9939 8 msgs #2200: find_executable fails to find .bat files on win32 http://bugs.python.org/issue2200 7 msgs #9583: Document startup option/environment interaction http://bugs.python.org/issue9583 7 msgs #678250: test_mmap failling on AIX http://bugs.python.org/issue678250 7 msgs Issues closed (57) ================== #1142: code sample showing errors reading large files with py 2.5/3.0 http://bugs.python.org/issue1142 closed by amaury.forgeotdarc #1441: Cycles through ob_type aren't freed http://bugs.python.org/issue1441 closed by benjamin.peterson #1458: installer crashes on attempted cancellation http://bugs.python.org/issue1458 closed by BreamoreBoy #1518: Fast globals/builtins access (patch) http://bugs.python.org/issue1518 closed by BreamoreBoy #1532: Refleak run of test_tcl fails http://bugs.python.org/issue1532 closed by BreamoreBoy #1605: Semi autogenerated _types module http://bugs.python.org/issue1605 closed by gvanrossum #1686: string.Template.safe_substitute fail when overriding pattern http://bugs.python.org/issue1686 closed by flox #1794: Hot keys must work in any keyboard layout http://bugs.python.org/issue1794 closed by r.david.murray #1866: const arg for PyInt_FromString http://bugs.python.org/issue1866 closed by ncoghlan #2027: Module containing C implementations of common text algorithms http://bugs.python.org/issue2027 closed by BreamoreBoy #2236: Distutils' mkpath implementation ignoring the "mode" parameter http://bugs.python.org/issue2236 closed by orsenthil #2643: mmap_object_dealloc calls time-consuming msync(), although clo http://bugs.python.org/issue2643 closed by pitrou #8432: buildbot: test_send_signal of test_subprocess failure http://bugs.python.org/issue8432 closed by pitrou #9131: test_set_reprs in test_pprint is fragile http://bugs.python.org/issue9131 closed by stutzbach #9252: PyImport_Import calls __import__ with dummy fromlist http://bugs.python.org/issue9252 closed by brett.cannon #9582: documentation line needs rewording http://bugs.python.org/issue9582 closed by benjamin.peterson #9597: mac: Install 2to3 in /usr/local/bin http://bugs.python.org/issue9597 closed by ned.deily #9786: Native TLS support for pthreads http://bugs.python.org/issue9786 closed by krisvale #9790: ntpath contains imports inside functions http://bugs.python.org/issue9790 closed by brian.curtin #9808: Implement os.getlogin on Windows http://bugs.python.org/issue9808 closed by janglin #9810: bzip2 build sometimes fails (VS8.0) http://bugs.python.org/issue9810 closed by ocean-city #9851: multiprocessing socket timeout will break client http://bugs.python.org/issue9851 closed by jnoller #9865: OrderedDict doesn't implement __sizeof__ http://bugs.python.org/issue9865 closed by rhettinger #9868: test_locale leaves locale changed http://bugs.python.org/issue9868 closed by ocean-city #9875: Garbage output when running setup.py on Windows http://bugs.python.org/issue9875 closed by eric.araujo #9887: distutil's build_scripts doesn't read utf-8 in all locales http://bugs.python.org/issue9887 closed by eric.araujo #9888: int overflow in datetime causes seg fault from datetime.ctime( http://bugs.python.org/issue9888 closed by amaury.forgeotdarc #9892: Event spends less time in wait() than requested http://bugs.python.org/issue9892 closed by pitrou #9894: wrong errno check http://bugs.python.org/issue9894 closed by pitrou #9895: Speed up test_subprocess http://bugs.python.org/issue9895 closed by pitrou #9898: cProfile.runctx doesn't allow sort argument http://bugs.python.org/issue9898 closed by amaury.forgeotdarc #9899: tkinter test_font fails on OS X with Aqua Tk http://bugs.python.org/issue9899 closed by pitrou #9900: Threading Bug or misuse of the api ? http://bugs.python.org/issue9900 closed by Rob.Watson #9901: GIL destruction can fail http://bugs.python.org/issue9901 closed by pitrou #9902: test_undecodable_env failure http://bugs.python.org/issue9902 closed by pitrou #9906: including elementary mathematical functions in default types http://bugs.python.org/issue9906 closed by rhettinger #9908: os.stat() fails on bytes paths under Windows 7 http://bugs.python.org/issue9908 closed by pitrou #9911: doc copyedits http://bugs.python.org/issue9911 closed by georg.brandl #9916: errno module is missing some symbols http://bugs.python.org/issue9916 closed by barry #9918: Installation "make test", two fails if non-ascii path http://bugs.python.org/issue9918 closed by pitrou #9928: weird oddity with bz2 context manager http://bugs.python.org/issue9928 closed by pitrou #9930: Incorrect semantics of __radd__ method for builtin types http://bugs.python.org/issue9930 closed by mark.dickinson #9932: List of sets initialization behavior problems http://bugs.python.org/issue9932 closed by eric.smith #9933: os module does not have the documented EX_NOTFOUND attribute http://bugs.python.org/issue9933 closed by georg.brandl #1672853: Error reading files larger than 4GB http://bugs.python.org/issue1672853 closed by amaury.forgeotdarc #1744752: Newline skipped in "for line in file" for huge file http://bugs.python.org/issue1744752 closed by rhettinger #1327971: HTTPResponse instance has no attribute 'fileno' http://bugs.python.org/issue1327971 closed by orsenthil #1633863: AIX: configure ignores $CC http://bugs.python.org/issue1633863 closed by pitrou #1729930: 2.5.1 latest svn fails test_curses and test_timeout http://bugs.python.org/issue1729930 closed by BreamoreBoy #1739789: Accelerate attr dict lookups http://bugs.python.org/issue1739789 closed by rhettinger #1752919: Exception in HTMLParser for special JavaScript code http://bugs.python.org/issue1752919 closed by BreamoreBoy #1451466: reading very large files http://bugs.python.org/issue1451466 closed by amaury.forgeotdarc #1730136: tkFont.__eq__ gives type error http://bugs.python.org/issue1730136 closed by amaury.forgeotdarc #1706323: Updated ASTVisitor Classes http://bugs.python.org/issue1706323 closed by amaury.forgeotdarc #1732367: Document the constants in the socket module http://bugs.python.org/issue1732367 closed by BreamoreBoy #1675951: Performance for small reads and fix seek problem http://bugs.python.org/issue1675951 closed by pitrou #1708652: Exact matching http://bugs.python.org/issue1708652 closed by rhettinger From brett at python.org Fri Sep 24 20:00:28 2010 From: brett at python.org (Brett Cannon) Date: Fri, 24 Sep 2010 11:00:28 -0700 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> References: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> Message-ID: I think every week where more bugs are closed than opened should be celebrated! =) Thanks to everyone who closed something this week (and to those that filed good bug reports). On Fri, Sep 24, 2010 at 09:14, Python tracker wrote: > > ACTIVITY SUMMARY (2010-09-17 - 2010-09-24) > Python tracker at http://bugs.python.org/ > > To view or respond to any of the issues listed below, click on the issue. > Do NOT respond to this message. > > Issues stats: > ?open ? ?2533 (+42) > ?closed 19189 (+57) > ?total ?21722 (+53) > > Open issues with patches: 1061 > > > Issues opened (42) > ================== > > #1763: Winpath module - easy access to Windows directories like My Do > http://bugs.python.org/issue1763 ?reopened by r.david.murray > > #6627: threading.local() does not work with C-created threads > http://bugs.python.org/issue6627 ?reopened by Nikratio > > #9441: increase logging handlers test coverage > http://bugs.python.org/issue9441 ?reopened by r.david.murray > > #9552: ssl build under Windows always rebuilds OpenSSL > http://bugs.python.org/issue9552 ?reopened by ocean-city > > #9877: Expose sysconfig._get_makefile_filename() in public API > http://bugs.python.org/issue9877 ?reopened by eric.araujo > > #9889: PyUnicode_FormatV and Py_UNICODE*? > http://bugs.python.org/issue9889 ?opened by mkleehammer > > #9890: Visual C++ Runtime Library Error is there a fix? > http://bugs.python.org/issue9890 ?opened by Hydro56 > > #9891: Minor doc typo at datamodel.rst: "list" -> "alist" > http://bugs.python.org/issue9891 ?opened by rbp > > #9893: Usefulness of the Misc/Vim/ files? > http://bugs.python.org/issue9893 ?opened by pitrou > > #9896: Introspectable range objects > http://bugs.python.org/issue9896 ?opened by durban > > #9897: multiprocessing problems > http://bugs.python.org/issue9897 ?opened by hume > > #9903: test_concurrent_futures writes on stderr > http://bugs.python.org/issue9903 ?opened by pitrou > > #9904: Cosmetic issues that may warrant a fix in symtable.h/c > http://bugs.python.org/issue9904 ?opened by eli.bendersky > > #9905: subprocess.Popen fails with stdout=PIPE, stderr=PIPE if standa > http://bugs.python.org/issue9905 ?opened by Thomas.Claveirole > > #9907: interactive mode TAB does not insert on OS X built with editli > http://bugs.python.org/issue9907 ?opened by ned.deily > > #9909: request for calendar.dayofyear() function > http://bugs.python.org/issue9909 ?opened by jfinkels > > #9910: Add Py_SetPath API for embeddint python > http://bugs.python.org/issue9910 ?opened by krisvale > > #9912: Fail when vsvarsall.bat produces stderr > http://bugs.python.org/issue9912 ?opened by flub > > #9913: Misc/SpecialBuilds.txt ?is out of date > http://bugs.python.org/issue9913 ?opened by belopolsky > > #9914: trace/profile conflict with the use of sys.modules[__name__] > http://bugs.python.org/issue9914 ?opened by belopolsky > > #9915: speeding up sorting with a key > http://bugs.python.org/issue9915 ?opened by stutzbach > > #9917: resource max value represented as signed when should be unsign > http://bugs.python.org/issue9917 ?opened by r.david.murray > > #9919: gdbinit lineno result is one line in excess > http://bugs.python.org/issue9919 ?opened by qpatata > > #9920: test_cmath on atan fails on AIX > http://bugs.python.org/issue9920 ?opened by sable > > #9921: os.path.join('x','') behavior > http://bugs.python.org/issue9921 ?opened by rgrig > > #9922: subprocess.getstatusoutput can fail with utf8 UnicodeDecodeErr > http://bugs.python.org/issue9922 ?opened by debatem1 > > #9923: mailcap module may not work on non-POSIX platforms if MAILCAPS > http://bugs.python.org/issue9923 ?opened by gnofi > > #9924: sqlite3 SELECT does not BEGIN a transaction, but should accord > http://bugs.python.org/issue9924 ?opened by zzzeek > > #9925: Idle doesn't launch > http://bugs.python.org/issue9925 ?opened by Stephan.Bellegy > > #9926: Wrapped TestSuite subclass does not get __call__ executed > http://bugs.python.org/issue9926 ?opened by loewis > > #9927: Leak around GetFinalPathNameByHandle (Windows) > http://bugs.python.org/issue9927 ?opened by ocean-city > > #9929: subprocess.Popen unbuffered not work > http://bugs.python.org/issue9929 ?opened by ocean-city > > #9931: test_ttk_guionly hangs on XP5 > http://bugs.python.org/issue9931 ?opened by ocean-city > > #9934: Python Docs Typo > http://bugs.python.org/issue9934 ?opened by Retro > > #9935: Faster pickling of instances > http://bugs.python.org/issue9935 ?opened by pitrou > > #9936: trace misreports "missing" lines > http://bugs.python.org/issue9936 ?opened by belopolsky > > #9937: _winreg.EnumValue causes MemoryError > http://bugs.python.org/issue9937 ?opened by kzmi > > #9938: Documentation for argparse interactive use > http://bugs.python.org/issue9938 ?opened by jayt > > #9939: Add a pipe type (FIFO) to the io module > http://bugs.python.org/issue9939 ?opened by pitrou > > #9940: Strange error reporting with "with" statement > http://bugs.python.org/issue9940 ?opened by pitrou > > #1724822: provide a shlex.split alternative for Windows shell syntax > http://bugs.python.org/issue1724822 ?reopened by r.david.murray > > #678250: test_mmap failling on AIX > http://bugs.python.org/issue678250 ?reopened by r.david.murray > > > > Most recent 15 issues with no replies (15) > ========================================== > > #9940: Strange error reporting with "with" statement > http://bugs.python.org/issue9940 > > #9938: Documentation for argparse interactive use > http://bugs.python.org/issue9938 > > #9935: Faster pickling of instances > http://bugs.python.org/issue9935 > > #9934: Python Docs Typo > http://bugs.python.org/issue9934 > > #9931: test_ttk_guionly hangs on XP5 > http://bugs.python.org/issue9931 > > #9914: trace/profile conflict with the use of sys.modules[__name__] > http://bugs.python.org/issue9914 > > #9913: Misc/SpecialBuilds.txt ?is out of date > http://bugs.python.org/issue9913 > > #9909: request for calendar.dayofyear() function > http://bugs.python.org/issue9909 > > #9904: Cosmetic issues that may warrant a fix in symtable.h/c > http://bugs.python.org/issue9904 > > #9903: test_concurrent_futures writes on stderr > http://bugs.python.org/issue9903 > > #9891: Minor doc typo at datamodel.rst: "list" -> "alist" > http://bugs.python.org/issue9891 > > #9889: PyUnicode_FormatV and Py_UNICODE*? > http://bugs.python.org/issue9889 > > #9886: Make operator.itemgetter/attrgetter/methodcaller easier to dis > http://bugs.python.org/issue9886 > > #9883: minidom: AttributeError: DocumentFragment instance has no attr > http://bugs.python.org/issue9883 > > #9861: subprocess module changed exposed attributes > http://bugs.python.org/issue9861 > > > > Most recent 15 issues waiting for review (15) > ============================================= > > #9936: trace misreports "missing" lines > http://bugs.python.org/issue9936 > > #9935: Faster pickling of instances > http://bugs.python.org/issue9935 > > #9929: subprocess.Popen unbuffered not work > http://bugs.python.org/issue9929 > > #9927: Leak around GetFinalPathNameByHandle (Windows) > http://bugs.python.org/issue9927 > > #9922: subprocess.getstatusoutput can fail with utf8 UnicodeDecodeErr > http://bugs.python.org/issue9922 > > #9919: gdbinit lineno result is one line in excess > http://bugs.python.org/issue9919 > > #9915: speeding up sorting with a key > http://bugs.python.org/issue9915 > > #9912: Fail when vsvarsall.bat produces stderr > http://bugs.python.org/issue9912 > > #9910: Add Py_SetPath API for embeddint python > http://bugs.python.org/issue9910 > > #9909: request for calendar.dayofyear() function > http://bugs.python.org/issue9909 > > #9907: interactive mode TAB does not insert on OS X built with editli > http://bugs.python.org/issue9907 > > #9896: Introspectable range objects > http://bugs.python.org/issue9896 > > #9891: Minor doc typo at datamodel.rst: "list" -> "alist" > http://bugs.python.org/issue9891 > > #9877: Expose sysconfig._get_makefile_filename() in public API > http://bugs.python.org/issue9877 > > #9869: long_subtype_new segfault in pure-Python code > http://bugs.python.org/issue9869 > > > > Top 10 most discussed issues (10) > ================================= > > #8998: add crypto routines to stdlib > http://bugs.python.org/issue8998 ?39 msgs > > #9552: ssl build under Windows always rebuilds OpenSSL > http://bugs.python.org/issue9552 ?17 msgs > > #9630: Reencode filenames when setting the filesystem encoding > http://bugs.python.org/issue9630 ?10 msgs > > #6627: threading.local() does not work with C-created threads > http://bugs.python.org/issue6627 ? 9 msgs > > #9360: nntplib cleanup > http://bugs.python.org/issue9360 ? 8 msgs > > #9864: email.utils.{parsedate,parsedate_tz} should have better return > http://bugs.python.org/issue9864 ? 8 msgs > > #9939: Add a pipe type (FIFO) to the io module > http://bugs.python.org/issue9939 ? 8 msgs > > #2200: find_executable fails to find .bat files on win32 > http://bugs.python.org/issue2200 ? 7 msgs > > #9583: Document startup option/environment interaction > http://bugs.python.org/issue9583 ? 7 msgs > > #678250: test_mmap failling on AIX > http://bugs.python.org/issue678250 ? 7 msgs > > > > Issues closed (57) > ================== > > #1142: code sample showing errors reading large files with py 2.5/3.0 > http://bugs.python.org/issue1142 ?closed by amaury.forgeotdarc > > #1441: Cycles through ob_type aren't freed > http://bugs.python.org/issue1441 ?closed by benjamin.peterson > > #1458: installer crashes on attempted cancellation > http://bugs.python.org/issue1458 ?closed by BreamoreBoy > > #1518: Fast globals/builtins access (patch) > http://bugs.python.org/issue1518 ?closed by BreamoreBoy > > #1532: Refleak run of test_tcl fails > http://bugs.python.org/issue1532 ?closed by BreamoreBoy > > #1605: Semi autogenerated _types module > http://bugs.python.org/issue1605 ?closed by gvanrossum > > #1686: string.Template.safe_substitute fail when overriding ?pattern > http://bugs.python.org/issue1686 ?closed by flox > > #1794: Hot keys must work in any keyboard layout > http://bugs.python.org/issue1794 ?closed by r.david.murray > > #1866: const arg for PyInt_FromString > http://bugs.python.org/issue1866 ?closed by ncoghlan > > #2027: Module containing C implementations of common text algorithms > http://bugs.python.org/issue2027 ?closed by BreamoreBoy > > #2236: Distutils' mkpath implementation ignoring the "mode" parameter > http://bugs.python.org/issue2236 ?closed by orsenthil > > #2643: mmap_object_dealloc calls time-consuming msync(), although clo > http://bugs.python.org/issue2643 ?closed by pitrou > > #8432: buildbot: test_send_signal of test_subprocess failure > http://bugs.python.org/issue8432 ?closed by pitrou > > #9131: test_set_reprs in test_pprint is fragile > http://bugs.python.org/issue9131 ?closed by stutzbach > > #9252: PyImport_Import calls __import__ with dummy fromlist > http://bugs.python.org/issue9252 ?closed by brett.cannon > > #9582: documentation line needs rewording > http://bugs.python.org/issue9582 ?closed by benjamin.peterson > > #9597: mac: Install 2to3 in /usr/local/bin > http://bugs.python.org/issue9597 ?closed by ned.deily > > #9786: Native TLS support for pthreads > http://bugs.python.org/issue9786 ?closed by krisvale > > #9790: ntpath contains imports inside functions > http://bugs.python.org/issue9790 ?closed by brian.curtin > > #9808: Implement os.getlogin on Windows > http://bugs.python.org/issue9808 ?closed by janglin > > #9810: bzip2 build sometimes fails (VS8.0) > http://bugs.python.org/issue9810 ?closed by ocean-city > > #9851: multiprocessing socket timeout will break client > http://bugs.python.org/issue9851 ?closed by jnoller > > #9865: OrderedDict doesn't implement __sizeof__ > http://bugs.python.org/issue9865 ?closed by rhettinger > > #9868: test_locale leaves locale changed > http://bugs.python.org/issue9868 ?closed by ocean-city > > #9875: Garbage output when running setup.py on Windows > http://bugs.python.org/issue9875 ?closed by eric.araujo > > #9887: distutil's build_scripts doesn't read utf-8 in all locales > http://bugs.python.org/issue9887 ?closed by eric.araujo > > #9888: int overflow in datetime causes seg fault from datetime.ctime( > http://bugs.python.org/issue9888 ?closed by amaury.forgeotdarc > > #9892: Event spends less time in wait() than requested > http://bugs.python.org/issue9892 ?closed by pitrou > > #9894: wrong errno check > http://bugs.python.org/issue9894 ?closed by pitrou > > #9895: Speed up test_subprocess > http://bugs.python.org/issue9895 ?closed by pitrou > > #9898: cProfile.runctx doesn't allow sort argument > http://bugs.python.org/issue9898 ?closed by amaury.forgeotdarc > > #9899: tkinter test_font fails on OS X with Aqua Tk > http://bugs.python.org/issue9899 ?closed by pitrou > > #9900: Threading Bug or misuse of the api ? > http://bugs.python.org/issue9900 ?closed by Rob.Watson > > #9901: GIL destruction can fail > http://bugs.python.org/issue9901 ?closed by pitrou > > #9902: test_undecodable_env failure > http://bugs.python.org/issue9902 ?closed by pitrou > > #9906: including elementary mathematical functions in default types > http://bugs.python.org/issue9906 ?closed by rhettinger > > #9908: os.stat() fails on bytes paths under Windows 7 > http://bugs.python.org/issue9908 ?closed by pitrou > > #9911: doc copyedits > http://bugs.python.org/issue9911 ?closed by georg.brandl > > #9916: errno module is missing some symbols > http://bugs.python.org/issue9916 ?closed by barry > > #9918: Installation "make test", two fails if non-ascii path > http://bugs.python.org/issue9918 ?closed by pitrou > > #9928: weird oddity with bz2 context manager > http://bugs.python.org/issue9928 ?closed by pitrou > > #9930: Incorrect semantics of __radd__ method for builtin types > http://bugs.python.org/issue9930 ?closed by mark.dickinson > > #9932: List of sets initialization behavior problems > http://bugs.python.org/issue9932 ?closed by eric.smith > > #9933: os module does not have the documented EX_NOTFOUND attribute > http://bugs.python.org/issue9933 ?closed by georg.brandl > > #1672853: Error reading files larger than 4GB > http://bugs.python.org/issue1672853 ?closed by amaury.forgeotdarc > > #1744752: Newline skipped in "for line in file" for huge file > http://bugs.python.org/issue1744752 ?closed by rhettinger > > #1327971: HTTPResponse instance has no attribute 'fileno' > http://bugs.python.org/issue1327971 ?closed by orsenthil > > #1633863: AIX: configure ignores $CC > http://bugs.python.org/issue1633863 ?closed by pitrou > > #1729930: 2.5.1 latest svn fails test_curses and test_timeout > http://bugs.python.org/issue1729930 ?closed by BreamoreBoy > > #1739789: Accelerate attr dict lookups > http://bugs.python.org/issue1739789 ?closed by rhettinger > > #1752919: Exception in HTMLParser for special JavaScript code > http://bugs.python.org/issue1752919 ?closed by BreamoreBoy > > #1451466: reading very large files > http://bugs.python.org/issue1451466 ?closed by amaury.forgeotdarc > > #1730136: tkFont.__eq__ gives type error > http://bugs.python.org/issue1730136 ?closed by amaury.forgeotdarc > > #1706323: Updated ASTVisitor Classes > http://bugs.python.org/issue1706323 ?closed by amaury.forgeotdarc > > #1732367: Document the constants in the socket module > http://bugs.python.org/issue1732367 ?closed by BreamoreBoy > > #1675951: Performance for small reads and fix seek problem > http://bugs.python.org/issue1675951 ?closed by pitrou > > #1708652: Exact matching > http://bugs.python.org/issue1708652 ?closed by rhettinger > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From tim.peters at gmail.com Fri Sep 24 21:14:32 2010 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 24 Sep 2010 15:14:32 -0400 Subject: [Python-Dev] PyObject_GC_UnTrack() no longer reliable in 2.7? Message-ID: Looks like 2.7 changes introduced to exempt dicts and tuples from cyclic gc if they obviously can't be in cycles has some unintended consequences. Specifically, if an extension module calls PyObject_GC_UnTrack() on a dict it _does not want tracked_, Python can start tracking the dict again. I assume this is unintended because (a) the docs weren't changed to warn about this; and, (b) it's wrong ;-) There are two main reasons an extension module may have been calling PyObject_GC_UnTrack(): 1. For correctness, if the extension is playing games with reference counts Python isn't expecting. 2. For speed, if the extension is creating dicts (or tuples) it knows cannot participate in cycles. This came up when Jim Fulton asked me for advice about assertion failures coming out of cyclic gc in a debug build when running ZODB's tests under 2.7. Turned out to be due to the "#1 reason" above: ZODB hand-rolled its own version of weak references long before Python had them, and has a dict mapping strings ("object IDs") to complex objects where the referenced objects' refcounts intentionally do _not_ account for the references due to this dict. This had been working fine for at least 8 years, thanks to calling PyObject_GC_UnTrack() on this dict right after it's created. But under 2.7, new code in Python apparently decides to track this dict again the first time its __setitem__ is called. Cyclic gc then discovers object references due to this dict that aren't accounted for in the referenced objects' refcounts, and dies early on with an assertion failure (which it should do - the refcounts are nuts as far as Python is concerned). Jim wormed around that for now by calling PyObject_GC_UnTrack() again every time the dict's content changes, but that was relatively easy in this case because the dict is an internal implementation detail all accesses to which are under ZODB's control. Best if no changes had been needed. "Better than nothing" if the docs are changed to warn that the effect of calling PyObject_GC_UnTrack() may be undone by Python a nanosecond later ;-) From vinay_sajip at yahoo.co.uk Fri Sep 24 21:32:41 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 24 Sep 2010 19:32:41 +0000 (UTC) Subject: [Python-Dev] Some changes to logging for 3.2 References: <4C9C7B3F.6060807@simplistix.co.uk> Message-ID: Chris Withers simplistix.co.uk> writes: > > Cool, how can I use it in Python 2.6? > > Chris Hi Chris, 1. Copy the top part (imports, QueueHandler and QueueListener classes) from the Gist linked to in the article - http://gist.github.com/591699 - into a utility module you can use again and again. 2. In your code, try to import them from logging.handlers and in an except ImportError: clause, import them from your utility module. 3. Profit ;-) Note that there's a slight change in the Gist from when the post was published - but it should still work as per the post. Until 3.2 is out, there may be other small changes: but you can copy the code over from the 3.2 branch in the Python SVN repository later and it should work fine under Python 2.6 and 2.7. Of course if you do find any problems (or have any other questions), please let me know asap :-) Regards, Vinay Sajip From solipsis at pitrou.net Fri Sep 24 21:36:00 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 24 Sep 2010 21:36:00 +0200 Subject: [Python-Dev] PyObject_GC_UnTrack() no longer reliable in 2.7? References: Message-ID: <20100924213600.549dc03c@pitrou.net> On Fri, 24 Sep 2010 15:14:32 -0400 Tim Peters wrote: > Looks like 2.7 changes introduced to exempt dicts and tuples from > cyclic gc if they obviously can't be in cycles has some unintended > consequences. Specifically, if an extension module calls > PyObject_GC_UnTrack() on a dict it _does not want tracked_, Python can > start tracking the dict again. > > I assume this is unintended because (a) the docs weren't changed to > warn about this; and, (b) it's wrong ;-) It was indeed unintended. I didn't know people were using PyObject_GC_(Un)Track in other places than constructors and destructors. > There are two main reasons an extension module may have been calling > PyObject_GC_UnTrack(): > > 1. For correctness, if the extension is playing games with reference > counts Python isn't expecting. Yikes :) > 2. For speed, if the extension is creating dicts (or tuples) it knows > cannot participate in cycles. The optimization is now automated in the simple cases (as you've found out!). > This came up when Jim Fulton asked me for advice about assertion > failures coming out of cyclic gc in a debug build when running ZODB's > tests under 2.7. Turned out to be due to the "#1 reason" above: ZODB > hand-rolled its own version of weak references long before Python had > them, and has a dict mapping strings ("object IDs") to complex objects > where the referenced objects' refcounts intentionally do _not_ account > for the references due to this dict. Perhaps ZODB should switch to standard weak references these days? (as a bonus, chances are it will be faster) > Best if no changes had been needed. "Better than nothing" if the docs > are changed to warn that the effect of calling PyObject_GC_UnTrack() > may be undone by Python a nanosecond later ;-) A doc addition will be enough, hopefully. Regards Antoine. From tjreedy at udel.edu Fri Sep 24 21:36:16 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 24 Sep 2010 15:36:16 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: <87d3s33dhh.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9BC975.5090801@v.loewis.de> <87d3s33dhh.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 9/24/2010 1:41 AM, Stephen J. Turnbull wrote: > Yes, and we'd all like more people to do more "real" work. But not > everybody has the time or skills. I think this is a case where > "agreeing to disagree" is the best we can do. There is also the matter of letting people start with something they feel condident with and grow into more complicated tasks. > Specifically, Terry has made a strong case that "a few minutes per > issue" *can* improve the tracker in ways that *are* noticable to some > of the developers (and some of them have acknowledged that). It would > be nice if the "tracker trimmers"[1] could assemble 60 of those into a > few hours, and do some "real work", but that's often just not possible > (especially for people with minimal programming expertise as yet). > > > Footnotes: > [1] Trawlers take fish out of the ocean: not really the best > metaphor. Gardening is a better metaphor. For instance, while 'gardening', I discovered 4! duplicate open issues about the bug created by the difflib.SequenceMatcher heuristic. I consolidated them into one, got intrigued, did some tests with 3.1, read difflib.py, ..., and now have a patch posted written with Eli Bendersky. -- Terry Jan Reedy From g.brandl at gmx.net Fri Sep 24 21:57:51 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 24 Sep 2010 21:57:51 +0200 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: References: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> Message-ID: Is it me, or is the "open" and "closed" count confusing to anyone else? I.e., shouldn't the "total" delta equal the sum of the "open" delta and the "closed" delta? Georg Am 24.09.2010 20:00, schrieb Brett Cannon: > I think every week where more bugs are closed than opened should be > celebrated! =) Thanks to everyone who closed something this week (and > to those that filed good bug reports). > > On Fri, Sep 24, 2010 at 09:14, Python tracker wrote: >> >> ACTIVITY SUMMARY (2010-09-17 - 2010-09-24) >> Python tracker at http://bugs.python.org/ >> >> To view or respond to any of the issues listed below, click on the issue. >> Do NOT respond to this message. >> >> Issues stats: >> open 2533 (+42) >> closed 19189 (+57) >> total 21722 (+53) >> >> Open issues with patches: 1061 From brett at python.org Fri Sep 24 22:00:45 2010 From: brett at python.org (Brett Cannon) Date: Fri, 24 Sep 2010 13:00:45 -0700 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: References: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> Message-ID: On Fri, Sep 24, 2010 at 12:57, Georg Brandl wrote: > Is it me, or is the "open" and "closed" count confusing to anyone else? > I.e., shouldn't the "total" delta equal the sum of the "open" delta and > the "closed" delta? The total delta is a complete count of bugs, while the open and closed deltas can apply to pre-existing bugs, e.g., a bug that was re-opened. -Brett > > Georg > > Am 24.09.2010 20:00, schrieb Brett Cannon: >> I think every week where more bugs are closed than opened should be >> celebrated! =) Thanks to everyone who closed something this week (and >> to those that filed good bug reports). >> >> On Fri, Sep 24, 2010 at 09:14, Python tracker wrote: >>> >>> ACTIVITY SUMMARY (2010-09-17 - 2010-09-24) >>> Python tracker at http://bugs.python.org/ >>> >>> To view or respond to any of the issues listed below, click on the issue. >>> Do NOT respond to this message. >>> >>> Issues stats: >>> ?open ? ?2533 (+42) >>> ?closed 19189 (+57) >>> ?total ?21722 (+53) >>> >>> Open issues with patches: 1061 > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From g.brandl at gmx.net Fri Sep 24 22:04:40 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 24 Sep 2010 22:04:40 +0200 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: References: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> Message-ID: So by opening and closing a bug 5 times within a week, the "open" and "close" counters both go up by 5? That would be stupid. Issues can't be open and closed at the same time. There is a count of open issues at the start of the week, and one at the end of the week. There's a difference between those two counts which in total must sum up to the total difference in issues. If I understand correctly how the counters work, they at least need to be renamed -- they do *not* count open/closed issues, they count openings/closings. Georg Am 24.09.2010 22:00, schrieb Brett Cannon: > On Fri, Sep 24, 2010 at 12:57, Georg Brandl wrote: >> Is it me, or is the "open" and "closed" count confusing to anyone else? >> I.e., shouldn't the "total" delta equal the sum of the "open" delta and >> the "closed" delta? > > The total delta is a complete count of bugs, while the open and closed > deltas can apply to pre-existing bugs, e.g., a bug that was re-opened. > > -Brett > >> >> Georg >> >> Am 24.09.2010 20:00, schrieb Brett Cannon: >>> I think every week where more bugs are closed than opened should be >>> celebrated! =) Thanks to everyone who closed something this week (and >>> to those that filed good bug reports). >>> >>> On Fri, Sep 24, 2010 at 09:14, Python tracker wrote: >>>> >>>> ACTIVITY SUMMARY (2010-09-17 - 2010-09-24) >>>> Python tracker at http://bugs.python.org/ >>>> >>>> To view or respond to any of the issues listed below, click on the issue. >>>> Do NOT respond to this message. >>>> >>>> Issues stats: >>>> open 2533 (+42) >>>> closed 19189 (+57) >>>> total 21722 (+53) >>>> >>>> Open issues with patches: 1061 >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From jim at zope.com Fri Sep 24 22:07:02 2010 From: jim at zope.com (Jim Fulton) Date: Fri, 24 Sep 2010 16:07:02 -0400 Subject: [Python-Dev] PyObject_GC_UnTrack() no longer reliable in 2.7? In-Reply-To: <20100924213600.549dc03c@pitrou.net> References: <20100924213600.549dc03c@pitrou.net> Message-ID: On Fri, Sep 24, 2010 at 3:36 PM, Antoine Pitrou wrote: > On Fri, 24 Sep 2010 15:14:32 -0400 > Tim Peters wrote: >> Looks like 2.7 changes introduced to exempt dicts and tuples from >> cyclic gc if they obviously can't be in cycles has some unintended >> consequences. ?Specifically, if an extension module calls >> PyObject_GC_UnTrack() on a dict it _does not want tracked_, Python can >> start tracking the dict again. >> >> I assume this is unintended because (a) the docs weren't changed to >> warn about this; and, (b) it's wrong ;-) > > It was indeed unintended. I didn't know people were using > PyObject_GC_(Un)Track in other places than constructors and destructors. > >> There are two main reasons an extension module may have been calling >> PyObject_GC_UnTrack(): >> >> 1. For correctness, if the extension is playing games with reference >> counts Python isn't expecting. > > Yikes :) > >> 2. For speed, if the extension is creating dicts (or tuples) it knows >> cannot participate in cycles. > > The optimization is now automated in the simple cases (as you've found > out!). > >> This came up when Jim Fulton asked me for advice about assertion >> failures coming out of cyclic gc in a debug build when running ZODB's >> tests under 2.7. ?Turned out to be due to the "#1 reason" above: ?ZODB >> hand-rolled its own version of weak references long before Python had >> them, and has a dict mapping strings ("object IDs") to complex objects >> where the referenced objects' refcounts intentionally do _not_ account >> for the references due to this dict. > > Perhaps ZODB should switch to standard weak references these days? > (as a bonus, chances are it will be faster) This is the long term plan. Switching is not going to be a small project and not high on the list of priorities. (Actually, ZODB invented its own weakref mechanism after Python had weakrefs, but before weakrefs were subclassable. Using standard weakrefs was deemed too expensive in terms of memory use.) For the record, I don't consider this a Python bug. This corner of ZODB is living on the edge and deserves what it gets. :) I'm just happy the fix was ultimately pretty simple. >> Best if no changes had been needed. ?"Better than nothing" if the docs >> are changed to warn that the effect of calling PyObject_GC_UnTrack() >> may be undone by Python a nanosecond later ;-) > > A doc addition will be enough, hopefully. Absolutely. Jim -- Jim Fulton From martin at v.loewis.de Fri Sep 24 23:09:37 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 24 Sep 2010 23:09:37 +0200 Subject: [Python-Dev] PyObject_GC_UnTrack() no longer reliable in 2.7? In-Reply-To: References: Message-ID: <4C9D1391.20504@v.loewis.de> > I assume this is unintended because (a) the docs weren't changed to > warn about this; and, (b) it's wrong ;-) It seems Jim is happy with (or has at least accepted) the behavior change. Would you still like to see it fixed (or, rather, have the 2.6 state restored)? I think it would be possible to have two versions of _PyGC_REFS_UNTRACKED, one being, say, -5. _PyGC_REFS_UNTRACKED_AND_KEEP_IT_THAT_WAY would be what you get when you call PyObject_GC_UnTrack; the code to do automatic tracking/untracking based on contents would use some other new API (which would be non-public in 2.7.x). Regards, Martin From daniel at stutzbachenterprises.com Fri Sep 24 23:22:15 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Fri, 24 Sep 2010 16:22:15 -0500 Subject: [Python-Dev] PyObject_GC_UnTrack() no longer reliable in 2.7? In-Reply-To: <4C9D1391.20504@v.loewis.de> References: <4C9D1391.20504@v.loewis.de> Message-ID: On Fri, Sep 24, 2010 at 4:09 PM, "Martin v. L?wis" wrote: > I think it would be possible to have two versions of > _PyGC_REFS_UNTRACKED, one being, say, -5. > _PyGC_REFS_UNTRACKED_AND_KEEP_IT_THAT_WAY would be what you get > when you call PyObject_GC_UnTrack; the code to do automatic > tracking/untracking based on contents would use some other > new API (which would be non-public in 2.7.x). > Where would the extra state information be stored? (to distinguish untracked and untracked-and-keep-it-that-way) -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sat Sep 25 00:10:48 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 25 Sep 2010 10:10:48 +1200 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> Message-ID: <4C9D21E8.1080005@canterbury.ac.nz> Paul Moore wrote: > I dug into this once, and as far as I could tell, it's possible to get > the information on Windows, but there's no way on Linux to "ask the > filesystem". Maybe we could use a heuristic such as: 1) Search the directory for an exact match to the name given, return it if found. 2) Look for a match ignoring case. If one is found, test it to see if it refers to the same file as the given path, and if so return it. 3) Otherwise, raise an exception. -- Greg From martin at v.loewis.de Sat Sep 25 00:10:39 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 25 Sep 2010 00:10:39 +0200 Subject: [Python-Dev] PyObject_GC_UnTrack() no longer reliable in 2.7? In-Reply-To: References: <4C9D1391.20504@v.loewis.de> Message-ID: <4C9D21DF.1000208@v.loewis.de> Am 24.09.2010 23:22, schrieb Daniel Stutzbach: > On Fri, Sep 24, 2010 at 4:09 PM, "Martin v. L?wis" > wrote: > > I think it would be possible to have two versions of > _PyGC_REFS_UNTRACKED, one being, say, -5. > _PyGC_REFS_UNTRACKED_AND_KEEP_IT_THAT_WAY would be what you get > when you call PyObject_GC_UnTrack; the code to do automatic > tracking/untracking based on contents would use some other > new API (which would be non-public in 2.7.x). > > > Where would the extra state information be stored? (to distinguish > untracked and untracked-and-keep-it-that-way) As everything else: in gc_refs. Regards, Martin From v+python at g.nevcal.com Sat Sep 25 00:43:22 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 24 Sep 2010 15:43:22 -0700 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: <4C9D21E8.1080005@canterbury.ac.nz> References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> <4C9D21E8.1080005@canterbury.ac.nz> Message-ID: <4C9D298A.3010407@g.nevcal.com> On 9/24/2010 3:10 PM, Greg Ewing wrote: > Paul Moore wrote: > >> I dug into this once, and as far as I could tell, it's possible to get >> the information on Windows, but there's no way on Linux to "ask the >> filesystem". > > Maybe we could use a heuristic such as: > > 1) Search the directory for an exact match to the name given, > return it if found. > > 2) Look for a match ignoring case. If one is found, test it to > see if it refers to the same file as the given path, and if so > return it. > > 3) Otherwise, raise an exception. Hmm. There is no need for the function on a case sensitive file system, because the name had better be spelled with matching case: that is, if it is spelled with non-matching case it is an attempt to reference a non-existent file (or at least a different file). So the API could do the "right thing" for case preserving or case ignoring file systems, but for case sensitive file systems, at most an existence check would be warranted. In other words, the API, should it be created, should be "What is the actual name of the file that matches this if it exists in the filesystem", so the first check is to see if it exists in the file system (this may raise an exception if it doesn't exist), and then if it does, then on those filesystems for which it might be different, obtain the different name. From ben+python at benfinney.id.au Sat Sep 25 01:15:25 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 25 Sep 2010 09:15:25 +1000 Subject: [Python-Dev] =?utf-8?q?os=2Epath_function_for_=E2=80=9Cget_the_re?= =?utf-8?q?al_filename=E2=80=9D_=28was=3A_os=2Epath=2Enormcase_rationale?= =?utf-8?b?Pyk=?= References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> <4C9D21E8.1080005@canterbury.ac.nz> Message-ID: <877hia4tte.fsf_-_@benfinney.id.au> Greg Ewing writes: > Maybe we could use a heuristic such as: Your heuristics seem to assume there will only ever be a maximum of one match, which is false. I present the following example: $ ls foo/ bAr.dat BaR.dat bar.DAT > 1) Search the directory for an exact match to the name given, > return it if found. And what if there are also matches for a case-insensitive search? e.g. searching for ?foo/bar.DAT? in the above example. > 2) Look for a match ignoring case. If one is found, test it to > see if it refers to the same file as the given path, and if so > return it. And what if several matches are found? e.g. searching for ?foo/BAR.DAT? in the above example. > 3) Otherwise, raise an exception. It seems to me this whole thing should be hashed out on ?python-ideas?. -- \ ?In case you haven't noticed, [the USA] are now almost as | `\ feared and hated all over the world as the Nazis were.? ?Kurt | _o__) Vonnegut, 2004 | Ben Finney From tim.peters at gmail.com Sat Sep 25 01:20:22 2010 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 24 Sep 2010 19:20:22 -0400 Subject: [Python-Dev] PyObject_GC_UnTrack() no longer reliable in 2.7? In-Reply-To: <4C9D1391.20504@v.loewis.de> References: <4C9D1391.20504@v.loewis.de> Message-ID: [Tim] >> I assume this is unintended because (a) the docs weren't changed to >> warn about this; and, (b) it's wrong ;-) [Martin v. L?wis] > It seems Jim is happy with (or has at least accepted) the behavior > change. Would you still like to see it fixed (or, rather, have the > 2.6 state restored)? "it's wrong ;-)" meant what it said - the track/untrack APIs weren't intended to be hints Python is free to ignore, they were meant to give the user control over whether and when their objects participated in cyclic gc. It's true that their (by far) most common uses are mandatory, to avoid tracking before a new object is sane, and to untrack again before it becomes insane when it's being torn down, but those were not the only intended uses. That said, the optimization 2.7 introduced probably has value that shouldn't be dismissed either. BTW, if it had taken Jim 1000 lines of new code instead of 2 to worm around the regression in ZODB under Python 2.7, I expect he'd be singing a different tune ;-) I view his experience as akin to the canary in the coal mine, albeit likely a mine with very few miners worldwide. > I think it would be possible to have two versions of > _PyGC_REFS_UNTRACKED, one being, say, -5. > _PyGC_REFS_UNTRACKED_AND_KEEP_IT_THAT_WAY would be what you get > when you call PyObject_GC_UnTrack; the code to do automatic > tracking/untracking based on contents would use some other > new API (which would be non-public in 2.7.x). Looks like a promising idea! gcmodule.c's IS_TRACKED macro would have to change to check both states, and likewise the debug assert in visit_reachable(). From guido at python.org Sat Sep 25 01:22:47 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 24 Sep 2010 16:22:47 -0700 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: <4C9D298A.3010407@g.nevcal.com> References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> <4C9D21E8.1080005@canterbury.ac.nz> <4C9D298A.3010407@g.nevcal.com> Message-ID: I think that, like os.path.realpath(), it should not fail if the file does not exist. Maybe the API could be called os.path.unnormpath(), since it is in a sense the opposite of normpath() (which removes case) ? But I would want to write it so that even on Unix it scans the filesystem, in case the filesystem is case-preserving (like the default fs on OS X). --Guido On Fri, Sep 24, 2010 at 3:43 PM, Glenn Linderman wrote: > ?On 9/24/2010 3:10 PM, Greg Ewing wrote: >> >> Paul Moore wrote: >> >>> I dug into this once, and as far as I could tell, it's possible to get >>> the information on Windows, but there's no way on Linux to "ask the >>> filesystem". >> >> Maybe we could use a heuristic such as: >> >> 1) Search the directory for an exact match to the name given, >> return it if found. >> >> 2) Look for a match ignoring case. If one is found, test it to >> see if it refers to the same file as the given path, and if so >> return it. >> >> 3) Otherwise, raise an exception. > > > Hmm. ?There is no need for the function on a case sensitive file system, > because the name had better be spelled with matching case: that is, if it is > spelled with non-matching case it is an attempt to reference a non-existent > file (or at least a different file). > > So the API could do the "right thing" for case preserving or case ignoring > file systems, but for case sensitive file systems, at most an existence > check would be warranted. > > In other words, the API, should it be created, should be "What is the actual > name of the file that matches this if it exists in the filesystem", so the > first check is to see if it exists in the file system (this may raise an > exception if it doesn't exist), and then if it does, then on those > filesystems for which it might be different, obtain the different name. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From guido at python.org Sat Sep 25 01:24:28 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 24 Sep 2010 16:24:28 -0700 Subject: [Python-Dev] =?windows-1252?q?os=2Epath_function_for_=93get_the_r?= =?windows-1252?q?eal_filename=94_=28was=3A_os=2Epath=2Enormcase_ra?= =?windows-1252?q?tionale=3F=29?= In-Reply-To: <877hia4tte.fsf_-_@benfinney.id.au> References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> <4C9D21E8.1080005@canterbury.ac.nz> <877hia4tte.fsf_-_@benfinney.id.au> Message-ID: I think searching a case-sensitive filename for a case-insensitive match should not be offered as part of os.path. Apps that really want to do things like """There is no file named "README", do you want to use "Readme" instead?""" can write their own inefficient code, thank you. --Guido On Fri, Sep 24, 2010 at 4:15 PM, Ben Finney wrote: > Greg Ewing writes: > >> Maybe we could use a heuristic such as: > > Your heuristics seem to assume there will only ever be a maximum of one > match, which is false. I present the following example: > > ? ?$ ls foo/ > ? ? ? ?bAr.dat ?BaR.dat ?bar.DAT > >> 1) Search the directory for an exact match to the name given, >> return it if found. > > And what if there are also matches for a case-insensitive search? e.g. > searching for ?foo/bar.DAT? in the above example. > >> 2) Look for a match ignoring case. If one is found, test it to >> see if it refers to the same file as the given path, and if so >> return it. > > And what if several matches are found? e.g. searching for ?foo/BAR.DAT? > in the above example. > >> 3) Otherwise, raise an exception. > > It seems to me this whole thing should be hashed out on ?python-ideas?. > > -- > ?\ ? ? ? ? ? ?In case you haven't noticed, [the USA] are now almost as | > ?`\ ? ? feared and hated all over the world as the Nazis were.? ?Kurt | > _o__) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Vonnegut, 2004 | > Ben Finney > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From foom at fuhm.net Sat Sep 25 01:36:42 2010 From: foom at fuhm.net (James Y Knight) Date: Fri, 24 Sep 2010 19:36:42 -0400 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> Message-ID: <026371F8-8933-4371-A2F5-EDF2F7D18DEE@fuhm.net> On Sep 24, 2010, at 10:53 AM, Paul Moore wrote: > On 24 September 2010 15:29, Guido van Rossum wrote: >> I don't think we should try to reimplement what the filesystem does. I >> think we should just ask the filesystem (how exactly I haven't figured >> out yet but I expect it will be more OS-specific than >> filesystem-specific). It will have to be a new API -- normcase() at >> least is *intended* to return a case-flattened name on OSes where >> case-preserving filesystems are the default, and changing it to look >> at the filesystem would break too much code. For a new use case we >> need a new API. > > I dug into this once, and as far as I could tell, it's possible to get > the information on Windows, but there's no way on Linux to "ask the > filesystem". From my researches, the standard interfaces a filesystem > has to implement on Linux don't offer any means of asking this > question. > > Of course, (a) I'm no Linux expert so what do I know, and (b) it may > well be possible to come up with a "good enough" solution by ignoring > pathologically annoying theoretical cases. > > I'm happy to provide Windows code if someone needs it. > Paul An OSX code sketch is available here (summary: call FSPathMakeRef to get an FSRef from a path string, then FSRefMakePath to make it back into a path, which will then have the correct case). And note that it only works if the file actually exists. http://stackoverflow.com/questions/370186/how-do-i-find-the-correct-case-of-a-filename It would indeed be useful to have that be available in Python. James From brett at python.org Sat Sep 25 03:45:01 2010 From: brett at python.org (Brett Cannon) Date: Fri, 24 Sep 2010 18:45:01 -0700 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: References: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> Message-ID: On Fri, Sep 24, 2010 at 13:04, Georg Brandl wrote: > So by opening and closing a bug 5 times within a week, the "open" and > "close" counters both go up by 5? ?That would be stupid. No, as in a bug was re-opened last week and then closed again this week. > > Issues can't be open and closed at the same time. ?There is a count of > open issues at the start of the week, and one at the end of the week. > There's a difference between those two counts which in total must sum > up to the total difference in issues. > > If I understand correctly how the counters work, they at least need to > be renamed -- they do *not* count open/closed issues, they count > openings/closings. Guess the only way to settle this is look at the code, but I don't care enough to bother. =) -Brett > > Georg > > Am 24.09.2010 22:00, schrieb Brett Cannon: >> On Fri, Sep 24, 2010 at 12:57, Georg Brandl wrote: >>> Is it me, or is the "open" and "closed" count confusing to anyone else? >>> I.e., shouldn't the "total" delta equal the sum of the "open" delta and >>> the "closed" delta? >> >> The total delta is a complete count of bugs, while the open and closed >> deltas can apply to pre-existing bugs, e.g., a bug that was re-opened. >> >> -Brett >> >>> >>> Georg >>> >>> Am 24.09.2010 20:00, schrieb Brett Cannon: >>>> I think every week where more bugs are closed than opened should be >>>> celebrated! =) Thanks to everyone who closed something this week (and >>>> to those that filed good bug reports). >>>> >>>> On Fri, Sep 24, 2010 at 09:14, Python tracker wrote: >>>>> >>>>> ACTIVITY SUMMARY (2010-09-17 - 2010-09-24) >>>>> Python tracker at http://bugs.python.org/ >>>>> >>>>> To view or respond to any of the issues listed below, click on the issue. >>>>> Do NOT respond to this message. >>>>> >>>>> Issues stats: >>>>> ?open ? ?2533 (+42) >>>>> ?closed 19189 (+57) >>>>> ?total ?21722 (+53) >>>>> >>>>> Open issues with patches: 1061 >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> http://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org >>> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org > > > -- > Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. > Four shall be the number of spaces thou shalt indent, and the number of thy > indenting shall be four. Eight shalt thou not indent, nor either indent thou > two, excepting that thou then proceed to four. Tabs are right out. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From greg.ewing at canterbury.ac.nz Sat Sep 25 03:55:55 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 25 Sep 2010 13:55:55 +1200 Subject: [Python-Dev] =?utf-8?q?os=2Epath_function_for_=E2=80=9Cget_the_re?= =?utf-8?q?al_filename=E2=80=9D?= In-Reply-To: <877hia4tte.fsf_-_@benfinney.id.au> References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> <4C9D21E8.1080005@canterbury.ac.nz> <877hia4tte.fsf_-_@benfinney.id.au> Message-ID: <4C9D56AB.2060602@canterbury.ac.nz> Ben Finney wrote: > Your heuristics seem to assume there will only ever be a maximum of one > match, which is false. I present the following example: > > $ ls foo/ > bAr.dat BaR.dat bar.DAT There should perhaps be an extra step at the beginning: 0) Test whether the specified path refers to an existing file. If not, raise an exception. If that passes, and the file system is case-sensitive, then there must be a directory entry that is an exact match, so it will be returned by step 1. If the file system is case-insensitive, then there can be at most one entry that matches except for case, and it must be the one we're looking for, so there is no need for the extra test in step 2. So the revised algorithm is: 0) Test whether the specified path refers to an existing file. If not, raise an exception. 1) Search the directory for an exact match, return it if found. 2) Search for a match ignoring case, and return one if found. 3) Otherwise, raise an exception. There's also some prior art that might be worth looking at: On Windows, Python checks to see whether the file name of an imported module has the same case as the name being imported, which is a similar problem in some ways. > It seems to me this whole thing should be hashed out on ?python-ideas?. Good point -- I've redirected the discussion there. -- Greg From greg.ewing at canterbury.ac.nz Sat Sep 25 03:56:06 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 25 Sep 2010 13:56:06 +1200 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> <4C9D21E8.1080005@canterbury.ac.nz> <4C9D298A.3010407@g.nevcal.com> Message-ID: <4C9D56B6.9050908@canterbury.ac.nz> Guido van Rossum wrote: > Maybe the API could be called os.path.unnormpath(), since it is in a > sense the opposite of normpath() (which removes case) ? Cute, but not very intuitive. Something like actualpath() might be better -- although that's somewhat arbitrarily different from realpath(). -- Greg From steve at pearwood.info Sat Sep 25 05:25:26 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 25 Sep 2010 13:25:26 +1000 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: References: <4C9531A7.10405@simplistix.co.uk> <4C9D298A.3010407@g.nevcal.com> Message-ID: <201009251325.28749.steve@pearwood.info> On Sat, 25 Sep 2010 09:22:47 am Guido van Rossum wrote: > I think that, like os.path.realpath(), it should not fail if the file > does not exist. > > Maybe the API could be called os.path.unnormpath(), since it is in a > sense the opposite of normpath() (which removes case) ? But I would > want to write it so that even on Unix it scans the filesystem, in > case the filesystem is case-preserving (like the default fs on OS X). It is not entirely clear to me what this function is meant to actually do? Should it: 1. Return the case of a filename in some canonical form which depends on the file system? 2. Return the case of a filename as it is actually stored on disk? 3. Something else? and just for completeness: 4. Return the case of a filename in some arbitrarily-chosen canonical form which does not depend on the file system? These are not the same, either conceptually or in practice. If you want #4, you already have it in os.path.normcase. I think that the OP, Chris, wants #1, but it isn't entirely clear to me. It's possible that he wants #2. Various people have posted links to recipes that solve case #2. Note though that this necessarily demands that if the file doesn't exist, it should raise an exception. In the case of #1, if the file system doesn't exist, we can't predict what the canonical form should be. The very concept of canonical form for file names is troublesome. If the file system is case-preserving, the file system doesn't define a canonical form: the case of the file name will depend on how the file is initially named. If the file system is case-destructive the behaviour will depend on the file system itself: e.g. FAT12 and ISO 9660 both uppercase file names, but other file systems may make other choices. For some arbitrary path, where we don't know what file system it is, or if the path doesn't actually exist, we have no way of telling what the file system's canonical form will be, or even whether it will have one. Note that I've been talking about case preservation, not case sensitivity. That's because case preservation is orthogonal to sensitivity. You can see three of the four combinations, e.g.: Preserving + insensitive: fat32, NTFS under Win32, normally HFS+ Preserving + sensitive: ext3, NTFS under POSIX, optionally HFS+ Destructive + insensitive: fat12, fat16 without long file name support To the best of my knowledge, destructive + sensitive doesn't exist. It could, in principle, but it would be silly to do so. Note that just knowing the file system type is not enough to tell what its behaviour will be. Given an arbitrary file system, there's no obvious way to determine what it will do to file names short of trying to create a file and see what happens. -- Steven D'Aprano From techtonik at gmail.com Sat Sep 25 08:31:16 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Sat, 25 Sep 2010 09:31:16 +0300 Subject: [Python-Dev] Python development tools (Was: Goodbye) Message-ID: On Wed, Sep 22, 2010 at 1:47 PM, Antoine Pitrou wrote: > > Simply, situations like the above (Mark closing a bug just because > nobody would answer his message on a short delay) have happened > multiple times - despite people opposing, obviously -, I must say that the same attitude is present in meta tracker proposals as well. People, who "need a closure" quite often tend to mark issues (enhancement proposals) as won't fix just because they can't (or don't want) to see solution or just because they think that solution is too complicated to implement. They judge it from the point of annoyed developer even though the feature can actually be handy for users of the tracker. The reason here could be that developers are lost in the amount of issues and can't filter them and see who does what. For example, I could triage issues for some modules I am interested in, but I can't. So, to overcome the complexity of global bug space developers are tempted to squash confusing issues. There can be only one solution - concentrate on the tools. Add an "omnibox" to tracker that will use autocompletion and labels (Google Code). Useful ticket query interface (Trac). Favorite issues (Google stars). Personal tags, issue sets, message filters (Slashdot). I've already brought the issue about necessity to enhance Python tools some six months ago. At that time Richard Leland promised to do a research of Python process to further work on proposal to improve it. This was due to June, but still no result. I should blame myself for waiting, because I had some plans to propose, but at that time it was impossible to get going, because everybody had big hopes for that research. > There was a whole python-dev thread some time (weeks? months?) ago where > several of us already tried to suggest more fruitful ways of > contributing, suggestions which weren't received very welcomingly AFAIR. This will never give any results if you did not collecting the outcomes, summaries and links for the future. My vision is that other fruitful ways of contributing are not fun. The most "modern" tool with more or less sane gameplay here is tracker. But it suxx in many ways. Mostly because it is too old and too few are able to patch it due to either lack of information or experience (or time to get one or another). Mercurial is insane, and there isn't really anything else (except Sphinx). Well, there are lists, but there is nothing you can really do about them (except prevent Mailman from dropping archives from time to time or setup a Google Groups mirror). -- anatoly t. From techtonik at gmail.com Sat Sep 25 10:20:22 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Sat, 25 Sep 2010 11:20:22 +0300 Subject: [Python-Dev] Python wiki In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> Message-ID: On Fri, Sep 24, 2010 at 1:27 AM, Nick Coghlan wrote: >> >> ? ?Antoine> Given that few or none of us seem to (want to) actively >> ? ?Antoine> contribute to the wiki, I'm afraid python-dev is not the place >> ? ?Antoine> to ask. ?Perhaps a call should be issued on c.l.py ... >> >> It would be nice if you could actually send messages to the people who do >> actually update wiki content. ?Unfortunately, without donning my cape and >> blue tights, then digging into the users files on the wiki there's no real >> way to do that. That's bad. I'd really like to see the amount of my contributions so far. How about recording a session on MoinMoin hacking and drafting an upgrade plan? Who's gonna be the driver? > That's a good point actually... why *isn't* there a pydotorg-wiki-sig? > (Aside from the obvious point of nobody ever asking for one). Because Yet Another Mailing List doesn't solve the problem. If you need one - go Google Groups like packaging folks did. Python ML are: 1. require dedicated admin to update, who is not a member of the group 2. don't have search 3. don't have optional thread subscription That's already enough to seek better platform for collaboration. > I must admit, that the various things I've thrown on there myself have > been pretty much fire-and-forget. Without anyone that feels like they > collectively "own" the wiki, the much needed pruning never happens. Community can perfectly manage the stuff without dedicated admins if there is a gameplay in it. I am doing the wiki works when I am redirected to outdated wiki pages from search. But I do it only if it doesn't take me more than 5 minutes, and if I can remember the password (and I know where an edit button is). My advice - subscribe people editing page by default (a checkbox near submit button). This way more people will receive notifications when a page is changed and will be more interested to contribute themselves. Of course, there must be a setting to opt out. > With an admin team behind it, you can also make more use of ACLs to > flag certain parts of the wiki as "official" by making them only > editable by certain people (e.g. only devs, only the triage team, only > the wiki admins). But keeping those user lists up to date is itself > something that requires a strong wiki admin team. That's a dead way. Wiki should be open for everyone. Just need more people subscribed to revert unwelcome changes. That is to make timeline more visible, because on wiki.python.org it is _not_ intuitive. It may worth to see how Mercurial wiki is managed - I've picked up the bookmarks monitoring habit from it. Maybe a design change will help, but again - there is no entrypoint for people with design skills to start. A lot of problems. All on the surface. Mailing list won't help. What's next? -- anatoly t. From p.f.moore at gmail.com Sat Sep 25 12:47:28 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 25 Sep 2010 11:47:28 +0100 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: <4C9D298A.3010407@g.nevcal.com> References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> <4C9D21E8.1080005@canterbury.ac.nz> <4C9D298A.3010407@g.nevcal.com> Message-ID: On 24 September 2010 23:43, Glenn Linderman wrote: > Hmm. ?There is no need for the function on a case sensitive file system, > because the name had better be spelled with matching case: that is, if it is > spelled with non-matching case it is an attempt to reference a non-existent > file (or at least a different file). On Linux, I don't believe there's a way to ask "is this filesystem case insensitive?" In fact, with userfs, I believe it's possible to do massively pathological things like having a filesystem which treats anagrams as the same file (foo is the same file as oof or ofo). (More realistically, MacOS does Unicode normalisation). Windows has (I believe) user definable filesystems, too, but the OS has "get me the real filename" style calls, which the filesystem should support, so no matter how nasty a filesystem implementer gets, he has to deal with his own mess :-) Paul. From g.brandl at gmx.net Sat Sep 25 13:11:41 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 25 Sep 2010 13:11:41 +0200 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: References: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> Message-ID: Am 25.09.2010 03:45, schrieb Brett Cannon: > On Fri, Sep 24, 2010 at 13:04, Georg Brandl wrote: >> So by opening and closing a bug 5 times within a week, the "open" and >> "close" counters both go up by 5? That would be stupid. > > No, as in a bug was re-opened last week and then closed again this week. > >> >> Issues can't be open and closed at the same time. There is a count of >> open issues at the start of the week, and one at the end of the week. >> There's a difference between those two counts which in total must sum >> up to the total difference in issues. >> >> If I understand correctly how the counters work, they at least need to >> be renamed -- they do *not* count open/closed issues, they count >> openings/closings. > > Guess the only way to settle this is look at the code, but I don't > care enough to bother. =) I'll bother Ezio when he's back. It just feels strange to me that the bit of statistic I feel is most interesting -- whether there are less open bugs at the end of the week than at the start -- is not obvious from the report. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From martin at v.loewis.de Sat Sep 25 14:10:47 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 25 Sep 2010 14:10:47 +0200 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: References: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> Message-ID: <4C9DE6C7.5040909@v.loewis.de> >> Guess the only way to settle this is look at the code, but I don't >> care enough to bother. =) > > I'll bother Ezio when he's back. It just feels strange to me that the bit > of statistic I feel is most interesting -- whether there are less open bugs > at the end of the week than at the start -- is not obvious from the report. The total numbers reported are really the totals. Also, the delta reported for the totals is the difference to the last report. The number reported with +/- for open/closed are *not* deltas, but the number of issues opened since last week. As some open issues were closed and some closed issues were opened, they don't sum up the way you expect. An example: old: open: #1 #2 closed: #3 #4 new: open: #1 #3 #5 closed: #2 #4 The report would be open: 3 (+2, namely #3 and #5); delta would be +1 closed: 2 (+1, namely #4); delta would be 0 IOW, the numbers after +/- match the counts in the lists shown below, not the delta since last week. HTH, Martin From techtonik at gmail.com Sat Sep 25 14:22:32 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Sat, 25 Sep 2010 15:22:32 +0300 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9C7DF1.5040500@voidspace.org.uk> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> Message-ID: On Fri, Sep 24, 2010 at 1:31 PM, Michael Foord wrote: > > More wiki and website maintainers needed! That's the consequence. You need to seek the reason why there are no maintainers or active members on pydotorg-www lists. I've expressed my thoughts earlier. On Fri, Sep 24, 2010 at 6:40 AM, Steven D'Aprano wrote: > > For me, the number one roadblock is unfamiliarity -- I always forget > that there is a Python wiki. For me a major annoyance is the empty page with two links on wiki.python.org While it allows to tell new people that there is also a Jython wiki, my vision that it should be instead be oriented on existing contributors immediately providing instruments to work with Python wiki. So if smb. need Jython wiki - it should be moved to wiki.jython.org I'd start from there. Who can do this? -- anatoly t. From g.brandl at gmx.net Sat Sep 25 14:22:06 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 25 Sep 2010 14:22:06 +0200 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: <4C9DE6C7.5040909@v.loewis.de> References: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> <4C9DE6C7.5040909@v.loewis.de> Message-ID: Am 25.09.2010 14:10, schrieb "Martin v. L?wis": >>> Guess the only way to settle this is look at the code, but I don't >>> care enough to bother. =) >> >> I'll bother Ezio when he's back. It just feels strange to me that the bit >> of statistic I feel is most interesting -- whether there are less open bugs >> at the end of the week than at the start -- is not obvious from the report. > > The total numbers reported are really the totals. Also, the delta > reported for the totals is the difference to the last report. > > The number reported with +/- for open/closed are *not* deltas, but the > number of issues opened since last week. As some open issues were closed > and some closed issues were opened, they don't sum up the way you > expect. An example: > > old: > open: #1 #2 > closed: #3 #4 > new: > open: #1 #3 #5 > closed: #2 #4 > > The report would be > > open: 3 (+2, namely #3 and #5); delta would be +1 > closed: 2 (+1, namely #4); delta would be 0 > > IOW, the numbers after +/- match the counts in the lists shown below, > not the delta since last week. Yes, and that's what I complained about. However, your example doesn't demonstrate my problem, since its deltas *are* real deltas, and +1 + +0 == +1. I guess a better example would be old: open #1 #2 closed #3 new: open #1 closed #2 #3 #4 #5 which results in +2 for open (since #4 and #5 were opened) and +3 for closed (since #2, #4 and #5 were closed), however the total issue delta is +2. This is why I think these numbers should be labeled "openings" and "closings". Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From techtonik at gmail.com Sat Sep 25 15:15:12 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Sat, 25 Sep 2010 16:15:12 +0300 Subject: [Python-Dev] Relative imports in Py3k Message-ID: Hi, I wonder if situation with relative imports in packages is improved in Python 3k or we are still doomed to a chain of hacks? My user story: I am currently debugging project, which consists of many modules in one package. Each module has tests or other useful stuff for debug in its main section, but it is a disaster to use it, because I can't just execute the module file and expect it to find relatives. All imports are like: from spyderlib.config import get_icon from spyderlib.utils.qthelpers import translate, add_actions, create_action PEP 328 http://www.python.org/dev/peps/pep-0328/ proposes: from ... import config from ..utils.qthelpers import translate, add_actions, create_action But this doesn't work, and I couldn't find any short user level explanation why it is not possible to make this work at least in Py3k without additional magic. -- anatoly t. From dstanek at dstanek.com Sat Sep 25 15:15:53 2010 From: dstanek at dstanek.com (David Stanek) Date: Sat, 25 Sep 2010 09:15:53 -0400 Subject: [Python-Dev] Python wiki In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> Message-ID: On Sat, Sep 25, 2010 at 8:22 AM, anatoly techtonik wrote: > > For me a major annoyance is the empty page with two links on wiki.python.org > While it allows to tell new people that there is also a Jython wiki, > my vision that it should be instead be oriented on existing > contributors immediately providing instruments to work with Python > wiki. So if smb. need Jython wiki - it should be moved to > wiki.jython.org > That's funny, I've never seen that page before. Does it get linked to from somewhere? -- David blog: http://www.traceback.org twitter: http://twitter.com/dstanek From g.brandl at gmx.net Sat Sep 25 15:52:10 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 25 Sep 2010 15:52:10 +0200 Subject: [Python-Dev] Relative imports in Py3k In-Reply-To: References: Message-ID: Am 25.09.2010 15:15, schrieb anatoly techtonik: > Hi, > > I wonder if situation with relative imports in packages is improved in > Python 3k or we are still doomed to a chain of hacks? > > My user story: > I am currently debugging project, which consists of many modules in one package. > Each module has tests or other useful stuff for debug in its main > section, but it is a > disaster to use it, because I can't just execute the module file and > expect it to find > relatives. All imports are like: > > from spyderlib.config import get_icon > from spyderlib.utils.qthelpers import translate, add_actions, create_action > > PEP 328 http://www.python.org/dev/peps/pep-0328/ proposes: > > from ... import config > from ..utils.qthelpers import translate, add_actions, create_action > > But this doesn't work, and I couldn't find any short user level > explanation why it is > not possible to make this work at least in Py3k without additional magic. Uh, "this doesn't work" is a report that every developer loves. I suppose Python does raise an exception with a message? For diagnosing this, you should also at least include from which module you're trying to executing these import statements. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From g.brandl at gmx.net Sat Sep 25 15:53:31 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 25 Sep 2010 15:53:31 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> Message-ID: Am 25.09.2010 15:15, schrieb David Stanek: > On Sat, Sep 25, 2010 at 8:22 AM, anatoly techtonik wrote: >> >> For me a major annoyance is the empty page with two links on wiki.python.org >> While it allows to tell new people that there is also a Jython wiki, >> my vision that it should be instead be oriented on existing >> contributors immediately providing instruments to work with Python >> wiki. So if smb. need Jython wiki - it should be moved to >> wiki.jython.org >> > > That's funny, I've never seen that page before. Does it get linked to > from somewhere? It's at , and FTR, it has annoyed me as well. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From eric at trueblade.com Sat Sep 25 15:58:01 2010 From: eric at trueblade.com (Eric Smith) Date: Sat, 25 Sep 2010 09:58:01 -0400 Subject: [Python-Dev] Relative imports in Py3k In-Reply-To: References: Message-ID: <4C9DFFE9.5040501@trueblade.com> On 9/25/2010 9:15 AM, anatoly techtonik wrote: > from ... import config > from ..utils.qthelpers import translate, add_actions, create_action > > But this doesn't work, and I couldn't find any short user level > explanation why it is > not possible to make this work at least in Py3k without additional magic. This doesn't belong on python-dev, as it's not about developing python. Also, it's a horrible bug report, if that's what it is. I'd suggest trying python-list and reading something like http://itscommonsensestupid.blogspot.com/2008/07/tips-to-write-good-bug-report.html Eric. From stephen at xemacs.org Sat Sep 25 15:58:34 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 25 Sep 2010 22:58:34 +0900 Subject: [Python-Dev] Goodbye In-Reply-To: <4C9C6A6F.6010202@netwok.org> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9C6A6F.6010202@netwok.org> Message-ID: <87vd5t2ad1.fsf@uwakimon.sk.tsukuba.ac.jp> ?ric Araujo writes: > How about revamping the type/versions fields? > > Issue type () Feature request (blocked by moratorium: () yes () no) I think the information about "blocked by moratorium" is not something that users or devs will care about much. The users can be informed about the fact of the moratorium ("There is currently a feature moratorium in effect. If this feature is determined to be a change in the language, it will not appear until Python 3.4 or later. Changes to the standard library functions are acceptable. If no progress is made on the issue in a reasonable time, you may request discussion on python-dev at python.org.") in the reply page that confirms receipt of the issue. However, devs already know about the moratorium, and if there's a question of interpretation it will be discussed on python-dev. Users only care that their request is (not) being addressed; the moratorium is only one of many reasons why action is delayed. From stephen at xemacs.org Sat Sep 25 16:13:27 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 25 Sep 2010 23:13:27 +0900 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> <4C9D21E8.1080005@canterbury.ac.nz> <4C9D298A.3010407@g.nevcal.com> Message-ID: <87tyld29o8.fsf@uwakimon.sk.tsukuba.ac.jp> Paul Moore writes: > In fact, with userfs, I believe it's possible to do massively > pathological things like having a filesystem which treats anagrams > as the same file (foo is the same file as oof or ofo). (More > realistically, MacOS does Unicode normalisation). Nitpick: Mac OS X doesn't do Unicode normalization. The default filesystem implementation does. From guido at python.org Sat Sep 25 16:45:38 2010 From: guido at python.org (Guido van Rossum) Date: Sat, 25 Sep 2010 07:45:38 -0700 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: <201009251325.28749.steve@pearwood.info> References: <4C9531A7.10405@simplistix.co.uk> <4C9D298A.3010407@g.nevcal.com> <201009251325.28749.steve@pearwood.info> Message-ID: On Fri, Sep 24, 2010 at 8:25 PM, Steven D'Aprano wrote: > On Sat, 25 Sep 2010 09:22:47 am Guido van Rossum wrote: > >> I think that, like os.path.realpath(), it should not fail if the file >> does not exist. >> >> Maybe the API could be called os.path.unnormpath(), since it is in a >> sense the opposite of normpath() (which removes case) ? But I would >> want to write it so that even on Unix it scans the filesystem, in >> case the filesystem is case-preserving (like the default fs on OS X). > > It is not entirely clear to me what this function is meant to actually > do? Should it: > > 1. Return the case of a filename in some canonical form which depends > ? on the file system? > 2. Return the case of a filename as it is actually stored on disk? This one. This is actually useful (on case-preserving filesystems). There is no doubt in my mind that this is the requested and needed functionality. > 3. Something else? > > and just for completeness: > > 4. Return the case of a filename in some arbitrarily-chosen canonical > ? form which does not depend on the file system? > > These are not the same, either conceptually or in practice. > > If you want #4, you already have it in os.path.normcase. > > I think that the OP, Chris, wants #1, but it isn't entirely clear to me. I don't think this is where the issue lies. > It's possible that he wants #2. > > Various people have posted links to recipes that solve case #2. Note > though that this necessarily demands that if the file doesn't exist, it > should raise an exception. No it needn't; realpath() uses the filesystem but leaves non-existing parts alone. Also some of the path may exist (e.g. a parent directory). > In the case of #1, if the file system doesn't exist, we can't predict > what the canonical form should be. > > The very concept of canonical form for file names is troublesome. If the > file system is case-preserving, the file system doesn't define a > canonical form: the case of the file name will depend on how the file > is initially named. If the file system is case-destructive the > behaviour will depend on the file system itself: e.g. FAT12 and ISO > 9660 both uppercase file names, but other file systems may make other > choices. For some arbitrary path, where we don't know what file system > it is, or if the path doesn't actually exist, we have no way of telling > what the file system's canonical form will be, or even whether it will > have one. > > Note that I've been talking about case preservation, not case > sensitivity. That's because case preservation is orthogonal to > sensitivity. You can see three of the four combinations, e.g.: > > Preserving + insensitive: ?fat32, NTFS under Win32, normally HFS+ > Preserving + sensitive: ?ext3, NTFS under POSIX, optionally HFS+ > Destructive + insensitive: ?fat12, fat16 without long file name support > > To the best of my knowledge, destructive + sensitive doesn't exist. It > could, in principle, but it would be silly to do so. > > Note that just knowing the file system type is not enough to tell what > its behaviour will be. Given an arbitrary file system, there's no > obvious way to determine what it will do to file names short of trying > to create a file and see what happens. This operation should not do any writes. The solution may well be OS specific. Solutions for Windows and OS X have already been pointed out. If it can't be done for other Unix versions, I think returning the input unchanged on those platform is a fine fallback (as it is for non-existent filenames). -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Sat Sep 25 18:42:50 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 25 Sep 2010 18:42:50 +0200 Subject: [Python-Dev] r85000 - python/branches/py3k/Lib/test/test_trace.py References: <20100924220423.14A22EEB87@mail.python.org> Message-ID: <20100925184250.7f4c6713@pitrou.net> On Sat, 25 Sep 2010 00:04:23 +0200 (CEST) alexander.belopolsky wrote: > Author: alexander.belopolsky > Date: Sat Sep 25 00:04:22 2010 > New Revision: 85000 > > Log: > This should fix buildbot failure introduced by r84994 Can you also backport it to 2.7? Thanks Antoine. From tjreedy at udel.edu Sat Sep 25 18:53:48 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 25 Sep 2010 12:53:48 -0400 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: References: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> Message-ID: On 9/25/2010 7:11 AM, Georg Brandl wrote: > I'll bother Ezio when he's back. It just feels strange to me that the bit > of statistic I feel is most interesting -- whether there are less open bugs > at the end of the week than at the start -- is not obvious from the report. As of just now, the default search for all open issues returns 2475. That is down steadily over the last 8 or so weeks from a peak of about 2750 in early June. About 30 people have closed at least one issue in this period. The current figure is up from about 1200 a few years ago. Part of the increase is due to the 2.x to 3.x transition and the bulge from 2 to 4 active issues. I would roughly guess that there are perhap a couple hundred 2.7-only issue. You just closed one today. -- Terry Jan Reedy From g.brandl at gmx.net Sat Sep 25 19:02:06 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 25 Sep 2010 19:02:06 +0200 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: References: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> Message-ID: Am 25.09.2010 18:53, schrieb Terry Reedy: > On 9/25/2010 7:11 AM, Georg Brandl wrote: > >> I'll bother Ezio when he's back. It just feels strange to me that the bit >> of statistic I feel is most interesting -- whether there are less open bugs >> at the end of the week than at the start -- is not obvious from the report. > > As of just now, the default search for all open issues returns 2475. > That is down steadily over the last 8 or so weeks from a peak of about > 2750 in early June. About 30 people have closed at least one issue in > this period. That's really promising. (And that's also why I want to see a negative delta for the "open" count.) Thanks for these numbers! Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From solipsis at pitrou.net Sat Sep 25 19:08:22 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 25 Sep 2010 19:08:22 +0200 Subject: [Python-Dev] Summary of Python tracker Issues References: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> Message-ID: <20100925190822.70cc7ed8@pitrou.net> On Sat, 25 Sep 2010 19:02:06 +0200 Georg Brandl wrote: > Am 25.09.2010 18:53, schrieb Terry Reedy: > > On 9/25/2010 7:11 AM, Georg Brandl wrote: > > > >> I'll bother Ezio when he's back. It just feels strange to me that the bit > >> of statistic I feel is most interesting -- whether there are less open bugs > >> at the end of the week than at the start -- is not obvious from the report. > > > > As of just now, the default search for all open issues returns 2475. > > That is down steadily over the last 8 or so weeks from a peak of about > > 2750 in early June. About 30 people have closed at least one issue in > > this period. > > That's really promising. (And that's also why I want to see a negative delta > for the "open" count.) Thanks for these numbers! Without any precision on how and why these bugs were closed, these numbers don't mean anything. We would need a breakdown of bug closings according to the "resolution" field, at the minimum. Antoine. From paul at boddie.org.uk Sat Sep 25 21:12:37 2010 From: paul at boddie.org.uk (Paul Boddie) Date: Sat, 25 Sep 2010 21:12:37 +0200 Subject: [Python-Dev] Python wiki Message-ID: <201009252112.37935.paul@boddie.org.uk> Hello, I've been following this thread all week at work, but I thought it might be time to respond to the different remarks in a single response, given that I am involved in editing and maintaining the Wiki on python.org, and I had a strong enough opinion about such things to give a talk about it at EuroPython that some of you attended: http://wiki.europython.eu/Talks/Web_Collaboration_And_Python/Talk Guido van Rossum wrote: > On Thu, Sep 23, 2010 at 3:35 PM, "Martin v. L?wis" wrote: > > There actually is an admin team, and they actually do set ACLs. > > Who are they? > > > IIUC, this is primarily for spam protection, though. > > So would they object against additions to the team? The administrators are generally the people on the following page: http://wiki.python.org/moin/AdminGroup Someone may have special powers, particularly if they have shell access to the machine running the Wiki, but there's no "secret brotherhood". In fact, I recall advocating that Carl Trachte get admin powers given his continuing high-quality contributions, so we can say that people aren't denying others administrative privileges if that person is clearly doing good work and would benefit from being able to have those privileges. [Later on...] Guido van Rossum wrote: > On Fri, Sep 24, 2010 at 3:31 AM, Michael Foord > > wrote: > > Wiki maintenance is discussed, along with other python.org maintenance > > topics, on the pydotorg-www mailing list: > > > > http://mail.python.org/mailman/listinfo/pydotorg-www > > Which has hidden its membership (even to members). Does it really need > to appear that secretive? At least the message archive is open and has > all the usual suspects. As we're now seeing, people don't feel that it's acceptable to publish the subscribers list, and I think that it's a complete distraction to seek to do so, anyway. It's not as if everyone on that list has special privileges and is preventing newcomers from joining it. > > More wiki and website maintainers needed! > > Maybe a prominent wiki page with info about how to join the list and > the responsibilities / needs would help? > > Also, I gotta say that the wiki login process is awkward. MoinMoin supports OpenID, although I did find and report issues with Moin 1.9 in this regard. Something on my now-huge list of things to do is to make Moin authentication better, especially where there might be a choice of authentication methods. Georg Brandl wrote: > Am 23.09.2010 22:25, schrieb anatoly techtonik: > > On Thu, Sep 23, 2010 at 6:57 PM, Barry Warsaw wrote: > >> I certainly agree with that. So, how can we solve those problems? > >> Radomir has shell access now so perhaps we can ask him to make the > >> Python wiki theme more visually appealing. What roadblocks do people > >> encounter when they want to help garden or reorganize the wiki? > > > > First of all Wiki is outdated. Correct me, but python.org specific > > customizations are not modules, but patches that makes it hard to > > update the Wiki to latest version or add new customizations. > > That's why we have a Moin core developer on the team. ISTM that Moin 1.x > is notoriously hard to extend -- once you go beyond plugins adding new wiki > macros -- which is part of what the team wants to fix in 2.x. Although I understand the sentiments about "specific customizations", python.org should only have its theme as something that isn't a generic extension to MoinMoin, and that theme should be actively maintained. When python.org switched its architecture a while ago, the special theme presumably came with the deal, but it's been so out of date for so long that I switched to one of the standard themes years ago. Fortunately, Radomir's EuroPython theme is actively maintained, works with recent MoinMoin releases, looks a lot better than the old theme, and is used elsewhere. As for Moin 1.x being "notoriously hard to extend" beyond macros, you can get a long way with macros and actions, although I agree that sometimes it's possible to hit architectural constraints. > > Second - ugly Creole syntax. I am for inter-cooperation between wikis, > > and I understand that for non-developer communities [] symbols > > imposing problems, but as an open source developer I am addicted to > > Trac and Google Code syntax and never had problems with those. > > This isn't Creole syntax, it's Moin wiki syntax. And face it, it's not > going to change. It's also not so much different from Trac wiki syntax. I never agreed with this complaint, anyway. When discussing this previously with Anatoly, I went as far as to ask on #moin where Radomir actually posted a good summary of the syntax differences: http://wiki.python.org/moin/SiteImprovements/WikiSyntaxComparison I invite anyone to justify claims that the old style (which Trac adopted) was better. Complaints about double bracketing are specious: MediaWiki has different bracketing levels and it's really confusing. > > Fourth. GPL license. I personally don't have interest to waste my time > > for the code I won't be able to use in some projects, unless the > > project is either exceptional (Mercurial) or interesting. To make > > python.org MoinMoin interesting - there should be an inviting > > entrypoint (see point three above) and the list of tasks to choose > > form. Something that is better than > > http://wiki.python.org/moin/SiteImprovements > > Yes, that needs to be updated indeed. On the matter of the GPL, Anatoly and I more or less agreed to disagree when we last discussed the Web site. On the matter of site improvements and the page which attempts to document ideas, we could move such things to a tracker (where they will probably get as much attention as the Web site bugs get right now), or we could actually prune a lot of the issues by addressing them with fixes that are readily available (applying to a lot of "xyz doesn't work" complaints). Getting people to accept that things aren't going to change for the sake of it (like the Wiki syntax) is more difficult, but at some point such discussions just have to end. anatoly techtonik wrote: > That's a dead way. Wiki should be open for everyone. Just need more > people subscribed to revert unwelcome changes. That is to make > timeline more visible, because on wiki.python.org it is _not_ > intuitive. It may worth to see how Mercurial wiki is managed - I've > picked up the bookmarks monitoring habit from it. Maybe a design > change will help, but again - there is no entrypoint for people with > design skills to start. The Wiki is generally "open for everyone", but as I noted in my talk you often get people who decide that their opinion is worth more than the cumulative knowledge and experience of everyone who has been maintaining a particular resource [1]. You also have to prevent spamming and vandalism which is quite well taken care of with the TextCha support, although I accept that mechanisms should be in place to promote users to higher levels of privilege after a while - another little project on my long "to do" list - so that they don't have to answer editing questions. As for the Mercurial Wiki, I have some involvement with that, and I don't see how it is significantly differently managed. I've also proposed various changes to that Wiki including a theme that could be deployed straight away, but I suspect that it's more a case of people being too busy to take me up on any such offer than a malicious cabal denying me the chance to improve a particular Internet resource. Some final notes on the matter of Wiki editing: The Wiki has for a long time been a product of many co-existing contributions that occasionally overlap with each other and are tied together as respectfully and gently as possible. Although there is a temptation to overhaul much of the content, respect for the existing structure and content has mostly restrained any cleaning-up attempts. I wouldn't mind a bit of reorganisation, but that brings us to the role of the Wiki. If other resources are meant to provide the bulk of what people consider the "important" content, then the Wiki will always have to be shaped to respect that; if the Wiki is to hold much of that content, then issues such as navigability become even more critical. But what is certain is that tucking the Wiki away in some obscure location (I had to request that a link to it be re-added to the front page of python.org recently, as I recall) and treating it like a playpen will only encourage casual edits with little concern for the resulting quality. Anyone now on the verge of concluding that this is purely a Wiki phenomenon should consult my "common myths" slide: http://wiki.europython.eu/Talks/Web_Collaboration_And_Python/Talk#Some_Common_Myths High-quality documentation isn't something you get for nothing, nor is it a property of some magic solution or tool: there's always some hard work involved for human beings. Paul [1] One "happy" memory of mine involves someone who decided that their style and terminology edits were so important that they felt the need to tell me "don't modify it again" when it was they who were doing the modifying. From fuzzyman at voidspace.org.uk Sat Sep 25 21:33:52 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 25 Sep 2010 20:33:52 +0100 Subject: [Python-Dev] Python wiki In-Reply-To: <201009252112.37935.paul@boddie.org.uk> References: <201009252112.37935.paul@boddie.org.uk> Message-ID: <4C9E4EA0.8040502@voidspace.org.uk> On 25/09/2010 20:12, Paul Boddie wrote: > [snip...] > Guido van Rossum wrote: >> On Fri, Sep 24, 2010 at 3:31 AM, Michael Foord >> >> wrote: >>> Wiki maintenance is discussed, along with other python.org maintenance >>> topics, on the pydotorg-www mailing list: >>> >>> http://mail.python.org/mailman/listinfo/pydotorg-www >> Which has hidden its membership (even to members). Does it really need >> to appear that secretive? At least the message archive is open and has >> all the usual suspects. > As we're now seeing, people don't feel that it's acceptable to publish the > subscribers list, To be fair, quite a few people said they thought it was fine / a good thing. A couple (maybe 3?) said that as the list was originally advertised with the member list hidden it would be unfair to change. So based on responses, more people think it *is* acceptable. So long as we give notice for the change, so that anyone who doesn't want their name associated with the list can unsubscribe, I don't see why there should be a problem. All the best, Michael > and I think that it's a complete distraction to seek to do > so, anyway. It's not as if everyone on that list has special privileges and > is preventing newcomers from joining it. > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From g.brandl at gmx.net Sat Sep 25 21:41:12 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 25 Sep 2010 21:41:12 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: <201009252112.37935.paul@boddie.org.uk> References: <201009252112.37935.paul@boddie.org.uk> Message-ID: Am 25.09.2010 21:12, schrieb Paul Boddie: > Georg Brandl wrote: >> Am 23.09.2010 22:25, schrieb anatoly techtonik: >> > On Thu, Sep 23, 2010 at 6:57 PM, Barry Warsaw wrote: >> >> I certainly agree with that. So, how can we solve those problems? >> >> Radomir has shell access now so perhaps we can ask him to make the >> >> Python wiki theme more visually appealing. What roadblocks do people >> >> encounter when they want to help garden or reorganize the wiki? >> > >> > First of all Wiki is outdated. Correct me, but python.org specific >> > customizations are not modules, but patches that makes it hard to >> > update the Wiki to latest version or add new customizations. >> >> That's why we have a Moin core developer on the team. ISTM that Moin 1.x >> is notoriously hard to extend -- once you go beyond plugins adding new wiki >> macros -- which is part of what the team wants to fix in 2.x. > > Although I understand the sentiments about "specific customizations", > python.org should only have its theme as something that isn't a generic > extension to MoinMoin, and that theme should be actively maintained. When > python.org switched its architecture a while ago, the special theme > presumably came with the deal, but it's been so out of date for so long that > I switched to one of the standard themes years ago. Fortunately, Radomir's > EuroPython theme is actively maintained, works with recent MoinMoin releases, > looks a lot better than the old theme, and is used elsewhere. As a disclaimer, I have no idea about the specific configuration of Moin used at wiki.python.org. > As for Moin 1.x being "notoriously hard to extend" beyond macros, you can get > a long way with macros and actions, although I agree that sometimes it's > possible to hit architectural constraints. It's just that every Moin installation I've come across used custom patches, and as a consequence was difficult to update. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From pje at telecommunity.com Sat Sep 25 21:56:59 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sat, 25 Sep 2010 15:56:59 -0400 Subject: [Python-Dev] WSGI is now Python 3-friendly Message-ID: <20100925195711.60FB13A4079@sparrow.telecommunity.com> I have only done the Python 3-specific changes at this point; the diff is here if anybody wants to review, nitpick or otherwise comment: http://svn.python.org/view/peps/trunk/pep-0333.txt?r1=85014&r2=85013&pathrev=85014 For that matter, if anybody wants to take a crack at updating Python 3's wsgiref based on the above, feel free. ;-) I'll be happy to answer any questions I can that come up in the process. (Please note: I went with Ian Bicking's "headers are strings, bodies are bytes" proposal, rather than my original "bodies and outputs are bytes" one, as there were not only some good arguments in its favor, but because it also resulted in fewer changes to the PEP, especially in the code samples.) I will continue to work on adding the other addenda/errata mentioned here: http://mail.python.org/pipermail/web-sig/2010-September/004655.html But because these are "shoulds" rather than musts, and apply to both Python 2 and 3, they are not as high priority for immediate implementation in wsgiref and do not necessarily need to hold up the 3.2 release. (Nonetheless, if anybody is willing to implement them in the Python 3 version, I will happily review the changes for backport into the Python 2 standalone version of wsgiref, and issue an updated release to include them.) Thanks! From rdmurray at bitdance.com Sat Sep 25 23:02:18 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 25 Sep 2010 17:02:18 -0400 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: References: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> <4C9DE6C7.5040909@v.loewis.de> Message-ID: <20100925210218.4E6722186B4@kimball.webabinitio.net> On Sat, 25 Sep 2010 14:22:06 +0200, Georg Brandl wrote: > Am 25.09.2010 14:10, schrieb "Martin v. L=F6wis": > > The total numbers reported are really the totals. Also, the delta > > reported for the totals is the difference to the last report. > > > The number reported with +/- for open/closed are *not* deltas, but the > > number of issues opened since last week. As some open issues were closed > > and some closed issues were opened, they don't sum up the way you > > expect. An example: > > > old: > > open: #1 #2 > > closed: #3 #4 > > new: > > open: #1 #3 #5 > > closed: #2 #4 > > > The report would be > > > open: 3 (+2, namely #3 and #5); delta would be +1 > > closed: 2 (+1, namely #4); delta would be 0 > > > IOW, the numbers after +/- match the counts in the lists shown below, > > not the delta since last week. > > Yes, and that's what I complained about. However, your example doesn't > demonstrate my problem, since its deltas *are* real deltas, and > +1 + +0 = +1. > > I guess a better example would be > > old: > open #1 #2 > closed #3 > new: > open #1 > closed #2 #3 #4 #5 > > which results in +2 for open (since #4 and #5 were opened) and +3 for closed > (since #2, #4 and #5 were closed), however the total issue delta is +2. Th= > is why I think these numbers should be labeled "openings" and "closings". I haven't looked at the code, so I don't know the details of the algorithm that is actually used, but from what Ezio said your example is *not* correct. The numbers in parenthesis are the number of issues opened/closed in the past week that are *still* open or closed. So open would certainly not be +2. I'm not sure if it would be +0 or -1 without looking at the code. I agree that having the delta against open from the previous week would be the most helpful. --David From guido at python.org Sat Sep 25 23:05:31 2010 From: guido at python.org (Guido van Rossum) Date: Sat, 25 Sep 2010 14:05:31 -0700 Subject: [Python-Dev] Python wiki In-Reply-To: <201009252112.37935.paul@boddie.org.uk> References: <201009252112.37935.paul@boddie.org.uk> Message-ID: On Sat, Sep 25, 2010 at 12:12 PM, Paul Boddie wrote: > Guido van Rossum wrote: >> Also, I gotta say that the wiki login process is awkward. > > MoinMoin supports OpenID, although I did find and report issues with Moin 1.9 > in this regard. Something on my now-huge list of things to do is to make Moin > authentication better, especially where there might be a choice of > authentication methods. Unfortunately, most sites using OpenID seem have an awkward login process. Maybe it's just me (I don't use OpenID much) but I expect that without a lot more handholding of new users, OpenID actually turns more people away than any other registration/login process. The people who like OpenID are often the ultra-geeks who cannot imagine the worldview of a typical internet user, and when they design a website they are often clueless about what it feels like to start using OpenID for the first time or to remember your OpenID handle if the last time you used it was a month ago. Not that the native Moin login method is much better... Either way, Python's wikis have some of the most awkward auth systems I've come across. -- --Guido van Rossum (python.org/~guido) From guido at python.org Sat Sep 25 23:07:05 2010 From: guido at python.org (Guido van Rossum) Date: Sat, 25 Sep 2010 14:07:05 -0700 Subject: [Python-Dev] WSGI is now Python 3-friendly In-Reply-To: <20100925195711.60FB13A4079@sparrow.telecommunity.com> References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> Message-ID: This is a very laudable initiative and I approve of the changes -- but I really think it ought to be a separate PEP rather than pretending it is just a set of textual corrections on the existing PEP 333. --Guido On Sat, Sep 25, 2010 at 12:56 PM, P.J. Eby wrote: > I have only done the Python 3-specific changes at this point; the diff is > here if anybody wants to review, nitpick or otherwise comment: > > ?http://svn.python.org/view/peps/trunk/pep-0333.txt?r1=85014&r2=85013&pathrev=85014 > > For that matter, if anybody wants to take a crack at updating Python 3's > wsgiref based on the above, feel free. ?;-) ?I'll be happy to answer any > questions I can that come up in the process. > > (Please note: I went with Ian Bicking's "headers are strings, bodies are > bytes" proposal, rather than my original "bodies and outputs are bytes" one, > as there were not only some good arguments in its favor, but because it also > resulted in fewer changes to the PEP, especially in the code samples.) > > I will continue to work on adding the other addenda/errata mentioned here: > > ?http://mail.python.org/pipermail/web-sig/2010-September/004655.html > > But because these are "shoulds" rather than musts, and apply to both Python > 2 and 3, they are not as high priority for immediate implementation in > wsgiref and do not necessarily need to hold up the 3.2 release. > > (Nonetheless, if anybody is willing to implement them in the Python 3 > version, I will happily review the changes for backport into the Python 2 > standalone version of wsgiref, and issue an updated release to include > them.) > > Thanks! > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Sat Sep 25 23:37:26 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 25 Sep 2010 23:37:26 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: References: <201009252112.37935.paul@boddie.org.uk> Message-ID: <4C9E6B96.2080401@v.loewis.de> > Unfortunately, most sites using OpenID seem have an awkward login > process. Maybe it's just me (I don't use OpenID much) but I expect > that without a lot more handholding of new users, OpenID actually > turns more people away than any other registration/login process. So how do you like the OpenID login of PyPI, which has a Google, MyOpenID and Launchpad icon, which users need to click on to create in account or login? The ultra geeks demanded and got a separate page where they can enter long URLs. Regards, Martin From martin at v.loewis.de Sat Sep 25 23:41:45 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 25 Sep 2010 23:41:45 +0200 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: References: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> <4C9DE6C7.5040909@v.loewis.de> Message-ID: <4C9E6C99.2060406@v.loewis.de> > I guess a better example would be > > old: > open #1 #2 > closed #3 > new: > open #1 > closed #2 #3 #4 #5 > > which results in +2 for open (since #4 and #5 were opened) and +3 for closed > (since #2, #4 and #5 were closed), however the total issue delta is +2. This > is why I think these numbers should be labeled "openings" and "closings". I *have* looked at the code, any I'm certain that this is not how the computation goes. Instead, I'm also certain that it goes as in the message I sent. Notice that it prints the numbers I put into parentheses (+2, +1), and they do *not* sum up to +1. The numbers I posted as "delta would be" are not currently included in the report. Regards, Martin From martin at v.loewis.de Sat Sep 25 23:43:20 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 25 Sep 2010 23:43:20 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> Message-ID: <4C9E6CF8.4080705@v.loewis.de> >>> For me a major annoyance is the empty page with two links on wiki.python.org >>> While it allows to tell new people that there is also a Jython wiki, >>> my vision that it should be instead be oriented on existing >>> contributors immediately providing instruments to work with Python >>> wiki. So if smb. need Jython wiki - it should be moved to >>> wiki.jython.org >>> >> >> That's funny, I've never seen that page before. Does it get linked to >> from somewhere? > > It's at , and FTR, it has annoyed me as well. So how would you propose to resolve this? Keep in mind that existing links need to continue to work. Regards, Martin From g.brandl at gmx.net Sat Sep 25 23:45:42 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 25 Sep 2010 23:45:42 +0200 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: <4C9E6C99.2060406@v.loewis.de> References: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> <4C9DE6C7.5040909@v.loewis.de> <4C9E6C99.2060406@v.loewis.de> Message-ID: Am 25.09.2010 23:41, schrieb "Martin v. L?wis": > >> I guess a better example would be >> >> old: >> open #1 #2 >> closed #3 >> new: >> open #1 >> closed #2 #3 #4 #5 >> >> which results in +2 for open (since #4 and #5 were opened) and +3 for closed >> (since #2, #4 and #5 were closed), however the total issue delta is +2. This >> is why I think these numbers should be labeled "openings" and "closings". > > I *have* looked at the code, any I'm certain that this is not how the > computation goes. > > Instead, I'm also certain that it goes as in the message I sent. > Notice that it prints the numbers I put into parentheses (+2, +1), > and they do *not* sum up to +1. The numbers I posted as "delta would > be" are not currently included in the report. Ah. I misunderstood, sorry. Well, let's bury this thread then and I will put it on my todo list to modify it to work as expected. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From martin at v.loewis.de Sat Sep 25 23:46:11 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 25 Sep 2010 23:46:11 +0200 Subject: [Python-Dev] Relative imports in Py3k In-Reply-To: <4C9DFFE9.5040501@trueblade.com> References: <4C9DFFE9.5040501@trueblade.com> Message-ID: <4C9E6DA3.8080105@v.loewis.de> > Also, it's a horrible bug report, if that's what it is. It's not a bug report, and I don't think it was meant to be one. It started with "I wonder if", suggesting that it is really a request for help. What you read as a bug report was labeled "user story", which I think is anatoly's way of saying "it's not a bug report". Regards, Martin From g.brandl at gmx.net Sat Sep 25 23:53:43 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 25 Sep 2010 23:53:43 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9E6CF8.4080705@v.loewis.de> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> <4C9E6CF8.4080705@v.loewis.de> Message-ID: Am 25.09.2010 23:43, schrieb "Martin v. L?wis": >>>> For me a major annoyance is the empty page with two links on wiki.python.org >>>> While it allows to tell new people that there is also a Jython wiki, >>>> my vision that it should be instead be oriented on existing >>>> contributors immediately providing instruments to work with Python >>>> wiki. So if smb. need Jython wiki - it should be moved to >>>> wiki.jython.org >>>> >>> >>> That's funny, I've never seen that page before. Does it get linked to >>> from somewhere? >> >> It's at , and FTR, it has annoyed me as well. > > So how would you propose to resolve this? Keep in mind that existing > links need to continue to work. Redirect wiki.python.org to the Python wiki front page, and put the Jython wiki somewhere on its own (whether it's wiki.jython.org or not). The only people who will need a pointer are those who went directly to wiki.python.org/ and intended to go to the Jython wiki; a link might be added on the front page of the Python wiki. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From skip at pobox.com Sun Sep 26 00:13:30 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 25 Sep 2010 17:13:30 -0500 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9E4EA0.8040502@voidspace.org.uk> References: <201009252112.37935.paul@boddie.org.uk> <4C9E4EA0.8040502@voidspace.org.uk> Message-ID: <19614.29706.324339.825570@montanaro.dyndns.org> >> As we're now seeing, people don't feel that it's acceptable to >> publish the subscribers list, Michael> To be fair, quite a few people said they thought it was fine / Michael> a good thing. A couple (maybe 3?) said that as the list was Michael> originally advertised with the member list hidden it would be Michael> unfair to change. So based on responses, more people think it Michael> *is* acceptable. I've not yet responded to this aspect of the thread. I see no reason not to make the membership list visible to the list members themselves. Beyond that, I don't think it serves any benefit, and of course, runs the risk of exposing email addresses to spam harvesters. OTOH, those of us who are visible enough on the web that we get to the point of helping maintain a very public website have probably already exposed our email addresses dozens or hundreds of times (or more) to spammers. Googling for "skip at pobox.com" (the quotes are required to constrain the search properly) yields about 22,000 hits. "guido at python.org" yields about 38,000 hits. Paul's and Michael's addresses returned multiple thousands of hits. And so on. As I recall, one reason for the formation of pydotorg-www was to create a more visible list than pydotorg so the maintenance of the website didn't appear so cabalistic to the interested observer. Still, I see that most/all lists hosted on mail.python.org seem to restrict the list membership to the list admins, so leaving the status quo probably won't hurt anything. Skip From martin at v.loewis.de Sun Sep 26 00:16:48 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 26 Sep 2010 00:16:48 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> <4C9E6CF8.4080705@v.loewis.de> Message-ID: <4C9E74D0.3050704@v.loewis.de> > Redirect wiki.python.org to the Python wiki front page, and put the Jython > wiki somewhere on its own (whether it's wiki.jython.org or not). But that can't work: then off-site links into either wiki break. Regards, Martin From g.brandl at gmx.net Sun Sep 26 00:48:56 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 26 Sep 2010 00:48:56 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9E74D0.3050704@v.loewis.de> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> <4C9E6CF8.4080705@v.loewis.de> <4C9E74D0.3050704@v.loewis.de> Message-ID: Am 26.09.2010 00:16, schrieb "Martin v. L?wis": >> Redirect wiki.python.org to the Python wiki front page, and put the Jython >> wiki somewhere on its own (whether it's wiki.jython.org or not). > > But that can't work: then off-site links into either wiki break. Why -- they can be redirected easily. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From greg.ewing at canterbury.ac.nz Sun Sep 26 00:57:47 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 26 Sep 2010 10:57:47 +1200 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> <4C9D21E8.1080005@canterbury.ac.nz> <4C9D298A.3010407@g.nevcal.com> Message-ID: <4C9E7E6B.2060409@canterbury.ac.nz> Paul Moore wrote: > Windows has (I believe) user definable filesystems, too, but the OS > has "get me the real filename" style calls, Does it really, though? The suggestions I've seen for doing this involve abusing the short/long filename translation machinery, and I'm not sure they're guaranteed to return the actual case rather than something that happens to work. -- Greg From martin at v.loewis.de Sun Sep 26 01:18:50 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 26 Sep 2010 01:18:50 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> <4C9E6CF8.4080705@v.loewis.de> <4C9E74D0.3050704@v.loewis.de> Message-ID: <4C9E835A.6010606@v.loewis.de> Am 26.09.2010 00:48, schrieb Georg Brandl: > Am 26.09.2010 00:16, schrieb "Martin v. L?wis": >>> Redirect wiki.python.org to the Python wiki front page, and put the Jython >>> wiki somewhere on its own (whether it's wiki.jython.org or not). >> >> But that can't work: then off-site links into either wiki break. > > Why -- they can be redirected easily. Ok, so please be more specific: what exactly should the new structure look like? Regards, Martin From steve at holdenweb.com Sun Sep 26 03:12:53 2010 From: steve at holdenweb.com (Steve Holden) Date: Sat, 25 Sep 2010 21:12:53 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: <1285339858.3243.4.camel@localhost.localdomain> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9C6A6F.6010202@netwok.org> <20100924142639.0e212df3@pitrou.net> <1285338660.3243.1.camel@localhost.localdomain> <1285339858.3243.4.camel@localhost.localdomain> Message-ID: <4C9E9E15.1030403@holdenweb.com> On 9/24/2010 10:50 AM, Antoine Pitrou wrote: > Le samedi 25 septembre 2010 ? 00:42 +1000, Nick Coghlan a ?crit : >>> >>> I have often used searches on "performance" or "resource usage" to find >>> what was needing a review or a patch. I think it would be a mistake to >>> remove those two categories. >> >> That purpose would be served just as well by keywords though >> (particularly since those attributes aren't mutually exclusive - >> resource usage problems will usually *cause* performance problems, and >> you may notice the latter first). >> Keywords are generally better than a restricted vocabulary for richness of content, but worse for finding the appropriate search term. You "pays yer money and takes yer chance". Given that we could (if necessary) overlay some sort of Python-specific semantic category generation to searches if we needed to, I tend to favor the "liberally tag with keywords" approach. But without any smarts being applied to the problem it's always to know whether you are searching for the right keywords. >> A generic "bug" classification would also better suit documentation >> bugs. The simpler we can make the more common fields, while still >> providing the essential information, the better. > > But how should a performance improvement be filed? Bug? Feature request? > Or should "feature request" be renamed "improvement"? > Not all features would be improvements (and I'm thinking specifically here of 2.x's "print >> f" as an egregious design wart added for entirely pragmatic reasons). They could, however, be classified along with performance improvement requests as "Enhancement requests". The big problem, I suspect, is that we don't give clear enough guidance to almost total noobs about how to fill in the issue tracker form. If you can't remember what your first month as a programmer was like then you probably aren't qualified to be writing int on-line help for the tracker. (The tracker does *have* on-line help, right?) regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From jnoller at gmail.com Sun Sep 26 03:22:53 2010 From: jnoller at gmail.com (Jesse Noller) Date: Sat, 25 Sep 2010 21:22:53 -0400 Subject: [Python-Dev] WSGI is now Python 3-friendly In-Reply-To: <20100925195711.60FB13A4079@sparrow.telecommunity.com> References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> Message-ID: On Sat, Sep 25, 2010 at 3:56 PM, P.J. Eby wrote: > I have only done the Python 3-specific changes at this point; the diff is > here if anybody wants to review, nitpick or otherwise comment: > > ?http://svn.python.org/view/peps/trunk/pep-0333.txt?r1=85014&r2=85013&pathrev=85014 > > For that matter, if anybody wants to take a crack at updating Python 3's > wsgiref based on the above, feel free. ?;-) ?I'll be happy to answer any > questions I can that come up in the process. > > (Please note: I went with Ian Bicking's "headers are strings, bodies are > bytes" proposal, rather than my original "bodies and outputs are bytes" one, > as there were not only some good arguments in its favor, but because it also > resulted in fewer changes to the PEP, especially in the code samples.) > > I will continue to work on adding the other addenda/errata mentioned here: > > ?http://mail.python.org/pipermail/web-sig/2010-September/004655.html > > But because these are "shoulds" rather than musts, and apply to both Python > 2 and 3, they are not as high priority for immediate implementation in > wsgiref and do not necessarily need to hold up the 3.2 release. > > (Nonetheless, if anybody is willing to implement them in the Python 3 > version, I will happily review the changes for backport into the Python 2 > standalone version of wsgiref, and issue an updated release to include > them.) > > Thanks! This looks like good progress (and does not seem to block the progress of PEP 444) but would it be possible to do it as a different PEP rather then just an update; or adding an explicit "these are the differences between v1 and v2" section? It seems like it will end up different enough to be a different specification, closely related to the original, but different enough to trip up all the people maintaining current WSGI servers and apps. jesse From pje at telecommunity.com Sun Sep 26 03:45:13 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sat, 25 Sep 2010 21:45:13 -0400 Subject: [Python-Dev] WSGI is now Python 3-friendly In-Reply-To: References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> Message-ID: <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> At 09:22 PM 9/25/2010 -0400, Jesse Noller wrote: >It seems like it will end up >different enough to be a different specification, closely related to >the original, but different enough to trip up all the people >maintaining current WSGI servers and apps. The only actual *change* to the spec is mandating the use of the 'bytes' type or equivalent for HTTP bodies when using Python 3. Seriously, that's *it*. Everything else that's (planned to be) added is either 100% truly just clarifications (e.g. nothing in the spec *ever* said SERVER_PORT could be an int, but apparently some people somehow interpreted it so), or else best-practice recommendations from people who actually implemented WSGI servers. For example, the readline() size hint is "not supported" in the original spec (meaning clients can't call it and be compliant). The planned modification is "servers should implement it" (best practice), but you can't call an implementation that *doesn't* implement it noncompliant. (This just addresses the fact that most practical implementations *did* in fact support it, and code out there relies on this.) So, no (previously-)compliant implementations were harmed in the making of the updated spec. If they were compliant before, they're compliant now. I'm actually a bit surprised people are bringing this up now, since when I announced the plan to make these changes, I said that nothing would be changed that would break anything... even for what I believe are the only Python 3 WSGI implementations right now (by Graham Dumpleton and Robert Brewer). Indeed, all of the changes (except the bytes thing) are stuff previously discussed endlessly on the Web-SIG (years ago in most cases) and widely agreed on as, "this should have been made clear in the original PEP". And, I also explicitly deferred and/or rejected items that *can't* be done in a 100% backward-compatible way, and would have to be WSGI 1.1 or higher -- indeed, I have a long list of changes from Graham that I've pronounced "can't be done without a 1.1". Indeed, the entire point of the my scope choices were to allow all this to happen *without* a whole new spec. ;-) From tjreedy at udel.edu Sun Sep 26 03:51:01 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 25 Sep 2010 21:51:01 -0400 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: <20100925190822.70cc7ed8@pitrou.net> References: <20100924161427.1CFDB78282@psf.upfronthosting.co.za> <20100925190822.70cc7ed8@pitrou.net> Message-ID: On 9/25/2010 1:08 PM, Antoine Pitrou wrote: > On Sat, 25 Sep 2010 19:02:06 +0200 > Georg Brandl wrote: > >> Am 25.09.2010 18:53, schrieb Terry Reedy: >>> On 9/25/2010 7:11 AM, Georg Brandl wrote: >>> >>>> I'll bother Ezio when he's back. It just feels strange to me that the bit >>>> of statistic I feel is most interesting -- whether there are less open bugs >>>> at the end of the week than at the start -- is not obvious from the report. >>> >>> As of just now, the default search for all open issues returns 2475. >>> That is down steadily over the last 8 or so weeks from a peak of about >>> 2750 in early June. About 30 people have closed at least one issue in >>> this period. >> >> That's really promising. (And that's also why I want to see a negative delta >> for the "open" count.) Thanks for these numbers! > > Without any precision on how and why these bugs were closed, these > numbers don't mean anything. We would need a breakdown of bug closings > according to the "resolution" field, at the minimum. Doing some hand-counting because I do not have proper (as in sql query) access to the database, closed issues so far for Activity: 2010-09 174 fixed 36 invalid 31 out of date 1 postponed 18 reject 18 won't fix 7 works for me --- 285 -- Terry Jan Reedy From pje at telecommunity.com Sun Sep 26 04:00:19 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sat, 25 Sep 2010 22:00:19 -0400 Subject: [Python-Dev] WSGI is now Python 3-friendly In-Reply-To: References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> Message-ID: <20100926020021.454453A4079@sparrow.telecommunity.com> At 02:07 PM 9/25/2010 -0700, Guido van Rossum wrote: >This is a very laudable initiative and I approve of the changes -- but >I really think it ought to be a separate PEP rather than pretending it >is just a set of textual corrections on the existing PEP 333. With the exception of the bytes change, I ruled out accepting any proposed amendments that would actually alter the protocol. The amendments are all either textual clarifications, clarifications of ambiguous/unspecified areas, or best-practice recommendations by implementors. (i.e., which are generally already implemented in major servers) The full list of things Graham and others have asked for or recommended would indeed require a 1.1 version at minimum, and thus a new PEP. But I really don't want to start down that road right now, and therefore hope that I can talk Graham or some other poor soul into shepherding a 1.1 PEP instead. ;-) (Seriously: through an ironic twist of fate, I have done nearly *zero* Python web programming since around the time I drafted the first spec in 2004, so even if it makes sense for me to finish PEP 333, it makes little sense for me to be starting a *new* one on the topic now!) From guido at python.org Sun Sep 26 04:15:52 2010 From: guido at python.org (Guido van Rossum) Date: Sat, 25 Sep 2010 19:15:52 -0700 Subject: [Python-Dev] WSGI is now Python 3-friendly In-Reply-To: <20100926020021.454453A4079@sparrow.telecommunity.com> References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926020021.454453A4079@sparrow.telecommunity.com> Message-ID: On Sat, Sep 25, 2010 at 7:00 PM, P.J. Eby wrote: > At 02:07 PM 9/25/2010 -0700, Guido van Rossum wrote: >> >> This is a very laudable initiative and I approve of the changes -- but >> I really think it ought to be a separate PEP rather than pretending it >> is just a set of textual corrections on the existing PEP 333. > > With the exception of the bytes change, I ruled out accepting any proposed > amendments that would actually alter the protocol. ?The amendments are all > either textual clarifications, clarifications of ambiguous/unspecified > areas, or best-practice recommendations by implementors. ?(i.e., which are > generally already implemented in major servers) Of those, IMO only textual clarifications ought to be made to an existing, accepted, widely implemented standards-track PEP. Clarifications of ambiguous/unspecified behavior can possibly rule as non-conforming implementations that used to get the benefit of the doubt. Best-practice recommendations also have the effect of changing (perceived) compliance. Really, what's the problem with creating a new PEP? PEPs are cheap -- it's getting the acceptance that's costly, and you've already gotten that part. It doesn't have to be long. You can still make pure textual clarifications to PEP 333 (basically, improve grammar) and add a pointer to the new PEP. Or, you can create a new PEP that is an updated copy of PEP 333, and just add a pointer to PEP 333 saying "(especially in the context of Python 3) this PEP is now superseded by PEP XXXX". I strongly disagree with using (standards-track) PEPs as mutable documents -- before you know it people will have to use weasel words like "compliant with PEP 333 as of date X" to describe their software's conformity. If you add a new PEP #, software declared compliant with PEP 333 remains compliant (and some such software can now also claim compliance with the new PEP without any changes). > The full list of things Graham and others have asked for or recommended > would indeed require a 1.1 version at minimum, and thus a new PEP. ?But I > really don't want to start down that road right now, and therefore hope that > I can talk Graham or some other poor soul into shepherding a 1.1 PEP > instead. ?;-) That's fine. It will just be another PEP. > (Seriously: through an ironic twist of fate, I have done nearly *zero* > Python web programming since around the time I drafted the first spec in > 2004, so even if it makes sense for me to finish PEP 333, it makes little > sense for me to be starting a *new* one on the topic now!) Don't see this as a new spec. See it as a procedural issue. -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Sun Sep 26 08:31:16 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 26 Sep 2010 08:31:16 +0200 Subject: [Python-Dev] WSGI is now Python 3-friendly In-Reply-To: <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> Message-ID: <4C9EE8B4.7090405@v.loewis.de> Am 26.09.2010 03:45, schrieb P.J. Eby: > I'm actually a bit surprised people are bringing this up now, since when > I announced the plan to make these changes, I said that nothing would be > changed that would break anything I think people read this as "nothing would be changed, period." However, you did make substantial changes to the specification (or else the whole exercise would have been pointless, I suppose, and you couldn't have claimed that WSGI is now Python 3-friendly when it previously was not). So this is essentially a new version of the spec. As PEPs themselves are not versioned (unlike, say, ISO standards), Guido insists it ought to get a new PEP number. Then, people declaring compliance can identify what specification they actually comply to. Declaring compliance to PEP 333 as-of-last-week-but-not-as-of-today is now difficult. This particularly puzzles people some of the existing WSGI servers are now incompatible to the PEP, when they still were compatible last week. Regards, Martin From scott+python-dev at scottdial.com Sun Sep 26 08:14:42 2010 From: scott+python-dev at scottdial.com (Scott Dial) Date: Sun, 26 Sep 2010 02:14:42 -0400 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9E6B96.2080401@v.loewis.de> References: <201009252112.37935.paul@boddie.org.uk> <4C9E6B96.2080401@v.loewis.de> Message-ID: <4C9EE4D2.80502@scottdial.com> On 9/25/2010 5:37 PM, Martin v. L?wis wrote: >> Unfortunately, most sites using OpenID seem have an awkward login >> process. Maybe it's just me (I don't use OpenID much) but I expect >> that without a lot more handholding of new users, OpenID actually >> turns more people away than any other registration/login process. > > So how do you like the OpenID login of PyPI, which has a Google, > MyOpenID and Launchpad icon, which users need to click on to create > in account or login? > > The ultra geeks demanded and got a separate page where they > can enter long URLs. Having just tried this out. A few comments: 1) Registering via OpenID is a bit clumsy since there is a "Register" link that does not mention OpenID. 2) The URL registered with the OpenID provider is a bit of a wart: "http://pypi.python.org/pypi?:action=openid_return" vs. "http://bitbucket.org/" 3) The email I received asked me to "Complete your Cheese Shop registration" which I think is just an oversight since the relabeling to pypi. 4) It's a bit clumsy that "Login" pops up an HTTP Authentication prompt, which is useless to someone who only has never set a password and relies only on an OpenID credential. Furthermore, the 401 page does not provide a quick way to get to use OpenID. In general, I am pretty happy with pypi's support of OpenID considering it allowed me to use my own provider, which often has not been the case with other sites. My experience with trying to adopt OpenID as a way of life has been poor mostly because many sites fail to support anything but a few major OpenID providers (e.g., Google). I appreciate has a fast-path for those providers and yet let's me still use my own. Although, I think it would be nice if I didn't have to go to another page to do that, but I may be biased by having such a short OpenID URI. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From martin at v.loewis.de Sun Sep 26 08:36:19 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 26 Sep 2010 08:36:19 +0200 Subject: [Python-Dev] Goodbye In-Reply-To: <4C9E9E15.1030403@holdenweb.com> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9C6A6F.6010202@netwok.org> <20100924142639.0e212df3@pitrou.net> <1285338660.3243.1.camel@localhost.localdomain> <1285339858.3243.4.camel@localhost.localdomain> <4C9E9E15.1030403@holdenweb.com> Message-ID: <4C9EE9E3.3040500@v.loewis.de> > Keywords are generally better than a restricted vocabulary for richness > of content, but worse for finding the appropriate search term. You "pays > yer money and takes yer chance". I think you are unaware what a "roundup keyword" is, here. > But without any smarts being applied to the problem > it's always to know whether you are searching for the right > keywords. The Python bug tracker has only 5 keywords at the moment, which all have a straight-forward meaning. Adding "performance" and "resource usage" would make it 7, which still would be manageable. Regards, Martin From martin at v.loewis.de Sun Sep 26 09:12:30 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 26 Sep 2010 09:12:30 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9EE4D2.80502@scottdial.com> References: <201009252112.37935.paul@boddie.org.uk> <4C9E6B96.2080401@v.loewis.de> <4C9EE4D2.80502@scottdial.com> Message-ID: <4C9EF25E.4050805@v.loewis.de> > 1) Registering via OpenID is a bit clumsy since there is a "Register" > link that does not mention OpenID. Thanks. Fixed. > 2) The URL registered with the OpenID provider is a bit of a wart: > "http://pypi.python.org/pypi?:action=openid_return" vs. > "http://bitbucket.org/" You mean, as this is what the provider then shows you for confirmation? Unfortunately, this can't be changed anymore, or many of the existing accounts break. When I started this, I was more unclear about the relationship of "realm" and "return URL" (I'm still unclear, not having used a realm yet). > 3) The email I received asked me to "Complete your Cheese Shop > registration" which I think is just an oversight since the relabeling to > pypi. Ok, fixed. > 4) It's a bit clumsy that "Login" pops up an HTTP Authentication > prompt, which is useless to someone who only has never set a password > and relies only on an OpenID credential. Furthermore, the 401 page does > not provide a quick way to get to use OpenID. I think there is no way out wrt. to the basic auth prompt. I could label the "Login" link "Password login" if you think this would help. Preventing the browser from prompting the user on the chance they might want to enter an OpenID is not possible, and stopping to use basic authentication is not feasible. > In general, I am pretty happy with pypi's support of OpenID considering > it allowed me to use my own provider, which often has not been the case > with other sites. I guess you are then not in the class of users Guido was referring to, but rather in the "ultra geeks" class. What regular user is actively searching for an "OpenID provider"? If you were using your facebook account (or some such) to log in (i.e. a service that "the masses" likely use and which happens to be an OpenID provider), I'd rather add another provider icon to the front page. > Although, I think it would be nice if I didn't have to go to another > page to do that, but I may be biased by having such a short OpenID URI. This is actually deliberate. I don't want to clutter the front page with a wide entry field. And again, enjoying a short OpenID URI probably does put you in the "ultra geek" category (which I seriously don't mean as an offense). I've learned that OpenID really is a mystery even to the fairly technical usership of PyPI. As an anecdote, a user was puzzled that, after registering the Google OpenID, all you need to do to login is to click on the google logo, and that no user interaction at all was required. This counters established expectations about security so much to actually confuse long-term internet users. Regards, Martin From p.f.moore at gmail.com Sun Sep 26 10:01:32 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 26 Sep 2010 09:01:32 +0100 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: <4C9E7E6B.2060409@canterbury.ac.nz> References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> <4C9D21E8.1080005@canterbury.ac.nz> <4C9D298A.3010407@g.nevcal.com> <4C9E7E6B.2060409@canterbury.ac.nz> Message-ID: On 25 September 2010 23:57, Greg Ewing wrote: > Paul Moore wrote: > >> Windows has (I believe) user definable filesystems, too, but the OS >> has "get me the real filename" style calls, > > Does it really, though? The suggestions I've seen for doing > this involve abusing the short/long filename translation > machinery, and I'm not sure they're guaranteed to return the > actual case rather than something that happens to work. There's another call available. I've been too lazy to go and look it up, but I'll do so sometime today. Paul. From ncoghlan at gmail.com Sun Sep 26 12:54:01 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Sep 2010 20:54:01 +1000 Subject: [Python-Dev] PyObject_GC_UnTrack() no longer reliable in 2.7? In-Reply-To: References: <4C9D1391.20504@v.loewis.de> Message-ID: On Sat, Sep 25, 2010 at 9:20 AM, Tim Peters wrote: [MvL] >> I think it would be possible to have two versions of >> _PyGC_REFS_UNTRACKED, one being, say, -5. >> _PyGC_REFS_UNTRACKED_AND_KEEP_IT_THAT_WAY would be what you get >> when you call PyObject_GC_UnTrack; the code to do automatic >> tracking/untracking based on contents would use some other >> new API (which would be non-public in 2.7.x). > > Looks like a promising idea! ?gcmodule.c's IS_TRACKED macro would have > to change to check both states, and likewise the debug assert in > visit_reachable(). Given the intent is to restore the 2.6 state, perhaps it is the "UNTRACK_ALLOW_RETRACK" optimisation that should get the new special value? Other than that, MvL's suggestion looks like a good idea. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From dirkjan at ochtman.nl Sun Sep 26 12:55:16 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sun, 26 Sep 2010 12:55:16 +0200 Subject: [Python-Dev] hg conversion: tags Message-ID: Hi all, I've recently been working on the conversion more (since my thesis got finished). I finally wrote the script that splits the release branches from the feature branches, so that we can include the former in the main repository and keep the latter in separate clones as needed. Next, I wanted to look into tags. There's a big list of tags (see below), and I wonder if I should clean that up or if we should leave it as-is. For example, it might be interesting to bring old release tags in line with newer tags (so Release_1_0 would become r10), or maybe use clean tags similar to what we do with hg itself (r266 would become just 2.6.6), or just remove some tags. Is this a good idea at all, or should we just leave everything the way it is now? Cheers, Dirkjan r32a2 r266 r266rc2 r266rc1 r32a1 r27 r27rc2 r27rc1 r27b2 r27b1 r312 r265 r265rc2 r312rc1 r27a4 r265rc1 r27a3 r255 r255c2 r255c1 r27a2 r27a1 r311 r264 r264rc2 r264rc1 r263 r263rc1 r311rc1 r31 r31rc2 r31rc1 r31b1 r262 r262c1 r31a2 r31a1 r301 r254 r253 r246 r253c1 r246c1 r261 r30 r30rc3 r30rc2 r26 r26rc2 r30rc1 r26rc1 r30b3 r26b3 r26b2 r30b2 r26b1 r30b1 r26a3 r30a5 r26a2 r30a4 r237 r245 r237c1 r245c1 r30a3 r26a1 r252 r252c1 r251c1 r30a2 r30a1 py3k-before-rstdocs py26-before-rstdocs r251 r236 r236c1 r244 r244c1 r25 r25c2 r25c1 r25b3 r25b2 r25b1 r25a2 r25a1 r25a0 r243 r25a0/trunk r243c1 before-ast-merge-to-head mrg_to_ast-branch_15OCT05 IDLE-syntax-root r242 r242c1 email_Release_2_5_6 r241 r241c2 r241c1 r235 r235c1 merged_from_MAIN_07JAN05 mrg_to_ast-branch_05JAN05 pre-sf-818006 bsddb_version_4_3_0 release24-fork r24 r24c1 r24b2 r24b1 r24a3 tim-doctest-closed r24a2 tim-doctest-merge-24a2 r24a1 pre-sf-1149508 r234 email_Release_2_5_5 r234c1 start-sf-965425 before-ast-merge-from-head bsddb_version_4_2_4 r233 r233c1 r232 r232c1 r231 pybsddb_after_bsddb42_01 pybsddb_before_bsddb42 r23-mac r23 release23-fork r23c2 r23rc1-fork r23c1 bsddb_version_4_1_6 r23b2 r23b2-fork anthony-parser-branchpoint IDLELIB_CREATED r09b1 r223 email_Release_2_5_3 r223c1 email_Release_2_5_2 r23b1-mac r23b1 r23b1-fork COPY_TO_PYTHON mrg_to_ast-branch_24APR03 email_Release_2_5_1 cache-attr-fork email_Release_2_5 r09a2nt email_Release_2_5b1 r23a2 r23a2-fork r09a2 LAST_OLD_IDLE r09a1 r23a1 r23a1-fork r09a0 before-bgen-pep252 r222-mac r222 email_Release_2_4_3 email_Release_2_4_2 Distutils-1_0_3 r222b1 email_Release_2_4_1 email_Release_2_4 py-cvs-2002_09_13-merged py-cvs-2002_09_13 MERGED_FROM_DS_RPC_13SEP02 MERGED_TO_MAIN_13SEP02 PRE_DS_RPC_MERGE email_Release_2_3_1 email_Release_2_3 BEFORE_RESTART email_Release_2_2 email_Release_2_1 final_classic_builds email_Release_2_0_5 Release_2_0_4 TAG_pre_teyc_gvr_rpc_patch r221 r213 r22p1 r221c2 r221c1 r1_95_2 r212 r212c1 release22-mac release22-macmerge release22 release22-fork r22c1-mac r22c1-macmergefromtrunk r22c1 r22rc1-fork final_CW6_projects universal_headers_332 v0_10 r22b2-mac r22b2 r22b2-fork v0_09 r22b1-mac v0_08 v0_07 v0_06 r22b1 r22b1-fork r22b1-docs-prep r22a4 r22a4-fork r22a3 r22a3-fork r22a3-docs-prep r22a2 r22a2-docs-prep r22a2-fork before-carbon-package after-descr-branch-merge date2001-08-01 date2001-07-30 date2001-07-28 IDLEFORK_081_RELEASE date2001-07-21 r211 r22a1 date2001-07-17b date2001-07-17a date2001-07-16 date2001-07-15 py-cvs-2001_07_13-merged py-cvs-2001_07_13 py-cvs-rel2_1-merged date2001-07-13 r211c1 py-cvs-rel2_1 py-cvs-2000_03_09 descr-fork date2001-07-06 base-VP-idle r201 r201c1 merged-with-main-repository after-call-reorg before-call-reorg mac210 Distutils-1_0_2 release21 r21c2 r21c1 mac210b2 r21b2 mac210b1a mac210b1 r21b1 mac210a3 r161 mac210a1 r21a2 r21a1 pre_amk last_cw_pro_53 mac200 release20 Distutils-1_0_1 r20c1 Distutils-1_0 Distutils-0_9_4 r20b2 Distutils-0_9_3 r20b11 mac200b1 r20b1 release16 Distutils-0_9_2 last_68k_projects cw_pro_5 Distutils-0_9_1 arelease r16b1 r16b1-win Distutils-0_9 Distutils-0_8_2 Distutils-0_8_1 Distutils-0_8 r16a2 Distutils-0_1_5 pre_GUSI2 Distutils-0_1_4 Distutils-0_1_3-branch r16a1 release152p2 release152p1-branch-tag pre-unicode idle05 pre_0_2_breakage Distutils-0_1_3 Distutils-0_1_2 Distutils-0_1_1 Distutils-0_1 mx_builder-alpha2 release152p1 pre-string-meths Release_1_0 release152 r152 mac152c1 r152c1 r06 release152b2 r152b2 mac152b1 release152b1 idle-r02 r152b1 Release_0_1 r01 r152a2 Public_Release_20-Aug-1998 PYIDE_APR98 PYTHONSCRIPT_0_5b1 OSAM_APR98 BBPY_0_2_3 release152a1 r152a1 release151p1 mac151 r151 Public_Release_27-Mar-1998 Public_13-Mar-1998 Public_11-Mar-1998 release15 r15b2 r15b1 r15a4 r15a4near r15a3 r15a2 r15a1 lastoldnames lastoldname release14 Python1_4final r14beta3 Beta_20-Aug-1996 r14beta2 Beta_09-Aug-1996 Beta_05-Jul-1996 r14beta1 cwisync1 cnrisync2 chameleon-1 cnrisync release13 r13beta1 Public_05-Jul-1995 release12 proof3 r12beta4 Python_1_2_release Beta_20-Mar-1995 proof2 Beta_15-Mar-1995-#2 Beta_15-Mar-1995 Beta_14-Mar-1995-#3 Beta_14-Mar-1995-#2 Beta_14-Mar-1995 proof1 r12beta3 r12beta2 r12beta1 release111 release11 mhammond mac102 release102 release101 release100 last099 release099 release098 pre_jh From ncoghlan at gmail.com Sun Sep 26 13:07:38 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Sep 2010 21:07:38 +1000 Subject: [Python-Dev] Relative imports in Py3k In-Reply-To: <4C9E6DA3.8080105@v.loewis.de> References: <4C9DFFE9.5040501@trueblade.com> <4C9E6DA3.8080105@v.loewis.de> Message-ID: On Sun, Sep 26, 2010 at 7:46 AM, "Martin v. L?wis" wrote: > What you read as a bug report was labeled "user story", > which I think is anatoly's way of saying "it's not a bug > report". Skimming the original post, I actually thought of two possible candidates that fit the "it doesn't work" moniker when it comes to imports: http://bugs.python.org/issue8098 (multi-threaded dodginess courtesy of some deadlock avoidance we added a while back) http://bugs.python.org/issue992389 (ah, the joys of circular imports - our dear little 6 year old tracker issue...) Not quite what anatoly is talking about though (see other post). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Sun Sep 26 13:32:47 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Sep 2010 21:32:47 +1000 Subject: [Python-Dev] Relative imports in Py3k In-Reply-To: References: Message-ID: On Sat, Sep 25, 2010 at 11:15 PM, anatoly techtonik wrote: > Hi, > > I wonder if situation with relative imports in packages is improved in > Python 3k or > we are still doomed to a chain of hacks? > > My user story: > I am currently debugging project, which consists of many modules in one package. > Each module has tests or other useful stuff for debug in its main > section, but it is a > disaster to use it, because I can't just execute the module file and > expect it to find > relatives. All imports are like: > > from spyderlib.config import get_icon > from spyderlib.utils.qthelpers import translate, add_actions, create_action This is almost certainly failing because the directory containing the spyderlib package isn't on sys.path anywhere (instead, whichever directory contains the script you executed directly will be in there, which will be somewhere inside the package instead of outside it). Put the appropriate directory in PYTHONPATH and these tests should start working. > PEP 328 http://www.python.org/dev/peps/pep-0328/ ?proposes: > > from ... import config > from ..utils.qthelpers import translate, add_actions, create_action This fails for two reasons: 1. "__main__" is missing the module namespace information PEP 328 needs to do its thing 2. Even if 1 is resolved, PEP 328 will resolve to the same absolute imports you're already using and likely fail for the same reason (i.e. spyderlib not found on sys.path) > But this doesn't work, and I couldn't find any short user level > explanation why it is > not possible to make this work at least in Py3k without additional magic. If you use the -m switch to run your module from the directory that contains the spyderlib package directory, it will work. The use of -m provides the module namespace information that PEP 328 needs, while running from the directory that contains the spyderlib package ensures it is visible through sys.path. The one caveat is that the specified module is run as "__main__" and hence does not exist in sys.modules under its normal name. If it gets imported by another module, then you will end up with two copies of the module (one as "__main__" and one under the normal name). This may or may not cause problems, depending on the nature of the module being executed. While PEP 366 describes the boilerplate you can put at the top of your module to allow a directly executed module to try to find its siblings, it still requires that PYTHONPATH be set appropriately. And if you set PYTHONPATH appropriately, then direct execution with absolute imports will work. (The explanation of the failures applies for all Python versions that I am aware of, but the -m based fix only became available in 2.6) (The impact of various command line options and the PYTHONPATH environment variable on sys.path are described at http://docs.python.org/using/cmdline.html) (The basic import search algorithm is described in the tutorial: http://docs.python.org/tutorial/modules.html#the-module-search-path) (And the full gory specification details are in PEP 302, with a few subsequent tweaks courtesy of PEP 328 and PEP 366). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From p.f.moore at gmail.com Sun Sep 26 13:36:56 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 26 Sep 2010 12:36:56 +0100 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> <4C9D21E8.1080005@canterbury.ac.nz> <4C9D298A.3010407@g.nevcal.com> <4C9E7E6B.2060409@canterbury.ac.nz> Message-ID: On 26 September 2010 09:01, Paul Moore wrote: > On 25 September 2010 23:57, Greg Ewing wrote: >> Paul Moore wrote: >> >>> Windows has (I believe) user definable filesystems, too, but the OS >>> has "get me the real filename" style calls, >> >> Does it really, though? The suggestions I've seen for doing >> this involve abusing the short/long filename translation >> machinery, and I'm not sure they're guaranteed to return the >> actual case rather than something that happens to work. > > There's another call available. I've been too lazy to go and look it > up, but I'll do so sometime today. Hmm, I can't find the one I was thinking of. GetLongFileName correctly sets the case of all but the final part, and FindFile can be used to find the last part, but that's not what I recall. GetFinalPathNameByHandle works, and is documented to do so, but (a) it works on an open file handle, so you need to open the file, and (b) it's Vista and later only... Paul. From martin at v.loewis.de Sun Sep 26 13:40:15 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 26 Sep 2010 13:40:15 +0200 Subject: [Python-Dev] hg conversion: tags In-Reply-To: References: Message-ID: <4C9F311F.1070507@v.loewis.de> > For example, it might be interesting to bring old release > tags in line with newer tags (so Release_1_0 would become r10), or > maybe use clean tags similar to what we do with hg itself (r266 would > become just 2.6.6), or just remove some tags. Is this a good idea at > all, or should we just leave everything the way it is now? Removing tags sounds fine with me. Renaming them not so; I fail to see the point. > before-ast-merge-to-head > mrg_to_ast-branch_15OCT05 These could go, IMO. Anything that's there for merge maintenance can go, IMO. > IDLE-syntax-root This one can go as well. In general, I propose to remove all tags which aren't copies of trunk or some branch (i.e. tagging only a part of the Python source tree). I suppose hg can't represent that adequately, anyway (they often come from CVS, and even CVS can only barely represent tags that cover only one or a few files). Regards, Martin From ncoghlan at gmail.com Sun Sep 26 13:43:55 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Sep 2010 21:43:55 +1000 Subject: [Python-Dev] Goodbye In-Reply-To: <1285341131.3243.7.camel@localhost.localdomain> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9C6A6F.6010202@netwok.org> <20100924142639.0e212df3@pitrou.net> <1285338660.3243.1.camel@localhost.localdomain> <1285339858.3243.4.camel@localhost.localdomain> <1285341131.3243.7.camel@localhost.localdomain> Message-ID: On Sat, Sep 25, 2010 at 1:12 AM, Antoine Pitrou wrote: > >> > But how should a performance improvement be filed? Bug? Feature request? >> > Or should "feature request" be renamed "improvement"? >> >> It's a feature request (since we won't backport it unless there is a >> genuine performance problem being addressed as a bug fix). Whether >> that warrants changing the name, I don't know. > > I think most people won't intuitively file performance issues as > "feature requests", since it doesn't sound like a feature. Agreed. >> ?A third option for >> "other improvement" may also work (since that would also cover things >> like clarifying doc wording, fixing comments, adjusting code to be >> more readable/obviously correct, etc). > > But then why not keep a clear categorization of these "other > improvements"? Because it's asking people to make a decision too early in the process. The clear categorisation as to what *kind* of bug/feature/improvement it is can be supplied later on by selecting the appropriate components and keywords. > By the way, doc wording fixes and cosmetic code changes often get > backported. :) Yep, hence why I think the basic "bug, feature, other" split may be a good way to go. It's a quick and easy decision most of the time (including a clear choice for "I don't know if this is a bug or not"), and makes a meaningful procedural distinction (bugs are usually backported, new features are not, and other changes may be backported depending on the details). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From dirkjan at ochtman.nl Sun Sep 26 13:57:52 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sun, 26 Sep 2010 13:57:52 +0200 Subject: [Python-Dev] hg conversion: tags In-Reply-To: <4C9F311F.1070507@v.loewis.de> References: <4C9F311F.1070507@v.loewis.de> Message-ID: On Sun, Sep 26, 2010 at 13:40, "Martin v. L?wis" wrote: > This one can go as well. In general, I propose to remove all tags which > aren't copies of trunk or some branch (i.e. tagging only a part of the > Python source tree). I suppose hg can't represent that adequately, > anyway (they often come from CVS, and even CVS can only barely represent > tags that cover only one or a few files). Ah, yeah, that would definitely be a good reason to remove some of these. Cheers, Dirkjan From ncoghlan at gmail.com Sun Sep 26 13:58:31 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Sep 2010 21:58:31 +1000 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9E74D0.3050704@v.loewis.de> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> <4C9E6CF8.4080705@v.loewis.de> <4C9E74D0.3050704@v.loewis.de> Message-ID: On Sun, Sep 26, 2010 at 8:16 AM, "Martin v. L?wis" wrote: > But that can't work: then off-site links into either wiki break. Georg isn't suggesting a general structural change, just a change to have the URL when you land *directly* on wiki.python.org automatically rewritten to resolve to wiki.python.org/moin. Any URL with additional path info would be handled the same as it is now. I've certainly run into this, since Firefox will turn "wiki.p" into wiki.python.org for me and it (mildly) irritates me every time that that doesn't actually take me to the front page of the wiki. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Sun Sep 26 14:02:57 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Sep 2010 22:02:57 +1000 Subject: [Python-Dev] Python wiki In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> Message-ID: On Sat, Sep 25, 2010 at 6:20 PM, anatoly techtonik wrote: > My advice - subscribe people editing page by default (a checkbox near > submit button). This way more people will receive notifications when a > page is changed and will be more interested to contribute themselves. > Of course, there must be a setting to opt out. My experience with the auto-nosy setting on Roundup actually makes me quite positive towards this idea. It's especially useful when coupled with the query for "Issues followed by me" and the new "add me to the nosy list" button. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From dirkjan at ochtman.nl Sun Sep 26 14:02:48 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sun, 26 Sep 2010 14:02:48 +0200 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> <4C9D21E8.1080005@canterbury.ac.nz> <4C9D298A.3010407@g.nevcal.com> <4C9E7E6B.2060409@canterbury.ac.nz> Message-ID: On Sun, Sep 26, 2010 at 13:36, Paul Moore wrote: > Hmm, I can't find the one I was thinking of. GetLongFileName correctly > sets the case of all but the final part, and FindFile can be used to > find the last part, but that's not what I recall. > > GetFinalPathNameByHandle works, and is documented to do so, but (a) it > works on an open file handle, so you need to open the file, and (b) > it's Vista and later only... FWIW, here's what Mercurial uses to get the real path name on Windows: http://hg.intevation.org/mercurial/crew/file/66a07fb76ceb/mercurial/util.py#l633 (I don't know much about that code or this topic, but maybe someone finds it useful. It doesn't use any "special" Windows API, so if there is any, it's something the hg hackers don't know about.) Cheers, Dirkjan From foom at fuhm.net Sun Sep 26 14:37:37 2010 From: foom at fuhm.net (James Y Knight) Date: Sun, 26 Sep 2010 08:37:37 -0400 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> <4C9D21E8.1080005@canterbury.ac.nz> <4C9D298A.3010407@g.nevcal.com> <4C9E7E6B.2060409@canterbury.ac.nz> Message-ID: <3707CDA2-BBE0-46DD-A1C1-61AFC393336C@fuhm.net> On Sep 26, 2010, at 7:36 AM, Paul Moore wrote: > On 26 September 2010 09:01, Paul Moore wrote: >> On 25 September 2010 23:57, Greg Ewing wrote: >>> Paul Moore wrote: >>> >>>> Windows has (I believe) user definable filesystems, too, but the OS >>>> has "get me the real filename" style calls, >>> >>> Does it really, though? The suggestions I've seen for doing >>> this involve abusing the short/long filename translation >>> machinery, and I'm not sure they're guaranteed to return the >>> actual case rather than something that happens to work. >> >> There's another call available. I've been too lazy to go and look it >> up, but I'll do so sometime today. > > Hmm, I can't find the one I was thinking of. GetLongFileName correctly > sets the case of all but the final part, and FindFile can be used to > find the last part, but that's not what I recall. > > GetFinalPathNameByHandle works, and is documented to do so, but (a) it > works on an open file handle, so you need to open the file, and (b) > it's Vista and later only... Were you thinking of SHGetFileInfo? http://stackoverflow.com/questions/74451/getting-actual-file-name-with-proper-casing-on-windows James From martin at v.loewis.de Sun Sep 26 14:59:55 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 26 Sep 2010 14:59:55 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> <4C9E6CF8.4080705@v.loewis.de> <4C9E74D0.3050704@v.loewis.de> Message-ID: <4C9F43CB.6010500@v.loewis.de> Am 26.09.2010 13:58, schrieb Nick Coghlan: > On Sun, Sep 26, 2010 at 8:16 AM, "Martin v. L?wis" wrote: >> But that can't work: then off-site links into either wiki break. > > Georg isn't suggesting a general structural change, just a change to > have the URL when you land *directly* on wiki.python.org automatically > rewritten to resolve to wiki.python.org/moin. I suppose you mean "to redirect to"? I think there would be problems if wiki.python.org actually returned the same content as wiki.python.org/moin. Regards, Martin From p.f.moore at gmail.com Sun Sep 26 15:06:38 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 26 Sep 2010 14:06:38 +0100 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: <3707CDA2-BBE0-46DD-A1C1-61AFC393336C@fuhm.net> References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> <4C9D21E8.1080005@canterbury.ac.nz> <4C9D298A.3010407@g.nevcal.com> <4C9E7E6B.2060409@canterbury.ac.nz> <3707CDA2-BBE0-46DD-A1C1-61AFC393336C@fuhm.net> Message-ID: On 26 September 2010 13:37, James Y Knight wrote: > > Were you thinking of SHGetFileInfo? > > http://stackoverflow.com/questions/74451/getting-actual-file-name-with-proper-casing-on-windows It wasn't, but it looks possible. Only gives the last component, though, so you still have to walk up the path components :-( I suspect I was thinking of GetLongFileName, which puts everything *but* the last component into the right case. I missed the problem with the last component :-( Paul. From ncoghlan at gmail.com Sun Sep 26 15:45:35 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Sep 2010 23:45:35 +1000 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9F43CB.6010500@v.loewis.de> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> <4C9E6CF8.4080705@v.loewis.de> <4C9E74D0.3050704@v.loewis.de> <4C9F43CB.6010500@v.loewis.de> Message-ID: On Sun, Sep 26, 2010 at 10:59 PM, "Martin v. L?wis" wrote: > Am 26.09.2010 13:58, schrieb Nick Coghlan: >> On Sun, Sep 26, 2010 at 8:16 AM, "Martin v. L?wis" wrote: >>> But that can't work: then off-site links into either wiki break. >> >> Georg isn't suggesting a general structural change, just a change to >> have the URL when you land *directly* on wiki.python.org automatically >> rewritten to resolve to wiki.python.org/moin. > > I suppose you mean "to redirect to"? I think there would be problems if > wiki.python.org actually returned the same content as wiki.python.org/moin. I don't actually have any specific preference as to implementation details, aside from it being something that works without breaking the rest of the web server :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From pje at telecommunity.com Sun Sep 26 15:47:35 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 26 Sep 2010 09:47:35 -0400 Subject: [Python-Dev] WSGI is now Python 3-friendly In-Reply-To: References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926020021.454453A4079@sparrow.telecommunity.com> Message-ID: <20100926134738.184093A40B2@sparrow.telecommunity.com> At 07:15 PM 9/25/2010 -0700, Guido van Rossum wrote: >Don't see this as a new spec. See it as a procedural issue. As a procedural issue, PEP 333 is an Informational PEP, in "Draft" status, which I'd like to make Final after these amendments. See http://www.wsgi.org/wsgi/Amendments_1.0, which Graham created in 2007, stating: """This page is intended to collect any ideas related to amendments to the original WSGI 1.0 so that it can be marked as 'Final'.""" IOW, there is no intention to treat the PEP as "mutable" going forward; this is just cleanup so we can mark it Final. After that, it's an ex-parrot. >Clarifications of ambiguous/unspecified behavior can possibly rule as >non-conforming implementations that used to get the benefit of the >doubt. Best-practice recommendations also have the effect of changing >(perceived) compliance. I understand the general principle, but with respect to these *specific* changes, any perceived-compliance arguments that were going to happen, already happened years ago. The changes are merely to officially document the way those arguments already turned out, so the PEP can become Final. Specifically, the changes all fall into one of three categories: 1. Textual clarification (SERVER_PORT is not an int, iteration can stop before all output is consumed) 2. Practical issues with wsgi.input arising from the fact that real-world programs needed its behavior to be more "file-like" than the specification required... and which essentially forced servers that were not using socket.makefile() to make their emulations work like that, anyway (or else be rejected by users). 3. Clarification of behavior that would break HTTP compliance (apps or servers sending more than Content-Length bytes) and is therefore *already a bug* in any implementation that does it. Since in all three categories any implementation that did not end up following the recommendations on its own is going to have been considered buggy by its users (regardless of its formal "compliance"), and because the changes do not actually declare the buggy behaviors in categories 2 and 3 to be non-compliant, I do not see how any of these changes can produce the type of problems you're worried about here. Certainly, if I thought such problems were possible, I wouldn't have accepted these amendments. Likewise, if I thought that changes would continue to be made to the PEP past this point, the goal wouldn't be getting it to Final status. From g.brandl at gmx.net Sun Sep 26 15:53:58 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 26 Sep 2010 15:53:58 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9F43CB.6010500@v.loewis.de> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> <4C9E6CF8.4080705@v.loewis.de> <4C9E74D0.3050704@v.loewis.de> <4C9F43CB.6010500@v.loewis.de> Message-ID: Am 26.09.2010 14:59, schrieb "Martin v. L?wis": > Am 26.09.2010 13:58, schrieb Nick Coghlan: >> On Sun, Sep 26, 2010 at 8:16 AM, "Martin v. L?wis" wrote: >>> But that can't work: then off-site links into either wiki break. >> >> Georg isn't suggesting a general structural change, just a change to >> have the URL when you land *directly* on wiki.python.org automatically >> rewritten to resolve to wiki.python.org/moin. > > I suppose you mean "to redirect to"? I think there would be problems if > wiki.python.org actually returned the same content as wiki.python.org/moin. I've been talking about redirecting all the time, haven't I? Plan is simple: * redirect from wiki.python.org to wiki.python.org/moin * (optionally) redirect from wiki.jython.org to wiki.python.org/jython -- or -- make wiki.jython.org the canonical URI for the Jython wiki and redirect from old /jython URIs there. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From martin at v.loewis.de Sun Sep 26 16:25:13 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 26 Sep 2010 16:25:13 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> <4C9E6CF8.4080705@v.loewis.de> <4C9E74D0.3050704@v.loewis.de> <4C9F43CB.6010500@v.loewis.de> Message-ID: <4C9F57C9.6040205@v.loewis.de> > I've been talking about redirecting all the time, haven't I? You said "put the Jython wiki somewhere on its own". That seemed to suggest it won't be anymore at wiki.python.org/jython. > * redirect from wiki.python.org to wiki.python.org/moin > > * (optionally) redirect from wiki.jython.org to wiki.python.org/jython > -- or -- > make wiki.jython.org the canonical URI for the Jython wiki and redirect > from old /jython URIs there. I've asked Frank Wierzbicki which one he prefers. Regards, Martin From tseaver at palladion.com Sun Sep 26 16:52:47 2010 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 26 Sep 2010 10:52:47 -0400 Subject: [Python-Dev] hg conversion: tags In-Reply-To: <4C9F311F.1070507@v.loewis.de> References: <4C9F311F.1070507@v.loewis.de> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/26/2010 07:40 AM, "Martin v. L?wis" wrote: >> For example, it might be interesting to bring old release >> tags in line with newer tags (so Release_1_0 would become r10), or >> maybe use clean tags similar to what we do with hg itself (r266 would >> become just 2.6.6), or just remove some tags. Is this a good idea at >> all, or should we just leave everything the way it is now? > > Removing tags sounds fine with me. Renaming them not so; I fail to see > the point. One point is to remove inconsistencies tied to CVS-era restrictions: the tags which correspond to (define) actual numbered releases have goofy names due to CVSism ('r266' is a ridiculously hard-to-figure-out alternative to '2.6.6'). Given that nobody will be able to update checkouts in place anyway, I think taking this opportunity to use "real" version IDs for tags would be a much better than sticking with the old cruft. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkyfXj4ACgkQ+gerLs4ltQ6fZQCeJIQzz4CO9D/Jc1ryeO9zhiUA /EcAn3IkPDOpJetd9kC6/gqpNqN8JHkC =0roB -----END PGP SIGNATURE----- From tseaver at palladion.com Sun Sep 26 16:58:43 2010 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 26 Sep 2010 10:58:43 -0400 Subject: [Python-Dev] WSGI is now Python 3-friendly In-Reply-To: <4C9EE8B4.7090405@v.loewis.de> References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/26/2010 02:31 AM, "Martin v. L?wis" wrote: > Am 26.09.2010 03:45, schrieb P.J. Eby: >> I'm actually a bit surprised people are bringing this up now, since when >> I announced the plan to make these changes, I said that nothing would be >> changed that would break anything > > I think people read this as "nothing would be changed, period." > > However, you did make substantial changes to the specification (or else > the whole exercise would have been pointless, I suppose, and you > couldn't have claimed that WSGI is now Python 3-friendly when it > previously was not). The clarifications remove Python3-only ambiguities, making it possible for to implement both sides the spec sanely and consistently under Python 3. > So this is essentially a new version of the spec. As PEPs themselves > are not versioned (unlike, say, ISO standards), Guido insists it ought > to get a new PEP number. Then, people declaring compliance can identify > what specification they actually comply to. Declaring compliance to > PEP 333 as-of-last-week-but-not-as-of-today is now difficult. This > particularly puzzles people some of the existing WSGI servers are > now incompatible to the PEP, when they still were compatible last > week. PJE's claim is that there are *no* such servers: > So, no (previously-)compliant implementations were harmed in the > making of the updated spec. If they were compliant before, they're > compliant now. I hadn't realized that PEP 333 was never actually in the 'Final' status (de facto, it has been so for years, of course). Given that fact, and PJEs assurances, I think amending the PEP and then immediately declaring it final is reasonable. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkyfX6MACgkQ+gerLs4ltQ7tjgCfXP4SamlyjLenSsHib0O8E03d MbEAnR+lsNoUb7PH4NkdKNL1rToWXTsi =s5d7 -----END PGP SIGNATURE----- From guido at python.org Sun Sep 26 17:20:26 2010 From: guido at python.org (Guido van Rossum) Date: Sun, 26 Sep 2010 08:20:26 -0700 Subject: [Python-Dev] WSGI is now Python 3-friendly In-Reply-To: References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> Message-ID: On Sun, Sep 26, 2010 at 7:58 AM, Tres Seaver wrote: > I hadn't realized that PEP 333 was never actually in the 'Final' status > (de facto, it has been so for years, of course). ?Given that fact, and > PJEs assurances, I think amending the PEP and then immediately declaring > it final is reasonable. And who is going to give the PEP its stamp of approval? I'm sorry, but all this weasel-wording about how these changes aren't really changes and how this standard wasn't really a standard make me very uncomfortable. I'm happy approving Final status for the *original* PEP 333 and I'm happy to approve a new PEP which includes PJE's corrections. I'm not going to approve?Final status for a history-rewrite of PEP 333. -- --Guido van Rossum (python.org/~guido) From brian.curtin at gmail.com Sun Sep 26 18:31:22 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Sun, 26 Sep 2010 11:31:22 -0500 Subject: [Python-Dev] os.path.normcase rationale? In-Reply-To: References: <4C9531A7.10405@simplistix.co.uk> <4C9C79DA.7000506@simplistix.co.uk> <20100924121737.309071FA5C2@kimball.webabinitio.net> <4C9D21E8.1080005@canterbury.ac.nz> <4C9D298A.3010407@g.nevcal.com> <4C9E7E6B.2060409@canterbury.ac.nz> Message-ID: On Sun, Sep 26, 2010 at 06:36, Paul Moore wrote: > On 26 September 2010 09:01, Paul Moore wrote: > > On 25 September 2010 23:57, Greg Ewing > wrote: > >> Paul Moore wrote: > >> > >>> Windows has (I believe) user definable filesystems, too, but the OS > >>> has "get me the real filename" style calls, > >> > >> Does it really, though? The suggestions I've seen for doing > >> this involve abusing the short/long filename translation > >> machinery, and I'm not sure they're guaranteed to return the > >> actual case rather than something that happens to work. > > > > There's another call available. I've been too lazy to go and look it > > up, but I'll do so sometime today. > > GetFinalPathNameByHandle works, and is documented to do so, but (a) it > works on an open file handle, so you need to open the file, and (b) > it's Vista and later only... FYI, this is currently exposed as nt._getfinalpathname, and is used for os.path.samefile on Vista and beyond. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Sun Sep 26 18:50:20 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 26 Sep 2010 18:50:20 +0200 Subject: [Python-Dev] WSGI is now Python 3-friendly In-Reply-To: References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> Message-ID: <4C9F79CC.80306@v.loewis.de> >> I think people read this as "nothing would be changed, period." > >> However, you did make substantial changes to the specification (or else >> the whole exercise would have been pointless, I suppose, and you >> couldn't have claimed that WSGI is now Python 3-friendly when it >> previously was not). > > The clarifications remove Python3-only ambiguities, making it possible > for to implement both sides the spec sanely and consistently under Python 3. Sure. I don't argue whether this is an improvement. I claim that it is a specification change, long after the specification has been implemented multiple times. It thus deserves to get a new name (i.e. PEP number). PJE says that strictly speaking, the PEP had not been accepted yet. However, at this stage, claiming that it is still open for edits is really confusing, IMO. >> So this is essentially a new version of the spec. As PEPs themselves >> are not versioned (unlike, say, ISO standards), Guido insists it ought >> to get a new PEP number. Then, people declaring compliance can identify >> what specification they actually comply to. Declaring compliance to >> PEP 333 as-of-last-week-but-not-as-of-today is now difficult. This >> particularly puzzles people some of the existing WSGI servers are >> now incompatible to the PEP, when they still were compatible last >> week. > > PJE's claim is that there are *no* such servers: Does that claim include wsgiref? > I hadn't realized that PEP 333 was never actually in the 'Final' status > (de facto, it has been so for years, of course). Given that fact, and > PJEs assurances, I think amending the PEP and then immediately declaring > it final is reasonable. I'm still with Guido here: I'd accept PEP 333 as final in the state it had last week, give PJE's edits a new PEP number, and accept that as final right away also. Regards, Martin From martin at v.loewis.de Sun Sep 26 18:51:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 26 Sep 2010 18:51:07 +0200 Subject: [Python-Dev] WSGI is now Python 3-friendly In-Reply-To: References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> Message-ID: <4C9F79FB.3040901@v.loewis.de> > I'm sorry, but all this weasel-wording about how these changes aren't > really changes and how this standard wasn't really a standard make me > very uncomfortable. I'm happy approving Final status for the > *original* PEP 333 and I'm happy to approve a new PEP which includes > PJE's corrections. I'm not going to approve Final status for a > history-rewrite of PEP 333. +1 Martin From martin at v.loewis.de Sun Sep 26 18:54:24 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 26 Sep 2010 18:54:24 +0200 Subject: [Python-Dev] hg conversion: tags In-Reply-To: References: <4C9F311F.1070507@v.loewis.de> Message-ID: <4C9F7AC0.3080509@v.loewis.de> > One point is to remove inconsistencies tied to CVS-era restrictions: > the tags which correspond to (define) actual numbered releases have > goofy names due to CVSism ('r266' is a ridiculously hard-to-figure-out > alternative to '2.6.6'). > > Given that nobody will be able to update checkouts in place anyway, I > think taking this opportunity to use "real" version IDs for tags would > be a much better than sticking with the old cruft. Please understand that this will also delay the introduction of Mercurial. Not only will the tag names have to be changed, but also all software doing automatic processing of tag names will have to be updated (in addition to being ported to Mercurial, which is necessary either way). But then, if somebody volunteers to make these changes, I'm -0 to the renaming (I slightly prefer calling even future release tags rXYZ). Regards, Martin From martin at v.loewis.de Sun Sep 26 18:55:58 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 26 Sep 2010 18:55:58 +0200 Subject: [Python-Dev] PyObject_GC_UnTrack() no longer reliable in 2.7? In-Reply-To: References: <4C9D1391.20504@v.loewis.de> Message-ID: <4C9F7B1E.5040404@v.loewis.de> Am 26.09.2010 12:54, schrieb Nick Coghlan: > On Sat, Sep 25, 2010 at 9:20 AM, Tim Peters wrote: > [MvL] >>> I think it would be possible to have two versions of >>> _PyGC_REFS_UNTRACKED, one being, say, -5. >>> _PyGC_REFS_UNTRACKED_AND_KEEP_IT_THAT_WAY would be what you get >>> when you call PyObject_GC_UnTrack; the code to do automatic >>> tracking/untracking based on contents would use some other >>> new API (which would be non-public in 2.7.x). >> >> Looks like a promising idea! gcmodule.c's IS_TRACKED macro would have >> to change to check both states, and likewise the debug assert in >> visit_reachable(). > > Given the intent is to restore the 2.6 state, perhaps it is the > "UNTRACK_ALLOW_RETRACK" optimisation that should get the new special > value? Other than that, MvL's suggestion looks like a good idea. It would work either way, of course. Regards, Martin From pje at telecommunity.com Sun Sep 26 19:33:44 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 26 Sep 2010 13:33:44 -0400 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> Message-ID: <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> At 08:20 AM 9/26/2010 -0700, Guido van Rossum wrote: >I'm happy approving Final status for the >*original* PEP 333 and I'm happy to approve a new PEP which includes >PJE's corrections. Can we make it PEP 3333, then? ;-) That number would at least communicate that it's the same thing, but for Python 3. Really, my reason for trying to do the (non Py3-specific) amendments in a way that didn't require a new PEP number was because of the many ancillary questions that it raises for the community, such as: * Is this is some sort of competition/replacement to PEP 444? * What happened to the old one, why can't we just use that? * Why isn't there a different protocol version? * How is this different from the old one? To be fair, I *also* wanted to avoid all the work associated with *answering* them. ;-) (Heck, I really wanted to avoid the work of having to even *think* about which questions *might* arise and how they'd need to be addressed.) OTOH, I can certainly see that my attempt to avoid this has *already* failed: it simply brought up a different set of questions, just on Python-Dev instead of Web-SIG or Python-list. Oh well. Perhaps making the numbering appear to be a continuation will help a bit. Another option would be to make a PEP that consists solely of the amendments and errata themselves, as this would answer most of the above questions directly. Still another would be to abandon the effort to amend the PEP, and simply leave things as they are now: AFAICT, the fact that these amendments aren't in the PEP hasn't stopped anybody from *treating* most of them as if they were. (Because everyone understands that failure to follow them constitutes a bug in your program, even if it technically complies with the spec.) From vinay_sajip at yahoo.co.uk Sun Sep 26 20:01:32 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 26 Sep 2010 18:01:32 +0000 (UTC) Subject: [Python-Dev] WSGI is now Python 3-friendly References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <4C9F79CC.80306@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > I'm still with Guido here: I'd accept PEP 333 as final in the state it > had last week, give PJE's edits a new PEP number, and accept that as > final right away also. > This sounds like it should make everyone happy - no rewriting of history, and no procedural roadblocks or delays to the amendments PJE wants to make. Are we done here, or am I missing something? From tjreedy at udel.edu Sun Sep 26 20:25:26 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 26 Sep 2010 14:25:26 -0400 Subject: [Python-Dev] Goodbye In-Reply-To: References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9C6A6F.6010202@netwok.org> <20100924142639.0e212df3@pitrou.net> <1285338660.3243.1.camel@localhost.localdomain> <1285339858.3243.4.camel@localhost.localdomain> <1285341131.3243.7.camel@localhost.localdomain> Message-ID: On 9/26/2010 7:43 AM, Nick Coghlan wrote: > Yep, hence why I think the basic "bug, feature, other" split may be a > good way to go. It's a quick and easy decision most of the time > (including a clear choice for "I don't know if this is a bug or not"), > and makes a meaningful procedural distinction (bugs are usually > backported, new features are not, and other changes may be backported > depending on the details). +1 on 3 categories. The categories other than the main two are used so seldom as to make the differentiation pretty useless. -- Terry Jan Reedy From tjreedy at udel.edu Sun Sep 26 20:31:31 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 26 Sep 2010 14:31:31 -0400 Subject: [Python-Dev] hg conversion: tags In-Reply-To: <4C9F7AC0.3080509@v.loewis.de> References: <4C9F311F.1070507@v.loewis.de> <4C9F7AC0.3080509@v.loewis.de> Message-ID: On 9/26/2010 12:54 PM, "Martin v. L?wis" wrote: > But then, if somebody volunteers to make these changes, I'm -0 to > the renaming (I slightly prefer calling even future release tags rXYZ). Except that r311 could be either 3.1.1 or 3.11 (should be ever get that far ;-). -- Terry Jan Reedy From steve at holdenweb.com Sun Sep 26 20:31:25 2010 From: steve at holdenweb.com (Steve Holden) Date: Sun, 26 Sep 2010 14:31:25 -0400 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9F57C9.6040205@v.loewis.de> References: <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> <4C9E6CF8.4080705@v.l oewis.de> <4C9E74D0.3050704@v.loewis.de> <4C9F43CB.6010500@v.loewis.de> <4C9F57C9.6040205@v.loewis.de> Message-ID: <4C9F917D.2020001@holdenweb.com> On 9/26/2010 10:25 AM, "Martin v. L?wis" wrote: >> I've been talking about redirecting all the time, haven't I? > > You said "put the Jython wiki somewhere on its own". That seemed to > suggest it won't be anymore at wiki.python.org/jython. > >> * redirect from wiki.python.org to wiki.python.org/moin >> >> * (optionally) redirect from wiki.jython.org to wiki.python.org/jython >> -- or -- >> make wiki.jython.org the canonical URI for the Jython wiki and redirect >> from old /jython URIs there. > > I've asked Frank Wierzbicki which one he prefers. > Strikes me this is a much needed change. Thanks! regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From tjreedy at udel.edu Sun Sep 26 20:59:22 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 26 Sep 2010 14:59:22 -0400 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> Message-ID: On 9/26/2010 1:33 PM, P.J. Eby wrote: Thank you do doing the needed rewrite. > Can we make it PEP 3333, then? ;-) > > That number would at least communicate that it's the same thing, but for > Python 3. A new rewriten PEP gives you a bit more freedom than doing it in place. It will be easier to refer to the existing PEP 333 rather than "an earlier version of this PEP". > > Really, my reason for trying to do the (non Py3-specific) amendments in > a way that didn't require a new PEP number was because of the many > ancillary questions that it raises for the community, such as: > > * Is this is some sort of competition/replacement to PEP 444? > * What happened to the old one, why can't we just use that? > * Why isn't there a different protocol version? You can also (briefly) answer questions like these in a new section. I would refer people to the web-sig if they have further questions. > * How is this different from the old one? You could mark added material is a way that does not conflict with rst or html. Or use .rst to make new text stand out in the .html web verion (bold, underlined, red, or whatever). People familiar with 333 can focus on the marked sections. New readers can ignore the marking. > To be fair, I *also* wanted to avoid all the work associated with > *answering* them. ;-) (Heck, I really wanted to avoid the work of having > to even *think* about which questions *might* arise and how they'd need > to be addressed.) > > OTOH, I can certainly see that my attempt to avoid this has *already* > failed: it simply brought up a different set of questions, just on > Python-Dev instead of Web-SIG or Python-list. You can't win in situations like this. > Oh well. Perhaps making the numbering appear to be a continuation will > help a bit. > > Another option would be to make a PEP that consists solely of the > amendments and errata themselves, as this would answer most of the above > questions directly. Please no. Terrible to read. Mark important changes, as suggested above, in a complete text. > > Still another would be to abandon the effort to amend the PEP, and > simply leave things as they are now: AFAICT, the fact that these > amendments aren't in the PEP hasn't stopped anybody from *treating* most > of them as if they were. (Because everyone understands that failure to > follow them constitutes a bug in your program, even if it technically > complies with the spec.) Please no ;-). -- Terry Jan Reedy From martin at v.loewis.de Sun Sep 26 21:08:59 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 26 Sep 2010 21:08:59 +0200 Subject: [Python-Dev] hg conversion: tags In-Reply-To: References: <4C9F311F.1070507@v.loewis.de> <4C9F7AC0.3080509@v.loewis.de> Message-ID: <4C9F9A4B.60402@v.loewis.de> Am 26.09.2010 20:31, schrieb Terry Reedy: > On 9/26/2010 12:54 PM, "Martin v. L?wis" wrote: > >> But then, if somebody volunteers to make these changes, I'm -0 to >> the renaming (I slightly prefer calling even future release tags rXYZ). > > Except that r311 could be either 3.1.1 or 3.11 (should be ever get that > far ;-). And indeed, we cannot possibly go that far. Guido has Pronounced long ago that version numbers can go only from 0 to 9. Regards, Martin From p.f.moore at gmail.com Sun Sep 26 21:14:00 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 26 Sep 2010 20:14:00 +0100 Subject: [Python-Dev] WSGI is now Python 3-friendly In-Reply-To: References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <4C9F79CC.80306@v.loewis.de> Message-ID: On 26 September 2010 19:01, Vinay Sajip wrote: > Martin v. L?wis v.loewis.de> writes: > >> >> I'm still with Guido here: I'd accept PEP 333 as final in the state it >> had last week, give PJE's edits a new PEP number, and accept that as >> final right away also. >> > > This sounds like it should make everyone happy - no rewriting of history, and no > procedural roadblocks or delays to the amendments PJE wants to make. Are we done > here, or am I missing something? Sounds fine to me (PJE's response to Guido seems to imply he might still have an issue, but I really hope not - let's get the new version finalised and move on). Paul. From barry at python.org Sun Sep 26 21:47:48 2010 From: barry at python.org (Barry Warsaw) Date: Sun, 26 Sep 2010 15:47:48 -0400 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> Message-ID: On Sep 26, 2010, at 1:33 PM, P.J. Eby wrote: > At 08:20 AM 9/26/2010 -0700, Guido van Rossum wrote: >> I'm happy approving Final status for the >> *original* PEP 333 and I'm happy to approve a new PEP which includes >> PJE's corrections. > > Can we make it PEP 3333, then? ;-) That works for me. -Barry From guido at python.org Sun Sep 26 22:44:15 2010 From: guido at python.org (Guido van Rossum) Date: Sun, 26 Sep 2010 13:44:15 -0700 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> Message-ID: On Sun, Sep 26, 2010 at 12:47 PM, Barry Warsaw wrote: > On Sep 26, 2010, at 1:33 PM, P.J. Eby wrote: > >> At 08:20 AM 9/26/2010 -0700, Guido van Rossum wrote: >>> I'm happy approving Final status for the >>> *original* PEP 333 and I'm happy to approve a new PEP which includes >>> PJE's corrections. >> >> Can we make it PEP 3333, then? ?;-) > > That works for me. Go for it. -- --Guido van Rossum (python.org/~guido) From pje at telecommunity.com Sun Sep 26 22:50:07 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 26 Sep 2010 16:50:07 -0400 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> Message-ID: <20100926205006.8A00A3A411A@sparrow.telecommunity.com> At 01:44 PM 9/26/2010 -0700, Guido van Rossum wrote: >On Sun, Sep 26, 2010 at 12:47 PM, Barry Warsaw wrote: > > On Sep 26, 2010, at 1:33 PM, P.J. Eby wrote: > > > >> At 08:20 AM 9/26/2010 -0700, Guido van Rossum wrote: > >>> I'm happy approving Final status for the > >>> *original* PEP 333 and I'm happy to approve a new PEP which includes > >>> PJE's corrections. > >> > >> Can we make it PEP 3333, then? ;-) > > > > That works for me. > >Go for it. Shall I just "svn cp" it, then (to preserve edit history), or wait for the PEP editor do it? From guido at python.org Sun Sep 26 22:56:54 2010 From: guido at python.org (Guido van Rossum) Date: Sun, 26 Sep 2010 13:56:54 -0700 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: <20100926205006.8A00A3A411A@sparrow.telecommunity.com> References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> <20100926205006.8A00A3A411A@sparrow.telecommunity.com> Message-ID: Since you have commit privileges, just do it. The PEP editor position mostly exists to assure non-committers are not prevented from authoring PEPs. Please do add a prominent note at the top of PEP 333 pointing to PEP 3333 for further information on Python 3 compliance or some such words. Add a similar note at the top of PEP 3333 -- maybe mark up the differences in PEP 3333 so people can easily tell what was added. And move PEP 333 to Final status. --Guido On Sun, Sep 26, 2010 at 1:50 PM, P.J. Eby wrote: > At 01:44 PM 9/26/2010 -0700, Guido van Rossum wrote: >> >> On Sun, Sep 26, 2010 at 12:47 PM, Barry Warsaw wrote: >> > On Sep 26, 2010, at 1:33 PM, P.J. Eby wrote: >> > >> >> At 08:20 AM 9/26/2010 -0700, Guido van Rossum wrote: >> >>> I'm happy approving Final status for the >> >>> *original* PEP 333 and I'm happy to approve a new PEP which includes >> >>> PJE's corrections. >> >> >> >> Can we make it PEP 3333, then? ?;-) >> > >> > That works for me. >> >> Go for it. > > Shall I just "svn cp" it, then (to preserve edit history), or wait for the > PEP editor do it? > > -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Sun Sep 26 23:34:24 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 27 Sep 2010 07:34:24 +1000 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> <20100926205006.8A00A3A411A@sparrow.telecommunity.com> Message-ID: On Mon, Sep 27, 2010 at 6:56 AM, Guido van Rossum wrote: > Since you have commit privileges, just do it. The PEP editor position > mostly exists to assure non-committers are not prevented from > authoring PEPs. > > Please do add a prominent note at the top of PEP 333 pointing to PEP > 3333 for further information on Python 3 compliance or some such > words. Add a similar note at the top of PEP 3333 -- maybe mark up the > differences in PEP 3333 so people can easily tell what was added. And > move PEP 333 to Final status. Since PEP 333 is referred to as both by both its PEP number and as WSGI 1.0, should the new PEP explicitly state that PEP 3333 is WSGI 1.0.1? (i.e. not a full update, even to 1.1 status, just some small tweaks to make it usable in the Python 3 world along with some generally applicable clarifications). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From pje at telecommunity.com Mon Sep 27 01:57:35 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 26 Sep 2010 19:57:35 -0400 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> <20100926205006.8A00A3A411A@sparrow.telecommunity.com> Message-ID: <20100926235731.CF1BD3A4114@sparrow.telecommunity.com> Done. The other amendments were never actually made, so I just reverted the Python 3 bit after moving it to the new PEP. I'll make the changes to 3333 instead as soon as I have another time slot free. At 01:56 PM 9/26/2010 -0700, Guido van Rossum wrote: >Since you have commit privileges, just do it. The PEP editor position >mostly exists to assure non-committers are not prevented from >authoring PEPs. > >Please do add a prominent note at the top of PEP 333 pointing to PEP >3333 for further information on Python 3 compliance or some such >words. Add a similar note at the top of PEP 3333 -- maybe mark up the >differences in PEP 3333 so people can easily tell what was added. And >move PEP 333 to Final status. > >--Guido > >On Sun, Sep 26, 2010 at 1:50 PM, P.J. Eby wrote: > > At 01:44 PM 9/26/2010 -0700, Guido van Rossum wrote: > >> > >> On Sun, Sep 26, 2010 at 12:47 PM, Barry Warsaw wrote: > >> > On Sep 26, 2010, at 1:33 PM, P.J. Eby wrote: > >> > > >> >> At 08:20 AM 9/26/2010 -0700, Guido van Rossum wrote: > >> >>> I'm happy approving Final status for the > >> >>> *original* PEP 333 and I'm happy to approve a new PEP which includes > >> >>> PJE's corrections. > >> >> > >> >> Can we make it PEP 3333, then? ;-) > >> > > >> > That works for me. > >> > >> Go for it. > > > > Shall I just "svn cp" it, then (to preserve edit history), or wait for the > > PEP editor do it? > > > > > > > >-- >--Guido van Rossum (python.org/~guido) >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >http://mail.python.org/mailman/options/python-dev/pje%40telecommunity.com From pje at telecommunity.com Mon Sep 27 02:13:42 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 26 Sep 2010 20:13:42 -0400 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> Message-ID: <20100927001338.492EE3A4114@sparrow.telecommunity.com> At 02:59 PM 9/26/2010 -0400, Terry Reedy wrote: >You could mark added material is a way that does not conflict with >rst or html. Or use .rst to make new text stand out in the .html web >verion (bold, underlined, red, or whatever). People familiar with >333 can focus on the marked sections. New readers can ignore the marking. If you (or anybody else) have any idea how to do that (highlight stuff in PEP-dialect .rst), let me know. (For that matter, if anybody knows how to make it not turn *every* PEP reference into a link, that'd be good too! It doesn't really need to turn 5 or 6 occurrences of "PEP 333" in the same paragraph into separate links. ;-) ) From ben+python at benfinney.id.au Mon Sep 27 03:15:06 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 27 Sep 2010 11:15:06 +1000 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> <20100927001338.492EE3A4114@sparrow.telecommunity.com> Message-ID: <87r5gg2did.fsf@benfinney.id.au> "P.J. Eby" writes: > (For that matter, if anybody knows how to make it not turn *every* PEP > reference into a link, that'd be good too! It doesn't really need to > turn 5 or 6 occurrences of "PEP 333" in the same paragraph into > separate links. ;-) ) reST, being designed explicitly for Python documentation, has support for PEP references built in: The :pep-reference: role is used to create an HTTP reference to a PEP (Python Enhancement Proposal). The :PEP: alias is usually used. For example: See :PEP:`287` for more information about reStructuredText. This is equivalent to: See `PEP 287`__ for more information about reStructuredText. __ http://www.python.org/peps/pep-0287.html . -- \ ?What is needed is not the will to believe but the will to find | `\ out, which is the exact opposite.? ?Bertrand Russell | _o__) | Ben Finney From pje at telecommunity.com Mon Sep 27 03:38:53 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 26 Sep 2010 21:38:53 -0400 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: <87r5gg2did.fsf@benfinney.id.au> References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> <20100927001338.492EE3A4114@sparrow.telecommunity.com> <87r5gg2did.fsf@benfinney.id.au> Message-ID: <20100927013849.688EC3A4114@sparrow.telecommunity.com> At 11:15 AM 9/27/2010 +1000, Ben Finney wrote: >"P.J. Eby" <pje >at telecommunity.com> writes: > > > (For that matter, if anybody knows how to make it not turn *every* PEP > > reference into a link, that'd be good too! It doesn't really need to > > turn 5 or 6 occurrences of "PEP 333" in the same paragraph into > > separate links. ;-) ) > >reST, being designed explicitly for Python documentation, has support >for PEP references built in: You misunderstand me; I wasn't asking how to *add* a link, but how to turn OFF the automatic conversion of the phrase "PEP 333" that happens without any special markup. Currently, the PEP 3333 preface is littered with unnecessary links, because the PEP pre-processor turns *every* mere textual mention of a PEP into a link to it. From ben+python at benfinney.id.au Mon Sep 27 03:46:48 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 27 Sep 2010 11:46:48 +1000 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> <20100927001338.492EE3A4114@sparrow.telecommunity.com> <87r5gg2did.fsf@benfinney.id.au> <20100927013849.688EC3A4114@sparrow.telecommunity.com> Message-ID: <87mxr42c1j.fsf@benfinney.id.au> "P.J. Eby" writes: > At 11:15 AM 9/27/2010 +1000, Ben Finney wrote: > > >reST, being designed explicitly for Python documentation, has support > >for PEP references built in: > > You misunderstand me; I wasn't asking how to *add* a link, but how to > turn OFF the automatic conversion of the phrase "PEP 333" that happens > without any special markup. [?] Right, I misread. Sorry for the noise. -- \ ?Beware of bugs in the above code; I have only proved it | `\ correct, not tried it.? ?Donald Knuth, 1977-03-29 | _o__) | Ben Finney From scott+python-dev at scottdial.com Mon Sep 27 03:56:20 2010 From: scott+python-dev at scottdial.com (Scott Dial) Date: Sun, 26 Sep 2010 21:56:20 -0400 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9EF25E.4050805@v.loewis.de> References: <201009252112.37935.paul@boddie.org.uk> <4C9E6B96.2080401@v.loewis.de> <4C9EE4D2.80502@scottdial.com> <4C9EF25E.4050805@v.loewis.de> Message-ID: <4C9FF9C4.5080502@scottdial.com> On 9/26/2010 3:12 AM, Martin v. L?wis wrote: >> 2) The URL registered with the OpenID provider is a bit of a wart: >> "http://pypi.python.org/pypi?:action=openid_return" vs. >> "http://bitbucket.org/" > > You mean, as this is what the provider then shows you for confirmation? The provider also lists the trusted sites by these return URLs, and that is where I saw it as being a bit of a wart. I use the OpenID plugin for WordPress as my provider, so it may be that it doesn't do this correctly. I noticed that Google shows just "pypi.python.org", but the WordPress plugin shows that return URL instead. Nevertheless, I agree that it's too late/not worth it to change that now. > I think there is no way out wrt. to the basic auth prompt. I could > label the "Login" link "Password login" if you think this would help. The basic auth prompt doesn't bother me so much as the fact that the 401 doesn't have a "Use OpenID [Google] [myOpenID] [Launchpad]" set of links; you have to use the brower's back button because the only links offered are to register or reset your password. > Preventing the browser from prompting the user on the chance they > might want to enter an OpenID is not possible, and stopping to use > basic authentication is not feasible. In theory, you could catch usernames that started with "http://", but I imagine that only "ultra geeks" know their URIs (I have no idea what the URI for a Google account is). But, I don't see this as being worthwhile either; I just think it would be nice if the 401 page gave a quick way to correct one's mistake that didn't involve the back button. > And again, enjoying a short OpenID URI > probably does put you in the "ultra geek" category (which I > seriously don't mean as an offense). No offense taken. :) -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From rdmurray at bitdance.com Mon Sep 27 05:45:26 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 26 Sep 2010 23:45:26 -0400 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9FF9C4.5080502@scottdial.com> References: <201009252112.37935.paul@boddie.org.uk> <4C9E6B96.2080401@v.loewis.de> <4C9EE4D2.80502@scottdial.com> <4C9EF25E.4050805@v.loewis.de> <4C9FF9C4.5080502@scottdial.com> Message-ID: <20100927034526.EE69221AE9A@kimball.webabinitio.net> On Sun, 26 Sep 2010 21:56:20 -0400, Scott Dial wrote: > On 9/26/2010 3:12 AM, Martin v. Loewis wrote: > > Preventing the browser from prompting the user on the chance they > > might want to enter an OpenID is not possible, and stopping to use > > basic authentication is not feasible. > > In theory, you could catch usernames that started with "http://", but I No, Martin really meant "not possible": once basic auth is started, the browser prompts for username and password and you are in basic-auth land thereafter; the web server has *no way* to tell the browser to *stop* using basic auth. > imagine that only "ultra geeks" know their URIs (I have no idea what the > URI for a Google account is). But, I don't see this as being worthwhile Well, my OpenId is 'david.bitdance.com', so even if you could get around the basic auth problem, looking for "http://" wouldn't work. -- R. David Murray www.bitdance.com From martin at v.loewis.de Mon Sep 27 06:10:39 2010 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 27 Sep 2010 06:10:39 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: <20100927034526.EE69221AE9A@kimball.webabinitio.net> References: <201009252112.37935.paul@boddie.org.uk> <4C9E6B96.2080401@v.loewis.de> <4C9EE4D2.80502@scottdial.com> <4C9EF25E.4050805@v.loewis.de> <4C9FF9C4.5080502@scottdial.com> <20100927034526.EE69221AE9A@kimball.webabinitio.net> Message-ID: <4CA0193F.5080805@v.loewis.de> > No, Martin really meant "not possible": once basic auth is started, > the browser prompts for username and password and you are in basic-auth > land thereafter; the web server has *no way* to tell the browser to > *stop* using basic auth. Yes, but Scott proposed that OpenID users might fill in their OpenID in the username field and leave the password field empty. Technically, this would work - the browser would then get the OpenID redirect in response to the original request. >> imagine that only "ultra geeks" know their URIs (I have no idea what the >> URI for a Google account is). But, I don't see this as being worthwhile > > Well, my OpenId is 'david.bitdance.com', so even if you could get around > the basic auth problem, looking for "http://" wouldn't work. Sure - however, it would actually be possible to determine that this is an OpenID: perform discovery on it. If that succeeds, try to establish a provider association; if that also succeeds, redirect the user to the OpenID login process. However, I'd rather not do that, since OpenID users don't expect that kind of interface. Providing OpenID links on the login failure 401 response is reasonable, though. Regards, Martin From scott+python-dev at scottdial.com Mon Sep 27 06:30:30 2010 From: scott+python-dev at scottdial.com (Scott Dial) Date: Mon, 27 Sep 2010 00:30:30 -0400 Subject: [Python-Dev] Python wiki In-Reply-To: <20100927034526.EE69221AE9A@kimball.webabinitio.net> References: <201009252112.37935.paul@boddie.org.uk> <4C9E6B96.2080401@v.loewis.de> <4C9EE4D2.80502@scottdial.com> <4C9EF25E.4050805@v.loewis.de> <4C9FF9C4.5080502@scottdial.com> <20100927034526.EE69221AE9A@kimball.webabinitio.net> Message-ID: <4CA01DE6.4010402@scottdial.com> On 9/26/2010 11:45 PM, R. David Murray wrote: > On Sun, 26 Sep 2010 21:56:20 -0400, Scott Dial wrote: >> On 9/26/2010 3:12 AM, Martin v. Loewis wrote: >>> Preventing the browser from prompting the user on the chance they >>> might want to enter an OpenID is not possible, and stopping to use >>> basic authentication is not feasible. >> >> In theory, you could catch usernames that started with "http://", but I > > No, Martin really meant "not possible": once basic auth is started, > the browser prompts for username and password and you are in basic-auth > land thereafter; the web server has *no way* to tell the browser to > *stop* using basic auth. I agree that once you reply with a 401 that you will prompt the user, but my point was what "username" means in the Authorization header is open to interpretation by the HTTP server and/or script handling the GET request. >> imagine that only "ultra geeks" know their URIs (I have no idea what the >> URI for a Google account is). But, I don't see this as being worthwhile > > Well, my OpenId is 'david.bitdance.com', so even if you could get around > the basic auth problem, looking for "http://" wouldn't work. That's actually not a valid OpenID[1], but the OpenID specification says a relaying party "MUST" normalize identifiers[2] (in this case, prepending the "http://"). I believe bugs.python.org does this by checking for a username first(?), and failing any matches, it normalizes it for OpenID discovery. Otherwise, I can always use the canonical form of my ID "http://scottdial.com" to login (assuming ':' and '/' are illegal characters for usernames). I say all this not with the intent of saying pypi *needs* this, but to refute the notion that OpenID must be clumsy to use. [1] http://openid.net/specs/openid-authentication-2_0.html """ Identifier: An Identifier is either a "http" or "https" URI, (commonly referred to as a "URL" within this document), or an XRI (Reed, D. and D. McAlpin, ?Extensible Resource Identifier (XRI) Syntax V2.0,? .) [XRI_Syntax_2.0]. """ [2] http://openid.net/specs/openid-authentication-2_0.html#normalization """ 3. Otherwise, the input SHOULD be treated as an http URL; if it does not include a "http" or "https" scheme, the Identifier MUST be prefixed with the string "http://". If the URL contains a fragment part, it MUST be stripped off together with the fragment delimiter character "#". See Section 11.5.2 (HTTP and HTTPS URL Identifiers) for more information. """ -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From amk at amk.ca Mon Sep 27 14:31:38 2010 From: amk at amk.ca (A.M. Kuchling) Date: Mon, 27 Sep 2010 08:31:38 -0400 Subject: [Python-Dev] Python wiki In-Reply-To: References: <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> <4C9E6CF8.4080705@v.loewis.de> <4C9E74D0.3050704@v.loewis.de> <4C9F43CB.6010500@v.loewis.de> Message-ID: <20100927123138.GA3241@amk-desktop.matrixgroup.net> On Sun, Sep 26, 2010 at 03:53:58PM +0200, Georg Brandl wrote: > * redirect from wiki.python.org to wiki.python.org/moin I've added a element to the top page of wiki.python.org, so browsers will now jump to the /moin/ page immediately. This won't help crawlers that don't parse the HTML, but that probably doesn't matter. --amk From g.brandl at gmx.net Mon Sep 27 16:52:37 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 27 Sep 2010 16:52:37 +0200 Subject: [Python-Dev] hg conversion: tags In-Reply-To: References: Message-ID: Am 26.09.2010 12:55, schrieb Dirkjan Ochtman: > Hi all, > > I've recently been working on the conversion more (since my thesis got > finished). I finally wrote the script that splits the release branches > from the feature branches, so that we can include the former in the > main repository and keep the latter in separate clones as needed. > Next, I wanted to look into tags. There's a big list of tags (see > below), and I wonder if I should clean that up or if we should leave > it as-is. For example, it might be interesting to bring old release > tags in line with newer tags (so Release_1_0 would become r10), or > maybe use clean tags similar to what we do with hg itself (r266 would > become just 2.6.6), or just remove some tags. Is this a good idea at > all, or should we just leave everything the way it is now? I'd remove as many tags as makes sense, only keeping the release tags. Most others were made to quickly go back to a version before some change happened; however nobody would want to go back there anymore now. Just like my *-before-rstdocs tags, which I guess nobody ever used. As for how to call the releases, while I'd prefer a bit less cryptic names, keeping the rXYZ convention is fine with me. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From g.brandl at gmx.net Mon Sep 27 16:57:40 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 27 Sep 2010 16:57:40 +0200 Subject: [Python-Dev] r85028 - in python/branches/py3k: Doc/c-api/init.rst Include/pythonrun.h Modules/getpath.c PC/getpathp.c In-Reply-To: <20100927053254.DAD4AEE9DC@mail.python.org> References: <20100927053254.DAD4AEE9DC@mail.python.org> Message-ID: Am 27.09.2010 07:32, schrieb kristjan.jonsson: > Author: kristjan.jonsson > Date: Mon Sep 27 07:32:54 2010 > New Revision: 85028 > > Log: > issue 9910 > Add a Py_SetPath api to override magic path computations when starting up python. > > Modified: > python/branches/py3k/Doc/c-api/init.rst > python/branches/py3k/Include/pythonrun.h > python/branches/py3k/Modules/getpath.c > python/branches/py3k/PC/getpathp.c > > Modified: python/branches/py3k/Doc/c-api/init.rst > +.. cfunction:: void Py_SetPath(const wchar_t *) > + > + .. index:: > + triple: module; search; path > + single: path (in module sys) > + single: Py_GetPath() > + > + Set the default module search path. If this function is called before > + :cfunc: `Py_Initialize` then :cfunc: Py_GetPath won't attempt to compute > + a default serarch path but uses the provided one in stead. This is useful > + if Python is being embedded by an application that has full knowledge > + of the location of all modules. The path components should be separated > + by semicolons. > + > + This also causes `sys.executable` to be set only to the raw program name > + (see :cfunc:`Py_SetProgramName`) and `for sys.prefix` and > + `sys.exec_prefix` to be empty. It is up to the caller to modify these if > + required after calling :cfunc: `Py_Initialize`. > + This needs a versionadded. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From barry at python.org Mon Sep 27 17:28:51 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 27 Sep 2010 11:28:51 -0400 Subject: [Python-Dev] hg conversion: tags In-Reply-To: References: Message-ID: <20100927112851.1455ea34@mission> On Sep 27, 2010, at 04:52 PM, Georg Brandl wrote: >I'd remove as many tags as makes sense, only keeping the release tags. >Most others were made to quickly go back to a version before some >change happened; however nobody would want to go back there anymore >now. Just like my *-before-rstdocs tags, which I guess nobody ever >used. > >As for how to call the releases, while I'd prefer a bit less cryptic >names, keeping the rXYZ convention is fine with me. We're still going to keep the Subversion repository in read-only even after the conversion. Will the mapping between svn and hg revision numbers be preserved somehow? If so, then I say in hg, nuke the tags we don't care about and sanitize the release tags to something that obvious and can't collide (e.g. r311 is 3.1.1 or 3.11? - yes despite Guido's Rule of Version Numbering). I'd personally be fine using the hexversion or even "3.1.1" as the tag name. Since we're going to live with the converted repository until the Next Big Thing In Version Control , let's take the opportunity now to clean things up and make them nice. I do think we should keep a mapping from new to old though. If that's not possible within the hg repository, can we at least generate a text file or some such and commit that? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From dirkjan at ochtman.nl Mon Sep 27 17:32:22 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Mon, 27 Sep 2010 17:32:22 +0200 Subject: [Python-Dev] hg conversion: tags In-Reply-To: <20100927112851.1455ea34@mission> References: <20100927112851.1455ea34@mission> Message-ID: On Mon, Sep 27, 2010 at 17:28, Barry Warsaw wrote: > I do think we should keep a mapping from new to old though. ?If that's not > possible within the hg repository, can we at least generate a text file or > some such and commit that? I'm planning an extension so that at least the web interface will take r432432 and give you the corresponding hg changeset (if available!). Those who want it will also be able to install it in their local clones. Cheers, Dirkjan From alexander.belopolsky at gmail.com Mon Sep 27 18:25:17 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 27 Sep 2010 12:25:17 -0400 Subject: [Python-Dev] hg conversion: tags In-Reply-To: <20100927112851.1455ea34@mission> References: <20100927112851.1455ea34@mission> Message-ID: On Mon, Sep 27, 2010 at 11:28 AM, Barry Warsaw wrote: .. > If so, then I say in hg, nuke the tags we don't care about and sanitize the > release tags to something that obvious and can't collide (e.g. r311 is 3.1.1 > or 3.11? - yes despite Guido's Rule of Version Numbering). ?I'd personally be > fine using the hexversion or even "3.1.1" as the tag name. > +1 Apart from being ambiguous, rXYZ notation conflicts with the tracker autoformat rule that links rNNN to svn revision. From solipsis at pitrou.net Mon Sep 27 18:36:35 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 27 Sep 2010 18:36:35 +0200 Subject: [Python-Dev] hg conversion: tags References: <20100927112851.1455ea34@mission> Message-ID: <20100927183635.5bbb337b@pitrou.net> On Mon, 27 Sep 2010 12:25:17 -0400 Alexander Belopolsky wrote: > On Mon, Sep 27, 2010 at 11:28 AM, Barry Warsaw wrote: > .. > > If so, then I say in hg, nuke the tags we don't care about and sanitize the > > release tags to something that obvious and can't collide (e.g. r311 is 3.1.1 > > or 3.11? - yes despite Guido's Rule of Version Numbering). ?I'd personally be > > fine using the hexversion or even "3.1.1" as the tag name. > > > > +1 > > Apart from being ambiguous, rXYZ notation conflicts with the tracker > autoformat rule that links rNNN to svn revision. And also, given that many hg commands can be specified a tag name using "-rXXX", having to write "-rr311" would be a bit awkward. From tjreedy at udel.edu Mon Sep 27 19:22:28 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 27 Sep 2010 13:22:28 -0400 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: <20100927013849.688EC3A4114@sparrow.telecommunity.com> References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> <20100927001338.492EE3A4114@sparrow.telecommunity.com> <87r5gg2did.fsf@benfinney.id.au> <20100927013849.688EC3A4114@sparrow.telecommunity.com> Message-ID: On 9/26/2010 9:38 PM, P.J. Eby wrote: > At 11:15 AM 9/27/2010 +1000, Ben Finney wrote: > You misunderstand me; I wasn't asking how to *add* a link, but how to > turn OFF the automatic conversion of the phrase "PEP 333" that happens > without any special markup. > Currently, the PEP 3333 preface is littered with unnecessary links, > because the PEP pre-processor turns *every* mere textual mention of a > PEP into a link to it. Ouch. This is about as annoying as Thunderbird's message editor popping up a windowed asking me what file I want to at.tach everytime I write the word "at-tach' or a derivative without the extra punctuation. It would definitely not be the vehicle for writing about at=mentment syndromes. Suggestion pending something better from rst/PEP experts: "This PEP extends PEP 333 (abbreviated P333 hereafter)." perhaps with "to avoid auto-link creation" added before ')' to pre-answer pesky questions and to avoid some editor re-expanding the abbreviations. -- Terry Jan Reedy From pje at telecommunity.com Mon Sep 27 19:49:45 2010 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 27 Sep 2010 13:49:45 -0400 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> <20100927001338.492EE3A4114@sparrow.telecommunity.com> <87r5gg2did.fsf@benfinney.id.au> <20100927013849.688EC3A4114@sparrow.telecommunity.com> Message-ID: <20100927174937.C173E3A40F5@sparrow.telecommunity.com> At 01:22 PM 9/27/2010 -0400, Terry Reedy wrote: >On 9/26/2010 9:38 PM, P.J. Eby wrote: >>At 11:15 AM 9/27/2010 +1000, Ben Finney wrote: > >>You misunderstand me; I wasn't asking how to *add* a link, but how to >>turn OFF the automatic conversion of the phrase "PEP 333" that happens >>without any special markup. > >>Currently, the PEP 3333 preface is littered with unnecessary links, >>because the PEP pre-processor turns *every* mere textual mention of a >>PEP into a link to it. > >Ouch. This is about as annoying as Thunderbird's message editor >popping up a windowed asking me what file I want to at.tach >everytime I write the word "at-tach' or a derivative without the >extra punctuation. It would definitely not be the vehicle for >writing about at=mentment syndromes. > >Suggestion pending something better from rst/PEP experts: >"This PEP extends PEP 333 (abbreviated P333 hereafter)." >perhaps with "to avoid auto-link creation" added before ')' to >pre-answer pesky questions and to avoid some editor re-expanding the >abbreviations. It turns out that using a backslash before the number (e.g. PEP \333) turns off the automatic conversion. The PEP still hasn't showed up on Python.org, though, so I'm wondering if maybe I broke something else somewhere. From martin at v.loewis.de Mon Sep 27 20:22:16 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 27 Sep 2010 20:22:16 +0200 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: <20100927174937.C173E3A40F5@sparrow.telecommunity.com> References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> <20100927001338.492EE3A4114@sparrow.telecommunity.com> <87r5gg2did.fsf@benfinney.id.au> <20100927013849.688EC3A4114@sparrow.telecommunity.com> <20100927174937.C173E3A40F5@sparrow.telecommunity.com> Message-ID: <4CA0E0D8.7080303@v.loewis.de> > The PEP still hasn't showed up on Python.org, though, so I'm wondering > if maybe I broke something else somewhere. See http://www.python.org/status/postcommitlog.txt Error processing PEP None (./pep-3333.txt), excluding: (./pep-3333.txt): "did not deal with u'Replaces' before having to handle 'Created'" make: *** [pep-0000.txt] Error 1 svn up make pep-0000.txt Traceback (most recent call last): File "/data/website-build/build/scripts/post-commit-svnup.py", line 160, in ? main() File "/data/website-build/build/scripts/post-commit-svnup.py", line 139, in main cmd("make pep-0000.txt") File "/data/website-build/build/scripts/post-commit-svnup.py", line 35, in cmd raise RuntimeError, '%s failed w/ exit code %d' % (command, err) RuntimeError: make pep-0000.txt failed w/ exit code 512 make: Circular pep-0000.txt <- pep-0000.txt dependency dropped. Regards, Martin From tjreedy at udel.edu Mon Sep 27 20:40:58 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 27 Sep 2010 14:40:58 -0400 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: <4CA0E0D8.7080303@v.loewis.de> References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> <20100927001338.492EE3A4114@sparrow.telecommunity.com> <87r5gg2did.fsf@benfinney.id.au> <20100927013849.688EC3A4114@sparrow.telecommunity.com> <20100927174937.C173E3A40F5@sparrow.telecommunity.com> <4CA0E0D8.7080303@v.loewis.de> Message-ID: On 9/27/2010 2:22 PM, "Martin v. L?wis" wrote: >> The PEP still hasn't showed up on Python.org, though, so I'm wondering >> if maybe I broke something else somewhere. > > See http://www.python.org/status/postcommitlog.txt Nasty link. That log begins back in 2008 and is so huge that it was still loading after about 20 seconds when I stopped it. Methinks it should periodically be archived into a separate file so it stays a manageable size. -- Terry Jan Reedy From fuzzyman at voidspace.org.uk Mon Sep 27 20:42:57 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 27 Sep 2010 19:42:57 +0100 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> <20100927001338.492EE3A4114@sparrow.telecommunity.com> <87r5gg2did.fsf@benfinney.id.au> <20100927013849.688EC3A4114@sparrow.telecommunity.com> Message-ID: <4CA0E5B1.4030000@voidspace.org.uk> On 27/09/2010 18:22, Terry Reedy wrote: > On 9/26/2010 9:38 PM, P.J. Eby wrote: >> At 11:15 AM 9/27/2010 +1000, Ben Finney wrote: > >> You misunderstand me; I wasn't asking how to *add* a link, but how to >> turn OFF the automatic conversion of the phrase "PEP 333" that happens >> without any special markup. > >> Currently, the PEP 3333 preface is littered with unnecessary links, >> because the PEP pre-processor turns *every* mere textual mention of a >> PEP into a link to it. > > Ouch. This is about as annoying as Thunderbird's message editor > popping up a windowed asking me what file I want to at.tach everytime > I write the word "at-tach' or a derivative without the extra > punctuation. It would definitely not be the vehicle for writing about > at=mentment syndromes. > I'd normally send this type of email off-list, but if I send it on list then hopefully you don't get twenty other emails saying the same thing... If you don't like Thunderbird warning you about attaching files then switch that off! Preferences -> Composition -> Check for missing attachments All the best, Michael Foord > Suggestion pending something better from rst/PEP experts: > "This PEP extends PEP 333 (abbreviated P333 hereafter)." > perhaps with "to avoid auto-link creation" added before ')' to > pre-answer pesky questions and to avoid some editor re-expanding the > abbreviations. > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From chris at simplistix.co.uk Mon Sep 27 20:16:16 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 27 Sep 2010 19:16:16 +0100 Subject: [Python-Dev] Some changes to logging for 3.2 In-Reply-To: <484041.20824.qm@web25807.mail.ukl.yahoo.com> References: <4C9C7B3F.6060807@simplistix.co.uk> <484041.20824.qm@web25807.mail.ukl.yahoo.com> Message-ID: <4CA0DF70.4060708@simplistix.co.uk> On 24/09/2010 12:06, Vinay Sajip wrote: >> http://plumberjack.blogspot.com/2010/09/improved-queuehandler-queuelistener.html >>> >> >> Cool, how can I use it in Python 2.6? :-) > > 1. Copy the top part (imports, QueueHandler and QueueListener classes) from the > Gist linked to in the article - http://gist.github.com/591699 - into a utility > module you can use again and again. Thanks! Although my comment was a light hearted dig from my point of view that the components of the standard library should be separately packaged and a python distribution should just be a "known good set" of these packages... Were that the case, users of Python 2.5, 2.6 *and* 2.7 could benefit from your good work :-) cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From brett at python.org Mon Sep 27 21:36:26 2010 From: brett at python.org (Brett Cannon) Date: Mon, 27 Sep 2010 12:36:26 -0700 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: <4CA0E0D8.7080303@v.loewis.de> References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> <20100927001338.492EE3A4114@sparrow.telecommunity.com> <87r5gg2did.fsf@benfinney.id.au> <20100927013849.688EC3A4114@sparrow.telecommunity.com> <20100927174937.C173E3A40F5@sparrow.telecommunity.com> <4CA0E0D8.7080303@v.loewis.de> Message-ID: All fixed. On Mon, Sep 27, 2010 at 11:22, "Martin v. L?wis" wrote: >> The PEP still hasn't showed up on Python.org, though, so I'm wondering >> if maybe I broke something else somewhere. > > See http://www.python.org/status/postcommitlog.txt > > Error processing PEP None (./pep-3333.txt), excluding: (./pep-3333.txt): > "did not deal with u'Replaces' before having to handle 'Created'" > make: *** [pep-0000.txt] Error 1 > svn up > make pep-0000.txt > Traceback (most recent call last): > ?File "/data/website-build/build/scripts/post-commit-svnup.py", line > 160, in ? > ? ?main() > ?File "/data/website-build/build/scripts/post-commit-svnup.py", line > 139, in main > ? ?cmd("make pep-0000.txt") > ?File "/data/website-build/build/scripts/post-commit-svnup.py", line > 35, in cmd > ? ?raise RuntimeError, '%s failed w/ exit code %d' % (command, err) > RuntimeError: make pep-0000.txt failed w/ exit code 512 > make: Circular pep-0000.txt <- pep-0000.txt dependency dropped. > > Regards, > Martin > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From pje at telecommunity.com Mon Sep 27 22:33:24 2010 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 27 Sep 2010 16:33:24 -0400 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly Message-ID: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> At 12:36 PM 9/27/2010 -0700, Brett Cannon wrote: >All fixed. Nope. I mean, sure, I checked in fixed PEP sources several hours ago, but python.org still doesn't show PEP 3333, or the updated version of PEP 333. From ncoghlan at gmail.com Mon Sep 27 22:34:10 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Sep 2010 06:34:10 +1000 Subject: [Python-Dev] Python wiki In-Reply-To: <20100927123138.GA3241@amk-desktop.matrixgroup.net> References: <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> <4C9E6CF8.4080705@v.loewis.de> <4C9E74D0.3050704@v.loewis.de> <4C9F43CB.6010500@v.loewis.de> <20100927123138.GA3241@amk-desktop.matrixgroup.net> Message-ID: On Mon, Sep 27, 2010 at 10:31 PM, A.M. Kuchling wrote: > On Sun, Sep 26, 2010 at 03:53:58PM +0200, Georg Brandl wrote: >> * redirect from wiki.python.org to wiki.python.org/moin > > I've added a element to the top page of > wiki.python.org, so browsers will now jump to the /moin/ page > immediately. ?This won't help crawlers that don't parse the HTML, but > that probably doesn't matter. That did the trick, thanks. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From merwok at netwok.org Mon Sep 27 22:36:25 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Mon, 27 Sep 2010 22:36:25 +0200 Subject: [Python-Dev] Python wiki In-Reply-To: References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> Message-ID: <4CA10049.1010509@netwok.org> Hello Le 25/09/2010 10:20, anatoly techtonik a ?crit : > On Fri, Sep 24, 2010 at 1:27 AM, Nick Coghlan wrote: >> That's a good point actually... why *isn't* there a pydotorg-wiki-sig? >> (Aside from the obvious point of nobody ever asking for one). > > Because Yet Another Mailing List doesn't solve the problem. > If you need one - go Google Groups like packaging folks did. > Python ML are: > 1. require dedicated admin to update, who is not a member of the group > 2. don't have search > 3. don't have optional thread subscription > That's already enough to seek better platform for collaboration. This is debatable. Google Groups require a Google account, are not controlled by python-dev, require JavaScript to view the archives, don?t send MIME digests. The public archives for python.org MLs are indexed by Web search engines and are therefore searchable. Re: thread subscription, there is a setting about topics in the mailman admin but I don?t understand it. I suppose a good email client can do something equivalent. > Community can perfectly manage the stuff without dedicated admins if > there is a gameplay in it. I don?t agree with the split between admins and community. Admins are just trusted people from the community (which includes python-dev). > My advice - subscribe people editing page by default (a checkbox near > submit button). +1 wholeheartedly. >> With an admin team behind it, you can also make more use of ACLs to >> flag certain parts of the wiki as "official" by making them only >> editable by certain people I don?t know if this would be noticed enough to change the image of the wiki. I had started reviewing and updating all pages pertaining to distutils and distutils2 some months ago, and I viewed the wiki as a collaborative area to hash things out, before they can become official code or docs. Regards From barry at python.org Mon Sep 27 22:52:06 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 27 Sep 2010 16:52:06 -0400 Subject: [Python-Dev] Python wiki In-Reply-To: <4CA10049.1010509@netwok.org> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4CA10049.1010509@netwok.org> Message-ID: <20100927165206.44b7d064@mission> On Sep 27, 2010, at 10:36 PM, ?ric Araujo wrote: >> Because Yet Another Mailing List doesn't solve the problem. >> If you need one - go Google Groups like packaging folks did. >> Python ML are: >> 1. require dedicated admin to update, who is not a member of the >> group 2. don't have search >> 3. don't have optional thread subscription >> That's already enough to seek better platform for collaboration. >This is debatable. Google Groups require a Google account, are not >controlled by python-dev, require JavaScript to view the archives, >don?t send MIME digests. The public archives for python.org MLs are >indexed by Web search engines and are therefore searchable. ObPlug: You can always read the archives at Gmane (e.g. via web or nntp) or mail-archive.com, both of which are searchable. Or you can help improve the world of open source mail archivers by getting involved in the Mailman project and helping us build the next generation archiver. :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From brett at python.org Mon Sep 27 22:55:43 2010 From: brett at python.org (Brett Cannon) Date: Mon, 27 Sep 2010 13:55:43 -0700 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> References: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> Message-ID: I spoke too soon. The Makefile stopped complaining before I committed, but turned out that was a lie. Fixed PEP 0 again, verifying there were no errors after a `make clean`, touching pep-3333.txt, or from just deleting pep-0000.txt. On Mon, Sep 27, 2010 at 13:33, P.J. Eby wrote: > At 12:36 PM 9/27/2010 -0700, Brett Cannon wrote: >> >> All fixed. > > Nope. ?I mean, sure, I checked in fixed PEP sources several hours ago, but > python.org still doesn't show PEP 3333, or the updated version of PEP 333. > > From g.brandl at gmx.net Mon Sep 27 22:56:39 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 27 Sep 2010 22:56:39 +0200 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> References: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> Message-ID: Am 27.09.2010 22:33, schrieb P.J. Eby: > At 12:36 PM 9/27/2010 -0700, Brett Cannon wrote: >>All fixed. > > Nope. I mean, sure, I checked in fixed PEP sources several hours > ago, but python.org still doesn't show PEP 3333, or the updated > version of PEP 333. It does now, for me, so I assume someone's gone and prodded it. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From martin at v.loewis.de Mon Sep 27 22:57:53 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 27 Sep 2010 22:57:53 +0200 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: <20100927201707.42D1A3A4105@sparrow.telecommunity.com> References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> <20100927001338.492EE3A4114@sparrow.telecommunity.com> <87r5gg2did.fsf@benfinney.id.au> <20100927013849.688EC3A4114@sparrow.telecommunity.com> <20100927174937.C173E3A40F5@sparrow.telecommunity.com> <4CA0E0D8.7080303@v.loewis.de> <20100927201707.42D1A3A4105@sparrow.telecommunity.com> Message-ID: <4CA10551.8060809@v.loewis.de> Am 27.09.2010 22:17, schrieb P.J. Eby: > At 12:36 PM 9/27/2010 -0700, Brett Cannon wrote: >> All fixed. > > Nope. Indeed. The immediate problem was that genpepindex tried to read pep-0000, and didn't like it. I worked around that in r85041, so that genpepindex now skips over pep-0000.txt. However, if it's really important that the fields in a PEP are in a certain order (why is that important?), then I guess PEP 0 should really comply with its own specification :-) Regards, Martin From ncoghlan at gmail.com Mon Sep 27 23:00:54 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Sep 2010 07:00:54 +1000 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> References: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> Message-ID: On Tue, Sep 28, 2010 at 6:33 AM, P.J. Eby wrote: > At 12:36 PM 9/27/2010 -0700, Brett Cannon wrote: >> >> All fixed. > > Nope. ?I mean, sure, I checked in fixed PEP sources several hours ago, but > python.org still doesn't show PEP 3333, or the updated version of PEP 333. I tweaked the post dates in the PEPs to try to kick the webserver into regenerating them when I committed the updated versions, and Brett and Martin have been tinkering with the PEP 0 generation code. Something or other in that process triggered a proper rebuild, since the updated PEP 333 and PEP 3333 are now visible on the site. Someone with web server access may want to double check the modification dates of the .txt files relative to the generated .html files for other PEPs though. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From guido at python.org Mon Sep 27 23:03:46 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 27 Sep 2010 14:03:46 -0700 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> References: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> Message-ID: On Mon, Sep 27, 2010 at 1:33 PM, P.J. Eby wrote: > At 12:36 PM 9/27/2010 -0700, Brett Cannon wrote: >> >> All fixed. > > Nope. ?I mean, sure, I checked in fixed PEP sources several hours ago, but > python.org still doesn't show PEP 3333, or the updated version of PEP 333. Seems Brett has fixed it. Both PEPs are now online. I wonder if it would make sense to change both from "Informational" to "Standard Track" ? -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Mon Sep 27 23:09:28 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Sep 2010 07:09:28 +1000 Subject: [Python-Dev] Mark PEP 3148 as Final? Message-ID: I saw the code for PEP 3148 go by on python-checkins the other day. Is there anything left to be done on that front, or can the PEP be marked Final? Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From martin at v.loewis.de Mon Sep 27 23:21:59 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 27 Sep 2010 23:21:59 +0200 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: References: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> Message-ID: <4CA10AF7.7010405@v.loewis.de> > Someone with web server access may want to double check the > modification dates of the .txt files relative to the generated .html > files for other PEPs though. make will deal with that just fine. If a PEP was modified, svn up will update the time stamp on the file. When then the rebuild fails, the html file will still have an old time stamp. So I'm unsure why you thought you needed to modify some of the files. Regards, Martin From guido at python.org Tue Sep 28 00:17:02 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 27 Sep 2010 15:17:02 -0700 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: <4CA10AF7.7010405@v.loewis.de> References: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> <4CA10AF7.7010405@v.loewis.de> Message-ID: On Mon, Sep 27, 2010 at 2:21 PM, "Martin v. L?wis" wrote: >> Someone with web server access may want to double check the >> modification dates of the .txt files relative to the generated .html >> files for other PEPs though. > > make will deal with that just fine. If a PEP was modified, svn up will > update the time stamp on the file. When then the rebuild fails, the > html file will still have an old time stamp. > > So I'm unsure why you thought you needed to modify some of the files. Because it's not clear to most of us on this thread what the failure modes and recovery strategies are? I know it's clear as mud to me how to debug these kinds of issues. -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Tue Sep 28 00:41:19 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 Sep 2010 00:41:19 +0200 Subject: [Python-Dev] Prefetching on buffered IO files Message-ID: <20100928004119.3963a4ad@pitrou.net> Hello, While trying to solve #3873 (poor performance of pickle on file objects, due to the overhead of calling read() with very small values), it occurred to me that the prefetching facilities offered by BufferedIOBase are not flexible and efficient enough. Indeed, if you use seek() and read(), 1) you limit yourself to seekable files 2) performance can be hampered by very bad seek() performance (this is true on GzipFile). If instead you use peek() and read(), the situation is better, but you end up doing multiple copies of data; also, you must call read() to advance the file pointer even though you don't care about the results. So I would propose adding the following method to BufferedIOBase: prefetch(self, buffer, skip, minread) Skip `skip` bytes from the stream. Then, try to read at least `minread` bytes and write them into `buffer`. The file pointer is advanced by at most `skip + minread`, or less if the end of file was reached. The total number of bytes written in `buffer` is returned, which can be more than `minread` if additional bytes could be prefetched (but, of course, cannot be more than `len(buffer)`). Arguments: - `buffer`: a writable buffer (e.g. bytearray) - `skip`: number of bytes to skip (must be >= 0) - `minread`: number of bytes to read (must be >= 0 and <= len(buffer)) Also, the BufferedIOBase ABC can then provide default implementations of read(), readinto() and peek(), simply by calling prefetch(). (how read1() can fit into the picture is not obvious) What do you think? Regards Antoine. From pje at telecommunity.com Tue Sep 28 01:29:13 2010 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 27 Sep 2010 19:29:13 -0400 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: References: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> Message-ID: <20100927232906.7231E3A40F5@sparrow.telecommunity.com> At 02:03 PM 9/27/2010 -0700, Guido van Rossum wrote: >On Mon, Sep 27, 2010 at 1:33 PM, P.J. Eby wrote: > > At 12:36 PM 9/27/2010 -0700, Brett Cannon wrote: > >> > >> All fixed. > > > > Nope. I mean, sure, I checked in fixed PEP sources several hours ago, but > > python.org still doesn't show PEP 3333, or the updated version of PEP 333. > >Seems Brett has fixed it. Both PEPs are now online. > >I wonder if it would make sense to change both from "Informational" to >"Standard Track" ? From PEP 1: """ There are three kinds of PEP: * A Standards Track PEP describes a new feature or implementation for Python. * An Informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore Informational PEPs or follow their advice. * A Process PEP describes a process surrounding Python, or proposes a change to (or an event in) a process. Process PEPs are like Standards Track PEPs but apply to areas other than the Python language itself. They may propose an implementation, but not to Python's codebase; they often require community consensus; unlike Informational PEPs, they are more than recommendations, and users are typically not free to ignore them. Examples include procedures, guidelines, changes to the decision-making process, and changes to the tools or environment used in Python development. Any meta-PEP is also considered a Process PEP. """ I don't think it qualifies as a Standards PEP under the above definitions. I made it Informational originally because it's rather like the DB API PEPs, which are Informational. I suppose we could say it's a Process PEP, or perhaps update PEP 1 to add a new category (into which the DB API PEPs would also fall), or maybe just tweak the above definitions a bit so that the Informational category makes more sense. From guido at python.org Tue Sep 28 02:39:45 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 27 Sep 2010 17:39:45 -0700 Subject: [Python-Dev] Prefetching on buffered IO files In-Reply-To: <20100928004119.3963a4ad@pitrou.net> References: <20100928004119.3963a4ad@pitrou.net> Message-ID: On Mon, Sep 27, 2010 at 3:41 PM, Antoine Pitrou wrote: > While trying to solve #3873 (poor performance of pickle on file > objects, due to the overhead of calling read() with very small values), > it occurred to me that the prefetching facilities offered by > BufferedIOBase are not flexible and efficient enough. I haven't read the whole bug but there seem to be lots of different smaller issues there, right? It seems that one (unfortunate) constraint is that reading pickles cannot use buffered I/O (at least not on a non-seekable file) because the API has been documented to leave the file positioned right after the last byte of the pickled data, right? > Indeed, if you use seek() and read(), 1) you limit yourself to seekable > files 2) performance can be hampered by very bad seek() performance > (this is true on GzipFile). Ow... I've always assumed that seek() is essentially free, because that's how a typical OS kernel implements it. If seek() is bad on GzipFile, how hard would it be to fix this? How common is the use case where you need to read a gzipped pickle *and* you need to leave the unzipped stream positioned exactly at the end of the pickle? > If instead you use peek() and read(), the situation is better, but you > end up doing multiple copies of data; also, you must call read() to > advance the file pointer even though you don't care about the results. Have you measured how bad the situation is if you do implement it this way? > So I would propose adding the following method to BufferedIOBase: > > prefetch(self, buffer, skip, minread) > > Skip `skip` bytes from the stream. ?Then, try to read at > least `minread` bytes and write them into `buffer`. The file > pointer is advanced by at most `skip + minread`, or less if > the end of file was reached. The total number of bytes written > in `buffer` is returned, which can be more than `minread` > if additional bytes could be prefetched (but, of course, > cannot be more than `len(buffer)`). > > Arguments: > - `buffer`: a writable buffer (e.g. bytearray) > - `skip`: number of bytes to skip (must be >= 0) > - `minread`: number of bytes to read (must be >= 0 and <= len(buffer)) I like the idea of an API that combines seek and read into a mutable buffer. However the semantics of this call seem really weird: there is no direct relationship between where it leaves the stream position and how much data it reads into the buffer. can you explain how exactly this will help solve the gzipped pickle performance problem? > Also, the BufferedIOBase ABC can then provide default implementations of > read(), readinto() and peek(), simply by calling prefetch(). > (how read1() can fit into the picture is not obvious) > > What do you think? Move to python-ideas? -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue Sep 28 02:41:44 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 27 Sep 2010 17:41:44 -0700 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: <20100927232906.7231E3A40F5@sparrow.telecommunity.com> References: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> <20100927232906.7231E3A40F5@sparrow.telecommunity.com> Message-ID: On Mon, Sep 27, 2010 at 4:29 PM, P.J. Eby wrote: > At 02:03 PM 9/27/2010 -0700, Guido van Rossum wrote: >> >> On Mon, Sep 27, 2010 at 1:33 PM, P.J. Eby wrote: >> > At 12:36 PM 9/27/2010 -0700, Brett Cannon wrote: >> >> >> >> All fixed. >> > >> > Nope. ?I mean, sure, I checked in fixed PEP sources several hours ago, >> > but >> > python.org still doesn't show PEP 3333, or the updated version of PEP >> > 333. >> >> Seems Brett has fixed it. Both PEPs are now online. >> >> I wonder if it would make sense to change both from "Informational" to >> "Standard Track" ? > > From PEP 1: > > """ > There are three kinds of PEP: > ? * A Standards Track PEP describes a new feature or implementation for > Python. > ? * An Informational PEP describes a Python design issue, or provides > general guidelines or information to the Python community, but does not > propose a new feature. Informational PEPs do not necessarily represent a > Python community consensus or recommendation, so users and implementors are > free to ignore Informational PEPs or follow their advice. > ? * A Process PEP describes a process surrounding Python, or proposes a > change to (or an event in) a process. Process PEPs are like Standards Track > PEPs but apply to areas other than the Python language itself. They may > propose an implementation, but not to Python's codebase; they often require > community consensus; unlike Informational PEPs, they are more than > recommendations, and users are typically not free to ignore them. Examples > include procedures, guidelines, changes to the decision-making process, and > changes to the tools or environment used in Python development. Any meta-PEP > is also considered a Process PEP. > """ > > I don't think it qualifies as a Standards PEP under the above definitions. > ?I made it Informational originally because it's rather like the DB API > PEPs, which are Informational. > > I suppose we could say it's a Process PEP, or perhaps update PEP 1 to add a > new category (into which the DB API PEPs would also fall), or maybe just > tweak the above definitions a bit so that the Informational category makes > more sense. Hm. I would rather extend the definition of Standards Track to include API standards that are important to the community even if they do not introduce a new feature for the language or standard library. WSGI and DB-API being the two most well-known examples but I wouldn't be surprised if there were others, possibly in the NumPy world. -- --Guido van Rossum (python.org/~guido) From stephen at xemacs.org Tue Sep 28 02:41:57 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 28 Sep 2010 09:41:57 +0900 Subject: [Python-Dev] Python wiki In-Reply-To: <4CA10049.1010509@netwok.org> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4CA10049.1010509@netwok.org> Message-ID: <87tylaya0a.fsf@uwakimon.sk.tsukuba.ac.jp> ?ric Araujo writes: > Le 25/09/2010 10:20, anatoly techtonik a ?crit : > > Python ML are: > > 1. require dedicated admin to update, who is not a member of the group > > 2. don't have search > > 3. don't have optional thread subscription > > That's already enough to seek better platform for collaboration. -------------- next part -------------- > This is debatable. Google Groups require a Google account, are not > controlled by python-dev, require JavaScript to view the archives, don?t > send MIME digests. The public archives for python.org MLs are indexed > by Web search engines and are therefore searchable. Indeed. Mailman doesn't require a dedicated admin to operate, only to set up. Some things go better if you've got part-time admins (moderation in particular). I agree with techtonik that a better platform is apparently needed, but this is a client-server system, and the problem is in the client. Ie, he should upgrade his MUA to one that does threading properly, and allows priorities to be given to threads. (Emacs/Gnus is the one I'm familiar with but surely there are others.) This is a contextual judgment, not an absolute: apparently most Python devs *do* use sufficiently powerful MUAs for their purposes. It makes sense for the minority to adopt the majority practice. > Re: thread subscription, there is a setting about topics in the mailman > admin but I don?t understand it. Mailman topics are a clumsy Subject-based filter, and require admin intervention to set up. They're quite valuable for low-to-medium- traffic lists with a variety of more or less defined topics (ie, if there were more traffic, you'd want to split the list), but are pretty much useless for this purpose. > > Community can perfectly manage the stuff without dedicated admins if > > there is a gameplay in it. > I don?t agree with the split between admins and community. Admins are > just trusted people from the community (which includes python-dev). No, they're not just trusted people. They're trusted people with greater privilege than the rest of us, and therefore a bottleneck for some operations. > > My advice - subscribe people editing page by default (a checkbox near > > submit button). > +1 wholeheartedly. That default needs to be user-configurable. I would not want to be spammed with typo corrections on a page where all I did was correct a typo. I probably wouldn't even want to see major changes by default: I came, I saw, I conquered the typo and got my info, now leave me alone! > >> With an admin team behind it, you can also make more use of ACLs to > >> flag certain parts of the wiki as "official" by making them only > >> editable by certain people > I don?t know if this would be noticed enough to change the image of the > wiki. I had started reviewing and updating all pages pertaining to > distutils and distutils2 some months ago, and I viewed the wiki as a > collaborative area to hash things out, before they can become official > code or docs. I don't think we want to change the image of the wiki (as in "anybody can edit")! What we may want is (1) to make it easy for those with the authority to update the official docs (but there's a very good story for putting them in a repo, perhaps associated with the sources, so this is a weak argument for reference-docs-in-wiki), and (2) make it easy for readers to cross reference the community documentation (often more detailed or less intimidatingly formal) and the reference manuals. (1) *may* suggest using the wiki engine to support editing, but I think most devs are strongly against that. (2) suggests using the wiki engine to easily or even automatically set up cross-references. I think that wiki is probably the best technology for this purpose at the present time, but I don't know if it's worth the effort to make the official documentation wiki-friendly (in the sense of allowing the wiki to generate links to community documentation from the reference manuals). From jnoller at gmail.com Tue Sep 28 03:19:41 2010 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 27 Sep 2010 21:19:41 -0400 Subject: [Python-Dev] Mark PEP 3148 as Final? In-Reply-To: References: Message-ID: On Mon, Sep 27, 2010 at 5:09 PM, Nick Coghlan wrote: > I saw the code for PEP 3148 go by on python-checkins the other day. Is > there anything left to be done on that front, or can the PEP be marked > Final? > > Cheers, > Nick. Argh, yes :) From greg.ewing at canterbury.ac.nz Tue Sep 28 00:35:39 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 28 Sep 2010 10:35:39 +1200 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: References: <20100925195711.60FB13A4079@sparrow.telecommunity.com> <20100926014515.CCA5C3A4079@sparrow.telecommunity.com> <4C9EE8B4.7090405@v.loewis.de> <20100926173409.24ECB3A40B2@sparrow.telecommunity.com> <20100927001338.492EE3A4114@sparrow.telecommunity.com> <87r5gg2did.fsf@benfinney.id.au> <20100927013849.688EC3A4114@sparrow.telecommunity.com> Message-ID: <4CA11C3B.6060204@canterbury.ac.nz> > On 9/26/2010 9:38 PM, P.J. Eby wrote: > >> Currently, the PEP 3333 preface is littered with unnecessary links, >> because the PEP pre-processor turns *every* mere textual mention of a >> PEP into a link to it. Perhaps the preprocessor should only do this for the first occurrence of each linkable phrase in a paragraph? -- Greg From jcea at jcea.es Tue Sep 28 04:44:08 2010 From: jcea at jcea.es (Jesus Cea) Date: Tue, 28 Sep 2010 04:44:08 +0200 Subject: [Python-Dev] Pronouncement needed in issue9675 Message-ID: <4CA15678.3070004@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 http://bugs.python.org/issue9675 Long history sort: Python 2.7 backported Capsule support and (incorrectly, in my opinion) marked CObject as deprecated. All C modules in the stdlib were updated to Capsule (with a CObject compatibility layer), except BSDDB, because this change was done late in the cycle, the proposed patch was buggy (solvable) and a pronouncement was done that CObject was not actually deprecated. But in python 2.7 release, CObject is marked as deprecated (arg!), so when executing python with -We (mark warnings as errors), bsddb fails. Since I think that adopting Capsule in BSDDB for 2.7.1 would break the API compatibility (maybe the CObject proxy would solve this), and since a previous pronouncement was done abour CObject not-deprecated in 2.7.x, I would like comments. Long history and links to previous pronouncements in http://bugs.python.org/issue9675 My proposal: CObject should not be marked as deprecated in 2.7.1. Thanks for your time and attention. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTKFWeJlgi5GaxT1NAQLlIgP+NAOBvwCW8gJNFspsjdLc2VqPbwXK1GJc mmESVsoZRlAROwCkTlaOeO8GGoEuWtfb32SwJ+21RTPdPo7UxbaxhFUNju3bRYzQ We8VGh/Qu8oJPk/toifCEw80mv4Vr9Pfli3qsR9MmGsCBFdqjLMmtwTZAkl3uMoY 6PCdo1hgAoY= =veqL -----END PGP SIGNATURE----- From meadori at gmail.com Tue Sep 28 05:15:48 2010 From: meadori at gmail.com (Meador Inge) Date: Mon, 27 Sep 2010 22:15:48 -0500 Subject: [Python-Dev] issue2180 and using 'tokenize' with Python 3 'str's Message-ID: Hi All, I was going through some of the open issues related to 'tokenize' and ran across 'issue2180'. The reproduction case for this issue is along the lines of: >>> tokenize.tokenize(io.StringIO("if 1:\n \\\n #hey\n print 1").readline) but, with 'py3k' I get: >>> tokenize.tokenize(io.StringIO("if 1:\n \\\n #hey\n print 1").readline) Traceback (most recent call last): File "", line 1, in File "/Users/minge/Code/python/py3k/Lib/tokenize.py", line 360, in tokenize encoding, consumed = detect_encoding(readline) File "/Users/minge/Code/python/py3k/Lib/tokenize.py", line 316, in detect_encoding if first.startswith(BOM_UTF8): TypeError: Can't convert 'bytes' object to str implicitly which, as seen in the trace, is because the 'detect_encoding' function in 'Lib/tokenize.py' searches for 'BOM_UTF8' (a 'bytes' object) in the string to tokenize 'first' (a 'str' object). It seems to me that strings should still be able to be tokenized, but maybe I am missing something. Is the implementation of 'detect_encoding' correct in how it attempts to determine an encoding or should I open an issue for this? --- Meador -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Tue Sep 28 05:27:06 2010 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 27 Sep 2010 22:27:06 -0500 Subject: [Python-Dev] issue2180 and using 'tokenize' with Python 3 'str's In-Reply-To: References: Message-ID: 2010/9/27 Meador Inge : > which, as seen in the trace, is because the 'detect_encoding' function in > 'Lib/tokenize.py' searches for 'BOM_UTF8' (a 'bytes' object) in the string > to tokenize 'first' (a 'str' object). ?It seems to me that strings should > still be able to be tokenized, but maybe I am missing something. > Is the implementation of 'detect_encoding' correct in how it attempts to > determine an encoding or should I open an issue for this? Tokenize only works on bytes. You can open a feature request if you desire. -- Regards, Benjamin From pje at telecommunity.com Tue Sep 28 05:34:26 2010 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 27 Sep 2010 23:34:26 -0400 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: References: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> <20100927232906.7231E3A40F5@sparrow.telecommunity.com> Message-ID: <20100928033419.D84853A40F5@sparrow.telecommunity.com> At 05:41 PM 9/27/2010 -0700, Guido van Rossum wrote: >On Mon, Sep 27, 2010 at 4:29 PM, P.J. Eby wrote: > > At 02:03 PM 9/27/2010 -0700, Guido van Rossum wrote: > >> > >> On Mon, Sep 27, 2010 at 1:33 PM, P.J. Eby wrote: > >> > At 12:36 PM 9/27/2010 -0700, Brett Cannon wrote: > >> >> > >> >> All fixed. > >> > > >> > Nope. I mean, sure, I checked in fixed PEP sources several hours ago, > >> > but > >> > python.org still doesn't show PEP 3333, or the updated version of PEP > >> > 333. > >> > >> Seems Brett has fixed it. Both PEPs are now online. > >> > >> I wonder if it would make sense to change both from "Informational" to > >> "Standard Track" ? > > > > From PEP 1: > > > > """ > > There are three kinds of PEP: > > * A Standards Track PEP describes a new feature or implementation for > > Python. > > * An Informational PEP describes a Python design issue, or provides > > general guidelines or information to the Python community, but does not > > propose a new feature. Informational PEPs do not necessarily represent a > > Python community consensus or recommendation, so users and implementors are > > free to ignore Informational PEPs or follow their advice. > > * A Process PEP describes a process surrounding Python, or proposes a > > change to (or an event in) a process. Process PEPs are like Standards Track > > PEPs but apply to areas other than the Python language itself. They may > > propose an implementation, but not to Python's codebase; they often require > > community consensus; unlike Informational PEPs, they are more than > > recommendations, and users are typically not free to ignore them. Examples > > include procedures, guidelines, changes to the decision-making process, and > > changes to the tools or environment used in Python development. > Any meta-PEP > > is also considered a Process PEP. > > """ > > > > I don't think it qualifies as a Standards PEP under the above definitions. > > I made it Informational originally because it's rather like the DB API > > PEPs, which are Informational. > > > > I suppose we could say it's a Process PEP, or perhaps update PEP 1 to add a > > new category (into which the DB API PEPs would also fall), or maybe just > > tweak the above definitions a bit so that the Informational category makes > > more sense. > >Hm. I would rather extend the definition of Standards Track to include >API standards that are important to the community even if they do not >introduce a new feature for the language or standard library. WSGI and >DB-API being the two most well-known examples but I wouldn't be >surprised if there were others, possibly in the NumPy world. Well, one of the tradeoffs here is that Informational track allows something to grow into a solid standard without also having to pass the same level of up-front scrutiny and commitment that a Standards track item does. I rather doubt that either the DBAPI *or* WSGI would've passed that scrutiny in early days, and the "free to ignore" part means that there's a lot less pushback on the minor points than generally occurs with Standards track PEPs. So, I'd hate for us to lose out on the *next* DBAPI or WSGI due to an implied pressure of needing to get it right in the first place. (Indeed, I think we need *more* Informational PEPs -- in retrospect there was probably some point at which I should have done some relating to setuptools and eggs and such.) Overall, though, I supposed there's no problem with promoting Final Informational PEPs to Standards, *unless* it creates an expectation that Informational PEPs will become Standards and they thus end up being debated in the same way anyway. (Of course, if it generally takes five or six years before an Informational PEP usually gets promoted, this is unlikely to be a major worry.) From steve at holdenweb.com Tue Sep 28 05:45:45 2010 From: steve at holdenweb.com (Steve Holden) Date: Mon, 27 Sep 2010 23:45:45 -0400 Subject: [Python-Dev] issue2180 and using 'tokenize' with Python 3 'str's In-Reply-To: References: Message-ID: <4CA164E9.70503@holdenweb.com> On 9/27/2010 11:27 PM, Benjamin Peterson wrote: > 2010/9/27 Meador Inge : >> which, as seen in the trace, is because the 'detect_encoding' function in >> 'Lib/tokenize.py' searches for 'BOM_UTF8' (a 'bytes' object) in the string >> to tokenize 'first' (a 'str' object). It seems to me that strings should >> still be able to be tokenized, but maybe I am missing something. >> Is the implementation of 'detect_encoding' correct in how it attempts to >> determine an encoding or should I open an issue for this? > > Tokenize only works on bytes. You can open a feature request if you desire. > Working only on bytes does seem rather perverse. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From martin at v.loewis.de Tue Sep 28 07:29:17 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 28 Sep 2010 07:29:17 +0200 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: References: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> <4CA10AF7.7010405@v.loewis.de> Message-ID: <4CA17D2D.2070809@v.loewis.de> > Because it's not clear to most of us on this thread what the failure > modes and recovery strategies are? I know it's clear as mud to me how > to debug these kinds of issues. As a starting point, look at the postcommitlog. It should contain the commands that got executed, and the error messages that got generated. Regards, Martin From martin at v.loewis.de Tue Sep 28 07:44:50 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 28 Sep 2010 07:44:50 +0200 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: <20100928033419.D84853A40F5@sparrow.telecommunity.com> References: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> <20100927232906.7231E3A40F5@sparrow.telecommunity.com> <20100928033419.D84853A40F5@sparrow.telecommunity.com> Message-ID: <4CA180D2.4070109@v.loewis.de> > Well, one of the tradeoffs here is that Informational track allows > something to grow into a solid standard without also having to pass the > same level of up-front scrutiny and commitment that a Standards track > item does. I rather doubt that either the DBAPI *or* WSGI would've > passed that scrutiny in early days, and the "free to ignore" part means > that there's a lot less pushback on the minor points than generally > occurs with Standards track PEPs. The downside of an informational PEP is that it is unilateral. The author can put anything into it, and the community doesn't really get a voice in deciding on it. It's bad for quality (as I think you also point out) if the author of a PEP is also the one who pronounces on it. Regards, Martin From ncoghlan at gmail.com Tue Sep 28 13:05:20 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Sep 2010 21:05:20 +1000 Subject: [Python-Dev] [Web-SIG] WSGI is now Python 3-friendly In-Reply-To: References: <20100927203316.CFFF93A4105@sparrow.telecommunity.com> <4CA10AF7.7010405@v.loewis.de> Message-ID: On Tue, Sep 28, 2010 at 8:17 AM, Guido van Rossum wrote: > On Mon, Sep 27, 2010 at 2:21 PM, "Martin v. L?wis" wrote: >>> Someone with web server access may want to double check the >>> modification dates of the .txt files relative to the generated .html >>> files for other PEPs though. >> >> make will deal with that just fine. If a PEP was modified, svn up will >> update the time stamp on the file. When then the rebuild fails, the >> html file will still have an old time stamp. >> >> So I'm unsure why you thought you needed to modify some of the files. > > Because it's not clear to most of us on this thread what the failure > modes and recovery strategies are? I know it's clear as mud to me how > to debug these kinds of issues. Yep, it was just a misunderstanding on my part as to possible failure modes for the PEP publication process. I've certainly seen plenty of other systems which need to be resent events if processing of the original events fails. Primarily, I didn't know that any PEP commit just triggered a full Makefile invocation for the PEP directory rather than trying to be selective. (Letting make sort it out is a better idea, I just didn't know it was set up that way). (And, like Terry, Firefox chokes on the postcommit log for me. The file appears to be too big for my system to handle gracefully, so checking it really didn't help me. I did try though.) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From amauryfa at gmail.com Tue Sep 28 13:18:53 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 28 Sep 2010 13:18:53 +0200 Subject: [Python-Dev] Pronouncement needed in issue9675 In-Reply-To: <4CA15678.3070004@jcea.es> References: <4CA15678.3070004@jcea.es> Message-ID: Hi, 2010/9/28 Jesus Cea : > http://bugs.python.org/issue9675 > > Long history sort: Python 2.7 backported Capsule support and > (incorrectly, in my opinion) marked CObject as deprecated. > > All C modules in the stdlib were updated to Capsule (with a CObject > compatibility layer), except BSDDB, because this change was done late in > the cycle, the proposed patch was buggy (solvable) and a pronouncement > was done that CObject was not actually deprecated. > > But in python 2.7 release, CObject is marked as deprecated (arg!), so > when executing python with -We (mark warnings as errors), bsddb fails. > > Since I think that adopting Capsule in BSDDB for 2.7.1 would break the > API compatibility (maybe the CObject proxy would solve this), and since > a previous pronouncement was done abour CObject not-deprecated in 2.7.x, > I would like comments. Is compatibility really broken? PyCObject_AsVoidPtr(), PyCObject_Import() accept Capsule objects as well. Or are there other usages of the api pointer? -- Amaury Forgeot d'Arc From solipsis at pitrou.net Tue Sep 28 13:19:18 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 Sep 2010 13:19:18 +0200 Subject: [Python-Dev] issue2180 and using 'tokenize' with Python 3 'str's References: <4CA164E9.70503@holdenweb.com> Message-ID: <20100928131918.6dba0ca6@pitrou.net> On Mon, 27 Sep 2010 23:45:45 -0400 Steve Holden wrote: > On 9/27/2010 11:27 PM, Benjamin Peterson wrote: > > 2010/9/27 Meador Inge : > >> which, as seen in the trace, is because the 'detect_encoding' function in > >> 'Lib/tokenize.py' searches for 'BOM_UTF8' (a 'bytes' object) in the string > >> to tokenize 'first' (a 'str' object). It seems to me that strings should > >> still be able to be tokenized, but maybe I am missing something. > >> Is the implementation of 'detect_encoding' correct in how it attempts to > >> determine an encoding or should I open an issue for this? > > > > Tokenize only works on bytes. You can open a feature request if you desire. > > > Working only on bytes does seem rather perverse. I agree, the morality of bytes objects could have been better :) From fuzzyman at voidspace.org.uk Tue Sep 28 13:29:18 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 28 Sep 2010 12:29:18 +0100 Subject: [Python-Dev] issue2180 and using 'tokenize' with Python 3 'str's In-Reply-To: <20100928131918.6dba0ca6@pitrou.net> References: <4CA164E9.70503@holdenweb.com> <20100928131918.6dba0ca6@pitrou.net> Message-ID: <4CA1D18E.1050601@voidspace.org.uk> On 28/09/2010 12:19, Antoine Pitrou wrote: > On Mon, 27 Sep 2010 23:45:45 -0400 > Steve Holden wrote: >> On 9/27/2010 11:27 PM, Benjamin Peterson wrote: >>> 2010/9/27 Meador Inge: >>>> which, as seen in the trace, is because the 'detect_encoding' function in >>>> 'Lib/tokenize.py' searches for 'BOM_UTF8' (a 'bytes' object) in the string >>>> to tokenize 'first' (a 'str' object). It seems to me that strings should >>>> still be able to be tokenized, but maybe I am missing something. >>>> Is the implementation of 'detect_encoding' correct in how it attempts to >>>> determine an encoding or should I open an issue for this? >>> Tokenize only works on bytes. You can open a feature request if you desire. >>> >> Working only on bytes does seem rather perverse. > I agree, the morality of bytes objects could have been better :) > The reason for working with bytes is that source data can only be correctly decoded to text once the encoding is known. The encoding is determined by reading the encoding cookie. I certainly wouldn't be opposed to an API that accepts a string as well though. All the best, Michael > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From fuzzyman at voidspace.org.uk Tue Sep 28 13:55:25 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 28 Sep 2010 12:55:25 +0100 Subject: [Python-Dev] issue2180 and using 'tokenize' with Python 3 'str's In-Reply-To: <4CA1D18E.1050601@voidspace.org.uk> References: <4CA164E9.70503@holdenweb.com> <20100928131918.6dba0ca6@pitrou.net> <4CA1D18E.1050601@voidspace.org.uk> Message-ID: On 28 September 2010 12:29, Michael Foord wrote: > On 28/09/2010 12:19, Antoine Pitrou wrote: > >> On Mon, 27 Sep 2010 23:45:45 -0400 >> Steve Holden wrote: >> >>> On 9/27/2010 11:27 PM, Benjamin Peterson wrote: >>> >>>> 2010/9/27 Meador Inge: >>>> >>>>> which, as seen in the trace, is because the 'detect_encoding' function >>>>> in >>>>> 'Lib/tokenize.py' searches for 'BOM_UTF8' (a 'bytes' object) in the >>>>> string >>>>> to tokenize 'first' (a 'str' object). It seems to me that strings >>>>> should >>>>> still be able to be tokenized, but maybe I am missing something. >>>>> Is the implementation of 'detect_encoding' correct in how it attempts >>>>> to >>>>> determine an encoding or should I open an issue for this? >>>>> >>>> Tokenize only works on bytes. You can open a feature request if you >>>> desire. >>>> >>>> Working only on bytes does seem rather perverse. >>> >> I agree, the morality of bytes objects could have been better :) >> >> The reason for working with bytes is that source data can only be > correctly decoded to text once the encoding is known. The encoding is > determined by reading the encoding cookie. > > I certainly wouldn't be opposed to an API that accepts a string as well > though. > > Ah, and to explain the design decision when tokenize was ported to py3k - the Python 2 APIs take the readline method of a file object (not a string). http://docs.python.org/library/tokenize.html For this to work correctly in Python 3 it *has* to be a file object open in binary read mode in order to decode the source code correctly. A new API that takes a string would certainly be nice. The Python 2 API for tokenize is 'interesting'... All the best, Michael Foord > All the best, > > Michael > > > >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk >> > > > -- > http://www.ironpythoninaction.com/ > http://www.voidspace.org.uk/blog > > READ CAREFULLY. By accepting and reading this email you agree, on behalf of > your employer, to release me from all obligations and waivers arising from > any and all NON-NEGOTIATED agreements, licenses, terms-of-service, > shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, > non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have > entered into with your employer, its partners, licensors, agents and > assigns, in perpetuity, without prejudice to my ongoing rights and > privileges. You further represent that you have the authority to release me > from any BOGUS AGREEMENTS on behalf of your employer. > > > -- http://www.voidspace.org.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Sep 28 14:09:48 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Sep 2010 22:09:48 +1000 Subject: [Python-Dev] issue2180 and using 'tokenize' with Python 3 'str's In-Reply-To: <4CA1D18E.1050601@voidspace.org.uk> References: <4CA164E9.70503@holdenweb.com> <20100928131918.6dba0ca6@pitrou.net> <4CA1D18E.1050601@voidspace.org.uk> Message-ID: On Tue, Sep 28, 2010 at 9:29 PM, Michael Foord wrote: > ?On 28/09/2010 12:19, Antoine Pitrou wrote: >> On Mon, 27 Sep 2010 23:45:45 -0400 >> Steve Holden ?wrote: >>> On 9/27/2010 11:27 PM, Benjamin Peterson wrote: >>>> Tokenize only works on bytes. You can open a feature request if you >>>> desire. >>>> >>> Working only on bytes does seem rather perverse. >> >> I agree, the morality of bytes objects could have been better :) >> > The reason for working with bytes is that source data can only be correctly > decoded to text once the encoding is known. The encoding is determined by > reading the encoding cookie. > > I certainly wouldn't be opposed to an API that accepts a string as well > though. A very quick scan of _tokenize suggests it is designed to support detect_encoding returning None to indicate the line iterator will return already decoded lines. This is confirmed by the fact the standard library uses it that way (via generate_tokens). An API that accepts a string, wraps a StringIO around it, then calls _tokenise with an encoding of None would appear to be the answer here. A feature request on the tracker is the best way to make that happen. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From meadori at gmail.com Tue Sep 28 15:19:59 2010 From: meadori at gmail.com (Meador Inge) Date: Tue, 28 Sep 2010 08:19:59 -0500 Subject: [Python-Dev] issue2180 and using 'tokenize' with Python 3 'str's In-Reply-To: References: <4CA164E9.70503@holdenweb.com> <20100928131918.6dba0ca6@pitrou.net> <4CA1D18E.1050601@voidspace.org.uk> Message-ID: On Tue, Sep 28, 2010 at 7:09 AM, Nick Coghlan wrote: > A feature request on the tracker is the best way to make that happen. > Done - http://bugs.python.org/issue9969. Thanks for the feedback everyone. -- Meador -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Sep 28 15:49:24 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 28 Sep 2010 06:49:24 -0700 Subject: [Python-Dev] Pronouncement needed in issue9675 In-Reply-To: <4CA15678.3070004@jcea.es> References: <4CA15678.3070004@jcea.es> Message-ID: On Mon, Sep 27, 2010 at 7:44 PM, Jesus Cea wrote: > http://bugs.python.org/issue9675 > > Long history sort: Python 2.7 backported Capsule support and > (incorrectly, in my opinion) marked CObject as deprecated. > > All C modules in the stdlib were updated to Capsule (with a CObject > compatibility layer), except BSDDB, because this change was done late in > the cycle, the proposed patch was buggy (solvable) and a pronouncement > was done that CObject was not actually deprecated. > > But in python 2.7 release, CObject is marked as deprecated (arg!), so > when executing python with -We (mark warnings as errors), bsddb fails. > > Since I think that adopting Capsule in BSDDB for 2.7.1 would break the > API compatibility (maybe the CObject proxy would solve this), and since > a previous pronouncement was done abour CObject not-deprecated in 2.7.x, > I would like comments. > > Long history and links to previous pronouncements in > http://bugs.python.org/issue9675 > > My proposal: CObject should not be marked as deprecated in 2.7.1. It strikes me that in general deprecation warnings in 2.7 don't do anybody any good unless they're Py3k warnings. It sounds to me that there is no shame in removing the warning in Python 2.7 (or turning it into a Py3k warning -- nobody should expect their code to behave well when Py3k warnings are turned into errors). My guess is that the warning was added to 2.7 before it was clear that there would never be a 2.8. At the same time I don't want to go so far as to remove all remaining deprecation warnings from 2.7.1 (too much churn) -- but for this specific one it sounds like a reasonable compromise. -- --Guido van Rossum (python.org/~guido) From dmalcolm at redhat.com Tue Sep 28 17:18:24 2010 From: dmalcolm at redhat.com (David Malcolm) Date: Tue, 28 Sep 2010 11:18:24 -0400 Subject: [Python-Dev] Pronouncement needed in issue9675 In-Reply-To: <4CA15678.3070004@jcea.es> References: <4CA15678.3070004@jcea.es> Message-ID: <1285687104.10362.44.camel@radiator.bos.redhat.com> On Tue, 2010-09-28 at 04:44 +0200, Jesus Cea wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > http://bugs.python.org/issue9675 > > Long history sort: Python 2.7 backported Capsule support and > (incorrectly, in my opinion) marked CObject as deprecated. > > All C modules in the stdlib were updated to Capsule (with a CObject > compatibility layer), except BSDDB, because this change was done late in > the cycle, the proposed patch was buggy (solvable) and a pronouncement > was done that CObject was not actually deprecated. > > But in python 2.7 release, CObject is marked as deprecated (arg!), so > when executing python with -We (mark warnings as errors), bsddb fails. > > Since I think that adopting Capsule in BSDDB for 2.7.1 would break the > API compatibility (maybe the CObject proxy would solve this), and since > a previous pronouncement was done abour CObject not-deprecated in 2.7.x, > I would like comments. > > Long history and links to previous pronouncements in > http://bugs.python.org/issue9675 > > My proposal: CObject should not be marked as deprecated in 2.7.1. This breaks quite a few third-party modules, some with segfaults; we (as in Fedora) ran into this building with 2.7 as the standard /usr/bin/python for Fedora 14. We fixed some of them, but are advising people not to turn on warnings as errors in our release notes [1] See this tracker bug for a survey of the gory details: https://bugzilla.redhat.com/showdependencytree.cgi?id=620842&hide_resolved=0 The list includes SWIG, numpy, and the GTK stack. I'd be more than happy to patch Fedora's python builds to remove the deprecation warning, but I don't want to stray too far from python.org's "official" behavior here. So, I'm +1 on removing this deprecation warning in 2.7, FWIW (though I'm not a committer yet) Hope this is helpful Dave [1] https://fedoraproject.org/wiki/Features/Python_2.7#Caveat:_PyCObject_and_warnings From dmalcolm at redhat.com Tue Sep 28 17:30:32 2010 From: dmalcolm at redhat.com (David Malcolm) Date: Tue, 28 Sep 2010 11:30:32 -0400 Subject: [Python-Dev] Pronouncement needed in issue9675 In-Reply-To: <1285687104.10362.44.camel@radiator.bos.redhat.com> References: <4CA15678.3070004@jcea.es> <1285687104.10362.44.camel@radiator.bos.redhat.com> Message-ID: <1285687832.10362.48.camel@radiator.bos.redhat.com> On Tue, 2010-09-28 at 11:18 -0400, David Malcolm wrote: > On Tue, 2010-09-28 at 04:44 +0200, Jesus Cea wrote: [snip] > > Long history and links to previous pronouncements in > > http://bugs.python.org/issue9675 Re-reading my post, I realize that my wording was really unclear; sorry. > > > > My proposal: CObject should not be marked as deprecated in 2.7.1. > > This breaks quite a few third-party modules, some with segfaults; we (as > in Fedora) ran into this building with 2.7 as the > standard /usr/bin/python for Fedora 14. Restating: the _deprecation_ breaks quite a few 3rd-party modules, and I'm in agreement with Jesus' proposal. [snip] From solipsis at pitrou.net Tue Sep 28 18:00:13 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 Sep 2010 18:00:13 +0200 Subject: [Python-Dev] Pronouncement needed in issue9675 References: <4CA15678.3070004@jcea.es> Message-ID: <20100928180013.0077ba38@pitrou.net> On Tue, 28 Sep 2010 04:44:08 +0200 Jesus Cea wrote: > > But in python 2.7 release, CObject is marked as deprecated (arg!), so > when executing python with -We (mark warnings as errors), bsddb fails. By "fails" you mean "crashes the interpreter". While the deprecation warning can be discussed, bsddb shouldn't crash when PyCObject_FromVoidPtr() returns NULL, but instead bail out cleanly (or perhaps even ignore the error and simply display a warning that the C API won't be available). Regards Antoine. From guido at python.org Tue Sep 28 18:32:03 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 28 Sep 2010 09:32:03 -0700 Subject: [Python-Dev] Pronouncement needed in issue9675 In-Reply-To: <1285687104.10362.44.camel@radiator.bos.redhat.com> References: <4CA15678.3070004@jcea.es> <1285687104.10362.44.camel@radiator.bos.redhat.com> Message-ID: On Tue, Sep 28, 2010 at 8:18 AM, David Malcolm wrote: [...] > This breaks quite a few third-party modules, some with segfaults; we (as > in Fedora) ran into this building with 2.7 as the > standard /usr/bin/python for Fedora 14. > > We fixed some of them, but are advising people not to turn on warnings > as errors in our release notes [1] Sure, but do realize that if an extension segfaults when warnings are turned into errors, that is ultimately a bug in the extension, and an indication that the extension is playing loose with error checking. There are likely other places where the module's error checking is *also* too loose, and this can be a cause of general instability (e.g. segfaulting instead of raising an exception when other rare errors occur, e.g. interrupts or out of memory or even I/O errors). > See this tracker bug for a survey of the gory details: > https://bugzilla.redhat.com/showdependencytree.cgi?id=620842&hide_resolved=0 > > The list includes SWIG, numpy, and the GTK stack. TBH, I'm not surprised. Those are very large code bases, and SWIG in particular is by its nature limited in the amount of error checking it can provide (as the error checking sometimes has to be in the C++ code it wraps). > I'd be more than happy to patch Fedora's python builds to remove the > deprecation warning, but I don't want to stray too far from python.org's > "official" behavior here. > > So, I'm +1 on removing this deprecation warning in 2.7, FWIW (though I'm > not a committer yet) FWIW, you don't have to be a committer for your +1 to count. But you do have to provide additional argumentation, which you did. (IOW thank you!) -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Tue Sep 28 23:59:55 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Sep 2010 07:59:55 +1000 Subject: [Python-Dev] Pronouncement needed in issue9675 In-Reply-To: References: <4CA15678.3070004@jcea.es> Message-ID: On Tue, Sep 28, 2010 at 11:49 PM, Guido van Rossum wrote: > It strikes me that in general deprecation warnings in 2.7 don't do > anybody any good unless they're Py3k warnings. It sounds to me that > there is no shame in removing the warning in Python 2.7 (or turning it > into a Py3k warning -- nobody should expect their code to behave well > when Py3k warnings are turned into errors). My guess is that the > warning was added to 2.7 before it was clear that there would never be > a 2.8. > > At the same time I don't want to go so far as to remove all remaining > deprecation warnings from 2.7.1 (too ?much churn) -- but for this > specific one it sounds like a reasonable compromise. Converting to a Py3k warning sounds like the best option. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From amauryfa at gmail.com Wed Sep 29 00:23:48 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 29 Sep 2010 00:23:48 +0200 Subject: [Python-Dev] Pronouncement needed in issue9675 In-Reply-To: References: <4CA15678.3070004@jcea.es> Message-ID: 2010/9/28 Nick Coghlan : > Converting to a Py3k warning sounds like the best option. Can someone please explain why converting to a PyCapsule object is not an option? PyCObject_AsVoidPtr() and PyCObject_Import() accept Capsules and will work as before. -- Amaury Forgeot d'Arc From guido at python.org Wed Sep 29 00:56:21 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 28 Sep 2010 15:56:21 -0700 Subject: [Python-Dev] Pronouncement needed in issue9675 In-Reply-To: References: <4CA15678.3070004@jcea.es> Message-ID: On Tue, Sep 28, 2010 at 3:23 PM, Amaury Forgeot d'Arc wrote: > 2010/9/28 Nick Coghlan : >> Converting to a Py3k warning sounds like the best option. > > Can someone please explain why converting to a PyCapsule object is not > an option? > PyCObject_AsVoidPtr() and PyCObject_Import() accept Capsules and will > work as before. Because bsddb is an external module? -- --Guido van Rossum (python.org/~guido) From amauryfa at gmail.com Wed Sep 29 01:02:31 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 29 Sep 2010 01:02:31 +0200 Subject: [Python-Dev] Pronouncement needed in issue9675 In-Reply-To: References: <4CA15678.3070004@jcea.es> Message-ID: 2010/9/29 Guido van Rossum : >> Can someone please explain why converting to a PyCapsule object is not >> an option? >> PyCObject_AsVoidPtr() and PyCObject_Import() accept Capsules and will >> work as before. > > Because bsddb is an external module? Yes, bsddb is compiled in a separate .pyd or .so. But what does this change? -- Amaury Forgeot d'Arc From jcea at jcea.es Wed Sep 29 01:03:44 2010 From: jcea at jcea.es (Jesus Cea) Date: Wed, 29 Sep 2010 01:03:44 +0200 Subject: [Python-Dev] Pronouncement needed in issue9675 In-Reply-To: References: <4CA15678.3070004@jcea.es> Message-ID: <4CA27450.9060106@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 29/09/10 00:56, Guido van Rossum wrote: > On Tue, Sep 28, 2010 at 3:23 PM, Amaury Forgeot d'Arc > wrote: >> 2010/9/28 Nick Coghlan : >>> Converting to a Py3k warning sounds like the best option. >> >> Can someone please explain why converting to a PyCapsule object is not >> an option? >> PyCObject_AsVoidPtr() and PyCObject_Import() accept Capsules and will >> work as before. > > Because bsddb is an external module? Uhm?. I am confused... The reason to rollback the deprecation warning is because it is breaking other external modules too, see a previous post. Since there is no 2.8 planned, a deprecation warning is not useful. See the thread linked in the tracker. About converting the deprecation warning to a py3k warning... Would a py3k warning be converted to an error when python is invoked as "-We"?. If that is the case, we are in the same situation. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTKJ0UJlgi5GaxT1NAQILeQP8C7zIMPG7BAAo2+m414c5jvy5f3wAlKG+ PGO+IOw/AF7FQSrLj59IWJ1E6oHWf/64TYS8tLTsHdo+sHm/gyHBXlqLUVIIZEMG YduS7G2zeU1XwDsSKqg3vhoDZXMNVhJkbEUegayuKEltAPK8McElx5Agqz26knHD mFA5aRtDw28= =5/5Y -----END PGP SIGNATURE----- From guido at python.org Wed Sep 29 01:06:26 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 28 Sep 2010 16:06:26 -0700 Subject: [Python-Dev] Pronouncement needed in issue9675 In-Reply-To: References: <4CA15678.3070004@jcea.es> Message-ID: On Tue, Sep 28, 2010 at 4:02 PM, Amaury Forgeot d'Arc wrote: > 2010/9/29 Guido van Rossum : >>> Can someone please explain why converting to a PyCapsule object is not >>> an option? >>> PyCObject_AsVoidPtr() and PyCObject_Import() accept Capsules and will >>> work as before. >> >> Because bsddb is an external module? > > Yes, bsddb is compiled in a separate .pyd or .so. But what does this change? Because it needs to support multiple Python versions from single source? -- --Guido van Rossum (python.org/~guido) From jcea at jcea.es Wed Sep 29 01:14:30 2010 From: jcea at jcea.es (Jesus Cea) Date: Wed, 29 Sep 2010 01:14:30 +0200 Subject: [Python-Dev] Pronouncement needed in issue9675 In-Reply-To: References: <4CA15678.3070004@jcea.es> Message-ID: <4CA276D6.30803@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 29/09/10 01:06, Guido van Rossum wrote: >>> Because bsddb is an external module? >> >> Yes, bsddb is compiled in a separate .pyd or .so. But what does this change? > > Because it needs to support multiple Python versions from single source? Well, that is my problem, no python-dev :-). I am replying in the mailing list only. It takes a while to propagate. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTKJ21plgi5GaxT1NAQLmGwP+OquVycOmX1MeCxx+wokmHIU7s6HurT4g BJJvCKLZvUByI/NQ9DJP5Ve/n79ZQk208NQou8fEpyvMyUldQJ8hdUbX26ZvDZ7a AZ78eef10Gog2GMVYMO+k41X0d0Pln1+r9RbD5vdLiCeM0iRGOYAKmmNiZQCBsAe LTC5O6+K2O0= =6l9I -----END PGP SIGNATURE----- From martin at v.loewis.de Wed Sep 29 01:18:01 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 29 Sep 2010 01:18:01 +0200 Subject: [Python-Dev] issue2180 and using 'tokenize' with Python 3 'str's In-Reply-To: <4CA164E9.70503@holdenweb.com> References: <4CA164E9.70503@holdenweb.com> Message-ID: <4CA277A9.6070200@v.loewis.de> Am 28.09.2010 05:45, schrieb Steve Holden: > On 9/27/2010 11:27 PM, Benjamin Peterson wrote: >> 2010/9/27 Meador Inge : >>> which, as seen in the trace, is because the 'detect_encoding' function in >>> 'Lib/tokenize.py' searches for 'BOM_UTF8' (a 'bytes' object) in the string >>> to tokenize 'first' (a 'str' object). It seems to me that strings should >>> still be able to be tokenized, but maybe I am missing something. >>> Is the implementation of 'detect_encoding' correct in how it attempts to >>> determine an encoding or should I open an issue for this? >> >> Tokenize only works on bytes. You can open a feature request if you desire. >> > Working only on bytes does seem rather perverse. Yeah, source code really should stop being stored on disks, or else disks should stop being byte-oriented. Let's go the Smalltalk way - they store all source code in the image, no need to deal with perversities like files anymore. Regards, Martin From martin at v.loewis.de Wed Sep 29 01:22:28 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 29 Sep 2010 01:22:28 +0200 Subject: [Python-Dev] issue2180 and using 'tokenize' with Python 3 'str's In-Reply-To: <4CA1D18E.1050601@voidspace.org.uk> References: <4CA164E9.70503@holdenweb.com> <20100928131918.6dba0ca6@pitrou.net> <4CA1D18E.1050601@voidspace.org.uk> Message-ID: <4CA278B4.7000801@v.loewis.de> > I certainly wouldn't be opposed to an API that accepts a string as well > though. Notice that this can't really work for Python 2 source code (but of course, it doesn't need to). In Python 2, if you have a string literal in the source code, you need to know the source encoding in order to get the bytes *back*. Now, if you parse a Unicode string as source code, and it contains byte string literals, you wouldn't know what encoding to apply. Fortunately, Python 3 byte literals ban non-ASCII literal characters, so assuming an ASCII-compatible encoding for the original source is fairly safe. Regards, Martin From amauryfa at gmail.com Wed Sep 29 01:25:23 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 29 Sep 2010 01:25:23 +0200 Subject: [Python-Dev] Pronouncement needed in issue9675 In-Reply-To: References: <4CA15678.3070004@jcea.es> Message-ID: 2010/9/29 Guido van Rossum : > On Tue, Sep 28, 2010 at 4:02 PM, Amaury Forgeot d'Arc > wrote: >> 2010/9/29 Guido van Rossum : >>>> Can someone please explain why converting to a PyCapsule object is not >>>> an option? >>>> PyCObject_AsVoidPtr() and PyCObject_Import() accept Capsules and will >>>> work as before. >>> >>> Because bsddb is an external module? >> >> Yes, bsddb is compiled in a separate .pyd or .so. But what does this change? > > Because it needs to support multiple Python versions from single source? But here we are talking about the bsddb module shipped with core python2.7 http://svn.python.org/view/python/branches/release27-maint/Modules/_bsddb.c This is a single source for a single version. And in any case a #ifdef is good enough. -- Amaury Forgeot d'Arc From benjamin at python.org Wed Sep 29 01:26:09 2010 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 28 Sep 2010 18:26:09 -0500 Subject: [Python-Dev] Pronouncement needed in issue9675 In-Reply-To: References: <4CA15678.3070004@jcea.es> Message-ID: 2010/9/28 Amaury Forgeot d'Arc : > 2010/9/29 Guido van Rossum : >>> Can someone please explain why converting to a PyCapsule object is not >>> an option? >>> PyCObject_AsVoidPtr() and PyCObject_Import() accept Capsules and will >>> work as before. >> >> Because bsddb is an external module? > > Yes, bsddb is compiled in a separate .pyd or .so. But what does this change? By external, I presume he means externally maintained. -- Regards, Benjamin From guido at python.org Wed Sep 29 01:52:33 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 28 Sep 2010 16:52:33 -0700 Subject: [Python-Dev] Pronouncement needed in issue9675 In-Reply-To: <4CA27450.9060106@jcea.es> References: <4CA15678.3070004@jcea.es> <4CA27450.9060106@jcea.es> Message-ID: On Tue, Sep 28, 2010 at 4:03 PM, Jesus Cea wrote: > On 29/09/10 00:56, Guido van Rossum wrote: >> On Tue, Sep 28, 2010 at 3:23 PM, Amaury Forgeot d'Arc >> wrote: >>> 2010/9/28 Nick Coghlan : >>>> Converting to a Py3k warning sounds like the best option. >>> >>> Can someone please explain why converting to a PyCapsule object is not >>> an option? >>> PyCObject_AsVoidPtr() and PyCObject_Import() accept Capsules and will >>> work as before. >> >> Because bsddb is an external module? > > Uhm?. I am confused... > > The reason to rollback the deprecation warning is because it is breaking > other external modules too, see a previous post. Since there is no 2.8 > planned, a deprecation warning is not useful. See the thread linked in > the tracker. > > About converting the deprecation warning to a py3k warning... Would a > py3k warning be converted to an error when python is invoked as "-We"?. > If that is the case, we are in the same situation. Hm, I am okay with programs not working if -We -3 is selected. Why would anyone want to run in that mode anyway? And if -We crashes you have a bug in your C code (not enouggh error checking),. -- --Guido van Rossum (python.org/~guido) From fuzzyman at voidspace.org.uk Wed Sep 29 02:08:56 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 29 Sep 2010 01:08:56 +0100 Subject: [Python-Dev] issue2180 and using 'tokenize' with Python 3 'str's In-Reply-To: <4CA278B4.7000801@v.loewis.de> References: <4CA164E9.70503@holdenweb.com> <20100928131918.6dba0ca6@pitrou.net> <4CA1D18E.1050601@voidspace.org.uk> <4CA278B4.7000801@v.loewis.de> Message-ID: On 29 Sep 2010, at 00:22, "Martin v. L?wis" wrote: >> I certainly wouldn't be opposed to an API that accepts a string as well >> though. > > Notice that this can't really work for Python 2 source code (but of > course, it doesn't need to). > > In Python 2, if you have a string literal in the source code, you need > to know the source encoding in order to get the bytes *back*. Now, > if you parse a Unicode string as source code, and it contains byte > string literals, you wouldn't know what encoding to apply. > > Fortunately, Python 3 byte literals ban non-ASCII literal characters, > so assuming an ASCII-compatible encoding for the original source is > fairly safe. > The new API couldn't be ported to Python 2 ?anyway?. As Nick pointed out, the underlying tokenization happens on decoded strings - so starting with source as Unicode will be fine. Michael > Regards, > Martin From steve at holdenweb.com Wed Sep 29 03:13:27 2010 From: steve at holdenweb.com (Steve Holden) Date: Tue, 28 Sep 2010 21:13:27 -0400 Subject: [Python-Dev] Atlassian and bitbucket merge Message-ID: I see that Atlassian have just taken over BitBucket, the Mercurial hosting company. IIRC Atlassian offered to host our issue tracking on JIRA, but in the end we decided to eat our own dog food and went with roundup. I'm wondering if they'd be similarly interested in supporting our Hg server. Or is self-hosting the only acceptable solution? From recent mail it looks likes we may be up and running on Hg fairly soon. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From brett at python.org Wed Sep 29 08:58:49 2010 From: brett at python.org (Brett Cannon) Date: Tue, 28 Sep 2010 23:58:49 -0700 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: References: Message-ID: On Tue, Sep 28, 2010 at 18:13, Steve Holden wrote: > I see that Atlassian have just taken over BitBucket, the Mercurial > hosting company. IIRC Atlassian offered to host our issue tracking on > JIRA, but in the end we decided to eat our own dog food and went with > roundup. That's right. Enough of a vocal stink was thrown when the infrastructure committee chose a closed source, Java issue tracker that we went with Roundup instead, even though Atlassian offered to host the tracker for us. > > I'm wondering if they'd be similarly interested in supporting our Hg > server. Or is self-hosting the only acceptable solution? From recent > mail it looks likes we may be up and running on Hg fairly soon. Looking at their pricing model, we don't need permission; public repos can have unlimited contributors. Plus bitbucket supports CNAMEs so we would also be able to still have hg.python.org for accessing the repos. The trick would be managing accounts. I would assume either everyone would need bitbucket accounts to add as contributors to a repo, or a dummy python-dev user account would be created where select core devs could add SSH keys to the python-dev user account (although I think the latter would destroy the commit history in terms of who made what change as I suspect bitbucket does it based on the bitbucket account and not one's .hgrc info although I could be wrong). Without knowing how this would be handled I can't really comment. > > regards > ?Steve > -- > Steve Holden ? ? ? ? ? +1 571 484 6266 ? +1 800 494 3119 > DjangoCon US September 7-9, 2010 ? ?http://djangocon.us/ > See Python Video! ? ? ? http://python.mirocommunity.org/ > Holden Web LLC ? ? ? ? ? ? ? ? http://www.holdenweb.com/ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From dirkjan at ochtman.nl Wed Sep 29 09:03:29 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 29 Sep 2010 09:03:29 +0200 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: References: Message-ID: On Wed, Sep 29, 2010 at 03:13, Steve Holden wrote: > I see that Atlassian have just taken over BitBucket, the Mercurial > hosting company. IIRC Atlassian offered to host our issue tracking on > JIRA, but in the end we decided to eat our own dog food and went with > roundup. > > I'm wondering if they'd be similarly interested in supporting our Hg > server. Or is self-hosting the only acceptable solution? From recent > mail it looks likes we may be up and running on Hg fairly soon. Don't know about acceptable, but as far as I know hosting Mercurial repositories doesn't require as much work as hosting Roundup instances (which the Mercurial project also has). I wouldn't mind maintaining hg.python.org, and there are probably other devs that could also easily get involved. Anyway, I don't think using Bitbucket buys us much. It could be nice to keep a mirror there for redundancy and because it might make contributing slightly easier for non-committers, but it won't allow doing all kinds of custom hooks the way we could do with hg.p.o, AFAICT. Cheers, Dirkjan From dirkjan at ochtman.nl Wed Sep 29 09:30:20 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 29 Sep 2010 09:30:20 +0200 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: References: Message-ID: On Wed, Sep 29, 2010 at 08:58, Brett Cannon wrote: > Looking at their pricing model, we don't need permission; public repos > can have unlimited contributors. Plus bitbucket supports CNAMEs so we > would also be able to still have hg.python.org for accessing the > repos. > > The trick would be managing accounts. I would assume either everyone > would need bitbucket accounts to add as contributors to a repo, or a > dummy python-dev user account would be created where select core devs > could add SSH keys to the python-dev user account (although I think > the latter would destroy the commit history in terms of who made what > change as I suspect bitbucket does it based on the bitbucket account > and not one's .hgrc info although I could be wrong). Without knowing > how this would be handled I can't really comment. Yeah, you can just add all the SSH keys to a single account. And usernames are always taken from the hgrc. Remember, the changeset authors are put into the changeset at commit time, not at push time, and the changeset ID depends on it, so you could never have a remote host changing the usernames without the history changing. Still, I think the flexibility of self-hosting (in terms of hooks and extension -- for example the one that would allow lookup by SVN rev) should win out here. Cheers, Dirkjan From hagen at zhuliguan.net Wed Sep 29 10:06:57 2010 From: hagen at zhuliguan.net (=?ISO-8859-1?Q?Hagen_F=FCrstenau?=) Date: Wed, 29 Sep 2010 10:06:57 +0200 Subject: [Python-Dev] Prefetching on buffered IO files In-Reply-To: References: <20100928004119.3963a4ad@pitrou.net> Message-ID: > Ow... I've always assumed that seek() is essentially free, because > that's how a typical OS kernel implements it. If seek() is bad on > GzipFile, how hard would it be to fix this? I'd imagine that there's no easy way to make arbitrary seeks on a GzipFile fast. But wouldn't it be enough to optimize small relative (backwards) seeks? > How common is the use case where you need to read a gzipped pickle > *and* you need to leave the unzipped stream positioned exactly at the > end of the pickle? Not uncommon, I think. You need this for unpickling objects which were dumped one after another into a GzipFile, right? ISTM that the immediate performance issue can be solved by the present patch, and there's room for future improvement by optimizing GzipFile seeks and/or extending the IO API. Cheers, Hagen From dirkjan at ochtman.nl Wed Sep 29 10:15:25 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 29 Sep 2010 10:15:25 +0200 Subject: [Python-Dev] hg conversion: tags In-Reply-To: <20100927183635.5bbb337b@pitrou.net> References: <20100927112851.1455ea34@mission> <20100927183635.5bbb337b@pitrou.net> Message-ID: Okay, so let's summarize this thread so far: Martin is in favor of removing some tags (certainly partial ones), but is -0 on renaming them. Tres is in favor of renaming release tags. Georg advocates removing non-release tags, and doesn't care much about renaming. Barry would like to clean up release tag names (either dotted or hexversion). Alexander is +1 on renaming tags on the premise that "r311" tags conflict with "r42543" SVN rev notation. Antoine is in favor of renaming tags on the premise that using hg log -rr311 is awkward in comparison to -r3.1.1. So it seems like most people here like the idea of renaming the tags. Martin and Georg would also like to get rid of some of the non-release tags, but it's unclear if we should get rid of all non-release tags or just the partial ones. Below is a grouped list. Based on that list, I propose the following scheme: - keep all "normal" release tags, renamed along these lines: r27 -> 2.7 r266 -> 2.6.6 r27rc2 -> 2.7rc2 r262c1 -> 2.6.2rc1 (rc and c should be normalized into the same thing, don't care which one) r30b3 -> 3.0b3 release22 -> 2.2 - keep Mac release tags that don't have the top-level Mac dir, renamed along these lines: r23b1-mac -> 2.3b1-mac - get rid of "special" release tags and Mac release tags with top-level Mac dir - get rid of email and distutils tags - get rid of all other tags Does that make sense? Cheers, Dirkjan ========================================== "Normal" release tags: r32a2 r266 r266rc2 r266rc1 r32a1 r27 r27rc2 r27rc1 r27b2 r27b1 r312 r265 r265rc2 r312rc1 r27a4 r265rc1 r27a3 r255 r255c2 r255c1 r27a2 r27a1 r311 r264 r264rc2 r264rc1 r263 r263rc1 r311rc1 r31 r31rc2 r31rc1 r31b1 r262 r262c1 r31a2 r31a1 r301 r254 r253 r246 r253c1 r246c1 r261 r30 r30rc3 r30rc2 r26 r26rc2 r30rc1 r26rc1 r30b3 r26b3 r26b2 r30b2 r26b1 r30b1 r26a3 r30a5 r26a2 r30a4 r237 r245 r237c1 r245c1 r30a3 r26a1 r252 r252c1 r251c1 r30a2 r30a1 r251 r236 r236c1 r244 r244c1 r25 r25c2 r25c1 r25b3 r25b2 r25b1 r25a2 r25a1 r25a0 r243 r243c1 r242 r242c1 r241 r241c2 r241c1 r235 r235c1 r24 r24c1 r24b2 r24b1 r24a3 r24a2 r24a1 r234 r234c1 r233 r233c1 r232 r232c1 r231 r23 r23c2 r23c1 r23b2 r09b1 r223 r223c1 r23b1 r23a2 r09a2 r09a1 r23a1 r09a0 r222 r222b1 r221 r213 r22p1 r221c2 r221c1 r212 r212c1 r22c1 r22b2 r22b1 r22a4 r22a3 r22a2 r211 r22a1 r211c1 r201 r201c1 r21c2 r21c1 r21b2 r21b1 r161 r21a2 r21a1 r20c1 r20b2 r20b11 r20b1 r16b1 r16a2 r16a1 r152 r152c1 r06 r152b2 r152b1 r01 r152a2 r152a1 r151 r15b2 r15b1 r15a4 r15a3 r15a2 r15a1 r14beta3 r14beta2 r14beta1 r13beta1 r12beta4 r12beta3 r12beta2 r12beta1 release22 release21 release20 release16 release15 release14 release13 release12 release111 release11 release102 release101 release100 release099 release098 "Special" release tags: release152p1-branch-tag r09a2nt r16b1-win r15a4near r22b1-docs-prep r22a3-docs-prep r22a2-docs-prep r23a1-fork r23rc1-fork r23b2-fork r23b1-fork r23a2-fork r22b2-fork r22rc1-fork r22a2-fork r22b1-fork r22a4-fork r22a3-fork release24-fork release23-fork release22-fork release152p2 (docs only) release152p1 (docs only) release152 (docs only) release152b2 (docs only) release152b1 (docs only) release152a1 (docs only) release151p1 (docs only) Mac releases: r23b1-mac r222-mac r23-mac r22c1-mac r22b2-mac r22b1-mac release22-mac mac102 r22c1-macmergefromtrunk (Mac dir) release22-macmerge (Mac dir) mac152c1 (Mac dir) mac152b1 (Mac dir) mac151 (Mac dir) mac210 (Mac dir) mac210b2 (Mac dir) mac210b1a (Mac dir) mac210b1 (Mac dir) mac210a3 (Mac dir) mac210a1 (Mac dir) mac200 (Mac dir) mac200b1 (Mac dir) distutils tags (distutils only): Distutils-1_0_3 Distutils-1_0_2 Distutils-1_0_1 Distutils-1_0 Distutils-0_9_4 Distutils-0_9_3 Distutils-0_9_2 Distutils-0_9_1 Distutils-0_9 Distutils-0_8_2 Distutils-0_8_1 Distutils-0_8 Distutils-0_1_5 Distutils-0_1_4 Distutils-0_1_3-branch Distutils-0_1_3 Distutils-0_1_2 Distutils-0_1_1 Distutils-0_1 email tags (email only): email_Release_2_5_6 email_Release_2_5_5 email_Release_2_5_3 email_Release_2_5_2 email_Release_2_5_1 email_Release_2_5 email_Release_2_5b1 email_Release_2_4_3 email_Release_2_4_2 email_Release_2_4_1 email_Release_2_4 email_Release_2_3_1 email_Release_2_3 email_Release_2_2 email_Release_2_1 Release_2_0_4 email_Release_2_0_5 Other tags: r25a0/trunk py3k-before-rstdocs py26-before-rstdocs before-ast-merge-to-head mrg_to_ast-branch_15OCT05 IDLE-syntax-root merged_from_MAIN_07JAN05 mrg_to_ast-branch_05JAN05 pre-sf-818006 bsddb_version_4_3_0 tim-doctest-closed tim-doctest-merge-24a2 pre-sf-1149508 start-sf-965425 before-ast-merge-from-head bsddb_version_4_2_4 pybsddb_after_bsddb42_01 pybsddb_before_bsddb42 bsddb_version_4_1_6 anthony-parser-branchpoint IDLELIB_CREATED COPY_TO_PYTHON mrg_to_ast-branch_24APR03 cache-attr-fork LAST_OLD_IDLE before-bgen-pep252 py-cvs-2002_09_13-merged py-cvs-2002_09_13 MERGED_FROM_DS_RPC_13SEP02 MERGED_TO_MAIN_13SEP02 PRE_DS_RPC_MERGE BEFORE_RESTART final_classic_builds TAG_pre_teyc_gvr_rpc_patch final_CW6_projects universal_headers_332 before-carbon-package after-descr-branch-merge date2001-08-01 date2001-07-30 date2001-07-28 IDLEFORK_081_RELEASE date2001-07-21 date2001-07-17b date2001-07-17a date2001-07-16 date2001-07-15 py-cvs-2001_07_13-merged py-cvs-2001_07_13 py-cvs-rel2_1-merged date2001-07-13 py-cvs-rel2_1 py-cvs-2000_03_09 descr-fork date2001-07-06 base-VP-idle merged-with-main-repository after-call-reorg before-call-reorg pre_amk last_cw_pro_53 last_68k_projects cw_pro_5 arelease pre_GUSI2 pre-unicode idle05 pre_0_2_breakage mx_builder-alpha2 pre-string-meths idle-r02 PYIDE_APR98 PYTHONSCRIPT_0_5b1 OSAM_APR98 BBPY_0_2_3 lastoldnames lastoldname cwisync1 cnrisync2 chameleon-1 cnrisync proof3 proof2 proof1 mhammond last099 pre_jh v0_10 (docs only) v0_09 (docs only) v0_08 (docs only) v0_07 (docs only) v0_06 (docs only) Public_Release_20-Aug-1998 (one file only) Public_Release_27-Mar-1998 (one file only) Public_13-Mar-1998 (one file only) Public_11-Mar-1998 (one file only) Beta_20-Aug-1996 (one file only) Beta_09-Aug-1996 (one file only) Beta_05-Jul-1996 (one file only) Beta_15-Mar-1995-#2 (one file only) Beta_15-Mar-1995 (one file only) Beta_14-Mar-1995-#3 (one file only) Beta_14-Mar-1995-#2 (one file only) Beta_14-Mar-1995 (one file only) Beta_20-Mar-1995 (one file only) Public_05-Jul-1995 (one file only) Release_1_0 (one file only) Release_0_1 (one file only) Python_1_2_release (one file only) Python1_4final (one file only) r1_95_2 (partial) From mal at egenix.com Wed Sep 29 10:29:05 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 29 Sep 2010 10:29:05 +0200 Subject: [Python-Dev] hg conversion: tags In-Reply-To: References: <20100927112851.1455ea34@mission> <20100927183635.5bbb337b@pitrou.net> Message-ID: <4CA2F8D1.1060704@egenix.com> Dirkjan Ochtman wrote: > Okay, so let's summarize this thread so far: > > Martin is in favor of removing some tags (certainly partial ones), but > is -0 on renaming them. > Tres is in favor of renaming release tags. > Georg advocates removing non-release tags, and doesn't care much about renaming. > Barry would like to clean up release tag names (either dotted or hexversion). > Alexander is +1 on renaming tags on the premise that "r311" tags > conflict with "r42543" SVN rev notation. > Antoine is in favor of renaming tags on the premise that using hg log > -rr311 is awkward in comparison to -r3.1.1. > > So it seems like most people here like the idea of renaming the tags. > Martin and Georg would also like to get rid of some of the non-release > tags, but it's unclear if we should get rid of all non-release tags or > just the partial ones. > > Below is a grouped list. Based on that list, I propose the following scheme: > > - keep all "normal" release tags, renamed along these lines: > r27 -> 2.7 > r266 -> 2.6.6 > r27rc2 -> 2.7rc2 > r262c1 -> 2.6.2rc1 (rc and c should be normalized into the same > thing, don't care which one) > r30b3 -> 3.0b3 > release22 -> 2.2 > > - keep Mac release tags that don't have the top-level Mac dir, renamed > along these lines: > r23b1-mac -> 2.3b1-mac > > - get rid of "special" release tags and Mac release tags with top-level Mac dir > - get rid of email and distutils tags > - get rid of all other tags > > Does that make sense? I'm not sure whether throwing away history in form of such tags is a good idea. I don't know how hg manages this, but can't we preserve the tag information of the tags that you've scheduled to be removed in some place that can easily be pulled in but doesn't affect the main repo size ? Renaming the release tags certainly is a good idea, since we're not stuck with CVS naming requirements anymore. I'd prefix the release tags with "release-" for additional context, though. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 29 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Wed Sep 29 10:55:56 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 29 Sep 2010 10:55:56 +0200 Subject: [Python-Dev] Prefetching on buffered IO files References: <20100928004119.3963a4ad@pitrou.net> Message-ID: <20100929105556.7976772a@pitrou.net> On Wed, 29 Sep 2010 10:06:57 +0200 Hagen F?rstenau wrote: > > Ow... I've always assumed that seek() is essentially free, because > > that's how a typical OS kernel implements it. If seek() is bad on > > GzipFile, how hard would it be to fix this? > > I'd imagine that there's no easy way to make arbitrary seeks on a > GzipFile fast. But wouldn't it be enough to optimize small relative > (backwards) seeks? As I explained to Guido, GzipFile doesn't know the buffering size of its consumer (apart from introducing couplings), and therefore has no way to know how much information it must retain. To reiterate, there's a complicated solution (optimize an implementation-dependent behaviour of GzipFile, with a non-trivial coding effort and performance tradeoff) which will not work on unseekable files anyway. And there's a more generic solution involving non-seeking primitives such as read() + peek(). (follow-up to python-ideas, if I didn't mess up the headers) Regards Antoine. From victor.stinner at haypocalc.com Wed Sep 29 11:35:28 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 29 Sep 2010 11:35:28 +0200 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: References: Message-ID: <201009291135.28572.victor.stinner@haypocalc.com> Le mercredi 29 septembre 2010 08:58:49, Brett Cannon a ?crit : > The trick would be managing accounts. I would assume either everyone > would need bitbucket accounts to add as contributors to a repo, or a > dummy python-dev user account would be created where select core devs > could add SSH keys to the python-dev user account (although I think > the latter would destroy the commit history in terms of who made what > change as I suspect bitbucket does it based on the bitbucket account > and not one's .hgrc info although I could be wrong). Can't we rewrite the history when converting from svn to hg to use real names instead of logins? Mercurial 1.6 (and maybe older versions) refuses to commit if the user didn't set its name (in ~/.hgrc). Bitbucket doesn't rewrite commit authors: it keeps original names. So developers have to configure correctly their Mercurial, but it allows to keep the name of the contributor on a contribution. Is it possible with Mercurial to add a tag on a commit to specify who commited a contribution? Like the Signed-By tag with git on the Linux kernel. -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Wed Sep 29 11:50:14 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 29 Sep 2010 11:50:14 +0200 Subject: [Python-Dev] Atlassian and bitbucket merge References: Message-ID: <20100929115014.2965f82d@pitrou.net> On Wed, 29 Sep 2010 09:03:29 +0200 Dirkjan Ochtman wrote: > > Anyway, I don't think using Bitbucket buys us much. It could be nice > to keep a mirror there for redundancy and because it might make > contributing slightly easier for non-committers, but it won't allow > doing all kinds of custom hooks the way we could do with hg.p.o, > AFAICT. Using Bitbucket seems mainly useful if you need the whole suite of services (issue tracker, wiki, etc.). Also, I find the Bitbucket UI horrible. Better than Launchpad probably, but still... Regards Antoine. From ben+python at benfinney.id.au Wed Sep 29 12:32:55 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 29 Sep 2010 20:32:55 +1000 Subject: [Python-Dev] Atlassian and bitbucket merge References: Message-ID: <87tyl82620.fsf@benfinney.id.au> Dirkjan Ochtman writes: > Still, I think the flexibility of self-hosting (in terms of hooks and > extension -- for example the one that would allow lookup by SVN rev) > should win out here. Not only the flexibility, but the autonomy. Hosting the source code on systems either paid for by PSF funds or donated to the PSF will allow all decisions about how the VCS repositories are administrated to be made without deferring to the repository provider as a separate entity. Or, more briefly: it's better for the PSF to be as much in control of the decisions about what happens to the VCS repositories as possible. -- \ ?We have to go forth and crush every world view that doesn't | `\ believe in tolerance and free speech.? ?David Brin | _o__) | Ben Finney From g.brandl at gmx.net Wed Sep 29 12:42:34 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 29 Sep 2010 12:42:34 +0200 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: References: Message-ID: Am 29.09.2010 09:03, schrieb Dirkjan Ochtman: > On Wed, Sep 29, 2010 at 03:13, Steve Holden wrote: >> I see that Atlassian have just taken over BitBucket, the Mercurial >> hosting company. IIRC Atlassian offered to host our issue tracking on >> JIRA, but in the end we decided to eat our own dog food and went with >> roundup. >> >> I'm wondering if they'd be similarly interested in supporting our Hg >> server. Or is self-hosting the only acceptable solution? From recent >> mail it looks likes we may be up and running on Hg fairly soon. > > Don't know about acceptable, but as far as I know hosting Mercurial > repositories doesn't require as much work as hosting Roundup instances > (which the Mercurial project also has). I wouldn't mind maintaining > hg.python.org, and there are probably other devs that could also > easily get involved. > > Anyway, I don't think using Bitbucket buys us much. It could be nice > to keep a mirror there for redundancy and because it might make > contributing slightly easier for non-committers, but it won't allow > doing all kinds of custom hooks the way we could do with hg.p.o, > AFAICT. Agreed on both counts. Bitbucket will be a good solution for non-core developers to host their clones and branches, and I'm sure there will be a mirror of the main Python repo somewhere. But as you say, hosting Mercurial is very easy, and doesn't require constant administration. The argument of hooks is a very strong one in favor of hg.python.org. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From dirkjan at ochtman.nl Wed Sep 29 12:46:29 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 29 Sep 2010 12:46:29 +0200 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: <201009291135.28572.victor.stinner@haypocalc.com> References: <201009291135.28572.victor.stinner@haypocalc.com> Message-ID: On Wed, Sep 29, 2010 at 11:35, Victor Stinner wrote: > Can't we rewrite the history when converting from svn to hg to use real names > instead of logins? I've been doing that since the start, look at the test repo on hg.p.o. Cheers, Dirkjan From python-dev at masklinn.net Wed Sep 29 12:58:48 2010 From: python-dev at masklinn.net (Xavier Morel) Date: Wed, 29 Sep 2010 12:58:48 +0200 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: <20100929115014.2965f82d@pitrou.net> References: <20100929115014.2965f82d@pitrou.net> Message-ID: <04B19C10-EC03-4ADC-BBB9-2AB0C1B4B72E@masklinn.net> On 2010-09-29, at 11:50 , Antoine Pitrou wrote: > On Wed, 29 Sep 2010 09:03:29 +0200 > Dirkjan Ochtman wrote: >> >> Anyway, I don't think using Bitbucket buys us much. It could be nice >> to keep a mirror there for redundancy and because it might make >> contributing slightly easier for non-committers, but it won't allow >> doing all kinds of custom hooks the way we could do with hg.p.o, >> AFAICT. > > Using Bitbucket seems mainly useful if you need the whole suite of > services (issue tracker, wiki, etc.). > The most useful features are probably the follow and fork, but for a project as big as Python I'm not sure those are going to be used a lot. The question then becomes whether Python development workflow will remain as-is or would more to a "pull-request" model via bitbucket. If it's negative, then I see no intrinsic value in the main server being on bitbucket. From solipsis at pitrou.net Wed Sep 29 13:01:56 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 29 Sep 2010 13:01:56 +0200 Subject: [Python-Dev] Resource leaks warnings References: <20100808221846.CFD80EEA3F@mail.python.org> Message-ID: <20100929130156.29767af1@pitrou.net> Hello, > I'd like to ask your opinion on this change; I think it should be reverted > or at least made silent by default. Basically, it prints a warning like > > gc: 2 uncollectable objects at shutdown: > Use gc.set_debug(gc.DEBUG_UNCOLLECTABLE) to list them. > > at interpreter shutdown if gc.garbage is nonempty. I would like to piggy-back on this discussion to suggest further warnings (either by default, or switchable). One feature I've often considered would be to add a warning in FileIO and socket dealloc if these objects haven't been closed explicitly. In most situations, relying on garbage collection to shutdown OS resources (here, file descriptors) is something we like to discourage. Furthermore, it can produce real bugs, especially under Windows when coupled with refererence cycles created by traceback objects (the random test_tarfile failures on the Windows buildbots were a symptom of that; their cause would have been obvious with such warnings). What do you think? Antoine. From ncoghlan at gmail.com Wed Sep 29 14:04:24 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Sep 2010 22:04:24 +1000 Subject: [Python-Dev] Pronouncement needed in issue9675 In-Reply-To: <4CA27450.9060106@jcea.es> References: <4CA15678.3070004@jcea.es> <4CA27450.9060106@jcea.es> Message-ID: On Wed, Sep 29, 2010 at 9:03 AM, Jesus Cea wrote: > About converting the deprecation warning to a py3k warning... Would a > py3k warning be converted to an error when python is invoked as "-We"?. > If that is the case, we are in the same situation. To unpack Guido's response slightly, Py3k specific deprecation warnings are only triggered if you pass the "-3" flag on the command line. So "-We" won't cause them to throw an exception, but "-We -3" will. (Python 2.6.5, started with "-We") >>> warnings.warnpy3k("Test") >>> sys.py3kwarning = True >>> warnings.warnpy3k("Test") Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.6/warnings.py", line 22, in warnpy3k warn(message, category, stacklevel+1) DeprecationWarning: Test Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Sep 29 14:16:42 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Sep 2010 22:16:42 +1000 Subject: [Python-Dev] hg conversion: tags In-Reply-To: <4CA2F8D1.1060704@egenix.com> References: <20100927112851.1455ea34@mission> <20100927183635.5bbb337b@pitrou.net> <4CA2F8D1.1060704@egenix.com> Message-ID: On Wed, Sep 29, 2010 at 6:29 PM, M.-A. Lemburg wrote: > I'm not sure whether throwing away history in form of such tags > is a good idea. > > I don't know how hg manages this, but can't we preserve the tag > information of the tags that you've scheduled to be removed > in some place that can easily be pulled in but doesn't > affect the main repo size ? But why bother? The tags are static, so grabbing them from svn instead of hg shouldn't be a big issue. If we had unlimited resources to support the transition my opinion would probably be different, but since we don't, applying the simple rule of culling the non-release tags seems good enough and better than spending too much time trying to figure out which tags are "important" enough to be worth preserving. > Renaming the release tags certainly is a good idea, since > we're not stuck with CVS naming requirements anymore. I'd prefix > the release tags with "release-" for additional context, > though. So long as we don't start using bare numbers for anything other than releases, I think that would just become redundant typing in fairly short order. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From benjamin at python.org Wed Sep 29 14:27:58 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 29 Sep 2010 07:27:58 -0500 Subject: [Python-Dev] Resource leaks warnings In-Reply-To: <20100929130156.29767af1@pitrou.net> References: <20100808221846.CFD80EEA3F@mail.python.org> <20100929130156.29767af1@pitrou.net> Message-ID: 2010/9/29 Antoine Pitrou : > > Hello, > >> I'd like to ask your opinion on this change; I think it should be reverted >> or at least made silent by default. ?Basically, it prints a warning like >> >> ? ? ? gc: 2 uncollectable objects at shutdown: >> ? ? ? ? ? Use gc.set_debug(gc.DEBUG_UNCOLLECTABLE) to list them. >> >> at interpreter shutdown if gc.garbage is nonempty. > > I would like to piggy-back on this discussion to suggest further > warnings (either by default, or switchable). > > One feature I've often considered would be to add a warning in FileIO > and socket dealloc if these objects haven't been closed explicitly. In > most situations, relying on garbage collection to shutdown OS resources > (here, file descriptors) is something we like to discourage. > Furthermore, it can produce real bugs, especially under Windows when > coupled with refererence cycles created by traceback objects (the > random test_tarfile failures on the Windows buildbots were a symptom of > that; their cause would have been obvious with such warnings). > > What do you think? It seems like a slippery slope. Sometimes you really don't care like when you're just hacking together a quick script. -- Regards, Benjamin From tseaver at palladion.com Wed Sep 29 14:32:24 2010 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 29 Sep 2010 08:32:24 -0400 Subject: [Python-Dev] hg conversion: tags In-Reply-To: References: <20100927112851.1455ea34@mission> <20100927183635.5bbb337b@pitrou.net> <4CA2F8D1.1060704@egenix.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/29/2010 08:16 AM, Nick Coghlan wrote: > On Wed, Sep 29, 2010 at 6:29 PM, M.-A. Lemburg wrote: >> I'm not sure whether throwing away history in form of such tags >> is a good idea. >> >> I don't know how hg manages this, but can't we preserve the tag >> information of the tags that you've scheduled to be removed >> in some place that can easily be pulled in but doesn't >> affect the main repo size ? > > But why bother? The tags are static, so grabbing them from svn instead > of hg shouldn't be a big issue. If we had unlimited resources to > support the transition my opinion would probably be different, but > since we don't, applying the simple rule of culling the non-release > tags seems good enough and better than spending too much time trying > to figure out which tags are "important" enough to be worth > preserving. I think the key heuristic is which information you want to use directly in Hg, e.g. to diff between tags, or diff a working branch against a tag. Based on how I use tags under SVN, the release tags account for nearly all of such cases. Having to go back to SVN to check out the rare exception seems like a good tradeoff. >> Renaming the release tags certainly is a good idea, since >> we're not stuck with CVS naming requirements anymore. I'd prefix >> the release tags with "release-" for additional context, >> though. > > So long as we don't start using bare numbers for anything other than > releases, I think that would just become redundant typing in fairly > short order. +1 for bare release numbers (Dirkjan's proposal). Although I would prefer 'c' over 'rc' in the normalized case, PEP 386 allows 'rc' as an alternative to 'c' precisely because some Python versions used it). I think we need to make the migrated version tags match the corresponding tarball version numbers exactly. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkyjMdgACgkQ+gerLs4ltQ4Y1ACfeIK6KtO7RAZyzcSp5ap2/Zn6 bH8AnjQGRFjrI7PaisUcSex3nsFp4AR/ =f8ZR -----END PGP SIGNATURE----- From dirkjan at ochtman.nl Wed Sep 29 14:34:33 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 29 Sep 2010 14:34:33 +0200 Subject: [Python-Dev] Resource leaks warnings In-Reply-To: References: <20100808221846.CFD80EEA3F@mail.python.org> <20100929130156.29767af1@pitrou.net> Message-ID: On Wed, Sep 29, 2010 at 14:27, Benjamin Peterson wrote: > It seems like a slippery slope. Sometimes you really don't care like > when you're just hacking together a quick script. Yeah, I often don't close files in scripts that I know are short running or only ever open one or two files, and I don't think I should be warned about that. Cheers, Dirkjan From exarkun at twistedmatrix.com Wed Sep 29 14:36:45 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 29 Sep 2010 12:36:45 -0000 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: References: Message-ID: <20100929123645.4943.388178770.divmod.xquotient.94@localhost.localdomain> On 01:13 am, steve at holdenweb.com wrote: >I see that Atlassian have just taken over BitBucket, the Mercurial >hosting company. IIRC Atlassian offered to host our issue tracking on >JIRA, but in the end we decided to eat our own dog food and went with >roundup. > >I'm wondering if they'd be similarly interested in supporting our Hg >server. Or is self-hosting the only acceptable solution? From recent >mail it looks likes we may be up and running on Hg fairly soon. I know of two medium sized projects (smaller than CPython) that are switching away from BitBucket's free services because of their poor reliability. Jean-Paul From benjamin at python.org Wed Sep 29 14:38:02 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 29 Sep 2010 07:38:02 -0500 Subject: [Python-Dev] hg conversion: tags In-Reply-To: References: <20100927112851.1455ea34@mission> <20100927183635.5bbb337b@pitrou.net> <4CA2F8D1.1060704@egenix.com> Message-ID: 2010/9/29 Tres Seaver : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 09/29/2010 08:16 AM, Nick Coghlan wrote: >> On Wed, Sep 29, 2010 at 6:29 PM, M.-A. Lemburg wrote: >>> I'm not sure whether throwing away history in form of such tags >>> is a good idea. >>> >>> I don't know how hg manages this, but can't we preserve the tag >>> information of the tags that you've scheduled to be removed >>> in some place that can easily be pulled in but doesn't >>> affect the main repo size ? >> >> But why bother? The tags are static, so grabbing them from svn instead >> of hg shouldn't be a big issue. If we had unlimited resources to >> support the transition my opinion would probably be different, but >> since we don't, applying the simple rule of culling the non-release >> tags seems good enough and better than spending too much time trying >> to figure out which tags are "important" enough to be worth >> preserving. > > I think the key heuristic is which information you want to use directly > in Hg, e.g. to diff between tags, or diff a working branch against a > tag. ?Based on how I use tags under SVN, the release tags account for > nearly all of such cases. ?Having to go back to SVN to check out the > rare exception seems like a good tradeoff. > >>> Renaming the release tags certainly is a good idea, since >>> we're not stuck with CVS naming requirements anymore. I'd prefix >>> the release tags with "release-" for additional context, >>> though. >> >> So long as we don't start using bare numbers for anything other than >> releases, I think that would just become redundant typing in fairly >> short order. > > +1 for bare release numbers (Dirkjan's proposal). ?Although I would > prefer 'c' over 'rc' in the normalized case, PEP 386 allows 'rc' as an > alternative to 'c' precisely because some Python versions used it). ?I > think we need to make the migrated version tags match the corresponding > tarball version numbers exactly. Well, the tarballs use rc, too. -- Regards, Benjamin From solipsis at pitrou.net Wed Sep 29 14:42:27 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 29 Sep 2010 14:42:27 +0200 Subject: [Python-Dev] Resource leaks warnings In-Reply-To: References: <20100808221846.CFD80EEA3F@mail.python.org> <20100929130156.29767af1@pitrou.net> Message-ID: <1285764147.3168.10.camel@localhost.localdomain> Le mercredi 29 septembre 2010 ? 07:27 -0500, Benjamin Peterson a ?crit : > > > > I would like to piggy-back on this discussion to suggest further > > warnings (either by default, or switchable). > > > > One feature I've often considered would be to add a warning in FileIO > > and socket dealloc if these objects haven't been closed explicitly. In > > most situations, relying on garbage collection to shutdown OS resources > > (here, file descriptors) is something we like to discourage. > > Furthermore, it can produce real bugs, especially under Windows when > > coupled with refererence cycles created by traceback objects (the > > random test_tarfile failures on the Windows buildbots were a symptom of > > that; their cause would have been obvious with such warnings). > > > > What do you think? > > It seems like a slippery slope. Sometimes you really don't care like > when you're just hacking together a quick script. Isn't the "with" statement appropriate in these cases? My assumption is/was that the benefit of warning against leaks in real applications (or even - sigh - the standard library) would outweigh the inconvenience when hacking together a quick script. But if it doesn't, what about enabling it with a command-line switch? From steve at holdenweb.com Wed Sep 29 14:52:07 2010 From: steve at holdenweb.com (Steve Holden) Date: Wed, 29 Sep 2010 08:52:07 -0400 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: <87tyl82620.fsf@benfinney.id.au> References: <87tyl82620.fsf@benfinney.id.au> Message-ID: <4CA33677.5080202@holdenweb.com> On 9/29/2010 6:32 AM, Ben Finney wrote: > Dirkjan Ochtman writes: > >> Still, I think the flexibility of self-hosting (in terms of hooks and >> extension -- for example the one that would allow lookup by SVN rev) >> should win out here. > > Not only the flexibility, but the autonomy. Hosting the source code on > systems either paid for by PSF funds or donated to the PSF will allow > all decisions about how the VCS repositories are administrated to be > made without deferring to the repository provider as a separate entity. > > Or, more briefly: it's better for the PSF to be as much in control of > the decisions about what happens to the VCS repositories as possible. > That's true, but we also have to balance that against the resources it takes to develop and manage our own hosting solutions. I'm not trying to push one way or the other (and Dirkan's points about tailored commit hooks seems compelling); I simply want to make sure that as much developer time as possible is actually spent developing Python rather than supporting the development effort in various ways that aren't as directly productive. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From steve at holdenweb.com Wed Sep 29 14:52:07 2010 From: steve at holdenweb.com (Steve Holden) Date: Wed, 29 Sep 2010 08:52:07 -0400 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: <87tyl82620.fsf@benfinney.id.au> References: <87tyl82620.fsf@benfinney.id.au> Message-ID: <4CA33677.5080202@holdenweb.com> On 9/29/2010 6:32 AM, Ben Finney wrote: > Dirkjan Ochtman writes: > >> Still, I think the flexibility of self-hosting (in terms of hooks and >> extension -- for example the one that would allow lookup by SVN rev) >> should win out here. > > Not only the flexibility, but the autonomy. Hosting the source code on > systems either paid for by PSF funds or donated to the PSF will allow > all decisions about how the VCS repositories are administrated to be > made without deferring to the repository provider as a separate entity. > > Or, more briefly: it's better for the PSF to be as much in control of > the decisions about what happens to the VCS repositories as possible. > That's true, but we also have to balance that against the resources it takes to develop and manage our own hosting solutions. I'm not trying to push one way or the other (and Dirkan's points about tailored commit hooks seems compelling); I simply want to make sure that as much developer time as possible is actually spent developing Python rather than supporting the development effort in various ways that aren't as directly productive. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ From tseaver at palladion.com Wed Sep 29 15:04:27 2010 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 29 Sep 2010 09:04:27 -0400 Subject: [Python-Dev] hg conversion: tags In-Reply-To: References: <20100927112851.1455ea34@mission> <20100927183635.5bbb337b@pitrou.net> <4CA2F8D1.1060704@egenix.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/29/2010 08:38 AM, Benjamin Peterson wrote: > 2010/9/29 Tres Seaver : >> +1 for bare release numbers (Dirkjan's proposal). Although I would >> prefer 'c' over 'rc' in the normalized case, PEP 386 allows 'rc' as an >> alternative to 'c' precisely because some Python versions used it). I >> think we need to make the migrated version tags match the corresponding >> tarball version numbers exactly. > > Well, the tarballs use rc, too. Right. I meant to indicate that we should map any 'rc' in the historical release tarball name onto 'rc' in the new Hg tag. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkyjOVsACgkQ+gerLs4ltQ4r9wCfYKYhfPTn/At4zqBuGCqYIwPf uQUAoNPsbKLkSDPflITveyBQ+VE24Tnl =vBUr -----END PGP SIGNATURE----- From steve at pearwood.info Wed Sep 29 15:11:44 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 29 Sep 2010 23:11:44 +1000 Subject: [Python-Dev] Resource leaks warnings In-Reply-To: <1285764147.3168.10.camel@localhost.localdomain> References: <20100808221846.CFD80EEA3F@mail.python.org> <1285764147.3168.10.camel@localhost.localdomain> Message-ID: <201009292311.45355.steve@pearwood.info> On Wed, 29 Sep 2010 10:42:27 pm Antoine Pitrou wrote: > My assumption is/was that the benefit of warning against leaks in > real applications (or even - sigh - the standard library) would > outweigh the inconvenience when hacking together a quick script. > > But if it doesn't, what about enabling it with a command-line switch? I think the ability to detect such file descriptor leaks would be valuable, but I'm not sure that it should be running all the time. At the risk of bike-shedding, is it something which could be controlled at runtime, like garbage collection? E.g. something like: gc.enable_file_warnings() run_my_tests_for_leakage() gc.disable_file_warnings() or similar. (I'm not wedded to it being in the gc module.) Otherwise, I'm +0.25 on enabling it with a command line switch, and -0 on turning it on by default. -- Steven D'Aprano From hrvoje.niksic at avl.com Wed Sep 29 15:16:14 2010 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Wed, 29 Sep 2010 15:16:14 +0200 Subject: [Python-Dev] Resource leaks warnings In-Reply-To: <1285764147.3168.10.camel@localhost.localdomain> References: <20100808221846.CFD80EEA3F@mail.python.org> <20100929130156.29767af1@pitrou.net> <1285764147.3168.10.camel@localhost.localdomain> Message-ID: <4CA33C1E.70702@avl.com> On 09/29/2010 02:42 PM, Antoine Pitrou wrote: >> It seems like a slippery slope. Sometimes you really don't care like >> when you're just hacking together a quick script. > > Isn't the "with" statement appropriate in these cases? A hacked-together quick script might contain code like: parse(open(bla).read()) Compared to this, "with" adds a new indentation level and a new variable, while breaking the flow of the code: with open(bla) as foo: contents = foo.read() parse(contents) People used to writing production code under stringent guidelines (introduced for good reason) will probably not be sympathetic to quick-hack usage patterns, but Python is used on both sides of the fence. From ziade.tarek at gmail.com Wed Sep 29 15:26:03 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 29 Sep 2010 15:26:03 +0200 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: <20100929123645.4943.388178770.divmod.xquotient.94@localhost.localdomain> References: <20100929123645.4943.388178770.divmod.xquotient.94@localhost.localdomain> Message-ID: On Wed, Sep 29, 2010 at 2:36 PM, wrote: > On 01:13 am, steve at holdenweb.com wrote: >> >> I see that Atlassian have just taken over BitBucket, the Mercurial >> hosting company. IIRC Atlassian offered to host our issue tracking on >> JIRA, but in the end we decided to eat our own dog food and went with >> roundup. >> >> I'm wondering if they'd be similarly interested in supporting our Hg >> server. Or is self-hosting the only acceptable solution? From recent >> mail it looks likes we may be up and running on Hg fairly soon. > > I know of two medium sized projects (smaller than CPython) that are > switching away from BitBucket's free services because of their poor > reliability. Me too, but I am pretty sure the reliability is going to drastically change in the upcoming months. If Atlassian took over this probably means Bitbucket will have more people to work on the project and some help from the Atlassian Ops. That's really a good news ! Although, I am also -1 because of the hooks management issues that were raised -- Tarek Ziad? | http://ziade.org From mal at egenix.com Wed Sep 29 15:36:46 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 29 Sep 2010 15:36:46 +0200 Subject: [Python-Dev] hg conversion: tags In-Reply-To: References: <20100927112851.1455ea34@mission> <20100927183635.5bbb337b@pitrou.net> <4CA2F8D1.1060704@egenix.com> Message-ID: <4CA340EE.10709@egenix.com> Nick Coghlan wrote: > On Wed, Sep 29, 2010 at 6:29 PM, M.-A. Lemburg wrote: >> I'm not sure whether throwing away history in form of such tags >> is a good idea. >> >> I don't know how hg manages this, but can't we preserve the tag >> information of the tags that you've scheduled to be removed >> in some place that can easily be pulled in but doesn't >> affect the main repo size ? > > But why bother? The tags are static, so grabbing them from svn instead > of hg shouldn't be a big issue. If we had unlimited resources to > support the transition my opinion would probably be different, but > since we don't, applying the simple rule of culling the non-release > tags seems good enough and better than spending too much time trying > to figure out which tags are "important" enough to be worth > preserving. You don't need to spend any extra time on this: just put all the tags that Dirkjan wants to delete into some other place. The separation work has already been done by Dirkjan. Note that the reason for keeping this history around is just that: for historic purposes and possible future research (e.g on copyright issues). It's not meant for development. >> Renaming the release tags certainly is a good idea, since >> we're not stuck with CVS naming requirements anymore. I'd prefix >> the release tags with "release-" for additional context, >> though. > > So long as we don't start using bare numbers for anything other than > releases, I think that would just become redundant typing in fairly > short order. That's a Perl argument ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 29 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From barry at python.org Wed Sep 29 15:40:54 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 29 Sep 2010 09:40:54 -0400 Subject: [Python-Dev] Resource leaks warnings In-Reply-To: <201009292311.45355.steve@pearwood.info> References: <20100808221846.CFD80EEA3F@mail.python.org> <1285764147.3168.10.camel@localhost.localdomain> <201009292311.45355.steve@pearwood.info> Message-ID: <20100929094054.4c200bea@mission> On Sep 29, 2010, at 11:11 PM, Steven D'Aprano wrote: >On Wed, 29 Sep 2010 10:42:27 pm Antoine Pitrou wrote: > >> My assumption is/was that the benefit of warning against leaks in >> real applications (or even - sigh - the standard library) would >> outweigh the inconvenience when hacking together a quick script. >> >> But if it doesn't, what about enabling it with a command-line switch? > >I think the ability to detect such file descriptor leaks would be >valuable, but I'm not sure that it should be running all the time. At >the risk of bike-shedding, is it something which could be controlled >at runtime, like garbage collection? E.g. something like: > >gc.enable_file_warnings() >run_my_tests_for_leakage() >gc.disable_file_warnings() > >or similar. (I'm not wedded to it being in the gc module.) I don't think it should be in the gc module, but I would prefer it be enabled and controlled through a separate module, rather than something Python does automatically for your convenience. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Sep 29 15:55:38 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 29 Sep 2010 09:55:38 -0400 Subject: [Python-Dev] hg conversion: tags In-Reply-To: References: <20100927112851.1455ea34@mission> <20100927183635.5bbb337b@pitrou.net> Message-ID: <20100929095538.7d1bbe4e@mission> On Sep 29, 2010, at 10:15 AM, Dirkjan Ochtman wrote: >Below is a grouped list. Based on that list, I propose the following >scheme: > >- keep all "normal" release tags, renamed along these lines: > r27 -> 2.7 > r266 -> 2.6.6 +1 > r27rc2 -> 2.7rc2 > r262c1 -> 2.6.2rc1 (rc and c should be normalized into the same >thing, don't care which one) Personally, I prefer 'c' to keep things regular. > r30b3 -> 3.0b3 > release22 -> 2.2 +1 >- keep Mac release tags that don't have the top-level Mac dir, renamed >along these lines: > r23b1-mac -> 2.3b1-mac +0 >- get rid of "special" release tags and Mac release tags with >top-level Mac dir >- get rid of email and distutils tags +0. If distutils-sig needs those tags, keep 'em. Similarly with email-sig, but being more involved with the separate email package releases, I don't think they'll be necessary. We're not going to diff against any earlier release and we have an email6 repository in Bazaar on Launchpad. When we integrate that back into Python, it'll likely be an undiffable change so I see no reason to keep the old tags (we can decide later if we'll want new tags in the main Python repo). >- get rid of all other tags +1. We'll always have the Subversion repository for the historical record. It'll make future mining a bit more difficult, but not insurmountably so and I'd rather plan for the benefit of future developers than the convenience of future archaeologists. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From digitalxero at gmail.com Wed Sep 29 16:29:54 2010 From: digitalxero at gmail.com (Dj Gilcrease) Date: Wed, 29 Sep 2010 10:29:54 -0400 Subject: [Python-Dev] hg conversion: tags In-Reply-To: <4CA340EE.10709@egenix.com> References: <20100927112851.1455ea34@mission> <20100927183635.5bbb337b@pitrou.net> <4CA2F8D1.1060704@egenix.com> <4CA340EE.10709@egenix.com> Message-ID: On Wed, Sep 29, 2010 at 9:36 AM, M.-A. Lemburg wrote: > You don't need to spend any extra time on this: just put all the tags > that Dirkjan wants to delete into some other place. The separation > work has already been done by Dirkjan. > > Note that the reason for keeping this history around is just that: > for historic purposes and possible future research (e.g on copyright > issues). It's not meant for development. Can just create a .hgtags_historic that has the full tag list so if someone wants to do historic tag data mining they can just replace the .hgtags to get the historic tags. mm that gives me an idea for a HG plugin to be able to set a tag as historic so it removes it from the .hgtags to .hgtags_historic and some bindings to be able to use historic tags just like normal tags (but ensure it is explicit to look in historic eg --historic) From barry at python.org Wed Sep 29 16:43:46 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 29 Sep 2010 10:43:46 -0400 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: References: Message-ID: <20100929104346.6088beb8@mission> On Sep 28, 2010, at 09:13 PM, Steve Holden wrote: >I see that Atlassian have just taken over BitBucket, the Mercurial >hosting company. IIRC Atlassian offered to host our issue tracking on >JIRA, but in the end we decided to eat our own dog food and went with >roundup. I was an advocate for JIRA at the time, and I think Atlassian is a fine organization with true fans of open source, but we have made a decision to manage our services ourselves, and I think until it's proven that we can't keep up, we should manage our source code repository. It's one of our most valuable assets, if not *the* most valuable asset. The beauty of dvcs of course is that we can have mirrors all over the place. But I certainly feel more comfortable with the official repository on our servers, with our admins in complete control. Do we expect Mercurial to require more, less, or about the same amount of babysitting as the current Subversion repository? I would think no more and Subversion hasn't been much of a problem. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From dirkjan at ochtman.nl Wed Sep 29 17:03:36 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 29 Sep 2010 17:03:36 +0200 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: <20100929104346.6088beb8@mission> References: <20100929104346.6088beb8@mission> Message-ID: On Wed, Sep 29, 2010 at 16:43, Barry Warsaw wrote: > Do we expect Mercurial to require more, less, or about the same amount of > babysitting as the current Subversion repository? ?I would think no more and > Subversion hasn't been much of a problem. Yeah, should be about the same. Cheers, Dirkjan From catch-all at masklinn.net Wed Sep 29 17:09:21 2010 From: catch-all at masklinn.net (Xavier Morel) Date: Wed, 29 Sep 2010 17:09:21 +0200 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: References: <20100929123645.4943.388178770.divmod.xquotient.94@localhost.localdomain> Message-ID: On 2010-09-29, at 15:26 , Tarek Ziad? wrote: > On Wed, Sep 29, 2010 at 2:36 PM, wrote: >> On 01:13 am, steve at holdenweb.com wrote: >>> I see that Atlassian have just taken over BitBucket, the Mercurial >>> hosting company. IIRC Atlassian offered to host our issue tracking on >>> JIRA, but in the end we decided to eat our own dog food and went with >>> roundup. >>> >>> I'm wondering if they'd be similarly interested in supporting our Hg >>> server. Or is self-hosting the only acceptable solution? From recent >>> mail it looks likes we may be up and running on Hg fairly soon. >> >> I know of two medium sized projects (smaller than CPython) that are >> switching away from BitBucket's free services because of their poor >> reliability. > > Me too, but I am pretty sure the reliability is going to drastically > change in the upcoming months. If Atlassian took over this probably > means Bitbucket will have more people to work on the project and some > help from the Atlassian Ops. That's really a good news ! According to Atlassian's announcement of the acquisition, they've already gotten started on that: >> Performance enhancements > Bitbucket's performance has lagged due to poor infrastructure and > lack of IT resources. Recently, Bitbucket customer repositories were > migrated from an EC2 storage system to the Contegix data center, the > same ISV that Atlassian uses for its hosted tools. Atlassian has hired > a full-time IT resource to continue to improve the Bitbucket service > to work on delivering even more services and improvements. It should > be noted that all Bitbucket users are now covered under the Atlassian > Terms of Use. There is heaps more work to do to provide the legendary > service that Atlassian customers have come to expect, and we fully > intend to live up to that promise. >> Feature updates > We've tripled the Bitbucket developer team to ramp up feature > improvements and the frequency of releases. Over the next few months, > users can expect to see more UI improvements, feature enhancements, > and integration with Atlassian's developer stack. Even though the team > has already tripled, we're still hiring (hint hint)! YMMV. From brett at python.org Wed Sep 29 18:31:44 2010 From: brett at python.org (Brett Cannon) Date: Wed, 29 Sep 2010 09:31:44 -0700 Subject: [Python-Dev] Resource leaks warnings In-Reply-To: <1285764147.3168.10.camel@localhost.localdomain> References: <20100808221846.CFD80EEA3F@mail.python.org> <20100929130156.29767af1@pitrou.net> <1285764147.3168.10.camel@localhost.localdomain> Message-ID: On Wed, Sep 29, 2010 at 05:42, Antoine Pitrou wrote: > > Le mercredi 29 septembre 2010 ? 07:27 -0500, Benjamin Peterson a ?crit : >> > >> > I would like to piggy-back on this discussion to suggest further >> > warnings (either by default, or switchable). >> > >> > One feature I've often considered would be to add a warning in FileIO >> > and socket dealloc if these objects haven't been closed explicitly. In >> > most situations, relying on garbage collection to shutdown OS resources >> > (here, file descriptors) is something we like to discourage. >> > Furthermore, it can produce real bugs, especially under Windows when >> > coupled with refererence cycles created by traceback objects (the >> > random test_tarfile failures on the Windows buildbots were a symptom of >> > that; their cause would have been obvious with such warnings). >> > >> > What do you think? >> >> It seems like a slippery slope. Sometimes you really don't care like >> when you're just hacking together a quick script. > > Isn't the "with" statement appropriate in these cases? Yes, which is why I suspect people saying they don't bother have been programming Python for a while and are not in the habit yet of using a 'with' statement. The amount of extra typing compared to inlining a call is minimal. > > My assumption is/was that the benefit of warning against leaks in real > applications (or even - sigh - the standard library) would outweigh the > inconvenience when hacking together a quick script. Does everyone here run all their code under py-debug? If not then I say switch it on when py-debug is on so that we at least detect the leaks in the stdlib without having to think about it. > > But if it doesn't, what about enabling it with a command-line switch? Sure, but I say always turn it on under py-debug. From debatem1 at gmail.com Wed Sep 29 18:56:08 2010 From: debatem1 at gmail.com (geremy condra) Date: Wed, 29 Sep 2010 09:56:08 -0700 Subject: [Python-Dev] Resource leaks warnings In-Reply-To: References: <20100808221846.CFD80EEA3F@mail.python.org> <20100929130156.29767af1@pitrou.net> <1285764147.3168.10.camel@localhost.localdomain> Message-ID: On Wed, Sep 29, 2010 at 9:31 AM, Brett Cannon wrote: > On Wed, Sep 29, 2010 at 05:42, Antoine Pitrou wrote: >> >> Le mercredi 29 septembre 2010 ? 07:27 -0500, Benjamin Peterson a ?crit : >>> > >>> > I would like to piggy-back on this discussion to suggest further >>> > warnings (either by default, or switchable). >>> > >>> > One feature I've often considered would be to add a warning in FileIO >>> > and socket dealloc if these objects haven't been closed explicitly. In >>> > most situations, relying on garbage collection to shutdown OS resources >>> > (here, file descriptors) is something we like to discourage. >>> > Furthermore, it can produce real bugs, especially under Windows when >>> > coupled with refererence cycles created by traceback objects (the >>> > random test_tarfile failures on the Windows buildbots were a symptom of >>> > that; their cause would have been obvious with such warnings). >>> > >>> > What do you think? >>> >>> It seems like a slippery slope. Sometimes you really don't care like >>> when you're just hacking together a quick script. >> >> Isn't the "with" statement appropriate in these cases? > > Yes, which is why I suspect people saying they don't bother have been > programming Python for a while and are not in the habit yet of using a > 'with' statement. The amount of extra typing compared to inlining a > call is minimal. > >> >> My assumption is/was that the benefit of warning against leaks in real >> applications (or even - sigh - the standard library) would outweigh the >> inconvenience when hacking together a quick script. > > Does everyone here run all their code under py-debug? If not then I > say switch it on when py-debug is on so that we at least detect the > leaks in the stdlib without having to think about it. > >> >> But if it doesn't, what about enabling it with a command-line switch? > > Sure, but I say always turn it on under py-debug. This would be a big +1 from me. Geremy Condra From dalcinl at gmail.com Wed Sep 29 19:43:10 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 29 Sep 2010 14:43:10 -0300 Subject: [Python-Dev] r85028 - in python/branches/py3k: Doc/c-api/init.rst Include/pythonrun.h Modules/getpath.c PC/getpathp.c In-Reply-To: References: <20100927053254.DAD4AEE9DC@mail.python.org> Message-ID: On 27 September 2010 11:57, Georg Brandl wrote: > Am 27.09.2010 07:32, schrieb kristjan.jonsson: >> Author: kristjan.jonsson >> Date: Mon Sep 27 07:32:54 2010 >> New Revision: 85028 >> >> Log: >> issue 9910 >> Add a Py_SetPath api to override magic path computations when starting up python. >> >> Modified: >> ? ?python/branches/py3k/Doc/c-api/init.rst >> ? ?python/branches/py3k/Include/pythonrun.h >> ? ?python/branches/py3k/Modules/getpath.c >> ? ?python/branches/py3k/PC/getpathp.c >> >> Modified: python/branches/py3k/Doc/c-api/init.rst > >> +.. cfunction:: ?void Py_SetPath(const wchar_t *) >> + >> + ? .. index:: >> + ? ? ?triple: module; search; path >> + ? ? ?single: path (in module sys) >> + ? ? ?single: Py_GetPath() >> + >> + ? Set the default module search path. ?If this function is called before >> + ? :cfunc: `Py_Initialize` then :cfunc: Py_GetPath won't attempt to compute >> + ? a default serarch path but uses the provided one in stead. ?This is useful >> + ? if Python is being embedded by an application that has full knowledge >> + ? of the location of all modules. ?The path components should be separated >> + ? by semicolons. >> + >> + ? This also causes `sys.executable` to be set only to the raw program name >> + ? (see :cfunc:`Py_SetProgramName`) and `for sys.prefix` and >> + ? `sys.exec_prefix` to be empty. ?It is up to the caller to modify these if >> + ? required after calling :cfunc: `Py_Initialize`. >> + > > This needs a versionadded. > > Georg > Did you noticed " ... The path components should be separated by semicolons." ? I would expect os.path.pathsep, after all paths are not OS-independent. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From guido at python.org Wed Sep 29 20:32:19 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 29 Sep 2010 11:32:19 -0700 Subject: [Python-Dev] We should be using a tool for code reviews Message-ID: I would like to recommend that the Python core developers start using a code review tool such as Rietveld or Reviewboard. I don't really care which tool we use (I'm sure there are plenty of pros and cons to each) but I do think we should get out of the stone age and start using a tool for the majority of our code reviews. While I would personally love to see Rietveld declared the official core Python code review tool, I realize that since I wrote as a Google engineer and it is running on Google infrastructure (App Engine), I can't be fully objective about the tool choice -- even though it is open source, has several non-Googler maintainers, and can be run outside App Engine as well. But I do think that using a specialized code review tool rather than unstructured email plus a general-purpose issue tracker can hugely improve developer performance and also increase community participation. (A code review tool makes it much more convenient for a senior reviewer to impart their wisdom to a junior developer without appearing judgmental or overbearing.) See also this buzz thread: http://www.google.com/buzz/115212051037621986145/At6Rj82Kret/When-will-the-Python-dev-community-start-using -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Wed Sep 29 20:41:25 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 29 Sep 2010 20:41:25 +0200 Subject: [Python-Dev] We should be using a tool for code reviews References: Message-ID: <20100929204125.416e2013@pitrou.net> On Wed, 29 Sep 2010 11:32:19 -0700 Guido van Rossum wrote: > I would like to recommend that the Python core developers start using > a code review tool such as Rietveld or Reviewboard. I don't really > care which tool we use (I'm sure there are plenty of pros and cons to > each) but I do think we should get out of the stone age and start > using a tool for the majority of our code reviews. He, several of us would like it too (although for short patches it doesn't really make a difference), but what's missing is some kind of Roundup integration. Something as trivial as a "start review" button in front of every uploaded patch file would do the trick; it has been suggested several times already, but what's needed is someone to write the code :) Regards Antoine. From brett at python.org Wed Sep 29 20:47:51 2010 From: brett at python.org (Brett Cannon) Date: Wed, 29 Sep 2010 11:47:51 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <20100929204125.416e2013@pitrou.net> References: <20100929204125.416e2013@pitrou.net> Message-ID: On Wed, Sep 29, 2010 at 11:41, Antoine Pitrou wrote: > On Wed, 29 Sep 2010 11:32:19 -0700 > Guido van Rossum wrote: >> I would like to recommend that the Python core developers start using >> a code review tool such as Rietveld or Reviewboard. I don't really >> care which tool we use (I'm sure there are plenty of pros and cons to >> each) but I do think we should get out of the stone age and start >> using a tool for the majority of our code reviews. > > He, several of us would like it too (although for short patches it > doesn't really make a difference), but what's missing is some kind of > Roundup integration. Something as trivial as a "start review" button in > front of every uploaded patch file would do the trick; it has been > suggested several times already, but what's needed is someone to write > the code :) The other option (as discussed on Buzz) is to add Rietveld's upload.py to Misc/ and tell people to use that to submit the patch. Then we simply say to the person submitting the patch, "upload it to Rietveld and paste in the link" or simply require it upfront to encourage people to do the upload in the first place. This would let usage to move forward until we get that "start review" button (wasn't Ezio looking into it?). From guido at python.org Wed Sep 29 20:49:31 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 29 Sep 2010 11:49:31 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <20100929204125.416e2013@pitrou.net> References: <20100929204125.416e2013@pitrou.net> Message-ID: On Wed, Sep 29, 2010 at 11:41 AM, Antoine Pitrou wrote: > On Wed, 29 Sep 2010 11:32:19 -0700 > Guido van Rossum wrote: >> I would like to recommend that the Python core developers start using >> a code review tool such as Rietveld or Reviewboard. I don't really >> care which tool we use (I'm sure there are plenty of pros and cons to >> each) but I do think we should get out of the stone age and start >> using a tool for the majority of our code reviews. > > He, several of us would like it too (although for short patches it > doesn't really make a difference), but what's missing is some kind of > Roundup integration. Something as trivial as a "start review" button in > front of every uploaded patch file would do the trick; it has been > suggested several times already, but what's needed is someone to write > the code :) As I tried to explain in the Buzz thread (not that I require you to read it -- I'll happily repeat myself here): Unfortunately taking the average patch posted to the tracker and importing it in Rietveld is very iffy -- it's very hard to find the right branch+rev needed to be able to apply the patch correctly -- not to mention that there are so many (slightly) different patch formats. It would take a fair bit of AI to get this right. The recommended way to work with Rietveld is to use a checkout (fortunately with Hg this will become easier to do) and use the "upload.py" script that you can download from Rietveld (log in and click on the Create link). Other projects that have adopted Rietveld (Chromum, GWT, Go) require its use for all reviews and have written their own scripts extending upload.py to implement the workflow they like (each one a bit different). They've also created their own branded Rietveld sites (which is easy to do using App Engine). -- --Guido van Rossum (python.org/~guido) From daniel at stutzbachenterprises.com Wed Sep 29 20:50:30 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Wed, 29 Sep 2010 13:50:30 -0500 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <20100929204125.416e2013@pitrou.net> References: <20100929204125.416e2013@pitrou.net> Message-ID: On Wed, Sep 29, 2010 at 1:41 PM, Antoine Pitrou wrote: > He, several of us would like it too (although for short patches it > doesn't really make a difference), but what's missing is some kind of > Roundup integration. Something as trivial as a "start review" button in > front of every uploaded patch file would do the trick; it has been > suggested several times already, but what's needed is someone to write > the code :) > Most patches won't import cleanly into Rietveld, because the patch typical does not contain precise information about the base revision for the patch. How about the opposite approach: make a Python-specific version of upload.py that lets the user attach the patch to an issue with an optional message? -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Sep 29 21:03:15 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 29 Sep 2010 12:03:15 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: On Wed, Sep 29, 2010 at 11:47 AM, Brett Cannon wrote: > On Wed, Sep 29, 2010 at 11:41, Antoine Pitrou wrote: >> On Wed, 29 Sep 2010 11:32:19 -0700 >> Guido van Rossum wrote: >>> I would like to recommend that the Python core developers start using >>> a code review tool such as Rietveld or Reviewboard. I don't really >>> care which tool we use (I'm sure there are plenty of pros and cons to >>> each) but I do think we should get out of the stone age and start >>> using a tool for the majority of our code reviews. >> >> He, several of us would like it too (although for short patches it >> doesn't really make a difference), but what's missing is some kind of >> Roundup integration. Something as trivial as a "start review" button in >> front of every uploaded patch file would do the trick; it has been >> suggested several times already, but what's needed is someone to write >> the code :) > > The other option (as discussed on Buzz) is to add Rietveld's upload.py > to Misc/ A problem with that is that we regularly make matching improvements to upload.py and the server-side code it talks to. While we tend to be conservative in these changes (because we don't control what version of upload.py people use) it would be a pain to maintain backwards compatibility with a version that was distributed in Misc/ two years ago -- that's kind of outside our horizon. Maybe the upload.py script distributed could just download the most current version from codereview.appspot.com/static/upload.py -- that URL is easy enough to keep stable. > and tell people to use that to submit the patch. Then we > simply say to the person submitting the patch, "upload it to Rietveld > and paste in the link" or simply require it upfront to encourage > people to do the upload in the first place. This would let usage to > move forward until we get that "start review" button (wasn't Ezio > looking into it?). Yeah, but it would still not work if they are working in an unpacked tarball -- upload.py requires that you have a VCS checkout of some sort (though it supports SVN, Hg, Git and Bzr). -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Wed Sep 29 21:07:32 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 29 Sep 2010 21:07:32 +0200 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: <20100929210732.185f1e31@pitrou.net> Guido, Brett, On Wed, 29 Sep 2010 11:47:51 -0700 Brett Cannon wrote: > > The other option (as discussed on Buzz) is to add Rietveld's upload.py > to Misc/ and tell people to use that to submit the patch. It sounds like a good option (or, even better, a customized version as suggested by Daniel). > Then we > simply say to the person submitting the patch, "upload it to Rietveld > and paste in the link" or simply require it upfront to encourage > people to do the upload in the first place. We shouldn't require it. Some people don't have a Google account (and/or don't want to have one). Also sometimes posting on Rietveld is overkill (patches less than 30 lines long fall in this category, as far as I'm concerned). > This would let usage to > move forward until we get that "start review" button (wasn't Ezio > looking into it?). Yes, I think Ezio was looking into it, but working on the tracker apparently has a tendency to burn out developers rather quickly :-/ (first Ajaksu, now Ezio) Regards Antoine. From barry at python.org Wed Sep 29 21:16:03 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 29 Sep 2010 15:16:03 -0400 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: Message-ID: <20100929151603.1dd6cee2@mission> On Sep 29, 2010, at 11:32 AM, Guido van Rossum wrote: >I would like to recommend that the Python core developers start using >a code review tool such as Rietveld or Reviewboard. I don't really >care which tool we use (I'm sure there are plenty of pros and cons to >each) but I do think we should get out of the stone age and start >using a tool for the majority of our code reviews. I like and have used Rietveld, both as a submitter of a patch and as a reviewer of someone else's code. It's very nice, and I think we should use it where appropriate, I don't think it should be a requirement. While it will be somewhat better integrated into our normal development processes whenever we move to Mercurial, it won't be seamless. I don't particularly like having to run a separate script (upload.py IIRC) in order to initiate a review and push updates. One thing I really like about Launchpad's merge proposals is that it's very well integrated into normal workflows. Updates against the target branch are automatically tracked in the generated diff and in fact, once a merge proposal has been accepted, it can be automatically landed by a 'bot if you want. Launchpad's merge proposal system doesn't have the really nice web-based ui that Rietveld has, but it is well integrated with an email-based workflow. When I see a merge proposal come into my inbox, with the diff against the target branch, I can just reply with my review inline right there, and those comments are visible to all subscribers. It lowers the barrier to performing the review immensely. Web is nice and should be available, but I really do not want to give up on email-based reviews (well, with Python give up on the possibility of them). Someone else mentioned that it should better integrate with Roundup, and I agree with that. I would much rather we concentrate on converting over to Mercurial as soon as possible, since I think a dvcs will do more to improve our processes than anything else at this point. Please, please, please, let's not let it wait until Pycon 2011 (*2 years* after pronouncement) [1]. -Barry [1] Apologies for sounding critical of any individual - that's not my intent. Dirkjan and folks have done a lot of great work to this point and ISTM that we're *really* close. Let's JFDI and work out remaining kinks as we go! -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From skip at pobox.com Wed Sep 29 21:57:34 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 29 Sep 2010 14:57:34 -0500 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: Message-ID: <19619.39470.822207.635187@montanaro.dyndns.org> Guido> I would like to recommend that the Python core developers start Guido> using a code review tool ... +1 I've suggested that something like Rietveld be integrated with our Roundup instance in the past. I suspect there is an open tracker case. Martin will be better able to find it than me. I became a convert watching the Unladen Swallow folks use it. Skip From chris at subtlety.com Wed Sep 29 22:06:16 2010 From: chris at subtlety.com (Chris Bergstresser) Date: Wed, 29 Sep 2010 16:06:16 -0400 Subject: [Python-Dev] Question on bz2 codec. Is this a bug? Message-ID: Hi all -- I looked through the bug tracker, but I didn't see this listed. I was trying to use the bz2 codec, but it seems like it's not very useful in the current form (and I'm not sure if it's getting added back to py3k, so maybe this is a moot point). It looks like the codec writes every piece of data fed to it as a separate compressed block. This results in compressed files which are significantly larger than the uncompressed files, if you're writing a lot of small bursts of data. It also leads to interesing oddities like this: import codecs with codecs.open('text.bz2', 'w', 'bz2') as f: for x in xrange(20): f.write('This is data %i\n' % x) with codecs.open('text.bz2', 'r', 'bz2') as f: print f.read() This prints "This is data 0" and exits, because the codec won't read beyond the first compressed block. My question is, is this known, intended behavior? Should I open a bug report? Is it going away in py3k, so there's no real point in fixing it? -- Chris From brett at python.org Wed Sep 29 22:12:56 2010 From: brett at python.org (Brett Cannon) Date: Wed, 29 Sep 2010 13:12:56 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: On Wed, Sep 29, 2010 at 12:03, Guido van Rossum wrote: > On Wed, Sep 29, 2010 at 11:47 AM, Brett Cannon wrote: >> On Wed, Sep 29, 2010 at 11:41, Antoine Pitrou wrote: >>> On Wed, 29 Sep 2010 11:32:19 -0700 >>> Guido van Rossum wrote: >>>> I would like to recommend that the Python core developers start using >>>> a code review tool such as Rietveld or Reviewboard. I don't really >>>> care which tool we use (I'm sure there are plenty of pros and cons to >>>> each) but I do think we should get out of the stone age and start >>>> using a tool for the majority of our code reviews. >>> >>> He, several of us would like it too (although for short patches it >>> doesn't really make a difference), but what's missing is some kind of >>> Roundup integration. Something as trivial as a "start review" button in >>> front of every uploaded patch file would do the trick; it has been >>> suggested several times already, but what's needed is someone to write >>> the code :) >> >> The other option (as discussed on Buzz) is to add Rietveld's upload.py >> to Misc/ > > A problem with that is that we regularly make matching improvements to > upload.py and the server-side code it talks to. While we tend to be > conservative in these changes (because we don't control what version > of upload.py people use) it would be a pain to maintain backwards > compatibility with a version that was distributed in Misc/ two years > ago -- that's kind of outside our horizon. Well, I would assume people are working from a checkout. Patches from an outdated checkout simply would fail and that's fine by me. > > Maybe the upload.py script distributed could just download the most > current version from codereview.appspot.com/static/upload.py -- that > URL is easy enough to keep stable. That's fine by me. > >> and tell people to use that to submit the patch. Then we >> simply say to the person submitting the patch, "upload it to Rietveld >> and paste in the link" or simply require it upfront to encourage >> people to do the upload in the first place. This would let usage to >> move forward until we get that "start review" button (wasn't Ezio >> looking into it?). > > Yeah, but it would still not work if they are working in an unpacked > tarball -- upload.py requires that you have a VCS checkout of some > sort (though it supports SVN, Hg, Git and Bzr). How often do we even get patches generated from a downloaded copy of Python? Is it enough to need to worry about this? From guido at python.org Wed Sep 29 22:23:24 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 29 Sep 2010 13:23:24 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: On Wed, Sep 29, 2010 at 1:12 PM, Brett Cannon wrote: > On Wed, Sep 29, 2010 at 12:03, Guido van Rossum wrote: >> A problem with that is that we regularly make matching improvements to >> upload.py and the server-side code it talks to. While we tend to be >> conservative in these changes (because we don't control what version >> of upload.py people use) it would be a pain to maintain backwards >> compatibility with a version that was distributed in Misc/ two years >> ago -- that's kind of outside our horizon. > > Well, I would assume people are working from a checkout. Patches from > an outdated checkout simply would fail and that's fine by me. Ok, but that's an extra barrier for contributions. Lots of people when asked for a patch just modify their distro in place and you can count yourself lucky if they send you a diff from a clean copy. But maybe with Hg it's less of a burden to ask people to use a checkout. > How often do we even get patches generated from a downloaded copy of > Python? Is it enough to need to worry about this? I used to get these frequently. I don't know what the experience of the current crop of core developers is though, so maybe my gut feelings here are outdated. -- --Guido van Rossum (python.org/~guido) From alexander.belopolsky at gmail.com Wed Sep 29 22:31:38 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 29 Sep 2010 16:31:38 -0400 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: On Wed, Sep 29, 2010 at 4:23 PM, Guido van Rossum wrote: .. > But maybe with Hg it's less of a burden to ask people to use a checkout. > I thought with Hg it would be more of a burden for casual contributors to use a checkout simply because the checkout is many times bigger. From solipsis at pitrou.net Wed Sep 29 22:32:43 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 29 Sep 2010 22:32:43 +0200 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: <1285792363.3194.41.camel@localhost.localdomain> > > Well, I would assume people are working from a checkout. Patches from > > an outdated checkout simply would fail and that's fine by me. > > Ok, but that's an extra barrier for contributions. Lots of people when > asked for a patch just modify their distro in place and you can count > yourself lucky if they send you a diff from a clean copy. Well, either they're doing a small patch and uploading to Rietveld doesn't bring much anyway; or they're doing a large patch and it may not apply cleanly if they do it against the stable release. So I don't think there's a real problem here. > I used to get these frequently. I don't know what the experience of > the current crop of core developers is though, so maybe my gut > feelings here are outdated. Most patches posted on the tracker are generated either by "SVN diff" or a DCVS equivalent (often hg). Regards Antoine. From guido at python.org Wed Sep 29 22:33:56 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 29 Sep 2010 13:33:56 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: On Wed, Sep 29, 2010 at 1:31 PM, Alexander Belopolsky wrote: > On Wed, Sep 29, 2010 at 4:23 PM, Guido van Rossum wrote: > .. >> But maybe with Hg it's less of a burden to ask people to use a checkout. >> > > I thought with Hg it would be more of a burden for casual contributors > to use a checkout simply because the checkout is many times bigger. Isn't it still faster though? -- --Guido van Rossum (python.org/~guido) From brett at python.org Wed Sep 29 22:35:45 2010 From: brett at python.org (Brett Cannon) Date: Wed, 29 Sep 2010 13:35:45 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: On Wed, Sep 29, 2010 at 13:23, Guido van Rossum wrote: > On Wed, Sep 29, 2010 at 1:12 PM, Brett Cannon wrote: >> On Wed, Sep 29, 2010 at 12:03, Guido van Rossum wrote: >>> A problem with that is that we regularly make matching improvements to >>> upload.py and the server-side code it talks to. While we tend to be >>> conservative in these changes (because we don't control what version >>> of upload.py people use) it would be a pain to maintain backwards >>> compatibility with a version that was distributed in Misc/ two years >>> ago -- that's kind of outside our horizon. >> >> Well, I would assume people are working from a checkout. Patches from >> an outdated checkout simply would fail and that's fine by me. > > Ok, but that's an extra barrier for contributions. Lots of people when > asked for a patch just modify their distro in place and you can count > yourself lucky if they send you a diff from a clean copy. > > But maybe with Hg it's less of a burden to ask people to use a checkout. > >> How often do we even get patches generated from a downloaded copy of >> Python? Is it enough to need to worry about this? > > I used to get these frequently. I don't know what the experience of > the current crop of core developers is though, so maybe my gut > feelings here are outdated. Well, we can start with strongly worded suggestions that patches submitted using Rietveld will typically get priority over patches submitted just to the issue tracker and that this means doing it from a checkout. From there we can see if there is a drop in submissions. From solipsis at pitrou.net Wed Sep 29 22:43:03 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 29 Sep 2010 22:43:03 +0200 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: <1285792983.3194.49.camel@localhost.localdomain> Le mercredi 29 septembre 2010 ? 13:35 -0700, Brett Cannon a ?crit : > > Well, we can start with strongly worded suggestions that patches > submitted using Rietveld will typically get priority over patches > submitted just to the issue tracker and that this means doing it from > a checkout. But will we (all of us) actually follow this rule? Granted, a patch is reviewed faster if it's easier to review. But in many cases (small patches) it doesn't really make a difference. I have from time to time suggested that a contributor post his/her patch to Rietveld. But that was for really large or nasty ones. Regards Antoine. From brett at python.org Wed Sep 29 22:56:46 2010 From: brett at python.org (Brett Cannon) Date: Wed, 29 Sep 2010 13:56:46 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: On Wed, Sep 29, 2010 at 13:31, Alexander Belopolsky wrote: > On Wed, Sep 29, 2010 at 4:23 PM, Guido van Rossum wrote: > .. >> But maybe with Hg it's less of a burden to ask people to use a checkout. >> > > I thought with Hg it would be more of a burden for casual contributors > to use a checkout simply because the checkout is many times bigger. Many times bigger than what? If you mean svn that's not true (the eval of the DVCS pegged Hg at only 50% larger than svn). If you mean compared to the tar.bz2 file, then sure, but a zip of an Hg checkout is under 50 MB which is less than 5x larger. But honestly, the line has to be drawn somewhere. Right now we won't take a patch submitted to the mailing list, but accepting patches not against a checkout wastes time for everyone involved as soon as it doesn't apply cleanly. At that point either a core developer has to fix it or (what typically happens) the person has to get a checkout, update their patch, and then re-submit. We might as well minimize that when possible by really pushing for checkout patches. From guido at python.org Wed Sep 29 22:58:45 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 29 Sep 2010 13:58:45 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <1285792983.3194.49.camel@localhost.localdomain> References: <20100929204125.416e2013@pitrou.net> <1285792983.3194.49.camel@localhost.localdomain> Message-ID: On Wed, Sep 29, 2010 at 1:43 PM, Antoine Pitrou wrote: > Le mercredi 29 septembre 2010 ? 13:35 -0700, Brett Cannon a ?crit : >> >> Well, we can start with strongly worded suggestions that patches >> submitted using Rietveld will typically get priority over patches >> submitted just to the issue tracker and that this means doing it from >> a checkout. > > But will we (all of us) actually follow this rule? > Granted, a patch is reviewed faster if it's easier to review. But in > many cases (small patches) it doesn't really make a difference. > > I have from time to time suggested that a contributor post his/her patch > to Rietveld. But that was for really large or nasty ones. More use of the tool also increases accountability and provides more opportunities for junior developers to learn. (And it increases familiarity of all involved with the tool for the future.) I agree it shouldn't be mandatory, but I would suggest you give it a try even for small changes. -- --Guido van Rossum (python.org/~guido) From p.f.moore at gmail.com Wed Sep 29 22:59:35 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 29 Sep 2010 21:59:35 +0100 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: On 29 September 2010 21:12, Brett Cannon wrote: > How often do we even get patches generated from a downloaded copy of > Python? Is it enough to need to worry about this? When I do simple bugfixes of library code, I'll often work from my "live" Python environment, patch it in place, test and generate a patch with diff. Very primitive, but surprisingly effective. Having said that, I don't think it's the end of the world if such patches couldn't use Reitveld. They are probably small enough that it would be overkill anyway. Paul From brett at python.org Wed Sep 29 22:59:58 2010 From: brett at python.org (Brett Cannon) Date: Wed, 29 Sep 2010 13:59:58 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <1285792983.3194.49.camel@localhost.localdomain> References: <20100929204125.416e2013@pitrou.net> <1285792983.3194.49.camel@localhost.localdomain> Message-ID: On Wed, Sep 29, 2010 at 13:43, Antoine Pitrou wrote: > Le mercredi 29 septembre 2010 ? 13:35 -0700, Brett Cannon a ?crit : >> >> Well, we can start with strongly worded suggestions that patches >> submitted using Rietveld will typically get priority over patches >> submitted just to the issue tracker and that this means doing it from >> a checkout. > > But will we (all of us) actually follow this rule? Does it matter? If it leads to an easier review process in some cases and doesn't impact reviews overall in a negative way who cares. > Granted, a patch is reviewed faster if it's easier to review. But in > many cases (small patches) it doesn't really make a difference. So it's a no-op in some cases, a benefit in others. I don't see the harm. > > I have from time to time suggested that a contributor post his/her patch > to Rietveld. But that was for really large or nasty ones. Same here, but that was because I didn't have an easy way to say "submit the patch to Rietveld and add the link to the issue". If we have it in a checkout we can easily say "please run code_review.py from your checkout and paste the link into the issue to move the review forward." From mal at egenix.com Wed Sep 29 23:05:38 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 29 Sep 2010 23:05:38 +0200 Subject: [Python-Dev] Question on bz2 codec. Is this a bug? In-Reply-To: References: Message-ID: <4CA3AA22.50304@egenix.com> Chris Bergstresser wrote: > Hi all -- > > I looked through the bug tracker, but I didn't see this listed. I > was trying to use the bz2 codec, but it seems like it's not very > useful in the current form (and I'm not sure if it's getting added > back to py3k, so maybe this is a moot point). It looks like the codec > writes every piece of data fed to it as a separate compressed block. > This results in compressed files which are significantly larger than > the uncompressed files, if you're writing a lot of small bursts of > data. It also leads to interesing oddities like this: > > import codecs > > with codecs.open('text.bz2', 'w', 'bz2') as f: > for x in xrange(20): > f.write('This is data %i\n' % x) > > with codecs.open('text.bz2', 'r', 'bz2') as f: > print f.read() > > This prints "This is data 0" and exits, because the codec won't read > beyond the first compressed block. > > My question is, is this known, intended behavior? Should I open a bug > report? Is it going away in py3k, so there's no real point in fixing > it? The codec is scheduled to be added back to Python3. However, it's main use is in working on whole chunks of data rather than the line-by-line approach you're after. This is provided by the codec's incremental encoder/decoders, but these are currently not used by codecs.open() and I'm not sure whether the io lib uses them, which could be used via the regular open(). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 29 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From robert.kern at gmail.com Wed Sep 29 23:11:36 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 29 Sep 2010 16:11:36 -0500 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: On 9/29/10 3:33 PM, Guido van Rossum wrote: > On Wed, Sep 29, 2010 at 1:31 PM, Alexander Belopolsky > wrote: >> On Wed, Sep 29, 2010 at 4:23 PM, Guido van Rossum wrote: >> .. >>> But maybe with Hg it's less of a burden to ask people to use a checkout. >> >> I thought with Hg it would be more of a burden for casual contributors >> to use a checkout simply because the checkout is many times bigger. > > Isn't it still faster though? For what it's worth, when I tried it, the relationship is reversed: [hg]$ time hg clone http://code.python.org/hg/branches/py3k/ ... hg clone http://code.python.org/hg/branches/py3k/ 24.44s user 3.43s system 10% cpu 4:15.48 total [hg]$ du -hsc py3k 105M py3k [svn]$ time svn co http://svn.python.org/projects/python/branches/py3k/ ... svn co http://svn.python.org/projects/python/branches/py3k/ 5.03s user 7.01s system 12% cpu 1:35.97 total [svn]$ du -hsc py3k 131M py3k Mercurial's checkout with the whole history is actually smaller than the SVN checkout because it applies some very nice compression to the history whereas the SVN checkout has an uncompressed copy of every single file tucked away in its .svn/ directory. My mercurial checkout happens to be slower, but I don't know whose fault that is. I'm still using Mercurial 1.5.1 on my OS X box, and while I wasn't doing much that would take up CPU or bandwidth, I wasn't trying especially hard to prevent such interference, either. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From solipsis at pitrou.net Wed Sep 29 23:21:16 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 29 Sep 2010 23:21:16 +0200 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: <20100929232116.487be5ae@pitrou.net> On Wed, 29 Sep 2010 13:56:46 -0700 Brett Cannon wrote: > On Wed, Sep 29, 2010 at 13:31, Alexander Belopolsky > wrote: > > On Wed, Sep 29, 2010 at 4:23 PM, Guido van Rossum wrote: > > .. > >> But maybe with Hg it's less of a burden to ask people to use a checkout. > >> > > > > I thought with Hg it would be more of a burden for casual contributors > > to use a checkout simply because the checkout is many times bigger. > > Many times bigger than what? If you mean svn that's not true (the eval > of the DVCS pegged Hg at only 50% larger than svn). If you mean > compared to the tar.bz2 file, then sure, but a zip of an Hg checkout > is under 50 MB which is less than 5x larger. The disk footprint is not very much larger, but I assume that the amount of bandwidth needed for a first checkout can easily be 10x or 50x. (then, if you keep the checkout around, pulling new revisions is very efficient) That said, it seems there are tools to optimize the internal structures of an hg repository so that it takes less disk space (and therefore less checkout bandwidth). I don't know whether Dirkjan intends to use that. Regards Antoine. From alexander.belopolsky at gmail.com Wed Sep 29 23:22:43 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 29 Sep 2010 17:22:43 -0400 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: On Wed, Sep 29, 2010 at 4:56 PM, Brett Cannon wrote: > On Wed, Sep 29, 2010 at 13:31, Alexander Belopolsky > wrote: >> On Wed, Sep 29, 2010 at 4:23 PM, Guido van Rossum wrote: >> .. >>> But maybe with Hg it's less of a burden to ask people to use a checkout. >>> >> >> I thought with Hg it would be more of a burden for casual contributors >> to use a checkout simply because the checkout is many times bigger. > > Many times bigger than what? If you mean svn that's not true (the eval > of the DVCS pegged Hg at only 50% larger than svn). My experience was different. I may misremember because I did not try to use Hg since about a year ago, but the Hg checkout was 3-4x of that of an SVN branch. However, my comment was simply a reaction to the argument that Hg checkout would be *less* of a burden than SVN. > > But honestly, the line has to be drawn somewhere. Right now we won't > take a patch submitted to the mailing list, but accepting patches not > against a checkout wastes time for everyone involved as soon as it > doesn't apply cleanly. At that point either a core developer has to > fix it or (what typically happens) the person has to get a checkout, > update their patch, and then re-submit. We might as well minimize that > when possible by really pushing for checkout patches. > Well, many patches do not apply cleanly by the time they are reviewed even when they are originally produced from a clean checkout. I actually wonder if it would be appropriate to encourage *reviewers* to upload the patches to a review tool. A reviewer is much more likely to have a clean checkout already than a casual contributor and oftentimes applying the patch is the first thing reviewers do anyways. If a review tool is one command away, it may be even easier to run it rather than to figure out how to reference the code in the patch in the roundup comment. From solipsis at pitrou.net Wed Sep 29 23:23:51 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 29 Sep 2010 23:23:51 +0200 Subject: [Python-Dev] Question on bz2 codec. Is this a bug? References: <4CA3AA22.50304@egenix.com> Message-ID: <20100929232351.3b9fce6d@pitrou.net> On Wed, 29 Sep 2010 23:05:38 +0200 "M.-A. Lemburg" wrote: > > The codec is scheduled to be added back to Python3. > > However, it's main use is in working on whole chunks of > data rather than the line-by-line approach you're after. > This is provided by the codec's incremental encoder/decoders, > but these are currently not used by codecs.open() and > I'm not sure whether the io lib uses them, which could > be used via the regular open(). The io lib will not work with a codec which doesn't return unicode strings. Anyway, the obvious way to write line-by-line to a bz2 file is to use the BZ2File class! Regards Antoine. From martin at v.loewis.de Wed Sep 29 23:25:42 2010 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 29 Sep 2010 23:25:42 +0200 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: <20100929104346.6088beb8@mission> References: <20100929104346.6088beb8@mission> Message-ID: <4CA3AED6.2050003@v.loewis.de> > Do we expect Mercurial to require more, less, or about the same amount of > babysitting as the current Subversion repository? The ongoing effort is to manage write access; this is not going to change with Mercurial. With a hosted service, you still need someone who gives write permissions to people. What you gain is not having to deal with lost credentials (forgotten passwords, new SSH keys). We *could* of course setup a web-based user management on python.org as well so committers can upload their SSH keys over the web, however, the current approach has not produced much problems. For Mercurial (or any other VCS), there is also an upfront effort to setup the system, which is primarily data conversion and hooks. Expect increased activity in creating hooks for one year after the switchover. Of course, with a hosted service, you often can't run hooks at all, so the effort to write them is also reduced :-) Regards, Martin From ncoghlan at gmail.com Wed Sep 29 23:25:54 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Sep 2010 07:25:54 +1000 Subject: [Python-Dev] Resource leaks warnings In-Reply-To: <20100929094054.4c200bea@mission> References: <20100808221846.CFD80EEA3F@mail.python.org> <1285764147.3168.10.camel@localhost.localdomain> <201009292311.45355.steve@pearwood.info> <20100929094054.4c200bea@mission> Message-ID: On Wed, Sep 29, 2010 at 11:40 PM, Barry Warsaw wrote: > I don't think it should be in the gc module, but I would prefer it be enabled > and controlled through a separate module, rather than something Python does > automatically for your convenience. The os module would seem to be the place to enable/disable tracking of OS level resource leaks (i.e. file descriptors and possible HANDLES on Windows). I'm not sure how practical this idea will prove to implement though. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Sep 29 23:28:48 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Sep 2010 07:28:48 +1000 Subject: [Python-Dev] r85028 - in python/branches/py3k: Doc/c-api/init.rst Include/pythonrun.h Modules/getpath.c PC/getpathp.c In-Reply-To: References: <20100927053254.DAD4AEE9DC@mail.python.org> Message-ID: On Thu, Sep 30, 2010 at 3:43 AM, Lisandro Dalcin wrote: > Did you noticed " ... The path components should be separated by > semicolons." ? I would expect os.path.pathsep, after all paths are not > OS-independent. That's true when reading from PYTHONPATH. For a programmatic API like this, being consistent is probably simpler in most cases. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From barry at python.org Wed Sep 29 23:30:10 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 29 Sep 2010 17:30:10 -0400 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <1285792363.3194.41.camel@localhost.localdomain> References: <20100929204125.416e2013@pitrou.net> <1285792363.3194.41.camel@localhost.localdomain> Message-ID: <20100929173010.2acf0b3d@mission> One other thought: IME patches in general are suboptimal to branches, so I think we should be encouraging people to publish their branches publicly for review. A diff is a decent way to get feedback about code changes, but that's often only part of the work involved in deciding whether a change should be accepted or not. A reviewer often wants to do a build with the changes, test them on various tasks and application, run the test suite, etc. For this, "merge" is much better than patch(1). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From martin at v.loewis.de Wed Sep 29 23:35:02 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 29 Sep 2010 23:35:02 +0200 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: Message-ID: <4CA3B106.30406@v.loewis.de> > While I would personally love to see Rietveld declared the official > core Python code review tool, I realize that since I wrote as a Google > engineer and it is running on Google infrastructure (App Engine), I > can't be fully objective about the tool choice -- even though it is > open source, has several non-Googler maintainers, and can be run > outside App Engine as well. I considered having roundup post all patches to Rietveld (trust me that I can solve the "what branch and revision problem"). However, the stumbling block is access control - I don't know how to get roundup to post to Rietveld, "anonymously", so to speak. So perhaps we should just run our own Rietveld instance next to Roundup. Regards, Martin From chris at subtlety.com Wed Sep 29 23:41:12 2010 From: chris at subtlety.com (Chris Bergstresser) Date: Wed, 29 Sep 2010 17:41:12 -0400 Subject: [Python-Dev] Question on bz2 codec. Is this a bug? In-Reply-To: <20100929232351.3b9fce6d@pitrou.net> References: <4CA3AA22.50304@egenix.com> <20100929232351.3b9fce6d@pitrou.net> Message-ID: On Wed, Sep 29, 2010 at 5:23 PM, Antoine Pitrou wrote: > Anyway, the obvious way to write line-by-line to a bz2 file is to use > the BZ2File class! The BZ2File class does not allow you to open a file for appending. Using the incremental encoder does work, which leads to the obvious question of why the codecs.open() method doesn't use the incremental method by default, at least in this case. -- Chris From daniel at stutzbachenterprises.com Wed Sep 29 23:41:52 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Wed, 29 Sep 2010 16:41:52 -0500 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: <4CA3AED6.2050003@v.loewis.de> References: <20100929104346.6088beb8@mission> <4CA3AED6.2050003@v.loewis.de> Message-ID: On Wed, Sep 29, 2010 at 4:25 PM, "Martin v. L?wis" wrote: > Of course, with a hosted service, you often can't run hooks at all, > so the effort to write them is also reduced :-) > It should be easy to write an automated script that pulls the latest changes from the hosted service and then runs hooks. Obviously, it would not be possible to write hooks that reject changesets, but it would be possible to write hooks that send email or notify buildbots. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Wed Sep 29 23:44:36 2010 From: eric at trueblade.com (Eric Smith) Date: Wed, 29 Sep 2010 17:44:36 -0400 Subject: [Python-Dev] [Python-checkins] r85101 - in python/branches/py3k/Doc/library: http.client.rst urllib.request.rst In-Reply-To: <20100929112423.4EFA7EE9AB@mail.python.org> References: <20100929112423.4EFA7EE9AB@mail.python.org> Message-ID: <4CA3B344.1040103@trueblade.com> On 9/29/2010 7:24 AM, antoine.pitrou wrote: > Modified: python/branches/py3k/Doc/library/urllib.request.rst > ============================================================================== > --- python/branches/py3k/Doc/library/urllib.request.rst (original) > +++ python/branches/py3k/Doc/library/urllib.request.rst Wed Sep 29 13:24:21 2010 > @@ -11,6 +11,10 @@ > opening URLs (mostly HTTP) in a complex world --- basic and digest > authentication, redirections, cookies and more. > > +.. warning:: When opening HTTPS (or FTPS) URLs, it is not attempted to > + validate the server certificate. Use at your own risk! > + > + > The :mod:`urllib.request` module defines the following functions: That wording is a little awkward. How about: "When opening HTTPS (or FTPS) URLs, no attempt is made to validate the server certificate. Use at your own risk." From ncoghlan at gmail.com Wed Sep 29 23:45:52 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Sep 2010 07:45:52 +1000 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: On Thu, Sep 30, 2010 at 4:47 AM, Brett Cannon wrote: > On Wed, Sep 29, 2010 at 11:41, Antoine Pitrou wrote: >> On Wed, 29 Sep 2010 11:32:19 -0700 >> Guido van Rossum wrote: >>> I would like to recommend that the Python core developers start using >>> a code review tool such as Rietveld or Reviewboard. I don't really >>> care which tool we use (I'm sure there are plenty of pros and cons to >>> each) but I do think we should get out of the stone age and start >>> using a tool for the majority of our code reviews. Somewhat amusing to get to this thread a few minutes after creating a Reitveld issue for the first pass of my urllib.parse patch :) >> He, several of us would like it too (although for short patches it >> doesn't really make a difference), but what's missing is some kind of >> Roundup integration. Something as trivial as a "start review" button in >> front of every uploaded patch file would do the trick; it has been >> suggested several times already, but what's needed is someone to write >> the code :) > > The other option (as discussed on Buzz) is to add Rietveld's upload.py > to Misc/ and tell people to use that to submit the patch. Then we > simply say to the person submitting the patch, "upload it to Rietveld > and paste in the link" or simply require it upfront to encourage > people to do the upload in the first place. This would let usage to > move forward until we get that "start review" button (wasn't Ezio > looking into it?). Make our script a "submit_patch.py" wrapper around the vanilla "upload.py" rather than a replacement of it and it sounds like a good idea to me. I'd want to be able to do a signature check on upload.py before we enabled runtime downloading of new versions from the codereview site though (in the meantime, a vetted version checked into SVN would do the trick). Even with Roundup integration, being able to create/update the Reitveld issue and make the appropriate tracker updates with a single command would still be handy. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From bcannon at gmail.com Wed Sep 29 23:46:57 2010 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 29 Sep 2010 14:46:57 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <4CA3B106.30406@v.loewis.de> References: <4CA3B106.30406@v.loewis.de> Message-ID: On Wed, Sep 29, 2010 at 14:35, "Martin v. L?wis" wrote: >> While I would personally love to see Rietveld declared the official >> core Python code review tool, I realize that since I wrote as a Google >> engineer and it is running on Google infrastructure (App Engine), I >> can't be fully objective about the tool choice -- even though it is >> open source, has several non-Googler maintainers, and can be run >> outside App Engine as well. > > I considered having roundup post all patches to Rietveld (trust me > that I can solve the "what branch and revision problem"). > > However, the stumbling block is access control - I don't know how > to get roundup to post to Rietveld, "anonymously", so to speak. > > So perhaps we should just run our own Rietveld instance next to Roundup. That shouldn't be too hard. Someone just has to create an App Engine project and handle the deployment. I guess the trickiest part is making sure enough people have admin access so multiple people can update the website, especially if we run a modified copy so that bugs.python.org can push "anonymous" (and probably be the only thing that can). From barry at python.org Wed Sep 29 23:52:25 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 29 Sep 2010 17:52:25 -0400 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: <20100929175225.41923d02@mission> On Sep 29, 2010, at 05:22 PM, Alexander Belopolsky wrote: >> Many times bigger than what? If you mean svn that's not true (the >> eval of the DVCS pegged Hg at only 50% larger than svn). > >My experience was different. I may misremember because I did not try >to use Hg since about a year ago, but the Hg checkout was 3-4x of that >of an SVN branch. However, my comment was simply a reaction to the >argument that Hg checkout would be *less* of a burden than SVN. With Bazaar, if you're a frequent contributor to a project, you can amortize the expensive initial checkout across all the branches you'll ever make. The first branch may take a while, but subsequent ones are very fast. I'd be surprised if Mercurial didn't have at least something similar. It's true though that for drive-by contributors who rarely develop a patch, generating a branch may be too high a barrier. In that case, I think it's fine to have diffs (which a more experienced developer can turn into a branch). >> But honestly, the line has to be drawn somewhere. Right now we won't >> take a patch submitted to the mailing list, but accepting patches not >> against a checkout wastes time for everyone involved as soon as it >> doesn't apply cleanly. At that point either a core developer has to >> fix it or (what typically happens) the person has to get a checkout, >> update their patch, and then re-submit. We might as well minimize >> that when possible by really pushing for checkout patches. >> > >Well, many patches do not apply cleanly by the time they are reviewed >even when they are originally produced from a clean checkout. I >actually wonder if it would be appropriate to encourage *reviewers* to >upload the patches to a review tool. A reviewer is much more likely >to have a clean checkout already than a casual contributor and >oftentimes applying the patch is the first thing reviewers do anyways. > If a review tool is one command away, it may be even easier to run it >rather than to figure out how to reference the code in the patch in >the roundup comment. While branches are far better than patches here, you can still have conflicts when merging the branch to the trunk (which I recommend doing when reviewing a branch). I have no problem halting a review right there and asking the developer to ensure a conflict-free merge. It's of course at the reviewer's discretion to assist them with this (e.g. by branching their branch, resolving the conflicts, committing and publishing this clean branch, for the original developer to merge into *their* branch before requesting a re-review -- see why branches are so much better?! :). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From martin at v.loewis.de Wed Sep 29 23:58:05 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 29 Sep 2010 23:58:05 +0200 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <4CA3B106.30406@v.loewis.de> Message-ID: <4CA3B66D.5070209@v.loewis.de> > That shouldn't be too hard. Someone just has to create an App Engine > project and handle the deployment. I guess the trickiest part is > making sure enough people have admin access so multiple people can > update the website, especially if we run a modified copy so that > bugs.python.org can push "anonymous" (and probably be the only thing > that can). I was hoping that I can run Rietveld on the same machine as roundup, with the added advantage that people can use the same logins, and even the same cookies. Of course, we could do an OpenID-style indirect communication, referring people to Roundup when they need to be authenticated to Rietveld (in particular when commenting). Regards, Martin From ncoghlan at gmail.com Wed Sep 29 23:57:48 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Sep 2010 07:57:48 +1000 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <4CA3B106.30406@v.loewis.de> References: <4CA3B106.30406@v.loewis.de> Message-ID: On Thu, Sep 30, 2010 at 7:35 AM, "Martin v. L?wis" wrote: >> While I would personally love to see Rietveld declared the official >> core Python code review tool, I realize that since I wrote as a Google >> engineer and it is running on Google infrastructure (App Engine), I >> can't be fully objective about the tool choice -- even though it is >> open source, has several non-Googler maintainers, and can be run >> outside App Engine as well. > > I considered having roundup post all patches to Rietveld (trust me > that I can solve the "what branch and revision problem"). > > However, the stumbling block is access control - I don't know how > to get roundup to post to Rietveld, "anonymously", so to speak. > > So perhaps we should just run our own Rietveld instance next to Roundup. Would it be possible to sync up the reitveld issue numbers with the roundup ones if you did that? Or would the fact that a single issue can have multiple attached patches prevent that? Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Wed Sep 29 23:59:00 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 29 Sep 2010 23:59:00 +0200 Subject: [Python-Dev] Question on bz2 codec. Is this a bug? In-Reply-To: References: <4CA3AA22.50304@egenix.com> <20100929232351.3b9fce6d@pitrou.net> Message-ID: <1285797540.3194.52.camel@localhost.localdomain> Le mercredi 29 septembre 2010 ? 17:41 -0400, Chris Bergstresser a ?crit : > On Wed, Sep 29, 2010 at 5:23 PM, Antoine Pitrou wrote: > > Anyway, the obvious way to write line-by-line to a bz2 file is to use > > the BZ2File class! > > The BZ2File class does not allow you to open a file for appending. > Using the incremental encoder does work, In what sense? Do you mean it adds a new bz2 stream at the end of the existing file? From benjamin at python.org Thu Sep 30 00:01:51 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 29 Sep 2010 17:01:51 -0500 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: 2010/9/29 Guido van Rossum : > On Wed, Sep 29, 2010 at 1:12 PM, Brett Cannon wrote: >> How often do we even get patches generated from a downloaded copy of >> Python? Is it enough to need to worry about this? > > I used to get these frequently. I don't know what the experience of > the current crop of core developers is though, so maybe my gut > feelings here are outdated. FWIW, I usually don't apply patches until the last stages of review (namely I'm going to apply it). So, it's a little annoying to download and apply a patch just to run upload.py. I'd prefer a pure web interface for starting reviews with a code review tool. -- Regards, Benjamin From martin at v.loewis.de Thu Sep 30 00:02:09 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 30 Sep 2010 00:02:09 +0200 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <4CA3B106.30406@v.loewis.de> Message-ID: <4CA3B761.2060905@v.loewis.de> >> So perhaps we should just run our own Rietveld instance next to Roundup. > > Would it be possible to sync up the reitveld issue numbers with the > roundup ones if you did that? Most certainly. However, this works fairly well today already. If you put [issueNNNN] into the Rietveld subject, it can already synchronize with roundup (even though it would use different issue numbers). For a local copy of Rietveld, I could certainly arrange it to use the very same numbers. > Or would the fact that a single issue > can have multiple attached patches prevent that? No, Rietveld can deal with multiple patch sets fine (not sure how it fares when they are unrelated patches, since it also computes deltas between subsets - but apparently only for the same files). Regards, Martin From solipsis at pitrou.net Thu Sep 30 00:04:20 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 30 Sep 2010 00:04:20 +0200 Subject: [Python-Dev] We should be using a tool for code reviews References: <20100929204125.416e2013@pitrou.net> <1285792363.3194.41.camel@localhost.localdomain> <20100929173010.2acf0b3d@mission> Message-ID: <20100930000420.3b7d4999@pitrou.net> On Wed, 29 Sep 2010 17:30:10 -0400 Barry Warsaw wrote: > One other thought: IME patches in general are suboptimal to branches, so I > think we should be encouraging people to publish their branches publicly for > review. A diff is a decent way to get feedback about code changes, but that's > often only part of the work involved in deciding whether a change should be > accepted or not. A reviewer often wants to do a build with the changes, test > them on various tasks and application, run the test suite, etc. For this, > "merge" is much better than patch(1). When I review a patch, I will tend to assume that the poster has already run the test suite (especially if it's another committer). Having to checkout a branch and generate a diff myself would make it much longer to produce a review, in most cases. Even rebuilding a new branch from scratch can be slower than applying the diff in an existing checkout and letting `make` rebuild a couple of files. Regards Antoine. From bcannon at gmail.com Thu Sep 30 00:11:50 2010 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 29 Sep 2010 15:11:50 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <4CA3B66D.5070209@v.loewis.de> References: <4CA3B106.30406@v.loewis.de> <4CA3B66D.5070209@v.loewis.de> Message-ID: On Wed, Sep 29, 2010 at 14:58, "Martin v. L?wis" wrote: >> That shouldn't be too hard. Someone just has to create an App Engine >> project and handle the deployment. I guess the trickiest part is >> making sure enough people have admin access so multiple people can >> update the website, especially if we run a modified copy so that >> bugs.python.org can push "anonymous" (and probably be the only thing >> that can). > > I was hoping that I can run Rietveld on the same machine as roundup, > with the added advantage that people can use the same logins, and > even the same cookies. If it can work that sure then great. I just don't know what it takes to run Rietveld sans App Engine. -Brett > > Of course, we could do an OpenID-style indirect communication, > referring people to Roundup when they need to be authenticated to > Rietveld (in particular when commenting). > > Regards, > Martin > From solipsis at pitrou.net Thu Sep 30 00:12:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 30 Sep 2010 00:12:33 +0200 Subject: [Python-Dev] We should be using a tool for code reviews References: <4CA3B106.30406@v.loewis.de> <4CA3B66D.5070209@v.loewis.de> Message-ID: <20100930001233.331484b4@pitrou.net> On Wed, 29 Sep 2010 23:58:05 +0200 "Martin v. L?wis" wrote: > > That shouldn't be too hard. Someone just has to create an App Engine > > project and handle the deployment. I guess the trickiest part is > > making sure enough people have admin access so multiple people can > > update the website, especially if we run a modified copy so that > > bugs.python.org can push "anonymous" (and probably be the only thing > > that can). > > I was hoping that I can run Rietveld on the same machine as roundup, > with the added advantage that people can use the same logins, and > even the same cookies. That would be very nice. Aren't there some App Engine emulation layers? Or are they only for development? From fdrake at acm.org Thu Sep 30 00:12:57 2010 From: fdrake at acm.org (Fred Drake) Date: Wed, 29 Sep 2010 18:12:57 -0400 Subject: [Python-Dev] Atlassian and bitbucket merge In-Reply-To: References: <20100929104346.6088beb8@mission> <4CA3AED6.2050003@v.loewis.de> Message-ID: On Wed, Sep 29, 2010 at 5:41 PM, Daniel Stutzbach wrote: > Obviously, it would not be possible to write hooks that reject changesets Of course, this is one of the more interesting ways to use hooks. Since there's no current expectation that running our own will be a problem, why don't we convert and see how it goes? If we discover problems doing so and think a hosted solution would solve them, we can reconsider then. ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From martin at v.loewis.de Thu Sep 30 00:15:43 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 30 Sep 2010 00:15:43 +0200 Subject: [Python-Dev] hg conversion: tags In-Reply-To: <4CA2F8D1.1060704@egenix.com> References: <20100927112851.1455ea34@mission> <20100927183635.5bbb337b@pitrou.net> <4CA2F8D1.1060704@egenix.com> Message-ID: <4CA3BA8F.9060000@v.loewis.de> > I don't know how hg manages this, but can't we preserve the tag > information of the tags that you've scheduled to be removed > in some place that can easily be pulled in but doesn't > affect the main repo size ? Most certainly, and this is the plan already: we will keep the subversion repository data "forever". If you want the CVS repository from the day when it was converted to subversion, you can still get it from http://svn.python.org/snapshots/python-cvsroot-final.tar.bz2 (Created 27-Oct-2005 04:34, FWIW, so we are close to 5 years into subversion usage. Given that the "public" usage of the CVS started in 2000, we should start to look into the hg successor around 2013, for a switchover in Oktober 2015 :-). Regards, Martin From fdrake at acm.org Thu Sep 30 00:19:41 2010 From: fdrake at acm.org (Fred Drake) Date: Wed, 29 Sep 2010 18:19:41 -0400 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <4CA3B106.30406@v.loewis.de> Message-ID: On Wed, Sep 29, 2010 at 5:57 PM, Nick Coghlan wrote: > Would it be possible to sync up the reitveld issue numbers with the > roundup ones if you did that? Or would the fact that a single issue > can have multiple attached patches prevent that? Another quirk would be that often several pieces are uploaded which really constitute a single change; often docs and/or tests are provided separately, just because we've had to go back and ask for them. Sometimes they're provided by a separate contributor. Keeping the numbers aligned could probably be done based on the file number containing the patch, assuming those are never reused (I don't think they are, but haven't dug deep enough into roundup). ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From barry at python.org Thu Sep 30 00:33:47 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 29 Sep 2010 18:33:47 -0400 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <20100930000420.3b7d4999@pitrou.net> References: <20100929204125.416e2013@pitrou.net> <1285792363.3194.41.camel@localhost.localdomain> <20100929173010.2acf0b3d@mission> <20100930000420.3b7d4999@pitrou.net> Message-ID: <20100929183347.0d95304e@mission> On Sep 30, 2010, at 12:04 AM, Antoine Pitrou wrote: >On Wed, 29 Sep 2010 17:30:10 -0400 >Barry Warsaw wrote: >> One other thought: IME patches in general are suboptimal to >> branches, so I think we should be encouraging people to publish >> their branches publicly for review. A diff is a decent way to get >> feedback about code changes, but that's often only part of the work >> involved in deciding whether a change should be accepted or not. A >> reviewer often wants to do a build with the changes, test them on >> various tasks and application, run the test suite, etc. For this, >> "merge" is much better than patch(1). > >When I review a patch, I will tend to assume that the poster has >already run the test suite (especially if it's another committer). >Having to checkout a branch and generate a diff myself would make it >much longer to produce a review, in most cases. Yep, it depends on who is submitting the branch, what the branch changes, etc. >Even rebuilding a new branch from scratch can be slower than applying >the diff in an existing checkout and letting `make` rebuild a couple of >files. You can have "co-located" branches[1] which essentially switch in-place, so if a branch is changing some .c files, you won't have to rebuild the whole world just to try out a patch. -Barry [1] I only have experience with these in Bazaar so Mercurial's might work differently or be called something different. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From guido at python.org Thu Sep 30 00:38:52 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 29 Sep 2010 15:38:52 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <20100930001233.331484b4@pitrou.net> References: <4CA3B106.30406@v.loewis.de> <4CA3B66D.5070209@v.loewis.de> <20100930001233.331484b4@pitrou.net> Message-ID: On Wed, Sep 29, 2010 at 3:12 PM, Antoine Pitrou wrote: > On Wed, 29 Sep 2010 23:58:05 +0200 > "Martin v. L?wis" wrote: >> > That shouldn't be too hard. Someone just has to create an App Engine >> > project and handle the deployment. I guess the trickiest part is >> > making sure enough people have admin access so multiple people can >> > update the website, especially if we run a modified copy so that >> > bugs.python.org can push "anonymous" (and probably be the only thing >> > that can). >> >> I was hoping that I can run Rietveld on the same machine as roundup, >> with the added advantage that people can use the same logins, and >> even the same cookies. > > That would be very nice. > Aren't there some App Engine emulation layers? Or are they only for > development? Yes, http://code.google.com/p/django-gae2django/ (see also http://code.google.com/appengine/articles/pure_django.html) -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Thu Sep 30 00:51:30 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 30 Sep 2010 00:51:30 +0200 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <20100930001233.331484b4@pitrou.net> References: <4CA3B106.30406@v.loewis.de> <4CA3B66D.5070209@v.loewis.de> <20100930001233.331484b4@pitrou.net> Message-ID: <4CA3C2F2.7080404@v.loewis.de> Am 30.09.2010 00:12, schrieb Antoine Pitrou: > On Wed, 29 Sep 2010 23:58:05 +0200 > "Martin v. L?wis" wrote: >>> That shouldn't be too hard. Someone just has to create an App Engine >>> project and handle the deployment. I guess the trickiest part is >>> making sure enough people have admin access so multiple people can >>> update the website, especially if we run a modified copy so that >>> bugs.python.org can push "anonymous" (and probably be the only thing >>> that can). >> >> I was hoping that I can run Rietveld on the same machine as roundup, >> with the added advantage that people can use the same logins, and >> even the same cookies. > > That would be very nice. > Aren't there some App Engine emulation layers? Or are they only for > development? There is dev_appserver.py, but it is indeed only for development. However, there is also gae2django, which I'm going to try. Regards, Martin From chris at subtlety.com Thu Sep 30 01:06:06 2010 From: chris at subtlety.com (Chris Bergstresser) Date: Wed, 29 Sep 2010 19:06:06 -0400 Subject: [Python-Dev] Question on bz2 codec. Is this a bug? In-Reply-To: <1285797540.3194.52.camel@localhost.localdomain> References: <4CA3AA22.50304@egenix.com> <20100929232351.3b9fce6d@pitrou.net> <1285797540.3194.52.camel@localhost.localdomain> Message-ID: On Wed, Sep 29, 2010 at 5:59 PM, Antoine Pitrou wrote: > Le mercredi 29 septembre 2010 ? 17:41 -0400, Chris Bergstresser a > ?crit : >> On Wed, Sep 29, 2010 at 5:23 PM, Antoine Pitrou wrote: >> > Anyway, the obvious way to write line-by-line to a bz2 file is to use >> > the BZ2File class! >> >> ? ?The BZ2File class does not allow you to open a file for appending. >> ? ?Using the incremental encoder does work, > > In what sense? Do you mean it adds a new bz2 stream at the end of the > existing file? Yes. If you open an existing bz2 file for appending and use the incremental encoder to encode the data you write to it, you end up with a single file containing two separate bz2 compressed blocks of data. The bunzip2 program handles multiple streams in a single file correctly, and there's a bug open (complete with working patch) in the Python tracker to handle them as well. -- Chris From chris at subtlety.com Thu Sep 30 01:06:06 2010 From: chris at subtlety.com (Chris Bergstresser) Date: Wed, 29 Sep 2010 19:06:06 -0400 Subject: [Python-Dev] Question on bz2 codec. Is this a bug? In-Reply-To: <1285797540.3194.52.camel@localhost.localdomain> References: <4CA3AA22.50304@egenix.com> <20100929232351.3b9fce6d@pitrou.net> <1285797540.3194.52.camel@localhost.localdomain> Message-ID: On Wed, Sep 29, 2010 at 5:59 PM, Antoine Pitrou wrote: > Le mercredi 29 septembre 2010 ? 17:41 -0400, Chris Bergstresser a > ?crit : >> On Wed, Sep 29, 2010 at 5:23 PM, Antoine Pitrou wrote: >> > Anyway, the obvious way to write line-by-line to a bz2 file is to use >> > the BZ2File class! >> >> ? ?The BZ2File class does not allow you to open a file for appending. >> ? ?Using the incremental encoder does work, > > In what sense? Do you mean it adds a new bz2 stream at the end of the > existing file? Yes. If you open an existing bz2 file for appending and use the incremental encoder to encode the data you write to it, you end up with a single file containing two separate bz2 compressed blocks of data. The bunzip2 program handles multiple streams in a single file correctly, and there's a bug open (complete with working patch) in the Python tracker to handle them as well. -- Chris From merwok at netwok.org Thu Sep 30 04:37:38 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 30 Sep 2010 04:37:38 +0200 Subject: [Python-Dev] distutils and distutils2 bugs (Was: Re: Goodbye) In-Reply-To: <4C9BBDED.5010701@netwok.org> References: <20100922124700.000ac012@pitrou.net> <20100923012449.63B1022904D@kimball.webabinitio.net> <4C9AFF59.9020400@v.loewis.de> <4C9BBDED.5010701@netwok.org> Message-ID: <4CA3F7F2.6050707@netwok.org> Le 23/09/2010 22:51, ?ric Araujo a ?crit : > Le 23/09/2010 19:22, Terry Reedy a ?crit : >> As of just now, if you were to wonder "What (security) bugs are open for >> 2.5" and search on open 2.5 issues, you would get a list of 44 issues. >> [...] It it 44 instead of perhaps 5 because Tarek >> and Eric insist on marking all disutils2 issues for all versions even >> though though these issues have nothing to do with maintenance releases. >> It is a real nuisance when trying to do tracker cleanup. > > Let?s fix this. [...] > > In short, setting versions other than ?3rd party? for distutils2 bugs > does not help distutils2 and raises unhelpful results for people > querying the status of CPython releases. +1 on changing that. There has been no further feedback after Georg (thanks Georg), so I went ahead and set ?3rd party? for distutils2 bugs instead of 2.5-3.2, for all the aforementioned reasons. Bugs applying to distutils and distutils2 have versions 2.7, 3.1, 3.2, 3rd party so that the forward-port is not forgotten. If you commit a change to distutils, sysconfig or pkgutil, it?s fine to assign the forward-port to me or, if you can use Mercurial, to publish a changeset somewhere and request a pull/import. (Those versions changes sent a lot of email, but this had to be done. Sorry for the inconvenience.) Thanks again to Terry and Mark for their triage work. Hope this helps! Kind regards From raymond.hettinger at gmail.com Thu Sep 30 05:50:32 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 29 Sep 2010 22:50:32 -0500 Subject: [Python-Dev] API for binary operations on Sets Message-ID: <62BC914E-8B5A-424C-A068-739D0E71FF55@gmail.com> I would like to solicit this group's thoughts on how to reconcile the Set abstract base class with the API for built-in set objects (see http://bugs.python.org/issue8743 ). I've been thinking about this issue for a good while and the RightThingToDo(tm) isn't clear. Here's the situation: Binary operators for the built-in set object restrict their "other" argument to instances of set, frozenset, or one of their subclasses. Otherwise, they return NotImplemented. This design was intentional (i.e. part of the original pure python version, it is unittested behavior, and it is a documented restriction). It allows other classes to "see" the NotImplemented and have a chance to take-over using __ror__, __rand__, etc. Also, by not accepting any iterable, it prevents little coding atrocities or possible mistakes like "s | 'abc'". This is a break with what is done for lists (Guido has previously lamented that list.__add__ accepting any iterable is one of his "regrets"). This design has been in place for several years and so far everyone has been happy with it (no bug reports, feature requests, or discussions on the newsgroup, etc). If someone needed to process a non-set iterable, the named set methods (like intersection, update, etc) all accept any iterable value and this provides an immediate, usable alternative. In contrast, the Set and MutableSet abstract base classes in Lib/_abcoll.py take a different approach. They specify that something claiming to be set-like will accept any-iterable for a binary operator (IOW, the builtin set object does not comply). The provided mixins (such as __or__, __and__, etc) are implemented that way and it works fine. Also, the Set and MutableSet API do not provide named methods such as update, intersection, difference, etc. They aren't really needed because the operator methods already provide the functionality and because it keeps the Set API to a reasonable minimum. All of this it well and good, but the two don't interoperate. You can't get an instance of the Set ABC to work with a regular set, nor do regular sets comply with the ABC. These are problems because they defeat some of the design goals for ABCs. We have a few options: 1. Liberalize setobject.c binary operator methods to accept anything registered to the Set ABC and add a backwards incompatible restriction to the Set ABC binary operator methods to only accept Set ABC instances (they currently accept any iterable). This approach has a backwards incompatible tightening of the Set ABC, but that will probably affect very few people. It also has the disadvantage of not providing a straight-forward way to handle general iterable arguments (either the implementer needs to write named binary methods like update, difference, etc for that purpose or the user will need to cast the the iterable to a set before operating on it). The positive side of this option is that keeps the current advantages of the setobject API and its NotImplemented return value. 1a. Liberalize setobject.c binary operator methods, restrict SetABC methods, and add named methods (like difference, update, etc) that accept any iterable. 2. We could liberalize builtin set objects to accept any iterable as an "other" argument to a binary set operator. This choice is not entirely backwards compatible because it would break code depending on being able run __ror__, __rand__, etc after a NotImplemented value is returned. That being said, I think it unlikely that such code exists. The real disadvantage is that it replicates the problems with list.__add__ and Guido has said before that he doesn't want to do that again. I was leaning towards #1 or #1a and the guys on IRC thought #2 would be better. Now I'm not sure and would like additional input so I can get this bug closed for 3.2. Any thoughts on the subject would be appreciated. Thanks, Raymond P.S. I also encountered a small difficulty in implementing #2 that would still need to be resolved if that option is chosen. -------------- next part -------------- An HTML attachment was scrubbed... URL: From debatem1 at gmail.com Thu Sep 30 06:11:50 2010 From: debatem1 at gmail.com (geremy condra) Date: Wed, 29 Sep 2010 21:11:50 -0700 Subject: [Python-Dev] API for binary operations on Sets In-Reply-To: <62BC914E-8B5A-424C-A068-739D0E71FF55@gmail.com> References: <62BC914E-8B5A-424C-A068-739D0E71FF55@gmail.com> Message-ID: On Wed, Sep 29, 2010 at 8:50 PM, Raymond Hettinger wrote: > I would like to solicit this group's thoughts on how to reconcile the Set > abstract base class with the API for built-in set objects > (see?http://bugs.python.org/issue8743?). ?I've been thinking about this > issue for a good while and the RightThingToDo(tm) isn't clear. > Here's the situation: > Binary operators for the built-in set object restrict their "other" argument > to instances of set, frozenset, or one of their subclasses. ? Otherwise, > they return NotImplemented. ?This design was intentional (i.e. part of the > original pure python version, it is unittested behavior, and it is a > documented restriction). ?It allows other classes to "see" the > NotImplemented and have a chance to take-over using __ror__, __rand__, etc. > ? ? Also, by not accepting any iterable, it prevents little coding > atrocities or possible mistakes like "s | 'abc'". ?This is a break with what > is done for lists (Guido has previously lamented that list.__add__ accepting > any iterable is one of his "regrets"). ?This design has been in place for > several years and so far everyone has been happy with it (no bug reports, > feature requests, or discussions on the newsgroup, etc). ?If someone needed > to process a non-set iterable, the named set methods (like intersection, > update, etc) all accept any iterable value and this provides an immediate, > usable alternative. > In contrast, the Set and MutableSet abstract base classes in Lib/_abcoll.py > take a different approach. ?They specify that something claiming to be > set-like will accept any-iterable for a binary operator (IOW, the builtin > set object does not comply). ? The provided mixins (such as __or__, __and__, > etc) are implemented that way and it works fine. ?Also, the Set and > MutableSet API do not provide named methods such as update, intersection, > difference, etc. ?They aren't really needed because the operator methods > already provide the functionality and because it keeps the Set API to a > reasonable minimum. > All of this it well and good, but the two don't interoperate. ?You can't get > an instance of the Set ABC to work with a regular set, nor do regular sets > comply with the ABC. ?These are problems because they defeat some of the > design goals for ABCs. > We have a few options: > 1. Liberalize setobject.c binary operator methods to accept anything > registered to the Set ABC and add a backwards incompatible restriction to > the Set ABC binary operator methods to only accept Set ABC instances (they > currently accept any iterable). > This approach has a backwards incompatible tightening of the Set ABC, but > that will probably affect very few people. ?It also has the disadvantage of > not providing a straight-forward way to handle general iterable arguments > (either the implementer needs to write named binary methods like update, > difference, etc for that purpose or the user will need to cast the the > iterable to a set before operating on it). ? The positive side of this > option is that keeps the current advantages of the setobject API and its > NotImplemented return value. > 1a. ?Liberalize setobject.c binary operator methods, restrict SetABC > methods, and add named methods (like difference, update, etc) that accept > any iterable. > 2. We could liberalize builtin set objects to accept any iterable as an > "other" argument to a binary set operator. ?This choice is not entirely > backwards compatible because it would break code depending on being able run > __ror__, __rand__, etc after a NotImplemented value is returned. ?That being > said, I think it unlikely that such code exists. ?The real disadvantage is > that it replicates the problems with list.__add__ and Guido has said before > that he doesn't want to do that again. > I was leaning towards #1 or #1a and the guys on IRC thought #2 would be > better. ?Now I'm not sure and would like additional input so I can get this > bug closed for 3.2. ?Any thoughts on the subject would be appreciated. I'm not clear on what the issues with list.__add__ were, but my first impression is to lean towards #2. What am I missing? > Thanks, > > Raymond > > P.S. I also encountered a small difficulty in implementing #2 that would > still need to be resolved if that option is chosen. What's the issue, if you don't mind me asking? Geremy Condra From tjreedy at udel.edu Thu Sep 30 06:29:10 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 30 Sep 2010 00:29:10 -0400 Subject: [Python-Dev] API for binary operations on Sets In-Reply-To: <62BC914E-8B5A-424C-A068-739D0E71FF55@gmail.com> References: <62BC914E-8B5A-424C-A068-739D0E71FF55@gmail.com> Message-ID: On 9/29/2010 11:50 PM, Raymond Hettinger wrote: > I would like to solicit this group's thoughts on how to reconcile the > Set abstract base class with the API for built-in set objects (see > http://bugs.python.org/issue8743 ). I've been thinking about this issue > for a good while and the RightThingToDo(tm) isn't clear. > > Here's the situation: > > Binary operators for the built-in set object restrict their "other" > argument to instances of set, frozenset, or one of their subclasses. > Otherwise, they return NotImplemented. This design was intentional (i.e. > part of the original pure python version, it is unittested behavior, and > it is a documented restriction). It allows other classes to "see" the > NotImplemented and have a chance to take-over using __ror__, __rand__, > etc. Also, by not accepting any iterable, it prevents little coding > atrocities or possible mistakes like "s | 'abc'". This is a break with > what is done for lists (Guido has previously lamented that list.__add__ > accepting any iterable is one of his "regrets"). I do not understand this. List.__add__ is currently *more* restrictive than set ops in that it will not even accept a 'frozenlist' (aka tuple). >>> ll + (4,5,6) Traceback (most recent call last): File "", line 1, in ll + (4,5,6) TypeError: can only concatenate list (not "tuple") to list >>> ll.__add__((5,6,7)) Traceback (most recent call last): File "", line 1, in ll.__add__((5,6,7)) TypeError: can only concatenate list (not "tuple") to list Does this violate the Sequence ABC (assuming there is one)? -- Terry Jan Reedy From raymond.hettinger at gmail.com Thu Sep 30 06:54:01 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 29 Sep 2010 23:54:01 -0500 Subject: [Python-Dev] API for binary operations on Sets In-Reply-To: References: <62BC914E-8B5A-424C-A068-739D0E71FF55@gmail.com> Message-ID: <0769A24E-8685-4FB1-BAD1-408EB6729B92@gmail.com> On Sep 29, 2010, at 11:29 PM, Terry Reedy wrote: > I do not understand this. List.__add__ is currently *more* restrictive than set ops in that it will not even accept a 'frozenlist' (aka tuple). Sorry, that should have been __iadd__(). >>> s = range(5) >>> s += 'abc' >>> s [0, 1, 2, 3, 4, 'a', 'b', 'c'] Raymond From jackdied at gmail.com Thu Sep 30 07:15:20 2010 From: jackdied at gmail.com (Jack Diederich) Date: Thu, 30 Sep 2010 01:15:20 -0400 Subject: [Python-Dev] API for binary operations on Sets In-Reply-To: <62BC914E-8B5A-424C-A068-739D0E71FF55@gmail.com> References: <62BC914E-8B5A-424C-A068-739D0E71FF55@gmail.com> Message-ID: I will say something snarky now and (hopefully) something useful tomorrow. When ABCs went in I was +0 because, like annotations, I was told I wouldn't have to care about them. That said; I do actually care about the set interface and what "set-y-ness" means for regular duck typing reasons. What people expect sets to do is what sets-alikes should do. -Jack From brett at python.org Thu Sep 30 07:20:57 2010 From: brett at python.org (Brett Cannon) Date: Wed, 29 Sep 2010 22:20:57 -0700 Subject: [Python-Dev] hg conversion: tags In-Reply-To: <4CA3BA8F.9060000@v.loewis.de> References: <20100927112851.1455ea34@mission> <20100927183635.5bbb337b@pitrou.net> <4CA2F8D1.1060704@egenix.com> <4CA3BA8F.9060000@v.loewis.de> Message-ID: On Wed, Sep 29, 2010 at 15:15, "Martin v. L?wis" wrote: >> I don't know how hg manages this, but can't we preserve the tag >> information of the tags that you've scheduled to be removed >> in some place that can easily be pulled in but doesn't >> affect the main repo size ? > > Most certainly, and this is the plan already: we will keep the > subversion repository data "forever". > > If you want the CVS repository from the day when it was converted > to subversion, you can still get it from > > http://svn.python.org/snapshots/python-cvsroot-final.tar.bz2 > > (Created 27-Oct-2005 04:34, FWIW, so we are close to 5 years > into subversion usage. Given that the "public" usage of the CVS > started in 2000, we should start to look into the hg successor > around 2013, for a switchover in Oktober 2015 :-). Just so people know, I am *not* leading that charge. From raymond.hettinger at gmail.com Thu Sep 30 08:32:42 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Thu, 30 Sep 2010 01:32:42 -0500 Subject: [Python-Dev] API for binary operations on Sets In-Reply-To: References: <62BC914E-8B5A-424C-A068-739D0E71FF55@gmail.com> Message-ID: On Sep 29, 2010, at 11:11 PM, geremy condra wrote: >> >> P.S. I also encountered a small difficulty in implementing #2 that would >> still need to be resolved if that option is chosen. > > What's the issue, if you don't mind me asking? IIRC, just commenting-out the Py_AnySet checks in set_or, set_xor, etc led to a segfault in TestOnlySetsInBinaryOps in Lib/test/test_set.py. There was something subtle going on and I haven't had time to trace though it yet. Somewhere a guard is probably missing. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Thu Sep 30 10:10:30 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 30 Sep 2010 10:10:30 +0200 Subject: [Python-Dev] hg conversion: tags In-Reply-To: <4CA3BA8F.9060000@v.loewis.de> References: <20100927112851.1455ea34@mission> <20100927183635.5bbb337b@pitrou.net> <4CA2F8D1.1060704@egenix.com> <4CA3BA8F.9060000@v.loewis.de> Message-ID: <4CA445F6.4080707@egenix.com> "Martin v. L?wis" wrote: >> I don't know how hg manages this, but can't we preserve the tag >> information of the tags that you've scheduled to be removed >> in some place that can easily be pulled in but doesn't >> affect the main repo size ? > > Most certainly, and this is the plan already: we will keep the > subversion repository data "forever". Fair enough. > If you want the CVS repository from the day when it was converted > to subversion, you can still get it from > > http://svn.python.org/snapshots/python-cvsroot-final.tar.bz2 > > (Created 27-Oct-2005 04:34, FWIW, so we are close to 5 years > into subversion usage. Given that the "public" usage of the CVS > started in 2000, we should start to look into the hg successor > around 2013, for a switchover in Oktober 2015 :-). Good idea :-) I suppose we'll have CVCSes by then - cloud version control systems. Let's reserve cloud number 9 for use by Python (cumulonimbus is too hard to remember) :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 30 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dirkjan at ochtman.nl Thu Sep 30 10:22:32 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Thu, 30 Sep 2010 10:22:32 +0200 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: Message-ID: On Wed, Sep 29, 2010 at 20:32, Guido van Rossum wrote: > I would like to recommend that the Python core developers start using > a code review tool such as Rietveld or Reviewboard. I don't really > care which tool we use (I'm sure there are plenty of pros and cons to > each) but I do think we should get out of the stone age and start > using a tool for the majority of our code reviews. Rambling thoughts about some of the things mentioned in this thread. I think hg-review looks interesting, though it may not (yet) have the level of sophistication of Rietveld. (Public test instance at http://review.stevelosh.com/.) It might be interesting to integrate Rietveld uploads in a Mercurial extension, particularly if it gets integrated with mq somehow. Email reviews seem to actually work really well for Mercurial (using the patchbomb extension to send out patches to mailing lists); the only problem we run into is that we can't keep track of things that have been submitted and reviewed. But it makes commenting inline effortless and provides a familiar interface for everyone. For the imparting wisdom thing, I think that's more a culture thing than a tool thing. If reviews happen in public as a standard part of the process, then it probably won't appear judgmental or overbearing either in a tool or in email (or issue tracker). I hope people will discover and like mq, which makes it easy to keep together a coherent series of patches. Cheers, Dirkjan From lvh at laurensvh.be Thu Sep 30 10:30:01 2010 From: lvh at laurensvh.be (Laurens Van Houtven) Date: Thu, 30 Sep 2010 10:30:01 +0200 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: Message-ID: Hi, On using code review tools: +1, no discussion. I've recently been doing a bit of research on these as a side effect of researching continuous deployment, so: 1. Barry is right about Launchpad's merge proposals (unsurprisingly) 2. hg has a review extension called hg-review, but I think it'll be too difficult to use properly for a big dev team like Python's with many people reading code reviews (hg-review stores reviews in the repository) -- there's no real canonical way AFAIK of saying "give me all pending reviews everywhere in the codebase", like you would have with a centralized place to publish patches vs a specific revision. (I *am* going to use hg-review personally for my startup, I'm not saying it's a bad tool at all! Just that it's not very good for big teams yet, because there's no real sensible way of getting a centralized UI for both publishing and reviewing suggested patches.) 3. FWIW, I agree Rietveld's a better tool than ReviewBoard, and I'm not attached to either of the authors, so Rietveld +1 cheers lvh -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Thu Sep 30 10:58:43 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 30 Sep 2010 17:58:43 +0900 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <20100929183347.0d95304e@mission> References: <20100929204125.416e2013@pitrou.net> <1285792363.3194.41.camel@localhost.localdomain> <20100929173010.2acf0b3d@mission> <20100930000420.3b7d4999@pitrou.net> <20100929183347.0d95304e@mission> Message-ID: <87bp7fboak.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > You can have "co-located" branches[1] which essentially switch > in-place, so if a branch is changing some .c files, you won't have > to rebuild the whole world just to try out a patch. In Mercurial these are called "named branches", and they are repo-local (by which I mean they must be part of the DAG). Named branches used to have some inconvenient aspects relevant to standalone branches (they could be fairly confusing to other users if pushed before being merge to mainline). It's not obvious to me that Mercurial style named branches would work well here ... it would take a little thought to design an appropriate workflow, anyway. From a.badger at gmail.com Thu Sep 30 11:10:14 2010 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 30 Sep 2010 05:10:14 -0400 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: <20100930091014.GG20328@unaka.lan> On Wed, Sep 29, 2010 at 01:23:24PM -0700, Guido van Rossum wrote: > On Wed, Sep 29, 2010 at 1:12 PM, Brett Cannon wrote: > > On Wed, Sep 29, 2010 at 12:03, Guido van Rossum wrote: > >> A problem with that is that we regularly make matching improvements to > >> upload.py and the server-side code it talks to. While we tend to be > >> conservative in these changes (because we don't control what version > >> of upload.py people use) it would be a pain to maintain backwards > >> compatibility with a version that was distributed in Misc/ two years > >> ago -- that's kind of outside our horizon. > > > > Well, I would assume people are working from a checkout. Patches from > > an outdated checkout simply would fail and that's fine by me. > > Ok, but that's an extra barrier for contributions. Lots of people when > asked for a patch just modify their distro in place and you can count > yourself lucky if they send you a diff from a clean copy. > > But maybe with Hg it's less of a burden to ask people to use a checkout. > > > How often do we even get patches generated from a downloaded copy of > > Python? Is it enough to need to worry about this? > > I used to get these frequently. I don't know what the experience of > the current crop of core developers is though, so maybe my gut > feelings here are outdated. > When helping out on a Linux distribution, dealing with patches against the latest tarball is a fairly frequent occurrence. The question would be whether these patches get filtered through the maintainer of the package before landing in roundup/rietveld and whether the distro maintainer is sufficiently in tune with python development that they're maintaining both patches against the last tarball and a checkout of trunk with the patches applied intelligently there. A few other random thoughts: * hg could be more of a burden in that it may be unfamiliar to the casual python user who happens to have found a fix for a bug and wants to submit it. cvs and svn are similar enough that people comfortable with one are usually comfortable with the other but hg has different semantics. * The barrier to entry seems to be higher the less well integrated the tools are. I occassionally try to contribute patches to bzr in launchpad and the integration there is horrid. You end up with two separate streams of comments and you don't automatically get subscribed to both. There's several UI elements for associating a branch with a bug but some of them are buggy (or else are very strict on what input they're expecting) while other ones are hard to find. Since I only contribute a patch two or three times a year, I have to re-figure out the process each time I try to contribute. * I like the idea of patch complexity being a measure of whether the patch needs to go into a code review tool in that it keeps simple things simple and gives more advanced tools to more advanced cases. I dislike it in that for someone who's just contributing a patch to fix a problem that they're encountering which happens to be somewhat complex, they end up having to learn a lot about tools that they may never use again. * It seems like code review will be a great aid to people who submit changes or review changes frequently. The trick will be making it non-intimidating for someone who's just going to contribute changes infrequently. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From orsenthil at gmail.com Thu Sep 30 11:12:08 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Thu, 30 Sep 2010 14:42:08 +0530 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: <20100930091208.GA29763@remy> On Thu, Sep 30, 2010 at 07:45:52AM +1000, Nick Coghlan wrote: > Somewhat amusing to get to this thread a few minutes after creating a > Reitveld issue for the first pass of my urllib.parse patch :) Hello Nick, could you please point me to that? Also, in general here are my points on Code Review using Rietveld. I think, Martin already indicated that Roundup can post to Rietveld instance. That should really be helpful. I imagine a work-flow like this. 1. Contributor creates a patch and uploads to Roundup tracker. 2. If the patch is very small or does not require review, the committer can just commit it. 3. If the patch does review review, one can click a link to do "Patch Review" wherein only at the moment, the Patch is POSTed the rietveld instance and is available for review. -- Senthil "I went to the museum where they had all the heads and arms from the statues that are in all the other museums." -- Steven Wright From stephen at xemacs.org Thu Sep 30 12:17:01 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 30 Sep 2010 19:17:01 +0900 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <87bp7fboak.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20100929204125.416e2013@pitrou.net> <1285792363.3194.41.camel@localhost.localdomain> <20100929173010.2acf0b3d@mission> <20100930000420.3b7d4999@pitrou.net> <20100929183347.0d95304e@mission> <87bp7fboak.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <8762xnbko2.fsf@uwakimon.sk.tsukuba.ac.jp> Sorry for following up to myself, but this typo might be very confusing: Stephen J. Turnbull writes: > Barry Warsaw writes: > > > You can have "co-located" branches[1] which essentially switch > > in-place, so if a branch is changing some .c files, you won't have > > to rebuild the whole world just to try out a patch. > > In Mercurial these are called "named branches", and they are > repo-local (by which I mean they must be part of the DAG). Named > branches used to have some inconvenient aspects relevant to standalone s/relevant/relative/ > branches (they could be fairly confusing to other users if pushed > before being merge to mainline). > > It's not obvious to me that Mercurial style named branches would work > well here ... it would take a little thought to design an appropriate > workflow, anyway. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/stephen%40xemacs.org From techtonik at gmail.com Thu Sep 30 13:07:56 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 30 Sep 2010 14:07:56 +0300 Subject: [Python-Dev] Python wiki In-Reply-To: <4C9E835A.6010606@v.loewis.de> References: <20100923100157.288a72a8@mission> <20100923103506.77307c5e@mission> <20100923105355.69068235@mission> <20100923173255.282841a1@pitrou.net> <20100923115719.2250e776@mission> <20100923184159.4ce2048e@pitrou.net> <19611.41449.475652.757621@montanaro.dyndns.org> <4C9BD64F.5020908@v.loewis.de> <4C9C3B44.3050401@v.loewis.de> <4C9C7DF1.5040500@voidspace.org.uk> <4C9E6CF8.4080705@v.loewis.de> <4C9E74D0.3050704@v.loewis.de> <4C9E835A.6010606@v.loewis.de> Message-ID: On Sun, Sep 26, 2010 at 2:18 AM, "Martin v. L?wis" wrote: > Am 26.09.2010 00:48, schrieb Georg Brandl: >> Am 26.09.2010 00:16, schrieb "Martin v. L?wis": >>>> Redirect wiki.python.org to the Python wiki front page, and put the Jython >>>> wiki somewhere on its own (whether it's wiki.jython.org or not). >>> >>> But that can't work: then off-site links into either wiki break. >> >> Why -- they can be redirected easily. > > Ok, so please be more specific: what exactly should the new structure > look like? >From http://wiki.python.org/moin/SiteImprovements * Shorten URLs - remove /moin/ prefix from http://wiki.python.org/moin/SiteImprovements#Wiki * This requires moving Jython wiki from http://wiki.python.org/jython/ to http://wiki.jython.org/ and placing a temporary redirect on the previous places. -- ?techtonik 2010-03-16 08:39:34 Expanding: 1. Move http://wiki.python.org/moin/ to http://wiki.python.org/ 2. Setup 301 redirect from http://wiki.python.org/moin/(.*) to http://wiki.python.org/$1 3. Setup 301 redirect from http://wiki.python.org/jython/(.*) to http://wiki.jython.org/$1 (optional tasks below) 4. Setup Analytics account to track sources of redirection to the old pages and make this list public 5. Crowdsource changing source links 6. Drop redirection after redirected hits drop below 5% (even for 5% new page can be looked up through Google) -- anatoly t. From daniel at stutzbachenterprises.com Thu Sep 30 13:33:59 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Thu, 30 Sep 2010 06:33:59 -0500 Subject: [Python-Dev] API for binary operations on Sets In-Reply-To: References: <62BC914E-8B5A-424C-A068-739D0E71FF55@gmail.com> Message-ID: On Wed, Sep 29, 2010 at 11:29 PM, Terry Reedy wrote: > Does this violate the Sequence ABC (assuming there is one)? > There is a Sequence ABC, but it does not define __add__. It only defines the following methods: __contains__, __getitem__, __iter__, __len__, __reversed__, count, and index tuple, range, and str types all register as following the Sequence ABC. list and bytearray types register as following the MutableSequence ABC, which is a subclass of the Sequence ABC. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From flub at devork.be Thu Sep 30 14:21:32 2010 From: flub at devork.be (Floris Bruynooghe) Date: Thu, 30 Sep 2010 13:21:32 +0100 Subject: [Python-Dev] Resource leaks warnings In-Reply-To: References: <20100808221846.CFD80EEA3F@mail.python.org> <1285764147.3168.10.camel@localhost.localdomain> <201009292311.45355.steve@pearwood.info> <20100929094054.4c200bea@mission> Message-ID: On 29 September 2010 22:25, Nick Coghlan wrote: > On Wed, Sep 29, 2010 at 11:40 PM, Barry Warsaw wrote: >> I don't think it should be in the gc module, but I would prefer it be enabled >> and controlled through a separate module, rather than something Python does >> automatically for your convenience. > > The os module would seem to be the place to enable/disable tracking of > OS level resource leaks (i.e. file descriptors and possible HANDLES on > Windows). Heh, I was expecting the sys module to be the natural choice because this would be changing interpreter behaviour. It's just random bikeshedding at this point however. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From ncoghlan at gmail.com Thu Sep 30 14:38:19 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Sep 2010 22:38:19 +1000 Subject: [Python-Dev] API for binary operations on Sets In-Reply-To: <62BC914E-8B5A-424C-A068-739D0E71FF55@gmail.com> References: <62BC914E-8B5A-424C-A068-739D0E71FF55@gmail.com> Message-ID: On Thu, Sep 30, 2010 at 1:50 PM, Raymond Hettinger wrote: > 1a. ?Liberalize setobject.c binary operator methods, restrict SetABC > methods, and add named methods (like difference, update, etc) that accept > any iterable. > 2. We could liberalize builtin set objects to accept any iterable as an > "other" argument to a binary set operator. ?This choice is not entirely > backwards compatible because it would break code depending on being able run > __ror__, __rand__, etc after a NotImplemented value is returned. ?That being > said, I think it unlikely that such code exists. ?The real disadvantage is > that it replicates the problems with list.__add__ and Guido has said before > that he doesn't want to do that again. > I was leaning towards #1 or #1a and the guys on IRC thought #2 would be > better. ?Now I'm not sure and would like additional input so I can get this > bug closed for 3.2. ?Any thoughts on the subject would be appreciated. > Thanks, My own inclination would be to go with #1a, *unless* Guido chimes in to say he's OK with having the set operators accept arbitrary iterators. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From barry at python.org Thu Sep 30 16:27:55 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 30 Sep 2010 10:27:55 -0400 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <8762xnbko2.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20100929204125.416e2013@pitrou.net> <1285792363.3194.41.camel@localhost.localdomain> <20100929173010.2acf0b3d@mission> <20100930000420.3b7d4999@pitrou.net> <20100929183347.0d95304e@mission> <87bp7fboak.fsf@uwakimon.sk.tsukuba.ac.jp> <8762xnbko2.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20100930102755.43ab03d3@mission> The torrential rains are causing havoc with my internet, so apologies for replying out of sequence. On Sep 30, 2010, at 07:17 PM, Stephen J. Turnbull wrote: >Sorry for following up to myself, but this typo might be very >confusing: > >Stephen J. Turnbull writes: > > Barry Warsaw writes: > > > > > You can have "co-located" branches[1] which essentially switch > > > in-place, so if a branch is changing some .c files, you won't > > > have to rebuild the whole world just to try out a patch. > > > > In Mercurial these are called "named branches", and they are > > repo-local (by which I mean they must be part of the DAG). Named > > branches used to have some inconvenient aspects relevant to > > standalone > >s/relevant/relative/ > > > branches (they could be fairly confusing to other users if pushed > > before being merge to mainline). > > > > It's not obvious to me that Mercurial style named branches would > > work well here ... it would take a little thought to design an > > appropriate workflow, anyway. I should note that I don't particularly like colocated/named branches. I personally much prefer separate directories for each feature or bug I'm working on. It helps me keep track of what I'm doing. I have a fast machine so recompiling all of Python is no big deal. I do like having the choice of being able to colocate or not, based on my own workflow preferences. But I suppose with Mercurial I can just have multiple copies of the same branch in different directories, and just start out with "hg update -C foo" -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From jnoller at gmail.com Thu Sep 30 16:47:11 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 30 Sep 2010 10:47:11 -0400 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: Message-ID: On Wed, Sep 29, 2010 at 2:32 PM, Guido van Rossum wrote: > I would like to recommend that the Python core developers start using > a code review tool such as Rietveld or Reviewboard. I don't really > care which tool we use (I'm sure there are plenty of pros and cons to > each) but I do think we should get out of the stone age and start > using a tool for the majority of our code reviews. > > While I would personally love to see Rietveld declared the official > core Python code review tool, I realize that since I wrote as a Google > engineer and it is running on Google infrastructure (App Engine), I > can't be fully objective about the tool choice -- even though it is > open source, has several non-Googler maintainers, and can be run > outside App Engine as well. > > But I do think that using a specialized code review tool rather than > unstructured email plus a general-purpose issue tracker can hugely > improve developer performance and also increase community > participation. (A code review tool makes it much more convenient for a > senior reviewer to impart their wisdom to a junior developer without > appearing judgmental or overbearing.) > > See also this buzz thread: > http://www.google.com/buzz/115212051037621986145/At6Rj82Kret/When-will-the-Python-dev-community-start-using > > -- > --Guido van Rossum (python.org/~guido) > Regardless of the tool(s) used, code reviews are a fantastic equalizer. If you have long time, experienced developers "submitting" to the same rules that newer contributors have to follow then it helps remove the idea that there is special treatment occurring. Additionally, a lot of people are terrified of code reviews as they view it as a "public flogging" - holding everyone to the same standards, and showing this is not the case helps fight this perception. Not to mention; there's a lot to be learned from doing them on both sides. At work, I learn about chunks of code I might not have otherwise known about or approaches to a problem I'd never considered. I sort of drank the kool-aid. From exarkun at twistedmatrix.com Thu Sep 30 16:52:18 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 30 Sep 2010 14:52:18 -0000 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: Message-ID: <20100930145218.2022.353483588.divmod.xquotient.55@localhost.localdomain> On 02:47 pm, jnoller at gmail.com wrote: >On Wed, Sep 29, 2010 at 2:32 PM, Guido van Rossum >wrote: >>I would like to recommend that the Python core developers start using >>a code review tool such as Rietveld or Reviewboard. I don't really >>care which tool we use (I'm sure there are plenty of pros and cons to >>each) but I do think we should get out of the stone age and start >>using a tool for the majority of our code reviews. >> >>While I would personally love to see Rietveld declared the official >>core Python code review tool, I realize that since I wrote as a Google >>engineer and it is running on Google infrastructure (App Engine), I >>can't be fully objective about the tool choice -- even though it is >>open source, has several non-Googler maintainers, and can be run >>outside App Engine as well. >> >>But I do think that using a specialized code review tool rather than >>unstructured email plus a general-purpose issue tracker can hugely >>improve developer performance and also increase community >>participation. (A code review tool makes it much more convenient for a >>senior reviewer to impart their wisdom to a junior developer without >>appearing judgmental or overbearing.) >> >>See also this buzz thread: >>http://www.google.com/buzz/115212051037621986145/At6Rj82Kret/When- >>will-the-Python-dev-community-start-using >> >>-- >>--Guido van Rossum (python.org/~guido) > >Regardless of the tool(s) used, code reviews are a fantastic >equalizer. If you have long time, experienced developers "submitting" >to the same rules that newer contributors have to follow then it helps >remove the idea that there is special treatment occurring. Of course, this is only true if the core developers *do* submit to the same rules. Is anyone proposing that current core committers have all their work reviewed before it is accepted? (I am strongly in favor of this, but I don't think many core committers are.) Jean-Paul From jnoller at gmail.com Thu Sep 30 16:56:56 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 30 Sep 2010 10:56:56 -0400 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <20100930145218.2022.353483588.divmod.xquotient.55@localhost.localdomain> References: <20100930145218.2022.353483588.divmod.xquotient.55@localhost.localdomain> Message-ID: On Thu, Sep 30, 2010 at 10:52 AM, wrote: > On 02:47 pm, jnoller at gmail.com wrote: >> >> On Wed, Sep 29, 2010 at 2:32 PM, Guido van Rossum >> wrote: >>> >>> I would like to recommend that the Python core developers start using >>> a code review tool such as Rietveld or Reviewboard. I don't really >>> care which tool we use (I'm sure there are plenty of pros and cons to >>> each) but I do think we should get out of the stone age and start >>> using a tool for the majority of our code reviews. >>> >>> While I would personally love to see Rietveld declared the official >>> core Python code review tool, I realize that since I wrote as a Google >>> engineer and it is running on Google infrastructure (App Engine), I >>> can't be fully objective about the tool choice -- even though it is >>> open source, has several non-Googler maintainers, and can be run >>> outside App Engine as well. >>> >>> But I do think that using a specialized code review tool rather than >>> unstructured email plus a general-purpose issue tracker can hugely >>> improve developer performance and also increase community >>> participation. (A code review tool makes it much more convenient for a >>> senior reviewer to impart their wisdom to a junior developer without >>> appearing judgmental or overbearing.) >>> >>> See also this buzz thread: >>> http://www.google.com/buzz/115212051037621986145/At6Rj82Kret/When- >>> will-the-Python-dev-community-start-using >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >> >> Regardless of the tool(s) used, code reviews are a fantastic >> equalizer. If you have long time, experienced developers "submitting" >> to the same rules that newer contributors have to follow then it helps >> remove the idea that there is special treatment occurring. > > Of course, this is only true if the core developers *do* submit to the same > rules. ?Is anyone proposing that current core committers have all their work > reviewed before it is accepted? > > (I am strongly in favor of this, but I don't think many core committers > are.) > > Jean-Paul > I'll propose it, knowing full well I won't win. Code reviews have saved my bacon on numerous occasions. The best unit tests on the planet won't protect you against a fundamentally bad assumption or logic error. Like I said - I think it helps "equalize" things. YMMV. From guido at python.org Thu Sep 30 16:56:59 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 30 Sep 2010 07:56:59 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <20100930145218.2022.353483588.divmod.xquotient.55@localhost.localdomain> References: <20100930145218.2022.353483588.divmod.xquotient.55@localhost.localdomain> Message-ID: On Thu, Sep 30, 2010 at 7:52 AM, wrote: > On 02:47 pm, jnoller at gmail.com wrote: >> Regardless of the tool(s) used, code reviews are a fantastic >> equalizer. If you have long time, experienced developers "submitting" >> to the same rules that newer contributors have to follow then it helps >> remove the idea that there is special treatment occurring. > > Of course, this is only true if the core developers *do* submit to the same > rules. ?Is anyone proposing that current core committers have all their work > reviewed before it is accepted? > > (I am strongly in favor of this, but I don't think many core committers > are.) Having worked in this style for almost 5 years now, I am also strongly in favor. Jesse expressed it better than I could. -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Thu Sep 30 17:07:05 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 30 Sep 2010 17:07:05 +0200 Subject: [Python-Dev] We should be using a tool for code reviews References: <20100930145218.2022.353483588.divmod.xquotient.55@localhost.localdomain> Message-ID: <20100930170705.5c1b8857@pitrou.net> On Thu, 30 Sep 2010 14:52:18 -0000 exarkun at twistedmatrix.com wrote: > > > >Regardless of the tool(s) used, code reviews are a fantastic > >equalizer. If you have long time, experienced developers "submitting" > >to the same rules that newer contributors have to follow then it helps > >remove the idea that there is special treatment occurring. > > Of course, this is only true if the core developers *do* submit to the > same rules. Is anyone proposing that current core committers have all > their work reviewed before it is accepted? > > (I am strongly in favor of this, but I don't think many core committers > are.) While I agree with non-trivial patches being *posted* for review, I don't agree that an actual review should happen in order for a patch to be committed. That's unless we get 10x the number of reviewers we currently have. (and you certainly know by experience that I would be glad to have reviews on my patches, especially the network-related ones ;-)) Regards Antoine. From daniel at stutzbachenterprises.com Thu Sep 30 17:31:36 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Thu, 30 Sep 2010 10:31:36 -0500 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <20100930145218.2022.353483588.divmod.xquotient.55@localhost.localdomain> References: <20100930145218.2022.353483588.divmod.xquotient.55@localhost.localdomain> Message-ID: On Thu, Sep 30, 2010 at 9:52 AM, wrote: > Of course, this is only true if the core developers *do* submit to the same > rules. Is anyone proposing that current core committers have all their work > reviewed before it is accepted? > I think most would welcome (or at least tolerate ;) ) additional review of their code. The hard part is encouraging contributors to find the time and motivation to thoroughly review code that they aren't personally interested in (and perhaps not even familiar with). -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From orsenthil at gmail.com Thu Sep 30 17:40:17 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Thu, 30 Sep 2010 21:10:17 +0530 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100930145218.2022.353483588.divmod.xquotient.55@localhost.localdomain> Message-ID: <20100930154017.GA14119@remy> > On Thu, Sep 30, 2010 at 9:52 AM, wrote: > > Of course, this is only true if the core developers *do* submit to the same > rules. ?Is anyone proposing that current core committers have all their > work reviewed before it is accepted? For large patches it is good idea. But enforcing it for smaller patches may cost us more time for each commit. -- Senthil Any sufficiently advanced bug is indistinguishable from a feature. -- Rich Kulawiec From martin at v.loewis.de Thu Sep 30 17:48:36 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 30 Sep 2010 17:48:36 +0200 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100930145218.2022.353483588.divmod.xquotient.55@localhost.localdomain> Message-ID: <4CA4B154.6060706@v.loewis.de> > The hard part is encouraging contributors to find the time and > motivation to thoroughly review code that they aren't personally > interested in (and perhaps not even familiar with). Not sure how well 'tit for tat' schemes work - we *could* require that people don't commit unreviewed changes, and also require that you can't commit unless you have reviewed somebody else's changes. So if you do 10 reviews, you are entitled to 10 commits... Of course, that would put more burden on those people who already do all the work. Regards, Martin From martin at v.loewis.de Thu Sep 30 17:51:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 30 Sep 2010 17:51:07 +0200 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <20100930154017.GA14119@remy> References: <20100930145218.2022.353483588.divmod.xquotient.55@localhost.localdomain> <20100930154017.GA14119@remy> Message-ID: <4CA4B1EB.9080106@v.loewis.de> Am 30.09.2010 17:40, schrieb Senthil Kumaran: >> On Thu, Sep 30, 2010 at 9:52 AM, wrote: >> >> Of course, this is only true if the core developers *do* submit to the same >> rules. Is anyone proposing that current core committers have all their >> work reviewed before it is accepted? > > For large patches it is good idea. But enforcing it for smaller > patches may cost us more time for each commit. I think the supporters of code review readily accept that: in return for the higher effort, they say, we also get more quality. So it's really a quality vs. quantity thing. It would be easy to just commit all patches that are uploaded to roundup, but of course, it would horribly break Python. With mandatory code review, even less patches get reviewed than today. Regards, Martin From daniel at stutzbachenterprises.com Thu Sep 30 18:02:19 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Thu, 30 Sep 2010 11:02:19 -0500 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <4CA4B154.6060706@v.loewis.de> References: <20100930145218.2022.353483588.divmod.xquotient.55@localhost.localdomain> <4CA4B154.6060706@v.loewis.de> Message-ID: On Thu, Sep 30, 2010 at 10:48 AM, "Martin v. L?wis" wrote: > Not sure how well 'tit for tat' schemes work - we *could* require > that people don't commit unreviewed changes, and also require that > you can't commit unless you have reviewed somebody else's changes. > I wonder if a "reputation" scheme would work better. Track and publicize patch submissions, reviews, and the review/patch ratio, but do not enforce any particular ratios. Perhaps provide a roundup query showing patches awaiting review sorted by the patch submitter's review/patch ratio? (in descending order) Obviously there would be many non-trivial details to work out. I'm just brainstorming. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Thu Sep 30 18:19:13 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 30 Sep 2010 18:19:13 +0200 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: Am 29.09.2010 20:49, schrieb Guido van Rossum: > Unfortunately taking the average patch posted to the tracker and > importing it in Rietveld is very iffy -- it's very hard to find the > right branch+rev needed to be able to apply the patch correctly -- not > to mention that there are so many (slightly) different patch formats. > It would take a fair bit of AI to get this right. > > The recommended way to work with Rietveld is to use a checkout > (fortunately with Hg this will become easier to do) and use the > "upload.py" script that you can download from Rietveld (log in and > click on the Create link). Then could we do it the other way round and let (our) upload.py open an issue with the patch automatically (after querying for information not already given to rietveld)? Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From barry at python.org Thu Sep 30 18:33:52 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 30 Sep 2010 12:33:52 -0400 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: Message-ID: <20100930123352.25b34d84@mission> On Sep 30, 2010, at 10:47 AM, Jesse Noller wrote: >Not to mention; there's a lot to be learned from doing them on both >sides. At work, I learn about chunks of code I might not have >otherwise known about or approaches to a problem I'd never considered. >I sort of drank the kool-aid. Tools aside, I completely agree. Many projects that I contribute to have policies such as "nothing lands without reviewer approval". For some that means one reviewer must approve it, for others two +1s and no -1s, or a coding approval and a ui approval, etc. When I was the review team lead on Launchpad, I had a goal that every developer would also eventually be a reviewer. We started with a small number of experienced developers, then ran a mentor program to train new reviewers (who we called "mentats"). A mentat approval was not enough to land a branch, but the mentor could basically say "yes, i agree with the review" and it would land. Eventually, by mutual consent a mentat would graduate to full reviewership (and hopefully be a mentor to new reviewers). This was hugely successful among many dimensions. It got everyone on the same page as to coding standards, it equalized the playing field, it got everyone to think collaboratively as a team, folks learned about parts of the system they didn't have day-to-day intimate knowledge about, and it got changes landed much more quickly. Here are a few things we learned along the way. Their applicability to Python will vary of course, and we need to find what works for us. * Keep branches *small*. We had a limit of 800 lines of diff, with special explicit permission from the person reviewing your change to exceed. 800 lines is about the maximum that a person can review in a reasonable amount of time without losing focus. * The *goal* was 5 minutes review, but the reality was that most reviews took about 15-20 minutes. If it's going longer, you weren't doing it right. This meant that there was a level of trust between the reviewer and the dev, so that the basic design had been previously discussed and agreed upon (we mandated pre-implementation chats), etc. A reviewer was there to sanity check the implementation, watch for obvious mistakes, ensure test coverage for the new code, etc. If you were questioning the basic design, you weren't doing a code review. It was okay to reject a change quickly if you found fatal problems. * The primary purpose of a code review was to learn and teach, and in a sense, the measurable increase in quality was a side-effect. What I mean by that is that the review process cannot catch all bugs. It can reduce them, but it's much more valuable to share expertise on how to do things. E.g. "Did you know that if X happens, you won't be decref'ing Y? We like to use goto statements to ensure that all objects are properly refcounted even in the case of exceptional conditions." That's a teaching moment that happens to improve quality. * Reviews are conversations, and it's a two way street. Many times a dev pushed back on one of my suggestions, and we'd have weekly reviewer meetings to hash out coding standards differences. E.g. Do you check for empty sequences with "if len(foo) == 0" or "if not foo"? The team would make those decisions and you'd live by them even if you didn't agree. It's also critical to document those decisions, and a wiki page style guide works very well (especially when you can point to PEP 8 as your basic style guide for example). * Reviews are collaborations. You're both there to get the code landed so work together, not at odds. Try to reach consensus, and don't be afraid to compromise. All code is ultimately owned by everyone and you never know who will have to read it 2 years from now, so keep things simple, clear, and well commented. These are all aesthetics that a reviewer can help with. * As a reviewer ASK QUESTIONS. The best reviews were the ones that asked lots of questions, such as "have you thought about the race conditions this might introduce?" or "what happens when foo is None?" A reviewer doesn't necessarily have to guess or work out every detail. If you don't understand something, ask a question and move on. Let the coder answer it to your satisfaction. As a reviewer, once all your questions are answered, you know you can approve the change. * Keep reviews fast, easy, and fun. I think this is especially true for Python, where we're all volunteers. Keeping it fast, easy, and fun greatly improves the odds that code will be reviewed for the good of the project. * Have a dispute resolution process. If a reviewer and coder can't agree, appeal to a higher authority. As review team leader, I did occasionally have to break ties. * Record the reviewer in the commit messages. We had highly structured commit messages that included the reviewer name, e.g. % commit -m"[r=barry] Bug 12345; fix bad frobnification in Foo.bar()" thus recording in the revision history both the coder and the reviewer, so that we could always ask someone about the change. * Don't let changes get stale. We had a goal that changes would go from ready-to-code (i.e. design and implementation strategy have already been worked out) to landed in 2 days. The longer a change goes before landing, the more stale it gets, the more conflicts you can have, and the less relevant the code becomes. I know this sounds like a lot of process, but it really was fairly lightweight in practice. And that's the most important! Keep things fast, easy, and fun and it'll get done. Also, these ideas evolved after 3 years of experimentation with various approaches, so let it take some time to evolve. And don't be so married to process that you're afraid to ditch steps that are wasteful and don't contribute value to the project. Certainly some of our techniques won't be relevant for Python. For example, we assigned people to do nothing but reviews for one day out of the week (we call it "on-call reviewers"). This worked for us because team velocity was much more important than individual velocity. Even though at first blush, it seemed like you personally were going to be 20% less productive, team productivity skyrocketed because changes were landing much faster, with much less waste built in. This probably won't work for Python where our involvement is not governed by paycheck, but by the whims of our real jobs and life commitments. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From debatem1 at gmail.com Thu Sep 30 18:53:00 2010 From: debatem1 at gmail.com (geremy condra) Date: Thu, 30 Sep 2010 09:53:00 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: <20100930123352.25b34d84@mission> References: <20100930123352.25b34d84@mission> Message-ID: On Thu, Sep 30, 2010 at 9:33 AM, Barry Warsaw wrote: > On Sep 30, 2010, at 10:47 AM, Jesse Noller wrote: > >>Not to mention; there's a lot to be learned from doing them on both >>sides. At work, I learn about chunks of code I might not have >>otherwise known about or approaches to a problem I'd never considered. >>I sort of drank the kool-aid. > > Tools aside, I completely agree. > > Many projects that I contribute to have policies such as "nothing lands > without reviewer approval". ?For some that means one reviewer must approve it, > for others two +1s and no -1s, or a coding approval and a ui approval, etc. > > When I was the review team lead on Launchpad, I had a goal that every > developer would also eventually be a reviewer. ?We started with a small number > of experienced developers, then ran a mentor program to train new reviewers > (who we called "mentats"). ?A mentat approval was not enough to land a branch, > but the mentor could basically say "yes, i agree with the review" and it would > land. ?Eventually, by mutual consent a mentat would graduate to full > reviewership (and hopefully be a mentor to new reviewers). > > This was hugely successful among many dimensions. ?It got everyone on the same > page as to coding standards, it equalized the playing field, it got everyone > to think collaboratively as a team, folks learned about parts of the system > they didn't have day-to-day intimate knowledge about, and it got changes > landed much more quickly. > > Here are a few things we learned along the way. ?Their applicability to Python > will vary of course, and we need to find what works for us. > > * Keep branches *small*. ?We had a limit of 800 lines of diff, with special > ?explicit permission from the person reviewing your change to exceed. ?800 > ?lines is about the maximum that a person can review in a reasonable amount > ?of time without losing focus. > > * The *goal* was 5 minutes review, but the reality was that most reviews took > ?about 15-20 minutes. ?If it's going longer, you weren't doing it right. > ?This meant that there was a level of trust between the reviewer and the dev, > ?so that the basic design had been previously discussed and agreed upon (we > ?mandated pre-implementation chats), etc. ?A reviewer was there to sanity > ?check the implementation, watch for obvious mistakes, ensure test coverage > ?for the new code, etc. ?If you were questioning the basic design, you > ?weren't doing a code review. ?It was okay to reject a change quickly if you > ?found fatal problems. > > * The primary purpose of a code review was to learn and teach, and in a sense, > ?the measurable increase in quality was a side-effect. ?What I mean by that > ?is that the review process cannot catch all bugs. ?It can reduce them, but > ?it's much more valuable to share expertise on how to do things. ?E.g. "Did > ?you know that if X happens, you won't be decref'ing Y? ?We like to use goto > ?statements to ensure that all objects are properly refcounted even in the > ?case of exceptional conditions." ?That's a teaching moment that happens to > ?improve quality. > > * Reviews are conversations, and it's a two way street. ?Many times a dev > ?pushed back on one of my suggestions, and we'd have weekly reviewer meetings > ?to hash out coding standards differences. ?E.g. Do you check for empty > ?sequences with "if len(foo) == 0" or "if not foo"? ?The team would make > ?those decisions and you'd live by them even if you didn't agree. ?It's also > ?critical to document those decisions, and a wiki page style guide works very > ?well (especially when you can point to PEP 8 as your basic style guide for > ?example). > > * Reviews are collaborations. ?You're both there to get the code landed so > ?work together, not at odds. ?Try to reach consensus, and don't be afraid to > ?compromise. ?All code is ultimately owned by everyone and you never know who > ?will have to read it 2 years from now, so keep things simple, clear, and > ?well commented. ?These are all aesthetics that a reviewer can help with. > > * As a reviewer ASK QUESTIONS. ?The best reviews were the ones that asked lots > ?of questions, such as "have you thought about the race conditions this might > ?introduce?" or "what happens when foo is None?" ?A reviewer doesn't > ?necessarily have to guess or work out every detail. ?If you don't understand > ?something, ask a question and move on. ?Let the coder answer it to your > ?satisfaction. ?As a reviewer, once all your questions are answered, you know > ?you can approve the change. > > * Keep reviews fast, easy, and fun. ?I think this is especially true for > ?Python, where we're all volunteers. ?Keeping it fast, easy, and fun greatly > ?improves the odds that code will be reviewed for the good of the project. > > * Have a dispute resolution process. ?If a reviewer and coder can't agree, > ?appeal to a higher authority. ?As review team leader, I did occasionally > ?have to break ties. > > * Record the reviewer in the commit messages. ?We had highly structured commit > ?messages that included the reviewer name, e.g. > > ?% commit -m"[r=barry] Bug 12345; fix bad frobnification in Foo.bar()" > > ?thus recording in the revision history both the coder and the reviewer, so > ?that we could always ask someone about the change. > > * Don't let changes get stale. ?We had a goal that changes would go from > ?ready-to-code (i.e. design and implementation strategy have already been > ?worked out) to landed in 2 days. ?The longer a change goes before landing, > ?the more stale it gets, the more conflicts you can have, and the less > ?relevant the code becomes. > > I know this sounds like a lot of process, but it really was fairly lightweight > in practice. ?And that's the most important! ?Keep things fast, easy, and fun > and it'll get done. ?Also, these ideas evolved after 3 years of > experimentation with various approaches, so let it take some time to evolve. > And don't be so married to process that you're afraid to ditch steps that are > wasteful and don't contribute value to the project. > > Certainly some of our techniques won't be relevant for Python. ?For example, > we assigned people to do nothing but reviews for one day out of the week (we > call it "on-call reviewers"). ?This worked for us because team velocity was > much more important than individual velocity. ?Even though at first blush, it > seemed like you personally were going to be 20% less productive, team > productivity skyrocketed because changes were landing much faster, with much > less waste built in. ?This probably won't work for Python where our > involvement is not governed by paycheck, but by the whims of our real jobs and > life commitments. Extremely well put. Could this kind of process be put in place for the code sprints Jesse's interested in? Seems like an ideal testbed. Geremy Condra From jnoller at gmail.com Thu Sep 30 19:19:12 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 30 Sep 2010 13:19:12 -0400 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100930123352.25b34d84@mission> Message-ID: On Thu, Sep 30, 2010 at 12:53 PM, geremy condra wrote: > On Thu, Sep 30, 2010 at 9:33 AM, Barry Warsaw wrote: >> On Sep 30, 2010, at 10:47 AM, Jesse Noller wrote: >> >>>Not to mention; there's a lot to be learned from doing them on both >>>sides. At work, I learn about chunks of code I might not have >>>otherwise known about or approaches to a problem I'd never considered. >>>I sort of drank the kool-aid. >> >> Tools aside, I completely agree. >> >> Many projects that I contribute to have policies such as "nothing lands >> without reviewer approval". ?For some that means one reviewer must approve it, >> for others two +1s and no -1s, or a coding approval and a ui approval, etc. >> >> When I was the review team lead on Launchpad, I had a goal that every >> developer would also eventually be a reviewer. ?We started with a small number >> of experienced developers, then ran a mentor program to train new reviewers >> (who we called "mentats"). ?A mentat approval was not enough to land a branch, >> but the mentor could basically say "yes, i agree with the review" and it would >> land. ?Eventually, by mutual consent a mentat would graduate to full >> reviewership (and hopefully be a mentor to new reviewers). >> >> This was hugely successful among many dimensions. ?It got everyone on the same >> page as to coding standards, it equalized the playing field, it got everyone >> to think collaboratively as a team, folks learned about parts of the system >> they didn't have day-to-day intimate knowledge about, and it got changes >> landed much more quickly. >> >> Here are a few things we learned along the way. ?Their applicability to Python >> will vary of course, and we need to find what works for us. >> >> * Keep branches *small*. ?We had a limit of 800 lines of diff, with special >> ?explicit permission from the person reviewing your change to exceed. ?800 >> ?lines is about the maximum that a person can review in a reasonable amount >> ?of time without losing focus. >> >> * The *goal* was 5 minutes review, but the reality was that most reviews took >> ?about 15-20 minutes. ?If it's going longer, you weren't doing it right. >> ?This meant that there was a level of trust between the reviewer and the dev, >> ?so that the basic design had been previously discussed and agreed upon (we >> ?mandated pre-implementation chats), etc. ?A reviewer was there to sanity >> ?check the implementation, watch for obvious mistakes, ensure test coverage >> ?for the new code, etc. ?If you were questioning the basic design, you >> ?weren't doing a code review. ?It was okay to reject a change quickly if you >> ?found fatal problems. >> >> * The primary purpose of a code review was to learn and teach, and in a sense, >> ?the measurable increase in quality was a side-effect. ?What I mean by that >> ?is that the review process cannot catch all bugs. ?It can reduce them, but >> ?it's much more valuable to share expertise on how to do things. ?E.g. "Did >> ?you know that if X happens, you won't be decref'ing Y? ?We like to use goto >> ?statements to ensure that all objects are properly refcounted even in the >> ?case of exceptional conditions." ?That's a teaching moment that happens to >> ?improve quality. >> >> * Reviews are conversations, and it's a two way street. ?Many times a dev >> ?pushed back on one of my suggestions, and we'd have weekly reviewer meetings >> ?to hash out coding standards differences. ?E.g. Do you check for empty >> ?sequences with "if len(foo) == 0" or "if not foo"? ?The team would make >> ?those decisions and you'd live by them even if you didn't agree. ?It's also >> ?critical to document those decisions, and a wiki page style guide works very >> ?well (especially when you can point to PEP 8 as your basic style guide for >> ?example). >> >> * Reviews are collaborations. ?You're both there to get the code landed so >> ?work together, not at odds. ?Try to reach consensus, and don't be afraid to >> ?compromise. ?All code is ultimately owned by everyone and you never know who >> ?will have to read it 2 years from now, so keep things simple, clear, and >> ?well commented. ?These are all aesthetics that a reviewer can help with. >> >> * As a reviewer ASK QUESTIONS. ?The best reviews were the ones that asked lots >> ?of questions, such as "have you thought about the race conditions this might >> ?introduce?" or "what happens when foo is None?" ?A reviewer doesn't >> ?necessarily have to guess or work out every detail. ?If you don't understand >> ?something, ask a question and move on. ?Let the coder answer it to your >> ?satisfaction. ?As a reviewer, once all your questions are answered, you know >> ?you can approve the change. >> >> * Keep reviews fast, easy, and fun. ?I think this is especially true for >> ?Python, where we're all volunteers. ?Keeping it fast, easy, and fun greatly >> ?improves the odds that code will be reviewed for the good of the project. >> >> * Have a dispute resolution process. ?If a reviewer and coder can't agree, >> ?appeal to a higher authority. ?As review team leader, I did occasionally >> ?have to break ties. >> >> * Record the reviewer in the commit messages. ?We had highly structured commit >> ?messages that included the reviewer name, e.g. >> >> ?% commit -m"[r=barry] Bug 12345; fix bad frobnification in Foo.bar()" >> >> ?thus recording in the revision history both the coder and the reviewer, so >> ?that we could always ask someone about the change. >> >> * Don't let changes get stale. ?We had a goal that changes would go from >> ?ready-to-code (i.e. design and implementation strategy have already been >> ?worked out) to landed in 2 days. ?The longer a change goes before landing, >> ?the more stale it gets, the more conflicts you can have, and the less >> ?relevant the code becomes. >> >> I know this sounds like a lot of process, but it really was fairly lightweight >> in practice. ?And that's the most important! ?Keep things fast, easy, and fun >> and it'll get done. ?Also, these ideas evolved after 3 years of >> experimentation with various approaches, so let it take some time to evolve. >> And don't be so married to process that you're afraid to ditch steps that are >> wasteful and don't contribute value to the project. >> >> Certainly some of our techniques won't be relevant for Python. ?For example, >> we assigned people to do nothing but reviews for one day out of the week (we >> call it "on-call reviewers"). ?This worked for us because team velocity was >> much more important than individual velocity. ?Even though at first blush, it >> seemed like you personally were going to be 20% less productive, team >> productivity skyrocketed because changes were landing much faster, with much >> less waste built in. ?This probably won't work for Python where our >> involvement is not governed by paycheck, but by the whims of our real jobs and >> life commitments. > > Extremely well put. Could this kind of process be put in place for the > code sprints Jesse's interested in? Seems like an ideal testbed. > > Geremy Condra We *should* encourage this within the Sprints docs/dev guide, etc. From brian.curtin at gmail.com Thu Sep 30 19:37:02 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Thu, 30 Sep 2010 12:37:02 -0500 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100930145218.2022.353483588.divmod.xquotient.55@localhost.localdomain> Message-ID: On Thu, Sep 30, 2010 at 10:31, Daniel Stutzbach < daniel at stutzbachenterprises.com> wrote: > On Thu, Sep 30, 2010 at 9:52 AM, wrote: > >> Of course, this is only true if the core developers *do* submit to the >> same rules. Is anyone proposing that current core committers have all their >> work reviewed before it is accepted? >> > > I think most would welcome (or at least tolerate ;) ) additional review of > their code. > > The hard part is encouraging contributors to find the time and motivation > to thoroughly review code that they aren't personally interested in (and > perhaps not even familiar with). > > -- > Daniel Stutzbach, Ph.D. > President, Stutzbach Enterprises, LLC > I definitely welcome additional, or in some cases, *any* review. Looking for reviews of Windows features/bugs sometimes equates to looking in a ghost town, but I have the self-inflicted problem of using Windows in the first place ;) Anyways, a big +1 to expanding review, especially incorporating something like Rietveld. Although I'm replying out of order, Barry's big response lays out a lot of good ideas that I think we can use. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Thu Sep 30 22:46:48 2010 From: bcannon at gmail.com (Brett Cannon) Date: Thu, 30 Sep 2010 13:46:48 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100930145218.2022.353483588.divmod.xquotient.55@localhost.localdomain> Message-ID: On Thu, Sep 30, 2010 at 08:31, Daniel Stutzbach wrote: > On Thu, Sep 30, 2010 at 9:52 AM, wrote: >> >> Of course, this is only true if the core developers *do* submit to the >> same rules. ?Is anyone proposing that current core committers have all their >> work reviewed before it is accepted? > > I think most would welcome (or at least tolerate ;) ) additional review of > their code. > The hard part is encouraging contributors to find the time and motivation to > thoroughly review code that they aren't personally interested in (and > perhaps not even familiar with). Once we have a good workflow in place we would have to start shifting our development culture towards requiring a review of code no matter who the author is (which I support doing). As for needing some point system or incentive to drive people to do it, I don't think that would be necessary. We are lucky to have a dev team that is friendly and respectful. I suspect that e.g., if Nick started doing reviews of importlib patches I would reciprocate the favour by reviewing runpy patches. Those of us who don't do reviews would most likely then just end up with patches sitting in the tracker waiting for a review. From brett at python.org Thu Sep 30 22:41:49 2010 From: brett at python.org (Brett Cannon) Date: Thu, 30 Sep 2010 13:41:49 -0700 Subject: [Python-Dev] We should be using a tool for code reviews In-Reply-To: References: <20100929204125.416e2013@pitrou.net> Message-ID: On Thu, Sep 30, 2010 at 09:19, Georg Brandl wrote: > Am 29.09.2010 20:49, schrieb Guido van Rossum: > >> Unfortunately taking the average patch posted to the tracker and >> importing it in Rietveld is very iffy -- it's very hard to find the >> right branch+rev needed to be able to apply the patch correctly -- not >> to mention that there are so many (slightly) different patch formats. >> It would take a fair bit of AI to get this right. >> >> The recommended way to work with Rietveld is to use a checkout >> (fortunately with Hg this will become easier to do) and use the >> "upload.py" script that you can download from Rietveld (log in and >> click on the Create link). > > Then could we do it the other way round and let (our) upload.py open > an issue with the patch automatically (after querying for information > not already given to rietveld)? Don't see why not, but those of us who use OpenID would need to start caring about a password which would be unfortunate. -Brett > > Georg > > -- > Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. > Four shall be the number of spaces thou shalt indent, and the number of thy > indenting shall be four. Eight shalt thou not indent, nor either indent thou > two, excepting that thou then proceed to four. Tabs are right out. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From mg at lazybytes.net Thu Sep 30 23:04:05 2010 From: mg at lazybytes.net (Martin Geisler) Date: Thu, 30 Sep 2010 23:04:05 +0200 Subject: [Python-Dev] We should be using a tool for code reviews References: <20100929204125.416e2013@pitrou.net> <1285792363.3194.41.camel@localhost.localdomain> <20100929173010.2acf0b3d@mission> <20100930000420.3b7d4999@pitrou.net> <20100929183347.0d95304e@mission> <87bp7fboak.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87ocbfgcze.fsf@hbox.dyndns.org> "Stephen J. Turnbull" writes: > Barry Warsaw writes: > > > You can have "co-located" branches[1] which essentially switch > > in-place, so if a branch is changing some .c files, you won't have > > to rebuild the whole world just to try out a patch. > > In Mercurial these are called "named branches", and they are > repo-local (by which I mean they must be part of the DAG). Think of it this way: you can have several clones and when you make different commits in each, they naturally begin to diverge: clone 1: ... [X] --- [A] --- [B] clone 2: ... [X] --- [R] clone 3: ... [X] --- [U] --- [V] Though you cannot see it yet, you have created branches in the history. If you pull one clone 2 into clone 1, then the branch suddenly becomes visible in that clone -- you have two "heads" (B and R): clone 1: ... [X] --- [A] --- [B] \ [R] You can of course pull in clone 3 as well: clone 1: ... [X] --- [A] --- [B] | \ | [R] \ [U] --- [V] You can now use 'hg update' to switch between these three anonymous branches. You can find the three heads by using 'hg heads' or you can use the bookmarks extension to assign labels to them. These labels will move forward when you commit, just like you move a bookmark forward in a book when you read through it. Btw, you can separate such a repository again with 'hg clone -r' which lets you ask for a clone containing just a subset of another repository. You can of course also split off named branches (see below) in this way. > Named branches used to have some inconvenient aspects relevant to > standalone branches (they could be fairly confusing to other users if > pushed before being merge to mainline). Now, "named branches" is sort of an extension of the above scenario. Along with the commit message, date, username, etc..., each changeset in Mercurial contains a label that tells us which "named branch" it belongs to. By default, changesets are on the branch called, well, "default". Named branches allow you to group or namespace your changesets. For Mercurial itself, we use two named branches: default and stable. Bugfixes go on the stable branch and we do our development on the default branch. Though a named branch is a collection of changesets, you often refer to a named branch in a context where you need just a single changeset. It is then interpreted as the tip-most changeset with that branch name. So $ hg update stable will checkout the latest (tip-most) changeset on the stable branch. If you now commit a bugfix, then that changeset will also be on the stable branch since the branch name is inherited from the parent changeset. In Mercurial we then merge the stable branch back into the default branch: # fix bug $ hg commit -m 'Fixed issue 123' $ hg update default $ hg merge stable $ hg commit -m 'Merged in fix for issue123' I've written a guide to such a workflow here: http://mercurial.aragost.com/kick-start/tasks.html > It's not obvious to me that Mercurial style named branches would work > well here ... it would take a little thought to design an appropriate > workflow, anyway. Yeah, you guys should try it out first -- it's easy to introduce named branches later if you want. -- Martin Geisler Mercurial links: http://mercurial.ch/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: