From greg.ewing at canterbury.ac.nz Mon Mar 1 00:22:19 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 01 Mar 2010 12:22:19 +1300 Subject: [Python-Dev] __file__ In-Reply-To: <4B89F5FA.2010604@g.nevcal.com> References: <20100130190005.058c8187@freewill.wooz.org> <20100225160832.6e7a3063@freewill.wooz.org> <4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk> <4B871227.2030707@canterbury.ac.nz> <4B885AB4.7040004@g.nevcal.com> <4B889AB1.4050402@g.nevcal.com> <4B89C5FB.40104@canterbury.ac.nz> <4B89F5FA.2010604@g.nevcal.com> Message-ID: <4B8AFAAB.5020106@canterbury.ac.nz> Glenn Linderman wrote: > if the > command line/runpy can do it, the importer could do it. Just a matter > of desire and coding. Whether it is worth pursuing further depends on > people's perceptions of "kookiness" vs. functional and performance > considerations. Having .py files around that aren't source text could lead to a lot of confusion, given that most platforms these days decide which application to open for a given file based solely on the filename extension. I wouldn't enjoy trying to open a .py file only to have my text editor blow up because it was actually a binary file. So on balance I think it's a bit too kooky for my taste. -- Greg From floris.bruynooghe at gmail.com Mon Mar 1 00:14:41 2010 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Sun, 28 Feb 2010 23:14:41 +0000 Subject: [Python-Dev] __file__ In-Reply-To: References: <4B871227.2030707@canterbury.ac.nz> <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> <4B8A6A8F.7050008@gmail.com> Message-ID: <20100228231441.GA18979@laurie.devork> On Sun, Feb 28, 2010 at 09:45:56PM +0100, Baptiste Carvello wrote: > However, making a difference between zipimport and the filesystem > importer means the application will stop working if I unzip the > library zip file, which is surprising. Unzipping the zip file can be > handy when debugging a bug caused by a forgotten module. That difference exists already, the zipimporter will happily run .pyo files inside the zipfile even when you're not running with -O or PYTHONOPTIMIZE. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From greg.ewing at canterbury.ac.nz Mon Mar 1 00:35:17 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 01 Mar 2010 12:35:17 +1300 Subject: [Python-Dev] __file__ In-Reply-To: <20100228121907.GA13564@laurie.devork> References: <20100225160832.6e7a3063@freewill.wooz.org> <4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk> <4B871227.2030707@canterbury.ac.nz> <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> Message-ID: <4B8AFDB5.8040302@canterbury.ac.nz> Floris Bruynooghe wrote: > Can't it look for a .py file in the source directory first (1st stat)? > When it's there check for the .pyc in the cache directory (2nd stat, > magic number encoded in filename), if it's not check for .pyc in the > source directory (2nd stat + read for magic number check). Yes, although that would then incur higher stat overheads for people distributing .pyc files. There doesn't seem to be a way of pleasing everyone. This is all assuming that the extra stat calls are actually a problem. Does anyone have any evidence that they would really take significant time compared to loading the module? Once you've looked for one file in a given directory, looking for another one in the same directory ought to be quite fast, since all the relevant directory blocks will be in the filesystem cache. -- Greg From florent.xicluna at gmail.com Mon Mar 1 00:36:27 2010 From: florent.xicluna at gmail.com (Florent XICLUNA) Date: Mon, 1 Mar 2010 00:36:27 +0100 Subject: [Python-Dev] Update xml.etree.ElementTree for Python 2.7 and 3.2 In-Reply-To: References: <4B7DB541.4000604@v.loewis.de> <4B7E2430.2070804@v.loewis.de> <4B7F95FE.20703@v.loewis.de> <4B7FD0C7.4050207@v.loewis.de> Message-ID: 2010/2/28 Stefan Behnel > I would actually encourage Florent to do the opposite: act now and prepare > a patch against the latest official ET 1.2 and cET releases (or their SVN > version respectively) that integrates everything that is considered safe, > i.e. everything that makes cET compatible with ET and everything that seems > clearly stable in ET 1.3 and does not break compatibility for existing code > that uses ET 1.2. If you send that to Fredrik, I expect little opposition > to making that the base for a 1.2.8 release, which can then be folded back > into the stdlib. > > I exchanged some e-mails with Fredrik last week. Not sure if it will be 1.2.8 or 1.3, but now he is positive on the goals of the patch. I've commited all the changes and external fixes to a branch of the Mercurial repo owned by Fredrik. I'm expecting an answer soon. Branch based on the official etree repository (Mercurial): http://bitbucket.org/flox/et-2009-provolone/ Patch based on this branch: http://codereview.appspot.com/207048 (patch set 7 almost identical to the tip of the Mercurial repo) -- Florent -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Mon Mar 1 00:56:20 2010 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 01 Mar 2010 10:56:20 +1100 Subject: [Python-Dev] __file__ In-Reply-To: <4B8AFDB5.8040302@canterbury.ac.nz> References: <20100225160832.6e7a3063@freewill.wooz.org> <4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk> <4B871227.2030707@canterbury.ac.nz> <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <4B8AFDB5.8040302@canterbury.ac.nz> Message-ID: <1267401380.25053.116.camel@lifeless-64> On Mon, 2010-03-01 at 12:35 +1300, Greg Ewing wrote: > > Yes, although that would then incur higher stat overheads for > people distributing .pyc files. There doesn't seem to be a > way of pleasing everyone. > > This is all assuming that the extra stat calls are actually > a problem. Does anyone have any evidence that they would > really take significant time compared to loading the module? > Once you've looked for one file in a given directory, looking > for another one in the same directory ought to be quite fast, > since all the relevant directory blocks will be in the > filesystem cache. We've done a bunch of testing in bzrlib. Basic things are: - statting /is/ expensive *if* you don't use the result. - loading code is the main cost *once* you have a hot disk cache Specifically, stats for files that are *not present* incur page-in costs for the dentries needed to determine the file is absent. In the special case of probing for $name.$ext1, ...$ext2, ...$ext3, you generally hit the same pages and don't incur additional page in costs. (you'll hit the same page in most file systems when you look for the second and third entries). In most file systems stats for files that *are present* also incur a page-in for the inode of the file. If you then do not read the file, this is I/O that doesn't really gain anything. Being able to disable .py file usage completely - so that only foo.pyc and foo/__init__.pyc are probed for, could have a very noticable change in the cold cache startup time. # Startup time for bzr (cold cache): $ drop-caches $ time bzr --no-plugins revno 5061 real 0m8.875s user 0m0.210s sys 0m0.140s # Hot cache $ time bzr --no-plugins revno 5061 real 0m0.307s user 0m0.250s sys 0m0.040s (revno is a small command that reads a small amount of data - just enough to trigger demand loading of the core repository layers and so on). strace timings for those two operations: cold cache: $ strace -c bzr --no-plugins revno 5061 % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 56.34 0.040000 76 527 read 28.98 0.020573 9 2273 1905 open 14.43 0.010248 14 734 625 stat 0.15 0.000107 0 533 fstat ... hot cache: % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 45.10 0.000368 92 4 getdents 19.49 0.000159 0 527 read 16.91 0.000138 1 163 munmap 10.05 0.000082 2 54 mprotect 8.46 0.000069 0 2273 1905 open 0.00 0.000000 0 8 write 0.00 0.000000 0 367 close 0.00 0.000000 0 734 625 stat ... Cheers, Rob -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From greg.ewing at canterbury.ac.nz Mon Mar 1 01:31:17 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 01 Mar 2010 13:31:17 +1300 Subject: [Python-Dev] __file__ In-Reply-To: <1267401380.25053.116.camel@lifeless-64> References: <20100225160832.6e7a3063@freewill.wooz.org> <4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk> <4B871227.2030707@canterbury.ac.nz> <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <4B8AFDB5.8040302@canterbury.ac.nz> <1267401380.25053.116.camel@lifeless-64> Message-ID: <4B8B0AD5.4090606@canterbury.ac.nz> Robert Collins wrote: > In the special > case of probing for $name.$ext1, ...$ext2, ...$ext3, you generally hit > the same pages and don't incur additional page in costs. So then looking for a .pyc alongside a .py or vice versa should be almost free, and we shouldn't be worrying about it. > hot cache: > % time seconds usecs/call calls errors syscall > ------ ----------- ----------- --------- --------- ---------------- > 45.10 0.000368 92 4 getdents > 0.00 0.000000 0 734 625 stat Further supporting the idea that stat calls are negligible once the cache is warmed up. -- Greg From pmaupin at gmail.com Mon Mar 1 01:25:39 2010 From: pmaupin at gmail.com (Patrick Maupin) Date: Sun, 28 Feb 2010 18:25:39 -0600 Subject: [Python-Dev] Draft PEP on RSON configuration file format Message-ID: All: Finding .ini configuration files too limiting, JSON and XML to hard to manually edit, and YAML too complex to parse quickly, I have started work on a new configuration file parser. I call the new format RSON (for "Readable Serial Object Notation"), and it is designed to be a superset of JSON. I would love for it to be considered valuable enough to be a part of the standard library, but even if that does not come to pass, I would be very interested in feedback to help me polish the specification, and then possibly help for implementation and testing. The documentation is in rst PEP form, at: http://rson.googlecode.com/svn/trunk/doc/draftpep.txt Thanks and best regards, Pat From benjamin at python.org Mon Mar 1 01:29:48 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 28 Feb 2010 18:29:48 -0600 Subject: [Python-Dev] Draft PEP on RSON configuration file format In-Reply-To: References: Message-ID: <1afaf6161002281629m5b270b40j51ae966cecb09f4e@mail.gmail.com> 2010/2/28 Patrick Maupin : > All: > > Finding .ini configuration files too limiting, JSON and XML to hard to > manually edit, and YAML too complex to parse quickly, I have started > work on a new configuration file parser. In that case, it should live in the user space for several years. If the community decides that it is an excellent format, then it should be considered for inclusion in the stand library. -- Regards, Benjamin From pmaupin at gmail.com Mon Mar 1 01:59:16 2010 From: pmaupin at gmail.com (Patrick Maupin) Date: Sun, 28 Feb 2010 18:59:16 -0600 Subject: [Python-Dev] Draft PEP on RSON configuration file format In-Reply-To: <1afaf6161002281629m5b270b40j51ae966cecb09f4e@mail.gmail.com> References: <1afaf6161002281629m5b270b40j51ae966cecb09f4e@mail.gmail.com> Message-ID: On Sun, Feb 28, 2010 at 6:29 PM, Benjamin Peterson wrote: > In that case, it should live in the user space for several years. If > the community decides that it is an excellent format, then it should > be considered for inclusion in the stand library. Agreed. However, there are too many things which became de facto standards without community input this way. PEP 1 itself says: Reference Implementation -- The reference implementation must be completed before any PEP is given status "Final", but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. So, I do not mind the code sitting outside the standard library, and the PEP not reaching "Final" for several years, but I do believe that the PEP process is itself a really good way to build a better mousetrap by consensus. If you do not care to participate in the building of this particular mousetrap, that is OK, too. Regards, Pat From solipsis at pitrou.net Mon Mar 1 02:39:45 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 28 Feb 2010 20:39:45 -0500 Subject: [Python-Dev] Draft PEP on RSON configuration file format References: <1afaf6161002281629m5b270b40j51ae966cecb09f4e@mail.gmail.com> Message-ID: <20100228203945.62e1ea4b@msiwind> Le Sun, 28 Feb 2010 18:59:16 -0600, Patrick Maupin a ?crit : > > So, I do not mind the code sitting outside the standard library, and > the PEP not reaching "Final" for several years, but I do believe that > the PEP process is itself a really good way to build a better > mousetrap by consensus. In this case it is *at best* python-ideas material, or even preferably comp.lang.python. Just for the record, my only reaction when giving the PEP a glance was "yet another configuration file format - yawn". Good luck though, Antoine. From pmaupin at gmail.com Mon Mar 1 02:46:30 2010 From: pmaupin at gmail.com (Patrick Maupin) Date: Sun, 28 Feb 2010 19:46:30 -0600 Subject: [Python-Dev] Draft PEP on RSON configuration file format In-Reply-To: <20100228203945.62e1ea4b@msiwind> References: <1afaf6161002281629m5b270b40j51ae966cecb09f4e@mail.gmail.com> <20100228203945.62e1ea4b@msiwind> Message-ID: On Sun, Feb 28, 2010 at 7:39 PM, Antoine Pitrou wrote: > In this case it is *at best* python-ideas material, or even > preferably comp.lang.python. I was thinking about comp.lang.python at some point, but thought I would try here first. > Just for the record, my only reaction when giving the PEP a glance was > "yet another configuration file format - yawn". I suppose I have that sort of reaction about areas I am not interested in, as well, but currently I am deeply interested in configuration files due to my circumstances. In any case, the observation that there are already several preexisting file formats used for configuration is certainly covered in the PEP draft, but if you have anything constructive to add *about* configuration file formats, I would certainly welcome the input. Best regards, Pat From solipsis at pitrou.net Mon Mar 1 02:51:20 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 28 Feb 2010 20:51:20 -0500 Subject: [Python-Dev] Draft PEP on RSON configuration file format References: <1afaf6161002281629m5b270b40j51ae966cecb09f4e@mail.gmail.com> <20100228203945.62e1ea4b@msiwind> Message-ID: <20100228205120.6ac82f01@msiwind> Le Sun, 28 Feb 2010 19:46:30 -0600, Patrick Maupin a ?crit : > > I suppose I have that sort of reaction about areas I am not interested > in, as well, but currently I am deeply interested in configuration > files due to my circumstances. In any case, the observation that > there are already several preexisting file formats used for > configuration is certainly covered in the PEP draft, but if you have > anything constructive to add *about* configuration file formats, I > would certainly welcome the input. Well, a constructive approach would involve approaching projects which have devised their own formats, so as to know what kind of unified format they would be likely to accept (or not). python-dev is probably not the place for such an approach, however. From brett at python.org Mon Mar 1 03:11:39 2010 From: brett at python.org (Brett Cannon) Date: Sun, 28 Feb 2010 18:11:39 -0800 Subject: [Python-Dev] __file__ In-Reply-To: <4B8B0AD5.4090606@canterbury.ac.nz> References: <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <4B8AFDB5.8040302@canterbury.ac.nz> <1267401380.25053.116.camel@lifeless-64> <4B8B0AD5.4090606@canterbury.ac.nz> Message-ID: On Sun, Feb 28, 2010 at 16:31, Greg Ewing wrote: > Robert Collins wrote: > >> In the special >> case of probing for $name.$ext1, ...$ext2, ...$ext3, you generally hit >> the same pages and don't incur additional page in costs. >> > > So then looking for a .pyc alongside a .py or vice versa > should be almost free, and we shouldn't be worrying about > it. > But that is making the assumption that all filesystems operate this way (.e.g does NFS have the same performance characteristics?). > > hot cache: >> % time seconds usecs/call calls errors syscall >> ------ ----------- ----------- --------- --------- ---------------- >> 45.10 0.000368 92 4 getdents >> 0.00 0.000000 0 734 625 stat >> > > Further supporting the idea that stat calls are negligible > once the cache is warmed up. But that's the point: once it's warmed up. This is not the case when executing a script once every once in a while compared to something bzr where you are most likely going to execute the command multiple times within a small timeframe. -Brett > > > -- > Greg > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Mar 1 03:27:04 2010 From: brett at python.org (Brett Cannon) Date: Sun, 28 Feb 2010 18:27:04 -0800 Subject: [Python-Dev] __file__ In-Reply-To: <4B8AD627.60103@gmail.com> References: <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> <4B8A6A8F.7050008@gmail.com> <4B8AD627.60103@gmail.com> Message-ID: On Sun, Feb 28, 2010 at 12:46, Nick Coghlan wrote: > Brett Cannon wrote: > > Actually it's four: name/__init__.py, name/__init__.pyc, name.py, and > > then name.pyc. And just so people have terminology to go with all of > > this, this search is what the finder does to say whether it can or > > cannot handle the requested module. > > Huh, I thought we checked for the directory first and only then checked > for the __init__ module within it (hence the generation of ImportWarning > when we don't find __init__ after finding a correctly named directory). > So a normal miss (i.e. no directory) only needs one stat call. > > (However, I'll grant that I haven't looked at this particular chunk of > code in a fairly long time, so I could easily be wrong). > > Robert raises a good point about the checks for extension modules as > well - we should get an accurate count here so Barry's PEP can pitch the > proportional reduction in stat calls accurately. > Here are the details (from Python/import.c:find_module) assuming that everything has failed to the point of trying for the implicit sys.path importers: stat_info = stat(name) if stat_info.exists and stat_info.is_dir: if stat(name/__init__.py) || stat(name/__init__.pyc): load(name) else: for ext in ('.so', 'module.so', '.py', 'pyc'): # Windows has an extra check for .pyw files. if open(name + ext): load(name) So there are a total of five to six depending on the OS (actually, VMS goes up to eight!) before a search path is considered not to contain a module. And thanks to doing this I realized importlib is not stat'ing the directory first which should fail faster than checking for the __init__ files every time. -Brett > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > --------------------------------------------------------------- > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Mar 1 03:29:42 2010 From: brett at python.org (Brett Cannon) Date: Sun, 28 Feb 2010 18:29:42 -0800 Subject: [Python-Dev] __file__ In-Reply-To: References: <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> <4B8A6A8F.7050008@gmail.com> Message-ID: On Sun, Feb 28, 2010 at 12:45, Baptiste Carvello wrote: > Nick Coghlan a ?crit : > > >> Another option is to remove bytecode-only support from the default >> filesystem importer, but keep it for zipimport (since the stat call >> savings don't apply in the latter case). >> >> > bytecode-only in a zip is used by py2exe, cx_freeze and the like, for space > reasons. Disabling it would probably hurt them. > > However, making a difference between zipimport and the filesystem importer > means the application will stop working if I unzip the library zip file, > which is surprising. Unzipping the zip file can be handy when debugging a > bug caused by a forgotten module. > > Is it really that hard to unzip a bunch of .pyc files, modify what you need to, and then zip it back up? And if you are given a zip file of only .pyc files you can't really debug anything anyway. -Brett > Cheers, > Baptiste > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Mar 1 03:31:54 2010 From: brett at python.org (Brett Cannon) Date: Sun, 28 Feb 2010 18:31:54 -0800 Subject: [Python-Dev] The fate of Distutils in Python 2.7 In-Reply-To: <94bdd2611002261415p5342b6dbv6a4a87ba0f4c7700@mail.gmail.com> References: <94bdd2611002261344q361e702av10fdf99ea5eb9f71@mail.gmail.com> <94bdd2611002261415p5342b6dbv6a4a87ba0f4c7700@mail.gmail.com> Message-ID: On Fri, Feb 26, 2010 at 14:15, Tarek Ziad? wrote: > On Fri, Feb 26, 2010 at 11:13 PM, Brett Cannon wrote: > [..] > > I assume you want the Distutils2 component to auto-assign to you like > > Distutils currently does? If so I can add the component for you if people > > don't object to the new component. > > Sounds good -- Thanks > Done. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From pmaupin at gmail.com Mon Mar 1 03:33:05 2010 From: pmaupin at gmail.com (Patrick Maupin) Date: Sun, 28 Feb 2010 20:33:05 -0600 Subject: [Python-Dev] Draft PEP on RSON configuration file format In-Reply-To: <20100228205120.6ac82f01@msiwind> References: <1afaf6161002281629m5b270b40j51ae966cecb09f4e@mail.gmail.com> <20100228203945.62e1ea4b@msiwind> <20100228205120.6ac82f01@msiwind> Message-ID: On Sun, Feb 28, 2010 at 7:51 PM, Antoine Pitrou wrote: > Well, a constructive approach would involve approaching projects > which have devised their own formats, so as to know what kind of > unified format they would be likely to accept (or not). Trying to poll "selected projects which have configuration files" may or may not be a constructive approach. Most projects which have predefined formats are unlikely to change, unless there is standardization on a new format. It is very much a chicken and egg problem, although I agree with (and have implemented) the suggestion that I discuss this on python-list. Having said that, one of the reasons I wrote the PEP and am working on a parser is because of a few projects I use and/or am personally involved in. For example, rst2pdf stylesheets are in JSON, e.g. http://rst2pdf.googlecode.com/svn/trunk/rst2pdf/styles/styles.json Now, we're all programmers here, and we can read this, and can even modify it, but it is easy to get wrong, and very verbose with lots of syntax gotchas. For example, unlike Python, JSON won't even let you have a trailing comma. But JSON *is* a great format, and RSON (like YAML) is designed to parse properly formatted JSON, so the goal is that any project which uses JSON could use RSON as a drop-in replacement, and then update its configuration data. Of course, it is extremely easy (hence your yawn) to create a new configuration format, even if it is specified that it is upwards compatible with JSON. The trick is to create the *correct* new format, that at least some people can agree on. In order to do this, I have chosen to poll, not preexisting projects, which have entrenched configuration data and a reluctance to change, but brand new projects which haven't been invented yet. Many of the inventors of those projects hang out on python-dev, so this seemed like a reasonable place to do polling. As I tried to make clear, I will not be too disappointed if I do not come up with something worthy of the standard library for a long time (if ever), but the PEP process is very valuable, and I would like to start off on the right foot by soliciting feedback before I do too much coding. Sorry if it feels like spam; this is my last message on the matter until and unless somebody wants to constructively discuss the actual contents of the PEP. Please feel free to email me privately if you don't want to clutter up this list. Thanks and best regards, Pat From v+python at g.nevcal.com Mon Mar 1 04:32:09 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sun, 28 Feb 2010 19:32:09 -0800 Subject: [Python-Dev] __file__ In-Reply-To: <4B8AFAAB.5020106@canterbury.ac.nz> References: <20100130190005.058c8187@freewill.wooz.org> <20100225160832.6e7a3063@freewill.wooz.org> <4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk> <4B871227.2030707@canterbury.ac.nz> <4B885AB4.7040004@g.nevcal.com> <4B889AB1.4050402@g.nevcal.com> <4B89C5FB.40104@canterbury.ac.nz> <4B89F5FA.2010604@g.nevcal.com> <4B8AFAAB.5020106@canterbury.ac.nz> Message-ID: <4B8B3539.5060709@g.nevcal.com> On approximately 2/28/2010 3:22 PM, came the following characters from the keyboard of Greg Ewing: > Glenn Linderman wrote: >> if the command line/runpy can do it, the importer could do it. Just >> a matter of desire and coding. Whether it is worth pursuing further >> depends on people's perceptions of "kookiness" vs. functional and >> performance considerations. > > Having .py files around that aren't source text could > lead to a lot of confusion, given that most platforms > these days decide which application to open for a given > file based solely on the filename extension. I wouldn't > enjoy trying to open a .py file only to have my text > editor blow up because it was actually a binary file. > > So on balance I think it's a bit too kooky for my > taste. I understand your thoughts, but have some rebuttal comments. Mind you, if there is a better solution that can improve performance for both the source+binary and the binary-only distributions, I'm all for it. But in general, I'm all for performance improvements, even if there is some kookiness :) Thankful for Brett's posting of the actual search code fragment. If your text editor blows up because it is binary, it is a sad text editor. If you have .py mapped to a text editor, that's sort of kooky too; I have it mapped to Python. The .py files that are binary would generally be part of an application distribution in binary form, and therefore would be installed in some place like /bin or C:\Program Files ... not the place you'd look for source code, to confuse your text editor. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From solipsis at pitrou.net Mon Mar 1 04:40:20 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 28 Feb 2010 22:40:20 -0500 Subject: [Python-Dev] __file__ References: <20100130190005.058c8187@freewill.wooz.org> <20100225160832.6e7a3063@freewill.wooz.org> <4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk> <4B871227.2030707@canterbury.ac.nz> <4B885AB4.7040004@g.nevcal.com> <4B889AB1.4050402@g.nevcal.com> <4B89C5FB.40104@canterbury.ac.nz> <4B89F5FA.2010604@g.nevcal.com> <4B8AFAAB.5020106@canterbury.ac.nz> <4B8B3539.5060709@g.nevcal.com> Message-ID: <20100228224020.46da4e23@msiwind> Le Sun, 28 Feb 2010 19:32:09 -0800, Glenn Linderman a ?crit : > > If your text editor blows up because it is binary, it is a sad text > editor. > > If you have .py mapped to a text editor, that's sort of kooky too; I > have it mapped to Python. File extensions exist for a reason, even if you find that "kooky" and have strong ideas about the psychology of text editors. Having some binary files named "foobar.py" would certainly annoy a lot of people, including me. Antoine. From greg.ewing at canterbury.ac.nz Mon Mar 1 06:21:52 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 01 Mar 2010 18:21:52 +1300 Subject: [Python-Dev] __file__ In-Reply-To: <4B8B3539.5060709@g.nevcal.com> References: <20100130190005.058c8187@freewill.wooz.org> <20100225160832.6e7a3063@freewill.wooz.org> <4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk> <4B871227.2030707@canterbury.ac.nz> <4B885AB4.7040004@g.nevcal.com> <4B889AB1.4050402@g.nevcal.com> <4B89C5FB.40104@canterbury.ac.nz> <4B89F5FA.2010604@g.nevcal.com> <4B8AFAAB.5020106@canterbury.ac.nz> <4B8B3539.5060709@g.nevcal.com> Message-ID: <4B8B4EF0.5020204@canterbury.ac.nz> Glenn Linderman wrote: > If your text editor blows up because it is binary, it is a sad text editor. Blow up is probably an exaggeration, but even just getting a screen full of gibberish when I think I'm opening a text file is a jarring experience. > If you have .py mapped to a text editor, that's sort of kooky too; I > have it mapped to Python. On Windows the action for double-clicking is usually mapped to running the file, but there's typically another action such as "Open with IDLE" or whatever available, and a bytecode file named with ".py" would allow you to apply that action to it. -- Greg From larry.bugbee at boeing.com Mon Mar 1 07:09:00 2010 From: larry.bugbee at boeing.com (Bugbee, Larry) Date: Sun, 28 Feb 2010 22:09:00 -0800 Subject: [Python-Dev] __file__ In-Reply-To: References: Message-ID: <261064E43172924BBADB00DA14BB4B672A18782E13@XCH-NW-14V.nw.nos.boeing.com> Greg Ewing wrote: > Having .py files around that aren't source text could lead > to a lot of confusion, given that most platforms these days > decide which application to open for a given file based > solely on the filename extension. I wouldn't enjoy trying > to open a .py file only to have my text editor blow up > because it was actually a binary file. > > So on balance I think it's a bit too kooky for my taste. +1 Add to that the inverse... I will cleanup directories based on the suffix keeping the .py and deleting .pyc and .pyo. Overloading a source file suffix is not good. Larry From baptiste13z at free.fr Mon Mar 1 08:25:08 2010 From: baptiste13z at free.fr (Baptiste Carvello) Date: Mon, 01 Mar 2010 08:25:08 +0100 Subject: [Python-Dev] __file__ In-Reply-To: References: <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> <4B8A6A8F.7050008@gmail.com> Message-ID: Brett Cannon a ?crit : > > However, making a difference between zipimport and the filesystem > importer means the application will stop working if I unzip the > library zip file, which is surprising. Unzipping the zip file can be > handy when debugging a bug caused by a forgotten module. > > > Is it really that hard to unzip a bunch of .pyc files, modify what you > need to, and then zip it back up? And if you are given a zip file of > only .pyc files you can't really debug anything anyway. > Well, this is a micro-use-case, I admit, I only mention it because it's something I've really done. It's only useful for debugging the building process, not the application (so I do have the source at hand), and the only reason for not rezipping is to test more quickly. I can definitely live without it! Cheers, Baptiste From stefan_ml at behnel.de Mon Mar 1 08:32:02 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 01 Mar 2010 08:32:02 +0100 Subject: [Python-Dev] Update xml.etree.ElementTree for Python 2.7 and 3.2 In-Reply-To: References: <4B7DB541.4000604@v.loewis.de> <4B7E2430.2070804@v.loewis.de> <4B7F95FE.20703@v.loewis.de> <4B7FD0C7.4050207@v.loewis.de> Message-ID: Florent XICLUNA, 01.03.2010 00:36: > I exchanged some e-mails with Fredrik last week. Not sure if it will be > 1.2.8 or 1.3, but now he is positive on the goals of the patch. I've > commited all the changes and external fixes to a branch of the Mercurial > repo owned by Fredrik. I'm expecting an answer soon. Happy to hear that. Thanks for putting so much work into this! > Branch based on the official etree repository (Mercurial): > http://bitbucket.org/flox/et-2009-provolone/ Interesting, I didn't even know Fredrik had continued to work on this. It even looks like lxml.etree has a bit to catch up API-wise before I release 2.3. Stefan From robertc at robertcollins.net Mon Mar 1 09:04:59 2010 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 01 Mar 2010 19:04:59 +1100 Subject: [Python-Dev] __file__ In-Reply-To: References: <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <4B8AFDB5.8040302@canterbury.ac.nz> <1267401380.25053.116.camel@lifeless-64> <4B8B0AD5.4090606@canterbury.ac.nz> Message-ID: <1267430699.9222.23.camel@lifeless-64> On Sun, 2010-02-28 at 18:11 -0800, Brett Cannon wrote: ... > So then looking for a .pyc alongside a .py or vice versa > should be almost free, and we shouldn't be worrying about > it. > But that is making the assumption that all filesystems operate this > way (.e.g does NFS have the same performance characteristics?). NFS doesn't cache pages, rather it caches individual entries. I do not know if adjacent data is pre-populated into an NFS client cache. I rather suspect not, but the general point is true - many filesystems cache by pages, not all. And some have unsorted lists or might have hash tables rather than b-trees or sorted lists, where locality of reference doesn't help at all. (VFAT is unsorted list, IIRC). > hot cache: > % time seconds usecs/call calls errors > syscall > ------ ----------- ----------- --------- --------- > ---------------- > 45.10 0.000368 92 4 > getdents > > 0.00 0.000000 0 734 625 stat > > > Further supporting the idea that stat calls are negligible > once the cache is warmed up. > > > But that's the point: once it's warmed up. This is not the case when > executing a script once every once in a while compared to something > bzr where you are most likely going to execute the command multiple > times within a small timeframe. bzr would /love/ cold cache times to come down. They are one of the most glaring performance differences between a large C program, and a large python program. Even though the second run is fast, it hurts the first time you run 'bzr st'. (**) -Rob (**) To the extent that I've seriously considered an import hook to disable normal imports under bzrlib, and special case how the search works so that we only load pyc's and assume all imports are absolute. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From baptiste13z at free.fr Mon Mar 1 09:09:09 2010 From: baptiste13z at free.fr (Baptiste Carvello) Date: Mon, 01 Mar 2010 09:09:09 +0100 Subject: [Python-Dev] __file__ In-Reply-To: References: <20100225160832.6e7a3063@freewill.wooz.org> <4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk> <4B871227.2030707@canterbury.ac.nz> <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> <4B8A6A8F.7050008@gmail.com> Message-ID: Antoine Pitrou a ?crit : > Le Sun, 28 Feb 2010 21:45:56 +0100, Baptiste Carvello a ?crit : >> bytecode-only in a zip is used by py2exe, cx_freeze and the like, for >> space reasons. Disabling it would probably hurt them. > > Source code compresses quite well. I'm not sure it would make much of a > difference. I did a quick check on the stdlib: a zip with .py and .pyc is about 80% bigger than one with .pyc only. If you use only the bytecode, this can be seen as waisted space. On the other hand, if you ever need to debug the application, source is very handy... Anyway, I'm a bit worried if bytecode-only is disabled from zipimport without some input from the developpers of py2exe/cx_freeze/etc, as they are big users of it. Cheers, Baptiste From fetchinson at googlemail.com Mon Mar 1 10:02:18 2010 From: fetchinson at googlemail.com (Daniel Fetchinson) Date: Mon, 1 Mar 2010 10:02:18 +0100 Subject: [Python-Dev] Draft PEP on RSON configuration file format In-Reply-To: References: Message-ID: > Finding .ini configuration files too limiting, JSON and XML to hard to > manually edit [snip] > I call the new format RSON (for "Readable Serial Object Notation"), > and it is designed to be a superset of JSON. Quick question: if JSON is too hard to manually edit, how can RSON be any easier when it is a *superset* of JSON? Cheers, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From ncoghlan at gmail.com Mon Mar 1 14:04:11 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 01 Mar 2010 23:04:11 +1000 Subject: [Python-Dev] __file__ In-Reply-To: References: <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> <4B8A6A8F.7050008@gmail.com> <4B8AD627.60103@gmail.com> Message-ID: <4B8BBB4B.7030900@gmail.com> Brett Cannon wrote: > So there are a total of five to six depending on the OS (actually, VMS > goes up to eight!) before a search path is considered not to contain a > module. The windows list is actually going to be slightly different (dir, pyd, py, pyw, py[co]). It looks for .pyd files rather than either flavour of .so file (we stopped allowing .dll files some time back due to the sqlite3 DLL naming conflict). So I believe it is always 5 stat calls on the major platform - dropping the bytecode filename check saves 20% of them on a miss. I'm not convinced that saving is worth the hassle of incurring a whole pile of subtle backward compatibility problems. It seems better to say "Python doesn't create in-place bytecode files anymore, but if you arrange to put them there yourself we'll still read them". I certainly wouldn't support removing the feature without some solid benchmarks to say that it really is going to significantly speed up typical import times for non-trivial modules (given that part of the definition of "non-trivial" is "significant amounts of code to be run when imported", that bar is likely to be a tough one to clear). > And thanks to doing this I realized importlib is not stat'ing the > directory first which should fail faster than checking for the __init__ > files every time. That would explain why we had different ideas as to what the interpreter was doing :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Mon Mar 1 15:21:21 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 1 Mar 2010 09:21:21 -0500 Subject: [Python-Dev] __file__ References: <20100225160832.6e7a3063@freewill.wooz.org> <4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk> <4B871227.2030707@canterbury.ac.nz> <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> <4B8A6A8F.7050008@gmail.com> Message-ID: <20100301092121.5882a882@msiwind> Le Mon, 01 Mar 2010 09:09:09 +0100, Baptiste Carvello a ?crit : > > I did a quick check on the stdlib: a zip with .py and .pyc is about > 80% bigger than one with .pyc only. If you use only the bytecode, > this can be seen as waisted space. On the other hand, if you ever > need to debug the application, source is very handy... My point is that the wasted size compared to the total bundle size (with interpreter and 3rd party C libs) would probably be small. > Anyway, I'm a bit worried if bytecode-only is disabled from zipimport > without some input from the developpers of py2exe/cx_freeze/etc, as > they are big users of it. Granted. Regards Antoine. From pmaupin at gmail.com Mon Mar 1 16:14:16 2010 From: pmaupin at gmail.com (Patrick Maupin) Date: Mon, 1 Mar 2010 09:14:16 -0600 Subject: [Python-Dev] Draft PEP on RSON configuration file format In-Reply-To: References: Message-ID: On Mon, Mar 1, 2010 at 3:02 AM, Daniel Fetchinson wrote: > Quick question: if JSON is too hard to manually edit, how can RSON be > any easier when it is a *superset* of JSON? > Well, Python is essentially a superset of JSON, with string escape handling being ever so slightly different, and using True instead of true, False instead of false, and None instead of null. YMMV, but I find it possible, even probable, to write Python that is far easier to edit than JSON, and in fact, I have used Python for configuration files that are only to be edited by programmers or other technical types. Regards, Pat -------------- next part -------------- An HTML attachment was scrubbed... URL: From rrr at ronadam.com Mon Mar 1 17:30:04 2010 From: rrr at ronadam.com (Ron Adam) Date: Mon, 01 Mar 2010 10:30:04 -0600 Subject: [Python-Dev] __file__ In-Reply-To: <4B8A6A8F.7050008@gmail.com> References: <20100225160832.6e7a3063@freewill.wooz.org> <4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk> <4B871227.2030707@canterbury.ac.nz> <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> <4B8A6A8F.7050008@gmail.com> Message-ID: Nick Coghlan wrote: > Michael Foord wrote: >>> Can't it look for a .py file in the source directory first (1st stat)? >>> When it's there check for the .pyc in the cache directory (2nd stat, >>> magic number encoded in filename), if it's not check for .pyc in the >>> source directory (2nd stat + read for magic number check). Or am I >>> missing a subtlety? >> The problem is doing this little dance for every path on sys.path. > > To unpack this a little bit for those not quite as familiar with the > import system (and to make it clear for my own benefit!): for a > top-level module/package, each path on sys.path needs to be eliminated > as a possible location before the interpreter can move on to check the > next path in the list. > > So the important number is the number of stat calls on a "miss" (i.e. > when the requested module/package is not present in a directory). > Currently, with builtin support for bytecode only files, there are 3 > checks (package directory, py source file, pyc/pyo bytecode file) to be > made for each path entry. > > The PEP proposes to reduce that to only two in the case of a miss, by > checking for the cached pyc only if the source file is present (there > would still be three checks for a "hit", but that only happens at most > once per module lookup). > > While the PEP is right in saying that a bytecode-only import hook could > be added, I believe it would actually be a little tricky to write one > that didn't severely degrade the performance of either normal imports or > bytecode-only imports. Keeping it in the core import, but turning it off > by default seems much less likely to have unintended performance > consequences when it is switched back on. > > Another option is to remove bytecode-only support from the default > filesystem importer, but keep it for zipimport (since the stat call > savings don't apply in the latter case). What if ... a bytecode-only mode is triggered by "__main__" loading from a bytecode file, otherwise the .py files are needed and are checked to make sure the bytecode files are current. Ron From jacob at jacobian.org Mon Mar 1 20:45:17 2010 From: jacob at jacobian.org (Jacob Kaplan-Moss) Date: Mon, 1 Mar 2010 13:45:17 -0600 Subject: [Python-Dev] Status of PEP 381 Message-ID: Hey folks -- Can someone point me to some information on what's going on with PEP 381 (PyPI mirroring)? There's a bunch of XXX'd material, and it doesn't appear that pypi.python.org implements the statistics or providing parts. I'd be willing -- happy, actually -- to fill in the missing pieces, but only if I'm not stepping on anyone's toes. I'd also need some help adding code to pypi.python.org if that's a necessary part -- I'm not very good at finding my way around that codebase. Thanks, Jacob From tjreedy at udel.edu Mon Mar 1 20:58:33 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 01 Mar 2010 14:58:33 -0500 Subject: [Python-Dev] Status of PEP 381 In-Reply-To: References: Message-ID: On 3/1/2010 2:45 PM, Jacob Kaplan-Moss wrote: > Hey folks -- > > Can someone point me to some information on what's going on with PEP > 381 (PyPI mirroring)? There's a bunch of XXX'd material, and it > doesn't appear that pypi.python.org implements the statistics or > providing parts. > > I'd be willing -- happy, actually -- to fill in the missing pieces, > but only if I'm not stepping on anyone's toes. I'd also need some help > adding code to pypi.python.org if that's a necessary part -- I'm not > very good at finding my way around that codebase. There is a separate catalog/pypi mailing list, mirrored as gmane.comp.python.catalog. Ask there. tjr From brett at python.org Mon Mar 1 21:37:10 2010 From: brett at python.org (Brett Cannon) Date: Mon, 1 Mar 2010 12:37:10 -0800 Subject: [Python-Dev] __file__ In-Reply-To: References: <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> <4B8A6A8F.7050008@gmail.com> Message-ID: On Mon, Mar 1, 2010 at 08:30, Ron Adam wrote: > > > Nick Coghlan wrote: > >> Michael Foord wrote: >> >>> Can't it look for a .py file in the source directory first (1st stat)? >>>> When it's there check for the .pyc in the cache directory (2nd stat, >>>> magic number encoded in filename), if it's not check for .pyc in the >>>> source directory (2nd stat + read for magic number check). Or am I >>>> missing a subtlety? >>>> >>> The problem is doing this little dance for every path on sys.path. >>> >> >> To unpack this a little bit for those not quite as familiar with the >> import system (and to make it clear for my own benefit!): for a >> top-level module/package, each path on sys.path needs to be eliminated >> as a possible location before the interpreter can move on to check the >> next path in the list. >> >> So the important number is the number of stat calls on a "miss" (i.e. >> when the requested module/package is not present in a directory). >> Currently, with builtin support for bytecode only files, there are 3 >> checks (package directory, py source file, pyc/pyo bytecode file) to be >> made for each path entry. >> >> The PEP proposes to reduce that to only two in the case of a miss, by >> checking for the cached pyc only if the source file is present (there >> would still be three checks for a "hit", but that only happens at most >> once per module lookup). >> >> While the PEP is right in saying that a bytecode-only import hook could >> be added, I believe it would actually be a little tricky to write one >> that didn't severely degrade the performance of either normal imports or >> bytecode-only imports. Keeping it in the core import, but turning it off >> by default seems much less likely to have unintended performance >> consequences when it is switched back on. >> >> Another option is to remove bytecode-only support from the default >> filesystem importer, but keep it for zipimport (since the stat call >> savings don't apply in the latter case). >> > > What if ... a bytecode-only mode is triggered by "__main__" loading from a > bytecode file, otherwise the .py files are needed and are checked to make > sure the bytecode files are current. > > That's way too magical for my tastes, especially if you mess up and accidentally leave behind a __main__.pyc after moving the __main__.py file. -Brett > Ron > > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Mar 1 21:44:33 2010 From: brett at python.org (Brett Cannon) Date: Mon, 1 Mar 2010 12:44:33 -0800 Subject: [Python-Dev] Who needs to be added to python-committers? Message-ID: Any people who have commit privileges not subscribed to python-committers? We have gained some new people recently so I just want to make sure everyone is subscribed to the list. Also, anyone have an email address for Hirokazu Yamamoto? His mailbox is full so Mailman unsubscribed him. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From ziade.tarek at gmail.com Mon Mar 1 21:59:11 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 1 Mar 2010 21:59:11 +0100 Subject: [Python-Dev] Status of PEP 381 In-Reply-To: References: Message-ID: <94bdd2611003011259v56aa6ccaq430fe7ed6e73b504@mail.gmail.com> On Mon, Mar 1, 2010 at 8:45 PM, Jacob Kaplan-Moss wrote: > Hey folks -- > > Can someone point me to some information on what's going on with PEP > 381 (PyPI mirroring)? There's a bunch of XXX'd material, and it > doesn't appear that pypi.python.org implements the statistics or > providing parts. > > I'd be willing -- happy, actually -- to fill in the missing pieces, > but only if I'm not stepping on anyone's toes. I'd also need some help > adding code to pypi.python.org if that's a necessary part -- I'm not > very good at finding my way around that codebase. Hey Jacob, I have started to implement the server parts (some are already available at pypi but not activated) and Martin worked on the client part on his side to provide a library for mirrors. I need to finish editing the PEP text but any help is welcome of course on any part. As Terry suggested you can send a mail at Catalog, I'll catch up on the thread once its started, to list possible tasks. Regards, Tarek -- Tarek Ziad? | http://ziade.org From martin at v.loewis.de Mon Mar 1 22:40:35 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 01 Mar 2010 22:40:35 +0100 Subject: [Python-Dev] Python and Windows 2000 Message-ID: <4B8C3453.7030507@v.loewis.de> I don't recall whether we have already decided about continued support for Windows 2000. If not, I'd like to propose that we phase out that support: the Windows 2.7 installer should display a warning; 3.2 will stop supporting Windows 2000. Opinions? Regards, Martin From greg.ewing at canterbury.ac.nz Mon Mar 1 22:53:35 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 02 Mar 2010 10:53:35 +1300 Subject: [Python-Dev] __file__ In-Reply-To: References: <20100225160832.6e7a3063@freewill.wooz.org> <4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk> <4B871227.2030707@canterbury.ac.nz> <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> <4B8A6A8F.7050008@gmail.com> Message-ID: <4B8C375F.30508@canterbury.ac.nz> Ron Adam wrote: > What if ... a bytecode-only mode is triggered by "__main__" loading from > a bytecode file, otherwise the .py files are needed and are checked to > make sure the bytecode files are current. That would preclude having a bytecode-only library that could be used by a sourceful program. Such a situation might arise if you have an application with a scripting interface that is used by importing stuff from the application's internal libraries. -- Greg From guido at python.org Mon Mar 1 22:53:45 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 1 Mar 2010 13:53:45 -0800 Subject: [Python-Dev] Python and Windows 2000 In-Reply-To: <4B8C3453.7030507@v.loewis.de> References: <4B8C3453.7030507@v.loewis.de> Message-ID: On Mon, Mar 1, 2010 at 1:40 PM, "Martin v. L?wis" wrote: > I don't recall whether we have already decided about continued support > for Windows 2000. > > If not, I'd like to propose that we phase out that support: the Windows > 2.7 installer should display a warning; 3.2 will stop supporting Windows > 2000. > > Opinions? +2000 -- --Guido van Rossum (python.org/~guido) From ssteinerx at gmail.com Mon Mar 1 23:33:21 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Mon, 1 Mar 2010 17:33:21 -0500 Subject: [Python-Dev] Python and Windows 2000 In-Reply-To: References: <4B8C3453.7030507@v.loewis.de> Message-ID: <339E3993-FFF8-4B86-BE17-59AC56540CFB@gmail.com> On Mar 1, 2010, at 4:53 PM, Guido van Rossum wrote: > On Mon, Mar 1, 2010 at 1:40 PM, "Martin v. L?wis" wrote: >> I don't recall whether we have already decided about continued support >> for Windows 2000. >> >> If not, I'd like to propose that we phase out that support: the Windows >> 2.7 installer should display a warning; 3.2 will stop supporting Windows >> 2000. >> >> Opinions? > > +2000 I guess i'll finally have to retire my ten year old MSDN cd's and my favorite version of Windows -- the last one I ever used. S From collinwinter at google.com Tue Mar 2 00:35:15 2010 From: collinwinter at google.com (Collin Winter) Date: Mon, 1 Mar 2010 15:35:15 -0800 Subject: [Python-Dev] Packaging JIT-less versions of Python Message-ID: <3c8293b61003011535x6190351ald0640855bbfa7c4b@mail.gmail.com> Hey packaging guys, We recently committed a change to Unladen Swallow [1] that moves all the JIT infrastructure into a Python extension module. The theory [2] behind this patch was that this would make it easier for downstream packagers to ship a JIT-less Python package, with the JIT compiler available via an optional add-on package. Some questions for you, so we're not shooting blind here: - Have you guys thought about how a JIT-enabled Python 3 installation would be packaged by your respective distros? - Would you prefer the default python3.x package to have a JIT, or would you omit the JIT by default? - How would you prefer to build the JIT-less package (current options: via a ./configure flag; or by deleting _llvmjit.so from the JIT-enabled package)? - Would the two packages be able to exist side-by-side, or would they be mutually exclusive? My strong preference would be to have the JIT included by default so that it receives as much testing as possible. Thanks, Collin Winter [1] - http://code.google.com/p/unladen-swallow/source/detail?r=1110 [2] - http://code.google.com/p/unladen-swallow/issues/detail?id=136 From barry at python.org Tue Mar 2 01:32:41 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 1 Mar 2010 19:32:41 -0500 Subject: [Python-Dev] __file__ In-Reply-To: <20100228224020.46da4e23@msiwind> References: <20100130190005.058c8187@freewill.wooz.org> <20100225160832.6e7a3063@freewill.wooz.org> <4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk> <4B871227.2030707@canterbury.ac.nz> <4B885AB4.7040004@g.nevcal.com> <4B889AB1.4050402@g.nevcal.com> <4B89C5FB.40104@canterbury.ac.nz> <4B89F5FA.2010604@g.nevcal.com> <4B8AFAAB.5020106@canterbury.ac.nz> <4B8B3539.5060709@g.nevcal.com> <20100228224020.46da4e23@msiwind> Message-ID: <20100301193241.38a81a93@freewill.wooz.org> On Feb 28, 2010, at 10:40 PM, Antoine Pitrou wrote: >File extensions exist for a reason, even if you find that "kooky" and >have strong ideas about the psychology of text editors. > >Having some binary files named "foobar.py" would certainly annoy a lot >of people, including me. I completely agree. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From barry at python.org Tue Mar 2 01:41:52 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 1 Mar 2010 19:41:52 -0500 Subject: [Python-Dev] __file__ In-Reply-To: <4B89CC14.5070902@canterbury.ac.nz> References: <20100202231252.77a79b0e@freewill.wooz.org> <20100225160832.6e7a3063@freewill.wooz.org> <4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk> <4B871227.2030707@canterbury.ac.nz> <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> Message-ID: <20100301194152.766d0de8@freewill.wooz.org> On Feb 28, 2010, at 02:51 PM, Greg Ewing wrote: >A solution might be to look for the presence of the >cache directory, and only look for a .pyc in the source >directory if there is no cache directory. Testing for >the cache directory would only have to be done once >per package and the result remembered, so it would >add very little overhead. I think the other thing that bothers me about continuing to support pyc-only imports, is that people will then want tools to create them. Right now, it's probably just as easy as byte-compiling everything, then finding the .py files and removing them. After PEP 3147 is implemented, and the default, you'll have to byte-compile the files, then find the pycs in the __pycache__ directory, move them up a level and rename them. Then of course remove the .py files. It's not insurmountable of course, I think if we support pyc-only imports, people are rightly going to want us to write and support the tool to create those imports. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From barry at python.org Tue Mar 2 01:49:46 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 1 Mar 2010 19:49:46 -0500 Subject: [Python-Dev] __file__ In-Reply-To: <4B8A6A8F.7050008@gmail.com> References: <20100225160832.6e7a3063@freewill.wooz.org> <4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk> <4B871227.2030707@canterbury.ac.nz> <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> <4B8A6A8F.7050008@gmail.com> Message-ID: <20100301194946.188a8855@freewill.wooz.org> On Feb 28, 2010, at 11:07 PM, Nick Coghlan wrote: >While the PEP is right in saying that a bytecode-only import hook could >be added, I believe it would actually be a little tricky to write one >that didn't severely degrade the performance of either normal imports or >bytecode-only imports. Keeping it in the core import, but turning it off >by default seems much less likely to have unintended performance >consequences when it is switched back on. Except that even users of bytecode-only imports probably don't want or care about that for *every* package directory. So really, all-or-nothing hits them too. One option to help with that is to have a flag or marker in a package's __init__.py that signals pyc-only imports for that package directory. It's getting complicated again. ;) >Another option is to remove bytecode-only support from the default >filesystem importer, but keep it for zipimport (since the stat call >savings don't apply in the latter case). I'd be okay with this, but even here I'd argue that it would be fine to require the source files by default. The primary use case I've seen mentioned for pyc-only imports is to make it more difficult for users to accidentally shoot themselves in the foot. I think the very presence of a zip file for importing is enough without the extra step of removing the source. But that's just me. :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From barry at python.org Tue Mar 2 01:59:55 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 1 Mar 2010 19:59:55 -0500 Subject: [Python-Dev] __file__ In-Reply-To: <20100301194946.188a8855@freewill.wooz.org> References: <20100225160832.6e7a3063@freewill.wooz.org> <4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk> <4B871227.2030707@canterbury.ac.nz> <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> <4B8A6A8F.7050008@gmail.com> <20100301194946.188a8855@freewill.wooz.org> Message-ID: <20100301195955.3091dc19@freewill.wooz.org> Thanks everybody for providing great input on this aspect of the PEP. I've updated the open issues section to include a list of the possible resolutions for bytecode-only imports. Unless anybody has more ideas, it might just be time to get a BDFL pronouncement. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From brett at python.org Tue Mar 2 03:02:57 2010 From: brett at python.org (Brett Cannon) Date: Mon, 1 Mar 2010 18:02:57 -0800 Subject: [Python-Dev] Python and Windows 2000 In-Reply-To: <4B8C3453.7030507@v.loewis.de> References: <4B8C3453.7030507@v.loewis.de> Message-ID: On Mon, Mar 1, 2010 at 13:40, "Martin v. L?wis" wrote: > I don't recall whether we have already decided about continued support > for Windows 2000. > > If not, I'd like to propose that we phase out that support: the Windows > 2.7 installer should display a warning; 3.2 will stop supporting Windows > 2000. > > Opinions? > Considering MS is retiring support for Win2K in July ( http://support.microsoft.com/lifecycle/?p1=3071) I think we can be faster than them in dropping it. =) +1 from me. > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From meadori at gmail.com Tue Mar 2 03:05:49 2010 From: meadori at gmail.com (Meador Inge) Date: Mon, 1 Mar 2010 20:05:49 -0600 Subject: [Python-Dev] PEP 3188: Implementation Questions In-Reply-To: References: <4095897c1002252051w24377ec2tc60d0b04adee8c5d@mail.gmail.com> <4B88464E.9080700@canterbury.ac.nz> <4095897c1002270858j5e14579dhe01c1fb4958f087b@mail.gmail.com> Message-ID: <4095897c1003011805r72f33112j1891e3dac13c7707@mail.gmail.com> On Sat, Feb 27, 2010 at 11:20 AM, Thomas Heller wrote: > See issue 887237: > > http://bugs.python.org/issue887237 > Thanks for the link Thomas. Since there is already interest in adding arithmetic to ctypes, perhaps that is an option. One question that raises in my mind, though, is should only 'long double' unpack to a ctype in that case? Or should all items unpack to ctypes now? It seems to me that you would want everything to unpack to types from the same family (e.g. Python builtins or ctypes). This seems conceptually cleaner and the interoperability between types in the same "family" are (or can be in the case of modifying ctypes) more clearly defined. Thanks, -- Meador -------------- next part -------------- An HTML attachment was scrubbed... URL: From nyamatongwe at gmail.com Tue Mar 2 03:11:28 2010 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Tue, 2 Mar 2010 13:11:28 +1100 Subject: [Python-Dev] Python and Windows 2000 In-Reply-To: <4B8C3453.7030507@v.loewis.de> References: <4B8C3453.7030507@v.loewis.de> Message-ID: <50862ebd1003011811t5ce65f23x1a89d65faac76997@mail.gmail.com> Martin v. L?wis: > I don't recall whether we have already decided about continued support > for Windows 2000. > > If not, I'd like to propose that we phase out that support: the Windows > 2.7 installer should display a warning; 3.2 will stop supporting Windows > 2000. Is there any reason for this? I can understand dropping Windows 9x due to the lack of Unicode support but is there anything missing from Windows 2000 that makes supporting it difficult? Neil From dmalcolm at redhat.com Tue Mar 2 04:29:32 2010 From: dmalcolm at redhat.com (David Malcolm) Date: Mon, 01 Mar 2010 22:29:32 -0500 Subject: [Python-Dev] Packaging JIT-less versions of Python In-Reply-To: <3c8293b61003011535x6190351ald0640855bbfa7c4b@mail.gmail.com> References: <3c8293b61003011535x6190351ald0640855bbfa7c4b@mail.gmail.com> Message-ID: <1267500572.8882.351.camel@brick> On Mon, 2010-03-01 at 15:35 -0800, Collin Winter wrote: > Hey packaging guys, > > We recently committed a change to Unladen Swallow [1] that moves all > the JIT infrastructure into a Python extension module. The theory [2] > behind this patch was that this would make it easier for downstream > packagers to ship a JIT-less Python package, with the JIT compiler > available via an optional add-on package. > > Some questions for you, so we're not shooting blind here: > - Have you guys thought about how a JIT-enabled Python 3 installation > would be packaged by your respective distros? I suspect that there may be architecture-specific issues both in getting LLVM to work, and in getting the Python JIT code running (e.g. I've no idea how well it works on, say, s390). The answers may vary between CPU architecture (e.g. "on by default for i386 and x86_64, off by default on ppc", or whatever). > - Would you prefer the default python3.x package to have a JIT, or > would you omit the JIT by default? I'd be inclined to turn it on by default in Fedora's "rawhide" development branch as soon as it's meaningful to test it (with a suitable warning on our mailing list); we like to try this kind of thing! If no major issues are found, I'd leave it turned on. I'd expect to run into "fun" CPU-arch-specific bugs, though. Multicore 64-bit big-endian springs to mind. I can't speak for what I'd do for RHEL at this time. It's likely to be affected by how experience in Fedora goes, but goes through more formal QA, and on more "enterprisey" hardware (e.g. s390). > - How would you prefer to build the JIT-less package (current options: > via a ./configure flag; or by deleting _llvmjit.so from the > JIT-enabled package)? > - Would the two packages be able to exist side-by-side, or would they > be mutually exclusive? I have a particular interest in ABI compatibility: if turning JIT on and off is going to change the ABI of extension modules, that would be a major pain, as I hope that we will have dozens of C extension modules available via RPM for our Python 3 stack by the time of the "great unladen merger". So I'm keen for the ability to toggle the JIT code in the face of bugs and have it not affect ABI. "-Xjit" will do this at runtime (once that's renamed), but I think it would be useful to be able to toggle the JIT on/off default during the build, so that I can fix a broken architecture for non-technical users, but have individual testers opt back in with -Xjit whilst tracking down a major bug. In either case, I don't want to have to recompile 30 extension modules to try with/without JIT; that would introduce too much change during bug-hunts, and be no fun at all. (In the blue-sky nirvana future, I'd love to be able to ship ahead-of-time compiled versions of the stdlib, pre-optimized based on realworld workloads. Back in my reality, though, I have bugs to fix before I can work on _that_ patch :( ) > My strong preference would be to have the JIT included by default so > that it receives as much testing as possible. Sounds reasonable. Hope the above made sense and is useful. Dave > > Thanks, > Collin Winter > > [1] - http://code.google.com/p/unladen-swallow/source/detail?r=1110 > [2] - http://code.google.com/p/unladen-swallow/issues/detail?id=136 From martin at v.loewis.de Tue Mar 2 06:17:04 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 02 Mar 2010 06:17:04 +0100 Subject: [Python-Dev] Python and Windows 2000 In-Reply-To: <50862ebd1003011811t5ce65f23x1a89d65faac76997@mail.gmail.com> References: <4B8C3453.7030507@v.loewis.de> <50862ebd1003011811t5ce65f23x1a89d65faac76997@mail.gmail.com> Message-ID: <4B8C9F50.7010709@v.loewis.de> >> If not, I'd like to propose that we phase out that support: the Windows >> 2.7 installer should display a warning; 3.2 will stop supporting Windows >> 2000. > > Is there any reason for this? I can understand dropping Windows 9x > due to the lack of Unicode support but is there anything missing from > Windows 2000 that makes supporting it difficult? See http://bugs.python.org/issue6926 The SDK currently hides symbolic constants from us that people are asking for. In addition, we could simplify the code in dl_nt.c around GetCurrentActCtx and friends, by linking to these functions directly. Regards, Martni From stefan_ml at behnel.de Tue Mar 2 10:06:06 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 02 Mar 2010 10:06:06 +0100 Subject: [Python-Dev] ElementTree.tostring() returning str by default in Py3.2 Message-ID: Hi, I just noticed that the xml.etree.ElementTree.tostring() function in Py3.2 returns a str object by default, unless an encoding is specified. This is a backwards incompatible change compared to ET 1.2. For one, it breaks tons of tests in lxml's compatibility test suite. Previously, the default encoding was 'ASCII', and tostring() was guaranteed to always return a byte string. According to svn blame, this was changed by GvR in rev. 56841. Is there any reasoning behind this? Stefan From nyamatongwe at gmail.com Tue Mar 2 10:08:06 2010 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Tue, 2 Mar 2010 20:08:06 +1100 Subject: [Python-Dev] Python and Windows 2000 In-Reply-To: <4B8C9F50.7010709@v.loewis.de> References: <4B8C3453.7030507@v.loewis.de> <50862ebd1003011811t5ce65f23x1a89d65faac76997@mail.gmail.com> <4B8C9F50.7010709@v.loewis.de> Message-ID: <50862ebd1003020108i6ebeef69l2e7987a887438058@mail.gmail.com> Martin v. L?wis: > See http://bugs.python.org/issue6926 > > The SDK currently hides symbolic constants from us that people are > asking for. Setting the version to 0x501 (XP) doesn't actively try to stop running on version 0x500 (2K), it just reveals the symbols and APIs from 0x501. Including a call to an 0x501-specific API will then fail at load. IPPROTO_IPV6 (the cause of issue 6926) isn't a new symbol that started working in Windows XP - it was present in older SDKs without a version guard so was visible when compiling for any version of Windows. > In addition, we could simplify the code in dl_nt.c around > GetCurrentActCtx and friends, by linking to these functions directly. It would be simpler but its not as if this code needs any changes at this point. I don't really have a strong need for Windows 2000 although I keep an instance for checking compatibility of my code and I do still get queries from people using old versions of Windows, including 9x. There is the question of whether to force failure on Windows 2000 or just remove it from the list of known-working platforms while still allowing it to run. Neil From greg.ewing at canterbury.ac.nz Tue Mar 2 10:40:47 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 02 Mar 2010 22:40:47 +1300 Subject: [Python-Dev] Python and Windows 2000 In-Reply-To: <50862ebd1003020108i6ebeef69l2e7987a887438058@mail.gmail.com> References: <4B8C3453.7030507@v.loewis.de> <50862ebd1003011811t5ce65f23x1a89d65faac76997@mail.gmail.com> <4B8C9F50.7010709@v.loewis.de> <50862ebd1003020108i6ebeef69l2e7987a887438058@mail.gmail.com> Message-ID: <4B8CDD1F.5020501@canterbury.ac.nz> Neil Hodgson wrote: > There > is the question of whether to force failure on Windows 2000 or just > remove it from the list of known-working platforms while still > allowing it to run. I'd be grateful if you could refrain from doing anything to actively break it. Win 2000 was the last version to be free of Microsoft DRM crap, and I'd be unhappy if I were forced to downgrade to a later version just so I could use Py3 on it. -- Greg From stefan_ml at behnel.de Tue Mar 2 10:47:06 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 02 Mar 2010 10:47:06 +0100 Subject: [Python-Dev] ElementTree.tostring() returning str by default in Py3.2 In-Reply-To: References: Message-ID: Stefan Behnel, 02.03.2010 10:06: > I just noticed that the xml.etree.ElementTree.tostring() function in Py3.2 > returns a str object by default, unless an encoding is specified. This is a > backwards incompatible change compared to ET 1.2. For one, it breaks tons > of tests in lxml's compatibility test suite. Previously, the default > encoding was 'ASCII', and tostring() was guaranteed to always return a byte > string. > > According to svn blame, this was changed by GvR in rev. 56841. > > Is there any reasoning behind this? BTW, sorry for not reporting this earlier (the change is from 2007). It seems that bug 7173 (or some other bug) prevented me from running lxml's test suite in Py3 for quite a while now. This is the first time in ages that I get a complete run in Py3.x (x>0). Stefan From steve at pearwood.info Tue Mar 2 10:53:26 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 2 Mar 2010 20:53:26 +1100 Subject: [Python-Dev] __file__ In-Reply-To: <20100301194152.766d0de8@freewill.wooz.org> References: <20100202231252.77a79b0e@freewill.wooz.org> <4B89CC14.5070902@canterbury.ac.nz> <20100301194152.766d0de8@freewill.wooz.org> Message-ID: <201003022053.27049.steve@pearwood.info> On Tue, 2 Mar 2010 11:41:52 am Barry Warsaw wrote: > After PEP 3147 is implemented, and the default, you'll have to > byte-compile the files, then find the pycs in the __pycache__ > directory, move them up a level and rename them. Then of course > remove the .py files. > > It's not insurmountable of course, I think if we support pyc-only > imports, people are rightly going to want us to write and support the > tool to create those imports. Surely that's a job for a tiny Python script, or even a shell script? It doesn't sound hard, not from the description given. I imagine there will be recipes on ActiveState quite quickly, and if there isn't, that would be good evidence that demand for the feature is low. -- Steven D'Aprano From steve at pearwood.info Tue Mar 2 11:06:16 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 2 Mar 2010 21:06:16 +1100 Subject: [Python-Dev] __file__ In-Reply-To: <20100301195955.3091dc19@freewill.wooz.org> References: <20100301194946.188a8855@freewill.wooz.org> <20100301195955.3091dc19@freewill.wooz.org> Message-ID: <201003022106.17179.steve@pearwood.info> On Tue, 2 Mar 2010 11:59:55 am Barry Warsaw wrote: > Thanks everybody for providing great input on this aspect of the PEP. > I've updated the open issues section to include a list of the > possible resolutions for bytecode-only imports. Unless anybody has > more ideas, it might just be time to get a BDFL pronouncement. Please excuse me if these minor points have already been discussed, but I couldn't see them in the PEP. (1) What happens if the __cache__ directory doesn't exist and the enclosing directory is unwriteable, or if it does exist, but is unreadable? I expect that the byte code files will simply not be created, and everything will continue without them. (2) Presumably this only effects imports, not running python source code as a script. If I do this: python myscript.py from the shell, I would expect that no __cache__ directory will be created, just like today. BTW, you have some sort of automated warning in the PEP: System Message: WARNING/2 (pep-3147.txt, line 237) Title underline too short. http://www.python.org/dev/peps/pep-3147/#id47 -- Steven D'Aprano From dickinsm at gmail.com Tue Mar 2 13:38:47 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 2 Mar 2010 12:38:47 +0000 Subject: [Python-Dev] PEP 3188: Implementation Questions In-Reply-To: <4095897c1002252051w24377ec2tc60d0b04adee8c5d@mail.gmail.com> References: <4095897c1002252051w24377ec2tc60d0b04adee8c5d@mail.gmail.com> Message-ID: <5c6f2a5d1003020438g3d67e4falb02f9501e1b535fd@mail.gmail.com> On Fri, Feb 26, 2010 at 4:51 AM, Meador Inge wrote: > > Recently some discussion began in the issue 3132 thread (http://bugs.python.org/issue3132) regarding > implementation of the new struct string syntax for PEP 3118.? Mark Dickinson > suggested that I bring the discussion on over to Python Dev.? Below is a summary > of the questions\comments from the thread. Thanks for bringing this up here! > Unpacking a long-double > =================== > ... I don't want to dwell on this too much, since this is just one small part of the proposed struct module additions, and I'd like to get answers to some of the other questions you raised.? If interested parties (Carl?, Travis?) have time to comment on the other questions Meador raised, I'd *really* appreciate it! For long-double: I'm essentially -1 on any Decimal involvement here, on the basis that the semantics are messy and I doubt that the resulting behaviour would match users' needs or expectations.? I'm also not too keen on introducing a long-double-with-arithmetic type (wherever it ends up living);? I think having a single (binary) floating-point type has served Python very well, and adding other binary float precisions would risk either adding significant complexity, or leaving the extended precision type somewhat crippled. If there were a long double type that supported arithmetic, how should mixed-mode double + long double operations behave? What about conversions of long doubles to and from decimal strings? Should math module and cmath module functions accept long double arguments, and if so, what type result should they produce? The fact that there are at least 3 different common formats, before considering padding and byte order, for long double (IEEE 80-bit extended used on x86, IEEE binary128, and the so-called double-double format) isn't going to make things any easier here. I guess, in my currently not-very-informed state, I'd vote for: - packing and unpacking with the long double format expects and produces a ctypes.long_double in native mode; the long-double format would match that of the platform. - non-native mode packing and unpacking aren't permitted, and raise an exception. - to do arithmetic with long doubles, users would simply have to convert to and from Python floats, accepting the loss of precision that such conversion entails. But I think I'm failing to understand the intended use-cases for this (and other) additions. Would it be possible for Carl or Travis, or anyone else, to give some examples of situations where this would be useful? I don't doubt that such situations exist; I'm just not sure what they are. > Pointers > ====== > [...] > 2. How would unpacking a pointer to a Python Object work out?? Given an > ??? address how would the appropriate object to be unpacked be determined? Again, I'd really like to see some examples of how/when packing and unpacking pointers to Python objects would be used; I'm currently feeling too stupid to understand how this might work in practice. > 4. For the 'X{}' format (pointer to a function), is this supposed to mean a > ??? Python function or a C function? Ditto for this; assuming that we're talking about a Python function here. > String Syntax > ========== > > The syntax seems to have transcended verbal description.? I think we > need to put forth a grammar. Agreed: a clear specification is needed. > Use Cases > ======== > > ?? 1. What are the real world use cases for these struct string extensions? > ????? These should be fleshed out and documented. +many. Mark From dickinsm at gmail.com Tue Mar 2 13:50:43 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 2 Mar 2010 12:50:43 +0000 Subject: [Python-Dev] PEP 3188: Implementation Questions In-Reply-To: <4B89C946.6010403@canterbury.ac.nz> References: <4095897c1002252051w24377ec2tc60d0b04adee8c5d@mail.gmail.com> <4B88464E.9080700@canterbury.ac.nz> <4095897c1002270858j5e14579dhe01c1fb4958f087b@mail.gmail.com> <4B89C946.6010403@canterbury.ac.nz> Message-ID: <5c6f2a5d1003020450r3d4af2cdi1e29bf7f1a5254ad@mail.gmail.com> On Sun, Feb 28, 2010 at 1:39 AM, Greg Ewing wrote: > Meador Inge wrote: > >> Even with the user-defined precision capabilities of the 'Decimal' class? >> ?In other words, can I create an instance of a 'Decimal' that behaves (in >> all operations: arithmetic, comparison, etc...) exactly as the extended >> double precision type offered by a given machine? > > It's not precision that's the issue, it's that the number > base is different. That affects which numbers can > be represented exactly, and how results that can't be > represented exactly are rounded. > > I would be very surprised if there is a way of configuring > the Decimal type so that it gives identical results to that > of any IEEE binary floating point type, including rounding > behaviour, denormalisation, etc. I'd be astonished. :) Mark From ncoghlan at gmail.com Tue Mar 2 14:03:19 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 02 Mar 2010 23:03:19 +1000 Subject: [Python-Dev] Breaking bytecode only imports (was Re: __file__) In-Reply-To: <20100301195955.3091dc19@freewill.wooz.org> References: <20100225160832.6e7a3063@freewill.wooz.org> <4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk> <4B871227.2030707@canterbury.ac.nz> <4B88526E.7030808@voidspace.org.uk> <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> <4B8A6A8F.7050008@gmail.com> <20100301194946.188a8855@freewill.wooz.org> <20100301195955.3091dc19@freewill.wooz.org> Message-ID: <4B8D0C97.4020002@gmail.com> Barry Warsaw wrote: > Thanks everybody for providing great input on this aspect of the PEP. I've > updated the open issues section to include a list of the possible resolutions > for bytecode-only imports. Unless anybody has more ideas, it might just be > time to get a BDFL pronouncement. I think the benchmarking in the bytecode-only section is still too weak. "evidence shows that the extra stats can be fairly costly to start up time" isn't a valid justification for breaking working code. Doing 4 stat calls instead of 5 on a directory miss just doesn't excite me very much without some genuine benchmarks across different operating systems and filesystems showing that reducing the number of stat calls by at best 20% will result in a measurable reduction in import times for real modules (where we can expect the import time to be dominated by the execution of the actual module code rather than the time needed to find that code in the first place). Using the sample numbers Robert Collins posted: # Startup time for bzr (cold cache): $ drop-caches $ time bzr --no-plugins revno 5061 real 0m8.875s user 0m0.210s sys 0m0.140s # Hot cache $ time bzr --no-plugins revno 5061 real 0m0.307s user 0m0.250s sys 0m0.040s $ strace -c bzr --no-plugins revno 5061 % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 56.34 0.040000 76 527 read 28.98 0.020573 9 2273 1905 open 14.43 0.010248 14 734 625 stat 0.15 0.000107 0 533 fstat hot cache: % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 45.10 0.000368 92 4 getdents 19.49 0.000159 0 527 read 16.91 0.000138 1 163 munmap 10.05 0.000082 2 54 mprotect 8.46 0.000069 0 2273 1905 open 0.00 0.000000 0 8 write 0.00 0.000000 0 367 close 0.00 0.000000 0 734 625 stat Assuming all those stat errors are misses from the import system, we're looking at reducing that 625 figure down to 500: 125 fewer failed calls. With a hot cache, the impact is too small for strace to even measure. With a cold cache, it is 1.75 milliseconds: only 1.25% of the system time consumed in the script's execution, and not even registering relative to the 9 second wall clock time. Without significant measurable performance gains, a mere aesthetic preference isn't enough to justify inflicting subtle breakage on even a small subset of our users. Even aside from the issue of a lack of benchmarks to justify the breakage, bytecode only imports *cannot* legitimately be broken without at least one release where they generate Deprecation Warnings. Cheers, Nick. P.S. I actually started this thread as a +0 to the idea of dropping bytecode only imports. Over the course of the discussion I've shifted to a firm -1 in the absence of some proper comparative benchmarks to justify the change in semantics. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Tue Mar 2 16:32:12 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 2 Mar 2010 10:32:12 -0500 Subject: [Python-Dev] code.python.org dead? Message-ID: <20100302103212.69669f0f@msiwind> Hello, It seems someone has decided that code.python.org doesn't resolve anymore. This is annoying since there were Mercurial mirrors there ( at http://code.python.org/hg ). Until we make the official switch, these mirrors are still useful. Regards Antoine. From guido at python.org Tue Mar 2 18:34:26 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 2 Mar 2010 09:34:26 -0800 Subject: [Python-Dev] Breaking bytecode only imports (was Re: __file__) In-Reply-To: <4B8D0C97.4020002@gmail.com> References: <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> <4B8A6A8F.7050008@gmail.com> <20100301194946.188a8855@freewill.wooz.org> <20100301195955.3091dc19@freewill.wooz.org> <4B8D0C97.4020002@gmail.com> Message-ID: On Tue, Mar 2, 2010 at 5:03 AM, Nick Coghlan wrote: > P.S. I actually started this thread as a +0 to the idea of dropping > bytecode only imports. Over the course of the discussion I've shifted to > a firm -1 in the absence of some proper comparative benchmarks to > justify the change in semantics. FWIW, I started at -1 and am still -1. I think the PEP is overreaching in this aspect; it does not serve the stated purpose of the PEP to make life easier for distros that share code between Python versions. -- --Guido van Rossum (python.org/~guido) From barry at python.org Tue Mar 2 20:52:59 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 2 Mar 2010 14:52:59 -0500 Subject: [Python-Dev] Breaking bytecode only imports (was Re: __file__) In-Reply-To: References: <20100227105613.6581a198@freewill.wooz.org> <20100227173502.GA9522@laurie.devork> <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> <4B8A6A8F.7050008@gmail.com> <20100301194946.188a8855@freewill.wooz.org> <20100301195955.3091dc19@freewill.wooz.org> <4B8D0C97.4020002@gmail.com> Message-ID: <20100302145259.1442a324@heresy.wooz.org> On Mar 02, 2010, at 09:34 AM, Guido van Rossum wrote: >On Tue, Mar 2, 2010 at 5:03 AM, Nick Coghlan wrote: >> P.S. I actually started this thread as a +0 to the idea of dropping >> bytecode only imports. Over the course of the discussion I've shifted to >> a firm -1 in the absence of some proper comparative benchmarks to >> justify the change in semantics. > >FWIW, I started at -1 and am still -1. I think the PEP is overreaching >in this aspect; it does not serve the stated purpose of the PEP to >make life easier for distros that share code between Python versions. I think that's fair, and just the guidance I'm looking for. By now you understand the pros and cons, so if this is a pronouncement, I will cement it into the PEP. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From brett at python.org Tue Mar 2 21:06:53 2010 From: brett at python.org (Brett Cannon) Date: Tue, 2 Mar 2010 12:06:53 -0800 Subject: [Python-Dev] __file__ In-Reply-To: <201003022106.17179.steve@pearwood.info> References: <20100301194946.188a8855@freewill.wooz.org> <20100301195955.3091dc19@freewill.wooz.org> <201003022106.17179.steve@pearwood.info> Message-ID: On Tue, Mar 2, 2010 at 02:06, Steven D'Aprano wrote: > On Tue, 2 Mar 2010 11:59:55 am Barry Warsaw wrote: > > Thanks everybody for providing great input on this aspect of the PEP. > > I've updated the open issues section to include a list of the > > possible resolutions for bytecode-only imports. Unless anybody has > > more ideas, it might just be time to get a BDFL pronouncement. > > Please excuse me if these minor points have already been discussed, but > I couldn't see them in the PEP. > > (1) What happens if the __cache__ directory doesn't exist and the > enclosing directory is unwriteable, or if it does exist, but is > unreadable? > > I expect that the byte code files will simply not be created, and > everything will continue without them. > > (2) Presumably this only effects imports, not running python source code > as a script. If I do this: > > python myscript.py > > from the shell, I would expect that no __cache__ directory will be > created, just like today. > > BTW, you have some sort of automated warning in the PEP: > > System Message: WARNING/2 (pep-3147.txt, line 237) > Title underline too short. > > http://www.python.org/dev/peps/pep-3147/#id47 It's now fixed. Barry forgot to run the Makefile for the PEPs before checking in. Shame! =) -Brett > > > > > -- > Steven D'Aprano > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Tue Mar 2 22:33:56 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 02 Mar 2010 22:33:56 +0100 Subject: [Python-Dev] Python and Windows 2000 In-Reply-To: <50862ebd1003020108i6ebeef69l2e7987a887438058@mail.gmail.com> References: <4B8C3453.7030507@v.loewis.de> <50862ebd1003011811t5ce65f23x1a89d65faac76997@mail.gmail.com> <4B8C9F50.7010709@v.loewis.de> <50862ebd1003020108i6ebeef69l2e7987a887438058@mail.gmail.com> Message-ID: <4B8D8444.106@v.loewis.de> Neil Hodgson wrote: > Martin v. L?wis: > >> See http://bugs.python.org/issue6926 >> >> The SDK currently hides symbolic constants from us that people are >> asking for. > > Setting the version to 0x501 (XP) doesn't actively try to stop > running on version 0x500 (2K), it just reveals the symbols and APIs > from 0x501. Including a call to an 0x501-specific API will then fail > at load. > > IPPROTO_IPV6 (the cause of issue 6926) isn't a new symbol that > started working in Windows XP - it was present in older SDKs without a > version guard so was visible when compiling for any version of > Windows. I'm not proposing to actively break Windows 2000 support. I propose to stop supporting it, which means that we will not strive for Python running on Windows 2000, and that bug reports that report issues specific with Windows 2000 will be closed as "won't fix". I would not want to bump the SDK version while still aiming at Windows 2000 support - as that gets rarely tested, this API version is actually a mechanism to simplify ongoing support for Windows 2000. > I don't really have a strong need for Windows 2000 although I keep > an instance for checking compatibility of my code and I do still get > queries from people using old versions of Windows, including 9x. There > is the question of whether to force failure on Windows 2000 or just > remove it from the list of known-working platforms while still > allowing it to run. I would propose the latter; see above. However, patches that do actively break it would then be acceptable. In any case, we would need a warning in the installer that Windows 2000 supports is going away. Regards, Martin From martin at v.loewis.de Tue Mar 2 22:54:25 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 02 Mar 2010 22:54:25 +0100 Subject: [Python-Dev] code.python.org dead? In-Reply-To: <20100302103212.69669f0f@msiwind> References: <20100302103212.69669f0f@msiwind> Message-ID: <4B8D8911.1070003@v.loewis.de> > It seems someone has decided that code.python.org doesn't resolve > anymore. Correct - I deleted the DNS record, and the site, as it didn't work anymore (and because Barry confirmed that it is ok to delete the site) > This is annoying since there were Mercurial mirrors there > ( at http://code.python.org/hg ). Until we make the official switch, > these mirrors are still useful. Do you know who is in charge of creating the site? I could restore it, but I would need help to make it actually work (such as giving it a working DocumentRoot). Regards, Martin From barry at python.org Tue Mar 2 23:20:53 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 2 Mar 2010 17:20:53 -0500 Subject: [Python-Dev] code.python.org dead? In-Reply-To: <4B8D8911.1070003@v.loewis.de> References: <20100302103212.69669f0f@msiwind> <4B8D8911.1070003@v.loewis.de> Message-ID: <20100302172053.422d058c@heresy.wooz.org> On Mar 02, 2010, at 10:54 PM, Martin v. L?wis wrote: >Correct - I deleted the DNS record, and the site, as it didn't work >anymore (and because Barry confirmed that it is ok to delete the site) Yep, sorry. I didn't realize it was still in use. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From guido at python.org Wed Mar 3 00:27:05 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 2 Mar 2010 15:27:05 -0800 Subject: [Python-Dev] Breaking bytecode only imports (was Re: __file__) In-Reply-To: <20100302145259.1442a324@heresy.wooz.org> References: <4B89CC14.5070902@canterbury.ac.nz> <20100228121907.GA13564@laurie.devork> <690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> <4B8A6A8F.7050008@gmail.com> <20100301194946.188a8855@freewill.wooz.org> <20100301195955.3091dc19@freewill.wooz.org> <4B8D0C97.4020002@gmail.com> <20100302145259.1442a324@heresy.wooz.org> Message-ID: On Tue, Mar 2, 2010 at 11:52 AM, Barry Warsaw wrote: > On Mar 02, 2010, at 09:34 AM, Guido van Rossum wrote: > >>On Tue, Mar 2, 2010 at 5:03 AM, Nick Coghlan wrote: >>> P.S. I actually started this thread as a +0 to the idea of dropping >>> bytecode only imports. Over the course of the discussion I've shifted to >>> a firm -1 in the absence of some proper comparative benchmarks to >>> justify the change in semantics. >> >>FWIW, I started at -1 and am still -1. I think the PEP is overreaching >>in this aspect; it does not serve the stated purpose of the PEP to >>make life easier for distros that share code between Python versions. > > I think that's fair, and just the guidance I'm looking for. ?By now you > understand the pros and cons, so if this is a pronouncement, I will cement it > into the PEP. Yes, and thanks! -- --Guido van Rossum (python.org/~guido) From florent.xicluna at gmail.com Wed Mar 3 00:55:25 2010 From: florent.xicluna at gmail.com (Florent Xicluna) Date: Tue, 2 Mar 2010 23:55:25 +0000 (UTC) Subject: [Python-Dev] Deprecation warnings in Python 2.7 Message-ID: Hello, I would like to open a discussion on the meaning of deprecation warnings in 2.7. I assume, but I may be wrong, that 2.7 will be the last version in the 2.x branch. With this assumption, we should not find many things deprecated in 2.7 final. On the other hand, the list of py3k deprecation warnings is increasing a lot. And more users will probably start to move from 2.7 to the 3.1 release. While working on ticket #7092 and #7849, we started to discuss some proposals to simplify the deprecation warnings for 2.7. We discussed these 2 proposals: 1) in 2.6 there's no distinction between py3k and normal deprecations: they share the same category, DeprecationWarning. The "-3" switch enables the py3k DeprecationWarning and SyntaxWarning. Do we need to introduce a subclass Py3kDeprecationWarning in 2.7? It will make it easier to separate both kinds of deprecations (2.x and 3.x), and to filter them. 2) a different idea is to deprecate the "-3" switch and consider all py3k deprecations as normal deprecations. They will be hidden by default thanks to #7309. Since there's no future for the 2.x branch, it seems normal to consider the migration from 2.7 to 3.1 and show these warnings when the developer uses "-Wd" switch. What do you expect as DeprecationWarning in 2.7? I will post a list of the DeprecationWarnings in the python trunk, in a followup message, for a review, and to help the discussion. Best regards, -- Florent Xicluna From collinwinter at google.com Wed Mar 3 01:12:48 2010 From: collinwinter at google.com (Collin Winter) Date: Tue, 2 Mar 2010 16:12:48 -0800 Subject: [Python-Dev] Packaging JIT-less versions of Python In-Reply-To: <1267500572.8882.351.camel@brick> References: <3c8293b61003011535x6190351ald0640855bbfa7c4b@mail.gmail.com> <1267500572.8882.351.camel@brick> Message-ID: <3c8293b61003021612j2b76e7bcw98cc54995168c82d@mail.gmail.com> Hey David, On Mon, Mar 1, 2010 at 7:29 PM, David Malcolm wrote: > On Mon, 2010-03-01 at 15:35 -0800, Collin Winter wrote: [snip] >> - How would you prefer to build the JIT-less package (current options: >> via a ./configure flag; or by deleting _llvmjit.so from the >> JIT-enabled package)? >> - Would the two packages be able to exist side-by-side, or would they >> be mutually exclusive? > > I have a particular interest in ABI compatibility: if turning JIT on and > off is going to change the ABI of extension modules, that would be a > major pain, as I hope that we will have dozens of C extension modules > available via RPM for our Python 3 stack by the time of the "great > unladen merger". Do you have a good way of testing ABI compatibility, or is it just "build modules, see if they work"? Some general way of testing ABI compatibility would be really useful for PEP 384, too. > So I'm keen for the ability to toggle the JIT code in the face of bugs > and have it not affect ABI. ?"-Xjit" will do this at runtime (once > that's renamed), but I think it would be useful to be able to toggle the > JIT on/off default during the build, so that I can fix a broken > architecture for non-technical users, but have individual testers opt > back in with -Xjit whilst tracking down a major bug. That's something we can definitely do: you'd just change the default value for the -Xjit flag from "whenhot" to "never". Those individual testers would pass -Xjit=whenhot to opt back in. We could make that a ./configure flag if it would be useful to you and the other distros. > In either case, I don't want to have to recompile 30 extension modules > to try with/without JIT; that would introduce too much change during > bug-hunts, and be no fun at all. That would suck indeed; I want to avoid that. I think that kind of thing falls under PEP 384, which we will have to obey once it is accepted. > (In the blue-sky nirvana future, I'd love to be able to ship > ahead-of-time compiled versions of the stdlib, pre-optimized based on > realworld workloads. ?Back in my reality, though, I have bugs to fix > before I can work on _that_ patch :( ) Reid Kleckner may be looking at that for his Master's project. It's definitely doable. >> My strong preference would be to have the JIT included by default so >> that it receives as much testing as possible. > > Sounds reasonable. ?Hope the above made sense and is useful. Thanks for your perspective, Collin Winter From florent.xicluna at gmail.com Wed Mar 3 01:44:52 2010 From: florent.xicluna at gmail.com (Florent XICLUNA) Date: Wed, 3 Mar 2010 01:44:52 +0100 Subject: [Python-Dev] Deprecation warnings in Python 2.7 In-Reply-To: References: Message-ID: 2010/3/3 Florent Xicluna wrote: > > I will post a list of the DeprecationWarnings in the python trunk, in a > followup message, for a review, and to help the discussion. > Here is the list of the "Classic" DeprecationWarnings: http://paste.pocoo.org/show/184931/ And the list of the "Py3k" DeprecationWarnings and SyntaxWarnings: http://paste.pocoo.org/show/184932/ I expect most of the things in the first list to be removed in 2.7 instead of being only deprecated. Then what is the best approach: 1) to separate better the Py3k warnings giving them a specific subclass Py3kDeprecationWarnings 2) to merge both lists because we consider deprecation related to 3.1 (and newer) ? 3) to keep things unchanged? -- Florent From brett at python.org Wed Mar 3 01:51:37 2010 From: brett at python.org (Brett Cannon) Date: Tue, 2 Mar 2010 16:51:37 -0800 Subject: [Python-Dev] Deprecation warnings in Python 2.7 In-Reply-To: References: Message-ID: On Tue, Mar 2, 2010 at 15:55, Florent Xicluna wrote: > Hello, > > I would like to open a discussion on the meaning of deprecation warnings in > 2.7. > I assume, but I may be wrong, that 2.7 will be the last version in the 2.x > branch. With this assumption, we should not find many things deprecated in > 2.7 > final. > > On the other hand, the list of py3k deprecation warnings is increasing a > lot. > And more users will probably start to move from 2.7 to the 3.1 release. > > While working on ticket #7092 and #7849, we started to discuss some > proposals > to simplify the deprecation warnings for 2.7. > > We discussed these 2 proposals: > > 1) in 2.6 there's no distinction between py3k and normal deprecations: > they > share the same category, DeprecationWarning. The "-3" switch enables > the > py3k DeprecationWarning and SyntaxWarning. > Do we need to introduce a subclass Py3kDeprecationWarning in 2.7? It > will > make it easier to separate both kinds of deprecations (2.x and 3.x), and > to filter them. > > 2) a different idea is to deprecate the "-3" switch and consider all py3k > deprecations as normal deprecations. They will be hidden by default > thanks to #7309. Since there's no future for the 2.x branch, it seems > normal to consider the migration from 2.7 to 3.1 and show these warnings > when the developer uses "-Wd" switch. > > What do you expect as DeprecationWarning in 2.7? > I like option 2; we are serious about trying to make 2.7 the last in the 2.x series, which means if you care about deprecations then you are caring about Python 3. So I say make -3 the equivalent of -Wd in Python 2.7 -Brett > > I will post a list of the DeprecationWarnings in the python trunk, in a > followup message, for a review, and to help the discussion. > > Best regards, > > -- > Florent Xicluna > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Wed Mar 3 02:28:11 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Tue, 2 Mar 2010 17:28:11 -0800 Subject: [Python-Dev] Deprecation warnings in Python 2.7 In-Reply-To: References: Message-ID: <2DFA312B-72C0-4EC6-B703-4076F7C945F7@gmail.com> > > On Tue, Mar 2, 2010 at 15:55, Florent Xicluna wrote: > Hello, > > I would like to open a discussion on the meaning of deprecation warnings in 2.7. > I assume, but I may be wrong, that 2.7 will be the last version in the 2.x > branch. With this assumption, we should not find many things deprecated in 2.7 > final. > > On the other hand, the list of py3k deprecation warnings is increasing a lot. > And more users will probably start to move from 2.7 to the 3.1 release. > > While working on ticket #7092 and #7849, we started to discuss some proposals > to simplify the deprecation warnings for 2.7. > > We discussed these 2 proposals: > > 1) in 2.6 there's no distinction between py3k and normal deprecations: they > share the same category, DeprecationWarning. The "-3" switch enables the > py3k DeprecationWarning and SyntaxWarning. > Do we need to introduce a subclass Py3kDeprecationWarning in 2.7? It will > make it easier to separate both kinds of deprecations (2.x and 3.x), and > to filter them. > > 2) a different idea is to deprecate the "-3" switch and consider all py3k > deprecations as normal deprecations. -1 for several reasons. 1) We've advertised -3 as part of TheOneTrueWay(tm). It's a core part of our transition story, so we should keep that as clean/consistent as possible. Deprecating the -3 switch seems like shooting ourselves in the foot. 2) There is some chance that there will be a 2.8, so it isn't helpful to burn down our bridges. ISTM, nothing is currently broken and in need of "fixing" in 2.7. I don't see any advantage to conflating py3k warnings with other deprecations. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel at stutzbachenterprises.com Wed Mar 3 03:54:20 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Tue, 2 Mar 2010 20:54:20 -0600 Subject: [Python-Dev] Caching function pointers in type objects Message-ID: In CPython, is it safe to cache function pointers that are in type objects? For example, if I know that some_type->tp_richcompare is non-NULL, and I call it (which may execute arbitrary user code), can I assume that some_type->tp_richcompare is still non-NULL? -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Wed Mar 3 03:57:31 2010 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 2 Mar 2010 20:57:31 -0600 Subject: [Python-Dev] Caching function pointers in type objects In-Reply-To: References: Message-ID: <1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com> 2010/3/2 Daniel Stutzbach : > In CPython, is it safe to cache function pointers that are in type objects? > > For example, if I know that some_type->tp_richcompare is non-NULL, and I > call it (which may execute arbitrary user code), can I assume that > some_type->tp_richcompare is still non-NULL? Not unless it's builtin. Somebody could have deleted the rich comparison methods. -- Regards, Benjamin From rnk at mit.edu Wed Mar 3 04:06:41 2010 From: rnk at mit.edu (Reid Kleckner) Date: Tue, 2 Mar 2010 22:06:41 -0500 Subject: [Python-Dev] Caching function pointers in type objects In-Reply-To: <1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com> References: <1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com> Message-ID: <9a9942201003021906m22728b35p57608f199ee1f79d@mail.gmail.com> I don't think this will help you solve your problem, but one thing we've done in unladen swallow is to hack PyType_Modified to invalidate our own descriptor caches. We may eventually want to extend that into a callback interface, but it probably will never be considered an API that outside code should depend on. Reid On Tue, Mar 2, 2010 at 9:57 PM, Benjamin Peterson wrote: > 2010/3/2 Daniel Stutzbach : >> In CPython, is it safe to cache function pointers that are in type objects? >> >> For example, if I know that some_type->tp_richcompare is non-NULL, and I >> call it (which may execute arbitrary user code), can I assume that >> some_type->tp_richcompare is still non-NULL? > > Not unless it's builtin. Somebody could have deleted the rich > comparison methods. > > > > -- > Regards, > Benjamin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/reid.kleckner%40gmail.com > From barry at python.org Wed Mar 3 04:17:44 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 2 Mar 2010 22:17:44 -0500 Subject: [Python-Dev] Python 2.6.5 release candidate 1 now available Message-ID: <20100302221744.7ee4d3dd@heresy.wooz.org> Hello everyone, The source tarballs and Windows installer for Python 2.6.5 release candidate 1 are now available: http://www.python.org/download/releases/2.6.5/ Please download them, install them, and try to use them with your favorite projects and environments. If no regressions are found, we will do the final release on Monday March 15, 2010. Please test the release candidate as much as possible in the meantime, and help make 2.6.5 a rock solid release! Thanks, -Barry P.S. The Mac installer will hopefully be available soon. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From solipsis at pitrou.net Wed Mar 3 04:17:13 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 2 Mar 2010 22:17:13 -0500 Subject: [Python-Dev] code.python.org dead? In-Reply-To: <4B8D8911.1070003@v.loewis.de> References: <20100302103212.69669f0f@msiwind> <4B8D8911.1070003@v.loewis.de> Message-ID: <20100302221713.5882549b@msiwind> Le Tue, 02 Mar 2010 22:54:25 +0100, "Martin v. L?wis" a ?crit : > > > This is annoying since there were Mercurial mirrors there > > ( at http://code.python.org/hg ). Until we make the official switch, > > these mirrors are still useful. > > Do you know who is in charge of creating the site? I could restore it, > but I would need help to make it actually work (such as giving it a > working DocumentRoot). I actually sent a template of the actual Apache configuration long ago, to Neil Schemenauer I think (my memory may be failing on that). I normally have ssh access to the machine using the "svn2hg" account, but I don't think I can actually use that account until I return from vacation - that is, until mid-March. The setup was a quite standard hgwebdir wsgi script hooked into mod_wsgi. Also, if the mirrors themselves have been wiped as well, they will have to be regenerated (hopefully yielding the same changeset ids :-)). Regards Antoine. From solipsis at pitrou.net Wed Mar 3 04:24:00 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 2 Mar 2010 22:24:00 -0500 Subject: [Python-Dev] Deprecation warnings in Python 2.7 References: <2DFA312B-72C0-4EC6-B703-4076F7C945F7@gmail.com> Message-ID: <20100302222400.2e18ccef@msiwind> Le Tue, 2 Mar 2010 17:28:11 -0800, Raymond Hettinger a ?crit : > -1 for several reasons. > > 1) We've advertised -3 as part of TheOneTrueWay(tm). It's a core > part of our transition story, so we should keep that as > clean/consistent as possible. Deprecating the -3 switch seems like > shooting ourselves in the foot. > > 2) There is some chance that there will be a 2.8, so it isn't helpful > to burn down our bridges. That's my feeling too. I think introducing a Py3kDeprecationWarning subclass is the right thing to do. Regards Antoine. From benjamin at python.org Wed Mar 3 05:07:36 2010 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 2 Mar 2010 22:07:36 -0600 Subject: [Python-Dev] Deprecation warnings in Python 2.7 In-Reply-To: <20100302222400.2e18ccef@msiwind> References: <2DFA312B-72C0-4EC6-B703-4076F7C945F7@gmail.com> <20100302222400.2e18ccef@msiwind> Message-ID: <1afaf6161003022007q7f723630n555f0834dea33096@mail.gmail.com> 2010/3/2 Antoine Pitrou : > Le Tue, 2 Mar 2010 17:28:11 -0800, > Raymond Hettinger a ?crit : >> -1 for several reasons. >> >> 1) We've advertised -3 as part of TheOneTrueWay(tm). ?It's a core >> part of our transition story, so we should keep that as >> clean/consistent as possible. ?Deprecating the -3 switch seems like >> shooting ourselves in the foot. >> >> 2) There is some chance that there will be a 2.8, so it isn't helpful >> to burn down our bridges. > > That's my feeling too. I think introducing a Py3kDeprecationWarning > subclass is the right thing to do. Not to bikeshed, but I think it should be called Py3DeprecationWarning in light of the fact that Python 3 is no longer a mythical beast. :) -- Regards, Benjamin From stefan_ml at behnel.de Wed Mar 3 08:18:08 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 03 Mar 2010 08:18:08 +0100 Subject: [Python-Dev] ElementTree.tostring() returning str by default in Py3.2 In-Reply-To: References: Message-ID: Stefan Behnel, 02.03.2010 10:06: > I just noticed that the xml.etree.ElementTree.tostring() function in Py3.2 > returns a str object by default, unless an encoding is specified. This is a > backwards incompatible change compared to ET 1.2. For one, it breaks tons > of tests in lxml's compatibility test suite. Previously, the default > encoding was 'ASCII', and tostring() was guaranteed to always return a byte > string. > > According to svn blame, this was changed by GvR in rev. 56841. > > Is there any reasoning behind this? I filed a bug report for now. http://bugs.python.org/issue8047 Stefan From ncoghlan at gmail.com Wed Mar 3 11:35:33 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 03 Mar 2010 20:35:33 +1000 Subject: [Python-Dev] Deprecation warnings in Python 2.7 In-Reply-To: <20100302222400.2e18ccef@msiwind> References: <2DFA312B-72C0-4EC6-B703-4076F7C945F7@gmail.com> <20100302222400.2e18ccef@msiwind> Message-ID: <4B8E3B75.1000004@gmail.com> Antoine Pitrou wrote: > Le Tue, 2 Mar 2010 17:28:11 -0800, > Raymond Hettinger a ?crit : >> -1 for several reasons. >> >> 1) We've advertised -3 as part of TheOneTrueWay(tm). It's a core >> part of our transition story, so we should keep that as >> clean/consistent as possible. Deprecating the -3 switch seems like >> shooting ourselves in the foot. >> >> 2) There is some chance that there will be a 2.8, so it isn't helpful >> to burn down our bridges. > > That's my feeling too. I think introducing a Py3kDeprecationWarning > subclass is the right thing to do. Agreed for the reasons Raymond cited. As far as the subclass goes, I agree that's a good idea to allow people to readily detect whether they tripped over a normal deprecation or a Py3 only deprecation when running with -3 enabled. As far as naming goes, I agree with Py3kDeprecationWarning since the flag to enable them is called sys.py3kwarning. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From florent.xicluna at gmail.com Wed Mar 3 11:39:01 2010 From: florent.xicluna at gmail.com (Florent Xicluna) Date: Wed, 3 Mar 2010 11:39:01 +0100 Subject: [Python-Dev] Deprecation warnings in Python 2.7 In-Reply-To: <2DFA312B-72C0-4EC6-B703-4076F7C945F7@gmail.com> References: <2DFA312B-72C0-4EC6-B703-4076F7C945F7@gmail.com> Message-ID: 2010/3/3 Raymond Hettinger : > > -1 for several reasons. > 1) We've advertised -3 as part of TheOneTrueWay(tm). ?It's a core part of > our transition story, so we should keep that as clean/consistent as > possible. ?Deprecating the -3 switch seems like shooting ourselves in the > foot. Instead of deprecating the -3 switch, we can change it to a useful alias (see below). > 2) There is some chance that there will be a 2.8, so it isn't helpful to > burn down our bridges. > ISTM, nothing is currently broken and in need of "fixing" in 2.7. > I don't see any advantage to conflating py3k warnings with other > deprecations. > IMHO, the current deprecation and warning mechanism is not far from a Rube Goldberg machine. There's many options available, and it's not obvious both for the core developer and for the end user. On the python interpreter side, you have many options for the careful developer: -Wdefault: display all warnings -3: display all warnings except ImportWarning and PendingDeprecationWarning -Qwarn does nothing if -Wd is omitted, else warn about 2/1 or 2L/1 -Qwarnall does nothing if -Wd is omitted, else warn about 2.0/1 or 2j/1 -b/-bb (undocumented), warns about u'' == bytearray('') -t/-tt warns about tab inconsistencies Why -Qwarn needs -Wd to show its warnings? And you should know that -3 implies -Qwarn, but it still mask PendingDeprecationWarnings. So you will not see the PendingDeprecationWarning issued by cgi.parse_qs or shutil module (for example). At the same time, you could notice that the mhlib module yields a non-py3k DeprecationWarning about its removal in 3.0. And now you want to compare unicode with bytes (ok, it's a very bad idea, but it may happen when you sort dictionary keys, for example): ~ $ ./python >>> u'\xff' == bytearray('\xff') # No warning? False >>> u'\xff' == '\xff' __main__:1: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal False ~ $ ./python -Wd -bb >>> u'\xff' == bytearray('\xff') # It should raise an error because of '-bb' __main__:1: BytesWarning: Comparison between bytearray and string False >>> u'\xff' == '\xff' __main__:1: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal False Yeah, it may be confusing... "Comparison between bytearray and string" is not really the correct message in 2.7. And why to keep these 2 warnings categories which are not used anywhere else in Python source code? The behavior should be consistent between these 2 warnings, and they may share the same category. Why we don't use a RuntimeWarning here? I still support the simplification of the warnings mechanism: * -Qwarn/-Qwarnall should imply -Wdefault -Wignore::ImportWarning -Wignore::PendingDeprecationWarning (same as current -3 behavior) * BytesWarning should be replaced by UnicodeWarning or RuntimeWarning (and -b deprecated) * -Wdefault should show all warnings (including py3k warnings) * -3 should show the PendingDeprecationWarning in addition to its current behavior. (i.e. an alias for -Wdefault -Wignore::ImportWarning) -- Florent From cournape at gmail.com Wed Mar 3 11:51:54 2010 From: cournape at gmail.com (David Cournapeau) Date: Wed, 3 Mar 2010 19:51:54 +0900 Subject: [Python-Dev] PEP 3188: Implementation Questions In-Reply-To: <4095897c1002252051w24377ec2tc60d0b04adee8c5d@mail.gmail.com> References: <4095897c1002252051w24377ec2tc60d0b04adee8c5d@mail.gmail.com> Message-ID: <5b8d13221003030251r52ecc55do3ae7e9c773b8c4c9@mail.gmail.com> On Fri, Feb 26, 2010 at 1:51 PM, Meador Inge wrote: > Hi All, > > Recently some discussion began in the issue 3132 thread > (http://bugs.python.org/issue3132) regarding > implementation of the new struct string syntax for PEP 3118.? Mark Dickinson > suggested that I bring the discussion on over to Python Dev.? Below is a > summary > of the questions\comments from the thread. > > Unpacking a long-double > =================== > > 1. Should this return a Decimal object or a ctypes 'long double'? > 2. Using ctypes 'long double' is easier to implement, but precision is > ??? lost when needing to do arithmetic, since the value for cytpes 'long > double' > ??? is converted to a Python float. > 3. Using Decimal keeps the desired precision, but the implementation would > ??? be non-trivial and architecture specific (unless we just picked a > ??? fixed number of bytes regardless of the architecture). > 4. What representation should be used for standard size and alignment? > ??? IEEE 754 extended double precision? I think supporting even basic arithmetic correctly for long double would be a tremendous amount of work in python. First, as you know, there are many different formats which depend not only on the CPU but also on the OS and the compiler, but there are quite a few issues which are specific to long double (like converting to an integer which cannot fit in any C integer type on most implementations). Also, IEEE 754 does not define any alignment as far as I know, that's up to the CPU implementer I believe. In Numpy, long double usually maps to either 12 bytes (np.float96) or 16 bytes (np.float128). I would expect the long double to be mostly useful for data exchange - if you want to do arithmetic on long double, then the user of the buffer protocol would have to implement it by himself (like NumPy does ATM). So the important thing is to have enough information to use the long double: alignment and size are not enough. cheers, David From florent.xicluna at gmail.com Wed Mar 3 12:00:05 2010 From: florent.xicluna at gmail.com (Florent Xicluna) Date: Wed, 3 Mar 2010 12:00:05 +0100 Subject: [Python-Dev] Deprecation warnings in Python 2.7 In-Reply-To: References: <2DFA312B-72C0-4EC6-B703-4076F7C945F7@gmail.com> Message-ID: 2010/3/3 Florent Xicluna : > >>>> u'\xff' == bytearray('\xff') ? ?# It should raise an error because of '-bb' > __main__:1: BytesWarning: Comparison between bytearray and string > False >>>> u'\xff' == '\xff' > __main__:1: UnicodeWarning: Unicode equal comparison failed to convert > both arguments to Unicode - interpreting them as being unequal > False I see that the BytesWarning (and -b option) is a 3.x feature. I don't see the reason to keep it in 2.x, though.... On the other side, UnicodeWarning is unused in 3.x, why we keep it around? We may phase it out in 3.2? -- Florent From martin at v.loewis.de Wed Mar 3 20:43:58 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 03 Mar 2010 20:43:58 +0100 Subject: [Python-Dev] code.python.org dead? In-Reply-To: <20100302103212.69669f0f@msiwind> References: <20100302103212.69669f0f@msiwind> Message-ID: <4B8EBBFE.40401@v.loewis.de> > It seems someone has decided that code.python.org doesn't resolve > anymore. I have now restored it. Regards, Martin From henning.vonbargen at arcor.de Wed Mar 3 21:30:34 2010 From: henning.vonbargen at arcor.de (Henning von Bargen) Date: Wed, 03 Mar 2010 21:30:34 +0100 Subject: [Python-Dev] __file__ In-Reply-To: References: Message-ID: <4B8EC6EA.3050603@arcor.de> Nick Coghlan wrote: > Another option is to remove bytecode-only support from the default > filesystem importer, but keep it for zipimport (since the stat call > savings don't apply in the latter case). +1 Baptiste Carvello wrote: > However, making a difference between zipimport and the filesystem importer means > the application will stop working if I unzip the library zip file, which is > surprising. > Unzipping the zip file can be handy when debugging a bug caused by a > forgotten module. If the ZIP contains only bytecode files, it is just not intended for changing any code, so I don't think this is an argument. If you have access to the source code, you still can use that instead of messing around with byte code. Henning From daniel at stutzbachenterprises.com Wed Mar 3 22:24:08 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Wed, 3 Mar 2010 15:24:08 -0600 Subject: [Python-Dev] Caching function pointers in type objects In-Reply-To: <9a9942201003021906m22728b35p57608f199ee1f79d@mail.gmail.com> References: <1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com> <9a9942201003021906m22728b35p57608f199ee1f79d@mail.gmail.com> Message-ID: On Tue, Mar 2, 2010 at 9:06 PM, Reid Kleckner wrote: > I don't think this will help you solve your problem, but one thing > we've done in unladen swallow is to hack PyType_Modified to invalidate > our own descriptor caches. We may eventually want to extend that into > a callback interface, but it probably will never be considered an API > that outside code should depend on. > Thanks Reid and Benjamin for the information. I think I see a way to dramatically speed up PyObject_RichCompareBool when comparing immutable, built-in, non-container objects (int, float, str, etc.). It would speed up list.sort when the key is one of those types, as well as most operations on the ubiquitous dictionary with str keys. Is that a worthwhile avenue to pursue, or is it likely to be redundant with Unladen Swallow's optimizations? If I can find time to pursue it, would it be best for me to implement it as a patch to Unladen Swallow, CPython trunk, or CPython py3k? -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Wed Mar 3 22:29:25 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 3 Mar 2010 15:29:25 -0600 Subject: [Python-Dev] Caching function pointers in type objects In-Reply-To: References: <1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com> <9a9942201003021906m22728b35p57608f199ee1f79d@mail.gmail.com> Message-ID: <1afaf6161003031329t2574e8a9qf5fef6fb4b0f33e4@mail.gmail.com> 2010/3/3 Daniel Stutzbach : > I think I see a way to dramatically speed up PyObject_RichCompareBool when > comparing immutable, built-in, non-container objects (int, float, str, > etc.).? It would speed up list.sort when the key is one of those types, as > well as most operations on the ubiquitous dictionary with str keys. > > Is that a worthwhile avenue to pursue, or is it likely to be redundant with > Unladen Swallow's optimizations? Perhaps you could explain what exactly you want to do. :) That would help us make a judgment. > > If I can find time to pursue it, would it be best for me to implement it as > a patch to Unladen Swallow, CPython trunk, or CPython py3k? Your choice. -- Regards, Benjamin From python at mrabarnett.plus.com Wed Mar 3 22:34:14 2010 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 03 Mar 2010 21:34:14 +0000 Subject: [Python-Dev] Matching multiple regexes Message-ID: <4B8ED5D6.4060206@mrabarnett.plus.com> I've thought of a possible additional feature for my regex module and I'm looking for opinions. Occasionally there's a question about matching multiple regexes and someone might suggest combining them into one regex using "|". The feature would be to allow regex.compile, etc, to accept a list or tuple of regex strings. These would be combined into a single regex internally. The match object would get a new attribute .index which would give the index of the regex string which matched. An additional feature could be to allow regex.sub to accept as a replacement any object which has the __getitem__ method and call that method with the match index, if multiple regexes were used. (A replacement string would still behave the same.) This could mean that the following code would become possible: >>> subs = [("foo", "bar"), ("baz", "quux"), ("quuux", "foo")] >>> old_string, new_string = zip(*subs) >>> s = "fooxxxbazyyyquuux" >>> regex.sub(old_string, new_string, s) "barxxxquuxyyyfoo" From daniel at stutzbachenterprises.com Wed Mar 3 22:57:54 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Wed, 3 Mar 2010 15:57:54 -0600 Subject: [Python-Dev] Caching function pointers in type objects In-Reply-To: <1afaf6161003031329t2574e8a9qf5fef6fb4b0f33e4@mail.gmail.com> References: <1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com> <9a9942201003021906m22728b35p57608f199ee1f79d@mail.gmail.com> <1afaf6161003031329t2574e8a9qf5fef6fb4b0f33e4@mail.gmail.com> Message-ID: On Wed, Mar 3, 2010 at 3:29 PM, Benjamin Peterson wrote: > 2010/3/3 Daniel Stutzbach : > > I think I see a way to dramatically speed up PyObject_RichCompareBool > when > > comparing immutable, built-in, non-container objects (int, float, str, > > etc.). It would speed up list.sort when the key is one of those types, > as > > well as most operations on the ubiquitous dictionary with str keys. > (correcting myself) I just noticed that CPython already optimizes dictionaries with str-only keys and skips PyObject_RichCompareBool, so no speed up there. I think it would be redundant with optimization I have in mind, so it could perhaps be taken out which would simplify the dict code a bit and save a pointer in the dict structure. > Perhaps you could explain what exactly you want to do. :) That would > help us make a judgment. > It'd actually be a pretty small patch, so I think I should just explain it by actually writing it. ;-) -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdrake at acm.org Wed Mar 3 23:15:37 2010 From: fdrake at acm.org (Fred Drake) Date: Wed, 3 Mar 2010 17:15:37 -0500 Subject: [Python-Dev] Release blocker: urllib2 issue 7540 Message-ID: <9cee7ab81003031415p717420f3j89ebf1116a76f790@mail.gmail.com> Regarding: http://bugs.python.org/issue7540 The change made in response to this issue report breaks existing uses of Python 2.6.4. The mechanize library frequently re-initializes the data in the request without re-using the request. This worked fine in the past, and does not trigger the problem reported without the request being re-used. Applications (including tests) that use mechanize now break with this TypeError. This includes much (if not all) of the Zope community, which uses mechanize via zope.testbrowser. The proper response to this issue for Python 2.6 is to make no code changes (though a documentation enhancement may be in order). This change should be reverted from all branches. Python 2.6.5 should be blocked until this is done. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From benjamin at python.org Wed Mar 3 23:23:11 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 3 Mar 2010 16:23:11 -0600 Subject: [Python-Dev] [Pydotorg] FWD: Broken link in 2.6.5 rc 1 download page. In-Reply-To: <4B8EC488.10104@python.org> References: <4B8EC488.10104@python.org> Message-ID: <1afaf6161003031423q66d9babel67bac9b9282b79ca@mail.gmail.com> 2010/3/3 Michael Foord > > As far as I can tell there is no Python 2.6.5 documentation online and the link to 3.1 docs is actually for 3.1.1 whilst the link for 3.1.1 docs is dead. I've now fixed the 3.1.1 docs link. -- Regards, Benjamin From collinwinter at google.com Wed Mar 3 23:34:44 2010 From: collinwinter at google.com (Collin Winter) Date: Wed, 3 Mar 2010 14:34:44 -0800 Subject: [Python-Dev] Caching function pointers in type objects In-Reply-To: References: <1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com> <9a9942201003021906m22728b35p57608f199ee1f79d@mail.gmail.com> Message-ID: <3c8293b61003031434t5de492f7n368c92767e28376e@mail.gmail.com> Hey Daniel, On Wed, Mar 3, 2010 at 1:24 PM, Daniel Stutzbach wrote: > On Tue, Mar 2, 2010 at 9:06 PM, Reid Kleckner wrote: >> >> I don't think this will help you solve your problem, but one thing >> we've done in unladen swallow is to hack PyType_Modified to invalidate >> our own descriptor caches. ?We may eventually want to extend that into >> a callback interface, but it probably will never be considered an API >> that outside code should depend on. > > Thanks Reid and Benjamin for the information. > > I think I see a way to dramatically speed up PyObject_RichCompareBool when > comparing immutable, built-in, non-container objects (int, float, str, > etc.).? It would speed up list.sort when the key is one of those types, as > well as most operations on the ubiquitous dictionary with str keys. That definitely sounds worth pursuing. > Is that a worthwhile avenue to pursue, or is it likely to be redundant with > Unladen Swallow's optimizations? I don't believe it will be redundant with the optimizations in Unladen Swallow. > If I can find time to pursue it, would it be best for me to implement it as > a patch to Unladen Swallow, CPython trunk, or CPython py3k? I would recommend patching py3k, with a backport to trunk. Thanks, Collin Winter From daniel at stutzbachenterprises.com Wed Mar 3 23:41:50 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Wed, 3 Mar 2010 16:41:50 -0600 Subject: [Python-Dev] Caching function pointers in type objects In-Reply-To: <3c8293b61003031434t5de492f7n368c92767e28376e@mail.gmail.com> References: <1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com> <9a9942201003021906m22728b35p57608f199ee1f79d@mail.gmail.com> <3c8293b61003031434t5de492f7n368c92767e28376e@mail.gmail.com> Message-ID: On Wed, Mar 3, 2010 at 4:34 PM, Collin Winter wrote: > I would recommend patching py3k, with a backport to trunk. > After thinking it over, I'm inclined to patch trunk, so I can run the Unladen Swallow macro-benchmarks, then forward-port to py3k. I'm correct in understanding that it will be a while before the Unladen Swallow benchmarks can support Python 3, right? -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfoord at python.org Wed Mar 3 21:20:24 2010 From: mfoord at python.org (Michael Foord) Date: Wed, 03 Mar 2010 20:20:24 +0000 Subject: [Python-Dev] FWD: Broken link in 2.6.5 rc 1 download page. Message-ID: <4B8EC488.10104@python.org> As far as I can tell there is no Python 2.6.5 documentation online and the link to 3.1 docs is actually for 3.1.1 whilst the link for 3.1.1 docs is dead. The actual documentation doesn't seem to be in the pydotorg svn repository so I guess this a release manager issue? Michael Foord -------- Original Message -------- Subject: [Pydotorg] FWD: Broken link in 2.6.5 rc 1 download page. Date: Wed, 3 Mar 2010 08:45:36 -0800 From: Aahz Organization: The Cat & Dragon To: pydotorg at python.org ----- Forwarded message from Simon de Vlieger ----- > From: Simon de Vlieger > To: webmaster at python.org > Subject: Broken link in 2.6.5 rc 1 download page. > Date: Wed, 3 Mar 2010 09:57:29 +0100 > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Dear webmaster (I know no name), > > on this page: http://www.python.org/download/releases/2.6.5/ both links > for the documentation are incorrect. On this page: > http://www.python.org/doc/versions/ the link for the Python 3.1.1 is > broken. > > Hope to have informed you well enough. > > Regards, > > Simon de Vlieger > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.9 (Darwin) > > iQIcBAEBAgAGBQJLjiR5AAoJEBBSHP7i+JXfidsP/0fLCK0sn5+mj9CPsobDant7 > 1tkXK7pwrbujXhpAyhjhE/kB5BhNI/h9XpmmUkLSZzz7AoOpE6KzlqCzOmwBmdAk > 6LxO2c4ZojICVB5zgNYXibDtnSwbBhcefbEsN6BSh1kI/3ThcIdWRYFavBAlxj1T > 6P7y8MOHUn3INLEw8foxV20geivcqzmMhRtfglMVs9KcB/PZWrK8fWw8K2Ov0B+U > ihxaywfiJWcdkFKTTLNvQCbVr6ZGke728/7/lcWLvdPlwRPfVteKlLDglWX0beBy > tgQyxmtxFx+AeCiLeyybnxEnxRVgU1nYxsdWluJuwFT4l+/BsJG3CkM522rQ3MWZ > +DYB6xfONlvfCSXaPn3fOYesEvXek9BxKLiXpjWlJsmXvDcdFN2sr24aR1cJEXL8 > ZRkiJO5VFzngIxytk0Oy/SeMkxCUtsq4TMlMT2OyxblxxRoLikPSsfgKRuDkQ5s+ > QK2yNxTgKs2yA5ux9kmZscmVgHcWqJFjkE72dJoARyHMra9Jvcr2U7GVE2zqcOCS > rObaqq61TpZmvlPUXlkLUueTUI9Djz01hAQUp+Vcwu8omVlIgZC0GbuXK8JDfQJd > 60WFI1BvUQg7mDB1ThUU6HUE9rF71921lx02YyZFzGFn1K2/w/bRhxrrzD5XwJDQ > fFwYKj+pwgHO5AU5hMCR > =rTPq > -----END PGP SIGNATURE----- ----- End forwarded message ----- -- Aahz (aahz at pythoncraft.com)<*> http://www.pythoncraft.com/ "Many customs in this life persist because they ease friction and promote productivity as a result of universal agreement, and whether they are precisely the optimal choices is much less important." --Henry Spencer _______________________________________________ Pydotorg mailing list Pydotorg at python.org http://mail.python.org/mailman/listinfo/pydotorg -------------- next part -------------- An HTML attachment was scrubbed... URL: From collinwinter at google.com Wed Mar 3 23:50:02 2010 From: collinwinter at google.com (Collin Winter) Date: Wed, 3 Mar 2010 14:50:02 -0800 Subject: [Python-Dev] Caching function pointers in type objects In-Reply-To: References: <1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com> <9a9942201003021906m22728b35p57608f199ee1f79d@mail.gmail.com> <3c8293b61003031434t5de492f7n368c92767e28376e@mail.gmail.com> Message-ID: <3c8293b61003031450p4525e308qb7fdca502d90728a@mail.gmail.com> On Wed, Mar 3, 2010 at 2:41 PM, Daniel Stutzbach wrote: > On Wed, Mar 3, 2010 at 4:34 PM, Collin Winter > wrote: >> >> I would recommend patching py3k, with a backport to trunk. > > After thinking it over, I'm inclined to patch trunk, so I can run the > Unladen Swallow macro-benchmarks, then forward-port to py3k. > > I'm correct in understanding that it will be a while before the Unladen > Swallow benchmarks can support Python 3, right? That's correct; porting the full benchmark suite to Python 3 will require projects like Django to support Python 3. Collin Winter From barry at python.org Thu Mar 4 00:04:51 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 3 Mar 2010 18:04:51 -0500 Subject: [Python-Dev] __file__ In-Reply-To: <201003022106.17179.steve@pearwood.info> References: <20100301194946.188a8855@freewill.wooz.org> <20100301195955.3091dc19@freewill.wooz.org> <201003022106.17179.steve@pearwood.info> Message-ID: <20100303180451.61929d09@heresy.wooz.org> On Mar 02, 2010, at 09:06 PM, Steven D'Aprano wrote: >(1) What happens if the __cache__ directory doesn't exist and the >enclosing directory is unwriteable, or if it does exist, but is >unreadable? > >I expect that the byte code files will simply not be created, and >everything will continue without them. s/__cache__/__pycache__ but yes, just as it does today. >(2) Presumably this only effects imports, not running python source code >as a script. If I do this: > >python myscript.py > >from the shell, I would expect that no __cache__ directory will be >created, just like today. Correct. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ziade.tarek at gmail.com Thu Mar 4 00:54:02 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 4 Mar 2010 00:54:02 +0100 Subject: [Python-Dev] The fate of Distutils in Python 2.7 In-Reply-To: <94bdd2611002261344q361e702av10fdf99ea5eb9f71@mail.gmail.com> References: <94bdd2611002261344q361e702av10fdf99ea5eb9f71@mail.gmail.com> Message-ID: <94bdd2611003031554o495fcfd8kb24b61c3b6f63427@mail.gmail.com> Just FYI : I am post-poning the revert of Distutils to 2.6.x right after 2.7a4 has been tagged to avoid any problems (this is in 3 days) The revert is ready but 3 days is not long enough to make sure everything is going smooth. On Fri, Feb 26, 2010 at 10:44 PM, Tarek Ziad? wrote: > Hello, > > This is a follow-up of the Pycon summit + sprints on packaging. > > This is what we have planned to do: > > 1. refactor distutils in a new standalone version called distutils2 > [this is done already and we are actively working in the code] > 2. completely revert distutils in Lib/ and Doc/ so the code + doc is > the same than the current 2.6.x branch > 3. leave the new sysconfig module, that is used by the Makefile and > the site module > > The rest of the work will happen in distutils2 and we will try to > release a version asap for Python 2.x and 3.x (2.4 to 3.2), and the > goal > is to put it back in the stdlib in Python 3.3 > > Distutils in Python will be feature-frozen and I will only do bug > fixes there. ?All feature requests will be redirected to Distutils2. > > I think the easiest way to manage this for me and for the feedback of > the community is to add in bugs.python.org a "Distutils2" component, > so I can > start to reorganize the issues in there and reassign new issues to > Distutils2 when it applies. > > Regards > Tarek > > -- > Tarek Ziad? | http://ziade.org > -- Tarek Ziad? | http://ziade.org From jimjjewett at gmail.com Thu Mar 4 01:37:33 2010 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 3 Mar 2010 19:37:33 -0500 Subject: [Python-Dev] __file__ and bytecode-only Message-ID: I understand the need to ship without source -- but why does that require supporting .pyc (or .pyo) -only? Couldn't vendors just replace the real .py files with empty files? Then no one would need the extra stat call, and no one would be bitten by orphaned .pyc files after a rename. [Yes, zips could still allow unmatched names; yes, it would be helpful if a tool were available to sync the last-modification time; yes a deprecation release should still be needed.] -jJ From solipsis at pitrou.net Thu Mar 4 02:21:08 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 3 Mar 2010 20:21:08 -0500 Subject: [Python-Dev] code.python.org dead? In-Reply-To: <4B8EBBFE.40401@v.loewis.de> References: <20100302103212.69669f0f@msiwind> <4B8EBBFE.40401@v.loewis.de> Message-ID: <20100303202108.1f86979d@msiwind> Le Wed, 03 Mar 2010 20:43:58 +0100, "Martin v. L?wis" a ?crit : > > It seems someone has decided that code.python.org doesn't resolve > > anymore. > > I have now restored it. Thanks a lot! Antoine. From barry at python.org Thu Mar 4 02:49:23 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 3 Mar 2010 20:49:23 -0500 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: References: Message-ID: <20100303204923.6d597f03@heresy.wooz.org> On Mar 03, 2010, at 07:37 PM, Jim Jewett wrote: >I understand the need to ship without source -- but why does that >require supporting .pyc (or .pyo) -only? > >Couldn't vendors just replace the real .py files with empty files? Yes, I think that's a possibility. What would people think about that? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From brett at python.org Thu Mar 4 02:51:04 2010 From: brett at python.org (Brett Cannon) Date: Wed, 3 Mar 2010 17:51:04 -0800 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: References: Message-ID: On Wed, Mar 3, 2010 at 16:37, Jim Jewett wrote: > I understand the need to ship without source -- but why does that > require supporting .pyc (or .pyo) -only? > > Couldn't vendors just replace the real .py files with empty files? > Because if someone screws up the mod time on the source files the .pyc files will get recreated silently. -Brett > > Then no one would need the extra stat call, and no one would be bitten > by orphaned .pyc files after a rename. > > [Yes, zips could still allow unmatched names; yes, it would be helpful > if a tool were available to sync the last-modification time; yes a > deprecation release should still be needed.] > > -jJ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Thu Mar 4 03:33:21 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 03 Mar 2010 18:33:21 -0800 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <20100303204923.6d597f03@heresy.wooz.org> References: <20100303204923.6d597f03@heresy.wooz.org> Message-ID: <4B8F1BF1.6060608@g.nevcal.com> On approximately 3/3/2010 5:49 PM, came the following characters from the keyboard of Barry Warsaw: > On Mar 03, 2010, at 07:37 PM, Jim Jewett wrote: > >> >I understand the need to ship without source -- but why does that >> >require supporting .pyc (or .pyo) -only? >> > >> >Couldn't vendors just replace the real .py files with empty files? >> > Yes, I think that's a possibility. What would people think about that? > > -Barry > That's kooky, but not as kooky as my idea. As mentioned elsewhere, timestamps would have to be treated carefully. In this scenario, the .pyc files would still live in __pycache__ ? Complete with the foo..pyc naming ? So one could actually create a "fat .zip" application that could work with a variety of installed Python versions ??? Hmm... -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From greg.ewing at canterbury.ac.nz Thu Mar 4 04:18:09 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 04 Mar 2010 16:18:09 +1300 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <20100303204923.6d597f03@heresy.wooz.org> References: <20100303204923.6d597f03@heresy.wooz.org> Message-ID: <4B8F2671.1030601@canterbury.ac.nz> Barry Warsaw wrote: > On Mar 03, 2010, at 07:37 PM, Jim Jewett wrote: > >> Couldn't vendors just replace the real .py files with empty files? > > Yes, I think that's a possibility. What would people think about that? Seems like a perverse thing to have to do to me. Also a bit fragile, since you would have to make sure that the empty .py files were dated older than the .pyc files and stayed that way, lest Python try to recompile them and wipe out your code. You would also have to be careful to build installers that didn't recompile .py files on installation. (Haven't had much experience building installers using distutils, so I'm not sure how much of a problem that would be.) -- Greg From greg.ewing at canterbury.ac.nz Thu Mar 4 04:22:13 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 04 Mar 2010 16:22:13 +1300 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <4B8F1BF1.6060608@g.nevcal.com> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8F1BF1.6060608@g.nevcal.com> Message-ID: <4B8F2765.4030401@canterbury.ac.nz> Glenn Linderman wrote: > In this scenario, the .pyc files would still live in __pycache__ ? > Complete with the foo..pyc naming ? It might be neater to have a separate cache directory for each bytecode version, named __cache.__ or some such. -- Greg From glyph at twistedmatrix.com Thu Mar 4 04:28:45 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 3 Mar 2010 22:28:45 -0500 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <4B8F2765.4030401@canterbury.ac.nz> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8F1BF1.6060608@g.nevcal.com> <4B8F2765.4030401@canterbury.ac.nz> Message-ID: On Mar 3, 2010, at 10:22 PM, Greg Ewing wrote: > Glenn Linderman wrote: > >> In this scenario, the .pyc files would still live in __pycache__ ? Complete with the foo..pyc naming ? > > It might be neater to have a separate cache directory > for each bytecode version, named __cache.__ or > some such. Okay, this is probably some pretty silly bikeshedding, but: if we're going to have it be something.something-else, can we please make sure that .something-else is a common extension that means "python bytecode cache"? It would be good to keep the file-manager and shell operations required to say "blow away bytecode cache directories" as simple as possible. From solipsis at pitrou.net Thu Mar 4 04:31:15 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 3 Mar 2010 22:31:15 -0500 Subject: [Python-Dev] __file__ and bytecode-only References: Message-ID: <20100303223115.51fe7276@msiwind> Le Wed, 3 Mar 2010 17:51:04 -0800, Brett Cannon a ?crit : > On Wed, Mar 3, 2010 at 16:37, Jim Jewett wrote: > > > I understand the need to ship without source -- but why does that > > require supporting .pyc (or .pyo) -only? > > > > Couldn't vendors just replace the real .py files with empty files? > > > > Because if someone screws up the mod time on the source files > the .pyc files will get recreated silently. Unless the .py files arrange to raise a syntax error on compiling. From solipsis at pitrou.net Thu Mar 4 04:32:53 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 3 Mar 2010 22:32:53 -0500 Subject: [Python-Dev] __file__ and bytecode-only References: <20100303204923.6d597f03@heresy.wooz.org> <4B8F1BF1.6060608@g.nevcal.com> <4B8F2765.4030401@canterbury.ac.nz> Message-ID: <20100303223253.502e1fb0@msiwind> Le Thu, 04 Mar 2010 16:22:13 +1300, Greg Ewing a ?crit : > Glenn Linderman wrote: > > > In this scenario, the .pyc files would still live in __pycache__ ? > > Complete with the foo..pyc naming ? > > It might be neater to have a separate cache directory > for each bytecode version, named __cache.__ or > some such. > Actually, I find it neater to have a single cache directory. It makes for much less clutter, and simpler ignore rules. From greg.ewing at canterbury.ac.nz Thu Mar 4 04:37:04 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 04 Mar 2010 16:37:04 +1300 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: References: <20100303204923.6d597f03@heresy.wooz.org> <4B8F1BF1.6060608@g.nevcal.com> <4B8F2765.4030401@canterbury.ac.nz> Message-ID: <4B8F2AE0.8010400@canterbury.ac.nz> Glyph Lefkowitz wrote: > if we're going to have it be something.something-else, can we please > make sure that .something-else is a common extension that means > "python bytecode cache"? Maybe something like __bytecode-__.pycache ? -- Greg From greg.ewing at canterbury.ac.nz Thu Mar 4 04:39:03 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 04 Mar 2010 16:39:03 +1300 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <20100303223115.51fe7276@msiwind> References: <20100303223115.51fe7276@msiwind> Message-ID: <4B8F2B57.2020404@canterbury.ac.nz> Antoine Pitrou wrote: > Unless the .py files arrange to raise a syntax error on compiling. I guess that prevents a total disaster, but the program is still broken and you have to hunt down the offending files and fix the timestamps -- if it's even evident what the problem is and how to fix it. -- Greg From greg.ewing at canterbury.ac.nz Thu Mar 4 04:48:36 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 04 Mar 2010 16:48:36 +1300 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <20100303223253.502e1fb0@msiwind> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8F1BF1.6060608@g.nevcal.com> <4B8F2765.4030401@canterbury.ac.nz> <20100303223253.502e1fb0@msiwind> Message-ID: <4B8F2D94.8020703@canterbury.ac.nz> Antoine Pitrou wrote: > Actually, I find it neater to have a single cache directory. It makes > for much less clutter, and simpler ignore rules. Another possibility would be to have a single top-level cache directory with a subdirectory for each version: __bytecode__.pycache//.pyc I don't think this ought to increase the stat call count, since the interpreter will always be looking for a particular version of bytecode, so it can just stat for the appropriate subdirectory directly. -- Greg From solipsis at pitrou.net Thu Mar 4 04:54:22 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 3 Mar 2010 22:54:22 -0500 Subject: [Python-Dev] __file__ and bytecode-only References: <20100303223115.51fe7276@msiwind> <4B8F2B57.2020404@canterbury.ac.nz> Message-ID: <20100303225422.39b030a7@msiwind> Le Thu, 04 Mar 2010 16:39:03 +1300, Greg Ewing a ?crit : > Antoine Pitrou wrote: > > > Unless the .py files arrange to raise a syntax error on compiling. > > I guess that prevents a total disaster, but the > program is still broken and you have to hunt down > the offending files and fix the timestamps -- if > it's even evident what the problem is and how to > fix it. > Bytecode-only distributions are supposed to be created by the vendor/distributor. It's up to him/her to get things right. Regards Antoine. From orsenthil at gmail.com Thu Mar 4 05:06:23 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Thu, 4 Mar 2010 09:36:23 +0530 Subject: [Python-Dev] Release blocker: urllib2 issue 7540 In-Reply-To: <9cee7ab81003031415p717420f3j89ebf1116a76f790@mail.gmail.com> References: <9cee7ab81003031415p717420f3j89ebf1116a76f790@mail.gmail.com> Message-ID: <7c42eba11003032006v182315bfn7efb62ebdd632de1@mail.gmail.com> On Thu, Mar 4, 2010 at 3:45 AM, Fred Drake wrote: > Regarding: ?http://bugs.python.org/issue7540 > > The proper response to this issue for Python 2.6 is to make no code > changes (though a documentation enhancement may be in order). > > This change should be reverted from all branches. > > Python 2.6.5 should be blocked until this is done. I have commented on that issue. Ready to revert the change from the branches and keep a doc fix and a warning. -- Senthil From ncoghlan at gmail.com Thu Mar 4 14:25:36 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 04 Mar 2010 23:25:36 +1000 Subject: [Python-Dev] __file__ In-Reply-To: <4B8EC6EA.3050603@arcor.de> References: <4B8EC6EA.3050603@arcor.de> Message-ID: <4B8FB4D0.5060008@gmail.com> Henning von Bargen wrote: > If the ZIP contains only bytecode files, it is just not intended > for changing any code, so I don't think this is an argument. > If you have access to the source code, you still can use that instead > of messing around with byte code. That doesn't apply when it is the app developer trying to figure out why their app is breaking when they throw it in a bytecode only zip :) It's a moot point anyway - Guido has pronounced that the bytecode-only support will be left alone by PEP 3147 even if the other caching changes are eventually accepted. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Mar 4 14:34:10 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 04 Mar 2010 23:34:10 +1000 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <20100303204923.6d597f03@heresy.wooz.org> References: <20100303204923.6d597f03@heresy.wooz.org> Message-ID: <4B8FB6D2.4040501@gmail.com> Barry Warsaw wrote: > On Mar 03, 2010, at 07:37 PM, Jim Jewett wrote: > >> I understand the need to ship without source -- but why does that >> require supporting .pyc (or .pyo) -only? >> >> Couldn't vendors just replace the real .py files with empty files? > > Yes, I think that's a possibility. What would people think about that? I actually thought of this, but didn't post it because it defeats the point of byte-code only distribution. - putting text in the .py file will break the application - touching the .py file in any way will break the application If someone wants to break the bytecode only support it should be its own PEP and not coupled with PEP 3147. The remaining open question to my mind is whether or not there should be a -X option to control the bytecode generation. E.g.: -Xcache_bytecode=no (don't write bytecode files at all) -Xcache_bytecode=file (write a classic "foo.pyc" file) -Xcache_bytecode=dir (write to the "__pycache__" directory) With cache_bytecode=dir being the default for future releases. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Thu Mar 4 16:33:24 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 4 Mar 2010 10:33:24 -0500 Subject: [Python-Dev] __file__ In-Reply-To: <4B8FB4D0.5060008@gmail.com> References: <4B8EC6EA.3050603@arcor.de> <4B8FB4D0.5060008@gmail.com> Message-ID: <20100304103324.219704b9@heresy.wooz.org> On Mar 04, 2010, at 11:25 PM, Nick Coghlan wrote: >It's a moot point anyway - Guido has pronounced that the bytecode-only >support will be left alone by PEP 3147 even if the other caching changes >are eventually accepted. Right. I should have stopped while I was ahead. We'll just keep what's in the PEP. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From brett at python.org Thu Mar 4 21:20:00 2010 From: brett at python.org (Brett Cannon) Date: Thu, 4 Mar 2010 12:20:00 -0800 Subject: [Python-Dev] Desired changes to Hg emails to python-checkins Message-ID: 1) I miss not having the affected files listed in the subject line. 2) The To field is set to hg at python.org which gets rejected as an invalid email address if you reply. Would be better to set it to python-checkins so that the habitual reply to a checkin won't get rejected. At least I would like to see these changes happen. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdrake at acm.org Thu Mar 4 21:37:20 2010 From: fdrake at acm.org (Fred Drake) Date: Thu, 4 Mar 2010 15:37:20 -0500 Subject: [Python-Dev] Desired changes to Hg emails to python-checkins In-Reply-To: References: Message-ID: <9cee7ab81003041237y19b42105y73728db31384964e@mail.gmail.com> On Thu, Mar 4, 2010 at 3:20 PM, Brett Cannon wrote: > 1) I miss not having the affected files listed in the subject line. > 2) The To field is set to hg at python.org which gets rejected as an invalid > email address if you reply. Would be better to set it to python-checkins so > that the habitual reply to a checkin won't get rejected. I concur on both points. You are not alone. :-) -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From ncoghlan at gmail.com Thu Mar 4 21:39:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 05 Mar 2010 06:39:50 +1000 Subject: [Python-Dev] Desired changes to Hg emails to python-checkins In-Reply-To: References: Message-ID: <4B901A96.2000100@gmail.com> Brett Cannon wrote: > 1) I miss not having the affected files listed in the subject line. > > 2) The To field is set to hg at python.org which > gets rejected as an invalid email address if you reply. Would be better > to set it to python-checkins so that the habitual reply to a checkin > won't get rejected. > > At least I would like to see these changes happen. +1 (and it's OK if the file list gets truncated for big checkins - it's still handy in typical cases) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From eric at trueblade.com Thu Mar 4 21:43:06 2010 From: eric at trueblade.com (Eric Smith) Date: Thu, 04 Mar 2010 15:43:06 -0500 Subject: [Python-Dev] Desired changes to Hg emails to python-checkins In-Reply-To: References: Message-ID: <4B901B5A.5040300@trueblade.com> Are the diffs gone for some deliberate reason? I realize the link tells me the changes, but I'll review a lot more code if the diffs show up in my inbox than if I have to fire up a browser, especially from my phone. Eric. Brett Cannon wrote: > 1) I miss not having the affected files listed in the subject line. > > 2) The To field is set to hg at python.org which > gets rejected as an invalid email address if you reply. Would be better > to set it to python-checkins so that the habitual reply to a checkin > won't get rejected. > > At least I would like to see these changes happen. > > -Brett > > > ------------------------------------------------------------------------ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com -- Eric. From ncoghlan at gmail.com Thu Mar 4 21:46:35 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 05 Mar 2010 06:46:35 +1000 Subject: [Python-Dev] Desired changes to Hg emails to python-checkins In-Reply-To: References: Message-ID: <4B901C2B.9010203@gmail.com> Brett Cannon wrote: > 1) I miss not having the affected files listed in the subject line. > > 2) The To field is set to hg at python.org which > gets rejected as an invalid email address if you reply. Would be better > to set it to python-checkins so that the habitual reply to a checkin > won't get rejected. > > At least I would like to see these changes happen. I'd like to keep the inline diffs as well - much easier to skim patches as I click through messages, whereas links to changesets, while useful if I decide to look at a change in more detail, don't tell me a lot at a glance. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Thu Mar 4 21:46:38 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 4 Mar 2010 15:46:38 -0500 Subject: [Python-Dev] Desired changes to Hg emails to python-checkins In-Reply-To: <4B901B5A.5040300@trueblade.com> References: <4B901B5A.5040300@trueblade.com> Message-ID: <20100304154638.0623468e@heresy.wooz.org> On Mar 04, 2010, at 03:43 PM, Eric Smith wrote: >Are the diffs gone for some deliberate reason? > >I realize the link tells me the changes, but I'll review a lot more code >if the diffs show up in my inbox than if I have to fire up a browser, >especially from my phone. Agreed, please restore the diffs! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From dirkjan at ochtman.nl Thu Mar 4 22:37:59 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Thu, 4 Mar 2010 22:37:59 +0100 Subject: [Python-Dev] Desired changes to Hg emails to python-checkins In-Reply-To: References: Message-ID: First off, this was just a first sample, it was by no means the definitive format. Second, the diffs are already back, in the next message (this first one was per changegroup, not per changeset -- new ones will be per changeset). On Thu, Mar 4, 2010 at 21:20, Brett Cannon wrote: > 1) I miss not having the affected files listed in the subject line. Well, I think it looks rather ugly, and Tarek said he'd like the first line of the commit message in the subject instead, so that's what I've implemented right now. But then if there's a majority who want the files, I'm happy to put them back. > 2) The To field is set to hg at python.org which gets rejected as an invalid > email address if you reply. Would be better to set it to python-checkins so > that the habitual reply to a checkin won't get rejected. Will fix. Cheers, Dirkjan From benjamin at python.org Thu Mar 4 22:44:11 2010 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 4 Mar 2010 15:44:11 -0600 Subject: [Python-Dev] Desired changes to Hg emails to python-checkins In-Reply-To: References: Message-ID: <1afaf6161003041344g20791bddpf13d66aa0c4d9438@mail.gmail.com> 2010/3/4 Dirkjan Ochtman : > Well, I think it looks rather ugly, and Tarek said he'd like the first > line of the commit message in the subject instead, so that's what I've > implemented right now. But then if there's a majority who want the > files, I'm happy to put them back. +1 to bringing back affected files. It helps tell at a glance whether a changeset is interesting or not. -- Regards, Benjamin From dirkjan at ochtman.nl Thu Mar 4 22:55:35 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Thu, 4 Mar 2010 22:55:35 +0100 Subject: [Python-Dev] Desired changes to Hg emails to python-checkins In-Reply-To: <1afaf6161003041344g20791bddpf13d66aa0c4d9438@mail.gmail.com> References: <1afaf6161003041344g20791bddpf13d66aa0c4d9438@mail.gmail.com> Message-ID: On Thu, Mar 4, 2010 at 22:44, Benjamin Peterson wrote: > +1 to bringing back affected files. It helps tell at a glance whether > a changeset is interesting or not. Do they need to be in the subject, or would it be fine to have them in the message? Cheers, Dirkjan From benjamin at python.org Thu Mar 4 22:56:58 2010 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 4 Mar 2010 15:56:58 -0600 Subject: [Python-Dev] Desired changes to Hg emails to python-checkins In-Reply-To: References: <1afaf6161003041344g20791bddpf13d66aa0c4d9438@mail.gmail.com> Message-ID: <1afaf6161003041356h6e9a971ex3b7c57c597c29226@mail.gmail.com> 2010/3/4 Dirkjan Ochtman : > On Thu, Mar 4, 2010 at 22:44, Benjamin Peterson wrote: >> +1 to bringing back affected files. It helps tell at a glance whether >> a changeset is interesting or not. > > Do they need to be in the subject, or would it be fine to have them in > the message? Both is good actually. -- Regards, Benjamin From brett at python.org Thu Mar 4 22:58:34 2010 From: brett at python.org (Brett Cannon) Date: Thu, 4 Mar 2010 13:58:34 -0800 Subject: [Python-Dev] Desired changes to Hg emails to python-checkins In-Reply-To: <1afaf6161003041356h6e9a971ex3b7c57c597c29226@mail.gmail.com> References: <1afaf6161003041344g20791bddpf13d66aa0c4d9438@mail.gmail.com> <1afaf6161003041356h6e9a971ex3b7c57c597c29226@mail.gmail.com> Message-ID: On Thu, Mar 4, 2010 at 13:56, Benjamin Peterson wrote: > 2010/3/4 Dirkjan Ochtman : > > On Thu, Mar 4, 2010 at 22:44, Benjamin Peterson > wrote: > >> +1 to bringing back affected files. It helps tell at a glance whether > >> a changeset is interesting or not. > > > > Do they need to be in the subject, or would it be fine to have them in > > the message? > > Both is good actually. > I prefer the subject so that I can easily skim them to see if someone edited a file I really care about, but if that is not possible then the body is acceptable, especially if it is the first thing in the body (that would let me at least see some of it in the initial snippet Gmail shows). -Brett > > > -- > Regards, > Benjamin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Fri Mar 5 01:43:09 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 05 Mar 2010 09:43:09 +0900 Subject: [Python-Dev] Desired changes to Hg emails to python-checkins In-Reply-To: References: <1afaf6161003041344g20791bddpf13d66aa0c4d9438@mail.gmail.com> <1afaf6161003041356h6e9a971ex3b7c57c597c29226@mail.gmail.com> Message-ID: <87vddbvbs2.fsf@uwakimon.sk.tsukuba.ac.jp> Brett Cannon writes: > I prefer the subject so that I can easily skim them to see if someone edited > a file I really care about, but if that is not possible then the body is > acceptable, especially if it is the first thing in the body (that would let > me at least see some of it in the initial snippet Gmail shows). I don't know if it's pedantically RFC-correct, but occasionally you'll see a Summary field in the header. I think using the first line of the log in the Subject field and the file list as the Summary field would be a good way to do it, *if* people's MUAs would display Summary. I suspect they don't (mine does, but mine does a lot of nifty things that cellphone MUAs don't :-( ). From martin at v.loewis.de Fri Mar 5 05:11:02 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 05 Mar 2010 05:11:02 +0100 Subject: [Python-Dev] doctest, unicode repr, and 2to3 Message-ID: <4B908456.2070505@v.loewis.de> Johan Harjano ran into an interesting problem when trying to run the Django test suite under Python 3.1. Django has doctests of the form >>> a6.headline u'Default headline' Even when converting the doctest with 2to3, the expected output is unmodified. However, in 3.x, the expected output will change (i.e. not produce an u"" prefix anymore). Now, it might be possible to reformulate the test case (e.g. use print() instead of relying on repr), however, this is undesirable as a) the test should continue to test in 2.x that the result object is a unicode string, and b) it makes the test less readable. I would like to find a solution where this gets automatically corrected, e.g. through 2to3, or through changes to doctest, or through changes of str.__repr__. Any proposal appreciated. Regards, Martin From barry at python.org Fri Mar 5 05:30:12 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 4 Mar 2010 23:30:12 -0500 Subject: [Python-Dev] doctest, unicode repr, and 2to3 In-Reply-To: <4B908456.2070505@v.loewis.de> References: <4B908456.2070505@v.loewis.de> Message-ID: <20100304233012.18ac0b41@freewill.wooz.org> On Mar 05, 2010, at 05:11 AM, Martin v. L?wis wrote: >Johan Harjano ran into an interesting problem when trying to run the >Django test suite under Python 3.1. > >Django has doctests of the form > >>>> a6.headline >u'Default headline' > >Even when converting the doctest with 2to3, the expected output is >unmodified. However, in 3.x, the expected output will change (i.e. not >produce an u"" prefix anymore). For this reason, I always recommend using print, even though... >Now, it might be possible to reformulate the test case (e.g. use print() >instead of relying on repr), however, this is undesirable as a) the test >should continue to test in 2.x that the result object is a unicode >string, and b) it makes the test less readable. If you really want to test that it's a unicode, shouldn't you actually test its type? (I'm not sure what would happen with that under 2to3.) Besides, the type of the string is very rarely important, so I think the u-prefix and quotes is mostly just noise. >I would like to find a solution where this gets automatically corrected, >e.g. through 2to3, or through changes to doctest, or through changes of >str.__repr__. I think Michael was also talking about changes to doctest that would automatically sort dictionaries and sets. Again, it's not hard to write doctests correctly, but it's surprisingly common to implicitly rely on sort order. I think the right place to change these is in doctest. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From glyph at twistedmatrix.com Fri Mar 5 05:33:16 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 4 Mar 2010 23:33:16 -0500 Subject: [Python-Dev] doctest, unicode repr, and 2to3 In-Reply-To: <20100304233012.18ac0b41@freewill.wooz.org> References: <4B908456.2070505@v.loewis.de> <20100304233012.18ac0b41@freewill.wooz.org> Message-ID: On Mar 4, 2010, at 11:30 PM, Barry Warsaw wrote: > If you really want to test that it's a unicode, shouldn't you actually test > its type? (I'm not sure what would happen with that under 2to3.) Presumably 2to3 will be smart enough to translate 'unicode' to 'str' and 'bytes' to... 'bytes'. Just don't use 'str' in 2.x and you should be okay :). -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at sweetapp.com Fri Mar 5 07:03:02 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Fri, 5 Mar 2010 17:03:02 +1100 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously Message-ID: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> Hi all, I recently submitted a daft PEP for a package designed to make it easier to execute Python functions asynchronously using threads and processes. It lets the user focus on their computational problem without having to build explicit thread/process pools and work queues. The package has been discussed on stdlib-sig but now I'd like this group's feedback. The PEP lives here: http://python.org/dev/peps/pep-3148/ Here are two examples to whet your appetites: """Determine if several numbers are prime.""" import futures import math PRIMES = [ 112272535095293, 112582705942171, 112272535095293, 115280095190773, 115797848077099, 1099726899285419] def is_prime(n): if n % 2 == 0: return False sqrt_n = int(math.floor(math.sqrt(n))) for i in range(3, sqrt_n + 1, 2): if n % i == 0: return False return True # Uses as many CPUs as your machine has. with futures.ProcessPoolExecutor() as executor: for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)): print('%d is prime: %s' % (number, is_prime)) """Print out the size of the home pages of various new sites (and Fox News).""" import futures import urllib.request URLS = ['http://www.foxnews.com/', 'http://www.cnn.com/', 'http://europe.wsj.com/', 'http://www.bbc.co.uk/', 'http://some-made-up-domain.com/'] def load_url(url, timeout): return urllib.request.urlopen(url, timeout=timeout).read() with futures.ThreadPoolExecutor(max_workers=5) as executor: # Create a future for each URL load. future_to_url = dict((executor.submit(load_url, url, 60), url) for url in URLS) # Iterate over the futures in the order that they complete. for future in futures.as_completed(future_to_url): url = future_to_url[future] if future.exception() is not None: print('%r generated an exception: %s' % (url, future.exception())) else: print('%r page is %d bytes' % (url, len(future.result()))) Cheers, Brian From ncoghlan at gmail.com Fri Mar 5 10:29:28 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 05 Mar 2010 19:29:28 +1000 Subject: [Python-Dev] doctest, unicode repr, and 2to3 In-Reply-To: References: <4B908456.2070505@v.loewis.de> <20100304233012.18ac0b41@freewill.wooz.org> Message-ID: <4B90CEF8.6000403@gmail.com> Glyph Lefkowitz wrote: > > On Mar 4, 2010, at 11:30 PM, Barry Warsaw wrote: > >> If you really want to test that it's a unicode, shouldn't you actually >> test >> its type? (I'm not sure what would happen with that under 2to3.) > > Presumably 2to3 will be smart enough to translate 'unicode' to 'str' and > 'bytes' to... 'bytes'. Just don't use 'str' in 2.x and you should be > okay :). In code (including doctest input lines) 2to3 can make those translations, but I believe the doctest output lines are a different story (since they're just arbitrary strings as far as the translator is concerned). I would expect 2to3 to be able to translate the following tests correctly because the expected output stays the same even though the commands change: >>> print a6.headline 'Default headline' >>> type(a6.headline) is type(u'') True But I don't see a ready way to support doctests where the expected *output* changes between versions. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From andrew at bemusement.org Fri Mar 5 12:53:06 2010 From: andrew at bemusement.org (Andrew Bennetts) Date: Fri, 5 Mar 2010 22:53:06 +1100 Subject: [Python-Dev] doctest, unicode repr, and 2to3 In-Reply-To: <4B908456.2070505@v.loewis.de> References: <4B908456.2070505@v.loewis.de> Message-ID: <20100305115306.GA9002@steerpike.home.puzzling.org> "Martin v. L?wis" wrote: [...] > Any proposal appreciated. I propose screaming ?help me, I have written a test suite using nothing but string matching assertions, what is wrong with me?!? -Andrew. From ironfroggy at gmail.com Fri Mar 5 13:45:46 2010 From: ironfroggy at gmail.com (Calvin Spealman) Date: Fri, 5 Mar 2010 07:45:46 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> Message-ID: <76fd5acf1003050445r1f8720f6w89599b8adb234666@mail.gmail.com> A young library solving an old problem in a way that conflicts with many of the other implementations available for years and with zero apparent users in the wild is not an appropriate candidate for a PEP. On Fri, Mar 5, 2010 at 1:03 AM, Brian Quinlan wrote: > Hi all, > > I recently submitted a daft PEP for a package designed to make it easier to > execute Python functions asynchronously using threads and processes. It lets > the user focus on their computational problem without having to build > explicit thread/process pools and work queues. > > The package has been discussed on stdlib-sig but now I'd like this group's > feedback. > > The PEP lives here: > http://python.org/dev/peps/pep-3148/ > > Here are two examples to whet your appetites: > > """Determine if several numbers are prime.""" > import futures > import math > > PRIMES = [ > ? ?112272535095293, > ? ?112582705942171, > ? ?112272535095293, > ? ?115280095190773, > ? ?115797848077099, > ? ?1099726899285419] > > def is_prime(n): > ? ?if n % 2 == 0: > ? ? ? ?return False > > ? ?sqrt_n = int(math.floor(math.sqrt(n))) > ? ?for i in range(3, sqrt_n + 1, 2): > ? ? ? ?if n % i == 0: > ? ? ? ? ? ?return False > ? ?return True > > # Uses as many CPUs as your machine has. > with futures.ProcessPoolExecutor() as executor: > ? ?for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)): > ? ? ? ?print('%d is prime: %s' % (number, is_prime)) > > > """Print out the size of the home pages of various new sites (and Fox > News).""" > import futures > import urllib.request > > URLS = ['http://www.foxnews.com/', > ? ? ? ?'http://www.cnn.com/', > ? ? ? ?'http://europe.wsj.com/', > ? ? ? ?'http://www.bbc.co.uk/', > ? ? ? ?'http://some-made-up-domain.com/'] > > def load_url(url, timeout): > ? ?return urllib.request.urlopen(url, timeout=timeout).read() > > with futures.ThreadPoolExecutor(max_workers=5) as executor: > ? ?# Create a future for each URL load. > ? ?future_to_url = dict((executor.submit(load_url, url, 60), url) > ? ? ? ? ? ? ? ? ? ? ? ? for url in URLS) > > ? ?# Iterate over the futures in the order that they complete. > ? ?for future in futures.as_completed(future_to_url): > ? ? ? ?url = future_to_url[future] > ? ? ? ?if future.exception() is not None: > ? ? ? ? ? ?print('%r generated an exception: %s' % (url, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? future.exception())) > ? ? ? ?else: > ? ? ? ? ? ?print('%r page is %d bytes' % (url, len(future.result()))) > > Cheers, > Brian > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com > -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy From jnoller at gmail.com Fri Mar 5 15:50:28 2010 From: jnoller at gmail.com (Jesse Noller) Date: Fri, 5 Mar 2010 09:50:28 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <76fd5acf1003050445r1f8720f6w89599b8adb234666@mail.gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <76fd5acf1003050445r1f8720f6w89599b8adb234666@mail.gmail.com> Message-ID: <4222a8491003050650j1f6be276tab614016564fe10d@mail.gmail.com> On Fri, Mar 5, 2010 at 7:45 AM, Calvin Spealman wrote: > A young library solving an old problem in a way that conflicts with > many of the other implementations available for years and with zero > apparent users in the wild is not an appropriate candidate for a PEP. > Baloney. A young library providing some syntactic sugar which uses primitives in the standard library to implement a common pattern is fine for a PEP. We've hashed this out pretty heavily on the stdlib-sig list prior to bringing it here. By the same argument, we should shunt all of the recent unittest changes and improvements into space, since golly, other people did it, why should we. This is something relatively simple, which I would gladly add in an instant to the multiprocessing package - but Brian's one-upped me in that regard and is providing something which works with both threads and processes handily. Take a look at multiprocessing.Pool for example - all that is some sugar on top of the primitives, but it's good sugar, and is used by a fair number of people. Let me also state - "my" vision of where futures would live would be in a concurrent package - for example: from concurrent import futures The reason *why* is that I would like to also move the abstractions I have in multiprocessing *out* of that module, make them work with both threads and processes (if it makes sense) and reduce the multiprocessing module to the base primitive Process object. A concurrent package which implements common patterns built on top of the primitives we support is an objectively Good Thing. For example, how many of us have sat down and implemented a thread pool on top of threading, I would hazard to say that most of us who use threading have done this, and probably more than once. It stands to reason that this is a common enough pattern to include in the standard library. In any case; consider me a strong +1 to adding it. jesse From ironfroggy at gmail.com Fri Mar 5 16:50:38 2010 From: ironfroggy at gmail.com (Calvin Spealman) Date: Fri, 5 Mar 2010 10:50:38 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4222a8491003050650j1f6be276tab614016564fe10d@mail.gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <76fd5acf1003050445r1f8720f6w89599b8adb234666@mail.gmail.com> <4222a8491003050650j1f6be276tab614016564fe10d@mail.gmail.com> Message-ID: <76fd5acf1003050750l5019e7deud2b5891647c5d7ac@mail.gmail.com> I revert my objections. I still would like to see this in use "in the wild" and I might even use it thusly, myself. On Fri, Mar 5, 2010 at 9:50 AM, Jesse Noller wrote: > On Fri, Mar 5, 2010 at 7:45 AM, Calvin Spealman wrote: >> A young library solving an old problem in a way that conflicts with >> many of the other implementations available for years and with zero >> apparent users in the wild is not an appropriate candidate for a PEP. >> > > Baloney. A young library providing some syntactic sugar which uses > primitives in the standard library to implement a common pattern is > fine for a PEP. We've hashed this out pretty heavily on the stdlib-sig > list prior to bringing it here. By the same argument, we should shunt > all of the recent unittest changes and improvements into space, since > golly, other people did it, why should we. > > This is something relatively simple, which I would gladly add in an > instant to the multiprocessing package - but Brian's one-upped me in > that regard and is providing something which works with both threads > and processes handily. Take a look at multiprocessing.Pool for example > - all that is some sugar on top of the primitives, but it's good > sugar, and is used by a fair number of people. > > Let me also state - "my" vision of where futures would live would be > in a concurrent package - for example: > > from concurrent import futures > > The reason *why* is that I would like to also move the abstractions I > have in multiprocessing *out* of that module, make them work with both > threads and processes (if it makes sense) and reduce the > multiprocessing module to the base primitive Process object. A > concurrent package which implements common patterns built on top of > the primitives we support is an objectively Good Thing. > > For example, how many of us have sat down and implemented a thread > pool on top of threading, I would hazard to say that most of us who > use threading have done this, and probably more than once. It stands > to reason that this is a common enough pattern to include in the > standard library. > > In any case; consider me a strong +1 to adding it. > > jesse > -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy From ncoghlan at gmail.com Fri Mar 5 16:53:27 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 06 Mar 2010 01:53:27 +1000 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4222a8491003050650j1f6be276tab614016564fe10d@mail.gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <76fd5acf1003050445r1f8720f6w89599b8adb234666@mail.gmail.com> <4222a8491003050650j1f6be276tab614016564fe10d@mail.gmail.com> Message-ID: <4B9128F7.3020301@gmail.com> Jesse Noller wrote: > The reason *why* is that I would like to also move the abstractions I > have in multiprocessing *out* of that module, make them work with both > threads and processes (if it makes sense) and reduce the > multiprocessing module to the base primitive Process object. A > concurrent package which implements common patterns built on top of > the primitives we support is an objectively Good Thing. Yes, I've often thought we should have a pool model that covers threads as well as processes. The reason I think "futures" works as a PEP and potential standard library addition is that it is small enough to be readily reviewed and maintained, and serves as a useful building block for more complex usage. For a developer to get anything similar from a third party library is almost certainly going to require buying into a much heavier framework. A simple futures module provides a way to farm out worker tasks in a standard fashion without having to build as much of your own infrastructure every time. I've read the various PEP checkins as they went by on the checkins list - it gets a +0 from me (the only reason it isn't a +1 is because I personally tend to write with a Thread+Queue style. However, I could easily become a futures convert if they were readily available in the standard library) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Fri Mar 5 16:56:09 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 5 Mar 2010 10:56:09 -0500 Subject: [Python-Dev] doctest, unicode repr, and 2to3 References: <4B908456.2070505@v.loewis.de> <20100304233012.18ac0b41@freewill.wooz.org> Message-ID: <20100305105609.16766c2c@msiwind> Le Thu, 4 Mar 2010 23:30:12 -0500, Barry Warsaw a ?crit : > > If you really want to test that it's a unicode, shouldn't you > actually test its type? (I'm not sure what would happen with that > under 2to3.) Besides, the type of the string is very rarely > important, so I think the u-prefix and quotes is mostly just noise. String type is actually very important, if you don't want your application/library to fail in the face of non-ASCII data. That's why we did all this thing in py3k, after all :) Regards Antoine. From fuzzyman at voidspace.org.uk Fri Mar 5 17:11:59 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 05 Mar 2010 16:11:59 +0000 Subject: [Python-Dev] doctest, unicode repr, and 2to3 In-Reply-To: <20100305105609.16766c2c@msiwind> References: <4B908456.2070505@v.loewis.de> <20100304233012.18ac0b41@freewill.wooz.org> <20100305105609.16766c2c@msiwind> Message-ID: <4B912D4F.7020700@voidspace.org.uk> On 05/03/2010 15:56, Antoine Pitrou wrote: > Le Thu, 4 Mar 2010 23:30:12 -0500, > Barry Warsaw a ?crit : > >> If you really want to test that it's a unicode, shouldn't you >> actually test its type? (I'm not sure what would happen with that >> under 2to3.) Besides, the type of the string is very rarely >> important, so I think the u-prefix and quotes is mostly just noise. >> > String type is actually very important, if you don't want your > application/library to fail in the face of non-ASCII data. > > That's why we did all this thing in py3k, after all :) > Well, I'd like to see a 'unicode agnostic' mode for different reasons - although given that it all changes in Python 3 and it is almost certainly too late to get changes to doctest before the 2.7 release the point is probably moot. IronPython has very different unicode behaviour to Python 2. IronPython is like Python 3 in this regard, all strings are unicode (str is unicode). This means that it rarely puts the u prefix on strings. So doctests that use string repr also fail on IronPython - the django doctest suite is a culprit in this regard too. Dictionary iteration order is also different in IronPython - so dict reprs also fail. For dict reprs it would be nice if doctest compared in an order agnostic way. (Internally reconstruct the dict as a dict of strings from the repr and then compare using dict equality.) I know it is easily *possible* to construct doctests that are robust against these differences, but the fact that the "obvious-thing" often fails in subtle ways on other platforms are *one* of the reasons I don't consider doctest to be a suitable tool for unit testing [1]. :-) Michael [1] It isn't the main reason and I realise that is entirely orthoganal anway... but doctest makes every line an assertion, and you have to jump through hoops if you don't want that behaviour. All besides the point. Doctest simply *rocks* for testing documentation examples - especially in conjunction with Sphinx. > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From barry at python.org Fri Mar 5 17:13:32 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 5 Mar 2010 11:13:32 -0500 Subject: [Python-Dev] doctest, unicode repr, and 2to3 In-Reply-To: <20100305105609.16766c2c@msiwind> References: <4B908456.2070505@v.loewis.de> <20100304233012.18ac0b41@freewill.wooz.org> <20100305105609.16766c2c@msiwind> Message-ID: <20100305111332.18650d1e@heresy.wooz.org> On Mar 05, 2010, at 10:56 AM, Antoine Pitrou wrote: >String type is actually very important, if you don't want your >application/library to fail in the face of non-ASCII data. > >That's why we did all this thing in py3k, after all :) That's not actually what I mean. I meant that in doctests, you probably don't need to be confronted by the string type every single time. It's just not that interesting for documentation, most of the time. I have no problem adding (unit)tests that ensure your outputs and values are of the expected type though! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From daniel at stutzbachenterprises.com Fri Mar 5 17:21:03 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Fri, 5 Mar 2010 10:21:03 -0600 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> Message-ID: On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan wrote: > import futures > +1 on the idea, -1 on the name. It's too similar to "from __future__ import ...". Also, the PEP should probably link to the discussions on stdlib-sig? -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnoller at gmail.com Fri Mar 5 17:35:54 2010 From: jnoller at gmail.com (Jesse Noller) Date: Fri, 5 Mar 2010 11:35:54 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> Message-ID: <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach wrote: > On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan wrote: >> >> import futures > > +1 on the idea, -1 on the name.? It's too similar to "from __future__ import > ...". Futures is a common term for this, and implemented named this in other languages. I don't think we should be adopting things that are common, and found elsewhere and then renaming them. From skip at pobox.com Fri Mar 5 17:56:52 2010 From: skip at pobox.com (skip at pobox.com) Date: Fri, 5 Mar 2010 10:56:52 -0600 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> Message-ID: <19345.14292.321112.787896@montanaro.dyndns.org> >>> import futures >> >> +1 on the idea, -1 on the name.? It's too similar to "from __future__ import >> ...". Jesse> Futures is a common term for this, and implemented named this in Jesse> other languages. I don't think we should be adopting things that Jesse> are common, and found elsewhere and then renaming them. Perhaps, but is it a common term for Python programmers (or the target population for Python)? I've never heard of it. "futures" to me are futures contracts in a trading environment (that's the industry I work in). No matter how well known the term is in the environment where it's used today you have to be sensitive to other meanings of the term. Skip From jnoller at gmail.com Fri Mar 5 18:03:58 2010 From: jnoller at gmail.com (Jesse Noller) Date: Fri, 5 Mar 2010 12:03:58 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <19345.14292.321112.787896@montanaro.dyndns.org> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <19345.14292.321112.787896@montanaro.dyndns.org> Message-ID: <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com> On Fri, Mar 5, 2010 at 11:56 AM, wrote: > ? ?>>> import futures > ? ?>> > ? ?>> +1 on the idea, -1 on the name.? It's too similar to "from __future__ import > ? ?>> ...". > > ? ?Jesse> Futures is a common term for this, and implemented named this in > ? ?Jesse> other languages. I don't think we should be adopting things that > ? ?Jesse> are common, and found elsewhere and then renaming them. > > Perhaps, but is it a common term for Python programmers (or the target > population for Python)? ?I've never heard of it. ?"futures" to me are > futures contracts in a trading environment (that's the industry I work in). > No matter how well known the term is in the environment where it's used > today you have to be sensitive to other meanings of the term. > It's a common programming term. I don't think we should make a new name - I mean, how many different names for green threads / coroutines or "flavors" of those concepts are out there because someone painted the bike shed a different color? I mean, there's prior art here: http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html jesse From curt at hagenlocher.org Fri Mar 5 18:06:44 2010 From: curt at hagenlocher.org (Curt Hagenlocher) Date: Fri, 5 Mar 2010 09:06:44 -0800 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> Message-ID: On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller wrote: > > On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach > wrote: > > On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan wrote: > >> > >> import futures > > > > +1 on the idea, -1 on the name.? It's too similar to "from __future__ import > > ...". > > Futures is a common term for this, and implemented named this in other > languages. I don't think we should be adopting things that are common, > and found elsewhere and then renaming them. Another common term for this is a "promise". -- Curt Hagenlocher curt at hagenlocher.org From daniel at stutzbachenterprises.com Fri Mar 5 18:28:42 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Fri, 5 Mar 2010 11:28:42 -0600 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <19345.14292.321112.787896@montanaro.dyndns.org> <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com> Message-ID: On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller wrote: > http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html > According to that link, Java has a module named "Concurrent" with an interface named "Future". You're proposing a module named "Futures" with a class named "Future". Why not name your module "concurrent"? That would eliminate the confusion with "from __future__". I don't see a problem with keeping the class name. Plus, a "concurrent" module might be useful for things other than Futures, in the future. ;-) Just my 0.02 cents, -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Fri Mar 5 18:38:10 2010 From: skip at pobox.com (skip at pobox.com) Date: Fri, 5 Mar 2010 11:38:10 -0600 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <19345.14292.321112.787896@montanaro.dyndns.org> <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com> Message-ID: <19345.16770.613942.887721@montanaro.dyndns.org> Jesse> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html Without reading that I can assure you that not everybody has drunk the Java Kool-Aid. Just because Sun thought it was a fine term doesn't mean everyone else will. I've been a professional programmer for about 30 years, have never touched Java, and the only times I've seen its effect in Python I have been unimpressed (logging and unittest). Google for "future". The Java doc is aways down on the front page. Now Google for "futures". It's nowhere to be found. As far as I can tell, all but one hit on the first page are for my definition of "futures". At the bottom of the page the "searches related to _futures_" are all related to trading futures or other derivatives except one. That appears to be related to the LPGA futures tour. Java and its definition of the term is nowhere to be found. I suppose you can argue that it should be called "future". I still contend that the name is *not* intuitive, no matter how well known it happens to be within the Java world. Don't name it "futures" though. That has nothing to do with common definitions of the term. Skip From jnoller at gmail.com Fri Mar 5 18:55:04 2010 From: jnoller at gmail.com (Jesse Noller) Date: Fri, 5 Mar 2010 12:55:04 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <19345.14292.321112.787896@montanaro.dyndns.org> <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com> Message-ID: <4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com> On Fri, Mar 5, 2010 at 12:28 PM, Daniel Stutzbach wrote: > On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller wrote: >> >> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html > > ?According to that link, Java has a module named "Concurrent" with an > interface named "Future".? You're proposing a module named "Futures" with a > class named "Future". > > Why not name your module "concurrent"?? That would eliminate the confusion > with "from __future__".? I don't see a problem with keeping the class name. > > Plus, a "concurrent" module might be useful for things other than Futures, > in the future. ;-) > Brian's module is named futures; I am +1'ing his proposal, and also suggesting we put it under concurrent/ package name. This means you would do the following: from concurrent import futures and in the future: from concurrent import pool And so on. From exarkun at twistedmatrix.com Fri Mar 5 19:30:35 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 05 Mar 2010 18:30:35 -0000 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> Message-ID: <20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain> On 05:06 pm, curt at hagenlocher.org wrote: >On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller wrote: >> >>On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach >> wrote: >> > On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan >>wrote: >> >> >> >> import futures >> > >> > +1 on the idea, -1 on the name.? It's too similar to "from >>__future__ import >> > ...". >> >>Futures is a common term for this, and implemented named this in other >>languages. I don't think we should be adopting things that are common, >>and found elsewhere and then renaming them. > >Another common term for this is a "promise". Promises aren't exactly the same. This would be a particularly bad name to apply here. Jean-Paul From guido at python.org Fri Mar 5 19:31:03 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 5 Mar 2010 10:31:03 -0800 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <19345.14292.321112.787896@montanaro.dyndns.org> <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com> <4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com> Message-ID: On Fri, Mar 5, 2010 at 9:55 AM, Jesse Noller wrote: > On Fri, Mar 5, 2010 at 12:28 PM, Daniel Stutzbach > wrote: >> On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller wrote: >>> >>> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html >> >> ?According to that link, Java has a module named "Concurrent" with an >> interface named "Future".? You're proposing a module named "Futures" with a >> class named "Future". >> >> Why not name your module "concurrent"?? That would eliminate the confusion >> with "from __future__".? I don't see a problem with keeping the class name. >> >> Plus, a "concurrent" module might be useful for things other than Futures, >> in the future. ;-) >> > > Brian's module is named futures; I am +1'ing his proposal, and also > suggesting we put it under concurrent/ package name. This means you > would do the following: > > from concurrent import futures > > and in the future: > from concurrent import pool > > And so on. "Future" is a pretty standard CS term for this concept (as noted "promise" is another), and it wasn't invented by Java. I am not worried at all about confusion with __future__ (which is after all surrounded by dunders and only appears in one specific context). FWIW, there are two different ways of designing the API for such a concept. In one, the "future" or "promise" object masquerades as an instance of the eventual result, like a proxy, and accessing any attribute would block until the result is available. I don't like this design much; the masquerading is never perfect, and attribute accesses become risks of exceptions related to the promised work. In the other design, which this PEP proposes, there is an explicit call to get the value, which blocks as needed. I found a wikipedia article (http://en.wikipedia.org/wiki/Futures_and_promises#Implicit_vs_explicit) that calls the former "implicit" and the latter "explicit". According to this article, the term "promise" was coined slightly older (by a year or so). I have no strong preference for either term; I suspect that if this were the only objection to the PEP Brian would be happy to rename futures to promises. -- --Guido van Rossum (python.org/~guido) From janssen at parc.com Fri Mar 5 19:45:50 2010 From: janssen at parc.com (Bill Janssen) Date: Fri, 5 Mar 2010 10:45:50 PST Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <19345.16770.613942.887721@montanaro.dyndns.org> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <19345.14292.321112.787896@montanaro.dyndns.org> <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com> <19345.16770.613942.887721@montanaro.dyndns.org> Message-ID: <26632.1267814750@parc.com> skip at pobox.com wrote: > > Jesse> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html > > Without reading that I can assure you that not everybody has drunk the Java > Kool-Aid. Just because Sun thought it was a fine term doesn't mean everyone > else will. I've been a professional programmer for about 30 years, have > never touched Java, and the only times I've seen its effect in Python I have > been unimpressed (logging and unittest). > > Google for "future". The Java doc is aways down on the front page. Now > Google for "futures". It's nowhere to be found. As far as I can tell, all > but one hit on the first page are for my definition of "futures". At the > bottom of the page the "searches related to _futures_" are all related to > trading futures or other derivatives except one. That appears to be related > to the LPGA futures tour. Java and its definition of the term is nowhere to > be found. This term, the "future", has been in use at least since 1985, even if the Web hasn't, and even if Google can't seem to separate this use from drivel about stock trading. See Halstead's 1985 TOPLAS paper on MultiLisp. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.97.1841 Bill From janssen at parc.com Fri Mar 5 19:49:27 2010 From: janssen at parc.com (Bill Janssen) Date: Fri, 5 Mar 2010 10:49:27 PST Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <19345.16770.613942.887721@montanaro.dyndns.org> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <19345.14292.321112.787896@montanaro.dyndns.org> <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com> <19345.16770.613942.887721@montanaro.dyndns.org> Message-ID: <26666.1267814967@parc.com> http://home.pipeline.com/~hbaker1/Futures.html (1977) From ndbecker2 at gmail.com Fri Mar 5 19:51:15 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 05 Mar 2010 13:51:15 -0500 Subject: [Python-Dev] argparse ugliness Message-ID: I generally enjoy argparse, but one thing I find rather ugly and unpythonic. parser.add_argument ('--plot', action='store_true') Specifying the argument 'action' as a string is IMO ugly. From guido at python.org Fri Mar 5 20:10:33 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 5 Mar 2010 11:10:33 -0800 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain> Message-ID: On Fri, Mar 5, 2010 at 10:30 AM, wrote: > On 05:06 pm, curt at hagenlocher.org wrote: >> >> On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller wrote: >>> >>> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach >>> wrote: >>> > On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan >>> > wrote: >>> >> >>> >> import futures >>> > >>> > +1 on the idea, -1 on the name.? It's too similar to "from __future__ >>> > import >>> > ...". >>> >>> Futures is a common term for this, and implemented named this in other >>> languages. I don't think we should be adopting things that are common, >>> and found elsewhere and then renaming them. >> >> Another common term for this is a "promise". > > Promises aren't exactly the same. ?This would be a particularly bad name to > apply here. Please explain. Even the Wikipedia article (http://en.wikipedia.org/wiki/Futures_and_promises), despite promising to explain the difference, didn't explain it. -- --Guido van Rossum (python.org/~guido) From brian.curtin at gmail.com Fri Mar 5 20:15:58 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Fri, 5 Mar 2010 13:15:58 -0600 Subject: [Python-Dev] argparse ugliness In-Reply-To: References: Message-ID: On Fri, Mar 5, 2010 at 12:51, Neal Becker wrote: > I generally enjoy argparse, but one thing I find rather > ugly and unpythonic. > > parser.add_argument ('--plot', action='store_true') > > Specifying the argument 'action' as a string is IMO ugly. > What else would you propose? FWIW, this is the same in optparse. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Mar 5 20:37:13 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 05 Mar 2010 14:37:13 -0500 Subject: [Python-Dev] doctest, unicode repr, and 2to3 In-Reply-To: <4B908456.2070505@v.loewis.de> References: <4B908456.2070505@v.loewis.de> Message-ID: On 3/4/2010 11:11 PM, "Martin v. L?wis" wrote: > Johan Harjano ran into an interesting problem when trying to run the > Django test suite under Python 3.1. > > Django has doctests of the form > >>>> a6.headline > u'Default headline' > > Even when converting the doctest with 2to3, the expected output is > unmodified. However, in 3.x, the expected output will change (i.e. not > produce an u"" prefix anymore). > > Now, it might be possible to reformulate the test case (e.g. use print() > instead of relying on repr), however, this is undesirable as a) the test > should continue to test in 2.x that the result object is a unicode > string, and b) it makes the test less readable. > > I would like to find a solution where this gets automatically corrected, > e.g. through 2to3, or through changes to doctest, or through changes of > str.__repr__. > > Any proposal appreciated. What is the easiest thing that works? If 2to3 can fix string literals and fix code within doc strings, would it be difficult to fix expected strings within doc strings? On the otherhand, as Foord pointed out, the 'u' prefix is something of a CPythonism not required by the language def, so an 'ignore leading u on expected output' flag would be useful. But 'correcting' >>> a,b u'ah', u'hah' is too much to expect. Doctest is both useful and fragile because it *almost* imitates human testing methods. If tests like the above were bunched, and my primary goal were to make Django tests work, and work now, without changing the style, I would be tempted to use something like the following: def u(s): #import as needed from a testutil module if type(s) is unicode: # str in 3.x print(s) else print('') #or any 'nevermatch' string >>> u(a6.headline) 'Default headline' which both retains the type test and readability. This would be a project specific solution, but then, the other comments suggest that wanting to combine a type and value test this way seems to be someone project specific. Terry Jan Reedy From martin at v.loewis.de Fri Mar 5 20:49:54 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 05 Mar 2010 20:49:54 +0100 Subject: [Python-Dev] doctest, unicode repr, and 2to3 In-Reply-To: <20100305115306.GA9002@steerpike.home.puzzling.org> References: <4B908456.2070505@v.loewis.de> <20100305115306.GA9002@steerpike.home.puzzling.org> Message-ID: <4B916062.90908@v.loewis.de> Andrew Bennetts wrote: > "Martin v. L?wis" wrote: > [...] >> Any proposal appreciated. > > I propose screaming ?help me, I have written a test suite using nothing > but string matching assertions, what is wrong with me?!? Thanks a lot. Regards, Martin From exarkun at twistedmatrix.com Fri Mar 5 21:18:22 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 05 Mar 2010 20:18:22 -0000 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain> Message-ID: <20100305201822.2792.1965005768.divmod.xquotient.976@localhost.localdomain> On 07:10 pm, guido at python.org wrote: >On Fri, Mar 5, 2010 at 10:30 AM, wrote: >>On 05:06 pm, curt at hagenlocher.org wrote: >>> >>>On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller >>>wrote: >>>> >>>>On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach >>>> wrote: >>>> > On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan >>>> >>>> > wrote: >>>> >> >>>> >> import futures >>>> > >>>> > +1 on the idea, -1 on the name.? It's too similar to "from >>>>__future__ >>>> > import >>>> > ...". >>>> >>>>Futures is a common term for this, and implemented named this in >>>>other >>>>languages. I don't think we should be adopting things that are >>>>common, >>>>and found elsewhere and then renaming them. >>> >>>Another common term for this is a "promise". >> >>Promises aren't exactly the same. ?This would be a particularly bad >>name to >>apply here. > >Please explain. Even the Wikipedia article >(http://en.wikipedia.org/wiki/Futures_and_promises), despite promising >to explain the difference, didn't explain it. The "explicit" futures on the wikipedia page seems to cover what is commonly referred to as a future. For example, Java's futures look like this. The "implicit" futures are what is generally called a promise. For example, E's promises look like this. Though the difference is mainly one of API, it turns out to make a significant difference in what you can accomplish. Promises are much more amenable to the pipelining optimization, for example. They're also much harder to implement in Python without core language changes. Jean-Paul From brett at python.org Fri Mar 5 21:31:29 2010 From: brett at python.org (Brett Cannon) Date: Fri, 5 Mar 2010 12:31:29 -0800 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <19345.14292.321112.787896@montanaro.dyndns.org> <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com> <4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com> Message-ID: On Fri, Mar 5, 2010 at 09:55, Jesse Noller wrote: > On Fri, Mar 5, 2010 at 12:28 PM, Daniel Stutzbach > wrote: > > On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller wrote: > >> > >> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html > > > > According to that link, Java has a module named "Concurrent" with an > > interface named "Future". You're proposing a module named "Futures" with > a > > class named "Future". > > > > Why not name your module "concurrent"? That would eliminate the > confusion > > with "from __future__". I don't see a problem with keeping the class > name. > > > > Plus, a "concurrent" module might be useful for things other than > Futures, > > in the future. ;-) > > > > Brian's module is named futures; I am +1'ing his proposal, and also > suggesting we put it under concurrent/ package name. This means you > would do the following: > > from concurrent import futures > > and in the future: > from concurrent import pool > > And so on. So I don't quite get what you are after here. Are you wanting to eventually have a generic pool class that you can simply import and use that is always set to the best option for the platform? And as for moving stuff from multiprocessing into the concurrent namespace, are you thinking like concurrent.multiprocessing? I guess I am just trying to figure out what the abstraction is you are after in the package namespace. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Fri Mar 5 21:33:25 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 05 Mar 2010 15:33:25 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jesse Noller wrote: > On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach > wrote: >> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan wrote: >>> import futures >> +1 on the idea, -1 on the name. It's too similar to "from __future__ import >> ...". > > Futures is a common term for this, and implemented named this in other > languages. I don't think we should be adopting things that are common, > and found elsewhere and then renaming them. - -1 to the name from me as well: it isn't "scoped" properly to make it clear what the module is about. If they were inside a pacakge named 'concurrency' or some such (as hinted by Jesse Noller, I think), the clash would go away. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkuRapAACgkQ+gerLs4ltQ7dBwCfRmMuq6X9VE8usYgSScXEA1D0 1PsAoI8MR5hjPjq9C7MFPTZhcO/T+NM4 =7wpK -----END PGP SIGNATURE----- From brett at python.org Fri Mar 5 21:38:43 2010 From: brett at python.org (Brett Cannon) Date: Fri, 5 Mar 2010 12:38:43 -0800 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> Message-ID: The PEP says that futures.wait() should only use keyword arguments past its first positional argument, but the PEP has the function signature as ``wait(fs, timeout=None, return_when=ALL_COMPLETED)``. Should it be ``wait(fs, *, timeout=None, return_when=ALL_COMPLETED)``? On Thu, Mar 4, 2010 at 22:03, Brian Quinlan wrote: > Hi all, > > I recently submitted a daft PEP for a package designed to make it easier to > execute Python functions asynchronously using threads and processes. It lets > the user focus on their computational problem without having to build > explicit thread/process pools and work queues. > > The package has been discussed on stdlib-sig but now I'd like this group's > feedback. > > The PEP lives here: > http://python.org/dev/peps/pep-3148/ > > Here are two examples to whet your appetites: > > """Determine if several numbers are prime.""" > import futures > import math > > PRIMES = [ > 112272535095293, > 112582705942171, > 112272535095293, > 115280095190773, > 115797848077099, > 1099726899285419] > > def is_prime(n): > if n % 2 == 0: > return False > > sqrt_n = int(math.floor(math.sqrt(n))) > for i in range(3, sqrt_n + 1, 2): > if n % i == 0: > return False > return True > > # Uses as many CPUs as your machine has. > with futures.ProcessPoolExecutor() as executor: > for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)): > print('%d is prime: %s' % (number, is_prime)) > > > """Print out the size of the home pages of various new sites (and Fox > News).""" > import futures > import urllib.request > > URLS = ['http://www.foxnews.com/', > 'http://www.cnn.com/', > 'http://europe.wsj.com/', > 'http://www.bbc.co.uk/', > 'http://some-made-up-domain.com/'] > > def load_url(url, timeout): > return urllib.request.urlopen(url, timeout=timeout).read() > > with futures.ThreadPoolExecutor(max_workers=5) as executor: > # Create a future for each URL load. > future_to_url = dict((executor.submit(load_url, url, 60), url) > for url in URLS) > > # Iterate over the futures in the order that they complete. > for future in futures.as_completed(future_to_url): > url = future_to_url[future] > if future.exception() is not None: > print('%r generated an exception: %s' % (url, > future.exception())) > else: > print('%r page is %d bytes' % (url, len(future.result()))) > > Cheers, > Brian > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnoller at gmail.com Fri Mar 5 21:51:59 2010 From: jnoller at gmail.com (Jesse Noller) Date: Fri, 5 Mar 2010 15:51:59 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <19345.14292.321112.787896@montanaro.dyndns.org> <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com> <4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com> Message-ID: <4222a8491003051251o4d467f37m81ad74aa1fff4a5c@mail.gmail.com> On Fri, Mar 5, 2010 at 3:31 PM, Brett Cannon wrote: > > So I don't quite get what you are after here. Are you wanting to eventually > have a generic pool class that you can simply import and use that is always > set to the best option for the platform? > And as for moving stuff from multiprocessing into the concurrent namespace, > are you thinking like concurrent.multiprocessing? I guess I am just trying > to figure out what the abstraction is you are after in the package > namespace. > -Brett My goal would be to put futures into a "concurrent" package - as it is an abstraction that allows for threads, and processes to be used. By default; I don't think we'd make a guess based on the platform, but rather pick a sane default (such as threads). After that; I would begin to remove chunks of multiprocessing (such as pool) and adapt it to the same type of "pick threads or processes" that futures uses. For example: from concurrent import Pool x = Pool(4, worker_primitive=Thread()) And so on. The end-goal would be to make concurrent.* a package containing common abstractions/patterns which can use threads or processes interchangeably. jesse From guido at python.org Fri Mar 5 21:55:46 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 5 Mar 2010 12:55:46 -0800 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100305201822.2792.1965005768.divmod.xquotient.976@localhost.localdomain> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain> <20100305201822.2792.1965005768.divmod.xquotient.976@localhost.localdomain> Message-ID: On Fri, Mar 5, 2010 at 12:18 PM, wrote: > The "explicit" futures on the wikipedia page seems to cover what is commonly > referred to as a future. ?For example, Java's futures look like this. > > The "implicit" futures are what is generally called a promise. ?For example, > E's promises look like this. Fair enough, though the article confuses the matter by using the words more or less interchangeably. > Though the difference is mainly one of API, it turns out to make a > significant difference in what you can accomplish. ?Promises are much more > amenable to the pipelining optimization, for example. ?They're also much > harder to implement in Python without core language changes. That's why implicit futures (by any name) aren't on the table. -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Mar 5 22:22:36 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 5 Mar 2010 13:22:36 -0800 Subject: [Python-Dev] doctest, unicode repr, and 2to3 In-Reply-To: <4B908456.2070505@v.loewis.de> References: <4B908456.2070505@v.loewis.de> Message-ID: On Thu, Mar 4, 2010 at 8:11 PM, "Martin v. L?wis" wrote: > Johan Harjano ran into an interesting problem when trying to run the > Django test suite under Python 3.1. > > Django has doctests of the form > >>>> a6.headline > u'Default headline' > > Even when converting the doctest with 2to3, the expected output is > unmodified. However, in 3.x, the expected output will change (i.e. not > produce an u"" prefix anymore). > > Now, it might be possible to reformulate the test case (e.g. use print() > instead of relying on repr), however, this is undesirable as a) the test > should continue to test in 2.x that the result object is a unicode > string, and b) it makes the test less readable. > > I would like to find a solution where this gets automatically corrected, > e.g. through 2to3, or through changes to doctest, or through changes of > str.__repr__. > > Any proposal appreciated. How about a heuristic rule (which you have to explicitly select) that changes u'XXX' into 'XXX' inside triply-quoted strings given certain contexts, e.g. only at the start of the line, only if there is a nearby preceding line starting with '>>>'? Exactly what context is of the right strength will have to be determined experimentally; if there are a lot of tests outputting things like [u'...'] or {u'...': u'...'} the context may have to be made more liberal. Possibly \bu('.*'|".*") would do it? The issue shows (yet again) a general problem with doctests being overspecified -- the test shouldn't care that the output starts with 'u', it should only care that the value is unicode, but there's no easy way to express this in doctests. But since these doctests exist I suggest that the practical way forward is to stick with them rather than trying to reformulate all the tests. -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Fri Mar 5 22:39:30 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 05 Mar 2010 22:39:30 +0100 Subject: [Python-Dev] doctest, unicode repr, and 2to3 In-Reply-To: References: <4B908456.2070505@v.loewis.de> Message-ID: <4B917A12.3030003@v.loewis.de> > The issue shows (yet again) a general problem with doctests being > overspecified -- the test shouldn't care that the output starts with > 'u', it should only care that the value is unicode, but there's no > easy way to express this in doctests. But since these doctests exist I > suggest that the practical way forward is to stick with them rather > than trying to reformulate all the tests. Ok, I think I would prefer that. One approach I considered was to override sys.displayhook, so that I can determine what parts of the output are "single" repr results. Of course, this wouldn't catch repr strings that were ultimately print()ed (not sure whether that restriction would affect Django). The other (more aggressive) approach is the heuristics you propose, which may end up with false negatives. doctest already has a wildcard matching flag, so it would grow another one. Regards, Martin From jnoller at gmail.com Fri Mar 5 22:42:51 2010 From: jnoller at gmail.com (Jesse Noller) Date: Fri, 5 Mar 2010 16:42:51 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> Message-ID: <4222a8491003051342y762fe18bi253cd1f81d8d36f6@mail.gmail.com> On Fri, Mar 5, 2010 at 3:33 PM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Jesse Noller wrote: >> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach >> wrote: >>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan wrote: >>>> import futures >>> +1 on the idea, -1 on the name. ?It's too similar to "from __future__ import >>> ...". >> >> Futures is a common term for this, and implemented named this in other >> languages. I don't think we should be adopting things that are common, >> and found elsewhere and then renaming them. > > - -1 to the name from me as well: ?it isn't "scoped" properly to make it > clear what the module is about. ?If they were inside a pacakge named > 'concurrency' or some such (as hinted by Jesse Noller, I think), the > clash would go away. If people agree with this; do you feel the proposal of said namespace should be a separate PEP, or piggy back on this? I don't want to piggy back on Brian's hard work. Jesse From g.brandl at gmx.net Fri Mar 5 23:01:23 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 05 Mar 2010 23:01:23 +0100 Subject: [Python-Dev] doctest, unicode repr, and 2to3 In-Reply-To: References: <4B908456.2070505@v.loewis.de> Message-ID: Am 05.03.2010 20:37, schrieb Terry Reedy: > On 3/4/2010 11:11 PM, "Martin v. L?wis" wrote: >> Johan Harjano ran into an interesting problem when trying to run the >> Django test suite under Python 3.1. >> >> Django has doctests of the form >> >>>>> a6.headline >> u'Default headline' >> >> Even when converting the doctest with 2to3, the expected output is >> unmodified. However, in 3.x, the expected output will change (i.e. not >> produce an u"" prefix anymore). >> >> Now, it might be possible to reformulate the test case (e.g. use print() >> instead of relying on repr), however, this is undesirable as a) the test >> should continue to test in 2.x that the result object is a unicode >> string, and b) it makes the test less readable. >> >> I would like to find a solution where this gets automatically corrected, >> e.g. through 2to3, or through changes to doctest, or through changes of >> str.__repr__. >> >> Any proposal appreciated. > > What is the easiest thing that works? > > If 2to3 can fix string literals and fix code within doc strings, would > it be difficult to fix expected strings within doc strings? Yes. Expected output from doctests can be anything. Doing conversions on it that are correct only for Python code is potentially producing many false positives. Heuristics need to be applied that can get very intricate. > On the otherhand, as Foord pointed out, Prefect? > the 'u' prefix is something of a > CPythonism not required by the language def, so an 'ignore leading u on > expected output' flag would be useful. But 'correcting' > > >>> a,b > u'ah', u'hah' > > is too much to expect. Doctest is both useful and fragile because it > *almost* imitates human testing methods. Yes, it is documentation and tests at the same time :) Georg From guido at python.org Fri Mar 5 23:01:04 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 5 Mar 2010 14:01:04 -0800 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4222a8491003051342y762fe18bi253cd1f81d8d36f6@mail.gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <4222a8491003051342y762fe18bi253cd1f81d8d36f6@mail.gmail.com> Message-ID: On Fri, Mar 5, 2010 at 1:42 PM, Jesse Noller wrote: > On Fri, Mar 5, 2010 at 3:33 PM, Tres Seaver wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Jesse Noller wrote: >>> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach >>> wrote: >>>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan wrote: >>>>> import futures >>>> +1 on the idea, -1 on the name. ?It's too similar to "from __future__ import >>>> ...". >>> >>> Futures is a common term for this, and implemented named this in other >>> languages. I don't think we should be adopting things that are common, >>> and found elsewhere and then renaming them. >> >> - -1 to the name from me as well: ?it isn't "scoped" properly to make it >> clear what the module is about. ?If they were inside a pacakge named >> 'concurrency' or some such (as hinted by Jesse Noller, I think), the >> clash would go away. > > If people agree with this; do you feel the proposal of said namespace > should be a separate PEP, or piggy back on this? I don't want to piggy > back on Brian's hard work. A simple renaming of futures to concurrency.futures seems easy enough to swallow. (Though I haven't kept track of what other modules the PEP proposes.) -- --Guido van Rossum (python.org/~guido) From g.brandl at gmx.net Fri Mar 5 23:06:33 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 05 Mar 2010 23:06:33 +0100 Subject: [Python-Dev] Desired changes to Hg emails to python-checkins In-Reply-To: References: Message-ID: Am 04.03.2010 21:20, schrieb Brett Cannon: > 1) I miss not having the affected files listed in the subject line. I actually like the (first line of the) commit message better. It communicates much better what the change is about (and for me, if I want to look at/review it). However, what I miss is the summary of affected files at the top of the message. Having to scroll down all of the diffs to find out if e.g. a change to a documentation file is included is tedious, even if the diffs are colored like they are for me. > 2) The To field is set to hg at python.org which > gets rejected as an invalid email address if you reply. Would be better > to set it to python-checkins so that the habitual reply to a checkin > won't get rejected. +1. Georg From solipsis at pitrou.net Fri Mar 5 23:54:04 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 5 Mar 2010 17:54:04 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> Message-ID: <20100305175404.7daeca3e@msiwind> Le Fri, 5 Mar 2010 17:03:02 +1100, Brian Quinlan a ?crit : > > The PEP lives here: > http://python.org/dev/peps/pep-3148/ Ok, here is my take on it: > cancel() > > Attempt to cancel the call. If the call is currently being executed > then it cannot be cancelled and the method will return False, > otherwise the call will be cancelled and the method will return > True. I think it shouldn't return anything, and raise an exception if cancelling failed. It is really an error condition, and ignoring the result doesn't seem right. > Future.running() > > Return True if the call is currently being executed and cannot be > cancelled. > > Future.done() > > Return True if the call was successfully cancelled or finished > running. These don't really make sense since the future is executing concurrently. By the time the result is returned, it can already be wrong. I advocate removing those two methods. > The following Future methods are meant for use in unit tests and > Executor implementations. Their names should then be preceded by an underscore '_'. We don't want people to think they are public APIs and start relying on them. > wait(fs, timeout=None, return_when=ALL_COMPLETED) > [...] > > This method should always be called using keyword arguments I don't think this is right. Keyword arguments are nice, but mandating them too often is IMO a nuisance (after all, it makes things longer to type and requires you to remember the exact parameter names). Especially when the method only takes at most 3 arguments. IMO, keyword-only arguments are mostly useful when there are a lot of positional arguments before, and you want to help the user use the right calling signature. Regards Antoine. From ziga.seilnacht at gmail.com Sat Mar 6 00:08:14 2010 From: ziga.seilnacht at gmail.com (=?windows-1252?Q?=8Eiga_Seilnacht?=) Date: Sat, 06 Mar 2010 00:08:14 +0100 Subject: [Python-Dev] doctest, unicode repr, and 2to3 In-Reply-To: <4B908456.2070505@v.loewis.de> References: <4B908456.2070505@v.loewis.de> Message-ID: <4B918EDE.6070506@gmail.com> Martin v. L?wis wrote: > Johan Harjano ran into an interesting problem when trying to run the > Django test suite under Python 3.1. > > Django has doctests of the form > > >>> a6.headline > u'Default headline' > > Even when converting the doctest with 2to3, the expected output is > unmodified. However, in 3.x, the expected output will change (i.e. not > produce an u"" prefix anymore). > > Now, it might be possible to reformulate the test case (e.g. use print() > instead of relying on repr), however, this is undesirable as a) the test > should continue to test in 2.x that the result object is a unicode > string, and b) it makes the test less readable. > > I would like to find a solution where this gets automatically corrected, > e.g. through 2to3, or through changes to doctest, or through changes of > str.__repr__. > > Any proposal appreciated. You can use a custom DocTestRunner that replaces sys.displayhook in its run() method and records the changed output. Something like the attached seems to do the trick. Regards, Ziga -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: convert_doctest.py URL: From tseaver at palladion.com Sat Mar 6 00:12:17 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 05 Mar 2010 18:12:17 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4222a8491003051342y762fe18bi253cd1f81d8d36f6@mail.gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <4222a8491003051342y762fe18bi253cd1f81d8d36f6@mail.gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jesse Noller wrote: > On Fri, Mar 5, 2010 at 3:33 PM, Tres Seaver wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Jesse Noller wrote: >>> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach >>> wrote: >>>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan wrote: >>>>> import futures >>>> +1 on the idea, -1 on the name. It's too similar to "from __future__ import >>>> ...". >>> Futures is a common term for this, and implemented named this in other >>> languages. I don't think we should be adopting things that are common, >>> and found elsewhere and then renaming them. >> - -1 to the name from me as well: it isn't "scoped" properly to make it >> clear what the module is about. If they were inside a pacakge named >> 'concurrency' or some such (as hinted by Jesse Noller, I think), the >> clash would go away. > > If people agree with this; do you feel the proposal of said namespace > should be a separate PEP, or piggy back on this? I don't want to piggy > back on Brian's hard work. I'm just expressiong a preference for scoping the name, and don't want to preempt the process. If your proposed work on factoring common stuff out of multiprocessing would sit in the same conceptual space, then sharing the package name seems like a good plan to me. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkuRj74ACgkQ+gerLs4ltQ5RNACeKYku88A9PBuQR46QTl7GrEwo mPEAoLdYyi+TLGYFw4SRAIM8zBsNvwxr =iPkb -----END PGP SIGNATURE----- From steven.bethard at gmail.com Sat Mar 6 02:37:15 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Fri, 5 Mar 2010 17:37:15 -0800 Subject: [Python-Dev] argparse ugliness In-Reply-To: References: Message-ID: On Fri, Mar 5, 2010 at 10:51 AM, Neal Becker wrote: > I generally enjoy argparse, but one thing I find rather > ugly and unpythonic. > > ? ?parser.add_argument ('--plot', action='store_true') > > Specifying the argument 'action' as a string is IMO ugly. If it really bothers you, you can use: action=argparse._StoreTrueAction instead. Undocumented (purposefully) but that's all argparse is doing under the covers. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From jyasskin at gmail.com Sat Mar 6 03:00:56 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Fri, 5 Mar 2010 18:00:56 -0800 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100305175404.7daeca3e@msiwind> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100305175404.7daeca3e@msiwind> Message-ID: <5d44f72f1003051800n70e57782i7589ad584db5ee9f@mail.gmail.com> On Fri, Mar 5, 2010 at 2:54 PM, Antoine Pitrou wrote: > Le Fri, 5 Mar 2010 17:03:02 +1100, > Brian Quinlan a ?crit : >> >> The PEP lives here: >> http://python.org/dev/peps/pep-3148/ > > Ok, here is my take on it: > >> cancel() >> >> Attempt to cancel the call. If the call is currently being executed >> then it cannot be cancelled and the method will return False, >> otherwise the call will be cancelled and the method will return >> True. > > I think it shouldn't return anything, and raise an exception if > cancelling failed. It is really an error condition, and ignoring the > result doesn't seem right. The caller can't avoid the error here by querying the future, because of the problem you point out below, so I'm inclined to think that "the future was already started" should be a return value rather than an exception (although that may be my C++ background showing through). Would calling the method try_cancel() work better? >> Future.running() >> >> Return True if the call is currently being executed and cannot be >> cancelled. >> >> Future.done() >> >> Return True if the call was successfully cancelled or finished >> running. > > These don't really make sense since the future is executing > concurrently. By the time the result is returned, it can already be > wrong. I advocate removing those two methods. "done()" can only be wrong in one direction though. If it returns True, it'll never again return False, which can be useful (although perhaps only for polling, which might be an argument for removing it anyway). "running()" becomes True and then False again, so I agree with your objection. A "started()" function would only go from False to True once. Maybe that's a better function? Jeffrey From solipsis at pitrou.net Sat Mar 6 03:28:44 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 5 Mar 2010 21:28:44 -0500 Subject: [Python-Dev] argparse ugliness References: Message-ID: <20100305212844.6edbb0b7@msiwind> Le Fri, 05 Mar 2010 13:51:15 -0500, Neal Becker a ?crit : > I generally enjoy argparse, but one thing I find rather > ugly and unpythonic. > > parser.add_argument ('--plot', action='store_true') I would argue that a string is actually more Pythonic than integers or anonymous objects repurposed as magic constants. (I'm looking at things such as SEEK_SET and friends) From stephen at xemacs.org Sat Mar 6 06:47:03 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 06 Mar 2010 14:47:03 +0900 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <19345.14292.321112.787896@montanaro.dyndns.org> <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com> <4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com> Message-ID: <87eijyuhm0.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > "Future" is a pretty standard CS term for this concept (as noted > "promise" is another), I like the term "promise" better. "Future" is very generic ("not now, but later"), whereas a "promise" is something I don't get from you now, but you will give me later. The wikipedia article is not very helpful on the implicit vs. explicit distinction. As far as I can tell from it, that distinction isn't really attached to "future" vs "promise." The only distinction the article described was in the context of the Alice language, where a future = promise (read-only) plus resolver (mutator). IMO that's not a compelling reason for adopting "future" in Python. From stephen at xemacs.org Sat Mar 6 06:55:25 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 06 Mar 2010 14:55:25 +0900 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100305201822.2792.1965005768.divmod.xquotient.976@localhost.localdomain> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain> <20100305201822.2792.1965005768.divmod.xquotient.976@localhost.localdomain> Message-ID: <87d3ziuh82.fsf@uwakimon.sk.tsukuba.ac.jp> exarkun at twistedmatrix.com writes: > The "explicit" futures on the wikipedia page seems to cover what is > commonly referred to as a future. For example, Java's futures look like > this. > > The "implicit" futures are what is generally called a promise. For > example, E's promises look like this. *sigh* All I can say is "it's a damned shame that there are no native speakers of English working in computer science." I have to admit Jean-Paul's explanation a pretty convincing reason for adopting "future" rather than "promise". But I'm with Skip, I would prefer that the module be named "future" rather than "futures". (Especially when wearing my Professional Economist sweatshirt. :-) From pje at telecommunity.com Sat Mar 6 07:11:14 2010 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 06 Mar 2010 01:11:14 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> Message-ID: <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> At 01:03 AM 3/5/2010, Brian Quinlan wrote: >Hi all, > >I recently submitted a daft PEP for a package designed to make it >easier to execute Python functions asynchronously using threads and >processes. It lets the user focus on their computational problem >without having to build explicit thread/process pools and work queues. > >The package has been discussed on stdlib-sig but now I'd like this >group's feedback. My immediate reaction is that this would be a lot more useful if it built on an API for coroutine yielding/interaction, similar to what's in say, Eventlet. That would seem to make it easier to write synchronous-looking code that operates on futures, and allow futures to be composed more cleanly. ISTM that if futures were standardized before a coroutine API, it would lead to effective orphaning ala what happened with asyncore, especially since the new library is, well, new. I'm somewhat concerned that, as described, the proposed API adds little over what's relatively easy to do with a mature coroutine framework like Eventlet, while at the same time creating yet another alternative (and mutually incompatible) event loop system in the stdlib, beyond the ones that are already in asyncore, tkinter, and the various SocketServer subclasses. As far as naming goes, Twisted uses the term "Deferred" for this concept (and also has a very mature API for handling them). And speaking of Twisted, it seems to me that the PEP would be much improved in general by learning from some of the lessons of other systems. I don't think that Java's example is really the best one to follow in this instance, compared to the many existing async frameworks that have Python-specific experience and APIs to learn from. Or, to put it another way, something that worries me about this PEP is that nearly all of its Python-related citations are for *discussions* of futures, with the only previous Python implementation cited being a crude sketch of a cookbook recipe. The PEP also doesn't address questions of interoperability with existing solutions, compare features with them, or even so much as say, "There are other production implementations of this concept in Python, but we are going to pretend they don't exist." ;-) From jyasskin at gmail.com Sat Mar 6 07:19:30 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Fri, 5 Mar 2010 22:19:30 -0800 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> Message-ID: <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> On Fri, Mar 5, 2010 at 10:11 PM, Phillip J. Eby wrote: > I'm somewhat concerned that, as described, the proposed API ... [creates] yet another alternative (and > mutually incompatible) event loop system in the stdlib ... Futures are a blocking construct; they don't involve an event loop. From pje at telecommunity.com Sat Mar 6 07:50:12 2010 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 06 Mar 2010 01:50:12 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com > References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> Message-ID: <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> At 01:19 AM 3/6/2010, Jeffrey Yasskin wrote: >On Fri, Mar 5, 2010 at 10:11 PM, Phillip J. Eby wrote: > > I'm somewhat concerned that, as described, the proposed API ... > [creates] yet another alternative (and > > mutually incompatible) event loop system in the stdlib ... > >Futures are a blocking construct; they don't involve an event loop. And where they block is in a loop, waiting for events (completed promises) coming back from other threads or processes. The Motivation section of the PEP also stresses avoiding reinvention of such loops, and points to the complication of using more than one at a time as a justification for the mechanism. It seems relevant to at least address why wrapping multiprocessing and multithreading is appropriate, but *not* dealing with any other form of sync/async boundary, *or* composition of futures. On which subject, I might add, the PEP is silent on whether executors are reentrant to the called code. That is, can I call a piece of code that uses futures, using the futures API? How will the called code know what executor to use? Must I pass it one explicitly? Will that work across threads and processes, without explicit support from the API? IOW, as far as I can tell from the PEP, it doesn't look like you can compose futures without *global* knowledge of the application... and in and of itself, this seems to negate the PEP's own motivation to prevent duplication of parallel execution handling! That is, if I use code from module A and module B that both want to invoke tasks asynchronously, and I want to invoke A and B asynchronously, what happens? Based on the design of the API, it appears there is nothing you can do except refactor A and B to take an executor in a parameter, instead of creating their own. It seems therefore to me that either the proposal does not define its scope/motivation very well, or it is not well-equipped to address the problem it's setting out to solve. If it's meant to be something less ambitious -- more like a recipe or example -- it should properly motivate that scope. If it's intended to be a robust tool for composing different pieces of code, OTOH, it should absolutely address the issue of writing composable code... since, that seems to be what it says the purpose of the API is. (I.e., composing code to use a common waiting loop.) And, existing Python async APIs (such as Twisted's Deferreds) actually *address* this issue of composition; the PEP does not. Hence my comments about not looking at existing implementations for API and implementation guidance. (With respect to what the API needs, and how it needs to do it, not necessarily directly copying actual APIs or implementations. Certainly some of the Deferred API naming has a rather, um, "twisted" vocabulary.) From jyasskin at gmail.com Sat Mar 6 08:14:36 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Fri, 5 Mar 2010 23:14:36 -0800 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <87eijyuhm0.fsf@uwakimon.sk.tsukuba.ac.jp> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <19345.14292.321112.787896@montanaro.dyndns.org> <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com> <4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com> <87eijyuhm0.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <5d44f72f1003052314l178f1f94sd90f55967ba83b43@mail.gmail.com> On Fri, Mar 5, 2010 at 9:47 PM, Stephen J. Turnbull wrote: > Guido van Rossum writes: > > ?> "Future" is a pretty standard CS term for this concept (as noted > ?> "promise" is another), > > I like the term "promise" better. ?"Future" is very generic ("not now, > but later"), whereas a "promise" is something I don't get from you > now, but you will give me later. > > The wikipedia article is not very helpful on the implicit vs. explicit > distinction. ?As far as I can tell from it, that distinction isn't > really attached to "future" vs "promise." ?The only distinction the > article described was in the context of the Alice language, where a > future = promise (read-only) plus resolver (mutator). ?IMO that's not > a compelling reason for adopting "future" in Python. It seems like a good idea to follow the choice other languages have used for the name (if they tend to agree) regardless of whether the evil Java followed it too. So let's take a poll: Io: Uses "future" to refer to the implicit kind (http://www.iolanguage.com/scm/io/docs/IoGuide.html#Concurrency-Futures) Alice ML: Uses "future" to refer to the implicit kind, and "promise" to refer to a handle that can fill in the future (http://www.ps.uni-saarland.de/alice/manual/futures.html) Java: Uses "future" to refer to the explicit kind. (http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html) AmbientTalk: Uses "future" to refer to something more like a deferred: you register callbacks to run when the future is resolved. (http://soft.vub.ac.be/amop/at/tutorial/actors#futures) C++0x: Uses "future" to refer to the explicit kind; "promise" similarly to AliceML, and "packaged_task" to get a future from a callable. (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3035.pdf section 30.6) E: Uses "promise" to refer to the implicit kind, and "resolver" to refer to a handle that can fill in the promise. (http://wiki.erights.org/wiki/Promise) Oz: A "future" is a read-only logic variable. I'm not entirely sure what that means. (http://www.mozart-oz.org/documentation/dstutorial/node2.html#label61) PLT Scheme: Uses "future" to refer to the explicit kind. (http://docs.plt-scheme.org/futures/index.html) C#: Uses "Task" to refer to the explicit kind. (http://msdn.microsoft.com/en-us/library/dd321424(VS.100).aspx) Id, from 1991: Used "I-structure" to refer to the implicit kind. (http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.4920) Scala: Uses "Future" to refer to the explicit kind. (http://www.scala-lang.org/docu/files/api/scala/actors/Actor.html#%21%21%28Any%2CPartialFunction%5BAny%2CA%5D%29) What languages did I miss? From this list, "future" seems to be the most popular choice, and it doesn't seem to distinguish between the implicit and explicit kinds. Jeffrey From brian at sweetapp.com Sat Mar 6 10:20:39 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sat, 6 Mar 2010 20:20:39 +1100 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> Message-ID: <7EFBC33B-06AD-4BAD-8173-005D6F08240E@sweetapp.com> On 6 Mar 2010, at 03:21, Daniel Stutzbach wrote: > On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan > wrote: > import futures > > +1 on the idea, -1 on the name. It's too similar to "from > __future__ import ...". > > Also, the PEP should probably link to the discussions on stdlib-sig? I thought about that but this discussion is spread over many threads and many months. Cheers, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at sweetapp.com Sat Mar 6 10:32:24 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sat, 6 Mar 2010 20:32:24 +1100 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> Message-ID: <258205C5-979D-416E-8D54-35438C909CFE@sweetapp.com> On 6 Mar 2010, at 07:38, Brett Cannon wrote: > The PEP says that futures.wait() should only use keyword arguments > past its first positional argument, but the PEP has the function > signature as ``wait(fs, timeout=None, return_when=ALL_COMPLETED)``. > Should it be ``wait(fs, *, timeout=None, return_when=ALL_COMPLETED)``? Hi Brett, That recommendation was designed to make it easy to change the API without breaking code. I'd don't think that recommendation makes sense anymore any I'll update the PEP. Cheers, Brian > On Thu, Mar 4, 2010 at 22:03, Brian Quinlan > wrote: > Hi all, > > I recently submitted a daft PEP for a package designed to make it > easier to execute Python functions asynchronously using threads and > processes. It lets the user focus on their computational problem > without having to build explicit thread/process pools and work queues. > > The package has been discussed on stdlib-sig but now I'd like this > group's feedback. > > The PEP lives here: > http://python.org/dev/peps/pep-3148/ > > Here are two examples to whet your appetites: > > """Determine if several numbers are prime.""" > import futures > import math > > PRIMES = [ > 112272535095293, > 112582705942171, > 112272535095293, > 115280095190773, > 115797848077099, > 1099726899285419] > > def is_prime(n): > if n % 2 == 0: > return False > > sqrt_n = int(math.floor(math.sqrt(n))) > for i in range(3, sqrt_n + 1, 2): > if n % i == 0: > return False > return True > > # Uses as many CPUs as your machine has. > with futures.ProcessPoolExecutor() as executor: > for number, is_prime in zip(PRIMES, executor.map(is_prime, > PRIMES)): > print('%d is prime: %s' % (number, is_prime)) > > > """Print out the size of the home pages of various new sites (and > Fox News).""" > import futures > import urllib.request > > URLS = ['http://www.foxnews.com/', > 'http://www.cnn.com/', > 'http://europe.wsj.com/', > 'http://www.bbc.co.uk/', > 'http://some-made-up-domain.com/'] > > def load_url(url, timeout): > return urllib.request.urlopen(url, timeout=timeout).read() > > with futures.ThreadPoolExecutor(max_workers=5) as executor: > # Create a future for each URL load. > future_to_url = dict((executor.submit(load_url, url, 60), url) > for url in URLS) > > # Iterate over the futures in the order that they complete. > for future in futures.as_completed(future_to_url): > url = future_to_url[future] > if future.exception() is not None: > print('%r generated an exception: %s' % (url, > > future.exception())) > else: > print('%r page is %d bytes' % (url, len(future.result()))) > > Cheers, > Brian > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at sweetapp.com Sat Mar 6 10:37:16 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sat, 6 Mar 2010 20:37:16 +1100 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4222a8491003051342y762fe18bi253cd1f81d8d36f6@mail.gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <4222a8491003051342y762fe18bi253cd1f81d8d36f6@mail.gmail.com> Message-ID: On 6 Mar 2010, at 08:42, Jesse Noller wrote: > If people agree with this; do you feel the proposal of said namespace > should be a separate PEP, or piggy back on this? I don't want to piggy > back on Brian's hard work. It doesn't really matter to me. We can either update this PEP to propose the concurrent.futures name or you can draft a more complete PEP that describes what other functionality should live in the concurrent package. Cheers, Brian From brian at sweetapp.com Sat Mar 6 10:52:46 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sat, 6 Mar 2010 20:52:46 +1100 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100305175404.7daeca3e@msiwind> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100305175404.7daeca3e@msiwind> Message-ID: <7E1039CA-2768-47BF-AD8C-8B12D3D6C20B@sweetapp.com> On 6 Mar 2010, at 09:54, Antoine Pitrou wrote: > Le Fri, 5 Mar 2010 17:03:02 +1100, > Brian Quinlan a ?crit : >> >> The PEP lives here: >> http://python.org/dev/peps/pep-3148/ > > Ok, here is my take on it: > >> cancel() >> >> Attempt to cancel the call. If the call is currently being executed >> then it cannot be cancelled and the method will return False, >> otherwise the call will be cancelled and the method will return >> True. > > I think it shouldn't return anything, and raise an exception if > cancelling failed. It is really an error condition, and ignoring the > result doesn't seem right. In my experience with futures, canceling them is a best-effort optimization that people use when another future fails. For example: futures = [executor.submit(CopyDirectory, src, dest) for dest in ...] finished, unfinished = wait(futures, return_when=FIRST_EXCEPTION) # If there are unfinished futures then there must have been a failure for f in unfinished: # No reason to waste bandwidth copying files if the operation has already failed. f.cancel() for f in finished(): if f.exception(): raise f.exception() >> Future.running() >> >> Return True if the call is currently being executed and cannot be >> cancelled. >> >> Future.done() >> >> Return True if the call was successfully cancelled or finished >> running. > > These don't really make sense since the future is executing > concurrently. By the time the result is returned, it can already be > wrong. I advocate removing those two methods. There methods are useful for logging - by displaying the count of pending, running and completed futures you can estimate the progress of the system. >> The following Future methods are meant for use in unit tests and >> Executor implementations. > > Their names should then be preceded by an underscore '_'. We don't > want > people to think they are public APIs and start relying on them. Actually, as discussed on the stdlib-sig, these methods are designed to make it possible for users to implement their own Executors so we'll have keep the interface stable. >> wait(fs, timeout=None, return_when=ALL_COMPLETED) >> [...] >> >> This method should always be called using keyword arguments > > I don't think this is right. Keyword arguments are nice, but mandating > them too often is IMO a nuisance (after all, it makes things longer to > type and requires you to remember the exact parameter names). > Especially when the method only takes at most 3 arguments. > > IMO, keyword-only arguments are mostly useful when there are a lot of > positional arguments before, and you want to help the user use the > right calling signature. I agree, I'll change this. Cheers, Brian From brian at sweetapp.com Sat Mar 6 11:32:50 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sat, 6 Mar 2010 21:32:50 +1100 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> Message-ID: On 6 Mar 2010, at 17:50, Phillip J. Eby wrote: > At 01:19 AM 3/6/2010, Jeffrey Yasskin wrote: >> On Fri, Mar 5, 2010 at 10:11 PM, Phillip J. Eby > > wrote: >> > I'm somewhat concerned that, as described, the proposed API ... >> [creates] yet another alternative (and >> > mutually incompatible) event loop system in the stdlib ... >> >> Futures are a blocking construct; they don't involve an event loop. > > And where they block is in a loop, waiting for events (completed > promises) coming back from other threads or processes. > > The Motivation section of the PEP also stresses avoiding reinvention > of such loops, and points to the complication of using more than one > at a time as a justification for the mechanism. It seems relevant > to at least address why wrapping multiprocessing and multithreading > is appropriate, but *not* dealing with any other form of sync/async > boundary, *or* composition of futures. > > On which subject, I might add, the PEP is silent on whether > executors are reentrant to the called code. That is, can I call a > piece of code that uses futures, using the futures API? How will > the called code know what executor to use? Must I pass it one > explicitly? Will that work across threads and processes, without > explicit support from the API? Executors are reentrant but deadlock is possible. There are two deadlock examples in the PEP. > > IOW, as far as I can tell from the PEP, it doesn't look like you can > compose futures without *global* knowledge of the application... > and in and of itself, this seems to negate the PEP's own motivation > to prevent duplication of parallel execution handling! > > That is, if I use code from module A and module B that both want to > invoke tasks asynchronously, and I want to invoke A and B > asynchronously, what happens? Based on the design of the API, it > appears there is nothing you can do except refactor A and B to take > an executor in a parameter, instead of creating their own. A and B could both use their own executor instances. You would need to refactor A and B if you wanted to manage thread and process counts globally. > It seems therefore to me that either the proposal does not define > its scope/motivation very well, or it is not well-equipped to > address the problem it's setting out to solve. If it's meant to be > something less ambitious -- more like a recipe or example -- it > should properly motivate that scope. If it's intended to be a > robust tool for composing different pieces of code, OTOH, it should > absolutely address the issue of writing composable code... since, > that seems to be what it says the purpose of the API is. (I.e., > composing code to use a common waiting loop.) My original motivation when designing this module was having to deal with a lot of code that looks like this: def get_some_user_info(user): x = make_ldap_call1(user) y = make_ldap_call2(user) z = [make_db_call(user, i) for i in something] # Do some processing with x, y, z and return a result Doing these operations serially is too slow. So how do I parallelize them? Using the threading module is the obvious choice but having to create my own work/result queue every time I encounter this pattern is annoying. The futures module lets you write this as: def get_some_user_info(user): with ThreadPoolExecutor(max_threads=10) as executor: x_future = executor.submit(make_ldap_call1, user) y_future = executor.submit(make_ldap_call2, user) z_futures = [executor.submit(make_db_call, user, i) for i in something] finished, _ = wait([x_future, y_future] + z_futures, return_when=FIRST_EXCEPTION) for f in finished: if f.exception(): raise f.exception() x = x_future.result() y = y_future.result() z = [f.result() for f in z_futures] # Do some processing with x, y, z and return a result > And, existing Python async APIs (such as Twisted's Deferreds) > actually *address* this issue of composition; the PEP does not. > Hence my comments about not looking at existing implementations for > API and implementation guidance. (With respect to what the API > needs, and how it needs to do it, not necessarily directly copying > actual APIs or implementations. Certainly some of the Deferred API > naming has a rather, um, "twisted" vocabulary.) Using twisted (or any other asynchronous I/O framework) forces you to rewrite your I/O code. Futures do not. Cheers, Brian From ncoghlan at gmail.com Sat Mar 6 13:28:08 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 06 Mar 2010 22:28:08 +1000 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <4222a8491003051342y762fe18bi253cd1f81d8d36f6@mail.gmail.com> Message-ID: <4B924A58.4030000@gmail.com> Brian Quinlan wrote: > > On 6 Mar 2010, at 08:42, Jesse Noller wrote: >> If people agree with this; do you feel the proposal of said namespace >> should be a separate PEP, or piggy back on this? I don't want to piggy >> back on Brian's hard work. > > It doesn't really matter to me. > > We can either update this PEP to propose the concurrent.futures name or > you can draft a more complete PEP that describes what other > functionality should live in the concurrent package. I think a "concurrent.futures" name works - it gives the scoping desired by the folks with an finance background and gives us a bucket for future thread/process agnostic concurrency tools (such as a pools and message queues). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From stephen at xemacs.org Sat Mar 6 13:46:23 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 06 Mar 2010 21:46:23 +0900 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <5d44f72f1003052314l178f1f94sd90f55967ba83b43@mail.gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <19345.14292.321112.787896@montanaro.dyndns.org> <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com> <4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com> <87eijyuhm0.fsf@uwakimon.sk.tsukuba.ac.jp> <5d44f72f1003052314l178f1f94sd90f55967ba83b43@mail.gmail.com> Message-ID: <871vfxvcrk.fsf@uwakimon.sk.tsukuba.ac.jp> Jeffrey Yasskin writes: > It seems like a good idea to follow the choice other languages have > used for the name (if they tend to agree) For the record, I've conceded that point. From ncoghlan at gmail.com Sat Mar 6 13:43:40 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 06 Mar 2010 22:43:40 +1000 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> Message-ID: <4B924DFC.9070207@gmail.com> Brian Quinlan wrote: >> IOW, as far as I can tell from the PEP, it doesn't look like you can >> compose futures without *global* knowledge of the application... and >> in and of itself, this seems to negate the PEP's own motivation to >> prevent duplication of parallel execution handling! >> >> That is, if I use code from module A and module B that both want to >> invoke tasks asynchronously, and I want to invoke A and B >> asynchronously, what happens? Based on the design of the API, it >> appears there is nothing you can do except refactor A and B to take an >> executor in a parameter, instead of creating their own. > > A and B could both use their own executor instances. You would need to > refactor A and B if you wanted to manage thread and process counts > globally. You may want to consider providing global thread and process executors in the futures module itself. Code which just wants to say "do this in the background" without having to manage the lifecycle of its own executor instance is then free to do so. I've had a lot of experience with a framework that provides this and it is *very* convenient (it's also a good way to avoid deadlocks due to synchronous notification APIs). On PJE's broader point, async event loops with non-blocking I/O and messages passed back to the event loop to indicate completion of operations and relying on threads and processes to farm out tasks (which you will later block on in order to retrieve the results) are completely different programming models. This PEP doesn't change that - it just makes certain aspects of the latter approach easier to handle. Trying to design an API that can cope with either model strikes me as a fool's errand. They differ at such a fundamental level that I don't see how a hybrid API could be particularly optimal for either approach. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sat Mar 6 13:50:47 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 06 Mar 2010 22:50:47 +1000 Subject: [Python-Dev] argparse ugliness In-Reply-To: References: Message-ID: <4B924FA7.4050807@gmail.com> Neal Becker wrote: > I generally enjoy argparse, but one thing I find rather > ugly and unpythonic. > > parser.add_argument ('--plot', action='store_true') > > Specifying the argument 'action' as a string is IMO ugly. It isn't ideal, but having to grab constants from the module isn't ideal either. Using a string here allows argparse (and optparse before it) to spit out an explicit "unknown action" message. Using named constants a typo would just get you a generic NameError instead. With variable names only being checked at runtime, you wouldn't get the static name checking that is the primary benefit of using named constants in less dynamic languages. So, agreed that using strings isn't pretty, but there aren't any great alternatives available. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From daniel at stutzbachenterprises.com Sat Mar 6 16:35:53 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Sat, 6 Mar 2010 09:35:53 -0600 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4B924DFC.9070207@gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <4B924DFC.9070207@gmail.com> Message-ID: On Sat, Mar 6, 2010 at 6:43 AM, Nick Coghlan wrote: > You may want to consider providing global thread and process executors > in the futures module itself. Code which just wants to say "do this in > the background" without having to manage the lifecycle of its own > executor instance is then free to do so. +1 -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Sat Mar 6 17:04:26 2010 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 06 Mar 2010 11:04:26 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> Message-ID: <20100306160434.39FBD3A4077@sparrow.telecommunity.com> At 05:32 AM 3/6/2010, Brian Quinlan wrote: >Using twisted (or any other asynchronous I/O framework) forces you to >rewrite your I/O code. Futures do not. Twisted's "Deferred" API has nothing to do with I/O. From gregg.lind at gmail.com Sat Mar 6 17:27:46 2010 From: gregg.lind at gmail.com (Gregg Lind) Date: Sat, 6 Mar 2010 10:27:46 -0600 Subject: [Python-Dev] Modifying Grammar/grammar and other foul acts Message-ID: Python-devs, I'm writing to you for some help in understanding the Python grammar. As an excuse to deep dive into Python's tokenizer / grammar, I decided (as a hideous, hideous joke) to want to allow braces where colons are allowed (as flow control). Starting from PEP 306 (and branch r311), I hacked on Grammar/Grammer As a first example: funcdef: ('def' NAME parameters ['->' test] ':' suite | 'def' NAME parameters ['->' test] '{' suite '}' ) I reran Parser/pgen and the dfa changes, but python (3.1) when recompiled, throws errors on things like: def a() { None } Strangely enough: lambdef: ( 'lambda' [varargslist] ':' test | 'lambda' [varargslist] '{' test '}' ) works fine! I this simplely some difference between "test" and "suite". I have tried tackling this with gdb, looking at err_input clearly isn't enough. (gdb) break err_input (gdb) break PyParser_ASTFromString import sys b = compile("def a() {pass}","sys.stdout","single") # yet a simple grammar fix is enough for this! c = compile("lambda x {None}","sys.stdout","single") I'm in over my head! Any insights / help would be appreciated. Full-on flaming is also appropriate, but would be less appreciated. Specific questions 1.) I assume the Grammar/grammar is read top to bottom. Confirm? 2.) More help figuring out how to debug what python *thinks* it's seeing when it see "def a() {pass}". It's not getting to the ast construction stage, as near as I can tell. What additional breakpoints can I set to see where it's failing. Gregg L. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Sat Mar 6 17:56:20 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 06 Mar 2010 17:56:20 +0100 Subject: [Python-Dev] Modifying Grammar/grammar and other foul acts In-Reply-To: References: Message-ID: <4B928934.2070103@v.loewis.de> > 1.) I assume the Grammar/grammar is read top to bottom. Confirm? Confirm - but this is not surprising: *any* source file is typically read from top to bottom. Randoma access reading is typically done for binary files, only. So you must be asking something else, but I can't guess what that might be. > 2.) More help figuring out how to debug what python *thinks* it's > seeing when it see "def a() {pass}". It's not getting to the ast > construction stage, as near as I can tell. What additional breakpoints > can I set to see where it's failing. Enable the D() debugging in parser.c (configure with --with-pydebug, and set PYTHONDEBUG). Regards, Martin From python-dev at masklinn.net Sat Mar 6 18:26:00 2010 From: python-dev at masklinn.net (Xavier Morel) Date: Sat, 6 Mar 2010 18:26:00 +0100 Subject: [Python-Dev] argparse ugliness In-Reply-To: <4B924FA7.4050807@gmail.com> References: <4B924FA7.4050807@gmail.com> Message-ID: I don't believe argparse's action specification scheme is bad either, but On 6 Mar 2010, at 13:50 , Nick Coghlan wrote: > > you wouldn't get the static name checking that is > the primary benefit of using named constants in less dynamic languages. There are quite a few tools which do handle static checking of that kind of stuff, so while Python itself doesn't provide it (as a builtin) it's not like you can't get it some other way. From gregg.lind at gmail.com Sat Mar 6 19:26:25 2010 From: gregg.lind at gmail.com (Gregg Lind) Date: Sat, 6 Mar 2010 12:26:25 -0600 Subject: [Python-Dev] Modifying Grammar/grammar and other foul acts In-Reply-To: <4B928934.2070103@v.loewis.de> References: <4B928934.2070103@v.loewis.de> Message-ID: Sorry, re: question one, forgive the ill-formed question. I meant more, are the parser rules applied "first matching". Essentially trying to confirm that the parser is "top down" or "bottom up" or whether or not it even matters. Thanks for the tip -- it seems to be exactly what I want. To make it explicit, this seems to be fuller (unix) recipe for how to make this style of debugging happen. $ ./configure --with-pydebug $ make $ set PYTHONDEBUG=1 $ ./python -d # then this shows the parsing info On Sat, Mar 6, 2010 at 10:56 AM, "Martin v. L?wis" wrote: > > 1.) I assume the Grammar/grammar is read top to bottom. Confirm? > > Confirm - but this is not surprising: *any* source file is typically > read from top to bottom. Randoma access reading is typically done for > binary files, only. > > So you must be asking something else, but I can't guess what that might be. > > > 2.) More help figuring out how to debug what python *thinks* it's > > seeing when it see "def a() {pass}". It's not getting to the ast > > construction stage, as near as I can tell. What additional breakpoints > > can I set to see where it's failing. > > Enable the D() debugging in parser.c (configure with --with-pydebug, > and set PYTHONDEBUG). > > Regards, > Martin > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Mar 6 20:43:20 2010 From: brett at python.org (Brett Cannon) Date: Sat, 6 Mar 2010 11:43:20 -0800 Subject: [Python-Dev] Silencing DeprecationWarning by default in Python 2.7 is silencing division warnings Message-ID: I see two possible fixes for this. One is to not silence DeprecationWarning if Py_DivisionWarningFlag is set to >= 1. The other is to introduce a new subclass of DeprecationWarning called IntegerDivisionWarning and have that added to the warnings filter so that if it is triggered it is handled separately from what DeprecationWarning triggers. The former means that you might get more than you bargained for in terms of warnings as you are suddenly switching on all DeprecationWarnings on top of division warnings. The latter means that you now have to explicit care about IntegerDivisionWarning on top of DeprecationWarning (to minimize this I could have IntegerDivisionWarning added to the warnings filter only in the case of when Py_DivisionWarningFlag is set instead of blindly adding it). Thoughts? -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sat Mar 6 20:51:05 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 6 Mar 2010 13:51:05 -0600 Subject: [Python-Dev] Silencing DeprecationWarning by default in Python 2.7 is silencing division warnings In-Reply-To: References: Message-ID: <1afaf6161003061151i5415116ex3d13c713f46a3ab4@mail.gmail.com> 2010/3/6 Brett Cannon : > I see two possible fixes for this. One is to not silence DeprecationWarning > if Py_DivisionWarningFlag is set to >= 1. The other is to introduce a new > subclass of DeprecationWarning called IntegerDivisionWarning and have that > added to the warnings filter so that if it is triggered it is handled > separately from what DeprecationWarning triggers. > The former means that you might get more than you bargained for in terms of > warnings as you are suddenly switching on all DeprecationWarnings on top of > division warnings. The latter means that you now have to explicit care about > IntegerDivisionWarning on top of DeprecationWarning (to minimize this I > could have IntegerDivisionWarning added to the warnings filter only in the > case of when Py_DivisionWarningFlag is set instead of blindly adding it). > Thoughts? How about just setting the warnings filter based on the integer division warning message? Might be a little brittle, but I don't much third party code is warning about classic division. -- Regards, Benjamin From digitalxero at gmail.com Sat Mar 6 22:20:10 2010 From: digitalxero at gmail.com (Dj Gilcrease) Date: Sat, 6 Mar 2010 14:20:10 -0700 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100306160434.39FBD3A4077@sparrow.telecommunity.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> Message-ID: I have been playing with the feedback branch of this package for py3 and there seems to be a rather serious bug in the Process version. Using the code @ http://dpaste.com/hold/168795/ When I was running in debug mode I found that as soon as p = multiprocessing.Process( target=_process_worker, args=(self._call_queue, self._result_queue, self._shutdown_process_event)) was called (yes even before p.start() was called) the processes just started launching all by themselves. I am also wondering why you are launching the process directly instead of using a Pool since you are limiting the number of processes always wouldnt it be better to launch the worker processes up front then just add worker items to the queue? From digitalxero at gmail.com Sat Mar 6 22:40:34 2010 From: digitalxero at gmail.com (Dj Gilcrease) Date: Sat, 6 Mar 2010 14:40:34 -0700 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> Message-ID: I am also getting an odd error on odd error on exit Error in atexit._run_exitfuncs: TypeError: print_exception(): Exception expected for value, str found From guido at python.org Sat Mar 6 22:41:01 2010 From: guido at python.org (Guido van Rossum) Date: Sat, 6 Mar 2010 13:41:01 -0800 Subject: [Python-Dev] Modifying Grammar/grammar and other foul acts In-Reply-To: References: <4B928934.2070103@v.loewis.de> Message-ID: On Sat, Mar 6, 2010 at 10:26 AM, Gregg Lind wrote: > Sorry, re: question one, forgive the ill-formed question.? I meant more, are > the parser rules applied "first matching".?? Essentially trying to confirm > that the parser is "top down" or "bottom up" or whether or not it even > matters. That's not how it works at all. I can't explain it in a few words -- but any text on LL(1) parsing should clarify this. The parser uses no backtracking and a 1-token lookahead. The only unusual thing is that individual rules use a regex-like notation, but that is all converted to a DFA. If one token is not enough to know which path to take through the DFA (this may invoke another rule -- but you always know which one) you're hosed. I suspect you've introduced ambiguities, though I don't immediately see where (they could be in the combination of different rules). Another possibility is that you may be running into problems where the parser expects a newline at the end of a suite. (FWIW since you're not proposing a language change, this is technically off-topic for python-dev. :-) --Guido > Thanks for the tip -- it seems to be exactly what I want.? To make it > explicit, this seems to be fuller (unix) recipe for how to make this style > of debugging happen. > > ??? $ ./configure --with-pydebug > ??? $ make > ??? $ set PYTHONDEBUG=1 > ??? $ ./python -d?? # then this shows the parsing info > > > On Sat, Mar 6, 2010 at 10:56 AM, "Martin v. L?wis" > wrote: >> >> > 1.) ?I assume the Grammar/grammar is read top to bottom. ?Confirm? >> >> Confirm - but this is not surprising: *any* source file is typically >> read from top to bottom. Randoma access reading is typically done for >> binary files, only. >> >> So you must be asking something else, but I can't guess what that might >> be. >> >> > 2.) ?More help figuring out how to debug what python *thinks* it's >> > seeing when it see "def a() {pass}". ?It's not getting to the ast >> > construction stage, as near as I can tell. ?What additional breakpoints >> > can I set to see where it's failing. >> >> Enable the D() debugging in parser.c (configure with --with-pydebug, >> and set PYTHONDEBUG). >> >> Regards, >> Martin >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) From jnoller at gmail.com Sat Mar 6 22:58:19 2010 From: jnoller at gmail.com (Jesse Noller) Date: Sat, 6 Mar 2010 16:58:19 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> Message-ID: <52B6BD6E-5678-4F74-89DE-EF3A47AD5C6A@gmail.com> On Mar 6, 2010, at 4:20 PM, Dj Gilcrease wrote: > I have been playing with the feedback branch of this package for py3 > and there seems to be a rather serious bug in the Process version. > Using the code @ http://dpaste.com/hold/168795/ > > When I was running in debug mode I found that as soon as > > p = multiprocessing.Process( > target=_process_worker, > args=(self._call_queue, > self._result_queue, > self._shutdown_process_event)) > > was called (yes even before p.start() was called) the processes just > started launching all by themselves. > Did you run the provided example code on windows by chance? If so, look at the multiprocessing docs, there are restrictions on windows (see the __main__ note) - not following the guidelines can result in lots of processes spawning. From benjamin at python.org Sat Mar 6 23:13:46 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 6 Mar 2010 16:13:46 -0600 Subject: [Python-Dev] [RELEASED] Python 3.1.2 release candidate Message-ID: <1afaf6161003061413k646e972eta515d9322793e208@mail.gmail.com> On behalf of the Python development team, I'm pleased to announce a release candidate for the second bugfix release of the Python 3.1 series, Python 3.1.2. This bug fix release fixes numerous issues found in 3.1.1. This release candidate has been released to solicit testing and feedback over an possible regressions from 3.1.1. Please consider testing it with your library or application and reporting an bugs you encounter. This will help make the final 3.1.2 release, planned in 2 weeks time, all the more stable. The Python 3.1 version series focuses on the stabilization and optimization of the features and changes that Python 3.0 introduced. For example, the new I/O system has been rewritten in C for speed. File system APIs that use unicode strings now handle paths with undecodable bytes in them. Other features include an ordered dictionary implementation, a condensed syntax for nested with statements, and support for ttk Tile in Tkinter. For a more extensive list of changes in 3.1, see http://doc.python.org/3.1/whatsnew/3.1.html or Misc/NEWS in the Python distribution. To download Python 3.1.2rc1 visit: http://www.python.org/download/releases/3.1.2/ A list of changes in 3.1.2rc1 can be found here: http://svn.python.org/projects/python/tags/r312rc1/Misc/NEWS The 3.1 documentation can be found at: http://docs.python.org/3.1 Bugs can always be reported to: http://bugs.python.org Enjoy! -- Benjamin Peterson Release Manager benjamin at python.org (on behalf of the entire python-dev team and 3.1.2's contributors) From benjamin at python.org Sat Mar 6 23:41:33 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 6 Mar 2010 16:41:33 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 4 Message-ID: <1afaf6161003061441j5621bbe7jdb8ba4c22f81f46e@mail.gmail.com> On behalf of the Python development team, I'm overjoyed to announce the fourth alpha release of Python 2.7. Python 2.7 is scheduled (by Guido and Python-dev) to be the last major version in the 2.x series. Though more major releases have not been absolutely ruled out, it's likely that the 2.7 release will an extended period of maintenance for the 2.x series. 2.7 includes many features that were first released in Python 3.1. The faster io module, the new nested with statement syntax, improved float repr, set literals, dictionary views, and the memoryview object have been backported from 3.1. Other features include an ordered dictionary implementation, unittests improvements, a new sysconfig module, and support for ttk Tile in Tkinter. For a more extensive list of changes in 2.7, see http://doc.python.org/dev/whatsnew/2.7.html or Misc/NEWS in the Python distribution. To download Python 2.7 visit: http://www.python.org/download/releases/2.7/ Please note that this is a development release, intended as a preview of new features for the community, and is thus not suitable for production use. The 2.7 documentation can be found at: http://docs.python.org/2.7 Please consider trying Python 2.7 with your code and reporting any bugs you may notice to: http://bugs.python.org Enjoy! -- Benjamin Peterson 2.7 Release Manager benjamin at python.org (on behalf of the entire python-dev team and 2.7's contributors) From ben+python at benfinney.id.au Sat Mar 6 23:47:57 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 07 Mar 2010 09:47:57 +1100 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain> <20100305201822.2792.1965005768.divmod.xquotient.976@localhost.localdomain> <87d3ziuh82.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87hbot12zm.fsf@benfinney.id.au> "Stephen J. Turnbull" writes: > I have to admit Jean-Paul's explanation a pretty convincing reason for > adopting "future" rather than "promise". But I'm with Skip, I would > prefer that the module be named "future" rather than "futures". Has anyone in this very long thread raised the issue that Python *already* uses this term for the name of a module with a totally unrelated purpose; the ?__future__? pseudo-module? That alone seems a pretty strong reason to avoid the word ?future? (singular or plural) for some other module name. -- \ ?Creativity can be a social contribution, but only in so far as | `\ society is free to use the results.? ?Richard Stallman | _o__) | Ben Finney From jnoller at gmail.com Sat Mar 6 23:55:08 2010 From: jnoller at gmail.com (Jesse Noller) Date: Sat, 6 Mar 2010 17:55:08 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <87hbot12zm.fsf@benfinney.id.au> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain> <20100305201822.2792.1965005768.divmod.xquotient.976@localhost.localdomain> <87d3ziuh82.fsf@uwakimon.sk.tsukuba.ac.jp> <87hbot12zm.fsf@benfinney.id.au> Message-ID: <07376A42-ECA8-483D-BDDA-EEE6864EA6E8@gmail.com> On Mar 6, 2010, at 5:47 PM, Ben Finney wrote: > "Stephen J. Turnbull" writes: > >> I have to admit Jean-Paul's explanation a pretty convincing reason >> for >> adopting "future" rather than "promise". But I'm with Skip, I would >> prefer that the module be named "future" rather than "futures". > > Has anyone in this very long thread raised the issue that Python > *already* uses this term for the name of a module with a totally > unrelated purpose; the ?__future__? pseudo-module? > > That alone seems a pretty strong reason to avoid the word ?future? > (singular or plural) for some other module name. > Yes, they have, and putting it in a sub namespace has also come up. In the thread. From greg.ewing at canterbury.ac.nz Sun Mar 7 00:19:42 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 07 Mar 2010 12:19:42 +1300 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <5d44f72f1003051800n70e57782i7589ad584db5ee9f@mail.gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100305175404.7daeca3e@msiwind> <5d44f72f1003051800n70e57782i7589ad584db5ee9f@mail.gmail.com> Message-ID: <4B92E30E.9070400@canterbury.ac.nz> Jeffrey Yasskin wrote: > The caller can't avoid the error here by querying the future, because > of the problem you point out below, so I'm inclined to think that "the > future was already started" should be a return value rather than an > exception (although that may be my C++ background showing through). I think it's your C++ background showing. In Python philosophy, there's no particlular connection between whether something can be tested for and whether it should raise an exception. The thing to consider, I think, is whether it makes sense in a large proportion of use cases to ignore the fact that the cancel failed and carry on regardless. If not, then raising an exception makes it much harder to accidentally ignore the situation. -- Greg From greg.ewing at canterbury.ac.nz Sun Mar 7 00:28:17 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 07 Mar 2010 12:28:17 +1300 Subject: [Python-Dev] argparse ugliness In-Reply-To: <20100305212844.6edbb0b7@msiwind> References: <20100305212844.6edbb0b7@msiwind> Message-ID: <4B92E511.3070304@canterbury.ac.nz> Antoine Pitrou wrote: > I would argue that a string is actually more Pythonic than > integers or anonymous objects repurposed as magic constants. > (I'm looking at things such as SEEK_SET and friends) Strings are certainly preferable to ints, one reason being that they display as something meaningful in debugging output. But it also doesn't hurt to export named constants for them, in the interests of catching typos. -- Greg From greg.ewing at canterbury.ac.nz Sun Mar 7 00:37:26 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 07 Mar 2010 12:37:26 +1300 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> Message-ID: <4B92E736.2010005@canterbury.ac.nz> Phillip J. Eby wrote: > while at the same time creating yet another alternative (and > mutually incompatible) event loop system in the stdlib, beyond the ones > that are already in asyncore, tkinter, and the various SocketServer > subclasses. Aaargh... that's the *last* thing we need! I've been thinking for a while that it would be a big help if there were one, standardised module in the stdlib for handling async events, and all the other gui toolkits etc. were made to use it. -- Greg From fuzzyman at voidspace.org.uk Sun Mar 7 00:38:51 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 06 Mar 2010 23:38:51 +0000 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4B92E736.2010005@canterbury.ac.nz> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <4B92E736.2010005@canterbury.ac.nz> Message-ID: <4B92E78B.6060503@voidspace.org.uk> On 06/03/2010 23:37, Greg Ewing wrote: > Phillip J. Eby wrote: >> while at the same time creating yet another alternative (and mutually >> incompatible) event loop system in the stdlib, beyond the ones that >> are already in asyncore, tkinter, and the various SocketServer >> subclasses. > > Aaargh... that's the *last* thing we need! > > I've been thinking for a while that it would be a big help > if there were one, standardised module in the stdlib for > handling async events, and all the other gui toolkits > etc. were made to use it. > Wouldn't it have to be the Tcl event loop then? Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From greg.ewing at canterbury.ac.nz Sun Mar 7 00:59:20 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 07 Mar 2010 12:59:20 +1300 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <258205C5-979D-416E-8D54-35438C909CFE@sweetapp.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <258205C5-979D-416E-8D54-35438C909CFE@sweetapp.com> Message-ID: <4B92EC58.4090501@canterbury.ac.nz> Brian Quinlan wrote: > That recommendation was designed to make it easy to change the API > without breaking code. > > I'd don't think that recommendation makes sense anymore any I'll update > the PEP. I don't think there's anything wrong with stating that the order of the arguments is not a guaranteed part of the API. There isn't necessarily any need to enforce that, though. -- Greg From daniel at stutzbachenterprises.com Sun Mar 7 00:52:01 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Sat, 6 Mar 2010 17:52:01 -0600 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4B92E78B.6060503@voidspace.org.uk> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <4B92E736.2010005@canterbury.ac.nz> <4B92E78B.6060503@voidspace.org.uk> Message-ID: On Sat, Mar 6, 2010 at 5:38 PM, Michael Foord wrote: > On 06/03/2010 23:37, Greg Ewing wrote: > >> I've been thinking for a while that it would be a big help >> if there were one, standardised module in the stdlib for >> handling async events, and all the other gui toolkits >> etc. were made to use it. >> >> Wouldn't it have to be the Tcl event loop then? > I image he means a standardized Abstract Base Class, which each GUI toolkit would subclass. That's more or less how Twisted's "reactors" work. That way non-GUI async code can just use the ABC and not worry about what event loop is running underneath (be it TCL, GTK, or just poll()). -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sun Mar 7 01:19:41 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 07 Mar 2010 13:19:41 +1300 Subject: [Python-Dev] Modifying Grammar/grammar and other foul acts In-Reply-To: References: <4B928934.2070103@v.loewis.de> Message-ID: <4B92F11D.30402@canterbury.ac.nz> Gregg Lind wrote: > Sorry, re: question one, forgive the ill-formed question. I meant more, > are the parser rules applied "first matching". Essentially trying to > confirm that the parser is "top down" or "bottom up" or whether or not > it even matters. I think pgen would complain if you had two rules that could both match at the same time, so the question doesn't apply. BTW, be careful with terminology here -- the terms "top down" and "bottom up" have a different meaning in parsing theory (they refer to the way the parse tree is built, not the order of matching in a list of rules). -- Greg From greg.ewing at canterbury.ac.nz Sun Mar 7 01:37:42 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 07 Mar 2010 13:37:42 +1300 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4B92E78B.6060503@voidspace.org.uk> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <4B92E736.2010005@canterbury.ac.nz> <4B92E78B.6060503@voidspace.org.uk> Message-ID: <4B92F556.1080309@canterbury.ac.nz> Michael Foord wrote: > Wouldn't it have to be the Tcl event loop then? No, tcl/tk would have to be threatened with the comfy chair until it allowed itself to be spliced into the official event loop somehow. -- Greg From skip at pobox.com Sun Mar 7 02:02:48 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 6 Mar 2010 19:02:48 -0600 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <87hbot12zm.fsf@benfinney.id.au> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> <20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain> <20100305201822.2792.1965005768.divmod.xquotient.976@localhost.localdomain> <87d3ziuh82.fsf@uwakimon.sk.tsukuba.ac.jp> <87hbot12zm.fsf@benfinney.id.au> Message-ID: <19346.64312.597466.830641@montanaro.dyndns.org> Ben> Has anyone in this very long thread raised the issue that Python Ben> *already* uses this term for the name of a module with a totally Ben> unrelated purpose; the ???__future__??? pseudo-module? Yes, it's already come up. While not quite the same it does remind me of the __builtin__/__builtins__ confusion. Skip From digitalxero at gmail.com Sun Mar 7 02:47:46 2010 From: digitalxero at gmail.com (Dj Gilcrease) Date: Sat, 6 Mar 2010 18:47:46 -0700 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <52B6BD6E-5678-4F74-89DE-EF3A47AD5C6A@gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <52B6BD6E-5678-4F74-89DE-EF3A47AD5C6A@gmail.com> Message-ID: On Sat, Mar 6, 2010 at 2:58 PM, Jesse Noller wrote: > Did you run the provided example code on windows by chance? If so, look at > the multiprocessing docs, there are restrictions on windows (see the > __main__ note) - not following the guidelines can result in lots of > processes spawning. Yes, on win7. I might recommend then that the examples in the PEP be restructured to work correctly on windows by including the if __name__ == '__main__': From brian at sweetapp.com Sun Mar 7 03:10:26 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sun, 7 Mar 2010 13:10:26 +1100 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100306160434.39FBD3A4077@sparrow.telecommunity.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> Message-ID: <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> On 7 Mar 2010, at 03:04, Phillip J. Eby wrote: > At 05:32 AM 3/6/2010, Brian Quinlan wrote: >> Using twisted (or any other asynchronous I/O framework) forces you to >> rewrite your I/O code. Futures do not. > > Twisted's "Deferred" API has nothing to do with I/O. I see, you just mean the API and not the underlying model. We discussed the Deferred API on the stdlib-sig and I don't think that anyone expressed a preference for it over the one described in the PEP. Do you have any concrete criticism? Cheers, Brian From exarkun at twistedmatrix.com Sun Mar 7 03:34:54 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sun, 07 Mar 2010 02:34:54 -0000 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> Message-ID: <20100307023454.2792.1549472893.divmod.xquotient.1332@localhost.localdomain> On 02:10 am, brian at sweetapp.com wrote: >On 7 Mar 2010, at 03:04, Phillip J. Eby wrote: >>At 05:32 AM 3/6/2010, Brian Quinlan wrote: >>>Using twisted (or any other asynchronous I/O framework) forces you to >>>rewrite your I/O code. Futures do not. >> >>Twisted's "Deferred" API has nothing to do with I/O. > >I see, you just mean the API and not the underlying model. > >We discussed the Deferred API on the stdlib-sig and I don't think that >anyone expressed a preference for it over the one described in the PEP. > >Do you have any concrete criticism? From reading some of the stdlib-sig archives, it sounds like there is general agreement that Deferreds and Futures can be used to complement each other, and that getting code that is primarily Deferred-based to integrate with Future-based code or vice versa should eventually be possible. Do I have the right sense of people's feelings? And relatedly, once Futures are accepted and implemented, are people going to use them as an argument to exclude Deferreds from the stdlib (or be swayed by other people making such arguments)? Hopefully not, given what I read on stdlib-sig, but it doesn't hurt to check... Jean-Paul From jnoller at gmail.com Sun Mar 7 03:40:54 2010 From: jnoller at gmail.com (Jesse Noller) Date: Sat, 6 Mar 2010 21:40:54 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100307023454.2792.1549472893.divmod.xquotient.1332@localhost.localdomain> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307023454.2792.1549472893.divmod.xquotient.1332@localhost.localdomain> Message-ID: <4222a8491003061840j78f84ecfs4dbb7997d1c27d21@mail.gmail.com> On Sat, Mar 6, 2010 at 9:34 PM, wrote: > On 02:10 am, brian at sweetapp.com wrote: >> >> On 7 Mar 2010, at 03:04, Phillip J. Eby wrote: >>> >>> At 05:32 AM 3/6/2010, Brian Quinlan wrote: >>>> >>>> Using twisted (or any other asynchronous I/O framework) forces you to >>>> rewrite your I/O code. Futures do not. >>> >>> Twisted's "Deferred" API has nothing to do with I/O. >> >> I see, you just mean the API and not the underlying model. >> >> We discussed the Deferred API on the stdlib-sig and I don't think that >> anyone expressed a preference for it over the one described in the PEP. >> >> Do you have any concrete criticism? > > From reading some of the stdlib-sig archives, it sounds like there is > general agreement that Deferreds and Futures can be used to complement each > other, and that getting code that is primarily Deferred-based to integrate > with Future-based code or vice versa should eventually be possible. > > Do I have the right sense of people's feelings? > > And relatedly, once Futures are accepted and implemented, are people going > to use them as an argument to exclude Deferreds from the stdlib (or be > swayed by other people making such arguments)? ?Hopefully not, given what I > read on stdlib-sig, but it doesn't hurt to check... > > Jean-Paul Generally speaking; I don't see futures as an exclusion to Deferreds, or other asynchronous doodads. I just see it as a useful construct on top of threads and processes primarily. So in my mind, no. jesse From digitalxero at gmail.com Sun Mar 7 04:09:59 2010 From: digitalxero at gmail.com (Dj Gilcrease) Date: Sat, 6 Mar 2010 20:09:59 -0700 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> Message-ID: After playing with the API for a while & running into many issues with the examples & tests crashing windows I decided to modify the API a little and fix up the examples so they dont crash windows based computers. http://code.google.com/p/pythonfutures/issues/detail?id=1 API Change that changes the current Executor to ExecutorBase and adds a new Executor class that is used like futures.Executor() # creates an executor that uses threading and a max_workers = to the number of cpus futures.Executor(use='process') # Creates an executor that uses multiprocessing and a max_workers = to the number of cpus futures.Executor(max_workers=5) # threading again, just specifying the number of workers futures.Executor(use='process', max_workers=5) # back to multiprocessing, but with the max_workers specified From pje at telecommunity.com Sun Mar 7 04:56:57 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sat, 06 Mar 2010 22:56:57 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> Message-ID: <20100307035657.111563A4077@sparrow.telecommunity.com> At 01:10 PM 3/7/2010 +1100, Brian Quinlan wrote: >On 7 Mar 2010, at 03:04, Phillip J. Eby wrote: > >>At 05:32 AM 3/6/2010, Brian Quinlan wrote: >>>Using twisted (or any other asynchronous I/O framework) forces you to >>>rewrite your I/O code. Futures do not. >> >>Twisted's "Deferred" API has nothing to do with I/O. > >I see, you just mean the API and not the underlying model. > >We discussed the Deferred API on the stdlib-sig and I don't think that >anyone expressed a preference for it over the one described in the PEP. > >Do you have any concrete criticism? Of the PEP, yes, absolutely, and I've already stated much of it. My quibbles are with the PEP *itself*, not so much the API or implementation. I think that said API and implementation is fine, but FAR too narrowly scoped to claim to be "futures" or "execute computations asynchronously", as the PEP calls it. It's really just a nice task queuing system. Now, if the PEP were *scoped* as such, i.e., "hey, let's just have a nice multithread/multiprocess task queuing implementation in the stdlib", I would be +1. It's a handy utility to have. But I think that the scope given by the PEP appears overly ambitious compared to what is actually being delivered; this seems less of a "futures API" and more like a couple of utility functions for waiting on threads and processes. To rise to the level of an API, it seems to me that it would need to address interop with coroutines and async frameworks, where the idea of "futures" seems much more relevant than simple synchronous-but-parallel scripts. (It should also have better tools for working with futures asynchronously, because, hey, it says right there in the title, "execute computations asynchronously".) Anyway, I'd like to see the answers to (at *least*) the following issues fleshed out in the PEP, if you want it to really be a "futures API", vs. "nice task queue in the stdlib": * Compare/contrast alternatives now available * Address the issue of competing event loops and sharing/passing executors among code * Either offer a way for executed code to re-enter its own executor (e.g. via an optional parameter), or explain why this was considered and rejected * Address interoperability with coroutines and async frameworks, or clearly explain why such is out of scope (Personally, I think it would be better to just drop the ambitious title and scope, and go for the "nice task queue" scope. I imagine, too, that in that case Jean-Paul wouldn't need to worry about it being raised as a future objection to Deferreds or some such getting into the stdlib.) From ncoghlan at gmail.com Sun Mar 7 05:43:15 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 07 Mar 2010 14:43:15 +1000 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4B92E30E.9070400@canterbury.ac.nz> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100305175404.7daeca3e@msiwind> <5d44f72f1003051800n70e57782i7589ad584db5ee9f@mail.gmail.com> <4B92E30E.9070400@canterbury.ac.nz> Message-ID: <4B932EE3.80908@gmail.com> Greg Ewing wrote: > The thing to consider, I think, is whether it makes sense in > a large proportion of use cases to ignore the fact that the > cancel failed and carry on regardless. If not, then raising an > exception makes it much harder to accidentally ignore the > situation. Cancelling operations is generally a best effort activity - having to wrap it an exception handler would be annoying. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sun Mar 7 05:49:02 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 07 Mar 2010 14:49:02 +1000 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100307035657.111563A4077@sparrow.telecommunity.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> Message-ID: <4B93303E.9050301@gmail.com> P.J. Eby wrote: > (Personally, I think it would be better to just drop the ambitious title > and scope, and go for the "nice task queue" scope. I imagine, too, that > in that case Jean-Paul wouldn't need to worry about it being raised as a > future objection to Deferreds or some such getting into the stdlib.) This may be a terminology thing - to me futures *are* just a nice way to handle farming tasks out to worker threads or processes. You seem to see them as something more comprehensive than that. I agree the PEP should just target what the current implementation provides and put whatever scope limitations are needed in the preamble text to make that clear. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From g.brandl at gmx.net Sun Mar 7 10:10:08 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 07 Mar 2010 10:10:08 +0100 Subject: [Python-Dev] argparse ugliness In-Reply-To: <20100305212844.6edbb0b7@msiwind> References: <20100305212844.6edbb0b7@msiwind> Message-ID: Am 06.03.2010 03:28, schrieb Antoine Pitrou: > Le Fri, 05 Mar 2010 13:51:15 -0500, > Neal Becker a ?crit : >> I generally enjoy argparse, but one thing I find rather >> ugly and unpythonic. >> >> parser.add_argument ('--plot', action='store_true') > > I would argue that a string is actually more Pythonic than > integers or anonymous objects repurposed as magic constants. > (I'm looking at things such as SEEK_SET and friends) Me too. In absence of a separate symbol type, Python itself uses strings for this purpose (hence the intern()ing business). Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From stefan_ml at behnel.de Sun Mar 7 11:10:52 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 07 Mar 2010 11:10:52 +0100 Subject: [Python-Dev] [RELEASED] Python 3.1.2 release candidate In-Reply-To: <1afaf6161003061413k646e972eta515d9322793e208@mail.gmail.com> References: <1afaf6161003061413k646e972eta515d9322793e208@mail.gmail.com> Message-ID: Benjamin Peterson, 06.03.2010 23:13: > A list of changes in 3.1.2rc1 can be found here: > > http://svn.python.org/projects/python/tags/r312rc1/Misc/NEWS Any reason the fix for bug 7173 didn't make it in? The test case that Amaury came up with seems to indicate that this a pretty serious issue. http://bugs.python.org/issue7173 Stefan From ndbecker2 at gmail.com Sun Mar 7 13:29:25 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Sun, 07 Mar 2010 07:29:25 -0500 Subject: [Python-Dev] argparse ugliness References: Message-ID: Brian Curtin wrote: > On Fri, Mar 5, 2010 at 12:51, Neal Becker wrote: > >> I generally enjoy argparse, but one thing I find rather >> ugly and unpythonic. >> >> parser.add_argument ('--plot', action='store_true') >> >> Specifying the argument 'action' as a string is IMO ugly. >> > > What else would you propose? > FWIW, this is the same in optparse. I would have thought use the object itself, instead of a string that spells the object's name. From jnoller at gmail.com Sun Mar 7 14:50:03 2010 From: jnoller at gmail.com (Jesse Noller) Date: Sun, 7 Mar 2010 08:50:03 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> Message-ID: <4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> On Sat, Mar 6, 2010 at 10:09 PM, Dj Gilcrease wrote: > After playing with the API for a while & running into many issues with > the examples & tests crashing windows I decided to modify the API a > little and fix up the examples so they dont crash windows based > computers. > > http://code.google.com/p/pythonfutures/issues/detail?id=1 > > API Change that changes the current Executor to ExecutorBase and > adds a new Executor class that is used like > > futures.Executor() # creates an executor that uses threading and a > max_workers = to the number of cpus > > futures.Executor(use='process') # Creates an executor that uses > multiprocessing and a max_workers = to the number of cpus > > futures.Executor(max_workers=5) # threading again, just specifying the > number of workers > > futures.Executor(use='process', max_workers=5) # back to multiprocessing, > but with the max_workers specified Making the tests and examples happy on windows is fine; but some explanation is needed for the API changes. From solipsis at pitrou.net Sun Mar 7 15:54:21 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 7 Mar 2010 09:54:21 -0500 Subject: [Python-Dev] Silencing DeprecationWarning by default in Python 2.7 is silencing division warnings References: Message-ID: <20100307095421.6cf6c596@msiwind> Le Sat, 6 Mar 2010 11:43:20 -0800, Brett Cannon a ?crit : > I see two possible fixes for this. One is to not silence > DeprecationWarning if Py_DivisionWarningFlag is set to >= 1. The > other is to introduce a new subclass of DeprecationWarning called > IntegerDivisionWarning and have that added to the warnings filter so > that if it is triggered it is handled separately from what > DeprecationWarning triggers. The latter would have my preference. Being able to discriminate between various classes of deprecations is good. Especially when what you want is to test for them :) From solipsis at pitrou.net Sun Mar 7 15:57:16 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 7 Mar 2010 09:57:16 -0500 Subject: [Python-Dev] [RELEASED] Python 3.1.2 release candidate References: <1afaf6161003061413k646e972eta515d9322793e208@mail.gmail.com> Message-ID: <20100307095716.53afe4cb@msiwind> Le Sun, 07 Mar 2010 11:10:52 +0100, Stefan Behnel a ?crit : > > Any reason the fix for bug 7173 didn't make it in? The test case that > Amaury came up with seems to indicate that this a pretty serious > issue. The patch needs to properly include an unit test reproducing the test case. Also, a committer must step up and apply it. I am currently on vacation and refrain from doing commits myself. Regards Antoine. From pje at telecommunity.com Sun Mar 7 16:48:09 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 07 Mar 2010 10:48:09 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4B93303E.9050301@gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> Message-ID: <20100307154809.742E73A4062@sparrow.telecommunity.com> At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote: >P.J. Eby wrote: > > (Personally, I think it would be better to just drop the ambitious title > > and scope, and go for the "nice task queue" scope. I imagine, too, that > > in that case Jean-Paul wouldn't need to worry about it being raised as a > > future objection to Deferreds or some such getting into the stdlib.) > >This may be a terminology thing - to me futures *are* just a nice way to >handle farming tasks out to worker threads or processes. You seem to see >them as something more comprehensive than that. Actual futures are, yes. Specifically, futures are a mechanism for asynchronous computation, whereas the PEP seems to be all about synchronously managing parallel tasks. That's a huge difference. Technically, the things in the PEP (and by extension, Java's futures) match the letter of the definition of a future, but not (IMO) the spirit. There's no clean way to compose them, and at base they're more about parallelism than asynchrony. >I agree the PEP should just target what the current implementation >provides and put whatever scope limitations are needed in the preamble >text to make that clear. Yep. I'm just saying "parallel task queueing" is a much better description of what the implementation is/does, and would suggest renaming Future -> Task and Executor -> WorkerPool or some such. These names would be *much* clearer to people who've never heard of futures, as well as more appropriate to the actual scope of what this does. From jyasskin at gmail.com Sun Mar 7 17:39:55 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Sun, 7 Mar 2010 08:39:55 -0800 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100307154809.742E73A4062@sparrow.telecommunity.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> <20100307154809.742E73A4062@sparrow.telecommunity.com> Message-ID: <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com> On Sun, Mar 7, 2010 at 7:48 AM, P.J. Eby wrote: > At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote: >> >> P.J. Eby wrote: >> > (Personally, I think it would be better to just drop the ambitious title >> > and scope, and go for the "nice task queue" scope. ?I imagine, too, that >> > in that case Jean-Paul wouldn't need to worry about it being raised as a >> > future objection to Deferreds or some such getting into the stdlib.) >> >> This may be a terminology thing - to me futures *are* just a nice way to >> handle farming tasks out to worker threads or processes. You seem to see >> them as something more comprehensive than that. > > Actual futures are, yes. ?Specifically, futures are a mechanism for > asynchronous computation, whereas the PEP seems to be all about > synchronously managing parallel tasks. ?That's a huge difference. > > Technically, the things in the PEP (and by extension, Java's futures) match > the letter of the definition of a future, but not (IMO) the spirit. ?There's > no clean way to compose them, and at base they're more about parallelism > than asynchrony. Do you have an example of a language or library that uses the term "future" to refer to what you're talking about? I'm curious to see what it looks like. From millbr02 at luther.edu Sun Mar 7 18:36:17 2010 From: millbr02 at luther.edu (Brad Miller) Date: Sun, 7 Mar 2010 11:36:17 -0600 Subject: [Python-Dev] [RELEASED] Python 3.1.2 release candidate In-Reply-To: <20100307095716.53afe4cb@msiwind> References: <1afaf6161003061413k646e972eta515d9322793e208@mail.gmail.com> <20100307095716.53afe4cb@msiwind> Message-ID: I download the source snapshot ./configure --enable-framework make ... /usr/bin/install -c -d -m 755 Python.framework/Versions/3.1 if test ""; then \ gcc -o Python.framework/Versions/3.1/Python -dynamiclib \ -isysroot "" \ -all_load libpython3.1.a -Wl,-single_module \ -install_name /Library/Frameworks/Python.framework/Versions/3.1/Python \ -compatibility_version 3.1 \ -current_version 3.1 \ -framework CoreFoundation -ldl -framework CoreFoundation; \ else \ /usr/bin/libtool -o Python.framework/Versions/3.1/Python -dynamic libpython3.1.a \ -lSystem -lSystemStubs -arch_only x86_64 -install_name /Library/Frameworks/Python.framework/Versions/3.1/Python -compatibility_version 3.1 -current_version 3.1 -framework CoreFoundation -ldl -framework CoreFoundation;\ fi Undefined symbols: "_libintl_bindtextdomain", referenced from: _PyIntl_bindtextdomain in libpython3.1.a(_localemodule.o) "_libintl_textdomain", referenced from: _PyIntl_textdomain in libpython3.1.a(_localemodule.o) "_libintl_dgettext", referenced from: _PyIntl_dgettext in libpython3.1.a(_localemodule.o) "_libintl_gettext", referenced from: _PyIntl_gettext in libpython3.1.a(_localemodule.o) "_libintl_dcgettext", referenced from: _PyIntl_dcgettext in libpython3.1.a(_localemodule.o) ld: symbol(s) not found /usr/bin/libtool: internal link edit command failed I'm on 10.6.2, gcc version 4.2.1 Any hints on where to go from here? On Sun, Mar 7, 2010 at 8:57 AM, Antoine Pitrou wrote: > Le Sun, 07 Mar 2010 11:10:52 +0100, > Stefan Behnel a ?crit : > > > > Any reason the fix for bug 7173 didn't make it in? The test case that > > Amaury came up with seems to indicate that this a pretty serious > > issue. > > The patch needs to properly include an unit test reproducing the test > case. > Also, a committer must step up and apply it. I am currently on vacation > and refrain from doing commits myself. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/bmiller%40luther.edu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Sun Mar 7 18:57:31 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 07 Mar 2010 12:57:31 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.co m> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> <20100307154809.742E73A4062@sparrow.telecommunity.com> <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com> Message-ID: <20100307175732.307513A4062@sparrow.telecommunity.com> At 08:39 AM 3/7/2010 -0800, Jeffrey Yasskin wrote: >On Sun, Mar 7, 2010 at 7:48 AM, P.J. Eby wrote: > > At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote: > >> > >> P.J. Eby wrote: > >> > (Personally, I think it would be better to just drop the ambitious title > >> > and scope, and go for the "nice task queue" scope. ? I > imagine, too, that > >> > in that case Jean-Paul wouldn't need to worry about it being raised as a > >> > future objection to Deferreds or some such getting into the stdlib.) > >> > >> This may be a terminology thing - to me futures *are* just a nice way to > >> handle farming tasks out to worker threads or processes. You seem to see > >> them as something more comprehensive than that. > > > > Actual futures are, yes. ? Specifically, futures are a mechanism for > > asynchronous computation, whereas the PEP seems to be all about > > synchronously managing parallel tasks. ? That's a huge difference. > > > > Technically, the things in the PEP (and by extension, Java's futures) match > > the letter of the definition of a future, but not (IMO) the > spirit. ? There's > > no clean way to compose them, and at base they're more about parallelism > > than asynchrony. > >Do you have an example of a language or library that uses the term >"future" to refer to what you're talking about? I'm curious to see >what it looks like. The wikipedia page menetioned earlier in the discussion has several examples. Twisted's Deferreds are probably the best example of such a system in Python, though, as far as I know. The features proposed in the PEP are basically akin to Twisted's deferToThread(), i.e. "return a future for running this code in some other thread". However, you could easily make your own thread or process pool for executing tasks, using Deferreds as your "future" objects; deferToThread() is not particularly special, it's just one possible "executor", to use the PEP's terminology. Likewise, deferreds are not limited to representing parallel task execution - they can also represent asynchronous events such as incoming I/O or a waited-for mouse click. Deferreds allow composing calculation by stacking callbacks (in continuation-passing style), but they're most Pythonic (IMO) when used with a coroutine wrapping system like Eventlet or my earlier peak.events framework, so that you can use them in "yield" expressions. That way you can write synchronous-looking code that nonetheless executes asynchronously. Twisted uses Deferreds extensively, both for I/O and IPC, and of course for database access. However, the 'defer' module itself is conceptually distinct from both the I/O system, event loop, and threads: it's a largely-generic framework for managing asynchronous computation. Parallel task queueing is simply *one* possible application of Deferreds; their actual purpose is to provide a common API for working with values that aren't yet known. That is to say, futures. In contrast, the PEP manages only a very limited sort of "future" - the completion of a single task, which must be synchronously waited for. If one "future" in the PEP needs to wait for another, you tie up that thread or process in a busy loop... which rather limits how much parallelism you can have. Deferreds also allow you to write code that doesn't *care* whether the value has been determined or not... you can simply stack on what code you want to have run when the value is available, and it runs immediately if the value is there. Likewise, a deferred is not tied to the execution of a single piece of code - it's more of a rendezvous or synchronization point, that can be given a value to "fire" with. (In contrast, the PEP's "futures" are tightly coupled to a task execution model: they have a notion of running and being cancelled, for example.) For these reasons, I'd say that Deferreds are much more deserving of the title "futures", especially since there is no need to call a simple task queue a "futures" system. (Unless you're Java, I suppose, in which case you might want the name for marketing reasons. ;-) But in Python I assume we'd like to call a spade a spade, and a task queue a task queue.) From rdmurray at bitdance.com Sun Mar 7 19:45:52 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 07 Mar 2010 13:45:52 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100307154809.742E73A4062@sparrow.telecommunity.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> <20100307154809.742E73A4062@sparrow.telecommunity.com> Message-ID: <20100307184552.BF15E1AB54F@kimball.webabinitio.net> On Sun, 07 Mar 2010 10:48:09 -0500, "P.J. Eby" wrote: > At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote: > >I agree the PEP should just target what the current implementation > >provides and put whatever scope limitations are needed in the preamble > >text to make that clear. > > Yep. I'm just saying "parallel task queueing" is a much better > description of what the implementation is/does, and would suggest > renaming Future -> Task and Executor -> WorkerPool or some > such. These names would be *much* clearer to people who've never > heard of futures, as well as more appropriate to the actual scope of > what this does. For what it's worth: I don't have any particular knowledge in this area. I did loosely follow the stdlib-sig discussion. I wasn't really sure exactly what the module was about or what a 'future' was, or why I would want to use one. I did get that it was about parallel execution of tasks, but it seemed like there had to be more to it than that. Hearing it called a 'worker pool' makes a lightbulb go off and I can now understand why this would be a useful facility to have in the standard library. -- R. David Murray www.bitdance.com From jyasskin at gmail.com Sun Mar 7 19:59:50 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Sun, 7 Mar 2010 10:59:50 -0800 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100307175732.307513A4062@sparrow.telecommunity.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> <20100307154809.742E73A4062@sparrow.telecommunity.com> <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com> <20100307175732.307513A4062@sparrow.telecommunity.com> Message-ID: <5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com> On Sun, Mar 7, 2010 at 9:57 AM, P.J. Eby wrote: > At 08:39 AM 3/7/2010 -0800, Jeffrey Yasskin wrote: >> Do you have an example of a language or library that uses the term >> "future" to refer to what you're talking about? I'm curious to see >> what it looks like. > > The wikipedia page menetioned earlier in the discussion has several > examples. ?Twisted's Deferreds are probably the best example of such a > system in Python, though, as far as I know. ?The features proposed in the > PEP are basically akin to Twisted's deferToThread(), i.e. "return a future > for running this code in some other thread". Nearly all of the wikipedia page's examples refer to exactly this kind of system, but with syntax to support it instead of putting it in a library. The distinction between futures and deferreds is that futures are designed to block, while deferreds are designed to continue through callbacks. We could say that futures are to deferreds as threading.Lock is to mutex.mutex. >?That way you > can write synchronous-looking code that nonetheless executes asynchronously. "Asynchronous" in the PEP refers to how the task behind the future runs, not how users wait for the future. Is there a better way to say that? > Twisted uses Deferreds extensively, both for I/O and IPC, and of course for > database access. ?However, the 'defer' module itself is conceptually > distinct from both the I/O system, event loop, and threads: it's a > largely-generic framework for managing asynchronous computation. ?Parallel > task queueing is simply *one* possible application of Deferreds; their > actual purpose is to provide a common API for working with values that > aren't yet known. > > That is to say, futures. > > In contrast, the PEP manages only a very limited sort of "future" - the > completion of a single task, which must be synchronously waited for. ?If one > "future" in the PEP needs to wait for another, you tie up that thread or > process in a busy loop... ?which rather limits how much parallelism you can > have. So is it that you just don't like the idea of blocking, and want to stop anything that relies on it from getting into the standard library? Given the set_result and set_exception methods, it's pretty straightforward to fill in the value of a future from something that isn't purely computational. Given a way to register "on-done" callbacks with the future, it would be straightforward to wait for a future without blocking, too. Jeffrey From guido at python.org Sun Mar 7 20:49:29 2010 From: guido at python.org (Guido van Rossum) Date: Sun, 7 Mar 2010 11:49:29 -0800 Subject: [Python-Dev] argparse ugliness In-Reply-To: References: Message-ID: On Sun, Mar 7, 2010 at 4:29 AM, Neal Becker wrote: > Brian Curtin wrote: > >> On Fri, Mar 5, 2010 at 12:51, Neal Becker wrote: >> >>> I generally enjoy argparse, but one thing I find rather >>> ugly and unpythonic. >>> >>> ? ?parser.add_argument ('--plot', action='store_true') >>> >>> Specifying the argument 'action' as a string is IMO ugly. >>> >> >> What else would you propose? >> FWIW, this is the same in optparse. > > I would have thought use the object itself, instead of a string that spells > the object's name. What object? How would you write the example instead then? -- --Guido van Rossum (python.org/~guido) From pje at telecommunity.com Sun Mar 7 20:56:13 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 07 Mar 2010 14:56:13 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.co m> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> <20100307154809.742E73A4062@sparrow.telecommunity.com> <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com> <20100307175732.307513A4062@sparrow.telecommunity.com> <5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com> Message-ID: <20100307195614.BBD693A4062@sparrow.telecommunity.com> At 10:59 AM 3/7/2010 -0800, Jeffrey Yasskin wrote: >So is it that you just don't like the idea of blocking, and want to >stop anything that relies on it from getting into the standard library? Um, no. As I said before, call it a "parallel task queue" or "parallel task manager" or something to that general effect and I'm on board. It may not be in the Zen of Python, but ISTM that names should generally follow use cases. It is something of a corollary to "one obvious way to do it", in that if you see something whose name matches what you want to do, then it should be obvious that that's the way in question. ;-) The use cases for "parallel task queues", however, are a subset of those for "futures" in the general case. Since the proposed module addresses most of the former but very little of the latter, calling it futures is inappropriate. Specifically, it's: 1. Confusing to people who don't know what futures are (see e.g R.D. Murray's post), and 2. Underpowered for people who expect/want a more fully-featured futures system along the lines of E or Deferreds. It seems that the only people for whom it's an intuitively correct description are people who've only had experience with more limited futures models (like Java's). However, these people should not have a problem understanding the notion of parallel task queueing or task management, so changing the name isn't really a loss for them, and it's a gain for everybody else. > Given the set_result and set_exception methods, it's pretty > straightforward to fill in the value of a future from something > that isn't purely computational. Those are described as "internal" methods in the PEP; by contrast, the Deferred equivalents are part of the public API. > Given a way to register "on-done" callbacks with the future, it > would be straightforward to wait for a future without blocking, too. Yes, and with a few more additions besides that one, you might be on the way to an actual competitor for Deferreds. For example: retry support, chaining, logging, API for transparent result processing, coroutine support, co-ordination tools like locks, sempaphores and queues, etc. These are all things you would very likely want or need if you actually wanted to write a program using futures as *your main computational model*, vs. just needing to toss out some parallel tasks in a primarily synchronous program. Of course, Deferreds are indeed overkill if all you're ever going to want is a few parallel tasks, unless you're already skilled in using Twisted or some wrapper for it. So, I totally support having a simple task queue in the stdlib, as there are definitely times I would've used such a thing for a quick script, if it were available. However, I've *also* had use cases for using futures as a computational model, and so that's what I originally thought this PEP was about. After the use cases were clarified, though, it seems to me that *calling* it futures is a bad idea, because it's really just a nice task queuing system. I'm +1 on adding a nice task queuing system, -1 on calling it by any other name. ;-) From brett at python.org Sun Mar 7 21:35:02 2010 From: brett at python.org (Brett Cannon) Date: Sun, 7 Mar 2010 12:35:02 -0800 Subject: [Python-Dev] [RELEASED] Python 3.1.2 release candidate In-Reply-To: References: <1afaf6161003061413k646e972eta515d9322793e208@mail.gmail.com> <20100307095716.53afe4cb@msiwind> Message-ID: Do you have MacPorts or Fink installed? If you do then you have a version of gettext installed that is not compiled properly to work with Python (usually its a 32-bit/64-bit thing). Best piece of advice is to not use MacPorts/Fink and use Homebrew instead. =) On Sun, Mar 7, 2010 at 09:36, Brad Miller wrote: > > I download the source snapshot > > ./configure --enable-framework > make > > ... > > /usr/bin/install -c -d -m 755 Python.framework/Versions/3.1 > if test ""; then \ > gcc -o Python.framework/Versions/3.1/Python -dynamiclib \ > -isysroot "" \ > -all_load libpython3.1.a -Wl,-single_module \ > -install_name > /Library/Frameworks/Python.framework/Versions/3.1/Python \ > -compatibility_version 3.1 \ > -current_version 3.1 \ > -framework CoreFoundation -ldl -framework > CoreFoundation; \ > else \ > /usr/bin/libtool -o Python.framework/Versions/3.1/Python > -dynamic libpython3.1.a \ > -lSystem -lSystemStubs -arch_only x86_64 > -install_name /Library/Frameworks/Python.framework/Versions/3.1/Python > -compatibility_version 3.1 -current_version 3.1 -framework CoreFoundation > -ldl -framework CoreFoundation;\ > fi > > Undefined symbols: > "_libintl_bindtextdomain", referenced from: > _PyIntl_bindtextdomain in libpython3.1.a(_localemodule.o) > "_libintl_textdomain", referenced from: > _PyIntl_textdomain in libpython3.1.a(_localemodule.o) > "_libintl_dgettext", referenced from: > _PyIntl_dgettext in libpython3.1.a(_localemodule.o) > "_libintl_gettext", referenced from: > _PyIntl_gettext in libpython3.1.a(_localemodule.o) > "_libintl_dcgettext", referenced from: > _PyIntl_dcgettext in libpython3.1.a(_localemodule.o) > ld: symbol(s) not found > /usr/bin/libtool: internal link edit command failed > > I'm on 10.6.2, gcc version 4.2.1 > > Any hints on where to go from here? > > > On Sun, Mar 7, 2010 at 8:57 AM, Antoine Pitrou wrote: > >> Le Sun, 07 Mar 2010 11:10:52 +0100, >> Stefan Behnel a ?crit : >> > >> > Any reason the fix for bug 7173 didn't make it in? The test case that >> > Amaury came up with seems to indicate that this a pretty serious >> > issue. >> >> The patch needs to properly include an unit test reproducing the test >> case. >> Also, a committer must step up and apply it. I am currently on vacation >> and refrain from doing commits myself. >> >> Regards >> >> Antoine. >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/bmiller%40luther.edu >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Sun Mar 7 21:56:12 2010 From: barry at python.org (Barry Warsaw) Date: Sun, 7 Mar 2010 15:56:12 -0500 Subject: [Python-Dev] Python 2.6.5 rc 2 Message-ID: <20100307155612.59d9db94@heresy.wooz.org> We are definitely going to need a 2.6.5 rc 2. We've had a number of critical issues fixed since rc1 and we have two more critical patches approved for landing which fix issues on OS X. I've given Ronald approval in the tracker to land these two patches. 8066 OS X installer: readline module breaks when targeting on 10.5 or 10.6 8067 OS X Installer: build errors on 10.6 when targeting 10.4 and earlier These are the only two release blockers left for 2.6.5 (at the moment). I would like to cut the rc2 release tomorrow, Monday 2010-03-08, so if anybody knows of any other bugs you think must go into 2.6.5, mark them release blockers now so that I can evaluate them. I really don't want there to be an rc3, and I'd still like to get 2.6.5 final out on Monday 2010-03-15. We'll see. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From digitalxero at gmail.com Sun Mar 7 22:44:31 2010 From: digitalxero at gmail.com (Dj Gilcrease) Date: Sun, 7 Mar 2010 14:44:31 -0700 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> Message-ID: On Sun, Mar 7, 2010 at 6:50 AM, Jesse Noller wrote: > Making the tests and examples happy on windows is fine; but some > explanation is needed for the API changes. > My primary motivation behind the API change is so there is just a single public Executor class that you tell what system to use instead of two separate classes. The use case I was thinking about is when a user is is unsure which system (threads or processes) they want to use so they just build the system with the defaults (which is threads) then it is a little easier to switch it to processes in the future instead of having to change imports and all instances of the class you just change the use keyword to switch between systems From guido at python.org Mon Mar 8 01:47:50 2010 From: guido at python.org (Guido van Rossum) Date: Sun, 7 Mar 2010 16:47:50 -0800 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100307195614.BBD693A4062@sparrow.telecommunity.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> <20100307154809.742E73A4062@sparrow.telecommunity.com> <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com> <20100307175732.307513A4062@sparrow.telecommunity.com> <5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com> <20100307195614.BBD693A4062@sparrow.telecommunity.com> Message-ID: On Sun, Mar 7, 2010 at 11:56 AM, P.J. Eby wrote: > At 10:59 AM 3/7/2010 -0800, Jeffrey Yasskin wrote: >> >> So is it that you just don't like the idea of blocking, and want to stop >> anything that relies on it from getting into the standard library? > > Um, no. ?As I said before, call it a "parallel task queue" or "parallel task > manager" or something to that general effect and I'm on board. > > It may not be in the Zen of Python, but ISTM that names should generally > follow use cases. ?It is something of a corollary to "one obvious way to do > it", in that if you see something whose name matches what you want to do, > then it should be obvious that that's the way in question. ?;-) > > The use cases for "parallel task queues", however, are a subset of those for > "futures" in the general case. ?Since the proposed module addresses most of > the former but very little of the latter, calling it futures is > inappropriate. > > Specifically, it's: > > 1. Confusing to people who don't know what futures are (see e.g R.D. > Murray's post), and This is not a problem. We will document what we consider a future. > 2. Underpowered for people who expect/want a more fully-featured futures > system along the lines of E or Deferreds. This sounds like an underhanded slur towards the PEP. > It seems that the only people for whom it's an intuitively correct > description are people who've only had experience with more limited futures > models (like Java's). ?However, these people should not have a problem > understanding the notion of parallel task queueing or task management, so > changing the name isn't really a loss for them, and it's a gain for > everybody else. I expect that the majority of Python users fall either in camp #1 (never heard of futures, will be happy to learn about what Python calls futures) or camp #3 (have used Java futures). The users of E can be counted on a few hands. Deferreds are used heavily in some Python circles but most Python users (myself included) have at most a very vague idea of them. Also, as you clarify below, Deferreds are so much more powerful that they can't possibly be mistaken for futures (as defined by this PEP). Plus they already have a name. >> ?Given the set_result and set_exception methods, it's pretty >> straightforward to fill in the value of a future from something that isn't >> purely computational. > > Those are described as "internal" methods in the PEP; by contrast, the > Deferred equivalents are part of the public API. > > >> ?Given a way to register "on-done" callbacks with the future, it would be >> straightforward to wait for a future without blocking, too. > > Yes, and with a few more additions besides that one, you might be on the way > to an actual competitor for Deferreds. ?For example: retry support, > chaining, logging, API for transparent result processing, coroutine support, > co-ordination tools like locks, sempaphores and queues, etc. > > These are all things you would very likely want or need if you actually > wanted to write a program using futures as *your main computational model*, > vs. just needing to toss out some parallel tasks in a primarily synchronous > program. > > Of course, Deferreds are indeed overkill if all you're ever going to want is > a few parallel tasks, unless you're already skilled in using Twisted or some > wrapper for it. > > So, I totally support having a simple task queue in the stdlib, as there are > definitely times I would've used such a thing for a quick script, if it were > available. > > However, I've *also* had use cases for using futures as a computational > model, and so that's what I originally thought this PEP was about. ?After > the use cases were clarified, though, it seems to me that *calling* it > futures is a bad idea, because it's really just a nice task queuing system. > > I'm +1 on adding a nice task queuing system, -1 on calling it by any other > name. ?;-) So let's focus on the functionality of the task queuing system, and stick to roughly the functionality proposed in the PEP. The name is a non-issue and further discussion ought to be sent to null-dev at python.org. -- --Guido van Rossum (python.org/~guido) From fuzzyman at voidspace.org.uk Mon Mar 8 01:51:02 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 08 Mar 2010 00:51:02 +0000 Subject: [Python-Dev] unittest2 plus class and module level fixtures in unittest Message-ID: <4B9449F6.9080206@voidspace.org.uk> Hello all, unittest has seen quite a few new features since Python 2.6. For those of you who might have missed the announcements in other places, unittest2 is a backport of the new features in unittest to work with Python 2.4-2.6. It is already being used for the development of distutils2: http://pypi.python.org/pypi/unittest2/ In other news, class and module fixtures (setUpClass / tearDownClass / setUpModule / tearDownModule) are now implemented in unittest (in trunk - not yet merged to py3k). These features are tested but I'm sure there are some lurking bugs or oddities, so please try them out. I have not yet added documentation for them; I'll pull it out from this email as a starting point. I'd rather this thread didn't become *another* debate on the merit of these features, but perhaps that is too much to hope for. Below are some notes on how class and module fixtures work. Class and module level fixtures are implemented in unittest.TestSuite. When the test suite encounters a test from a new class then tearDownClass from the previous class (if there is one) is called, followed by setUpClass from the new class. Similarly if a test is from a different module from the previous test then tearDownModule from the previous module is run, followed by setUpModule from the new module. After all the tests have run the final tearDownClass and tearDownModule are run. Note that shared fixtures do not play well with [potential] features like test parallelization and they break test isolation. They should be used with care. setUpClass and tearDownClass -------------------------------------- These must be implemented as class methods. import unittest class Test(unittest.TestCase): @classmethod def setUpClass(cls): cls._connection = createExpensiveConnectionObject() @classmethod def tearDownClass(cls): cls._connection.destroy() If you want the setUpClass and tearDownClass on base classes called then you must call up to them yourself. The implementations in unittest2.TestCase are empty. If an exception is raised during a setUpClass then the tests in the class are not run and the tearDownClass is not run. Skipped classes will not have setUpClass or tearDownClass run. A setUpClass that raises a unittest2.SkipTest exception will currently be reported as an error rather than a skip (although the effect is the same). This will be fixed at some point in the future. setUpModule and tearDownModule -------------------------------------------- These should be implemented as functions. def setUpModule(): createConnection() def tearDownModule(): closeConnection() If an exception is raised in a setUpModule then none of the tests in the module will be run and the tearDownModule will not be run. A setUpModule that raises a unittest.SkipTest exception will currently be reported as an error rather than a skip (although the effect is the same). This will be fixed at some point in the future. The Gory Details ---------------------- The default ordering of tests created by the unittest test loaders is to group all tests from the same modules and classes together. This will lead to setUpClass / setUpModule (etc) being called exactly once per class and module. If you randomize the order, so that tests from different modules and classes are adjacent to each other, then these shared fixture functions may be called multiple times in a single test run. This particular issue was a point of some disussion. Shared fixtures are not intended to work with suites with non-standard (incompatible) ordering (a BaseTestSuite still exists for frameworks that don't want to support shared fixtures), so this is already a pathological situation. There are a couple of things unittest could do in this situation: 1) Work out the best order to run the setUp and tearDowns, based on the position of the first and last test from each class / module in the suite. 2) Run setUp and tearDowns appropriately if consecutive tests are from different classes and modules. If your setUps do unrelated things then strategy 1 will be the best one, but (so long as your setups are repeatable) 2 will be fine just slower. If your setups do related things, for example pushing large test datasets into the same database, then having multiple setups active before the corresponding teardowns have been run will actually do the wrong thing and your tests will be broken. In addition strategy 1 relies on unrolling the test suite and introspecting all tests prior to the run, not all custom test suites implementations support this. Given that strategy 2 was substantially easier to implement, and less likely to cause broken test runs, that was what I went with. It is of course possible to add alternative strategies, plus the metadata to support them, if there proves to be a substantial need for this. One of the major use cases is for test order randomization. It would be easy to add a 'randomize' method to the suite that randomizes the order of test modules in a suite, the order of test classes within modules and the order of tests within a class. This would allow for tests to be run in a random order whilst remaining compatible with shared fixtures. If there are any exceptions raised during one of the shared fixture functions the test is reported as an error. Because there is no corresponding test instance an _ErrorHolder object (that has the same interface as a TestCase) is created to represent the error. If you are just using the standard unittest test runner then this detail doesn't matter, but if you are a framework author it may be relevant. I have a few minor features I'd like to add to unittest before 2.7 beta - standard out capturing, fail fast and ctrl-c handling (first ctrl-c will wait for current test to exit, second will force an exit - results will still be reported). After that the *really* big problem for unittest is the extensibility story. I'll be working on extensibility points for unittest that allow people to implement extensions without having to override functionality. This will take a fair bit of discussion and experimentation, so I will probably trial this in unittest2 first. It would be nice to have something in place for the release of 3.2, but that may be ambitious. All the best, Michael Foord -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Mar 8 02:11:43 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 7 Mar 2010 20:11:43 -0500 Subject: [Python-Dev] unittest2 plus class and module level fixtures in unittest References: <4B9449F6.9080206@voidspace.org.uk> Message-ID: <20100307201143.48f0fd3e@msiwind> Le Mon, 08 Mar 2010 00:51:02 +0000, Michael Foord a ?crit : > > In other news, class and module fixtures (setUpClass / > tearDownClass / setUpModule / tearDownModule) are now implemented in > unittest (in trunk > - not yet merged to py3k). These features are tested but I'm sure > there are some lurking bugs or oddities, so please try them out. I > have not yet added documentation for them; I'll pull it out from this > email as a starting point. > > I'd rather this thread didn't become *another* debate on the merit of > these features, but perhaps that is too much to hope for. Just for the record, could you sum up the reasons why you/we decided on this route rather than e.g. adding a test-resources-like facility? (I'm not saying I disagree, it's just that I didn't remember a decision had already been made on this point) Thanks for all your work on this! cheers Antoine. From markroddy at gmail.com Mon Mar 8 05:04:15 2010 From: markroddy at gmail.com (Mark Roddy) Date: Sun, 7 Mar 2010 23:04:15 -0500 Subject: [Python-Dev] unittest2 plus class and module level fixtures in unittest In-Reply-To: <20100307201143.48f0fd3e@msiwind> References: <4B9449F6.9080206@voidspace.org.uk> <20100307201143.48f0fd3e@msiwind> Message-ID: <7930f1201003072004q25b57535q86e4e0c9cc04736c@mail.gmail.com> On Sun, Mar 7, 2010 at 8:11 PM, Antoine Pitrou wrote: > Le Mon, 08 Mar 2010 00:51:02 +0000, > Michael Foord a ?crit : >> >> In other news, class and module fixtures (setUpClass / >> tearDownClass / setUpModule / tearDownModule) are now implemented in >> unittest (in trunk >> - not yet merged to py3k). These features are tested but I'm sure >> there are some lurking bugs or oddities, so please try them out. I >> have not yet added documentation for them; I'll pull it out from this >> email as a starting point. >> >> I'd rather this thread didn't become *another* debate on the merit of >> these features, but perhaps that is too much to hope for. > > Just for the record, could you sum up the reasons why you/we decided on > this route rather than e.g. adding a test-resources-like facility? > > (I'm not saying I disagree, it's just that I didn't remember a decision > had already been made on this point) > > Thanks for all your work on this! > > cheers > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/markroddy%40gmail.com > There were some lengthy threads on python-ideas (http://mail.python.org/pipermail/python-ideas/2010-January/006736.html) and python-dev (http://mail.python.org/pipermail/python-dev/2010-February/097521.html). I'm sure Michael has a set of factors that are most important to him for the decision (which I don't blame him for not wanting to get into, the two previous threads were contentious), but for me personally I don't view class/module setup and test-resources as necessarily competing options. For the simplest use case ( create some expensive connection and/or object which is essentially stateless and won't need to be torn down), I personally find a test-resources solution to require an unnecessary amount of work to implement. After exploring it a bit in the course of those discussions I hope that test-resources (or something similar, maybe something like junit rules which was also brought up) makes it into the stdlib at some point as some of the more complicated integration/scenario tests I support would be made much simpler with such functionality. -Mark From mark.russell at zen.co.uk Mon Mar 8 11:25:16 2010 From: mark.russell at zen.co.uk (Mark Russell) Date: Mon, 8 Mar 2010 10:25:16 +0000 Subject: [Python-Dev] argparse ugliness In-Reply-To: References: Message-ID: <67850396-A793-4547-9532-CDF0F65E0977@zen.co.uk> On 7 Mar 2010, at 19:49, Guido van Rossum wrote: > How would you write the example instead then? Boolean flags are a common enough case that I'd be inclined to add a wrapper method, so you could just say: parser.add_bool_argument('--plot') As you can always fall back to the more general add_argument() call, the wrapper could just handle the usual case (default false, presence of option sets flag to True). So the signature would be pretty simple - something like: def add_bool_argument(self, help=None, dest=None): Mark Russell From ncoghlan at gmail.com Mon Mar 8 12:14:56 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 08 Mar 2010 21:14:56 +1000 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100307195614.BBD693A4062@sparrow.telecommunity.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> <20100307154809.742E73A4062@sparrow.telecommunity.com> <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com> <20100307175732.307513A4062@sparrow.telecommunity.com> <5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com> <20100307195614.BBD693A4062@sparrow.telecommunity.com> Message-ID: <4B94DC30.2030904@gmail.com> P.J. Eby wrote: > I'm +1 on adding a nice task queuing system, -1 on calling it by any > other name. ;-) As Guido said, let's call the nice task queuing system "futures" and point people wanting a full-power asynchronous process model to Twisted - while the Deferred API may technically be independent of the rest of the framework, you need at least some of the other tools for asynchronous operations to make it really shine. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Mon Mar 8 12:25:10 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 08 Mar 2010 21:25:10 +1000 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306061134.5AEE23A40A7@sparrow.telecommunity.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> Message-ID: <4B94DE96.8080100@gmail.com> Dj Gilcrease wrote: > On Sun, Mar 7, 2010 at 6:50 AM, Jesse Noller wrote: >> Making the tests and examples happy on windows is fine; but some >> explanation is needed for the API changes. >> > > My primary motivation behind the API change is so there is just a > single public Executor class that you tell what system to use instead > of two separate classes. The use case I was thinking about is when a > user is is unsure which system (threads or processes) they want to use > so they just build the system with the defaults (which is threads) > then it is a little easier to switch it to processes in the future > instead of having to change imports and all instances of the class you > just change the use keyword to switch between systems Wouldn't a factory function serve that purpose just as well? Or even just "from concurrent.futures import ProcessPoolExecutor as TaskExecutor". That last form has the virtue that you can retrieve your executor from anywhere rather than being limited to the two provided by the concurrent.futures model. I think the string based approach actually unduly constrains the API despite superficially appearing to make it more flexible. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From victor.stinner at haypocalc.com Mon Mar 8 14:12:51 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Mon, 8 Mar 2010 14:12:51 +0100 Subject: [Python-Dev] Catch SIGINT at Python startup Message-ID: <201003081412.51255.victor.stinner@haypocalc.com> Hi, I wrote patches for the issue #3137 and I would like your opinion about my solution (not directly about the patches): I propose to consider a SIGINT during startup as a fatal error. If you press CTRL+c while Python is starting, Python displays an error and exit with the code 1. Currently, if you press CTRL+c while Python is starting: the SIGINT signal will be *ignored* somewhere between Python initialization and the first Python instruction. First problem is that if site import is interrupted, Python is not completly initialized (if a SIGINT is sent just after the call to initsigs(), initsite() is completly skipped). Example: ------------------ $ python ^C Traceback (most recent call last): File "/home/SHARE/SVN/python-trunk/Lib/site.py", line 62, in import os File "/home/SHARE/SVN/python-trunk/Lib/os.py", line 44, in from posix import * KeyboardInterrupt $ echo $? 1 ------------------ It's not easy to reproduce this issue because Python initialization is really fast (yeah really! or at least to fast for me ;-)). I'm using "valgrind --log- file=/dev/null ./python -v" to simulate a very slow computer :-) -- Victor Stinner http://www.haypocalc.com/ From steven.bethard at gmail.com Mon Mar 8 16:40:36 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Mon, 8 Mar 2010 07:40:36 -0800 Subject: [Python-Dev] argparse ugliness In-Reply-To: References: Message-ID: On Sun, Mar 7, 2010 at 11:49 AM, Guido van Rossum wrote: > On Sun, Mar 7, 2010 at 4:29 AM, Neal Becker wrote: >> Brian Curtin wrote: >> >>> On Fri, Mar 5, 2010 at 12:51, Neal Becker wrote: >>> >>>> I generally enjoy argparse, but one thing I find rather >>>> ugly and unpythonic. >>>> >>>> ? ?parser.add_argument ('--plot', action='store_true') >>>> >>>> Specifying the argument 'action' as a string is IMO ugly. >>>> >>> >>> What else would you propose? >>> FWIW, this is the same in optparse. >> >> I would have thought use the object itself, instead of a string that spells >> the object's name. > > What object? How would you write the example instead then? In argparse, unlike optparse, actions are actually defined by objects with a particular API, and the string is just a shorthand for referring to that. So: parser.add_argument ('--plot', action='store_true') is equivalent to: parser.add_argument('--plot', argparse._StoreTrueAction) Because the names are so long and you'd have to import them, I've left them as private attributes of the module, but if there's really demand, we could rename them to argparse.StoreTrueAction, etc. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From steven.bethard at gmail.com Mon Mar 8 16:41:09 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Mon, 8 Mar 2010 07:41:09 -0800 Subject: [Python-Dev] argparse ugliness In-Reply-To: References: Message-ID: On Mon, Mar 8, 2010 at 7:40 AM, Steven Bethard wrote: > On Sun, Mar 7, 2010 at 11:49 AM, Guido van Rossum wrote: >> On Sun, Mar 7, 2010 at 4:29 AM, Neal Becker wrote: >>> Brian Curtin wrote: >>> >>>> On Fri, Mar 5, 2010 at 12:51, Neal Becker wrote: >>>> >>>>> I generally enjoy argparse, but one thing I find rather >>>>> ugly and unpythonic. >>>>> >>>>> ? ?parser.add_argument ('--plot', action='store_true') >>>>> >>>>> Specifying the argument 'action' as a string is IMO ugly. >>>>> >>>> >>>> What else would you propose? >>>> FWIW, this is the same in optparse. >>> >>> I would have thought use the object itself, instead of a string that spells >>> the object's name. >> >> What object? How would you write the example instead then? > > In argparse, unlike optparse, actions are actually defined by objects > with a particular API, and the string is just a shorthand for > referring to that. So: > > ?parser.add_argument ('--plot', action='store_true') > > is equivalent to: > > ?parser.add_argument('--plot', argparse._StoreTrueAction) Sorry, that should have been: parser.add_argument('--plot', action=argparse._StoreTrueAction) > > Because the names are so long and you'd have to import them, I've left > them as private attributes of the module, but if there's really > demand, we could rename them to argparse.StoreTrueAction, etc. > > Steve > -- > Where did you get that preposterous hypothesis? > Did Steve tell you that? > ? ? ? ?--- The Hiphopopotamus > -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From dstanek at dstanek.com Mon Mar 8 16:53:43 2010 From: dstanek at dstanek.com (David Stanek) Date: Mon, 8 Mar 2010 10:53:43 -0500 Subject: [Python-Dev] argparse ugliness In-Reply-To: References: Message-ID: On Mon, Mar 8, 2010 at 10:40 AM, Steven Bethard wrote: > > In argparse, unlike optparse, actions are actually defined by objects > with a particular API, and the string is just a shorthand for > referring to that. So: > > ?parser.add_argument ('--plot', action='store_true') > > is equivalent to: > > ?parser.add_argument('--plot', argparse._StoreTrueAction) > > Because the names are so long and you'd have to import them, I've left > them as private attributes of the module, but if there's really > demand, we could rename them to argparse.StoreTrueAction, etc. > Any reason not to do something like: from argparse import actions ... parser.add_argument('--plot', actions.store_true) Basically a small namespace for the constants. -- David blog: http://www.traceback.org twitter: http://twitter.com/dstanek From tjreedy at udel.edu Mon Mar 8 17:16:33 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 08 Mar 2010 11:16:33 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4B94DC30.2030904@gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> <20100307154809.742E73A4062@sparrow.telecommunity.com> <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com> <20100307175732.307513A4062@sparrow.telecommunity.com> <5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com> <20100307195614.BBD693A4062@sparrow.telecommunity.com> <4B94DC30.2030904@gmail.com> Message-ID: On 3/8/2010 6:14 AM, Nick Coghlan wrote: > P.J. Eby wrote: >> I'm +1 on adding a nice task queuing system, -1 on calling it by any >> other name. ;-) > > As Guido said, let's call the nice task queuing system "futures" and I was confused by 'futures' also until Philip explained it as task-queue or task-pool, and hence also do not like it. Since the examples in the PEP do *NOT* give example output, it was not clear to me whether execution or the termination thereof is ordered (queue) or not (pool). Looking more close, I gather that the prime results will be printed 'in order' (waiting on each even if others are done) while the url results will be printed 'as available'. Adding 'will print ...' and 'might print ...' outputs would help. > point people wanting a full-power asynchronous process model to Twisted That could be done in the PEP to clarify its scope. Terry Jan Reedy From tjreedy at udel.edu Mon Mar 8 17:21:42 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 08 Mar 2010 11:21:42 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <7EFBC33B-06AD-4BAD-8173-005D6F08240E@sweetapp.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <7EFBC33B-06AD-4BAD-8173-005D6F08240E@sweetapp.com> Message-ID: On 3/6/2010 4:20 AM, Brian Quinlan wrote: > > On 6 Mar 2010, at 03:21, Daniel Stutzbach wrote: > >> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan > > wrote: >> >> import futures >> >> >> +1 on the idea, -1 on the name. It's too similar to "from __future__ >> import ...". >> >> Also, the PEP should probably link to the discussions on stdlib-sig? > > I thoug ht about that but this discussion is spread over many threads > and many months. This is pretty typical. I would say just that, and link to the first. "This PEP was discussed over many months in many threads in the stdlib-sig list. The first was .... . Python-dev discussion occured in . tjr From ndbecker2 at gmail.com Mon Mar 8 17:44:15 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 8 Mar 2010 11:44:15 -0500 Subject: [Python-Dev] argparse ugliness In-Reply-To: References: Message-ID: <201003081144.16370.ndbecker2@gmail.com> On Monday 08 March 2010, David Stanek wrote: > On Mon, Mar 8, 2010 at 10:40 AM, Steven Bethard > > wrote: > > In argparse, unlike optparse, actions are actually defined by objects > > with a particular API, and the string is just a shorthand for > > referring to that. So: > > > > parser.add_argument ('--plot', action='store_true') > > > > is equivalent to: > > > > parser.add_argument('--plot', argparse._StoreTrueAction) > > > > Because the names are so long and you'd have to import them, I've left > > them as private attributes of the module, but if there's really > > demand, we could rename them to argparse.StoreTrueAction, etc. > > Any reason not to do something like: > > from argparse import actions > ... > parser.add_argument('--plot', actions.store_true) > > Basically a small namespace for the constants. +1 From python-dev at masklinn.net Mon Mar 8 17:49:42 2010 From: python-dev at masklinn.net (Xavier Morel) Date: Mon, 8 Mar 2010 17:49:42 +0100 Subject: [Python-Dev] argparse ugliness In-Reply-To: References: Message-ID: On 8 Mar 2010, at 16:53 , David Stanek wrote: > > On Mon, Mar 8, 2010 at 10:40 AM, Steven Bethard > wrote: >> >> In argparse, unlike optparse, actions are actually defined by objects >> with a particular API, and the string is just a shorthand for >> referring to that. So: >> >> parser.add_argument ('--plot', action='store_true') >> >> is equivalent to: >> >> parser.add_argument('--plot', argparse._StoreTrueAction) >> >> Because the names are so long and you'd have to import them, I've left >> them as private attributes of the module, but if there's really >> demand, we could rename them to argparse.StoreTrueAction, etc. >> > > Any reason not to do something like: > > from argparse import actions > ... > parser.add_argument('--plot', actions.store_true) > > Basically a small namespace for the constants. action is taken from **kwargs, not from a positional argument as *args is a sequence of option strings (so you can write add_argument('-p', '/p', '--plot', '--land-mass')). So you'd have to write add_argument('--plot', action=actions.store_true) which is straight from the department of redundant redundancies. An option would be parser.add(actions.StoreTrue('--plot')) but I'm not sure this makes any sense API-wise, and it would probably make the code a lot messier as the parser would have to reach into the action to get the information it needs. Either that, or the action would be an *arg and argparse would have to walk all of its *args type-testing each one to find if there's an action anywhere. From dstanek at dstanek.com Mon Mar 8 17:55:09 2010 From: dstanek at dstanek.com (David Stanek) Date: Mon, 8 Mar 2010 11:55:09 -0500 Subject: [Python-Dev] argparse ugliness In-Reply-To: References: Message-ID: On Mon, Mar 8, 2010 at 11:49 AM, Xavier Morel wrote: > On 8 Mar 2010, at 16:53 , David Stanek wrote: >> >> On Mon, Mar 8, 2010 at 10:40 AM, Steven Bethard >> wrote: >>> >>> In argparse, unlike optparse, actions are actually defined by objects >>> with a particular API, and the string is just a shorthand for >>> referring to that. So: >>> >>> ?parser.add_argument ('--plot', action='store_true') >>> >>> is equivalent to: >>> >>> ?parser.add_argument('--plot', argparse._StoreTrueAction) >>> >>> Because the names are so long and you'd have to import them, I've left >>> them as private attributes of the module, but if there's really >>> demand, we could rename them to argparse.StoreTrueAction, etc. >>> >> >> Any reason not to do something like: >> >> ?from argparse import actions >> ?... >> ?parser.add_argument('--plot', actions.store_true) >> >> Basically a small namespace for the constants. > > action is taken from **kwargs, not from a positional argument as *args > is a sequence of option strings (so you can write > ?add_argument('-p', '/p', '--plot', '--land-mass')). So you'd have to write > add_argument('--plot', action=actions.store_true) which is straight from > the department of redundant redundancies. > > An option would be > > ?parser.add(actions.StoreTrue('--plot')) > > but I'm not sure this makes any sense API-wise, and it would probably make > the code a lot messier as the parser would have to reach into the action > to get the information it needs. Either that, or the action would be an *arg > and argparse would have to walk all of its *args type-testing each one to find > if there's an action anywhere. You could just change my example to: ?from argparse.actions import store_true ?... ?parser.add_argument('--plot', action=store_true) I would probably still import the actions namespace directly, but wouldn't have to. -- David blog: http://www.traceback.org twitter: http://twitter.com/dstanek From fuzzyman at voidspace.org.uk Mon Mar 8 18:14:35 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 08 Mar 2010 17:14:35 +0000 Subject: [Python-Dev] Catch SIGINT at Python startup In-Reply-To: <201003081412.51255.victor.stinner@haypocalc.com> References: <201003081412.51255.victor.stinner@haypocalc.com> Message-ID: <4B95307B.7060202@voidspace.org.uk> On 08/03/2010 13:12, Victor Stinner wrote: > Hi, > > I wrote patches for the issue #3137 and I would like your opinion about my > solution (not directly about the patches): I propose to consider a SIGINT > during startup as a fatal error. If you press CTRL+c while Python is starting, > Python displays an error and exit with the code 1. > > That sounds like a good solution to me. +1 Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From rrr at ronadam.com Mon Mar 8 18:17:13 2010 From: rrr at ronadam.com (Ron Adam) Date: Mon, 08 Mar 2010 11:17:13 -0600 Subject: [Python-Dev] argparse ugliness In-Reply-To: References: Message-ID: Steven Bethard wrote: > On Mon, Mar 8, 2010 at 7:40 AM, Steven Bethard wrote: >> On Sun, Mar 7, 2010 at 11:49 AM, Guido van Rossum wrote: >>> On Sun, Mar 7, 2010 at 4:29 AM, Neal Becker wrote: >>>> Brian Curtin wrote: >>>> >>>>> On Fri, Mar 5, 2010 at 12:51, Neal Becker wrote: >>>>> >>>>>> I generally enjoy argparse, but one thing I find rather >>>>>> ugly and unpythonic. >>>>>> >>>>>> parser.add_argument ('--plot', action='store_true') >>>>>> >>>>>> Specifying the argument 'action' as a string is IMO ugly. >>>>>> >>>>> What else would you propose? >>>>> FWIW, this is the same in optparse. >>>> I would have thought use the object itself, instead of a string that spells >>>> the object's name. >>> What object? How would you write the example instead then? >> In argparse, unlike optparse, actions are actually defined by objects >> with a particular API, and the string is just a shorthand for >> referring to that. So: >> >> parser.add_argument ('--plot', action='store_true') >> >> is equivalent to: >> >> parser.add_argument('--plot', argparse._StoreTrueAction) > > Sorry, that should have been: > > parser.add_argument('--plot', action=argparse._StoreTrueAction) > >> Because the names are so long and you'd have to import them, I've left >> them as private attributes of the module, but if there's really >> demand, we could rename them to argparse.StoreTrueAction, etc. I like the strings. They are simple and easy to use/read and they don't have to be created or imported before the parser is defined. That allows me to put the parser setup right after the 'if __name__ == '__main__':' so it's easy to find and read. It also allows me to not import or create objects that may be expensive in either time or resources if I'm not going to use them. Also the strings help separate the parser from the rest of your program in a much cleaner way. This can make your programs more modular and easier to modify and maintain. Ron From guido at python.org Mon Mar 8 18:47:18 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 8 Mar 2010 09:47:18 -0800 Subject: [Python-Dev] Catch SIGINT at Python startup In-Reply-To: <4B95307B.7060202@voidspace.org.uk> References: <201003081412.51255.victor.stinner@haypocalc.com> <4B95307B.7060202@voidspace.org.uk> Message-ID: Actually it sounds like there's some overly general except clause somewhere that should be adjusted to catch just "Exception" instead of "*". On Mon, Mar 8, 2010 at 9:14 AM, Michael Foord wrote: > On 08/03/2010 13:12, Victor Stinner wrote: >> >> Hi, >> >> I wrote patches for the issue #3137 and I would like your opinion about my >> solution (not directly about the patches): I propose to consider a SIGINT >> during startup as a fatal error. If you press CTRL+c while Python is >> starting, >> Python displays an error and exit with the code 1. >> >> > > That sounds like a good solution to me. +1 > > Michael > > -- > http://www.ironpythoninaction.com/ > http://www.voidspace.org.uk/blog > > READ CAREFULLY. By accepting and reading this email you agree, on behalf of > your employer, to release me from all obligations and waivers arising from > any and all NON-NEGOTIATED agreements, licenses, terms-of-service, > shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, > non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have > entered into with your employer, its partners, licensors, agents and > assigns, in perpetuity, without prejudice to my ongoing rights and > privileges. You further represent that you have the authority to release me > from any BOGUS AGREEMENTS on behalf of your employer. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From guido at python.org Mon Mar 8 18:49:55 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 8 Mar 2010 09:49:55 -0800 Subject: [Python-Dev] argparse ugliness In-Reply-To: References: Message-ID: On Mon, Mar 8, 2010 at 9:17 AM, Ron Adam wrote: > I like the strings. ?They are simple and easy to use/read and they don't > have to be created or imported before the parser is defined. I like them too. I don't see anything unpythonic about them. > That allows me > to put the parser setup right after the 'if __name__ == '__main__':' so it's > easy to find and read. ?It also allows me to not import or create objects > that may be expensive in either time or resources if I'm not going to use > them. > > Also the strings help separate the parser from the rest of your program in a > much cleaner way. This can make your programs more modular and easier to > modify and maintain. Also in this particular case a typo in the string will be caught just about at the same time as a typo in the name would be caught. -- --Guido van Rossum (python.org/~guido) From fijall at gmail.com Mon Mar 8 19:21:20 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 8 Mar 2010 11:21:20 -0700 Subject: [Python-Dev] Catch SIGINT at Python startup In-Reply-To: References: <201003081412.51255.victor.stinner@haypocalc.com> <4B95307B.7060202@voidspace.org.uk> Message-ID: <693bc9ab1003081021icc69731i719df4b4f5d81430@mail.gmail.com> On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum wrote: > Actually it sounds like there's some overly general except clause > somewhere that should be adjusted to catch just "Exception" instead of > "*". > There is at least one that prints "import 'site' failed" and continues to run your program. Cheers, fijal From guido at python.org Mon Mar 8 19:34:12 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 8 Mar 2010 10:34:12 -0800 Subject: [Python-Dev] Catch SIGINT at Python startup In-Reply-To: <693bc9ab1003081021icc69731i719df4b4f5d81430@mail.gmail.com> References: <201003081412.51255.victor.stinner@haypocalc.com> <4B95307B.7060202@voidspace.org.uk> <693bc9ab1003081021icc69731i719df4b4f5d81430@mail.gmail.com> Message-ID: On Mon, Mar 8, 2010 at 10:21 AM, Maciej Fijalkowski wrote: > On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum wrote: >> Actually it sounds like there's some overly general except clause >> somewhere that should be adjusted to catch just "Exception" instead of >> "*". >> > > There is at least one that prints "import 'site' failed" and continues > to run your program. Yeah, it shouldn't do that. The ^C should fall through. -- --Guido van Rossum (python.org/~guido) From digitalxero at gmail.com Mon Mar 8 20:04:39 2010 From: digitalxero at gmail.com (Dj Gilcrease) Date: Mon, 8 Mar 2010 12:04:39 -0700 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4B94DE96.8080100@gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> <4B94DE96.8080100@gmail.com> Message-ID: On Mon, Mar 8, 2010 at 4:25 AM, Nick Coghlan wrote: > Wouldn't a factory function serve that purpose just as well? Or even > just "from concurrent.futures import ProcessPoolExecutor as TaskExecutor". > > That last form has the virtue that you can retrieve your executor from > anywhere rather than being limited to the two provided by the > concurrent.futures model. > > I think the string based approach actually unduly constrains the API > despite superficially appearing to make it more flexible. mm you are correct, I went with the string approach because I was experimenting with 3 additional executor types and wanted to be able to switch between or intermix them them without having to change imports and didnt feel like writing a register class with a factory method. A style I have used in my own code in the past is a Singleton class with register and create methods, where the register takes a name(string) and the class and the create method takes the name and *args, **kwargs and acts as a factory. Would this style be better or would it be better to just leave it with the two executor classes? I tend to dislike multiple classes for a what is essentially a Strategy of a concept and factories are something I tend to forget about until well after my initial idea has formed into a proof of concept. From fuzzyman at voidspace.org.uk Mon Mar 8 20:20:50 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 08 Mar 2010 19:20:50 +0000 Subject: [Python-Dev] unittest2 plus class and module level fixtures in unittest In-Reply-To: <20100307201143.48f0fd3e@msiwind> References: <4B9449F6.9080206@voidspace.org.uk> <20100307201143.48f0fd3e@msiwind> Message-ID: <4B954E12.40901@voidspace.org.uk> On 08/03/2010 01:11, Antoine Pitrou wrote: > Le Mon, 08 Mar 2010 00:51:02 +0000, > Michael Foord a ?crit : > >> In other news, class and module fixtures (setUpClass / >> tearDownClass / setUpModule / tearDownModule) are now implemented in >> unittest (in trunk >> - not yet merged to py3k). These features are tested but I'm sure >> there are some lurking bugs or oddities, so please try them out. I >> have not yet added documentation for them; I'll pull it out from this >> email as a starting point. >> >> I'd rather this thread didn't become *another* debate on the merit of >> these features, but perhaps that is too much to hope for. >> > Just for the record, could you sum up the reasons why you/we decided on > this route rather than e.g. adding a test-resources-like facility? > > (I'm not saying I disagree, it's just that I didn't remember a decision > had already been made on this point) > Well, Guido had already stated on python-ideas that he wanted these features (class and module fixtures) in unittest. I (and others) suggested testresources as an alternative but it became clear that it didn't offer a suitable abstraction for module level fixtures, instead you would have to manually attach the resources to all of the classes in the module yourself. In addition to the lack of a module level fixture abstraction Guido felt that the testresources API itself was not yet mature or polished enough to come into the standard library. At that point I figured we had reverted to the decision that setUpClass / setUpModule should go in. testresources still works of course, and is a better general solution for shared fixtures where people are pushing beyond what setUpClass and setUpModule allow. I'll probably link to testresources in the shared fixture documentation - and it has the great advantage of *not* being in the standard library and is therefore free to evolve faster. All the best, Michael > Thanks for all your work on this! > > cheers > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From darren at ontrenet.com Mon Mar 8 21:06:07 2010 From: darren at ontrenet.com (Darren Govoni) Date: Mon, 08 Mar 2010 15:06:07 -0500 Subject: [Python-Dev] Building thread-safe sqlite3 with Python 2.6.5rc1 Message-ID: <1268078767.5175.4.camel@kratos> Hi, I'm trying to get the thread-safe enabled for sqlite3 in my build of 2.6.5rc1, but its not taking and throws the old thread access error. I am running Ubuntu with other versions of Python, but I successfully built and installed sqlite 3.6.22, pysqlite 2.6.0. Do I have to set something explicit when building python 2.6.5 to find the updated sqlite3? thanks for any tips. Darren -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at haypocalc.com Mon Mar 8 21:50:02 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Mon, 8 Mar 2010 21:50:02 +0100 Subject: [Python-Dev] Catch SIGINT at Python startup In-Reply-To: References: <201003081412.51255.victor.stinner@haypocalc.com> <4B95307B.7060202@voidspace.org.uk> Message-ID: <201003082150.02469.victor.stinner@haypocalc.com> Hi, Le lundi 08 mars 2010 18:47:18, Guido van Rossum a ?crit : > Actually it sounds like there's some overly general except clause > somewhere that should be adjusted to catch just "Exception" instead of > "*". Most of my patches (for SIGINT) are exactly doing that: check the exception type instead of "if (result == NULL) PyErr_Clear();" (try/except: pass). About initsite(): I consider any error as fatal. If initsite() failed, something goes wrong. I think that my patches are enough to catch SIGINT during the Python initialization. I used different tricks to test Python: add kill(getpid(), SIGINT), use "signal SIGINT" in gdb, add PyErr_SetNone(PyExc_KeyboardInterrupt), ... valgrind is a very good tool to slow down any process :-D -- Victor Stinner http://www.haypocalc.com/ From skip at pobox.com Mon Mar 8 21:53:58 2010 From: skip at pobox.com (skip at pobox.com) Date: Mon, 8 Mar 2010 14:53:58 -0600 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4B94DC30.2030904@gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> <20100307154809.742E73A4062@sparrow.telecommunity.com> <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com> <20100307175732.307513A4062@sparrow.telecommunity.com> <5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com> <20100307195614.BBD693A4062@sparrow.telecommunity.com> <4B94DC30.2030904@gmail.com> Message-ID: <19349.25574.880099.668647@montanaro.dyndns.org> >> I'm +1 on adding a nice task queuing system, -1 on calling it by any >> other name. ;-) Nick> As Guido said, let's call the nice task queuing system "futures" Nick> and point people wanting a full-power asynchronous process model Nick> to Twisted Can this module at least be pushed down into a package? I think "concurrent" or "concurrency" were both suggested at one point. Skip From digitalxero at gmail.com Mon Mar 8 21:56:24 2010 From: digitalxero at gmail.com (Dj Gilcrease) Date: Mon, 8 Mar 2010 13:56:24 -0700 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> <4B94DE96.8080100@gmail.com> Message-ID: On Mon, Mar 8, 2010 at 12:04 PM, Dj Gilcrease wrote: > A style I have used in my own code in the past is a Singleton class > with register and create methods, where the register takes a > name(string) and the class and the create method takes the name and > *args, **kwargs and acts as a factory. So I decided to play with this design a little and since I made it a singleton I decided to place all the thread/process tracking and exit handle code in it instead of having the odd semi-global scoped _shutdown, _thread_references, _remove_dead_thread_references and _python_exit objects floating around in each executor file, seems to work well. The API would be from concurrent.futures import executors executor = executors.create(NAME, *args, **kwargs) # NAME is 'process' or 'thread' by default To create your own executor you create your executor class and add the following at the end from concurrent.futures import executors, ExecutorBase class MyExecutor(ExecutorBase): ... executors.register(NAME, MyExecutor) It checks to see if your executor is a subclass of ExecutorBase, but only does a UserWarning if it is not since you should know what methods are required to be an executor like object, so if you are not subclassing ExecutorBase you should suppress the UserWarning before you register you class and un-suppress it after Some Helper Methods/Properties on the executors Singleton add_joinable_ref - This replaces the _thread_references.add system of tracking threads, and it allows for adding processes as well hence the joinable_ref name. It does check to make sure the ref being passes has a join method then creates a weakref to it and adds it to a set. For every thread or process your executor creates you should call this with so it will be tracked properly cleanup_joinable_refs - This replaces the _remove_dead_thread_references that had to be written for each executor individually. This should be called periodically, currently it is only called when you create a new executor since it is a blocking method (it uses a thread lock to make sure the set of references does not change while it is discarding old ones) shutdown - is a readonly property and replaces the _shutdown global var that had to be created for each executor individually, it is set in the executors destructor __del__ - replaces the _python_exit method that had to be written for each executor individually If this API change isnt accepted its no big deal, since I am only changing it due to personal pet peeves and the only real issue I had with the package was scoping which has already been addressed by adding it to a concurrent package From guido at python.org Mon Mar 8 22:06:36 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 8 Mar 2010 13:06:36 -0800 Subject: [Python-Dev] Catch SIGINT at Python startup In-Reply-To: <201003082150.02469.victor.stinner@haypocalc.com> References: <201003081412.51255.victor.stinner@haypocalc.com> <4B95307B.7060202@voidspace.org.uk> <201003082150.02469.victor.stinner@haypocalc.com> Message-ID: On Mon, Mar 8, 2010 at 12:50 PM, Victor Stinner wrote: > Le lundi 08 mars 2010 18:47:18, Guido van Rossum a ?crit : >> Actually it sounds like there's some overly general except clause >> somewhere that should be adjusted to catch just "Exception" instead of >> "*". > > Most of my patches (for SIGINT) are exactly doing that: check the exception > type instead of "if (result == NULL) PyErr_Clear();" (try/except: pass). About > initsite(): I consider any error as fatal. If initsite() failed, something > goes wrong. > > I think that my patches are enough to catch SIGINT during the Python > initialization. I used different tricks to test Python: add kill(getpid(), > SIGINT), use "signal SIGINT" in gdb, add > PyErr_SetNone(PyExc_KeyboardInterrupt), ... valgrind is a very good tool to > slow down any process :-D I see. I thought for a moment that by "fatal" you meant something like calling Py_FatalError(). I am trying to remember why I made site.py failures non-fatal in the first place. I don't have any specific recollection but it must've been either from before the separation between site.py (part of the stdlib) and sitecustomize.py (site-specific) or out of a worry that if some external cause broke site.py (which does a lot of I/O) it would be a fatal breakdown of all Python execution. I am fine with making an exception for KeyboardInterrupt (and other exceptions not deriving from Exception but only from BaseException) but I don't think I want to go as far as making all errors coming out of site.py fatal. (At least not without more understanding of the use case for making these non-fatal in the first place.) OTOH I think the try/except in site.py around the execution of sitecustomize.py might be changed so that it prints a full traceback whenever it raises an exception other than ImportError or, again, exceptions inheriting from BaseException but not from Exception. IOW I think that exceptions coming out of sitecustomize.py should continued to be treated non-fatally, apart from ^C and friends, but I think that exceptions coming out of site.py *might* be considered more fatally once the change to exceptions coming out of sitecustomize.py is made. -- --Guido van Rossum (python.org/~guido) From greg.ewing at canterbury.ac.nz Mon Mar 8 22:20:38 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 09 Mar 2010 10:20:38 +1300 Subject: [Python-Dev] argparse ugliness In-Reply-To: <67850396-A793-4547-9532-CDF0F65E0977@zen.co.uk> References: <67850396-A793-4547-9532-CDF0F65E0977@zen.co.uk> Message-ID: <4B956A26.6010204@canterbury.ac.nz> Mark Russell wrote: > Boolean flags are a common enough case that I'd be inclined to add a wrapper method, > > parser.add_bool_argument('--plot') +1, this looks good. -- Greg From exarkun at twistedmatrix.com Mon Mar 8 22:11:45 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 08 Mar 2010 21:11:45 -0000 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> <4B94DE96.8080100@gmail.com> Message-ID: <20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain> On 08:56 pm, digitalxero at gmail.com wrote: >On Mon, Mar 8, 2010 at 12:04 PM, Dj Gilcrease >wrote: >>A style I have used in my own code in the past is a Singleton class >>with register and create methods, where the register takes a >>name(string) and the class and the create method takes the name and >>*args, **kwargs and acts as a factory. > > >So I decided to play with this design a little and since I made it a >singleton I decided to place all the thread/process tracking and exit >handle code in it instead of having the odd semi-global scoped >_shutdown, _thread_references, _remove_dead_thread_references and >_python_exit objects floating around in each executor file, seems to >work well. The API would be > >from concurrent.futures import executors > >executor = executors.create(NAME, *args, **kwargs) # NAME is 'process' >or 'thread' by default > > >To create your own executor you create your executor class and add the >following at the end Getting rid of the process-global state like this simplifies testing (both testing of the executors themselves and of application code which uses them). It also eliminates the unpleasant interpreter shutdown/module globals interactions that have plagued a number of stdlib systems that keep global state. Jean-Paul From glyph at twistedmatrix.com Mon Mar 8 22:18:34 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 8 Mar 2010 16:18:34 -0500 Subject: [Python-Dev] Catch SIGINT at Python startup In-Reply-To: References: <201003081412.51255.victor.stinner@haypocalc.com> <4B95307B.7060202@voidspace.org.uk> <201003082150.02469.victor.stinner@haypocalc.com> Message-ID: On Mar 8, 2010, at 4:06 PM, Guido van Rossum wrote: > I am trying to remember why I made site.py failures non-fatal in the > first place. I don't have any specific recollection but it must've > been either from before the separation between site.py (part of the > stdlib) and sitecustomize.py (site-specific) or out of a worry that if > some external cause broke site.py (which does a lot of I/O) it would > be a fatal breakdown of all Python execution. The thing that occurs to me is that one might want to write an administrative tool in Python to manipulate site.py, or even just some data that something in site.py would load. If exceptions from site.py were fatal, then bugs in such a tool would be completely unrecoverable; in trying to run it to un-do the buggy operation, it would crash immediately. On the other hand, such a tool should *really* be invoked with the -S option anyway, so... maybe not that pressing of a concern. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Mar 8 22:31:16 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 09 Mar 2010 10:31:16 +1300 Subject: [Python-Dev] argparse ugliness In-Reply-To: References: Message-ID: <4B956CA4.4050108@canterbury.ac.nz> Steven Bethard wrote: > Because the names are so long and you'd have to import them, I've left > them as private attributes of the module, but if there's really > demand, we could rename them to argparse.StoreTrueAction, etc. What's wrong with just StoreTrue? -- Greg From greg.ewing at canterbury.ac.nz Mon Mar 8 22:39:14 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 09 Mar 2010 10:39:14 +1300 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> <20100307154809.742E73A4062@sparrow.telecommunity.com> <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com> <20100307175732.307513A4062@sparrow.telecommunity.com> <5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com> <20100307195614.BBD693A4062@sparrow.telecommunity.com> <4B94DC30.2030904@gmail.com> Message-ID: <4B956E82.4080707@canterbury.ac.nz> Terry Reedy wrote: > Looking more close, I gather that the prime > results will be printed 'in order' (waiting on each even if others are > done) while the url results will be printed 'as available'. Seems to me that if you care about the order of the results, you should be able to just wait for each result separately in the order you want them. Something like task1 = start_task(proc1) task2 = start_task(proc2) task3 = start_task(proc3) result1 = task1.wait_for_result() result2 = task2.wait_for_result() result3 = task3.wait_for_result() This would also be a natural way to write things even if you don't care about the order, but you need all the results before proceeding. You're going to be held up until the longest-running task completes anyway, so it doesn't matter if some of them finish earlier and have to sit around waiting for you to collect the result. -- Greg From robert.kern at gmail.com Mon Mar 8 22:35:46 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 08 Mar 2010 15:35:46 -0600 Subject: [Python-Dev] argparse ugliness In-Reply-To: <4B956A26.6010204@canterbury.ac.nz> References: <67850396-A793-4547-9532-CDF0F65E0977@zen.co.uk> <4B956A26.6010204@canterbury.ac.nz> Message-ID: On 2010-03-08 15:20 PM, Greg Ewing wrote: > Mark Russell wrote: >> Boolean flags are a common enough case that I'd be inclined to add a >> wrapper method, >> >> parser.add_bool_argument('--plot') > > +1, this looks good. I've added it to the argparse bugtracker, along with my suggested spelling add_flag(): http://code.google.com/p/argparse/issues/detail?id=62 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From greg.ewing at canterbury.ac.nz Mon Mar 8 22:46:56 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 09 Mar 2010 10:46:56 +1300 Subject: [Python-Dev] argparse ugliness In-Reply-To: References: Message-ID: <4B957050.90808@canterbury.ac.nz> Xavier Morel wrote: > So you'd have to write > add_argument('--plot', action=actions.store_true) which is straight from > the department of redundant redundancies. This could easily be fixed with from argparse.actions import store_true > An option would be > > parser.add(actions.StoreTrue('--plot')) > > but I'm not sure this makes any sense API-wise, and it would probably make > the code a lot messier as the parser would have to reach into the action > to get the information it needs. It would make more sense if the object concerned were called an Option or some such instead of an Action: parser.add(options.Bool('--plot')) -- Greg From guido at python.org Mon Mar 8 22:40:33 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 8 Mar 2010 13:40:33 -0800 Subject: [Python-Dev] Catch SIGINT at Python startup In-Reply-To: References: <201003081412.51255.victor.stinner@haypocalc.com> <4B95307B.7060202@voidspace.org.uk> <201003082150.02469.victor.stinner@haypocalc.com> Message-ID: On Mon, Mar 8, 2010 at 1:18 PM, Glyph Lefkowitz wrote: > > On Mar 8, 2010, at 4:06 PM, Guido van Rossum wrote: > > I am trying to remember why I made site.py failures non-fatal in the > first place. I don't have any specific recollection but it must've > been either from before the separation between site.py (part of the > stdlib) and sitecustomize.py (site-specific) or out of a worry that if > some external cause broke site.py (which does a lot of I/O) it would > be a fatal breakdown of all Python execution. > > The thing that occurs to me is that one might want to write an > administrative tool in Python to manipulate site.py, or even just some data > that something in site.py would load. This would be a more likely theory if we didn't have sitecustomize.py to be manipulated. > If exceptions from site.py were > fatal, then bugs in such a tool would be completely unrecoverable; in trying > to run it to un-do the buggy operation, it would crash immediately. > On the other hand, such a tool should *really* be invoked with the -S option > anyway, so... maybe not that pressing of a concern. Right. I'm leaning towards the theory that treating site.py failures as non-fatal is older than the separation between site.py and sitecustomize.py. -- --Guido van Rossum (python.org/~guido) From victor.stinner at haypocalc.com Mon Mar 8 22:43:45 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Mon, 8 Mar 2010 22:43:45 +0100 Subject: [Python-Dev] Catch SIGINT at Python startup In-Reply-To: References: <201003081412.51255.victor.stinner@haypocalc.com> Message-ID: <201003082243.45999.victor.stinner@haypocalc.com> Le lundi 08 mars 2010 22:18:34, Glyph Lefkowitz a ?crit : > On Mar 8, 2010, at 4:06 PM, Guido van Rossum wrote: > > I am trying to remember why I made site.py failures non-fatal in the > > first place. I don't have any specific recollection but it must've > > been either from before the separation between site.py (part of the > > stdlib) and sitecustomize.py (site-specific) or out of a worry that if > > some external cause broke site.py (which does a lot of I/O) it would > > be a fatal breakdown of all Python execution. > > The thing that occurs to me is that one might want to write an > administrative tool in Python to manipulate site.py, or even just some > data that something in site.py would load. If exceptions from site.py > were fatal, then bugs in such a tool would be completely unrecoverable; in > trying to run it to un-do the buggy operation, it would crash immediately. I don't think that modifying the site.py file is a good idea. Why not editing/generating the sitecustomize or usercustomize module? Anyway, if you write such tool: use a different language or use a failsafe Python interpreter dedicated to this tool ;-) -- Victor Stinner http://www.haypocalc.com/ From martin at v.loewis.de Mon Mar 8 22:52:50 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 08 Mar 2010 22:52:50 +0100 Subject: [Python-Dev] Building thread-safe sqlite3 with Python 2.6.5rc1 In-Reply-To: <1268078767.5175.4.camel@kratos> References: <1268078767.5175.4.camel@kratos> Message-ID: <4B9571B2.2040601@v.loewis.de> > thanks for any tips. Set sqlite_setup_debug to True in setup.py Regards, Martin From greg.ewing at canterbury.ac.nz Mon Mar 8 23:03:47 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 09 Mar 2010 11:03:47 +1300 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> <4B94DE96.8080100@gmail.com> Message-ID: <4B957443.7060904@canterbury.ac.nz> Dj Gilcrease wrote: > executor = executors.create(NAME, *args, **kwargs) # NAME is 'process' > or 'thread' by default > > from concurrent.futures import executors, ExecutorBase > class MyExecutor(ExecutorBase): ... > executors.register(NAME, MyExecutor) I don't understand the reason for using a registration system rather than just importing names from a module. You mentioned wanting to globally change the executor class being used by a program without having to make changes throughout. Registering a different class under the same name would be one way to do that, but you could achieve the same thing just by assigning to a name in a module. In other words, instead of inventing your own mechanism for managing a namespace, just use a module as your namespace. -- Greg From ben+python at benfinney.id.au Mon Mar 8 22:55:50 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 09 Mar 2010 08:55:50 +1100 Subject: [Python-Dev] argparse ugliness References: Message-ID: <87lje2zdeh.fsf@benfinney.id.au> David Stanek writes: > On Mon, Mar 8, 2010 at 10:40 AM, Steven Bethard > wrote: > > ?parser.add_argument ('--plot', action='store_true') [?] +1. > Any reason not to do something like: > > from argparse import actions > ... > parser.add_argument('--plot', actions.store_true) > > Basically a small namespace for the constants. -1. That ignores the value to the programmer in specifying keyword arguments (there are a lot of arguments to the function, making keyword arguments a better mean-what-you-say choice). To be equivalent, it would have to be: from argparse import actions # ... parser.add_argument('--plot', action=actions.store_true) I prefer the string values. -- \ ?Value your freedom or you will lose it, teaches history. | `\ ?Don't bother us with politics,? respond those who don't want | _o__) to learn.? ?Richard Stallman, 2002 | Ben Finney From victor.stinner at haypocalc.com Mon Mar 8 23:01:34 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Mon, 8 Mar 2010 23:01:34 +0100 Subject: [Python-Dev] Catch SIGINT at Python startup In-Reply-To: References: <201003081412.51255.victor.stinner@haypocalc.com> <201003082150.02469.victor.stinner@haypocalc.com> Message-ID: <201003082301.34649.victor.stinner@haypocalc.com> Le lundi 08 mars 2010 22:06:36, vous avez ?crit : > OTOH I think the try/except in site.py around the execution of > sitecustomize.py might be changed so that it prints a full traceback > whenever it raises an exception other than ImportError or, again, > exceptions inheriting from BaseException but not from Exception. IOW I > think that exceptions coming out of sitecustomize.py should continued > to be treated non-fatally, apart from ^C and friends, but I think that > exceptions coming out of site.py *might* be considered more fatally > once the change to exceptions coming out of sitecustomize.py is made. Do you mean something like the following code? ---------------------- def execsitecustomize(): """Run custom site specific code, if available.""" try: import sitecustomize except ImportError: pass except Exception: if sys.flags.verbose: sys.excepthook(*sys.exc_info()) else: print >>sys.stderr, "'import sitecustomize' failed; use -v for traceback" def execusercustomize(): """Run custom user specific code, if available.""" try: import usercustomize except ImportError: pass except Exception: if sys.flags.verbose: sys.excepthook(*sys.exc_info()) else: print >>sys.stderr, "'import usercustomize' failed; use -v for traceback" ---------------------- Using these functions, even if sitecustomize fails, usercustomize is imported. Can it be a problem? Does usercustomize requires that sitecustomize has been initialized correctly? (The code can be changed to only import usercustomize if sitecutomize succeed or failed with an ImportError, but not if it fails with a different error) -- Victor Stinner http://www.haypocalc.com/ From guido at python.org Mon Mar 8 23:14:51 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 8 Mar 2010 14:14:51 -0800 Subject: [Python-Dev] Catch SIGINT at Python startup In-Reply-To: <201003082301.34649.victor.stinner@haypocalc.com> References: <201003081412.51255.victor.stinner@haypocalc.com> <201003082150.02469.victor.stinner@haypocalc.com> <201003082301.34649.victor.stinner@haypocalc.com> Message-ID: On Mon, Mar 8, 2010 at 2:01 PM, Victor Stinner wrote: > Le lundi 08 mars 2010 22:06:36, vous avez ?crit : >> OTOH I think the try/except in site.py around the execution of >> sitecustomize.py might be changed so that it prints a full traceback >> whenever it raises an exception other than ImportError or, again, >> exceptions inheriting from BaseException but not from Exception. IOW I >> think that exceptions coming out of sitecustomize.py should continued >> to be treated non-fatally, apart from ^C and friends, but I think that >> exceptions coming out of site.py *might* be considered more fatally >> once the change to exceptions coming out of sitecustomize.py is made. > > Do you mean something like the following code? > ---------------------- > def execsitecustomize(): > ? ?"""Run custom site specific code, if available.""" > ? ?try: > ? ? ? ?import sitecustomize > ? ?except ImportError: > ? ? ? ?pass > ? ?except Exception: > ? ? ? ?if sys.flags.verbose: > ? ? ? ? ? ?sys.excepthook(*sys.exc_info()) > ? ? ? ?else: > ? ? ? ? ? ?print >>sys.stderr, "'import sitecustomize' failed; use -v for > traceback" > > def execusercustomize(): > ? ?"""Run custom user specific code, if available.""" > ? ?try: > ? ? ? ?import usercustomize > ? ?except ImportError: > ? ? ? ?pass > ? ?except Exception: > ? ? ? ?if sys.flags.verbose: > ? ? ? ? ? ?sys.excepthook(*sys.exc_info()) > ? ? ? ?else: > ? ? ? ? ? ?print >>sys.stderr, "'import usercustomize' failed; use -v for > traceback" > ---------------------- Yes, roughly. > Using these functions, even if sitecustomize fails, usercustomize is imported. Until 5 minutes ago I didn't even know we had usercustomize. :-) > Can it be a problem? Does usercustomize requires that sitecustomize has been > initialized correctly? I don't know, but if it is, execsitecustomize() could return a flag to be checked by the code that calls execusercustomize(). > (The code can be changed to only import usercustomize if sitecutomize succeed > or failed with an ImportError, but not if it fails with a different error) Right. It doesn't strike me as a big deal either way. -- --Guido van Rossum (python.org/~guido) From darren at ontrenet.com Mon Mar 8 23:28:10 2010 From: darren at ontrenet.com (Darren Govoni) Date: Mon, 08 Mar 2010 17:28:10 -0500 Subject: [Python-Dev] Building thread-safe sqlite3 with Python 2.6.5rc1 In-Reply-To: <4B9571B2.2040601@v.loewis.de> References: <1268078767.5175.4.camel@kratos> <4B9571B2.2040601@v.loewis.de> Message-ID: <1268087290.5175.9.camel@kratos> Thanks. It says. sqlite: found /usr/local/include/sqlite3.h /usr/local/include/sqlite3.h: version 3.6.22 Which is correct. Yet when I run my progam using the newly built python executable and attempt to use sqlite3 functionality across threads it still says: ProgrammingError: SQLite objects created in a thread can only be used in that same thread.The object was created in thread id -1217128768 and this is thread id -1218753680 I set the -DSQLITE_THREADSAFE=1 flag on sqlite3 when I configured, built and installed the lib. Darren On Mon, 2010-03-08 at 22:52 +0100, "Martin v. L?wis" wrote: > > thanks for any tips. > > Set sqlite_setup_debug to True in setup.py > > Regards, > Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Mon Mar 8 23:39:19 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 08 Mar 2010 23:39:19 +0100 Subject: [Python-Dev] Building thread-safe sqlite3 with Python 2.6.5rc1 In-Reply-To: <1268087290.5175.9.camel@kratos> References: <1268078767.5175.4.camel@kratos> <4B9571B2.2040601@v.loewis.de> <1268087290.5175.9.camel@kratos> Message-ID: <4B957C97.2080508@v.loewis.de> > Which is correct. Yet when I run my progam using the newly built python > executable and attempt to use sqlite3 > functionality across threads it still says: > > ProgrammingError: SQLite objects created in a thread can only be used in > that same thread.The object was created in thread id -1217128768 and > this is thread id -1218753680 > > I set the -DSQLITE_THREADSAFE=1 flag on sqlite3 when I configured, built > and installed the lib. So you'll need to run Python in a debugger, trying to find out under what conditions the exception is raised. This kind of question is off-topic for python-dev (which is about development of Python, not about using it). Regards, Martin From phd at phd.pp.ru Mon Mar 8 23:52:41 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Tue, 9 Mar 2010 01:52:41 +0300 Subject: [Python-Dev] Building thread-safe sqlite3 with Python 2.6.5rc1 In-Reply-To: <1268087290.5175.9.camel@kratos> References: <1268078767.5175.4.camel@kratos> <4B9571B2.2040601@v.loewis.de> <1268087290.5175.9.camel@kratos> Message-ID: <20100308225241.GA14115@phd.pp.ru> On Mon, Mar 08, 2010 at 05:28:10PM -0500, Darren Govoni wrote: > ProgrammingError: SQLite objects created in a thread can only be used in > that same thread.The object was created in thread id -1217128768 and > this is thread id -1218753680 Darren, try to pass check_same_thread=False when creating a connection. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From barry at python.org Tue Mar 9 13:56:36 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 9 Mar 2010 07:56:36 -0500 Subject: [Python-Dev] Python 2.6.5 rc 2 Message-ID: <20100309075636.09f52b27@heresy.wooz.org> Hi Python hackateers! It looks like we finally have no more release blockers for 2.6.5rc2. I would like to tag the tree tonight for rc2 so that Martin can build the Windows installer for a release tomorrow. I am also moving the final release back to Friday March 19. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From darren at ontrenet.com Tue Mar 9 14:05:52 2010 From: darren at ontrenet.com (Darren Govoni) Date: Tue, 09 Mar 2010 08:05:52 -0500 Subject: [Python-Dev] Building thread-safe sqlite3 with Python 2.6.5rc1 In-Reply-To: <20100308225241.GA14115@phd.pp.ru> References: <1268078767.5175.4.camel@kratos> <4B9571B2.2040601@v.loewis.de> <1268087290.5175.9.camel@kratos> <20100308225241.GA14115@phd.pp.ru> Message-ID: <1268139952.18649.0.camel@kratos> That worked great Oleg! Thank you! On Tue, 2010-03-09 at 01:52 +0300, Oleg Broytman wrote: > On Mon, Mar 08, 2010 at 05:28:10PM -0500, Darren Govoni wrote: > > ProgrammingError: SQLite objects created in a thread can only be used in > > that same thread.The object was created in thread id -1217128768 and > > this is thread id -1218753680 > > Darren, try to pass check_same_thread=False when creating a connection. > > Oleg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy at alum.mit.edu Tue Mar 9 14:42:28 2010 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Tue, 9 Mar 2010 08:42:28 -0500 Subject: [Python-Dev] Modifying Grammar/grammar and other foul acts In-Reply-To: References: Message-ID: On Sat, Mar 6, 2010 at 11:27 AM, Gregg Lind wrote: > Python-devs, > > I'm writing to you for some help in understanding the Python grammar.? As an > excuse to deep dive into Python's tokenizer / grammar, I decided (as a > hideous, hideous joke) to want to allow braces where colons are allowed (as > flow control). > > Starting from PEP 306 (and branch r311), I hacked on Grammar/Grammer > > As a first example: > > funcdef: ('def' NAME parameters ['->' test] ':' suite | > ????????? 'def' NAME parameters ['->' test] '{' suite '}' ) > > I reran Parser/pgen and the dfa changes, but python (3.1) when recompiled, > throws errors on things like: > > def a() { None } > > Strangely enough: > > lambdef: ( 'lambda' [varargslist] ':' test? | > ?????????????? 'lambda' [varargslist] '{' test '}' ) > > works fine!? I this simplely some difference between "test" and "suite". > > I have tried tackling this with gdb, looking at err_input clearly isn't > enough. > > ??? (gdb) break err_input > ??? (gdb) break PyParser_ASTFromString > ??? import sys > ??? b = compile("def a() {pass}","sys.stdout","single") > ??? # yet a simple grammar fix is enough for this! > ??? c = compile("lambda x {None}","sys.stdout","single") > > I'm in over my head! You don't say what errors occur when you try to compile strings in your new language. You may have changed the Grammar, which allows you to tokenize the input. That isn't enough to get the input to compile. You also need to change the compiler to understand the new tokens. Jeremy > Any insights / help would be appreciated.? Full-on flaming is also > appropriate, but would be less appreciated. > > Specific questions > > 1.)? I assume the Grammar/grammar is read top to bottom.? Confirm? > 2.)? More help figuring out how to debug what python *thinks* it's seeing > when it see "def a() {pass}".? It's not getting to the ast construction > stage, as near as I can tell.? What additional breakpoints can I set to see > where it's failing. > > Gregg L. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu > > From aahz at pythoncraft.com Tue Mar 9 17:08:29 2010 From: aahz at pythoncraft.com (Aahz) Date: Tue, 9 Mar 2010 08:08:29 -0800 Subject: [Python-Dev] FWD: http://www.python.org/dev/patches/ Message-ID: <20100309160828.GA22698@panix.com> ----- Forwarded message from Bob Vadnais ----- > Date: Tue, 09 Mar 2010 00:37:33 -0500 > From: Bob Vadnais > To: webmaster at python.org > Subject: http://www.python.org/dev/patches/ > > > Submit documentation patches the same way. When adding the patch, be > > sure to set the "Category" field to "Documentation". > > There doesn't appear to be a field named "Category". Perhaps there used > to be? The correct field name now appears to be "Components". ----- End forwarded message ----- -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Many customs in this life persist because they ease friction and promote productivity as a result of universal agreement, and whether they are precisely the optimal choices is much less important." --Henry Spencer From solipsis at pitrou.net Tue Mar 9 00:23:23 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 8 Mar 2010 18:23:23 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> <4B94DE96.8080100@gmail.com> <20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain> Message-ID: <20100308182323.1f7b991e@msiwind> Le Mon, 08 Mar 2010 21:11:45 -0000, exarkun at twistedmatrix.com a ?crit : > > Getting rid of the process-global state like this simplifies testing > (both testing of the executors themselves and of application code > which uses them). It also eliminates the unpleasant interpreter > shutdown/module globals interactions that have plagued a number of > stdlib systems that keep global state. +1. From tjreedy at udel.edu Tue Mar 9 06:14:28 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 09 Mar 2010 00:14:28 -0500 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4B956E82.4080707@canterbury.ac.nz> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> <20100307154809.742E73A4062@sparrow.telecommunity.com> <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com> <20100307175732.307513A4062@sparrow.telecommunity.com> <5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com> <20100307195614.BBD693A4062@sparrow.telecommunity.com> <4B94DC30.2030904@gmail.com> <4B956E82.4080707@canterbury.ac.nz> Message-ID: On 3/8/2010 4:39 PM, Greg Ewing wrote: > Terry Reedy wrote: >> Looking more close, I gather that the prime results will be printed >> 'in order' (waiting on each even if others are done) while the url >> results will be printed 'as available'. > > Seems to me that if you care about the order of the results, > you should be able to just wait for each result separately > in the order you want them. Something like > > task1 = start_task(proc1) > task2 = start_task(proc2) > task3 = start_task(proc3) > result1 = task1.wait_for_result() > result2 = task2.wait_for_result() > result3 = task3.wait_for_result() *If* I understand the first example correctly, this is effectively what the first example does with two loops. But I was hoping for clarification and amplification in the PEP. > > This would also be a natural way to write things even if > you don't care about the order, but you need all the results > before proceeding. You're going to be held up until the > longest-running task completes anyway, so it doesn't matter > if some of them finish earlier and have to sit around > waiting for you to collect the result. > From rdmurray at bitdance.com Tue Mar 9 18:31:23 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 09 Mar 2010 12:31:23 -0500 Subject: [Python-Dev] argparse ugliness In-Reply-To: References: <67850396-A793-4547-9532-CDF0F65E0977@zen.co.uk> <4B956A26.6010204@canterbury.ac.nz> Message-ID: <20100309173123.8268E1AB57F@kimball.webabinitio.net> On Mon, 08 Mar 2010 15:35:46 -0600, Robert Kern wrote: > On 2010-03-08 15:20 PM, Greg Ewing wrote: > > Mark Russell wrote: > >> Boolean flags are a common enough case that I'd be inclined to add a > >> wrapper method, > >> > >> parser.add_bool_argument('--plot') > > > > +1, this looks good. > > I've added it to the argparse bugtracker, along with my suggested spelling > add_flag(): > > http://code.google.com/p/argparse/issues/detail?id=62 Shouldn't argparse bugs/enhancement tickets be going to bugs.python.org now? -- R. David Murray www.bitdance.com From robert.kern at gmail.com Tue Mar 9 18:46:17 2010 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 9 Mar 2010 11:46:17 -0600 Subject: [Python-Dev] argparse ugliness In-Reply-To: <20100309173123.8268E1AB57F@kimball.webabinitio.net> References: <67850396-A793-4547-9532-CDF0F65E0977@zen.co.uk> <4B956A26.6010204@canterbury.ac.nz> <20100309173123.8268E1AB57F@kimball.webabinitio.net> Message-ID: <3d375d731003090946p4e3e3d90x70649b4b25c43843@mail.gmail.com> On Tue, Mar 9, 2010 at 11:31, R. David Murray wrote: > On Mon, 08 Mar 2010 15:35:46 -0600, Robert Kern wrote: >> On 2010-03-08 15:20 PM, Greg Ewing wrote: >> > Mark Russell wrote: >> >> Boolean flags are a common enough case that I'd be inclined to add a >> >> wrapper method, >> >> >> >> parser.add_bool_argument('--plot') >> > >> > +1, this looks good. >> >> I've added it to the argparse bugtracker, along with my suggested spelling >> add_flag(): >> >> ? ?http://code.google.com/p/argparse/issues/detail?id=62 > > Shouldn't argparse bugs/enhancement tickets be going to bugs.python.org > now? Probably. Having used the the argparse tracker before, though, it was the first place I thought to file it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From digitalxero at gmail.com Tue Mar 9 22:32:53 2010 From: digitalxero at gmail.com (Dj Gilcrease) Date: Tue, 9 Mar 2010 14:32:53 -0700 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> <4B94DE96.8080100@gmail.com> <20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain> Message-ID: On Mon, Mar 8, 2010 at 2:11 PM, wrote: > Getting rid of the process-global state like this simplifies testing (both > testing of the executors themselves and of application code which uses > them). ?It also eliminates the unpleasant interpreter shutdown/module > globals interactions that have plagued a number of stdlib systems that keep > global state. Ok the new patch is submitted @ http://code.google.com/p/pythonfutures/issues/detail?id=1 *note there are 2 tests that fail and 1 test that dead locks on windows even without this patch, the deadlock test I am skipping in the patch and the two that fail do so for a reason that does not make sense to me. From brian at sweetapp.com Wed Mar 10 09:19:35 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Wed, 10 Mar 2010 19:19:35 +1100 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> <4B94DE96.8080100@gmail.com> <20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain> Message-ID: <40C6F203-726D-4543-AE75-D090A1DA580E@sweetapp.com> On 10 Mar 2010, at 08:32, Dj Gilcrease wrote: > On Mon, Mar 8, 2010 at 2:11 PM, wrote: >> Getting rid of the process-global state like this simplifies >> testing (both >> testing of the executors themselves and of application code which >> uses >> them). It also eliminates the unpleasant interpreter shutdown/module >> globals interactions that have plagued a number of stdlib systems >> that keep >> global state. > > > Ok the new patch is submitted @ > http://code.google.com/p/pythonfutures/issues/detail?id=1 Cool, thanks. > *note there are 2 tests that fail and 1 test that dead locks on > windows even without this patch, the deadlock test I am skipping in > the patch and the two that fail do so for a reason that does not make > sense to me. I'll investigate but I don't have convenient access to a windows machine. Cheers, Brian From brian at sweetapp.com Wed Mar 10 09:19:38 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Wed, 10 Mar 2010 19:19:38 +1100 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4B956E82.4080707@canterbury.ac.nz> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> <20100307154809.742E73A4062@sparrow.telecommunity.com> <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com> <20100307175732.307513A4062@sparrow.telecommunity.com> <5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com> <20100307195614.BBD693A4062@sparrow.telecommunity.com> <4B94DC30.2030904@gmail.com> <4B956E82.4080707@canterbury.ac.nz> Message-ID: <9DDE3DD4-29F2-4062-B28A-C9394BB565B6@sweetapp.com> On 9 Mar 2010, at 08:39, Greg Ewing wrote: > Terry Reedy wrote: >> Looking more close, I gather that the prime results will be printed >> 'in order' (waiting on each even if others are done) while the url >> results will be printed 'as available'. > > Seems to me that if you care about the order of the results, > you should be able to just wait for each result separately > in the order you want them. Something like > > task1 = start_task(proc1) > task2 = start_task(proc2) > task3 = start_task(proc3) > result1 = task1.wait_for_result() > result2 = task2.wait_for_result() > result3 = task3.wait_for_result() You can write this as: executor = ... future1 = executor.submit(proc1) future2 = executor.submit(proc2) future3 = executor.submit(proc3) result1 = task1.result() result2 = task2.result() result3 = task3.result() > This would also be a natural way to write things even if > you don't care about the order, but you need all the results > before proceeding. You're going to be held up until the > longest-running task completes anyway, so it doesn't matter > if some of them finish earlier and have to sit around > waiting for you to collect the result. Often you don't want to continue if there is a failure. In the example that you gave, if "proc3" raises an exception immediately, you still wait for "proc1" and "proc2" to complete even though you will end up discarding their results. Cheers, Brian From brian at sweetapp.com Wed Mar 10 09:19:42 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Wed, 10 Mar 2010 19:19:42 +1100 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> <4B94DE96.8080100@gmail.com> <20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain> Message-ID: On 9 Mar 2010, at 08:11, exarkun at twistedmatrix.com wrote: > On 08:56 pm, digitalxero at gmail.com wrote: >> On Mon, Mar 8, 2010 at 12:04 PM, Dj Gilcrease >> wrote: >>> A style I have used in my own code in the past is a Singleton class >>> with register and create methods, where the register takes a >>> name(string) and the class and the create method takes the name and >>> *args, **kwargs and acts as a factory. >> >> >> So I decided to play with this design a little and since I made it a >> singleton I decided to place all the thread/process tracking and exit >> handle code in it instead of having the odd semi-global scoped >> _shutdown, _thread_references, _remove_dead_thread_references and >> _python_exit objects floating around in each executor file, seems to >> work well. The API would be >> >> from concurrent.futures import executors >> >> executor = executors.create(NAME, *args, **kwargs) # NAME is >> 'process' >> or 'thread' by default >> >> >> To create your own executor you create your executor class and add >> the >> following at the end > > Getting rid of the process-global state like this simplifies testing > (both testing of the executors themselves and of application code > which uses them). It also eliminates the unpleasant interpreter > shutdown/module globals interactions that have plagued a number of > stdlib systems that keep global state. I'm not sure what you mean, could you clarify? Cheers, Brian From brian at sweetapp.com Wed Mar 10 09:19:53 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Wed, 10 Mar 2010 19:19:53 +1100 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <7EFBC33B-06AD-4BAD-8173-005D6F08240E@sweetapp.com> Message-ID: <8AAB0679-897B-4D08-8C42-0DE1732055DC@sweetapp.com> On 9 Mar 2010, at 03:21, Terry Reedy wrote: > On 3/6/2010 4:20 AM, Brian Quinlan wrote: >> >> On 6 Mar 2010, at 03:21, Daniel Stutzbach wrote: >> >>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan >> > wrote: >>> >>> import futures >>> >>> >>> +1 on the idea, -1 on the name. It's too similar to "from __future__ >>> import ...". >>> >>> Also, the PEP should probably link to the discussions on stdlib-sig? >> >> I thoug ht about that but this discussion is spread over many threads >> and many months. > > This is pretty typical. I would say just that, and link to the first. > "This PEP was discussed over many months in many threads in the > stdlib-sig list. The first was .... . Python-dev discussion occured > in . I'll add that. Cheers, Brian From ncoghlan at gmail.com Wed Mar 10 13:32:15 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 10 Mar 2010 22:32:15 +1000 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> <4B94DE96.8080100@gmail.com> <20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain> Message-ID: <4B97914F.50402@gmail.com> Brian Quinlan wrote: >> Getting rid of the process-global state like this simplifies testing >> (both testing of the executors themselves and of application code >> which uses them). It also eliminates the unpleasant interpreter >> shutdown/module globals interactions that have plagued a number of >> stdlib systems that keep global state. > > I'm not sure what you mean, could you clarify? Assuming your question refers to the second sentence, Jean-Paul is referring to a trick of the CPython interpreter when it terminates. To maximise the chances of objects being deleted properly rather than just dumped from memory when the process exits, module dictionaries are filled with None values before the interpreter shuts down. This can cause weirdness (usually intermittent name errors during shutdown) when __del__ methods directly or indirectly reference module globals. One of the easiest ways to avoid that is to put the state on a singleton object, then give the affected classes a reference to that object. Cheers, Nick. P.S. This problem is actually the reason we don't have a context manager for temporary directories yet. Something that should have been simple became a twisty journey down the rabbit hole: http://bugs.python.org/issue5178 -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Mar 10 13:37:38 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 10 Mar 2010 22:37:38 +1000 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <19349.25574.880099.668647@montanaro.dyndns.org> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> <20100307154809.742E73A4062@sparrow.telecommunity.com> <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com> <20100307175732.307513A4062@sparrow.telecommunity.com> <5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com> <20100307195614.BBD693A4062@sparrow.telecommunity.com> <4B94DC30.2030904@gmail.com> <19349.25574.880099.668647@montanaro.dyndns.org> Message-ID: <4B979292.4080906@gmail.com> skip at pobox.com wrote: > >> I'm +1 on adding a nice task queuing system, -1 on calling it by any > >> other name. ;-) > > Nick> As Guido said, let's call the nice task queuing system "futures" > Nick> and point people wanting a full-power asynchronous process model > Nick> to Twisted > > Can this module at least be pushed down into a package? I think > "concurrent" or "concurrency" were both suggested at one point. Yep, I believe "concurrent.futures" was picked as the name elsewhere in the thread. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Mar 10 13:46:14 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 10 Mar 2010 22:46:14 +1000 Subject: [Python-Dev] argparse ugliness In-Reply-To: <4B956CA4.4050108@canterbury.ac.nz> References: <4B956CA4.4050108@canterbury.ac.nz> Message-ID: <4B979496.1080705@gmail.com> Greg Ewing wrote: > Steven Bethard wrote: > >> Because the names are so long and you'd have to import them, I've left >> them as private attributes of the module, but if there's really >> demand, we could rename them to argparse.StoreTrueAction, etc. > > What's wrong with just StoreTrue? All of this discussion about the class names is ignoring the main benefit of using the string names: - with Python variables, you just get a generic name error at the reference site (which may or may not be useful, depending on the program structure) - with a string, the parser *tells* you that the problem is with the requested action for a particular argument The second, explicit error is going to be more informative in most cases. Using strings also reduces the verbosity of the code, avoiding either an "argparse." buried in the middle of the function call, or else a "from argparse import ". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Mar 10 13:48:23 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 10 Mar 2010 22:48:23 +1000 Subject: [Python-Dev] argparse ugliness In-Reply-To: <4B957050.90808@canterbury.ac.nz> References: <4B957050.90808@canterbury.ac.nz> Message-ID: <4B979517.10406@gmail.com> Greg Ewing wrote: > Xavier Morel wrote: >> So you'd have to write add_argument('--plot', >> action=actions.store_true) which is straight from the department of >> redundant redundancies. > > This could easily be fixed with > > from argparse.actions import store_true Converting argparse from a module to a package (which would be necessary to make the above work) is above the bar I set for "easy". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From eric at trueblade.com Wed Mar 10 14:13:17 2010 From: eric at trueblade.com (Eric Smith) Date: Wed, 10 Mar 2010 08:13:17 -0500 Subject: [Python-Dev] argparse ugliness In-Reply-To: <4B979496.1080705@gmail.com> References: <4B956CA4.4050108@canterbury.ac.nz> <4B979496.1080705@gmail.com> Message-ID: <4B979AED.8010300@trueblade.com> Nick Coghlan wrote: > Greg Ewing wrote: >> Steven Bethard wrote: >> >>> Because the names are so long and you'd have to import them, I've left >>> them as private attributes of the module, but if there's really >>> demand, we could rename them to argparse.StoreTrueAction, etc. >> What's wrong with just StoreTrue? > > All of this discussion about the class names is ignoring the main > benefit of using the string names: > > - with Python variables, you just get a generic name error at the > reference site (which may or may not be useful, depending on the program > structure) > - with a string, the parser *tells* you that the problem is with the > requested action for a particular argument > > The second, explicit error is going to be more informative in most cases. > > Using strings also reduces the verbosity of the code, avoiding either an > "argparse." buried in the middle of the function call, or else a "from > argparse import ". I agree that it should be left as a string. About the only reason I would suggest changing it from a string to an object is that it makes it obvious that argparse does accept an object that conforms to the Action API. But this is already well documented, so I don't see the need. Just leave it as a string. -- Eric. From fdrake at acm.org Wed Mar 10 14:22:44 2010 From: fdrake at acm.org (Fred Drake) Date: Wed, 10 Mar 2010 08:22:44 -0500 Subject: [Python-Dev] argparse ugliness In-Reply-To: <4B979496.1080705@gmail.com> References: <4B956CA4.4050108@canterbury.ac.nz> <4B979496.1080705@gmail.com> Message-ID: <9cee7ab81003100522h1a6bcac6ib4061a48327750e@mail.gmail.com> On Wed, Mar 10, 2010 at 7:46 AM, Nick Coghlan wrote: > All of this discussion about the class names is ignoring the main > benefit of using the string names: Another benefit of strings is that data-driven argparse configuration will usually be slightly simpler. Some of us find things like that useful. :-) -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From barry at python.org Wed Mar 10 18:50:10 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 10 Mar 2010 12:50:10 -0500 Subject: [Python-Dev] Python 2.6.5 release candidate 2 now available Message-ID: <20100310125010.415f838e@heresy.wooz.org> Hi everyone, The source tarballs and Windows installer for Python 2.6.5 release candidate 2 are now available: http://www.python.org/download/releases/2.6.5/ As usual, please download, install, and test them with your favorite projects and environments. A number of regressions and build problems on OS X have been fixed since rc1, and I'm really hoping we will not have to do an rc3. I'm currently planning on releasing 2.6.5 final on March 19, 2010. Enjoy, -Barry P.S. The Mac installer will hopefully be available soon. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From fuzzyman at voidspace.org.uk Thu Mar 11 01:45:00 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 11 Mar 2010 00:45:00 +0000 Subject: [Python-Dev] FWD: http://www.python.org/dev/patches/ In-Reply-To: <20100309160828.GA22698@panix.com> References: <20100309160828.GA22698@panix.com> Message-ID: <4B983D0C.1030309@voidspace.org.uk> On 09/03/2010 16:08, Aahz wrote: > ----- Forwarded message from Bob Vadnais > ----- > > >> Date: Tue, 09 Mar 2010 00:37:33 -0500 >> From: Bob Vadnais >> To: webmaster at python.org >> Subject: http://www.python.org/dev/patches/ >> >> >>> Submit documentation patches the same way. When adding the patch, be> >>> >> sure to set the "Category" field to "Documentation". >> >> There doesn't appear to be a field named "Category". Perhaps there used >> to be? The correct field name now appears to be "Components". >> Yep, 'Components' is correct. I'm amending that page appropriately. Thanks Michael Foord > ----- End forwarded message ----- > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From victor.stinner at haypocalc.com Thu Mar 11 02:21:42 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Thu, 11 Mar 2010 02:21:42 +0100 Subject: [Python-Dev] Catch SIGINT at Python startup In-Reply-To: <693bc9ab1003081021icc69731i719df4b4f5d81430@mail.gmail.com> References: <201003081412.51255.victor.stinner@haypocalc.com> <693bc9ab1003081021icc69731i719df4b4f5d81430@mail.gmail.com> Message-ID: <201003110221.42480.victor.stinner@haypocalc.com> Hi, Some news about my patch. Le lundi 08 mars 2010 19:21:20, Maciej Fijalkowski a ?crit : > On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum wrote: > > Actually it sounds like there's some overly general except clause > > somewhere that should be adjusted to catch just "Exception" instead of > > "*". > > There is at least one that prints "import 'site' failed" and continues > to run your program. I commited to patch and broke all buildbots \o/ There were errors in test_sysconfig and test_subprocess related to issues #7774 and #7880 (problems with sys.executable and modified sys.argv[0]). I consider that it's a good point to detect the errors earlier, but it breaks code based on the errors :-) I will no backport the fix to 2.6 (or at least, not the fix about importing the site module) Should I backport the fix to 3.1? (The backport to py3k is not done yet) -- Victor Stinner http://www.haypocalc.com/ From brian at sweetapp.com Thu Mar 11 12:00:38 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Thu, 11 Mar 2010 22:00:38 +1100 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4B97914F.50402@gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> <4B94DE96.8080100@gmail.com> <20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain> <4B97914F.50402@gmail.com> Message-ID: <50A261E0-C876-4FA3-91ED-9BDE55572839@sweetapp.com> On 10 Mar 2010, at 23:32, Nick Coghlan wrote: > Brian Quinlan wrote: >>> Getting rid of the process-global state like this simplifies testing >>> (both testing of the executors themselves and of application code >>> which uses them). It also eliminates the unpleasant interpreter >>> shutdown/module globals interactions that have plagued a number of >>> stdlib systems that keep global state. >> >> I'm not sure what you mean, could you clarify? > > Assuming your question refers to the second sentence, Jean-Paul is > referring to a trick of the CPython interpreter when it terminates. To > maximise the chances of objects being deleted properly rather than > just > dumped from memory when the process exits, module dictionaries are > filled with None values before the interpreter shuts down. > > This can cause weirdness (usually intermittent name errors during > shutdown) when __del__ methods directly or indirectly reference module > globals. Ah. I'm familiar with this problem. My approach was to install an exit handler that ensures that all pending futures are complete and all threads and processes exit before allowing the interpreter to exit. Cheers, Brian > One of the easiest ways to avoid that is to put the state on a > singleton > object, then give the affected classes a reference to that object. > > Cheers, > Nick. > > P.S. This problem is actually the reason we don't have a context > manager > for temporary directories yet. Something that should have been simple > became a twisty journey down the rabbit hole: > http://bugs.python.org/issue5178 > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > --------------------------------------------------------------- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com From victor.stinner at haypocalc.com Thu Mar 11 14:56:39 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Thu, 11 Mar 2010 14:56:39 +0100 Subject: [Python-Dev] Bug tracker: no confirmation before deleting a message Message-ID: <201003111456.39941.victor.stinner@haypocalc.com> Hi, I removed two messages by error in the bug tracker :-( The first message is msg100432 (first message of #8063) which has been restored by "admin". Thank you admin! The second message is msg98284, part of #7774. How can I restore it? Or can "admin" restore it please? :-) There is an open issue on Roundup tracker to add a button to restore a message or a deleted file: http://psf.upfronthosting.co.za/roundup/meta/issue267 I tried to following URL to restore the message, but it didn't work: http://bugs.python.org/issue8063?@action=edit&@add at messages=100432 Could you at least add a Javascript confirmation dialog please? :-) -- Victor Stinner http://www.haypocalc.com/ From martin at v.loewis.de Thu Mar 11 15:59:01 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 11 Mar 2010 15:59:01 +0100 Subject: [Python-Dev] Bug tracker: no confirmation before deleting a message In-Reply-To: <201003111456.39941.victor.stinner@haypocalc.com> References: <201003111456.39941.victor.stinner@haypocalc.com> Message-ID: <4B990535.6070700@v.loewis.de> > I removed two messages by error in the bug tracker :-( Please discuss tracker issues on tracker-discuss or, better yet, file a report on the meta tracker. > The second message is msg98284, part of #7774. How can I restore it? Or can > "admin" restore it please? :-) Done. > There is an open issue on Roundup tracker to add a button to restore a message > or a deleted file: > > http://psf.upfronthosting.co.za/roundup/meta/issue267 Unfortunately, I didn't have time to look into this yet (and may not for many more weeks to come). > I tried to following URL to restore the message, but it didn't work: > > http://bugs.python.org/issue8063?@action=edit&@add at messages=100432 Roundup has stopped accepting GET requests that edit the database, for XSS concerns. > Could you at least add a Javascript confirmation dialog please? :-) That would be even more effort than looking into the existing issue, unless code is provided. Regards, Martin From brett at python.org Thu Mar 11 21:58:11 2010 From: brett at python.org (Brett Cannon) Date: Thu, 11 Mar 2010 12:58:11 -0800 Subject: [Python-Dev] Catch SIGINT at Python startup In-Reply-To: <201003110221.42480.victor.stinner@haypocalc.com> References: <201003081412.51255.victor.stinner@haypocalc.com> <693bc9ab1003081021icc69731i719df4b4f5d81430@mail.gmail.com> <201003110221.42480.victor.stinner@haypocalc.com> Message-ID: On Wed, Mar 10, 2010 at 17:21, Victor Stinner wrote: > Hi, > > Some news about my patch. > > Le lundi 08 mars 2010 19:21:20, Maciej Fijalkowski a ?crit : > > On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum > wrote: > > > Actually it sounds like there's some overly general except clause > > > somewhere that should be adjusted to catch just "Exception" instead of > > > "*". > > > > There is at least one that prints "import 'site' failed" and continues > > to run your program. > > I commited to patch and broke all buildbots \o/ There were errors in > test_sysconfig and test_subprocess related to issues #7774 and #7880 > (problems > with sys.executable and modified sys.argv[0]). > > I consider that it's a good point to detect the errors earlier, but it > breaks > code based on the errors :-) I will no backport the fix to 2.6 (or at > least, > not the fix about importing the site module) > > Should I backport the fix to 3.1? (The backport to py3k is not done yet) > > Once you are happy with the stability then please port it to 3.1. -Brett > -- > Victor Stinner > http://www.haypocalc.com/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Thu Mar 11 22:33:43 2010 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 11 Mar 2010 15:33:43 -0600 Subject: [Python-Dev] Catch SIGINT at Python startup In-Reply-To: References: <201003081412.51255.victor.stinner@haypocalc.com> <693bc9ab1003081021icc69731i719df4b4f5d81430@mail.gmail.com> <201003110221.42480.victor.stinner@haypocalc.com> Message-ID: <1afaf6161003111333k791e21afy8c1f47d30338ca76@mail.gmail.com> 2010/3/11 Brett Cannon : > > > On Wed, Mar 10, 2010 at 17:21, Victor Stinner > wrote: >> >> Hi, >> >> Some news about my patch. >> >> Le lundi 08 mars 2010 19:21:20, Maciej Fijalkowski a ?crit : >> > On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum >> > wrote: >> > > Actually it sounds like there's some overly general except clause >> > > somewhere that should be adjusted to catch just "Exception" instead of >> > > "*". >> > >> > There is at least one that prints "import 'site' failed" and continues >> > to run your program. >> >> I commited to patch and broke all buildbots \o/ There were errors in >> test_sysconfig and test_subprocess related to issues #7774 and #7880 >> (problems >> with sys.executable and modified sys.argv[0]). >> >> I consider that it's a good point to detect the errors earlier, but it >> breaks >> code based on the errors :-) I will no backport the fix to 2.6 (or at >> least, >> not the fix about importing the site module) >> >> Should I backport the fix to 3.1? (The backport to py3k is not done yet) >> > > Once you are happy with the stability then please port it to 3.1. > -Brett But after 3.1.2 please! -- Regards, Benjamin From brett at python.org Thu Mar 11 22:35:48 2010 From: brett at python.org (Brett Cannon) Date: Thu, 11 Mar 2010 13:35:48 -0800 Subject: [Python-Dev] Catch SIGINT at Python startup In-Reply-To: <1afaf6161003111333k791e21afy8c1f47d30338ca76@mail.gmail.com> References: <201003081412.51255.victor.stinner@haypocalc.com> <693bc9ab1003081021icc69731i719df4b4f5d81430@mail.gmail.com> <201003110221.42480.victor.stinner@haypocalc.com> <1afaf6161003111333k791e21afy8c1f47d30338ca76@mail.gmail.com> Message-ID: On Thu, Mar 11, 2010 at 13:33, Benjamin Peterson wrote: > 2010/3/11 Brett Cannon : > > > > > > On Wed, Mar 10, 2010 at 17:21, Victor Stinner < > victor.stinner at haypocalc.com> > > wrote: > >> > >> Hi, > >> > >> Some news about my patch. > >> > >> Le lundi 08 mars 2010 19:21:20, Maciej Fijalkowski a ?crit : > >> > On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum > >> > wrote: > >> > > Actually it sounds like there's some overly general except clause > >> > > somewhere that should be adjusted to catch just "Exception" instead > of > >> > > "*". > >> > > >> > There is at least one that prints "import 'site' failed" and continues > >> > to run your program. > >> > >> I commited to patch and broke all buildbots \o/ There were errors in > >> test_sysconfig and test_subprocess related to issues #7774 and #7880 > >> (problems > >> with sys.executable and modified sys.argv[0]). > >> > >> I consider that it's a good point to detect the errors earlier, but it > >> breaks > >> code based on the errors :-) I will no backport the fix to 2.6 (or at > >> least, > >> not the fix about importing the site module) > >> > >> Should I backport the fix to 3.1? (The backport to py3k is not done yet) > >> > > > > Once you are happy with the stability then please port it to 3.1. > > -Brett > > But after 3.1.2 please! > Sorry, I meant py3k (aka 3.2) and not 3.1. Since this is a behavior change I don't think it should be in 2.6/3.1. -Brett > > > > -- > Regards, > Benjamin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at haypocalc.com Fri Mar 12 01:05:39 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 12 Mar 2010 01:05:39 +0100 Subject: [Python-Dev] Catch SIGINT at Python startup In-Reply-To: References: <201003081412.51255.victor.stinner@haypocalc.com> <201003110221.42480.victor.stinner@haypocalc.com> Message-ID: <201003120105.39900.victor.stinner@haypocalc.com> Le jeudi 11 mars 2010 21:58:11, Brett Cannon a ?crit : > > (...) > > Should I backport the fix to 3.1? (The backport to py3k is not done yet) > > Once you are happy with the stability then please port it to 3.1. My commit broke test_sysconfig, test_platform and test_subprocess. Some commits later, #7880 is fixed and part of #7774 is also fixed. So yes, it will be possible to port the fix to py3k. -- Victor Stinner http://www.haypocalc.com/ From ndbecker2 at gmail.com Fri Mar 12 02:52:57 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 11 Mar 2010 20:52:57 -0500 Subject: [Python-Dev] interesting article on regex performance Message-ID: http://swtch.com/~rsc/regexp/regexp1.html From peter.a.portante at gmail.com Fri Mar 12 03:12:12 2010 From: peter.a.portante at gmail.com (Peter Portante) Date: Thu, 11 Mar 2010 21:12:12 -0500 Subject: [Python-Dev] interesting article on regex performance In-Reply-To: Message-ID: http://code.google.com/p/re2/ On 3/11/10 8:52 PM, "Neal Becker" wrote: > http://swtch.com/~rsc/regexp/regexp1.html > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com From g.brandl at gmx.net Fri Mar 12 11:13:10 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 12 Mar 2010 11:13:10 +0100 Subject: [Python-Dev] Modifying Grammar/grammar and other foul acts In-Reply-To: References: Message-ID: Am 09.03.2010 14:42, schrieb Jeremy Hylton: > On Sat, Mar 6, 2010 at 11:27 AM, Gregg Lind wrote: >> Python-devs, >> >> I'm writing to you for some help in understanding the Python grammar. As an >> excuse to deep dive into Python's tokenizer / grammar, I decided (as a >> hideous, hideous joke) to want to allow braces where colons are allowed (as >> flow control). >> >> Starting from PEP 306 (and branch r311), I hacked on Grammar/Grammer >> >> As a first example: >> >> funcdef: ('def' NAME parameters ['->' test] ':' suite | >> 'def' NAME parameters ['->' test] '{' suite '}' ) >> >> I reran Parser/pgen and the dfa changes, but python (3.1) when recompiled, >> throws errors on things like: >> >> def a() { None } >> >> Strangely enough: >> >> lambdef: ( 'lambda' [varargslist] ':' test | >> 'lambda' [varargslist] '{' test '}' ) >> >> works fine! I this simplely some difference between "test" and "suite". >> >> I have tried tackling this with gdb, looking at err_input clearly isn't >> enough. >> >> (gdb) break err_input >> (gdb) break PyParser_ASTFromString >> import sys >> b = compile("def a() {pass}","sys.stdout","single") >> # yet a simple grammar fix is enough for this! >> c = compile("lambda x {None}","sys.stdout","single") >> >> I'm in over my head! > > You don't say what errors occur when you try to compile strings in > your new language. You may have changed the Grammar, which allows you > to tokenize the input. That isn't enough to get the input to compile. > You also need to change the compiler to understand the new tokens. In particular, many AST creation functions check for specific counts of children on many nodes. I haven't checked, but in the case of the "funcdef" rule, it may check for either 7 or 5 children to determine whether the optional return annotation ['->' test] is present. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From ncoghlan at gmail.com Fri Mar 12 17:12:23 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 13 Mar 2010 02:12:23 +1000 Subject: [Python-Dev] interesting article on regex performance In-Reply-To: References: Message-ID: <4B9A67E7.30801@gmail.com> Peter Portante wrote: > http://code.google.com/p/re2/ > > On 3/11/10 8:52 PM, "Neal Becker" wrote: > >> http://swtch.com/~rsc/regexp/regexp1.html Both interesting links. I'll add another one to the list: http://google-opensource.blogspot.com/2010/03/re2-principled-approach-to-regular.html To bring this on-topic for python-dev by considering how it could apply to Python's default re engine, I think the key issue is that any updates to the default engine would need to remain backwards compatible with all of the tricks that re2 doesn't support. There are major practical problems associated with making such a leap directly (Google's re2 engine is in C++ rather than C and we'd have to keep the existing implementation around regardless to handle the features that re2 doesn't support). I would say it is better to let re2 bake for a while and see if anyone is motivated to come up with Python bindings for it and release them on PyPI. Once that happens (and assuming the bindings earn a good reputation), the first step towards integration would be to include a "See Also" in the re module documentation to point people towards the more limited (but more robust) regex engine implementation. The next step would probably be a hybrid third party library that exploits the NFA approach when it can, but resorts to backtracking when it has to in order to handle full regex functionality. (Although developers would need to be able to disable the backtracking support in order to restore re2's guarantees of linear time execution) Only once such a hybrid library had been proven in practice could the default re module be considered for possible update. Basically, I see some interesting ideas there, but I don't see anything that will impact the Python core or standard library in the short term. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From tseaver at palladion.com Fri Mar 12 18:02:46 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 12 Mar 2010 12:02:46 -0500 Subject: [Python-Dev] interesting article on regex performance In-Reply-To: <4B9A67E7.30801@gmail.com> References: <4B9A67E7.30801@gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nick Coghlan wrote: > Peter Portante wrote: >> http://code.google.com/p/re2/ >> >> On 3/11/10 8:52 PM, "Neal Becker" wrote: >> >>> http://swtch.com/~rsc/regexp/regexp1.html > > Both interesting links. I'll add another one to the list: > http://google-opensource.blogspot.com/2010/03/re2-principled-approach-to-regular.html > > To bring this on-topic for python-dev by considering how it could apply > to Python's default re engine, I think the key issue is that any updates > to the default engine would need to remain backwards compatible with all > of the tricks that re2 doesn't support. > > There are major practical problems associated with making such a leap > directly (Google's re2 engine is in C++ rather than C and we'd have to > keep the existing implementation around regardless to handle the > features that re2 doesn't support). > > I would say it is better to let re2 bake for a while and see if anyone > is motivated to come up with Python bindings for it and release them on > PyPI. Once that happens (and assuming the bindings earn a good > reputation), the first step towards integration would be to include a > "See Also" in the re module documentation to point people towards the > more limited (but more robust) regex engine implementation. > > The next step would probably be a hybrid third party library that > exploits the NFA approach when it can, but resorts to backtracking when > it has to in order to handle full regex functionality. (Although > developers would need to be able to disable the backtracking support in > order to restore re2's guarantees of linear time execution) > > Only once such a hybrid library had been proven in practice could the > default re module be considered for possible update. > > Basically, I see some interesting ideas there, but I don't see anything > that will impact the Python core or standard library in the short term. Agreed on the whole. However, it might be interesting to explore the "hybrid mode" using some of the older, C-based libraries referenced by the original (Russ Cox) article, e.g. Rob Pike's Unicode-aware 1992 implmentation, as ported for Unix: http://swtch.com/plan9port/unix/ Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkuac7AACgkQ+gerLs4ltQ6t3wCgxQPsnpbyE1N9BEGg7nHqcBVP q04AnA6RMJw83+uc5u6Oyud89SXEoP5C =+0aL -----END PGP SIGNATURE----- From brent.longborough at gmail.com Fri Mar 12 18:42:13 2010 From: brent.longborough at gmail.com (Brent Longborough) Date: Fri, 12 Mar 2010 17:42:13 +0000 (UTC) Subject: [Python-Dev] interesting article on regex performance References: <4B9A67E7.30801@gmail.com> Message-ID: I have some regex-intensive Python that might benefit from this, but I don't think I have enough skill to do a set of Python bindings. An ideal first cut would be to enable this: import re2 as re I live in hope... Brent L From arigo at tunes.org Fri Mar 12 19:31:21 2010 From: arigo at tunes.org (Armin Rigo) Date: Fri, 12 Mar 2010 19:31:21 +0100 Subject: [Python-Dev] PyPy 1.2, JIT included Message-ID: <20100312183121.GA17630@code0.codespeak.net> ================================== PyPy 1.2: Just-in-Time Compilation ================================== PyPy 1.2 has been released. The highlight of this release is to be the first that ships with a Just-in-Time compiler that is known to be faster than CPython (and unladen swallow) on some real-world applications (or the best benchmarks we could get for them). The main theme for the 1.2 release is speed. Main site: http://pypy.org/ The JIT is stable and we don't observe crashes. Nevertheless we would recommend you to treat it as beta software and as a way to try out the JIT to see how it works for you. Highlights of This Release ========================== * The JIT compiler. * Various interpreter optimizations that improve performance as well as help save memory. * Introducing a new PyPy website at http://pypy.org/ , made by tav and improved by the PyPy team. * Introducing http://speed.pypy.org/ , a new service that monitors our performance nightly, made by Miquel Torres. * There will be ubuntu packages on "PyPy's PPA" made by Bartosz Skowron; however various troubles prevented us from having them as of now. Known JIT problems (or why you should consider this beta software): * The only supported platform is 32bit x86 for now, we're looking for help with other platforms. * It is still memory-hungry. There is no limit on the amount of RAM that the assembler can consume; it is thus possible (although unlikely) that the assembler ends up using unreasonable amounts of memory. If you want to try PyPy, go to the "download page" on our excellent new site at http://pypy.org/download.html and find the binary for your platform. If the binary does not work (e.g. on Linux, because of different versions of external .so dependencies), or if your platform is not supported, you can try building from the source. What is PyPy? ============= Technically, PyPy is both a Python interpreter implementation and an advanced compiler, or more precisely a framework for implementing dynamic languages and generating virtual machines for them. The focus of this release is the introduction of a new transformation, the JIT Compiler Generator, and its application to the Python interpreter. Socially, PyPy is a collaborative effort of many individuals working together in a distributed and sprint-driven way since 2003. PyPy would not have gotten as far as it has without the coding, feedback and general support from numerous people. The PyPy release team, Armin Rigo, Maciej Fijalkowski and Amaury Forgeot d'Arc Together with Antonio Cuni, Carl Friedrich Bolz, Holger Krekel and Samuele Pedroni and many others: http://codespeak.net/pypy/dist/pypy/doc/contributor.html From collinwinter at google.com Fri Mar 12 20:08:44 2010 From: collinwinter at google.com (Collin Winter) Date: Fri, 12 Mar 2010 11:08:44 -0800 Subject: [Python-Dev] interesting article on regex performance In-Reply-To: <4B9A67E7.30801@gmail.com> References: <4B9A67E7.30801@gmail.com> Message-ID: <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> On Fri, Mar 12, 2010 at 8:12 AM, Nick Coghlan wrote: [snip] > To bring this on-topic for python-dev by considering how it could apply > to Python's default re engine, I think the key issue is that any updates > to the default engine would need to remain backwards compatible with all > of the tricks that re2 doesn't support. > > There are major practical problems associated with making such a leap > directly (Google's re2 engine is in C++ rather than C and we'd have to > keep the existing implementation around regardless to handle the > features that re2 doesn't support). I don't see why C++ would be a deal-breaker in this case, since it would be restricted to an extension module. > I would say it is better to let re2 bake for a while and see if anyone > is motivated to come up with Python bindings for it and release them on > PyPI. FWIW, re2 is heavily, heavily used in production at Google. Stabilizing any proposed Python bindings would be a good idea, but I'm not sure how much more "baking" re2's core functionality needs. > Once that happens (and assuming the bindings earn a good > reputation), the first step towards integration would be to include a > "See Also" in the re module documentation to point people towards the > more limited (but more robust) regex engine implementation. > > The next step would probably be a hybrid third party library that > exploits the NFA approach when it can, but resorts to backtracking when > it has to in order to handle full regex functionality. (Although > developers would need to be able to disable the backtracking support in > order to restore re2's guarantees of linear time execution) We considered such a hybrid approach for Unladen Swallow, but rejected it on the advice of the V8 team [1]: you end up maintaining two totally separate, incompatible regex engines; the hybrid system comes with interesting, possibly unintuitive performance/correctness issues when bailing from one implementation to another; performance is unstable as small changes are made to the regexes; and it's not obvious that enough Python regexes would benefit from re2's performance/restrictions tradeoff to make such a hybrid system worthwhile in the long term. (There is no representative corpus of real-world Python regexes weighted for dynamic execution frequency to use in assessing such tradeoffs empirically like there is for JavaScript.) re2 is very useful when you want to run user-provided regexes and want to protect your backends against pathological/malicious regex input, but I'm not sure how applicable it is to Python. I think there are more promising strategies to improve regex performance, such as reusing the new JIT infrastructure to JIT-compile regular expressions to machine code (along the lines of V8's irregexp). Some work has already been done in this direction, and I'd be thrilled to mentor any GSoC students interested in working on such a project this summer. Lastly, anyone interested in working on Python regex performance should take a look at the regex benchmarks in the standard benchmark suite [2]. Thanks, Collin Winter [1] - http://blog.chromium.org/2009/02/irregexp-google-chromes-new-regexp.html#c4843826268005492354 [2] - http://hg.python.org/benchmarks/file/5b8fe389710b/performance From skip at pobox.com Fri Mar 12 20:29:09 2010 From: skip at pobox.com (skip at pobox.com) Date: Fri, 12 Mar 2010 13:29:09 -0600 Subject: [Python-Dev] interesting article on regex performance In-Reply-To: <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> Message-ID: <19354.38405.67977.218983@montanaro.dyndns.org> >> There are major practical problems associated with making such a leap >> directly (Google's re2 engine is in C++ rather than C and we'd have >> to keep the existing implementation around regardless to handle the >> features that re2 doesn't support). Collin> I don't see why C++ would be a deal-breaker in this case, since Collin> it would be restricted to an extension module. Traditionally Python has run on some (minority) platforms where C++ was unavailable. While the re module is a dynamically linked extension module and thus could be considered "optional", I doubt anybody thinks of it as optional nowadays. It's used in the regression test suite anyway. It would be tough to run unit tests on such minority platforms without it. You'd have to maintain both the current sre implementation and the new re2 implementation for a long while into the future. As I was reading the code I thought, "Great! This stuff is so simple. It's even all written in C." Then I looked at the re2 page. :-( Skip From barry at python.org Fri Mar 12 20:53:12 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 12 Mar 2010 14:53:12 -0500 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <4B8FB6D2.4040501@gmail.com> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com> Message-ID: <20100312145312.1f0a0aa0@heresy.wooz.org> On Mar 04, 2010, at 11:34 PM, Nick Coghlan wrote: >The remaining open question to my mind is whether or not there should be >a -X option to control the bytecode generation. E.g.: > >-Xcache_bytecode=no (don't write bytecode files at all) -B and $PYTHONDONTWRITEBYTECODE will still be supported and serve this use case. >-Xcache_bytecode=file (write a classic "foo.pyc" file) >-Xcache_bytecode=dir (write to the "__pycache__" directory) > >With cache_bytecode=dir being the default for future releases. I believe we've decided /not/ to support creation of bytecode-only distributions out of the box. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From collinwinter at google.com Fri Mar 12 22:40:50 2010 From: collinwinter at google.com (Collin Winter) Date: Fri, 12 Mar 2010 13:40:50 -0800 Subject: [Python-Dev] interesting article on regex performance In-Reply-To: <19354.38405.67977.218983@montanaro.dyndns.org> References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> <19354.38405.67977.218983@montanaro.dyndns.org> Message-ID: <3c8293b61003121340i70e5cca7p602aa03d76d69855@mail.gmail.com> On Fri, Mar 12, 2010 at 11:29 AM, wrote: > > ? ?>> There are major practical problems associated with making such a leap > ? ?>> directly (Google's re2 engine is in C++ rather than C and we'd have > ? ?>> to keep the existing implementation around regardless to handle the > ? ?>> features that re2 doesn't support). > > ? ?Collin> I don't see why C++ would be a deal-breaker in this case, since > ? ?Collin> it would be restricted to an extension module. > > Traditionally Python has run on some (minority) platforms where C++ was > unavailable. ?While the re module is a dynamically linked extension module > and thus could be considered "optional", I doubt anybody thinks of it as > optional nowadays. ?It's used in the regression test suite anyway. ?It would > be tough to run unit tests on such minority platforms without it. ?You'd > have to maintain both the current sre implementation and the new re2 > implementation for a long while into the future. re2 is not a full replacement for Python's current regex semantics: it would only serve as an accelerator for a subset of the current regex language. Given that, it makes perfect sense that it would be optional on such minority platforms (much like the incoming JIT). Collin From solipsis at pitrou.net Fri Mar 12 23:03:08 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 12 Mar 2010 17:03:08 -0500 Subject: [Python-Dev] C++ References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> <19354.38405.67977.218983@montanaro.dyndns.org> Message-ID: <20100312170308.111b90a5@msiwind> Le Fri, 12 Mar 2010 13:29:09 -0600, skip at pobox.com a ?crit : > > Traditionally Python has run on some (minority) platforms where C++ > was unavailable. Is this concern still valid? We are in the 2010s now. I'm not saying I want us to put some C++ in the core interpreter, but the portability argument sounds a little old... Regards Antoine. From barry at python.org Fri Mar 12 23:15:20 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 12 Mar 2010 17:15:20 -0500 Subject: [Python-Dev] C++ In-Reply-To: <20100312170308.111b90a5@msiwind> References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> <19354.38405.67977.218983@montanaro.dyndns.org> <20100312170308.111b90a5@msiwind> Message-ID: <20100312171520.0409e36d@heresy.wooz.org> On Mar 12, 2010, at 05:03 PM, Antoine Pitrou wrote: >Le Fri, 12 Mar 2010 13:29:09 -0600, >skip at pobox.com a ?crit : >> >> Traditionally Python has run on some (minority) platforms where C++ >> was unavailable. > >Is this concern still valid? Certainly not if Unladen Swallow blazes the trail. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From nyamatongwe at gmail.com Fri Mar 12 23:18:41 2010 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Sat, 13 Mar 2010 09:18:41 +1100 Subject: [Python-Dev] C++ In-Reply-To: <20100312170308.111b90a5@msiwind> References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> <19354.38405.67977.218983@montanaro.dyndns.org> <20100312170308.111b90a5@msiwind> Message-ID: <50862ebd1003121418s5e7dd608o31185b42d3fdb468@mail.gmail.com> Antoine Pitrou: > Is this concern still valid? We are in the 2010s now. > I'm not saying I want us to put some C++ in the core interpreter, but > the portability argument sounds a little old... There are still viable platforms which only support subsets of C++. IIRC, Android does not support exceptions in C++. Neil From fuzzyman at voidspace.org.uk Fri Mar 12 23:48:42 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 12 Mar 2010 22:48:42 +0000 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <20100312145312.1f0a0aa0@heresy.wooz.org> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com> <20100312145312.1f0a0aa0@heresy.wooz.org> Message-ID: <4B9AC4CA.3070302@voidspace.org.uk> On 12/03/2010 19:53, Barry Warsaw wrote: > On Mar 04, 2010, at 11:34 PM, Nick Coghlan wrote: > > >> The remaining open question to my mind is whether or not there should be >> a -X option to control the bytecode generation. E.g.: >> >> -Xcache_bytecode=no (don't write bytecode files at all) >> > -B and $PYTHONDONTWRITEBYTECODE will still be supported and serve this use > case. > > >> -Xcache_bytecode=file (write a classic "foo.pyc" file) >> -Xcache_bytecode=dir (write to the "__pycache__" directory) >> >> With cache_bytecode=dir being the default for future releases. >> > I believe we've decided /not/ to support creation of bytecode-only > distributions out of the box. > I thought Guido said on this topic [1]: " FWIW, I started at -1 and am still -1. I think the PEP is overreaching in this aspect; it does not serve the stated purpose of the PEP to make life easier for distros that share code between Python versions." Has something changed since then? Michael [1] http://mail.python.org/pipermail/python-dev/2010-March/098098.html > -Barry > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Fri Mar 12 23:54:19 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 12 Mar 2010 22:54:19 +0000 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <4B9AC4CA.3070302@voidspace.org.uk> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com> <20100312145312.1f0a0aa0@heresy.wooz.org> <4B9AC4CA.3070302@voidspace.org.uk> Message-ID: <4B9AC61B.6010007@voidspace.org.uk> On 12/03/2010 22:48, Michael Foord wrote: > On 12/03/2010 19:53, Barry Warsaw wrote: >> On Mar 04, 2010, at 11:34 PM, Nick Coghlan wrote: >> >> >>> The remaining open question to my mind is whether or not there should be >>> a -X option to control the bytecode generation. E.g.: >>> >>> -Xcache_bytecode=no (don't write bytecode files at all) >>> >> -B and $PYTHONDONTWRITEBYTECODE will still be supported and serve this use >> case. >> >> >>> -Xcache_bytecode=file (write a classic "foo.pyc" file) >>> -Xcache_bytecode=dir (write to the "__pycache__" directory) >>> >>> With cache_bytecode=dir being the default for future releases. >>> >> I believe we've decided /not/ to support creation of bytecode-only >> distributions out of the box. >> > > I thought Guido said on this topic [1]: > Ok - so oops, I misunderstood. I'll go back into my corner now. :-) All the best, Michael > " FWIW, I started at -1 and am still -1. I think the PEP is overreaching > in this aspect; it does not serve the stated purpose of the PEP to > make life easier for distros that share code between Python versions." > > Has something changed since then? > > Michael > > [1] http://mail.python.org/pipermail/python-dev/2010-March/098098.html >> -Barry >> >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe:http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk >> > > > -- > http://www.ironpythoninaction.com/ > http://www.voidspace.org.uk/blog > > READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Mar 12 23:56:57 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 12 Mar 2010 17:56:57 -0500 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <4B9AC4CA.3070302@voidspace.org.uk> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com> <20100312145312.1f0a0aa0@heresy.wooz.org> <4B9AC4CA.3070302@voidspace.org.uk> Message-ID: <20100312175657.287bb456@heresy.wooz.org> On Mar 12, 2010, at 10:48 PM, Michael Foord wrote: >On 12/03/2010 19:53, Barry Warsaw wrote: >> I believe we've decided /not/ to support creation of bytecode-only >> distributions out of the box. >> > >I thought Guido said on this topic [1]: > >" FWIW, I started at -1 and am still -1. I think the PEP is overreaching >in this aspect; it does not serve the stated purpose of the PEP to >make life easier for distros that share code between Python versions." > >Has something changed since then? Nope, sorry I should have been clearer. *creation* is the key here. As per BDFL pronouncement, we'll support reading pyc-only modules just like we do today. This is in PEP 3147. We won't support creating them though. BTW, I'm actually starting to work on the implementation now, so if you want to play along: % bzr branch lp:~barry/python/pep3147 -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From jyasskin at gmail.com Fri Mar 12 23:59:16 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Fri, 12 Mar 2010 14:59:16 -0800 Subject: [Python-Dev] C++ In-Reply-To: <50862ebd1003121418s5e7dd608o31185b42d3fdb468@mail.gmail.com> References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> <19354.38405.67977.218983@montanaro.dyndns.org> <20100312170308.111b90a5@msiwind> <50862ebd1003121418s5e7dd608o31185b42d3fdb468@mail.gmail.com> Message-ID: <5d44f72f1003121459u6b91f1dbwe0bc33c2011b3f6b@mail.gmail.com> On Fri, Mar 12, 2010 at 2:18 PM, Neil Hodgson wrote: > Antoine Pitrou: > >> Is this concern still valid? We are in the 2010s now. >> I'm not saying I want us to put some C++ in the core interpreter, but >> the portability argument sounds a little old... > > ? There are still viable platforms which only support subsets of C++. > IIRC, Android does not support exceptions in C++. Looks like they'll be getting exceptions "soon": http://groups.google.com/group/android-ndk/browse_thread/thread/89db67ed1fbf6450/4de3dd6105eb26ce?#4de3dd6105eb26ce But yeah, thanks for the concrete example, and I'd agree that Python should compile with -fno-exceptions, for a couple more years at least. From g.brandl at gmx.net Sat Mar 13 00:23:11 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 13 Mar 2010 00:23:11 +0100 Subject: [Python-Dev] interesting article on regex performance In-Reply-To: <19354.38405.67977.218983@montanaro.dyndns.org> References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> <19354.38405.67977.218983@montanaro.dyndns.org> Message-ID: Am 12.03.2010 20:29, schrieb skip at pobox.com: > > >> There are major practical problems associated with making such a leap > >> directly (Google's re2 engine is in C++ rather than C and we'd have > >> to keep the existing implementation around regardless to handle the > >> features that re2 doesn't support). > > Collin> I don't see why C++ would be a deal-breaker in this case, since > Collin> it would be restricted to an extension module. > > Traditionally Python has run on some (minority) platforms where C++ was > unavailable. While the re module is a dynamically linked extension module > and thus could be considered "optional", I doubt anybody thinks of it as > optional nowadays. Actually, _sre is a builtin module by default since at least Python 2.5. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From skip at pobox.com Sat Mar 13 00:22:13 2010 From: skip at pobox.com (skip at pobox.com) Date: Fri, 12 Mar 2010 17:22:13 -0600 Subject: [Python-Dev] interesting article on regex performance In-Reply-To: <3c8293b61003121340i70e5cca7p602aa03d76d69855@mail.gmail.com> References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> <19354.38405.67977.218983@montanaro.dyndns.org> <3c8293b61003121340i70e5cca7p602aa03d76d69855@mail.gmail.com> Message-ID: <19354.52389.642682.376209@montanaro.dyndns.org> Collin> re2 is not a full replacement for Python's current regex Collin> semantics: it would only serve as an accelerator for a subset of Collin> the current regex language. Given that, it makes perfect sense Collin> that it would be optional on such minority platforms (much like Collin> the incoming JIT). Sure, but over the years Python has supported at least four different regular expression modules that I'm aware of (regex, regexp, and the current re module with different extension modules underneath it, perhaps there were others). During some of that time more than one module was distributed with Python proper. I think the desire today would be that only one regular expression module be distributed with Python (that would be my vote anyway). Getting people to move off the older libraries was difficult. If re2 can't replace sre under the covers than I think it belongs in PyPI, not the Python distribution. That said, that suggests to me that a different NFA or DFA implementation written in C would replace sre, one not written in C++. Hopefully that provides some context for my earlier response. Skip From jared.grubb at gmail.com Sat Mar 13 02:59:12 2010 From: jared.grubb at gmail.com (Jared Grubb) Date: Fri, 12 Mar 2010 17:59:12 -0800 Subject: [Python-Dev] interesting article on regex performance In-Reply-To: <19354.52389.642682.376209@montanaro.dyndns.org> References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> <19354.38405.67977.218983@montanaro.dyndns.org> <3c8293b61003121340i70e5cca7p602aa03d76d69855@mail.gmail.com> <19354.52389.642682.376209@montanaro.dyndns.org> Message-ID: On 12 Mar 2010, at 15:22, skip at pobox.com wrote: > > Collin> re2 is not a full replacement for Python's current regex > Collin> semantics: it would only serve as an accelerator for a subset of > Collin> the current regex language. Given that, it makes perfect sense > Collin> that it would be optional on such minority platforms (much like > Collin> the incoming JIT). > > Sure, but over the years Python has supported at least four different > regular expression modules that I'm aware of (regex, regexp, and the current > re module with different extension modules underneath it, perhaps there were > others). During some of that time more than one module was distributed with > Python proper. I think the desire today would be that only one regular > expression module be distributed with Python (that would be my vote anyway). > Getting people to move off the older libraries was difficult. If re2 can't > replace sre under the covers than I think it belongs in PyPI, not the Python > distribution. That said, that suggests to me that a different NFA or DFA > implementation written in C would replace sre, one not written in C++. re2 would be a supplement to re -- it is not a replacement, and Python would run fine if it's not present on some platforms. It's like a floating-point processor: you can do all math you need with just an integer processor. But if you have an FPU present, then it makes sense to use it for the FP operations. Jared From meadori at gmail.com Sat Mar 13 04:17:05 2010 From: meadori at gmail.com (Meador Inge) Date: Fri, 12 Mar 2010 21:17:05 -0600 Subject: [Python-Dev] C++ In-Reply-To: <20100312170308.111b90a5@msiwind> References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> <19354.38405.67977.218983@montanaro.dyndns.org> <20100312170308.111b90a5@msiwind> Message-ID: <4095897c1003121917m36e9cd12je0965c7c3a99c615@mail.gmail.com> On Fri, Mar 12, 2010 at 4:03 PM, Antoine Pitrou wrote: > Le Fri, 12 Mar 2010 13:29:09 -0600, > skip at pobox.com a ?crit : > > > > Traditionally Python has run on some (minority) platforms where C++ > > was unavailable. > > Is this concern still valid? We are in the 2010s now. > I'm not saying I want us to put some C++ in the core interpreter, but > the portability argument sounds a little old... > One area where this _may_ be a problem is with embedded systems. I believe there are some instances where folks have built Python into an embedded system (with an RTOS say VxWorks, Symbian, QNX Neutrino, Nucleus, etc...) where C++ is not always the easiest to develop in. Admittedly, though, these types of systems are by far a minority with respect to Python. Just thought I would mention it anyhow. -- Meador -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Sat Mar 13 04:54:35 2010 From: skip at pobox.com (skip at pobox.com) Date: Fri, 12 Mar 2010 21:54:35 -0600 Subject: [Python-Dev] C++ In-Reply-To: <20100312170308.111b90a5@msiwind> References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> <19354.38405.67977.218983@montanaro.dyndns.org> <20100312170308.111b90a5@msiwind> Message-ID: <19355.3195.511985.327292@montanaro.dyndns.org> Antoine> skip at pobox.com a ??crit : >> >> Traditionally Python has run on some (minority) platforms where C++ >> was unavailable. Antoine> Is this concern still valid? We are in the 2010s now. Like I said, *minority* platforms. Here are some which come to mind as quite possibly not supporting C++ well, if at all, yet all have some dialect of Python available: Windows CE: http://sourceforge.net/projects/pythonce/ Palm: http://pippy.sourceforge.net/ iPod: http://ciberjacobo.com/python-ipod OS/2: http://www.andymac.org/ QNX: http://sourceforge.net/projects/pyqnx/ If all you care about are mainstream Windows, Unix/Linux and Mac OSX environments then C++ is no problem. Skip From jyasskin at gmail.com Sat Mar 13 07:54:59 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Fri, 12 Mar 2010 22:54:59 -0800 Subject: [Python-Dev] C++ In-Reply-To: <19355.3195.511985.327292@montanaro.dyndns.org> References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> <19354.38405.67977.218983@montanaro.dyndns.org> <20100312170308.111b90a5@msiwind> <19355.3195.511985.327292@montanaro.dyndns.org> Message-ID: <5d44f72f1003122254s4c015780ia7c72a1cab48f5ba@mail.gmail.com> On Fri, Mar 12, 2010 at 7:54 PM, wrote: > ? ?Antoine> skip at pobox.com a ??crit : > ? ?>> > ? ?>> Traditionally Python has run on some (minority) platforms where C++ > ? ?>> was unavailable. > > ? ?Antoine> Is this concern still valid? We are in the 2010s now. > > Like I said, *minority* platforms. ?Here are some which come to mind as > quite possibly not supporting C++ well, if at all, yet all have some dialect > of Python available: > > ? ?Windows CE: http://sourceforge.net/projects/pythonce/ http://www.wirelessdevnet.com/channels/pda/training/vcce.html > ? ?Palm: http://pippy.sourceforge.net/ http://prc-tools.sourceforge.net/ > ? ?iPod: http://ciberjacobo.com/python-ipod http://ipodlinux.org/wiki/Toolchain > ? ?OS/2: http://www.andymac.org/ I can't find a modern c++ compiler for OS/2. Then again, your link only provides python 2.4. > ? ?QNX: http://sourceforge.net/projects/pyqnx/ http://www.qnx.com/products/tools/ > If all you care about are mainstream Windows, Unix/Linux and Mac OSX > environments then C++ is no problem. If you care about any of these minority platforms except OS/2, C++ is also available, as a google search for " C++" quickly finds. OS/2 might be available too; I just didn't look for very long. If you know of platforms that don't support particular features of C++, please link to documentation of that like Neil Hodgson did. If not, please stop spreading FUD. From g.brandl at gmx.net Sat Mar 13 14:04:08 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 13 Mar 2010 14:04:08 +0100 Subject: [Python-Dev] Desired changes to Hg emails to python-checkins In-Reply-To: References: Message-ID: Am 04.03.2010 21:20, schrieb Brett Cannon: > 1) I miss not having the affected files listed in the subject line. > > 2) The To field is set to hg at python.org which > gets rejected as an invalid email address if you reply. Would be better > to set it to python-checkins so that the habitual reply to a checkin > won't get rejected. Before I forget: for CPython, if all branches are to be in one hg repo with named branches, the branch name also needs to be in the subject line. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From ncoghlan at gmail.com Sat Mar 13 14:59:57 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 13 Mar 2010 23:59:57 +1000 Subject: [Python-Dev] interesting article on regex performance In-Reply-To: <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> Message-ID: <4B9B9A5D.5090800@gmail.com> Collin Winter wrote: > On Fri, Mar 12, 2010 at 8:12 AM, Nick Coghlan wrote: >> There are major practical problems associated with making such a leap >> directly (Google's re2 engine is in C++ rather than C and we'd have to >> keep the existing implementation around regardless to handle the >> features that re2 doesn't support). > > I don't see why C++ would be a deal-breaker in this case, since it > would be restricted to an extension module. It wouldn't be a dealbreaker for an optional regex acceleration module, I agree. It's still a practical problem in the sense that C++ code is very much in the minority in the CPython code base. >> I would say it is better to let re2 bake for a while and see if anyone >> is motivated to come up with Python bindings for it and release them on >> PyPI. > > FWIW, re2 is heavily, heavily used in production at Google. > Stabilizing any proposed Python bindings would be a good idea, but I'm > not sure how much more "baking" re2's core functionality needs. It was more the bindings side of things I was referring to - the algorithm's performance is no doubt well established, but how easy it is for non-Google users to pick up and integrate with other software is still to be determined. > We considered such a hybrid approach for Unladen Swallow, but rejected > it on the advice of the V8 team [1]: you end up maintaining two > totally separate, incompatible regex engines; the hybrid system comes > with interesting, possibly unintuitive performance/correctness issues > when bailing from one implementation to another; performance is > unstable as small changes are made to the regexes; and it's not > obvious that enough Python regexes would benefit from re2's > performance/restrictions tradeoff to make such a hybrid system > worthwhile in the long term. (There is no representative corpus of > real-world Python regexes weighted for dynamic execution frequency to > use in assessing such tradeoffs empirically like there is for > JavaScript.) > > re2 is very useful when you want to run user-provided regexes and want > to protect your backends against pathological/malicious regex input, > but I'm not sure how applicable it is to Python. I think there are > more promising strategies to improve regex performance, such as > reusing the new JIT infrastructure to JIT-compile regular expressions > to machine code (along the lines of V8's irregexp). Some work has > already been done in this direction, and I'd be thrilled to mentor any > GSoC students interested in working on such a project this summer. > > Lastly, anyone interested in working on Python regex performance > should take a look at the regex benchmarks in the standard benchmark > suite [2]. This is the kind of baking that I was really talking about - there's an awful lot of investigation to be done before moving Python in the direction of an NFA based regex engine could be seriously considered. Good to hear the perspective of someone that has already looked into this in some detail though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ndbecker2 at gmail.com Sat Mar 13 15:00:33 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Sat, 13 Mar 2010 09:00:33 -0500 Subject: [Python-Dev] C++ References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> <19354.38405.67977.218983@montanaro.dyndns.org> <20100312170308.111b90a5@msiwind> <50862ebd1003121418s5e7dd608o31185b42d3fdb468@mail.gmail.com> Message-ID: Neil Hodgson wrote: > Antoine Pitrou: > >> Is this concern still valid? We are in the 2010s now. >> I'm not saying I want us to put some C++ in the core interpreter, but >> the portability argument sounds a little old... > > There are still viable platforms which only support subsets of C++. > IIRC, Android does not support exceptions in C++. > > Neil Does re2 require exceptions? From ncoghlan at gmail.com Sat Mar 13 15:17:09 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Mar 2010 00:17:09 +1000 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <20100312175657.287bb456@heresy.wooz.org> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com> <20100312145312.1f0a0aa0@heresy.wooz.org> <4B9AC4CA.3070302@voidspace.org.uk> <20100312175657.287bb456@heresy.wooz.org> Message-ID: <4B9B9E65.4080902@gmail.com> Barry Warsaw wrote: > *creation* is the key here. As per BDFL pronouncement, we'll support reading > pyc-only modules just like we do today. This is in PEP 3147. We won't > support creating them though. The creation side could be made a little more explicit in the PEP. We could also do something via the compileall module. (Pause while Nick goes and reads the source code for compileall for the first time ever...) Hmm - methinks the PEP actually needs to talk explicitly about the py_compile and compileall modules. These compile the files directly rather than using the import system's side-effect, so they'll need to understand the intricacies of the new system. While it's probably OK if the import side-effects only create files using the new scheme, the standard library modules will likely need to support both schemes (although I'm not sure if "same as import system" or "same as Python 3.1" make more sense as the default semantics - probably the former). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From skip at pobox.com Sat Mar 13 18:07:49 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 13 Mar 2010 11:07:49 -0600 Subject: [Python-Dev] C++ In-Reply-To: <5d44f72f1003122254s4c015780ia7c72a1cab48f5ba@mail.gmail.com> References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> <19354.38405.67977.218983@montanaro.dyndns.org> <20100312170308.111b90a5@msiwind> <19355.3195.511985.327292@montanaro.dyndns.org> <5d44f72f1003122254s4c015780ia7c72a1cab48f5ba@mail.gmail.com> Message-ID: <19355.50789.388095.93289@montanaro.dyndns.org> Jeffrey> If you know of platforms that don't support particular features Jeffrey> of C++, please link to documentation of that like Neil Hodgson Jeffrey> did. If not, please stop spreading FUD. No need to get your knickers in a twist. Every couple years someone asks, "why isn't Python written in C++?" The lack of good C++ support for many (minority) platforms is often the primary explanation. If that's changing, I have no problem with it. I spent a fair amount of time with Google trying to find representative threads here or in comp.lang.python but failed to find any needles in the haystack. Perhaps you will have better luck. Skip From dudek at cim.mcgill.ca Sat Mar 13 18:24:27 2010 From: dudek at cim.mcgill.ca (gregory dudek) Date: Sat, 13 Mar 2010 12:24:27 -0500 Subject: [Python-Dev] Patch to telnetlib.py Message-ID: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca> The Telnet module telnetlib.py can be very slow -- unusably slow -- for large automated data transfers. There are typically done in raw mode. The attached patch greatly increased the speed of telnet interactions in raw mode. I submitted this a couple of year ago, but it was for an older branch of python. There are 2 key things being done: 1) concatenations string with string.join instead of '+' (which is probably a minor issue) 2) wholesale appending the raw and processed buffers when the IAC character is not found. The should be examined carefully since I am not an expert in the Telnet protocol, but it seems to work very well giving me a 5x speedup. --- installed/telnetlib.py 2010-02-02 22:57:58.000000000 -0500 +++ telnetlib.py 2010-03-13 12:17:02.000000000 -0500 @@ -30,6 +30,7 @@ - timeout should be intrinsic to the connection object instead of an option on one of the read calls only +Modified by G. Dudek for greater efficiency. """ @@ -420,6 +421,14 @@ """ buf = ['', ''] try: + if self.rawq: + if not IAC in self.rawq: + # speed hack, no IAC just grab whole queue. --Dudek + buf[self.sb] = "".join((buf[self.sb] , self.rawq)) + self.cookedq = "".join((self.cookedq , buf[0] )) + self.sbdataq = "".join((self.sbdataq , buf[1] )) + self.rawq_flush() + return while self.rawq: c = self.rawq_getchar() if not self.iacseq: @@ -428,7 +437,7 @@ if c == "\021": continue if c != IAC: - buf[self.sb] = buf[self.sb] + c + buf[self.sb] = "".join((buf[self.sb] , c)) continue else: self.iacseq += c @@ -480,8 +489,14 @@ self.iacseq = '' # Reset on EOF self.sb = 0 pass - self.cookedq = self.cookedq + buf[0] - self.sbdataq = self.sbdataq + buf[1] + self.cookedq = "".join((self.cookedq , buf[0] )) + self.sbdataq = "".join((self.sbdataq , buf[1] )) + + + def rawq_flush(self): + """ Set the queue to empty status """ + self.rawq = '' + self.irawq = 0 def rawq_getchar(self): """Get next char from raw queue. @@ -516,7 +531,7 @@ buf = self.sock.recv(50) self.msg("recv %r", buf) self.eof = (not buf) - self.rawq = self.rawq + buf + self.rawq = "".join((self.rawq,buf)) def sock_avail(self): """Test whether data is available on the socket.""" From tjreedy at udel.edu Sat Mar 13 19:22:56 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 13 Mar 2010 13:22:56 -0500 Subject: [Python-Dev] Patch to telnetlib.py In-Reply-To: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca> References: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca> Message-ID: On 3/13/2010 12:24 PM, gregory dudek wrote: > The Telnet module telnetlib.py can be > very slow -- unusably slow -- for large automated data transfers. There are typically done in raw mode. > > The attached patch Please submit to the tracker. If there is an existing issue, attach it to that. From eric at trueblade.com Sat Mar 13 19:23:16 2010 From: eric at trueblade.com (Eric Smith) Date: Sat, 13 Mar 2010 13:23:16 -0500 Subject: [Python-Dev] Patch to telnetlib.py In-Reply-To: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca> References: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca> Message-ID: <4B9BD814.50806@trueblade.com> Can you create an issue on the bug tracker? Otherwise this will get lost. Eric. On 3/13/2010 12:24 PM, gregory dudek wrote: > The Telnet module telnetlib.py can be > very slow -- unusably slow -- for large automated data transfers. There are typically done in raw mode. > > The attached patch greatly increased the speed of telnet interactions in raw mode. I submitted this a couple of year ago, but it was for an older branch of python. > > There are 2 key things being done: > 1) concatenations string with string.join instead of '+' (which is probably a minor issue) > 2) wholesale appending the raw and processed buffers when the IAC character is not found. The should be examined > carefully since I am not an expert in the Telnet protocol, but it seems to work very well giving me a 5x speedup. > > > --- installed/telnetlib.py 2010-02-02 22:57:58.000000000 -0500 > +++ telnetlib.py 2010-03-13 12:17:02.000000000 -0500 > @@ -30,6 +30,7 @@ > - timeout should be intrinsic to the connection object instead of an > option on one of the read calls only > > +Modified by G. Dudek for greater efficiency. > """ > > > @@ -420,6 +421,14 @@ > """ > buf = ['', ''] > try: > + if self.rawq: > + if not IAC in self.rawq: > + # speed hack, no IAC just grab whole queue. --Dudek > + buf[self.sb] = "".join((buf[self.sb] , self.rawq)) > + self.cookedq = "".join((self.cookedq , buf[0] )) > + self.sbdataq = "".join((self.sbdataq , buf[1] )) > + self.rawq_flush() > + return > while self.rawq: > c = self.rawq_getchar() > if not self.iacseq: > @@ -428,7 +437,7 @@ > if c == "\021": > continue > if c != IAC: > - buf[self.sb] = buf[self.sb] + c > + buf[self.sb] = "".join((buf[self.sb] , c)) > continue > else: > self.iacseq += c > @@ -480,8 +489,14 @@ > self.iacseq = '' # Reset on EOF > self.sb = 0 > pass > - self.cookedq = self.cookedq + buf[0] > - self.sbdataq = self.sbdataq + buf[1] > + self.cookedq = "".join((self.cookedq , buf[0] )) > + self.sbdataq = "".join((self.sbdataq , buf[1] )) > + > + > + def rawq_flush(self): > + """ Set the queue to empty status """ > + self.rawq = '' > + self.irawq = 0 > > def rawq_getchar(self): > """Get next char from raw queue. > @@ -516,7 +531,7 @@ > buf = self.sock.recv(50) > self.msg("recv %r", buf) > self.eof = (not buf) > - self.rawq = self.rawq + buf > + self.rawq = "".join((self.rawq,buf)) > > def sock_avail(self): > """Test whether data is available on the socket.""" > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com > From guido at python.org Sat Mar 13 19:22:59 2010 From: guido at python.org (Guido van Rossum) Date: Sat, 13 Mar 2010 10:22:59 -0800 Subject: [Python-Dev] Patch to telnetlib.py In-Reply-To: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca> References: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca> Message-ID: On Sat, Mar 13, 2010 at 9:24 AM, gregory dudek wrote: > The Telnet module telnetlib.py can be > very slow -- unusably slow -- for large automated data transfers. ?There are typically done in raw mode. > > The attached patch greatly increased the speed of telnet interactions in raw mode. ?I submitted this a couple of year ago, but it was for an older branch of python. For which Python version is your new patch? > There are 2 key things being done: > ?1) concatenations string with string.join instead of '+' ?(which is probably a minor issue) This looks like superstition in the patch below -- there is no reason why "".join((a, b)) should be any faster than a+b. (The join trick is important when joining *many* strings, but for two it doesn't make a difference, both strings have to be copied at least once no matter what you try.) > ?2) wholesale appending the raw and processed buffers when the IAC character is not found. ?The should be examined > ? ? carefully since I am not an expert in the Telnet protocol, but it seems to work very well giving me a 5x speedup. This makes sense and is likely the reason your patch is faster. You should really submit this to the bug tracker at bugs.python.org. > --- installed/telnetlib.py ? ? ?2010-02-02 22:57:58.000000000 -0500 > +++ telnetlib.py ? ? ? ?2010-03-13 12:17:02.000000000 -0500 > @@ -30,6 +30,7 @@ > ?- timeout should be intrinsic to the connection object instead of an > ? option on one of the read calls only > > +Modified by G. Dudek for greater efficiency. You're gonna need to submit a Python contributor agreement (http://www.python.org/psf/contrib/contrib-form/) if/when your patch gets accepted. Welcome to the club! > ?""" > > > @@ -420,6 +421,14 @@ > ? ? ? ? """ > ? ? ? ? buf = ['', ''] > ? ? ? ? try: > + ? ? ? ? ? ?if self.rawq: > + ? ? ? ? ? ? ? ?if not IAC in self.rawq: > + ? ? ? ? ? ? ? ? ? ?# speed hack, no IAC just grab whole queue. --Dudek > + ? ? ? ? ? ? ? ? ? ?buf[self.sb] = "".join((buf[self.sb] , self.rawq)) > + ? ? ? ? ? ? ? ? ? ?self.cookedq = "".join((self.cookedq , buf[0] )) > + ? ? ? ? ? ? ? ? ? ?self.sbdataq = "".join((self.sbdataq , buf[1] )) > + ? ? ? ? ? ? ? ? ? ?self.rawq_flush() > + ? ? ? ? ? ? ? ? ? ?return > ? ? ? ? ? ? while self.rawq: > ? ? ? ? ? ? ? ? c = self.rawq_getchar() > ? ? ? ? ? ? ? ? if not self.iacseq: > @@ -428,7 +437,7 @@ > ? ? ? ? ? ? ? ? ? ? if c == "\021": > ? ? ? ? ? ? ? ? ? ? ? ? continue > ? ? ? ? ? ? ? ? ? ? if c != IAC: > - ? ? ? ? ? ? ? ? ? ? ? ?buf[self.sb] = buf[self.sb] + c > + ? ? ? ? ? ? ? ? ? ? ? ?buf[self.sb] = "".join((buf[self.sb] , c)) > ? ? ? ? ? ? ? ? ? ? ? ? continue > ? ? ? ? ? ? ? ? ? ? else: > ? ? ? ? ? ? ? ? ? ? ? ? self.iacseq += c > @@ -480,8 +489,14 @@ > ? ? ? ? ? ? self.iacseq = '' # Reset on EOF > ? ? ? ? ? ? self.sb = 0 > ? ? ? ? ? ? pass > - ? ? ? ?self.cookedq = self.cookedq + buf[0] > - ? ? ? ?self.sbdataq = self.sbdataq + buf[1] > + ? ? ? ?self.cookedq = "".join((self.cookedq , buf[0] )) > + ? ? ? ?self.sbdataq = "".join((self.sbdataq , buf[1] )) > + > + > + ? ?def rawq_flush(self): > + ? ? ? ?""" Set the queue to empty status """ > + ? ? ? ?self.rawq = '' > + ? ? ? ?self.irawq = 0 > > ? ? def rawq_getchar(self): > ? ? ? ? """Get next char from raw queue. > @@ -516,7 +531,7 @@ > ? ? ? ? buf = self.sock.recv(50) > ? ? ? ? self.msg("recv %r", buf) > ? ? ? ? self.eof = (not buf) > - ? ? ? ?self.rawq = self.rawq + buf > + ? ? ? ?self.rawq = "".join((self.rawq,buf)) > > ? ? def sock_avail(self): > ? ? ? ? """Test whether data is available on the socket.""" > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From greg at krypto.org Sat Mar 13 19:42:22 2010 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 13 Mar 2010 10:42:22 -0800 Subject: [Python-Dev] C++ In-Reply-To: References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> <19354.38405.67977.218983@montanaro.dyndns.org> <20100312170308.111b90a5@msiwind> <50862ebd1003121418s5e7dd608o31185b42d3fdb468@mail.gmail.com> Message-ID: <52dc1c821003131042i4e3ec141g84155a1c4a85a855@mail.gmail.com> On Sat, Mar 13, 2010 at 6:00 AM, Neal Becker wrote: > Neil Hodgson wrote: > >> Antoine Pitrou: >> >>> Is this concern still valid? We are in the 2010s now. >>> I'm not saying I want us to put some C++ in the core interpreter, but >>> the portability argument sounds a little old... >> >> ? ?There are still viable platforms which only support subsets of C++. >> IIRC, Android does not support exceptions in C++. >> >> ? ?Neil > > Does re2 require exceptions? Nope. No exceptions. It was written following this style guide: http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Exceptions From jackdied at gmail.com Sat Mar 13 19:47:58 2010 From: jackdied at gmail.com (Jack Diederich) Date: Sat, 13 Mar 2010 13:47:58 -0500 Subject: [Python-Dev] Patch to telnetlib.py In-Reply-To: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca> References: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca> Message-ID: On Sat, Mar 13, 2010 at 12:24 PM, gregory dudek wrote: > The Telnet module telnetlib.py can be > very slow -- unusably slow -- for large automated data transfers. ?There are typically done in raw mode. > > The attached patch greatly increased the speed of telnet interactions in raw mode. ?I submitted this a couple of year ago, but it was for an older branch of python. > > There are 2 key things being done: > ?1) concatenations string with string.join instead of '+' ?(which is probably a minor issue) > ?2) wholesale appending the raw and processed buffers when the IAC character is not found. ?The should be examined > ? ? carefully since I am not an expert in the Telnet protocol, but it seems to work very well giving me a 5x speedup. As others mentioned, please post the bug to the tracker. Also, please assign the patch to me ("jackdied") and mention the previous bug number - I thought I had reviewed all the telnetlib bugs including those that were closed WONTFIX. Thanks for the kick in the pants, I have a whole new inner loop for data processing but I haven't applied it. I've been adding unit tests to the module so I could be sure I wouldn't break anything but never finished the job. -Jack From solipsis at pitrou.net Sat Mar 13 22:46:51 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 13 Mar 2010 16:46:51 -0500 Subject: [Python-Dev] "Fixing" the new GIL Message-ID: <20100313164651.7eb322b1@msiwind> Hello, As some of you may know, Dave Beazley recently exhibited a situation where the new GIL shows quite a poor behaviour (the old GIL isn't very good either, but still a little better). This issue is followed in http://bugs.python.org/issue7946 This situation is when an IO-bound thread wants to process a lot of incoming packets, while one (or several) CPU-bound thread is also running. Each time the IO-bound thread releases the GIL, the CPU-bound thread gets it and keeps holding it for at least 5 milliseconds (default setting), which limits the number of individual packets which can be recv()'ed and processed per second. I have proposed two mechanisms, based on the same idea: IO-bound threads should be able to steal the GIL very quickly, rather than having to wait for the whole "thread switching interval" (again, 5 ms by default). They differ in how they detect an "IO-bound threads": - the first mechanism is actually the same mechanism which was embodied in the original new GIL patch before being removed. In this approach, IO methods (such as socket.read() in socketmodule.c) releasing the GIL must use a separate C macro when trying to get the GIL back again. - the second mechanism dynamically computes the "interactiveness" of a thread and allows interactive threads to steal the GIL quickly. In this approach, IO methods don't have to be modified at all. Both approaches show similar benchmark results (for the benchmarks that I know of) and basically fix the issue put forward by Dave Beazley. Any thoughts? Regards Antoine. From benjamin at python.org Sat Mar 13 22:52:03 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 13 Mar 2010 15:52:03 -0600 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <20100313164651.7eb322b1@msiwind> References: <20100313164651.7eb322b1@msiwind> Message-ID: <1afaf6161003131352n3a7bd423w75a8a36ce9ba7a70@mail.gmail.com> 2010/3/13 Antoine Pitrou : > Any thoughts? \ The latter solution seems best to me as it would help any 3rd party IO libraries and require less code modification. -- Regards, Benjamin From martin at v.loewis.de Sat Mar 13 23:03:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 13 Mar 2010 23:03:07 +0100 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <20100313164651.7eb322b1@msiwind> References: <20100313164651.7eb322b1@msiwind> Message-ID: <4B9C0B9B.3080805@v.loewis.de> > Any thoughts? My initial reaction to this report, and still my current opinion is: This issue is not a problem. It's a boundary case, so it's not clear whether Python should be able to deal with it gracefully. I doubt it's a realistic case. Regards, Martin From daniel at stutzbachenterprises.com Sat Mar 13 23:18:23 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Sat, 13 Mar 2010 16:18:23 -0600 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <20100313164651.7eb322b1@msiwind> References: <20100313164651.7eb322b1@msiwind> Message-ID: On Sat, Mar 13, 2010 at 3:46 PM, Antoine Pitrou wrote: > - the second mechanism dynamically computes the "interactiveness" of a > thread and allows interactive threads to steal the GIL quickly. In > this approach, IO methods don't have to be modified at all. > I like the second approach as well, assuming "interactiveness" can be computed cheaply. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Sun Mar 14 00:11:41 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 13 Mar 2010 17:11:41 -0600 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <20100313164651.7eb322b1@msiwind> References: <20100313164651.7eb322b1@msiwind> Message-ID: <19356.7085.340102.469770@montanaro.dyndns.org> If the 5ms interval is too long would it be possible to adaptively reduce it in this situation? How would you detect it? I assume this would be akin to your interactiveness computation. Skip From solipsis at pitrou.net Sun Mar 14 00:23:00 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 13 Mar 2010 18:23:00 -0500 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <19356.7085.340102.469770@montanaro.dyndns.org> References: <20100313164651.7eb322b1@msiwind> <19356.7085.340102.469770@montanaro.dyndns.org> Message-ID: <20100313182300.4f5376a0@msiwind> Le Sat, 13 Mar 2010 17:11:41 -0600, skip at pobox.com a ?crit : > > If the 5ms interval is too long would it be possible to adaptively > reduce it in this situation? How would you detect it? I assume this > would be akin to your interactiveness computation. I think modulating the interval is what Dave Beazley's own patch proposal does. It has a full-blown priority system. It is more complicated than either of the two approaches I am more proposing, though. (also, per Dave's own admission, his patch is a proof-of-concept rather than a finished piece of work) From brett at python.org Sun Mar 14 01:53:00 2010 From: brett at python.org (Brett Cannon) Date: Sat, 13 Mar 2010 16:53:00 -0800 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <1afaf6161003131352n3a7bd423w75a8a36ce9ba7a70@mail.gmail.com> References: <20100313164651.7eb322b1@msiwind> <1afaf6161003131352n3a7bd423w75a8a36ce9ba7a70@mail.gmail.com> Message-ID: On Sat, Mar 13, 2010 at 13:52, Benjamin Peterson wrote: > 2010/3/13 Antoine Pitrou : > > Any thoughts? > \ > The latter solution seems best to me as it would help any 3rd party IO > libraries and require less code modification. > Plus the interactiveness approach has been tested by OS thread schedulers for years and is shown to work. So I vote for the latter approach as well. -Brett > > > > -- > Regards, > Benjamin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Mar 14 04:32:51 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Mar 2010 13:32:51 +1000 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: References: <20100313164651.7eb322b1@msiwind> <1afaf6161003131352n3a7bd423w75a8a36ce9ba7a70@mail.gmail.com> Message-ID: <4B9C58E3.6000305@gmail.com> Brett Cannon wrote: > > > On Sat, Mar 13, 2010 at 13:52, Benjamin Peterson > wrote: > > 2010/3/13 Antoine Pitrou >: > > Any thoughts? > \ > The latter solution seems best to me as it would help any 3rd party IO > libraries and require less code modification. > > > Plus the interactiveness approach has been tested by OS thread > schedulers for years and is shown to work. So I vote for the latter > approach as well. +1 Cheap to calculate and means developer don't need to guess if they should be using the "I am CPU bound" or the "I am IO bound" GIL acquisition macro. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From cs at zip.com.au Sun Mar 14 06:16:34 2010 From: cs at zip.com.au (Cameron Simpson) Date: Sun, 14 Mar 2010 16:16:34 +1100 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <4B9C0B9B.3080805@v.loewis.de> References: <4B9C0B9B.3080805@v.loewis.de> Message-ID: <20100314051634.GA9503@cskk.homeip.net> On 13Mar2010 23:03, Martin v. L?wis wrote: | > Any thoughts? | | My initial reaction to this report, and still my current opinion is: | This issue is not a problem. | It's a boundary case, so it's not clear whether Python should be able to | deal with it gracefully. I doubt it's a realistic case. How was it tripped over? Speaking for myself, I have an app with a daemon mode which I expect will sometimes behave as described; it answers requests and thus has I/O bound threads waiting for requests and dispatching replies, and threads doing data handling, which make constant use of the zlib library. On the client side the same app is often throughput bound by a data examination process that is compute bound; I can easily see it having compute bound threads and I/O bound threads talking to daemon instances. Currently all the above is somewhat "batchy"; the client end looks like an archiving command line tool, but it's intended to have a FUSE mode to present the archive as a filesystem; that I can imagine tripping over this issue as a user uses the filesystem for "stuff". Still, it is unusual and I suspect it will self limit to an extent anyway; I can't fairly claim to have had it cause me trouble yet. (Indeed, I've not used the new GIL at all - it's mostly using python 2.6). Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ I very strongly suggest that you periodically place ice packs over the abused areas. - Steve Garnier From nir at winpdb.org Sun Mar 14 12:31:51 2010 From: nir at winpdb.org (Nir Aides) Date: Sun, 14 Mar 2010 13:31:51 +0200 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <20100313164651.7eb322b1@msiwind> References: <20100313164651.7eb322b1@msiwind> Message-ID: <5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com> There are two possible problems with Dave's benchmark: 1) On my system setting TCP_NODELAY option on the accepted server socket changes results dramatically. 2) What category of socket servers is dave's spin() function intended to simulate? In a server which involves CPU intensive work in response to a socket request the behavior may be significantly different. In such a system, high CPU load will significantly reduce socket responsiveness which in turn will reduce CPU load and increase socket responsiveness. Testing with a modified server that reflects the above indicates the new GIL behaves just fine in terms of throughput. So a change to the GIL may not be required at all. There is still the question of latency - a single request which takes long time to process will affect the latency of other "small" requests. However, it can be argued if such a scenario is practical, or if modifying the GIL is the solution. If a change is still required, then I vote for the simpler approach - that of having a special macro for socket code. I remember there was reluctance in the past to repeat the OS scheduling functionality and for a good reason. Nir On Sat, Mar 13, 2010 at 11:46 PM, Antoine Pitrou wrote: > > Hello, > > As some of you may know, Dave Beazley recently exhibited a situation > where the new GIL shows quite a poor behaviour (the old GIL isn't very > good either, but still a little better). This issue is followed in > http://bugs.python.org/issue7946 > > This situation is when an IO-bound thread wants to process a lot of > incoming packets, while one (or several) CPU-bound thread is also > running. Each time the IO-bound thread releases the GIL, the CPU-bound > thread gets it and keeps holding it for at least 5 milliseconds > (default setting), which limits the number of individual packets which > can be recv()'ed and processed per second. > > I have proposed two mechanisms, based on the same idea: IO-bound > threads should be able to steal the GIL very quickly, rather than > having to wait for the whole "thread switching interval" (again, 5 ms > by default). They differ in how they detect an "IO-bound threads": > > - the first mechanism is actually the same mechanism which was > embodied in the original new GIL patch before being removed. In this > approach, IO methods (such as socket.read() in socketmodule.c) > releasing the GIL must use a separate C macro when trying to get the > GIL back again. > > - the second mechanism dynamically computes the "interactiveness" of a > thread and allows interactive threads to steal the GIL quickly. In > this approach, IO methods don't have to be modified at all. > > Both approaches show similar benchmark results (for the benchmarks > that I know of) and basically fix the issue put forward by Dave Beazley. > > Any thoughts? > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andymac at bullseye.apana.org.au Sun Mar 14 12:28:46 2010 From: andymac at bullseye.apana.org.au (Andrew MacIntyre) Date: Sun, 14 Mar 2010 22:28:46 +1100 Subject: [Python-Dev] C++ In-Reply-To: <5d44f72f1003122254s4c015780ia7c72a1cab48f5ba@mail.gmail.com> References: <4B9A67E7.30801@gmail.com> <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> <19354.38405.67977.218983@montanaro.dyndns.org> <20100312170308.111b90a5@msiwind> <19355.3195.511985.327292@montanaro.dyndns.org> <5d44f72f1003122254s4c015780ia7c72a1cab48f5ba@mail.gmail.com> Message-ID: <4B9CC86E.7000505@bullseye.andymac.org> Jeffrey Yasskin wrote: > On Fri, Mar 12, 2010 at 7:54 PM, wrote: >> OS/2: http://www.andymac.org/ > > I can't find a modern c++ compiler for OS/2. Then again, your link > only provides python 2.4. While the gcc used as part of the EMX toolchain is 2.8.1, there are ports of gcc 3.3.5 and 4.4.x using a somewhat compatible runtime (known as klibc). I have an unreleased 2.6 binary package which I'm holding up pending 2.6.5. OpenWatcom is also supported on OS/2, though I don't know whether the C++ support is advanced enough... while I like this compiler, the runtime support (or lack thereof) for Posixish features makes platform support a lot of work. Andrew. -- ------------------------------------------------------------------------- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac at bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac at pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From peter.a.portante at gmail.com Sun Mar 14 14:54:13 2010 From: peter.a.portante at gmail.com (Peter Portante) Date: Sun, 14 Mar 2010 08:54:13 -0500 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com> Message-ID: On 3/14/10 7:31 AM, "Nir Aides" wrote: > There are two possible problems with Dave's benchmark: > > 1) On my system setting TCP_NODELAY option on the accepted server socket > changes results dramatically. Could you document what you saw and explain how you think TCP_NODELAY makes a difference, including what kind of system you ran your tests and what the application was that demonstrates those dramatic results? > 2) What category of socket servers is dave's spin() function intended to > simulate? What is the problem you are trying to get at with this question? Does Dave? spin() function have to have a category? Or perhaps the question is, for these solutions, what category of application do they hurt? Perhaps we can look at the solutions as general, but consider their impact in various categories. > In a server which involves CPU intensive work in response to a socket request > the behavior may be significantly different. In such a system, high CPU load > will significantly reduce socket responsiveness which in turn will reduce CPU > load and increase socket responsiveness. Not sure I follow how high CPU load will in turn reduce CPU load. :) Can you explain more about what you are trying to say here? > Testing with a modified server that reflects the above indicates the new GIL > behaves just fine in terms of throughput. So a change to the GIL may not be > required at all. Are you saying that a change to the new GIL may not be required at all? Did your modified server run any worse with any of the proposed changes? Could you document what you saw? > There is still the question of latency - a single request which takes long > time to process will affect the latency of other "small" requests. However, it > can be argued if such a scenario is practical, or if modifying the GIL is the > solution. Perhaps Dave already documented this effect in his visualizations, no? > If a change is still required, then I vote for the simpler approach - that of > having a special macro for socket code. What is that simpler approach? How would that special macro work? > I remember there was reluctance in the past to repeat the OS?scheduling > functionality and for a good reason. Folks, Do we believe that every other application that has contention for one resource leaves the solution of that problem to the OS scheduler? In what ways do we consider the CPython interpreter to be different than another application that has multiple threads and contention for one resource? Perhaps we have a unique problem among all other user space applications. Perhaps we don?t. Do we see Antoine?s or Dave?s proposed solutions as being solutions to a niche problem? Or are they algorithms for handling one or more of the behaviors the CPython interpreter will encounter when running the many python applications across OS/CPU environments? As for the behavior of the GIL, how are the proposed solutions repeating OS scheduling functionality? Could we instead look at these solutions as repeating what other applications have done that have contention for one resource? Can anyone show that Antoine?s proposed solution hurts performance? -peter > Nir > > > On Sat, Mar 13, 2010 at 11:46 PM, Antoine Pitrou wrote: >> >> Hello, >> >> As some of you may know, Dave Beazley recently exhibited a situation >> where the new GIL shows quite a poor behaviour (the old GIL isn't very >> good either, but still a little better). This issue is followed in >> http://bugs.python.org/issue7946 >> >> This situation is when an IO-bound thread wants to process a lot of >> incoming packets, while one (or several) CPU-bound thread is also >> running. Each time the IO-bound thread releases the GIL, the CPU-bound >> thread gets it and keeps holding it for at least 5 milliseconds >> (default setting), which limits the number of individual packets which >> can be recv()'ed and processed per second. >> >> I have proposed two mechanisms, based on the same idea: IO-bound >> threads should be able to steal the GIL very quickly, rather than >> having to wait for the whole "thread switching interval" (again, 5 ms >> by default). They differ in how they detect an "IO-bound threads": >> >> - the first mechanism is actually the same mechanism which was >> ?embodied in the original new GIL patch before being removed. In this >> ?approach, IO methods (such as socket.read() in socketmodule.c) >> ?releasing the GIL must use a separate C macro when trying to get the >> ?GIL back again. >> >> - the second mechanism dynamically computes the "interactiveness" of a >> ?thread and allows interactive threads to steal the GIL quickly. In >> ?this approach, IO methods don't have to be modified at all. >> >> Both approaches show similar benchmark results (for the benchmarks >> that I know of) and basically fix the issue put forward by Dave Beazley. >> >> Any thoughts? >> >> Regards >> >> Antoine. >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon+python-dev at unequivocal.co.uk Sun Mar 14 13:59:50 2010 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Sun, 14 Mar 2010 12:59:50 +0000 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <20100312175657.287bb456@heresy.wooz.org> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com> <20100312145312.1f0a0aa0@heresy.wooz.org> <4B9AC4CA.3070302@voidspace.org.uk> <20100312175657.287bb456@heresy.wooz.org> Message-ID: <20100314125950.GH19240@snowy.squish.net> On Fri, Mar 12, 2010 at 05:56:57PM -0500, Barry Warsaw wrote: > Nope, sorry I should have been clearer. > > *creation* is the key here. As per BDFL pronouncement, we'll support reading > pyc-only modules just like we do today. This is in PEP 3147. We won't > support creating them though. > > BTW, I'm actually starting to work on the implementation now, so if you want > to play along: > > % bzr branch lp:~barry/python/pep3147 Sorry if I missed it, but why on earth is the bytecode directory __pycache__ and not .pycache? (Or indeed anything else that starts with a '.') Surely this is a classic ideal use case for a "hidden" directory? From steve at pearwood.info Sun Mar 14 16:07:23 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 15 Mar 2010 02:07:23 +1100 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <20100314125950.GH19240@snowy.squish.net> References: <20100312175657.287bb456@heresy.wooz.org> <20100314125950.GH19240@snowy.squish.net> Message-ID: <201003150207.24396.steve@pearwood.info> On Sun, 14 Mar 2010 11:59:50 pm Jon Ribbens wrote: > Sorry if I missed it, but why on earth is the bytecode directory > __pycache__ and not .pycache? (Or indeed anything else that starts > with a '.') Surely this is a classic ideal use case for a "hidden" > directory? I disagree with your assumption that there is *any* use-case for a hidden directory, let alone an ideal one. I despise hidden directories and dot files. I know it is the "Unix way", and I suppose it made sense back in 1975 when users had two or three dot files in their home directory, but I count 215 dot files in my home directory compared to only 77 visible files, and I have no idea how most of them got there or what they do. Programs that litter the file system with dot files are bad enough when they do it in $HOME, but sprinkling dot files everywhere they can is inexcusable. This is not the place for me to rant over the evil that is dot files, so I'll just say this: Python works on platforms other than Unix/Linux, and some of those platforms don't treat dot files as anything more than a file with a leading dot in the file name. -- Steven D'Aprano From solipsis at pitrou.net Sun Mar 14 16:24:11 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 14 Mar 2010 11:24:11 -0400 Subject: [Python-Dev] __file__ and bytecode-only References: <20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com> <20100312145312.1f0a0aa0@heresy.wooz.org> <4B9AC4CA.3070302@voidspace.org.uk> <20100312175657.287bb456@heresy.wooz.org> <20100314125950.GH19240@snowy.squish.net> Message-ID: <20100314112411.2e5e1604@msiwind> Le Sun, 14 Mar 2010 12:59:50 +0000, Jon Ribbens a ?crit : > > Sorry if I missed it, but why on earth is the bytecode directory > __pycache__ and not .pycache? (Or indeed anything else that starts > with a '.') Surely this is a classic ideal use case for a "hidden" > directory? I suppose the same reason pyc files are not hidden nowadays. We want people to be aware that these files exist. From p.f.moore at gmail.com Sun Mar 14 17:37:57 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 14 Mar 2010 16:37:57 +0000 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <4B9B9E65.4080902@gmail.com> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com> <20100312145312.1f0a0aa0@heresy.wooz.org> <4B9AC4CA.3070302@voidspace.org.uk> <20100312175657.287bb456@heresy.wooz.org> <4B9B9E65.4080902@gmail.com> Message-ID: <79990c6b1003140937x783bfa02pc5c0a949a5f42890@mail.gmail.com> On 13 March 2010 14:17, Nick Coghlan wrote: > The creation side could be made a little more explicit in the PEP. We > could also do something via the compileall module. > > (Pause while Nick goes and reads the source code for compileall for the > first time ever...) > > Hmm - methinks the PEP actually needs to talk explicitly about the > py_compile and compileall modules. These compile the files directly > rather than using the import system's side-effect, so they'll need to > understand the intricacies of the new system. The bdist_wininst installer also compiles modules explicitly on install (as does the python.org Windows MSI installer). I've always assumed that this worked via compileall, but haven't checked. Regardless, these should probably also be covered in the PEP. Paul. From rdmurray at bitdance.com Sun Mar 14 17:46:11 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 14 Mar 2010 12:46:11 -0400 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <20100314125950.GH19240@snowy.squish.net> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com> <20100312145312.1f0a0aa0@heresy.wooz.org> <4B9AC4CA.3070302@voidspace.org.uk> <20100312175657.287bb456@heresy.wooz.org> <20100314125950.GH19240@snowy.squish.net> Message-ID: <20100314164611.0B5051AB5B9@kimball.webabinitio.net> On Sun, 14 Mar 2010 12:59:50 -0000, Jon Ribbens wrote: > On Fri, Mar 12, 2010 at 05:56:57PM -0500, Barry Warsaw wrote: > > Nope, sorry I should have been clearer. > > > > *creation* is the key here. As per BDFL pronouncement, we'll support reading > > pyc-only modules just like we do today. This is in PEP 3147. We won't > > support creating them though. > > > > BTW, I'm actually starting to work on the implementation now, so if you want > > to play along: > > > > % bzr branch lp:~barry/python/pep3147 > > Sorry if I missed it, but why on earth is the bytecode directory > __pycache__ and not .pycache? (Or indeed anything else that starts > with a '.') Surely this is a classic ideal use case for a "hidden" > directory? You did miss it. Perhaps a link to the discussion should be added to the PEP, or a summary of it. If I remember correctly the BDFL even weighed in on this point. -- R. David Murray www.bitdance.com From martin at v.loewis.de Sun Mar 14 19:32:52 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 14 Mar 2010 19:32:52 +0100 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <20100314051634.GA9503@cskk.homeip.net> References: <4B9C0B9B.3080805@v.loewis.de> <20100314051634.GA9503@cskk.homeip.net> Message-ID: <4B9D2BD4.8010307@v.loewis.de> > Speaking for myself, I have an app with a daemon mode which I expect > will sometimes behave as described; it answers requests and thus has I/O > bound threads waiting for requests and dispatching replies, and threads > doing data handling, which make constant use of the zlib library. This is then already different from the scenario described. Every call into zlib will release the GIL, for the period of the zlib computation, allowing other threads to run. > On the > client side the same app is often throughput bound by a data examination > process that is compute bound; I can easily see it having compute bound > threads and I/O bound threads talking to daemon instances. I can't see that. I would expect that typically (and specifically including your application), the compute bound threads will synchronize with the IO bound ones, asking for more requests to perform. That's the whole point of using the "bound" adjective (?): execution time is *bound* by the computation time. It can't get faster than what the computation can process. Regards, Martni From janssen at parc.com Sun Mar 14 20:37:48 2010 From: janssen at parc.com (Bill Janssen) Date: Sun, 14 Mar 2010 12:37:48 PDT Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <20100314051634.GA9503@cskk.homeip.net> References: <4B9C0B9B.3080805@v.loewis.de> <20100314051634.GA9503@cskk.homeip.net> Message-ID: <70987.1268595468@parc.com> Cameron Simpson wrote: > Currently all the above is somewhat "batchy"; the client end looks like > an archiving command line tool, but it's intended to have a FUSE mode to > present the archive as a filesystem; that I can imagine tripping over > this issue as a user uses the filesystem for "stuff". Me, too :-). Though I also do a lot of processing in pure Python... > Still, it is unusual and I suspect it will self limit to an extent > anyway; I can't fairly claim to have had it cause me trouble yet. > (Indeed, I've not used the new GIL at all - it's mostly using python > 2.6). The old GIL is causing me a great deal of grief. My server is heavily threaded, and when I get a number of Python 2.6 threads doing layout analysis, my 4 and 8 core machines turn into slow thrashers. I'm dismayed that the backport of the new GIL work to the 2.x line has been stopped (see issue 7753); I see a future of OS X 10.7 with presumably 64-bit Python 2.7 on, say, 32-core machines running even more slowly than it currently does. That's a pretty hard-core backwards compatibility decision. Maybe the Mac-SIG should adopt that patch... Bill From nir at winpdb.org Sun Mar 14 22:11:44 2010 From: nir at winpdb.org (Nir Aides) Date: Sun, 14 Mar 2010 23:11:44 +0200 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: References: <5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com> Message-ID: <5d2328511003141411j2bfb8c12s72e2ed54744fe0d4@mail.gmail.com> inline: On Sun, Mar 14, 2010 at 3:54 PM, Peter Portante wrote: > On 3/14/10 7:31 AM, "Nir Aides" wrote: > > There are two possible problems with Dave's benchmark: > > 1) On my system setting TCP_NODELAY option on the accepted server socket > changes results dramatically. > > Could you document what you saw and explain how you think TCP_NODELAY makes > a difference, including what kind of system you ran your tests and what the > application was that demonstrates those dramatic results? > I first disabled the call to spin() but client running time remained around 30 seconds. I then added TCP_NODELAY and running time dropped to a few dozen milliseconds for the entire no-spin run. The system is Ubuntu Karmic 64bit with latest revision of python 3.2. 2) What category of socket servers is dave's spin() function intended to > simulate? > > What is the problem you are trying to get at with this question? > > Does Dave? spin() function have to have a category? Or perhaps the question > is, for these solutions, what category of application do they hurt? Perhaps > we can look at the solutions as general, but consider their impact in > various categories. > In Dave's code sample, spin() is loading the CPU regardless of requests. This may demonstrate how short IO bound requests will behave while the server is processing a long Python-algorithmic CPU intensive request, or an ongoing CPU intensive task unrelated to incoming requests. However is this typical for socket servers? If you change the sample server code to start a CPU bound thread with work X for each incoming request you will see different behavior. There is still the question of latency - a single request which takes long > time to process will affect the latency of other "small" requests. However, > it can be argued if such a scenario is practical, or if modifying the GIL is > the solution. > > Perhaps Dave already documented this effect in his visualizations, no? > Naturally. I did not imply otherwise. His analysis is brilliant. > If a change is still required, then I vote for the simpler approach - that > of having a special macro for socket code. > > What is that simpler approach? How would that special macro work? > The special macro for socket code is one of the alternatives proposed by Antoine above. However, thinking about it again, with this approach as soon as the new incoming request tries to read a file, query the DB, decompress some data or do anything which releases the GIL, it goes back to square one. no? I remember there was reluctance in the past to repeat the OS scheduling > functionality and for a good reason. > > > In what ways do we consider the CPython interpreter to be different than > another application that has multiple threads and contention for one > resource? Perhaps we have a unique problem among all other user space > applications. Perhaps we don?t. > I think a rule of thumb is to protect a resource (typically a data structure), as tightly as possible, avoiding for example locking across function calls, etc, if possible. In contrast CPython is "locked" the entire time. As for the behavior of the GIL, how are the proposed solutions repeating OS > scheduling functionality? > Dave discussed this in his analysis. Nir -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Mar 14 22:43:12 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Mar 2010 07:43:12 +1000 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <20100314164611.0B5051AB5B9@kimball.webabinitio.net> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com> <20100312145312.1f0a0aa0@heresy.wooz.org> <4B9AC4CA.3070302@voidspace.org.uk> <20100312175657.287bb456@heresy.wooz.org> <20100314125950.GH19240@snowy.squish.net> <20100314164611.0B5051AB5B9@kimball.webabinitio.net> Message-ID: <4B9D5870.8080507@gmail.com> R. David Murray wrote: > On Sun, 14 Mar 2010 12:59:50 -0000, Jon Ribbens wrote: >> Sorry if I missed it, but why on earth is the bytecode directory >> __pycache__ and not .pycache? (Or indeed anything else that starts >> with a '.') Surely this is a classic ideal use case for a "hidden" >> directory? > > You did miss it. Perhaps a link to the discussion should be added to > the PEP, or a summary of it. If I remember correctly the BDFL even > weighed in on this point. He did (in favour of keeping the directory visible). Here's one reference (although not the main one): http://osdir.com/ml/python-dev/2010-02/msg00114.html My Google-fu is failing me, so I don't have a link to the other part of the thread that I refer to in that message. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Sun Mar 14 23:09:29 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 14 Mar 2010 18:09:29 -0400 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <70987.1268595468@parc.com> References: <4B9C0B9B.3080805@v.loewis.de> <20100314051634.GA9503@cskk.homeip.net> <70987.1268595468@parc.com> Message-ID: <20100314180929.4053d683@msiwind> Le Sun, 14 Mar 2010 12:37:48 PDT, Bill Janssen a ?crit : > > The old GIL is causing me a great deal of grief. My server is heavily > threaded, and when I get a number of Python 2.6 threads doing layout > analysis, my 4 and 8 core machines turn into slow thrashers. Have you checked whether the 2.x patch improved your use case? It would be nice to get real-world feedback of which situations the new GIL helps (or doesn't help) improve. Thanks Antoine. From solipsis at pitrou.net Sun Mar 14 23:17:03 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 14 Mar 2010 18:17:03 -0400 Subject: [Python-Dev] "Fixing" the new GIL References: <5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com> <5d2328511003141411j2bfb8c12s72e2ed54744fe0d4@mail.gmail.com> Message-ID: <20100314181703.0f65fdcf@msiwind> Le Sun, 14 Mar 2010 23:11:44 +0200, Nir Aides a ?crit : > > I first disabled the call to spin() but client running time remained > around 30 seconds. > I then added TCP_NODELAY and running time dropped to a few dozen > milliseconds for the entire no-spin run. You don't want the benchmark to be dependent on the TCP stack's implementation specifics. Therefore, you should use the UDP version of Dave Beazley's benchmark. I have posted it on http://bugs.python.org/issue7946, and included a variant of it in ccbench. > The special macro for socket code is one of the alternatives proposed > by Antoine above. > > However, thinking about it again, with this approach as soon as the > new incoming request tries to read a file, query the DB, decompress > some data or do anything which releases the GIL, it goes back to > square one. no? Indeed. This approach involves using the macro in every place where releasing the GIL is meant to cover OS- or IO-dependent latencies. Even then, though, there may be cases (such as doing a very small amount of zlib compression) where the background CPU-bound thread will get "too much time" compared to the "mostly IO" thread. But the other approach isn't perfect either. These are heuristics anyway. > I remember there was reluctance in the past to repeat the OS > scheduling > functionality and for a good reason. I agree with this. Hence my posting on this list, about whether we should go with an additional amount of complexity to solve what may be considered an annoying performance problem. Regards Antoine. From martin at v.loewis.de Sun Mar 14 23:23:53 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 14 Mar 2010 23:23:53 +0100 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com> References: <20100313164651.7eb322b1@msiwind> <5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com> Message-ID: <4B9D61F9.7070106@v.loewis.de> > Testing with a modified server that reflects the above indicates the new > GIL behaves just fine in terms of throughput. > So a change to the GIL may not be required at all. Thanks - I think you put it better than I did. Regards, Martin From greg.ewing at canterbury.ac.nz Mon Mar 15 00:10:52 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 15 Mar 2010 12:10:52 +1300 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <20100314125950.GH19240@snowy.squish.net> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com> <20100312145312.1f0a0aa0@heresy.wooz.org> <4B9AC4CA.3070302@voidspace.org.uk> <20100312175657.287bb456@heresy.wooz.org> <20100314125950.GH19240@snowy.squish.net> Message-ID: <4B9D6CFC.7020709@canterbury.ac.nz> Jon Ribbens wrote: > Sorry if I missed it, but why on earth is the bytecode directory > __pycache__ and not .pycache? (Or indeed anything else that starts > with a '.') Surely this is a classic ideal use case for a "hidden" > directory? Please don't try to hide it. On MacOSX, it makes it impossible to look inside using the Finder if you want to, and it won't have any effect on Windows anyway. -- Greg From greg at krypto.org Mon Mar 15 00:29:48 2010 From: greg at krypto.org (Gregory P. Smith) Date: Sun, 14 Mar 2010 16:29:48 -0700 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com> References: <20100313164651.7eb322b1@msiwind> <5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com> Message-ID: <52dc1c821003141629o1a48c7fp83847c342ef2d9e7@mail.gmail.com> On Sun, Mar 14, 2010 at 4:31 AM, Nir Aides wrote: > There are two possible problems with Dave's benchmark: > 1) On my system setting TCP_NODELAY option on the accepted server socket > changes results dramatically. > 2) What category of socket servers is dave's spin() function intended to > simulate? > In a server which involves CPU intensive work in response to a socket > request the behavior may be significantly different. > In such a system, high CPU load will significantly reduce socket > responsiveness which in turn will reduce CPU load and increase socket > responsiveness. > Testing with a modified server that reflects the above indicates the new GIL > behaves just fine in terms of throughput. > So a change to the GIL may not be required at all. > There is still the question of latency - a single request which takes long > time to process will affect the latency of other "small" requests. However, > it can be argued if such a scenario is practical, or if modifying the GIL is > the solution. Such a scenario is practical and is an area that we really should not fall flat on our face in. Python should not regress to performing worse when an application adds a cpu intensive thread that isn't tempered by the activity of its IO threads. As for the argument that an application with cpu intensive work being driven by the IO itself will work itself out... No it won't, it can get into beat patterns where it is handling requests quite rapidly up until one that causes a long computation to start comes in. At that point it'll stop performing well on other requests for as long (it could be a significant amount of time) as the cpu intensive request threads are running. That is not a graceful degration in serving capacity / latency as one would normally expect. It is a sudden drop off. (followed by a sudden ramp up, and a sudden drop off and a sudden ramp up, and drop off... etc. not pretty. not predictable. not easy to deploy and manage as a service) If we don't fix this issue, we need to document as an official part of the Python threading module docs that people should not mix computation with IO in threaded applications when using CPython if they care about IO performance. This will blow people's mind. > If a change is still required, then I vote for the simpler approach - that > of having a special macro for socket code. > I remember there was reluctance in the past to repeat the OS?scheduling > functionality and for a good reason. I don't consider needing to modify code specifically for it to indicate if it is IO bound or CPU bound to be the simpler approach. So +1 on the interactiveness calculation for me. +0.5 on annotating the IO lock release/acquisitions. Basically: I want to see this fixed, even if its not my preferred approach I still think it is important. -gps From janssen at parc.com Mon Mar 15 02:46:25 2010 From: janssen at parc.com (Bill Janssen) Date: Sun, 14 Mar 2010 18:46:25 PDT Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <20100314180929.4053d683@msiwind> References: <4B9C0B9B.3080805@v.loewis.de> <20100314051634.GA9503@cskk.homeip.net> <70987.1268595468@parc.com> <20100314180929.4053d683@msiwind> Message-ID: <74196.1268617585@parc.com> Antoine Pitrou wrote: > Le Sun, 14 Mar 2010 12:37:48 PDT, > Bill Janssen a ?crit : > > > > The old GIL is causing me a great deal of grief. My server is heavily > > threaded, and when I get a number of Python 2.6 threads doing layout > > analysis, my 4 and 8 core machines turn into slow thrashers. > > Have you checked whether the 2.x patch improved your use case? > It would be nice to get real-world feedback of which situations the > new GIL helps (or doesn't help) improve. Yes, good thought. I'll try it and see if I can get some numbers. I do have a mix of I/O and CPU bound threads, and we are seeing core usage by Python similar to that David originally discovered. Bill From dave at dabeaz.com Mon Mar 15 04:07:05 2010 From: dave at dabeaz.com (David Beazley) Date: Sun, 14 Mar 2010 22:07:05 -0500 Subject: [Python-Dev] "Fixing" the new GIL Message-ID: <49653.1268622425@dabeaz.com> I have been following this discussion about fixing the GIL and just wanted to make a few comments about it. To the doubters who don't think this is a real problem worth fixing, I must respectfully disagree. Multicore systems and parallel programming issues are not going away any time in the foreseeable future. Moreover, there are plenty of programmers who are using Python to build applications involving concurrency, distributed computing, and compute intensive processing. While everyone agrees that it would be great if the GIL simply disappeared, the next best option is for the GIL to have well-defined and reasonably predictable behavior (i.e., behavior that can be easily explained and justified). Specifically, if multiple threads happen to be performing CPU intensive work at the same time, it would be nice if they didn't thrash on multiple cores (the problem with the old GIL) and if I/O is mixed with CPU-bound processing it would be nice if I/O requests didn't have their responsiveness and priority penalized (the problem with the new GIL). So, just to be clear about the my bug report, it is directly related to the problem of overlapping I/O requests with CPU-bound processing. This kind of scenario comes up in the context of many applications--especially those based on cooperating processes, multiprocessing, and message passing. Ironically, these are exactly the kinds of parallel programming techniques that people are going to use because of the GIL in the first place. So, why would you want the new GIL to behave poorly here? In any case, if the GIL can be improved in a way that is simple and which either improves or doesn't negatively impact the performance of existing applications, why wouldn't you want to do it? Seems like a no-brainer. Cheers, Dave From martin at v.loewis.de Mon Mar 15 09:28:40 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 15 Mar 2010 09:28:40 +0100 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <52dc1c821003141629o1a48c7fp83847c342ef2d9e7@mail.gmail.com> References: <20100313164651.7eb322b1@msiwind> <5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com> <52dc1c821003141629o1a48c7fp83847c342ef2d9e7@mail.gmail.com> Message-ID: <4B9DEFB8.3010107@v.loewis.de> > As for the argument that an application with cpu intensive work being > driven by the IO itself will work itself out... No it won't, it can > get into beat patterns where it is handling requests quite rapidly up > until one that causes a long computation to start comes in. At that > point it'll stop performing well on other requests for as long (it > could be a significant amount of time) as the cpu intensive request > threads are running. That is not a graceful degration in serving > capacity / latency as one would normally expect. It is a sudden drop > off. Why do you say that? The other threads continue to be served - and Python couldn't use more than one CPU, anyway. Can you demonstrate that in an example? Regards, Martin From martin at v.loewis.de Mon Mar 15 09:34:03 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 15 Mar 2010 09:34:03 +0100 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <49653.1268622425@dabeaz.com> References: <49653.1268622425@dabeaz.com> Message-ID: <4B9DF0FB.8000002@v.loewis.de> > So, just to be clear about the my bug report, it is directly related > to the problem of overlapping I/O requests with CPU-bound processing. > This kind of scenario comes up in the context of many > applications--especially those based on cooperating processes, > multiprocessing, and message passing. How so? if you have cooperating processes, multiprocessing, and message passing, you *don't* have CPU bound threads along with IO "bound" threads in the same process - you may not have threads at all!!! > In any case, if the GIL can be improved in a way that is simple and > which either improves or doesn't negatively impact the performance of > existing applications, why wouldn't you want to do it? Seems like a > no-brainer. Unfortunately, a simple improvement doesn't really seem to exist. Regards, Martin From dave at dabeaz.com Mon Mar 15 13:21:29 2010 From: dave at dabeaz.com (David Beazley) Date: Mon, 15 Mar 2010 07:21:29 -0500 Subject: [Python-Dev] (no subject) Message-ID: <50994.1268655689@dabeaz.com> On Mon 15/03/10 4:34 AM , "Martin v. L?wis" martin at v.loewis.de sent: > > So, just to be clear about the my bug report, it > is directly related> to the problem of overlapping I/O requests with > CPU-bound processing.> This kind of scenario comes up in the context of > many> applications--especially those based on > cooperating processes,> multiprocessing, and message passing. > > How so? if you have cooperating processes, multiprocessing, and message > passing, you *don't* have CPU bound threads along with IO > "bound"threads in the same process - you may not have threads at all!!! > You're right in that end user will probably not be using threads in this situation. However, threads are still often used inside the associated programming libraries and frameworks that provide communication. For example, threads might be used to implement queuing, background monitoring, event notification, routing, and other similar low-level features. Just as an example, the multiprocessing module currently uses background threads as part of its implementation of queues. In a similar vein, threads are sometimes used in asynchronous I/O frameworks (e.g., Twisted) to handle certain kinds of deferred operations. Bottom line: just because a user isn't directly programming with threads doesn't mean that threads aren't there. > > In any case, if the GIL can be improved in a way > that is simple and> which either improves or doesn't negatively > impact the performance of> existing applications, why wouldn't you want to > do it? Seems like a> no-brainer. > > Unfortunately, a simple improvement doesn't really seem to exist. > Well, I think this problem can be fixed in a manner that is pretty well isolated to just one specific part of Python (the GIL)--especially since several prototypes, including my own, have already demonstrated that it's possible. In any case, as it stands now, the current "new GIL" ignores about 40 years of work concerning operating system thread/process scheduling and the efficient handling of I/O requests. I suspect that I'm not the only one who be disappointed if the Python community simply blew it off and said it's not an issue. Trying to fix it is a worthwhile goal. Cheers, Dave From barry at python.org Mon Mar 15 16:16:54 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 15 Mar 2010 11:16:54 -0400 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <4B9B9E65.4080902@gmail.com> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com> <20100312145312.1f0a0aa0@heresy.wooz.org> <4B9AC4CA.3070302@voidspace.org.uk> <20100312175657.287bb456@heresy.wooz.org> <4B9B9E65.4080902@gmail.com> Message-ID: <20100315111654.1aa7acf5@heresy.wooz.org> On Mar 14, 2010, at 12:17 AM, Nick Coghlan wrote: >Hmm - methinks the PEP actually needs to talk explicitly about the >py_compile and compileall modules. These compile the files directly >rather than using the import system's side-effect, so they'll need to >understand the intricacies of the new system. Good point. I'll add this to the PEP. >While it's probably OK if the import side-effects only create files >using the new scheme, the standard library modules will likely need to >support both schemes (although I'm not sure if "same as import system" >or "same as Python 3.1" make more sense as the default semantics - >probably the former). I don't understand this point. compileall probably /could/ be extended to understand bytecode-only (i.e. legacy or <3.2) layout. I've added that to the PEP too. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Mon Mar 15 16:18:34 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 15 Mar 2010 11:18:34 -0400 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <79990c6b1003140937x783bfa02pc5c0a949a5f42890@mail.gmail.com> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com> <20100312145312.1f0a0aa0@heresy.wooz.org> <4B9AC4CA.3070302@voidspace.org.uk> <20100312175657.287bb456@heresy.wooz.org> <4B9B9E65.4080902@gmail.com> <79990c6b1003140937x783bfa02pc5c0a949a5f42890@mail.gmail.com> Message-ID: <20100315111834.66190720@heresy.wooz.org> On Mar 14, 2010, at 04:37 PM, Paul Moore wrote: >The bdist_wininst installer also compiles modules explicitly on >install (as does the python.org Windows MSI installer). I've always >assumed that this worked via compileall, but haven't checked. >Regardless, these should probably also be covered in the PEP. Thanks; added to PEP. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Mon Mar 15 16:28:29 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 15 Mar 2010 11:28:29 -0400 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <4B9D5870.8080507@gmail.com> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com> <20100312145312.1f0a0aa0@heresy.wooz.org> <4B9AC4CA.3070302@voidspace.org.uk> <20100312175657.287bb456@heresy.wooz.org> <20100314125950.GH19240@snowy.squish.net> <20100314164611.0B5051AB5B9@kimball.webabinitio.net> <4B9D5870.8080507@gmail.com> Message-ID: <20100315112829.4b36f160@heresy.wooz.org> On Mar 15, 2010, at 07:43 AM, Nick Coghlan wrote: >He did (in favour of keeping the directory visible). http://www.mail-archive.com/python-dev at python.org/msg45203.html (added to pep) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From cs at zip.com.au Tue Mar 16 06:34:45 2010 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 16 Mar 2010 16:34:45 +1100 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <4B9DEFB8.3010107@v.loewis.de> References: <4B9DEFB8.3010107@v.loewis.de> Message-ID: <20100316053445.GA11229@cskk.homeip.net> On 15Mar2010 09:28, Martin v. L?wis wrote: | > As for the argument that an application with cpu intensive work being | > driven by the IO itself will work itself out... No it won't, it can | > get into beat patterns where it is handling requests quite rapidly up | > until one that causes a long computation to start comes in. At that | > point it'll stop performing well on other requests for as long (it | > could be a significant amount of time) as the cpu intensive request | > threads are running. That is not a graceful degration in serving | > capacity / latency as one would normally expect. It is a sudden drop | > off. | | Why do you say that? The other threads continue to be served - and | Python couldn't use more than one CPU, anyway. Can you demonstrate that | in an example? Real example: I have a FuncMultiQueue class which manages a pool of worker threads. Other tasks can make requests of it by passing a callable (usually obtained via partial()) to its .call(), .bgcall() or .qbgcall() methods depending on how they want to collect the result of the callable. The idea here is that one has a few threads receiving requests (eg a daemon watching a socket or monitoring a db queue table) which then use the FuncMultiQueue to manage how many actual requests are processed in parallel (yes, a semaphore can cover a lot of this, but not the asynchronous call modes). So, suppose a couple of CPU-intensive callables get queued which work for a substantial time, and meanwhile a bunch of tiny tiny cheap requests arrive. Their timely response will be impacted by this issue. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ hybrid rather than pure; compromising rather than clean; distorted rather than straightforward; ambiguous rather than articulated; both-and rather than either-or; the difficult unity of inclusion rather than the easy unity of exclusion. - Paul Barton-Davis From cs at zip.com.au Tue Mar 16 06:38:47 2010 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 16 Mar 2010 16:38:47 +1100 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <4B9D2BD4.8010307@v.loewis.de> References: <4B9D2BD4.8010307@v.loewis.de> Message-ID: <20100316053847.GA12552@cskk.homeip.net> On 14Mar2010 19:32, "Martin v. L?wis" wrote: | > Speaking for myself, I have an app with a daemon mode which I expect | > will sometimes behave as described; it answers requests and thus has I/O | > bound threads waiting for requests and dispatching replies, and threads | > doing data handling, which make constant use of the zlib library. | | This is then already different from the scenario described. Every call | into zlib will release the GIL, for the period of the zlib computation, | allowing other threads to run. | | > On the | > client side the same app is often throughput bound by a data examination | > process that is compute bound; I can easily see it having compute bound | > threads and I/O bound threads talking to daemon instances. | | I can't see that. I would expect that typically (and specifically | including your application), the compute bound threads will synchronize | with the IO bound ones, asking for more requests to perform. In the single threaded case, sure. The usual command line "archive this" mode fit this. But... | That's the whole point of using the "bound" adjective (?): execution | time is *bound* by the computation time. It can't get faster than what | the computation can process. If it's lurking behind a filesystem interface or in its daemon mode (remote archive store), multiple client processes can be using it at once, and it will be processing multiple tasks somewhat in parallel. Here one can get a compute bound thread answering one request, impacting quick response to other parallel-and-cheap requests. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ Here was a man who not only had a ready mind and a quick wit, but could also sing. - _Rope_ From martin at v.loewis.de Tue Mar 16 08:59:16 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 16 Mar 2010 08:59:16 +0100 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <20100316053445.GA11229@cskk.homeip.net> References: <4B9DEFB8.3010107@v.loewis.de> <20100316053445.GA11229@cskk.homeip.net> Message-ID: <4B9F3A54.3040907@v.loewis.de> Cameron Simpson wrote: > On 15Mar2010 09:28, Martin v. L?wis wrote: > | > As for the argument that an application with cpu intensive work being > | > driven by the IO itself will work itself out... No it won't, it can > | > get into beat patterns where it is handling requests quite rapidly up > | > until one that causes a long computation to start comes in. At that > | > point it'll stop performing well on other requests for as long (it > | > could be a significant amount of time) as the cpu intensive request > | > threads are running. That is not a graceful degration in serving > | > capacity / latency as one would normally expect. It is a sudden drop > | > off. > | > | Why do you say that? The other threads continue to be served - and > | Python couldn't use more than one CPU, anyway. Can you demonstrate that > | in an example? > > Real example: ... unfortunately without a demonstration. What's the throughput under the old GIL? What's the throughput of this application under the new GIL? How can I observe the "beat pattern"? > The idea here is that one has a few threads receiving requests (eg a > daemon watching a socket or monitoring a db queue table) which then use > the FuncMultiQueue to manage how many actual requests are processed > in parallel (yes, a semaphore can cover a lot of this, but not the > asynchronous call modes). Why do you say processing is in parallel? In Python, processing is normally never in parallel, but always sequential (potentially interleaving). Are you releasing the GIL for processing? > So, suppose a couple of CPU-intensive callables get queued which work for a > substantial time, and meanwhile a bunch of tiny tiny cheap requests arrive. > Their timely response will be impacted by this issue. By how much exactly? What operating system? Regards, Martin From martin at v.loewis.de Tue Mar 16 09:02:41 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 16 Mar 2010 09:02:41 +0100 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <20100316053847.GA12552@cskk.homeip.net> References: <4B9D2BD4.8010307@v.loewis.de> <20100316053847.GA12552@cskk.homeip.net> Message-ID: <4B9F3B21.1090106@v.loewis.de> > If it's lurking behind a filesystem interface or in its daemon mode > (remote archive store), multiple client processes can be using it at once, > and it will be processing multiple tasks somewhat in parallel. Here one > can get a compute bound thread answering one request, impacting quick > response to other parallel-and-cheap requests. However, "impacting" will always be the case. *Of course* any thread that performs computation impacts everything else on the same processor. There is nothing to prevent that, unless you schedule the long-running activity at a point in time where you know the system will be idle for the entire run of that task. Regards, Martin From cs at zip.com.au Tue Mar 16 10:15:44 2010 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 16 Mar 2010 20:15:44 +1100 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <4B9F3A54.3040907@v.loewis.de> References: <4B9F3A54.3040907@v.loewis.de> Message-ID: <20100316091544.GA3799@cskk.homeip.net> On 16Mar2010 08:59, "Martin v. L?wis" wrote: | Cameron Simpson wrote: | > On 15Mar2010 09:28, Martin v. L?wis wrote: | > | > As for the argument that an application with cpu intensive work being | > | > driven by the IO itself will work itself out... No it won't, it can | > | > get into beat patterns where it is handling requests quite rapidly up | > | > until one that causes a long computation to start comes in. At that | > | > point it'll stop performing well on other requests for as long (it | > | > could be a significant amount of time) as the cpu intensive request | > | > threads are running. That is not a graceful degration in serving | > | > capacity / latency as one would normally expect. It is a sudden drop | > | > off. | > | | > | Why do you say that? The other threads continue to be served - and | > | Python couldn't use more than one CPU, anyway. Can you demonstrate that | > | in an example? | > | > Real example: | | ... unfortunately without a demonstration. What's the throughput under | the old GIL? What's the throughput of this application under the new | GIL? How can I observe the "beat pattern"? You can't. This isn't an example where I am personally bitten by the GIL. I may be, but there's plenty of other stuff in terms of tuning or optimisation in my particular app before I get anywhere near the GIL. My point is not that it's biting me right now but that earlier you said the scenario was probably unrealistic, but I have an app which can probably exhibit exactly the issue under discussion. | > The idea here is that one has a few threads receiving requests (eg a | > daemon watching a socket or monitoring a db queue table) which then use | > the FuncMultiQueue to manage how many actual requests are processed | > in parallel (yes, a semaphore can cover a lot of this, but not the | > asynchronous call modes). | | Why do you say processing is in parallel? | In Python, processing is | normally never in parallel, but always sequential (potentially | interleaving). Are you releasing the GIL for processing? I mean that I can have multiple actual requests being served, and they are _not_ handled sequentially - the multi queue is to permit more than one to be in progress at once so that an expensive request need not inherently delay a following cheap request. And in my it's-a-filesystem mode there will often be a bottleneck in the pure-python data scanning part. And thus the CPU bound python active while still accepting requests in I/O blocked threads. | > So, suppose a couple of CPU-intensive callables get queued which work for a | > substantial time, and meanwhile a bunch of tiny tiny cheap requests arrive. | > Their timely response will be impacted by this issue. | | By how much exactly? What operating system? Can't tell you - I'm not claiming the current GIL behaviour is biting me right now; I'm saying your claim that the scenario is unrealistic is a bit unfair; my app wil probably be just the kind of app to be affected, even only subtly. And it can hardly be alone in the "daemon" world. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ There's two kinds of climbers...smart ones, and dead ones. - Don Whillans From cs at zip.com.au Tue Mar 16 10:21:20 2010 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 16 Mar 2010 20:21:20 +1100 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <4B9F3B21.1090106@v.loewis.de> References: <4B9F3B21.1090106@v.loewis.de> Message-ID: <20100316092120.GA6286@cskk.homeip.net> On 16Mar2010 09:02, "Martin v. L?wis" wrote: | > If it's lurking behind a filesystem interface or in its daemon mode | > (remote archive store), multiple client processes can be using it at once, | > and it will be processing multiple tasks somewhat in parallel. Here one | > can get a compute bound thread answering one request, impacting quick | > response to other parallel-and-cheap requests. | | However, "impacting" will always be the case. *Of course* any thread | that performs computation impacts everything else on the same processor. | There is nothing to prevent that, unless you schedule the long-running | activity at a point in time where you know the system will be idle for | the entire run of that task. Sure, but one has only to dip one's toe in the long running Linux interactivity scheduling wars to know that there's a _lot_ of attraction in a scheduling approach that favours the usually-blocked task over the CPU bound task, and various approaches like the early low-latency patches to the later in-the-scheduler changes were greeted with much rejoicing. Certainly there's only so much CPU to go around, but if the tasks blocked waiting for something to happen get to run more immediately when an event does occur, you tend to get better "responsiveness". Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ Be smart, be safe, be paranoid. - Ryan Cousineau, courier at compdyn.com DoD#863, KotRB, KotKWaWCRH From ncoghlan at gmail.com Tue Mar 16 10:44:27 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Mar 2010 19:44:27 +1000 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <20100315111654.1aa7acf5@heresy.wooz.org> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com> <20100312145312.1f0a0aa0@heresy.wooz.org> <4B9AC4CA.3070302@voidspace.org.uk> <20100312175657.287bb456@heresy.wooz.org> <4B9B9E65.4080902@gmail.com> <20100315111654.1aa7acf5@heresy.wooz.org> Message-ID: <4B9F52FB.2090701@gmail.com> Barry Warsaw wrote: > On Mar 14, 2010, at 12:17 AM, Nick Coghlan wrote: >> While it's probably OK if the import side-effects only create files >> using the new scheme, the standard library modules will likely need to >> support both schemes (although I'm not sure if "same as import system" >> or "same as Python 3.1" make more sense as the default semantics - >> probably the former). > > I don't understand this point. > > compileall probably /could/ be extended to understand bytecode-only > (i.e. legacy or <3.2) layout. I've added that to the PEP too. In Python 3.1, *invoking* py_compile.compile() will create 2.x style bytecode. Similarly, when force==False, compileall.compile_dir() and compileall.compile_path() will check for 2.x style bytecode in order to decide whether or not to compile the module. The question for 3.2 is what bytecode layout py_compile.compile() should generate. For the precompile-a-system-library use case it should clearly generate a PEP 3147 layout and this probably makes sense as the default behaviour in 3.2. However, for production of bytecode-only packages, it would be convenient to be able to explicitly invoke the 2.x style behaviour without having to specify the target filename explicitly using the 'cfile' parameter (which isn't exposed at the compileall layer anyway). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Mar 16 11:37:52 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Mar 2010 20:37:52 +1000 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <4B9F3A54.3040907@v.loewis.de> References: <4B9DEFB8.3010107@v.loewis.de> <20100316053445.GA11229@cskk.homeip.net> <4B9F3A54.3040907@v.loewis.de> Message-ID: <4B9F5F80.7030102@gmail.com> Martin v. L?wis wrote: > Cameron Simpson wrote: >> The idea here is that one has a few threads receiving requests (eg a >> daemon watching a socket or monitoring a db queue table) which then use >> the FuncMultiQueue to manage how many actual requests are processed >> in parallel (yes, a semaphore can cover a lot of this, but not the >> asynchronous call modes). > > Why do you say processing is in parallel? In Python, processing is > normally never in parallel, but always sequential (potentially > interleaving). Are you releasing the GIL for processing? We know the GIL is timeslicing - that's the whole point. The real question is *how often* is it time slicing and *which threads* are getting to run. The key issue is that the current new GIL implementation means that, in the presence of a CPU bound GIL-holding thread, the *minimum* duration of *any* C call that releases the GIL becomes sys.getcheckinterval(). The CPU bound thread *always* wants the GIL and once it gets it, it won't give it back until sys.getcheckinterval() expires. This is in contrast to the I/O bound (or non-GIL holding CPU bound) threads that nearly always yield the GIL early, since they're waiting for something else to happen (be it I/O or a native calculation). Antoine's interactiveness patch resolves this by rewarding threads that regularly release the GIL early: the interpreter quickly recognises them as doing so and they are subsequently given priority over the threads that only release the GIL when the check interval expires. When threads that are flagged as interactive ask for the GIL back, the interpreter gives it to them immediately instead of forcing them to wait until the check interval expires (because it trusts those threads not to hang onto the GIL for too long). I like this significantly better than the explicit priority patch, because it means threads are given priority based on their behaviour (i.e. frequently holding the GIL for less than the check interval) rather than relying on developers to correctly decide between using a normal GIL request and a priority request. Handling of thread pools that mix I/O bound and CPU bound GIL-holding tasks isn't quite optimal with this approach (since the nature of any given thread will change over time), but it isn't terrible either - the interpreter doesn't need much time to reclassify a thread as interactive or non-interactive (and there are a couple of parameters that can be tuned to control how quickly these changes in status can happen). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From kristjan at ccpgames.com Tue Mar 16 11:52:28 2010 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Tue, 16 Mar 2010 10:52:28 +0000 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <49653.1268622425@dabeaz.com> References: <49653.1268622425@dabeaz.com> Message-ID: <930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local> How about attacking the original problem, then? The reason they thrash on pthreads implementation is that a pthreads mutex is assumed to be a short-held resource. Therefore it will be optimized in the following ways for multicore machines: 1) There is a certain amount of spinning done, to try to acquire it before blocking 2) It will employ un-fair tactics to avoid lock-convoying, meaning that a thread coming in to acquire the mutex may get in before others that are queued. This is why "ticking" the GIL works so badly: The thread that releases the lock is usually the one that reaquires it even though others may be waiting. See e.g. http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-616ef7b031aa.aspx for a discussion of this (albeit on windows.) On Windows, this isn't a problem. The reason is, that the GIL on windows is implemented using Event objects that don't cut these corners. The Event provides you with a strict FIFO queue of objects waiting for the event. If pthreads doesn't provide a synchronization primitive similar to that, someone that doesn't thrash and has a true FIFO queue, it is possible to construct such a thing using condition variables and critical sections. Perhaps the posix semaphore api is more appropriate in this case. By the way, this also shows another problem with (old) python. There is only one core locking primitive, the PyThread_type_lock. It is being used both as a critical section in the traditional sense, and also as this sort-of-inverse lock that the GIL is. In the modern world, where the intended behaviour of these is quite different, there is no one-size-fits all. On windows in particular, the use of the Event object based lock is not ideal for other uses than the GIL. In the new GIL, there appear to be several problems: 1) There is no FIFO queue of threads wanting the queue, thus thread scheduling becomes non-deterministic 2) The "ticking" of the GIL is now controled by a condition variable timeout. There appears to be no way to prevent many such timeouts to be in progress at the same time, thus you may have an unnecessarily high rate of ticking going on. 3) There isn't an immediate gil request made when an IO thread requests the gil back, only after an initial timeout. What we are trying to write here is a thread scheduler, and that is complex business. K > -----Original Message----- > From: python-dev-bounces+kristjan=ccpgames.com at python.org > [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf > Of David Beazley > Sent: 15. mars 2010 03:07 > To: python-dev at python.org > Subject: Re: [Python-Dev] "Fixing" the new GIL > > happen to be performing CPU intensive work at the same time, it would > be nice if they didn't thrash on multiple cores (the problem with the > old GIL) and if I/O is From dave at dabeaz.com Tue Mar 16 12:02:32 2010 From: dave at dabeaz.com (David Beazley) Date: Tue, 16 Mar 2010 06:02:32 -0500 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local> References: <49653.1268622425@dabeaz.com> <930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local> Message-ID: <5AD14AA1-72D3-4076-91BF-63B6C402139D@dabeaz.com> Python doesn't use a pthreads mutex for the GIL. It has always used a binary semaphore implemented with condition variables (or just a pthreads semaphore if available). The reason the performance is so bad is precisely due to the fact that it is using this implementation and the fact that there *IS* a FIFO queue of threads (associated with the condition variable). The I/O performance problem with the new GIL is gets much worse with many CPU-bound threads precisely because there is a FIFO queue involved. This has been covered in my past GIL presentations. -Dave On Mar 16, 2010, at 5:52 AM, Kristj?n Valur J?nsson wrote: > How about attacking the original problem, then? > > The reason they thrash on pthreads implementation is that a pthreads mutex is assumed to be a short-held resource. Therefore it will be optimized in the following ways for multicore machines: > 1) There is a certain amount of spinning done, to try to acquire it before blocking > 2) It will employ un-fair tactics to avoid lock-convoying, meaning that a thread coming in to acquire the mutex may get in before others that are queued. This is why "ticking" the GIL works so badly: The thread that releases the lock is usually the one that reaquires it even though others may be waiting. See e.g. http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-616ef7b031aa.aspx for a discussion of this (albeit on windows.) > > On Windows, this isn't a problem. The reason is, that the GIL on windows is implemented using Event objects that don't cut these corners. The Event provides you with a strict FIFO queue of objects waiting for the event. > > If pthreads doesn't provide a synchronization primitive similar to that, someone that doesn't thrash and has a true FIFO queue, it is possible to construct such a thing using condition variables and critical sections. Perhaps the posix semaphore api is more appropriate in this case. > > By the way, this also shows another problem with (old) python. There is only one core locking primitive, the PyThread_type_lock. It is being used both as a critical section in the traditional sense, and also as this sort-of-inverse lock that the GIL is. In the modern world, where the intended behaviour of these is quite different, there is no one-size-fits all. On windows in particular, the use of the Event object based lock is not ideal for other uses than the GIL. > > > In the new GIL, there appear to be several problems: > 1) There is no FIFO queue of threads wanting the queue, thus thread scheduling becomes non-deterministic > 2) The "ticking" of the GIL is now controled by a condition variable timeout. There appears to be no way to prevent many such timeouts to be in progress at the same time, thus you may have an unnecessarily high rate of ticking going on. > 3) There isn't an immediate gil request made when an IO thread requests the gil back, only after an initial timeout. > > What we are trying to write here is a thread scheduler, and that is complex business. > K > > > >> -----Original Message----- >> From: python-dev-bounces+kristjan=ccpgames.com at python.org >> [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf >> Of David Beazley >> Sent: 15. mars 2010 03:07 >> To: python-dev at python.org >> Subject: Re: [Python-Dev] "Fixing" the new GIL >> >> happen to be performing CPU intensive work at the same time, it would >> be nice if they didn't thrash on multiple cores (the problem with the >> old GIL) and if I/O is > From solipsis at pitrou.net Tue Mar 16 13:00:11 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 16 Mar 2010 08:00:11 -0400 Subject: [Python-Dev] "Fixing" the new GIL References: <49653.1268622425@dabeaz.com> <930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local> Message-ID: <20100316080011.34589b00@msiwind> Hi Kristj?n, > In the new GIL, there appear to be several problems: > 1) There is no FIFO queue of threads wanting the queue, thus thread > scheduling becomes non-deterministic Thread scheduling has always been non-deterministic. > 2) The "ticking" of the GIL is > now controled by a condition variable timeout. There appears to be > no way to prevent many such timeouts to be in progress at the same > time, thus you may have an unnecessarily high rate of ticking going > on. Unless there's a bug, no, there isn't. > 3) There isn't an immediate gil request made when an IO thread > requests the gil back, only after an initial timeout. This is what we are looking to fix (perhaps). > What we are trying to write here is a thread scheduler, and that is > complex business. K I would have rephrased: "What we are trying to avoid here is a thread scheduler". regards Antoine. From kristjan at ccpgames.com Tue Mar 16 14:46:08 2010 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Tue, 16 Mar 2010 13:46:08 +0000 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <5AD14AA1-72D3-4076-91BF-63B6C402139D@dabeaz.com> References: <49653.1268622425@dabeaz.com> <930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local> <5AD14AA1-72D3-4076-91BF-63B6C402139D@dabeaz.com> Message-ID: <930F189C8A437347B80DF2C156F7EC7F0A1821825F@exchis.ccp.ad.local> You are right, I was assuming a mutex. (why not then, use a semaphore?) But the implementation isn't a simple binary semaphore. It suffers from case 2) that I explained before: Greedy aquisition of the "semaphore" even if someone is waiting. Also, I should add, a condition variable doesn't specify that it employs a FIFO. pythread_cond_signals() wakes up _a_ thread. If you want particular behaviour then you have to add code to deal with that. But it tries to be fair, and we can probably assum e FIFO behavioud, so let's ignore that part. So, in the following, let's assume that the condition variable is fair, and that it queues up the waiting threads in a FIFO manner: Fixing 2) in this case is easy: Make sure you wait if others are waiting. In python pseudocode: def Aquire(gil): with gil.condition: if gil.locked or gil.n_waiting: gil.n_waiting+=1 while gil.locked: gil.condition.wait() gil.n_waiting-=1 gil.locked = True def Release(gil): with gil.condition: gil.locked=False if gil.n_waiting: gil.condition.notify() This ought to fix problem 2) below. Multicore CPUs ought to not behave worse than single core anymore. (Btw, the C code is erroneous, one should not call pthread_cond_signal without having the lock held, but that has been brought up before.) Now, if you want to improve IO performance, by ensuring that IO threads get woken up before (yielding) cpu threads, you can do something like this: class GIL(): def __init__(self, npri): self.lock=RLock() self.queues = [[Condition(self.lock), 0] for i in range(npri)] self.locked = False def Acquire(self, pri): """Acquire the lock with priority "pri", where 0 is highest priority""" with self.lock: if self.locked or sum([q[1] for q in self.queues[:pri+1]]) #locked or someone with same or higher priority waiting self.queues[pri][1]+=1 while self.locked: self.queues[pri][0].wait() self.queues[pri][1]-=1 self.locked = True def Release(self): with self.lock: self.locked = False for q in self.queues: if q[1]: q[0].notify() break def Blip(self): """volountailily give up the lock and allow higher priority requests to run""" self.Release() self.Acquire(len(self.queues)-1) Typically you would have only two priority classes. IO would then be done with: gil.Release() r = socket.recv(l) gil.Acquire(0) But here we are on dangerous grounds. Priority scheduling is full of pitfalls. I should add here maybe, that I am a strong proponent of strict FIFO scheduling with perhaps few special cases. We have been using FIFO tasklet scheduling for all tasklets, including those that are returing with IO, in EVE Online for three years now. Before, we had somewhat chaotic scheduling (immediate switching to tasklets) and behavior was very erratic. Switching to FIFO cured that and was very wholesome for the cluster in general.. I have been toying with the idea of adding a special IO tasklet priority class to stackless for a while, but haven't done it yet. I am doubtful that it will on the whole change matters much. Cheers, Kristj?n > -----Original Message----- > From: David Beazley [mailto:dave at dabeaz.com] > Sent: 16. mars 2010 11:03 > To: Kristj?n Valur J?nsson > Cc: David Beazley; python-dev at python.org > Subject: Re: [Python-Dev] "Fixing" the new GIL > > Python doesn't use a pthreads mutex for the GIL. It has always used > a binary semaphore implemented with condition variables (or just a > pthreads semaphore if available). The reason the performance is so > bad is precisely due to the fact that it is using this implementation > and the fact that there *IS* a FIFO queue of threads (associated with > the condition variable). The I/O performance problem with the new GIL > is gets much worse with many CPU-bound threads precisely because there > is a FIFO queue involved. This has been covered in my past GIL > presentations. > > -Dave > > > > On Mar 16, 2010, at 5:52 AM, Kristj?n Valur J?nsson wrote: > > > How about attacking the original problem, then? > > > > The reason they thrash on pthreads implementation is that a pthreads > mutex is assumed to be a short-held resource. Therefore it will be > optimized in the following ways for multicore machines: > > 1) There is a certain amount of spinning done, to try to acquire it > before blocking > > 2) It will employ un-fair tactics to avoid lock-convoying, meaning > that a thread coming in to acquire the mutex may get in before others > that are queued. This is why "ticking" the GIL works so badly: The > thread that releases the lock is usually the one that reaquires it even > though others may be waiting. See e.g. > http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f- > 8f85-616ef7b031aa.aspx for a discussion of this (albeit on windows.) > > > > On Windows, this isn't a problem. The reason is, that the GIL on > windows is implemented using Event objects that don't cut these > corners. The Event provides you with a strict FIFO queue of objects > waiting for the event. > > > > If pthreads doesn't provide a synchronization primitive similar to > that, someone that doesn't thrash and has a true FIFO queue, it is > possible to construct such a thing using condition variables and > critical sections. Perhaps the posix semaphore api is more appropriate > in this case. > > > > By the way, this also shows another problem with (old) python. There > is only one core locking primitive, the PyThread_type_lock. It is > being used both as a critical section in the traditional sense, and > also as this sort-of-inverse lock that the GIL is. In the modern > world, where the intended behaviour of these is quite different, there > is no one-size-fits all. On windows in particular, the use of the > Event object based lock is not ideal for other uses than the GIL. > > > > > > In the new GIL, there appear to be several problems: > > 1) There is no FIFO queue of threads wanting the queue, thus thread > scheduling becomes non-deterministic > > 2) The "ticking" of the GIL is now controled by a condition variable > timeout. There appears to be no way to prevent many such timeouts to > be in progress at the same time, thus you may have an unnecessarily > high rate of ticking going on. > > 3) There isn't an immediate gil request made when an IO thread > requests the gil back, only after an initial timeout. > > > > What we are trying to write here is a thread scheduler, and that is > complex business. > > K > > > > > > > >> -----Original Message----- > >> From: python-dev-bounces+kristjan=ccpgames.com at python.org > >> [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On > Behalf > >> Of David Beazley > >> Sent: 15. mars 2010 03:07 > >> To: python-dev at python.org > >> Subject: Re: [Python-Dev] "Fixing" the new GIL > >> > >> happen to be performing CPU intensive work at the same time, it > would > >> be nice if they didn't thrash on multiple cores (the problem with > the > >> old GIL) and if I/O is > > > From kristjan at ccpgames.com Tue Mar 16 15:10:33 2010 From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=) Date: Tue, 16 Mar 2010 14:10:33 +0000 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <20100316080011.34589b00@msiwind> References: <49653.1268622425@dabeaz.com> <930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local> <20100316080011.34589b00@msiwind> Message-ID: <930F189C8A437347B80DF2C156F7EC7F0A1821827D@exchis.ccp.ad.local> > -----Original Message----- > From: python-dev-bounces+kristjan=ccpgames.com at python.org > [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf > Of Antoine Pitrou > Sent: 16. mars 2010 12:00 > To: python-dev at python.org > Subject: Re: [Python-Dev] "Fixing" the new GIL > > > Hi Kristj?n, > > > In the new GIL, there appear to be several problems: > > 1) There is no FIFO queue of threads wanting the queue, thus thread > > scheduling becomes non-deterministic > > Thread scheduling has always been non-deterministic. Yes, but not the scheduling of "which thread has the GIL" which is what we are talking about. With a gil, you have neutered the non-deterministic OS scheduler and forced the system to _effectively_ schedule threads as you with them to be scheduled. The key to this is implementing your GIL in such a way that you (and not the system) chooses which threads runs next. On Windows it works in a nice, determinitstic FIFO order becoause that's how the underlying Event object is supposed to work. under pthreads, we rely on the behaviour of the condition variable which _probably_ is well behaved in this manner too. > > > 2) The "ticking" of the GIL is > > now controled by a condition variable timeout. There appears to be > > no way to prevent many such timeouts to be in progress at the same > > time, thus you may have an unnecessarily high rate of ticking going > > on. > > Unless there's a bug, no, there isn't. You're right, I had forgotten about "gil_switch_number". Never mind. > > > 3) There isn't an immediate gil request made when an IO thread > > requests the gil back, only after an initial timeout. > > This is what we are looking to fix (perhaps). > > > What we are trying to write here is a thread scheduler, and that is > > complex business. K > > I would have rephrased: > "What we are trying to avoid here is a thread scheduler". I can't argue with that, but I'm afraid that in the end, you have to. Python threads do behave very much in the same way as stackless python tasklets do. They own the cpu until they volounteeringly give it up. You want to decide, then, which of the threads next gets its turn. You are working on a completely different level than what the OS usually uses for its scheduling (timeslices, IO) so _you_ have to be in control. By a happy coincidence, in the past on a single CPU, the old GIL did give us this, without us being explicit about it: FIFO execution of tasklets. This fell out as a byproduct of the very simple way that the GIL was implemented, using a single Condition variable. Now that we have a different mechanism, _where threads time out_ on the condition, this behaviour no longer falls out of the implementation on its own. It may turn out, in this new, more complicated world, that you have to explicitly schedule the threads, by chaining them and giving each their own condition to wait on. Btw, if you want to take up a private converstaion with me, I'd like to point out to you some problems with the Windows version of the Condition variable :) Cheers, Kristj?n From solipsis at pitrou.net Tue Mar 16 15:16:23 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 16 Mar 2010 15:16:23 +0100 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F0A1821827D@exchis.ccp.ad.local> References: <49653.1268622425@dabeaz.com> <930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local> <20100316080011.34589b00@msiwind> <930F189C8A437347B80DF2C156F7EC7F0A1821827D@exchis.ccp.ad.local> Message-ID: <20100316151623.05e8e570@msiwind> Le Tue, 16 Mar 2010 14:10:33 +0000, Kristj?n Valur J?nsson a ?crit : > > The key to this > is implementing your GIL in such a way that you (and not the system) > chooses which threads runs next. On Windows it works in a nice, > determinitstic FIFO order becoause that's how the underlying Event > object is supposed to work. Well, I don't think this has ever been by design, and it's not obvious this is desirable either (see Dave Beazley's benchmark). > Btw, if you want to take up a private converstaion with me, I'd like > to point out to you some problems with the Windows version of the > Condition variable :) You can report a bug. cheers Antoine. From dickinsm at gmail.com Tue Mar 16 15:41:26 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 16 Mar 2010 14:41:26 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. Message-ID: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> Hello all, Currently in Python 2.x, Decimal-to-float comparisons behave as follows: >>> Decimal(1) < float(4) False >>> Decimal(4) < float(1) False That is, any Decimal sorts before any float (though this is arbitrary: it's possible that on some implementations any float sorts before any Decimal). This causes (a) confusion, and (b) bugs, especially when floats and Decimals are accidentally combined in a program. There probably aren't too many legitimate reasons for deliberately mixing floats and Decimals in the same calculation, so preventing accidents is the main concern here. In Python 3.x, however, such comparisons raise a TypeError ("unorderable types: Decimal() < float()"). http://bugs.python.org/issue2531 ('float compared to decimal is silently incorrect') was opened for this a while ago. I'm planning to commit a change to trunk that changes the behaviour so that the comparison result is based on the respective values of the arguments (so the results above would be True and False respectively). Question for python-dev people (and the point of this email): should this change be forward ported to py3k? On the one hand there's something to be said for maintaining a clean separation between the float and Decimal types, allowing only explicit conversions from one to the other; mixed-type arithmetic between floats and Decimals was very deliberately not permitted in the original PEP, and that's unlikely to change in a hurry. On the other hand, there's value in keeping 2.x and 3.x aligned where possible for the sake of 2-to-3 porters, and the new behaviour may even be useful. Even with the TypeError above, there are still some py3k surprises arising from the ability to compare ints and Decimals, and ints and floats, but not floats and Decimals. A quick tour of some of these surprises, in trunk: >>> from decimal import Decimal >>> Decimal(1) < 2 < float(3) < Decimal(1) # < is non-transitive True >>> Decimal(1) == 1 == float(1) # so is equality True >>> Decimal(1) == float(1) False >>> d1, i1, f1 = Decimal(1), float(1), 1 >>> set([d1, i1, f1]) == set([f1, i1, d1]) # sets with the same elements are different False >>> sorted([d1, i1, f1]) == sorted([f1, i1, d1]) False and in py3k: >>> from decimal import Decimal >>> Decimal(1) < 2 < float(3) < Decimal(1) Traceback (most recent call last): File "", line 1, in TypeError: unorderable types: float() < Decimal() >>> Decimal(1) == 1 == float(1) True >>> Decimal(1) == float(1) False >>> d1, i1, f1 = Decimal(1), float(1), 1 >>> set([d1, i1, f1]) == set([f1, i1, d1]) False >>> sorted([Decimal(1), 2, float(3)]) [Decimal('1'), 2, 3.0] >>> sorted([2, Decimal(1), float(3)]) Traceback (most recent call last): File "", line 1, in TypeError: unorderable types: float() < Decimal() >>> sorted([float(3), 2, Decimal(1)]) [Decimal('1'), 2, 3.0] By the way, even with the patch there are still problems with other numeric types: comparisons or set operations involving both Fraction and Decimal instances are going to cause similar difficulties to those above. In practice I think this is much less of an issue than the float/Decimal problem, since the chance of accidentally combining Fraction and Decimal types in a calculation seems significantly smaller than the chance of accidentally combining float and Decimal types. -- Mark From facundobatista at gmail.com Tue Mar 16 15:58:45 2010 From: facundobatista at gmail.com (Facundo Batista) Date: Tue, 16 Mar 2010 09:58:45 -0500 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> Message-ID: On Tue, Mar 16, 2010 at 9:41 AM, Mark Dickinson wrote: > On the one hand there's something to be said for maintaining a clean > separation between the float and Decimal types, allowing only explicit > conversions from one to the other; ?mixed-type arithmetic between > floats and Decimals was very deliberately not permitted in the > original PEP, and that's unlikely to change in a hurry. ?On the other But, to be fair, we didn't have "true value of the float at that time". I'm +0 to allow these comparisons, being "Decimal(1) < .3" the same as "Decimal(1) < Decimal.from_float(.3)" -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From janssen at parc.com Tue Mar 16 16:11:39 2010 From: janssen at parc.com (Bill Janssen) Date: Tue, 16 Mar 2010 08:11:39 PDT Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <4B9F3B21.1090106@v.loewis.de> References: <4B9D2BD4.8010307@v.loewis.de> <20100316053847.GA12552@cskk.homeip.net> <4B9F3B21.1090106@v.loewis.de> Message-ID: <82842.1268752299@parc.com> Martin v. L?wis wrote: > > If it's lurking behind a filesystem interface or in its daemon mode > > (remote archive store), multiple client processes can be using it at once, > > and it will be processing multiple tasks somewhat in parallel. Here one > > can get a compute bound thread answering one request, impacting quick > > response to other parallel-and-cheap requests. > > However, "impacting" will always be the case. *Of course* any thread > that performs computation impacts everything else on the same processor. That's not the problem, Martin. The problem is that it also impacts other threads on other cores, because of the interlock between the OS thread scheduler and the internal Pythonic struggle to obtain the GIL. And it shouldn't. Bill From dickinsm at gmail.com Tue Mar 16 16:17:47 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 16 Mar 2010 15:17:47 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> Message-ID: <5c6f2a5d1003160817jdfea407q4af516f1770bda19@mail.gmail.com> On Tue, Mar 16, 2010 at 2:58 PM, Facundo Batista wrote: > On Tue, Mar 16, 2010 at 9:41 AM, Mark Dickinson wrote: > >> On the one hand there's something to be said for maintaining a clean >> separation between the float and Decimal types, allowing only explicit >> conversions from one to the other; ?mixed-type arithmetic between >> floats and Decimals was very deliberately not permitted in the >> original PEP, and that's unlikely to change in a hurry. ?On the other > > But, to be fair, we didn't have "true value of the float at that time". That's true. I'd still be reluctant to start supporting operations like Decimal('1.2') + 0.71, though. At least comparisons have the nice feature that the return type is unambiguous. > > I'm +0 to allow these comparisons, being "Decimal(1) < .3" the same as > "Decimal(1) < Decimal.from_float(.3)" Yes, I should have clarified that those are exactly the semantics I'm proposing. Thanks! Mark From pje at telecommunity.com Tue Mar 16 16:58:22 2010 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 16 Mar 2010 11:58:22 -0400 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> Message-ID: <20100316155815.DAE983A4097@sparrow.telecommunity.com> At 09:58 AM 3/16/2010 -0500, Facundo Batista wrote: >I'm +0 to allow these comparisons, being "Decimal(1) < .3" the same as >"Decimal(1) < Decimal.from_float(.3)" Does Decimal.from_float() use the "shortest decimal representation" approach? If not, it might be confusing if a number that prints as '.1' compares unequal to Decimal('.1'). From dickinsm at gmail.com Tue Mar 16 17:11:38 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 16 Mar 2010 16:11:38 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <20100316155815.DAE983A4097@sparrow.telecommunity.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <20100316155815.DAE983A4097@sparrow.telecommunity.com> Message-ID: <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com> On Tue, Mar 16, 2010 at 3:58 PM, P.J. Eby wrote: > At 09:58 AM 3/16/2010 -0500, Facundo Batista wrote: >> >> I'm +0 to allow these comparisons, being "Decimal(1) < .3" the same as >> "Decimal(1) < Decimal.from_float(.3)" > > Does Decimal.from_float() use the "shortest decimal representation" > approach? No. It does exact conversions: >>> Decimal.from_float(1.1) Decimal('1.100000000000000088817841970012523233890533447265625') >>> Decimal.from_float(1.1) == 1.1 False >>> Decimal('1.1') == float('1.1') # returns False both pre- and post-patch False > If not, it might be confusing if a number that prints as '.1' compares > unequal to Decimal('.1'). Agreed, but this is just your everyday floating-point confusion, to be dealt with by social means (e.g., educating the programmer). Any technical solution that made "Decimal('1.1') == float('1.1')" evaluate to True would, I suspect, be a cure worse than the original disease. Mark From victor.stinner at haypocalc.com Tue Mar 16 17:18:50 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Tue, 16 Mar 2010 17:18:50 +0100 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <20100316155815.DAE983A4097@sparrow.telecommunity.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <20100316155815.DAE983A4097@sparrow.telecommunity.com> Message-ID: <201003161718.50634.victor.stinner@haypocalc.com> Le mardi 16 mars 2010 16:58:22, P.J. Eby a ?crit : > At 09:58 AM 3/16/2010 -0500, Facundo Batista wrote: > >I'm +0 to allow these comparisons, being "Decimal(1) < .3" the same as > >"Decimal(1) < Decimal.from_float(.3)" > > Does Decimal.from_float() use the "shortest decimal representation" > approach? > > If not, it might be confusing if a number that prints as '.1' > compares unequal to Decimal('.1'). In py3k, comparing bytes and str raise a TypeError("unorderable types: bytes() < str()"). I like this behaviour :-) If comparaison of Decimal and float can have "unpredictable" result, I would suggest the same behaviour (raise an error). -- Victor Stinner http://www.haypocalc.com/ From dickinsm at gmail.com Tue Mar 16 17:23:30 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 16 Mar 2010 16:23:30 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <20100316155815.DAE983A4097@sparrow.telecommunity.com> <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com> Message-ID: <5c6f2a5d1003160923v1519a3advf95df39cb7f884af@mail.gmail.com> On Tue, Mar 16, 2010 at 4:11 PM, Mark Dickinson wrote: [...] >>>> Decimal.from_float(1.1) == 1.1 > False Whoops. To clarify, this is the pre-patch behaviour; post-patch, this gives True. Mark From dickinsm at gmail.com Tue Mar 16 17:25:32 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 16 Mar 2010 16:25:32 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <201003161718.50634.victor.stinner@haypocalc.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <20100316155815.DAE983A4097@sparrow.telecommunity.com> <201003161718.50634.victor.stinner@haypocalc.com> Message-ID: <5c6f2a5d1003160925n637b9987t9af2c0bd02330bd3@mail.gmail.com> On Tue, Mar 16, 2010 at 4:18 PM, Victor Stinner wrote: > If comparaison of Decimal and float can have "unpredictable" result, I would > suggest the same behaviour (raise an error). Well, it's not really `unpredictable': the new behaviour is perfectly predictable and sane, provided only that you remember the basic fact that float("0.1") is not exactly 0.1. Mark From guido at python.org Tue Mar 16 17:41:48 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 16 Mar 2010 08:41:48 -0800 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> Message-ID: I'd say if you're not going to forward-port this to Python 3, it shouldn't go into Python 2 -- in that case it would make more sense to me to back-port the exception-raising behavior. Also supporting comparisons but not other mixed operations is going to be confusing. If you are sticking to that behavior I think mixed comparisons should also be ruled out. --Guido On Tue, Mar 16, 2010 at 7:41 AM, Mark Dickinson wrote: > Hello all, > > Currently in Python 2.x, Decimal-to-float comparisons behave as follows: > >>>> Decimal(1) < float(4) > False >>>> Decimal(4) < float(1) > False > > That is, any Decimal sorts before any float (though this is arbitrary: > ?it's possible that on some implementations any float sorts before any > Decimal). ?This causes (a) confusion, and (b) bugs, especially when > floats and Decimals are accidentally combined in a program. ?There > probably aren't too many legitimate reasons for deliberately mixing > floats and Decimals in the same calculation, so preventing accidents > is the main concern here. > > In Python 3.x, however, such comparisons raise a TypeError > ("unorderable types: Decimal() < float()"). > > http://bugs.python.org/issue2531 ('float compared to decimal is > silently incorrect') was opened for this a while ago. > > I'm planning to commit a change to trunk that changes the behaviour so > that the comparison result is based on the respective values of the > arguments (so the results above would be True and False respectively). > > Question for python-dev people (and the point of this email): should > this change be forward ported to py3k? > > On the one hand there's something to be said for maintaining a clean > separation between the float and Decimal types, allowing only explicit > conversions from one to the other; ?mixed-type arithmetic between > floats and Decimals was very deliberately not permitted in the > original PEP, and that's unlikely to change in a hurry. ?On the other > hand, there's value in keeping 2.x and 3.x aligned where possible for > the sake of 2-to-3 porters, and the new behaviour may even be useful. > Even with the TypeError above, there are still some py3k surprises > arising from the ability to compare ints and Decimals, and ints and > floats, but not floats and Decimals. > > A quick tour of some of these surprises, in trunk: > >>>> from decimal import Decimal >>>> Decimal(1) < 2 < float(3) < Decimal(1) ?# < is non-transitive > True >>>> Decimal(1) == 1 == float(1) ? # so is equality > True >>>> Decimal(1) == float(1) > False >>>> d1, i1, f1 = Decimal(1), float(1), 1 >>>> set([d1, i1, f1]) == set([f1, i1, d1]) ?# sets with the same elements are different > False >>>> sorted([d1, i1, f1]) == sorted([f1, i1, d1]) > False > > and in py3k: > >>>> from decimal import Decimal >>>> Decimal(1) < 2 < float(3) < Decimal(1) > Traceback (most recent call last): > ?File "", line 1, in > TypeError: unorderable types: float() < Decimal() >>>> Decimal(1) == 1 == float(1) > True >>>> Decimal(1) == float(1) > False >>>> d1, i1, f1 = Decimal(1), float(1), 1 >>>> set([d1, i1, f1]) == set([f1, i1, d1]) > False >>>> sorted([Decimal(1), 2, float(3)]) > [Decimal('1'), 2, 3.0] >>>> sorted([2, Decimal(1), float(3)]) > Traceback (most recent call last): > ?File "", line 1, in > TypeError: unorderable types: float() < Decimal() >>>> sorted([float(3), 2, Decimal(1)]) > [Decimal('1'), 2, 3.0] > > By the way, even with the patch there are still problems with other > numeric types: comparisons or set operations involving both Fraction > and Decimal instances are going to cause similar difficulties to those > above. ?In practice I think this is much less of an issue than the > float/Decimal problem, since the chance of accidentally combining > Fraction and Decimal types in a calculation seems significantly > smaller than the chance of accidentally combining float and Decimal > types. > > -- > Mark > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From kristjan at ccpgames.com Tue Mar 16 18:02:13 2010 From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=) Date: Tue, 16 Mar 2010 17:02:13 +0000 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <20100316151623.05e8e570@msiwind> References: <49653.1268622425@dabeaz.com> <930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local> <20100316080011.34589b00@msiwind> <930F189C8A437347B80DF2C156F7EC7F0A1821827D@exchis.ccp.ad.local> <20100316151623.05e8e570@msiwind> Message-ID: <930F189C8A437347B80DF2C156F7EC7F0A18218371@exchis.ccp.ad.local> > -----Original Message----- > From: python-dev-bounces+kristjan=ccpgames.com at python.org > [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf > Of Antoine Pitrou > > The key to this > > is implementing your GIL in such a way that you (and not the system) > > chooses which threads runs next. On Windows it works in a nice, > > determinitstic FIFO order becoause that's how the underlying Event > > object is supposed to work. > > Well, I don't think this has ever been by design, and it's not obvious > this is desirable either (see Dave Beazley's benchmark). > Did Dave benchmark a straight FIFO system? I can tell you out of experience, engineering a similar system (stacklessIO) that you _do_ want a deterministic scheduling algorithm. The most straightforward is round-robin (or FIFO), and the only sane option really if you know nothing about your threads. Upon this, you can start to build, by for example adding another priority layer, but you have to have a solid foundation. We have never had high priority for IO threads in python (and its not-by-design round robin scheduler on single core cpus) and it is unclear why that should be required now as a fix. > > Btw, if you want to take up a private converstaion with me, I'd like > > to point out to you some problems with the Windows version of the > > Condition variable :) > > You can report a bug. Or I could commit a fix. I just thought You might be interested, is all. I'll submit a patch. K From dickinsm at gmail.com Tue Mar 16 18:07:48 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 16 Mar 2010 17:07:48 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> Message-ID: <5c6f2a5d1003161007q2503f7acy3bc131fc14b9ae10@mail.gmail.com> On Tue, Mar 16, 2010 at 4:41 PM, Guido van Rossum wrote: > I'd say if you're not going to forward-port this to Python 3, it > shouldn't go into Python 2 -- in that case it would make more sense to > me to back-port the exception-raising behavior. That's also a possible solution, and the one that I'd personally be happiest with. The main problem is that this has the potential to break code: lists containing both floats and Decimals are sortable in 2.6, but would no longer be sortable in 2.7. If such breakage is deemed acceptable then I'd happily backport the exception; I really don't have a good feeling for how much real-world code could break, if any. > Also supporting comparisons but not other mixed operations is going to > be confusing. If you are sticking to that behavior I think mixed > comparisons should also be ruled out. Confusing, yes, but at least not bug-prone. The current 2.x behaviour has provoked complaints from a number of different people in various different fora (I recently saw this come up on StackOverflow), and after initially being skeptical I'm now convinced that it would be a good idea to change it if at all possible. Mark From guido at python.org Tue Mar 16 18:15:28 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 16 Mar 2010 09:15:28 -0800 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003161007q2503f7acy3bc131fc14b9ae10@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003161007q2503f7acy3bc131fc14b9ae10@mail.gmail.com> Message-ID: On Tue, Mar 16, 2010 at 9:07 AM, Mark Dickinson wrote: > On Tue, Mar 16, 2010 at 4:41 PM, Guido van Rossum wrote: >> I'd say if you're not going to forward-port this to Python 3, it >> shouldn't go into Python 2 -- in that case it would make more sense to >> me to back-port the exception-raising behavior. > > That's also a possible solution, and the one that I'd personally be > happiest with. ?The main problem is that this has the potential to > break code: ?lists containing both floats and Decimals are sortable in > 2.6, but would no longer be sortable in 2.7. ?If such breakage is > deemed acceptable then I'd happily backport the exception; ?I really > don't have a good feeling for how much real-world code could break, if > any. Definitely some. Stricter comparison rules are a frequent cause of problems when code is first ported to 3.x. While you'd think that code comparing a float and a Decimal is *already* broken, there's a surprising number of situations where that's not necessary the case, e.g. when an arbitrary but stable ordering is needed. >> Also supporting comparisons but not other mixed operations is going to >> be confusing. If you are sticking to that behavior I think mixed >> comparisons should also be ruled out. > > Confusing, yes, but at least not bug-prone. ?The current 2.x behaviour > has provoked complaints from a number of different people in various > different fora (I recently saw this come up on StackOverflow), and > after initially being skeptical I'm now convinced that it would be a > good idea to change it if at all possible. Yeah, it should have raised an exception all along. But it's too late for that now. I wonder if it should just become a py3k warning? -- --Guido van Rossum (python.org/~guido) From nir at winpdb.org Tue Mar 16 19:11:00 2010 From: nir at winpdb.org (Nir Aides) Date: Tue, 16 Mar 2010 20:11:00 +0200 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <5AD14AA1-72D3-4076-91BF-63B6C402139D@dabeaz.com> References: <49653.1268622425@dabeaz.com> <930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local> <5AD14AA1-72D3-4076-91BF-63B6C402139D@dabeaz.com> Message-ID: <5d2328511003161111vbf198d1y36e974d474e38a06@mail.gmail.com> Hello Dave, The following documentation suggests ordering in Linux is not FIFO: http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait.html#tag_03_518_08_06 "Threads waiting on mutexes and condition variables are selected to proceed in an order dependent upon the scheduling policy rather than in some fixed order (for example, FIFO or priority). Thus, the scheduling policy determines which thread(s) are awakened and allowed to proceed." Here is the code: http://www.google.com/codesearch/p?hl=en#5ge3gHPB4K4/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz%7CaeB7Uqo7T9g/linuxthreads/queue.h&q=pthread_cond_timedwait&exact_package=http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz If this is so then it should affect the proposed fixes. For example waiting with timeout should be avoided, no? Nir 2010/3/16 David Beazley > Python doesn't use a pthreads mutex for the GIL. It has always used a > binary semaphore implemented with condition variables (or just a pthreads > semaphore if available). The reason the performance is so bad is > precisely due to the fact that it is using this implementation and the fact > that there *IS* a FIFO queue of threads (associated with the condition > variable). The I/O performance problem with the new GIL is gets much worse > with many CPU-bound threads precisely because there is a FIFO queue > involved. This has been covered in my past GIL presentations. > > -Dave > > > > On Mar 16, 2010, at 5:52 AM, Kristj?n Valur J?nsson wrote: > > > How about attacking the original problem, then? > > > > The reason they thrash on pthreads implementation is that a pthreads > mutex is assumed to be a short-held resource. Therefore it will be > optimized in the following ways for multicore machines: > > 1) There is a certain amount of spinning done, to try to acquire it > before blocking > > 2) It will employ un-fair tactics to avoid lock-convoying, meaning that a > thread coming in to acquire the mutex may get in before others that are > queued. This is why "ticking" the GIL works so badly: The thread that > releases the lock is usually the one that reaquires it even though others > may be waiting. See e.g. > http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-616ef7b031aa.aspxfor a discussion of this (albeit on windows.) > > > > On Windows, this isn't a problem. The reason is, that the GIL on windows > is implemented using Event objects that don't cut these corners. The Event > provides you with a strict FIFO queue of objects waiting for the event. > > > > If pthreads doesn't provide a synchronization primitive similar to that, > someone that doesn't thrash and has a true FIFO queue, it is possible to > construct such a thing using condition variables and critical sections. > Perhaps the posix semaphore api is more appropriate in this case. > > > > By the way, this also shows another problem with (old) python. There is > only one core locking primitive, the PyThread_type_lock. It is being used > both as a critical section in the traditional sense, and also as this > sort-of-inverse lock that the GIL is. In the modern world, where the > intended behaviour of these is quite different, there is no one-size-fits > all. On windows in particular, the use of the Event object based lock is > not ideal for other uses than the GIL. > > > > > > In the new GIL, there appear to be several problems: > > 1) There is no FIFO queue of threads wanting the queue, thus thread > scheduling becomes non-deterministic > > 2) The "ticking" of the GIL is now controled by a condition variable > timeout. There appears to be no way to prevent many such timeouts to be in > progress at the same time, thus you may have an unnecessarily high rate of > ticking going on. > > 3) There isn't an immediate gil request made when an IO thread requests > the gil back, only after an initial timeout. > > > > What we are trying to write here is a thread scheduler, and that is > complex business. > > K > > > > > > > >> -----Original Message----- > >> From: python-dev-bounces+kristjan=ccpgames.com at python.org > >> [mailto:python-dev-bounces+kristjan = > ccpgames.com at python.org] On Behalf > >> Of David Beazley > >> Sent: 15. mars 2010 03:07 > >> To: python-dev at python.org > >> Subject: Re: [Python-Dev] "Fixing" the new GIL > >> > >> happen to be performing CPU intensive work at the same time, it would > >> be nice if they didn't thrash on multiple cores (the problem with the > >> old GIL) and if I/O is > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.a.portante at gmail.com Tue Mar 16 20:22:34 2010 From: peter.a.portante at gmail.com (Peter Portante) Date: Tue, 16 Mar 2010 14:22:34 -0500 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <5d2328511003161111vbf198d1y36e974d474e38a06@mail.gmail.com> Message-ID: Yes, having another thread wait with a timeout is not good. CPU cycles waisted doing things that don?t help the thread holding the GIL release it. They just note that they are present and then the GIL-holding-thread should be responsible for handling the rest. -peter On 3/16/10 1:11 PM, "Nir Aides" wrote: > Hello Dave,? > > The following documentation suggests ordering in Linux is not FIFO: > http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait > .html#tag_03_518_08_06 > "Threads waiting on mutexes and condition variables are selected to proceed in > an order dependent upon the scheduling policy rather than in some fixed order > (for example, FIFO or priority). Thus, the scheduling policy determines which > thread(s) are awakened and allowed to proceed." > > Here is the code: > http://www.google.com/codesearch/p?hl=en#5ge3gHPB4K4/gnu/glibc/glibc-linuxthre > ads-2.1.1.tar.gz%7CaeB7Uqo7T9g/linuxthreads/queue.h&q=pthread_cond_timedwait&e > xact_package=http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz > > If this is so then it should affect the proposed fixes.? > For example?waiting with timeout should be avoided, no? > > Nir > > 2010/3/16 David Beazley >> Python doesn't use a pthreads mutex for the GIL. ? ?It has always used a >> binary semaphore implemented with condition variables (or just a pthreads >> semaphore if available). ? ?The reason the performance is so bad is precisely >> due to the fact that it is using this implementation and the fact that there >> *IS* a FIFO queue of threads (associated with the condition variable). ? The >> I/O performance problem with the new GIL is gets much worse with many >> CPU-bound threads precisely because there is a FIFO queue involved. ? This >> has been covered in my past GIL presentations. >> >> -Dave >> >> >> >> On Mar 16, 2010, at 5:52 AM, Kristj?n Valur J?nsson wrote: >> >>> > How about attacking the original problem, then? >>> > >>> > The reason they thrash on pthreads implementation is that a pthreads mutex >>> is assumed to be a short-held resource. ?Therefore it will be optimized in >>> the following ways for multicore machines: >>> > 1) There is a certain amount of spinning done, to try to acquire it before >>> blocking >>> > 2) It will employ un-fair tactics to avoid lock-convoying, meaning that a >>> thread coming in to acquire the mutex may get in before others that are >>> queued. ?This is why "ticking" the GIL works so badly: ?The thread that >>> releases the lock is usually the one that reaquires it even though others >>> may be waiting. ?See e.g. >>> http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85- >>> 616ef7b031aa.aspx for a discussion of this (albeit on windows.) >>> > >>> > On Windows, this isn't a problem. ?The reason is, that the GIL on windows >>> is implemented using Event objects that don't cut these corners. ?The Event >>> provides you with a strict FIFO queue of objects waiting for the event. >>> > >>> > If pthreads doesn't provide a synchronization primitive similar to that, >>> someone that doesn't thrash and has a true FIFO queue, it is possible to >>> construct such a thing using condition variables and critical sections. >>> ?Perhaps the posix semaphore api is more appropriate in this case. >>> > >>> > By the way, this also shows another problem with (old) python. ?There is >>> only one core locking primitive, the PyThread_type_lock. ?It is being used >>> both as a critical section in the traditional sense, and also as this >>> sort-of-inverse lock that the GIL is. ?In the modern world, where the >>> intended behaviour of these is quite different, there is no one-size-fits >>> all. ?On windows in particular, the use of the Event object based lock is >>> not ideal for other uses than the GIL. >>> > >>> > >>> > In the new GIL, there appear to be several problems: >>> > 1) There is no FIFO queue of threads wanting the queue, thus thread >>> scheduling becomes non-deterministic >>> > 2) The "ticking" of the GIL is now controled by a condition variable >>> timeout. ?There appears to be no way to prevent many such timeouts to be in >>> progress at the same time, thus you may have an unnecessarily high rate of >>> ticking going on. >>> > 3) There isn't an immediate gil request made when an IO thread requests >>> the gil back, only after an initial timeout. >>> > >>> > What we are trying to write here is a thread scheduler, and that is >>> complex business. >>> > K >>> > >>> > >>> > >>>> >> -----Original Message----- >>>> >> From: python-dev-bounces+kristjan=ccpgames.com >>>> @python.org >>>> >> [mailto:python-dev-bounces+kristjan >>>> =ccpgames.com >>>> @python.org ] On Behalf >>>> >> Of David Beazley >>>> >> Sent: 15. mars 2010 03:07 >>>> >> To: python-dev at python.org >>>> >> Subject: Re: [Python-Dev] "Fixing" the new GIL >>>> >> >>>> >> happen to be performing CPU intensive work at the same time, it would >>>> >> be nice if they didn't thrash on multiple cores (the problem with the >>>> >> old GIL) and if I/O is >>> > >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From hancock.robert at gmail.com Tue Mar 16 21:09:39 2010 From: hancock.robert at gmail.com (Robert Hancock) Date: Tue, 16 Mar 2010 16:09:39 -0400 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: References: <5d2328511003161111vbf198d1y36e974d474e38a06@mail.gmail.com> Message-ID: <613d56581003161309p147c45f5ub0f142f1cd160776@mail.gmail.com> The Linux kernel scheduler deals with two types of ratings and has a heuristic algorithm that rewards and penalizes on a rapid basis. To determine the next thread it inspects the priority array to find the highest priority task that is runnable and then selects the first task in that priority. This is a gross simplification of an extremely complex system, but it is not simply a simple queue. To duplicate this exact type of scheduling in the interpreter would be a daunting task, but Dave's example illustrates that some system of reward and penalty can be beneficial. Peter has brought up the point, both here and at the Open Space at Pycon, that before attempting a downsized version of the a priority thread scheduler that we should look at other programs that have had to deal with similar problems. Peter, since you have dealt with the nitty-gritty of concurrency before, can you suggest some applications that could serve as models for research? On Tue, Mar 16, 2010 at 3:22 PM, Peter Portante wrote: > Yes, having another thread wait with a timeout is not good. CPU cycles > waisted doing things that don?t help the thread holding the GIL release it. > They just note that they are present and then the GIL-holding-thread should > be responsible for handling the rest. -peter > > > > On 3/16/10 1:11 PM, "Nir Aides" wrote: > > Hello Dave, > > The following documentation suggests ordering in Linux is not FIFO: > > http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait.html#tag_03_518_08_06 > "Threads waiting on mutexes and condition variables are selected to > proceed in an order dependent upon the scheduling policy rather than in some > fixed order (for example, FIFO or priority). Thus, the scheduling policy > determines which thread(s) are awakened and allowed to proceed." > > Here is the code: > > http://www.google.com/codesearch/p?hl=en#5ge3gHPB4K4/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz%7CaeB7Uqo7T9g/linuxthreads/queue.h&q=pthread_cond_timedwait&exact_package=http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz > > If this is so then it should affect the proposed fixes. > For example waiting with timeout should be avoided, no? > > Nir > > 2010/3/16 David Beazley > > Python doesn't use a pthreads mutex for the GIL. It has always used a > binary semaphore implemented with condition variables (or just a pthreads > semaphore if available). The reason the performance is so bad is > precisely due to the fact that it is using this implementation and the fact > that there *IS* a FIFO queue of threads (associated with the condition > variable). The I/O performance problem with the new GIL is gets much worse > with many CPU-bound threads precisely because there is a FIFO queue > involved. This has been covered in my past GIL presentations. > > -Dave > > > > On Mar 16, 2010, at 5:52 AM, Kristj?n Valur J?nsson wrote: > > > How about attacking the original problem, then? > > > > The reason they thrash on pthreads implementation is that a pthreads > mutex is assumed to be a short-held resource. Therefore it will be > optimized in the following ways for multicore machines: > > 1) There is a certain amount of spinning done, to try to acquire it > before blocking > > 2) It will employ un-fair tactics to avoid lock-convoying, meaning that a > thread coming in to acquire the mutex may get in before others that are > queued. This is why "ticking" the GIL works so badly: The thread that > releases the lock is usually the one that reaquires it even though others > may be waiting. See e.g. > http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-616ef7b031aa.aspxfor a discussion of this (albeit on windows.) > > > > On Windows, this isn't a problem. The reason is, that the GIL on windows > is implemented using Event objects that don't cut these corners. The Event > provides you with a strict FIFO queue of objects waiting for the event. > > > > If pthreads doesn't provide a synchronization primitive similar to that, > someone that doesn't thrash and has a true FIFO queue, it is possible to > construct such a thing using condition variables and critical sections. > Perhaps the posix semaphore api is more appropriate in this case. > > > > By the way, this also shows another problem with (old) python. There is > only one core locking primitive, the PyThread_type_lock. It is being used > both as a critical section in the traditional sense, and also as this > sort-of-inverse lock that the GIL is. In the modern world, where the > intended behaviour of these is quite different, there is no one-size-fits > all. On windows in particular, the use of the Event object based lock is > not ideal for other uses than the GIL. > > > > > > In the new GIL, there appear to be several problems: > > 1) There is no FIFO queue of threads wanting the queue, thus thread > scheduling becomes non-deterministic > > 2) The "ticking" of the GIL is now controled by a condition variable > timeout. There appears to be no way to prevent many such timeouts to be in > progress at the same time, thus you may have an unnecessarily high rate of > ticking going on. > > 3) There isn't an immediate gil request made when an IO thread requests > the gil back, only after an initial timeout. > > > > What we are trying to write here is a thread scheduler, and that is > complex business. > > K > > > > > > > >> -----Original Message----- > >> From: python-dev-bounces+kristjan=ccpgames.com @ > python.org > >> [mailto:python-dev-bounces+kristjan < > mailto:python-dev-bounces%2Bkristjan > = > ccpgames.com @python.org ] On > Behalf > >> Of David Beazley > >> Sent: 15. mars 2010 03:07 > >> To: python-dev at python.org > >> Subject: Re: [Python-Dev] "Fixing" the new GIL > >> > >> happen to be performing CPU intensive work at the same time, it would > >> be nice if they didn't thrash on multiple cores (the problem with the > >> old GIL) and if I/O is > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org > > > > ------------------------------ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/hancock.robert%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sridharr at activestate.com Tue Mar 16 21:31:54 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Tue, 16 Mar 2010 13:31:54 -0700 Subject: [Python-Dev] Python 2.6.5 rc 2 In-Reply-To: <20100309075636.09f52b27@heresy.wooz.org> References: <20100309075636.09f52b27@heresy.wooz.org> Message-ID: <314E71ED-C3BA-47B2-9BD3-0D6BFE6B6AEA@activestate.com> I just verified with our ActivePython build that 2.6.4rc2 builds fine on Linux, Windows, Mac, HP-UX, AIX and Solaris. 3.1.2rc1 builds fine except on AIX[1] and HP-UX[2] but those issues existed in 3.1.1 too, I believe. -srid [1] http://bugs.python.org/issue6645 [2] http://bugs.python.org/issue5999 On 2010-03-09, at 4:56 AM, Barry Warsaw wrote: > Hi Python hackateers! > > It looks like we finally have no more release blockers for 2.6.5rc2. I would > like to tag the tree tonight for rc2 so that Martin can build the Windows > installer for a release tomorrow. I am also moving the final release back to > Friday March 19. > > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/sridharr%40activestate.com From benjamin at python.org Tue Mar 16 21:51:10 2010 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 16 Mar 2010 15:51:10 -0500 Subject: [Python-Dev] tagging 3.1.2 Message-ID: <1afaf6161003161351x664aa033i38cd7deb387012da@mail.gmail.com> My plan is to tag 3.1.2 sometime on Thursday, so binaries can be built for the final release on Saturday. Agreeable? -- Regards, Benjamin From brian at sweetapp.com Tue Mar 16 21:52:22 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Tue, 16 Mar 2010 13:52:22 -0700 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4B979292.4080906@gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> <20100307154809.742E73A4062@sparrow.telecommunity.com> <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com> <20100307175732.307513A4062@sparrow.telecommunity.com> <5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com> <20100307195614.BBD693A4062@sparrow.telecommunity.com> <4B94DC30.2030904@gmail.com> <19349.25574.880099.668647@montanaro.dyndns.org> <4B979292.4080906@gmail.com> Message-ID: Nick Coghlan wrote: > You may want to consider providing global thread and process executors > in the futures module itself. Code which just wants to say "do this in > the background" without having to manage the lifecycle of its own > executor instance is then free to do so. I've had a lot of experience > with a framework that provides this and it is *very* convenient (it's > also a good way to avoid deadlocks due to synchronous notification APIs). This seems like a reasonable idea to me. I take it that the thread/process pool should be unlimited in size. Should every thread/process exit when it finishes its job or should there be a smarter collection strategy? Cheers, Brian From barry at python.org Tue Mar 16 22:06:36 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 16 Mar 2010 17:06:36 -0400 Subject: [Python-Dev] tagging 2.6.5 (Re: tagging 3.1.2) In-Reply-To: <1afaf6161003161351x664aa033i38cd7deb387012da@mail.gmail.com> References: <1afaf6161003161351x664aa033i38cd7deb387012da@mail.gmail.com> Message-ID: <20100316170636.69c6a141@heresy> On Mar 16, 2010, at 03:51 PM, Benjamin Peterson wrote: >My plan is to tag 3.1.2 sometime on Thursday, so binaries can be built >for the final release on Saturday. Agreeable? I might as well do the same for 2.6.5, though I plan on releasing it Friday. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From greg.ewing at canterbury.ac.nz Tue Mar 16 23:10:41 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 17 Mar 2010 11:10:41 +1300 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> Message-ID: <4BA001E1.5050106@canterbury.ac.nz> Mark Dickinson wrote: > On the one hand there's something to be said for maintaining a clean > separation between the float and Decimal types, allowing only explicit > conversions from one to the other; mixed-type arithmetic between > floats and Decimals was very deliberately not permitted in the > original PEP, and that's unlikely to change in a hurry. I think that as long as you're disallowing arithmetic between float and decimal, comparison should be disallowed as well. So if you're going to change anything, it would be better to make such comparisons raise a TypeError in 2.7 to match the current 3.x behaviour. -- Greg From greg.ewing at canterbury.ac.nz Tue Mar 16 23:16:11 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 17 Mar 2010 11:16:11 +1300 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <20100316155815.DAE983A4097@sparrow.telecommunity.com> <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com> Message-ID: <4BA0032B.4080307@canterbury.ac.nz> Mark Dickinson wrote: > On Tue, Mar 16, 2010 at 3:58 PM, P.J. Eby wrote: > >>If not, it might be confusing if a number that prints as '.1' compares >>unequal to Decimal('.1'). > > Agreed, but this is just your everyday floating-point confusion, to be > dealt with by social means (e.g., educating the programmer). Seems to me that this education would mostly consist of saying "don't compare floats and decimals", which is why I think that disallowing them in the first place would be better. Then if a programmer truly needs to compare them for some reason, he has to be explicit about how to do it. -- Greg From ncoghlan at gmail.com Tue Mar 16 23:08:14 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 17 Mar 2010 08:08:14 +1000 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F0A18218371@exchis.ccp.ad.local> References: <49653.1268622425@dabeaz.com> <930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local> <20100316080011.34589b00@msiwind> <930F189C8A437347B80DF2C156F7EC7F0A1821827D@exchis.ccp.ad.local> <20100316151623.05e8e570@msiwind> <930F189C8A437347B80DF2C156F7EC7F0A18218371@exchis.ccp.ad.local> Message-ID: <4BA0014E.9010207@gmail.com> Kristj?n Valur J?nsson wrote: > We have never had high priority for IO threads in > python (and its not-by-design round robin scheduler on single core > cpus) and it is unclear why that should be required now as a fix. David explained that in the issue tracker - 2.x typically doesn't do that much work per bytecode instruction, so the interpreter releases and reacquires the GIL far more frequently in a CPU-bound thread than it does under the new time-based check interval. The current settings mean we have less GIL overhead in the normal case, but worse worst-case I/O latency. There are two fairly straightforward ways to handle that: - cut the check interval duration drastically (trading GIL overhead for I/O responsiveness, as in 2.x) - add some form of I/O thread prioritisation, such as Antoine's gilinter patch (which automatically identifies and prioritises "interactive" threads at the cost of a dozen or so additional lines of C code and a small amount of overhead on GIL context switches to adjust thread interactivity counters). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Mar 16 23:10:41 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 17 Mar 2010 08:10:41 +1000 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> <20100307154809.742E73A4062@sparrow.telecommunity.com> <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com> <20100307175732.307513A4062@sparrow.telecommunity.com> <5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com> <20100307195614.BBD693A4062@sparrow.telecommunity.com> <4B94DC30.2030904@gmail.com> <19349.25574.880099.668647@montanaro.dyndns.org> <4B979292.4080906@gmail.com> Message-ID: <4BA001E1.8080105@gmail.com> Brian Quinlan wrote: > I take it that the thread/process pool should be unlimited in size. > Should every thread/process exit when it finishes its job or should > there be a smarter collection strategy? I'd be inclined to do something slightly smarter, similar to what we do with memory overallocation for mutable containers. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Tue Mar 16 23:22:25 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 16 Mar 2010 23:22:25 +0100 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <4BA0014E.9010207@gmail.com> References: <49653.1268622425@dabeaz.com> <930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local> <20100316080011.34589b00@msiwind> <930F189C8A437347B80DF2C156F7EC7F0A1821827D@exchis.ccp.ad.local> <20100316151623.05e8e570@msiwind> <930F189C8A437347B80DF2C156F7EC7F0A18218371@exchis.ccp.ad.local> <4BA0014E.9010207@gmail.com> Message-ID: <1268778145.32204.38.camel@localhost> > David explained that in the issue tracker - 2.x typically doesn't do > that much work per bytecode instruction, Oh, but that's wrong in general. Dave's *spinning loop* doesn't do much work per bytecode instruction, however ;) > The current settings mean we have less GIL overhead in the normal case, > but worse worst-case I/O latency. Actually, ccbench shows that worst case IO latency is much worse in 2.x (when executing bytecodes which do a lot of work, e.g. matching a regex). What happens though is that best case IO latency is better in 2.x (e.g. spinning loop, or short opcodes approaching the spinning loop case :-)). cheers Antoine. From steve at pearwood.info Tue Mar 16 23:32:08 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 17 Mar 2010 09:32:08 +1100 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003160923v1519a3advf95df39cb7f884af@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com> <5c6f2a5d1003160923v1519a3advf95df39cb7f884af@mail.gmail.com> Message-ID: <201003170932.09201.steve@pearwood.info> On Wed, 17 Mar 2010 03:23:30 am Mark Dickinson wrote: > On Tue, Mar 16, 2010 at 4:11 PM, Mark Dickinson > wrote: [...] > > >>>> Decimal.from_float(1.1) == 1.1 > > > > False > > Whoops. To clarify, this is the pre-patch behaviour; post-patch, > this gives True. Whew! You had me worried there for a second. Just to clarify, you are proposing: Decimal.from_float(1.1) == 1.1 Decimal.('1.1') != float('1.1') +1 on this behaviour, even in the absence of supporting mixed Decimal and float arithmetic operations. Both Decimals and floats are representations of real numbers, and not being able to compare two numbers is just weird: refusing to compare (say) Decimal(1) with float(1) makes as little sense to me as refusing to compare int(1) with float(1). But mixed arithmetic runs into the problem, what do you want the result type to be? Given (say) decimal+float, returning either a Decimal or a float will be the wrong thing to do some of the time, so better to prohibit mixed arithmetic and let folks handle their own conversions. So +1 on continuing to prohibit mixed arithmetic. But no such problems arise with comparisons, which will always return a bool, and will avoid the current ... interesting ... behaviour. In 3.1: >>> Decimal(1) == 1 == 1.0 True >>> Decimal(1) == 1.0 False >>> Decimal.from_float(1.0) == 1 == 1.0 True >>> Decimal.from_float(1.0) == 1.0 False Replacing False with an exception doesn't make it any less bizarre. -- Steven D'Aprano From solipsis at pitrou.net Tue Mar 16 23:37:36 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 16 Mar 2010 22:37:36 +0000 (UTC) Subject: [Python-Dev] "Fixing" the new GIL References: <49653.1268622425@dabeaz.com> <930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local> <20100316080011.34589b00@msiwind> <930F189C8A437347B80DF2C156F7EC7F0A1821827D@exchis.ccp.ad.local> <20100316151623.05e8e570@msiwind> <930F189C8A437347B80DF2C156F7EC7F0A18218371@exchis.ccp.ad.local> Message-ID: Le Tue, 16 Mar 2010 17:02:13 +0000, Kristj?n Valur J?nsson a ?crit?: >> >> Well, I don't think this has ever been by design, and it's not obvious >> this is desirable either (see Dave Beazley's benchmark). >> > Did Dave benchmark a straight FIFO system? Well, he presented benchmark figures with two threads, which is a FIFO system with the new GIL (because when a thread releases the GIL, it ensures that another thread takes it). > We have never had high priority for IO threads in python > (and its not-by-design round robin scheduler on single core cpus) and it > is unclear why that should be required now as a fix. I agree that Dave's situation doesn't seem new to me, although he often presents it as a deficiency of the new GIL only. Regards Antoine. From martin at v.loewis.de Tue Mar 16 23:44:25 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 16 Mar 2010 23:44:25 +0100 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <20100316091544.GA3799@cskk.homeip.net> References: <4B9F3A54.3040907@v.loewis.de> <20100316091544.GA3799@cskk.homeip.net> Message-ID: <4BA009C9.8090000@v.loewis.de> > You can't. This isn't an example where I am personally bitten by the GIL. > I may be, but there's plenty of other stuff in terms of tuning or > optimisation in my particular app before I get anywhere near the GIL. > > My point is not that it's biting me right now but that earlier you > said the scenario was probably unrealistic, but I have an app which can > probably exhibit exactly the issue under discussion. I don't understand. First you say that there is plenty of other stuff .... before you get anywhere near the GIL, and then you say that your app can probably exhibit the issue under discussion. Which one is it? I can easily believe that there is tons of other stuff that make the GIL contention irrelevant, I can not believe that it will exhibit the issue under discussion. > | > The idea here is that one has a few threads receiving requests (eg a > | > daemon watching a socket or monitoring a db queue table) which then use > | > the FuncMultiQueue to manage how many actual requests are processed > | > in parallel (yes, a semaphore can cover a lot of this, but not the > | > asynchronous call modes). > | > | Why do you say processing is in parallel? > | In Python, processing is > | normally never in parallel, but always sequential (potentially > | interleaving). Are you releasing the GIL for processing? > > > I mean that I can have multiple actual requests being served, and they > are _not_ handled sequentially - the multi queue is to permit more than > one to be in progress at once so that an expensive request need not > inherently delay a following cheap request But, in Python, due to the GIL, *everything* is sequential, *nothing* is in parallel. > Can't tell you - I'm not claiming the current GIL behaviour is biting me > right now; I'm saying your claim that the scenario is unrealistic is a > bit unfair; my app wil probably be just the kind of app to be affected, > even only subtly. And it can hardly be alone in the "daemon" world. Unfortunately, you are not able to demonstrate this (because the effect would only be subtle). My claim is that the scenario where the effect is *visible* is unrealistic, i.e. that, in realistic scenarios, tons of other effects actually lessen the problem. Regards, Martin From ncoghlan at gmail.com Tue Mar 16 23:46:04 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 17 Mar 2010 08:46:04 +1000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <201003170932.09201.steve@pearwood.info> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com> <5c6f2a5d1003160923v1519a3advf95df39cb7f884af@mail.gmail.com> <201003170932.09201.steve@pearwood.info> Message-ID: <4BA00A2C.8070204@gmail.com> Steven D'Aprano wrote: > But no such problems arise with comparisons, which will always return a > bool, and will avoid the current ... interesting ... behaviour. In 3.1: > >>>> Decimal(1) == 1 == 1.0 > True >>>> Decimal(1) == 1.0 > False >>>> Decimal.from_float(1.0) == 1 == 1.0 > True >>>> Decimal.from_float(1.0) == 1.0 > False > > Replacing False with an exception doesn't make it any less bizarre. Allowing the comparisons also doesn't introduce the potential for large cumulative errors which are possible when actual implicit arithmetic conversions are allowed. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From steve at pearwood.info Tue Mar 16 23:46:39 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 17 Mar 2010 09:46:39 +1100 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <4BA0032B.4080307@canterbury.ac.nz> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com> <4BA0032B.4080307@canterbury.ac.nz> Message-ID: <201003170946.39457.steve@pearwood.info> On Wed, 17 Mar 2010 09:16:11 am Greg Ewing wrote: > Mark Dickinson wrote: > > On Tue, Mar 16, 2010 at 3:58 PM, P.J. Eby wrote: > >>If not, it might be confusing if a number that prints as '.1' > >> compares unequal to Decimal('.1'). > > > > Agreed, but this is just your everyday floating-point confusion, to > > be dealt with by social means (e.g., educating the programmer). > > Seems to me that this education would mostly consist of saying > "don't compare floats and decimals", which is why I think that > disallowing them in the first place would be better. I'm sure you don't mean to suggest that the only (or even the main) source of floating point confusion comes from mixed Decimal/float operations. > Then if a programmer truly needs to compare them for some > reason, he has to be explicit about how to do it. More explicit than someDecimal == someFloat? Seems pretty explicit to me. -- Steven D'Aprano From guido at python.org Wed Mar 17 00:01:12 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 16 Mar 2010 15:01:12 -0800 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <201003170932.09201.steve@pearwood.info> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com> <5c6f2a5d1003160923v1519a3advf95df39cb7f884af@mail.gmail.com> <201003170932.09201.steve@pearwood.info> Message-ID: On Tue, Mar 16, 2010 at 2:32 PM, Steven D'Aprano wrote: > But mixed arithmetic runs into the problem, what do you want the result > type to be? Given (say) decimal+float, returning either a Decimal or a > float will be the wrong thing to do some of the time, so better to > prohibit mixed arithmetic and let folks handle their own conversions. > So +1 on continuing to prohibit mixed arithmetic. I'm not disagreeing, but I really wonder, what is the value of supporting mixed comparisons then? Just because you *can* assign a meaning to it doesn't mean you should. OTOH I'm sure a lot of people would like to see mixed arithmetic supported, the PEP be damned, and they would probably be happy with any simple rule about the return type even if it's not always ideal. I note that there are cases where converting a long to a float also is the wrong thing to do, and yet mixed long/float operations always return floats. If you are amenable to this argument, I would propose to make the result of mixed operations return a Decimal, since in some "intuitive complexity" sense an int is a simpler type than a float and a float is a simpler type than a Decimal -- so results return the more complex type. But my intuition on this isn't super strong and I could live with always returning a float as well -- there are always casts to force the issue. -- --Guido van Rossum (python.org/~guido) From nir at winpdb.org Wed Mar 17 00:05:45 2010 From: nir at winpdb.org (Nir Aides) Date: Wed, 17 Mar 2010 01:05:45 +0200 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <5d2328511003161111vbf198d1y36e974d474e38a06@mail.gmail.com> References: <49653.1268622425@dabeaz.com> <930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local> <5AD14AA1-72D3-4076-91BF-63B6C402139D@dabeaz.com> <5d2328511003161111vbf198d1y36e974d474e38a06@mail.gmail.com> Message-ID: <5d2328511003161605j771a2feai83b981594b33b77d@mail.gmail.com> Hi, I posted a small patch to the GIL which demonstrates the scheduling policy of pthreads conditions. It seems that with pthreads a good policy is to allow (and help) the OS to manage scheduling of threads via the condition queue without introducing scheduling logic. Some changes include removing the timeout from the new GIL wait, and allowing CPU bound threads to run long enough to let the OS recognize them as such. The patch uses ticks countdown-to-switch but it should be changed to a time based countdown (If taking a look at the clock is expensive maybe it can be done once every N ticks during countdown). Remains to explore what can be done on other platforms. Nir 2010/3/16 Nir Aides > Hello Dave, > > The following documentation suggests ordering in Linux is not FIFO: > > http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait.html#tag_03_518_08_06 > "Threads waiting on mutexes and condition variables are selected to > proceed in an order dependent upon the scheduling policy rather than in some > fixed order (for example, FIFO or priority). Thus, the scheduling policy > determines which thread(s) are awakened and allowed to proceed." > > Here is the code: > > http://www.google.com/codesearch/p?hl=en#5ge3gHPB4K4/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz%7CaeB7Uqo7T9g/linuxthreads/queue.h&q=pthread_cond_timedwait&exact_package=http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz > > If this is so then it should affect the proposed fixes. > For example waiting with timeout should be avoided, no? > > Nir > > 2010/3/16 David Beazley > >> Python doesn't use a pthreads mutex for the GIL. It has always used a >> binary semaphore implemented with condition variables (or just a pthreads >> semaphore if available). The reason the performance is so bad is >> precisely due to the fact that it is using this implementation and the fact >> that there *IS* a FIFO queue of threads (associated with the condition >> variable). The I/O performance problem with the new GIL is gets much worse >> with many CPU-bound threads precisely because there is a FIFO queue >> involved. This has been covered in my past GIL presentations. >> >> >> -Dave >> >> >> >> On Mar 16, 2010, at 5:52 AM, Kristj?n Valur J?nsson wrote: >> >> > How about attacking the original problem, then? >> > >> > The reason they thrash on pthreads implementation is that a pthreads >> mutex is assumed to be a short-held resource. Therefore it will be >> optimized in the following ways for multicore machines: >> > 1) There is a certain amount of spinning done, to try to acquire it >> before blocking >> > 2) It will employ un-fair tactics to avoid lock-convoying, meaning that >> a thread coming in to acquire the mutex may get in before others that are >> queued. This is why "ticking" the GIL works so badly: The thread that >> releases the lock is usually the one that reaquires it even though others >> may be waiting. See e.g. >> http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-616ef7b031aa.aspxfor a discussion of this (albeit on windows.) >> > >> > On Windows, this isn't a problem. The reason is, that the GIL on >> windows is implemented using Event objects that don't cut these corners. >> The Event provides you with a strict FIFO queue of objects waiting for the >> event. >> > >> > If pthreads doesn't provide a synchronization primitive similar to that, >> someone that doesn't thrash and has a true FIFO queue, it is possible to >> construct such a thing using condition variables and critical sections. >> Perhaps the posix semaphore api is more appropriate in this case. >> > >> > By the way, this also shows another problem with (old) python. There is >> only one core locking primitive, the PyThread_type_lock. It is being used >> both as a critical section in the traditional sense, and also as this >> sort-of-inverse lock that the GIL is. In the modern world, where the >> intended behaviour of these is quite different, there is no one-size-fits >> all. On windows in particular, the use of the Event object based lock is >> not ideal for other uses than the GIL. >> > >> > >> > In the new GIL, there appear to be several problems: >> > 1) There is no FIFO queue of threads wanting the queue, thus thread >> scheduling becomes non-deterministic >> > 2) The "ticking" of the GIL is now controled by a condition variable >> timeout. There appears to be no way to prevent many such timeouts to be in >> progress at the same time, thus you may have an unnecessarily high rate of >> ticking going on. >> > 3) There isn't an immediate gil request made when an IO thread requests >> the gil back, only after an initial timeout. >> > >> > What we are trying to write here is a thread scheduler, and that is >> complex business. >> > K >> > >> > >> > >> >> -----Original Message----- >> >> From: python-dev-bounces+kristjan=ccpgames.com at python.org >> >> [mailto:python-dev-bounces+kristjan = >> ccpgames.com at python.org] On Behalf >> >> Of David Beazley >> >> Sent: 15. mars 2010 03:07 >> >> To: python-dev at python.org >> >> Subject: Re: [Python-Dev] "Fixing" the new GIL >> >> >> >> happen to be performing CPU intensive work at the same time, it would >> >> be nice if they didn't thrash on multiple cores (the problem with the >> >> old GIL) and if I/O is >> > >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Wed Mar 17 00:06:55 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 17 Mar 2010 00:06:55 +0100 Subject: [Python-Dev] tagging 3.1.2 In-Reply-To: <1afaf6161003161351x664aa033i38cd7deb387012da@mail.gmail.com> References: <1afaf6161003161351x664aa033i38cd7deb387012da@mail.gmail.com> Message-ID: <4BA00F0F.8070504@v.loewis.de> Benjamin Peterson wrote: > My plan is to tag 3.1.2 sometime on Thursday, so binaries can be built > for the final release on Saturday. Agreeable? Fine with me. Martin From raymond.hettinger at gmail.com Wed Mar 17 01:36:36 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Tue, 16 Mar 2010 17:36:36 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> Message-ID: <594D6395-4B01-4C9C-8417-36866992C338@gmail.com> On Mar 16, 2010, at 9:41 AM, Guido van Rossum wrote: > I'd say if you're not going to forward-port this to Python 3, it > shouldn't go into Python 2 -- in that case it would make more sense to > me to back-port the exception-raising behavior. Python 3 doesn't need it because it is possible to not give a result at all. Python 2 does need it because we have to give *some* result. > > Also supporting comparisons but not other mixed operations is going to > be confusing. If you are sticking to that behavior I think mixed > comparisons should also be ruled out. The difference is that mixed comparisons currently do give a result, but one that is non-sensical. The proposal is a make in give a meaningful result, not as an extra feature, but in an effort to not be wrong. Since 2.x has to give a result, we should make it useful. Since 3.x does not make the comparison, it is okay to punt. Raymond From guido at python.org Wed Mar 17 02:05:29 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 16 Mar 2010 18:05:29 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <594D6395-4B01-4C9C-8417-36866992C338@gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <594D6395-4B01-4C9C-8417-36866992C338@gmail.com> Message-ID: On Tue, Mar 16, 2010 at 5:36 PM, Raymond Hettinger wrote: > > > > On Mar 16, 2010, at 9:41 AM, Guido van Rossum wrote: > >> I'd say if you're not going to forward-port this to Python 3, it >> shouldn't go into Python 2 -- in that case it would make more sense to >> me to back-port the exception-raising behavior. > > Python 3 doesn't need it because it is possible to not give a result > at all. ?Python 2 does need it because we have to give *some* > result. > >> >> Also supporting comparisons but not other mixed operations is going to >> be confusing. If you are sticking to that behavior I think mixed >> comparisons should also be ruled out. > > The difference is that mixed comparisons currently do give a result, > but one that is non-sensical. ?The proposal is a make in give a > meaningful result, not as an extra feature, but in an effort to not > be wrong. > > Since 2.x has to give a result, we should make it useful. > Since 3.x does not make the comparison, it is okay to punt. No. You are talking like there is no path from 2.7 to 3.x. Since the result in 2.x was never useful, you should not add a dead-end feature. And there is no reason (other than backwards compatibility) why the comparison couldn't raise an exception in 2.x as well -- e.g. str<->unicode comparisons do this already. (It used to be required that comparisons had to always return a result, but that was done away with long before Decimal was introduced.) -- --Guido van Rossum (python.org/~guido) From raymond.hettinger at gmail.com Wed Mar 17 02:27:01 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Tue, 16 Mar 2010 18:27:01 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <4BA0032B.4080307@canterbury.ac.nz> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <20100316155815.DAE983A4097@sparrow.telecommunity.com> <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com> <4BA0032B.4080307@canterbury.ac.nz> Message-ID: <32790831-2CF6-425D-8DA1-4D45C1A1F083@gmail.com> On Mar 16, 2010, at 3:16 PM, Greg Ewing wrote: > > Seems to me that this education would mostly consist of saying > "don't compare floats and decimals", which is why I think that > disallowing them in the first place would be better. That makes sense. I do worry that 2.x currently does make the comparison and gives the wrong answer. We have the ability to make it a correct answer. But, it seems like the mood here is that wrong-is-better-than-right for an action that someone shouldn't be doing in the first place. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Mar 17 11:45:51 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 17 Mar 2010 21:45:51 +1100 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <32790831-2CF6-425D-8DA1-4D45C1A1F083@gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <4BA0032B.4080307@canterbury.ac.nz> <32790831-2CF6-425D-8DA1-4D45C1A1F083@gmail.com> Message-ID: <201003172145.52425.steve@pearwood.info> On Wed, 17 Mar 2010 12:27:01 pm Raymond Hettinger wrote: > On Mar 16, 2010, at 3:16 PM, Greg Ewing wrote: > > Seems to me that this education would mostly consist of saying > > "don't compare floats and decimals", which is why I think that > > disallowing them in the first place would be better. > > That makes sense. > > I do worry that 2.x currently does make the comparison > and gives the wrong answer. We have the ability to > make it a correct answer. But, it seems like the mood > here is that wrong-is-better-than-right for an action > that someone shouldn't be doing in the first place. I don't get this. Why is it "wrong" to compare Decimals to floats, and why shouldn't I do so? What harm is there? If the argument is that naive users who don't understand floats may be confused by the results, then the problem lies with floats, and if you really want to avoid confusing the float-naive, then we should prohibit all comparisons on floats: >>> 1e20 + 1e4 < 1e20 + 2e4 False I don't mean that as a serious suggestion -- it would be absurd to cripple floats for the sake of avoiding confusion of those who don't understand floats. Why are Decimals different? I can't see why comparing Decimal(1) to float(1) is wrong in any sense. I can see that comparing Decimal("1.1") to float("1.1") may confuse the float-naive, but the float naive will be confused by this too: >>> x = 1.0/3 >>> x + 1.0 - 1.0 == x False There's an awful lot about floats that is confusing to naive users, I don't see that the behaviour of Decimals will make it worse. -- Steven D'Aprano From kristjan at ccpgames.com Wed Mar 17 12:01:16 2010 From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=) Date: Wed, 17 Mar 2010 11:01:16 +0000 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <613d56581003161309p147c45f5ub0f142f1cd160776@mail.gmail.com> References: <5d2328511003161111vbf198d1y36e974d474e38a06@mail.gmail.com> <613d56581003161309p147c45f5ub0f142f1cd160776@mail.gmail.com> Message-ID: <930F189C8A437347B80DF2C156F7EC7F0A182184A4@exchis.ccp.ad.local> I could also point you at EVE online, a large scale IO machine based on stackless python. Our game proxies (front end servers) handle 30.000 client connections each using tasklets. The problem is coarse grained scheduling of IO and other tasks, same as with threads in regular Python. We have found that adhering to FIFO scheduling provides good all round performance, particularly the variance of IO latency is very low. As I?v mentioned in another message, I am toying with the idea of adding another priority layer to the Stackless scheduler for IO tasks. It?s not trivial though (requires me to do some stackless surgery) and the turnaround time to test new approaches in production for us is a bit high ? K From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Robert Hancock Sent: 16. mars 2010 20:10 To: Peter Portante Cc: David Beazley; python-dev at python.org Subject: Re: [Python-Dev] "Fixing" the new GIL The Linux kernel scheduler deals with two types of ratings and has a heuristic algorithm that rewards and penalizes on a rapid basis. To determine the next thread it inspects the priority array to find the highest priority task that is runnable and then selects the first task in that priority. This is a gross simplification of an extremely complex system, but it is not simply a simple queue. To duplicate this exact type of scheduling in the interpreter would be a daunting task, but Dave's example illustrates that some system of reward and penalty can be beneficial. Peter has brought up the point, both here and at the Open Space at Pycon, that before attempting a downsized version of the a priority thread scheduler that we should look at other programs that have had to deal with similar problems. Peter, since you have dealt with the nitty-gritty of concurrency before, can you suggest some applications that could serve as models for research? -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristjan at ccpgames.com Wed Mar 17 12:08:39 2010 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 17 Mar 2010 11:08:39 +0000 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <4BA0014E.9010207@gmail.com> References: <49653.1268622425@dabeaz.com> <930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local> <20100316080011.34589b00@msiwind> <930F189C8A437347B80DF2C156F7EC7F0A1821827D@exchis.ccp.ad.local> <20100316151623.05e8e570@msiwind> <930F189C8A437347B80DF2C156F7EC7F0A18218371@exchis.ccp.ad.local> <4BA0014E.9010207@gmail.com> Message-ID: <930F189C8A437347B80DF2C156F7EC7F0A182184B0@exchis.ccp.ad.local> > -----Original Message----- > From: Nick Coghlan [mailto:ncoghlan at gmail.com] > Sent: 16. mars 2010 22:08 > To: Kristj?n Valur J?nsson > Cc: Antoine Pitrou; python-dev at python.org > Subject: Re: [Python-Dev] "Fixing" the new GIL > > Kristj?n Valur J?nsson wrote: > > We have never had high priority for IO threads in > > python (and its not-by-design round robin scheduler on single core > > cpus) and it is unclear why that should be required now as a fix. > > David explained that in the issue tracker - 2.x typically doesn't do > that much work per bytecode instruction, so the interpreter releases > and > reacquires the GIL far more frequently in a CPU-bound thread than it > does under the new time-based check interval. > > Okay, but my original post in this thread was "why don't we fix the original problem with the GIL as implemented in 2.x" rather than to try to beat the new, more complicated, GIL into submission? In fact, I probably should submit a patch to do that since 2.x isn't disappearing quite yet. Cheers, K From steve at pearwood.info Wed Mar 17 12:43:35 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 17 Mar 2010 22:43:35 +1100 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003170932.09201.steve@pearwood.info> Message-ID: <201003172243.35336.steve@pearwood.info> On Wed, 17 Mar 2010 10:01:12 am Guido van Rossum wrote: > On Tue, Mar 16, 2010 at 2:32 PM, Steven D'Aprano wrote: > > But mixed arithmetic runs into the problem, what do you want the > > result type to be? Given (say) decimal+float, returning either a > > Decimal or a float will be the wrong thing to do some of the time, > > so better to prohibit mixed arithmetic and let folks handle their > > own conversions. So +1 on continuing to prohibit mixed arithmetic. > > I'm not disagreeing, but I really wonder, what is the value of > supporting mixed comparisons then? Just because you *can* assign a > meaning to it doesn't mean you should. Any mixed comparison with floats is already risky: >>> 1e20 + 1e4 == 10**20 + 10**4 False but we allow them anyway because they're too useful and too obvious not to. The problems with (say) floating point equality are far too well known to need me give examples, but I will anyway *wink*: >>> 1.3 == 3.7 - 2.4 False but we allow it anyway, partly from tradition ("floats have always been like that") and partly because, frankly, laziness is a virtue. It would be painful not to be able to say (e.g.): >>> 1.25 == 3.75 - 2.5 True >>> 10.0 > 1 True My defence of mixed comparisons is the same. If I'm doing serious maths work, then I'm going to be comparing numbers carefully, and not just tossing a comparison operator between them. But for simple calculations, or using the interactive interpreter as a calculator, or if I need to sort a list of mixed numbers without caring whether they are ints or floats or Decimals, laziness is a virtue. If your needs for accuracy aren't high, you can go far by comparing floats with ==. Why shouldn't the same apply to mixed floats and Decimals? Having said all that, I've just re-read the PEP, and spotted a fly in the ointment... hash. If we allow Decimals to compare equal to floats, that presumably implies that they need to hash equal. That may be simple enough for integer valued floats, but what non-integer values? -- Steven D'Aprano From dave at dabeaz.com Wed Mar 17 12:53:24 2010 From: dave at dabeaz.com (David Beazley) Date: Wed, 17 Mar 2010 06:53:24 -0500 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F0A182184A4@exchis.ccp.ad.local> References: <5d2328511003161111vbf198d1y36e974d474e38a06@mail.gmail.com> <613d56581003161309p147c45f5ub0f142f1cd160776@mail.gmail.com> <930F189C8A437347B80DF2C156F7EC7F0A182184A4@exchis.ccp.ad.local> Message-ID: <66A064A8-E328-49C7-9E5E-F5E8DC6D3D64@dabeaz.com> I'm not sure I see any inherent problem in putting all I/O bound threads in some kind of FIFO queue. However, there still should be some mechanism that gives those threads priority over a CPU-bound thread that never wants to give up the GIL. A common OS solution is to simply have multiple FIFO queues, each with varying priorities. Threads with low priority don't get to run unless the higher priority queues are empty. One could probably try it with Python by simply having two queues (I/O and CPU) and separating threads into two categories depending on whether they are forcefully timed out or not. An OS would tend to have many more levels, but maybe just two queues would be enough to solve the thread scheduling problems being discussed here. Cheers, Dave On Mar 17, 2010, at 6:01 AM, Kristj?n Valur J?nsson wrote: > I could also point you at EVE online, a large scale IO machine based on stackless python. Our game proxies (front end servers) handle 30.000 client connections each using tasklets. The problem is coarse grained scheduling of IO and other tasks, same as with threads in regular Python. We have found that adhering to FIFO scheduling provides good all round performance, particularly the variance of IO latency is very low. > As I?v mentioned in another message, I am toying with the idea of adding another priority layer to the Stackless scheduler for IO tasks. It?s not trivial though (requires me to do some stackless surgery) and the turnaround time to test new approaches in production for us is a bit high J > > K > > From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Robert Hancock > Sent: 16. mars 2010 20:10 > To: Peter Portante > Cc: David Beazley; python-dev at python.org > Subject: Re: [Python-Dev] "Fixing" the new GIL > > The Linux kernel scheduler deals with two types of ratings and has a heuristic algorithm that rewards and penalizes on a rapid basis. To determine the next thread it inspects the priority array to find the highest priority task that is runnable and then selects the first task in that priority. This is a gross simplification of an extremely complex system, but it is not simply a simple queue. > > To duplicate this exact type of scheduling in the interpreter would be a daunting task, but Dave's example illustrates that some system of reward and penalty can be beneficial. > > Peter has brought up the point, both here and at the Open Space at Pycon, that before attempting a downsized version of the a priority thread scheduler that we should look at other programs that have had to deal with similar problems. > > Peter, since you have dealt with the nitty-gritty of concurrency before, can you suggest some applications that could serve as models for research? > > From dickinsm at gmail.com Wed Mar 17 14:39:14 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 17 Mar 2010 13:39:14 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <4BA0032B.4080307@canterbury.ac.nz> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <20100316155815.DAE983A4097@sparrow.telecommunity.com> <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com> <4BA0032B.4080307@canterbury.ac.nz> Message-ID: <5c6f2a5d1003170639r73ed8811ydad84fcd0faa3d3a@mail.gmail.com> On Tue, Mar 16, 2010 at 10:16 PM, Greg Ewing wrote: > Mark Dickinson wrote: >> >> On Tue, Mar 16, 2010 at 3:58 PM, P.J. Eby wrote: >> >>> If not, it might be confusing if a number that prints as '.1' compares >>> unequal to Decimal('.1'). >> >> Agreed, but this is just your everyday floating-point confusion, to be >> dealt with by social means (e.g., educating the programmer). > > Seems to me that this education would mostly consist of saying > "don't compare floats and decimals", which is why I think that > disallowing them in the first place would be better. I was thinking of something more along the lines of: "Sure, go ahead and compare floats and decimals, but be aware that float('1.1') is not exactly 1.1, so don't complain when 1.1 == Decimal('1.1') returns False." For me, this actually a plus of allowing these comparisons: it makes the education easier. "Look, the binary float stored for 1.1 is actually larger than 1.1, and here's the proof: >>> 1.1 > Decimal('1.1') -> True." -- Mark From dickinsm at gmail.com Wed Mar 17 14:40:41 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 17 Mar 2010 13:40:41 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <201003170932.09201.steve@pearwood.info> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com> <5c6f2a5d1003160923v1519a3advf95df39cb7f884af@mail.gmail.com> <201003170932.09201.steve@pearwood.info> Message-ID: <5c6f2a5d1003170640i957d1d6yfca64e26e3b4e7b4@mail.gmail.com> On Tue, Mar 16, 2010 at 10:32 PM, Steven D'Aprano wrote: > On Wed, 17 Mar 2010 03:23:30 am Mark Dickinson wrote: >> On Tue, Mar 16, 2010 at 4:11 PM, Mark Dickinson >> wrote: [...] >> >> >>>> Decimal.from_float(1.1) == 1.1 >> > >> > False >> >> Whoops. ?To clarify, this is the pre-patch behaviour; ?post-patch, >> this gives True. > > Whew! You had me worried there for a second. Just to clarify, you are > proposing: > > Decimal.from_float(1.1) == 1.1 > Decimal.('1.1') != float('1.1') Exactly, yes. -- Mark From dickinsm at gmail.com Wed Mar 17 15:19:51 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 17 Mar 2010 14:19:51 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <594D6395-4B01-4C9C-8417-36866992C338@gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <594D6395-4B01-4C9C-8417-36866992C338@gmail.com> Message-ID: <5c6f2a5d1003170719m4e5abe27t1d74371da53b4914@mail.gmail.com> On Wed, Mar 17, 2010 at 12:36 AM, Raymond Hettinger wrote: > > On Mar 16, 2010, at 9:41 AM, Guido van Rossum wrote: >> >> Also supporting comparisons but not other mixed operations is going to >> be confusing. If you are sticking to that behavior I think mixed >> comparisons should also be ruled out. > > The difference is that mixed comparisons currently do give a result, > but one that is non-sensical. ?The proposal is a make in give a > meaningful result, not as an extra feature, but in an effort to not > be wrong. Yes, exactly. A number of people (including me, early in the issue tracker discussion) have suggested that it would be confusing/inconsistent/strange to allow mixed-type comparisons but not mixed-type arithmetic. But it's not clear to me exactly why this would be a bad thing. It seems likely that for serious applications there's little benefit in allowing float<->Decimal comparisons if mixed-type arithmetic isn't allowed: if you're checking that b > a, it's probably because you're about to subtract b from a, or divide b by a, or do some other piece of arithmetic that depends on that condition being true. But for simply playing around with numbers on the command line, and for helping free people from their floating-point misconceptions, I think having these float<->Decimal comparisons available is potentially useful. I understand the argument about not adding this feature to 2.x if it's not going to go into 3.x; but I'm convinced enough that the 2.x behaviour should change that I'd prefer to add it to both 2.x and 3.x than to neither. The other option would be to leave 3.x alone and just add a py3k warning to 2.x, so that at least there's some indication of where strange results are coming from. -- Mark From dickinsm at gmail.com Wed Mar 17 15:41:49 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 17 Mar 2010 14:41:49 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <201003172243.35336.steve@pearwood.info> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003170932.09201.steve@pearwood.info> <201003172243.35336.steve@pearwood.info> Message-ID: <5c6f2a5d1003170741v20c64f29t87aa1bffa82af3df@mail.gmail.com> On Wed, Mar 17, 2010 at 11:43 AM, Steven D'Aprano wrote: > Having said all that, I've just re-read the PEP, and spotted a fly in > the ointment... hash. > > If we allow Decimals to compare equal to floats, that presumably implies > that they need to hash equal. That may be simple enough for integer > valued floats, but what non-integer values? That is indeed an issue. It can be taken care by checking whether the Decimal instance is exactly representable as a float, and returning the hash of the corresponding float if so. From the patch (http://bugs.python.org/file16544/issue2531.patch): + self_as_float = float(self) + if Decimal.from_float(self_as_float) == self: + return hash(self_as_float) Strictly speaking this only works if the Decimal -> float conversion (which relies on the platform strtod) is accurate to within <1ulp, but that's going to be true in 2.7/3.x on all platforms using David Gay's str<->float conversion code, and is *probably* true for the system strtod on any sane platform, so I don't think it's a real issue. This also slows down the (already slow) hash computation for Decimal a touch. Until people start complaining, or show evidence of applications that require making large dictionaries of Decimal instances, I'm not going to worry about that either. It would be easy to cache the hash value of a Decimal instance if it became necessary. A less messy (but more intrusive) alternative would be to come up with a single sane hash function defined for all rational numbers (plus infinities and nans), and use this function for int, long, float, Decimal and Fraction types. I have a candidate function in mind... Mark From dickinsm at gmail.com Wed Mar 17 16:04:10 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 17 Mar 2010 15:04:10 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003161007q2503f7acy3bc131fc14b9ae10@mail.gmail.com> Message-ID: <5c6f2a5d1003170804s35eed086j14c1e48d81a53193@mail.gmail.com> On Tue, Mar 16, 2010 at 5:15 PM, Guido van Rossum wrote: > Definitely some. Stricter comparison rules are a frequent cause of > problems when code is first ported to 3.x. While you'd think that code > comparing a float and a Decimal is *already* broken, there's a > surprising number of situations where that's not necessary the case, > e.g. when an arbitrary but stable ordering is needed. Hmm. Okay. It seems like backporting the exception isn't a real option, then. A nitpick: the current 2.x behaviour fails to give an arbitrary but stable ordering; it merely gives an arbitrary ordering: >>> sorted([Decimal(1), 2, 3.0]) [Decimal('1'), 2, 3.0] >>> sorted([2, Decimal(1), 3.0]) [3.0, Decimal('1'), 2] So long as your list contains only floats and Decimals you're fine, but for a list containing floats, integers and Decimals the ordering is no longer stable: the problem, of course, being that int <-> float and int <-> Decimal comparisons use a rule (compare by numeric value) that's not compatible with the way that floats and Decimals are compared. This seems like yet another possible cause of subtle bugs, and again would be fixed by the proposed change in behaviour. On the other hand, I've not seen any reports of anyone encountering this in real life. Mark Mark From guido at python.org Wed Mar 17 18:09:27 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 17 Mar 2010 10:09:27 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003170804s35eed086j14c1e48d81a53193@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003161007q2503f7acy3bc131fc14b9ae10@mail.gmail.com> <5c6f2a5d1003170804s35eed086j14c1e48d81a53193@mail.gmail.com> Message-ID: On Wed, Mar 17, 2010 at 8:04 AM, Mark Dickinson wrote: > On Tue, Mar 16, 2010 at 5:15 PM, Guido van Rossum wrote: >> Definitely some. Stricter comparison rules are a frequent cause of >> problems when code is first ported to 3.x. While you'd think that code >> comparing a float and a Decimal is *already* broken, there's a >> surprising number of situations where that's not necessary the case, >> e.g. when an arbitrary but stable ordering is needed. > > Hmm. ?Okay. ?It seems like backporting the exception isn't a real option, then. > > A nitpick: ?the current 2.x behaviour fails to give an arbitrary but > stable ordering; ?it merely gives an arbitrary ordering: > >>>> sorted([Decimal(1), 2, 3.0]) > [Decimal('1'), 2, 3.0] >>>> sorted([2, Decimal(1), 3.0]) > [3.0, Decimal('1'), 2] > > So long as your list contains only floats and Decimals you're fine, > but for a list containing floats, integers and Decimals the ordering > is no longer stable: ?the problem, of course, being that int <-> float > and int <-> Decimal comparisons use a rule (compare by numeric value) > that's not compatible with the way that floats and Decimals are > compared. ?This seems like yet another possible cause of subtle bugs, > and again would be fixed by the proposed change in behaviour. ?On the > other hand, I've not seen any reports of anyone encountering this in > real life. Ok, I'll try to stay out of the discussion of which solution is best of our users, and if the outcome is that mixed operations in general are bad but mixed comparisons are good, I'll trust you. However I want to reiterate that you really shouldn't improve the situation for 2.7 unless you also forward-port the solution to 3.x. -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Wed Mar 17 19:54:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 17 Mar 2010 18:54:33 +0000 (UTC) Subject: [Python-Dev] test_bigmem and test_bigaddrspace broken Message-ID: Hello, The title says it all. test_bigmem and test_bigaddrspace are partly broken under py3k, and no buildbot ever tests that they run successfully. As an example, here is a test_bigmem run under a 16GB machine. It gets "killed" at some point, probably by the Linux kernel's OOM killer, although I instruct the test suite to only use a fraction of the available RAM. There are also a couple of failures: $ ./python -m test.regrtest -v -M 8Gb test_bigmem == CPython 3.2a0 (py3k, Mar 17 2010, 18:38:10) [GCC 4.3.2] == Linux-a.b.c.d-grsec-xxxx-grs-ipv4-64-x86_64-with-debian-lenny-sid == /home/antoine/py3k/py3k/build/test_python_10071 test_bigmem test_capitalize (test.test_bigmem.StrTest) ... Skipping test_capitalize because of memory constraint ok test_center (test.test_bigmem.StrTest) ... ok test_compare (test.test_bigmem.StrTest) ... Skipping test_compare because of memory constraint ok test_concat (test.test_bigmem.StrTest) ... ok test_contains (test.test_bigmem.StrTest) ... Skipping test_contains because of memory constraint ok test_count (test.test_bigmem.StrTest) ... Skipping test_count because of memory constraint ok test_encode (test.test_bigmem.StrTest) ... ERROR test_encode_ascii (test.test_bigmem.StrTest) ... ok test_encode_raw_unicode_escape (test.test_bigmem.StrTest) ... ok test_encode_utf32 (test.test_bigmem.StrTest) ... ok test_encode_utf7 (test.test_bigmem.StrTest) ... ok test_endswith (test.test_bigmem.StrTest) ... Skipping test_endswith because of memory constraint ok test_expandtabs (test.test_bigmem.StrTest) ... Skipping test_expandtabs because of memory constraint ok test_find (test.test_bigmem.StrTest) ... Skipping test_find because of memory constraint ok test_format (test.test_bigmem.StrTest) ... Skipping test_format because of memory constraint ok test_hash (test.test_bigmem.StrTest) ... ok test_index (test.test_bigmem.StrTest) ... Skipping test_index because of memory constraint ok test_isalnum (test.test_bigmem.StrTest) ... Skipping test_isalnum because of memory constraint ok test_isalpha (test.test_bigmem.StrTest) ... Skipping test_isalpha because of memory constraint ok test_isdigit (test.test_bigmem.StrTest) ... Skipping test_isdigit because of memory constraint ok test_islower (test.test_bigmem.StrTest) ... Skipping test_islower because of memory constraint ok test_isspace (test.test_bigmem.StrTest) ... Skipping test_isspace because of memory constraint ok test_istitle (test.test_bigmem.StrTest) ... Skipping test_istitle because of memory constraint ok test_isupper (test.test_bigmem.StrTest) ... Skipping test_isupper because of memory constraint ok test_join (test.test_bigmem.StrTest) ... Skipping test_join because of memory constraint ok test_ljust (test.test_bigmem.StrTest) ... ok test_lower (test.test_bigmem.StrTest) ... Skipping test_lower because of memory constraint ok test_lstrip (test.test_bigmem.StrTest) ... ok test_repeat (test.test_bigmem.StrTest) ... ok test_replace (test.test_bigmem.StrTest) ... Skipping test_replace because of memory constraint ok test_repr_large (test.test_bigmem.StrTest) ... Skipping test_repr_large because of memory constraint ok test_repr_small (test.test_bigmem.StrTest) ... Skipping test_repr_small because of memory constraint ok test_rfind (test.test_bigmem.StrTest) ... Skipping test_rfind because of memory constraint ok test_rindex (test.test_bigmem.StrTest) ... Skipping test_rindex because of memory constraint ok test_rjust (test.test_bigmem.StrTest) ... ok test_rstrip (test.test_bigmem.StrTest) ... ok test_slice_and_getitem (test.test_bigmem.StrTest) ... Skipping test_slice_and_getitem because of memory constraint ok test_split_large (test.test_bigmem.StrTest) ... Skipping test_split_large because of memory constraint ok test_split_small (test.test_bigmem.StrTest) ... Skipping test_split_small because of memory constraint ok test_splitlines (test.test_bigmem.StrTest) ... Skipping test_splitlines because of memory constraint ok test_startswith (test.test_bigmem.StrTest) ... Skipping test_startswith because of memory constraint ok test_strip (test.test_bigmem.StrTest) ... ok test_swapcase (test.test_bigmem.StrTest) ... Skipping test_swapcase because of memory constraint ok test_title (test.test_bigmem.StrTest) ... Skipping test_title because of memory constraint ok test_translate (test.test_bigmem.StrTest) ... Skipping test_translate because of memory constraint ok test_unicode_repr (test.test_bigmem.StrTest) ... Skipping test_unicode_repr because of memory constraint ok test_unicode_repr_overflow (test.test_bigmem.StrTest) ... Skipping test_unicode_repr_overflow because of memory constraint ok test_unicode_repr_wide (test.test_bigmem.StrTest) ... Skipping test_unicode_repr_wide because of memory constraint ok test_upper (test.test_bigmem.StrTest) ... Skipping test_upper because of memory constraint ok test_zfill (test.test_bigmem.StrTest) ... ok test_capitalize (test.test_bigmem.BytesTest) ... ok test_center (test.test_bigmem.BytesTest) ... ok test_compare (test.test_bigmem.BytesTest) ... ok test_concat (test.test_bigmem.BytesTest) ... ok test_contains (test.test_bigmem.BytesTest) ... FAIL test_count (test.test_bigmem.BytesTest) ... ok test_decode (test.test_bigmem.BytesTest) ... ok test_endswith (test.test_bigmem.BytesTest) ... ok test_expandtabs (test.test_bigmem.BytesTest) ... ok test_find (test.test_bigmem.BytesTest) ... ok test_hash (test.test_bigmem.BytesTest) ... ok test_index (test.test_bigmem.BytesTest) ... ok test_isalnum (test.test_bigmem.BytesTest) ... ok test_isalpha (test.test_bigmem.BytesTest) ... ok test_isdigit (test.test_bigmem.BytesTest) ... ok test_islower (test.test_bigmem.BytesTest) ... ok test_isspace (test.test_bigmem.BytesTest) ... ok test_istitle (test.test_bigmem.BytesTest) ... ok test_isupper (test.test_bigmem.BytesTest) ... ok test_join (test.test_bigmem.BytesTest) ... ok test_ljust (test.test_bigmem.BytesTest) ... ok test_lower (test.test_bigmem.BytesTest) ... ok test_lstrip (test.test_bigmem.BytesTest) ... ok test_repeat (test.test_bigmem.BytesTest) ... ok test_replace (test.test_bigmem.BytesTest) ... ok test_rfind (test.test_bigmem.BytesTest) ... ok test_rindex (test.test_bigmem.BytesTest) ... ok test_rjust (test.test_bigmem.BytesTest) ... ok test_rstrip (test.test_bigmem.BytesTest) ... ok test_slice_and_getitem (test.test_bigmem.BytesTest) ... ok test_split_large (test.test_bigmem.BytesTest) ... Skipping test_split_large because of memory constraint ok test_split_small (test.test_bigmem.BytesTest) ... ok test_splitlines (test.test_bigmem.BytesTest) ... ok test_startswith (test.test_bigmem.BytesTest) ... ok test_strip (test.test_bigmem.BytesTest) ... ok test_swapcase (test.test_bigmem.BytesTest) ... ok test_title (test.test_bigmem.BytesTest) ... ok test_translate (test.test_bigmem.BytesTest) ... ok test_upper (test.test_bigmem.BytesTest) ... ok test_zfill (test.test_bigmem.BytesTest) ... ok test_capitalize (test.test_bigmem.BytearrayTest) ... ok test_center (test.test_bigmem.BytearrayTest) ... ok test_compare (test.test_bigmem.BytearrayTest) ... ok test_concat (test.test_bigmem.BytearrayTest) ... ok test_contains (test.test_bigmem.BytearrayTest) ... FAIL test_count (test.test_bigmem.BytearrayTest) ... ok test_decode (test.test_bigmem.BytearrayTest) ... ok test_endswith (test.test_bigmem.BytearrayTest) ... ok test_expandtabs (test.test_bigmem.BytearrayTest) ... ok test_find (test.test_bigmem.BytearrayTest) ... ok test_index (test.test_bigmem.BytearrayTest) ... ok test_isalnum (test.test_bigmem.BytearrayTest) ... ok test_isalpha (test.test_bigmem.BytearrayTest) ... ok test_isdigit (test.test_bigmem.BytearrayTest) ... ok test_islower (test.test_bigmem.BytearrayTest) ... ok test_isspace (test.test_bigmem.BytearrayTest) ... ok test_istitle (test.test_bigmem.BytearrayTest) ... ok test_isupper (test.test_bigmem.BytearrayTest) ... ok test_join (test.test_bigmem.BytearrayTest) ... ok test_ljust (test.test_bigmem.BytearrayTest) ... ok test_lower (test.test_bigmem.BytearrayTest) ... ok test_lstrip (test.test_bigmem.BytearrayTest) ... ok test_repeat (test.test_bigmem.BytearrayTest) ... ok test_replace (test.test_bigmem.BytearrayTest) ... ok test_rfind (test.test_bigmem.BytearrayTest) ... ok test_rindex (test.test_bigmem.BytearrayTest) ... ok test_rjust (test.test_bigmem.BytearrayTest) ... ok test_rstrip (test.test_bigmem.BytearrayTest) ... ok test_slice_and_getitem (test.test_bigmem.BytearrayTest) ... ok test_split_small (test.test_bigmem.BytearrayTest) ... ok test_splitlines (test.test_bigmem.BytearrayTest) ... ok test_startswith (test.test_bigmem.BytearrayTest) ... ok test_strip (test.test_bigmem.BytearrayTest) ... ok test_swapcase (test.test_bigmem.BytearrayTest) ... ok test_title (test.test_bigmem.BytearrayTest) ... ok test_translate (test.test_bigmem.BytearrayTest) ... ok test_upper (test.test_bigmem.BytearrayTest) ... ok test_zfill (test.test_bigmem.BytearrayTest) ... ok test_compare (test.test_bigmem.TupleTest) ... Skipping test_compare because of memory constraint ok test_concat_large (test.test_bigmem.TupleTest) ... Skipping test_concat_large because of memory constraint ok test_concat_small (test.test_bigmem.TupleTest) ... Skipping test_concat_small because of memory constraint ok test_contains (test.test_bigmem.TupleTest) ... Skipping test_contains because of memory constraint ok test_from_2G_generator (test.test_bigmem.TupleTest) ... Skipping test_from_2G_generator because of memory constraint ok test_from_almost_2G_generator (test.test_bigmem.TupleTest) ... Skipping test_from_almost_2G_generator because of memory constraint ok test_hash (test.test_bigmem.TupleTest) ... Skipping test_hash because of memory constraint ok test_index_and_slice (test.test_bigmem.TupleTest) ... Skipping test_index_and_slice because of memory constraint ok test_repeat_large (test.test_bigmem.TupleTest) ... Skipping test_repeat_large because of memory constraint ok test_repeat_large_2 (test.test_bigmem.TupleTest) ... Skipping test_repeat_large_2 because of memory constraint ok test_repeat_small (test.test_bigmem.TupleTest) ... Skipping test_repeat_small because of memory constraint ok test_repr_large (test.test_bigmem.TupleTest) ... Skipping test_repr_large because of memory constraint ok test_repr_small (test.test_bigmem.TupleTest) ... ERROR test_append (test.test_bigmem.ListTest) ... Skipping test_append because of memory constraint ok test_compare (test.test_bigmem.ListTest) ... Skipping test_compare because of memory constraint ok test_concat_large (test.test_bigmem.ListTest) ... Skipping test_concat_large because of memory constraint ok test_concat_small (test.test_bigmem.ListTest) ... Skipping test_concat_small because of memory constraint ok test_contains (test.test_bigmem.ListTest) ... Skipping test_contains because of memory constraint ok test_count (test.test_bigmem.ListTest) ... Skipping test_count because of memory constraint ok test_extend_large (test.test_bigmem.ListTest) ... Skipping test_extend_large because of memory constraint ok test_extend_small (test.test_bigmem.ListTest) ... Skipping test_extend_small because of memory constraint ok test_hash (test.test_bigmem.ListTest) ... Skipping test_hash because of memory constraint ok test_index (test.test_bigmem.ListTest) ... Skipping test_index because of memory constraint ok test_index_and_slice (test.test_bigmem.ListTest) ... Skipping test_index_and_slice because of memory constraint ok test_inplace_concat_large (test.test_bigmem.ListTest) ... Skipping test_inplace_concat_large because of memory constraint ok test_inplace_concat_small (test.test_bigmem.ListTest) ... Skipping test_inplace_concat_small because of memory constraint ok test_inplace_repeat_large (test.test_bigmem.ListTest) ... Skipping test_inplace_repeat_large because of memory constraint ok test_inplace_repeat_small (test.test_bigmem.ListTest) ... Skipping test_inplace_repeat_small because of memory constraint ok test_insert (test.test_bigmem.ListTest) ... Skipping test_insert because of memory constraint ok test_pop (test.test_bigmem.ListTest) ... Skipping test_pop because of memory constraint ok test_remove (test.test_bigmem.ListTest) ... Skipping test_remove because of memory constraint ok test_repeat_large (test.test_bigmem.ListTest) ... Skipping test_repeat_large because of memory constraint ok test_repeat_small (test.test_bigmem.ListTest) ... Skipping test_repeat_small because of memory constraint ok test_repr_large (test.test_bigmem.ListTest) ... Skipping test_repr_large because of memory constraint ok test_repr_small (test.test_bigmem.ListTest) ... Killed From solipsis at pitrou.net Wed Mar 17 20:08:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 17 Mar 2010 20:08:33 +0100 Subject: [Python-Dev] test_bigmem and test_bigaddrspace broken In-Reply-To: <20100317185831.GA7296@idyll.org> References: <20100317185831.GA7296@idyll.org> Message-ID: <1268852913.3507.15.camel@localhost> Hello, Le mercredi 17 mars 2010 ? 11:58 -0700, C. Titus Brown a ?crit : > On Wed, Mar 17, 2010 at 06:54:33PM +0000, Antoine Pitrou wrote: > > The title says it all. test_bigmem and test_bigaddrspace are partly > > broken under py3k, and no buildbot ever tests that they run successfully. > > As an example, here is a test_bigmem run under a 16GB machine. It gets > > "killed" at some point, probably by the Linux kernel's OOM killer, > > although I instruct the test suite to only use a fraction of the > > available RAM. There are also a couple of failures: > > I can provide login access to a 24gb machine (Linux), if you or others are > interested. Sometimes it's running other stuff, but it's got lots of swap :) Thanks for the proposal. That wasn't really the point of my message, though :) As mentioned I have access to a 16GB machine which is able to run the tests. The problem is that nobody seems interested in maintaining these tests. As far as I can say, the initial committer is Thomas Wouters, and not much happened since then. It seems to me that big companies such as Google and Apple would be the primarily concerned ones by such scenarios, and willing to ensure they run fine. (by the way, even without swapping and in non-debug mode, the tests are quite long to run) Regards Antoine. From tjreedy at udel.edu Wed Mar 17 20:34:19 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 17 Mar 2010 15:34:19 -0400 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003161007q2503f7acy3bc131fc14b9ae10@mail.gmail.com> <5c6f2a5d1003170804s35eed086j14c1e48d81a53193@mail.gmail.com> Message-ID: On 3/17/2010 1:09 PM, Guido van Rossum wrote: > Ok, I'll try to stay out of the discussion of which solution is best > of our users, and if the outcome is that mixed operations in general > are bad but mixed comparisons are good, I'll trust you. However I want > to reiterate that you really shouldn't improve the situation for 2.7 > unless you also forward-port the solution to 3.x. I agree. Improving 2.7 and not 3.2+ would give people a reason to not move to 3.x. tjr From tlesher at gmail.com Wed Mar 17 20:53:45 2010 From: tlesher at gmail.com (Tim Lesher) Date: Wed, 17 Mar 2010 15:53:45 -0400 Subject: [Python-Dev] Removing Windows 95-related code in posixmodule.c Message-ID: <9613db601003171253y389a17aam68823b225c97a34@mail.gmail.com> Now that Windows 9x is no longer supported (per PEP 11), would it be appropriate to replace the ANSI ("A") versions of Windows OS calls in posixmodule.c with their Unicode ("W") equivalents and remove the unicode_file_names() function that determines which version to call? There are a number of places in that module where we try to decide whether to call ANSI or UNICODE versions (because only the ANSI versions were fully supported on Windows 9x). This is still fine on Win32, where the ANSI versions are still supported for backwards compatibility, but it causes issues with Windows CE, which doesn't have the ANSI versions at all. I've been maintaining some fairly ugly patches to cope with this in my company's Python 2.5.x for CE build, and I'd love to drop them as we move to a post-2.6 version. Thanks! -- Tim Lesher -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Wed Mar 17 21:44:21 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 17 Mar 2010 13:44:21 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003161007q2503f7acy3bc131fc14b9ae10@mail.gmail.com> <5c6f2a5d1003170804s35eed086j14c1e48d81a53193@mail.gmail.com> Message-ID: <826E9179-6933-4316-91AB-044F5453A97A@gmail.com> On Mar 17, 2010, at 12:34 PM, Terry Reedy wrote: > > I agree. Improving 2.7 and not 3.2+ would give people a reason to not move to 3.x. > FWIW, I think this is mischaracterizing the proposal. The spectrum of options from worst to best is 1) compare but give the wrong answer 2) compare but give the right answer 3) refuse to compare. Py3.x is already in the best position, it refuses to compare. IOW, is already is as improved as it can get. P2.6 is in the worst position. The proposal is to make it better, but not as good as 3.x. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Mar 17 21:59:24 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 18 Mar 2010 07:59:24 +1100 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <826E9179-6933-4316-91AB-044F5453A97A@gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <826E9179-6933-4316-91AB-044F5453A97A@gmail.com> Message-ID: <201003180759.25229.steve@pearwood.info> On Thu, 18 Mar 2010 07:44:21 am Raymond Hettinger wrote: > The spectrum of options from worst to best is > 1) compare but give the wrong answer > 2) compare but give the right answer > 3) refuse to compare. Why is 3 the best? If there is a right answer to give, surely giving the right answer it is better than not? -- Steven D'Aprano From martin at v.loewis.de Wed Mar 17 22:16:45 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 17 Mar 2010 22:16:45 +0100 Subject: [Python-Dev] test_bigmem and test_bigaddrspace broken In-Reply-To: <1268852913.3507.15.camel@localhost> References: <20100317185831.GA7296@idyll.org> <1268852913.3507.15.camel@localhost> Message-ID: <4BA146BD.2010909@v.loewis.de> > As mentioned I have access to a 16GB machine which is able to run the > tests. The problem is that nobody seems interested in maintaining these > tests. So what do you propose to do? I personally don't see that as a problem. Many rarely-used features get infrequent testing - that's the nature of rarely-used features. In turn, they tend to break over time, until somebody comes along and fixes them. That's open source: scratch your own itches. > As far as I can say, the initial committer is Thomas Wouters, and not > much happened since then. It seems to me that big companies such as > Google and Apple would be the primarily concerned ones by such > scenarios, and willing to ensure they run fine. I'm not sure Apple has really Python applications running that use a lot of memory - being a big company doesn't necessarily mean you operate large machines. I'm sure Google has applications that use that much memory, and I wouldn't be surprised if Google has also bug fixes for various bugs in large objects that they haven't been contributing (yet). In any case, I think it's a good thing that these tests are there for people to run if they have the hardware and the time. If somebody wanted to contribute such hardware as a build slave - that could certainly be arranged. Regards, Martin From martin at v.loewis.de Wed Mar 17 22:22:43 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 17 Mar 2010 22:22:43 +0100 Subject: [Python-Dev] Removing Windows 95-related code in posixmodule.c In-Reply-To: <9613db601003171253y389a17aam68823b225c97a34@mail.gmail.com> References: <9613db601003171253y389a17aam68823b225c97a34@mail.gmail.com> Message-ID: <4BA14823.10603@v.loewis.de> Tim Lesher wrote: > Now that Windows 9x is no longer supported (per PEP 11), would it be > appropriate to replace the ANSI ("A") versions of Windows OS calls in > posixmodule.c with their Unicode ("W") equivalents and remove the > unicode_file_names() function that determines which version to call? Not directly. The A versions are still used if you pass byte strings as file names. It would be possible to convert byte strings to Unicode strings (using CP_ACP), but I would only want to consider that if the code complexity does not increase under that approach (which I doubt). FWIW, the unicode_file_names function *is* removed in Python 2.7. Regards, Martin From solipsis at pitrou.net Wed Mar 17 22:26:06 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 17 Mar 2010 21:26:06 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?test=5Fbigmem_and_test=5Fbigaddrspace_brok?= =?utf-8?q?en?= References: <20100317185831.GA7296@idyll.org> <1268852913.3507.15.camel@localhost> <4BA146BD.2010909@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > > As mentioned I have access to a 16GB machine which is able to run the > > tests. The problem is that nobody seems interested in maintaining these > > tests. > > So what do you propose to do? Nothing. I was pointing out the issue here because it is not obvious that interested people would follow tracker updates. Of course, there is also the problem of having tests which never get run and whose failures don't get noticed (test_zipfile64 is another example, although the last time I tried it ran ok). And the additional problem of buggy tests (test_bigmem shouldn't OOM if I ask it to use 8GB on a machine with around 14GB of free mem). Regards Antoine. From ncoghlan at gmail.com Wed Mar 17 22:33:19 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 18 Mar 2010 07:33:19 +1000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <201003180759.25229.steve@pearwood.info> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <826E9179-6933-4316-91AB-044F5453A97A@gmail.com> <201003180759.25229.steve@pearwood.info> Message-ID: <4BA14A9F.10902@gmail.com> Steven D'Aprano wrote: > On Thu, 18 Mar 2010 07:44:21 am Raymond Hettinger wrote: >> The spectrum of options from worst to best is >> 1) compare but give the wrong answer >> 2) compare but give the right answer >> 3) refuse to compare. > > Why is 3 the best? If there is a right answer to give, surely giving the > right answer it is better than not? I agree with Steven here - for mixed *arithmetic* refusing to get involved is a reasonable choice, because the whole point of Decimals is to get the answers according to base 10 expectations. Allowing implicit conversion of base 2 floats puts that foundation at risk. In what way do comparisons carry the same risk? Decimal.from_float shows that a perfectly adequate mapping from float into the Decimal space can be made, and the comparisons have a clear well-defined answer. It may be slightly confusing to those not familiar with the subtleties of limited precision base 2 vs larger precision base 10, but the boolean result places them in a clearly different category to my mind. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From raymond.hettinger at gmail.com Wed Mar 17 22:58:25 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 17 Mar 2010 14:58:25 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <201003180759.25229.steve@pearwood.info> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <826E9179-6933-4316-91AB-044F5453A97A@gmail.com> <201003180759.25229.steve@pearwood.info> Message-ID: <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> On Mar 17, 2010, at 1:59 PM, Steven D'Aprano wrote: > On Thu, 18 Mar 2010 07:44:21 am Raymond Hettinger wrote: >> The spectrum of options from worst to best is >> 1) compare but give the wrong answer >> 2) compare but give the right answer >> 3) refuse to compare. > > Why is 3 the best? If there is a right answer to give, surely giving the > right answer it is better than not? From the early days of the decimal module, we've thought that mixed float-decimal operations are 1) a bit perilous and 2) have few, if any good use cases. Accordingly, any mixed operations should be explicit rather than implicit: Decimal('1.1') + Decimal.from_float(2.2) is better than: Decimal('1.1') + 2.2 To help the user avoid confusion, we flag the latter with a TypeError: unsupported operand type(s) for +: 'Decimal' and 'float'. Unfortunately, in Py2.x, implicit mixed comparisons do not raise an exception, and instead will silently fail by giving an incorrect answer: >>> Decimal('1.1') < 2.2 False IMO, raising the exception is the right thing to do. Short of that though, if we're going to give a result, it should at least be a correct one. Raymond New zen: * right answers are better wrong * but ill-formed questions are best not answered at all -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Mar 18 00:07:44 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 17 Mar 2010 16:07:44 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <826E9179-6933-4316-91AB-044F5453A97A@gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003161007q2503f7acy3bc131fc14b9ae10@mail.gmail.com> <5c6f2a5d1003170804s35eed086j14c1e48d81a53193@mail.gmail.com> <826E9179-6933-4316-91AB-044F5453A97A@gmail.com> Message-ID: On Wed, Mar 17, 2010 at 1:44 PM, Raymond Hettinger wrote: > > On Mar 17, 2010, at 12:34 PM, Terry Reedy wrote: > > I agree. Improving 2.7 and not 3.2+ would give people a reason to not move > to 3.x. > > > FWIW, I think this is mischaracterizing the proposal. > The spectrum of options from worst to best is > 1) compare but give the wrong answer > 2) compare but give the right answer > 3) refuse to compare. > Py3.x is already in the best position, it refuses to compare. > IOW, is already is as improved as it can get. > P2.6 is in the worst position. ?The proposal is to make > it better, but not as good as 3.x. Some people have argued that (2) is better than (3). At the very least, there is a marked discontinuity in this spectrum between (2) and (3). With (1), users are less likely to rely on this feature in 2.x and then have no recourse when converting to 3.x. -- --Guido van Rossum (python.org/~guido) From greg.ewing at canterbury.ac.nz Thu Mar 18 00:49:24 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 18 Mar 2010 12:49:24 +1300 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <201003170946.39457.steve@pearwood.info> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com> <4BA0032B.4080307@canterbury.ac.nz> <201003170946.39457.steve@pearwood.info> Message-ID: <4BA16A84.9010307@canterbury.ac.nz> Steven D'Aprano wrote: > More explicit than someDecimal == someFloat? Seems pretty explicit to > me. Yes. I mean at least specifying either float(someDecimal) == someFloat or someDecimal == Decimal(someFloat). Preferably also whether the conversion is to be as exact as possible or on a minimum-digits basis. -- Greg From greg.ewing at canterbury.ac.nz Thu Mar 18 00:58:40 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 18 Mar 2010 12:58:40 +1300 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com> <5c6f2a5d1003160923v1519a3advf95df39cb7f884af@mail.gmail.com> <201003170932.09201.steve@pearwood.info> Message-ID: <4BA16CB0.4040908@canterbury.ac.nz> Guido van Rossum wrote: > in some > "intuitive complexity" sense an int is a simpler type than a float and > a float is a simpler type than a Decimal I don't think this analogy holds. In a mathematical sense, ints are a subset of reals, but binary and decimal floats are just alternative approximate representations of reals, neither one being inherently preferable over the other. One could argue that since all binary floats are exactly representable in decimal but not vice versa, decimal should be regarded as the wider type. But even this doesn't hold when you have a limited number of decimal digits available, which you always do at any given moment with the Decimal type. And even if there are enough digits, an exact conversion mightn't be what you really want. This problem doesn't arise with int->float conversion -- there is only one obvious way of chopping it to fit. -- Greg From greg.ewing at canterbury.ac.nz Thu Mar 18 01:02:31 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 18 Mar 2010 13:02:31 +1300 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <594D6395-4B01-4C9C-8417-36866992C338@gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <594D6395-4B01-4C9C-8417-36866992C338@gmail.com> Message-ID: <4BA16D97.2010502@canterbury.ac.nz> Raymond Hettinger wrote: > Python 3 doesn't need it because it is possible to not give a result > at all. Python 2 does need it because we have to give *some* > result. That's not true -- it's possible for comparisons to raise an exception in 2.x, and they sometimes do already: Python 2.5.4 (r254:67916, May 15 2009, 15:21:20) [GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> 1+2j < 3+4j Traceback (most recent call last): File "", line 1, in TypeError: no ordering relation is defined for complex numbers -- Greg From raymond.hettinger at gmail.com Thu Mar 18 02:03:36 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 17 Mar 2010 18:03:36 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <4BA16D97.2010502@canterbury.ac.nz> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <594D6395-4B01-4C9C-8417-36866992C338@gmail.com> <4BA16D97.2010502@canterbury.ac.nz> Message-ID: On Mar 17, 2010, at 5:02 PM, Greg Ewing wrote: > Raymond Hettinger wrote: > >> Python 3 doesn't need it because it is possible to not give a result >> at all. Python 2 does need it because we have to give *some* >> result. > > That's not true -- it's possible for comparisons to raise > an exception in 2.x, and they sometimes do already: Complex objects do not support __float__. Decimal objects do. If an object supports __float__, then a float comparison coerces its other argument via __float__ and the other argument never gets a chance to raise an exception. >>> class D: def __float__(self): return 3.14 >>> float(D()) 3.1400000000000001 >>> float(complex(3.14)) Traceback (most recent call last): File "", line 1, in float(complex(3.14)) TypeError: can't convert complex to float >>> D() < 10.0 True >>> complex(3.14) < 10.0 Traceback (most recent call last): File "", line 1, in complex(3.14) < 10.0 TypeError: no ordering relation is defined for complex numbers Raymond From jafo at tummy.com Thu Mar 18 12:58:44 2010 From: jafo at tummy.com (Sean Reifschneider) Date: Thu, 18 Mar 2010 05:58:44 -0600 Subject: [Python-Dev] Bug? syslog.openlog using ident "python" by default. Message-ID: <20100318115844.GA7374@tummy.com> I'd appreciate some opinions on this. Personally, I'm in the "the current code is buggy" camp. :-) I can code up the changes to the syslog module if we decide that's the right way to go. Looks like Raymond, Guido, and I are the last ones to do syslog-specific changes to this module in the last 15 years. If you call: from syslog import syslog, openlog syslog('My error message') Something like the following gets logged: Mar 18 05:20:22 guin python: My error message ^^^^^^ Where I'm annoyed by the "python" in the above. This is pulled from the C argv[0]. IMHO, what we really want is the Python sys.argv[0]. Because of this feature, I always have to do the following when I'm using syslog: openlog(os.path.basename(sys.argv[0])) I would propose changing the Python syslog() call to do the C equivalent of: if openlog_hasnt_been_called_before_now: if sys.argv[0]: syslog.openlog(os.path.basename(sys.argv[0])) In other words, if there's a script name and openlog hasn't already been called, call openlog with the basename of the script name. This is effectively what happens in normal C code that calls syslog. Note that the Python syslog.openlog says that the default for ident is "(usually) 'syslog'". The benefit of this change is that you get a more identifiable ident string in Python programs, so when you look at the logs you can tell what script it came from, not just that it came from some Python program. One way of looking at it would be that the syslog module logging "python" as the program name is a bug. The downside I see is that there might be some users doing log scraping depending on this (IMHO, buggy) log ident. Something else just occurred to me though, a nice enhancement would be for openlog() to have a "None" default for ident, something like: orig_openlog = openlog def openlog(ident = None, logopt = 0, facility = LOG_USER) if ident == None: if sys.argv[0]: ident = os.path.basename(sys.argv[0]) else: ident = 'python' orig_openlog(ident, logopt, facility) So you could call openlog and rely on the default ident (sys.argv[0]), and set the logopt and facility. The gory details of why this is occurring are: If you don't call "openlog()" before "syslog()", the system syslog() function calls something similar to: "openlog(basename(argv[0]))", which causes the "ident" of the syslog messages to be "python" rather than the specific program. Thoughts? Thanks, Sean -- "Every increased possession loads us with new weariness." -- John Ruskin Sean Reifschneider, Member of Technical Staff tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability From steve at pearwood.info Thu Mar 18 13:23:18 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 18 Mar 2010 23:23:18 +1100 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> Message-ID: <201003182323.19458.steve@pearwood.info> On Thu, 18 Mar 2010 08:58:25 am Raymond Hettinger wrote: > On Mar 17, 2010, at 1:59 PM, Steven D'Aprano wrote: > > On Thu, 18 Mar 2010 07:44:21 am Raymond Hettinger wrote: > >> The spectrum of options from worst to best is > >> 1) compare but give the wrong answer > >> 2) compare but give the right answer > >> 3) refuse to compare. > > > > Why is 3 the best? If there is a right answer to give, surely > > giving the right answer it is better than not? > > From the early days of the decimal module, > we've thought that mixed float-decimal operations > are 1) a bit perilous and 2) have few, if any good > use cases. When it comes to *arithmetic* operations, I agree. Is there anyone on python-dev willing to argue the case for allowing implicit mixed float/Decimal arithmetic operations? The arguments in the PEP seem pretty convincing to me, and I'm not suggesting we change that. But comparison operations are different. For starters, you don't need to worry about whether to return a float or a Decimal, because you always get a bool. In theory, both Decimals and floats are representations of the same underlying thing, namely real numbers, and it seems strange to me that I can't ask whether two such real numbers are equal just because their storage implementation is different. I can see three reasonable reasons for avoiding mixed comparisons: (1) To avoid confusing float-naive users (but they're confused by pure float comparisons too). (2) To avoid mixed arithmetic operations (but comparisons aren't arithmetic). (3) If Decimals and floats compare equal, they must hash equal, and currently they don't (but Mark Dickinson thinks he has a solution for that). > Accordingly, any mixed operations should be explicit > rather than implicit: > > Decimal('1.1') + Decimal.from_float(2.2) > > is better than: > > Decimal('1.1') + 2.2 Agreed. The user should explicitly choose whether they want a float answer or a Decimal answer. > To help the user avoid confusion, we flag the latter with a > TypeError: unsupported operand type(s) for +: 'Decimal' and 'float'. > > Unfortunately, in Py2.x, implicit mixed comparisons do not > raise an exception, and instead will silently fail by giving > > an incorrect answer: > >>> Decimal('1.1') < 2.2 > False That is clearly the wrong thing to do. Do you envisage any problems from allowing this instead? >>> Decimal('1.1') < 2.2 True > IMO, raising the exception is the right thing to do. > Short of that though, if we're going to give a result, > it should at least be a correct one. +1 -- Steven D'Aprano From raymond.hettinger at gmail.com Thu Mar 18 18:55:41 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Thu, 18 Mar 2010 10:55:41 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <201003182323.19458.steve@pearwood.info> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> Message-ID: <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> On Mar 18, 2010, at 5:23 AM, Steven D'Aprano wrote: > On Thu, 18 Mar 2010 08:58:25 am Raymond Hettinger wrote: >> On Mar 17, 2010, at 1:59 PM, Steven D'Aprano wrote: >>> On Thu, 18 Mar 2010 07:44:21 am Raymond Hettinger wrote: >>>> The spectrum of options from worst to best is >>>> 1) compare but give the wrong answer >>>> 2) compare but give the right answer >>>> 3) refuse to compare. >>> >>> Why is 3 the best? If there is a right answer to give, surely >>> giving the right answer it is better than not? >> >> From the early days of the decimal module, >> we've thought that mixed float-decimal operations >> are 1) a bit perilous and 2) have few, if any good >> use cases. > > When it comes to *arithmetic* operations, I agree. Is there anyone on > python-dev willing to argue the case for allowing implicit mixed > float/Decimal arithmetic operations? The arguments in the PEP seem > pretty convincing to me, and I'm not suggesting we change that. > > But comparison operations are different. For starters, you don't need to > worry about whether to return a float or a Decimal, because you always > get a bool. In theory, both Decimals and floats are representations of > the same underlying thing, namely real numbers, and it seems strange to > me that I can't ask whether two such real numbers are equal just > because their storage implementation is different. > > I can see three reasonable reasons for avoiding mixed comparisons: > > (1) To avoid confusing float-naive users (but they're confused by pure > float comparisons too). > > (2) To avoid mixed arithmetic operations (but comparisons aren't > arithmetic). > > (3) If Decimals and floats compare equal, they must hash equal, and > currently they don't (but Mark Dickinson thinks he has a solution for > that). Thanks for the thoughtful post. The question is which behavior is most useful, most of the time. If a person truly wants to do mixed comparisons, it's trivially easy to do so in a way that is explicit: somedecimal < Decimal.from_float(somefloat) My thought is that intentional mixed compares of float and decimal are very rare relative to unintentional cases. IOW, most of the time that x float comparisons in py3k. In-Reply-To: <201003182323.19458.steve@pearwood.info> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> Message-ID: <4BA2707A.80801@g.nevcal.com> On 3/18/2010 5:23 AM, Steven D'Aprano wrote: > On Thu, 18 Mar 2010 08:58:25 am Raymond Hettinger wrote: > >> On Mar 17, 2010, at 1:59 PM, Steven D'Aprano wrote: >> >>> On Thu, 18 Mar 2010 07:44:21 am Raymond Hettinger wrote: >>> >>>> The spectrum of options from worst to best is >>>> 1) compare but give the wrong answer >>>> 2) compare but give the right answer >>>> 3) refuse to compare. >>>> >>> Why is 3 the best? If there is a right answer to give, surely >>> giving the right answer it is better than not? >>> >> From the early days of the decimal module, >> we've thought that mixed float-decimal operations >> are 1) a bit perilous and 2) have few, if any good >> use cases. >> > When it comes to *arithmetic* operations, I agree. Is there anyone on > python-dev willing to argue the case for allowing implicit mixed > float/Decimal arithmetic operations? The arguments in the PEP seem > pretty convincing to me, and I'm not suggesting we change that. > > But comparison operations are different. For starters, you don't need to > worry about whether to return a float or a Decimal, because you always > get a bool. In theory, both Decimals and floats are representations of > the same underlying thing, namely real numbers, and it seems strange to > me that I can't ask whether two such real numbers are equal just > because their storage implementation is different. > > I can see three reasonable reasons for avoiding mixed comparisons: > > (1) To avoid confusing float-naive users (but they're confused by pure > float comparisons too). > > (2) To avoid mixed arithmetic operations (but comparisons aren't > arithmetic). > > (3) If Decimals and floats compare equal, they must hash equal, and > currently they don't (but Mark Dickinson thinks he has a solution for > that). > > > > >> Accordingly, any mixed operations should be explicit >> rather than implicit: >> >> Decimal('1.1') + Decimal.from_float(2.2) >> >> is better than: >> >> Decimal('1.1') + 2.2 >> > Agreed. The user should explicitly choose whether they want a float > answer or a Decimal answer. > > > >> To help the user avoid confusion, we flag the latter with a >> TypeError: unsupported operand type(s) for +: 'Decimal' and 'float'. >> >> Unfortunately, in Py2.x, implicit mixed comparisons do not >> raise an exception, and instead will silently fail by giving >> >> an incorrect answer: >> >>> Decimal('1.1')< 2.2 >> False >> > That is clearly the wrong thing to do. > > Do you envisage any problems from allowing this instead? > > >>>> Decimal('1.1')< 2.2 >>>> > True > Yes. As any non-na?ve float user is aware, the proper form of float comparisons is not to use < or > or == or !=, but rather, instead of using < (to follow along with your example), one should use: Decimal('1.1') - 2.2 < epsilon However, while even this is only useful in certain circumstances as a gross simplification [1], it immediately shows the need to do mixed arithmetic to produce (sometimes) correct results. More correct comparisons require much more code (even the 20-line C code in [1], which understands the float format to some extent, admits to being deficient in some circumstances). For all the reasons that mixed decimal and float arithmetic is bad, mixed decimal and float comparisons are also bad. To do proper comparisons, you need to know the number of significant digits of both numbers, and the precision and numeric ranges being dealt with by the application. For the single purpose of sorting, one could make an argument that not knowing the significant digits, precision, and numeric ranges, that the sort would probably produce results where floats and decimals that should compare equal would be clustered similarly as they would if the significant digits, precision, and numeric ranges were known, and that would probably be close to truth, but only if the decimal vs float key were the last in the composite sort key. I don't think Python informs its comparison operations that it is being used as part of a sort, nor would there be a way for user-written sorts to inform the comparison operations of that fact. Seems like it would be better to raise an exception, and in the documentation for the exception point out that turning off the exception (if it should be decided that that should be possible, which could be good for compatibility), would regress to the current behavior, which doesn't sort numerically, but by type. [1] http://www.cprogramming.com/tutorial/floating_point/understanding_floating_point_representation.html Glenn From dickinsm at gmail.com Thu Mar 18 19:41:08 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 18 Mar 2010 18:41:08 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> Message-ID: <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> On Thu, Mar 18, 2010 at 5:55 PM, Raymond Hettinger wrote: > My thought is that intentional mixed compares of float and decimal > are very rare relative to unintentional cases. ?IOW, most of the > time that x (or the user simply doesn't understand what his or her code is > actually doing). ?That user is best served by refusing the temptation > to guess that they really wanted to go down this path. In this case, could we consider again the idea of making these comparisons produce TypeErrors in 2.7? That's what's been requested or proposed multiple times on the tracker issue (by Imri Goldberg[1], Jeremy Dunck[2], Bert Hughes[3] and Stefan Krah[4]), and also by participants in this discussion (Greg Ewing). I'm only seeing two arguments against this at the moment: (1) it has the potential to break code that relies on being able to sort heterogeneous lists. But given that heterogeneous lists can't be sorted stably anyway (see my earlier post about lists containing ints, floats and Decimals), perhaps this is an acceptable risk. (2) A few of the posters here (Steven, Nick, and me) feel that it's slightly more natural to allow these comparisons; but I think the argument's fairly evenly balanced at the moment between those who'd prefer an exception and those who'd prefer to allow the comparisons. I'd really like to get this sorted for 2.7: as far as I'm concerned, either of the proposed behaviours (raise an exception, or allow comparisons) would be an improvement on the current 2.7 behaviour. Could everyone live with making float<->Decimal comparisons raise an exception in 2.7? Mark [1] http://bugs.python.org/issue2531#msg83691 [2] http://bugs.python.org/issue2531#msg83818 [3] http://bugs.python.org/issue2531#msg97891 [4] http://bugs.python.org/issue2531#msg98217 From solipsis at pitrou.net Thu Mar 18 19:44:52 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 18 Mar 2010 18:44:52 +0000 (UTC) Subject: [Python-Dev] Decimal <-> float comparisons in py3k. References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <4BA2707A.80801@g.nevcal.com> Message-ID: Glenn Linderman g.nevcal.com> writes: > > For all the reasons that mixed decimal and float arithmetic is bad, > mixed decimal and float comparisons are also bad. To do proper > comparisons, you need to know the number of significant digits of both > numbers, and the precision and numeric ranges being dealt with by the > application. What is a "proper comparison"? A comparison is a lossy operation by construction (it returns only 1 bit of output, or 2 bits for rich comparison), so the loss in precision that results from mixing float and decimal operands should not be a concern here. Unless there are situations where the comparison algorithm might return wrong results (are there?), I don't see why we should forbid such comparisons. > Seems like it would be better to raise an exception, and in the > documentation for the exception point out that turning off the exception > (if it should be decided that that should be possible, which could be > good for compatibility), would regress to the current behavior, which > doesn't sort numerically, but by type. Regressing to an useless behaviour sounds like a deliberate annoyance. I don't think this proposal should be implemented. Regards Antoine. From fuzzyman at voidspace.org.uk Thu Mar 18 19:48:48 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 18 Mar 2010 18:48:48 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <4BA2707A.80801@g.nevcal.com> Message-ID: <4BA27590.9020304@voidspace.org.uk> On 18/03/2010 18:44, Antoine Pitrou wrote: > [snip...] >> Seems like it would be better to raise an exception, and in the >> documentation for the exception point out that turning off the exception >> (if it should be decided that that should be possible, which could be >> good for compatibility), would regress to the current behavior, which >> doesn't sort numerically, but by type. >> > Regressing to an useless behaviour sounds like a deliberate annoyance. I don't > think this proposal should be implemented. > I agree, comparisons here have completely defined semantics - it sounds crazy not to allow it. (The argument 'because some programmers might do it without realising' doesn't hold much water with me.) Michael > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From steve at pearwood.info Thu Mar 18 20:20:41 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 19 Mar 2010 06:20:41 +1100 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> Message-ID: <201003190620.42295.steve@pearwood.info> On Fri, 19 Mar 2010 05:41:08 am Mark Dickinson wrote: > I'd really like to get this sorted for 2.7: as far as I'm concerned, > either of the proposed behaviours (raise an exception, or allow > comparisons) would be an improvement on the current 2.7 behaviour. > > Could everyone live with making float<->Decimal comparisons raise an > exception in 2.7? Yes. I would far prefer an exception than the current incorrect results in 2.6. -- Steven D'Aprano From steve at pearwood.info Thu Mar 18 20:34:19 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 19 Mar 2010 06:34:19 +1100 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <4BA2707A.80801@g.nevcal.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003182323.19458.steve@pearwood.info> <4BA2707A.80801@g.nevcal.com> Message-ID: <201003190634.19250.steve@pearwood.info> On Fri, 19 Mar 2010 05:27:06 am Glenn Linderman wrote: > > Do you envisage any problems from allowing this instead? > > > >>>> Decimal('1.1')< 2.2 > > > > True > > Yes. > > As any non-na?ve float user is aware, the proper form of float > comparisons is not to use < or > or == or !=, but rather, instead of > using < (to follow along with your example), one should use: > > Decimal('1.1') - 2.2 < epsilon And yet we allow 1.1 < 2.2 instead of forcing users to do the "proper form". One can only wonder why the various standards (actual and de-facto) for floating point allows comparisons at all. But they do, and so does Python, and frankly even if the only reason is to satisfy lazy coders who don't have a requirement for high accuracy, then that's a good reason in my book, and one equally applicable to Decimal and float. -- Steven D'Aprano From robert.kern at gmail.com Thu Mar 18 20:45:50 2010 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 18 Mar 2010 14:45:50 -0500 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <4BA2707A.80801@g.nevcal.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <4BA2707A.80801@g.nevcal.com> Message-ID: On 2010-03-18 13:27 PM, Glenn Linderman wrote: > As any non-na?ve float user is aware, the proper form of float > comparisons is not to use < or > or == or !=, but rather, instead of > using < (to follow along with your example), one should use: > > Decimal('1.1') - 2.2 < epsilon Not at all. This is quite incorrect for most use cases. Fuzzy comparisons are sometimes useful for equality testing, but almost never for ordering. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From v+python at g.nevcal.com Thu Mar 18 21:35:55 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 18 Mar 2010 13:35:55 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <4BA2707A.80801@g.nevcal.com> Message-ID: <4BA28EAB.3000707@g.nevcal.com> On 3/18/2010 12:45 PM, Robert Kern wrote: > On 2010-03-18 13:27 PM, Glenn Linderman wrote: > >> As any non-na?ve float user is aware, the proper form of float >> comparisons is not to use < or > or == or !=, but rather, instead of >> using < (to follow along with your example), one should use: >> >> Decimal('1.1') - 2.2 < epsilon > > Not at all. This is quite incorrect for most use cases. Fuzzy > comparisons are sometimes useful for equality testing, but almost > never for ordering. > I wondered if anyone would catch me on that! I did it mostly to keep the same example going, but partly to see if there were any real floating point experts in the discussion! You'll have noted my reference was to using the technique for equality. From omega_force2003 at yahoo.com Thu Mar 18 21:54:20 2010 From: omega_force2003 at yahoo.com (OMEGA RED) Date: Thu, 18 Mar 2010 13:54:20 -0700 (PDT) Subject: [Python-Dev] Is Scala better than Python ??? Message-ID: <405517.58108.qm@web34408.mail.mud.yahoo.com> To All, ? I have done some more review of python and Scala and I am very impressed with both languages.? However, it seems as though that Scala is far more able to support parallel processing than Python ??? Is this true ?? ? O.R. -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phd.pp.ru Thu Mar 18 22:04:40 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Fri, 19 Mar 2010 00:04:40 +0300 Subject: [Python-Dev] Is Scala better than Python ??? In-Reply-To: <405517.58108.qm@web34408.mail.mud.yahoo.com> References: <405517.58108.qm@web34408.mail.mud.yahoo.com> Message-ID: <20100318210440.GA4142@phd.pp.ru> Hello. We'are sorry but we cannot help you. This mailing list is for working on developing Python (fixing bugs and adding new features to Python itself); if you're having problems using Python, please find another forum. Probably python-list (comp.lang.python) news group/mailing list is the best place. See http://www.python.org/community/lists/ for other lists/news groups/forums. Thank you for understanding. On Thu, Mar 18, 2010 at 01:54:20PM -0700, OMEGA RED wrote: > To All, > ? > I have done some more review of python and Scala and I am very impressed with both languages.? However, it seems as though that Scala is far more able to support parallel processing than Python ??? Is this true ?? > ? > O.R. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From tjreedy at udel.edu Thu Mar 18 22:38:06 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 18 Mar 2010 17:38:06 -0400 Subject: [Python-Dev] Joel Spolsky on Mercurial Message-ID: In what might be his last Joel on Software post (for awhile, at least) http://www.joelonsoftware.com/items/2010/03/17.html Summary: Early 2009: Joel disses DVCSes. His programmers switch from subversion to hg. Joel grumbles. His programmers develop an hg-related product. Joel takes a better look. Joel sees the light: Subversion controls versions, hg controls changes; this make merging much easier. Joel flips (his opinion): Switch now! Joel write hg tutorial: http://hginit.com/ It starts with "Subversion Re-education" for existing version-oriented system users and continue with "Ground up Mercurial" where new vcs users can start. It has several simple examples. Having gotten that far, I think this might be worth referencing in new dev docs. Terry Jan Reedy From ncoghlan at gmail.com Thu Mar 18 22:48:33 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 19 Mar 2010 07:48:33 +1000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> Message-ID: <4BA29FB1.9020909@gmail.com> Mark Dickinson wrote: > Could everyone live with making float<->Decimal comparisons raise an > exception in 2.7? I could, with the caveat that *if* this causes problems for real world code, then changing it to produce the correct answer (as per your patch) should be applied as a bug fix in both 2.7 and 3.2. Note that even in Py3k there are some fairly serious weirdnesses kicking around due to the intransitive nature of numeric equality though: >>> from decimal import Decimal as dec >>> set((1, 1.0, dec("1.0"))) {1} >>> set((1.0, dec("1.0"))) {1.0, Decimal('1.0')} >>> d = {} >>> from decimal import Decimal as dec >>> d[1] = d[1.0] = d[dec("1.0")] = 42 >>> d {1: 42} >>> d[1.0] = d[dec("1.0")] = 42 >>> d {1: 42} >>> del d[1] >>> d[1.0] = d[dec("1.0")] = 42 >>> d {1.0: 42, Decimal('1.0'): 42} When there is a clear, correct way (based on Decimal.from_float) to make numeric comparison behave in accordance with the rules of mathematics, do we really want to preserve strange, unintuitive behaviour like the above? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From glenn at nevcal.com Thu Mar 18 21:31:33 2010 From: glenn at nevcal.com (Glenn Linderman) Date: Thu, 18 Mar 2010 13:31:33 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <201003190634.19250.steve@pearwood.info> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003182323.19458.steve@pearwood.info> <4BA2707A.80801@g.nevcal.com> <201003190634.19250.steve@pearwood.info> Message-ID: <4BA28DA5.8050608@nevcal.com> On 3/18/2010 12:34 PM, Steven D'Aprano wrote: > On Fri, 19 Mar 2010 05:27:06 am Glenn Linderman wrote: > > >>> Do you envisage any problems from allowing this instead? >>> >>> >>>>>> Decimal('1.1')< 2.2 >>>>>> >>> True >>> >> Yes. >> >> As any non-na?ve float user is aware, the proper form of float >> comparisons is not to use< or> or == or !=, but rather, instead of >> using< (to follow along with your example), one should use: >> >> Decimal('1.1') - 2.2< epsilon >> > And yet we allow > > 1.1< 2.2 > > instead of forcing users to do the "proper form". One can only wonder > why the various standards (actual and de-facto) for floating point > allows comparisons at all. > Hard to tell 1.1 < 2.2 from the second line of diff = 1.1 - 2.2 diff < epsilon Hard to enforce the "proper form", without being omniscient, and the "proper form" isn't always as simple as my example, to be truly proper often requires a lot more code (and a lot more omniscience). I'm +1 on adding an exception for Decimal/float comparisons, -0 on allowing it to be turned off to achieve compatible behavior, since Raymond pointed out that the compatible sorting behavior isn't stable in the presence of int, Decimal, and float. > But they do, and so does Python, and frankly even if the only reason is > to satisfy lazy coders who don't have a requirement for high accuracy, > then that's a good reason in my book, and one equally applicable to > Decimal and float. > A point well understood, but I personally would rather force Decimal/float comparisons to be explicitly converted to be lazily compared, if the arithmetic operations are forced to be explicit. From v+python at g.nevcal.com Fri Mar 19 00:25:53 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 18 Mar 2010 16:25:53 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <4BA29FB1.9020909@gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> Message-ID: <4BA2B681.6090709@g.nevcal.com> On 3/18/2010 2:48 PM, Nick Coghlan wrote: > When there is a clear, correct way (based on Decimal.from_float) to make > numeric comparison behave in accordance with the rules of mathematics, > do we really want to preserve strange, unintuitive behaviour like the above? > > Cheers, > Nick. > I'm aware of nothing that prevents the lazy coder from having a class unifiedNumber in his toolbox that implements his favorite type of conversions from various numeric types to whatever he thinks is the "best" one for his application, and then using it places where sources might be of various other numeric types. I'm aware of nothing that would prevent the lazy coder from implement comparison operators and even arithmetic operators on such a class. I believe, but haven't proven, and haven't used Python long enough to "just know", that such class could even implement what would appear to be operators that would seem to provide implicit conversion from the various other numeric classes, as long as one item was of the class. So I think it would be possible, with such a class, to have one's cake (nothing implicit), and eat it too (providing a way do comparisons and numerical operations, mostly implicitly). Glenn From tjreedy at udel.edu Fri Mar 19 01:27:25 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 18 Mar 2010 20:27:25 -0400 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <4BA29FB1.9020909@gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> Message-ID: On 3/18/2010 5:48 PM, Nick Coghlan wrote: > Mark Dickinson wrote: >> Could everyone live with making float<->Decimal comparisons raise an >> exception in 2.7? > > I could, with the caveat that *if* this causes problems for real world > code, then changing it to produce the correct answer (as per your patch) > should be applied as a bug fix in both 2.7 and 3.2. > > Note that even in Py3k there are some fairly serious weirdnesses kicking > around due to the intransitive nature of numeric equality though: > >>>> from decimal import Decimal as dec >>>> set((1, 1.0, dec("1.0"))) > {1} >>>> set((1.0, dec("1.0"))) > {1.0, Decimal('1.0')} > >>>> d = {} >>>> from decimal import Decimal as dec >>>> d[1] = d[1.0] = d[dec("1.0")] = 42 >>>> d > {1: 42} >>>> d[1.0] = d[dec("1.0")] = 42 >>>> d > {1: 42} >>>> del d[1] >>>> d[1.0] = d[dec("1.0")] = 42 >>>> d > {1.0: 42, Decimal('1.0'): 42} > > When there is a clear, correct way (based on Decimal.from_float) to make > numeric comparison behave in accordance with the rules of mathematics, > do we really want to preserve strange, unintuitive behaviour like the above? I would still strongly prefer that equality intransitivity be fixed one way or another (I suggested two on the tracker). The many ways it makes sets misbehave was discussed in both of http://mail.python.org/pipermail/python-list/2008-September/508859.html http://bugs.python.org/issue4087 Terry Jan Reedy From solipsis at pitrou.net Fri Mar 19 02:18:27 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 19 Mar 2010 01:18:27 +0000 (UTC) Subject: [Python-Dev] Decimal <-> float comparisons in py3k. References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> Message-ID: Glenn Linderman g.nevcal.com> writes: > > On 3/18/2010 2:48 PM, Nick Coghlan wrote: > > When there is a clear, correct way (based on Decimal.from_float) to make > > numeric comparison behave in accordance with the rules of mathematics, > > do we really want to preserve strange, unintuitive behaviour like the above? > > I'm aware of nothing that prevents the lazy coder from having a class > unifiedNumber in his toolbox [snip] Please stick to the topic. We are talking about Python's default behaviour here. Antoine. From glenn at nevcal.com Fri Mar 19 03:08:56 2010 From: glenn at nevcal.com (Glenn Linderman) Date: Thu, 18 Mar 2010 19:08:56 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> Message-ID: <4BA2DCB8.2040406@nevcal.com> On 3/18/2010 6:18 PM, Antoine Pitrou wrote: > Glenn Linderman g.nevcal.com> writes: > >> On 3/18/2010 2:48 PM, Nick Coghlan wrote: >> >>> When there is a clear, correct way (based on Decimal.from_float) to make >>> numeric comparison behave in accordance with the rules of mathematics, >>> do we really want to preserve strange, unintuitive behaviour like the above? >>> >> I'm aware of nothing that prevents the lazy coder from having a class >> unifiedNumber in his toolbox [snip] >> > Please stick to the topic. We are talking about Python's default behaviour here. > Yes, I consider my comment relevant, and think that you should apologize for claiming it is off topic. There are two choices on the table -- doing comparisons implicitly between Decimal and float, and raising an exception. It seems the current behavior, sorting by type, is universally disliked, but doing nothing is a third choice. So if the default behavior is to raise an exception, my comment pointed out the way comparisons could be provided, for those that need them. This allows both behaviors to exist concurrently. Python developers could even consider including such a library in the standard library, although my suggestion didn't include that. On the other hand, if the default behavior is to do an implicit conversion, I don't know of any way that that could be turned into an exception for those coders that don't want or don't like the particular type of implicit conversion chosen. Glenn From ctb at msu.edu Fri Mar 19 03:36:30 2010 From: ctb at msu.edu (C. Titus Brown) Date: Thu, 18 Mar 2010 19:36:30 -0700 Subject: [Python-Dev] GSoC 2010 is on -- projects? Message-ID: <20100319023630.GC1617@idyll.org> Hi all, once again, the PSF has been accepted as a mentoring foundation for the Google Summer of Code! This year, we're going to emphasize python 3 porting, so please think of projects you'd like to see tackled. Please submit ideas for projects as soon as possible, as students will be able to start applying soon. You can add them to this page: http://wiki.python.org/moin/SummerOfCode/2010 You can subscribe to the mentors list here, http://mail.python.org/mailman/listinfo/soc2010-mentors and the general discussion list (students included) here: http://mail.python.org/mailman/listinfo/soc2010-general thanks, --titus -- C. Titus Brown, ctb at msu.edu From benjamin at python.org Fri Mar 19 04:13:42 2010 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 18 Mar 2010 22:13:42 -0500 Subject: [Python-Dev] GSoC 2010 is on -- projects? In-Reply-To: <20100319023630.GC1617@idyll.org> References: <20100319023630.GC1617@idyll.org> Message-ID: <1afaf6161003182013j43de82aeheae2982bc6e4dc8b@mail.gmail.com> 2010/3/18 C. Titus Brown : > Hi all, > > once again, the PSF has been accepted as a mentoring foundation for the Google > Summer of Code! ?This year, we're going to emphasize python 3 porting, so > please think of projects you'd like to see tackled. This is not completely to the exclusion of other, projects, though, correct? For example, I think building a 64 bit PyPy backend would be a very worthy task. -- Regards, Benjamin From ctb at msu.edu Fri Mar 19 04:15:54 2010 From: ctb at msu.edu (C. Titus Brown) Date: Thu, 18 Mar 2010 20:15:54 -0700 Subject: [Python-Dev] GSoC 2010 is on -- projects? In-Reply-To: <1afaf6161003182013j43de82aeheae2982bc6e4dc8b@mail.gmail.com> References: <20100319023630.GC1617@idyll.org> <1afaf6161003182013j43de82aeheae2982bc6e4dc8b@mail.gmail.com> Message-ID: <20100319031554.GB5376@idyll.org> On Thu, Mar 18, 2010 at 10:13:42PM -0500, Benjamin Peterson wrote: > 2010/3/18 C. Titus Brown : > > Hi all, > > > > once again, the PSF has been accepted as a mentoring foundation for the Google > > Summer of Code! ??This year, we're going to emphasize python 3 porting, so > > please think of projects you'd like to see tackled. > > This is not completely to the exclusion of other, projects, though, > correct? For example, I think building a 64 bit PyPy backend would be > a very worthy task. Right -- emphasize only ;). Good projects + good students + good mentors are always welcome (and, actually, we'd like to focus on good students -- better that something get done than that something particularly relevant get chosen but NOT done by a poor student). thanks, --titus -- C. Titus Brown, ctb at msu.edu From benjamin at python.org Fri Mar 19 04:20:17 2010 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 18 Mar 2010 22:20:17 -0500 Subject: [Python-Dev] 3.1.2 tagging delayed a little Message-ID: <1afaf6161003182020l1dd73856k30942c590ebcfb8f@mail.gmail.com> Pending the resolution of a few Mac OS issues (#8068, #8069, and #8133), I'm not going to tag the release at the moment. Hopefully, we'll still be able to have a release in the next week. -- Regards, Benjamin From lgautier at gmail.com Fri Mar 19 07:23:20 2010 From: lgautier at gmail.com (Laurent Gautier) Date: Fri, 19 Mar 2010 07:23:20 +0100 Subject: [Python-Dev] GSoC 2010 is on -- projects? In-Reply-To: <20100319023630.GC1617@idyll.org> References: <20100319023630.GC1617@idyll.org> Message-ID: <4BA31858.3010904@gmail.com> On 3/19/10 3:36 AM, C. Titus Brown wrote: > Hi all, > > once again, the PSF has been accepted as a mentoring foundation for the Google > Summer of Code! This year, we're going to emphasize python 3 porting, so > please think of projects you'd like to see tackled. Hi, Does this mean that any other python project could potentially see itself ported to Python 3 in the course of this SoC ? If so, can any project owner submit a request for help, or is there going to be a list of projects that would nice to port, or will a voting system of some sort be put in place ? Best, Laurent From tjreedy at udel.edu Fri Mar 19 07:45:52 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 19 Mar 2010 02:45:52 -0400 Subject: [Python-Dev] GSoC 2010 is on -- projects? In-Reply-To: <4BA31858.3010904@gmail.com> References: <20100319023630.GC1617@idyll.org> <4BA31858.3010904@gmail.com> Message-ID: On 3/19/2010 2:23 AM, Laurent Gautier wrote: > On 3/19/10 3:36 AM, C. Titus Brown wrote: >> Hi all, >> >> once again, the PSF has been accepted as a mentoring foundation for >> the Google >> Summer of Code! This year, we're going to emphasize python 3 porting, so >> please think of projects you'd like to see tackled. > > Hi, > > > Does this mean that any other python project could potentially see > itself ported to Python 3 in the course of this SoC ? The theme should include both general porting tools and specific projects, especially infrastructure projects like numeric, which are blocking the porting of other projects. It would be nice if those doing specific projects contributed their experience/knowledge to a central pool. > If so, can any project owner submit a request for help, Any project owner is *always* free to ask for help (on python-list, but now here in this thread). Those who can also mentor might be more likely to get it. If I were a student, I would consider serious interest from a project owner (and a promise to distribute a port, when ready), a prerequisite. > or is there going to be a list > of projects that would nice to port, or will a voting system of some > sort be put in place ? Like most contributors, students choose projects, within the limits of what they can get mentors for, that scratch their itches. They may or may not otherwise be swayed by requests and opinions. My views. Terry Jan Reedy From techtonik at gmail.com Fri Mar 19 09:09:16 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 19 Mar 2010 10:09:16 +0200 Subject: [Python-Dev] Tracker reviews workflow and flags Message-ID: I want to push some of my patches before 2.7 and use 5-1 rule for that, but I can't come up with any review workflow other than mailing status of my comments to the issues here. I can't mark issues in any way. How about giving users ability to set flags or keywords? Maybe entering a separate field like "Review status"? Real world example with issue8151. It is an issue with a trivial patch in it. Everything what is needed is to dispatch it to stable `commit queue` and port to trunk. It is not 'easy' - it is 'trivial', but I have no means to mark it as 'easy' either, so even this trivial fix lies in tracker for three days waiting for review. About 'easy' flag: "6 easy This is an easy task (e.g. suitable for GHOP or bug day beginners)" It seems that it is for the issue that requires a patch, but if the patch is already there and nobody wants to commit it - what should I do? Finally, review queue proposal. To follow 5-1 rule, I need to review 5 issues, before mine is reviewed, so I need some help from tracker. 1. I lack expertise in certain areas (C/C++, Mac OS, etc.), so I would like to hide some issues under "Needs review" query, so they won't bother me (if too many people marked issues in this way - the stats may become a good reason for highly professional bug party) 2. I need ability to set "Needs review" flag back. Some issues were once reviewed, but since then they've got modified and need further comments. The need the review again. That means pushed back _into the end_ of patch queue. 3. Setting bug dependency by users. Often nobody wants to care about issue, because it is just too long, there is much irrelevant comments and nobody has time to read it. We don't have personal digg-like comment filters to cope with the amount of text to be brain-loaded. But it is possible to refactor such issue thread into separate digestable issue/issues. P.S. About personal comment filters if anybody is interested in that. Digg-like +1, -1 are good for voting, but for filtering it would be nice to: 1. have several filters for different aspects of the thread, 2. have JS filter by marking individual phrases in comments and adding ranges to filter using jquery / ajax This way reviews will be more fun and easy. -- anatoly t. From dirkjan at ochtman.nl Fri Mar 19 10:00:07 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Fri, 19 Mar 2010 10:00:07 +0100 Subject: [Python-Dev] Joel Spolsky on Mercurial In-Reply-To: References: Message-ID: On Thu, Mar 18, 2010 at 22:38, Terry Reedy wrote: > Having gotten that far, I think this might be worth referencing in new dev > docs. Will do. I finally read hginit yesterday, after having seen people rave about it on twitter for a few weeks, and it's a very friendly introduction. Cheers, Dirkjan From dickinsm at gmail.com Fri Mar 19 10:37:23 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Fri, 19 Mar 2010 09:37:23 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <4BA29FB1.9020909@gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> Message-ID: <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com> On Thu, Mar 18, 2010 at 9:48 PM, Nick Coghlan wrote: > Note that even in Py3k there are some fairly serious weirdnesses kicking > around due to the intransitive nature of numeric equality though: Yep. My personal favourite is: >>> from decimal import Decimal as dec >>> set((1, 1.0, dec(1))) == set((1.0, 1, dec(1))) False (The sets even have different sizes!) Note that while the originally proposed change does fix problems with sets of Decimals, ints and floats, and with sorting of lists of those types, it's not a complete fix: as soon as you throw Fractions into the mix, all the same problems arise again. Making hashes of int, float, Decimal *and* Fraction all compatible with one another, efficient for ints and floats, and not grossly inefficient for Fractions and Decimals, is kinda hairy, though I have a couple of ideas of how it could be done. Making Decimal-to-Fraction comparisons give correct results isn't easy either: the conversion in one direction (Fraction to Decimal) is lossy, while the conversion in the other direction (Decimal to Fraction) can't be performed efficiently if the Decimal has a large exponent (Decimal('1e123456')); so you can't just do the Decimal.from_float trick of whacking everything into a single domain and comparing there. Again, this is solvable, but not trivially so. See, this is what happens when a language conflates numeric equality with the equivalence relation used for membership testing. ;-). (I'm not really complaining, but this is definitely one of the downsides of this particular design decision.) > When there is a clear, correct way (based on Decimal.from_float) to make > numeric comparison behave in accordance with the rules of mathematics, > do we really want to preserve strange, unintuitive behaviour like the above? No. Not really, no. :) Mark From dickinsm at gmail.com Fri Mar 19 11:07:55 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Fri, 19 Mar 2010 10:07:55 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com> Message-ID: <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com> On Fri, Mar 19, 2010 at 9:37 AM, Mark Dickinson wrote: > Making hashes of int, > float, Decimal *and* Fraction all compatible with one another, > efficient for ints and floats, and not grossly inefficient for > Fractions and Decimals, is kinda hairy, though I have a couple of > ideas of how it could be done. To elaborate, here's a cute scheme for the above, which is actually remarkably close to what Python already does for ints and floats, and which doesn't require any of the numeric types to figure out whether it's exactly equal to an instance of some other numeric type. After throwing out infinities and nans (which can easily be dealt with separately), everything we care about is a rational number, so it's enough to come up with some mapping from the set of all rationals to the set of possible hashes, subject to the requirement that the mapping can be computed efficiently for the types we care about. For any prime P there's a natural 'reduce modulo P' map reduce : {rational numbers} --> {0, 1, ..., P-1, infinity} defined in pseudocode by: reduce(m/n) = ((m % P) * inv(n, P)) % P if n % P != 0 else infinity where inv(n, P) represents the modular inverse to n modulo P. Now let P be the handy Mersenne prime P = 2**31-1 (on a 64-bit machine, the almost equally handy prime 2**61-1 could be used instead), and define a hash function from the set of rationals to the set [-2**31, 2**31) by: hash(0) = 0 hash(m/n) = 1 + reduce(m/n - 1) if m/n > 0 # i.e., reduce m/n modulo P, but to [1..P] rather than [0..P-1]. hash(m/n) = -hash(-m/n) if m/n < 0. and in the last two cases, map a result of infinity to the unused hash value -2**31. For ints, this hash function is almost identical to what Python already has, except that the current int hash does a reduction modulo 2**32-1 or 2**64-1 rather than 2**31-1. For all small ints, hash(n) == n, as currently. Either way, the hash can be computed digit-by-digit in exactly the same manner. For floats, it's also easy to compute: express the float as m * 2**e for some integers m and e, compute hash(m), and rotate e bits in the appropriate direction. And it's straightforward to implement for the Decimal and Fraction types, too. (One minor detail: as usual, some postprocessing would be required to replace a hash of -1 with something else, since a hash value of -1 is invalid.) Mark From solipsis at pitrou.net Fri Mar 19 12:50:16 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 19 Mar 2010 11:50:16 +0000 (UTC) Subject: [Python-Dev] Decimal & lt; -& gt; float comparisons in py3k. References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> Message-ID: Glenn Linderman nevcal.com> writes: > > On the other hand, if the default behavior is to do an implicit > conversion, I don't know of any way that that could be turned into an > exception for those coders that don't want or don't like the particular > type of implicit conversion chosen. You still haven't given a good reason why we should raise an exception rather than do a comparison, though. The fact that some particular coders don't like "the particular type of implicit conversion chosen" is a particularly weak argument. Python isn't a language construction framework; we try to choose useful defaults rather than simply give out a box of tools. If some people don't like the defaults (significant indentation, limitless precision integers, etc.), there are other choices out there. Antoine. From ncoghlan at gmail.com Fri Mar 19 12:52:57 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 19 Mar 2010 21:52:57 +1000 Subject: [Python-Dev] Joel Spolsky on Mercurial In-Reply-To: References: Message-ID: <4BA36599.7000106@gmail.com> Dirkjan Ochtman wrote: > On Thu, Mar 18, 2010 at 22:38, Terry Reedy wrote: >> Having gotten that far, I think this might be worth referencing in new dev >> docs. > > Will do. I finally read hginit yesterday, after having seen people > rave about it on twitter for a few weeks, and it's a very friendly > introduction. The key insight for me was the "manage versions" vs "manage change sets" distinction between a traditional VCS and a DVCS. I kind of knew that already, but that page expressed it really well. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From arcriley at gmail.com Fri Mar 19 12:57:26 2010 From: arcriley at gmail.com (Arc Riley) Date: Fri, 19 Mar 2010 07:57:26 -0400 Subject: [Python-Dev] GSoC 2010 is on -- projects? In-Reply-To: <4BA31858.3010904@gmail.com> References: <20100319023630.GC1617@idyll.org> <4BA31858.3010904@gmail.com> Message-ID: Hi Laurent If your community project would like help porting to Python 3, and you feel this work is enough for a student to work full time for several weeks on, then please do add it to the GSoC ideas page on the wiki. There will be another program running for high school students which is more suitable for smaller tasks (2-3 days each), more on-par with the actual time it takes to port most Python packages. On Fri, Mar 19, 2010 at 2:23 AM, Laurent Gautier wrote: > > Hi, > > > Does this mean that any other python project could potentially see itself > ported to Python 3 in the course of this SoC ? If so, can any project owner > submit a request for help, or is there going to be a list of projects that > would nice to port, or will a voting system of some sort be put in place ? > > > Best, > > > Laurent > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/arcriley%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aries.shuaib at gmail.com Fri Mar 19 12:59:38 2010 From: aries.shuaib at gmail.com (M. Shuaib Khan) Date: Fri, 19 Mar 2010 16:59:38 +0500 Subject: [Python-Dev] Joel Spolsky on Mercurial In-Reply-To: <4BA36599.7000106@gmail.com> References: <4BA36599.7000106@gmail.com> Message-ID: On Fri, Mar 19, 2010 at 4:52 PM, Nick Coghlan wrote: > Dirkjan Ochtman wrote: > > On Thu, Mar 18, 2010 at 22:38, Terry Reedy wrote: > >> Having gotten that far, I think this might be worth referencing in new > dev > >> docs. > > > > Will do. I finally read hginit yesterday, after having seen people > > rave about it on twitter for a few weeks, and it's a very friendly > > introduction. > > The key insight for me was the "manage versions" vs "manage change sets" > distinction between a traditional VCS and a DVCS. I kind of knew that > already, but that page expressed it really well. > Hi, Is Python-dev going to consider shifting their repo to mercurial/git instead of SVN? :) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > --------------------------------------------------------------- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/aries.shuaib%40gmail.com > -- M. Shuaib Khan http://mshuaibkhan.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Mar 19 13:03:25 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 19 Mar 2010 22:03:25 +1000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com> <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com> Message-ID: <4BA3680D.8070909@gmail.com> Mark Dickinson wrote: > On Fri, Mar 19, 2010 at 9:37 AM, Mark Dickinson wrote: > For ints, this hash function is almost identical to what Python > already has, except that the current int hash does a reduction modulo > 2**32-1 or 2**64-1 rather than 2**31-1. For all small ints, hash(n) > == n, as currently. Either way, the hash can be computed > digit-by-digit in exactly the same manner. For floats, it's also easy > to compute: express the float as m * 2**e for some integers m and e, > compute hash(m), and rotate e bits in the appropriate direction. And > it's straightforward to implement for the Decimal and Fraction types, > too. It seems to me that given the existing conflation of numeric equivalence and containment testing, going the whole hog and fixing the set membership problem for all of our rational types would be the right thing to do. Would it be worth providing the underlying implementation of the hash algorithm as a math.hash_rational function and then use it for Decimal and Fraction? That would have the virtue of making it easy for others to define numeric types that "played well" with numeric equivalence. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From phd at phd.pp.ru Fri Mar 19 13:09:44 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Fri, 19 Mar 2010 15:09:44 +0300 Subject: [Python-Dev] Joel Spolsky on Mercurial In-Reply-To: References: <4BA36599.7000106@gmail.com> Message-ID: <20100319120944.GA12240@phd.pp.ru> On Fri, Mar 19, 2010 at 04:59:38PM +0500, M. Shuaib Khan wrote: > Is Python-dev going to consider shifting their repo to mercurial/git instead > of SVN? :) http://mail.python.org/pipermail/python-dev/2009-March/087931.html The decision was made about a year ago. http://www.python.org/dev/peps/pep-0374/ http://www.python.org/dev/peps/pep-0385/ Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From alex.nanou at gmail.com Fri Mar 19 13:46:07 2010 From: alex.nanou at gmail.com (Alex A. Naanou) Date: Fri, 19 Mar 2010 15:46:07 +0300 Subject: [Python-Dev] binary operation heuristics -- bug or undocumented? Message-ID: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com> Hi, A friend of mine stumbled upon the following behavior: ---cut--- >>> class A(object): pass ... >>> class B(object): ... def __add__(self, other): ... print 'B: adding B and %s objects.' % other.__class__.__name__ ... >>> class C(object): ... def __radd__(self, other): ... print 'C: adding C and %s objects.' % other.__class__.__name__ ... >>> a, b, c = A(), B(), C() >>> b + c B: adding B and C objects. >>> a + c C: adding C and A objects. # so far, quite logical. now let's do this: >>> 1 + c C: adding C and int objects. --uncut-- My first expectation would be to get a TypeError here, as ints indeed have an __add__ method, and they do not know anything about C objects (obviously :) ). On second thought, giving client code priority to handle things has it's merits. The problem is that I found no mention of this behavior in the docs. P.S. tested in 2.5 through 3.0 and PyPy Thanks. -- Alex. From dickinsm at gmail.com Fri Mar 19 13:55:58 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Fri, 19 Mar 2010 12:55:58 +0000 Subject: [Python-Dev] binary operation heuristics -- bug or undocumented? In-Reply-To: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com> References: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com> Message-ID: <5c6f2a5d1003190555l56cd5237g9b7a410a835390f6@mail.gmail.com> On Fri, Mar 19, 2010 at 12:46 PM, Alex A. Naanou wrote: > A friend of mine stumbled upon the following behavior: > > > ---cut--- > >>>> class A(object): pass > ... >>>> class B(object): > ... ? ? def __add__(self, other): > ... ? ? ? ? print 'B: adding B and %s objects.' % other.__class__.__name__ > ... >>>> class C(object): > ... ? ? def __radd__(self, other): > ... ? ? ? ? print 'C: adding C and %s objects.' % other.__class__.__name__ > ... >>>> a, b, c = A(), B(), C() > >>>> b + c > B: adding B and C objects. > >>>> a + c > C: adding C and A objects. > > > # so far, quite logical. now let's do this: > >>>> 1 + c > C: adding C and int objects. > > > --uncut-- > > My first expectation would be to get a TypeError here, as ints indeed > have an __add__ method, and they do not know anything about C objects > (obviously :) ). Yes: the int.__add__ method is tried first. Since it doesn't know anything about C objects it returns NotImplemented, and then C.__radd__ is given a chance to do the addition. The rules are documented here: http://docs.python.org/reference/datamodel.html#coercion-rules Mark From fuzzyman at voidspace.org.uk Fri Mar 19 14:05:20 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 19 Mar 2010 13:05:20 +0000 Subject: [Python-Dev] binary operation heuristics -- bug or undocumented? In-Reply-To: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com> References: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com> Message-ID: <4BA37690.1040601@voidspace.org.uk> On 19/03/2010 12:46, Alex A. Naanou wrote: > [snip...] >>>> class C(object): >>>> > ... def __radd__(self, other): > ... print 'C: adding C and %s objects.' % other.__class__.__name__ > ... > >>>> a, b, c = A(), B(), C() >>>> 1 + c >>>> > C: adding C and int objects. > > > That is the whole point of the __radd__ method. ints don't know how to add themselves to C objects, so int.__add__ will return NotImplemented and then Python will try C.__radd__. All the best, Michael > --uncut-- > > My first expectation would be to get a TypeError here, as ints indeed > have an __add__ method, and they do not know anything about C objects > (obviously :) ). On second thought, giving client code priority to > handle things has it's merits. > > The problem is that I found no mention of this behavior in the docs. > > > P.S. tested in 2.5 through 3.0 and PyPy > > Thanks. > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From phd at phd.pp.ru Fri Mar 19 14:07:34 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Fri, 19 Mar 2010 16:07:34 +0300 Subject: [Python-Dev] right-side binary operations In-Reply-To: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com> References: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com> Message-ID: <20100319130734.GA20925@phd.pp.ru> On Fri, Mar 19, 2010 at 03:46:07PM +0300, Alex A. Naanou wrote: > >>> class C(object): > ... def __radd__(self, other): > ... print 'C: adding C and %s objects.' % other.__class__.__name__ > ... > >>> 1 + c > C: adding C and int objects. > > My first expectation would be to get a TypeError here, as ints indeed > have an __add__ method, and they do not know anything about C objects > (obviously :) ). On second thought, giving client code priority to > handle things has it's merits. > > The problem is that I found no mention of this behavior in the docs. It's well-known and documented behavior. It's what r-methods are for. See http://docs.python.org/reference/datamodel.html#emulating-numeric-types "These methods are called to implement the binary arithmetic operations (+, -, *, /, %, divmod(), pow(), **, <<, >>, &, ^, |) with reflected (swapped) operands. These functions are only called if the left operand does not support the corresponding operation and the operands are of different types. [3] For instance, to evaluate the expression x - y, where y is an instance of a class that has an __rsub__() method, y.__rsub__(x) is called if x.__sub__(y) returns NotImplemented." Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From skip at pobox.com Fri Mar 19 14:13:18 2010 From: skip at pobox.com (skip at pobox.com) Date: Fri, 19 Mar 2010 08:13:18 -0500 Subject: [Python-Dev] Joel Spolsky on Mercurial In-Reply-To: References: <4BA36599.7000106@gmail.com> Message-ID: <19363.30830.383690.400879@montanaro.dyndns.org> M> Is Python-dev going to consider shifting their repo to mercurial/git M> instead of SVN? :) Yes, Dirkjan has been working on it. My initial experience with hg as a quick way to revision control anything anywhere on a small scale ("hmmm, I should track changes to /etc/hosts... cd /etc ; hg init ; hg add hosts") worked quite well, but when I tried to use it for something where I was actually working collaboratively (my lockfile module) I managed to get my code/branches/heads/whatever completely f**ked up. I gave up and just went back to Subversion. Skip From alex.nanou at gmail.com Fri Mar 19 14:16:42 2010 From: alex.nanou at gmail.com (Alex A. Naanou) Date: Fri, 19 Mar 2010 16:16:42 +0300 Subject: [Python-Dev] binary operation heuristics -- bug or undocumented? In-Reply-To: <4BA37690.1040601@voidspace.org.uk> References: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com> <4BA37690.1040601@voidspace.org.uk> Message-ID: <36f889221003190616j25053655h11f1d9417f77821b@mail.gmail.com> Thanks! On Fri, Mar 19, 2010 at 16:05, Michael Foord wrote: > On 19/03/2010 12:46, Alex A. Naanou wrote: >> >> [snip...] >>>>> >>>>> class C(object): >>>>> >> >> ... ? ? def __radd__(self, other): >> ... ? ? ? ? print 'C: adding C and %s objects.' % other.__class__.__name__ >> ... >> >>>>> >>>>> a, b, c = A(), B(), C() >>>>> 1 + c >>>>> >> >> C: adding C and int objects. >> >> >> > > That is the whole point of the __radd__ method. ints don't know how to add > themselves to C objects, so int.__add__ will return NotImplemented and then > Python will try C.__radd__. > > All the best, > > Michael > >> --uncut-- >> >> My first expectation would be to get a TypeError here, as ints indeed >> have an __add__ method, and they do not know anything about C objects >> (obviously :) ). On second thought, giving client code priority to >> handle things has it's merits. >> >> The problem is that I found no mention of this behavior in the docs. >> >> >> P.S. tested in 2.5 through 3.0 and PyPy >> >> Thanks. >> >> > > > -- > http://www.ironpythoninaction.com/ > http://www.voidspace.org.uk/blog > > READ CAREFULLY. By accepting and reading this email you agree, on behalf of > your employer, to release me from all obligations and waivers arising from > any and all NON-NEGOTIATED agreements, licenses, terms-of-service, > shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, > non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have > entered into with your employer, its partners, licensors, agents and > assigns, in perpetuity, without prejudice to my ongoing rights and > privileges. You further represent that you have the authority to release me > from any BOGUS AGREEMENTS on behalf of your employer. P.S. like the licence :) > > > -- Alex. From casevh at gmail.com Fri Mar 19 14:17:53 2010 From: casevh at gmail.com (Case Vanhorsen) Date: Fri, 19 Mar 2010 06:17:53 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com> <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com> Message-ID: <99e0b9531003190617w28c5da2etf8b35c54f223a036@mail.gmail.com> On Fri, Mar 19, 2010 at 3:07 AM, Mark Dickinson wrote: > On Fri, Mar 19, 2010 at 9:37 AM, Mark Dickinson wrote: >> Making hashes of int, >> float, Decimal *and* Fraction all compatible with one another, >> efficient for ints and floats, and not grossly inefficient for >> Fractions and Decimals, is kinda hairy, though I have a couple of >> ideas of how it could be done. > > To elaborate, here's a cute scheme for the above, which is actually > remarkably close to what Python already does for ints and floats, and > which doesn't require any of the numeric types to figure out whether > it's exactly equal to an instance of some other numeric type. > > After throwing out infinities and nans (which can easily be dealt with > separately), everything we care about is a rational number, so it's > enough to come up with some mapping from the set of all rationals to > the set of possible hashes, subject to the requirement that the > mapping can be computed efficiently for the types we care about. > > For any prime P there's a natural 'reduce modulo P' map > > ?reduce : {rational numbers} --> {0, 1, ..., P-1, infinity} > > defined in pseudocode by: > > ?reduce(m/n) = ((m % P) * inv(n, P)) % P ?if n % P != 0 ?else infinity > > where inv(n, P) represents the modular inverse to n modulo P. > > Now let P be the handy Mersenne prime P = 2**31-1 (on a 64-bit > machine, the almost equally handy prime 2**61-1 could be used > instead), and define a hash function from the set of rationals to the > set [-2**31, 2**31) by: > > hash(0) = 0 > hash(m/n) = 1 + reduce(m/n - 1) if m/n > 0 ? # i.e., reduce m/n modulo > P, but to [1..P] rather than [0..P-1]. > hash(m/n) = -hash(-m/n) if m/n < 0. > > and in the last two cases, map a result of infinity to the unused hash > value -2**31. > > For ints, this hash function is almost identical to what Python > already has, except that the current int hash does a reduction modulo > 2**32-1 or 2**64-1 rather than 2**31-1. ?For all small ints, hash(n) > == n, as currently. ?Either way, the hash can be computed > digit-by-digit in exactly the same manner. ?For floats, it's also easy > to compute: ?express the float as m * 2**e for some integers m and e, > compute hash(m), and rotate e bits in the appropriate direction. ?And > it's straightforward to implement for the Decimal and Fraction types, > too. Will this change the result of hashing a long? I know that both gmpy and SAGE use their own hash implementations for performance reasons. I understand that CPython's hash function is an implementation detail, but there are external modules that rely on the existing hash behavior. FWIW, I'd prefer 2.7 and 3.2 have the same behavior. I don't mind the existing 3.1 behavior and I'd rather not have a difference between 3.1 and 3.2. casevh > > (One minor detail: ?as usual, some postprocessing would be required to > replace a hash of -1 with something else, since a hash value of -1 is > invalid.) > > Mark > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/casevh%40gmail.com > From eric at trueblade.com Fri Mar 19 14:33:15 2010 From: eric at trueblade.com (Eric Smith) Date: Fri, 19 Mar 2010 09:33:15 -0400 Subject: [Python-Dev] Bug? syslog.openlog using ident "python" by default. In-Reply-To: <20100318115844.GA7374@tummy.com> References: <20100318115844.GA7374@tummy.com> Message-ID: <4BA37D1B.9020306@trueblade.com> Sean Reifschneider wrote: > If you call: > > from syslog import syslog, openlog > syslog('My error message') > > Something like the following gets logged: > > Mar 18 05:20:22 guin python: My error message > ^^^^^^ > > Where I'm annoyed by the "python" in the above. This is pulled from > the C argv[0]. IMHO, what we really want is the Python sys.argv[0]. I agree with this. The existing behavior is very annoying. I don't have any particular thoughts about the implementation. Eric. From brian.curtin at gmail.com Fri Mar 19 15:20:00 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Fri, 19 Mar 2010 09:20:00 -0500 Subject: [Python-Dev] Tracker reviews workflow and flags In-Reply-To: References: Message-ID: On Fri, Mar 19, 2010 at 03:09, anatoly techtonik wrote: > I want to push some of my patches before 2.7 and use 5-1 rule for > that, but I can't come up with any review workflow other than mailing > status of my comments to the issues here. I can't mark issues in any > way. How about giving users ability to set flags or keywords? Maybe > entering a separate field like "Review status"? > We already have "Patch Review" as a stage, and "needs review" as a flag, which I feel is more than enough. If you don't have the privileges to modify an issue, you can always comment on the issue with something like "can this issue be set to the patch review stage?" Someone will probably take a look at it and set it accordingly. Real world example with issue8151. It is an issue with a trivial patch > in it. Everything what is needed is to dispatch it to stable `commit > queue` and port to trunk. It is not 'easy' - it is 'trivial', but I > have no means to mark it as 'easy' either, so even this trivial fix > lies in tracker for three days waiting for review. > Given that we have hundreds of issues with patches waiting for review, three days isn't so bad :) As with any project, there are only so many people and so much time to get the work done. > About 'easy' flag: > "6 easy This is an easy task (e.g. suitable for GHOP or bug day > beginners)" > It seems that it is for the issue that requires a patch, but if the > patch is already there and nobody wants to commit it - what should I > do? > Post a comment on the issue asking what the status is. If an approved patch has been sitting for a few weeks, make a comment. If it has been there for a few days, I'd let it slide but keep it on your radar. > > Finally, review queue proposal. > To follow 5-1 rule, I need to review 5 issues, before mine is > reviewed, so I need some help from tracker. > 1. I lack expertise in certain areas (C/C++, Mac OS, etc.), so I would > like to hide some issues under "Needs review" query, so they won't > bother me (if too many people marked issues in this way - the stats > may become a good reason for highly professional bug party) > There already exists a component field which has Macintosh as an option, so you could filter on that. Same goes for Windows. Drilling down to the language(s) involved in the issue feels like just another step that would get limited use. I do see what you are saying, though, because I'm sure it's frustrating to want to look out there for a nice pure-Python issue, then the first 10 issues you click on require C code (or vice versa). > 2. I need ability to set "Needs review" flag back. Some issues were > once reviewed, but since then they've got modified and need further > comments. The need the review again. That means pushed back _into the > end_ of patch queue. > I don't think the "needs review" flag gets unset, but if it has been unset manually and you've made changes, you can ask for another review. If you can get the changes made quickly, you might be able to get the previous reviewer to look at it again while it's still fresh in their mind. > 3. Setting bug dependency by users. Often nobody wants to care about > issue, because it is just too long, there is much irrelevant comments > and nobody has time to read it. We don't have personal digg-like > comment filters to cope with the amount of text to be brain-loaded. > But it is possible to refactor such issue thread into separate > digestable issue/issues. > > > P.S. About personal comment filters if anybody is interested in that. > Digg-like +1, -1 are good for voting, but for filtering it would be > nice to: 1. have several filters for different aspects of the thread, > 2. have JS filter by marking individual phrases in comments and adding > ranges to filter using jquery / ajax > > This way reviews will be more fun and easy. > -- > anatoly t. > I think that would take away from the goal of fixing the issue. Even some -1 comments could contain helpful information, but you might have to explicitly click on that hidden comment to find out. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Fri Mar 19 15:22:05 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 19 Mar 2010 10:22:05 -0400 Subject: [Python-Dev] Tracker reviews workflow and flags In-Reply-To: References: Message-ID: <20100319142205.E11FC1FDA39@kimball.webabinitio.net> On Fri, 19 Mar 2010 10:09:16 +0200, anatoly techtonik wrote: > I want to push some of my patches before 2.7 and use 5-1 rule for > that, but I can't come up with any review workflow other than mailing > status of my comments to the issues here. I can't mark issues in any > way. How about giving users ability to set flags or keywords? Maybe > entering a separate field like "Review status"? With our current workflow, mailing the status of your reviews here is pretty much the only option. ("I've reviewed issues a, b, c, d, and e, could someone please review y?"). > Real world example with issue8151. It is an issue with a trivial patch > in it. Everything what is needed is to dispatch it to stable `commit > queue` and port to trunk. It is not 'easy' - it is 'trivial', but I > have no means to mark it as 'easy' either, so even this trivial fix > lies in tracker for three days waiting for review. Three days? That's nothing :) > About 'easy' flag: > "6 easy This is an easy task (e.g. suitable for GHOP or bug day beginners)" > It seems that it is for the issue that requires a patch, but if the > patch is already there and nobody wants to commit it - what should I > do? If an issue has a patch the 'easy' flag doesn't really apply any more, but it doesn't always get deleted, either. Issues that have a patch should get set to "patch review" stage, unless tests are missing, in which case it should get set to "needs test". Making changes like that (and also updating keywords) requires tracker privileges, but we give those out more freely than we do commit privs. You could try hanging out on #python-dev on Freenode, where there will (usually) be people who can respond to your requests to update the status of tracker issues. Make enough good requests and we'll get you tracker privs so you can do it yourself. ("Enough" is probably a pretty small number, by the way :) > Finally, review queue proposal. > To follow 5-1 rule, I need to review 5 issues, before mine is > reviewed, so I need some help from tracker. > 1. I lack expertise in certain areas (C/C++, Mac OS, etc.), so I would > like to hide some issues under "Needs review" query, so they won't > bother me (if too many people marked issues in this way - the stats > may become a good reason for highly professional bug party) To avoid issues involving C coding, try restricting your search to 'Library' issues. There is C code in some library modules, but less so than in the core ;). It would be nice if you could search on more than one component, but currently the interface doesn't allow that. (You might be able to do it by hand constructing a query string, I haven't checked.) > 2. I need ability to set "Needs review" flag back. Some issues were > once reviewed, but since then they've got modified and need further > comments. The need the review again. That means pushed back _into the > end_ of patch queue. I'm not sure exactly what the 'needs review' keyword is for, frankly :) Talking about pushing things back into the end of the patch queue makes the assumption that there *is* a queue. There isn't, there's just the set of open issues with patches that committers look at when they can or when they have an itch to scratch. Making a comment on an issue (such as doing a review) generates an email to everyone on the nosy list and to the python-bugs list (for those people who are subscribed), and thus in a fashion brings the issue to the 'top of the pile' in some sense and for a short period. Often things happen to the issue when a comment is posted, but not always. As with everything else, it depends on whether anyone has both the time and the interest to take action. > 3. Setting bug dependency by users. Often nobody wants to care about > issue, because it is just too long, there is much irrelevant comments > and nobody has time to read it. We don't have personal digg-like > comment filters to cope with the amount of text to be brain-loaded. > But it is possible to refactor such issue thread into separate > digestable issue/issues. > > P.S. About personal comment filters if anybody is interested in that. > Digg-like +1, -1 are good for voting, but for filtering it would be > nice to: 1. have several filters for different aspects of the thread, > 2. have JS filter by marking individual phrases in comments and adding > ranges to filter using jquery / ajax -1 on comment filtering. The text in the issue is the history of the issue. Reading it is part of doing an issue review. I can't imagine that the effort of manually blocking certain comments so that you didn't see them when later looking at the issue again would be worth it. It seems to me you'd be much more likely to accidentally block something you should be reading than you would be to make the issue usefully more readable. If an issue needs to be refactored, new issues can be opened with pointers to the old issue, and pointers added to the old issue to the new. This can be done by anyone just by saying, eg: 'see issue 1234' (roundup turns that into a link), and you can also suggest that the bug be added to the dependency list by someone who can, if that is appropriate. This happens fairly regularly with complex issues, or issues where we discover more than one bug is involved. Of course, it doesn't always happen when it should. The roundup code lives in the SVN repository, and you can set up your own instance of the tracker to experiment with changes. Patches are welcome, although we have a few in the meta-tracker[1] already that haven't been reviewed. Currently Martin is pretty much the only one doing code-level tracker work, and while I hope to get involved in that at some point it hasn't happened yet. So if you want to contribute some time in that area, that would be great. You could join the tracker-discuss list (though it is very low volume). [1] The 'report tracker problems' link on the lower left, if you haven't noticed it. -- R. David Murray www.bitdance.com From lgautier at gmail.com Fri Mar 19 15:23:33 2010 From: lgautier at gmail.com (Laurent Gautier) Date: Fri, 19 Mar 2010 15:23:33 +0100 Subject: [Python-Dev] GSoC 2010 is on -- projects? In-Reply-To: References: <20100319023630.GC1617@idyll.org> <4BA31858.3010904@gmail.com> Message-ID: <4BA388E5.1030703@gmail.com> On 3/19/10 12:57 PM, Arc Riley wrote: > Hi Laurent > > If your community project would like help porting to Python 3, and you > feel this work is enough for a student to work full time for several > weeks on, then please do add it to the GSoC ideas page on the wiki. Whether this is worth weeks of work or not will depend on a given student's knowledge about Python 3, and I'd suspect that the GSoC would be an opportunity for a number of applicant to actually learn the intricacies of Python 3. Developing Python 3-specific features could be used to increase the amount of work on the project, but I am uncertain of whether this is worth a full GSoC. > There will be another program running for high school students which is > more suitable for smaller tasks (2-3 days each), more on-par with the > actual time it takes to port most Python packages. > My project is roughly 6000 lines of Python and 6000 lines of C (Mostly C-level Python API bindings). I tried porting to Python 3 in the past, and it went fast (less than a day). The hurdle is that this resulted in segfaults for a lot of string-related features (You know, the byte/string thing). Tracing issues at the C level can be time-consuming, so I am hesitating to claim that 2-3 days of an high-school student would be enough. Could several ports be bundled in a GSoC (the target projects would be grouped by theme, somehow). L. > On Fri, Mar 19, 2010 at 2:23 AM, Laurent Gautier > wrote: > > > Hi, > > > Does this mean that any other python project could potentially see > itself ported to Python 3 in the course of this SoC ? If so, can any > project owner submit a request for help, or is there going to be a > list of projects that would nice to port, or will a voting system of > some sort be put in place ? > > > Best, > > > Laurent > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/arcriley%40gmail.com > > From rhamph at gmail.com Fri Mar 19 16:22:12 2010 From: rhamph at gmail.com (Adam Olsen) Date: Fri, 19 Mar 2010 09:22:12 -0600 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> Message-ID: On Thu, Mar 18, 2010 at 12:41, Mark Dickinson wrote: > I'm only seeing two arguments against this at the moment: (1) it has > the potential to break code that relies on being able to sort > heterogeneous lists. ?But given that heterogeneous lists can't be > sorted stably anyway (see my earlier post about lists containing ints, > floats and Decimals), perhaps this is an acceptable risk. (2) A few of > the posters here (Steven, Nick, and me) feel that it's slightly more > natural to allow these comparisons; ?but I think the argument's fairly > evenly balanced at the moment between those who'd prefer an exception > and those who'd prefer to allow the comparisons. Conceptually I like the idea of them all being comparable, but are there any real use cases involving heterogeneous lists? All the examples I've seen have focused on how they're broken, not on how they'd be correct (including possible math after the comparison!) if the language compared properly. Without such use cases allowing comparison seems like a lot of work for nothing. -- Adam Olsen, aka Rhamphoryncus From raymond.hettinger at gmail.com Fri Mar 19 17:38:27 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 19 Mar 2010 09:38:27 -0700 Subject: [Python-Dev] Decimal & lt; -& gt; float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> Message-ID: <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> On Mar 19, 2010, at 4:50 AM, Antoine Pitrou wrote: > Glenn Linderman nevcal.com> writes: >> >> On the other hand, if the default behavior is to do an implicit >> conversion, I don't know of any way that that could be turned into an >> exception for those coders that don't want or don't like the particular >> type of implicit conversion chosen. > > You still haven't given a good reason why we should raise an exception rather > than do a comparison, though. > > The fact that some particular coders don't like "the particular type of implicit > conversion chosen" is a particularly weak argument. Python isn't a language > construction framework; we try to choose useful defaults rather than simply give > out a box of tools. If some people don't like the defaults (significant > indentation, limitless precision integers, etc.), there are other choices out > there. The reason to prefer an exception is that decimal/float comparisons are more likely to be a programmer error than an intended behavior. Real use cases for decimal/float comparisons are rare and would be more clearly expressed with an explicit conversion using Decimal.from_float(). Of course there is a precedent, I can compare "120" < 140 in AWK and get an automatic implicit conversion ;-) Just because we can compare, doesn't mean we should. Raymond From amk at amk.ca Fri Mar 19 17:39:58 2010 From: amk at amk.ca (A.M. Kuchling) Date: Fri, 19 Mar 2010 12:39:58 -0400 Subject: [Python-Dev] Tracker reviews workflow and flags In-Reply-To: <20100319142205.E11FC1FDA39@kimball.webabinitio.net> References: <20100319142205.E11FC1FDA39@kimball.webabinitio.net> Message-ID: <20100319163958.GB8716@amk-desktop.matrixgroup.net> On Fri, Mar 19, 2010 at 10:22:05AM -0400, R. David Murray wrote: > > Real world example with issue8151. It is an issue with a trivial patch > > in it. Everything what is needed is to dispatch it to stable `commit > > queue` and port to trunk. It is not 'easy' - it is 'trivial', but I > > have no means to mark it as 'easy' either, so even this trivial fix > > lies in tracker for three days waiting for review. > > Three days? That's nothing :) I think Anatoly has a point, though; there are issues where the people discussing the fix have come to a consensus, but none of them are committers, or the committers involved are too busy. There's a patch ready to apply, but no committer knows about it. > To avoid issues involving C coding, try restricting your search to > 'Library' issues. There is C code in some library modules, but less so > than in the core ;). It would be nice if you could search on more than Actually Library should pretty much always be pure-Python. Issues related to C modules such as socket are classified under the 'Extension modules' component. Of course, sometimes the ultimate cause will migrate (a bug in SocketServer.py is traced to a flaw in socket, for example). --amk From tjreedy at udel.edu Fri Mar 19 18:15:46 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 19 Mar 2010 13:15:46 -0400 Subject: [Python-Dev] Tracker reviews workflow and flags In-Reply-To: <20100319142205.E11FC1FDA39@kimball.webabinitio.net> References: <20100319142205.E11FC1FDA39@kimball.webabinitio.net> Message-ID: On 3/19/2010 10:22 AM, R. David Murray wrote: > This can be done by anyone just by saying, eg: 'see issue 1234' (roundup > turns that into a link), That should be 'see issue #1234' to get the autolink. From http://wiki.python.org/moin/TrackerDocs/ The tracker converts some specially formatted words in messages into links. The list includes "#" links to the issue "msg" links to the message "r, "rev", "revision " links to svn.python.org displaying the checked in changes. From tjreedy at udel.edu Fri Mar 19 18:20:33 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 19 Mar 2010 13:20:33 -0400 Subject: [Python-Dev] Joel Spolsky on Mercurial In-Reply-To: References: Message-ID: On 3/19/2010 5:00 AM, Dirkjan Ochtman wrote: > On Thu, Mar 18, 2010 at 22:38, Terry Reedy wrote: >> Having gotten that far, I think this might be worth referencing in new dev >> docs. > > Will do. I finally read hginit yesterday, after having seen people > rave about it on twitter for a few weeks, and it's a very friendly > introduction. Having read further 'lessons', you might also want to note that the Python group workflow differs (and briefly how so) from that given as a possibility, assuming that it does. Terry Jan Reedy From tjreedy at udel.edu Fri Mar 19 18:33:14 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 19 Mar 2010 13:33:14 -0400 Subject: [Python-Dev] binary operation heuristics -- bug or undocumented? In-Reply-To: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com> References: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com> Message-ID: On 3/19/2010 8:46 AM, Alex A. Naanou wrote: > A friend of mine stumbled upon the following behavior: [snip] Questions like this are more appropriately posted to python-list, where you would have gotten the same answer. tjr From solipsis at pitrou.net Fri Mar 19 19:11:18 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 19 Mar 2010 18:11:18 +0000 (UTC) Subject: [Python-Dev] =?utf-8?b?RGVjaW1hbCAmYW1wO2FtcDsgbHQ7IC0mYW1wO2Ft?= =?utf-8?q?p=3B_gt=3B=09float_comparisons_in_py3k=2E?= References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> Message-ID: Raymond Hettinger gmail.com> writes: > > The reason to prefer an exception is that decimal/float comparisons > are more likely to be a programmer error than an intended behavior. Not more so than float/int or decimal/int or bool/int comparisons, which all work. We forbid comparisons when there is a real danger or ambiguity, such as unicode vs. bytes. There is no such danger or ambiguity when comparing a decimal with a float. I don't see the point of being so restrictive; Python is not Java, and typing is not supposed to be a synonym for bondage. > Of course there is a precedent, I can compare "120" < 140 in AWK > and get an automatic implicit conversion The proper precedent in this context, though, is this one (py3k): >>> 1 < 2.0 True >>> 1 < Decimal("2.0") True >>> 1 > Decimal("2.0") False >>> 1 > 2.0 False >>> True > 0.5 True >>> True > 1.5 False >>> True > Decimal("0.5") True >>> True > Decimal("1.5") False Are you suggesting to change all the above comparisons to raise a TypeError? cheers Antoine. From zvezdan at zope.com Fri Mar 19 19:09:28 2010 From: zvezdan at zope.com (Zvezdan Petkovic) Date: Fri, 19 Mar 2010 14:09:28 -0400 Subject: [Python-Dev] Tracker reviews workflow and flags In-Reply-To: References: <20100319142205.E11FC1FDA39@kimball.webabinitio.net> Message-ID: <46EF5EEA-CA98-44C5-92AD-0BC83E39090D@zope.com> On Mar 19, 2010, at 1:15 PM, Terry Reedy wrote: > On 3/19/2010 10:22 AM, R. David Murray wrote: > >> This can be done by anyone just by saying, eg: 'see issue 1234' (roundup >> turns that into a link), > > That should be 'see issue #1234' to get the autolink. > From http://wiki.python.org/moin/TrackerDocs/ The list in the docs is not complete then. David is right. The words 'issue 1234' will become a link in roundup automatically. As well as 'issue1234'. Zvezdan From raymond.hettinger at gmail.com Fri Mar 19 19:42:42 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 19 Mar 2010 11:42:42 -0700 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> Message-ID: <7A59AEF6-A4F9-4D81-BE9A-EAD89053AD7F@gmail.com> On Mar 19, 2010, at 11:11 AM, Antoine Pitrou wrote: > Raymond Hettinger gmail.com> writes: >> >> The reason to prefer an exception is that decimal/float comparisons >> are more likely to be a programmer error than an intended behavior. > > Not more so than float/int or decimal/int or bool/int comparisons, which all > work. The float/int and decimal/int comparisons have valid and common use cases. In contrast, comparing binary floats and decimal floats is rare, and more likely to be an accident. When an int is converted to a float or decimal, the result usually isn't surprising. The conversion of a float to a decimal is not as straight-forward (most people don't even know that an exact conversion is possible). Also, the float and decimal constructors also natively accept ints as inputs, but the decimal constructor intentionally excludes a float input. Raymond From v+python at g.nevcal.com Fri Mar 19 19:42:39 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 19 Mar 2010 11:42:39 -0700 Subject: [Python-Dev] Decimal & lt; -& gt; float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> Message-ID: <4BA3C59F.7030508@g.nevcal.com> On 3/19/2010 4:50 AM, Antoine Pitrou wrote: > Glenn Linderman nevcal.com> writes: > >> On the other hand, if the default behavior is to do an implicit >> conversion, I don't know of any way that that could be turned into an >> exception for those coders that don't want or don't like the particular >> type of implicit conversion chosen. >> > You still haven't given a good reason why we should raise an exception rather > than do a comparison, though. > > The fact that some particular coders don't like "the particular type of implicit > conversion chosen" is a particularly weak argument. Python isn't a language > construction framework; we try to choose useful defaults rather than simply give > out a box of tools. If some people don't like the defaults (significant > indentation, limitless precision integers, etc.), there are other choices out > there. > The whole point of providing Decimal is for applications for which float is inappropriate. I didn't think I needed to reproduce PEP 327 in my email. So when a coder choose to use Decimal, it is because float is inappropriate. Because float is inappropriate, mixing Decimal and float is inappropriate. Having the language coerce implicitly, is inappropriate. All this is in the PEP. Note that the PEP actually says that the problem is not doing implicit arithmetic (as some have reported in this thread) but in doing implicit coercions. In order to do implicit comparisons, one must do an implicit coercion. Hence the PEP actually already prohibits implicit comparisons, as well as implicit arithmetic. Now the proposal is to start down the slippery slope by allowing comparisons. To start with, neither decimal nor float comparisons, are, in general, exact. (Although it is more common for decimal calculations to contain the code to do rounding and significant digit calculations than float code, due to its use in monetary calculations. People that don't care about the precision tend to just use float, and no significant digit concerns are coded.) Comparisons need to be done with full knowledge of the precision of the numbers. The additional information necessary to do so cannot be encoded in a binary operator. People that do care about precision would rather not have imprecise data introduced by accidentally forgetting to include a coercion constructor, and have it implicitly converted; hence, the exception is much better for them. My original message pointed out that providing an exception solves the problem for people that care about precision, and doesn't hinder a solution (only a convenience) for people that somehow bothered to use Decimal but then suddenly lost interest in doing correct math. For those people, the class I proposed can workaround the existence of the exception, so the language can better serve both types of people, those that care, and those that don't. My personal opinion is that real applications that use Decimal are much more likely to care, and much more likely to appreciate the exception, whereas applications that don't care, aren't likely to use Decimal in the first place, and so won't encounter the problem anyway. And if they do, for some reason, hop on the Decimal bandwagon, then it seems to be a simple matter to implement a class with implicit conversions using whatever parameters they choose to implement, for their sloppy endeavors. Yes, if implicit coercions for comparisons exist, it is possible to write code that avoids using them... but much friendlier for people that wish to avoid them, if the exception remains in place, unless explicitly sidestepped through coding another class. Implementing an exception for Decimal/float comparison attempts, rather than comparing their types, is a giant step forward. Implementing an implicit coercion for Decimal/float comparisons is a step down a slippery slope, to reduce the benefits of Decimal for its users. Glenn From tjreedy at udel.edu Fri Mar 19 19:43:01 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 19 Mar 2010 14:43:01 -0400 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> Message-ID: On 3/19/2010 2:11 PM, Antoine Pitrou wrote: > Raymond Hettinger gmail.com> writes: >> >> The reason to prefer an exception is that decimal/float comparisons >> are more likely to be a programmer error than an intended behavior. If you really believe that, then equality comparisons should also be disabled by raising NotImplemented or whatever. Clearly, someone who writes 'if somefloat == somedecimal:'assumes (now wrongly) that the test might be true. This is just as buggy as an order comparison. Raising an exception would consistently isolate decimals from other numbers and eliminate the equality intransitivity mess and its nasty effect on sets. It still strikes me as a bit crazy for Python to say that 0.0 == 0 and 0 == Decimal(0) but that 0.0 != Decimal(0). Who would expect such a thing? Terry Jan Reedy From raymond.hettinger at gmail.com Fri Mar 19 19:47:45 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 19 Mar 2010 11:47:45 -0700 Subject: [Python-Dev] Decimal & lt; -& gt; float comparisons in py3k. In-Reply-To: <4BA3C59F.7030508@g.nevcal.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4BA3C59F.7030508@g.nevcal.com> Message-ID: <3909DF13-B2FA-43E7-9B15-22326271BA86@gmail.com> On Mar 19, 2010, at 11:42 AM, Glenn Linderman wrote: >> > > The whole point of providing Decimal is for applications for which float is inappropriate. I didn't think I needed to reproduce PEP 327 in my email. > > So when a coder choose to use Decimal, it is because float is inappropriate. Because float is inappropriate, mixing Decimal and float is inappropriate. Having the language coerce implicitly, is inappropriate. All this is in the PEP. Note that the PEP actually says that the problem is not doing implicit arithmetic (as some have reported in this thread) but in doing implicit coercions. In order to do implicit comparisons, one must do an implicit coercion. Hence the PEP actually already prohibits implicit comparisons, as well as implicit arithmetic. Well said. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Mar 19 19:53:26 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 19 Mar 2010 18:53:26 +0000 (UTC) Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4BA3C59F.7030508@g.nevcal.com> Message-ID: Glenn Linderman g.nevcal.com> writes: > > So when a coder choose to use Decimal, it is because float is > inappropriate. Because float is inappropriate, mixing Decimal and float > is inappropriate. Having the language coerce implicitly, is > inappropriate. I'm sorry but this is very dogmatic. What is the concrete argument against an accurate comparison between floats and decimals? > Comparisons need to be done with > full knowledge of the precision of the numbers. The additional > information necessary to do so cannot be encoded in a binary operator. This doesn't have anything to do with the mixing of floats and decimals, though, since it also applies to unmixed comparisons. Again, is there an argument specific to mixed comparisons? From solipsis at pitrou.net Fri Mar 19 20:02:35 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 19 Mar 2010 19:02:35 +0000 (UTC) Subject: [Python-Dev] =?utf-8?b?RGVjaW1hbCAmYW1wO2FtcDsgYW1wOyBsdDsgLSZh?= =?utf-8?q?mp=3Bamp=3B_amp=3B_gt=3B=09float_comparisons_in_py3k=2E?= References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <7A59AEF6-A4F9-4D81-BE9A-EAD89053AD7F@gmail.com> Message-ID: Raymond Hettinger gmail.com> writes: > > The conversion of a float to a decimal is not as straight-forward > (most people don't even know that an exact conversion is possible). I still don't follow you. You are saying that an exact conversion is possible, but don't want it to be done because it is "not as straight-forward"? From v+python at g.nevcal.com Fri Mar 19 20:16:35 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 19 Mar 2010 12:16:35 -0700 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> Message-ID: <4BA3CD93.8080502@g.nevcal.com> On 3/19/2010 11:43 AM, Terry Reedy wrote: > On 3/19/2010 2:11 PM, Antoine Pitrou wrote: >> Raymond Hettinger gmail.com> writes: >>> >>> The reason to prefer an exception is that decimal/float comparisons >>> are more likely to be a programmer error than an intended behavior. > > If you really believe that, then equality comparisons should also be > disabled by raising NotImplemented or whatever. Totally agree. While the example most used in this thread is a less than operator, the text of the thread has never (as far as I have read) distinguished between equality, inequality, or relative signed magnitude comparisons. Sort has also been mentioned explicitly as an example. I perceive that the whole thread is about _all_ comparison operators with one float and one decimal operand currently producing an exception (3.x) or a type-based ordering (2.x). The type-based ordering has been demonstrated to produce unstable sorts in the presence of other types also, so is more of a problem than first perceived, and should be changed. > Clearly, someone who writes 'if somefloat == somedecimal:'assumes (now > wrongly) that the test might be true. This is just as buggy as an > order comparison. Raising an exception would consistently isolate > decimals from other numbers and eliminate the equality intransitivity > mess and its nasty effect on sets. Totally agree. > It still strikes me as a bit crazy for Python to say that 0.0 == 0 and > 0 == Decimal(0) but that 0.0 != Decimal(0). Who would expect such a > thing? The same person that would expect both 0 == "0" 0.0 == "0.0" to be False... i.e. anyone that hasn't coded in Perl for too many years. Glenn From chambon.pascal at wanadoo.fr Fri Mar 19 19:58:42 2010 From: chambon.pascal at wanadoo.fr (Pascal Chambon) Date: Fri, 19 Mar 2010 19:58:42 +0100 Subject: [Python-Dev] Attribute lookup ambiguity Message-ID: <4BA3C962.7080500@wanadoo.fr> Hello I've already crossed a bunch of articles detailing python's attribute lookup semantic (__dict__, descriptors, order of base class traversing...), but I have never seen, so far, an explanation of WHICH method did waht, exactly. I assumed that getattr(a, b) was the same as a.__getattribute__(b), and that this __getattribute__ method (or the hidden routine replacing it when we don't override it in our class) was in charge of doing the whole job of traversing the object tree, checking descriptors, binding methods, calling __getattr__ on failure etc. However, the test case below shows that __getattribute__ does NOT call __getattr__ on failure. So it seems it's an upper levl machinery, in getattr(), which is in chrge of that last action. Is that on purpose ? Considering that __getattribute__ (at lest, object.__getattribute__) does 90% of the hard job, why are these 10% left ? Can we find somewhere the details of "who must do what" when customizing attribute access ? Shouldn't we inform people about the fact that __getattribute__ isn't sufficient in itself to lookup an attribute ? Thanks for the attention, regards, Pascal ======= INPUT ======= class A(object): def __getattribute__(self, name): print "A getattribute", name return object.__getattribute__(self, name) def __getattr__(self, name): print "A getattr", name return "hello A" class B(A): def __getattribute__(self, name): print "B getattribute", name return A.__getattribute__(self, name) def __getattr__(self, name): print "B getattr", name return "hello B" print A().obj print "---" print B().obj print "---" print getattr(B(), "obj") print "-----" print object.__getattribute__(B(), "obj") # DOES NOT CALL __getattr__() !!! =========== OUTPUT =========== A getattribute obj A getattr obj hello A --- B getattribute obj A getattribute obj B getattr obj hello B --- B getattribute obj A getattribute obj B getattr obj hello B ----- Traceback (most recent call last): File "C:\Users\Pakal\Desktop\test_object_model.py", line 34, in print object.__getattribute__(B(), "obj") # DOES NOT CALL __getattr__() !!!??? AttributeError: 'B' object has no attribute 'obj' From brett at python.org Fri Mar 19 20:27:46 2010 From: brett at python.org (Brett Cannon) Date: Fri, 19 Mar 2010 12:27:46 -0700 Subject: [Python-Dev] 3.1.2 tagging delayed a little In-Reply-To: <1afaf6161003182020l1dd73856k30942c590ebcfb8f@mail.gmail.com> References: <1afaf6161003182020l1dd73856k30942c590ebcfb8f@mail.gmail.com> Message-ID: On Thu, Mar 18, 2010 at 20:20, Benjamin Peterson wrote: > Pending the resolution of a few Mac OS issues (#8068, #8069, and > #8133), I'm not going to tag the release at the moment. Hopefully, > we'll still be able to have a release in the next week. Do you want issue8133 to be a release blocker? If you do I can make sure it passes by the end of this weekend. -Brett > > -- > Regards, > Benjamin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From rdmurray at bitdance.com Fri Mar 19 20:35:33 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 19 Mar 2010 15:35:33 -0400 Subject: [Python-Dev] Tracker reviews workflow and flags In-Reply-To: <46EF5EEA-CA98-44C5-92AD-0BC83E39090D@zope.com> References: <20100319142205.E11FC1FDA39@kimball.webabinitio.net> <46EF5EEA-CA98-44C5-92AD-0BC83E39090D@zope.com> Message-ID: <20100319193533.E63481FDAFD@kimball.webabinitio.net> On Fri, 19 Mar 2010 14:09:28 -0400, Zvezdan Petkovic wrote: > On Mar 19, 2010, at 1:15 PM, Terry Reedy wrote: > > On 3/19/2010 10:22 AM, R. David Murray wrote: > >> This can be done by anyone just by saying, eg: 'see issue 1234' (roundup > >> turns that into a link), > > > > That should be 'see issue #1234' to get the autolink. > > From http://wiki.python.org/moin/TrackerDocs/ > > The list in the docs is not complete then. > David is right. The words 'issue 1234' will become a link in roundup automatically. As well as 'issue1234'. I've updated the wiki page. -- R. David Murray www.bitdance.com From rdmurray at bitdance.com Fri Mar 19 20:46:35 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 19 Mar 2010 15:46:35 -0400 Subject: [Python-Dev] Tracker reviews workflow and flags In-Reply-To: <20100319163958.GB8716@amk-desktop.matrixgroup.net> References: <20100319142205.E11FC1FDA39@kimball.webabinitio.net> <20100319163958.GB8716@amk-desktop.matrixgroup.net> Message-ID: <20100319194635.3DE491FDAE7@kimball.webabinitio.net> On Fri, 19 Mar 2010 12:39:58 -0400, "A.M. Kuchling" wrote: > On Fri, Mar 19, 2010 at 10:22:05AM -0400, R. David Murray wrote: > > > Real world example with issue8151. It is an issue with a trivial patch > > > in it. Everything what is needed is to dispatch it to stable `commit > > > queue` and port to trunk. It is not 'easy' - it is 'trivial', but I > > > have no means to mark it as 'easy' either, so even this trivial fix > > > lies in tracker for three days waiting for review. > > > > Three days? That's nothing :) > > I think Anatoly has a point, though; there are issues where the people > discussing the fix have come to a consensus, but none of them are > committers, or the committers involved are too busy. There's a patch > ready to apply, but no committer knows about it. Yes, this is a general problem with the workflow. This is one reason for the creation of the 'committers.rst' file in py3k: submitters can find committers there to add to the nosy list of issues that are ready for commit, so that they do notice. However, as you note, sometimes 'ready' bugs sit in the tracker for a while even when one or more committers is already nosy. As someone else said, that's the nature of a volunteer driven process. The best solution as far as I know is to expand the pool of volunteers, so I hope Anatoloy is inspired to continue to contribute so he can eventually become a committer! > > To avoid issues involving C coding, try restricting your search to > > 'Library' issues. There is C code in some library modules, but less so > > than in the core ;). It would be nice if you could search on more than > > Actually Library should pretty much always be pure-Python. Issues > related to C modules such as socket are classified under the > 'Extension modules' component. Of course, sometimes the ultimate > cause will migrate (a bug in SocketServer.py is traced to a flaw in > socket, for example). Good point. Or at least, that's the theory. In practice bugs sometimes get miscategorized. So, Anatoly, if you find bugs like that you can ask that they be properly recategorized. -- R. David Murray www.bitdance.com From dickinsm at gmail.com Fri Mar 19 20:50:38 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Fri, 19 Mar 2010 19:50:38 +0000 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> Message-ID: <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com> On Fri, Mar 19, 2010 at 6:43 PM, Terry Reedy wrote: > On 3/19/2010 2:11 PM, Antoine Pitrou wrote: >> >> Raymond Hettinger ?gmail.com> ?writes: >>> >>> The reason to prefer an exception is that decimal/float comparisons >>> are more likely to be a programmer error than an intended behavior. > > If you really believe that, then equality comparisons should also be > disabled by raising NotImplemented or whatever. Hah. This is a very good point, and one I'd somehow missed up until now. I don't think we *can* reasonably make equality comparisons raise NotImplemented (in either 2.x or 3.x), since that messes up containment tests: something like "1.0 in {2, Decimal(3)}" would raise a TypeError instead of correctly returning False. (So the patch I've put on the issue tracker is wrong, since it does raise TypeError for equality and inequality, as well as for <, >, <= and >=.) > Clearly, someone who writes > 'if somefloat == somedecimal:'assumes (now wrongly) that the test might be > true. This is just as buggy as an order comparison. Raising an exception > would consistently isolate decimals from other numbers and eliminate the > equality intransitivity mess and its nasty effect on sets. > > It still strikes me as a bit crazy for Python to say that 0.0 == 0 and 0 == > Decimal(0) but that 0.0 != Decimal(0). Who would expect such a thing? Agreed. A solution to the original problem that still has 0.0 == Decimal(0) evaluating to False isn't much of a solution. This puts me firmly in the 'make numeric comparisons give the right answer' camp. Thanks, Mark From dickinsm at gmail.com Fri Mar 19 20:55:58 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Fri, 19 Mar 2010 19:55:58 +0000 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com> Message-ID: <5c6f2a5d1003191255u14d74227j583ec140fea534ab@mail.gmail.com> On Fri, Mar 19, 2010 at 7:50 PM, Mark Dickinson wrote: > So the patch > I've put on the issue tracker is wrong, since it does raise TypeError ... s/I've put/I have yet to put/ I really shouldn't admit to errors in things that I haven't even been made public yet. :) Mark From v+python at g.nevcal.com Fri Mar 19 21:05:32 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 19 Mar 2010 13:05:32 -0700 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com> Message-ID: <4BA3D90C.5060009@g.nevcal.com> On 3/19/2010 12:50 PM, Mark Dickinson wrote: > Hah. This is a very good point, and one I'd somehow missed up until > now. I don't think we*can* reasonably make equality comparisons > raise NotImplemented (in either 2.x or 3.x), since that messes up > containment tests: something like "1.0 in {2, Decimal(3)}" would > raise a TypeError instead of correctly returning False. (So the patch > I've put on the issue tracker is wrong, since it does raise TypeError > for equality and inequality, as well as for<,>,<= and>=.) > Sounds to me like containment checking is wrong; that if it gets an exception during the comparison that it should assume unequal, rather than aborting, and continue to the next entry. Wouldn't the same issue arise for containment tests when disparate, incomparable objects are included in the same set? Why is this different? -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Fri Mar 19 21:23:43 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 19 Mar 2010 20:23:43 +0000 Subject: [Python-Dev] Attribute lookup ambiguity In-Reply-To: <4BA3C962.7080500@wanadoo.fr> References: <4BA3C962.7080500@wanadoo.fr> Message-ID: <4BA3DD4F.1080209@voidspace.org.uk> On 19/03/2010 18:58, Pascal Chambon wrote: > Hello > > I've already crossed a bunch of articles detailing python's attribute > lookup semantic (__dict__, descriptors, order of base class > traversing...), but I have never seen, so far, an explanation of WHICH > method did waht, exactly. > > I assumed that getattr(a, b) was the same as a.__getattribute__(b), > and that this __getattribute__ method (or the hidden routine replacing > it when we don't override it in our class) was in charge of doing the > whole job of traversing the object tree, checking descriptors, binding > methods, calling __getattr__ on failure etc. > > However, the test case below shows that __getattribute__ does NOT call > __getattr__ on failure. So it seems it's an upper levl machinery, in > getattr(), which is in chrge of that last action. Python 3 has the behavior you are asking for. It would be a backwards incompatible change to do it in Python 2 as __getattribute__ *not* calling __getattr__ is the documented behaviour. Python 3.2a0 (py3k:78770, Mar 7 2010, 20:32:50) [GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin >>> class x: ... def __getattribute__(s, name): ... print ('__getattribute__', name) ... raise AttributeError ... def __getattr__(s, name): ... print ('__getattr__', name) ... >>> a = x() >>> a.b __getattribute__ b __getattr__ b This list is not really an appropriate place to ask questions like this though, comp.lang.python would be better. All the best, Michael Fooord > > Is that on purpose ? Considering that __getattribute__ (at lest, > object.__getattribute__) does 90% of the hard job, why are these 10% > left ? > Can we find somewhere the details of "who must do what" when > customizing attribute access ? > Shouldn't we inform people about the fact that __getattribute__ isn't > sufficient in itself to lookup an attribute ? > > Thanks for the attention, > regards, > Pascal > > > > ======= > INPUT > ======= > > class A(object): > > def __getattribute__(self, name): > print "A getattribute", name > return object.__getattribute__(self, name) > > def __getattr__(self, name): > print "A getattr", name > return "hello A" > > > class B(A): > > > def __getattribute__(self, name): > print "B getattribute", name > return A.__getattribute__(self, name) > > def __getattr__(self, name): > print "B getattr", name > return "hello B" > > print A().obj > print "---" > print B().obj > print "---" > print getattr(B(), "obj") > print "-----" > print object.__getattribute__(B(), "obj") # DOES NOT CALL > __getattr__() !!! > > > =========== > OUTPUT > =========== > > A getattribute obj > A getattr obj > hello A > --- > B getattribute obj > A getattribute obj > B getattr obj > hello B > --- > B getattribute obj > A getattribute obj > B getattr obj > hello B > ----- > Traceback (most recent call last): > File "C:\Users\Pakal\Desktop\test_object_model.py", line 34, in > print object.__getattribute__(B(), "obj") # DOES NOT CALL > __getattr__() !!!??? > AttributeError: 'B' object has no attribute 'obj' > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From rdmurray at bitdance.com Fri Mar 19 21:33:56 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 19 Mar 2010 16:33:56 -0400 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <4BA3CD93.8080502@g.nevcal.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <4BA3CD93.8080502@g.nevcal.com> Message-ID: <20100319203356.DED911FDAE6@kimball.webabinitio.net> On Fri, 19 Mar 2010 12:16:35 -0700, Glenn Linderman wrote: > On 3/19/2010 11:43 AM, Terry Reedy wrote: > > It still strikes me as a bit crazy for Python to say that 0.0 == 0 and > > 0 == Decimal(0) but that 0.0 != Decimal(0). Who would expect such a > > thing? > > The same person that would expect both > > 0 == "0" > 0.0 == "0.0" > > to be False... i.e. anyone that hasn't coded in Perl for too many years. No, those two situations are not comparable. "if a equals b and b equals c, then a equals c" is what most people will expect. In fact, only mathematicians are likely to know that any other situation could possibly be valid. Programmers are generally going to view it as a bug. -- R. David Murray www.bitdance.com From benjamin at python.org Fri Mar 19 21:56:44 2010 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 19 Mar 2010 15:56:44 -0500 Subject: [Python-Dev] 3.1.2 tagging delayed a little In-Reply-To: References: <1afaf6161003182020l1dd73856k30942c590ebcfb8f@mail.gmail.com> Message-ID: <1afaf6161003191356l517eda26yd978a986ea6d4dc6@mail.gmail.com> 2010/3/19 Brett Cannon : > On Thu, Mar 18, 2010 at 20:20, Benjamin Peterson wrote: >> Pending the resolution of a few Mac OS issues (#8068, #8069, and >> #8133), I'm not going to tag the release at the moment. Hopefully, >> we'll still be able to have a release in the next week. > > Do you want issue8133 to be a release blocker? If you do I can make > sure it passes by the end of this weekend. That would be wonderful. -- Regards, Benjamin From guido at python.org Fri Mar 19 22:06:59 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 19 Mar 2010 14:06:59 -0700 Subject: [Python-Dev] Decimal & amp; amp; lt; -& amp; amp; gt; float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <7A59AEF6-A4F9-4D81-BE9A-EAD89053AD7F@gmail.com> Message-ID: Antoine, I think your email client turns replies whose subject contains '&' into new subjects containing sequences like this: &amp; amp; lt; -&amp; amp; gt; There's a number of new threads with ever-more occurrences of "amp" in the subject, and you are the first poster in each thread (and the first post's subject already starts with "Re:"). -- --Guido van Rossum (python.org/~guido) From aditya at wefoundland.com Fri Mar 19 22:10:53 2010 From: aditya at wefoundland.com (aditya bhargava) Date: Fri, 19 Mar 2010 16:10:53 -0500 Subject: [Python-Dev] "expected a character buffer object" messages Message-ID: <4d91314c1003191410r4fe5e9adv264a90d2a68abb18@mail.gmail.com> I found an old thread on Python-Dev that suggested replacing the error "TypeError: expected a character buffer object" with something more helpful for non-experts like "expected a string or character buffer". Here's the thread: http://mail.python.org/pipermail/python-dev/2006-May/065301.html I think this is an excellent idea, but besides this ticket: http://bugs.python.org/issue6780 I haven't seen anything further about it. Does anyone know the status on this? It would be nice to have a more descriptive error. Aditya -- wefoundland.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Mar 19 22:48:04 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 19 Mar 2010 21:48:04 +0000 (UTC) Subject: [Python-Dev] Decimal [...] float comparisons in py3k. References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <7A59AEF6-A4F9-4D81-BE9A-EAD89053AD7F@gmail.com> Message-ID: Guido van Rossum python.org> writes: > > Antoine, I think your email client turns replies whose subject > contains '&' into new subjects containing sequences like this: > > &amp; amp; lt; -&amp; amp; gt; > > There's a number of new threads with ever-more occurrences of "amp" in > the subject, and you are the first poster in each thread (and the > first post's subject already starts with "Re:"). Hmm, indeed. It's the gmane Web posting interface, actually. Regards Antoine. From guido at python.org Fri Mar 19 22:50:04 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 19 Mar 2010 14:50:04 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot Message-ID: I'd like to reboot this thread. I've been spinning this topic in my head for most of the morning, and I think we should seriously reconsider allowing mixed arithmetic involving Decimal, not just mixed comparisons. [Quick summary: embed Decimal in the numeric tower but add a context flag to disallow implicit mixing of float and Decimal.] I tried to find the argumentation against it in PEP 327 (Decimal Data Type) and found that it didn't make much of an argument against mixed arithmetic beyond "it's not needed" and "it's not urgent". (It even states that initially Decimal.from_float() was omitted for simplicity -- but it got added later.) We now also have PEP 3141 (A Type Hierarchy for Numbers) which proposes a numeric tower. It has an explicit exclusion for Decimal, but that exclusion is provisional: "After consultation with its authors it has been decided that the ``Decimal`` type should not at this time be made part of the numeric tower." That was a compromise because at the time some contributors to Decimal were fiercely opposed to including Decimal into the numeric tower, and I didn't want to have a endless discussion at the time (there were many more pressing issues to be resolved). However now the subject is coming up again, and my gut keeps telling me that Decimal ought to be properly embedded in Python's numeric tower. Decimal is already *touching* the numeric tower by allowing mixed arithmetic with ints. This causes the anomaly that Mark mentioned earlier: the three values 1, 1.0 and Decimal(1) do not satisfy the rule "if x == y and y == z then it follows that x == z". We have 1.0 == 1 == Decimal(1) but 1 == 1.0 != Decimal(1). This also causes problems with hashing, where {Decimal(1), 1, 1.0} != {Decimal(1), 1.0, 1}. I'd like to look at the issue by comparing the benefits and drawbacks of properly embedding Decimal into the numeric tower. As advantages, I see consistent behavior in situations like the above and more intuitive behavior for beginners. Also, this would be a possible road towards eventually supporting a language extension where floating point literals produce Decimal values instead of binary floats. (A possible syntax could be "from __options__ import decimal_float", which would work similar to "from __future__ import ..." except it's a permanent part of the language rather than a forward compatibility feature.) As a downside, there is the worry that inadvertent mixing of Decimal and float can compromise the correctness of programs in a way that is hard to detect. But the anomalies above indicate that not fixing the situation can *also* compromise correctness in a similar way. Maybe a way out would be to add a new flag to the decimal Context class indicating whether to disallow mixing Decimal and float -- that way programs that care about this can force the issue, while the default behavior can be more intuitive. Possibly the flag should not affect comparisons. There is one choice which I'm not sure about. Should a mixed float/Decimal operation return a float or a Decimal? I note that Fraction (which *is* properly embedded in the numeric tower) supports this and returns a float result in this case. While I earlier proposed to return the most "complicated" type of the two, i.e. Decimal, I now think it may also make sense to return a float, being the most "fuzzy" type in the numeric tower. This would also make checking for accidental floats easier, since floats now propagate throughout the computation (like NaN) and a simple assertion that the result is a Decimal instance suffices to check that no floats were implicitly mixed into the computation. The implementation of __hash__ will be complicated, and it may make sense to tweak the hash function of float, Fraction and Decimal to make it easier to ensure that for values that can be represented in either type the hash matches the equality. But this sounds a worthwhile price to pay for proper embedding in the numeric tower. -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Fri Mar 19 23:02:00 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 19 Mar 2010 22:02:00 +0000 (UTC) Subject: [Python-Dev] containment checking References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com> <4BA3D90C.5060009@g.nevcal.com> Message-ID: Glenn Linderman g.nevcal.com> writes: > > Sounds to me like containment checking is wrong; that if it gets an > exception during the comparison that it should assume unequal, rather > than aborting, and continue to the next entry. Well as the Zen says: Errors should never pass silently. Unless explicitly silenced. If there's a bug in your __eq__ method, rather than getting wrong containment results, you get the proper exception. > Wouldn't the same issue arise for containment tests when disparate, > incomparable objects are included in the same set?? Why is this > different? That's why equality testing almost never fails between standard types. We do have an (IMO unfortunate) exception in the datetime module, though: >>> from datetime import tzinfo, timedelta, datetime >>> ZERO = timedelta(0) >>> class UTC(tzinfo): # UTC class ripped off from the official doc ... """UTC""" ... def utcoffset(self, dt): ... return ZERO ... def tzname(self, dt): ... return "UTC" ... def dst(self, dt): ... return ZERO ... >>> utc = UTC() >>> a = datetime.now() >>> b = datetime.now(utc) >>> a == b Traceback (most recent call last): File "", line 1, in TypeError: can't compare offset-naive and offset-aware datetimes Regards Antoine. From raymond.hettinger at gmail.com Fri Mar 19 23:36:38 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 19 Mar 2010 15:36:38 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: Message-ID: <28258875-7132-4FA0-9926-24A6346FF943@gmail.com> On Mar 19, 2010, at 2:50 PM, Guido van Rossum wrote: > I'd like to reboot this thread. I've been spinning this topic in my > head for most of the morning, and I think we should seriously > reconsider allowing mixed arithmetic involving Decimal, not just mixed > comparisons. [Quick summary: embed Decimal in the numeric tower but > add a context flag to disallow implicit mixing of float and Decimal.] Making decimals first class citizens would sure help eliminate some special cases and anomalies. If a context flag were added, am wondering whether it should simply provide a warning rather than flat-out disallowing the transaction. The whole point is to highlight accidental mixing. If the mixed arithmetic were allowed, then the decimal constructor should be changed to match (i.e. accept float inputs instead of requiring Decimal.from_float()). Raymond From barry at python.org Fri Mar 19 23:56:55 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 19 Mar 2010 18:56:55 -0400 Subject: [Python-Dev] RELEASED Python 2.6.5 Message-ID: <20100319185655.26e1bf1a@heresy> On behalf of the Python community, I'm happy to announce the availability of Python 2.6.5 final. This is the latest production-ready version in the Python 2.6 series. Python 2.6.5 fixes dozens of issues in the core, built-in modules, libraries, and documentation since Python 2.6.4 was released back in October 2009. We highly recommend that you upgrade to Python 2.6.5. Please see the NEWS file for all the gory details. http://www.python.org/download/releases/2.6.5/NEWS.txt Source tarballs and the Windows installers can be downloaded from the Python 2.6.5 page. The Mac OS X disk image will be uploaded soon. http://www.python.org/download/releases/2.6.5/ For more information on Python 2.6 in general, please see http://docs.python.org/whatsnew/2.6.html Please report bugs for any Python version in the Python tracker. http://bugs.python.org Enjoy, -Barry Barry Warsaw barry at python.org Python 2.6 Release Manager (on behalf of the entire python-dev team) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From martin at v.loewis.de Sat Mar 20 00:00:48 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 20 Mar 2010 00:00:48 +0100 Subject: [Python-Dev] GSoC 2010 is on -- projects? In-Reply-To: <4BA388E5.1030703@gmail.com> References: <20100319023630.GC1617@idyll.org> <4BA31858.3010904@gmail.com> <4BA388E5.1030703@gmail.com> Message-ID: <4BA40220.30701@v.loewis.de> > Whether this is worth weeks of work or not will depend on a given > student's knowledge about Python 3, and I'd suspect that the GSoC would > be an opportunity for a number of applicant to actually learn the > intricacies of Python 3. > Developing Python 3-specific features could be used to increase the > amount of work on the project, but I am uncertain of whether this is > worth a full GSoC. I'd say this would definitely make a GSoC project; any non-trivial porting will be. As you get experienced with porting, it can indeed go fairly quickly. However, the learning curve is not to be underestimated: the first few changes will rather be in the "one change per week" range. Regards, Martin From tjreedy at udel.edu Sat Mar 20 00:03:59 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 19 Mar 2010 19:03:59 -0400 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <4BA3CD93.8080502@g.nevcal.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <4BA3CD93.8080502@g.nevcal.com> Message-ID: On 3/19/2010 3:16 PM, Glenn Linderman wrote: > I perceive that the whole thread is about _all_ comparison operators with > one float and one decimal operand currently producing an exception (3.x) Not true for equality comparison. That returns False regardless of value, just as order comparisons return what they do in 2.x regardless of value. I claim that the former is at least as bad as the latter for numbers. > or a type-based ordering (2.x). 3.x mixes type-based and value-based equality testing for Decimals and other numbers. Terry Jan Reedy From dickinsm at gmail.com Sat Mar 20 00:13:29 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Fri, 19 Mar 2010 23:13:29 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: Message-ID: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> Just a couple of quick side comments on this; I haven't got my head around the whole mixed-operations idea yet. On Fri, Mar 19, 2010 at 9:50 PM, Guido van Rossum wrote: > There is one choice which I'm not sure about. Should a mixed > float/Decimal operation return a float or a Decimal? I'll just say that it's much easier to return a Decimal if you want to be able to make guarantees about rounding behaviour, basically because floats can be converted losslessly to Decimals. I also like the fact that the decimal module offers more control (rounding mode, precision, flags, wider exponent range) than float. In general, the correct semantics for an arithmetic operation are to produce a result that's equivalent to what would have been obtained by performing the operation to infinite precision and then doing a single round to fit that result into the output type format. This is trivially easy to do if mixed float and Decimal operations produce Decimal, and much much harder if they produce floats; if mixed-type operations produced floats we'd probably have to go with algorithms that involve two rounds (i.e., first coerce the Decimal to a float, then do the operation as usual on the two floats), and there would likely be some (small) numeric surprises as a result. > The implementation of __hash__ will be complicated, and it may make > sense to tweak the hash function of float, Fraction and Decimal to > make it easier to ensure that for values that can be represented in > either type the hash matches the equality. But this sounds a > worthwhile price to pay for proper embedding in the numeric tower. I don't think this is going to be a problem. I've implemented most of the scheme I outlined earlier (it's working for ints, floats and Decimals; still need to implement it for Fractions and complex numbers) and it seems to work just fine, with essentially no more code than was there before. I'll post a proof-of-concept patch when I've filled in the missing bits. Mark From facundobatista at gmail.com Sat Mar 20 00:17:24 2010 From: facundobatista at gmail.com (Facundo Batista) Date: Fri, 19 Mar 2010 19:17:24 -0400 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: Message-ID: On Fri, Mar 19, 2010 at 5:50 PM, Guido van Rossum wrote: > As a downside, there is the worry that inadvertent mixing of Decimal > and float can compromise the correctness of programs in a way that is > hard to detect. But the anomalies above indicate that not fixing the Decimal already has something that we can use in this case, and fits very nice here: Signals. Signals represent conditions that arise during computation. Each corresponds to one context flag and one context trap enabler. So, if we add a signal like "MixedWithFloats", users will have a flag in the context that they could check to see if a float was mixed in the operations executed (and if the user set the trap accordingly, an exception will be raised when the signal happens). OTOH, returning a float the first time both are mixed is easy to check... but if it has downsides, and we prefer to return a Decimal in that case, note that we have a mechanism in Decimal we can use. Furthermore, in case we want to ease the transition we can do the following: - add this signal - set *by default* the trap to raise an exception when float and Decimal is mixed So, the behaviour will be the same as we have now, but users can easily change it. Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From v+python at g.nevcal.com Sat Mar 20 00:50:17 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 19 Mar 2010 16:50:17 -0700 Subject: [Python-Dev] containment checking In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com> <4BA3D90C.5060009@g.nevcal.com> Message-ID: <4BA40DB9.3020302@g.nevcal.com> On 3/19/2010 3:02 PM, Antoine Pitrou wrote: > Glenn Linderman g.nevcal.com> writes: > >> > >> > Sounds to me like containment checking is wrong; that if it gets an >> > exception during the comparison that it should assume unequal, rather >> > than aborting, and continue to the next entry. >> > Well as the Zen says: > > Errors should never pass silently. > Unless explicitly silenced. > > If there's a bug in your __eq__ method, rather than getting wrong containment > results, you get the proper exception. > If there's a bug in your __eq__ method, it may or may not raise an exception, which may or may not get you wrong containment results. But it will probably get you buggy results, somehow or another. That's what design, code reviews, and testing are for. If the your __eq__ method uses exceptions (the only available method of out-of-band signalling for binary operators; not all exceptions are errors) to declare that it can't perform the comparison and produce a boolean result, that is a case where producing an exception is not an error, so your quoted Zen doesn't apply. Glenn From solipsis at pitrou.net Sat Mar 20 00:58:05 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 19 Mar 2010 23:58:05 +0000 (UTC) Subject: [Python-Dev] containment checking References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com> <4BA3D90C.5060009@g.nevcal.com> <4BA40DB9.3020302@g.nevcal.com> Message-ID: Glenn Linderman g.nevcal.com> writes: > > If there's a bug in your __eq__ method, it may or may not raise an > exception, which may or may not get you wrong containment results. But > it will probably get you buggy results, somehow or another. That's what > design, code reviews, and testing are for. We'll have to "agree to disagree" then. If you want error silencing by default, Python is not the language you are looking for. From greg.ewing at canterbury.ac.nz Sat Mar 20 01:10:25 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 20 Mar 2010 13:10:25 +1300 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <4BA3680D.8070909@gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com> <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com> <4BA3680D.8070909@gmail.com> Message-ID: <4BA41271.3010003@canterbury.ac.nz> Nick Coghlan wrote: > Mark Dickinson wrote: > It seems to me that given the existing conflation of numeric equivalence > and containment testing, going the whole hog and fixing the set > membership problem for all of our rational types would be the right > thing to do. Isn't this only solving half the problem, though? Even if you find that the hashes are equal, you still have to decide whether the values themselves are equal. Is there some similarly clever way of comparing two rational numbers without having explicit access to the numerators and denominators? -- Greg From v+python at g.nevcal.com Sat Mar 20 01:13:54 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 19 Mar 2010 17:13:54 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: Message-ID: <4BA41342.8040603@g.nevcal.com> On 3/19/2010 2:50 PM, Guido van Rossum wrote: > I'd like to reboot this thread. I'll go along with that idea! > I've been spinning this topic in my > head for most of the morning, and I think we should seriously > reconsider allowing mixed arithmetic involving Decimal, not just mixed > comparisons. [Quick summary: embed Decimal in the numeric tower but > add a context flag to disallow implicit mixing of float and Decimal.] > As long as there is a way to avoid implicit mixing of float and Decimal, or to easily detect (preferably with an exception) the implicit mixing, then I think that solves the concerns of people trying to write numerically correct code using Decimal. And if Mark (or someone) can solve the hashing anomoly problem without huge expense, then it could be a winner. > I tried to find the argumentation against it in PEP 327 > Should Aahz be consulted, as some of the objections in PEP 327 are attributed to him, but he is pretty scarce around here these days? > Also, this would be a possible road > towards eventually supporting a language extension where floating > point literals produce Decimal values instead of binary floats. (A > possible syntax could be "from __options__ import decimal_float", > which would work similar to "from __future__ import ..." except it's a > permanent part of the language rather than a forward compatibility > feature.) > Nice touch here... rather than being forced to quote Decimal values as strings and convert from string, or use a tuple to represent the parts, both of which are warts. Not sure what context would be used, though. Glenn From glenn at nevcal.com Sat Mar 20 01:15:06 2010 From: glenn at nevcal.com (Glenn Linderman) Date: Fri, 19 Mar 2010 17:15:06 -0700 Subject: [Python-Dev] containment checking In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com> <4BA3D90C.5060009@g.nevcal.com> <4BA40DB9.3020302@g.nevcal.com> Message-ID: <4BA4138A.9090003@nevcal.com> On 3/19/2010 4:58 PM, Antoine Pitrou wrote: > Glenn Linderman g.nevcal.com> writes: > >> > >> > If there's a bug in your __eq__ method, it may or may not raise an >> > exception, which may or may not get you wrong containment results. But >> > it will probably get you buggy results, somehow or another. That's what >> > design, code reviews, and testing are for. >> > We'll have to "agree to disagree" then. If you want error silencing by default, > Python is not the language you are looking for. > We can agree to disagree, if you like. But taken to the limit, the Zen you quoted would prevent the try except clause from being used. From greg.ewing at canterbury.ac.nz Sat Mar 20 01:27:09 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 20 Mar 2010 13:27:09 +1300 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> Message-ID: <4BA4165D.2070808@canterbury.ac.nz> Antoine Pitrou wrote: > We forbid comparisons when there is a real danger or ambiguity, such as unicode > vs. bytes. There is no such danger or ambiguity when comparing a decimal with a > float. So do you think that float("0.1") and Decimal("0.1") should be equal or not, and why? -- Greg From fuzzyman at voidspace.org.uk Sat Mar 20 01:18:09 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 20 Mar 2010 00:18:09 +0000 Subject: [Python-Dev] containment checking In-Reply-To: <4BA4138A.9090003@nevcal.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com> <4BA3D90C.5060009@g.nevcal.com> <4BA40DB9.3020302@g.nevcal.com> <4BA4138A.9090003@nevcal.com> Message-ID: <4BA41441.8030404@voidspace.org.uk> On 20/03/2010 00:15, Glenn Linderman wrote: > On 3/19/2010 4:58 PM, Antoine Pitrou wrote: >> Glenn Linderman g.nevcal.com> writes: >>> > > If there's a bug in your __eq__ method, it may or may not raise an >>> > exception, which may or may not get you wrong containment results. >>> But >>> > it will probably get you buggy results, somehow or another. That's >>> what >>> > design, code reviews, and testing are for. >> We'll have to "agree to disagree" then. If you want error silencing >> by default, >> Python is not the language you are looking for. > > We can agree to disagree, if you like. But taken to the limit, the Zen > you quoted would prevent the try except clause from being used. No, that is what "unless explicitly silenced" means - you are proposing to silence them *without* an explicit try except clause. Michael > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From v+python at g.nevcal.com Sat Mar 20 01:19:28 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 19 Mar 2010 17:19:28 -0700 Subject: [Python-Dev] containment checking In-Reply-To: <4BA41441.8030404@voidspace.org.uk> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com> <4BA3D90C.5060009@g.nevcal.com> <4BA40DB9.3020302@g.nevcal.com> <4BA4138A.9090003@nevcal.com> <4BA41441.8030404@voidspace.org.uk> Message-ID: <4BA41490.1000803@g.nevcal.com> On 3/19/2010 5:18 PM, Michael Foord wrote: >>>> will probably get you buggy results, somehow or another. That's what >>>> > design, code reviews, and testing are for. >>> We'll have to "agree to disagree" then. If you want error silencing >>> by default, >>> Python is not the language you are looking for. >> >> We can agree to disagree, if you like. But taken to the limit, the >> Zen you quoted would prevent the try except clause from being used. > > No, that is what "unless explicitly silenced" means - you are > proposing to silence them *without* an explicit try except clause. > > Michael Who, me? The containment checking code would contain the try/except, I was proposing. Glenn -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Sat Mar 20 01:20:57 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 20 Mar 2010 00:20:57 +0000 Subject: [Python-Dev] containment checking In-Reply-To: <4BA41490.1000803@g.nevcal.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com> <4BA3D90C.5060009@g.nevcal.com> <4BA40DB9.3020302@g.nevcal.com> <4BA4138A.9090003@nevcal.com> <4BA41441.8030404@voidspace.org.uk> <4BA41490.1000803@g.nevcal.com> Message-ID: <4BA414E9.2000105@voidspace.org.uk> On 20/03/2010 00:19, Glenn Linderman wrote: > On 3/19/2010 5:18 PM, Michael Foord wrote: >>>>> will probably get you buggy results, somehow or another. That's what >>>>> > design, code reviews, and testing are for. >>>> We'll have to "agree to disagree" then. If you want error silencing >>>> by default, >>>> Python is not the language you are looking for. >>> >>> We can agree to disagree, if you like. But taken to the limit, the >>> Zen you quoted would prevent the try except clause from being used. >> >> No, that is what "unless explicitly silenced" means - you are >> proposing to silence them *without* an explicit try except clause. >> >> Michael > > Who, me? The containment checking code would contain the try/except, > I was proposing. Explicit by the programmer. That is what explicit means... Caught and silenced for you by Python is implicit. Michael > > Glenn > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Mar 20 01:34:39 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 20 Mar 2010 00:34:39 +0000 (UTC) Subject: [Python-Dev] 0.1 References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <4BA4165D.2070808@canterbury.ac.nz> Message-ID: Greg Ewing canterbury.ac.nz> writes: > > So do you think that float("0.1") and Decimal("0.1") should be > equal or not, and why? Given that float("0.1") is not the mathematical 0.1, while Decimal("0.1") is, I'd say no. From greg.ewing at canterbury.ac.nz Sat Mar 20 01:50:50 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 20 Mar 2010 13:50:50 +1300 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <4BA3D90C.5060009@g.nevcal.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com> <4BA3D90C.5060009@g.nevcal.com> Message-ID: <4BA41BEA.6070309@canterbury.ac.nz> Glenn Linderman wrote: > Sounds to me like containment checking is wrong; that if it gets an > exception during the comparison that it should assume unequal, rather > than aborting, and continue to the next entry. What exception would it catch, though? Catching something as generic as TypeError would be a very bad idea, I think -- there would be too much potential for it to mask bugs. It might be acceptable if there were a special subclass of TypeError for this, such as NotComparableError. What happens in Py3 here, BTW? It must have the same problem if it refuses to compare some things for equality. -- Greg From greg.ewing at canterbury.ac.nz Sat Mar 20 02:22:16 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 20 Mar 2010 14:22:16 +1300 Subject: [Python-Dev] Decimal & lt; -& gt; float comparisons in py3k. In-Reply-To: <4BA3C59F.7030508@g.nevcal.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4BA3C59F.7030508@g.nevcal.com> Message-ID: <4BA42348.4050203@canterbury.ac.nz> Glenn Linderman wrote: > In order to do > implicit comparisons, one must do an implicit coercion. Hence the PEP > actually already prohibits implicit comparisons, as well as implicit > arithmetic. Not necessarily -- you could compare them as though they had both been converted to the equivalent rational numbers, which can be done without loss of precision. -- Greg From v+python at g.nevcal.com Sat Mar 20 03:20:54 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 19 Mar 2010 19:20:54 -0700 Subject: [Python-Dev] containment checking In-Reply-To: <4BA414E9.2000105@voidspace.org.uk> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com> <4BA3D90C.5060009@g.nevcal.com> <4BA40DB9.3020302@g.nevcal.com> <4BA4138A.9090003@nevcal.com> <4BA41441.8030404@voidspace.org.uk> <4BA41490.1000803@g.nevcal.com> <4BA414E9.2000105@voidspace.org.uk> Message-ID: <4BA43106.3000600@g.nevcal.com> On 3/19/2010 5:20 PM, Michael Foord wrote: > On 20/03/2010 00:19, Glenn Linderman wrote: >> On 3/19/2010 5:18 PM, Michael Foord wrote: >>>>>> will probably get you buggy results, somehow or another. That's what >>>>>> > design, code reviews, and testing are for. >>>>> We'll have to "agree to disagree" then. If you want error >>>>> silencing by default, >>>>> Python is not the language you are looking for. >>>> >>>> We can agree to disagree, if you like. But taken to the limit, the >>>> Zen you quoted would prevent the try except clause from being used. >>> >>> No, that is what "unless explicitly silenced" means - you are >>> proposing to silence them *without* an explicit try except clause. >>> >>> Michael >> >> Who, me? The containment checking code would contain the try/except, >> I was proposing. > > Explicit by the programmer. That is what explicit means... Caught and > silenced for you by Python is implicit. Should I really believe that there are no try/except clauses in the Python source code (nor their C equivalent, if (errno == blahblahblah) ... )? I mean, I haven't read very much of the source code... but that statement makes me want to download and grep... -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Mar 20 03:40:47 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 20 Mar 2010 13:40:47 +1100 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: Message-ID: <201003201340.47883.steve@pearwood.info> On Sat, 20 Mar 2010 08:50:04 am Guido van Rossum wrote: > I'd like to reboot this thread. I've been spinning this topic in my > head for most of the morning, and I think we should seriously > reconsider allowing mixed arithmetic involving Decimal, not just > mixed comparisons. [Quick summary: embed Decimal in the numeric tower > but add a context flag to disallow implicit mixing of float and > Decimal.] The last few days, I've argued against changing the prohibition against mixed arithmetic operations. But you've inspired me to go take a look at a module I've been neglecting, fractions, and I've learned that the Fraction type already fully supports arithmetic and comparisons with floats and ints. I'm extremely impressed -- I had no idea the numeric tower in 2.6 was this advanced. (I still do most of my work with 2.5.) Decimal appears to be the odd one: >>> f = Fraction(0) >>> d = Decimal(0) >>> 0 == 0.0 == 0j == f True >>> 0 == 0.0 == 0j == f == d False Not just odd in the sense of "different", but also odd in the sense of "weird": >>> d == 0 == 0.0 == 0j == f True [...] > I'd like to look at the issue by comparing the benefits and drawbacks > of properly embedding Decimal into the numeric tower. As advantages, > I see consistent behavior in situations like the above and more > intuitive behavior for beginners. Also, this would be a possible road > towards eventually supporting a language extension where floating > point literals produce Decimal values instead of binary floats. (A > possible syntax could be "from __options__ import decimal_float", > which would work similar to "from __future__ import ..." except it's > a permanent part of the language rather than a forward compatibility > feature.) That's far more ambitious than I was willing to even imagine, but now that you've suggested it, I like it. -- Steven D'Aprano From jafo at tummy.com Sat Mar 20 03:40:18 2010 From: jafo at tummy.com (Sean Reifschneider) Date: Fri, 19 Mar 2010 20:40:18 -0600 Subject: [Python-Dev] Needs Review: 2to3 fixer for sys.exitfunc to atexit. Message-ID: <20100320024018.GA7310@tummy.com> Issue 2356 ( http://bugs.python.org/issue2356 ) looks like it has a patch candidate, but just needs review to put in place. Seems like it may be low hanging fruit critical that just needs a quick review to either apply or at least get back to the folks who are working on a patch. Thanks, Sean -- The most important thing in communication is to hear what isn't being said. -- Peter Drucker Sean Reifschneider, Member of Technical Staff tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability From benjamin at python.org Sat Mar 20 03:45:44 2010 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 19 Mar 2010 21:45:44 -0500 Subject: [Python-Dev] Needs Review: 2to3 fixer for sys.exitfunc to atexit. In-Reply-To: <20100320024018.GA7310@tummy.com> References: <20100320024018.GA7310@tummy.com> Message-ID: <1afaf6161003191945w20bfaeabvb003c67e51d7cbb5@mail.gmail.com> 2010/3/19 Sean Reifschneider : > Issue 2356 ( http://bugs.python.org/issue2356 ) looks like it has a patch > candidate, but just needs review to put in place. ?Seems like it may be > low hanging fruit critical that just needs a quick review to either apply > or at least get back to the folks who are working on a patch. "critical"? I'm not sure that a new 2to3 fixer or py3k warning is a critical issue. -- Regards, Benjamin From ctb at msu.edu Sat Mar 20 04:13:26 2010 From: ctb at msu.edu (C. Titus Brown) Date: Fri, 19 Mar 2010 20:13:26 -0700 Subject: [Python-Dev] GSoC 2010 is on -- projects? In-Reply-To: <4BA40220.30701@v.loewis.de> References: <20100319023630.GC1617@idyll.org> <4BA31858.3010904@gmail.com> <4BA388E5.1030703@gmail.com> <4BA40220.30701@v.loewis.de> Message-ID: <20100320031326.GM1623@idyll.org> On Sat, Mar 20, 2010 at 12:00:48AM +0100, "Martin v. L?wis" wrote: > > Whether this is worth weeks of work or not will depend on a given > > student's knowledge about Python 3, and I'd suspect that the GSoC would > > be an opportunity for a number of applicant to actually learn the > > intricacies of Python 3. > > Developing Python 3-specific features could be used to increase the > > amount of work on the project, but I am uncertain of whether this is > > worth a full GSoC. > > I'd say this would definitely make a GSoC project; any non-trivial > porting will be. Sounds good to me. Line up a good student and bob's your uncle. --titus -- C. Titus Brown, ctb at msu.edu From alexander.belopolsky at gmail.com Sat Mar 20 04:20:52 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 19 Mar 2010 23:20:52 -0400 Subject: [Python-Dev] Tracker reviews workflow and flags In-Reply-To: <20100319194635.3DE491FDAE7@kimball.webabinitio.net> References: <20100319142205.E11FC1FDA39@kimball.webabinitio.net> <20100319163958.GB8716@amk-desktop.matrixgroup.net> <20100319194635.3DE491FDAE7@kimball.webabinitio.net> Message-ID: I am generally happy with the tracker workflow and in my experience, issues that are lingering are lingering for a good reason. I've decided to add my two cents to this thread because I've just had a negative experience with how issue 8154 and my input to it were handled. I will not go into details here beyond referring to http://bugs.python.org/issue8154, but if you follow the link, you'll see that there was not a consensus on how the issue should be addressed and even whether or not it was a bug. Nevertheless the patch was committed both to the trunk and to 2.6 without any answer to my concerns and without even an rNNN link to the committed revision. I think it would be nice if committers would not cut the discussion short without at least a note explaining their decision. From ncoghlan at gmail.com Sat Mar 20 05:20:49 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 20 Mar 2010 14:20:49 +1000 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <4BA3CD93.8080502@g.nevcal.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <4BA3CD93.8080502@g.nevcal.com> Message-ID: <4BA44D21.1070401@gmail.com> Glenn Linderman wrote: > The same person that would expect both > > 0 == "0" > 0.0 == "0.0" > > to be False... i.e. anyone that hasn't coded in Perl for too many years. Completely different - that is comparing numbers to strings. What this discussion is about is comparison between the different kinds of real number in the interpreter core and standard library (ints, floats, decimal.Decimal, fractions.Fraction). For arithmetic, decimal.Decimal deliberately fails to interoperate with binary floating point numbers (hence why it isn't registered as an example of the numbers.Real ABC). For comparison, it only sort of fails to interoperate - it gives either an exception or silently returns the wrong answer, depending on Python version and the exact operator involved. Fractions are nowhere near as purist about things - they will happily allow themselves to be implicitly converted to floating point values. So currently (for arithmetic): int op int -> int (or potentially float due to division) int op float, float op int -> float int op Fraction, Fraction op int -> Fraction int op Decimal, Decimal op int -> Decimal Fraction op Fraction -> Fraction Fraction op float, float op Fraction -> float Fraction op Decimal, Decimal op Fraction -> TypeError Decimal op Decimal -> Decimal Decimal op float, float op Decimal -> TypeError float op float -> float Nobody is arguing in this thread for any changes to that. I just want to contrast it with the situation for the comparison operators: int op int -> bool int op float, float op int -> bool int op Fraction, Fraction op int -> bool int op Decimal, Decimal op int -> bool Fraction op Fraction -> bool Fraction op float, float op Fraction -> bool Fraction op Decimal, Decimal op Fraction -> TypeError Decimal op Decimal -> bool Decimal op float, float op Decimal -> TypeError float op float -> bool In the case of floats and Decimals, there's no ambiguity here that creates any temptation to guess - to determine a true/false result for a comparison, floats can be converted explicitly to Decimals without any loss of accuracy. For Fractions, the precedent has already been set by allowing implicit (potentially lossy) conversion to binary floats - a lossy conversion to Decimal wouldn't be any worse. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From v+python at g.nevcal.com Sat Mar 20 07:14:24 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 19 Mar 2010 23:14:24 -0700 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <4BA44D21.1070401@gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <4BA3CD93.8080502@g.nevcal.com> <4BA44D21.1070401@gmail.com> Message-ID: <4BA467C0.1000300@g.nevcal.com> On 3/19/2010 9:20 PM, Nick Coghlan wrote: > Glenn Linderman wrote: > >> > The same person that would expect both >> > >> > 0 == "0" >> > 0.0 == "0.0" >> > >> > to be False... i.e. anyone that hasn't coded in Perl for too many years. >> > Completely different - that is comparing numbers to strings. One can argue that either way, that it is completely different, or completely the same. Is the map the territory, or is the territory the map? The human representation of the numbers in string form is meaningful numerically, and there are explicit conversions between them that prove that it is so. But it is completely different, because Python doesn't do implicit coercion of strings to numbers, so they don't belong in the tree. But it is completely the same, because Python doesn't do implicit coercion of Decimal to numbers, so they don't belong in the tree. Completely different, because Python also doesn't do numeric operations on strings either... so the analogy only goes so far, admittedly. Even Perl, which will implicitly coerce string to numbers, has very limited numeric operations that are performed on the string form. Thanks, though for the nice chart from an internals guy, you confirmed all that I thought I knew about how this presently works, from PEP and manual reading, a bit of experimentation, and the discussions here. I have no problem with Guido's latest proposal in the rebooted thread, as long as there is a way to wall off the Decimal type from creeping errors due to implicit conversions and inaccurate types. It is presently an excellent feature of Python to have a way to be sure that conversions are explicit to retain proper accuracy. Few languages have that without requiring the user to write the whole class himself (or beg, borrow, or buy it). As he implicitly points out, while declaring that mixed arithmetic might be appropriate, that it either needs to be done correctly and completely, or not at all. Glenn From dickinsm at gmail.com Sat Mar 20 12:06:30 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 20 Mar 2010 11:06:30 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <99e0b9531003190617w28c5da2etf8b35c54f223a036@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com> <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com> <99e0b9531003190617w28c5da2etf8b35c54f223a036@mail.gmail.com> Message-ID: <5c6f2a5d1003200406o7b7c8a6am839925e0f07d0222@mail.gmail.com> On Fri, Mar 19, 2010 at 1:17 PM, Case Vanhorsen wrote: > On Fri, Mar 19, 2010 at 3:07 AM, Mark Dickinson wrote: >> On Fri, Mar 19, 2010 at 9:37 AM, Mark Dickinson wrote: >>> Making hashes of int, >>> float, Decimal *and* Fraction all compatible with one another, >>> efficient for ints and floats, and not grossly inefficient for >>> Fractions and Decimals, is kinda hairy, though I have a couple of >>> ideas of how it could be done. >> >> To elaborate, here's a cute scheme for the above, which is actually >> remarkably close to what Python already does for ints and floats, and >> which doesn't require any of the numeric types to figure out whether >> it's exactly equal to an instance of some other numeric type. > Will this change the result of hashing a long? I know that both gmpy > and SAGE use their own hash implementations for performance reasons. I > understand that CPython's hash function is an implementation detail, > but there are external modules that rely on the existing hash > behavior. Yes, it would change the hash of a long. What external modules are there that rely on existing hash behaviour? And exactly what behaviour do they rely on? Mark From dickinsm at gmail.com Sat Mar 20 12:16:39 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 20 Mar 2010 11:16:39 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <4BA41271.3010003@canterbury.ac.nz> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com> <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com> <4BA3680D.8070909@gmail.com> <4BA41271.3010003@canterbury.ac.nz> Message-ID: <5c6f2a5d1003200416i59ce880fk9cec234caaac2eb0@mail.gmail.com> On Sat, Mar 20, 2010 at 12:10 AM, Greg Ewing wrote: > Nick Coghlan wrote: >> >> Mark Dickinson wrote: > >> It seems to me that given the existing conflation of numeric equivalence >> and containment testing, going the whole hog and fixing the set >> membership problem for all of our rational types would be the right >> thing to do. > > Isn't this only solving half the problem, though? Yes. > Even if > you find that the hashes are equal, you still have to decide > whether the values themselves are equal. True. The reason I was concentrating on the hashes is that it's not immediately obvious that it's even *possible* to find a decent hash function that's efficient to compute and gives equal results for numerically equal inputs (regardless of type); this is especially true if you don't want to significantly slow down the existing hashes for int and float. But once that problem is solved, it shouldn't be too hard to implement all the comparisons. It *is* kinda messy, because as far as I can see the oddities of the various types mean that you end up producing specialized code for comparing each pair of types (one block of code for float<->Fraction comparisons, another for float<->Decimal, yet another for Decimal<->Fraction, and so on), but it's doable. Mark From p.f.moore at gmail.com Sat Mar 20 12:41:36 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 20 Mar 2010 11:41:36 +0000 Subject: [Python-Dev] Fraction arithmetic (Was: Decimal ... float comparisons in py3k) Message-ID: <79990c6b1003200441k69f1899fsfd1e8b40871a95cc@mail.gmail.com> On 20 March 2010 04:20, Nick Coghlan wrote: > In the case of floats and Decimals, there's no ambiguity here that > creates any temptation to guess - to determine a true/false result for a > comparison, floats can be converted explicitly to Decimals without any > loss of accuracy. For Fractions, the precedent has already been set by > allowing implicit (potentially lossy) conversion to binary floats - a > lossy conversion to Decimal wouldn't be any worse. Hmm, given that a float can be converted losslessly to a fraction, why was the decision taken to convert the fraction to a float rather than the other way round? I don't see a PEP for the fractions module, and my google-fu has failed to find anything. Was there a discussion on this? Paul. From dickinsm at gmail.com Sat Mar 20 12:58:38 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 20 Mar 2010 11:58:38 +0000 Subject: [Python-Dev] Fraction arithmetic (Was: Decimal ... float comparisons in py3k) In-Reply-To: <79990c6b1003200441k69f1899fsfd1e8b40871a95cc@mail.gmail.com> References: <79990c6b1003200441k69f1899fsfd1e8b40871a95cc@mail.gmail.com> Message-ID: <5c6f2a5d1003200458q64c033b7u796c12b19eb04d7e@mail.gmail.com> On Sat, Mar 20, 2010 at 11:41 AM, Paul Moore wrote: > On 20 March 2010 04:20, Nick Coghlan wrote: >> In the case of floats and Decimals, there's no ambiguity here that >> creates any temptation to guess - to determine a true/false result for a >> comparison, floats can be converted explicitly to Decimals without any >> loss of accuracy. For Fractions, the precedent has already been set by >> allowing implicit (potentially lossy) conversion to binary floats - a >> lossy conversion to Decimal wouldn't be any worse. > > Hmm, given that a float can be converted losslessly to a fraction, why > was the decision taken to convert the fraction to a float rather than > the other way round? I'm not sure of the actual reason for this decision, but one argument I've seen used for other languages is that it's desirable for the inexactness of the float type to be contagious: rationals are perceived as exact, while floats are perceived as approximations. Note that this only applies to arithmetic operations: for comparisons, an exact conversion *is* performed. This is much like what currently happens with ints and floats in the core: a mixed-type arithmetic operation between an int and a float first converts the int to a float (possibly changing the value in the process). A mixed-type comparison makes an exact comparison without doing such a conversion. For example (in any non-ancient version of Python): >>> n = 2**53 + 1 >>> x = 2.**53 >>> n > x # compares exact values; no conversion performed True >>> n - x # converts n to a float before subtracting 0.0 > I don't see a PEP for the fractions module, and my google-fu has > failed to find anything. Was there a discussion on this? There's PEP 3141 (http://www.python.org/dev/peps/pep-3141/), which was the motivation for adding the fractions module in the first place, and there's the issue tracker item for the fractions module (http://bugs.python.org/issue1682). Mark From chambon.pascal at wanadoo.fr Sat Mar 20 13:00:30 2010 From: chambon.pascal at wanadoo.fr (Pascal Chambon) Date: Sat, 20 Mar 2010 13:00:30 +0100 Subject: [Python-Dev] Attribute lookup ambiguity In-Reply-To: <4BA3DD4F.1080209@voidspace.org.uk> References: <4BA3C962.7080500@wanadoo.fr> <4BA3DD4F.1080209@voidspace.org.uk> Message-ID: <4BA4B8DE.8060109@wanadoo.fr> Michael Foord a ?crit : > > On 19/03/2010 18:58, Pascal Chambon wrote: >> Hello >> >> I've already crossed a bunch of articles detailing python's attribute >> lookup semantic (__dict__, descriptors, order of base class >> traversing...), but I have never seen, so far, an explanation of >> WHICH method did waht, exactly. >> >> I assumed that getattr(a, b) was the same as a.__getattribute__(b), >> and that this __getattribute__ method (or the hidden routine >> replacing it when we don't override it in our class) was in charge of >> doing the whole job of traversing the object tree, checking >> descriptors, binding methods, calling __getattr__ on failure etc. >> >> However, the test case below shows that __getattribute__ does NOT >> call __getattr__ on failure. So it seems it's an upper levl >> machinery, in getattr(), which is in chrge of that last action. > > Python 3 has the behavior you are asking for. It would be a backwards > incompatible change to do it in Python 2 as __getattribute__ *not* > calling __getattr__ is the documented behaviour. > > Python 3.2a0 (py3k:78770, Mar 7 2010, 20:32:50) > [GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin > >>> class x: > ... def __getattribute__(s, name): > ... print ('__getattribute__', name) > ... raise AttributeError > ... def __getattr__(s, name): > ... print ('__getattr__', name) > ... > >>> a = x() > >>> a.b > __getattribute__ b > __getattr__ b I'm confused there, because the script you gave behaves the same in python 2.6. And according to the doc, it's normal, getattr() reacts to an AttributeError from __getattribute__, by calling __getattr__ : """ Python 2.6.5 documentation object.__getattribute__(/self/, /name/) Called unconditionally to implement attribute accesses for instances of the class. If the class also defines __getattr__() , the latter will not be called unless __getattribute__() either calls it explicitly or raises an AttributeError . This method should return the (computed) attribute value or raise an AttributeError exception. In order to avoid infinite recursion in this method, its implementation should always call the base class method with the same name to access any attributes it needs, for example, object.__getattribute__(self, name). """ But the point which for me is still unclear, is : does the default implementation of __getattribute__ (the one of "object") call __getattr__ by himself, or does it rely on its caller for that, by raising an AttributeError ? For Python2, it's blatantly the latter case which is favoured, but since it looks like an implementation detail at the moment, I propose we settle it (and document it) once for all. > > > This list is not really an appropriate place to ask questions like > this though, comp.lang.python would be better. > > All the best, > > Michael Fooord Sorry if I misposted, I just (wrongly ?) assumed that it was more an undecided, implementation-specific point (since the doc gave possible behaviours for __getattribute__, without precising which one was the default one), and thus targetted the hands-in-core-code audience only. Regards, Pascal -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Mar 20 16:11:12 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 20 Mar 2010 15:11:12 +0000 (UTC) Subject: [Python-Dev] Mixing float and Decimal -- thread reboot References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> Message-ID: Mark Dickinson gmail.com> writes: > > On Fri, Mar 19, 2010 at 9:50 PM, Guido van Rossum python.org> wrote: > > There is one choice which I'm not sure about. Should a mixed > > float/Decimal operation return a float or a Decimal? > > I'll just say that it's much easier to return a Decimal if you want to > be able to make guarantees about rounding behaviour, basically because > floats can be converted losslessly to Decimals. I also like the fact > that the decimal module offers more control (rounding mode, precision, > flags, wider exponent range) than float. A problem, though, is that decimals are much slower than floats. If you have a decimal creeping in some part of a calculation it could degrade performance quite a bit. Regards Antoine. From casevh at gmail.com Sat Mar 20 16:17:56 2010 From: casevh at gmail.com (Case Vanhorsen) Date: Sat, 20 Mar 2010 08:17:56 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003200406o7b7c8a6am839925e0f07d0222@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com> <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com> <99e0b9531003190617w28c5da2etf8b35c54f223a036@mail.gmail.com> <5c6f2a5d1003200406o7b7c8a6am839925e0f07d0222@mail.gmail.com> Message-ID: <99e0b9531003200817j6c47c915rf0d77f2be75d306f@mail.gmail.com> On Sat, Mar 20, 2010 at 4:06 AM, Mark Dickinson wrote: > On Fri, Mar 19, 2010 at 1:17 PM, Case Vanhorsen wrote: >> On Fri, Mar 19, 2010 at 3:07 AM, Mark Dickinson wrote: >>> On Fri, Mar 19, 2010 at 9:37 AM, Mark Dickinson wrote: >>>> Making hashes of int, >>>> float, Decimal *and* Fraction all compatible with one another, >>>> efficient for ints and floats, and not grossly inefficient for >>>> Fractions and Decimals, is kinda hairy, though I have a couple of >>>> ideas of how it could be done. >>> >>> To elaborate, here's a cute scheme for the above, which is actually >>> remarkably close to what Python already does for ints and floats, and >>> which doesn't require any of the numeric types to figure out whether >>> it's exactly equal to an instance of some other numeric type. > >> Will this change the result of hashing a long? I know that both gmpy >> and SAGE use their own hash implementations for performance reasons. I >> understand that CPython's hash function is an implementation detail, >> but there are external modules that rely on the existing hash >> behavior. > > Yes, it would change the hash of a long. > > What external modules are there that rely on existing hash behaviour? I'm only aware of gmpy and SAGE. > And exactly what behaviour do they rely on? Instead of calculating hash(long(mpz)), they calculate hash(mpz) directly. It avoids creation of a temporary object that could be quite large and is faster than the two-step process. I would need to modify the code so that it continues to produce the same result. casevh > Mark > From rdmurray at bitdance.com Sat Mar 20 16:30:08 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 20 Mar 2010 11:30:08 -0400 Subject: [Python-Dev] Tracker reviews workflow and flags In-Reply-To: References: <20100319142205.E11FC1FDA39@kimball.webabinitio.net> <20100319163958.GB8716@amk-desktop.matrixgroup.net> <20100319194635.3DE491FDAE7@kimball.webabinitio.net> Message-ID: <20100320153009.0F0C01FD9F8@kimball.webabinitio.net> On Fri, 19 Mar 2010 23:20:52 -0400, Alexander Belopolsky wrote: > I will not go into details here beyond referring to > http://bugs.python.org/issue8154, but if you follow the link, you'll > see that there was not a consensus on how the issue should be > addressed and even whether or not it was a bug. Nevertheless the > patch was committed both to the trunk and to 2.6 without any answer to > my concerns and without even an rNNN link to the committed revision. > > I think it would be nice if committers would not cut the discussion > short without at least a note explaining their decision. I think the reason this happened in this bug is that it *appeared* as though the same problem had been encountered and already fixed in python3, and all that was needed was a backport. We were groping our way toward deciding that that wasn't actually the case when Matthias committed the patch. Possibly my fault, since I said that I agreed that it should be fixed. As I've now noted on the tracker (and in the Ubuntu tracker just for good measure), I don't think that commit was appropriate, at least for 2.6, and probably not for 2.7 either. I also hope that Matthias will put in commit numbers in the future, since it is very helpful. Even better, I hope that we can automate this after the switch to Mercurial (but someone will need to write the code to do it, of course...) --David From tjreedy at udel.edu Sat Mar 20 17:29:10 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 20 Mar 2010 12:29:10 -0400 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003200406o7b7c8a6am839925e0f07d0222@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com> <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com> <99e0b9531003190617w28c5da2etf8b35c54f223a036@mail.gmail.com> <5c6f2a5d1003200406o7b7c8a6am839925e0f07d0222@mail.gmail.com> Message-ID: On 3/20/2010 7:06 AM, Mark Dickinson wrote: >> Will this change the result of hashing a long? I know that both gmpy >> and SAGE use their own hash implementations for performance reasons. I >> understand that CPython's hash function is an implementation detail, >> but there are external modules that rely on the existing hash >> behavior. > > Yes, it would change the hash of a long. > > What external modules are there that rely on existing hash behaviour? > And exactly what behaviour do they rely on? Depending on specifics of CPython's hash is a 'do at your own risk' enterprise, like CPython bytecode hacks. From solipsis at pitrou.net Sat Mar 20 17:45:04 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 20 Mar 2010 16:45:04 +0000 (UTC) Subject: [Python-Dev] Bug? syslog.openlog using ident " python" by default. References: <20100318115844.GA7374@tummy.com> Message-ID: Sean Reifschneider tummy.com> writes: > > I would propose changing the Python syslog() call to do the C equivalent of: > > if openlog_hasnt_been_called_before_now: > if sys.argv[0]: > syslog.openlog(os.path.basename(sys.argv[0])) > > In other words, if there's a script name and openlog hasn't already been > called, call openlog with the basename of the script name. This looks fine to me. The compatibility breakage doesn't look too serious, given that "python" is an inappropriate name and therefore people would have changed it anyway. By the way, why does logging have its own syslog implementation? Regards Antoine. From tjreedy at udel.edu Sat Mar 20 18:02:02 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 20 Mar 2010 13:02:02 -0400 Subject: [Python-Dev] Tracker opened/closed issue list: missing? Message-ID: I did not receive the usual Friday tracker post on issues opened and closed during the past week. Did anyone else? I am reading via gmane. Terry Jan Reedy From rhamph at gmail.com Sat Mar 20 18:04:15 2010 From: rhamph at gmail.com (Adam Olsen) Date: Sat, 20 Mar 2010 11:04:15 -0600 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> Message-ID: On Sat, Mar 20, 2010 at 09:11, Antoine Pitrou wrote: > Mark Dickinson gmail.com> writes: >> >> On Fri, Mar 19, 2010 at 9:50 PM, Guido van Rossum python.org> > wrote: >> > There is one choice which I'm not sure about. Should a mixed >> > float/Decimal operation return a float or a Decimal? >> >> I'll just say that it's much easier to return a Decimal if you want to >> be able to make guarantees about rounding behaviour, basically because >> floats can be converted losslessly to Decimals. ?I also like the fact >> that the decimal module offers more control (rounding mode, precision, >> flags, wider exponent range) than float. > > A problem, though, is that decimals are much slower than floats. If you have a > decimal creeping in some part of a calculation it could degrade performance > quite a bit. For a little context, we have this numeric tower: int -> Fraction -> float -> complex And coincidentally (or not), we have an unspoken rule that you can go right, but never left (int/int -> float, int/Fraction -> Fraction, Fraction/float -> float, Fraction/complex -> complex, etc). This gives us a preference for fast, inexact results. Decimal is more precise, and pays a performance cost for it. It also seems odd to stick it between float and complex (nobody's planning a ComplexDecimal, right?) That suggests it should go between Fraction and float. Decimal/float -> float. -- Adam Olsen, aka Rhamphoryncus From dickinsm at gmail.com Sat Mar 20 18:05:56 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 20 Mar 2010 17:05:56 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <99e0b9531003200817j6c47c915rf0d77f2be75d306f@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com> <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com> <99e0b9531003190617w28c5da2etf8b35c54f223a036@mail.gmail.com> <5c6f2a5d1003200406o7b7c8a6am839925e0f07d0222@mail.gmail.com> <99e0b9531003200817j6c47c915rf0d77f2be75d306f@mail.gmail.com> Message-ID: <5c6f2a5d1003201005n363b8b4w154ce799312f307f@mail.gmail.com> On Sat, Mar 20, 2010 at 3:17 PM, Case Vanhorsen wrote: > On Sat, Mar 20, 2010 at 4:06 AM, Mark Dickinson wrote: >> What external modules are there that rely on existing hash behaviour? > > I'm only aware of ?gmpy and SAGE. > >> And exactly what behaviour do they rely on? > > Instead of calculating hash(long(mpz)), they calculate hash(mpz) > directly. It avoids creation of a temporary object that could be quite > large and is faster than the two-step process. I would need to modify > the code so that it continues to produce the same result. Does gmpy only do this for Python 2.6? Or does it use different algorithms for 2.4/2.5 and 2.6? As far as I can tell, there was no reasonable way to compute long_hash directly at all before the algorithm was changed for 2.6, unless you imitate exactly what Python was doing (break up into 15-bit pieces, and do all the rotation and addition exactly the same way), in which case you might as well be calling long_hash directly. Mark From brian.curtin at gmail.com Sat Mar 20 18:11:17 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Sat, 20 Mar 2010 11:11:17 -0600 Subject: [Python-Dev] Tracker opened/closed issue list: missing? In-Reply-To: References: Message-ID: On Sat, Mar 20, 2010 at 11:02, Terry Reedy wrote: > I did not receive the usual Friday tracker post on issues opened and closed > during the past week. > Did anyone else? > I am reading via gmane. > > Terry Jan Reedy > I actually haven't seen that mail in a few weeks (subscribed to the list via my gmail account). -------------- next part -------------- An HTML attachment was scrubbed... URL: From casevh at gmail.com Sat Mar 20 18:47:40 2010 From: casevh at gmail.com (Case Vanhorsen) Date: Sat, 20 Mar 2010 10:47:40 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003201005n363b8b4w154ce799312f307f@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com> <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com> <99e0b9531003190617w28c5da2etf8b35c54f223a036@mail.gmail.com> <5c6f2a5d1003200406o7b7c8a6am839925e0f07d0222@mail.gmail.com> <99e0b9531003200817j6c47c915rf0d77f2be75d306f@mail.gmail.com> <5c6f2a5d1003201005n363b8b4w154ce799312f307f@mail.gmail.com> Message-ID: <99e0b9531003201047n31d853f2mc85de92e17655720@mail.gmail.com> On Sat, Mar 20, 2010 at 10:05 AM, Mark Dickinson wrote: > On Sat, Mar 20, 2010 at 3:17 PM, Case Vanhorsen wrote: >> On Sat, Mar 20, 2010 at 4:06 AM, Mark Dickinson wrote: >>> What external modules are there that rely on existing hash behaviour? >> >> I'm only aware of ?gmpy and SAGE. >> >>> And exactly what behaviour do they rely on? >> >> Instead of calculating hash(long(mpz)), they calculate hash(mpz) >> directly. It avoids creation of a temporary object that could be quite >> large and is faster than the two-step process. I would need to modify >> the code so that it continues to produce the same result. > > Does gmpy only do this for Python 2.6? ?Or does it use different > algorithms for 2.4/2.5 and 2.6? ?As far as I can tell, there was no > reasonable way to compute long_hash directly at all before the > algorithm was changed for 2.6, unless you imitate exactly what Python > was doing (break up into 15-bit pieces, and do all the rotation and > addition exactly the same way), in which case you might as well be > calling long_hash directly. > > Mark > It does the later: it converts from GMP's internal format to CPython's long format and calculates the hash along the way. I may (should :) ) revert back to converting to long and then calling long_hash. The majority of the speed increase came from the conversion improvements, not the hash calculation. I am in favor of any change that makes 2.7 and 3.2 behave the same. casevh From rdmurray at bitdance.com Sat Mar 20 18:49:10 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 20 Mar 2010 13:49:10 -0400 Subject: [Python-Dev] Tracker opened/closed issue list: missing? In-Reply-To: References: Message-ID: <20100320174910.7E2D61FCDF1@kimball.webabinitio.net> On Sat, 20 Mar 2010 11:11:17 -0600, Brian Curtin wrote: > On Sat, Mar 20, 2010 at 11:02, Terry Reedy wrote: > > I did not receive the usual Friday tracker post on issues opened and closed > > during the past week. > > Did anyone else? > > I am reading via gmane. > > > > Terry Jan Reedy > > > > I actually haven't seen that mail in a few weeks (subscribed to the list via > my gmail account). Darn. I bet I broke it when I added the 'languishing' status. I even thought about that at the time but didn't follow up on it. I'll see if I can figure out where the script lives. -- R. David Murray www.bitdance.com From guido at python.org Sat Mar 20 20:04:08 2010 From: guido at python.org (Guido van Rossum) Date: Sat, 20 Mar 2010 12:04:08 -0700 Subject: [Python-Dev] Fraction arithmetic (Was: Decimal ... float comparisons in py3k) In-Reply-To: <5c6f2a5d1003200458q64c033b7u796c12b19eb04d7e@mail.gmail.com> References: <79990c6b1003200441k69f1899fsfd1e8b40871a95cc@mail.gmail.com> <5c6f2a5d1003200458q64c033b7u796c12b19eb04d7e@mail.gmail.com> Message-ID: On Sat, Mar 20, 2010 at 4:58 AM, Mark Dickinson wrote: > On Sat, Mar 20, 2010 at 11:41 AM, Paul Moore wrote: >> On 20 March 2010 04:20, Nick Coghlan wrote: >>> In the case of floats and Decimals, there's no ambiguity here that >>> creates any temptation to guess - to determine a true/false result for a >>> comparison, floats can be converted explicitly to Decimals without any >>> loss of accuracy. For Fractions, the precedent has already been set by >>> allowing implicit (potentially lossy) conversion to binary floats - a >>> lossy conversion to Decimal wouldn't be any worse. >> >> Hmm, given that a float can be converted losslessly to a fraction, why >> was the decision taken to convert the fraction to a float rather than >> the other way round? > > I'm not sure of the actual reason for this decision, but one argument > I've seen used for other languages is that it's desirable for the > inexactness of the float type to be contagious: ?rationals are > perceived as exact, while floats are perceived as approximations. You hit the nail on the head. > Note that this only applies to arithmetic operations: ?for > comparisons, an exact conversion *is* performed. ?This is much like > what currently happens with ints and floats in the core: ?a mixed-type > arithmetic operation between an int and a float first converts the int > to a float (possibly changing the value in the process). ?A mixed-type > comparison makes an exact comparison without doing such a conversion. > For example (in any non-ancient version of Python): > >>>> n = 2**53 + 1 >>>> x = 2.**53 >>>> n > x ? # compares exact values; ?no conversion performed > True >>>> n - x ? ?# converts n to a float before subtracting > 0.0 > >> I don't see a PEP for the fractions module, and my google-fu has >> failed to find anything. Was there a discussion on this? > > There's PEP 3141 (http://www.python.org/dev/peps/pep-3141/), which was > the motivation for adding the fractions module in the first place, and > there's the issue tracker item for the fractions module > (http://bugs.python.org/issue1682). -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Sat Mar 20 20:29:50 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 20 Mar 2010 20:29:50 +0100 Subject: [Python-Dev] Tracker opened/closed issue list: missing? In-Reply-To: <20100320174910.7E2D61FCDF1@kimball.webabinitio.net> References: <20100320174910.7E2D61FCDF1@kimball.webabinitio.net> Message-ID: <4BA5222E.3020008@v.loewis.de> > Darn. I bet I broke it when I added the 'languishing' status. I even > thought about that at the time but didn't follow up on it. I'll see > if I can figure out where the script lives. My guess is that the Debian upgrade killed it. Regards, Martin From guido at python.org Sat Mar 20 20:56:24 2010 From: guido at python.org (Guido van Rossum) Date: Sat, 20 Mar 2010 12:56:24 -0700 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003200416i59ce880fk9cec234caaac2eb0@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com> <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com> <4BA3680D.8070909@gmail.com> <4BA41271.3010003@canterbury.ac.nz> <5c6f2a5d1003200416i59ce880fk9cec234caaac2eb0@mail.gmail.com> Message-ID: On Sat, Mar 20, 2010 at 4:16 AM, Mark Dickinson wrote: > True. ?The reason I was concentrating on the hashes is that it's not > immediately obvious that it's even *possible* to find a decent hash > function that's efficient to compute and gives equal results for > numerically equal inputs (regardless of type); this is especially true > if you don't want to significantly slow down the existing hashes for > int and float. ?But once that problem is solved, it shouldn't be too > hard to implement all the comparisons. > > It *is* kinda messy, because as far as I can see the oddities of the > various types mean that you end up producing specialized code for > comparing each pair of types (one block of code for float<->Fraction > comparisons, another for float<->Decimal, yet another for > Decimal<->Fraction, and so on), but it's doable. I propose to reduce all hashes to the hash of a normalized fraction, which we can define as a combination of the hashes for the numerator and the denominator. Then all we have to do is figure fairly efficient ways to convert floats and decimals to normalized fractions (not necessarily Fractions). I may be naive but this seems doable: for a float, the denominator is always a power of 2 and removing factors of 2 from the denominator is easy (just right-shift until the last bit is zero). For Decimal, the unnormalized denominator is always a power of 10, and the normalization is a bit messier, but doesn't seem excessively so. The resulting numerator and denominator may be large numbers, but for typical use of Decimal and float they will rarely be excessively large, and I'm not too worried about slowing things down when they are (everything slows down when you're using really large integers anyway). -- --Guido van Rossum (python.org/~guido) From dickinsm at gmail.com Sat Mar 20 21:38:46 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 20 Mar 2010 20:38:46 +0000 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com> <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com> <4BA3680D.8070909@gmail.com> <4BA41271.3010003@canterbury.ac.nz> <5c6f2a5d1003200416i59ce880fk9cec234caaac2eb0@mail.gmail.com> Message-ID: <5c6f2a5d1003201338p18aebd40y2c345b6dc8b861a6@mail.gmail.com> On Sat, Mar 20, 2010 at 7:56 PM, Guido van Rossum wrote: > I propose to reduce all hashes to the hash of a normalized fraction, > which we can define as a combination of the hashes for the numerator > and the denominator. Then all we have to do is figure fairly efficient > ways to convert floats and decimals to normalized fractions (not > necessarily Fractions). I may be naive but this seems doable: for a > float, the denominator is always a power of 2 and removing factors of > 2 from the denominator is easy (just right-shift until the last bit is > zero). For Decimal, the unnormalized denominator is always a power of > 10, and the normalization is a bit messier, but doesn't seem > excessively so. The resulting numerator and denominator may be large > numbers, but for typical use of Decimal and float they will rarely be > excessively large, and I'm not too worried about slowing things down > when they are (everything slows down when you're using really large > integers anyway). I *am* worried about slowing things down for large Decimals: if you can't put Decimal('1e1234567') into a dict or set without waiting for an hour for the hash computation to complete (because it's busy computing 10**1234567), I consider that a problem. But it's solvable! I've just put a patch on the bug tracker: http://bugs.python.org/issue8188 It demonstrates how hashes can be implemented efficiently and compatibly for all numeric types, even large Decimals like the above. It needs a little tidying up, but it works. Mark From brett at python.org Sat Mar 20 21:40:19 2010 From: brett at python.org (Brett Cannon) Date: Sat, 20 Mar 2010 13:40:19 -0700 Subject: [Python-Dev] Silencing DeprecationWarning by default in Python 2.7 is silencing division warnings In-Reply-To: References: Message-ID: On Sat, Mar 6, 2010 at 12:43, Brett Cannon wrote: > I see two possible fixes for this. One is to not silence DeprecationWarning > if Py_DivisionWarningFlag is set to >= 1. The other is to introduce a new > subclass of DeprecationWarning called IntegerDivisionWarning and have that > added to the warnings filter so that if it is triggered it is handled > separately from what DeprecationWarning triggers. > The former means that you might get more than you bargained for in terms of > warnings as you are suddenly switching on all DeprecationWarnings on top of > division warnings. The latter means that you now have to explicit care about > IntegerDivisionWarning on top of DeprecationWarning (to minimize this I > could have IntegerDivisionWarning added to the warnings filter only in the > case of when Py_DivisionWarningFlag is set instead of blindly adding it). > Thoughts? > -Brett Just to follow up on this with my current thinking, I am leaning towards having Py_DivisionWarningFlag disable the suppression of DeprecationWarning. Since the silencing is new in 2.7, people using -Q will not see any different semantics than they were with Python 2.6. Plus if they are bothering to be notified about division warnings they probably want to know about other warnings as well. From greg.ewing at canterbury.ac.nz Sat Mar 20 23:43:01 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 21 Mar 2010 10:43:01 +1200 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <4BA467C0.1000300@g.nevcal.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <4BA3CD93.8080502@g.nevcal.com> <4BA44D21.1070401@gmail.com> <4BA467C0.1000300@g.nevcal.com> Message-ID: <4BA54F75.4050004@canterbury.ac.nz> Glenn Linderman wrote: > One can argue that either way, that it is completely different, or > completely the same. An important difference is that there is no intermediate type that can be compared with both ints and strings. Another relevant difference is that numbers are just one of many possible things that a string could be interpreted as representing. Decimals, on the other hand, are just numbers and nothing else. (Even strings which happen to look like numbers might not be -- e.g. telephone "numbers", which are better thought of as strings with a limited character set... despite some misguided souls occasionally storing them in floating point. :-) -- Greg From greg.ewing at canterbury.ac.nz Sun Mar 21 00:20:56 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 21 Mar 2010 11:20:56 +1200 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> Message-ID: <4BA55858.2000800@canterbury.ac.nz> Adam Olsen wrote: > For a little context, we have this numeric tower: > > int -> Fraction -> float -> complex > > Decimal is more precise, and pays a performance cost for it. It also > seems odd to stick it between float and complex (nobody's planning a > ComplexDecimal, right?) That suggests it should go between Fraction > and float. Decimal/float -> float. There are two ways in which that linear tower is overly simplistic: * It conflates the notions of exactness and width. They're really orthogonal concepts, and to reflect this you would need two parallel towers, with exact and inexact versions of each type. * Decimal and float really belong side-by-side in the tower, rather than one above the other. Neither of them is inherently any more precise or exact than the other. There doesn't seem to be any good solution here. For every use case in which Decimal+float->float appears better, there seems to be another one for which Decimal+float->Decimal appears better. -- Greg From jyasskin at gmail.com Sun Mar 21 00:27:54 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Sat, 20 Mar 2010 16:27:54 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA55858.2000800@canterbury.ac.nz> References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> <4BA55858.2000800@canterbury.ac.nz> Message-ID: <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> On Sat, Mar 20, 2010 at 4:20 PM, Greg Ewing wrote: > Adam Olsen wrote: > >> For a little context, we have this numeric tower: >> >> int -> Fraction -> float -> complex >> >> Decimal is more precise, and pays a performance cost for it. ?It also >> seems odd to stick it between float and complex (nobody's planning a >> ComplexDecimal, right?) ?That suggests it should go between Fraction >> and float. ?Decimal/float -> float. > > There are two ways in which that linear tower is overly > simplistic: > > * It conflates the notions of exactness and width. They're > really orthogonal concepts, and to reflect this you would > need two parallel towers, with exact and inexact versions > of each type. It's representing the mathematical concepts Integral -> Rational -> Real -> Complex When designing it, I tried to include a notion of exact/inexact types, but we couldn't find anything practical to do with them, so we took them out. It's reasonably easy to design inexact Integral and Rational types, but pretty hard to design a useful, exact Real type (things like '==' get uncomputable quickly), so we probably couldn't actually implement two whole parallel towers. > * Decimal and float really belong side-by-side in the > tower, rather than one above the other. Yep. From dickinsm at gmail.com Sun Mar 21 00:46:09 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 20 Mar 2010 23:46:09 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA55858.2000800@canterbury.ac.nz> References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> <4BA55858.2000800@canterbury.ac.nz> Message-ID: <5c6f2a5d1003201646h11f38bc0jcbd621a6591c709c@mail.gmail.com> On Sat, Mar 20, 2010 at 11:20 PM, Greg Ewing wrote: > * Decimal and float really belong side-by-side in the > tower, rather than one above the other. Neither of them is > inherently any more precise or exact than the other. Except that float is fixed-width (typically 53 bits of precision), while Decimal allows a user-specified, arbitrarily large, precision; so in that sense the two floating-point types aren't on an equal footing. Mark From guido at python.org Sun Mar 21 00:52:42 2010 From: guido at python.org (Guido van Rossum) Date: Sat, 20 Mar 2010 16:52:42 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003201646h11f38bc0jcbd621a6591c709c@mail.gmail.com> References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> <4BA55858.2000800@canterbury.ac.nz> <5c6f2a5d1003201646h11f38bc0jcbd621a6591c709c@mail.gmail.com> Message-ID: On Sat, Mar 20, 2010 at 4:46 PM, Mark Dickinson wrote: > On Sat, Mar 20, 2010 at 11:20 PM, Greg Ewing > wrote: >> * Decimal and float really belong side-by-side in the >> tower, rather than one above the other. Neither of them is >> inherently any more precise or exact than the other. > > Except that float is fixed-width (typically 53 bits of precision), > while Decimal allows a user-specified, arbitrarily large, precision; > so in that sense the two floating-point types aren't on an equal > footing. But this doesn't really help deciding which should be positioned at the end does it? Both Fraction and Decimal can represent every float value (barring NaN/Inf). I wonder if we need to look at use cases or see what other languages do for guidance. -- --Guido van Rossum (python.org/~guido) From rhamph at gmail.com Sun Mar 21 00:58:28 2010 From: rhamph at gmail.com (Adam Olsen) Date: Sat, 20 Mar 2010 17:58:28 -0600 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA55858.2000800@canterbury.ac.nz> References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> <4BA55858.2000800@canterbury.ac.nz> Message-ID: On Sat, Mar 20, 2010 at 17:20, Greg Ewing wrote: > There are two ways in which that linear tower is overly > simplistic: > > * It conflates the notions of exactness and width. They're > really orthogonal concepts, and to reflect this you would > need two parallel towers, with exact and inexact versions > of each type. > > * Decimal and float really belong side-by-side in the > tower, rather than one above the other. Neither of them is > inherently any more precise or exact than the other. > > There doesn't seem to be any good solution here. For every > use case in which Decimal+float->float appears better, there > seems to be another one for which Decimal+float->Decimal > appears better. Sure, from a purist point of view my post is completely wrong. It doesn't correspond to the mathematical reality. What it does correspond to is the code. Only going rightward through the types is what we have today. A linear progression is a lot simpler to understand than any sort of cycle; parallel progressions isn't even on the table. float has been the king of inexact types right for a long time. All other things being equal, that's good enough for me. -- Adam Olsen, aka Rhamphoryncus From ncoghlan at gmail.com Sun Mar 21 02:10:08 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 21 Mar 2010 11:10:08 +1000 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <4BA4165D.2070808@canterbury.ac.nz> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <4BA4165D.2070808@canterbury.ac.nz> Message-ID: <4BA571F0.3090800@gmail.com> Greg Ewing wrote: > Antoine Pitrou wrote: > >> We forbid comparisons when there is a real danger or ambiguity, such >> as unicode >> vs. bytes. There is no such danger or ambiguity when comparing a >> decimal with a >> float. > > So do you think that float("0.1") and Decimal("0.1") should be > equal or not, and why? Note that Antoine's point was that float("0.1") and Decimal.from_float(0.1) should compare equal. The latter exactly matches the underlying binary floating point value: >>> dec.from_float(0.1) Decimal('0.1000000000000000055511151231257827021181583404541015625') Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sun Mar 21 02:17:09 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 21 Mar 2010 11:17:09 +1000 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <4BA467C0.1000300@g.nevcal.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <4BA3CD93.8080502@g.nevcal.com> <4BA44D21.1070401@gmail.com> <4BA467C0.1000300@g.nevcal.com> Message-ID: <4BA57395.3040803@gmail.com> Glenn Linderman wrote: > Thanks, though for the nice chart from an internals guy, you confirmed > all that I thought I knew about how this presently works, from PEP and > manual reading, a bit of experimentation, and the discussions here. I'll confess to a bit of interpreter prompt experimentation to confirm the behaviour, especially in the relation to Fractions - I didn't know all of that off the top of my head. The implicit coercion from Fraction -> float definitely surprised me a bit. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From fuzzyman at voidspace.org.uk Sun Mar 21 02:30:23 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 21 Mar 2010 01:30:23 +0000 Subject: [Python-Dev] Attribute lookup ambiguity In-Reply-To: <4BA4B8DE.8060109@wanadoo.fr> References: <4BA3C962.7080500@wanadoo.fr> <4BA3DD4F.1080209@voidspace.org.uk> <4BA4B8DE.8060109@wanadoo.fr> Message-ID: <4BA576AF.2050208@voidspace.org.uk> On 20/03/2010 12:00, Pascal Chambon wrote: > Michael Foord a ?crit : >> >> On 19/03/2010 18:58, Pascal Chambon wrote: >>> Hello >>> >>> I've already crossed a bunch of articles detailing python's >>> attribute lookup semantic (__dict__, descriptors, order of base >>> class traversing...), but I have never seen, so far, an explanation >>> of WHICH method did waht, exactly. >>> >>> I assumed that getattr(a, b) was the same as a.__getattribute__(b), >>> and that this __getattribute__ method (or the hidden routine >>> replacing it when we don't override it in our class) was in charge >>> of doing the whole job of traversing the object tree, checking >>> descriptors, binding methods, calling __getattr__ on failure etc. >>> >>> However, the test case below shows that __getattribute__ does NOT >>> call __getattr__ on failure. So it seems it's an upper levl >>> machinery, in getattr(), which is in chrge of that last action. >> >> Python 3 has the behavior you are asking for. It would be a backwards >> incompatible change to do it in Python 2 as __getattribute__ *not* >> calling __getattr__ is the documented behaviour. >> >> Python 3.2a0 (py3k:78770, Mar 7 2010, 20:32:50) >> [GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin >> >>> class x: >> ... def __getattribute__(s, name): >> ... print ('__getattribute__', name) >> ... raise AttributeError >> ... def __getattr__(s, name): >> ... print ('__getattr__', name) >> ... >> >>> a = x() >> >>> a.b >> __getattribute__ b >> __getattr__ b > I'm confused there, because the script you gave behaves the same in > python 2.6. And according to the doc, it's normal, getattr() reacts to > an AttributeError from __getattribute__, by calling __getattr__ : > > """ > Python 2.6.5 documentation > > object.__getattribute__(/self/, /name/) > > Called unconditionally to implement attribute accesses for > instances of the class. If the class also defines __getattr__() > , > the latter will not be called unless __getattribute__() > > either calls it explicitly or raises an AttributeError > . > This method should return the (computed) attribute value or raise > an AttributeError > > exception. In order to avoid infinite recursion in this method, > its implementation should always call the base class method with > the same name to access any attributes it needs, for example, > object.__getattribute__(self, name). > > """ > > But the point which for me is still unclear, is : does the default > implementation of __getattribute__ (the one of "object") call > __getattr__ by himself, or does it rely on its caller for that, by > raising an AttributeError ? For Python2, it's blatantly the latter > case which is favoured, but since it looks like an implementation > detail at the moment, I propose we settle it (and document it) once > for all. Ah right, my apologies. So it is still documented behaviour - __getattr__ is obviously called by the Python runtime and not by __getattribute__. (It isn't just by getattr as the same behaviour is shown when doing a normal attribute lookup and not via the getattr function.) > >> >> >> This list is not really an appropriate place to ask questions like >> this though, comp.lang.python would be better. >> >> All the best, >> >> Michael Fooord > Sorry if I misposted, I just (wrongly ?) assumed that it was more an > undecided, implementation-specific point (since the doc gave possible > behaviours for __getattribute__, without precising which one was the > default one), and thus targetted the hands-in-core-code audience only. > Well, the documentation you pointed to specifies that __getattr__ will be called if __getattribute__ raises an AttributeError, it just doesn't specify that it is done by object.__getattribute__ (which it isn't). Michael Foord > Regards, > Pascal > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Mar 21 02:54:34 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 21 Mar 2010 11:54:34 +1000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: Message-ID: <4BA57C5A.9060403@gmail.com> Guido van Rossum wrote: > I think we should seriously reconsider allowing mixed arithmetic > involving Decimal, not just mixed comparisons. I'm glad I'm not the only one that started wondering that. I wasn't quite game enough to actually suggest it though :) > There is one choice which I'm not sure about. Should a mixed > float/Decimal operation return a float or a Decimal? I note that > Fraction (which *is* properly embedded in the numeric tower) supports > this and returns a float result in this case. While I earlier > proposed to return the most "complicated" type of the two, i.e. > Decimal, I now think it may also make sense to return a float, being > the most "fuzzy" type in the numeric tower. This would also make > checking for accidental floats easier, since floats now propagate > throughout the computation (like NaN) and a simple assertion that the > result is a Decimal instance suffices to check that no floats were > implicitly mixed into the computation. To blend a couple of different ideas from the thread reboot together: I suggest a 'linearised' numeric tower that looks like: int -> Decimal -> Fraction -> float -> complex As Adam stated, this is a pragmatic tower stating which implicit coercions are defined by the code, not a formal mathematical relationship. Note that this would involve adding mixed Fraction/Decimal arithmetic as well as Decimal/float arithmetic. I placed Decimal to the left of Fraction to keep Decimal's dependencies clear and because Decimal -> Fraction conversions appear straightforward (using power of 10 denominators) without introducing the precision issues that would arise in Decimal -> Fraction conversions. I also like the idea of adding the decimal context signal that Facundo suggests (e.g. under the name ImplicitCoercionToBinaryFloat). So "quick and dirty don't need a perfect answer" operations end up with ordinary binary floats, while more precise code can enable the signal trap to ensure the calculation fails noisily if binary floats are introduced. Allowing implicit operations would also finally allow Decimal to be registered with numbers.Real. Whatever we decide to do will need to be codified in a PEP though. "Cleaning Up Python's Numeric Tower" or something along those lines. > The implementation of __hash__ will be complicated, and it may make > sense to tweak the hash function of float, Fraction and Decimal to > make it easier to ensure that for values that can be represented in > either type the hash matches the equality. But this sounds a > worthwhile price to pay for proper embedding in the numeric tower. And Mark appears to already have a good answer to that problem. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sun Mar 21 02:56:17 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 21 Mar 2010 11:56:17 +1000 Subject: [Python-Dev] Attribute lookup ambiguity In-Reply-To: <4BA576AF.2050208@voidspace.org.uk> References: <4BA3C962.7080500@wanadoo.fr> <4BA3DD4F.1080209@voidspace.org.uk> <4BA4B8DE.8060109@wanadoo.fr> <4BA576AF.2050208@voidspace.org.uk> Message-ID: <4BA57CC1.1050404@gmail.com> Michael Foord wrote: > Well, the documentation you pointed to specifies that __getattr__ will > be called if __getattribute__ raises an AttributeError, it just doesn't > specify that it is done by object.__getattribute__ (which it isn't). And as for why not: because __getattribute__ implementations need to be able to call object.__getattribute__ without triggering the fallback behaviour. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From raymond.hettinger at gmail.com Sun Mar 21 03:44:09 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 20 Mar 2010 19:44:09 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA57C5A.9060403@gmail.com> References: <4BA57C5A.9060403@gmail.com> Message-ID: <043F9D0F-AFC7-4D09-8514-0C7AD59BCD9B@gmail.com> On Mar 20, 2010, at 6:54 PM, Nick Coghlan wrote: > > I suggest a 'linearised' numeric tower that looks like: > > int -> Decimal -> Fraction -> float -> complex Is that a typo? Shouldn't Decimal and float go between Fraction and complex? The abstract numeric tower is: Number Complex Real Rational Integral where both Decimal and float have operations associated with reals. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Sun Mar 21 04:46:15 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 20 Mar 2010 20:46:15 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> <4BA55858.2000800@canterbury.ac.nz> <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> Message-ID: On Mar 20, 2010, at 4:27 PM, Jeffrey Yasskin wrote: > > When designing it, I tried to include a notion of exact/inexact types, > but we couldn't find anything practical to do with them, so we took > them out. The were also other reasons that they were taken out. The notion of inexactness is a taint, not a property of a type. The design documents for the decimal spec aimed for the ability to do either inexact or exact calculations (and to encompass integer and fixed-point arithmetic). That is in-part why decimal is suitable for accounting work. A person can forbid any inexactness by setting a context flag that would raise an exception if any inexact calculation occurs. Another design principle for decimal is the notion that numbers are always exact, it is the results of operations that are subject to rounding. That is why the decimal constructor is not context sensitive. So, as Jeffrey says, the notion of exactness and inexactness is not built in to the numeric tower. Instead, it is about which operations or methods can be expected to apply to a given type (i.e. both float and Decimal support the necessary methods to register as a Real). In terms of interoperability of concrete types, we're in the fortunate position that any float can be exactly converted to a Decimal and any Decimal can be exactly converted to a Fraction. Raymond From greg.ewing at canterbury.ac.nz Sun Mar 21 05:40:19 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 21 Mar 2010 16:40:19 +1200 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003201646h11f38bc0jcbd621a6591c709c@mail.gmail.com> References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> <4BA55858.2000800@canterbury.ac.nz> <5c6f2a5d1003201646h11f38bc0jcbd621a6591c709c@mail.gmail.com> Message-ID: <4BA5A333.9070404@canterbury.ac.nz> Mark Dickinson wrote: > Except that float is fixed-width (typically 53 bits of precision), > while Decimal allows a user-specified, arbitrarily large, precision; Yes, but it still has *some* fixed limit at any given moment, so the result of an operation on Decimals always has the potential to produce an inexact result. It's not like an int or Fraction where the result can expand to whatever size is needed. -- Greg From greg.ewing at canterbury.ac.nz Sun Mar 21 05:48:18 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 21 Mar 2010 16:48:18 +1200 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <4BA571F0.3090800@gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <4BA4165D.2070808@canterbury.ac.nz> <4BA571F0.3090800@gmail.com> Message-ID: <4BA5A512.5060005@canterbury.ac.nz> Nick Coghlan wrote: > Note that Antoine's point was that float("0.1") and > Decimal.from_float(0.1) should compare equal. That would mean that Decimal("0.1") != float("0.1"), which might be surprising to someone who didn't realise they were mixing floats and decimals. -- Greg From greg.ewing at canterbury.ac.nz Sun Mar 21 05:54:18 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 21 Mar 2010 16:54:18 +1200 Subject: [Python-Dev] Attribute lookup ambiguity In-Reply-To: <4BA576AF.2050208@voidspace.org.uk> References: <4BA3C962.7080500@wanadoo.fr> <4BA3DD4F.1080209@voidspace.org.uk> <4BA4B8DE.8060109@wanadoo.fr> <4BA576AF.2050208@voidspace.org.uk> Message-ID: <4BA5A67A.9000703@canterbury.ac.nz> Michael Foord wrote: > Well, the documentation you pointed to specifies that __getattr__ will > be called if __getattribute__ raises an AttributeError, it just doesn't > specify that it is done by object.__getattribute__ (which it isn't). If __getattribute__ raises an exception, it won't get a chance to do anything else, so something outside of __getattribute__ must catch the AttributeError and calling __getattr__. So I think the docs *are* specifying the behaviour here, if only by implication. -- Greg From greg.ewing at canterbury.ac.nz Sun Mar 21 06:00:47 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 21 Mar 2010 17:00:47 +1200 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA57C5A.9060403@gmail.com> References: <4BA57C5A.9060403@gmail.com> Message-ID: <4BA5A7FF.7080501@canterbury.ac.nz> Nick Coghlan wrote: > int -> Decimal -> Fraction -> float -> complex I don't think it's a good idea to put Decimal below Fraction, because Decimal has to be considered an implicitly inexact type like float, and we don't want to coerce from an inexact type to an exact one. -- Greg From rhamph at gmail.com Sun Mar 21 06:54:17 2010 From: rhamph at gmail.com (Adam Olsen) Date: Sat, 20 Mar 2010 23:54:17 -0600 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <4BA5A512.5060005@canterbury.ac.nz> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <4BA4165D.2070808@canterbury.ac.nz> <4BA571F0.3090800@gmail.com> <4BA5A512.5060005@canterbury.ac.nz> Message-ID: On Sat, Mar 20, 2010 at 22:48, Greg Ewing wrote: > Nick Coghlan wrote: > >> Note that Antoine's point was that float("0.1") and >> Decimal.from_float(0.1) should compare equal. > > That would mean that Decimal("0.1") != float("0.1"), which might be > surprising to someone who didn't realise they were mixing floats > and decimals. Much like Fraction("0.1") != float("0.1") is surprising. The only way to get rid of surprises around float is to get rid of float, and that ain't happening. -- Adam Olsen, aka Rhamphoryncus From ncoghlan at gmail.com Sun Mar 21 07:58:24 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 21 Mar 2010 16:58:24 +1000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <043F9D0F-AFC7-4D09-8514-0C7AD59BCD9B@gmail.com> References: <4BA57C5A.9060403@gmail.com> <043F9D0F-AFC7-4D09-8514-0C7AD59BCD9B@gmail.com> Message-ID: <4BA5C390.8030606@gmail.com> Raymond Hettinger wrote: > > On Mar 20, 2010, at 6:54 PM, Nick Coghlan wrote: >> >> I suggest a 'linearised' numeric tower that looks like: >> >> int -> Decimal -> Fraction -> float -> complex > > Is that a typo? Shouldn't Decimal and float go between Fraction and > complex? > > The abstract numeric tower is: > > Number Complex Real Rational Integral > > where both Decimal and float have operations associated with reals. I don't actually mind either way - the pragmatic tower is about coding convenience rather than numeric purity (and mixing Fractions and Decimals in the same algorithm is somewhat nonsensical - they're designed for two completely different problem domains). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sun Mar 21 07:59:49 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 21 Mar 2010 16:59:49 +1000 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <4BA5A512.5060005@canterbury.ac.nz> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <4BA4165D.2070808@canterbury.ac.nz> <4BA571F0.3090800@gmail.com> <4BA5A512.5060005@canterbury.ac.nz> Message-ID: <4BA5C3E5.6080805@gmail.com> Greg Ewing wrote: > Nick Coghlan wrote: > >> Note that Antoine's point was that float("0.1") and >> Decimal.from_float(0.1) should compare equal. > > That would mean that Decimal("0.1") != float("0.1"), which might be > surprising to someone who didn't realise they were mixing floats > and decimals. That's fine - binary floats *are* surprising. That's why Decimal exists in the first place. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From webmaster at python.org Sun Mar 21 14:55:18 2010 From: webmaster at python.org (webmaster at python.org) Date: Sun, 21 Mar 2010 13:55:18 +0000 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) Message-ID: <4BA62546.10906@python.org> Hey all, This seems to happen whenever we do a new release (we've had a couple of emails to webmaster at python.org about it since 2.6.5 was released). The main download page for Python has a broken link for the Mac installer (because it hasn't been built yet I assume): http://python.org/download/ The link 404s, with no explanation or alternate link - so for the casual user who wants to install Python 2.6 on Mac OS X they are sorely-out-of-luck. Not being able to provide a mac installer at the same time as other platforms is one thing (and I accept that is unavoidable), breaking the download links for Mac users for unspecified lengths of time is just bad practise. If we create a new stable release without a Mac installer can we at least provide a brief explanation and link to the *previous version* until the new version is ready? All the best, Michael -------- Original Message -------- Subject: Broken link to down Date: Sun, 21 Mar 2010 13:40:36 +0000 From: Ben Hodgson To: webmaster at python.org Hey there, In case you don't know, the link on http://www.python.org/download/ to the Python 2.6.5 Mac Installer Disk Image (http://www.python.org/ftp/python/2.6.5/python-2.6.5_macosx10.3.dmg) is broken. Cheers, Ben -- Ben Hodgson http://benhodgson.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Mar 21 17:53:14 2010 From: guido at python.org (Guido van Rossum) Date: Sun, 21 Mar 2010 09:53:14 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA57C5A.9060403@gmail.com> References: <4BA57C5A.9060403@gmail.com> Message-ID: On Sat, Mar 20, 2010 at 6:54 PM, Nick Coghlan wrote: > Guido van Rossum wrote: >> I think we should seriously reconsider allowing mixed arithmetic >> involving Decimal, not just mixed comparisons. > > I'm glad I'm not the only one that started wondering that. I > wasn't quite game enough to actually suggest it though :) > >> There is one choice which I'm not sure about. Should a mixed >> float/Decimal operation return a float or a Decimal? I note that >> Fraction (which *is* properly embedded in the numeric tower) supports >> ?this and returns a float result in this case. While I earlier >> proposed to return the most "complicated" type of the two, i.e. >> Decimal, I now think it may also make sense to return a float, being >> ?the most "fuzzy" type in the numeric tower. This would also make >> checking for accidental floats easier, since floats now propagate >> throughout the computation (like NaN) and a simple assertion that the >> ?result is a Decimal instance suffices to check that no floats were >> implicitly mixed into the computation. > > To blend a couple of different ideas from the thread reboot together: > > I suggest a 'linearised' numeric tower that looks like: > > int -> Decimal -> Fraction -> float -> complex > > As Adam stated, this is a pragmatic tower stating which implicit > coercions are defined by the code, not a formal mathematical relationship. > > Note that this would involve adding mixed Fraction/Decimal arithmetic as > well as Decimal/float arithmetic. Yes, that was my intention too. > I placed Decimal to the left of > Fraction to keep Decimal's dependencies clear and because Decimal -> > Fraction conversions appear straightforward (using power of 10 > denominators) without introducing the precision issues that would arise > in Decimal -> Fraction conversions. This is just wrong. Decimal is much more like float than like int or Fraction, due to its rounding behavior. (You cannot produce exact results for 1/3 using either Decimal or float.) Clearly the difficult decision is whether Decimal is between Fraction and float, or between float and complex (I prefer not to say "to the left/right of" since PEP 3141 orders the types in the opposite way as people have been doing here). Both are floating point types that sometimes produce rounded results (e.g. 1/3); the difference is that they use different bases and that Decimal has configurable precision when rounding. I would make float the last station before complex, whereas Mark would prefer Decimal to go there. I defer to Mark, who has thought a lot more about Decimal than I have. > I also like the idea of adding the decimal context signal that Facundo > suggests (e.g. under the name ImplicitCoercionToBinaryFloat). Yes, this is what I was trying to say (but didn't find the right words for). > So "quick and dirty don't need a perfect answer" operations end up with > ordinary binary floats, while more precise code can enable the signal > trap to ensure the calculation fails noisily if binary floats are > introduced. But does this break a tie for the relative ordering of Decimal and float in the tower? > Allowing implicit operations would also finally allow Decimal to be > registered with numbers.Real. Right, that's one aspect of what I meant by "embedded in the numeric tower". > Whatever we decide to do will need to be codified in a PEP though. > "Cleaning Up Python's Numeric Tower" or something along those lines. The cleanup is really just specific to Decimal -- int, Fraction and float are already properly embedded in the tower (PEP 3141 doesn't advertise Fraction enough, since it predates it). That we may have to change the __hash__ implementation for the other types is merely a compromise towards efficiency. >> The implementation of __hash__ will be complicated, and it may make >> sense to tweak the hash function of float, Fraction and Decimal to >> make it easier to ensure that for values that can be represented in >> either type the hash matches the equality. But this sounds a >> worthwhile price to pay for proper embedding in the numeric tower. > > And Mark appears to already have a good answer to that problem. Which I still have to review. (Mark, if you're there, could you make a brief post here on the mathematical definition of the new hash you're proposing, and why it is both efficient to compute and good (enough) as a hash function?) -- --Guido van Rossum (python.org/~guido) From guido at python.org Sun Mar 21 18:02:30 2010 From: guido at python.org (Guido van Rossum) Date: Sun, 21 Mar 2010 10:02:30 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA57C5A.9060403@gmail.com> Message-ID: On Sun, Mar 21, 2010 at 9:53 AM, Guido van Rossum wrote: > Which I still have to review. (Mark, if you're there, could you make a > brief post here on the mathematical definition of the new hash you're > proposing, and why it is both efficient to compute and good (enough) > as a hash function?) Never mind, Mark. I found your explanation here: http://codereview.appspot.com/660042/diff/19001/11009?column_width=80 -- --Guido van Rossum (python.org/~guido) From guido at python.org Sun Mar 21 18:07:56 2010 From: guido at python.org (Guido van Rossum) Date: Sun, 21 Mar 2010 10:07:56 -0700 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <4BA5C3E5.6080805@gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <4BA4165D.2070808@canterbury.ac.nz> <4BA571F0.3090800@gmail.com> <4BA5A512.5060005@canterbury.ac.nz> <4BA5C3E5.6080805@gmail.com> Message-ID: On Sat, Mar 20, 2010 at 11:59 PM, Nick Coghlan wrote: > Greg Ewing wrote: >> Nick Coghlan wrote: >> >>> Note that Antoine's point was that float("0.1") and >>> Decimal.from_float(0.1) should compare equal. >> >> That would mean that Decimal("0.1") != float("0.1"), which might be >> surprising to someone who didn't realise they were mixing floats >> and decimals. > > That's fine - binary floats *are* surprising. That's why Decimal exists > in the first place. Decimals can be just as surprising: >>> Decimal(1) / Decimal(3) * Decimal(3) == Decimal(1) False >>> -- --Guido van Rossum (python.org/~guido) From benjamin at python.org Sun Mar 21 18:12:08 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 21 Mar 2010 12:12:08 -0500 Subject: [Python-Dev] [RELEASED] Python 3.1.2 Message-ID: <1afaf6161003211012r30bc0e77ub5dfe9f248668525@mail.gmail.com> On behalf of the Python development team, I'm joyful to announce the second bugfix release of the Python 3.1 series, Python 3.1.2. This bug fix release fixes numerous issues found in 3.1.1, and is considered a production release. The Python 3.1 version series focuses on the stabilization and optimization of the features and changes that Python 3.0 introduced. For example, the new I/O system has been rewritten in C for speed. File system APIs that use unicode strings now handle paths with undecodable bytes in them. Other features include an ordered dictionary implementation, a condensed syntax for nested with statements, and support for ttk Tile in Tkinter. For a more extensive list of changes in 3.1, see http://doc.python.org/3.1/whatsnew/3.1.html or Misc/NEWS in the Python distribution. To download Python 3.1.2 visit: http://www.python.org/download/releases/3.1.2/ A list of changes in 3.1.2 can be found here: http://svn.python.org/projects/python/tags/r312/Misc/NEWS The 3.1 documentation can be found at: http://docs.python.org/3.1 Bugs can always be reported to: http://bugs.python.org Enjoy! -- Benjamin Peterson Release Manager benjamin at python.org (on behalf of the entire python-dev team and 3.1.2's contributors) From raymond.hettinger at gmail.com Sun Mar 21 19:25:34 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 21 Mar 2010 11:25:34 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA57C5A.9060403@gmail.com> Message-ID: >> >> Note that this would involve adding mixed Fraction/Decimal arithmetic as >> well as Decimal/float arithmetic. > > Yes, that was my intention too. +1 > >> I placed Decimal to the left of >> Fraction to keep Decimal's dependencies clear and because Decimal -> >> Fraction conversions appear straightforward (using power of 10 >> denominators) without introducing the precision issues that would arise >> in Decimal -> Fraction conversions. > > This is just wrong. Decimal is much more like float than like int or > Fraction, due to its rounding behavior. (You cannot produce exact > results for 1/3 using either Decimal or float.) Right. We should be guided by: fractions are a superset of decimals which are a superset of binary floats. And by: binary floats and decimal floats both implement all of the operations for the Real abstract base class. > > Yes, this is what I was trying to say (but didn't find the right words for). > >> So "quick and dirty don't need a perfect answer" operations end up with >> ordinary binary floats, while more precise code can enable the signal >> trap to ensure the calculation fails noisily if binary floats are >> introduced. > > But does this break a tie for the relative ordering of Decimal and > float in the tower? > It seems to me that Decimals and floats should be considered at the same level (i.e. both implement Real). Mixed Decimal and float should coerce to Decimal because it can be done losslessly. There is no need to embed a notion of "imperfect answer". Numbers themselves are exact and many mixed operations can be exact is the coercions go the right way. Some folks who have had bad experiences with representation error (i.e. 1.1 cannot be exactly represented as a binary float) or with round-off error (i.e. 1.0 / 7.0 must be rounded) tend to think of both binary or decimal floats as necessarily inexact. But that is not the case, exact accounting work is perfectly feasable with decimals. Remember, the notion of inexactness is a taint, not an intrinsic property of a type. Even the Scheme numeric tower recognizes this. LIkewise, the decimal specification also spells-out this notion as basic to its design. > >> Whatever we decide to do will need to be codified in a PEP though. >> "Cleaning Up Python's Numeric Tower" or something along those lines. > > The cleanup is really just specific to Decimal -- int, Fraction and > float are already properly embedded in the tower (PEP 3141 doesn't > advertise Fraction enough, since it predates it). That we may have to > change the __hash__ implementation for the other types is merely a > compromise towards efficiency. I believe that no "clean-up" is necessary. Decimal already implements the Real ABC. All that is necessary is the common __hash__ algorithm and removing the restriction between decimal/float interaction so that any two instances of Real can interoperate with one another. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Sun Mar 21 19:31:06 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 21 Mar 2010 11:31:06 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA5A333.9070404@canterbury.ac.nz> References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> <4BA55858.2000800@canterbury.ac.nz> <5c6f2a5d1003201646h11f38bc0jcbd621a6591c709c@mail.gmail.com> <4BA5A333.9070404@canterbury.ac.nz> Message-ID: <834D7F8C-2F04-4927-8333-77868535A374@gmail.com> On Mar 20, 2010, at 9:40 PM, Greg Ewing wrote: > Mark Dickinson wrote: > >> Except that float is fixed-width (typically 53 bits of precision), >> while Decimal allows a user-specified, arbitrarily large, precision; > > Yes, but it still has *some* fixed limit at any given > moment, so the result of an operation on Decimals always > has the potential to produce an inexact result. It's > not like an int or Fraction where the result can expand > to whatever size is needed. I'm thinking that I need to do more work on the Decimal documentation. There still seems to be a profound misunderstanding of its capabilities (i.e. that an Inexact flag can be set to preclude any inexact operations, that the precision can be automatically extended as needed during a calculation, or that many types of calculations can be done exactly especially if floor division is used). If rounded is needed, it can be controlled explicitly. Raymond From raymond.hettinger at gmail.com Sun Mar 21 19:45:18 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 21 Mar 2010 11:45:18 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA57C5A.9060403@gmail.com> Message-ID: <341D3926-0D43-41AA-85BE-170A67BC4169@gmail.com> On Mar 21, 2010, at 10:02 AM, Guido van Rossum wrote: > On Sun, Mar 21, 2010 at 9:53 AM, Guido van Rossum wrote: >> Which I still have to review. (Mark, if you're there, could you make a >> brief post here on the mathematical definition of the new hash you're >> proposing, and why it is both efficient to compute and good (enough) >> as a hash function?) > > Never mind, Mark. I found your explanation here: > http://codereview.appspot.com/660042/diff/19001/11009?column_width=80 I'm often dazzled by Mark's brilliance, but this is an especially nice bit of reasoning. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Sun Mar 21 19:50:50 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 21 Mar 2010 14:50:50 -0400 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA57C5A.9060403@gmail.com> Message-ID: <20100321185050.D9C051FDBF4@kimball.webabinitio.net> On Sun, 21 Mar 2010 11:25:34 -0700, Raymond Hettinger wrote: > It seems to me that Decimals and floats should be considered at > the same level (i.e. both implement Real). > > Mixed Decimal and float should coerce to Decimal because it can be > done losslessly. > > There is no need to embed a notion of "imperfect answer". > Numbers themselves are exact and many mixed operations > can be exact is the coercions go the right way. I think the concern here is rather about operations such as: 1.1 + Decimal('1.1') The calculation may produce an "exact" result, but it won't be the exact result expected, because the conversion from string (at the program text file level) to float was lossy. Thus the desire for some mechanism to know that floats and decimals have been mixed anywhere in the calculations that led up to whatever result number you are looking at. And to have doing so trigger an exception if requested by the programmer. -- R. David Murray www.bitdance.com From rhamph at gmail.com Sun Mar 21 20:21:46 2010 From: rhamph at gmail.com (Adam Olsen) Date: Sun, 21 Mar 2010 13:21:46 -0600 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA5C390.8030606@gmail.com> References: <4BA57C5A.9060403@gmail.com> <043F9D0F-AFC7-4D09-8514-0C7AD59BCD9B@gmail.com> <4BA5C390.8030606@gmail.com> Message-ID: On Sun, Mar 21, 2010 at 00:58, Nick Coghlan wrote: > I don't actually mind either way - the pragmatic tower is about coding > convenience rather than numeric purity (and mixing Fractions and > Decimals in the same algorithm is somewhat nonsensical - they're > designed for two completely different problem domains). I think the rule I've been going on is that ideal types (int, Fraction) are on one end and pragmatic types (float, complex) are on the other. Since Decimal can be used exactly it clearly bridges both groups. *However*, there's other possible types out there, and would they fit into my system? I've just taken a look at sympy and although it's clearly an ideal type, it also allows mixing with float and complex, both producing sympy types. That puts it clearly past float and complex in the tower. I have no ideal where Decimal should go. -- Adam Olsen, aka Rhamphoryncus From raymond.hettinger at gmail.com Sun Mar 21 20:31:57 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 21 Mar 2010 12:31:57 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <20100321185050.D9C051FDBF4@kimball.webabinitio.net> References: <4BA57C5A.9060403@gmail.com> <20100321185050.D9C051FDBF4@kimball.webabinitio.net> Message-ID: On Mar 21, 2010, at 11:50 AM, R. David Murray wrote: > On Sun, 21 Mar 2010 11:25:34 -0700, Raymond Hettinger wrote: >> It seems to me that Decimals and floats should be considered at >> the same level (i.e. both implement Real). >> >> Mixed Decimal and float should coerce to Decimal because it can be >> done losslessly. >> >> There is no need to embed a notion of "imperfect answer". >> Numbers themselves are exact and many mixed operations >> can be exact is the coercions go the right way. > > I think the concern here is rather about operations such as: > > 1.1 + Decimal('1.1') > > The calculation may produce an "exact" result, but it won't be the exact > result expected, because the conversion from string (at the program text > file level) to float was lossy. Thus the desire for some mechanism to > know that floats and decimals have been mixed anywhere in the calculations > that led up to whatever result number you are looking at. And to have > doing so trigger an exception if requested by the programmer. That makes sense. That's why Guido proposed a context flag in decimal to issue a warning for implicit mixing of decimals and floats. What I was talking about was a different issue. The question of where to stack decimals in the hierarchy was erroneously being steered by the concept that both decimal and binary floats are intrinsically inexact. But that would be incorrect, inexactness is a taint, the numbers themselves are always exact. I really like Guido's idea of a context flag to control whether mixing of decimal and binary floats will issue a warning. The default should be to issue the warning (because unless you know what you're doing, it is most likely an error). Raymond From greg.ewing at canterbury.ac.nz Sun Mar 21 22:47:53 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 22 Mar 2010 09:47:53 +1200 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <4BA5C3E5.6080805@gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <201003180759.25229.steve@pearwood.info> <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com> <201003182323.19458.steve@pearwood.info> <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> <4BA2DCB8.2040406@nevcal.com> <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> <4BA4165D.2070808@canterbury.ac.nz> <4BA571F0.3090800@gmail.com> <4BA5A512.5060005@canterbury.ac.nz> <4BA5C3E5.6080805@gmail.com> Message-ID: <4BA69409.4040209@canterbury.ac.nz> Nick Coghlan wrote: > That's fine - binary floats *are* surprising. That's why Decimal exists > in the first place. This argument could equally well be used the other way -- someone using Decimal is doing so precisely because they *don't* want to be surprised, in which case they would probably prefer to get an exception. The fundamental problem here is that there are two possible reasons for a mixed float-decimal operation: 1) The user is thinking in terms of floats and has happened to get a Decimal mixed in somehow. 2) The user is thinking in terms of Decimals and has happened to get a float mixed in somehow. There is no way of distinguishing between these automatically. -- Greg From tjreedy at udel.edu Sun Mar 21 23:10:44 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 21 Mar 2010 18:10:44 -0400 Subject: [Python-Dev] [RELEASED] Python 3.1.2 In-Reply-To: <1afaf6161003211012r30bc0e77ub5dfe9f248668525@mail.gmail.com> References: <1afaf6161003211012r30bc0e77ub5dfe9f248668525@mail.gmail.com> Message-ID: On 3/21/2010 1:12 PM, Benjamin Peterson wrote: > On behalf of the Python development team, I'm joyful to announce the second > bugfix release of the Python 3.1 series, Python 3.1.2. Thanks for the work. > This bug fix release fixes numerous issues found in 3.1.1, and is considered a > production release. > > The Python 3.1 version series focuses on the stabilization and optimization of > the features and changes that Python 3.0 introduced. For example, the new I/O > system has been rewritten in C for speed. File system APIs that use unicode > strings now handle paths with undecodable bytes in them. Other features include > an ordered dictionary implementation, a condensed syntax for nested with > statements, and support for ttk Tile in Tkinter. For a more extensive list of > changes in 3.1, see http://doc.python.org/3.1/whatsnew/3.1.html or Misc/NEWS in > the Python distribution. > > To download Python 3.1.2 visit: > > http://www.python.org/download/releases/3.1.2/ > > A list of changes in 3.1.2 can be found here: > > http://svn.python.org/projects/python/tags/r312/Misc/NEWS There seem to be about a hundred fixes. > The 3.1 documentation can be found at: > > http://docs.python.org/3.1 > > Bugs can always be reported to: > > http://bugs.python.org From greg.ewing at canterbury.ac.nz Sun Mar 21 23:41:49 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 22 Mar 2010 10:41:49 +1200 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA57C5A.9060403@gmail.com> Message-ID: <4BA6A0AD.7090207@canterbury.ac.nz> Raymond Hettinger wrote: > > Remember, the notion of inexactness is a taint, > not an intrinsic property of a type. Even the Scheme numeric > tower recognizes this. LIkewise, the decimal specification also > spells-out this notion as basic to its design. I'm not sure it really does, otherwise every decimal value would have a flag indicating whether it was tainted with inexactness, and this flag would propagate through calculations. -- Greg From steve at pearwood.info Sun Mar 21 23:32:20 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 22 Mar 2010 09:32:20 +1100 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <4BA69409.4040209@canterbury.ac.nz> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <4BA5C3E5.6080805@gmail.com> <4BA69409.4040209@canterbury.ac.nz> Message-ID: <201003220932.22259.steve@pearwood.info> On Mon, 22 Mar 2010 08:47:53 am Greg Ewing wrote: > Nick Coghlan wrote: > > That's fine - binary floats *are* surprising. That's why Decimal > > exists in the first place. > > This argument could equally well be used the other way -- > someone using Decimal is doing so precisely because they > *don't* want to be surprised, in which case they would > probably prefer to get an exception. Then they're in for a terrible, terrible disappointment. Rounding issues don't go away because you're using Decimal instead of float, and I can't imagine anyone would like an exception in the following cases: >>> Decimal(1)/Decimal(3)*Decimal(3) == Decimal(1) False >>> Decimal(2).sqrt()**Decimal(2) == Decimal(2) False >>> Decimal(10**28)+Decimal(1)-Decimal(10**28) == Decimal(1) False Rounding isn't the only surprise: >>> x = Decimal("NAN"); x == x False Decimals are floats, but using radix 10 instead of radix 2. -- Steven D'Aprano From guido at python.org Sun Mar 21 23:35:51 2010 From: guido at python.org (Guido van Rossum) Date: Sun, 21 Mar 2010 15:35:51 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA57C5A.9060403@gmail.com> Message-ID: On Sun, Mar 21, 2010 at 11:25 AM, Raymond Hettinger wrote: > Right. ?We should be guided by: > ?? ?fractions are a superset of decimals which are a superset of binary floats. But mixed Fraction-float operations return floats, not Fractions. > And by: > ?? ?binary floats and decimal floats both implement all of the operations > ?? ?for the Real abstract base class. Sure, but that doesn't help us decide what mixed Decimal-float operations should return. > It seems to me that Decimals and floats should be considered at > the same level (i.e. both implement Real). Agreed, but doesn't help. (Except against the idea that Decimal goes on the "integer" side of Fraction, which is just wrong.) > Mixed Decimal and float should coerce to Decimal because it can be > done losslessly. But mixed Fraction-float returns float even though returning Fraction could be done losslessly. The real criterion should be what's more useful, not what can be done losslessly. > There is no need to embed a notion of "imperfect answer". > Numbers themselves are exact and many mixed operations > can be exact if the coercions go the right way. Division cannot, in general (I consider floor division a bastard child of the integers). And for multiplication it seems that rounding at some point becomes necessary since the alternative would be to use infinite precision. > Some folks who have had bad experiences with representation > error (i.e. 1.1 cannot be exactly represented as a binary float) or > with round-off error (i.e. 1.0 / 7.0 must be rounded) tend to think > of both binary or decimal floats as necessarily inexact. ?But that > is not the case, exact accounting work is perfectly feasable > with decimals. ?Remember, the notion of inexactness is a taint, > not an intrinsic property of a type. ?Even the Scheme numeric > tower recognizes this. ?LIkewise, the decimal specification also > spells-out this notion as basic to its design. I really don't think advertising Decimal as having exact operations is the right thing to do. Sure, it is the right data type for all accounting operations -- but that is a very specialized use case, where certain round-off errors are desirable (since nobody wants fractional pennies in their bill). > I believe that no "clean-up" is necessary. ?Decimal already implements > the Real ABC. ? All that is necessary is the common __hash__ algorithm > and removing the restriction between decimal/float interaction so that > any two instances of Real can interoperate with one another. Call it by any name you want. We're looking at revising the hash function and allowing mixed operations between Decimal and float, with signal that warns about these operations. I see two open issues: whether mixed Decimal-float operations should return Decimal and float, and whether the warning about such operations should be on or off by default. My gut tells me that the signal should be off by default. My gut doesn't say much about whether the result should be float or Decimal, but I want the reasoning to reach a decision to be sound. -- --Guido van Rossum (python.org/~guido) From greg.ewing at canterbury.ac.nz Sun Mar 21 23:50:26 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 22 Mar 2010 10:50:26 +1200 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> <4BA55858.2000800@canterbury.ac.nz> <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> Message-ID: <4BA6A2B2.6050207@canterbury.ac.nz> Raymond Hettinger wrote: > Since decimal also allows arbitrary sizes, all long ints can be > exactly represented (this was even one of the design goals > for the decimal module). There may be something we need to clarify here. I've been imagining that the implicit conversions to Decimal that we're talking about would be done to whatever precision is set in the context. Am I wrong about that? Is the intention to always use enough digits to get an exact representation? -- Greg From greg.ewing at canterbury.ac.nz Mon Mar 22 00:07:53 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 22 Mar 2010 11:07:53 +1200 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA57C5A.9060403@gmail.com> <20100321185050.D9C051FDBF4@kimball.webabinitio.net> Message-ID: <4BA6A6C9.3020708@canterbury.ac.nz> Raymond Hettinger wrote: > The question of > where to stack decimals in the hierarchy was erroneously > being steered by the concept that both decimal and binary floats > are intrinsically inexact. But that would be incorrect, inexactness > is a taint, the numbers themselves are always exact. I don't think that's correct. "Numbers are always exact" is a simplification due to choosing not to attach an inexactness flag to every value. Without such a flag, we don't really know whether any given value is exact or not, we can only guess. The reason for regarding certain types as "implicitly inexact" is something like this: If you start with exact ints, and do only int operations with them, you must end up with exact ints. But the same is not true of float or Decimal: even if you start with exact values, you can end up with inexact ones. > I really like Guido's idea of a context flag to control whether > mixing of decimal and binary floats will issue a warning. Personally I feel that far too much stuff concerning decimals is controlled by implicit context parameters. It gives me the uneasy feeling that I don't know what the heck any given decimal operation is going to do. It's probably justified in this case, though. -- Greg From greg.ewing at canterbury.ac.nz Mon Mar 22 00:08:11 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 22 Mar 2010 11:08:11 +1200 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA57C5A.9060403@gmail.com> <20100321185050.D9C051FDBF4@kimball.webabinitio.net> Message-ID: <4BA6A6DB.3010801@canterbury.ac.nz> Raymond Hettinger wrote: > The question of > where to stack decimals in the hierarchy was erroneously > being steered by the concept that both decimal and binary floats > are intrinsically inexact. But that would be incorrect, inexactness > is a taint, the numbers themselves are always exact. I don't think that's correct. "Numbers are always exact" is a simplification due to choosing not to attach an inexactness flag to every value. Without such a flag, we don't really know whether any given value is exact or not, we can only guess. The reason for regarding certain types as "implicitly inexact" is something like this: If you start with exact ints, and do only int operations with them, you must end up with exact ints. But the same is not true of float or Decimal: even if you start with exact values, you can end up with inexact ones. > I really like Guido's idea of a context flag to control whether > mixing of decimal and binary floats will issue a warning. Personally I feel that far too much stuff concerning decimals is controlled by implicit context parameters. It gives me the uneasy feeling that I don't know what the heck any given decimal operation is going to do. It's probably justified in this case, though. -- Greg From raymond.hettinger at gmail.com Sun Mar 21 23:58:14 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 21 Mar 2010 15:58:14 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA57C5A.9060403@gmail.com> Message-ID: <7FBF6CE1-9132-47BF-8397-76EFD6C63712@gmail.com> On Mar 21, 2010, at 3:35 PM, Guido van Rossum wrote: > >> It seems to me that Decimals and floats should be considered at >> the same level (i.e. both implement Real). > > Agreed, but doesn't help. (Except against the idea that Decimal goes > on the "integer" side of Fraction, which is just wrong.) Woohoo! Glad you agree. I was concerned that idea was gathering a following. > >> There is no need to embed a notion of "imperfect answer". >> Numbers themselves are exact and many mixed operations >> can be exact if the coercions go the right way. > > Division cannot, in general (I consider floor division a bastard child > of the integers). And for multiplication it seems that rounding at > some point becomes necessary since the alternative would be to use > infinite precision. Perception here is probably dictated by the use case of the beholder :-) In the accounting world, such operations are very common (part of getting three apples for one dollar recorded as 0.33, 0.33, and 0.34 in the individual apple accounts). Cost allocation tools use floor division all the time. Amortization computations always have to take into account that real payments and interest charges do not have fractional pennies (or sometimes fractional dollars) and have to readjust the amortization accordingly (usually with small roundings to the interest charges and a small adjustment to the final payment). > Call it by any name you want. We're looking at revising the hash > function and allowing mixed operations between Decimal and float, with > signal that warns about these operations. I see two open issues: > whether mixed Decimal-float operations should return Decimal and > float, and whether the warning about such operations should be on or > off by default. My gut tells me that the signal should be off by > default. My gut doesn't say much about whether the result should be > float or Decimal, but I want the reasoning to reach a decision to be > sound. My vote is for: decimal + float --> decimal and to emit a warning by default. Reasoning for coercion to decimal: 1) decimal + float can be done losslessly and meaningfully 2) decimal/float comparisons can give a useful and non-confusing result for: 1.1 == Decimal('1.1') This needs to return False since the two values: a) have different hash values (under either proposal) and b) are in-fact, not equal But, if the decimal were coerced to a binary float, the relation would return True: >>> float(decimal.Decimal('1.1')) == 1.1 True This is bad. Reasoning for emitting a warning by default: 1) Real actual use cases for mixed decimal / float operations are rare. 2) But accidental mixed decimal / float is an easy mistake to make. 3) Implicit coercion hides the error. 4) A warning flag gives you a chance to catch your error. 5) A warning is educational (it makes sure that you understand what your program is doing) 6). A warning is easily silenced either through a) the warnings module, b) setting a context flag in decimal, or c) by making the coercion explicit using Decimal.from_float(). Raymond From steve at pearwood.info Sun Mar 21 23:59:53 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 22 Mar 2010 09:59:53 +1100 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <20100321185050.D9C051FDBF4@kimball.webabinitio.net> Message-ID: <201003220959.53306.steve@pearwood.info> On Mon, 22 Mar 2010 06:31:57 am Raymond Hettinger wrote: > I really like Guido's idea of a context flag to control whether > mixing of decimal and binary floats will issue a warning. > The default should be to issue the warning (because unless > you know what you're doing, it is most likely an error). When you say "warning", do you mean warning.warn(), or an exception? I'd like to put in a vote for allowing naive users with low requirements for accuracy and precision to be able to type something like this in the interactive interpreter: >>> Decimal(1) + 1.0 and get two (in whatever type is decided on) without having to change the context or deal with an exception. Yes, this means that they may be surprised if they perform an operation which suffers from rounding errors, but that's no worse than what happens with floats. If naive users are going to use the interpreter as a calculator, they're going to start off using floats and ints simply because they require less typing. My idea is to allow a gentle learning curve with Decimal (and Fraction) without scaring them off with exceptions or excessive warnings: a single warning per session would be okay, a warning after every operation would be excessive in my opinion, and exceptions by default would be right out. -- Steven D'Aprano From greg.ewing at canterbury.ac.nz Mon Mar 22 00:12:25 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 22 Mar 2010 11:12:25 +1200 Subject: [Python-Dev] Decimal & amp; lt; -& amp; gt; float comparisons in py3k. In-Reply-To: <201003220932.22259.steve@pearwood.info> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <4BA5C3E5.6080805@gmail.com> <4BA69409.4040209@canterbury.ac.nz> <201003220932.22259.steve@pearwood.info> Message-ID: <4BA6A7D9.6030502@canterbury.ac.nz> Steven D'Aprano wrote: > Then they're in for a terrible, terrible disappointment. Rounding issues > don't go away because you're using Decimal instead of float, No, but what I mean is that they prefer to be surprised in unsurprising ways, so to speak. Everyone knows that floating point numbers have limited precision. What really surprises people is when *binary* floating point numbers behave differently from the decimal ones they're used to. -- Greg From raymond.hettinger at gmail.com Mon Mar 22 00:11:13 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 21 Mar 2010 16:11:13 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA6A2B2.6050207@canterbury.ac.nz> References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> <4BA55858.2000800@canterbury.ac.nz> <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> Message-ID: <465AE465-278C-4627-B285-BACF62FB731A@gmail.com> On Mar 21, 2010, at 3:50 PM, Greg Ewing wrote: > Raymond Hettinger wrote: > >> Since decimal also allows arbitrary sizes, all long ints can be >> exactly represented (this was even one of the design goals >> for the decimal module). > > There may be something we need to clarify here. I've been > imagining that the implicit conversions to Decimal that > we're talking about would be done to whatever precision > is set in the context. Am I wrong about that? Yes. The conversion to decimal is independent of the current context. The background docs for the decimal module makes its design intention clear. All numbers are exact. Rounding and context adjustments only apply to the *results (of operations. The module is designed that way. Its background documents confirm that viewpoint. And its operations are designed to support that world view (i.e. unary plus is an operation that can change a value). Also, we confirmed that point-of-view with the person who wrote the spec. > Is the intention > to always use enough digits to get an exact representation? Yes. That is in-fact what Decimal.fromfloat() does. That agrees with the decimal constructor itself which is not context sensitive: >>> decimal.getcontext().prec = 5 >>> decimal.Decimal('3.1415926535') # Notice this value doesn't get rounded Decimal('3.1415926535') >>> decimal.Decimal('3.1415926535') + 0 # This result does get rounded. Decimal('3.1416') >>> # Also, rounding does not get applied during an equality check >>> decimal.getcontext().prec = 5 >>> decimal.Decimal('3.1415926535') == decimal.Decimal('3.1416') False Raymond P.S. Thanks for asking the question. From raymond.hettinger at gmail.com Mon Mar 22 00:16:00 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 21 Mar 2010 16:16:00 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <201003220959.53306.steve@pearwood.info> References: <20100321185050.D9C051FDBF4@kimball.webabinitio.net> <201003220959.53306.steve@pearwood.info> Message-ID: <2042CE43-567E-43AB-A76B-23CE410F56DC@gmail.com> On Mar 21, 2010, at 3:59 PM, Steven D'Aprano wrote: > On Mon, 22 Mar 2010 06:31:57 am Raymond Hettinger wrote: >> I really like Guido's idea of a context flag to control whether >> mixing of decimal and binary floats will issue a warning. >> The default should be to issue the warning (because unless >> you know what you're doing, it is most likely an error). > > When you say "warning", do you mean warning.warn(), or an exception? I'm not sure I understand your question. I did mean warnings.warn(). But that does raise a catchable exception or it can be suppressed through the warnings module. It should probably be set to warn no more than once. Raymond From guido at python.org Mon Mar 22 02:24:26 2010 From: guido at python.org (Guido van Rossum) Date: Sun, 21 Mar 2010 18:24:26 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <2042CE43-567E-43AB-A76B-23CE410F56DC@gmail.com> References: <20100321185050.D9C051FDBF4@kimball.webabinitio.net> <201003220959.53306.steve@pearwood.info> <2042CE43-567E-43AB-A76B-23CE410F56DC@gmail.com> Message-ID: On Sun, Mar 21, 2010 at 4:16 PM, Raymond Hettinger wrote: > > On Mar 21, 2010, at 3:59 PM, Steven D'Aprano wrote: > >> On Mon, 22 Mar 2010 06:31:57 am Raymond Hettinger wrote: >>> I really like Guido's idea of a context flag to control whether >>> mixing of decimal and binary floats will issue a warning. >>> The default should be to issue the warning (because unless >>> you know what you're doing, it is most likely an error). >> >> When you say "warning", do you mean warning.warn(), or an exception? > > > I'm not sure I understand your question. ?I did mean warnings.warn(). > But that does raise a catchable exception or it can be suppressed > through the warnings module. ?It should probably be set to > warn no more than once. I would hope it could use whatever mechanism is already used for other conditions in the decimal module such as Underflow, Inexact, Rounded etc. But I have to admit I don't know exactly what those do. It appears they can either raise an exception or call a handle() method on the given exception. Are you thinking of putting the warn() call inside that handle() method? -- --Guido van Rossum (python.org/~guido) From raymond.hettinger at gmail.com Mon Mar 22 04:10:31 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 21 Mar 2010 20:10:31 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <20100321185050.D9C051FDBF4@kimball.webabinitio.net> <201003220959.53306.steve@pearwood.info> <2042CE43-567E-43AB-A76B-23CE410F56DC@gmail.com> Message-ID: <9C6B548B-E117-4FA9-9023-915E66D9C22F@gmail.com> On Mar 21, 2010, at 6:24 PM, Guido van Rossum wrote: > On Sun, Mar 21, 2010 at 4:16 PM, Raymond Hettinger > wrote: >> >> On Mar 21, 2010, at 3:59 PM, Steven D'Aprano wrote: >> >>> On Mon, 22 Mar 2010 06:31:57 am Raymond Hettinger wrote: >>>> I really like Guido's idea of a context flag to control whether >>>> mixing of decimal and binary floats will issue a warning. >>>> The default should be to issue the warning (because unless >>>> you know what you're doing, it is most likely an error). >>> >>> When you say "warning", do you mean warning.warn(), or an exception? >> >> >> I'm not sure I understand your question. I did mean warnings.warn(). >> But that does raise a catchable exception or it can be suppressed >> through the warnings module. It should probably be set to >> warn no more than once. > > I would hope it could use whatever mechanism is already used for other > conditions in the decimal module such as Underflow, Inexact, Rounded > etc. But I have to admit I don't know exactly what those do. It > appears they can either raise an exception or call a handle() method > on the given exception. Are you thinking of putting the warn() call > inside that handle() method? Yes. Raymond From rhamph at gmail.com Mon Mar 22 06:51:47 2010 From: rhamph at gmail.com (Adam Olsen) Date: Sun, 21 Mar 2010 23:51:47 -0600 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <201003220959.53306.steve@pearwood.info> References: <20100321185050.D9C051FDBF4@kimball.webabinitio.net> <201003220959.53306.steve@pearwood.info> Message-ID: On Sun, Mar 21, 2010 at 16:59, Steven D'Aprano wrote: > If naive users are going to use the interpreter as a calculator, they're > going to start off using floats and ints simply because they require > less typing. My idea is to allow a gentle learning curve with Decimal > (and Fraction) without scaring them off with exceptions or excessive > warnings: a single warning per session would be okay, a warning after > every operation would be excessive in my opinion, and exceptions by > default would be right out. That strikes me as a passive-aggressive way of saying we tolerate it for interactive use, but don't you dare mix them for real programs. A warning should be regarded as a bug in real programs ? unless it's a transitional measure ? so it might as well be an exception. Don't guess and all that. -- Adam Olsen, aka Rhamphoryncus From dickinsm at gmail.com Mon Mar 22 10:23:00 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 22 Mar 2010 09:23:00 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA6A2B2.6050207@canterbury.ac.nz> References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> <4BA55858.2000800@canterbury.ac.nz> <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> Message-ID: <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> On Sun, Mar 21, 2010 at 10:50 PM, Greg Ewing wrote: > Raymond Hettinger wrote: > >> Since decimal also allows arbitrary sizes, all long ints can be >> exactly represented (this was even one of the design goals >> for the decimal module). > > There may be something we need to clarify here. I've been > imagining that the implicit conversions to Decimal that > we're talking about would be done to whatever precision > is set in the context. Am I wrong about that? Is the intention > to always use enough digits to get an exact representation? I've been thinking about this, too. Currently, Decimal + integer -> Decimal converts the integer losslessly, with no reference to the Decimal context. But the Fraction type is going to mess this up: for Decimal + Fraction -> Decimal, I don't see any other sensible option than to convert the Fraction using the current context, since lossless conversion isn't generally possible. So with the above two conventions, we'd potentially end up with Decimal('1.23') + 314159 giving a different result from Decimal('1.23') + Fraction(314159, 1) (depending on the context). It may be that we should reconsider Decimal + int interactions, making the implicit int->Decimal conversion lossy. Decimal would then match the way that float behaves: float + int and float + Fraction both do a lossy conversion of the non-Fraction argument to Fraction. But then we're changing established behaviour of the Decimal module, which could cause problems for existing users. Note that comparisons are a separate issue: those always need to be done exactly (at least for equality, and once you're doing it for equality it makes sense to make the other comaprisons exact as well), else the rule that x == y implies hash(x) == hash(y) would become untenable. Again, this is the pattern that already exists for int<->float and Fraction<->float interactions: comparisons are exact, but arithmetic operations involve a lossy conversion. Mark From ncoghlan at gmail.com Mon Mar 22 11:32:35 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 22 Mar 2010 20:32:35 +1000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> <4BA55858.2000800@canterbury.ac.nz> <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> Message-ID: <4BA74743.90204@gmail.com> Mark Dickinson wrote: > But the Fraction type is going to mess this up: for Decimal + > Fraction -> Decimal, I don't see any other sensible option than to > convert the Fraction using the current context, since lossless > conversion isn't generally possible. Be able to duck this question was precisely why I put Decimal to the left of Fraction in my pragmatic tower, btw. There are all sorts of good reasons why that order was mathematically wrong (Real vs Rational, representability of NaN/Inf, etc), but from the point of view of providing clearly defined behaviour for an atypical operation, it should be a lot easier. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Mon Mar 22 11:41:10 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 22 Mar 2010 20:41:10 +1000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA6A0AD.7090207@canterbury.ac.nz> References: <4BA57C5A.9060403@gmail.com> <4BA6A0AD.7090207@canterbury.ac.nz> Message-ID: <4BA74946.9060302@gmail.com> Greg Ewing wrote: > Raymond Hettinger wrote: >> >> Remember, the notion of inexactness is a taint, >> not an intrinsic property of a type. Even the Scheme numeric >> tower recognizes this. LIkewise, the decimal specification also >> spells-out this notion as basic to its design. > > I'm not sure it really does, otherwise every decimal > value would have a flag indicating whether it was > tainted with inexactness, and this flag would propagate > through calculations. http://docs.python.org/library/decimal.html#decimal.Inexact (Part of the thread context rather than the individual decimal values, but if you use it properly it tells you whenever an inexact operation has occurred in the current thread) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Mon Mar 22 11:51:33 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 22 Mar 2010 20:51:33 +1000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <7FBF6CE1-9132-47BF-8397-76EFD6C63712@gmail.com> References: <4BA57C5A.9060403@gmail.com> <7FBF6CE1-9132-47BF-8397-76EFD6C63712@gmail.com> Message-ID: <4BA74BB5.8030807@gmail.com> Raymond Hettinger wrote: > On Mar 21, 2010, at 3:35 PM, Guido van Rossum wrote: >>> It seems to me that Decimals and floats should be considered at >>> the same level (i.e. both implement Real). >> Agreed, but doesn't help. (Except against the idea that Decimal goes >> on the "integer" side of Fraction, which is just wrong.) > > Woohoo! Glad you agree. > I was concerned that idea > was gathering a following. Heck no, it was just a random late night thought from me, and even I thought it was a somewhat dubious idea. I don't mind at all that it since has been knocked soundly (and deservedly) on the head :) > Reasoning for emitting a warning by default: > > 1) Real actual use cases for mixed decimal / float operations are rare. > 2) But accidental mixed decimal / float is an easy mistake to make. > 3) Implicit coercion hides the error. > 4) A warning flag gives you a chance to catch your error. > 5) A warning is educational (it makes sure that you understand > what your program is doing) > 6). A warning is easily silenced either through a) the warnings module, > b) setting a context flag in decimal, or c) by making the coercion explicit > using Decimal.from_float(). I'll add another one to that list: (7) For backwards compatible changes, it is easy to go from exception -> warning -> no warning (if we later decide to take that second step). Going from exception -> no warning -> warning (if we were to change our minds the other way) is a lot less user-friendly. (I was going to try to play devil's advocate and argue in favour of float results and/or no warning, but I got nuthin' - Raymond's points made too much sense to me). A warning is nice in that you can mix decimals and floats at the interpreter prompt with a single warning per session, but the warning can still act as a pointer into the weird and wonderful world of binary vs decimal floating point. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Mon Mar 22 14:56:04 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 22 Mar 2010 09:56:04 -0400 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask Message-ID: <20100322095604.3371039e@heresy> I have a pretty good start on PEP 3147 implementation [1], but I've encountered a situation that I'd like to get some feedback on. Here's the test case illustrating the problem. From test_import.py: def test_writable_directory(self): # The umask is not conducive to creating a writable __pycache__ # directory. with umask(0o222): __import__(TESTFN) self.assertTrue(os.path.exists('__pycache__')) self.assertTrue(os.path.exists(os.path.join( '__pycache__', '{}.{}.pyc'.format(TESTFN, self.tag)))) The __pycache__ directory does not exist before the import, and the import machinery creates the directory, but the umask leaves the directory unwritable by anybody. So of course when the import machinery goes to write the .pyc file inside __pycache__, it fails. This does not cause an ImportError though, just like if today the package directory were unwritable. This might be different than today's situation though because once the unwritable __pycache__ directory is created, nothing is going to change that without explicit user interaction, and that might be difficult after the fact. I'm not sure what the right answer is. Some possible choices: * Tough luck * Force the umask so that the directory is writable, but then the question is, by whom? ugo+w or something less? * Copy the permissions from the parent directory and ignore umask * Raise an exception or refuse to create __pycache__ if it's not writable (again, by whom?) Or maybe you have a better idea? What's the equivalent situation on Windows and how would things work there? -Barry [1] https://edge.launchpad.net/~barry/python/pep3147 P.S. I'm down to only 8 unit test failures. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From solipsis at pitrou.net Mon Mar 22 15:30:12 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 22 Mar 2010 14:30:12 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?PEP_3147=2C_=5F=5Fpycache=5F=5F_directorie?= =?utf-8?q?s_and_umask?= References: <20100322095604.3371039e@heresy> Message-ID: Barry Warsaw python.org> writes: > > I'm not sure what the right answer is. Some possible choices: > > * Tough luck > * Force the umask so that the directory is writable, but then the question is, > by whom? ugo+w or something less? > * Copy the permissions from the parent directory and ignore umask > * Raise an exception or refuse to create __pycache__ if it's not writable > (again, by whom?) Welcome to a problem PHP has known for years. The problem with solution 3 is that a non-root user can't change the owner of a directory, therefore copying the permissions doesn't ensure the directory will be writable by the right users. Solution 2 (chmod 777) is obviously too laxist. Anyone could put fake bytecode files in a root-owned application. Less laxist chmods have the same problem as solution 3, and can still be insecure. I think solution 4 would be the best compromise. In the face of ambiguity, refuse to guess. But, rather than : refuse to create __pycache__ if it's not writable, the test should be : refuse to create __pycache__ if I can't create it with the same ownership and permissions as the parent directory (and raise an ImportWarning). In light of this issue, I'm -0.5 on __pycache__ becoming the default caching mechanism. The directory ownership/permissions issue is too much of a mess, especially for Web applications (think __pycache__ files created by the Apache user). __pycache__ should only be created if explicitly activated (for example by distutils when installing stuff). Otherwise, if not present, the "legacy" mechanism (writing an untagged pyc file along the py file) should be used. Actually, __pycache__ creation doesn't have to be part of the import mechanism. It can be part of distutils instead (or whatever third-party tool such as distribute, or distro-specific packaging scripts). This would relax complexity of core Python a bit. Regards Antoine. From barry at python.org Mon Mar 22 16:21:08 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 22 Mar 2010 11:21:08 -0400 Subject: [Python-Dev] __file__ and bytecode-only In-Reply-To: <4B9F52FB.2090701@gmail.com> References: <20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com> <20100312145312.1f0a0aa0@heresy.wooz.org> <4B9AC4CA.3070302@voidspace.org.uk> <20100312175657.287bb456@heresy.wooz.org> <4B9B9E65.4080902@gmail.com> <20100315111654.1aa7acf5@heresy.wooz.org> <4B9F52FB.2090701@gmail.com> Message-ID: <20100322112108.68d47556@heresy> On Mar 16, 2010, at 07:44 PM, Nick Coghlan wrote: >In Python 3.1, *invoking* py_compile.compile() will create 2.x style >bytecode. Similarly, when force==False, compileall.compile_dir() and >compileall.compile_path() will check for 2.x style bytecode in order to >decide whether or not to compile the module. > >The question for 3.2 is what bytecode layout py_compile.compile() should >generate. For the precompile-a-system-library use case it should clearly >generate a PEP 3147 layout and this probably makes sense as the default >behaviour in 3.2. > >However, for production of bytecode-only packages, it would be >convenient to be able to explicitly invoke the 2.x style behaviour >without having to specify the target filename explicitly using the >'cfile' parameter (which isn't exposed at the compileall layer anyway). My working branch modifies py_compile to produce PEP 3147 layout by default. I agree that it should support traditional pyc output as an option, and compileall should support this as well. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From martin at v.loewis.de Mon Mar 22 16:52:03 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 22 Mar 2010 16:52:03 +0100 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: <20100322095604.3371039e@heresy> References: <20100322095604.3371039e@heresy> Message-ID: <4BA79223.1030806@v.loewis.de> > Or maybe you have a better idea? What's the equivalent situation on Windows > and how would things work there? On Windows, this problem is easy: create the directory with no specification of an ACL, and it will (usually) inherit the ACL of the parent directory. IOW, the same users will have write access to __pycache__ as to the parent directory. It is possible to defeat this machinery, by setting ACL entries on the parent that explicitly don't get inherited, or that only apply to subdirectories. In this case, I would say "tough luck": this is what the users requested, so it may have a reason. Regards, Martin From martin at v.loewis.de Mon Mar 22 16:58:47 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 22 Mar 2010 16:58:47 +0100 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: References: <20100322095604.3371039e@heresy> Message-ID: <4BA793B7.3060209@v.loewis.de> > refuse to create __pycache__ if I can't create it with the same ownership > and permissions as the parent directory (and raise an ImportWarning). I don't think an ImportWarning should be raised: AFAICT, we are not raising one, either, when the pyc file cannot be created (and it may well be the case that you don't have write permission at all, which would trigger lots of ImportWarnings). Regards, Martin From raymond.hettinger at gmail.com Mon Mar 22 17:39:23 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 22 Mar 2010 09:39:23 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> <4BA55858.2000800@canterbury.ac.nz> <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> Message-ID: <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> On Mar 22, 2010, at 2:23 AM, Mark Dickinson wrote: > Note that comparisons are a separate issue: those always need to be > done exactly (at least for equality, and once you're doing it for > equality it makes sense to make the other comaprisons exact as well), > else the rule that x == y implies hash(x) == hash(y) would become > untenable. Again, this is the pattern that already exists for > int<->float and Fraction<->float interactions: comparisons are exact, > but arithmetic operations involve a lossy conversion. My instinct says that we're asking for trouble if comparisons have different coercion rules than arithmetic operations. That would suggest that we follow the chain of lossless conversions: Fraction + float --> Fraction Fraction + decimal --> Fraction Decimal + float --> Decimal That way, arithmetic coercions match comparison coercions. We preserve get Mark's fast, clean new universal hash function. The equality/hash homomorphism is preserved. And a decimal context is not needed for any of the coercions (it is needed for the addition between two decimals once coercion has happened, but that is completely normal for Decimal so there are no surprises). Raymond From guido at python.org Mon Mar 22 18:00:34 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 22 Mar 2010 09:00:34 -0800 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> References: <4BA55858.2000800@canterbury.ac.nz> <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> Message-ID: On Mon, Mar 22, 2010 at 8:39 AM, Raymond Hettinger wrote: > My instinct says that we're asking for trouble if comparisons have > different coercion rules than arithmetic operations. Sorry, no, this is a station we passed long ago when float-long comparison was fixed to do the comparison exactly (but other mixed ops continue to convert the long to a float). Otherwise you'll get a violation of "if x < y and y < z then it follows that x < z". (FWIW, my instinct used to be the same as yours, but I got better. :-) > That would suggest that we follow the chain of lossless conversions: > > ? Fraction + float --> Fraction Besides the obvious backwards compatibility problem, I also just really think that this would be a mistake. Fraction corresponds to Rational in PEP 3141's numeric tower and float corresponds to Real (as does Decimal). The convention is to only move to the more general type in the numeric tower. Now we all know that floats (being a fixed number of bits) are actually representable as rationals as well, but *that's not how we think about them!* > ? Fraction + decimal --> Fraction This also violates the natural direction in the numeric tower. > ? Decimal + float --> Decimal If everybody associated with the Decimal implementation wants this I won't stop you; as I repeatedly said my intuition about this one (as opposed to the other two above) is very weak. > That way, arithmetic coercions match comparison coercions. Which is not a requirement. > We preserve get Mark's fast, clean new universal hash function. That hash function (or another that matches the requirements) is a given. > The equality/hash homomorphism is preserved. That is also a given (except I don't know if homomorphism is the right word to use). > And a decimal context is not needed for any of the coercions > (it is needed for the addition between two decimals once coercion > has happened, but that is completely normal for Decimal so there > are no surprises). Doesn't strike me as a deciding argument. -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Mon Mar 22 18:14:29 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 22 Mar 2010 17:14:29 +0000 (UTC) Subject: [Python-Dev] __pycache__ creation References: <20100322095604.3371039e@heresy> Message-ID: Oh, and by the way, there can be a race condition between __pycache__ creation and deletion (if it fails the test), where an attacker can stuff a hostile pyc file in the directory in the meantime (and the deletion then fails because the directory isn't empty). IMO, all these issues militate for putting __pycache__ creation out of the interpreter core, and in the hands of third-party package-time/ install-time tools (or distutils). Le Mon, 22 Mar 2010 14:30:12 +0000, Antoine Pitrou a ?crit?: > > __pycache__ should only be created if explicitly activated (for example > by distutils when installing stuff). Otherwise, if not present, the > "legacy" mechanism (writing an untagged pyc file along the py file) > should be used. > > Actually, __pycache__ creation doesn't have to be part of the import > mechanism. It can be part of distutils instead (or whatever third-party > tool such as distribute, or distro-specific packaging scripts). This > would relax complexity of core Python a bit. > > Regards > > Antoine. From raymond.hettinger at gmail.com Mon Mar 22 18:56:41 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 22 Mar 2010 10:56:41 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA55858.2000800@canterbury.ac.nz> <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> Message-ID: <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote: > >> Decimal + float --> Decimal > > If everybody associated with the Decimal implementation wants this I > won't stop you; as I repeatedly said my intuition about this one (as > opposed to the other two above) is very weak. That's my vote. I believe Nick chimed-in in agreement. Mark, do you concur? Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From ijmorlan at uwaterloo.ca Mon Mar 22 18:28:25 2010 From: ijmorlan at uwaterloo.ca (Isaac Morland) Date: Mon, 22 Mar 2010 13:28:25 -0400 (EDT) Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> Message-ID: On Mon, 22 Mar 2010, Antoine Pitrou wrote: > Oh, and by the way, there can be a race condition between __pycache__ > creation and deletion (if it fails the test), where an attacker can stuff > a hostile pyc file in the directory in the meantime (and the deletion > then fails because the directory isn't empty). Would creating it under a different name and then renaming help with this? > IMO, all these issues militate for putting __pycache__ creation out of > the interpreter core, and in the hands of third-party package-time/ > install-time tools (or distutils). Speaking only for myself, but really for anybody who likes tidy source directories, I hope some version of the __pycache__ proposal becomes part of standard Python, by which I ideally mean it's enabled by default but if that is just not a good idea then at most it should be required to set a command-line option to get this feature. If I just want to write some .py code and run it, I don't see why my directories need to clutter up with .pyc files. I've previously suggested a Python version of javac's -d ("destination directory") option, but putting all the .pyc's in a __pycache__ directory per source directory is good enough to make me happy (and is Pythonically simple, in my opinion). Isaac Morland CSCF Web Guru DC 2554C, x36650 WWW Software Specialist From chambon.pascal at wanadoo.fr Mon Mar 22 19:06:44 2010 From: chambon.pascal at wanadoo.fr (Pascal Chambon) Date: Mon, 22 Mar 2010 19:06:44 +0100 Subject: [Python-Dev] Attribute lookup ambiguity In-Reply-To: <4BA576AF.2050208@voidspace.org.uk> References: <4BA3C962.7080500@wanadoo.fr> <4BA3DD4F.1080209@voidspace.org.uk> <4BA4B8DE.8060109@wanadoo.fr> <4BA576AF.2050208@voidspace.org.uk> Message-ID: <4BA7B1B4.4010000@wanadoo.fr> Michael Foord a ?crit : > On 20/03/2010 12:00, Pascal Chambon wrote: >> >> >> But the point which for me is still unclear, is : does the default >> implementation of __getattribute__ (the one of "object") call >> __getattr__ by himself, or does it rely on its caller for that, by >> raising an AttributeError ? For Python2, it's blatantly the latter >> case which is favoured, but since it looks like an implementation >> detail at the moment, I propose we settle it (and document it) once >> for all. > > Ah right, my apologies. So it is still documented behaviour - > __getattr__ is obviously called by the Python runtime and not by > __getattribute__. (It isn't just by getattr as the same behaviour is > shown when doing a normal attribute lookup and not via the getattr > function.) > I really don't see the docs you're referring to ; until I tested myself, I think I had no obvious reasons to guess that __getattribute__ relied on the upper level caller instead of finishing the hard job himself. Nick Coghlan a ?crit : > Michael Foord wrote: > >> Well, the documentation you pointed to specifies that __getattr__ will >> be called if __getattribute__ raises an AttributeError, it just doesn't >> specify that it is done by object.__getattribute__ (which it isn't). >> > > And as for why not: because __getattribute__ implementations need to be > able to call object.__getattribute__ without triggering the fallback > behaviour. > > Cheers, > Nick. > > I guess there are cases in which it is beneficial indeed. > > Michael Foord wrote: > >> Well, the documentation you pointed to specifies that __getattr__ >> will be called if __getattribute__ raises an AttributeError, it just >> doesn't specify that it is done by object.__getattribute__ (which it >> isn't). > > If __getattribute__ raises an exception, it won't get a chance to > do anything else, so something outside of __getattribute__ must > catch the AttributeError and calling __getattr__. So I think the > docs *are* specifying the behaviour here, if only by implication. > I don't follow you there - in my mind, the default __getattribute__ could simply have wrapped all its operations inside soem kind of "try..catch AttributeError:" mechanism, and thus been able to fallback to __getattr__ in any way. If I sum it up properly the semantic is : -A.obj and getattr(A, "obj") are exactly the same -They trigger the calling of __getattribute__ on the object (or it's python core equivalent) -By default, this __getattribute__ browse the whole object hierarchy according to well known rules (__dict__, type, type's ancestors..), handling descriptor protocols and the like. But it doesn't fallback to __getattr__ - it raises an AttributeError instead. -getattr() falls back to __getattr__ if __getattribute__ fails -customized __getattribute__ methods have the choice between calling __getattr__ by themselves, or delegating it to getattr() by raising an exception. Wouldn't it be worth completing the doc with these point ? They really didn't seem obvious to me basically (even though, after analysis, some behaviours make more sense than others). I might submit a patch. regards, Pascal > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Mar 22 19:15:01 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 22 Mar 2010 18:15:01 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?=5F=5Fpycache=5F=5F_creation?= References: <20100322095604.3371039e@heresy> Message-ID: Isaac Morland uwaterloo.ca> writes: > > > IMO, all these issues militate for putting __pycache__ creation out of > > the interpreter core, and in the hands of third-party package-time/ > > install-time tools (or distutils). > > Speaking only for myself, but really for anybody who likes tidy source > directories, I hope some version of the __pycache__ proposal becomes part > of standard Python, by which I ideally mean it's enabled by default but if > that is just not a good idea then at most it should be required to set a > command-line option to get this feature. This doesn't contradict by my proposal. What I am proposing is that the creation of __pycache__ /directories/ be put outside of the core. It can be part of distutils, or of a separate module, or delegated to third-party tools. It could even be as simple as "python -m compileall --pycache", if someone implements it. Creation of the __pycache__ /contents/ (files inside the directory) would still be part of core Python, but only if the directory exists and is writable by the current process. Regards Antoine. From alexander.belopolsky at gmail.com Mon Mar 22 19:22:43 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 22 Mar 2010 14:22:43 -0400 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> References: <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> Message-ID: On Mon, Mar 22, 2010 at 1:56 PM, Raymond Hettinger wrote: > > On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote: > > ? Decimal + float --> Decimal > > If everybody associated with the Decimal implementation wants this I > won't stop you; as I repeatedly said my intuition about this one (as > opposed to the other two above) is very weak. > > That's my vote. I've been lurking on this thread so far, but let me add my +1 to this option. My reasoning is that Decimal is a "better" model of Real than float and mixed operations should not degrade the result. "Better" can mean different things to different people, but to me the tie breaker is the support for contexts. I would not want precision to suddenly change in the middle of calculation I add 1.0 instead of 1. This behavior will also be familiar to users of other "enhanced" numeric types such as NumPy scalars. Note that in the older Numeric, it was the other way around, but after considerable discussion, the behavior was changed. From guido at python.org Mon Mar 22 19:24:20 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 22 Mar 2010 10:24:20 -0800 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> Message-ID: On Mon, Mar 22, 2010 at 10:22 AM, Alexander Belopolsky wrote: > On Mon, Mar 22, 2010 at 1:56 PM, Raymond Hettinger > wrote: >> >> On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote: >> >> ? Decimal + float --> Decimal >> >> If everybody associated with the Decimal implementation wants this I >> won't stop you; as I repeatedly said my intuition about this one (as >> opposed to the other two above) is very weak. >> >> That's my vote. > > I've been lurking on this thread so far, but let me add my +1 to this > option. ?My reasoning is that Decimal is a "better" model of Real than > float and mixed operations should not degrade the result. ? "Better" > can mean different things to different people, but to me the tie > breaker is the support for contexts. ?I would not want precision to > suddenly change in the middle of calculation I add 1.0 instead of 1. > > This behavior will also be familiar to users of other "enhanced" > numeric types such as NumPy scalars. ? Note that in the older Numeric, > it was the other way around, but after considerable discussion, the > behavior was changed. Thanks, "better" is a great way to express this. -- --Guido van Rossum (python.org/~guido) From dickinsm at gmail.com Mon Mar 22 19:26:10 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 22 Mar 2010 18:26:10 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> References: <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> Message-ID: <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> On Mon, Mar 22, 2010 at 5:56 PM, Raymond Hettinger wrote: > > On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote: > > ? Decimal + float --> Decimal > > If everybody associated with the Decimal implementation wants this I > won't stop you; as I repeatedly said my intuition about this one (as > opposed to the other two above) is very weak. > > That's my vote. > I believe Nick chimed-in in agreement. > Mark, do you concur? Yes; Decimal + float -> Decimal works for me; the greater flexibility of the Decimal type is the main plus for me. I don't think my own intuition is particularly strong here, either. It would be interesting to hear from major users of the decimal module (I have to confess to not actually using it myself very much). I argued earlier for Decimal + float -> Decimal on the basis that the float->Decimal conversion can be done exactly; but having thought about Decimal + Fraction it's no longer clear to me that this makes sense. Having Decimal + float -> Decimal round the float using the current Decimal context still seems like a reasonable option. Just for the record, I'd also prefer Decimal + Fraction -> Decimal. I don't want to let the abstractions of the numeric tower get in the way of the practicalities: we should modify the abstractions if necessary! In particular, it's not clear to me that all numeric types have to be comparable with each other. It might make sense for Decimal + complex mixed-type operations to be disallowed, for example. Mark From raymond.hettinger at gmail.com Mon Mar 22 19:34:51 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 22 Mar 2010 11:34:51 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> References: <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> Message-ID: <23CC71BC-B7B4-4D51-B1E4-D70B2941391F@gmail.com> On Mar 22, 2010, at 11:26 AM, Mark Dickinson wrote: > On Mon, Mar 22, 2010 at 5:56 PM, Raymond Hettinger > wrote: >> >> On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote: >> >> Decimal + float --> Decimal >> >> If everybody associated with the Decimal implementation wants this I >> won't stop you; as I repeatedly said my intuition about this one (as >> opposed to the other two above) is very weak. >> >> That's my vote. >> I believe Nick chimed-in in agreement. >> Mark, do you concur? > > Yes; Decimal + float -> Decimal works for me; the greater flexibility > of the Decimal type is the main plus for me. > On Mar 22, 2010, at 11:22 AM, Alexander Belopolsky wrote: > >> I've been lurking on this thread so far, but let me add my +1 to this >> option. On Mar 22, 2010, at 11:24 AM, Guido van Rossum wrote: >> > > Thanks, "better" is a great way to express this. OMG ;-) It looks like there is a consensus on: Decimal + float --> Decimal We have progress. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhamph at gmail.com Mon Mar 22 19:45:41 2010 From: rhamph at gmail.com (Adam Olsen) Date: Mon, 22 Mar 2010 12:45:41 -0600 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> References: <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> Message-ID: On Mon, Mar 22, 2010 at 12:26, Mark Dickinson wrote: > I don't want to let the abstractions of the numeric tower get in the > way of the practicalities: ?we should modify the abstractions if > necessary! ?In particular, it's not clear to me that all numeric types > have to be comparable with each other. ?It might make sense for > Decimal + complex mixed-type operations to be disallowed, for example. Only until a needy user breaks out the duck tape and builds a ComplexDecimal type. ;) The nature of the beast is more will be added on later. So long as Decimal == complex works right I don't see a problem with Decimal + complex raising an exception. -- Adam Olsen, aka Rhamphoryncus From guido at python.org Mon Mar 22 19:54:11 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 22 Mar 2010 10:54:11 -0800 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> References: <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> Message-ID: On Mon, Mar 22, 2010 at 10:26 AM, Mark Dickinson wrote: > On Mon, Mar 22, 2010 at 5:56 PM, Raymond Hettinger > wrote: >> >> On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote: >> >> ? Decimal + float --> Decimal >> >> If everybody associated with the Decimal implementation wants this I >> won't stop you; as I repeatedly said my intuition about this one (as >> opposed to the other two above) is very weak. >> >> That's my vote. >> I believe Nick chimed-in in agreement. >> Mark, do you concur? > > Yes; ?Decimal + float -> Decimal works for me; the greater flexibility > of the Decimal type is the main plus for me. ?I don't think my own > intuition is particularly strong here, either. ?It would be > interesting to hear from major users of the decimal module (I have to > confess to not actually using it myself very much). You're unlikely to hear from them here... :-( Though maybe Raymond qualifies. > I argued earlier for Decimal + float -> Decimal on the basis that the > float->Decimal conversion can be done exactly; ?but having thought > about Decimal + Fraction it's no longer clear to me that this makes > sense. ?Having Decimal + float -> Decimal round the float using the > current Decimal context still seems like a reasonable option. So now we have a second-order decision to make -- whether Decimal+float should convert the float to Decimal using the current context's precision, or do it exactly. I think we should just follow Decimal.from_float() here, which AFAIK does the conversion exactly -- the operation will already round properly if needed, and I don't think it is a good idea to round twice in the same operation. (Even though this is what long+float does -- but the situation is different there since float has no variable precision.) > Just for the record, I'd also prefer Decimal + Fraction -> Decimal. I think that is pretty much uncontested. Though there are second-order details to decide here too: Fraction + float converts the Fraction to a float first and then just adds the floats; but but Fraction / Decimal cannot always be computed exactly as a Decimal so here I'd suggest rounding first because we have to in some cases (it's better to be consistent, except for comparisons, which should always be done exactly if at all possible). > I don't want to let the abstractions of the numeric tower get in the > way of the practicalities: ?we should modify the abstractions if > necessary! ?In particular, it's not clear to me that all numeric types > have to be comparable with each other. ?It might make sense for > Decimal + complex mixed-type operations to be disallowed, for example. One of the reasons for having a numeric tower is actually that unless you do at least comparisons right, you get into inconsistencies like a == b, b ==c, a != c (and similar for orderings). Of course complex doesn't play along with the orderings, but it does play along with ==. Real and Complex are distinct stations in the numeric tower, and the tower defines mixed Real-Complex operations as returning Complex results, but it allows raising exceptions as well. Since Decimal(-1) ** Decimal('0.5') already raises an exception instead of returning a complex number, I'm fine with following that example -- complex numbers are definitely a specialty area. PS. PEP 3141 defines some miscellaneous operations that any number in the numeric tower ought to implement, e.g. x.imag, x.real and x.conjugate(). -- --Guido van Rossum (python.org/~guido) From raymond.hettinger at gmail.com Mon Mar 22 20:00:37 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 22 Mar 2010 12:00:37 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> References: <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> Message-ID: On Mar 22, 2010, at 11:26 AM, Mark Dickinson wrote: > > Just for the record, I'd also prefer Decimal + Fraction -> Decimal. Guido was persuasive on why float + Fraction --> float, so this makes sense for the same reasons. For the implementation, is there a way to avoid the double rounding in myfloat + myfrac.numerator / myfrac.denominator? Perhaps translate it to: f = Fractions.from_decimal(myfloat) + myfract # Lossless, exact addition return f.numerator / f.denominator # Only one decimal context rounding applied. Elsewhere in the decimal module, there is a fundamental notion that numbers are exact and only the results of operations are rounded. For example, it is possible in decimal to add together two high precision numbers but do so in a low precision context. Raymond From rrr at ronadam.com Mon Mar 22 20:02:42 2010 From: rrr at ronadam.com (Ron Adam) Date: Mon, 22 Mar 2010 14:02:42 -0500 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> Message-ID: Antoine Pitrou wrote: > Isaac Morland uwaterloo.ca> writes: >>> IMO, all these issues militate for putting __pycache__ creation out of >>> the interpreter core, and in the hands of third-party package-time/ >>> install-time tools (or distutils). >> Speaking only for myself, but really for anybody who likes tidy source >> directories, I hope some version of the __pycache__ proposal becomes part >> of standard Python, by which I ideally mean it's enabled by default but if >> that is just not a good idea then at most it should be required to set a >> command-line option to get this feature. > > This doesn't contradict by my proposal. > > What I am proposing is that the creation of __pycache__ /directories/ be put > outside of the core. It can be part of distutils, or of a separate module, or > delegated to third-party tools. It could even be as simple as > "python -m compileall --pycache", if someone implements it. > > Creation of the __pycache__ /contents/ (files inside the directory) would still > be part of core Python, but only if the directory exists and is writable by the > current process. +1 If I understand correctly, we would have the current mode as the default, and can trigger __pycache__ behavior simply by manually creating a __pycache__ directory and deleting any byte-code files in the module/program directory. I like this, it is easy to understand and can be used without messing with flags or environment variables. Ron From guido at python.org Mon Mar 22 20:03:04 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 22 Mar 2010 11:03:04 -0800 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> Message-ID: On Mon, Mar 22, 2010 at 11:00 AM, Raymond Hettinger wrote: > > On Mar 22, 2010, at 11:26 AM, Mark Dickinson wrote: >> >> Just for the record, I'd also prefer Decimal + Fraction -> Decimal. > > > Guido was persuasive on why float + Fraction --> float, > so this makes sense for the same reasons. > > For the implementation, is there a way to avoid the double rounding > in ? myfloat + myfrac.numerator / myfrac.denominator? > > Perhaps translate it to: > > ? ? ?f = Fractions.from_decimal(myfloat) + myfract ? # Lossless, exact addition > ? ? ?return f.numerator / f.denominator ? ? ? ? ? # Only one decimal context rounding applied. > > Elsewhere in the decimal module, there is a fundamental notion > that numbers are exact and only the results of operations are > rounded. ? For example, it is possible in decimal to add together > two high precision numbers but do so in a low precision context. Works for me. -- --Guido van Rossum (python.org/~guido) From raymond.hettinger at gmail.com Mon Mar 22 20:03:55 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 22 Mar 2010 12:03:55 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> Message-ID: <8909CA79-4204-4F0C-A476-023B20C74CA4@gmail.com> On Mar 22, 2010, at 11:54 AM, Guido van Rossum wrote: > > So now we have a second-order decision to make -- whether > Decimal+float should convert the float to Decimal using the current > context's precision, or do it exactly. I think we should just follow > Decimal.from_float() here, which AFAIK does the conversion exactly -- > the operation will already round properly if needed, and I don't think > it is a good idea to round twice in the same operation. (Even though > this is what long+float does -- but the situation is different there > since float has no variable precision.) I concur. That is consistent with the basic design of the decimal module which treats inputs as exact and only applies rounding to the results of operations. FWIW, Mark and I have both been bitten severely by doubling rounding in the world of binary floats. Avoiding double rounding is a darned good idea. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at bytereef.org Mon Mar 22 20:26:51 2010 From: stefan at bytereef.org (Stefan Krah) Date: Mon, 22 Mar 2010 20:26:51 +0100 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA55858.2000800@canterbury.ac.nz> <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> Message-ID: <20100322192651.GA21459@yoda.bytereef.org> Guido van Rossum wrote: > > ? Decimal + float --> Decimal > > If everybody associated with the Decimal implementation wants this I > won't stop you; as I repeatedly said my intuition about this one (as > opposed to the other two above) is very weak. I've been following the discussion only passively so far, but I also think this is the most logical solution. Stefan Krah From dickinsm at gmail.com Mon Mar 22 20:32:54 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 22 Mar 2010 19:32:54 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> Message-ID: <5c6f2a5d1003221232l69fe8eenf0778d9c4a5e436b@mail.gmail.com> On Mon, Mar 22, 2010 at 7:00 PM, Raymond Hettinger wrote: > > On Mar 22, 2010, at 11:26 AM, Mark Dickinson wrote: >> >> Just for the record, I'd also prefer Decimal + Fraction -> Decimal. > > > Guido was persuasive on why float + Fraction --> float, > so this makes sense for the same reasons. > > For the implementation, is there a way to avoid the double rounding > in ? myfloat + myfrac.numerator / myfrac.denominator? > > Perhaps translate it to: > > ? ? ?f = Fractions.from_decimal(myfloat) + myfract ? # Lossless, exact addition > ? ? ?return f.numerator / f.denominator ? ? ? ? ? # Only one decimal context rounding applied. I'm not sure; I see a couple of problems with this. (1) It's fine for the basic arithmetic operations that Fraction already supports, but what about all the other Decimal methods that don't have Fraction counterparts. (2) It bothers me that the Decimal -> Fraction conversion can be inefficient in cases like Decimal('1e'); currently, all Decimal operations are relatively efficient (no exponential-time behaviour) provided only that the coefficients don't get too large; large exponents aren't a problem. I think getting this to work would involve a lot of extra code and significant 'cleverness'. I'd prefer the simple-to-implement and simple-to-explain option of rounding the Fraction before performing the operation, even if this means that the whole operation involves two rounding operations. It's not so different from what currently happens for Fraction+float, or even int+float. Mark From raymond.hettinger at gmail.com Mon Mar 22 20:36:18 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 22 Mar 2010 12:36:18 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> Message-ID: <027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com> One other thought. The Decimal constructor should now accept floats as a possible input type. Formerly, we separated that out to Decimal.from_float() because decimals weren't interoperable with floats. This will put decimal and float back on equal footing so that we have both: float(some_decimal) # coerce to binary float and Decimal(some_float) # coerce to decimal float That will also save us from odd idioms like: d = some_float + Decimal(0) # coerce to decimal then apply context rounding This also matches the behavior of other constructors: int(some_float) or int(some_decimal) or str(some_float) or str(some_decimal) Raymond From stefan at bytereef.org Mon Mar 22 20:46:28 2010 From: stefan at bytereef.org (Stefan Krah) Date: Mon, 22 Mar 2010 20:46:28 +0100 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> Message-ID: <20100322194628.GB21459@yoda.bytereef.org> Raymond Hettinger wrote: > > On Mar 22, 2010, at 11:26 AM, Mark Dickinson wrote: > > > > Just for the record, I'd also prefer Decimal + Fraction -> Decimal. > > > Guido was persuasive on why float + Fraction --> float, > so this makes sense for the same reasons. > > For the implementation, is there a way to avoid the double rounding > in myfloat + myfrac.numerator / myfrac.denominator? > > Perhaps translate it to: > > f = Fractions.from_decimal(myfloat) + myfract # Lossless, exact addition > return f.numerator / f.denominator # Only one decimal context rounding applied. > > Elsewhere in the decimal module, there is a fundamental notion > that numbers are exact and only the results of operations are > rounded. For example, it is possible in decimal to add together > two high precision numbers but do so in a low precision context. I don't think this will be practical for huge decimal exponents. Also, at first glance I wonder how to integrate this cleanly into the control flow. convert_other() will not work, so there would need to be a special case for fractions in each function. Or do you mean to outsource the whole computation to the fractions module, which calls decimal only for the final division? Stefan Krah From guido at python.org Mon Mar 22 20:52:24 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 22 Mar 2010 11:52:24 -0800 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com> References: <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> <027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com> Message-ID: On Mon, Mar 22, 2010 at 11:36 AM, Raymond Hettinger wrote: > One other thought. > > The Decimal constructor should now accept floats as a possible input type. > Formerly, we separated that out to Decimal.from_float() because > decimals weren't interoperable with floats. Not sure this follows; Fraction(1.1) raises an exception, you have to use Fraction.from_float(). > This will put decimal and float back on equal footing so that we have both: > ? ?float(some_decimal) ? ? ? ?# coerce to binary float > and > ? ?Decimal(some_float) ? ? ? # coerce to decimal float If you really want this I won't stop you. > That will also save us from odd idioms like: > > ? ?d = some_float + Decimal(0) ? # coerce to decimal then apply context rounding Although this isn't too bad: d = Decimal.from_float(some_float) > This also matches the behavior of other constructors: > ? ?int(some_float) > or > ? ?int(some_decimal) > or > ? ?str(some_float) > or > ? ?str(some_decimal) True, I don't care strongly. -- --Guido van Rossum (python.org/~guido) From pierrebai at hotmail.com Mon Mar 22 21:02:57 2010 From: pierrebai at hotmail.com (Pierre B.) Date: Mon, 22 Mar 2010 20:02:57 +0000 (UTC) Subject: [Python-Dev] Mixing float and Decimal -- thread reboot References: <4BA55858.2000800@canterbury.ac.nz> <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> Message-ID: Sorry to intervene out of the blue, but I find the suggested rule for fractional to decimal conversion not as clean as I'd expect. If fractions are converted to decimals when doing arithmetics, would it be worthwhile to at least provide a minimum of fractional conversion integrity? What I have in mind is the following rule: When doing conversion from fraction to decimal, always generate a whole number of repeating digits, always at least twice. Examples, with a precision of 5 in Decimal: 1/2 -> 0.50000 1/3 -> 0.33333 1/11 -> 0.090909 # Note that we produced 6 digits, because # the repeating pattern contains 2 digits. 1/7 -> 0.142857142857 # Always at least two full patterns. The benefits I see are that: 1. If a number can be represented exactly it will be converted exactly. 2. The minimum precision requested is respected. 3. The conversion yields something that will convert back more precisely. Not perfectly, but see the next point. 4. Since the repeating pattern is present at least twice at the end, one can augment the precision of the conversion by detecting the repetition and adding more. This detection is trivial. From dickinsm at gmail.com Mon Mar 22 21:12:27 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 22 Mar 2010 20:12:27 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> Message-ID: <5c6f2a5d1003221312u19c71ca3i85eea936fc9465fa@mail.gmail.com> On Mon, Mar 22, 2010 at 8:02 PM, Pierre B. wrote: > Sorry to intervene out of the blue, but I find the suggested rule for > fractional to decimal conversion not as clean as I'd expect. > > If fractions are converted to decimals when doing arithmetics, would it be > worthwhile to at least provide a minimum of fractional conversion integrity? > What I have in mind is the following rule: > > When doing conversion from fraction to decimal, always generate a whole number > of repeating digits, always at least twice. > > Examples, with a precision of 5 in Decimal: > > 1/2 -> 0.50000 > > 1/3 -> 0.33333 > > 1/11 -> 0.090909 > # Note that we produced 6 digits, because > # the repeating pattern contains 2 digits. > > 1/7 -> 0.142857142857 > ?# Always at least two full patterns. And for 1/123123127? The decimal expansion of this fraction has a period of over 15 million! Sorry, but this doesn't seem like a feasible or desirable strategy. Mark From pierrebai at hotmail.com Mon Mar 22 21:14:28 2010 From: pierrebai at hotmail.com (Pierre B.) Date: Mon, 22 Mar 2010 20:14:28 +0000 (UTC) Subject: [Python-Dev] Mixing float and Decimal -- thread reboot References: <4BA55858.2000800@canterbury.ac.nz> <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> Message-ID: Pierre B. hotmail.com> writes: > > 4. Since the repeating pattern is present at least twice at the end, > one can augment the precision of the conversion by detecting the > repetition and adding more. This detection is trivial. Wrong of me. This point is acutally invalid, since it is impossible to generally differentiate where the pattern ends. The counter example is: 0.177177177177... vs 0.177.17777777... From barry at python.org Mon Mar 22 21:16:26 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 22 Mar 2010 16:16:26 -0400 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: References: <20100322095604.3371039e@heresy> Message-ID: <20100322161626.6e56efac@heresy> On Mar 22, 2010, at 02:30 PM, Antoine Pitrou wrote: >Barry Warsaw python.org> writes: >> >> * Tough luck >> * Force the umask so that the directory is writable, but then the question is, >> by whom? ugo+w or something less? >> * Copy the permissions from the parent directory and ignore umask >> * Raise an exception or refuse to create __pycache__ if it's not writable >> (again, by whom?) > >Welcome to a problem PHP has known for years. Yay. >The problem with solution 3 is that a non-root user can't change the owner of >a directory, therefore copying the permissions doesn't ensure the directory >will be writable by the right users. > >Solution 2 (chmod 777) is obviously too laxist. Anyone could put fake bytecode >files in a root-owned application. Less laxist chmods have the same problem as >solution 3, and can still be insecure. > >I think solution 4 would be the best compromise. In the face of ambiguity, >refuse to guess. But, rather than : > refuse to create __pycache__ if it's not writable, >the test should be : > refuse to create __pycache__ if I can't create it with the same ownership >and permissions as the parent directory (and raise an ImportWarning). I actually think "tough luck" might be the right answer and that it's not really all that serious. Certainly not serious enough not to enable this by default. When Python is being installed, either by a from-source 'make install' or by the distro packager, then you'd expect the umask not to be insane. In the latter case, it's a bug and in the former case you screwed up so you should be able to delete and reinstall, or at the very least execute the same `find` command that Python's own Makefile calls (in my branch) for 'make clean'. When you're installing packages, again, I would expect that the system installer, or you via `easy_install` or whatever, would not have an insane umask. As with above, if you *did* have a bad umask, you could certainly also have a bad umask whenever you ran the third party tool as when you ran Python. You could also have a bad umask when you're doing local development, but that should be easy enough to fix, and besides, I think you'd have other problems if that were the case. One possibility would be to instrument compileall to remove __pycache__ directories with a flag. >In light of this issue, I'm -0.5 on __pycache__ becoming the default caching >mechanism. The directory ownership/permissions issue is too much of a mess, >especially for Web applications (think __pycache__ files created by the Apache >user). > >__pycache__ should only be created if explicitly activated (for example by >distutils when installing stuff). Otherwise, if not present, the "legacy" >mechanism (writing an untagged pyc file along the py file) should be used. I don't particularly like this much, because most people won't get the benefit of it in a local dev environment. While not the primary use case, it's a useful enough side benefit, and unless Guido changes his mind, we want to enable this by default. >Actually, __pycache__ creation doesn't have to be part of the import >mechanism. It can be part of distutils instead (or whatever third-party tool >such as distribute, or distro-specific packaging scripts). This would relax >complexity of core Python a bit. It's not really that complicated. :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Mon Mar 22 21:18:17 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 22 Mar 2010 16:18:17 -0400 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> Message-ID: <20100322161817.06a8911f@heresy> On Mar 22, 2010, at 06:15 PM, Antoine Pitrou wrote: >What I am proposing is that the creation of __pycache__ /directories/ be put >outside of the core. It can be part of distutils, or of a separate module, or >delegated to third-party tools. It could even be as simple as >"python -m compileall --pycache", if someone implements it. Except that most people won't do this, they'll just run Python. I wouldn't count this as "enabled by default". -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Mon Mar 22 21:20:02 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 22 Mar 2010 16:20:02 -0400 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> Message-ID: <20100322162002.6daff86f@heresy> On Mar 22, 2010, at 02:02 PM, Ron Adam wrote: >If I understand correctly, we would have the current mode as the default, and >can trigger __pycache__ behavior simply by manually creating a __pycache__ >directory and deleting any byte-code files in the module/program directory. > >I like this, it is easy to understand and can be used without messing with >flags or environment variables. Well, for a package with subpackages, it gets more complicated. Definitely not something you're likely to do manually. Antoine's suggestion of 'python -m compileall --pycache' would work, but I think it's also obscure enough that most Python users won't get the benefit. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From raymond.hettinger at gmail.com Mon Mar 22 21:33:20 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 22 Mar 2010 13:33:20 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> <027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com> Message-ID: <5F54E962-E216-4BF5-AB87-5237FB624DD7@gmail.com> While we're on the topic, I think you should consider allowing the Fraction() constructor to accept a decimal input. This corresponds to common schoolbook problems and simple client requests: "Express 3.5 as a fraction". >>> Fraction(Decimal('3.5')) Fraction(7, 2) Unlike typical binary floats which use full precision, it is not uncommon to have decimal floats with only a few digits of precision where the expression as a fraction is both useful and unsurprising. Raymond From solipsis at pitrou.net Mon Mar 22 21:33:14 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 22 Mar 2010 20:33:14 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?PEP_3147=2C_=5F=5Fpycache=5F=5F_directorie?= =?utf-8?q?s_and_umask?= References: <20100322095604.3371039e@heresy> <20100322161626.6e56efac@heresy> Message-ID: Barry Warsaw python.org> writes: > > When Python is being installed, either by a from-source 'make install' or by > the distro packager, then you'd expect the umask not to be insane. In the > latter case, it's a bug and in the former case you screwed up so you should be > able to delete and reinstall, or at the very least execute the same `find` > command that Python's own Makefile calls (in my branch) for 'make clean'. > > When you're installing packages, again, I would expect that the system > installer, or you via `easy_install` or whatever, would not have an insane > umask. Well, precisely. That's why I suggest that creating the __pycache__ directories be done *at install time* (or packaging time), and not via the core import machinery (that is, not at import time). That is, when you *know* you are the right user, with the right umask. Which also means that it can and probably should be pulled out of the core (and written in pure Python, which is also much more practical to maintain, test and debug). > I don't particularly like this much, because most people won't get the benefit > of it in a local dev environment. AFAICT it has no benefit in a local dev environment (apart from supposedly "cleaner" directories, which is not something anyone seems to consider a problem with the current CPythons). And again, this could be part of the distutils standard "setup.py install" (or "setup.py develop" with distutils2/distribute) if people care :) Regards Antoine. From guido at python.org Mon Mar 22 21:38:42 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 22 Mar 2010 12:38:42 -0800 Subject: [Python-Dev] __pycache__ creation In-Reply-To: <20100322162002.6daff86f@heresy> References: <20100322095604.3371039e@heresy> <20100322162002.6daff86f@heresy> Message-ID: On Mon, Mar 22, 2010 at 12:20 PM, Barry Warsaw wrote: > On Mar 22, 2010, at 02:02 PM, Ron Adam wrote: > >>If I understand correctly, we would have the current mode as the default, and >>can trigger __pycache__ behavior simply by manually creating a __pycache__ >>directory and deleting any byte-code files in the module/program directory. Huh? Last time I looked weren't we going to make __pycache__ the default (and eventually only) behavior? >>I like this, it is easy to understand and can be used without messing with >>flags or environment variables. > > Well, for a package with subpackages, it gets more complicated. ?Definitely > not something you're likely to do manually. ?Antoine's suggestion of 'python > -m compileall --pycache' would work, but I think it's also obscure enough that > most Python users won't get the benefit. I see only two reasonable solutions for __pycache__ creation -- either we change all setup/install scripts (both for core Python and for 3rd party packages) to always create a __pycache__ subdirectory for every directory (including package directories) installed; or we somehow create it the first time it's needed. But creating it as needed runs into at least similar problems with ownership as creating .pyc files when first needed (if the parent directory is root-owned a mere mortal can't create it at all). So even apart from the security issue (which I haven't thought about deeply) I think precreation should at least be an easily accessible option both for the core (where it can be done by compileall) and for 3rd party packages (where I guess it's up to distutils or whatever install mechanism is used). -- --Guido van Rossum (python.org/~guido) From p.f.moore at gmail.com Mon Mar 22 21:45:05 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 22 Mar 2010 20:45:05 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003221232l69fe8eenf0778d9c4a5e436b@mail.gmail.com> References: <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> <5c6f2a5d1003221232l69fe8eenf0778d9c4a5e436b@mail.gmail.com> Message-ID: <79990c6b1003221345n9d02b0dwf7d015ebfa812712@mail.gmail.com> On 22 March 2010 19:32, Mark Dickinson wrote: > I think getting this to work would involve a lot of extra code and > significant 'cleverness'. ?I'd prefer the simple-to-implement and > simple-to-explain option of rounding the Fraction before performing > the operation, even if this means that the whole operation involves > two rounding operations. ?It's not so different from what currently > happens for Fraction+float, or even int+float. My instinct throughout this discussion has been that a_decimal + a_fraction is in effect a_decimal + a_fraction.numerator / a_fraction.denominator and as the numerator and denomination can be converted into exact decimals, my instincts "expect" a double rounding. Anything else effectively means implementing partial cases of (a+b/c) treated as a ternary operation. So I guess I'm +1 on Mark's suggestion. Paul. From guido at python.org Mon Mar 22 21:44:47 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 22 Mar 2010 12:44:47 -0800 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5F54E962-E216-4BF5-AB87-5237FB624DD7@gmail.com> References: <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> <027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com> <5F54E962-E216-4BF5-AB87-5237FB624DD7@gmail.com> Message-ID: On Mon, Mar 22, 2010 at 12:33 PM, Raymond Hettinger wrote: > While we're on the topic, I think you should consider allowing the Fraction() > constructor to accept a decimal input. > > This corresponds to common schoolbook problems and simple client requests: > ? "Express 3.5 as a fraction". > > ? ?>>> Fraction(Decimal('3.5')) > ? Fraction(7, 2) > > Unlike typical binary floats which use full precision, it is not uncommon > to have decimal floats with only a few digits of precision where the > expression as a fraction is both useful and unsurprising. There is already a Fraction.from_decimal() class method. I understand the reasons for not allowing floats in the unadorned Fraction constructor don't quite hold for Decimal, but at the same time I'm not sure how compelling the use case is for such downcasts given that it can be done easily with an explicit call. (Same actually for requiring Decimal.from_float().) -- --Guido van Rossum (python.org/~guido) From dickinsm at gmail.com Mon Mar 22 21:45:39 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 22 Mar 2010 20:45:39 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5F54E962-E216-4BF5-AB87-5237FB624DD7@gmail.com> References: <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> <027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com> <5F54E962-E216-4BF5-AB87-5237FB624DD7@gmail.com> Message-ID: <5c6f2a5d1003221345n5560ab59m82381d1aaac4481d@mail.gmail.com> On Mon, Mar 22, 2010 at 8:33 PM, Raymond Hettinger wrote: > While we're on the topic, I think you should consider allowing the Fraction() > constructor to accept a decimal input. > > This corresponds to common schoolbook problems and simple client requests: > ? "Express 3.5 as a fraction". > > ? ?>>> Fraction(Decimal('3.5')) > ? Fraction(7, 2) > > Unlike typical binary floats which use full precision, it is not uncommon > to have decimal floats with only a few digits of precision where the > expression as a fraction is both useful and unsurprising. Sounds fine to me. Fraction already accepts decimal floating-point strings, so the implementation of this would be trivial (convert to string, then call the Fraction constructor). Mark From martin at v.loewis.de Mon Mar 22 21:47:40 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 22 Mar 2010 21:47:40 +0100 Subject: [Python-Dev] __pycache__ creation In-Reply-To: <20100322161817.06a8911f@heresy> References: <20100322095604.3371039e@heresy> <20100322161817.06a8911f@heresy> Message-ID: <4BA7D76C.9010502@v.loewis.de> Barry Warsaw wrote: > On Mar 22, 2010, at 06:15 PM, Antoine Pitrou wrote: > >> What I am proposing is that the creation of __pycache__ /directories/ be put >> outside of the core. It can be part of distutils, or of a separate module, or >> delegated to third-party tools. It could even be as simple as >> "python -m compileall --pycache", if someone implements it. > > Except that most people won't do this, they'll just run Python. I wouldn't > count this as "enabled by default". Therefore, I'm in favor of having it on by default. If certain use cases make it problematic (e.g. Apache creating directories which you then cannot delete), there should be a way to turn it *off*. Perhaps the existing machinery to turn of byte code generation at all might be sufficient. As for the original question (funny umasks), I think my proposal is "tough luck". Don't try to be super-smart; as Antoine explains, it gets worse, not better. If the user has arranged that Python will create unusable directories, the user better changes his setup. Regards, Martin From guido at python.org Mon Mar 22 21:48:09 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 22 Mar 2010 12:48:09 -0800 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003221345n5560ab59m82381d1aaac4481d@mail.gmail.com> References: <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> <027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com> <5F54E962-E216-4BF5-AB87-5237FB624DD7@gmail.com> <5c6f2a5d1003221345n5560ab59m82381d1aaac4481d@mail.gmail.com> Message-ID: On Mon, Mar 22, 2010 at 12:45 PM, Mark Dickinson wrote: > On Mon, Mar 22, 2010 at 8:33 PM, Raymond Hettinger > wrote: >> While we're on the topic, I think you should consider allowing the Fraction() >> constructor to accept a decimal input. >> >> This corresponds to common schoolbook problems and simple client requests: >> ? "Express 3.5 as a fraction". >> >> ? ?>>> Fraction(Decimal('3.5')) >> ? Fraction(7, 2) >> >> Unlike typical binary floats which use full precision, it is not uncommon >> to have decimal floats with only a few digits of precision where the >> expression as a fraction is both useful and unsurprising. > > Sounds fine to me. ?Fraction already accepts decimal floating-point > strings, so the implementation of this would be trivial (convert to > string, then call the Fraction constructor). Better, Fraction.from_decimal() already exists. ;-) -- --Guido van Rossum (python.org/~guido) From dickinsm at gmail.com Mon Mar 22 21:49:06 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 22 Mar 2010 20:49:06 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> <027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com> <5F54E962-E216-4BF5-AB87-5237FB624DD7@gmail.com> Message-ID: <5c6f2a5d1003221349o12e64d76qc5fc8e643a71fa5d@mail.gmail.com> On Mon, Mar 22, 2010 at 8:44 PM, Guido van Rossum wrote: > On Mon, Mar 22, 2010 at 12:33 PM, Raymond Hettinger > wrote: >> While we're on the topic, I think you should consider allowing the Fraction() >> constructor to accept a decimal input. >> >> This corresponds to common schoolbook problems and simple client requests: >> ? "Express 3.5 as a fraction". >> >> ? ?>>> Fraction(Decimal('3.5')) >> ? Fraction(7, 2) > > There is already a Fraction.from_decimal() class method. So there is; I'd failed to notice that! So decimal -> fraction conversion is implemented twice in the Fractions module---once in __new__ and once in from_decimal. Hmm. Mark From raymond.hettinger at gmail.com Mon Mar 22 21:53:25 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 22 Mar 2010 13:53:25 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> Message-ID: <926781A9-32A5-4814-952D-F8159A944B03@gmail.com> For the record, I thought I would take a stab at making a single post that recaps the trade-offs and reasoning behind the decision to have Fraction + decimal/float --> decimal/float. Pros: * While we know that both decimal and binary floats have a fixed internal precision and can be converted losslessly to a rational, that doesn't correspond to the way we think about them. We tend to think of floating point values as real numbers, not as rationals. * There is a notion of fractions being used for unrounded arithmetic and floats operations being rounded arithmetic. So, it doesn't make sense to create the illusion of an unrounded result from inputs that we already subject to rounding. * Backward compatibility. That is what the fractions module already does and we haven't have any problems with it. Cons: * The coercion logic for comparisons won't match the coercion logic for arithmetic operations. The former strives to be exact and to be consistent with hashing while the latter goes in the opposite direction. * Operations such as some_float + some_fraction are subject to double rounding. The potentially produces a different result than the single rounding in: float(Fraction.from_float(some_float) + some_fraction) Raymond From solipsis at pitrou.net Mon Mar 22 22:00:30 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 22 Mar 2010 21:00:30 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?=5F=5Fpycache=5F=5F_creation?= References: <20100322095604.3371039e@heresy> <20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > If certain use cases > make it problematic (e.g. Apache creating directories which you then > cannot delete), there should be a way to turn it *off*. Perhaps the > existing machinery to turn of byte code generation at all might be > sufficient. Except that you notice the problem when it happens, so turning it off is always too late. From martin at v.loewis.de Mon Mar 22 22:25:35 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 22 Mar 2010 22:25:35 +0100 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> <20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de> Message-ID: <4BA7E04F.7070308@v.loewis.de> Antoine Pitrou wrote: > Martin v. L?wis v.loewis.de> writes: >> If certain use cases >> make it problematic (e.g. Apache creating directories which you then >> cannot delete), there should be a way to turn it *off*. Perhaps the >> existing machinery to turn of byte code generation at all might be >> sufficient. > > Except that you notice the problem when it happens, so turning it off is always > too late. Why is it too late? Fix it, and get on. Regards, Martin From brett at python.org Mon Mar 22 22:28:26 2010 From: brett at python.org (Brett Cannon) Date: Mon, 22 Mar 2010 14:28:26 -0700 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: <20100322095604.3371039e@heresy> References: <20100322095604.3371039e@heresy> Message-ID: On Mon, Mar 22, 2010 at 06:56, Barry Warsaw wrote: > I have a pretty good start on PEP 3147 implementation [1], but I've > encountered a situation that I'd like to get some feedback on. ?Here's the > test case illustrating the problem. ?From test_import.py: > > ? ?def test_writable_directory(self): > ? ? ? ?# The umask is not conducive to creating a writable __pycache__ > ? ? ? ?# directory. > ? ? ? ?with umask(0o222): > ? ? ? ? ? ?__import__(TESTFN) > ? ? ? ?self.assertTrue(os.path.exists('__pycache__')) > ? ? ? ?self.assertTrue(os.path.exists(os.path.join( > ? ? ? ? ? ?'__pycache__', '{}.{}.pyc'.format(TESTFN, self.tag)))) > > The __pycache__ directory does not exist before the import, and the import > machinery creates the directory, but the umask leaves the directory unwritable > by anybody. ?So of course when the import machinery goes to write the .pyc > file inside __pycache__, it fails. ?This does not cause an ImportError though, > just like if today the package directory were unwritable. > > This might be different than today's situation though because once the > unwritable __pycache__ directory is created, nothing is going to change that > without explicit user interaction, and that might be difficult after the > fact. > > I'm not sure what the right answer is. ?Some possible choices: > > * Tough luck > * Force the umask so that the directory is writable, but then the question is, > ?by whom? ?ugo+w or something less? > * Copy the permissions from the parent directory and ignore umask > * Raise an exception or refuse to create __pycache__ if it's not writable > ?(again, by whom?) I say tough luck. At the moment if you can't write a .pyc it is a silent failure; don't see the difference as significant enough to warrant changing the practice. -Brett > > Or maybe you have a better idea? ?What's the equivalent situation on Windows > and how would things work there? > > -Barry > > [1] https://edge.launchpad.net/~barry/python/pep3147 > > P.S. I'm down to only 8 unit test failures. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > > From rdmurray at bitdance.com Mon Mar 22 22:39:26 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 22 Mar 2010 17:39:26 -0400 Subject: [Python-Dev] =?utf-8?q?=5F=5Fpycache=5F=5F_creation?= In-Reply-To: References: <20100322095604.3371039e@heresy> Message-ID: <20100322213926.927511FB7F0@kimball.webabinitio.net> On Mon, 22 Mar 2010 18:15:01 -0000, Antoine Pitrou wrote: > Isaac Morland uwaterloo.ca> writes: > > > > > IMO, all these issues militate for putting __pycache__ creation out of > > > the interpreter core, and in the hands of third-party package-time/ > > > install-time tools (or distutils). > > > > Speaking only for myself, but really for anybody who likes tidy source > > directories, I hope some version of the __pycache__ proposal becomes part > > of standard Python, by which I ideally mean it's enabled by default but if > > that is just not a good idea then at most it should be required to set a > > command-line option to get this feature. > > This doesn't contradict by my proposal. > > What I am proposing is that the creation of __pycache__ /directories/ be put > outside of the core. It can be part of distutils, or of a separate module, or > delegated to third-party tools. It could even be as simple as > "python -m compileall --pycache", if someone implements it. Or even as simple as 'mkdir __pycache__', if you are working in your own library and don't want .pyc clutter. -- R. David Murray www.bitdance.com From solipsis at pitrou.net Mon Mar 22 22:57:03 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 22 Mar 2010 21:57:03 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?=5F=5Fpycache=5F=5F_creation?= References: <20100322095604.3371039e@heresy> <20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de> <4BA7E04F.7070308@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > Antoine Pitrou wrote: > > Martin v. L?wis v.loewis.de> writes: > >> If certain use cases > >> make it problematic (e.g. Apache creating directories which you then > >> cannot delete), there should be a way to turn it *off*. Perhaps the > >> existing machinery to turn of byte code generation at all might be > >> sufficient. > > > > Except that you notice the problem when it happens, so turning it off is always > > too late. > > Why is it too late? Fix it, and get on. Sure, but it is annoying, and since it's the kind of things that noone (including sysadmins) ever thinks about in advance, it's bound to repeat itself quite often. It's especially annoying, of course, if you have to ask someone else to remove the directories for you (or if you have to write custom code and get it executed by the Apache or WSGI handler...). Really, it's unfriendly to users and it's certainly not outweighed by the "benefit" of having "cleaner" source directories. From martin at v.loewis.de Mon Mar 22 23:18:02 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 22 Mar 2010 23:18:02 +0100 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> <20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de> <4BA7E04F.7070308@v.loewis.de> Message-ID: <4BA7EC9A.1010206@v.loewis.de> >> Why is it too late? Fix it, and get on. > > Sure, but it is annoying, and since it's the kind of things that noone > (including sysadmins) ever thinks about in advance, it's bound to repeat itself > quite often. > > It's especially annoying, of course, if you have to ask someone else to remove > the directories for you (or if you have to write custom code and get it executed > by the Apache or WSGI handler...). > > Really, it's unfriendly to users and it's certainly not outweighed by the > "benefit" of having "cleaner" source directories. Whether it is outweighed would also depend on how likely and frequent the presumed problem is, no? If Apache creates a folder for me that I cannot remove, most likely, there was a configuration error in the first place: common practice tells that you should execute user code under user permissions, not as www-data. If your code does get run as Apache, this also opens a way of not asking for help: just put "os.system('chmod +w /.../__pycache__')" into your code, and have Apache run it again. So I don't think this is any more unfriendly than creating .pyc files in the first place, and the advantages of uniformity of this new approach certainly outweigh the disadvantages. Regards, Martin From ncoghlan at gmail.com Mon Mar 22 23:19:35 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 23 Mar 2010 08:19:35 +1000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <926781A9-32A5-4814-952D-F8159A944B03@gmail.com> References: <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> <926781A9-32A5-4814-952D-F8159A944B03@gmail.com> Message-ID: <4BA7ECF7.5020809@gmail.com> Raymond Hettinger wrote: > * The coercion logic for comparisons won't match the > coercion logic for arithmetic operations. The former > strives to be exact and to be consistent with hashing > while the latter goes in the opposite direction. Although Guido pointed out that float/long comparisons and coercions blazed the trail on this particular discrepancy years ago. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Mon Mar 22 23:53:35 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 22 Mar 2010 23:53:35 +0100 Subject: [Python-Dev] __pycache__ creation In-Reply-To: <4BA7EC9A.1010206@v.loewis.de> References: <20100322095604.3371039e@heresy> <20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de> <4BA7E04F.7070308@v.loewis.de> <4BA7EC9A.1010206@v.loewis.de> Message-ID: <1269298415.5792.11.camel@localhost> Le lundi 22 mars 2010 ? 23:18 +0100, "Martin v. L?wis" a ?crit : > If Apache creates a folder for me that I cannot remove, most likely, > there was a configuration error in the first place: common practice > tells that you should execute user code under user permissions, not as > www-data. I'm sure there can be reasons not to do so. One of them is mass hosting of Python apps: you don't want to create a separate process per Python user, and therefore you can't run the code under the "right" user. > If your code does get run as Apache, this also opens a way of > not asking for help: just put "os.system('chmod +w /.../__pycache__')" > into your code, and have Apache run it again. This is what I meant by "write custom code and get it executed by the Apache or WSGI handler". Having the Web server execute ad hoc system administration code is far from elegant and user-friendly. > So I don't think this is any more unfriendly than creating .pyc files in > the first place, and the advantages of uniformity of this new approach > certainly outweigh the disadvantages. I don't see the "advantages of uniformity". What we had initially was a way to solve problems which are specific to Ubuntu and Debian packagers, and which was only to be activated by those people on those systems. We now end with an alleged complete solution to a problem which doesn't seem to exist, or is at least vastly overblown (the idea that having pyc files along their source counterparts is a nuisance doesn't seem to be a common grief against Python). I would really recommend reexamining it, rather than falling for the shiny new thing. Regards Antoine. From greg.ewing at canterbury.ac.nz Tue Mar 23 00:05:47 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 23 Mar 2010 11:05:47 +1200 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> <4BA55858.2000800@canterbury.ac.nz> <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> Message-ID: <4BA7F7CB.7010603@canterbury.ac.nz> Mark Dickinson wrote: > But the Fraction type is going to mess this up: for Decimal + > Fraction -> Decimal, I don't see any other sensible option than to > convert the Fraction using the current context, since lossless > conversion isn't generally possible. You could convert the Decimal to a Fraction, do the arithmetic as Fractions, and convert the result back to Decimal using the current precision. I think this would satisfy the principle of performing the calculation as though infinite precision were available, and the final rounding is justified, since seen from the outside we are performing a single operation on two numbers. Another approach would be to convert the numerators and denominators separately to Decimal, and then do a/b + c/d = (a*d + b*c) / (b*d) using the usual rules of Decimal arithmetic. This would incur more than one rounding, but the rules for Decimals concern operations between two Decimals, and we have a Decimal and a Fraction here, so all bets could be considered off. -- Greg From martin at v.loewis.de Tue Mar 23 00:02:53 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 23 Mar 2010 00:02:53 +0100 Subject: [Python-Dev] __pycache__ creation In-Reply-To: <1269298415.5792.11.camel@localhost> References: <20100322095604.3371039e@heresy> <20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de> <4BA7E04F.7070308@v.loewis.de> <4BA7EC9A.1010206@v.loewis.de> <1269298415.5792.11.camel@localhost> Message-ID: <4BA7F71D.6030907@v.loewis.de> > We now end with an alleged complete solution to a problem which doesn't > seem to exist, or is at least vastly overblown (the idea that having pyc > files along their source counterparts is a nuisance doesn't seem to be a > common grief against Python). > > I would really recommend reexamining it, rather than falling for the > shiny new thing. I think the appropriate action at this point is to record this specific objection in the PEP. Regards, Martin From greg.ewing at canterbury.ac.nz Tue Mar 23 00:26:14 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 23 Mar 2010 11:26:14 +1200 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA74946.9060302@gmail.com> References: <4BA57C5A.9060403@gmail.com> <4BA6A0AD.7090207@canterbury.ac.nz> <4BA74946.9060302@gmail.com> Message-ID: <4BA7FC96.6020109@canterbury.ac.nz> Nick Coghlan wrote: > http://docs.python.org/library/decimal.html#decimal.Inexact > > (Part of the thread context rather than the individual decimal values, > but if you use it properly it tells you whenever an inexact operation > has occurred in the current thread) My problem was that the statement "All numbers are exact; inexactness is a taint" appears to be self-contradictory. Numbers *can* become tainted with inexactness whether you explicitly keep track of that or not. However, the excerpt by Cowlishaw posted earlier reveals where the confusion is coming from, I think. What Cowlishaw appears to mean by "numbers are exact" is that Decimals represent particular values, not *intervals*. This is not really the same thing as the notion of inexact numbers in the numeric tower. There, it means more like "this number may not quite represent the value the programmer had in mind". -- Greg From greg.ewing at canterbury.ac.nz Tue Mar 23 00:58:16 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 23 Mar 2010 11:58:16 +1200 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: References: <20100322095604.3371039e@heresy> Message-ID: <4BA80418.6030905@canterbury.ac.nz> Antoine Pitrou wrote: > In light of this issue, I'm -0.5 on __pycache__ becoming the default caching > mechanism. The directory ownership/permissions issue is too much of a mess, > especially for Web applications (think __pycache__ files created by the Apache > user). Doesn't the existing .pyc mechanism have the same problem? Seems to me it's just as insecure to allow the Apache user to create .pyc files, since an attacker could overwrite them with arbitrary bytecode. The only safe way is to pre-compile under a different user and make everything read-only to Apache. The same thing would apply under the __pycache__ regime. > Actually, __pycache__ creation doesn't have to be part of the import mechanism. > It can be part of distutils instead (or whatever third-party tool What about development, or if a user installs by dragging into site-packages instead of using an installer? I don't like the idea of being required to use an installation tool in order to get .pyc files. -- Greg From solipsis at pitrou.net Tue Mar 23 01:05:11 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 23 Mar 2010 00:05:11 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?PEP_3147=2C_=5F=5Fpycache=5F=5F_directorie?= =?utf-8?q?s_and_umask?= References: <20100322095604.3371039e@heresy> <4BA80418.6030905@canterbury.ac.nz> Message-ID: Greg Ewing canterbury.ac.nz> writes: > > Doesn't the existing .pyc mechanism have the same problem? Seems > to me it's just as insecure to allow the Apache user to create > .pyc files, since an attacker could overwrite them with arbitrary > bytecode. The problem is that you can't delete the __pycache__ directory if it doesn't have the right ownership and if it's non-empty. This problem doesn't exist with a pyc file situated in a directory you own. > > Actually, __pycache__ creation doesn't have to be part of the import mechanism. > > It can be part of distutils instead (or whatever third-party tool > > What about development, The main point of the __pycache__ proposal is to solve the needs of Ubuntu/Debian packagers. If you are developing (rather than deploying or building packages), you shouldn't have these needs AFAICT. > or if a user installs by dragging into > site-packages instead of using an installer? Well... do people actually do this? "python setup.py install" is simpler than finding the right place to drag your package to, and doing the dragging. It also gives you metadata for free. And there's less risk of screwing up. From greg.ewing at canterbury.ac.nz Tue Mar 23 01:23:41 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 23 Mar 2010 12:23:41 +1200 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> Message-ID: <4BA80A0D.6020704@canterbury.ac.nz> Antoine Pitrou wrote: > Oh, and by the way, there can be a race condition between __pycache__ > creation and deletion (if it fails the test) You can check whether the directory would be created with the right user beforehand, and if not, don't create one at all. To exploit a race condition there, the attacker would have to be capable of either changing the owner of the parent directory or removing it and replacing it with a different one, and if he can do that, he can do whatever he wants anyway. -- Greg From greg.ewing at canterbury.ac.nz Tue Mar 23 01:30:15 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 23 Mar 2010 12:30:15 +1200 Subject: [Python-Dev] Attribute lookup ambiguity In-Reply-To: <4BA7B1B4.4010000@wanadoo.fr> References: <4BA3C962.7080500@wanadoo.fr> <4BA3DD4F.1080209@voidspace.org.uk> <4BA4B8DE.8060109@wanadoo.fr> <4BA576AF.2050208@voidspace.org.uk> <4BA7B1B4.4010000@wanadoo.fr> Message-ID: <4BA80B97.6080606@canterbury.ac.nz> Pascal Chambon wrote: > I don't follow you there - in my mind, the default __getattribute__ > could simply have wrapped all its operations inside soem kind of > "try..catch AttributeError:" mechanism, and thus been able to fallback > to __getattr__ in any way. But then it would be incorrect to say that "__getattribute__ raises an exception". When we say that a function raises an exception, we normally mean that the exception propagates out of the function and can be seen by the caller, not that it was raised and caught somewhere inside the function. -- Greg From greg.ewing at canterbury.ac.nz Tue Mar 23 01:33:48 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 23 Mar 2010 12:33:48 +1200 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> References: <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> Message-ID: <4BA80C6C.5070809@canterbury.ac.nz> Mark Dickinson wrote: > It might make sense for > Decimal + complex mixed-type operations to be disallowed, for example. As long as you're allowing Decimal-float comparisons, Decimal-complex comparison for equality has an obvious interpretation. -- Greg From cs at zip.com.au Tue Mar 23 01:40:02 2010 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 23 Mar 2010 11:40:02 +1100 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: <20100322095604.3371039e@heresy> References: <20100322095604.3371039e@heresy> Message-ID: <20100323004002.GA29460@cskk.homeip.net> On 22Mar2010 09:56, Barry Warsaw wrote: | I have a pretty good start on PEP 3147 implementation [1], but I've | encountered a situation that I'd like to get some feedback on. Here's the | test case illustrating the problem. From test_import.py: | | def test_writable_directory(self): | # The umask is not conducive to creating a writable __pycache__ | # directory. | with umask(0o222): [...] | The __pycache__ directory does not exist before the import, and the import | machinery creates the directory, but the umask leaves the directory unwritable | by anybody. So of course when the import machinery goes to write the .pyc | file inside __pycache__, it fails. This does not cause an ImportError though, | just like if today the package directory were unwritable. | | This might be different than today's situation though because once the | unwritable __pycache__ directory is created, nothing is going to change that | without explicit user interaction, and that might be difficult after the | fact. Like any bad/suboptimal permission. | I'm not sure what the right answer is. Some possible choices: | | * Tough luck +1 I'd go with this one myself. | * Force the umask so that the directory is writable, but then the question is, | by whom? ugo+w or something less? -2 Racy and dangerous. The umask is a UNIX process global, and other threads may get bad results if they're active during this window. | * Copy the permissions from the parent directory and ignore umask -1 Maybe. But consider that you may not be the owner of the parent: then the new child will have different ownership than the parent but the same permission mask. Potentially a bad mix. This approach is very hard to get right. | * Raise an exception or refuse to create __pycache__ if it's not writable | (again, by whom?) -3 Bleah. My python program won't run because an obscure (to the user) directory had unusual permissions? Tough ove, it's the only way:-) -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ When asked what would I most want to try before doing it, I said Death. - Michael Burton, michaelb at compnews.co.uk From cs at zip.com.au Tue Mar 23 01:45:19 2010 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 23 Mar 2010 11:45:19 +1100 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: <20100323004002.GA29460@cskk.homeip.net> References: <20100323004002.GA29460@cskk.homeip.net> Message-ID: <20100323004519.GA30414@cskk.homeip.net> On 23Mar2010 11:40, I wrote: | | * Raise an exception or refuse to create __pycache__ if it's not writable | | (again, by whom?) | | -3 | Bleah. My python program won't run because an obscure (to the user) | directory had unusual permissions? Clarification: I'm -3 on the exception. Silent failure to make the __pycache__ would do, and may be better than silently making a useless (unwritable) one. How about: made_it = False ok = False try: if not isdir(__pycache__dir): mkdir(__pycache__dir) made_it = True write pyc content ... ok = True except OSError, IOerror etc: if not ok: os.remove pyc content file if made_it: rmdir(__pycache__dir) but be quiet if this fails, eg if it is not empty because another process or thread added stuff So silent tidyup attempt on failure, but no escaping exception. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ The govt MUST regulate the Net NOW! We can't have average people saying what's on their minds! - ezwriter at netcom.com From greg.ewing at canterbury.ac.nz Tue Mar 23 01:57:27 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 23 Mar 2010 12:57:27 +1200 Subject: [Python-Dev] __pycache__ creation In-Reply-To: <1269298415.5792.11.camel@localhost> References: <20100322095604.3371039e@heresy> <20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de> <4BA7E04F.7070308@v.loewis.de> <4BA7EC9A.1010206@v.loewis.de> <1269298415.5792.11.camel@localhost> Message-ID: <4BA811F7.2010003@canterbury.ac.nz> Antoine Pitrou wrote: > Having the Web server execute ad hoc system > administration code is far from elegant and user-friendly. With the right piece of code, you could create yourself a setuid-apache shell and solve this problem once and for all. :-) -- Greg From rrr at ronadam.com Tue Mar 23 01:53:28 2010 From: rrr at ronadam.com (Ron Adam) Date: Mon, 22 Mar 2010 19:53:28 -0500 Subject: [Python-Dev] __pycache__ creation In-Reply-To: <20100322162002.6daff86f@heresy> References: <20100322095604.3371039e@heresy> <20100322162002.6daff86f@heresy> Message-ID: Barry Warsaw wrote: > On Mar 22, 2010, at 02:02 PM, Ron Adam wrote: > >> If I understand correctly, we would have the current mode as the default, and >> can trigger __pycache__ behavior simply by manually creating a __pycache__ >> directory and deleting any byte-code files in the module/program directory. >> >> I like this, it is easy to understand and can be used without messing with >> flags or environment variables. > > Well, for a package with subpackages, it gets more complicated. Definitely > not something you're likely to do manually. Antoine's suggestion of 'python > -m compileall --pycache' would work, but I think it's also obscure enough that > most Python users won't get the benefit. May be a bit more complicated, but it should be easy to write tools to handle the repetitive stuff. When I'm writing python projects I usually only work on one or two packages at a time at most. Creating a couple of directories when I first get started on a project is nothing compared with the other 10 or more files of > 1,000 lines of code each. This has a very low mental hurdle too. All the other packages I'm importing would probably already be preinstalled complete with __pycache__ directories and bytecode files. As Antoine was suggesting that it would be up to the installer's (scrips) to create the __pycache__ directories at install time, either directly or by possibly issuing a 'python -m compileall --pycache' command? Ron From solipsis at pitrou.net Tue Mar 23 01:58:42 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 23 Mar 2010 00:58:42 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?=5F=5Fpycache=5F=5F_creation?= References: <20100322095604.3371039e@heresy> <20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de> <4BA7E04F.7070308@v.loewis.de> <4BA7EC9A.1010206@v.loewis.de> <1269298415.5792.11.camel@localhost> <4BA811F7.2010003@canterbury.ac.nz> Message-ID: Greg Ewing canterbury.ac.nz> writes: > > Antoine Pitrou wrote: > > Having the Web server execute ad hoc system > > administration code is far from elegant and user-friendly. > > With the right piece of code, you could create yourself > a setuid-apache shell and solve this problem once and for > all. Well, if I can create a setuid apache shell, I can probably su as root or apache as well. ("su -c rm -r whatever") Or are you talking about a Web-based shell? From rrr at ronadam.com Tue Mar 23 02:03:30 2010 From: rrr at ronadam.com (Ron Adam) Date: Mon, 22 Mar 2010 20:03:30 -0500 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> <20100322162002.6daff86f@heresy> Message-ID: Guido van Rossum wrote: > On Mon, Mar 22, 2010 at 12:20 PM, Barry Warsaw wrote: >> On Mar 22, 2010, at 02:02 PM, Ron Adam wrote: >> >>> If I understand correctly, we would have the current mode as the default, and >>> can trigger __pycache__ behavior simply by manually creating a __pycache__ >>> directory and deleting any byte-code files in the module/program directory. > Huh? Last time I looked weren't we going to make __pycache__ the > default (and eventually only) behavior? I expect that the __pycache__ directories would quickly become the recommended "defacto default" for writing and preinitializing modules and packages, with the current behavior of having bytecode in the same directory as the .py files only as the fall-back (what I meant by default) behavior when the __pycache__ directories do not exist. > I see only two reasonable solutions for __pycache__ creation -- either > we change all setup/install scripts (both for core Python and for 3rd > party packages) to always create a __pycache__ subdirectory for every > directory (including package directories) installed; or we somehow > create it the first time it's needed. > > But creating it as needed runs into at least similar problems with > ownership as creating .pyc files when first needed (if the parent > directory is root-owned a mere mortal can't create it at all). > > So even apart from the security issue (which I haven't thought about deeply) I > think precreation should at least be an easily accessible option both > for the core (where it can be done by compileall) and for 3rd party > packages (where I guess it's up to distutils or whatever install > mechanism is used). Yes, I think that is what Antoine was also getting at. Is there a need for python to use __pycache__ directories 100% of the time? For 2.x it seems like being flexible would be best, and if 3.x is going to be strict about it, it should be strict sooner than later rather than have a lot of 3rd party packages break at some point down the road. Ron From ben+python at benfinney.id.au Tue Mar 23 02:28:36 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 23 Mar 2010 12:28:36 +1100 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask References: <20100322095604.3371039e@heresy> <20100322161626.6e56efac@heresy> Message-ID: <87vdcnrfln.fsf@benfinney.id.au> Antoine Pitrou writes: > Barry Warsaw python.org> writes: > > > > When Python is being installed, either by a from-source 'make > > install' or by the distro packager, then you'd expect the umask not > > to be insane. In the latter case, it's a bug and in the former case > > you screwed up so you should be able to delete and reinstall, or at > > the very least execute the same `find` command that Python's own > > Makefile calls (in my branch) for 'make clean'. > > > > When you're installing packages, again, I would expect that the > > system installer, or you via `easy_install` or whatever, would not > > have an insane umask. > > Well, precisely. That's why I suggest that creating the __pycache__ > directories be done *at install time* (or packaging time), and not via > the core import machinery (that is, not at import time). That is, when > you *know* you are the right user, with the right umask. +1. Taking advantage of caching directories that have already been set up correctly in advance at install time is friendly. Littering the runtime directory with new subdirectories by default is not so friendly. Perhaps also of note is that the FHS recommends systems use ?/var/cache/foo/? for cached data from applications: /var/cache : Application cache data Purpose /var/cache is intended for cached data from applications. Such data is locally generated as a result of time-consuming I/O or calculation. The application must be able to regenerate or restore the data. Unlike /var/spool, the cached files can be deleted without data loss. The data must remain valid between invocations of the application and rebooting the system. This would suggest that Python could start using ?/var/cache/python/? for its cached bytecode tree on systems that implement the FHS. -- \ ?Simplicity and elegance are unpopular because they require | `\ hard work and discipline to achieve and education to be | _o__) appreciated.? ?Edsger W. Dijkstra | Ben Finney From tjreedy at udel.edu Tue Mar 23 04:06:21 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 22 Mar 2010 23:06:21 -0400 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> Message-ID: On 3/22/2010 2:15 PM, Antoine Pitrou wrote: > What I am proposing is that the creation of __pycache__ /directories/ be put > outside of the core. It can be part of distutils, or of a separate module, or > delegated to third-party tools. It could even be as simple as > "python -m compileall --pycache", if someone implements it. > > Creation of the __pycache__ /contents/ (files inside the directory) would still > be part of core Python, but only if the directory exists and is writable by the > current process. -1 If, as I have done several times recently, I create a directory and insert an empty __init__.py and several real module.py files, I want the .pycs to go into __pycache__ *automatically, by default, without me also having to remember to create an empty __pycache__ *directory*, *each time*. Ugh. Terry Jan Reedy From doko at ubuntu.com Tue Mar 23 05:55:03 2010 From: doko at ubuntu.com (Matthias Klose) Date: Tue, 23 Mar 2010 05:55:03 +0100 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: <87vdcnrfln.fsf@benfinney.id.au> References: <20100322095604.3371039e@heresy> <20100322161626.6e56efac@heresy> <87vdcnrfln.fsf@benfinney.id.au> Message-ID: <4BA849A7.8060504@ubuntu.com> On 23.03.2010 02:28, Ben Finney wrote: > Antoine Pitrou writes: > >> Barry Warsaw python.org> writes: >>> >>> When Python is being installed, either by a from-source 'make >>> install' or by the distro packager, then you'd expect the umask not >>> to be insane. In the latter case, it's a bug and in the former case >>> you screwed up so you should be able to delete and reinstall, or at >>> the very least execute the same `find` command that Python's own >>> Makefile calls (in my branch) for 'make clean'. >>> >>> When you're installing packages, again, I would expect that the >>> system installer, or you via `easy_install` or whatever, would not >>> have an insane umask. >> >> Well, precisely. That's why I suggest that creating the __pycache__ >> directories be done *at install time* (or packaging time), and not via >> the core import machinery (that is, not at import time). That is, when >> you *know* you are the right user, with the right umask. > > +1. > > Taking advantage of caching directories that have already been set up > correctly in advance at install time is friendly. Littering the runtime > directory with new subdirectories by default is not so friendly. > > Perhaps also of note is that the FHS recommends systems use > ?/var/cache/foo/? for cached data from applications: > > /var/cache : Application cache data > > Purpose > > /var/cache is intended for cached data from applications. Such data is > locally generated as a result of time-consuming I/O or calculation. The > application must be able to regenerate or restore the data. Unlike > /var/spool, the cached files can be deleted without data loss. The data > must remain valid between invocations of the application and rebooting > the system. > > > > This would suggest that Python could start using ?/var/cache/python/? > for its cached bytecode tree on systems that implement the FHS. it reads *data*, not code. From martin at v.loewis.de Tue Mar 23 09:00:50 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 23 Mar 2010 09:00:50 +0100 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> <20100322162002.6daff86f@heresy> Message-ID: <4BA87532.10603@v.loewis.de> > Is there a need for python to use __pycache__ directories 100% of the > time? For 2.x it seems like being flexible would be best, and if 3.x > is going to be strict about it, it should be strict sooner than later > rather than have a lot of 3rd party packages break at some point down > the road. For 2.x, nothing will happen, anyway (except for Linux distributions perhaps integrating a patch on their own): 2.7b1 is about to be released, after which point 2.x will not see any new features. Regards, Martin From dickinsm at gmail.com Tue Mar 23 10:16:03 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 23 Mar 2010 09:16:03 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA80C6C.5070809@canterbury.ac.nz> References: <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> <4BA80C6C.5070809@canterbury.ac.nz> Message-ID: <5c6f2a5d1003230216o50f86e5el18380add2c03a044@mail.gmail.com> On Tue, Mar 23, 2010 at 12:33 AM, Greg Ewing wrote: > Mark Dickinson wrote: >> >> It might make sense for >> Decimal + complex mixed-type operations to be disallowed, for example. > > As long as you're allowing Decimal-float comparisons, > Decimal-complex comparison for equality has an obvious > interpretation. Agreed. Decimal-to-complex equality and inequality comparisons should be permitted. Order comparisons would raise TypeError (just as complex-to-complex and float-to-complex comparisons do at the moment.) Mark From steve at pearwood.info Tue Mar 23 12:02:11 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 23 Mar 2010 22:02:11 +1100 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> <20100322162002.6daff86f@heresy> Message-ID: <201003232202.11749.steve@pearwood.info> On Tue, 23 Mar 2010 07:38:42 am Guido van Rossum wrote: > But creating it as needed runs into at least similar problems with > ownership as creating .pyc files when first needed (if the parent > directory is root-owned a mere mortal can't create it at all). Isn't that a feature though? I don't see why this is a problem, perhaps somebody can explain what I'm missing. Current system: If the user doesn't have write permission in the appropriate directory, no .pyc file is created and the import uses the .py file only. New system (proposal): If the user doesn't have write permission in the appropriate directory, no __pycache__ folder is created and the import uses the .py file only. If the user has write permission and the __pycache__ folder is created, but the umask is screwy and no .pyc files can be created, no .pyc file is created and the import uses the .py file only despite the existence of an empty __pycache__ folder. Why bother to delete the unwritable __pycache__ directory? If you can't write to it, neither can any process with the same or fewer permissions. If some other process has more permissions, it could write something nasty to the directory, but surely it's not Python's responsibility to be secure in the face of a compromised root or admin account? As I see it, the only question is, should we warn the user that we can't write to the newly-created __pycache__ directory? I'm +0 on that. To be clear: Can't create __pycache__? That's a feature. Fail silently and fall back to using the .py file alone. Existing __pycache__ directory, but can't write to it? That's a feature too, treat it just like the above. No __pycache__ directory, and creating it succeeds, but then import can't write to the freshly created directory? That's just weird, so I'm +0 on raising a warning and -1 on raising an exception. -- Steven D'Aprano From stefan at bytereef.org Tue Mar 23 13:09:59 2010 From: stefan at bytereef.org (Stefan Krah) Date: Tue, 23 Mar 2010 13:09:59 +0100 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: Message-ID: <20100323120959.GA486@mail.bytereef.org> Facundo Batista wrote: > On Fri, Mar 19, 2010 at 5:50 PM, Guido van Rossum wrote: > > > As a downside, there is the worry that inadvertent mixing of Decimal > > and float can compromise the correctness of programs in a way that is > > hard to detect. But the anomalies above indicate that not fixing the > > Decimal already has something that we can use in this case, and fits > very nice here: Signals. I like the simplicity of having a single signal (e.g. CoercionError), but a strictness context flag could offer greater control for people who only want pure decimal/integer operations. For example: strictness 0: completely promiscuous behaviour strictness 1: current py3k behaviour strictness 2: current py3k behaviour + pure equality comparisons strictness 3: current py3k behaviour + pure equality comparisons + disallow NaN equality comparisons [1] Just as an illustration, here is a quick and dirty diff using the DefaultContext for simplicity: Index: Lib/decimal.py =================================================================== --- Lib/decimal.py (revision 78352) +++ Lib/decimal.py (working copy) @@ -3765,8 +3765,8 @@ def __init__(self, prec=None, rounding=None, traps=None, flags=None, Emin=None, Emax=None, - capitals=None, _clamp=0, - _ignored_flags=None): + capitals=None, strictness=1, + _clamp=0, _ignored_flags=None): if flags is None: flags = [] if _ignored_flags is None: @@ -5785,7 +5785,9 @@ return other if isinstance(other, int): return Decimal(other) - if raiseit: + if isinstance(other, float) and DefaultContext.strictness == 0: + return Decimal.from_float(other) + if raiseit or DefaultContext.strictness > 1: raise TypeError("Unable to convert %s to Decimal" % other) return NotImplemented @@ -5800,7 +5802,8 @@ flags=[], Emax=999999999, Emin=-999999999, - capitals=1 + capitals=1, + strictness=1 ) Stefan Krah [1] See: http://mail.python.org/pipermail/python-dev/2009-November/093910.html, http://mail.python.org/pipermail/python-dev/2009-November/093952.html From dickinsm at gmail.com Tue Mar 23 14:20:03 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 23 Mar 2010 13:20:03 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <20100323120959.GA486@mail.bytereef.org> References: <20100323120959.GA486@mail.bytereef.org> Message-ID: <5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com> On Tue, Mar 23, 2010 at 12:09 PM, Stefan Krah wrote: > Facundo Batista wrote: >> On Fri, Mar 19, 2010 at 5:50 PM, Guido van Rossum wrote: >> >> > As a downside, there is the worry that inadvertent mixing of Decimal >> > and float can compromise the correctness of programs in a way that is >> > hard to detect. But the anomalies above indicate that not fixing the >> >> Decimal already has something that we can use in this case, and fits >> very nice here: Signals. > > I like the simplicity of having a single signal (e.g. CoercionError), but > a strictness context flag could offer greater control for people who only > want pure decimal/integer operations. Sounds worth considering. > For example: > > ?strictness 0: completely promiscuous behaviour > > ?strictness 1: current py3k behaviour > > ?strictness 2: current py3k behaviour + pure equality comparisons Can you explain what you mean by "+ pure equality comparisons" here? If I'm understanding correctly, this is a mode that's *more* strict than the current py3k behaviour; what's it disallowing that the current py3k behaviour allows? > ?strictness 3: current py3k behaviour + pure equality comparisons + > ? ? ? ? ? ? ? ?disallow NaN equality comparisons [1] Sorry, no. I think there are good reasons for the current NaN equality behaviour: 2.0 really *isn't* a NaN, and Decimal(2) == Decimal('nan') should return False rather than raising an exception. And the decimal module provides compare and compare_signal for those who want complete standards-backed control here. Besides, this seems to me to be an orthogonal issue to the issue of mixing Decimal with other numeric types. Mark From ben+python at benfinney.id.au Tue Mar 23 14:35:43 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 24 Mar 2010 00:35:43 +1100 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask References: <20100322095604.3371039e@heresy> <20100322161626.6e56efac@heresy> <87vdcnrfln.fsf@benfinney.id.au> <4BA849A7.8060504@ubuntu.com> Message-ID: <87iq8nqhxs.fsf@benfinney.id.au> Matthias Klose writes: > On 23.03.2010 02:28, Ben Finney wrote: > > Perhaps also of note is that the FHS recommends systems use > > ?/var/cache/foo/? for cached data from applications: > > > > /var/cache : Application cache data > > > > Purpose > > > > /var/cache is intended for cached data from applications. Such data is > > locally generated as a result of time-consuming I/O or calculation. The > > application must be able to regenerate or restore the data. Unlike > > /var/spool, the cached files can be deleted without data loss. The data > > must remain valid between invocations of the application and rebooting > > the system. > > > > > > > > This would suggest that Python could start using ?/var/cache/python/? > > for its cached bytecode tree on systems that implement the FHS. > > it reads *data*, not code. So what? There's no implication that data-which-happens-to-be-code is unsuitable for storage in ?/var/cache/foo/?. Easily-regenerated Python byte code for caching meets the description quite well, AFAICT. -- \ ?It seems intuitively obvious to me, which means that it might | `\ be wrong.? ?Chris Torek | _o__) | Ben Finney From ncoghlan at gmail.com Tue Mar 23 15:36:57 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 24 Mar 2010 00:36:57 +1000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA7F7CB.7010603@canterbury.ac.nz> References: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> <4BA55858.2000800@canterbury.ac.nz> <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> <4BA5AA34.8050709@canterbury.ac.nz> <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> <4BA6A2B2.6050207@canterbury.ac.nz> <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <4BA7F7CB.7010603@canterbury.ac.nz> Message-ID: <4BA8D209.2040609@gmail.com> Greg Ewing wrote: > Mark Dickinson wrote: > >> But the Fraction type is going to mess this up: for Decimal + >> Fraction -> Decimal, I don't see any other sensible option than to >> convert the Fraction using the current context, since lossless >> conversion isn't generally possible. > > You could convert the Decimal to a Fraction, do the arithmetic > as Fractions, and convert the result back to Decimal using the > current precision. It gets rather messy implementation-wise if you do it that way. As Stefan pointed out, given the Decimal module's existing execution model of invoking "convert_other" to create a Decimal instance and then proceeding from there, it seems most straightforward to adopt the same approach for floats and Fractions (i.e. keep the changes to handle the new types inside the existing argument conversion algorithm). Such a change should make Decimal/float and Decimal/Fraction arithmetic work in a comprehensible way, albeit with double-rounding in the case of the latter. Comparisons would still need to be handled separately to avoid the lossy conversions. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Mar 23 15:45:15 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 24 Mar 2010 00:45:15 +1000 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: References: <20100322095604.3371039e@heresy> <4BA80418.6030905@canterbury.ac.nz> Message-ID: <4BA8D3FB.5050603@gmail.com> Antoine Pitrou wrote: >> or if a user installs by dragging into >> site-packages instead of using an installer? > > Well... do people actually do this? Yes. We do it all the time with unpackaged only-for-internal-use Python code. I wouldn't expect it to work with random packages downloaded from the 'net, but it works fine for our own stuff (since it expects to be used this way). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From brian.curtin at gmail.com Tue Mar 23 15:50:21 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 23 Mar 2010 08:50:21 -0600 Subject: [Python-Dev] Request for commit access Message-ID: Hi all, Having been active in bug triage and patch writing/reviewing since late 2009, it was suggested in the python-dev IRC channel that I request commit access to the repository. I'm primarily a Windows user and have worked with many of the other active contributors to diagnose issues and test patches when they don't have direct access to Windows. Brian Curtin p.s. My contributor form in on file as of 2010-01-31. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Mar 23 15:55:23 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 24 Mar 2010 01:55:23 +1100 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: <87iq8nqhxs.fsf@benfinney.id.au> References: <20100322095604.3371039e@heresy> <4BA849A7.8060504@ubuntu.com> <87iq8nqhxs.fsf@benfinney.id.au> Message-ID: <201003240155.24604.steve@pearwood.info> On Wed, 24 Mar 2010 12:35:43 am Ben Finney wrote: > Matthias Klose writes: > > On 23.03.2010 02:28, Ben Finney wrote: > > > Perhaps also of note is that the FHS recommends systems use > > > ?/var/cache/foo/? for cached data from applications: > > > > > > /var/cache : Application cache data > > > > > > Purpose > > > > > > /var/cache is intended for cached data from applications. > > > Such data is locally generated as a result of time-consuming I/O > > > or calculation. The application must be able to regenerate or > > > restore the data. Unlike /var/spool, the cached files can be > > > deleted without data loss. The data must remain valid between > > > invocations of the application and rebooting the system. > > > > > > > > > > >#VARCACHEAPPLICATIONCACHEDATA> > > > > > > This would suggest that Python could start using > > > ?/var/cache/python/? for its cached bytecode tree on systems that > > > implement the FHS. > > > > it reads *data*, not code. > > So what? There's no implication that data-which-happens-to-be-code is > unsuitable for storage in ?/var/cache/foo/?. Easily-regenerated > Python byte code for caching meets the description quite well, > AFAICT. While I strongly approve of the concept of a central cache directory for many things, I don't think that .pyc files fit the bill. Since there is no privileged python user that runs all Python code, and since any unprivileged user needs to be able to write .pyc files, the cache directory needs to be world-writable. But since the data being stored is code, that opens up a fairly nasty security hole: user fred could overwrite the cached .pyc files used by user barney and cause barney to run any arbitrary code fred likes. The alternative would be to have individual caches for every user. Apart from being wasteful of disk space ("but who cares, bigger disks are cheap") that just complicates everything. You would need: /var/cache/python// which would essentially make it impossible to ship pre-compiled .pyc files, since the packaging system couldn't predict what usernames (note plural) to store them under. It is not true that one can necessarily delete the cached files without data loss. Python still supports .pyc-only packages, and AFAIK there are no plans to stop that, so deleting the .pyc file may delete the module. -- Steven D'Aprano From ezio.melotti at gmail.com Tue Mar 23 15:58:26 2010 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Tue, 23 Mar 2010 16:58:26 +0200 Subject: [Python-Dev] Request for commit access In-Reply-To: References: Message-ID: <4BA8D712.9070802@gmail.com> On 23/03/2010 16.50, Brian Curtin wrote: > Hi all, > > Having been active in bug triage and patch writing/reviewing since > late 2009, it was suggested in the python-dev IRC channel that I > request commit access to the repository. I'm primarily a Windows user > and have worked with many of the other active contributors to diagnose > issues and test patches when they don't have direct access to Windows. > > Brian Curtin > > p.s. My contributor form in on file as of 2010-01-31. +1 From dirkjan at ochtman.nl Tue Mar 23 16:05:00 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Tue, 23 Mar 2010 16:05:00 +0100 Subject: [Python-Dev] Request for commit access In-Reply-To: References: Message-ID: On Tue, Mar 23, 2010 at 15:50, Brian Curtin wrote: > Having been active in bug triage and patch writing/reviewing since late > 2009, it was suggested in the python-dev IRC channel that I request commit > access to the repository. I'm primarily a Windows user and have worked with > many of the other active contributors to diagnose issues and test patches > when they don't have direct access to Windows. I have usefully cooperated with Brian, so +1. Cheers, Dirkjan From eric at trueblade.com Tue Mar 23 16:09:10 2010 From: eric at trueblade.com (Eric Smith) Date: Tue, 23 Mar 2010 11:09:10 -0400 Subject: [Python-Dev] Request for commit access In-Reply-To: References: Message-ID: <4BA8D996.1060305@trueblade.com> Brian Curtin wrote: > Hi all, > > Having been active in bug triage and patch writing/reviewing since late > 2009, it was suggested in the python-dev IRC channel that I request > commit access to the repository. I'm primarily a Windows user and have > worked with many of the other active contributors to diagnose issues and > test patches when they don't have direct access to Windows. > > Brian Curtin Having worked with Brian at PyCon and via email, I support this. Especially if it can be done soon so he can work on committing http://bugs.python.org/issue1578269, which I have not had time to focus on. -- Eric. From stefan at bytereef.org Tue Mar 23 16:09:50 2010 From: stefan at bytereef.org (Stefan Krah) Date: Tue, 23 Mar 2010 16:09:50 +0100 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com> References: <20100323120959.GA486@mail.bytereef.org> <5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com> Message-ID: <20100323150950.GA1562@yoda.bytereef.org> Mark Dickinson wrote: > > I like the simplicity of having a single signal (e.g. CoercionError), but > > a strictness context flag could offer greater control for people who only > > want pure decimal/integer operations. > > Sounds worth considering. > > > For example: > > > > ?strictness 0: completely promiscuous behaviour > > > > ?strictness 1: current py3k behaviour > > > > ?strictness 2: current py3k behaviour + pure equality comparisons > > Can you explain what you mean by "+ pure equality comparisons" here? > If I'm understanding correctly, this is a mode that's *more* strict > than the current py3k behaviour; what's it disallowing that the > current py3k behaviour allows? It's disallowing all comparisons between e.g. float and decimal. The idea is that the context can provide a cheap way of enforcing types for people who like it: >>> from decimal import * >>> DefaultContext.strictness = 1 >>> Decimal(9) == 9.0 False >>> Decimal(9) in [1, 4.0 ,9] True >>> DefaultContext.strictness = 2 >>> Decimal(9) == 9.0 Traceback (most recent call last): File "", line 1, in File "/home/stefan/svn/py3k/Lib/decimal.py", line 858, in __eq__ other = _convert_other(other) File "/home/stefan/svn/py3k/Lib/decimal.py", line 5791, in _convert_other raise TypeError("Unable to convert %s to Decimal" % other) TypeError: Unable to convert 9.0 to Decimal >>> Decimal(9) in [1, 4.0 ,9] Traceback (most recent call last): File "", line 1, in File "/home/stefan/svn/py3k/Lib/decimal.py", line 858, in __eq__ other = _convert_other(other) File "/home/stefan/svn/py3k/Lib/decimal.py", line 5791, in _convert_other raise TypeError("Unable to convert %s to Decimal" % other) TypeError: Unable to convert 4.0 to Decimal >>> Decimal(9) in [1, 4 ,9] True This mode could help catch bugs like: n = 7 / 3 # Programmer thinks this is an integer x = Decimal(100) while x != n: pass # do something x -= 1 > > ?strictness 3: current py3k behaviour + pure equality comparisons + > > ? ? ? ? ? ? ? ?disallow NaN equality comparisons [1] > > Sorry, no. I think there are good reasons for the current NaN > equality behaviour: 2.0 really *isn't* a NaN, and Decimal(2) == > Decimal('nan') should return False rather than raising an exception. > And the decimal module provides compare and compare_signal for those > who want complete standards-backed control here. I'd like to make it an option for people who don't want to write: while x.compare_signal(7) != 0 And I think that an sNaN should really signal by default. > Besides, this seems > to me to be an orthogonal issue to the issue of mixing Decimal with > other numeric types. Yes, it would kind of overload the strictness parameter. I see it as another type of strictness, so I brought it up here. Level 3 would be a bit like the highest warning level of a compiler. But of course there's no need to discuss NaNs further in this thread other than to show a possible use of the flag. Stefan Krah From rdmurray at bitdance.com Tue Mar 23 16:27:48 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 23 Mar 2010 11:27:48 -0400 Subject: [Python-Dev] Request for commit access In-Reply-To: References: Message-ID: <20100323152748.50ECE1FD33F@kimball.webabinitio.net> On Tue, 23 Mar 2010 08:50:21 -0600, Brian Curtin wrote: > Having been active in bug triage and patch writing/reviewing since late > 2009, it was suggested in the python-dev IRC channel that I request commit > access to the repository. I'm primarily a Windows user and have worked with > many of the other active contributors to diagnose issues and test patches > when they don't have direct access to Windows. > > Brian Curtin +1 from me too. I've been working with Brian in IRC, and also at the Pycon sprint. Brian is careful with his code and very responsive to feedback. And we need more Windows devs :) -- R. David Murray www.bitdance.com From bob at redivi.com Tue Mar 23 16:43:11 2010 From: bob at redivi.com (Bob Ippolito) Date: Tue, 23 Mar 2010 11:43:11 -0400 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: <5c6f2a5d1003201338p18aebd40y2c345b6dc8b861a6@mail.gmail.com> References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> <4BA29FB1.9020909@gmail.com> <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com> <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com> <4BA3680D.8070909@gmail.com> <4BA41271.3010003@canterbury.ac.nz> <5c6f2a5d1003200416i59ce880fk9cec234caaac2eb0@mail.gmail.com> <5c6f2a5d1003201338p18aebd40y2c345b6dc8b861a6@mail.gmail.com> Message-ID: <6a36e7291003230843k3fa44402ob806d5fa47245436@mail.gmail.com> On Sat, Mar 20, 2010 at 4:38 PM, Mark Dickinson wrote: > On Sat, Mar 20, 2010 at 7:56 PM, Guido van Rossum wrote: >> I propose to reduce all hashes to the hash of a normalized fraction, >> which we can define as a combination of the hashes for the numerator >> and the denominator. Then all we have to do is figure fairly efficient >> ways to convert floats and decimals to normalized fractions (not >> necessarily Fractions). I may be naive but this seems doable: for a >> float, the denominator is always a power of 2 and removing factors of >> 2 from the denominator is easy (just right-shift until the last bit is >> zero). For Decimal, the unnormalized denominator is always a power of >> 10, and the normalization is a bit messier, but doesn't seem >> excessively so. The resulting numerator and denominator may be large >> numbers, but for typical use of Decimal and float they will rarely be >> excessively large, and I'm not too worried about slowing things down >> when they are (everything slows down when you're using really large >> integers anyway). > > I *am* worried about slowing things down for large Decimals: ?if you > can't put Decimal('1e1234567') into a dict or set without waiting for > an hour for the hash computation to complete (because it's busy > computing 10**1234567), I consider that a problem. > > But it's solvable! ?I've just put a patch on the bug tracker: > > http://bugs.python.org/issue8188 > > It demonstrates how hashes can be implemented efficiently and > compatibly for all numeric types, even large Decimals like the above. > It needs a little tidying up, but it works. I was interested in how the implementation worked yesterday, especially given the lack of explanation in the margins of numeric_hash3.patch. numeric_hash4.patch has much better comments, but I didn't see this patch until after I had sufficiently deciphered the previous patch and wrote most of this: http://bob.pythonmac.org/archives/2010/03/23/py3k-unified-numeric-hash/ I'm not really qualified to review the patch, what little formal math training I had has atrophied quite a bit over the years, but as far as I can tell it seems to work. The results also seem to match the Python implementations that I created. -bob From raymond.hettinger at gmail.com Tue Mar 23 18:01:14 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Tue, 23 Mar 2010 10:01:14 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <20100323120959.GA486@mail.bytereef.org> References: <20100323120959.GA486@mail.bytereef.org> Message-ID: On Mar 23, 2010, at 5:09 AM, Stefan Krah wrote: > > I like the simplicity of having a single signal (e.g. CoercionError), but > a strictness context flag could offer greater control for people who only > want pure decimal/integer operations. > > > For example: > > strictness 0: completely promiscuous behaviour > > strictness 1: current py3k behaviour > > strictness 2: current py3k behaviour + pure equality comparisons > > strictness 3: current py3k behaviour + pure equality comparisons + > disallow NaN equality comparisons [1] > The decimal module is already drowning in complexity, so it would be best to keep it simple: one boolean flag that if set would warn about any implicit decimal/float interaction. Raymond From dickinsm at gmail.com Tue Mar 23 18:31:40 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 23 Mar 2010 17:31:40 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <20100323150950.GA1562@yoda.bytereef.org> References: <20100323120959.GA486@mail.bytereef.org> <5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com> <20100323150950.GA1562@yoda.bytereef.org> Message-ID: <5c6f2a5d1003231031s3482b1a7w6a58d8c1afeb34e8@mail.gmail.com> On Tue, Mar 23, 2010 at 3:09 PM, Stefan Krah wrote: > Mark Dickinson wrote: >> [Stefan] >> > >> > ?strictness 2: current py3k behaviour + pure equality comparisons >> >> Can you explain what you mean by "+ pure equality comparisons" here? >> If I'm understanding correctly, this is a mode that's *more* strict >> than the current py3k behaviour; ?what's it disallowing that the >> current py3k behaviour allows? > > It's disallowing all comparisons between e.g. float and decimal. The idea > is that the context can provide a cheap way of enforcing types for people > who like it: >>>> DefaultContext.strictness = 2 >>>> Decimal(9) == 9.0 > Traceback (most recent call last): > ?File "", line 1, in > ?File "/home/stefan/svn/py3k/Lib/decimal.py", line 858, in __eq__ > ? ?other = _convert_other(other) > ?File "/home/stefan/svn/py3k/Lib/decimal.py", line 5791, in _convert_other > ? ?raise TypeError("Unable to convert %s to Decimal" % other) > TypeError: Unable to convert 9.0 to Decimal Hmm. It seems to me that deliberately making an __eq__ method between hashable types raise an exception isn't something that should be done lightly, since it can *really* screw up sets and dicts. For example, with your proposal, {9.0, Decimal(x)} would either raise or not, depending on whether Decimal(x) happened to hash equal to 9.0 (if they don't hash equal, then __eq__ will never be called). If the hash is regarded as essentially a black box (which is what it should be for most users) then you can easily end up with code that almost always works, but *very* occasionally and unpredicatably raises an exception. > And I think that an sNaN should really signal by default. Agreed, notwithstanding the above comments. Though to avoid the problems described above, I think the only way to make this acceptable would be to prevent hashing of signaling nans. (Which the decimal module current does; it also prevents hashing of quiet NaNs, but I can't see any good rationale for that.) Mark From rhamph at gmail.com Tue Mar 23 18:48:20 2010 From: rhamph at gmail.com (Adam Olsen) Date: Tue, 23 Mar 2010 11:48:20 -0600 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003231031s3482b1a7w6a58d8c1afeb34e8@mail.gmail.com> References: <20100323120959.GA486@mail.bytereef.org> <5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com> <20100323150950.GA1562@yoda.bytereef.org> <5c6f2a5d1003231031s3482b1a7w6a58d8c1afeb34e8@mail.gmail.com> Message-ID: On Tue, Mar 23, 2010 at 11:31, Mark Dickinson wrote: > Agreed, notwithstanding the above comments. ?Though to avoid the > problems described above, I think the only way to make this acceptable > would be to prevent hashing of signaling nans. ?(Which the decimal > module current does; it also prevents hashing of quiet NaNs, but I > can't see any good rationale for that.) a = Decimal('nan') a != a They don't follow the behaviour required for being hashable. float NaN should stop being hashable as well. -- Adam Olsen, aka Rhamphoryncus From dickinsm at gmail.com Tue Mar 23 19:04:37 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 23 Mar 2010 18:04:37 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <20100323120959.GA486@mail.bytereef.org> <5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com> <20100323150950.GA1562@yoda.bytereef.org> <5c6f2a5d1003231031s3482b1a7w6a58d8c1afeb34e8@mail.gmail.com> Message-ID: <5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com> On Tue, Mar 23, 2010 at 5:48 PM, Adam Olsen wrote: > a = Decimal('nan') > a != a > > They don't follow the behaviour required for being hashable. What's this required behaviour? The only rule I'm aware of is that if a == b then hash(a) == hash(b). That's not violated here. Note that containment tests check identity before equality, so there's no problem with putting (float) nans in sets or dicts: >>> x = float('nan') >>> s = {x} >>> x in s True Mark From rhamph at gmail.com Tue Mar 23 19:07:44 2010 From: rhamph at gmail.com (Adam Olsen) Date: Tue, 23 Mar 2010 12:07:44 -0600 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com> References: <20100323120959.GA486@mail.bytereef.org> <5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com> <20100323150950.GA1562@yoda.bytereef.org> <5c6f2a5d1003231031s3482b1a7w6a58d8c1afeb34e8@mail.gmail.com> <5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com> Message-ID: On Tue, Mar 23, 2010 at 12:04, Mark Dickinson wrote: > On Tue, Mar 23, 2010 at 5:48 PM, Adam Olsen wrote: >> a = Decimal('nan') >> a != a >> >> They don't follow the behaviour required for being hashable. > > What's this required behaviour? ?The only rule I'm aware of is that if > a == b then hash(a) == hash(b). ?That's not violated here. > > Note that containment tests check identity before equality, so there's > no problem with putting (float) nans in sets or dicts: > >>>> x = float('nan') >>>> s = {x} >>>> x in s > True Ergh, I thought that got changed. Nevermind then. -- Adam Olsen, aka Rhamphoryncus From dickinsm at gmail.com Tue Mar 23 19:55:36 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 23 Mar 2010 18:55:36 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <20100323120959.GA486@mail.bytereef.org> <5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com> <20100323150950.GA1562@yoda.bytereef.org> <5c6f2a5d1003231031s3482b1a7w6a58d8c1afeb34e8@mail.gmail.com> <5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com> Message-ID: <5c6f2a5d1003231155n211ee51fke7b62ced6f5d36c5@mail.gmail.com> On Tue, Mar 23, 2010 at 6:07 PM, Adam Olsen wrote: > On Tue, Mar 23, 2010 at 12:04, Mark Dickinson wrote: >> Note that containment tests check identity before equality, so there's >> no problem with putting (float) nans in sets or dicts: >> >>>>> x = float('nan') >>>>> s = {x} >>>>> x in s >> True > > Ergh, I thought that got changed. ?Nevermind then. Hmm. I think you're right: it did get changed at some point early in py3k's history; I seem to recall that the identity-checking behaviour got restored before 3.1 was released, though. There was an issue about this somewhere, but I'm failing to find it. Mark From rowen at uw.edu Tue Mar 23 20:49:26 2010 From: rowen at uw.edu (Russell E. Owen) Date: Tue, 23 Mar 2010 12:49:26 -0700 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask References: <20100322095604.3371039e@heresy> <4BA80418.6030905@canterbury.ac.nz> Message-ID: In article <4BA80418.6030905 at canterbury.ac.nz>, Greg Ewing wrote: > Antoine Pitrou wrote: > > > In light of this issue, I'm -0.5 on __pycache__ becoming the default > > caching > > mechanism. The directory ownership/permissions issue is too much of a mess, > > especially for Web applications (think __pycache__ files created by the > > Apache > > user). > > Doesn't the existing .pyc mechanism have the same problem? Seems > to me it's just as insecure to allow the Apache user to create > .pyc files, since an attacker could overwrite them with arbitrary > bytecode. > > The only safe way is to pre-compile under a different user and > make everything read-only to Apache. The same thing would apply > under the __pycache__ regime. This does sound like a bit security hole both in existing Python and the new __pycache__ proposed mechanism. It seems like this is the time to address it, while changing the caching mechanism. If .pyc files are to be shared, it seems essential to (by default) generate them at install time and make them read-only for unprivileged users. This in turn implies that we may have to give up some support for dragging python modules into site-packages, e.g. not generate .pyc files for such modules. At least if we go that route it will mostly affect power users, who can presumably cope. -- Russell From guido at python.org Tue Mar 23 21:19:14 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 23 Mar 2010 12:19:14 -0800 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003231155n211ee51fke7b62ced6f5d36c5@mail.gmail.com> References: <20100323120959.GA486@mail.bytereef.org> <5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com> <20100323150950.GA1562@yoda.bytereef.org> <5c6f2a5d1003231031s3482b1a7w6a58d8c1afeb34e8@mail.gmail.com> <5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com> <5c6f2a5d1003231155n211ee51fke7b62ced6f5d36c5@mail.gmail.com> Message-ID: On Tue, Mar 23, 2010 at 10:55 AM, Mark Dickinson wrote: > On Tue, Mar 23, 2010 at 6:07 PM, Adam Olsen wrote: >> On Tue, Mar 23, 2010 at 12:04, Mark Dickinson wrote: >>> Note that containment tests check identity before equality, so there's >>> no problem with putting (float) nans in sets or dicts: >>> >>>>>> x = float('nan') >>>>>> s = {x} >>>>>> x in s >>> True >> >> Ergh, I thought that got changed. ?Nevermind then. > > Hmm. ?I think you're right: ?it did get changed at some point early in > py3k's history; ?I seem to recall that the identity-checking behaviour > got restored before 3.1 was released, though. ?There was an issue > about this somewhere, but I'm failing to find it. Raymond and I don't see this the same way. It looks like he won. :-) -- --Guido van Rossum (python.org/~guido) From henning.vonbargen at arcor.de Tue Mar 23 21:27:47 2010 From: henning.vonbargen at arcor.de (Henning von Bargen) Date: Tue, 23 Mar 2010 21:27:47 +0100 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: Message-ID: <4BA92443.4050204@arcor.de> Antoine Pitrou wrote: > > or if a user installs by dragging into > > site-packages instead of using an installer? > > Well... do people actually do this? Why not? And it is also common to just set the PYTHONPATH environment variable instead of using setup.py install or copy the files to the lib/site-packages directory. I prefer it that way because it gives a clean separation between standard library files and application specific files. For example, if I want to use another Python version, I can just delete the python26 directory because I know it doesn't contain any application-specific code, just the standard library files. Considering the complexity, the risks and the benefit of the whole __pycache__ thingie, I'm -1... Henning From chambon.pascal at wanadoo.fr Tue Mar 23 21:57:22 2010 From: chambon.pascal at wanadoo.fr (Pascal Chambon) Date: Tue, 23 Mar 2010 21:57:22 +0100 Subject: [Python-Dev] Attribute lookup ambiguity In-Reply-To: <4BA80B97.6080606@canterbury.ac.nz> References: <4BA3C962.7080500@wanadoo.fr> <4BA3DD4F.1080209@voidspace.org.uk> <4BA4B8DE.8060109@wanadoo.fr> <4BA576AF.2050208@voidspace.org.uk> <4BA7B1B4.4010000@wanadoo.fr> <4BA80B97.6080606@canterbury.ac.nz> Message-ID: <4BA92B32.8050803@wanadoo.fr> Greg Ewing a ?crit : > > Pascal Chambon wrote: > >> I don't follow you there - in my mind, the default __getattribute__ >> could simply have wrapped all its operations inside soem kind of >> "try..catch AttributeError:" mechanism, and thus been able to >> fallback to __getattr__ in any way. > > But then it would be incorrect to say that "__getattribute__ > raises an exception". > > When we say that a function raises an exception, we normally > mean that the exception propagates out of the function and > can be seen by the caller, not that it was raised and caught > somewhere inside the function. > Indeed, but I've never run into any doc mentionning that the default __getattribute__ raised in exception instead of forwarding to __getattr__ by itself. All I've found is "If the class also defines __getattr__() , the latter will not be called unless __getattribute__() either calls it explicitly or raises an AttributeError "; that sentence which simply offers two alternatives for the behaviour of customized __gettattribute__ methods, without giving any hint on the behaviourthat was chosen when implementing object.__gettattribute__. Or am I missing some other doc which I'm supposed to know :? "In the face of ambiguity, refuse the temptation to guess", as we say anyway, so I propose we patch the doc to clarify this point for newcomers ^^ Regards, Pascal -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Mar 23 22:41:20 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 24 Mar 2010 09:41:20 +1200 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: References: <20100322095604.3371039e@heresy> <4BA80418.6030905@canterbury.ac.nz> Message-ID: <4BA93580.9000402@canterbury.ac.nz> Antoine Pitrou wrote: > The main point of the __pycache__ proposal is to solve the needs of > Ubuntu/Debian packagers. If you are developing (rather than deploying or > building packages), you shouldn't have these needs AFAICT. Maybe it's one point, but I'm not sure it's the *main* one. Personally I would benefit most from it during development. I hardly ever look in the directories of installed packages, so I don't care what they look like. -- Greg From greg.ewing at canterbury.ac.nz Tue Mar 23 22:48:51 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 24 Mar 2010 09:48:51 +1200 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> <20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de> <4BA7E04F.7070308@v.loewis.de> <4BA7EC9A.1010206@v.loewis.de> <1269298415.5792.11.camel@localhost> <4BA811F7.2010003@canterbury.ac.nz> Message-ID: <4BA93743.90006@canterbury.ac.nz> Antoine Pitrou wrote: > Well, if I can create a setuid apache shell, I can probably su as root or apache > as well. > ("su -c rm -r whatever") > > Or are you talking about a Web-based shell? I'm just saying that if there is any way of running code of your choice as the apache user, you can get it to make a copy of /bin/sh and suid it. Of course, if you have permission to su apache, then this is not necessary. But then you wouldn't have to go through web server contortions to fix apache-generated botchups either. -- Greg From steve at pearwood.info Tue Mar 23 23:00:30 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 24 Mar 2010 09:00:30 +1100 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com> References: <5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com> Message-ID: <201003240900.30334.steve@pearwood.info> On Wed, 24 Mar 2010 05:04:37 am Mark Dickinson wrote: > On Tue, Mar 23, 2010 at 5:48 PM, Adam Olsen wrote: > > a = Decimal('nan') > > a != a > > > > They don't follow the behaviour required for being hashable. > > What's this required behaviour? The only rule I'm aware of is that > if a == b then hash(a) == hash(b). That's not violated here. > > Note that containment tests check identity before equality, so > there's no problem with putting (float) nans in sets or dicts: > >>> x = float('nan') > >>> s = {x} > >>> x in s > True As usual though, NANs are unintuitive: >>> d = {float('nan'): 1} >>> d[float('nan')] = 2 >>> d {nan: 1, nan: 2} I suspect that's a feature, not a bug. -- Steven D'Aprano From greg.ewing at canterbury.ac.nz Tue Mar 23 23:14:12 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 24 Mar 2010 10:14:12 +1200 Subject: [Python-Dev] __pycache__ creation In-Reply-To: <201003232202.11749.steve@pearwood.info> References: <20100322095604.3371039e@heresy> <20100322162002.6daff86f@heresy> <201003232202.11749.steve@pearwood.info> Message-ID: <4BA93D34.3090407@canterbury.ac.nz> Steven D'Aprano wrote: > If the user has write permission and the __pycache__ folder is created, > but the umask is screwy and no .pyc files can be created, no .pyc file > is created and the import uses the .py file only despite the existence > of an empty __pycache__ folder. Sounds okay to me. -- Greg From greg.ewing at canterbury.ac.nz Tue Mar 23 23:37:12 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 24 Mar 2010 10:37:12 +1200 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: References: <20100322095604.3371039e@heresy> <4BA80418.6030905@canterbury.ac.nz> Message-ID: <4BA94298.2060904@canterbury.ac.nz> Russell E. Owen wrote: > If .pyc files are to be shared, it seems essential to (by default) > generate them at install time and make them read-only for unprivileged > users. > > This in turn implies that we may have to give up some support for > dragging python modules into site-packages No, I don't think so. Currently, when you install a package (by whatever means) in a directory that's not writable by the people who will be running it, there are two possibilities: 1) Precompiled .pyc files are generated at installation time, which are then read-only to users. 2) No .pyc files are installed, in which case none can or will be created by users either, since they don't have write permission to the directory. None of this would change if __pycache__ directories were used. The only difference would be that there would be an additional mode for failure to create .pyc files, i.e. __pycache__ could be created but nothing could be written to it because of a umask issue. If you install in a shared site-packages by dragging, you already have to be careful about setting the permissions. You'd just have to be sure to extended that diligence to any contained __pycache__ directories. -- Greg From barry at python.org Tue Mar 23 23:32:12 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 23 Mar 2010 18:32:12 -0400 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> <20100322162002.6daff86f@heresy> Message-ID: <20100323183212.7463e098@heresy> On Mar 22, 2010, at 12:38 PM, Guido van Rossum wrote: >Huh? Last time I looked weren't we going to make __pycache__ the >default (and eventually only) behavior? We definitely agreed it would be the default in Python 3.2. My recollection is that we agreed it would be the only on-demand way of writing pyc files, but that Python would read a lone .pyc file where the source would be if the source is missing, and that py_compile/compileall would support optional creation of those lone .pyc files. >I see only two reasonable solutions for __pycache__ creation -- either >we change all setup/install scripts (both for core Python and for 3rd >party packages) to always create a __pycache__ subdirectory for every >directory (including package directories) installed; or we somehow >create it the first time it's needed. > >But creating it as needed runs into at least similar problems with >ownership as creating .pyc files when first needed (if the parent >directory is root-owned a mere mortal can't create it at all). So even >apart from the security issue (which I haven't thought about deeply) I >think precreation should at least be an easily accessible option both >for the core (where it can be done by compileall) and for 3rd party >packages (where I guess it's up to distutils or whatever install >mechanism is used). So you're +1 on Tough Luck? I think that's the best answer. You will be able to arrange pre-creation though compileall with the right layout and presumably the right umask if you want. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From martin at v.loewis.de Tue Mar 23 23:34:42 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 23 Mar 2010 23:34:42 +0100 Subject: [Python-Dev] Request for commit access In-Reply-To: <20100323152748.50ECE1FD33F@kimball.webabinitio.net> References: <20100323152748.50ECE1FD33F@kimball.webabinitio.net> Message-ID: <4BA94202.6060502@v.loewis.de> >> Having been active in bug triage and patch writing/reviewing since late >> 2009, it was suggested in the python-dev IRC channel that I request commit >> access to the repository. I'm primarily a Windows user and have worked with >> many of the other active contributors to diagnose issues and test patches >> when they don't have direct access to Windows. >> >> Brian Curtin > > +1 from me too. I've been working with Brian in IRC, and also at the Pycon > sprint. Brian is careful with his code and very responsive to feedback. > And we need more Windows devs :) Procedurally, I wonder where people got the notion from that you can or need to apply for commit access. IIUC, it used to be the case that you would be recommended for commit access, by some (more or less senior) fellow committer. That person then would work on actually getting commit access to the new committer - perhaps by first asking other people in private, to avoid any public embarrassment if access is ultimately denied. IMO, that committer should then also mentor the new guy, both by helping out in difficult cases, and by closely following commits to see whether (possibly unstated) conventions are being followed. I'm not really picking on Brian here specifically, I just want to point out that I dislike this (apparent) change in process, primarily because of the risk of embarrassment. With all that said: I'm also fine with giving access to Brian (in particular due to the endorsements that have already been posted). I still would like somebody to step forward as a mentor, and I would need Brian to send me his SSH key. Regards, Martin From greg.ewing at canterbury.ac.nz Tue Mar 23 23:47:00 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 24 Mar 2010 10:47:00 +1200 Subject: [Python-Dev] Attribute lookup ambiguity In-Reply-To: <4BA92B32.8050803@wanadoo.fr> References: <4BA3C962.7080500@wanadoo.fr> <4BA3DD4F.1080209@voidspace.org.uk> <4BA4B8DE.8060109@wanadoo.fr> <4BA576AF.2050208@voidspace.org.uk> <4BA7B1B4.4010000@wanadoo.fr> <4BA80B97.6080606@canterbury.ac.nz> <4BA92B32.8050803@wanadoo.fr> Message-ID: <4BA944E4.9000403@canterbury.ac.nz> Pascal Chambon wrote: > All I've found is "If the class also defines __getattr__(), > the latter will not be called unless __getattribute__() > either calls it explicitly or raises an AttributeError Hmmm. Well, it still implies that there is some mechanism outside of __getattribute__ that will catch an AttributeError and call __getattr__ for you. Given the existence of that mechanism, it seems unlikely that the same thing would be implemented over again in the standard __getattribute__ method. So I don't think it requires very much guesswork to infer that the standard __getattribute__ won't call __getattr__ on its own. I concede that the wording could be improved to remove any possibility of doubt, though. -- Greg From barry at python.org Tue Mar 23 23:37:16 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 23 Mar 2010 18:37:16 -0400 Subject: [Python-Dev] __pycache__ creation In-Reply-To: <4BA7D76C.9010502@v.loewis.de> References: <20100322095604.3371039e@heresy> <20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de> Message-ID: <20100323183716.1e9aac6f@heresy> On Mar 22, 2010, at 09:47 PM, Martin v. L?wis wrote: >Therefore, I'm in favor of having it on by default. If certain use cases >make it problematic (e.g. Apache creating directories which you then >cannot delete), there should be a way to turn it *off*. Perhaps the >existing machinery to turn of byte code generation at all might be >sufficient. That's what I'm thinking. Of course it will always be possible to run compileall and get the __pycache__ directories pre-created, presumably with the right umask. >As for the original question (funny umasks), I think my proposal is >"tough luck". Don't try to be super-smart; as Antoine explains, it gets >worse, not better. If the user has arranged that Python will create >unusable directories, the user better changes his setup. I completely agree; +1 -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Tue Mar 23 23:42:22 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 23 Mar 2010 18:42:22 -0400 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> <20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de> <4BA7E04F.7070308@v.loewis.de> Message-ID: <20100323184222.28cb9a3a@heresy> On Mar 22, 2010, at 09:57 PM, Antoine Pitrou wrote: >It's especially annoying, of course, if you have to ask someone else to >remove the directories for you (or if you have to write custom code and get >it executed by the Apache or WSGI handler...). compileall should probably grow a --clean option which would be essentially equivalent to find . -name '__pycache__' | xargs rmdir -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From solipsis at pitrou.net Tue Mar 23 23:57:48 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 23 Mar 2010 22:57:48 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?PEP_3147=2C_=5F=5Fpycache=5F=5F_directorie?= =?utf-8?q?s_and_umask?= References: <20100322095604.3371039e@heresy> <4BA80418.6030905@canterbury.ac.nz> <4BA93580.9000402@canterbury.ac.nz> Message-ID: Greg Ewing canterbury.ac.nz> writes: > > The main point of the __pycache__ proposal is to solve the needs of > > Ubuntu/Debian packagers. If you are developing (rather than deploying or > > building packages), you shouldn't have these needs AFAICT. > > Maybe it's one point, but I'm not sure it's the *main* one. It's the only reason the PEP was originally designed, and proposed. > Personally I would benefit most from it during development. Why? What benefit would it bring to you? > I hardly ever look in the directories of installed packages, > so I don't care what they look like. Neither do I, but Ubuntu/Debian packagers want to share the source code of third-party libraries across Python versions while keeping distinct bytecode files (obviously). Again, that's the original point of the PEP as proposed by Barry. From barry at python.org Wed Mar 24 00:01:22 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 23 Mar 2010 19:01:22 -0400 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: References: <20100322095604.3371039e@heresy> <20100322161626.6e56efac@heresy> Message-ID: <20100323190122.54513b4a@heresy> On Mar 22, 2010, at 08:33 PM, Antoine Pitrou wrote: >Well, precisely. That's why I suggest that creating the __pycache__ >directories be done *at install time* (or packaging time), and not via the >core import machinery (that is, not at import time). That is, when you *know* >you are the right user, with the right umask. I don't think they're mutually exclusive. We will definitely give users the tool to do compilation at install time via compileall. That needn't preclude on-demand creation, which will generally Just Work. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Mar 24 00:02:47 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 23 Mar 2010 19:02:47 -0400 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: <87iq8nqhxs.fsf@benfinney.id.au> References: <20100322095604.3371039e@heresy> <20100322161626.6e56efac@heresy> <87vdcnrfln.fsf@benfinney.id.au> <4BA849A7.8060504@ubuntu.com> <87iq8nqhxs.fsf@benfinney.id.au> Message-ID: <20100323190247.29be43c7@heresy> On Mar 24, 2010, at 12:35 AM, Ben Finney wrote: >So what? There's no implication that data-which-happens-to-be-code is >unsuitable for storage in ?/var/cache/foo/?. Easily-regenerated Python >byte code for caching meets the description quite well, AFAICT. pyc files don't go there now, so why would PEP 3147 change that? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From solipsis at pitrou.net Wed Mar 24 00:01:43 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 23 Mar 2010 23:01:43 +0000 (UTC) Subject: [Python-Dev] Request for commit access References: <20100323152748.50ECE1FD33F@kimball.webabinitio.net> <4BA94202.6060502@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > Procedurally, I wonder where people got the notion from that you can or > need to apply for commit access. IIUC, it used to be the case that you > would be recommended for commit access, by some (more or less senior) > fellow committer. That person then would work on actually getting commit > access to the new committer - perhaps by first asking other people in > private, to avoid any public embarrassment if access is ultimately > denied. IMO, that committer should then also mentor the new guy, both by > helping out in difficult cases, and by closely following commits to see > whether (possibly unstated) conventions are being followed. > > I'm not really picking on Brian here specifically, I just want to point > out that I dislike this (apparent) change in process, primarily because > of the risk of embarrassment. For the record, I'm not opposing any point you are making, but all this is not clearly written out, and I think that's why people (including me) lately have been thinking that the candidate for commit rights had to declare himself on this mailing-list. If the process goes differently, it probably deserves documenting somewhere. Regards Antoine. From barry at python.org Wed Mar 24 00:07:24 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 23 Mar 2010 19:07:24 -0400 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: References: <20100322095604.3371039e@heresy> <4BA80418.6030905@canterbury.ac.nz> Message-ID: <20100323190724.46d5dc03@heresy> On Mar 23, 2010, at 12:49 PM, Russell E. Owen wrote: >If .pyc files are to be shared, it seems essential to (by default) >generate them at install time and make them read-only for unprivileged >users. I think in practice this is what's almost always going to happen for system Python source, either via your distribution's installer or distutils. I think most on-demand creation of pyc files will be when you're developing your own code. I don't want to have to run a separate step to get the benefit of __pycache__, but I admit it's outside the scope of the PEP's original intention. I leave it to the BDFL. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ben+python at benfinney.id.au Wed Mar 24 00:16:30 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 24 Mar 2010 10:16:30 +1100 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask References: <20100322095604.3371039e@heresy> <4BA80418.6030905@canterbury.ac.nz> Message-ID: <87eijar5m9.fsf@benfinney.id.au> Antoine Pitrou writes: Steven D'Aprano writes: > On Wed, 24 Mar 2010 12:35:43 am Ben Finney wrote: > > > On 23.03.2010 02:28, Ben Finney wrote: > > > > > > >#VARCACHEAPPLICATIONCACHEDATA> > > > > > > > > This would suggest that Python could start using > > > > ?/var/cache/python/? for its cached bytecode tree on systems > > > > that implement the FHS. > > [?] There's no implication that data-which-happens-to-be-code is > > unsuitable for storage in ?/var/cache/foo/?. Easily-regenerated > > Python byte code for caching meets the description quite well, > > AFAICT. > > While I strongly approve of the concept of a central cache directory > for many things, I don't think that .pyc files fit the bill. > > Since there is no privileged python user that runs all Python code, > and since any unprivileged user needs to be able to write .pyc files, Hold up; my understanding is that, as Antoine Pitrou says: > The main point of the __pycache__ proposal is to solve the needs of > Ubuntu/Debian packagers. If you are developing (rather than deploying > or building packages), you shouldn't have these needs AFAICT. So, the packaging system will, by definition, have access to write to FHS directories and those directories don't need to be world-writable. -- \ ?Pinky, are you pondering what I'm pondering?? ?I think so, | `\ Brain, but how will we get a pair of Abe Vigoda's pants?? | _o__) ?_Pinky and The Brain_ | Ben Finney From barry at python.org Wed Mar 24 00:18:25 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 23 Mar 2010 19:18:25 -0400 Subject: [Python-Dev] __pycache__ creation In-Reply-To: <4BA7F71D.6030907@v.loewis.de> References: <20100322095604.3371039e@heresy> <20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de> <4BA7E04F.7070308@v.loewis.de> <4BA7EC9A.1010206@v.loewis.de> <1269298415.5792.11.camel@localhost> <4BA7F71D.6030907@v.loewis.de> Message-ID: <20100323191825.27fe28e5@heresy> On Mar 23, 2010, at 12:02 AM, Martin v. L?wis wrote: >I think the appropriate action at this point is to record this specific >objection in the PEP. Done. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ijmorlan at uwaterloo.ca Wed Mar 24 01:50:20 2010 From: ijmorlan at uwaterloo.ca (Isaac Morland) Date: Tue, 23 Mar 2010 20:50:20 -0400 (EDT) Subject: [Python-Dev] =?utf-8?q?PEP_3147=2C_=5F=5Fpycache=5F=5F_directorie?= =?utf-8?q?s_and_umask?= In-Reply-To: References: <20100322095604.3371039e@heresy> <4BA80418.6030905@canterbury.ac.nz> <4BA93580.9000402@canterbury.ac.nz> Message-ID: On Tue, 23 Mar 2010, Antoine Pitrou wrote: > Greg Ewing canterbury.ac.nz> writes: >>> The main point of the __pycache__ proposal is to solve the needs of >>> Ubuntu/Debian packagers. If you are developing (rather than deploying or >>> building packages), you shouldn't have these needs AFAICT. >> >> Maybe it's one point, but I'm not sure it's the *main* one. > > It's the only reason the PEP was originally designed, and proposed. At least one additional use case has appeared. Actually, my use case was mentioned long ago, but I didn't really push (e.g. by writing a patch) and nobody jumped on it. But this PEP solves my case too, so it should not be ignored just because the immediate impetus for the PEP is another case. >> Personally I would benefit most from it during development. > > Why? What benefit would it bring to you? I'm sure Greg will jump in if I'm wrong about what he is saying, but the benefit to me and to Greg and to others writing .py code is that our directories will contain *.py and __pycache__, rather than *.py and *.pyc. So it will be much easier to see what is actually there. Or if we're using SVN and we do "svn status", the only spurious result will be "? __pycache__" rather than "? X.pyc" for every X.py in the directory. Or whatever other good effects come from having less junk in our source directories. Directory tidiness is a positive general feature with at least a few specific benefits. Isaac Morland CSCF Web Guru DC 2554C, x36650 WWW Software Specialist From solipsis at pitrou.net Wed Mar 24 02:05:16 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 24 Mar 2010 02:05:16 +0100 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: References: <20100322095604.3371039e@heresy> <4BA80418.6030905@canterbury.ac.nz> <4BA93580.9000402@canterbury.ac.nz> Message-ID: <1269392716.4874.3.camel@localhost> Le mardi 23 mars 2010 ? 20:50 -0400, Isaac Morland a ?crit : > > I'm sure Greg will jump in if I'm wrong about what he is saying, but the > benefit to me and to Greg and to others writing .py code is that our > directories will contain *.py and __pycache__, rather than *.py and *.pyc. > So it will be much easier to see what is actually there. I don't really get it. I have never had any problem to see "what is actually here" in a Python source directory, despite the presence of pyc files. > Or if we're using SVN and we do "svn status", the only spurious result > will be "? __pycache__" rather than "? X.pyc" for every X.py in the > directory. Well, I was assuming that everyone had been using svn:ignore, or .hgignore, for years. Similarly, you will configure svn, hg, or any other system, to ignore __pycache__ directories (or .pyc files) so that they don't appear in "svn status". > Directory tidiness is a positive general feature with at least a few > specific benefits. It's still mostly cosmetic and I don't think it's as serious as any positive or negative system administration effects the change may have. From rrr at ronadam.com Wed Mar 24 02:42:44 2010 From: rrr at ronadam.com (Ron Adam) Date: Tue, 23 Mar 2010 20:42:44 -0500 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> Message-ID: Terry Reedy wrote: > On 3/22/2010 2:15 PM, Antoine Pitrou wrote: > >> What I am proposing is that the creation of __pycache__ /directories/ >> be put >> outside of the core. It can be part of distutils, or of a separate >> module, or >> delegated to third-party tools. It could even be as simple as >> "python -m compileall --pycache", if someone implements it. >> >> Creation of the __pycache__ /contents/ (files inside the directory) >> would still >> be part of core Python, but only if the directory exists and is >> writable by the >> current process. > > -1 > > If, as I have done several times recently, I create a directory and > insert an empty __init__.py and several real module.py files, I want the > .pycs to go into __pycache__ *automatically, by default, without me also > having to remember to create an empty __pycache__ *directory*, *each > time*. Ugh. I think I misunderstood this at first. It looks like, while developing a python 3.2+ program, if you don't create an empty __pycache__ directory, everything will still work, you just won't get the .pyc files. That can be a good thing during development because you also will not have any problems with old .pyc files hanging around if you move or rename files. The startup time may just be a tad longer, but probably not enough to be much of a problem. If it is a problem you can just create the __pycache__ directory, but nothing bad will happen if you don't. Ron From lgautier at gmail.com Wed Mar 24 06:17:00 2010 From: lgautier at gmail.com (Laurent Gautier) Date: Wed, 24 Mar 2010 06:17:00 +0100 Subject: [Python-Dev] GSoC 2010 is on -- projects? In-Reply-To: <20100320031326.GM1623@idyll.org> References: <20100319023630.GC1617@idyll.org> <4BA31858.3010904@gmail.com> <4BA388E5.1030703@gmail.com> <4BA40220.30701@v.loewis.de> <20100320031326.GM1623@idyll.org> Message-ID: <4BA9A04C.4010405@gmail.com> On 3/20/10 4:13 AM, C. Titus Brown wrote: > On Sat, Mar 20, 2010 at 12:00:48AM +0100, "Martin v. L?wis" wrote: >>> Whether this is worth weeks of work or not will depend on a given >>> student's knowledge about Python 3, and I'd suspect that the GSoC would >>> be an opportunity for a number of applicant to actually learn the >>> intricacies of Python 3. >>> Developing Python 3-specific features could be used to increase the >>> amount of work on the project, but I am uncertain of whether this is >>> worth a full GSoC. >> >> I'd say this would definitely make a GSoC project; any non-trivial >> porting will be. > > Sounds good to me. Line up a good student and bob's your uncle. > > --titus I may have a student that is looking interested, and I have inserted the porting project to the Wiki table. Laurent. PS: Editing that table felt like a test in disguise ;-) From stephen at xemacs.org Wed Mar 24 06:36:40 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 24 Mar 2010 14:36:40 +0900 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <201003240900.30334.steve@pearwood.info> References: <5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com> <201003240900.30334.steve@pearwood.info> Message-ID: <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > As usual though, NANs are unintuitive: > > >>> d = {float('nan'): 1} > >>> d[float('nan')] = 2 > >>> d > {nan: 1, nan: 2} > > > I suspect that's a feature, not a bug. I don't see how it can be so. Aren't all of those entries garbage? To compute a histogram of results for computations on a series of cases would you not have to test each result for NaN-hood, then hash on a proxy such as the string "Nan"? From amauryfa at gmail.com Wed Mar 24 09:42:52 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 24 Mar 2010 09:42:52 +0100 Subject: [Python-Dev] argparse.py is licensed under the Apache License Message-ID: Hello, I noticed that the newly added argparse module has an unusual licence header,?included below. This is the only file in the Python tree that contains an explicit reference to the Apache License, and this leads me to some questions: - Is the Apache license compatible with the Python license? Will this cause problem for some organizations that redistribute Python, possibly with proprietary software? Are there additional constraints? - Does this addition require a paragraph in the python documentation? http://docs.python.org/license.html#licenses-and-acknowledgements-for-incorporated-software - The Apache License states that:: ?? ?You must cause any modified files to carry prominent notices stating ?? ?that You changed the files but r78749 already modified the file (to remove a py3k warning) didn't we break the License? - Did the contributor sign a Contributor agreement? In this case, shouldn't the code be marked as "Licensed to PSF under a Contributor Agreement", as?mentioned in the contribution form? http://www.python.org/psf/contrib/contrib-form And then, could this Apache License be removed? The first lines of Lib/argparse.py are: # Copyright 2006-2009 Steven J. Bethard . # # Licensed under the Apache License, Version 2.0 (the "License"); you may not # use this file except in compliance with the License. You may obtain a copy # of the License at # # ? ? http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. -- Amaury Forgeot d'Arc From dickinsm at gmail.com Wed Mar 24 10:51:36 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 24 Mar 2010 09:51:36 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com> <201003240900.30334.steve@pearwood.info> <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> On Wed, Mar 24, 2010 at 5:36 AM, Stephen J. Turnbull wrote: > Steven D'Aprano writes: > > ?> As usual though, NANs are unintuitive: > ?> > ?> >>> d = {float('nan'): 1} > ?> >>> d[float('nan')] = 2 > ?> >>> d > ?> {nan: 1, nan: 2} > ?> > ?> > ?> I suspect that's a feature, not a bug. Right: distinct nans (i.e., those with different id()) are treated as distinct set elements or dict keys. > I don't see how it can be so. ?Aren't all of those entries garbage? > To compute a histogram of results for computations on a series of > cases would you not have to test each result for NaN-hood, then hash > on a proxy such as the string "Nan"? So what alternative behaviour would you suggest, and how would you implement it? I agree that many aspects of the current treatment of nans aren't ideal, but I as far as I can see that's unavoidable. For sane containment testing, Python's == operator needs to give an equivalence relation. Meanwhile IEEE 754 requires that nans compare unequal to themselves, breaking reflexivity. So there have to be some compromises somewhere. The current compromise at least has the virtue that it doesn't require special-casing nans anywhere in the general containment-testing and hashing machinery. One alternative would be to prohibit putting nans into sets and dicts by making them unhashable; I'm not sure what that would gain, though. And there would still be some unintuitive behaviour for containment testing of nans in lists. Mark From techtonik at gmail.com Wed Mar 24 10:59:34 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 24 Mar 2010 11:59:34 +0200 Subject: [Python-Dev] At least one package management tool for 2.7 Message-ID: I wonder if there are many people here who don't use some kind of "easy_install" package for package management in their Python / virtualenv installations? I propose to include at least one such package that is capable to auto-update itself in Python 2.7 C:\~env\Python27>python.exe -m easy_install C:\~env\Python27\python.exe: No module named easy_install C:\~env\Python27>python.exe -m pip C:\~env\Python27\python.exe: No module named pip It bugs me when I have to troubleshoot things on yet another machine that doesn't have some kind of `setuptools` installed. Or when I have to test some bug in my package on different Python version with a clean install and need some dependencies. -- anatoly t. From ziade.tarek at gmail.com Wed Mar 24 11:26:20 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 24 Mar 2010 11:26:20 +0100 Subject: [Python-Dev] At least one package management tool for 2.7 In-Reply-To: References: Message-ID: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com> On Wed, Mar 24, 2010 at 10:59 AM, anatoly techtonik wrote: > I wonder if there are many people here who don't use some kind of > "easy_install" package for package management in their Python / > virtualenv installations? I propose to include at least one such > package that is capable to auto-update itself in Python 2.7 > > C:\~env\Python27>python.exe -m easy_install > C:\~env\Python27\python.exe: No module named easy_install > > C:\~env\Python27>python.exe -m pip > C:\~env\Python27\python.exe: No module named pip > > > It bugs me when I have to troubleshoot things on yet another machine > that doesn't have some kind of `setuptools` installed. Or when I have > to test some bug in my package on different Python version with a > clean install and need some dependencies. We are working on distutils2 right now to improve the situation, and Ian has proposed to work on the possible inclusion of virtualenv in the stldib as well. I'll talk for distutils2 : The plan is to provide a distutils2 standalone version that can be installed from 2.4 to 3.x, and that will provide a basic installer/uninstaller via -m. Distutils2 is planned to be reintegrated in the stdlib in Python 3.3, and my goal is to release it when Python 2.7 final is released. The open question is: do we want to include a full installer that takes care of installing / removing dependencies as well ? I think not. Pip already provides this feature on the top of distutils (and distutils2 later I guess) and is not hard to install on the top of Python. But the "auto-update" story seems interesting, can you expand on this ? Tarek -- Tarek Ziad? | http://ziade.org From techtonik at gmail.com Wed Mar 24 11:31:06 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 24 Mar 2010 12:31:06 +0200 Subject: [Python-Dev] Joel Spolsky on Mercurial In-Reply-To: References: Message-ID: On Fri, Mar 19, 2010 at 11:00 AM, Dirkjan Ochtman wrote: >> Having gotten that far, I think this might be worth referencing in new dev >> docs. > > Will do. I finally read hginit yesterday, after having seen people > rave about it on twitter for a few weeks, and it's a very friendly > introduction. I more like hands-on approach that does use real world workflow instead of academic examples. For example, tutorial like Ondrej Certik made for SymPy patches using Mercurial queues. http://docs.sympy.org/sympy-patches-tutorial.html It goes straight from the problem in SymPy development you'd like to resolve for yourself and then shows how Mercurial helps with it. If you have more time to dig - you may read a book or Joel series. -- anatoly t. From steve at pearwood.info Wed Mar 24 12:14:39 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 24 Mar 2010 22:14:39 +1100 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> References: <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp> <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> Message-ID: <201003242214.39778.steve@pearwood.info> On Wed, 24 Mar 2010 08:51:36 pm Mark Dickinson wrote: > On Wed, Mar 24, 2010 at 5:36 AM, Stephen J. Turnbull wrote: > > Steven D'Aprano writes: > > > > ?> As usual though, NANs are unintuitive: > > ?> > > ?> >>> d = {float('nan'): 1} > > ?> >>> d[float('nan')] = 2 > > ?> >>> d > > ?> {nan: 1, nan: 2} > > ?> > > ?> > > ?> I suspect that's a feature, not a bug. > > Right: distinct nans (i.e., those with different id()) are treated > as distinct set elements or dict keys. > > > I don't see how it can be so. ?Aren't all of those entries garbage? > > To compute a histogram of results for computations on a series of > > cases would you not have to test each result for NaN-hood, then > > hash on a proxy such as the string "Nan"? Not necessarily -- you could merely ignore any key which is a NaN, or you could pass each key through this first: def intern_nan(x, nan=float('nan')): if math.isnan(x): return nan return x thus ensuring that all NaN keys were the same NaN. > So what alternative behaviour would you suggest, and how would you > implement it? [...] > One alternative would be to prohibit putting nans into sets and dicts > by making them unhashable; I'm not sure what that would gain, > though. And there would still be some unintuitive behaviour for > containment testing of nans in lists. I think that would be worse than the current situation. That would mean that dict[some_float] would *nearly always* succeed, but occasionally would fail. I can't see that being a good thing. -- Steven D'Aprano From stephen at xemacs.org Wed Mar 24 12:15:55 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 24 Mar 2010 20:15:55 +0900 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> References: <5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com> <201003240900.30334.steve@pearwood.info> <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp> <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> Message-ID: <87eijagec4.fsf@uwakimon.sk.tsukuba.ac.jp> Mark Dickinson writes: > On Wed, Mar 24, 2010 at 5:36 AM, Stephen J. Turnbull wrote: > > Steven D'Aprano writes: > > ?> I suspect that's a feature, not a bug. > > Right: distinct nans (i.e., those with different id()) are treated as > distinct set elements or dict keys. > > > I don't see how it can be so. ?Aren't all of those entries garbage? > > To compute a histogram of results for computations on a series of > > cases would you not have to test each result for NaN-hood, then hash > > on a proxy such as the string "Nan"? > > So what alternative behaviour would you suggest, and how would you > implement it? I don't have an alternative behavior to suggest. I'm not suggesting that it's a bug, I'm suggesting that it's a wart: useless, ugly, and in some presumably rare/buggy cases, it could lead to nasty behavior. The example I have in mind is computing a histogram of function values for a very large sample of inputs. (This is a pathological example, of course: things where NaNs are representable generally won't be used directly as keys in a dictionary used to represent a histogram. Rather, they would be mapped to a representative value as the key.) If there are a lot of NaN's, the dictionary could get unexpectedly large. That's not Python's fault, of course: > Meanwhile IEEE 754 requires that nans compare unequal to > themselves, breaking reflexivity. So there have to be some > compromises somewhere. Indeed. IEEE 754 compatibility *is* a feature. > One alternative would be to prohibit putting nans into sets and > dicts by making them unhashable; I'm not sure what that would gain, > though. I would find that more intuitive. While NaNs aren't mutable, they're similar to mutable values in that their value is not deterministic in a certain sense. OTOH, since the only example I can think of where I would personally want to check whether a NaN is in a container is pathological, my intuition is hardly reliable. From techtonik at gmail.com Wed Mar 24 12:20:19 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 24 Mar 2010 13:20:19 +0200 Subject: [Python-Dev] At least one package management tool for 2.7 In-Reply-To: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com> References: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com> Message-ID: On Wed, Mar 24, 2010 at 12:26 PM, Tarek Ziad? wrote: > > Distutils2 is planned to be reintegrated in the stdlib in Python 3.3, > and my goal is to release it when Python 2.7 final is released. Does that means "after" Python 2.7, because I meant it to be "before" or at least "with"? > The open question is: do we want to include a full installer that > takes care of installing / removing dependencies as well ? If there is a risk to get nothing at all in 2.7 distribution, because it just won't be ready/accepted by that time, then I it is certainly optional. > But the "auto-update" story seems interesting, can you expand on this ? Sure. Package management tool should have an ability to update itself when required regardless of Python release. For example:: python.exe -m easy_install setuptools This will get you new version of `setuptools` and `easy_install` module from it automagically. You do not need to install new version of `setuptools` manually or copy files from SVN if you want to see fixes before next Python release. The stuff you would likely need to do with distutils bugs, which I personally find awkward. Package management is orthogonal to Python releases, and it is more oriented at Python users who don't like to wait or follow PEPs. That's why package management tool such as 'easy_install' has shorter development cycle, and it should faster react to user feedback. What can be one of the reasons that no package management tool is included with Python. In various README you may often see "requires setuptools > 0.6c9" or similar. I can't see why package management tool can not detect this dependency and propose to update itself. If it is impossible to ship the whole package management system then at least Python distribution may carry small bootstrap script for it. When user tries to execute package management tools, it warns him that these are not installed and gives a hint where to get them:: > python -m easy_install bla-bla-bla Error: easy_install module is not shipped with this Python release. Please execute the following command to install the latest version. python -m easy_bootstrap -- anatoly t. From techtonik at gmail.com Wed Mar 24 12:23:18 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 24 Mar 2010 13:23:18 +0200 Subject: [Python-Dev] At least one package management tool for 2.7 In-Reply-To: References: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com> Message-ID: > Sure. Package management tool should have an ability to update itself when required regardless of Python release. For example:: > > python.exe -m easy_install setuptools > This should be: python -m easy_install -U setuptools P.S. Wave effect. =) -- anatoly t. From ziade.tarek at gmail.com Wed Mar 24 12:30:19 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 24 Mar 2010 12:30:19 +0100 Subject: [Python-Dev] At least one package management tool for 2.7 In-Reply-To: References: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com> Message-ID: <94bdd2611003240430m1fb8f6e5o22fc24d204c043c9@mail.gmail.com> On Wed, Mar 24, 2010 at 12:20 PM, anatoly techtonik wrote: > On Wed, Mar 24, 2010 at 12:26 PM, Tarek Ziad? wrote: >> >> Distutils2 is planned to be reintegrated in the stdlib in Python 3.3, >> and my goal is to release it when Python 2.7 final is released. > > Does that means "after" Python 2.7, because I meant it to be "before" > or at least "with"? The goal is to provide a first version by the time 2.7 is out. > >> The open question is: do we want to include a full installer that >> takes care of installing / removing dependencies as well ? > > If there is a risk to get nothing at all in 2.7 distribution, because > it just won't be ready/accepted by that time, then I it is certainly > optional. > Understand that the distutils2 project is happening outside the stdlib at this time, so you will have to install it on the top of Python in any case. Last, its release cycle will be shorter until it gets back in the stdlib, that will make it easy to add features. >> But the "auto-update" story seems interesting, can you expand on this ? > > Sure. Package management tool should have an ability to update itself > when required regardless of Python release. For example:: > > ? ?python.exe -m easy_install setuptools > > This will get you new version of `setuptools` and `easy_install` > module from it automagically. You do not need to install new version > of `setuptools` manually or copy files from SVN if you want to see > fixes before next Python release. The stuff you would likely need to > do with distutils bugs, which I personally find awkward. > > Package management is orthogonal to Python releases, and it is more > oriented at Python users who don't like to wait or follow PEPs. That's > why package management tool such as 'easy_install' has shorter > development cycle, and it should faster react to user feedback. What > can be one of the reasons that no package management tool is included > with Python. > > In various README you may often see "requires setuptools > 0.6c9" or > similar. I can't see why package management tool can not detect this > dependency and propose to update itself. > > If it is impossible to ship the whole package management system then > at least Python distribution may carry small bootstrap script for it. > When user tries to execute package management tools, it warns him that > these are not installed and gives a hint where to get them:: > >> python -m easy_install bla-bla-bla > Error: easy_install module is not shipped with this Python release. > Please execute the following command to install the latest version. > > python -m easy_bootstrap Interesting, can you start a new thread on distutils-SIG for this part, so we can discuss it there ? Regards Tarek -- Tarek Ziad? | http://ziade.org From ncoghlan at gmail.com Wed Mar 24 12:47:26 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 24 Mar 2010 21:47:26 +1000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <201003242214.39778.steve@pearwood.info> References: <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp> <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> <201003242214.39778.steve@pearwood.info> Message-ID: <4BA9FBCE.5040908@gmail.com> Steven D'Aprano wrote: > On Wed, 24 Mar 2010 08:51:36 pm Mark Dickinson wrote: >>> I don't see how it can be so. Aren't all of those entries garbage? >>> To compute a histogram of results for computations on a series of >>> cases would you not have to test each result for NaN-hood, then >>> hash on a proxy such as the string "Nan"? > > Not necessarily -- you could merely ignore any key which is a NaN, or > you could pass each key through this first: > > def intern_nan(x, nan=float('nan')): > if math.isnan(x): return nan > return x > > thus ensuring that all NaN keys were the same NaN. Interning NaN certainly seems like it should be sufficient to eliminate the set/dict membership weirdness. That is, make it so that the first two lines of the following return True, while the latter two lines continue to return False: >>> float("nan") is float("nan") False >>> dec("nan") is dec("nan") False >>> float("nan") == float("nan") False >>> dec("nan") == dec("nan") False Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Mar 24 12:48:40 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 24 Mar 2010 21:48:40 +1000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <20100323120959.GA486@mail.bytereef.org> Message-ID: <4BA9FC18.90102@gmail.com> Raymond Hettinger wrote: > The decimal module is already drowning in complexity, > so it would be best to keep it simple: one boolean flag > that if set would warn about any implicit decimal/float > interaction. Agreed - those that want exceptions instead can use the usual warnings module mechanisms to trigger them. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From dsdale24 at gmail.com Wed Mar 24 12:50:44 2010 From: dsdale24 at gmail.com (Darren Dale) Date: Wed, 24 Mar 2010 07:50:44 -0400 Subject: [Python-Dev] [Distutils] At least one package management tool for 2.7 In-Reply-To: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com> References: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com> Message-ID: On Wed, Mar 24, 2010 at 6:26 AM, Tarek Ziad? wrote: > The open question is: do we want to include a full installer that > takes care of installing / removing dependencies as well ? > > I think not. Pip already provides this feature on the top of distutils > (and distutils2 later I guess) and is not hard to install on the top > of Python. Is pip able to determine and install dependencies recursively, like easy_install does? Or is it up to the requested package to it specify its dependencies (and its dependencies dependencies) in a pip requirements file that is distributed separately? Darren From ziade.tarek at gmail.com Wed Mar 24 12:53:01 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 24 Mar 2010 12:53:01 +0100 Subject: [Python-Dev] [Distutils] At least one package management tool for 2.7 In-Reply-To: References: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com> Message-ID: <94bdd2611003240453l79f7787ft101459a199a64061@mail.gmail.com> On Wed, Mar 24, 2010 at 12:50 PM, Darren Dale wrote: > On Wed, Mar 24, 2010 at 6:26 AM, Tarek Ziad? wrote: >> The open question is: do we want to include a full installer that >> takes care of installing / removing dependencies as well ? >> >> I think not. Pip already provides this feature on the top of distutils >> (and distutils2 later I guess) and is not hard to install on the top >> of Python. > > Is pip able to determine and install dependencies recursively, like > easy_install does? Or is it up to the requested package to it specify > its dependencies (and its dependencies dependencies) in a pip > requirements file that is distributed separately? Pip uses the same standard: the install_requires option in setuptools, that is put in a requires.txt file when the egg-info is computed. Notice that we have defined this field in the new version of the metadata (PEP 345) for future interoperability. Let's remove python-dev for packaging talks in the next answers Regards Tarek > > Darren > -- Tarek Ziad? | http://ziade.org From ncoghlan at gmail.com Wed Mar 24 13:02:06 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 24 Mar 2010 22:02:06 +1000 Subject: [Python-Dev] At least one package management tool for 2.7 In-Reply-To: References: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com> Message-ID: <4BA9FF3E.10009@gmail.com> anatoly techtonik wrote: > If it is impossible to ship the whole package management system then > at least Python distribution may carry small bootstrap script for it. > When user tries to execute package management tools, it warns him > that these are not installed and gives a hint where to get them:: > >> python -m easy_install bla-bla-bla > Error: easy_install module is not shipped with this Python release. > Please execute the following command to install the latest version. > > python -m easy_bootstrap Note that this idea has come up before and is *much* more likely to get traction than including a full package management system. The nature of package management is such that tying an installation system to the release cycle of the core interpreter is unlikely to end well. A bootstrapping tool that only knows how to download a single specific package would be much easier to keep stable. Even such a scaled back idea is still far from a certainty to be accepted though, given the number of people who vehemently object to duplicating any significant part of the functionality of the system package management tools (i.e. RPM, apt et al). At this point, the packaging story is in the hands of distutils-sig and they're pressing forward with several initiatives that will permit the construction of more robust and reliable Python-specific package management systems (such as supporting listing and uninstallation of installed packages). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Mar 24 13:04:48 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 24 Mar 2010 22:04:48 +1000 Subject: [Python-Dev] __pycache__ creation In-Reply-To: <20100323183212.7463e098@heresy> References: <20100322095604.3371039e@heresy> <20100322162002.6daff86f@heresy> <20100323183212.7463e098@heresy> Message-ID: <4BA9FFE0.3010800@gmail.com> Barry Warsaw wrote: > On Mar 22, 2010, at 12:38 PM, Guido van Rossum wrote: > >> Huh? Last time I looked weren't we going to make __pycache__ the >> default (and eventually only) behavior? > > We definitely agreed it would be the default in Python 3.2. > > My recollection is that we agreed it would be the only on-demand way of > writing pyc files, but that Python would read a lone .pyc file where the > source would be if the source is missing, and that py_compile/compileall would > support optional creation of those lone .pyc files. Yep, that's my recollection as well. I don't recall seeing an update to state that clearly in the PEP go by on the checkins list though :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From steve at pearwood.info Wed Mar 24 13:04:56 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 24 Mar 2010 23:04:56 +1100 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA9FBCE.5040908@gmail.com> References: <201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com> Message-ID: <201003242304.56454.steve@pearwood.info> On Wed, 24 Mar 2010 10:47:26 pm Nick Coghlan wrote: > Steven D'Aprano wrote: > > On Wed, 24 Mar 2010 08:51:36 pm Mark Dickinson wrote: > >>> I don't see how it can be so. Aren't all of those entries > >>> garbage? To compute a histogram of results for computations on a > >>> series of cases would you not have to test each result for > >>> NaN-hood, then hash on a proxy such as the string "Nan"? > > > > Not necessarily -- you could merely ignore any key which is a NaN, > > or you could pass each key through this first: > > > > def intern_nan(x, nan=float('nan')): > > if math.isnan(x): return nan > > return x > > > > thus ensuring that all NaN keys were the same NaN. > > Interning NaN certainly seems like it should be sufficient to > eliminate the set/dict membership weirdness. I didn't mean to suggest that Python should do that automatically! I meant that the developer could easily intern NaNs if needed. I wouldn't want Python to automatically intern NaNs, the reason being that this would throw away information (at least potentially, depending on the C library). According to the relevant IEEE standard, NaNs should (may?) carry a payload. For example, Apple's SANE math library back in the 1980s exposed this payload: NaNs created from different failures would have a consistent payload, allowing the programmer to tell how the NaN appeared in the calculation. E.g. INF-INF would give you a payload of 123 (or whatever it was), while log(-1) would give you a payload of 456. (I've made up the numbers, it's been far too many years for me to remember what they were.) The point is, whether Python currently exposes these payloads or not, we shouldn't prohibit it. If programmers want to explicitly fold all NaNs into one, it is easy to do so themselves. -- Steven D'Aprano From ncoghlan at gmail.com Wed Mar 24 13:07:36 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 24 Mar 2010 22:07:36 +1000 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> Message-ID: <4BAA0088.2010909@gmail.com> Ron Adam wrote: > I think I misunderstood this at first. > > It looks like, while developing a python 3.2+ program, if you don't > create an empty __pycache__ directory, everything will still work, you > just won't get the .pyc files. That can be a good thing during > development because you also will not have any problems with old .pyc > files hanging around if you move or rename files. The behaviour you described (not creating __pycache__ automatically) was just a suggestion in this thread. The behaviour in the actual PEP (and what will be implemented for 3.2+) is to create __pycache__ if it is missing. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Wed Mar 24 13:09:37 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 24 Mar 2010 08:09:37 -0400 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> Message-ID: <20100324080937.5343f791@heresy> On Mar 23, 2010, at 08:42 PM, Ron Adam wrote: >It looks like, while developing a python 3.2+ program, if you don't create >an empty __pycache__ directory, everything will still work, you just won't >get the .pyc files. That can be a good thing during development because >you also will not have any problems with old .pyc files hanging around if >you move or rename files. Not quite. In PEP-3147-land, you do not need to create the empty __pycache__ directory, Python will create it for you on-demand. If you subsequently move the .py source file, the __pycache__/...pyc file will be ignored. So this is actually better than today because you can't accidentally load stale pyc files -- if they live inside __pycache__. For backward compatibility we'll still support loading lone pyc files in the source file directory (i.e. outside of __pycache__). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ncoghlan at gmail.com Wed Mar 24 13:15:00 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 24 Mar 2010 22:15:00 +1000 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: References: <20100322095604.3371039e@heresy> <4BA80418.6030905@canterbury.ac.nz> Message-ID: <4BAA0244.401@gmail.com> Russell E. Owen wrote: > This in turn implies that we may have to give up some support for > dragging python modules into site-packages, e.g. not generate .pyc files > for such modules. At least if we go that route it will mostly affect > power users, who can presumably cope. And when someone drags a Python module into the per-user site-packages instead? [1] Yes, a shared Python needs to be managed carefully. Systems with a shared Python should also generally have a vaguely competent sysadmin running them. An unshared Python and associated packages under PEP 3147 should work just as well as they do under the existing pyc scheme (only without the source directory clutter). Cheers, Nick. [1] http://www.python.org/dev/peps/pep-0370/ -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From olemis at gmail.com Wed Mar 24 13:22:23 2010 From: olemis at gmail.com (Olemis Lang) Date: Wed, 24 Mar 2010 08:22:23 -0400 Subject: [Python-Dev] At least one package management tool for 2.7 In-Reply-To: References: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com> Message-ID: <24ea26601003240522n3f44d8b5k8105a13d4fb614bf@mail.gmail.com> On Wed, Mar 24, 2010 at 7:23 AM, anatoly techtonik wrote: >> Sure. Package management tool should have an ability to update itself when required regardless of Python release. For example:: >> >> python.exe -m easy_install setuptools >> > > This should be: > > ? ?python -m easy_install -U setuptools > JFTR More precisely, what I use for CI is {{{ #!sh $ easy_install -U setuptools==dev }}} but the `python -m` part should work as well ;o) PS: e.g. that allows to check out code from SVN in order to use setuptools 0.7.x `test_runner` switch like this {{{ python -W ignore::DeprecationWarning setup.py test -r ciutils:junitrunner }}} ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: TracRpc: API v2: Test cases for XML-RPC ... PASS - http://bitbucket.org/osimons/trac-rpc-mq/changeset/228ef43726b0/ From olemis at gmail.com Wed Mar 24 13:27:03 2010 From: olemis at gmail.com (Olemis Lang) Date: Wed, 24 Mar 2010 08:27:03 -0400 Subject: [Python-Dev] [Distutils] At least one package management tool for 2.7 In-Reply-To: References: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com> Message-ID: <24ea26601003240527t33c038b2ka1b865dcca172374@mail.gmail.com> On Wed, Mar 24, 2010 at 7:50 AM, Darren Dale wrote: > On Wed, Mar 24, 2010 at 6:26 AM, Tarek Ziad? wrote: >> The open question is: do we want to include a full installer that >> takes care of installing / removing dependencies as well ? >> >> I think not. Pip already provides this feature on the top of distutils >> (and distutils2 later I guess) and is not hard to install on the top >> of Python. > > Is pip able to determine and install dependencies recursively, like > easy_install does? Or is it up to the requested package to it specify > its dependencies (and its dependencies dependencies) in a pip > requirements file that is distributed separately? > My experience is that only `install_requires` is needed (unless you want to create app bundles AFAICR) , but in practice I've noticed that *some* easy_installable packages are not pip-able (though I had no time to figure out why :-/ ) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: On adding Hessian (RPC) support for Trac - http://feedproxy.google.com/~r/simelo-en/~3/Vit6dRudChU/on-adding-hessian-rpc-support-for-trac.html From barry at python.org Wed Mar 24 13:32:36 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 24 Mar 2010 08:32:36 -0400 Subject: [Python-Dev] __pycache__ creation In-Reply-To: <4BA9FFE0.3010800@gmail.com> References: <20100322095604.3371039e@heresy> <20100322162002.6daff86f@heresy> <20100323183212.7463e098@heresy> <4BA9FFE0.3010800@gmail.com> Message-ID: <20100324083236.6e48437e@heresy> On Mar 24, 2010, at 10:04 PM, Nick Coghlan wrote: >Yep, that's my recollection as well. I don't recall seeing an update to >state that clearly in the PEP go by on the checkins list though :) Check again . -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ncoghlan at gmail.com Wed Mar 24 14:06:54 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 24 Mar 2010 23:06:54 +1000 Subject: [Python-Dev] __pycache__ creation In-Reply-To: <20100324083236.6e48437e@heresy> References: <20100322095604.3371039e@heresy> <20100322162002.6daff86f@heresy> <20100323183212.7463e098@heresy> <4BA9FFE0.3010800@gmail.com> <20100324083236.6e48437e@heresy> Message-ID: <4BAA0E6E.8050408@gmail.com> Barry Warsaw wrote: > On Mar 24, 2010, at 10:04 PM, Nick Coghlan wrote: > >> Yep, that's my recollection as well. I don't recall seeing an update to >> state that clearly in the PEP go by on the checkins list though :) > > Check again . Ah yes, the recollection of seeing such a message is now quite fresh in my mind :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Wed Mar 24 14:19:43 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 24 Mar 2010 13:19:43 +0000 (UTC) Subject: [Python-Dev] At least one package management tool for 2.7 References: Message-ID: anatoly techtonik gmail.com> writes: > > I wonder if there are many people here who don't use some kind of > "easy_install" package for package management in their Python / > virtualenv installations? I propose to include at least one such > package that is capable to auto-update itself in Python 2.7 > > C:\~env\Python27>python.exe -m easy_install > C:\~env\Python27\python.exe: No module named easy_install > > C:\~env\Python27>python.exe -m pip > C:\~env\Python27\python.exe: No module named pip It will not happen in 2.7, which is almost in beta phase. Based on previous discussions, it will most likely not happen in 3.2 either. The consensus is that setuptools, which both pip and easy_install depend on, should not be included into the core. However, Tarek's work on "distutils2" (a mostly forward-compatible replacement for distutils) will include features such as dependencies, and make it much easier to create and perhaps bundle a pip-like utility. So perhaps in 3.3 you have a chance ;) Regards Antoine. From barry at python.org Wed Mar 24 14:22:49 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 24 Mar 2010 09:22:49 -0400 Subject: [Python-Dev] __pycache__ creation In-Reply-To: <4BAA0E6E.8050408@gmail.com> References: <20100322095604.3371039e@heresy> <20100322162002.6daff86f@heresy> <20100323183212.7463e098@heresy> <4BA9FFE0.3010800@gmail.com> <20100324083236.6e48437e@heresy> <4BAA0E6E.8050408@gmail.com> Message-ID: <20100324092249.659bd810@heresy> On Mar 24, 2010, at 11:06 PM, Nick Coghlan wrote: >Ah yes, the recollection of seeing such a message is now quite fresh in >my mind :) Just don't tell Guido I borrowed his time machine keys! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From stephen at xemacs.org Wed Mar 24 15:48:51 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 24 Mar 2010 23:48:51 +0900 Subject: [Python-Dev] __pycache__ creation In-Reply-To: <20100324092249.659bd810@heresy> References: <20100322095604.3371039e@heresy> <20100322162002.6daff86f@heresy> <20100323183212.7463e098@heresy> <4BA9FFE0.3010800@gmail.com> <20100324083236.6e48437e@heresy> <4BAA0E6E.8050408@gmail.com> <20100324092249.659bd810@heresy> Message-ID: <877hp1hj1o.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > On Mar 24, 2010, at 11:06 PM, Nick Coghlan wrote: > > >Ah yes, the recollection of seeing such a message is now quite fresh in > >my mind :) > > Just don't tell Guido I borrowed his time machine keys! Wouldn't that be preferable to revealing you've learned to hotwire it? From steven.bethard at gmail.com Wed Mar 24 17:05:36 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 24 Mar 2010 09:05:36 -0700 Subject: [Python-Dev] argparse.py is licensed under the Apache License In-Reply-To: References: Message-ID: On Wed, Mar 24, 2010 at 1:42 AM, Amaury Forgeot d'Arc wrote: > I noticed that the newly added argparse module has an unusual > licence header,?included below. This is the only file in the Python tree > that contains an explicit reference to the Apache License, > and this leads me to some questions: Sorry, I forgot to remove this when moving argparse to the stdlib. There's probably one test\test_argparse too. > - Did the contributor sign a Contributor agreement? In this case, > shouldn't the code be marked as "Licensed to PSF under a Contributor Agreement", > as?mentioned in the contribution form? > http://www.python.org/psf/contrib/contrib-form > And then, could this Apache License be removed? Yes, I have signed a contributor agreement. I was thinking of just removing the license entirely, but if it's better to replace it with "Licensed to PSF under a Contributor Agreement", that's fine too. Let me know, and I'll take care of this today. Thanks for catching this! Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From dickinsm at gmail.com Wed Mar 24 17:22:29 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 24 Mar 2010 16:22:29 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA9FBCE.5040908@gmail.com> References: <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp> <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> <201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com> Message-ID: <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> On Wed, Mar 24, 2010 at 11:47 AM, Nick Coghlan > Interning NaN certainly seems like it should be sufficient to eliminate > the set/dict membership weirdness. > > That is, make it so that the first two lines of the following return > True, while the latter two lines continue to return False: > >>>> float("nan") is float("nan") > False >>>> dec("nan") is dec("nan") > False >>>> float("nan") == float("nan") > False >>>> dec("nan") == dec("nan") > False Yes; that could be done. Though as Steven points out, not all NaNs are equivalent (possibility of different payloads and different signs), so float nans with different underlying bit patterns, and Decimal nans with different string representations, would ideally be interned separately. For floats it might be possible to get away with pretending that there's only one nan. For decimal, I don't think that's true, since the payload and sign are part of the standard, and are very visible (e.g. in the repr of the nan). The obvious way to do this nan interning for floats would be to put the interning code into PyFloat_FromDouble. I'm not sure whether this would be worth the cost in terms of added code (and possibly reduced performance, since the nan check would be done every time a float was returned), but I'd be willing to review a patch. Mark From rrr at ronadam.com Wed Mar 24 17:46:28 2010 From: rrr at ronadam.com (Ron Adam) Date: Wed, 24 Mar 2010 11:46:28 -0500 Subject: [Python-Dev] __pycache__ creation In-Reply-To: <4BAA0088.2010909@gmail.com> References: <20100322095604.3371039e@heresy> <4BAA0088.2010909@gmail.com> Message-ID: Nick Coghlan wrote: > Ron Adam wrote: >> I think I misunderstood this at first. >> >> It looks like, while developing a python 3.2+ program, if you don't >> create an empty __pycache__ directory, everything will still work, you >> just won't get the .pyc files. That can be a good thing during >> development because you also will not have any problems with old .pyc >> files hanging around if you move or rename files. > > The behaviour you described (not creating __pycache__ automatically) was > just a suggestion in this thread. > > The behaviour in the actual PEP (and what will be implemented for 3.2+) > is to create __pycache__ if it is missing. > > Cheers, > Nick. OK :-) hmmmm... unless there is a __pycache__ *file* located there first. ;-) Not that I can think of any good reason to do that at this moment. Ron From amentajo at msu.edu Wed Mar 24 18:16:14 2010 From: amentajo at msu.edu (Joe Amenta) Date: Wed, 24 Mar 2010 13:16:14 -0400 Subject: [Python-Dev] GSoC 2010 is on -- projects? In-Reply-To: References: <20100319023630.GC1617@idyll.org> <4BA31858.3010904@gmail.com> Message-ID: <4dc473a51003241016va2503cexf5c0ec55d5c30bda@mail.gmail.com> On Fri, Mar 19, 2010 at 2:45 AM, Terry Reedy wrote: > On 3/19/2010 2:23 AM, Laurent Gautier wrote: > >> On 3/19/10 3:36 AM, C. Titus Brown wrote: >> >>> Hi all, >>> >>> once again, the PSF has been accepted as a mentoring foundation for >>> the Google >>> Summer of Code! This year, we're going to emphasize python 3 porting, so >>> please think of projects you'd like to see tackled. >>> >> >> Hi, >> >> >> Does this mean that any other python project could potentially see >> itself ported to Python 3 in the course of this SoC ? >> > > The theme should include both general porting tools and specific projects, > especially infrastructure projects like numeric, which are blocking the > porting of other projects. It would be nice if those doing specific projects > contributed their experience/knowledge to a central pool. > > > If so, can any project owner submit a request for help, >> > > Any project owner is *always* free to ask for help (on python-list, but now > here in this thread). Those who can also mentor might be more likely to get > it. If I were a student, I would consider serious interest from a project > owner (and a promise to distribute a port, when ready), a prerequisite. > > > > or is there going to be a list > >> of projects that would nice to port, or will a voting system of some >> sort be put in place ? >> > > Like most contributors, students choose projects, within the limits of what > they can get mentors for, that scratch their itches. They may or may not > otherwise be swayed by requests and opinions. > > My views. > > Terry Jan Reedy > > Would anyone be interested in mentoring further lib3to2 work? I'm planning on applying again as a student. --Joe Amenta -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Mar 24 18:08:59 2010 From: brett at python.org (Brett Cannon) Date: Wed, 24 Mar 2010 10:08:59 -0700 Subject: [Python-Dev] argparse.py is licensed under the Apache License In-Reply-To: References: Message-ID: On Wed, Mar 24, 2010 at 09:05, Steven Bethard wrote: > On Wed, Mar 24, 2010 at 1:42 AM, Amaury Forgeot d'Arc > wrote: > > I noticed that the newly added argparse module has an unusual > > licence header, included below. This is the only file in the Python tree > > that contains an explicit reference to the Apache License, > > and this leads me to some questions: > > Sorry, I forgot to remove this when moving argparse to the stdlib. > There's probably one test\test_argparse too. > > > - Did the contributor sign a Contributor agreement? In this case, > > shouldn't the code be marked as "Licensed to PSF under a Contributor > Agreement", > > as mentioned in the contribution form? > > http://www.python.org/psf/contrib/contrib-form > > And then, could this Apache License be removed? > > Yes, I have signed a contributor agreement. I was thinking of just > removing the license entirely, but if it's better to replace it with > "Licensed to PSF under a Contributor Agreement", that's fine too. Let > me know, and I'll take care of this today. > > I say just strip it out. If you leave it the PSF board will get an email from some large corporation at some point in the future about whether you have actually signed an agreement or not. Much easier to simply remove it so people can rightfully assume it is licensed under the PSF license. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ianb at colorstudy.com Wed Mar 24 18:19:27 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 24 Mar 2010 12:19:27 -0500 Subject: [Python-Dev] [Distutils] At least one package management tool for 2.7 In-Reply-To: <24ea26601003240527t33c038b2ka1b865dcca172374@mail.gmail.com> References: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com> <24ea26601003240527t33c038b2ka1b865dcca172374@mail.gmail.com> Message-ID: On Wed, Mar 24, 2010 at 7:27 AM, Olemis Lang wrote: > My experience is that only `install_requires` is needed (unless you > want to create app bundles AFAICR) , but in practice I've noticed that > *some* easy_installable packages are not pip-able (though I had no > time to figure out why :-/ ) Usually this is because Setuptools is poking at objects to do its work, while pip tries to work mostly with subprocesses. Though to complicate things a bit, pip makes sure the Setuptools monkeypatches to distutils are applied, so that it's always as though the setup.py says "from setuptools import setup". easy_install *also* does this. But then easy_install starts calling methods and whatnot, while pip just does: setup.py install --single-version-externally-managed --no-deps --record some_tmp_file The --no-deps keeps Setuptools from resolving dependencies (because it does so using easy_install), and --single-version-externally-managed keeps Setuptools doing egg directories. And --record keeps track of installed files, which are later moved around to facilitate uninstall. But some distributions pay extra attention to those options, or do other tricky things in their setup.py, and as a result this causes failures. Because easy_install is just calling internal methods, it kind of sidelines those tricks (also people don't tend to give these options to easy_install, and some don't even exist, so some code paths just aren't exercised with typical easy_install usage). Oh, the other reason is the link searching mechanism is slightly different between the two installers. Not particularly intentionally, just incidentally. Also because pip doesn't install anything but source packages, some packages are installable via easy_install but not pip -- usually this is an oversight on the part of the person doing the packaging, but they just never noticed. -- Ian Bicking | http://blog.ianbicking.org | http://twitter.com/ianbicking From raymond.hettinger at gmail.com Wed Mar 24 18:39:54 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 24 Mar 2010 10:39:54 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> References: <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp> <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> <201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com> <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> Message-ID: <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com> On Mar 24, 2010, at 9:22 AM, Mark Dickinson wrote: > > The obvious way to do this nan interning for floats would be to put > the interning code into PyFloat_FromDouble. I'm not sure whether this > would be worth the cost in terms of added code (and possibly reduced > performance, since the nan check would be done every time a float was > returned), but I'd be willing to review a patch. -1 Propagating support for NaNs has already consumed an enormous amount of your time. The code for the math module doubled in complexity as a result of adding Nan/Inf support. At each stage, some new weirdness emerges (such as complex(Nan, Nan), etc). And each change introduces the risk of new bugs in code that had been stable for a long time. IIRC, the original purpose of a NaN was to serve as a placeholder value in a long series of floating point ops so that the programmer would not have to introduce edge case tests at each stage of a calculation. Yet, I look at the code for the decimal module and the C code for the math module and see that the opposite result occurred, the code is littered with is_special(x) tests and handlers. In terms of practicality, NaNs work great as a way to mark missing values and to propagate through subsequent calculations. IMO, their least useful feature is the property of being not equal to themselves -- that causes more problems than it solves because it impairs a programmer's ability to reason about their programs. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From dsdale24 at gmail.com Wed Mar 24 18:53:46 2010 From: dsdale24 at gmail.com (Darren Dale) Date: Wed, 24 Mar 2010 13:53:46 -0400 Subject: [Python-Dev] [Distutils] At least one package management tool for 2.7 In-Reply-To: References: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com> <24ea26601003240527t33c038b2ka1b865dcca172374@mail.gmail.com> Message-ID: On Wed, Mar 24, 2010 at 1:19 PM, Ian Bicking wrote: > On Wed, Mar 24, 2010 at 7:27 AM, Olemis Lang wrote: >> My experience is that only `install_requires` is needed (unless you >> want to create app bundles AFAICR) , but in practice I've noticed that >> *some* easy_installable packages are not pip-able (though I had no >> time to figure out why :-/ ) > > Usually this is because Setuptools is poking at objects to do its > work, while pip tries to work mostly with subprocesses. ?Though to > complicate things a bit, pip makes sure the Setuptools monkeypatches > to distutils are applied, so that it's always as though the setup.py > says "from setuptools import setup". ?easy_install *also* does this. > > But then easy_install starts calling methods and whatnot, while pip just does: > > ?setup.py install --single-version-externally-managed --no-deps > --record some_tmp_file > > The --no-deps keeps Setuptools from resolving dependencies Seeking clarification: how can pip recursively install dependencies *and* keep Setuptools from resolving dependencies? Darren From alexander.belopolsky at gmail.com Wed Mar 24 19:26:12 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 24 Mar 2010 14:26:12 -0400 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com> References: <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp> <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> <201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com> <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com> Message-ID: On Wed, Mar 24, 2010 at 1:39 PM, Raymond Hettinger wrote: .. > IMO,?their least useful feature is the property of being not equal > to themselves -- that causes more problems than it solves > because it impairs a programmer's ability to reason about > their programs. I agree. An often cited rationale for IEEE 754 behavior is that it eliminates branching in some high performance numerical algorithms. While this is likely to be true, I have never seen it benefiting real life applications, certainly not those written in Python. I wonder why Python did not follow Java model where Float NaN objects unlike raw float NaNs compare equal to themselves. One reason may be that Python does not have raw floats, but if someone needs IEEE 754 NaNs, one can use numpy scalars or add arithmetics to ctypes numerical types. Mark, I wonder if you could describe an algorithm off the top of your head that relies on NaN == NaN being false. From jbaker at zeomega.com Wed Mar 24 19:06:03 2010 From: jbaker at zeomega.com (Jason Baker) Date: Wed, 24 Mar 2010 13:06:03 -0500 Subject: [Python-Dev] [Distutils] At least one package management tool for 2.7 In-Reply-To: References: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com> <24ea26601003240527t33c038b2ka1b865dcca172374@mail.gmail.com> Message-ID: On Wed, Mar 24, 2010 at 12:53 PM, Darren Dale wrote: > On Wed, Mar 24, 2010 at 1:19 PM, Ian Bicking wrote: > > On Wed, Mar 24, 2010 at 7:27 AM, Olemis Lang wrote: > >> My experience is that only `install_requires` is needed (unless you > >> want to create app bundles AFAICR) , but in practice I've noticed that > >> *some* easy_installable packages are not pip-able (though I had no > >> time to figure out why :-/ ) > > > > Usually this is because Setuptools is poking at objects to do its > > work, while pip tries to work mostly with subprocesses. Though to > > complicate things a bit, pip makes sure the Setuptools monkeypatches > > to distutils are applied, so that it's always as though the setup.py > > says "from setuptools import setup". easy_install *also* does this. > > > > But then easy_install starts calling methods and whatnot, while pip just > does: > > > > setup.py install --single-version-externally-managed --no-deps > > --record some_tmp_file > > > > The --no-deps keeps Setuptools from resolving dependencies > > Seeking clarification: how can pip recursively install dependencies > *and* keep Setuptools from resolving dependencies? > > Using the --no-deps option to setup.py -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Mar 24 19:36:41 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 24 Mar 2010 11:36:41 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp> <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> <201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com> <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com> Message-ID: On Wed, Mar 24, 2010 at 11:26 AM, Alexander Belopolsky wrote: > I wonder why Python did not follow Java model where Float NaN objects > unlike raw float NaNs compare equal to themselves. ? ?One reason may > be that Python does not have raw floats, but if someone needs IEEE 754 > NaNs, one can use numpy scalars or add arithmetics to ctypes numerical > types. Probably because we were blindly following the IEEE standard without understanding it in every detail. -- --Guido van Rossum (python.org/~guido) From dickinsm at gmail.com Wed Mar 24 19:50:41 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 24 Mar 2010 18:50:41 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp> <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> <201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com> <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com> Message-ID: <5c6f2a5d1003241150q24557b16o772953d9aa404f30@mail.gmail.com> On Wed, Mar 24, 2010 at 6:26 PM, Alexander Belopolsky wrote: > Mark, I wonder if you could describe an algorithm off the top of your > head that relies on NaN == NaN being false. > No, I certainly couldn't! And I often wonder if the original IEEE 754 committee, given 20/20 foresight, would have made the same decisions regarding comparisons of nans. It's certainly not one of my favourite features of IEEE 754. (Though sqrt(-0.) -> -0. ranks lower for me. Grr.) A bogus application that I've often seen mentioned is that it allows checking whether a float 'x' is a nan by doing `x == x'; but the proper way to do this is to have an 'isnan' function or method, so this isn't particularly convincing. Slightly more convincing is history: this is the way that nan comparisons behave in other languages (Fortran, C) used for numerics. If Python were to do something different then a naively translated algorithm from another language would fail. It's the behaviour that numerically-aware people expect, and I'd expect to get complaints from those people if it changed. Mark From alexander.belopolsky at gmail.com Wed Mar 24 19:55:53 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 24 Mar 2010 14:55:53 -0400 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp> <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> <201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com> <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com> Message-ID: On Wed, Mar 24, 2010 at 2:36 PM, Guido van Rossum wrote: .. > Probably because we were blindly following the IEEE standard without > understanding it in every detail. Are you talking about "accidental" support for NaNs in older versions of Python or about recent efforts to support them properly in math and decimal modules? I feel you are too harsh on the developers that worked in this area. I dare to suggest that the current situation is not due to lack of understanding of the standard, but due to pragmatic decisions made in early development and desire for backward compatibility in the recent versions. Is this an area where design changes are feasible? IIRC, NaN support was never "officially" in the language, but it may have changed with the decimal module. From alexander.belopolsky at gmail.com Wed Mar 24 20:04:22 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 24 Mar 2010 15:04:22 -0400 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003241150q24557b16o772953d9aa404f30@mail.gmail.com> References: <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp> <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> <201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com> <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com> <5c6f2a5d1003241150q24557b16o772953d9aa404f30@mail.gmail.com> Message-ID: On Wed, Mar 24, 2010 at 2:50 PM, Mark Dickinson wrote: .. > ?If Python were to do something different then a naively translated > algorithm from another language would fail. ?It's the behaviour that > numerically-aware people expect, and I'd expect to get complaints from > those people if it changed. Numerically-aware people are likely to be aware of the differences in languages that they use. I think in this day and age you are more likely to hear from confused Java programmers than from Fortran or even C folks. From dickinsm at gmail.com Wed Mar 24 20:26:11 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 24 Mar 2010 19:26:11 +0000 Subject: [Python-Dev] Why is nan != nan? Message-ID: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> [Changing the subject line, since we're way off the original topic] On Wed, Mar 24, 2010 at 7:04 PM, Alexander Belopolsky wrote: > On Wed, Mar 24, 2010 at 2:50 PM, Mark Dickinson wrote: > .. >> ?If Python were to do something different then a naively translated >> algorithm from another language would fail. ?It's the behaviour that >> numerically-aware people expect, and I'd expect to get complaints from >> those people if it changed. > > Numerically-aware people are likely to be aware of the differences in > languages that they use. Sure. But I'd still expect them to complain. :) Here's an interesting recent blog post on this subject, from the creator of Eiffel: http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/ Mark From steve at pearwood.info Wed Mar 24 20:34:13 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 25 Mar 2010 06:34:13 +1100 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> References: <4BA9FBCE.5040908@gmail.com> <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> Message-ID: <201003250634.13861.steve@pearwood.info> On Thu, 25 Mar 2010 03:22:29 am Mark Dickinson wrote: > The obvious way to do this nan interning for floats would be to put > the interning code into PyFloat_FromDouble. ?I'm not sure whether > this would be worth the cost in terms of added code (and possibly > reduced performance, since the nan check would be done every time a > float was returned), but I'd be willing to review a patch. I hope that it's obvious from my previous post that I do NOT want such interning done, but since I put the idea in people's heads, I wish to reiterate that I'm against the idea: -1 on interning NaNs. For the rare application where it might be useful, it is easy to do in the application code. -- Steven D'Aprano From stefan at bytereef.org Wed Mar 24 20:36:13 2010 From: stefan at bytereef.org (Stefan Krah) Date: Wed, 24 Mar 2010 20:36:13 +0100 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BA9FC18.90102@gmail.com> References: <20100323120959.GA486@mail.bytereef.org> <4BA9FC18.90102@gmail.com> Message-ID: <20100324193613.GA470@mail.bytereef.org> Nick Coghlan wrote: > Raymond Hettinger wrote: > > The decimal module is already drowning in complexity, > > so it would be best to keep it simple: one boolean flag > > that if set would warn about any implicit decimal/float > > interaction. > > Agreed - those that want exceptions instead can use the usual warnings > module mechanisms to trigger them. I'm not sure about the warnings module. If lower complexity is a goal, I would prefer Facundo's original proposal of just adding a single new signal. Users who just want to know if a NonIntegerConversion has occurred can check the flags, users who want an exception set the trap. With the warnings module, users have to know (and deal with) two exception handling/suppressing mechanisms. Stefan Krah From guido at python.org Wed Mar 24 20:51:13 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 24 Mar 2010 12:51:13 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp> <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> <201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com> <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com> Message-ID: On Wed, Mar 24, 2010 at 11:55 AM, Alexander Belopolsky wrote: > On Wed, Mar 24, 2010 at 2:36 PM, Guido van Rossum wrote: > .. >> Probably because we were blindly following the IEEE standard without >> understanding it in every detail. > > Are you talking about "accidental" support for NaNs in older versions > of Python or about recent efforts to support them properly in math and > decimal modules? My recollection include recent efforts, such as the math.isnan() function. I don't recall ever hearing an argument for the peculiar behavior of NaN in comparisons beyond "this is what the IEEE standard prescribes." > I feel you are too harsh on the developers that worked in this area. Maybe. I didn't mean to put down individuals complicit in the decision -- in fact I would say that I am to blame myself for not probing deeper. > I dare to suggest that the current situation is not due to lack of > understanding of the standard, but due to pragmatic decisions made in > early development and desire for backward compatibility in the recent > versions. I think that originally NaN (and Inf) behavior was so platform-dependent that it really wouldn't have mattered if we had changed it. > Is this an area where design changes are feasible? ?IIRC, NaN support > was never "officially" in the language, but it may have changed with > the decimal module. It has been at least since 2.6 introduced math.isnan(), and ISTR there was a proposal to add a separate module to deal with NaN and Inf properly in a pure-python library module. -- --Guido van Rossum (python.org/~guido) From alexander.belopolsky at gmail.com Wed Mar 24 20:51:46 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 24 Mar 2010 15:51:46 -0400 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003241150q24557b16o772953d9aa404f30@mail.gmail.com> References: <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp> <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> <201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com> <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com> <5c6f2a5d1003241150q24557b16o772953d9aa404f30@mail.gmail.com> Message-ID: On Wed, Mar 24, 2010 at 2:50 PM, Mark Dickinson wrote: > On Wed, Mar 24, 2010 at 6:26 PM, Alexander Belopolsky > wrote: >> Mark, I wonder if you could describe an algorithm off the top of your >> head that relies on NaN == NaN being false. >> > > No, I certainly couldn't! ?And I often wonder if the original IEEE 754 > committee, given 20/20 foresight, would have made the same decisions > regarding comparisons of nans. ?It's certainly not one of my favourite > features of IEEE 754. I tried to google the rationale for the IEEE 754 decision, but came up with nothing. Here are a few representative results: """ So without fear let me not stop at the arguments that ?the committee must have voted on this point and they obviously knew what they were doing? and ?it is the standard and implemented on zillions of machines, you cannot change it now?. """ - "Reflexivity, and other pillars of civilization" by Bertrand Meyer http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/ """ I suppose this simplifies numerical computations in some way, but I couldn't find an explicitly stated reason, not even in the Lecture Notes on the Status of IEEE 754 by Kahan which discusses other design decisions in detail. """ - "What is the rationale for all comparisons returning false for IEEE754 NaN values?" http://stackoverflow.com/questions/1565164/what-is-the-rationale-for-all-comparisons-returning-false-for-ieee754-nan-values From arcriley at gmail.com Wed Mar 24 21:18:53 2010 From: arcriley at gmail.com (Arc Riley) Date: Wed, 24 Mar 2010 16:18:53 -0400 Subject: [Python-Dev] GSoC 2010 is on -- projects? In-Reply-To: <4dc473a51003241016va2503cexf5c0ec55d5c30bda@mail.gmail.com> References: <20100319023630.GC1617@idyll.org> <4BA31858.3010904@gmail.com> <4dc473a51003241016va2503cexf5c0ec55d5c30bda@mail.gmail.com> Message-ID: I'm sure we can find you a mentor. On Wed, Mar 24, 2010 at 1:16 PM, Joe Amenta wrote: > On Fri, Mar 19, 2010 at 2:45 AM, Terry Reedy wrote: > >> On 3/19/2010 2:23 AM, Laurent Gautier wrote: >> >>> On 3/19/10 3:36 AM, C. Titus Brown wrote: >>> >>>> Hi all, >>>> >>>> once again, the PSF has been accepted as a mentoring foundation for >>>> the Google >>>> Summer of Code! This year, we're going to emphasize python 3 porting, so >>>> please think of projects you'd like to see tackled. >>>> >>> >>> Hi, >>> >>> >>> Does this mean that any other python project could potentially see >>> itself ported to Python 3 in the course of this SoC ? >>> >> >> The theme should include both general porting tools and specific projects, >> especially infrastructure projects like numeric, which are blocking the >> porting of other projects. It would be nice if those doing specific projects >> contributed their experience/knowledge to a central pool. >> >> >> If so, can any project owner submit a request for help, >>> >> >> Any project owner is *always* free to ask for help (on python-list, but >> now here in this thread). Those who can also mentor might be more likely to >> get it. If I were a student, I would consider serious interest from a >> project owner (and a promise to distribute a port, when ready), a >> prerequisite. >> >> >> > or is there going to be a list >> >>> of projects that would nice to port, or will a voting system of some >>> sort be put in place ? >>> >> >> Like most contributors, students choose projects, within the limits of >> what they can get mentors for, that scratch their itches. They may or may >> not otherwise be swayed by requests and opinions. >> >> My views. >> >> Terry Jan Reedy >> >> > > Would anyone be interested in mentoring further lib3to2 work? I'm planning > on applying again as a student. > > --Joe Amenta > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/arcriley%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Wed Mar 24 21:26:05 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 24 Mar 2010 13:26:05 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp> <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> <201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com> <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com> <5c6f2a5d1003241150q24557b16o772953d9aa404f30@mail.gmail.com> Message-ID: <56FC12CB-0A9C-46EC-A793-8ACAB8CAC1A3@gmail.com> On Mar 24, 2010, at 12:51 PM, Alexander Belopolsky wrote: > > - "Reflexivity, and other pillars of civilization" by Bertrand Meyer > http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/ Excellent link. Thanks for the research. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Wed Mar 24 21:37:05 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 24 Mar 2010 16:37:05 -0400 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <56FC12CB-0A9C-46EC-A793-8ACAB8CAC1A3@gmail.com> References: <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> <201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com> <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com> <5c6f2a5d1003241150q24557b16o772953d9aa404f30@mail.gmail.com> <56FC12CB-0A9C-46EC-A793-8ACAB8CAC1A3@gmail.com> Message-ID: Thanks. Same link reported concurrently by Mark in the "Why is nan != nan?" thread. On Wed, Mar 24, 2010 at 4:26 PM, Raymond Hettinger wrote: > > On Mar 24, 2010, at 12:51 PM, Alexander Belopolsky wrote: > > - "Reflexivity, and other pillars of civilization" by Bertrand Meyer > http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/ > > Excellent link. > Thanks for the research. > > Raymond > From raymond.hettinger at gmail.com Wed Mar 24 21:56:14 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 24 Mar 2010 13:56:14 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <20100324193613.GA470@mail.bytereef.org> References: <20100323120959.GA486@mail.bytereef.org> <4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org> Message-ID: FWIW, my viewpoint on this is softening over time and I no longer feel a need to push for a new context flag. It is probably simplest for users if implicit coercions didn't come with control knobs. We already have Fraction+float-->float occurring without any exceptions or warnings, and nothing bad has happened as a result. Also, I'm reminded of Tim Peter's admonition to resist extending the decimal spec. I used to worry that any decimal/float interactions were most likely errors and shouldn't pass silently. Now, I've just stopped worrying and I feel better already ;-) Adding a FAQ entry is simpler than building-out Context object circuitry and documenting it in an understandable way. Raymond On Mar 24, 2010, at 12:36 PM, Stefan Krah wrote: > Nick Coghlan wrote: >> Raymond Hettinger wrote: >>> The decimal module is already drowning in complexity, >>> so it would be best to keep it simple: one boolean flag >>> that if set would warn about any implicit decimal/float >>> interaction. >> >> Agreed - those that want exceptions instead can use the usual warnings >> module mechanisms to trigger them. > > > I'm not sure about the warnings module. If lower complexity is a goal, > I would prefer Facundo's original proposal of just adding a single new > signal. Users who just want to know if a NonIntegerConversion has occurred > can check the flags, users who want an exception set the trap. > > With the warnings module, users have to know (and deal with) two exception > handling/suppressing mechanisms. > > > Stefan Krah From greg.ewing at canterbury.ac.nz Wed Mar 24 22:13:52 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 25 Mar 2010 09:13:52 +1200 Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask In-Reply-To: References: <20100322095604.3371039e@heresy> <4BA80418.6030905@canterbury.ac.nz> <4BA93580.9000402@canterbury.ac.nz> Message-ID: <4BAA8090.9080508@canterbury.ac.nz> Isaac Morland wrote: > the > benefit to me and to Greg and to others writing .py code is that our > directories will contain *.py and __pycache__, rather than *.py and > *.pyc. So it will be much easier to see what is actually there. Yes. When using MacOSX I do most of my work using the Finder's column view. With two windows open one above the other, there's room for about a dozen files to be seen at once. There's no way to filter the view or sort by anything other than name, so with the current .pyc scheme, half of that valuable screen space is wasted. -- Greg From dickinsm at gmail.com Wed Mar 24 22:09:26 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 24 Mar 2010 21:09:26 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <20100323120959.GA486@mail.bytereef.org> <4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org> Message-ID: <5c6f2a5d1003241409g806d79dw4cd245b97ce47ae9@mail.gmail.com> Slight change of topic. I've been implementing the extra comparisons required for the Decimal type and found an anomaly while testing. Currently in py3k, order comparisons (but not ==, !=) between a complex number and another complex, float or int raise TypeError: >>> z = complex(0, 0) >>> z < int() Traceback (most recent call last): File "", line 1, in TypeError: unorderable types: complex() < int() >>> z < float() Traceback (most recent call last): File "", line 1, in TypeError: unorderable types: complex() < float() >>> z < complex() Traceback (most recent call last): File "", line 1, in TypeError: unorderable types: complex() < complex() But Fraction is the odd man out: a comparison between a Fraction and a complex raises a TypeError for complex numbers with nonzero imaginary component, but returns a boolean value if the complex number has zero imaginary component: >>> z < Fraction() False >>> complex(0, 1) < Fraction() Traceback (most recent call last): File "", line 1, in TypeError: unorderable types: complex() < Fraction() I'm tempted to call this Fraction behaviour a bug, but maybe it arises from the numeric integration themes of PEP 3141. Any ideas? Mark From guido at python.org Wed Mar 24 22:09:59 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 24 Mar 2010 14:09:59 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <20100323120959.GA486@mail.bytereef.org> <4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org> Message-ID: W00t! On Wed, Mar 24, 2010 at 1:56 PM, Raymond Hettinger wrote: > FWIW, my viewpoint on this is softening over time > and I no longer feel a need to push for a new context flag. > > It is probably simplest for users if implicit coercions didn't come > with control knobs. ?We already have Fraction+float-->float > occurring without any exceptions or warnings, and nothing > bad has happened as a result. > > Also, I'm reminded of Tim Peter's admonition to resist > extending the decimal spec. > > I used to worry that any decimal/float interactions were > most likely errors and shouldn't pass silently. ?Now, I've > just stopped worrying and I feel better already ;-) > Adding a FAQ entry is simpler than building-out > Context object circuitry and documenting it in an > understandable way. > > Raymond > > > > On Mar 24, 2010, at 12:36 PM, Stefan Krah wrote: > >> Nick Coghlan wrote: >>> Raymond Hettinger wrote: >>>> The decimal module is already drowning in complexity, >>>> so it would be best to keep it simple: ?one boolean flag >>>> that if set would warn about any implicit decimal/float >>>> interaction. >>> >>> Agreed - those that want exceptions instead can use the usual warnings >>> module mechanisms to trigger them. >> >> >> I'm not sure about the warnings module. If lower complexity is a goal, >> I would prefer Facundo's original proposal of just adding a single new >> signal. Users who just want to know if a NonIntegerConversion has occurred >> can check the flags, users who want an exception set the trap. >> >> With the warnings module, users have to know (and deal with) two exception >> handling/suppressing mechanisms. >> >> >> Stefan Krah > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From dickinsm at gmail.com Wed Mar 24 22:11:26 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 24 Mar 2010 21:11:26 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <20100323120959.GA486@mail.bytereef.org> <4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org> Message-ID: <5c6f2a5d1003241411y21d9d661p16501c48d4bfcd13@mail.gmail.com> On Wed, Mar 24, 2010 at 8:56 PM, Raymond Hettinger wrote: > FWIW, my viewpoint on this is softening over time > and I no longer feel a need to push for a new context flag. > > It is probably simplest for users if implicit coercions didn't come > with control knobs. ?We already have Fraction+float-->float > occurring without any exceptions or warnings, and nothing > bad has happened as a result. I agree with this; I'd be happy to avoid the control knobs. Mark From raymond.hettinger at gmail.com Wed Mar 24 22:29:27 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 24 Mar 2010 14:29:27 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003241409g806d79dw4cd245b97ce47ae9@mail.gmail.com> References: <20100323120959.GA486@mail.bytereef.org> <4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org> <5c6f2a5d1003241409g806d79dw4cd245b97ce47ae9@mail.gmail.com> Message-ID: <341CCB69-D532-4AF3-B811-34BA8263DEAD@gmail.com> On Mar 24, 2010, at 2:09 PM, Mark Dickinson wrote: > Slight change of topic. I've been implementing the extra comparisons > required for the Decimal type and found an anomaly while testing. > Currently in py3k, order comparisons (but not ==, !=) between a > complex number and another complex, float or int raise TypeError: > >>>> z = complex(0, 0) >>>> z < int() > Traceback (most recent call last): > File "", line 1, in > TypeError: unorderable types: complex() < int() >>>> z < float() > Traceback (most recent call last): > File "", line 1, in > TypeError: unorderable types: complex() < float() >>>> z < complex() > Traceback (most recent call last): > File "", line 1, in > TypeError: unorderable types: complex() < complex() > > But Fraction is the odd man out: a comparison between a Fraction and > a complex raises a TypeError for complex numbers with nonzero > imaginary component, but returns a boolean value if the complex number > has zero imaginary component: > >>>> z < Fraction() > False >>>> complex(0, 1) < Fraction() > Traceback (most recent call last): > File "", line 1, in > TypeError: unorderable types: complex() < Fraction() > > I'm tempted to call this Fraction behaviour a bug, but maybe it arises > from the numeric integration themes of PEP 3141. Any ideas? Conceptually, it's a bug. The numeric tower treats non-complex numbers as special cases of complex where the imaginary component is zero (that's why the non-complex types all support real/imag), and since complex numbers are not allowed to compare to themselves, they shouldn't compare to anything else either. To confirm, we should ask Jeffrey Y to opine. Raymond From alexander.belopolsky at gmail.com Wed Mar 24 22:31:57 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 24 Mar 2010 17:31:57 -0400 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> Message-ID: On Wed, Mar 24, 2010 at 3:26 PM, Mark Dickinson wrote: .. > Here's an interesting recent blog post on this subject, from the > creator of Eiffel: > > http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/ It appears that Meyer's view has evolved over the years: """ In this context it doesn't particularly shock me that operations on NaN should cause invariant violations. After all, isn't NaN supposed to mean "not a number"? If it's not a number, it doesn't have to satisfy the properties of numbers. """ "NaN and floating point exceptions" by Roger Browne quoting an ISE Eiffel mailing list post by Bertrand Meyer http://teameiffel.blogspot.com/2006/04/nan-and-floating-point-exceptions.html Compare this with the conclusion he reached in "Pillars:" "It is rather dangerous indeed to depart from the fundamental laws of mathematics. " To bring the discussion back on topic for python-dev, I would argue that reflexivity should hold for hashable values. Having it otherwise would lead to unsurmountable problems when storing such values in sets or using them as keys in dictionaries. If x == x is False stays for x = float('nan'), I would argue that hash(float('nan')) should raise NotImplemented or ValueError. From v+python at g.nevcal.com Wed Mar 24 23:09:29 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 24 Mar 2010 15:09:29 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <20100323120959.GA486@mail.bytereef.org> <4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org> Message-ID: <4BAA8D99.4000606@g.nevcal.com> On 3/24/2010 1:56 PM, Raymond Hettinger wrote: > FWIW, my viewpoint on this is softening over time > and I no longer feel a need to push for a new context flag. > To make Decimal useful for people that want to control its numerical quality, there must be a way to exclude accidental operations, and preferably an option for producing exceptions. Otherwise the status quo in 3.x is preferable to adding it to the numeric tree. I agree that not worrying can make you feel better, though. :) > IIRC, the original purpose of a NaN was to serve as a placeholder > value in a long series of floating point ops so that the programmer > would not have to introduce edge case tests at each stage > of a calculation. Yet, I look at the code for the decimal module > and the C code for the math module and see that the opposite > result occurred, the code is littered with is_special(x) tests > and handlers. You are looking at the wrong code. The point is that the code _using_ the math module and decimal module doesn't have to be littered with edge case tests; not that the implementation, which the good folks at IEEE figured would be in custom silicon that could do the edge case checking in parallel with other operations and with no net slowdown, would not have to be littered with edge cases. Glenn From martin at v.loewis.de Wed Mar 24 23:14:18 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 24 Mar 2010 23:14:18 +0100 Subject: [Python-Dev] argparse.py is licensed under the Apache License In-Reply-To: References: Message-ID: <4BAA8EBA.8030302@v.loewis.de> > Yes, I have signed a contributor agreement. I was thinking of just > removing the license entirely, but if it's better to replace it with > "Licensed to PSF under a Contributor Agreement", that's fine too. Let > me know, and I'll take care of this today. Technically, your *contribution* should have included that line, whereas the code that we (i.e. the PSF) immediately relicense won't. IIUC, and IANAL, the line indicates that you do intend your contribution to be covered by your agreement (just so you couldn't claim we stole it from your laptop, and you didn't really mean to contribute it). Of course, committing it to subversion is an even stronger indication that you wanted to contribute it :-) The PSF then immediately exercises its right to relicense the code, currently under the PSF license. Regards, Martin From raymond.hettinger at gmail.com Wed Mar 24 23:21:53 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 24 Mar 2010 15:21:53 -0700 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> Message-ID: On Mar 24, 2010, at 2:31 PM, Alexander Belopolsky wrote: > """ > In this context it doesn't particularly shock me that operations on > NaN should cause invariant violations. After all, isn't NaN supposed > to mean "not a number"? If it's not a number, it doesn't have to > satisfy the properties of numbers. > """ This sound like a good, universal reply to "bug reports" about various containers/tools not working with NaNs :-) Bug report: "Container X or Function Y doesn't behave well when I give it a NaN as an input". Closed -- Won't fix: "It does not shock me that a NaN violates that tool or container's invariants." > > To bring the discussion back on topic for python-dev, I would argue > that reflexivity should hold for hashable values. Having it otherwise > would lead to unsurmountable problems when storing such values in sets > or using them as keys in dictionaries. If x == x is False stays for x > = float('nan'), I would argue that hash(float('nan')) should raise > NotImplemented or ValueError. Hashability isn't the only place where you have a problem. Even unordered collections are affected: >>> class ListBasedSet(collections.Set): ''' Alternate set implementation favoring space over speed and not requiring the set elements to be hashable. ''' def __init__(self, iterable): self.elements = lst = [] for value in iterable: if value not in lst: lst.append(value) def __iter__(self): return iter(self.elements) def __contains__(self, value): return any(value == elem for elem in self.elements) def __len__(self): return len(self.elements) >>> n = float('Nan') >>> s = ListBasedSet([n]) >>> n in s # unexpected result False >>> len(s) # expected result 1 >>> len(s & s) # unexpected result 0 If we want to be able to reason about our programs, then we need to rely on equality relations being reflexsive, symmetric, and transitive. Otherwise, containers and functions can't even make minimal guarantees about what they do. Anything else smells of a Douglas Hofstadter style or Betrand Russell style logic bomb: * this sentence is a lie * this object isn't equal to itself * this is a set containing sets that are not members of themselves The property of NaN objects not being equal to themselves is more harmful than helpful. We should probably draw the line at well-defined numeric contexts such as the decimal module and stop trying to propagate NaN awareness throughout the entire object model. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Mar 24 23:23:11 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 24 Mar 2010 15:23:11 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <341CCB69-D532-4AF3-B811-34BA8263DEAD@gmail.com> References: <20100323120959.GA486@mail.bytereef.org> <4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org> <5c6f2a5d1003241409g806d79dw4cd245b97ce47ae9@mail.gmail.com> <341CCB69-D532-4AF3-B811-34BA8263DEAD@gmail.com> Message-ID: On Wed, Mar 24, 2010 at 2:29 PM, Raymond Hettinger wrote: > > On Mar 24, 2010, at 2:09 PM, Mark Dickinson wrote: > >> Slight change of topic. ?I've been implementing the extra comparisons >> required for the Decimal type and found an anomaly while testing. >> Currently in py3k, order comparisons (but not ==, !=) between a >> complex number and another complex, float or int raise TypeError: >> >>>>> z = complex(0, 0) >>>>> z < int() >> Traceback (most recent call last): >> ?File "", line 1, in >> TypeError: unorderable types: complex() < int() >>>>> z < float() >> Traceback (most recent call last): >> ?File "", line 1, in >> TypeError: unorderable types: complex() < float() >>>>> z < complex() >> Traceback (most recent call last): >> ?File "", line 1, in >> TypeError: unorderable types: complex() < complex() >> >> But Fraction is the odd man out: ?a comparison between a Fraction and >> a complex raises a TypeError for complex numbers with nonzero >> imaginary component, but returns a boolean value if the complex number >> has zero imaginary component: >> >>>>> z < Fraction() >> False >>>>> complex(0, 1) < Fraction() >> Traceback (most recent call last): >> ?File "", line 1, in >> TypeError: unorderable types: complex() < Fraction() >> >> I'm tempted to call this Fraction behaviour a bug, but maybe it arises >> from the numeric integration themes of PEP 3141. ?Any ideas? > > Conceptually, it's a bug. ?The numeric tower treats non-complex > numbers as special cases of complex where the imaginary > component is zero (that's why the non-complex types all support > real/imag), and since complex numbers are not allowed to compare > to themselves, they shouldn't compare to anything else either. That's how I read the PEP too. PEP 3141 doesn't define any ordering operations on Complex, they only show up on Real. > To confirm, we should ask Jeffrey Y to opine. CC'ed him. After all looks like it was he who added it to Fraction. :-) -- --Guido van Rossum (python.org/~guido) From raymond.hettinger at gmail.com Wed Mar 24 23:25:09 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 24 Mar 2010 15:25:09 -0700 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> Message-ID: > Even unordered collections are affected: s/unordered/unhashed, equality-based/ Raymond From alexander.belopolsky at gmail.com Wed Mar 24 23:30:17 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 24 Mar 2010 18:30:17 -0400 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> Message-ID: On Wed, Mar 24, 2010 at 6:21 PM, Raymond Hettinger wrote: .. > If we want to be able to reason about our programs, > then we need to rely on equality relations being > reflexsive, symmetric, and transitive. ?Otherwise, > containers and functions can't even make minimal > guarantees?about what they do. +1 > .. ?We should probably draw the > line at well-defined numeric contexts such as the decimal module > and stop trying to propagate NaN awareness throughout the > entire object model. I am not sure what this means in practical terms. Should float('nan') == float('nan') return True or should float('nan') raise an exception to begin with? I would prefer the former. From dickinsm at gmail.com Wed Mar 24 23:31:42 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 24 Mar 2010 22:31:42 +0000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> Message-ID: <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> On Wed, Mar 24, 2010 at 10:30 PM, Alexander Belopolsky wrote: > On Wed, Mar 24, 2010 at 6:21 PM, Raymond Hettinger > wrote: > .. >> If we want to be able to reason about our programs, >> then we need to rely on equality relations being >> reflexsive, symmetric, and transitive. ?Otherwise, >> containers and functions can't even make minimal >> guarantees?about what they do. > > +1 > >> .. ?We should probably draw the >> line at well-defined numeric contexts such as the decimal module >> and stop trying to propagate NaN awareness throughout the >> entire object model. > > I am not sure what this means in practical terms. ? Should > float('nan') == float('nan') return True or should float('nan') raise > an exception to begin with? ? I would prefer the former. > Neither is necessary, because Python doesn't actually use == as the equivalence relation for containment testing: the actual equivalence relation is: x equivalent to y iff id(x) == id(y) or x == y. This restores the missing reflexivity (besides being a useful optimization). Mark From alexander.belopolsky at gmail.com Wed Mar 24 23:36:49 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 24 Mar 2010 18:36:49 -0400 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> Message-ID: On Wed, Mar 24, 2010 at 6:31 PM, Mark Dickinson wrote: .. > Neither is necessary, because Python doesn't actually use == as the > equivalence relation for containment testing: ?the actual equivalence > relation is: ?x equivalent to y iff id(x) == id(y) or x == y. ?This > restores the missing reflexivity (besides being a useful > optimization). No, it does not: >>> float('nan') in [float('nan')] False It would if NaNs were always interned, but they are not. From alexander.belopolsky at gmail.com Wed Mar 24 23:43:03 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 24 Mar 2010 18:43:03 -0400 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> Message-ID: Not to mention the following aberrations: >>> {x for x in [float('nan')] * 10} {nan} >>> {float(x) for x in ['nan'] * 10} {nan, nan, nan, nan, nan, nan, nan, nan, nan, nan} >>> {float('nan')} | {float('nan')} {nan, nan} >>> {float('nan')} & {float('nan')} set() On Wed, Mar 24, 2010 at 6:36 PM, Alexander Belopolsky wrote: > On Wed, Mar 24, 2010 at 6:31 PM, Mark Dickinson wrote: > .. >> Neither is necessary, because Python doesn't actually use == as the >> equivalence relation for containment testing: ?the actual equivalence >> relation is: ?x equivalent to y iff id(x) == id(y) or x == y. ?This >> restores the missing reflexivity (besides being a useful >> optimization). > > No, it does not: > >>>> float('nan') in [float('nan')] > False > > > It would if NaNs were always interned, but they are not. > From dickinsm at gmail.com Wed Mar 24 23:47:42 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 24 Mar 2010 22:47:42 +0000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> Message-ID: <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> On Wed, Mar 24, 2010 at 10:36 PM, Alexander Belopolsky wrote: > On Wed, Mar 24, 2010 at 6:31 PM, Mark Dickinson wrote: > .. >> Neither is necessary, because Python doesn't actually use == as the >> equivalence relation for containment testing: ?the actual equivalence >> relation is: ?x equivalent to y iff id(x) == id(y) or x == y. ?This >> restores the missing reflexivity (besides being a useful >> optimization). > > No, it does not: > >>>> float('nan') in [float('nan')] > False Sure, but just think of it as having two different nans there. (You could imagine thinking of the id of the nan as part of the payload.) There's no ideal solution here; IMO, the compromise that currently exists is an acceptable one. Mark From alexander.belopolsky at gmail.com Wed Mar 24 23:52:38 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 24 Mar 2010 18:52:38 -0400 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> Message-ID: On Wed, Mar 24, 2010 at 6:47 PM, Mark Dickinson wrote: .. > There's no ideal solution here; ?IMO, the compromise that currently > exists is an acceptable one. I don't see a compromise. So far I failed to find a use case that benefits from NaN violating reflexivity. From dickinsm at gmail.com Thu Mar 25 00:02:39 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 24 Mar 2010 23:02:39 +0000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> Message-ID: <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> On Wed, Mar 24, 2010 at 10:52 PM, Alexander Belopolsky wrote: > On Wed, Mar 24, 2010 at 6:47 PM, Mark Dickinson wrote: > .. >> There's no ideal solution here; ?IMO, the compromise that currently >> exists is an acceptable one. > > I don't see a compromise. ? So far I failed to find a use case that > benefits from NaN violating reflexivity. > So if I understand correctly, you propose that float('nan') == float('nan') return True. Would you also suggest extending this behaviour to Decimal nans? From alexander.belopolsky at gmail.com Thu Mar 25 00:11:09 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 24 Mar 2010 19:11:09 -0400 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> Message-ID: On Wed, Mar 24, 2010 at 7:02 PM, Mark Dickinson wrote: .. > > So if I understand correctly, you propose that float('nan') == > float('nan') return True. ?Would you also suggest extending this > behaviour to Decimal nans? > yes From steven.bethard at gmail.com Thu Mar 25 00:18:02 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 24 Mar 2010 16:18:02 -0700 Subject: [Python-Dev] argparse.py is licensed under the Apache License In-Reply-To: References: Message-ID: On Wed, Mar 24, 2010 at 10:08 AM, Brett Cannon wrote: > On Wed, Mar 24, 2010 at 09:05, Steven Bethard > wrote: >> On Wed, Mar 24, 2010 at 1:42 AM, Amaury Forgeot d'Arc >> wrote: >> > I noticed that the newly added argparse module has an unusual >> > licence header,?included below. This is the only file in the Python tree >> > that contains an explicit reference to the Apache License, >> > and this leads me to some questions: >> >> Sorry, I forgot to remove this when moving argparse to the stdlib. >> There's probably one test\test_argparse too. > > I say just strip it out. Benjamin beat me to the punch for argparse.py, but I also fixed this in test_argparse.py. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From gruszczy at gmail.com Thu Mar 25 00:18:21 2010 From: gruszczy at gmail.com (=?UTF-8?Q?Filip_Gruszczy=C5=84ski?=) Date: Thu, 25 Mar 2010 00:18:21 +0100 Subject: [Python-Dev] Issue 6081: format string using mapping rather than kwargs Message-ID: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com> Hello everyone, I was told to bring this issue to python-dev, but I was reluctant to do this until I was confident I can solve the problem. I have managed to provide the patch that works, so now I can talk about how this should be done, just as Eric told me. Here is the link to the ticket: http://bugs.python.org/issue6081. In the issue Raymond Hettinger proposed adding method to string, that would allow to explicitly pass subscriptable object, from which formatting of a string would be done. Two questions arise: 1) whether add such functionality at all? 2) how the method should be called? Should formatting options based on *args be kept (like using {0} in formatted string) or not and how argument should be passed. Eric suggested passing only mapping and only adding *args, if that is found useful by users; I have though about having both mapping and *args, which would keep it similar to the old formatting. Another option is to have *args and one keyword with the mapping (instead of kwargs). I would appreciate any advice on this topic, even if this ticket would be dismissed altogether, as I would like to learn as much as possible on practices on developing Python. -- Filip Gruszczy?ski From dickinsm at gmail.com Thu Mar 25 00:31:57 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 24 Mar 2010 23:31:57 +0000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> Message-ID: <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> On Wed, Mar 24, 2010 at 11:11 PM, Alexander Belopolsky wrote: > On Wed, Mar 24, 2010 at 7:02 PM, Mark Dickinson wrote: > .. >> >> So if I understand correctly, you propose that float('nan') == >> float('nan') return True. ?Would you also suggest extending this >> behaviour to Decimal nans? >> > > yes > Okay. So now it seems to me that there are many decisions to make: should any Decimal nan compare equal to any other? What if the two nans have different payloads or signs? How about comparing a signaling nan with either an arbitrary quiet nan, or with the exact quiet nan that corresponds to the signaling nan? How do decimal nans compare with float nans? Should Decimal.compare(Decimal('nan'), Decimal('nan')) return 0 rather than nan? If not, how do you justify the difference between == and compare? If so, how do you justify the deviation from the standard on which the decimal modulo is based? In answering all these questions, you effectively end up developing your own standard, and hoping that all the answers you chose are sensible, consistent, and useful. Alternatively, we could do what we're currently doing: make use of *existing* standards to answer these questions, and rely on the expertise of the many who've thought about this in depth. You say that you don't see any compromise: I say that there's value in adhering to (de facto and de jure) standards, and I see a compromise between standards adherence and Python pragmatics. Mark From greg.ewing at canterbury.ac.nz Thu Mar 25 01:32:49 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 25 Mar 2010 13:32:49 +1300 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <341CCB69-D532-4AF3-B811-34BA8263DEAD@gmail.com> References: <20100323120959.GA486@mail.bytereef.org> <4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org> <5c6f2a5d1003241409g806d79dw4cd245b97ce47ae9@mail.gmail.com> <341CCB69-D532-4AF3-B811-34BA8263DEAD@gmail.com> Message-ID: <4BAAAF31.5060709@canterbury.ac.nz> Raymond Hettinger wrote: > Conceptually, it's a bug. The numeric tower treats non-complex > numbers as special cases of complex where the imaginary > component is zero (that's why the non-complex types all support > real/imag), and since complex numbers are not allowed to compare > to themselves, they shouldn't compare to anything else either. There's a contradiction in there somewhere. If you believe that a non-complex is equivalent to a complex with zero imaginary part, then you *should* be able to compare two complexes provided that their imaginary parts are both zero. (I don't think that should be the case, BTW -- complex numbers live on a two-dimensional plane, and from a geometrical point of view there's no reason to single out the x-axis and give it special treatment.) -- Greg From jyasskin at gmail.com Thu Mar 25 02:15:58 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Wed, 24 Mar 2010 18:15:58 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5c6f2a5d1003241409g806d79dw4cd245b97ce47ae9@mail.gmail.com> References: <20100323120959.GA486@mail.bytereef.org> <4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org> <5c6f2a5d1003241409g806d79dw4cd245b97ce47ae9@mail.gmail.com> Message-ID: <5d44f72f1003241815g109236fco1ab9607544352cf0@mail.gmail.com> On Wed, Mar 24, 2010 at 2:09 PM, Mark Dickinson wrote: > Slight change of topic. ?I've been implementing the extra comparisons > required for the Decimal type and found an anomaly while testing. > Currently in py3k, order comparisons (but not ==, !=) between a > complex number and another complex, float or int raise TypeError: > >>>> z = complex(0, 0) >>>> z < int() > Traceback (most recent call last): > ?File "", line 1, in > TypeError: unorderable types: complex() < int() >>>> z < float() > Traceback (most recent call last): > ?File "", line 1, in > TypeError: unorderable types: complex() < float() >>>> z < complex() > Traceback (most recent call last): > ?File "", line 1, in > TypeError: unorderable types: complex() < complex() > > But Fraction is the odd man out: ?a comparison between a Fraction and > a complex raises a TypeError for complex numbers with nonzero > imaginary component, but returns a boolean value if the complex number > has zero imaginary component: > >>>> z < Fraction() > False >>>> complex(0, 1) < Fraction() > Traceback (most recent call last): > ?File "", line 1, in > TypeError: unorderable types: complex() < Fraction() > > I'm tempted to call this Fraction behaviour a bug, but maybe it arises > from the numeric integration themes of PEP 3141. ?Any ideas? I'd call it a bug. From g.brandl at gmx.net Thu Mar 25 07:41:26 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 25 Mar 2010 06:41:26 +0000 Subject: [Python-Dev] Request for commit access In-Reply-To: References: <20100323152748.50ECE1FD33F@kimball.webabinitio.net> <4BA94202.6060502@v.loewis.de> Message-ID: Am 23.03.2010 23:01, schrieb Antoine Pitrou: > Martin v. L?wis v.loewis.de> writes: >> >> Procedurally, I wonder where people got the notion from that you can or >> need to apply for commit access. IIUC, it used to be the case that you >> would be recommended for commit access, by some (more or less senior) >> fellow committer. That person then would work on actually getting commit >> access to the new committer - perhaps by first asking other people in >> private, to avoid any public embarrassment if access is ultimately >> denied. IMO, that committer should then also mentor the new guy, both by >> helping out in difficult cases, and by closely following commits to see >> whether (possibly unstated) conventions are being followed. >> >> I'm not really picking on Brian here specifically, I just want to point >> out that I dislike this (apparent) change in process, primarily because >> of the risk of embarrassment. > > For the record, I'm not opposing any point you are making, but all this is not > clearly written out, and I think that's why people (including me) lately have > been thinking that the candidate for commit rights had to declare himself on > this mailing-list. I don't think it is "wrong" to do it this way -- but of course the hopeful new committer needs to be prepared for a "no". I still would like every new committer to have a "mentor" from the pool of more experienced committers; however for people who are very active on IRC like Brian, there's usually one of the around to review a patch before it is committed. Georg From g.brandl at gmx.net Thu Mar 25 07:54:39 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 25 Mar 2010 06:54:39 +0000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> Message-ID: Am 24.03.2010 22:47, schrieb Mark Dickinson: > On Wed, Mar 24, 2010 at 10:36 PM, Alexander Belopolsky > wrote: >> On Wed, Mar 24, 2010 at 6:31 PM, Mark Dickinson wrote: >> .. >>> Neither is necessary, because Python doesn't actually use == as the >>> equivalence relation for containment testing: the actual equivalence >>> relation is: x equivalent to y iff id(x) == id(y) or x == y. This >>> restores the missing reflexivity (besides being a useful >>> optimization). >> >> No, it does not: >> >>>>> float('nan') in [float('nan')] >> False > > Sure, but just think of it as having two different nans there. (You > could imagine thinking of the id of the nan as part of the payload.) That's interesting. Thinking of each value created by float('nan') as a different nan makes sense to my naive mind, and it also explains nicely the behavior present right now. Each nan comes from a different operation and therefore is a "different" non-number. Of course, float being an immutable type, there is some reason to expect that all values created by float('nan') should be identical, but after all, datetime is an immutable type as well, but you wouldn't expect datetime.now() in [datetime.now()] to be true. The only wart left is that you can't distinguish different nans by their string representation -- this could be remedied by making it ``"nan-%s" % id(self)``, but that looks a bit ugly to me. Georg From mal at egenix.com Thu Mar 25 10:10:26 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 25 Mar 2010 10:10:26 +0100 Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk: Doc/c-api/capsule.rst Doc/c-api/cobject.rst Doc/c-api/concrete.rst Doc/data/refcounts.dat Doc/extending/extending.rst Include/Python.h Include/cStringIO.h Include/cobject.h Include/datetime.h Include/py_curses.h Include/pycapsule.h Include/pyexpat.h Include/ucnhash.h Lib/test/test_sys.py Makefile.pre.in Misc/NEWS Modules/_ctypes/callproc.c Modules/_ctypes/cfield.c Modules/_ctypes/ctypes.h Modules/_cursesmodule.c Modules/_elementtree.c Modules/_testcapimodule.c Modules/cStringIO.c Modules/cjkcodecs/cjkcodecs.h Modules/cjkcodecs/multibytecodec.c Modules/cjkcodecs/multibytecodec.h Modules/datetimemodule.c Modules/pyexpat.c Modules/socketmodule.c Modules/socketmodule.h Modules/unicodedata.c Objects/capsule.c Objects/object.c Objects/unicodeobject.c PC/VS7.1/pythoncore.vcproj PC/VS8.0/pythoncore.vcproj PC/os2emx/python27.def PC/os2vacpp/python.def Python/compile.c Python/getargs.c In-Reply-To: <20100325005454.B1AC5FCCD@mail.python.org> References: <20100325005454.B1AC5FCCD@mail.python.org> Message-ID: <4BAB2882.6070508@egenix.com> larry.hastings wrote: > Author: larry.hastings > Date: Thu Mar 25 01:54:54 2010 > New Revision: 79397 > > Log: > Backported PyCapsule from 3.1, and converted most uses of > CObject to PyCapsule. Backporting PyCapsule is fine, but the changes you made to all those PyCObject uses does not look backwards compatible. The C APIs exposed by the modules (e.g. the datetime module) are used in lots of 3rd party extension modules and changing them from PyCObject to PyCapsule is a major change in the module API. > Added: > python/trunk/Doc/c-api/capsule.rst > python/trunk/Include/pycapsule.h > python/trunk/Objects/capsule.c > Modified: > python/trunk/Doc/c-api/cobject.rst > python/trunk/Doc/c-api/concrete.rst > python/trunk/Doc/data/refcounts.dat > python/trunk/Doc/extending/extending.rst > python/trunk/Include/Python.h > python/trunk/Include/cStringIO.h > python/trunk/Include/cobject.h > python/trunk/Include/datetime.h > python/trunk/Include/py_curses.h > python/trunk/Include/pyexpat.h > python/trunk/Include/ucnhash.h > python/trunk/Lib/test/test_sys.py > python/trunk/Makefile.pre.in > python/trunk/Misc/NEWS > python/trunk/Modules/_ctypes/callproc.c > python/trunk/Modules/_ctypes/cfield.c > python/trunk/Modules/_ctypes/ctypes.h > python/trunk/Modules/_cursesmodule.c > python/trunk/Modules/_elementtree.c > python/trunk/Modules/_testcapimodule.c > python/trunk/Modules/cStringIO.c > python/trunk/Modules/cjkcodecs/cjkcodecs.h > python/trunk/Modules/cjkcodecs/multibytecodec.c > python/trunk/Modules/cjkcodecs/multibytecodec.h > python/trunk/Modules/datetimemodule.c > python/trunk/Modules/pyexpat.c > python/trunk/Modules/socketmodule.c > python/trunk/Modules/socketmodule.h > python/trunk/Modules/unicodedata.c > python/trunk/Objects/object.c > python/trunk/Objects/unicodeobject.c > python/trunk/PC/VS7.1/pythoncore.vcproj > python/trunk/PC/VS8.0/pythoncore.vcproj > python/trunk/PC/os2emx/python27.def > python/trunk/PC/os2vacpp/python.def > python/trunk/Python/compile.c > python/trunk/Python/getargs.c -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 25 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ _______________________________________________ Python-checkins mailing list Python-checkins at python.org http://mail.python.org/mailman/listinfo/python-checkins From steve at pearwood.info Thu Mar 25 11:18:42 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 25 Mar 2010 21:18:42 +1100 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com> Message-ID: <201003252118.43146.steve@pearwood.info> On Thu, 25 Mar 2010 05:26:12 am Alexander Belopolsky wrote: > Mark, I wonder if you could describe an algorithm off the top of your > head that relies on NaN == NaN being false. I don't know whether "relies on" is appropriate, but consider: def myfunc(x, y): if x == y: return 1.0 else: return something_complicated**(x-y) Optimising floating point code is fraught with dangers (the above fails for x=y=INF as well as NAN) but anything that make Not A Numbers pretend to be numbers is a bad thing. I'd like to turn the question around ... what algorithms are there that rely on NaN == NaN being True? -- Steven D'Aprano From ncoghlan at gmail.com Thu Mar 25 11:28:03 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 25 Mar 2010 20:28:03 +1000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <201003250634.13861.steve@pearwood.info> References: <4BA9FBCE.5040908@gmail.com> <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> <201003250634.13861.steve@pearwood.info> Message-ID: <4BAB3AB3.4030204@gmail.com> Steven D'Aprano wrote: > On Thu, 25 Mar 2010 03:22:29 am Mark Dickinson wrote: >> The obvious way to do this nan interning for floats would be to put >> the interning code into PyFloat_FromDouble. I'm not sure whether >> this would be worth the cost in terms of added code (and possibly >> reduced performance, since the nan check would be done every time a >> float was returned), but I'd be willing to review a patch. > > I hope that it's obvious from my previous post that I do NOT want such > interning done, but since I put the idea in people's heads, I wish to > reiterate that I'm against the idea: -1 on interning NaNs. For the rare > application where it might be useful, it is easy to do in the > application code. Yep, and I'll freely admit I didn't know about the potential additional state on NaN values, or I wouldn't have suggested interning automatically. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Mar 25 11:31:33 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 25 Mar 2010 20:31:33 +1000 Subject: [Python-Dev] __pycache__ creation In-Reply-To: References: <20100322095604.3371039e@heresy> <4BAA0088.2010909@gmail.com> Message-ID: <4BAB3B85.701@gmail.com> Ron Adam wrote: > hmmmm... unless there is a __pycache__ *file* located there first. ;-) Just a specific reason why attempting to create __pycache__ can fail (which has defined behaviour in the PEP - running directly from the source without caching the bytecode file). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Mar 25 12:00:35 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 25 Mar 2010 21:00:35 +1000 Subject: [Python-Dev] Issue 6081: format string using mapping rather than kwargs In-Reply-To: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com> References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com> Message-ID: <4BAB4253.8090605@gmail.com> Filip Gruszczy?ski wrote: > I would appreciate any advice on this topic, even if this ticket would > be dismissed altogether, as I would like to learn as much as possible > on practices on developing Python. Raymond's use case is valid, but the currently proposed method name is way too long to be usable and the use of *args is a definite misfeature. I suggest approaching the issue with the exact spec needed to cover the problem Raymond described: Add a method to strings that allows a format string to be applied to an existing mapping without implicitly converting the mapping to a dict first. >From that spec, a straightforward API falls out: def format_mapping(self, kwds): # Method body actually written in C, so it can # easily invoke the internal formatting operation return do_string_format(self, NULL, kwds) Sure, you can't mix positional and keyword arguments the way you can with .format(), but you can't mix and match those with mod-formatting either. *If* support for a sequence were to be added (and that's a big if), it should be added as an explicit second argument, not as a *args argument. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Mar 25 12:22:12 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 25 Mar 2010 21:22:12 +1000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> Message-ID: <4BAB4764.4000500@gmail.com> Mark Dickinson wrote: > Here's an interesting recent blog post on this subject, from the > creator of Eiffel: > > http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/ Interesting. So the natural tweak that would arise from that perspective is for us to restore reflexivity by declaring that any given NaN is equal to *itself* but not to any other NaN (even one with the same payload). With NaN (in general) not being interned, that would actually fit the idea of a NaN implicitly carrying the operation that created the NaN as part of its definition of equivalence. So, I'm specifically putting that proposal on the table for both float and Decimal NaNs in Python: "Not a Number" is not a single floating point value. Instead each instance is a distinct value representing the precise conditions that created it. Thus, two "NaN" values x and y will compare equal iff they are the exact same NaN object (i.e. "if isnan(x) then x == y iff x is y". As stated above, such a change would allow us to restore reflexivity (eliminating a bunch of weirdness) while still honouring the idea of NaN being a set of values rather than a single value. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Mar 25 12:25:35 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 25 Mar 2010 21:25:35 +1000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <201003252118.43146.steve@pearwood.info> References: <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com> <201003252118.43146.steve@pearwood.info> Message-ID: <4BAB482F.4030608@gmail.com> Steven D'Aprano wrote: > I'd like to turn the question around ... what algorithms are there that > rely on NaN == NaN being True? Absolutely anything that expects "x is y" to imply that "x == y". The builtin containers enforce this by checking identity before they check equality, but there are plenty of algorithms that won't. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From gruszczy at gmail.com Thu Mar 25 12:48:28 2010 From: gruszczy at gmail.com (=?UTF-8?Q?Filip_Gruszczy=C5=84ski?=) Date: Thu, 25 Mar 2010 12:48:28 +0100 Subject: [Python-Dev] Issue 6081: format string using mapping rather than kwargs In-Reply-To: <4BAB4253.8090605@gmail.com> References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com> <4BAB4253.8090605@gmail.com> Message-ID: <1be78d221003250448v54b39b9fob81b1eb7364fbf73@mail.gmail.com> > From that spec, a straightforward API falls out: > > ? ?def format_mapping(self, kwds): > ? ? ?# Method body actually written in C, so it can > ? ? ?# easily invoke the internal formatting operation > ? ? ?return do_string_format(self, NULL, kwds) Thanks a lot for the advice, I'll provide according patch hopefully in the few days. -- Filip Gruszczy?ski From dickinsm at gmail.com Thu Mar 25 12:55:57 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 25 Mar 2010 11:55:57 +0000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BAB4764.4000500@gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> Message-ID: <5c6f2a5d1003250455o5f1e2d29ld18c9893d11d3e1@mail.gmail.com> On Thu, Mar 25, 2010 at 11:22 AM, Nick Coghlan wrote: > Mark Dickinson wrote: >> Here's an interesting recent blog post on this subject, from the >> creator of Eiffel: >> >> http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/ > > Interesting. So the natural tweak that would arise from that perspective > is for us to restore reflexivity by declaring that any given NaN is > equal to *itself* but not to any other NaN (even one with the same payload). > > With NaN (in general) not being interned, that would actually fit the > idea of a NaN implicitly carrying the operation that created the NaN as > part of its definition of equivalence. > > So, I'm specifically putting that proposal on the table for both float > and Decimal NaNs in Python: > > ?"Not a Number" is not a single floating point value. Instead each > ?instance is a distinct value representing the precise conditions that > ?created it. Thus, two "NaN" values x and y will compare equal iff they > ?are the exact same NaN object (i.e. "if isnan(x) then x == y iff > ?x is y". In other words, this would make explicit, at the level of ==, what Python's already doing under the hood (e.g. in PyObjectRichCompareBool) for membership testing---at least for nans. > As stated above, such a change would allow us to restore reflexivity > (eliminating a bunch of weirdness) while still honouring the idea of NaN > being a set of values rather than a single value. +0.2 from me. I could happily live with this change; but could also equally live with the existing weirdness. It's still a little odd for an immutable type to care about object identity, but I guess oddness comes with the floating-point territory. :) Mark From eric at trueblade.com Thu Mar 25 12:58:14 2010 From: eric at trueblade.com (Eric Smith) Date: Thu, 25 Mar 2010 07:58:14 -0400 Subject: [Python-Dev] Issue 6081: format string using mapping rather than kwargs In-Reply-To: <1be78d221003250448v54b39b9fob81b1eb7364fbf73@mail.gmail.com> References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com> <4BAB4253.8090605@gmail.com> <1be78d221003250448v54b39b9fob81b1eb7364fbf73@mail.gmail.com> Message-ID: <4BAB4FD6.80007@trueblade.com> Filip Gruszczy?ski wrote: >> From that spec, a straightforward API falls out: >> >> def format_mapping(self, kwds): >> # Method body actually written in C, so it can >> # easily invoke the internal formatting operation >> return do_string_format(self, NULL, kwds) > > Thanks a lot for the advice, I'll provide according patch hopefully in > the few days. I think this is basically my patch, with the method renamed, and needing tests. From mal at egenix.com Thu Mar 25 08:59:49 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 25 Mar 2010 08:59:49 +0100 Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk: Doc/c-api/capsule.rst Doc/c-api/cobject.rst Doc/c-api/concrete.rst Doc/data/refcounts.dat Doc/extending/extending.rst Include/Python.h Include/cStringIO.h Include/cobject.h Include/datetime.h Include/py_curses.h Include/pycapsule.h Include/pyexpat.h Include/ucnhash.h Lib/test/test_sys.py Makefile.pre.in Misc/NEWS Modules/_ctypes/callproc.c Modules/_ctypes/cfield.c Modules/_ctypes/ctypes.h Modules/_cursesmodule.c Modules/_elementtree.c Modules/_testcapimodule.c Modules/cStringIO.c Modules/cjkcodecs/cjkcodecs.h Modules/cjkcodecs/multibytecodec.c Modules/cjkcodecs/multibytecodec.h Modules/datetimemodule.c Modules/pyexpat.c Modules/socketmodule.c Modules/socketmodule.h Modules/unicodedata.c Objects/capsule.c Objects/object.c Objects/unicodeobject.c PC/VS7.1/pythoncore.vcproj PC/VS8.0/pythoncore.vcproj PC/os2emx/python27.def PC/os2vacpp/python.def Python/compile.c Python/getargs.c In-Reply-To: <20100325005454.B1AC5FCCD@mail.python.org> References: <20100325005454.B1AC5FCCD@mail.python.org> Message-ID: <4BAB17F5.1000305@egenix.com> larry.hastings wrote: > Author: larry.hastings > Date: Thu Mar 25 01:54:54 2010 > New Revision: 79397 > > Log: > Backported PyCapsule from 3.1, and converted most uses of > CObject to PyCapsule. Backporting PyCapsule is fine, but the changes you made to all those PyCObject uses does not look backwards compatible. The C APIs exposed by the modules (e.g. the datetime module) are used in lots of 3rd party extension modules and changing them from PyCObject to PyCapsule is a major change in the module API. > Added: > python/trunk/Doc/c-api/capsule.rst > python/trunk/Include/pycapsule.h > python/trunk/Objects/capsule.c > Modified: > python/trunk/Doc/c-api/cobject.rst > python/trunk/Doc/c-api/concrete.rst > python/trunk/Doc/data/refcounts.dat > python/trunk/Doc/extending/extending.rst > python/trunk/Include/Python.h > python/trunk/Include/cStringIO.h > python/trunk/Include/cobject.h > python/trunk/Include/datetime.h > python/trunk/Include/py_curses.h > python/trunk/Include/pyexpat.h > python/trunk/Include/ucnhash.h > python/trunk/Lib/test/test_sys.py > python/trunk/Makefile.pre.in > python/trunk/Misc/NEWS > python/trunk/Modules/_ctypes/callproc.c > python/trunk/Modules/_ctypes/cfield.c > python/trunk/Modules/_ctypes/ctypes.h > python/trunk/Modules/_cursesmodule.c > python/trunk/Modules/_elementtree.c > python/trunk/Modules/_testcapimodule.c > python/trunk/Modules/cStringIO.c > python/trunk/Modules/cjkcodecs/cjkcodecs.h > python/trunk/Modules/cjkcodecs/multibytecodec.c > python/trunk/Modules/cjkcodecs/multibytecodec.h > python/trunk/Modules/datetimemodule.c > python/trunk/Modules/pyexpat.c > python/trunk/Modules/socketmodule.c > python/trunk/Modules/socketmodule.h > python/trunk/Modules/unicodedata.c > python/trunk/Objects/object.c > python/trunk/Objects/unicodeobject.c > python/trunk/PC/VS7.1/pythoncore.vcproj > python/trunk/PC/VS8.0/pythoncore.vcproj > python/trunk/PC/os2emx/python27.def > python/trunk/PC/os2vacpp/python.def > python/trunk/Python/compile.c > python/trunk/Python/getargs.c -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 25 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Thu Mar 25 13:10:27 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 25 Mar 2010 22:10:27 +1000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <5c6f2a5d1003250455o5f1e2d29ld18c9893d11d3e1@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <5c6f2a5d1003250455o5f1e2d29ld18c9893d11d3e1@mail.gmail.com> Message-ID: <4BAB52B3.2010702@gmail.com> Mark Dickinson wrote: > +0.2 from me. I could happily live with this change; but could also > equally live with the existing weirdness. > > It's still a little odd for an immutable type to care about object > identity, but I guess oddness comes with the floating-point territory. > :) The trick for me came in thinking of NaN as a set of values rather than a single value - at that point, the different id values just reflect the multitude of members of that set. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From dickinsm at gmail.com Thu Mar 25 13:24:14 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 25 Mar 2010 12:24:14 +0000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BAB4764.4000500@gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> Message-ID: <5c6f2a5d1003250524i2f5414cdta7a8d9defa5c3aad@mail.gmail.com> On Thu, Mar 25, 2010 at 11:22 AM, Nick Coghlan wrote: > So, I'm specifically putting that proposal on the table for both float > and Decimal NaNs in Python: > > ?"Not a Number" is not a single floating point value. Instead each > ?instance is a distinct value representing the precise conditions that > ?created it. Thus, two "NaN" values x and y will compare equal iff they > ?are the exact same NaN object (i.e. "if isnan(x) then x == y iff > ?x is y". I'd also suggest that the language make no guarantees about whether two distinct calls to float('nan') or Decimal('nan') (or any other function call returning a nan) return identical values or not, but leave implementations free to do what's convenient or efficient. For example, with the current decimal module: Decimal('nan') returns a new nan each time, but Decimal(-1).sqrt() always returns the same nan object (assuming that InvalidOperation isn't trapped). I think it's fine to regard this as an implementation detail. Python 2.6.2 (r262:71600, Aug 26 2009, 09:40:44) [GCC 4.2.1 (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from decimal import * >>> getcontext().traps[InvalidOperation] = 0 >>> x, y = Decimal('nan'), Decimal('nan') >>> id(x), id(y) (47309953516000, 47309930620880) >>> x, y = Decimal(-1).sqrt(), Decimal(-1).sqrt() >>> id(x), id(y) (9922272, 9922272) Mark From jcea at jcea.es Thu Mar 25 13:36:28 2010 From: jcea at jcea.es (Jesus Cea) Date: Thu, 25 Mar 2010 13:36:28 +0100 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> Message-ID: <4BAB58CC.9070004@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/25/2010 07:54 AM, Georg Brandl wrote: >>>>>> float('nan') in [float('nan')] >>> False >> >> Sure, but just think of it as having two different nans there. (You >> could imagine thinking of the id of the nan as part of the payload.) > > That's interesting. Thinking of each value created by float('nan') as > a different nan makes sense to my naive mind, and it also explains > nicely the behavior present right now. Each nan comes from a different > operation and therefore is a "different" non-number. Infinites are "not equal" for a good reason, for example. 1/0 and 2/0 are both infinites, but one is "greater" than the other. Or (1/0)^(1/0), an infinite infinitelly "bigger". - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS6tYzJlgi5GaxT1NAQL7kgP+LqjzNKOkSOZ+gTvgKfUTrF1poNP1VMC4 1LTkCcpFQYUoc4d8kk6lmzN7RdBesidbnVC2SApKTdNTAfbKMB3hjkTIzoxbx9wf sLb5IUSqhtc+xJ+JQFepQwA7YLa64AVI23/wZcJCkqCBIg6S5DuGxhWErr3TXVgF GqcZjhvD2lA= =AVOO -----END PGP SIGNATURE----- From jcea at jcea.es Thu Mar 25 13:39:25 2010 From: jcea at jcea.es (Jesus Cea) Date: Thu, 25 Mar 2010 13:39:25 +0100 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BAB4764.4000500@gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> Message-ID: <4BAB597D.1060508@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/25/2010 12:22 PM, Nick Coghlan wrote: > "Not a Number" is not a single floating point value. Instead each > instance is a distinct value representing the precise conditions that > created it. Thus, two "NaN" values x and y will compare equal iff they > are the exact same NaN object (i.e. "if isnan(x) then x == y iff > x is y". > > As stated above, such a change would allow us to restore reflexivity > (eliminating a bunch of weirdness) while still honouring the idea of NaN > being a set of values rather than a single value. Sounds good. But IEEE 754 was created by pretty clever guys and sure they had a reason for define things in the way they are. Probably we are missing something. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS6tZfZlgi5GaxT1NAQJZtQP/bxi6l5TaiiOzv+no6cLaVXbkWXb9v6OL jmejzrlAosXzzd/4CuinN2mdFs7bd9Y3O9gHoQ2nUfbfWQc4SwxpxjK67j10PODJ MMz7wXgz075A8S7gUlpwWznByU2VfAys6ZVxZCv/uogW9SXIHqEBC/sXwWN5Hwvn uHImzIL4bfs= =RI3T -----END PGP SIGNATURE----- From eric at trueblade.com Thu Mar 25 13:57:00 2010 From: eric at trueblade.com (Eric Smith) Date: Thu, 25 Mar 2010 08:57:00 -0400 Subject: [Python-Dev] Issue 6081: format string using mapping rather than kwargs In-Reply-To: <4BAB4253.8090605@gmail.com> References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com> <4BAB4253.8090605@gmail.com> Message-ID: <4BAB5D9C.1010001@trueblade.com> Nick Coghlan wrote: > Filip Gruszczy?ski wrote: >> I would appreciate any advice on this topic, even if this ticket would >> be dismissed altogether, as I would like to learn as much as possible >> on practices on developing Python. > > Raymond's use case is valid, but the currently proposed method name is > way too long to be usable and the use of *args is a definite misfeature. > > I suggest approaching the issue with the exact spec needed to cover the > problem Raymond described: > > Add a method to strings that allows a format string to > be applied to an existing mapping without implicitly > converting the mapping to a dict first. It occurs to me that Raymond's use case could be satisfied using existing Python, by slightly changing the format string. After all, str.format() supports mapping lookup already: $ ./python.exe Python 2.6.5+ (release26-maint:79421, Mar 25 2010, 08:51:39) [GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> class Default(dict): ... def __missing__(self, key): ... return key ... >>> s = '{m[name]} was born in {m[country]}' >>> s.format(m=Default(name='Guido')) 'Guido was born in country' >>> Considering that, maybe the best thing is to do nothing. I'll update the issue with this note. -- Eric. From dickinsm at gmail.com Thu Mar 25 14:17:44 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 25 Mar 2010 13:17:44 +0000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BAB58CC.9070004@jcea.es> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <4BAB58CC.9070004@jcea.es> Message-ID: <5c6f2a5d1003250617n57008aa8p9fa0ccccaa21eaa0@mail.gmail.com> On Thu, Mar 25, 2010 at 12:36 PM, Jesus Cea wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 03/25/2010 07:54 AM, Georg Brandl wrote: >>>>>>> float('nan') in [float('nan')] >>>> False >>> >>> Sure, but just think of it as having two different nans there. ?(You >>> could imagine thinking of the id of the nan as part of the payload.) >> >> That's interesting. ?Thinking of each value created by float('nan') as >> a different nan makes sense to my naive mind, and it also explains >> nicely the behavior present right now. ?Each nan comes from a different >> operation and therefore is a "different" non-number. > > Infinites are "not equal" for a good reason, for example. Well, that depends on your mathematical model. The underlying mathematical model for IEEE 754 floating-point is the doubly extended real line: that is, the set of all real numbers augmented with two extra elements "infinity" and "-infinity", with the obvious total order. This is made explicit in section 3.2 of the standard: "The mathematical structure underpinning the arithmetic in this standard is the extended reals, that is, the set of real numbers together with positive and negative infinity." This is the same model that one typically uses in a first course in calculus when studying limits of functions; it's an appropriate model for dealing with computer approximations to real numbers and continuous functions. So the model has precisely two infinities, and 1/0, 2/0 and (1/0)**(1/0) all give the same infinity. The decision to make 1/0 "infinity" rather than "-infinity" is admittedly a little arbitrary. For floating-point (but not for calculus!), it makes sense in the light of the decision to have both positive and negative floating-point zeros; 1/(-0) is -infinity, of course. Other models of "reals + one or more infinities" are possible, of course, but they're not relevant to IEEE 754 floating point. There's a case for using a floating-point model with a single infinity, especially for those who care more about algebraic functions (polynomials, rational functions) than transcendental ones; however, IEEE 754 doesn't make provisions for this. Mark From dickinsm at gmail.com Thu Mar 25 14:25:02 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 25 Mar 2010 13:25:02 +0000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BAB597D.1060508@jcea.es> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> Message-ID: <5c6f2a5d1003250625o40f31568sa795e332866c1cae@mail.gmail.com> On Thu, Mar 25, 2010 at 12:39 PM, Jesus Cea wrote: > > But IEEE 754 was created by pretty clever guys and sure they had a > reason for define things in the way they are. Probably we are missing > something. Well, if we are, then nobody seems to know what! See the Bertrand Meyer blog post that was linked to up-thread. Mark From ncoghlan at gmail.com Thu Mar 25 15:08:48 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 26 Mar 2010 00:08:48 +1000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BAB597D.1060508@jcea.es> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> Message-ID: <4BAB6E70.3050008@gmail.com> Jesus Cea wrote: > But IEEE 754 was created by pretty clever guys and sure they had a > reason for define things in the way they are. Probably we are missing > something. Yes, this is where their "implementable in a hardware circuit" focus comes in. They were primarily thinking of a floating point representation where the 32/64 bits are *it* - you can't have "multiple NaNs" because you don't have the bits available to describe them. We don't have that limitation - by bring id() into play for NaN equality tests we have a lot more bits to play with (effectively adding an extra 32 or 64 bits to the floating point value just to say "which NaN is this one?"). Doubling the size of your storage just to have multiple kinds of NaN would be insane, but in our case the extra storage is already in use - we would just be applying an additional interpretation to it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Mar 25 15:16:30 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 26 Mar 2010 00:16:30 +1000 Subject: [Python-Dev] Issue 6081: format string using mapping rather than kwargs In-Reply-To: <4BAB5D9C.1010001@trueblade.com> References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com> <4BAB4253.8090605@gmail.com> <4BAB5D9C.1010001@trueblade.com> Message-ID: <4BAB703E.10701@gmail.com> Eric Smith wrote: > Considering that, maybe the best thing is to do nothing. I'll update the > issue with this note. I agree this slightly weakens the case for change, but it's not really the same thing. Adding a "format_mapping" method allows an arbitrary mapping to be used with any keyword-based format string. Crafting a special format string in order to use an arbitrary mapping means that the same format string can no longer be used with an ordinary keyword based call - you have to do "format(m=dict(name=y, country=z))" instead. Moving the decision of "how am I going to be called" to the time of writing the format string is a bit odd. On the other hand, the specially crafted format string does have the virtue of travelling far more easily through any APIs that wrap the basic format() method (since it doesn't need special treatment to make its way through the wrapping code). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From steve at pearwood.info Thu Mar 25 15:19:09 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 26 Mar 2010 01:19:09 +1100 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BAB58CC.9070004@jcea.es> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB58CC.9070004@jcea.es> Message-ID: <201003260119.10967.steve@pearwood.info> On Thu, 25 Mar 2010 11:36:28 pm Jesus Cea wrote: > Infinites are "not equal" for a good reason, for example. > > 1/0 and 2/0 are both infinites, but one is "greater" than the other. > Or (1/0)^(1/0), an infinite infinitelly "bigger". I think you're mistaken. In Python 3.1: >>> x = float('inf') >>> y = float('inf') + 1 >>> x == y True >>> x is y False In cardinal arithmetic, there are an infinity of different sized infinities, but in ordinal arithmetic there are only two: +inf and -inf, corresponding to the infinities on the real number line. (I hope that I'm not over-simplifying -- it's been more than a decade since I've needed to care about this.) But in any case, the IEEE standard doesn't deal with cardinals: it only uses two signed infinities. -- Steven D'Aprano From curt at hagenlocher.org Thu Mar 25 15:19:24 2010 From: curt at hagenlocher.org (Curt Hagenlocher) Date: Thu, 25 Mar 2010 07:19:24 -0700 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BAB6E70.3050008@gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> <4BAB6E70.3050008@gmail.com> Message-ID: On Thu, Mar 25, 2010 at 7:08 AM, Nick Coghlan wrote: > Jesus Cea wrote: > > But IEEE 754 was created by pretty clever guys and sure they had a > > reason for define things in the way they are. Probably we are missing > > something. > > Yes, this is where their "implementable in a hardware circuit" focus > comes in. They were primarily thinking of a floating point > representation where the 32/64 bits are *it* - you can't have "multiple > NaNs" because you don't have the bits available to describe them. > Wait, what? I haven't been paying much attention, but this is backwards. There are multiple representations of NaN in the IEEE encoding; that's actually part of the problem with saying that NaN = NaN or NaN != NaN. If you want to ignore the "payload" in the NaN, then you're not just comparing bits any more. -- Curt Hagenlocher curt at hagenlocher.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From dickinsm at gmail.com Thu Mar 25 15:22:33 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 25 Mar 2010 14:22:33 +0000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BAB6E70.3050008@gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> <4BAB6E70.3050008@gmail.com> Message-ID: <5c6f2a5d1003250722g28b130edxda76a1f8b6f9f0dc@mail.gmail.com> On Thu, Mar 25, 2010 at 2:08 PM, Nick Coghlan wrote: > Jesus Cea wrote: >> But IEEE 754 was created by pretty clever guys and sure they had a >> reason for define things in the way they are. Probably we are missing >> something. > > Yes, this is where their "implementable in a hardware circuit" focus > comes in. They were primarily thinking of a floating point > representation where the 32/64 bits are *it* - you can't have "multiple > NaNs" because you don't have the bits available to describe them. I'm not so sure about this: standard 64-bit binary IEEE 754 doubles allow for 2**53-2 different nans (2**52-2 signaling nans, 2**52 quiet nans): anything with bit pattern (msb to lsb) x1111111 1111xxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx is an infinity or a nan, and there are only 2 infinities. Mark From solipsis at pitrou.net Thu Mar 25 15:26:05 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 25 Mar 2010 14:26:05 +0000 (UTC) Subject: [Python-Dev] Why is nan != nan? References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> <4BAB6E70.3050008@gmail.com> Message-ID: Le Thu, 25 Mar 2010 07:19:24 -0700, Curt Hagenlocher a ?crit?: > On Thu, Mar 25, 2010 at 7:08 AM, Nick Coghlan > wrote: > >> Jesus Cea wrote: >> > But IEEE 754 was created by pretty clever guys and sure they had a >> > reason for define things in the way they are. Probably we are missing >> > something. >> >> Yes, this is where their "implementable in a hardware circuit" focus >> comes in. They were primarily thinking of a floating point >> representation where the 32/64 bits are *it* - you can't have "multiple >> NaNs" because you don't have the bits available to describe them. >> > Wait, what? I haven't been paying much attention, but this is backwards. > There are multiple representations of NaN in the IEEE encoding; that's > actually part of the problem with saying that NaN = NaN or NaN != NaN. > If you want to ignore the "payload" in the NaN, then you're not just > comparing bits any more. This sounds a bit sophistic, if the (Python) user doesn't have access to the payload anyway. Antoine. From dickinsm at gmail.com Thu Mar 25 15:42:14 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 25 Mar 2010 14:42:14 +0000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> <4BAB6E70.3050008@gmail.com> Message-ID: <5c6f2a5d1003250742g1f945514t8017a897d6f6223f@mail.gmail.com> On Thu, Mar 25, 2010 at 2:26 PM, Antoine Pitrou wrote: > Le Thu, 25 Mar 2010 07:19:24 -0700, Curt Hagenlocher a ?crit?: >> Wait, what? I haven't been paying much attention, but this is backwards. >> There are multiple representations of NaN in the IEEE encoding; that's >> actually part of the problem with saying that NaN = NaN or NaN != NaN. >> If you want to ignore the "payload" in the NaN, then you're not just >> comparing bits any more. > > This sounds a bit sophistic, if the (Python) user doesn't have access to > the payload anyway. Well, you can get at the payload using the struct module, if you care enough. But yes, it's true that Python doesn't take much care with the payload: e.g., ideally, an operation on a nan (3.0 + nan, sqrt(nan), ...) should return exactly the same nan, to make sure that information in the payload is preserved. Python doesn't bother, for floats (though it does for decimal). Mark From pierrebai at hotmail.com Thu Mar 25 15:53:59 2010 From: pierrebai at hotmail.com (Pierre B.) Date: Thu, 25 Mar 2010 14:53:59 +0000 (UTC) Subject: [Python-Dev] Why is nan != nan? References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> <5c6f2a5d1003250625o40f31568sa795e332866c1cae@mail.gmail.com> Message-ID: Mark Dickinson gmail.com> writes: > > On Thu, Mar 25, 2010 at 12:39 PM, Jesus Cea jcea.es> wrote: > > > > But IEEE 754 was created by pretty clever guys and sure they had a > > reason for define things in the way they are. Probably we are missing > > something. > > Well, if we are, then nobody seems to know what! See the Bertrand > Meyer blog post that was linked to up-thread. The missing part, IMO, is that allowing a given NaN value to compare equal to itself only pushes the problem up one level. Any single operation yielding a NaN will still be unequal to itself. That is, under what is being proposed, with a function func() returning the same result of some calculation: x = func() s1 = (x) print x in s1 print func() in s1 This would print True and False, even though func() is perfoming the same calculation and thus logically returning the same NaN. I think the IEEE NaN represent the fact that you have a number of an undefined set, but it doesn't specify which. The only way out, IMO, is to make *all* NaN comparisons yield False, but identity yield true. No interning necessary. At most, you could make the identity function return False for the different types of NaN. From dickinsm at gmail.com Thu Mar 25 15:54:55 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 25 Mar 2010 14:54:55 +0000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <5c6f2a5d1003250742g1f945514t8017a897d6f6223f@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> <4BAB6E70.3050008@gmail.com> <5c6f2a5d1003250742g1f945514t8017a897d6f6223f@mail.gmail.com> Message-ID: <5c6f2a5d1003250754t7d7ab207i346825c79951195a@mail.gmail.com> On Thu, Mar 25, 2010 at 2:42 PM, Mark Dickinson wrote: > On Thu, Mar 25, 2010 at 2:26 PM, Antoine Pitrou wrote: >> This sounds a bit sophistic, if the (Python) user doesn't have access to >> the payload anyway. > > Well, you can get at the payload using the struct module, if you care > enough. ?But yes, it's true that Python doesn't take much care with > the payload: ?e.g., ideally, an operation on a nan (3.0 + nan, > sqrt(nan), ...) should return exactly the same nan, to make sure that > information in the payload is preserved. ?Python doesn't bother, for > floats (though it does for decimal). Hmm. I take it back. I was being confused by the fact that sqrt(nan) returns a nan with a new identity; but it does apparently preserve the payload. An example: >>> from struct import pack, unpack >>> from math import sqrt >>> x = unpack('>> y = sqrt(x) >>> bin(unpack('>> bin(unpack(' References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com> <4BAB4253.8090605@gmail.com> <4BAB5D9C.1010001@trueblade.com> <4BAB703E.10701@gmail.com> Message-ID: <4BAB78F6.8080905@trueblade.com> Nick Coghlan wrote: > Moving the decision of "how am I going to be called" to the time of > writing the format string is a bit odd. > > On the other hand, the specially crafted format string does have the > virtue of travelling far more easily through any APIs that wrap the > basic format() method (since it doesn't need special treatment to make > its way through the wrapping code). Completely agree on all points. Now we're just left with "is it worth expanding the str api for this?". I don't feel strongly either way. -- Eric. From curt at hagenlocher.org Thu Mar 25 16:01:54 2010 From: curt at hagenlocher.org (Curt Hagenlocher) Date: Thu, 25 Mar 2010 08:01:54 -0700 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <5c6f2a5d1003250754t7d7ab207i346825c79951195a@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> <4BAB6E70.3050008@gmail.com> <5c6f2a5d1003250742g1f945514t8017a897d6f6223f@mail.gmail.com> <5c6f2a5d1003250754t7d7ab207i346825c79951195a@mail.gmail.com> Message-ID: On Thu, Mar 25, 2010 at 7:54 AM, Mark Dickinson wrote: > > Hmm. I take it back. ?I was being confused by the fact that sqrt(nan) > returns a nan with a new identity; ?but it does apparently preserve > the payload. ?An example: I played with this some a few months ago, and both the FPU and the C libraries I tested will preserve the payload. I imagine Python just inherits their behavior. -- Curt Hagenlocher curt at hagenlocher.org From dickinsm at gmail.com Thu Mar 25 16:05:32 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 25 Mar 2010 15:05:32 +0000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> <4BAB6E70.3050008@gmail.com> <5c6f2a5d1003250742g1f945514t8017a897d6f6223f@mail.gmail.com> <5c6f2a5d1003250754t7d7ab207i346825c79951195a@mail.gmail.com> Message-ID: <5c6f2a5d1003250805q455b2445h41479708e69d6463@mail.gmail.com> On Thu, Mar 25, 2010 at 3:01 PM, Curt Hagenlocher wrote: > On Thu, Mar 25, 2010 at 7:54 AM, Mark Dickinson wrote: >> >> Hmm. I take it back. ?I was being confused by the fact that sqrt(nan) >> returns a nan with a new identity; ?but it does apparently preserve >> the payload. ?An example: > > I played with this some a few months ago, and both the FPU and the C > libraries I tested will preserve the payload. I imagine Python just > inherits their behavior. Pretty much, yes. I think we've also taken care to preserve payloads in functions that have been added to the math library as well (e.g., the gamma function). Not that that's particularly hard: it's just a matter of making sure to do "if (isnan(x)) return x;" rather than "if (isnan(x)) return standard_python_nan;". If that's not true, then there's a minor bug to be corrected. Mark From ncoghlan at gmail.com Thu Mar 25 16:05:36 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 26 Mar 2010 01:05:36 +1000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <5c6f2a5d1003250722g28b130edxda76a1f8b6f9f0dc@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> <4BAB6E70.3050008@gmail.com> <5c6f2a5d1003250722g28b130edxda76a1f8b6f9f0dc@mail.gmail.com> Message-ID: <4BAB7BC0.8020306@gmail.com> Mark Dickinson wrote: > On Thu, Mar 25, 2010 at 2:08 PM, Nick Coghlan wrote: >> Jesus Cea wrote: >>> But IEEE 754 was created by pretty clever guys and sure they had a >>> reason for define things in the way they are. Probably we are missing >>> something. >> Yes, this is where their "implementable in a hardware circuit" focus >> comes in. They were primarily thinking of a floating point >> representation where the 32/64 bits are *it* - you can't have "multiple >> NaNs" because you don't have the bits available to describe them. > > I'm not so sure about this: standard 64-bit binary IEEE 754 doubles > allow for 2**53-2 different nans (2**52-2 signaling nans, 2**52 quiet > nans): anything with bit pattern (msb to lsb) > > x1111111 1111xxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx > > is an infinity or a nan, and there are only 2 infinities. I stand corrected :) It still seems to me that the problems mostly arise when we're trying to get floats and Decimals to behave like Python *objects* (i.e. with reflexive equality) rather than like IEEE defined numbers. It's an extra element that isn't part of the problem the numeric standards are trying to solve. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From cournape at gmail.com Thu Mar 25 16:07:10 2010 From: cournape at gmail.com (David Cournapeau) Date: Fri, 26 Mar 2010 00:07:10 +0900 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BAB597D.1060508@jcea.es> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> Message-ID: <5b8d13221003250807h21dcd79fj37f65cc81af8e3b7@mail.gmail.com> On Thu, Mar 25, 2010 at 9:39 PM, Jesus Cea wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 03/25/2010 12:22 PM, Nick Coghlan wrote: >> ? "Not a Number" is not a single floating point value. Instead each >> ? instance is a distinct value representing the precise conditions that >> ? created it. Thus, two "NaN" values x and y will compare equal iff they >> ? are the exact same NaN object (i.e. "if isnan(x) then x == y iff >> ? x is y". >> >> As stated above, such a change would allow us to restore reflexivity >> (eliminating a bunch of weirdness) while still honouring the idea of NaN >> being a set of values rather than a single value. > > Sounds good. > > But IEEE 754 was created by pretty clever guys and sure they had a > reason for define things in the way they are. Probably we are missing > something. Yes, indeed. I don't claim having a deep understanding myself, but up to now, everytime I thought something in IEE 754 was weird, it ended up being for good reasons. I think the fundamental missing point in this discussion about Nan is exception handling: a lot of NaN quircky behavior becomes much more natural once you take into account which operations are invalid under which condition. Unless I am mistaken, python itself does not support for FPU exception handling. For example, the reason why x != x for x Nan is because != (and ==) are about the only operations where you can have NaN as operands without risking raising an exception, and support for creating and detecting NaN in languages have been coming only quite lately (e.g. C99). Concerning the lack of rationale: a relatively short reference concerned about FPU exception and NaN handling is from Kahan himself http://www.eecs.berkeley.edu/~wkahan/ieee754status/ieee754.ps David From ncoghlan at gmail.com Thu Mar 25 16:09:00 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 26 Mar 2010 01:09:00 +1000 Subject: [Python-Dev] Issue 6081: format string using mapping rather than kwargs In-Reply-To: <4BAB78F6.8080905@trueblade.com> References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com> <4BAB4253.8090605@gmail.com> <4BAB5D9C.1010001@trueblade.com> <4BAB703E.10701@gmail.com> <4BAB78F6.8080905@trueblade.com> Message-ID: <4BAB7C8C.6020004@gmail.com> Eric Smith wrote: > Nick Coghlan wrote: >> Moving the decision of "how am I going to be called" to the time of >> writing the format string is a bit odd. >> >> On the other hand, the specially crafted format string does have the >> virtue of travelling far more easily through any APIs that wrap the >> basic format() method (since it doesn't need special treatment to make >> its way through the wrapping code). > > Completely agree on all points. Now we're just left with "is it worth > expanding the str api for this?". I don't feel strongly either way. For something as core as the string API, we better feel darn strongly about it before we mess with it :) I'm inclined to leave it alone unless/until Raymond or somebody else steps up to really champion it. Cheers, Nick. P.S. There's also the language moratorium to consider - since this idea affect the methods of a builtin type, I believe the moratorium applies. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From dickinsm at gmail.com Thu Mar 25 16:13:04 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 25 Mar 2010 15:13:04 +0000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BAB7BC0.8020306@gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> <4BAB6E70.3050008@gmail.com> <5c6f2a5d1003250722g28b130edxda76a1f8b6f9f0dc@mail.gmail.com> <4BAB7BC0.8020306@gmail.com> Message-ID: <5c6f2a5d1003250813h15e5610bseb4a9b3b541585ff@mail.gmail.com> On Thu, Mar 25, 2010 at 3:05 PM, Nick Coghlan wrote: > Mark Dickinson wrote: >> On Thu, Mar 25, 2010 at 2:08 PM, Nick Coghlan wrote: >>> Jesus Cea wrote: >>>> But IEEE 754 was created by pretty clever guys and sure they had a >>>> reason for define things in the way they are. Probably we are missing >>>> something. >>> Yes, this is where their "implementable in a hardware circuit" focus >>> comes in. They were primarily thinking of a floating point >>> representation where the 32/64 bits are *it* - you can't have "multiple >>> NaNs" because you don't have the bits available to describe them. >> >> I'm not so sure about this: ?standard 64-bit binary IEEE 754 doubles >> allow for 2**53-2 different nans (2**52-2 signaling nans, 2**52 quiet >> nans): ?anything with bit pattern (msb to lsb) >> >> x1111111 1111xxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx >> >> is an infinity or a nan, and there are only 2 infinities. > > I stand corrected :) > > It still seems to me that the problems mostly arise when we're trying to > get floats and Decimals to behave like Python *objects* (i.e. with > reflexive equality) rather than like IEEE defined numbers. > > It's an extra element that isn't part of the problem the numeric > standards are trying to solve. Agreed. We don't have to be "missing something"; rather, the IEEE folks (quite understandably) almost certainly didn't anticipate this kind of usage. So I'll concede that it's reasonable to consider deviating from the standard in the light of this. Mark From solipsis at pitrou.net Thu Mar 25 16:24:38 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 25 Mar 2010 15:24:38 +0000 (UTC) Subject: [Python-Dev] Issue 6081: format string using mapping rather than kwargs References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com> <4BAB4253.8090605@gmail.com> <4BAB5D9C.1010001@trueblade.com> <4BAB703E.10701@gmail.com> <4BAB78F6.8080905@trueblade.com> <4BAB7C8C.6020004@gmail.com> Message-ID: Nick Coghlan gmail.com> writes: > > For something as core as the string API, we better feel darn strongly > about it before we mess with it :) Having two very similar methods with different names and subtly different APIs sounds bad IMHO. Also, the use case doesn't look very strong to me. > P.S. There's also the language moratorium to consider - since this idea > affect the methods of a builtin type, I believe the moratorium applies. So do I. Regards Antoine. From eric at trueblade.com Thu Mar 25 16:55:54 2010 From: eric at trueblade.com (Eric Smith) Date: Thu, 25 Mar 2010 11:55:54 -0400 Subject: [Python-Dev] Issue 6081: format string using mapping rather than kwargs In-Reply-To: <4BAB7C8C.6020004@gmail.com> References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com> <4BAB4253.8090605@gmail.com> <4BAB5D9C.1010001@trueblade.com> <4BAB703E.10701@gmail.com> <4BAB78F6.8080905@trueblade.com> <4BAB7C8C.6020004@gmail.com> Message-ID: <4BAB878A.3080106@trueblade.com> Nick Coghlan wrote: >> Completely agree on all points. Now we're just left with "is it worth >> expanding the str api for this?". I don't feel strongly either way. > > For something as core as the string API, we better feel darn strongly > about it before we mess with it :) > > I'm inclined to leave it alone unless/until Raymond or somebody else > steps up to really champion it. I'm okay with that. > P.S. There's also the language moratorium to consider - since this idea > affect the methods of a builtin type, I believe the moratorium applies. I don't think there's any effort to get this in before the moratorium expires. Eric. From raymond.hettinger at gmail.com Thu Mar 25 17:03:04 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Thu, 25 Mar 2010 09:03:04 -0700 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BAB4764.4000500@gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> Message-ID: On Mar 25, 2010, at 4:22 AM, Nick Coghlan wrote: > Mark Dickinson wrote: >> Here's an interesting recent blog post on this subject, from the >> creator of Eiffel: >> >> http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/ > > Interesting. So the natural tweak that would arise from that perspective > is for us to restore reflexivity by declaring that any given NaN is > equal to *itself* but not to any other NaN (even one with the same payload). > > With NaN (in general) not being interned, that would actually fit the > idea of a NaN implicitly carrying the operation that created the NaN as > part of its definition of equivalence. > > So, I'm specifically putting that proposal on the table for both float > and Decimal NaNs in Python: > > "Not a Number" is not a single floating point value. Instead each > instance is a distinct value representing the precise conditions that > created it. Thus, two "NaN" values x and y will compare equal iff they > are the exact same NaN object (i.e. "if isnan(x) then x == y iff > x is y". > > As stated above, such a change would allow us to restore reflexivity > (eliminating a bunch of weirdness) while still honouring the idea of NaN > being a set of values rather than a single value. > +1 Raymond From larry at hastings.org Thu Mar 25 18:38:54 2010 From: larry at hastings.org (Larry Hastings) Date: Thu, 25 Mar 2010 10:38:54 -0700 Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk: Doc/c-api/capsule.rst Doc/c-api/cobject.rst Doc/c-api/concrete.rst Doc/data/refcounts.dat Doc/extending/extending.rst Include/Python.h Include/cStringIO.h Include/cobject.h Include/datetime.h Include/py_curses.h Include/pycapsule.h Include/pyexpat.h Include/ucnhash.h Lib/test/test_sys.py Makefile.pre.in Misc/NEWS Modules/_ctypes/callproc.c Modules/_ctypes/cfield.c Modules/_ctypes/ctypes.h Modules/_cursesmodule.c Modules/_elementtree.c Modules/_testcapimodule.c Modules/cStringIO.c Modules/cjkcodecs/cjkcodecs.h Modules/cjkcodecs/multibytecodec.c Modules/cjkcodecs/multibytecodec.h Modules/datetimemodule.c Modules/pyexpat.c Modules/socketmodule.c Modules/socketmodule.h Modules/unicodedata.c Objects/capsule.c Objects/object.c Objects/unicodeobject.c PC/VS7.1/pythoncore.vcproj PC/VS8.0/pythoncore.vcproj PC/os2emx/python27.def PC/os2vacpp/python.def Python/compile.c Python/getargs.c In-Reply-To: <4BAB2882.6070508@egenix.com> References: <20100325005454.B1AC5FCCD@mail.python.org> <4BAB2882.6070508@egenix.com> Message-ID: <4BAB9FAE.1070206@hastings.org> M.-A. Lemburg wrote: > Backporting PyCapsule is fine, but the changes you made to all > those PyCObject uses does not look backwards compatible. > > The C APIs exposed by the modules (e.g. the datetime module) > are used in lots of 3rd party extension modules and changing > them from PyCObject to PyCapsule is a major change in the > module API. You're right, my changes aren't backwards compatible. I thought it was reasonable for four reasons: 1. The CObject API isn't safe. It's easy to crash Python 2.6 in just a few lines by mixing and matching CObjects. Switching Python to capsules prevents a class of exploits. I've included a script at the bottom of this message that demonstrates three such crashes. The script runs in Python 2 and 3, but 3.1 doesn't crash because it's using capsules. 2. As I just mentioned, Python 3.1 already uses capsules everywhere instead of CObjects. Since part of the purpose of Python 2.7 is to prepare developers for the to upgrade to 3.1, getting them to switch to capsules now is just one more way they are prepared. 3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we ever made a 2.8 I want to remove it completely. 4. When Python publishes an API using a CObject, it describes the thing the CObject points to in a header file. In nearly all cases that header file also provides a macro or inline function that does the importing work for you. I changed those to use capsules too. So if the third-party code uses the macro or inline function, all you need do is recompile it against 2.7 and it works fine. Sadly I know of one exception: pyexpat.expat_CAPI. The header file just describes the struct pointed to by the CObject, but callers I can suggest four ways to ameliorate the problem. First, we could do as Antoine Pitrou suggests on the bug (issue 7992): wherever the CObject used to be published as a module attribute to expose an API, we could provide both a CObject and a capsule; internally Python would only use the capsules. This would allow third-party libraries to run against 2.7 unchanged. The major problem with this is that third-party libraries would still be vulnerable to the mix-and-match CObject crash. A secondary, minor concern: obviously we'd store the CObject attribute with the existing name, and the capsule attribute would have to get some new name. But in Python 3.1, these attributes already expose a capsule. Therefore, people who convert to using the capsules now would have to convert again when moving to 3.1. Second, we could make CObject internally support unpacking capsules. If you gave a capsule to PyCObject_AsVoidPtr() it would unpack it and return the pointer within. (We could probably also map the capsule "context" to the CObject "desc", if any of the Python use cases needed it.) I wouldn't change anything else about CObjects; creating and using them would continue to work as normal. This would also allow third-party libraries to run against Python 2.7 unchanged. The only problem is that it's unsafe, as indeed allowing any use of PyCObject_AsVoidPtr() is unsafe. Third, I've been pondering writing a set of preprocessor macros, shipped in their own header file distributed independently of Python and released to the public domain, that would make it easy to use either CObjects or capsules depending on what version of Python you were compiling against. Obviously, using these macros would require a source code change in the third-party library. But these macros would make it a five-minute change. This could compliment the first or second approaches. Fourth, we could back out of the changes to published APIs and convert them back to CObjects. -1. Your thoughts? /larry/ ----- import sys def log(message): print(message) sys.stdout.flush() def crash1(): log("Running crash1...") try: import datetime import cStringIO cStringIO.cStringIO_CAPI = datetime.datetime_CAPI import cPickle s = cPickle.dumps([1, 2, 3]) except ImportError: # This test isn't translatable to Python 3. pass log("Survived crash1!") def crash2(): log("Running crash2...") try: import unicodedata import _socket _socket.CAPI = unicodedata.ucnhash_CAPI import ssl except AttributeError: # Congratulations, you didn't crash. pass log("Survived crash2!") def crash3(): log("Running crash3...") try: import unicodedata import _multibytecodec _multibytecodec.__create_codec(unicodedata.ucnhash_CAPI) except ValueError: # Congratulations, you didn't crash. pass log("Survived crash3!") import sys if len(sys.argv) > 1: if sys.argv[1] == '1': crash1() sys.exit(0) elif sys.argv[1] == '2': crash2() sys.exit(0) elif sys.argv[1] == '3': crash3() sys.exit(0) crash1() crash2() crash3() From jcea at jcea.es Thu Mar 25 18:39:09 2010 From: jcea at jcea.es (Jesus Cea) Date: Thu, 25 Mar 2010 18:39:09 +0100 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <201003260119.10967.steve@pearwood.info> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB58CC.9070004@jcea.es> <201003260119.10967.steve@pearwood.info> Message-ID: <4BAB9FBD.8010508@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/25/2010 03:19 PM, Steven D'Aprano wrote: > On Thu, 25 Mar 2010 11:36:28 pm Jesus Cea wrote: > >> Infinites are "not equal" for a good reason, for example. >> >> 1/0 and 2/0 are both infinites, but one is "greater" than the other. >> Or (1/0)^(1/0), an infinite infinitelly "bigger". > > > I think you're mistaken. In Python 3.1: I was refering to mathematical infinites and why inf!=inf is sensible and a natural consequence calculus and limits (mathematically). In any case, I am an engineer, not a mathematical or a language designer :). IANAL. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS6ufvJlgi5GaxT1NAQKrpQP+PbbNP6tNtPi480/ovki89DEDNqvKH4RU A/R6jLXwm7byF0RQ+3B9gUh8SfANQaTOeCOYufUJHXPDM3BHsmt2S1kcPoW5sUYe a38nUuD0sTyCV23h2QtzZpNGG7qNa6iHTMEc6vYgY3CdfCw+301xdqH0ZkmXV1B9 OIrj7ec7m+s= =rTUf -----END PGP SIGNATURE----- From jcea at jcea.es Thu Mar 25 18:47:54 2010 From: jcea at jcea.es (Jesus Cea) Date: Thu, 25 Mar 2010 18:47:54 +0100 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <5b8d13221003250807h21dcd79fj37f65cc81af8e3b7@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> <5b8d13221003250807h21dcd79fj37f65cc81af8e3b7@mail.gmail.com> Message-ID: <4BABA1CA.5050706@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/25/2010 04:07 PM, David Cournapeau wrote: > Yes, indeed. I don't claim having a deep understanding myself, but up > to now, everytime I thought something in IEE 754 was weird, it ended > up being for good reasons. I was wondering if we could bring the question to news:comp.arch newsgroup. They have the knowledge, and I know there are people from the IEEE 754 group lurking there. I only have read-only access, nevertheless. Another relevant group could be news:comp.arch.arithmetic, but I am not familiar with it. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS6uhyZlgi5GaxT1NAQIZJwP/bAx5vXBLMLI8f724Hf0OtfATpV4SFQ84 RaLHNBPPkE5+cdFxWIv6VmFThtYfKnjutmLNU1TJYFoDwgOvqigYO8hOTFnWlfML Sx5B3LFdtGSZAfSsd+rMF23wKpbpAy/TicE+B6zg+Qy1LFv1V+OVn/Y3xBPGxVW5 m4yAKWT5T4U= =mwXx -----END PGP SIGNATURE----- From mal at egenix.com Thu Mar 25 19:05:52 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 25 Mar 2010 19:05:52 +0100 Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk: Doc/c-api/capsule.rst Doc/c-api/cobject.rst Doc/c-api/concrete.rst Doc/data/refcounts.dat Doc/extending/extending.rst Include/Python.h Include/cStringIO.h Include/cobject.h Include/datetime.h Include/py_curses.h Include/pycapsule.h Include/pyexpat.h Include/ucnhash.h Lib/test/test_sys.py Makefile.pre.in Misc/NEWS Modules/_ctypes/callproc.c Modules/_ctypes/cfield.c Modules/_ctypes/ctypes.h Modules/_cursesmodule.c Modules/_elementtree.c Modules/_testcapimodule.c Modules/cStringIO.c Modules/cjkcodecs/cjkcodecs.h Modules/cjkcodecs/multibytecodec.c Modules/cjkcodecs/multibytecodec.h Modules/datetimemodule.c Modules/pyexpat.c Modules/socketmodule.c Modules/socketmodule.h Modules/unicodedata.c Objects/capsule.c Objects/object.c Objects/unicodeobject.c PC/VS7.1/pythoncore.vcproj PC/VS8.0/pythoncore.vcproj PC/os2emx/python27.def PC/os2vacpp/python.def Python/compile.c Python/getargs.c In-Reply-To: <4BAB9FAE.1070206@hastings.org> References: <20100325005454.B1AC5FCCD@mail.python.org> <4BAB2882.6070508@egenix.com> <4BAB9FAE.1070206@hastings.org> Message-ID: <4BABA600.8090103@egenix.com> Larry Hastings wrote: > > M.-A. Lemburg wrote: >> Backporting PyCapsule is fine, but the changes you made to all >> those PyCObject uses does not look backwards compatible. >> >> The C APIs exposed by the modules (e.g. the datetime module) >> are used in lots of 3rd party extension modules and changing >> them from PyCObject to PyCapsule is a major change in the >> module API. > > You're right, my changes aren't backwards compatible. I thought it was > reasonable for four reasons: > > 1. The CObject API isn't safe. It's easy to crash Python 2.6 in just a > few lines by mixing and matching CObjects. Switching Python to capsules > prevents a class of exploits. I've included a script at the bottom of > this message that demonstrates three such crashes. The script runs in > Python 2 and 3, but 3.1 doesn't crash because it's using capsules. > > 2. As I just mentioned, Python 3.1 already uses capsules everywhere > instead of CObjects. Since part of the purpose of Python 2.7 is to > prepare developers for the to upgrade to 3.1, getting them to switch to > capsules now is just one more way they are prepared. > > 3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we > ever made a 2.8 I want to remove it completely. > > 4. When Python publishes an API using a CObject, it describes the thing > the CObject points to in a header file. In nearly all cases that header > file also provides a macro or inline function that does the importing > work for you. I changed those to use capsules too. So if the > third-party code uses the macro or inline function, all you need do is > recompile it against 2.7 and it works fine. Sadly I know of one > exception: pyexpat.expat_CAPI. The header file just describes the > struct pointed to by the CObject, but callers > > > I can suggest four ways to ameliorate the problem. > > First, we could do as Antoine Pitrou suggests on the bug (issue 7992): > wherever the CObject used to be published as a module attribute to > expose an API, we could provide both a CObject and a capsule; internally > Python would only use the capsules. This would allow third-party > libraries to run against 2.7 unchanged. The major problem with this is > that third-party libraries would still be vulnerable to the > mix-and-match CObject crash. A secondary, minor concern: obviously we'd > store the CObject attribute with the existing name, and the capsule > attribute would have to get some new name. But in Python 3.1, these > attributes already expose a capsule. Therefore, people who convert to > using the capsules now would have to convert again when moving to 3.1. > > Second, we could make CObject internally support unpacking capsules. If > you gave a capsule to PyCObject_AsVoidPtr() it would unpack it and > return the pointer within. (We could probably also map the capsule > "context" to the CObject "desc", if any of the Python use cases needed > it.) I wouldn't change anything else about CObjects; creating and using > them would continue to work as normal. This would also allow > third-party libraries to run against Python 2.7 unchanged. The only > problem is that it's unsafe, as indeed allowing any use of > PyCObject_AsVoidPtr() is unsafe. > > Third, I've been pondering writing a set of preprocessor macros, shipped > in their own header file distributed independently of Python and > released to the public domain, that would make it easy to use either > CObjects or capsules depending on what version of Python you were > compiling against. Obviously, using these macros would require a source > code change in the third-party library. But these macros would make it > a five-minute change. This could compliment the first or second > approaches. > > Fourth, we could back out of the changes to published APIs and convert > them back to CObjects. -1. > > > Your thoughts? More later, have to run. For now, I'm with Antoine on this one: * adding support for PyCapsules to 2.7 is fine. * exposing the various module C APIs as PyCapsules in addition to the existing PyCObject C APIS is fines as well and indeed a good idea, since it makes porting 2.7 applications to 3.1 easier. * adding more C APIs using PyCapsules in Python 2.7 is fine as well, e.g. for the decimal module. * removing the PyCObject C APIs is no option in Python 2.7. Please remember that you're dealing with Python 2.7. Major backwards incompatible changes are not allowed in that branch and you're pretty much breaking all 3rd party modules that have just started using e.g the datetime module C API. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 25 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Thu Mar 25 19:15:28 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 25 Mar 2010 18:15:28 +0000 (UTC) Subject: [Python-Dev] [Python-checkins] PyCapsule backport References: <20100325005454.B1AC5FCCD@mail.python.org> <4BAB2882.6070508@egenix.com> <4BAB9FAE.1070206@hastings.org> Message-ID: Hello Larry, > You're right, my changes aren't backwards compatible. I thought it was > reasonable for four reasons: > > 1. The CObject API isn't safe. It's easy to crash Python 2.6 in just a > few lines by mixing and matching CObjects. Switching Python to capsules > prevents a class of exploits. I've included a script at the bottom of > this message that demonstrates three such crashes. The script runs in > Python 2 and 3, but 3.1 doesn't crash because it's using capsules. Ok, but this is IMHO not a good enough reason to break the API. No prior announcement (prior to 2.7) has been made that PyCObject APIs would be replaced by their PyCapsule equivalents. You must leave people time to adapt. > 2. As I just mentioned, Python 3.1 already uses capsules everywhere > instead of CObjects. Since part of the purpose of Python 2.7 is to > prepare developers for the to upgrade to 3.1, getting them to switch to > capsules now is just one more way they are prepared. Not /forcing them/ to switch, though. If you agree to put a bit more work into it, and make the PyCObject APIs coexist with the corresponding PyCapsule APIs in 2.7, then I think it would be fine. The point where we force people to switch their APIs is the 2.x -> 3.x transition. That's the whole point of breaking compatibility at one defined point. If we force them to switch when migrating to 2.7 instead, it becomes messy and quite unfriendly to them. > 3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we > ever made a 2.8 I want to remove it completely. I don't think there is a point in removing it in 2.8. If 2.8 ever exists, its purpose will be to serve users who don't /want/ to switch to 3.x yet, so breaking compatibility for them kind of ruins the purpose. > First, we could do as Antoine Pitrou suggests on the bug (issue 7992): > wherever the CObject used to be published as a module attribute to > expose an API, we could provide both a CObject and a capsule; internally > Python would only use the capsules. [...] > > Second, we could make CObject internally support unpacking capsules. If > you gave a capsule to PyCObject_AsVoidPtr() it would unpack it and > return the pointer within. [...] > > Third, I've been pondering writing a set of preprocessor macros, shipped > in their own header file distributed independently of Python and > released to the public domain, that would make it easy to use either > CObjects or capsules depending on what version of Python you were > compiling against. Obviously, using these macros would require a source > code change in the third-party library. But these macros would make it > a five-minute change. This could compliment the first or second approaches. > > Fourth, we could back out of the changes to published APIs and convert > them back to CObjects. -1. I think solution #1 would be the best one. #2 looks too complicated. #3 is not much friendlier than the current compatibility breakage. #4 is the necessary fallback if you don't want to implement #1. Regards Antoine. From larry at hastings.org Thu Mar 25 19:16:27 2010 From: larry at hastings.org (Larry Hastings) Date: Thu, 25 Mar 2010 11:16:27 -0700 Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk: Doc/c-api/capsule.rst Doc/c-api/cobject.rst Doc/c-api/concrete.rst Doc/data/refcounts.dat Doc/extending/extending.rst Include/Python.h Include/cStringIO.h Include/cobject.h Include/datetime.h Include/py_curses.h Include/pycapsule.h Include/pyexpat.h Include/ucnhash.h Lib/test/test_sys.py Makefile.pre.in Misc/NEWS Modules/_ctypes/callproc.c Modules/_ctypes/cfield.c Modules/_ctypes/ctypes.h Modules/_cursesmodule.c Modules/_elementtree.c Modules/_testcapimodule.c Modules/cStringIO.c Modules/cjkcodecs/cjkcodecs.h Modules/cjkcodecs/multibytecodec.c Modules/cjkcodecs/multibytecodec.h Modules/datetimemodule.c Modules/pyexpat.c Modules/socketmodule.c Modules/socketmodule.h Modules/unicodedata.c Objects/capsule.c Objects/object.c Objects/unicodeobject.c PC/VS7.1/pythoncore.vcproj PC/VS8.0/pythoncore.vcproj PC/os2emx/python27.def PC/os2vacpp/python.def Python/compile.c Python/getargs.c In-Reply-To: <4BABA600.8090103@egenix.com> References: <20100325005454.B1AC5FCCD@mail.python.org> <4BAB2882.6070508@egenix.com> <4BAB9FAE.1070206@hastings.org> <4BABA600.8090103@egenix.com> Message-ID: <4BABA87B.2030306@hastings.org> M.-A. Lemburg wrote: > [Y]ou're pretty much breaking all 3rd party modules that have just started using e.g the datetime module C API. Not if they're using PyDateTime_IMPORT. My understanding is that requiring a recompile is okay, and if you use PyDateTime_IMPORT and recompile against 2.7 it works fine. As I already mentioned, the exception to this is pyexpat, which provides no such importer macro / function. I shall await your more fully-fleshed out reply, /larry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdrake at acm.org Thu Mar 25 19:26:35 2010 From: fdrake at acm.org (Fred Drake) Date: Thu, 25 Mar 2010 14:26:35 -0400 Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk: Doc/c-api/capsule.rst Doc/c-api/cobject.rst Doc/c-api/concrete.rst Doc/data/refcounts.dat Doc/extending/extending.rst Include/Python.h Include/cStringIO.h Include/cobject.h Include/datetime Message-ID: <9cee7ab81003251126n28bca927j91f31d1d6d025c08@mail.gmail.com> On Thu, Mar 25, 2010 at 2:16 PM, Larry Hastings wrote: > My understanding is that requiring a recompile is okay This has always been a point of contention. I'm not even sure what the current official position is. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From larry at hastings.org Thu Mar 25 19:49:39 2010 From: larry at hastings.org (Larry Hastings) Date: Thu, 25 Mar 2010 11:49:39 -0700 Subject: [Python-Dev] [Python-checkins] PyCapsule backport In-Reply-To: References: <20100325005454.B1AC5FCCD@mail.python.org> <4BAB2882.6070508@egenix.com> <4BAB9FAE.1070206@hastings.org> Message-ID: <4BABB043.9080607@hastings.org> Just a reply to one part of your message, as I really need to get back to work for now. Antoine Pitrou wrote: > I think solution #1 would be the best one. > #2 looks too complicated. > #2 is a seven-line diff, below. /larry/ Index: cobject.c =================================================================== --- cobject.c (revision 79396) +++ cobject.c (working copy) @@ -50,7 +50,12 @@ PyCObject_AsVoidPtr(PyObject *self) { if (self) { - if (self->ob_type == &PyCObject_Type) + if (PyCapsule_CheckExact(self)) { + const char *name = PyCapsule_GetName(self); + const void *value = PyCapsule_GetPointer(self, name); + return value; + } + else if (self->ob_type == &PyCObject_Type) return ((PyCObject *)self)->cobject; PyErr_SetString(PyExc_TypeError, "PyCObject_AsVoidPtr with non-C-object"); From stefan_ml at behnel.de Thu Mar 25 20:15:28 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 25 Mar 2010 20:15:28 +0100 Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk: Doc/c-api/capsule.rst Doc/c-api/cobject.rst Doc/c-api/concrete.rst Doc/data/refcounts.dat Doc/extending/extending.rst Include/Python.h Include/cStringIO.h Include/cobject.h Include/datetime.h Include/py_curses.h Include/pycapsule.h Include/pyexpat.h Include/ucnhash.h Lib/test/test_sys.py Makefile.pre.in Misc/NEWS Modules/_ctypes/callproc.c Modules/_ctypes/cfield.c Modules/_ctypes/ctypes.h Modules/_cursesmodule.c Modules/_elementtree.c Modules/_testcapimodule.c Modules/cStringIO.c Modules/cjkcodecs/cjkcodecs.h Modules/cjkcodecs/multibytecodec.c Modules/cjkcodecs/multibytecodec.h Modules/datetimemodule.c Modules/pyexpat.c Modules/socketmodule.c Modules/socketmodule.h Modules/unicodedata.c Objects/capsule.c Objects/object.c Objects/unicodeobject.c PC/VS7.1/pythoncore.vcproj PC/VS8.0/pythoncore.vcproj PC/os2emx/python27.def In-Reply-To: <4BAB2882.6070508@egenix.com> References: <20100325005454.B1AC5FCCD@mail.python.org> <4BAB2882.6070508@egenix.com> Message-ID: M.-A. Lemburg, 25.03.2010 10:10: > larry.hastings wrote: >> Author: larry.hastings >> Date: Thu Mar 25 01:54:54 2010 >> New Revision: 79397 >> >> Log: >> Backported PyCapsule from 3.1, and converted most uses of >> CObject to PyCapsule. > > Backporting PyCapsule is fine, but the changes you made to all > those PyCObject uses does not look backwards compatible. I got this in Cython's continuous integration server after this change: Traceback (most recent call last): File "runtests.py", line 19, in import cPickle as pickle TypeError: PyCObject_AsVoidPtr with non-C-object Although this looks like an internal problem in CPython, I'm certainly against any backwards incompatible changes of this size in the 2.x series. Stefan From v+python at g.nevcal.com Thu Mar 25 20:31:28 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 25 Mar 2010 12:31:28 -0700 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <5c6f2a5d1003250813h15e5610bseb4a9b3b541585ff@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> <4BAB6E70.3050008@gmail.com> <5c6f2a5d1003250722g28b130edxda76a1f8b6f9f0dc@mail.gmail.com> <4BAB7BC0.8020306@gmail.com> <5c6f2a5d1003250813h15e5610bseb4a9b3b541585ff@mail.gmail.com> Message-ID: <4BABBA10.6000300@g.nevcal.com> On 3/25/2010 8:13 AM, Mark Dickinson wrote: > On Thu, Mar 25, 2010 at 3:05 PM, Nick Coghlan wrote: > >> Mark Dickinson wrote: >> >>> On Thu, Mar 25, 2010 at 2:08 PM, Nick Coghlan wrote: >>> >>>> Jesus Cea wrote: >>>> >>>>> But IEEE 754 was created by pretty clever guys and sure they had a >>>>> reason for define things in the way they are. Probably we are missing >>>>> something. >>>>> >>>> Yes, this is where their "implementable in a hardware circuit" focus >>>> comes in. They were primarily thinking of a floating point >>>> representation where the 32/64 bits are *it* - you can't have "multiple >>>> NaNs" because you don't have the bits available to describe them. >>>> >>> I'm not so sure about this: standard 64-bit binary IEEE 754 doubles >>> allow for 2**53-2 different nans (2**52-2 signaling nans, 2**52 quiet >>> nans): anything with bit pattern (msb to lsb) >>> >>> x1111111 1111xxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx >>> >>> is an infinity or a nan, and there are only 2 infinities. >>> >> I stand corrected :) >> >> It still seems to me that the problems mostly arise when we're trying to >> get floats and Decimals to behave like Python *objects* (i.e. with >> reflexive equality) rather than like IEEE defined numbers. >> >> It's an extra element that isn't part of the problem the numeric >> standards are trying to solve. >> > Agreed. We don't have to be "missing something"; rather, the IEEE > folks (quite understandably) almost certainly didn't anticipate this > kind of usage. So I'll concede that it's reasonable to consider > deviating from the standard in the light of this. > It is my understand that even bit-for-bit identical NaN values will compare unequal according to IEEE 754 rules. I would have no problem with Python interning each encountered NaN value, to avoid having bit-for-bit identical NaN values with different Python IDs, but having them compare equal seems inappropriate. Glenn From gruszczy at gmail.com Thu Mar 25 22:29:39 2010 From: gruszczy at gmail.com (=?UTF-8?Q?Filip_Gruszczy=C5=84ski?=) Date: Thu, 25 Mar 2010 22:29:39 +0100 Subject: [Python-Dev] Issue 6081: format string using mapping rather than kwargs In-Reply-To: <4BAB878A.3080106@trueblade.com> References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com> <4BAB4253.8090605@gmail.com> <4BAB5D9C.1010001@trueblade.com> <4BAB703E.10701@gmail.com> <4BAB78F6.8080905@trueblade.com> <4BAB7C8C.6020004@gmail.com> <4BAB878A.3080106@trueblade.com> Message-ID: <1be78d221003251429v2652e330tff1e7577ce2a03e@mail.gmail.com> >> I'm inclined to leave it alone unless/until Raymond or somebody else >> steps up to really champion it. > > I'm okay with that. And I am looking for another bug, that I could get some python-fu from ;-) -- Filip Gruszczy?ski From steve at pearwood.info Thu Mar 25 23:05:10 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 26 Mar 2010 09:05:10 +1100 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <4BAB482F.4030608@gmail.com> References: <201003252118.43146.steve@pearwood.info> <4BAB482F.4030608@gmail.com> Message-ID: <201003260905.11105.steve@pearwood.info> On Thu, 25 Mar 2010 10:25:35 pm Nick Coghlan wrote: > Steven D'Aprano wrote: > > I'd like to turn the question around ... what algorithms are there > > that rely on NaN == NaN being True? > > Absolutely anything that expects "x is y" to imply that "x == y". The > builtin containers enforce this by checking identity before they > check equality, but there are plenty of algorithms that won't. Fair point, but I was actually thinking about mathematical algorithms. Builtin containers may also fail with any object that violates the expectation that identity implies equality, e.g.: class AlwaysDifferent: def __eq__(self, other): return False def __ne__(self, other): return True I don't see this as a problem -- if you choose to use such objects, you're responsible for understanding what is going on. If you choose to use floats, then you need to understand that NANs are weird. Personally, I'm less concerned about sets of floats ending up with strange combinations of NANs than I am about the possibility of disastrous maths errors caused by allowing NANs to test as equal. Here's a simplistic example: def func(a, b): if a == b: return 1.0 return math.sin(a*b)**(a-b) (I say "simplistic" because it currently fails for a=b=INF.) Currently this function will do the right thing for a, b both NANs: >>> func(float('nan'), float('nan')) nan but making NANs test as equal will cause it to give the wrong answer. I fear that kind of error far more than the current funny behaviour of builtin containers with NANs. -- Steven D'Aprano From greg.ewing at canterbury.ac.nz Thu Mar 25 23:45:51 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 26 Mar 2010 11:45:51 +1300 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> Message-ID: <4BABE79F.7010802@canterbury.ac.nz> Georg Brandl wrote: > Thinking of each value created by float('nan') as > a different nan makes sense to my naive mind, and it also explains > nicely the behavior present right now. Not entirely: x = float('NaN') y = x if x == y: ... There it's hard to argue that the NaNs being compared result from different operations. It does suggest a potential compromise, though: a single NaN object compares equal to itself, but different NaN objects are never equal (more or less what dict membership testing does now, but extended to all == comparisons). Whether that's a *sane* compromise I'm not sure. -- Greg From solipsis at pitrou.net Fri Mar 26 00:06:06 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 25 Mar 2010 23:06:06 +0000 (UTC) Subject: [Python-Dev] Mixing float and Decimal -- thread reboot References: <201003252118.43146.steve@pearwood.info> <4BAB482F.4030608@gmail.com> <201003260905.11105.steve@pearwood.info> Message-ID: Steven D'Aprano pearwood.info> writes: > > Personally, I'm less concerned about sets of floats ending up with > strange combinations of NANs than I am about the possibility of > disastrous maths errors caused by allowing NANs to test as equal. > Here's a simplistic example: You just said "if you choose to use floats, then you need to understand that NANs are weird". I wonder why this saying shouldn't apply to your "simplistic example" of NAN usage. (is your example even from real life?) From rhamph at gmail.com Fri Mar 26 00:10:56 2010 From: rhamph at gmail.com (Adam Olsen) Date: Thu, 25 Mar 2010 17:10:56 -0600 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <201003252118.43146.steve@pearwood.info> References: <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com> <201003252118.43146.steve@pearwood.info> Message-ID: On Thu, Mar 25, 2010 at 04:18, Steven D'Aprano wrote: > def myfunc(x, y): > ? ?if x == y: > ? ? ? ?return 1.0 > ? ?else: > ? ? ? ?return something_complicated**(x-y) > > > Optimising floating point code is fraught with dangers (the above fails > for x=y=INF as well as NAN) but anything that make Not A Numbers > pretend to be numbers is a bad thing. What about this: def myfunc(x): if x >= THRESHOLD: return 1.0 else: return something_complicated(x) If one behaves right it's more likely a fluke, not a designed in feature. It's certainly not obvious without covering every comparison with comments. Maybe that's the solution. Signal by default on comparison, but add a collection of naneq/naneg/etc functions (math module, methods, whatever) that use a particular quiet mapping, making the whole thing explicit? -- Adam Olsen, aka Rhamphoryncus From guido at python.org Fri Mar 26 00:14:30 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 25 Mar 2010 16:14:30 -0700 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BABBA10.6000300@g.nevcal.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> <4BAB6E70.3050008@gmail.com> <5c6f2a5d1003250722g28b130edxda76a1f8b6f9f0dc@mail.gmail.com> <4BAB7BC0.8020306@gmail.com> <5c6f2a5d1003250813h15e5610bseb4a9b3b541585ff@mail.gmail.com> <4BABBA10.6000300@g.nevcal.com> Message-ID: On Thu, Mar 25, 2010 at 12:31 PM, Glenn Linderman wrote: > It is my understand that even bit-for-bit identical NaN values will compare > unequal according to IEEE 754 rules. > > I would have no problem with Python interning each encountered NaN value, to > avoid having bit-for-bit identical NaN values with different Python IDs, but > having them compare equal seems inappropriate. Let's please not intern NaNs. Interning is for performance. If you have enough NaNs to affect your performance I think you have bigger worries! -- --Guido van Rossum (python.org/~guido) From g.brandl at gmx.net Fri Mar 26 00:21:59 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 25 Mar 2010 23:21:59 +0000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BABE79F.7010802@canterbury.ac.nz> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <4BABE79F.7010802@canterbury.ac.nz> Message-ID: Am 25.03.2010 22:45, schrieb Greg Ewing: > Georg Brandl wrote: >> Thinking of each value created by float('nan') as >> a different nan makes sense to my naive mind, and it also explains >> nicely the behavior present right now. > > Not entirely: > > x = float('NaN') > y = x > if x == y: > ... > > There it's hard to argue that the NaNs being compared > result from different operations. > > It does suggest a potential compromise, though: a single > NaN object compares equal to itself, but different NaN > objects are never equal (more or less what dict membership > testing does now, but extended to all == comparisons). > > Whether that's a *sane* compromise I'm not sure. FWIW, I like it. Georg From raymond.hettinger at gmail.com Fri Mar 26 01:00:03 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Thu, 25 Mar 2010 17:00:03 -0700 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <4BABE79F.7010802@canterbury.ac.nz> Message-ID: <71121792-9ED3-40A3-87B1-5ED7EEDE1B65@gmail.com> On Mar 25, 2010, at 4:21 PM, Georg Brandl wrote: > Am 25.03.2010 22:45, schrieb Greg Ewing: >> Georg Brandl wrote: >>> Thinking of each value created by float('nan') as >>> a different nan makes sense to my naive mind, and it also explains >>> nicely the behavior present right now. >> >> Not entirely: >> >> x = float('NaN') >> y = x >> if x == y: >> ... >> >> There it's hard to argue that the NaNs being compared >> result from different operations. >> >> It does suggest a potential compromise, though: a single >> NaN object compares equal to itself, but different NaN >> objects are never equal (more or less what dict membership >> testing does now, but extended to all == comparisons). >> >> Whether that's a *sane* compromise I'm not sure. > > FWIW, I like it. > > Georg > +1 Raymond From steve at pearwood.info Fri Mar 26 01:57:38 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 26 Mar 2010 11:57:38 +1100 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> Message-ID: <201003261157.39250.steve@pearwood.info> On Thu, 25 Mar 2010 06:26:11 am Mark Dickinson wrote: > Here's an interesting recent blog post on this subject, from the > creator of Eiffel: > > http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of- >civilization/ Sorry, but he lost me right at the beginning when he quoted someone else: "there is no reason to believe that the result of one calculation with unclear value should match that of another calculation with unclear value" and then argued: "The exact same argument can be used to assert that the result should not be False: ? there is no reason to believe that the result of one calculation with unclear value should not match that of another calculation with unclear value. Just as convincing! Both arguments complement each other: there is no compelling reason for demanding that the values be equal; and there is no compelling argument either to demand that they be different. If you ignore one of the two sides, you are biased." This whole argument is invalid on at least three levels. I'll get the first two out the way briefly #1: Bertrand starts by treating NANs as "unclear values", and concludes that we shouldn't prefer "two unclear values are different" as more compelling than "two unclear values are the same". But this is ridiculous -- if you ask me a pair of questions, and I answer "I'm not sure" to both of them, why would you assume that the right answer to both questions is actually the same? #2: But in fact NANs aren't "unclear values", they are not values at all. The answer to "what is the non-complex logarithm of -1?" is not "I'm not sure" but "there is no such value". Bertrand spends an awful lot of time trying to demonstrate why the reflexivity of equality (every x is equal to itself) should apply to NANs as well as the other floats, but RoE is a property of equivalence relations, which does not (and should not) hold for "there is no such value". By analogy: the Lizard King of Russia does not exist; the Vampire Queen of New Orleans also does not exist. We don't therefore conclude that the Lizard King and the Vampire Queen are therefore the same person. #3: We could, if we wish, violate the IEEE standard and treat equality of NANs as an equivalence relation. It's our language, we're free to follow whatever standards we like, and reflexivity of equality is a very useful axiom to have. Since it applies to all non-NAN floats (and virtually every object in Python, other than those with funny __eq__ methods), perhaps we should extend it to NANs as well? I hope to convince you that the cost of doing so is worse than the disease. Since NANs are usually found in mathematical contexts, we should follow the IEEE standard even at the cost of rare anomalies in non-mathematical code containing NANs. Simply put: we should treat "two unclear values are different" as more compelling than "two unclear values are the same" as it leads to fewer, smaller, errors. Consider: log(-1) = NAN # maths equality, not assignment log(-2) = NAN If we allow NAN = NAN, then we permit the error: log(-1) = NAN = log(-2) therefore log(-1) = log(-2) and 1 = 2 But if make NAN != NAN, then we get: log(-1) != log(-2) and all of mathematics does not collapse into a pile of rubble. I think that is a fairly compelling reason to prefer inequality over equality. One objection might be that while log(-1) and log(-2) should be considered different NANs, surely NANs should be equal to themselves? -1 = -1 implies log(-1) = log(-1) But consider the practicalities: there are far more floats than available NAN payloads. We simply can't map every invalid calculation to a unique NAN, and therefore there *must* be cases like: log(-123.456789e-8) = log(-9.876e47) implies 123.456789e-8 = 9.876e47 So we mustn't consider NANs equal just because their payloads are equal. What about identity? Even if we don't dare allow this: x = log(-1) # assignment y = log(-1) # another NAN with the same payload assert x is not y assert x == y surely we can allow this? assert x == x But this is dangerous. Don't be fooled by the simplicity of the above example. Just because you have two references to the same (as in identity) NAN, doesn't mean they represent "the same thing" or came from the same place: data = [1, 2, float('nan'), float('nan'), 3] x = harmonic_mean(data) y = 1 - geometric_mean(data) It is an accident of implementation whether x and y happen to be the same object or not. Why should their inequality depend on such a fragile thing? In fact, identity of NANs is itself an implementation quirk of programming languages like Python: logically, NANs don't have identity at all. To put it another way: all ONEs are the same ONE, even if they come from different sources, are in different memory locations, or have different identities; but all NANs are different, even if they come from the same source, are in the same memory location, or have the same identity. The fundamental problem here is that NANs are not values. If you treat them as if they were values, then you want reflexivity of equality. But they're not -- they're *signals* for "your calculation has gone screwy and the result you get is garbage", so to speak. You shouldn't even think of a specific NAN as a piece of specific garbage, but merely a label on the *kind* of garbage you've got (the payload): INF-INF is, in some sense, a different kind of error to log(-1). In the same way you might say "INF-INF could be any number at all, therefore we return NAN", you might say "since INF-INF could be anything, there's no reason to think that INF-INF == INF-INF." -- Steven D'Aprano From steve at pearwood.info Fri Mar 26 02:10:54 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 26 Mar 2010 12:10:54 +1100 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BABE79F.7010802@canterbury.ac.nz> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BABE79F.7010802@canterbury.ac.nz> Message-ID: <201003261210.54863.steve@pearwood.info> On Fri, 26 Mar 2010 09:45:51 am Greg Ewing wrote: > Georg Brandl wrote: > > Thinking of each value created by float('nan') as > > a different nan makes sense to my naive mind, and it also explains > > nicely the behavior present right now. > > Not entirely: > > x = float('NaN') > y = x > if x == y: > ... > > There it's hard to argue that the NaNs being compared > result from different operations. But unlike us, the equality operator only has a pinhole view of the operands. It can't distinguish between your example and this: x = float('nan') y = some_complex_calculation(x) if x == y: ... where y merely happens to end up with the same object as x by some quirk of implementation. > It does suggest a potential compromise, though: a single > NaN object compares equal to itself, but different NaN > objects are never equal (more or less what dict membership > testing does now, but extended to all == comparisons). > > Whether that's a *sane* compromise I'm not sure. What do we do with Decimal? Aren't we committed to matching the Decimal standard, in which case aren't we committed to this? x = Decimal('nan') assert x != x If that's the case, then float NANs and Decimal NANs will behave differently. I think that's a mistake. For what it's worth, I'm -1 on allowing NANs to test equal to any NAN, including itself. However, I would be -0 on the following compromise: Allow NANs to test equal to themselves (identity testing). math module to grow a function ieee_equals(x, y) that keeps the current behaviour. -- Steven D'Aprano From greg.ewing at canterbury.ac.nz Fri Mar 26 02:15:54 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 26 Mar 2010 14:15:54 +1300 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <201003252118.43146.steve@pearwood.info> References: <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com> <201003252118.43146.steve@pearwood.info> Message-ID: <4BAC0ACA.7030401@canterbury.ac.nz> Steven D'Aprano wrote: > I'd like to turn the question around ... what algorithms are there that > rely on NaN == NaN being True? That seems to be a straw question, since AFAIK nobody has suggested that there are any such algorithms. On the other hand, it has been claimed that some algorithms exist that benefit from Nan == NaN being false, so it's fair to ask for examples to back that up. -- Greg From pje at telecommunity.com Fri Mar 26 02:19:06 2010 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 25 Mar 2010 21:19:06 -0400 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <201003261157.39250.steve@pearwood.info> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> Message-ID: <20100326011901.6F4133A4080@sparrow.telecommunity.com> At 11:57 AM 3/26/2010 +1100, Steven D'Aprano wrote: >But they're not -- they're *signals* for "your calculation has gone >screwy and the result you get is garbage", so to speak. You >shouldn't even think of a specific NAN as a piece of specific >garbage, but merely a label on the *kind* of garbage you've got (the >payload): INF-INF is, in some sense, a different kind of error to >log(-1). In the same way you might say "INF-INF could be any number >at all, therefore we return NAN", you might say "since INF-INF could >be anything, there's no reason to think that INF-INF == INF-INF." So, are you suggesting that maybe the Pythonic thing to do in that case would be to cause any operation on a NAN (including perhaps comparison) to fail, rather than allowing garbage to silently propagate? In other words, if NAN is only a signal that you have garbage, is there really any reason to keep it as an *object*, instead of simply raising an exception? Then, you could at least identify what calculation created the garbage, instead of it percolating up through other calculations. In low-level languages like C or Fortran, it obviously makes sense to represent NAN as a value, because there's no other way to represent it. But in a language with exceptions, is there a use case for it existing as a value? From fijall at gmail.com Fri Mar 26 02:31:57 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 25 Mar 2010 19:31:57 -0600 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <201003261157.39250.steve@pearwood.info> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> Message-ID: <693bc9ab1003251831l226687d2m9331b6dc086f0ab3@mail.gmail.com> > -1 = -1 > implies log(-1) = log(-1) Mathematically speaking this is incorrect. x = y implies log(x) = log(y) for x > 0 and y > 0 Cheers, fijal From jcea at jcea.es Fri Mar 26 02:37:48 2010 From: jcea at jcea.es (Jesus Cea) Date: Fri, 26 Mar 2010 02:37:48 +0100 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <201003261157.39250.steve@pearwood.info> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> Message-ID: <4BAC0FEC.8060308@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/26/2010 01:57 AM, Steven D'Aprano wrote: > In fact, identity of NANs is itself an implementation quirk of > programming languages like Python: logically, NANs don't have identity > at all. > > To put it another way: all ONEs are the same ONE, even if they come from > different sources, are in different memory locations, or have different > identities; but all NANs are different, even if they come from the same > source, are in the same memory location, or have the same identity. +inf. Bravo!. :-) - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS6wP7Jlgi5GaxT1NAQJXFgP+KHshsGGOM1GO/f1lEazul1+vipw6tUuF Hf8Pm39srX3pKV7OUabWJWBaRURnJWIHEymjwILBwVo9X4604klbTRi3MrqbOeZv G8NFeMzagXLwEOlAnPbVcfDb3KcW4C/Zm0A5TOPY4X5T/8FRmhfbrC4Ip2klyyMh 24fGUWTr5DI= =d1RL -----END PGP SIGNATURE----- From rhamph at gmail.com Fri Mar 26 02:53:31 2010 From: rhamph at gmail.com (Adam Olsen) Date: Thu, 25 Mar 2010 19:53:31 -0600 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <201003261157.39250.steve@pearwood.info> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> Message-ID: On Thu, Mar 25, 2010 at 18:57, Steven D'Aprano wrote: > Simply put: we should treat "two unclear values are different" as more > compelling than "two unclear values are the same" as it leads to fewer, > smaller, errors. Consider: > > log(-1) = NAN ?# maths equality, not assignment > log(-2) = NAN > > If we allow NAN = NAN, then we permit the error: > > log(-1) = NAN = log(-2) > therefore log(-1) = log(-2) > and 1 = 2 > > But if make NAN != NAN, then we get: > > log(-1) != log(-2) > > and all of mathematics does not collapse into a pile of rubble. I think > that is a fairly compelling reason to prefer inequality over equality. > > One objection might be that while log(-1) and log(-2) should be > considered different NANs, surely NANs should be equal to themselves? > > -1 = -1 > implies log(-1) = log(-1) IMO, this just shows how ludicrous it is to compare NaNs. No matter what we do it will imply some insane mathematical consequence implied and code that will break. They are, after all, an error passed silently. Why is it complex can raise an exception when sorted, forcing you to use a sane (and explicit) method, but for NaN it's okay to silently fail? -- Adam Olsen, aka Rhamphoryncus From v+python at g.nevcal.com Fri Mar 26 02:56:00 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 25 Mar 2010 18:56:00 -0700 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> <4BAB6E70.3050008@gmail.com> <5c6f2a5d1003250722g28b130edxda76a1f8b6f9f0dc@mail.gmail.com> <4BAB7BC0.8020306@gmail.com> <5c6f2a5d1003250813h15e5610bseb4a9b3b541585ff@mail.gmail.com> <4BABBA10.6000300@g.nevcal.com> Message-ID: <4BAC1430.4030301@g.nevcal.com> On 3/25/2010 4:14 PM, Guido van Rossum wrote: > On Thu, Mar 25, 2010 at 12:31 PM, Glenn Linderman wrote: > >> It is my understand that even bit-for-bit identical NaN values will compare >> unequal according to IEEE 754 rules. >> >> I would have no problem with Python interning each encountered NaN value, to >> avoid having bit-for-bit identical NaN values with different Python IDs, but >> having them compare equal seems inappropriate. >> > Let's please not intern NaNs. Interning is for performance. If you > have enough NaNs to affect your performance I think you have bigger > worries! > I'm OK with interning NaNs, and I'm OK with not interning NaNs. But I would much prefer to see Python conform to the IEEE 754 for float, and the Decimal standard for Decimal, where at all possible, including keeping NaNs not comparing equal to anything, themselves included. And retaining a mode where Decimal is standoffish, and doesn't accept non-Decimal comparisons or arithmetic seems like a big win for program correctness to me. Glenn From greg.ewing at canterbury.ac.nz Fri Mar 26 03:25:30 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 26 Mar 2010 15:25:30 +1300 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> <4BAB6E70.3050008@gmail.com> Message-ID: <4BAC1B1A.6030500@canterbury.ac.nz> Curt Hagenlocher wrote: > Wait, what? I haven't been paying much attention, but this is backwards. > There are multiple representations of NaN in the IEEE encoding; I think Nick's point is that there aren't enough bits to give the result of every operation its own unique NaN. The payload of a NaN in typical hardware implementations is quite small, because it has to fit into the exponent field. -- Greg From greg.ewing at canterbury.ac.nz Fri Mar 26 05:32:46 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 26 Mar 2010 17:32:46 +1300 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <201003261157.39250.steve@pearwood.info> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> Message-ID: <4BAC38EE.9090600@canterbury.ac.nz> Steven D'Aprano wrote: > By analogy: the Lizard King of Russia does not exist; the Vampire Queen > of New Orleans also does not exist. We don't therefore conclude that > the Lizard King and the Vampire Queen are therefore the same person. But it's equally invalid to say that they're *not* the same person, because it's meaningless to say *anything* about the properties of nonexistent people. -- Greg From greg.ewing at canterbury.ac.nz Fri Mar 26 05:35:55 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 26 Mar 2010 17:35:55 +1300 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <201003261210.54863.steve@pearwood.info> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BABE79F.7010802@canterbury.ac.nz> <201003261210.54863.steve@pearwood.info> Message-ID: <4BAC39AB.2040400@canterbury.ac.nz> Steven D'Aprano wrote: > What do we do with Decimal? Aren't we committed to matching the Decimal > standard, It's been pointed out that the Decimal standard only defines some abstract operations, and doesn't mandate that they be mapped onto any particular language syntax. That gives us enough flexibility to make == do what we want and still claim compliance with the standard. BTW, does IEEE754 give us the same flexibility? If so, we may not have much of a problem in the first place. -- Greg From greg.ewing at canterbury.ac.nz Fri Mar 26 05:40:23 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 26 Mar 2010 17:40:23 +1300 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BAC1B1A.6030500@canterbury.ac.nz> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es> <4BAB6E70.3050008@gmail.com> <4BAC1B1A.6030500@canterbury.ac.nz> Message-ID: <4BAC3AB7.5040809@canterbury.ac.nz> I impulsively wrote: > The payload of a NaN in typical hardware implementations > is quite small, because it has to fit into the exponent > field. ...which turns out to be precisely wrong. Some day I'll learn to wait until somebody else in the thread has checked the facts for me before posting. :-) -- Greg From stephen at xemacs.org Fri Mar 26 05:39:55 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 26 Mar 2010 13:39:55 +0900 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <201003261210.54863.steve@pearwood.info> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BABE79F.7010802@canterbury.ac.nz> <201003261210.54863.steve@pearwood.info> Message-ID: <87ocibg0h0.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > But unlike us, the equality operator only has a pinhole view of the > operands. It can't distinguish between your example and this: > > x = float('nan') > y = some_complex_calculation(x) > if x == y: > ... > > where y merely happens to end up with the same object as x by some quirk > of implementation. Note that Mark has already provided a related example of such a quirk (Decimal(-1).sqrt(), I think it was). From v+python at g.nevcal.com Fri Mar 26 05:47:53 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 25 Mar 2010 21:47:53 -0700 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BAC39AB.2040400@canterbury.ac.nz> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <4BABE79F.7010802@canterbury.ac.nz> <201003261210.54863.steve@pearwood.info> <4BAC39AB.2040400@canterbury.ac.nz> Message-ID: <4BAC3C79.400@g.nevcal.com> On 3/25/2010 9:35 PM, Greg Ewing wrote: > Steven D'Aprano wrote: > >> What do we do with Decimal? Aren't we committed to matching the >> Decimal standard, > > It's been pointed out that the Decimal standard only defines > some abstract operations, and doesn't mandate that they > be mapped onto any particular language syntax. That gives > us enough flexibility to make == do what we want and still > claim compliance with the standard. > > BTW, does IEEE754 give us the same flexibility? If so, > we may not have much of a problem in the first place. > I propose that the abstract Decimal operation for addition be mapped to the syntax of operator - and that the abstract Decimal operation for subtraction be mapped to the syntax of operator +. Then people will have to actually read the manual to learn how to use the Decimal type in Python, rather than assuming that things might work the way they expect. This will lead to more robust and correct programs, because people will have read the manual. Or at least it seems that it should work that way... ?? ? (Hmm, that might not render consistently for everyone, so I'll throw in a couple :) :) also.) Glenn From mal at egenix.com Fri Mar 26 10:20:08 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 26 Mar 2010 10:20:08 +0100 Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk: Doc/c-api/capsule.rst Doc/c-api/cobject.rst Doc/c-api/concrete.rst Doc/data/refcounts.dat Doc/extending/extending.rst Include/Python.h Include/cStringIO.h Include/cobject.h Include/datetime.h Include/py_curses.h Include/pycapsule.h Include/pyexpat.h Include/ucnhash.h Lib/test/test_sys.py Makefile.pre.in Misc/NEWS Modules/_ctypes/callproc.c Modules/_ctypes/cfield.c Modules/_ctypes/ctypes.h Modules/_cursesmodule.c Modules/_elementtree.c Modules/_testcapimodule.c Modules/cStringIO.c Modules/cjkcodecs/cjkcodecs.h Modules/cjkcodecs/multibytecodec.c Modules/cjkcodecs/multibytecodec.h Modules/datetimemodule.c Modules/pyexpat.c Modules/socketmodule.c Modules/socketmodule.h Modules/unicodedata.c Objects/capsule.c Objects/object.c Objects/unicodeobject.c PC/VS7.1/pythoncore.vcproj PC/VS8.0/pythoncore.vcproj PC/os2emx/python27.def PC/os2vacpp/python.def Python/compile.c Python/getargs.c In-Reply-To: <4BAB9FAE.1070206@hastings.org> References: <20100325005454.B1AC5FCCD@mail.python.org> <4BAB2882.6070508@egenix.com> <4BAB9FAE.1070206@hastings.org> Message-ID: <4BAC7C48.3080600@egenix.com> Larry Hastings wrote: > > M.-A. Lemburg wrote: >> Backporting PyCapsule is fine, but the changes you made to all >> those PyCObject uses does not look backwards compatible. >> >> The C APIs exposed by the modules (e.g. the datetime module) >> are used in lots of 3rd party extension modules and changing >> them from PyCObject to PyCapsule is a major change in the >> module API. > > You're right, my changes aren't backwards compatible. I thought it was > reasonable for four reasons: Just as reminder of the process we have in place for such changes: Please discuss any major breakage on python-dev before checking in the patch. > 1. The CObject API isn't safe. It's easy to crash Python 2.6 in just a > few lines by mixing and matching CObjects. Switching Python to capsules > prevents a class of exploits. I've included a script at the bottom of > this message that demonstrates three such crashes. The script runs in > Python 2 and 3, but 3.1 doesn't crash because it's using capsules. That's good, but then again: deliberate wrong use of APIs will always cause crashes and at least I don't know of any report about PyCObjects posing a problem in all their years of existence. > 2. As I just mentioned, Python 3.1 already uses capsules everywhere > instead of CObjects. Since part of the purpose of Python 2.7 is to > prepare developers for the to upgrade to 3.1, getting them to switch to > capsules now is just one more way they are prepared. Sure, but forcing them is not a good idea, even less so if you can easily expose both a PyCObject and PyCapsule interface to the same C API. > 3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we > ever made a 2.8 I want to remove it completely. Please remember that PyCObjects are not only used internally in CPython, but also in other 3rd party modules to expose C APIs and those will generally have to support more than just the latest Python release. If you deprecate those C APIs, the process will at least have to cover one more release, i.e. run through the whole deprecation process: 1. pending deprecation (2.7) 2. deprecation (2.8) 3. removal (2.9) I think it's better to add a -3 warning when using PyCObjects in 2.7. > 4. When Python publishes an API using a CObject, it describes the thing > the CObject points to in a header file. In nearly all cases that header > file also provides a macro or inline function that does the importing > work for you. I changed those to use capsules too. So if the > third-party code uses the macro or inline function, all you need do is > recompile it against 2.7 and it works fine. Sadly I know of one > exception: pyexpat.expat_CAPI. The header file just describes the > struct pointed to by the CObject, but callers I know about those macros... I introduced that idea with mxDateTime and then added the same logic in CPython in a couple of places a long while ago, e.g. socketmodule.h. IIRC, Jim Fulton added PyCObjects for exchanging module C APIs a few years before that. Later on PyCObject_Import() was added to simplify the C API import a bit. A recompile is certainly a possibility to have 3rd party modules switch to the capsule interfaces of the stdlib modules, but we should still be more careful about this. Wouldn't it be possible to have PyCObject_AsVoidPtr() et al. work with PyCapsules as well ? After all, the caller is only interested in the pointer and doesn't really care about what object was used to wrap it. > I can suggest four ways to ameliorate the problem. > > First, we could do as Antoine Pitrou suggests on the bug (issue 7992): > wherever the CObject used to be published as a module attribute to > expose an API, we could provide both a CObject and a capsule; internally > Python would only use the capsules. This would allow third-party > libraries to run against 2.7 unchanged. The major problem with this is > that third-party libraries would still be vulnerable to the > mix-and-match CObject crash. True, but we've been happy with this vulnerability for years, just as we've been happy with the fact that it's easy to crash the VM by passing hand-crafted byte-code to it, or using ctypes to call an OS function with the wrong parameters, etc. Like I said: there are many ways to deliberately crash Python. We don't have a concept of interface signatures in Python, it's mostly based on trust. > A secondary, minor concern: obviously we'd > store the CObject attribute with the existing name, and the capsule > attribute would have to get some new name. But in Python 3.1, these > attributes already expose a capsule. Therefore, people who convert to > using the capsules now would have to convert again when moving to 3.1. This should be manageable with some aliasing. > Second, we could make CObject internally support unpacking capsules. If > you gave a capsule to PyCObject_AsVoidPtr() it would unpack it and > return the pointer within. (We could probably also map the capsule > "context" to the CObject "desc", if any of the Python use cases needed > it.) I wouldn't change anything else about CObjects; creating and using > them would continue to work as normal. This would also allow > third-party libraries to run against Python 2.7 unchanged. The only > problem is that it's unsafe, as indeed allowing any use of > PyCObject_AsVoidPtr() is unsafe. That's a good idea. Yes, playing with fire is unsafe, but fire is useful in a lot of places as well :-) > Third, I've been pondering writing a set of preprocessor macros, shipped > in their own header file distributed independently of Python and > released to the public domain, that would make it easy to use either > CObjects or capsules depending on what version of Python you were > compiling against. Obviously, using these macros would require a source > code change in the third-party library. But these macros would make it > a five-minute change. This could compliment the first or second > approaches. I'm not sure whether that would worth the effort. 3rd party modules for Python 2.x are likely not going to use them, since they typically have to support more than just Python 2.7. E.g. look at Plone - they are still running on Python 2.4 and use lots of 3rd party modules. It took Zope years to switch from 2.4 to 2.6. You have the same situation with other large systems, whether OSS or not. If a required 3rd party module would suddenly only support Python 2.7, developers would either have to find a replacement that (still) works for them (rather unlikely), or try to port their whole app to 2.7, which often enough is not easily possible due to other restrictions. This aspect is often not considered when discussing such backwards incompatible changes - at least not in the early stages. Fortunately, we have so far - and most of the time - found a way to keep everyone happy. > Fourth, we could back out of the changes to published APIs and convert > them back to CObjects. -1. See above. That's still an option as well, but I agree that either making PyCObjects compatible with PyCapsules (via changes to the PyCObject functions) or having them exist side-by-side is a better option. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 26 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From stefan_ml at behnel.de Fri Mar 26 11:03:32 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 26 Mar 2010 11:03:32 +0100 Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk: Doc/c-api/capsule.rst Doc/c-api/cobject.rst Doc/c-api/concrete.rst Doc/data/refcounts.dat Doc/extending/extending.rst Include/Python.h Include/cStringIO.h Include/cobject.h Include/datetime.h Include/py_curses.h Include/pycapsule.h Include/pyexpat.h Include/ucnhash.h Lib/test/test_sys.py Makefile.pre.in Misc/NEWS Modules/_ctypes/callproc.c Modules/_ctypes/cfield.c Modules/_ctypes/ctypes.h Modules/_cursesmodule.c Modules/_elementtree.c Modules/_testcapimodule.c Modules/cStringIO.c Modules/cjkcodecs/cjkcodecs.h Modules/cjkcodecs/multibytecodec.c Modules/cjkcodecs/multibytecodec.h Modules/datetimemodule.c Modules/pyexpat.c Modules/socketmodule.c Modules/socketmodule.h Modules/unicodedata.c Objects/capsule.c Objects/object.c Objects/unicodeobject.c PC/VS7.1/pythoncore.vcproj PC/VS8.0/pythoncore.vcproj PC/os2emx/python27.def In-Reply-To: <4BAC7C48.3080600@egenix.com> References: <20100325005454.B1AC5FCCD@mail.python.org> <4BAB2882.6070508@egenix.com> <4BAB9FAE.1070206@hastings.org> <4BAC7C48.3080600@egenix.com> Message-ID: M.-A. Lemburg, 26.03.2010 10:20: > Larry Hastings wrote: >> 3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we >> ever made a 2.8 I want to remove it completely. > > Please remember that PyCObjects are not only used internally > in CPython, but also in other 3rd party modules to expose C APIs > and those will generally have to support more than just the latest > Python release. The removal of PyCObject from Py3.2 pretty much caught the Cython project by surprise, as they had been The One Obvious Way for public C-APIs for ages. We quickly switched to PyCapsule back then, but only for newer 3.x versions. This is easy to do when your code relies on Cython (or its generated header files) for importing and exporting this C-API, as a rebuild will just do all the work for you. However, if you have to do this manually, you will basically have to write all the code yourself that Cython now generates. If this can be done using preprocessor macros, fine. But the underlying ABI should not be changed in Py2.x. The normal 2.x approach would be to add the new API *in addition* to the old one, potentially by providing macros that map the new functionality to the old one, but not the other way round. That would allow code to be written that works in both Py2.7 and Py3.2+, without breaking backwards compatibility with existing Py2.x libraries. And, no, recompilation is not always an option and certainly shouldn't be required for the soon to be discontinued 2.x line. The *only* reason for the very existence of 2.7 is to not break backwards compatibility. > If you deprecate those C APIs, the process will at least have to cover > one more release, i.e. run through the whole deprecation process: > > 1. pending deprecation (2.7) > 2. deprecation (2.8) > 3. removal (2.9) If you replace 2.8 by 3.1 and 2.9 by 3.2, that's more or less what has happened already. No need to put any more thoughts into Py2.x for that purpose. Stefan From cournape at gmail.com Fri Mar 26 18:04:06 2010 From: cournape at gmail.com (David Cournapeau) Date: Sat, 27 Mar 2010 02:04:06 +0900 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <20100326011901.6F4133A4080@sparrow.telecommunity.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> Message-ID: <5b8d13221003261004r249dbfacr6a16a8f2212cc080@mail.gmail.com> On Fri, Mar 26, 2010 at 10:19 AM, P.J. Eby wrote: > At 11:57 AM 3/26/2010 +1100, Steven D'Aprano wrote: >> >> But they're not -- they're *signals* for "your calculation has gone screwy >> and the result you get is garbage", so to speak. You shouldn't even think of >> a specific NAN as a piece of specific garbage, but merely a label on the >> *kind* of garbage you've got (the payload): INF-INF is, in some sense, a >> different kind of error to log(-1). In the same way you might say "INF-INF >> could be any number at all, therefore we return NAN", you might say "since >> INF-INF could be anything, there's no reason to think that INF-INF == >> INF-INF." > > So, are you suggesting that maybe the Pythonic thing to do in that case > would be to cause any operation on a NAN (including perhaps comparison) to > fail, rather than allowing garbage to silently propagate? Nan behavior being tightly linked to FPU exception handling, I think this is a good idea. One of the goal of Nan is to avoid many testing in intermediate computation (for efficiency reason), which may not really apply to python. Generally, you want to detect errors/exceptional situations as early as possible, and if you use python, you don't care about potential slowdown caused by those checks. David From casey at pandora.com Fri Mar 26 18:40:43 2010 From: casey at pandora.com (Casey Duncan) Date: Fri, 26 Mar 2010 11:40:43 -0600 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <20100326011901.6F4133A4080@sparrow.telecommunity.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> Message-ID: On Mar 25, 2010, at 7:19 PM, P.J. Eby wrote: > At 11:57 AM 3/26/2010 +1100, Steven D'Aprano wrote: >> But they're not -- they're *signals* for "your calculation has gone screwy and the result you get is garbage", so to speak. You shouldn't even think of a specific NAN as a piece of specific garbage, but merely a label on the *kind* of garbage you've got (the payload): INF-INF is, in some sense, a different kind of error to log(-1). In the same way you might say "INF-INF could be any number at all, therefore we return NAN", you might say "since INF-INF could be anything, there's no reason to think that INF-INF == INF-INF." > > So, are you suggesting that maybe the Pythonic thing to do in that case would be to cause any operation on a NAN (including perhaps comparison) to fail, rather than allowing garbage to silently propagate? > > In other words, if NAN is only a signal that you have garbage, is there really any reason to keep it as an *object*, instead of simply raising an exception? Then, you could at least identify what calculation created the garbage, instead of it percolating up through other calculations. > > In low-level languages like C or Fortran, it obviously makes sense to represent NAN as a value, because there's no other way to represent it. But in a language with exceptions, is there a use case for it existing as a value? If a NaN object is allowed to exist, that is a float operation that does not return a real number does not itself raise an exception immediately, then it will always be possible to get (seemingly) nonsensical behavior when it is used in containers that do not themselves "operate" on their elements. So even provided that performing any "operation" on a NaN object raises an exception, it would still be possible to add such an object to a list or tuple and have subsequent containment checks for that object return false. So this "solution" would simply narrow the problem posed, but not eliminate it. None of the solution posed seem very ideal, in particular when they deviate from the standard in arbitrary ways someone deems "better". It's obvious to me that no ideal solution exists so long as you attempt to represent non-numeric values in a numeric type. So unless you simply eliminate NaNs (thus breaking the standard), you are going to confuse somebody. And I think having float deviate from the IEEE standard is ill advised unless there is no alternative (i.e., the standard cannot be practically implemented), and breaking it will confuse people too (and probably the ones that know this domain). I propose that the current behavior stands as is and that the documentation make mention of the fact that NaN values are unordered, thus some float values may not behave intuitively wrt hashing, equality, etc. The fact of the matter is that using floats as dict keys or set values or even just checking equality is much more complex in practice than you would expect. I mean even representing 1.1 is problematic ;^). Unless the float values you are using are constants, how would you practically use them as dict keys, or hsah set members anyway? I'm not saying it can't be done, but is a hash table with float keys ever a data structure that someone on this list would recommend? If so good luck and god speed 8^) -Casey From python-dev at masklinn.net Fri Mar 26 22:16:19 2010 From: python-dev at masklinn.net (Xavier Morel) Date: Fri, 26 Mar 2010 22:16:19 +0100 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> Message-ID: <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net> On 26 Mar 2010, at 18:40 , Casey Duncan wrote: > > > On Mar 25, 2010, at 7:19 PM, P.J. Eby wrote: > >> At 11:57 AM 3/26/2010 +1100, Steven D'Aprano wrote: >>> But they're not -- they're *signals* for "your calculation has gone screwy and the result you get is garbage", so to speak. You shouldn't even think of a specific NAN as a piece of specific garbage, but merely a label on the *kind* of garbage you've got (the payload): INF-INF is, in some sense, a different kind of error to log(-1). In the same way you might say "INF-INF could be any number at all, therefore we return NAN", you might say "since INF-INF could be anything, there's no reason to think that INF-INF == INF-INF." >> >> So, are you suggesting that maybe the Pythonic thing to do in that case would be to cause any operation on a NAN (including perhaps comparison) to fail, rather than allowing garbage to silently propagate? >> >> In other words, if NAN is only a signal that you have garbage, is there really any reason to keep it as an *object*, instead of simply raising an exception? Then, you could at least identify what calculation created the garbage, instead of it percolating up through other calculations. >> >> In low-level languages like C or Fortran, it obviously makes sense to represent NAN as a value, because there's no other way to represent it. But in a language with exceptions, is there a use case for it existing as a value? > > If a NaN object is allowed to exist, that is a float operation that does not return a real number does not itself raise an exception immediately, then it will always be possible to get (seemingly) nonsensical behavior when it is used in containers that do not themselves "operate" on their elements. How about raising an exception instead of creating nans in the first place, except maybe within specific contexts (so that the IEEE-754 minded can get their nans working as they currently do)? That way, there cannot be any nan-induced seemingly nonsensical behavior except within known scopes. From casey at pandora.com Fri Mar 26 22:39:44 2010 From: casey at pandora.com (Casey Duncan) Date: Fri, 26 Mar 2010 15:39:44 -0600 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net> Message-ID: <16E0E0B5-9E14-4386-815E-742F344ADEA3@pandora.com> On Mar 26, 2010, at 3:16 PM, Xavier Morel wrote: > On 26 Mar 2010, at 18:40 , Casey Duncan wrote: >> >> >> On Mar 25, 2010, at 7:19 PM, P.J. Eby wrote: >> >>> At 11:57 AM 3/26/2010 +1100, Steven D'Aprano wrote: >>>> But they're not -- they're *signals* for "your calculation has gone screwy and the result you get is garbage", so to speak. You shouldn't even think of a specific NAN as a piece of specific garbage, but merely a label on the *kind* of garbage you've got (the payload): INF-INF is, in some sense, a different kind of error to log(-1). In the same way you might say "INF-INF could be any number at all, therefore we return NAN", you might say "since INF-INF could be anything, there's no reason to think that INF-INF == INF-INF." >>> >>> So, are you suggesting that maybe the Pythonic thing to do in that case would be to cause any operation on a NAN (including perhaps comparison) to fail, rather than allowing garbage to silently propagate? >>> >>> In other words, if NAN is only a signal that you have garbage, is there really any reason to keep it as an *object*, instead of simply raising an exception? Then, you could at least identify what calculation created the garbage, instead of it percolating up through other calculations. >>> >>> In low-level languages like C or Fortran, it obviously makes sense to represent NAN as a value, because there's no other way to represent it. But in a language with exceptions, is there a use case for it existing as a value? >> >> If a NaN object is allowed to exist, that is a float operation that does not return a real number does not itself raise an exception immediately, then it will always be possible to get (seemingly) nonsensical behavior when it is used in containers that do not themselves "operate" on their elements. > How about raising an exception instead of creating nans in the first place, except maybe within specific contexts (so that the IEEE-754 minded can get their nans working as they currently do)? > > That way, there cannot be any nan-induced seemingly nonsensical behavior except within known scopes. Having NaN creation raise an exception would undoubtedly break plenty of existing code that either expects and deals with NaNs itself or works accidentally because the NaNs do not cause harm. I don't sympathize much with the latter case since they are just hidden bugs probably, but the former makes it hard to justify raising exceptions for NaNs as the default behavior. But since I assume we're talking Python 3 here, maybe arguments containing the phase "existing code" can be dutifully ignored, I dunno. -Casey From raymond.hettinger at gmail.com Sat Mar 27 00:16:36 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 26 Mar 2010 16:16:36 -0700 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net> Message-ID: On Mar 26, 2010, at 2:16 PM, Xavier Morel wrote: >> > How about raising an exception instead of creating nans in the first place, except maybe within specific contexts (so that the IEEE-754 minded can get their nans working as they currently do)? -1 The numeric community uses NaNs as placeholders in vectorized calculations. People do use them and there's no point in breaking their code. Of the ideas I've seen in this thread, only two look reasonable: * Do nothing. This is attractive because it doesn't break anything. * Have float.__eq__(x, y) return True whenever x and y are the same NaN object. This is attractive because it is a minimal change that provides a little protection for simple containers. I support either of those options. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Mar 27 01:05:27 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 27 Mar 2010 11:05:27 +1100 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <20100326011901.6F4133A4080@sparrow.telecommunity.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> Message-ID: <201003271105.28852.steve@pearwood.info> On Fri, 26 Mar 2010 12:19:06 pm P.J. Eby wrote: > At 11:57 AM 3/26/2010 +1100, Steven D'Aprano wrote: > >But they're not -- they're *signals* for "your calculation has gone > >screwy and the result you get is garbage", so to speak. You > >shouldn't even think of a specific NAN as a piece of specific > >garbage, but merely a label on the *kind* of garbage you've got (the > >payload): INF-INF is, in some sense, a different kind of error to > >log(-1). In the same way you might say "INF-INF could be any number > >at all, therefore we return NAN", you might say "since INF-INF could > >be anything, there's no reason to think that INF-INF == INF-INF." > > So, are you suggesting that maybe the Pythonic thing to do in that > case would be to cause any operation on a NAN (including perhaps > comparison) to fail, rather than allowing garbage to silently > propagate? Certainly not. That defeats the whole purpose of NANs. I wish floating point calculations in Python would return NANs rather than raise the exceptions they do now. I can't speak for others, but in my experience NANs are a much nicer way to do maths-related programming. I've programmed with a system that supported NANs extensively (Apple's SANE, circa 1990), and I miss it so. Note also that NANs do not necessarily contaminate every expression or function call. The standard allows for them to "cancel out", so to speak, where it is mathematically justifiable: >>> nan = float('nan') >>> 1.0**nan 1.0 so you shouldn't assume that the presence of a NAN in a calculation is the kiss of death. > In other words, if NAN is only a signal that you have garbage, is > there really any reason to keep it as an *object*, instead of simply > raising an exception? Then, you could at least identify what > calculation created the garbage, instead of it percolating up through > other calculations. The standard distinguishes between signalling NANs and quiet NANs (which propagate as values). By default, signalling NANs are usually converted to quiet NANs, but the caller is supposed to be able to be able to change that behaviour to a floating point signal which can be trapped. In Python, the equivalent would be an exception. -- Steven D'Aprano From fuzzyman at voidspace.org.uk Sat Mar 27 01:59:21 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 27 Mar 2010 00:59:21 +0000 Subject: [Python-Dev] Optional delta argument for assertAlmostEqual Message-ID: <4BAD5869.6010607@voidspace.org.uk> Hello all, A user has suggested an optional argument to unittest.TestCase.assertAlmostEqual for specifying a maximum difference between the expected and actual values, instead of using rounding. This sounds great to me as the default implementation of assertAlmostEqual has *never* been useful to me (YMMV). In fact one of the first things I do on setting up a test suite is provide a TestCase that overrides assertAlmostEqual with an implementation that uses a delta rather than rounding. The implementation would be effectively: assert abs(actual - expected) < delta This has the advantage that it allows things like: self.assertAlmostEqual(timeStamp, expected, delta=datetime.timedelta(seconds=5)) The issue is this would make the signature of assertAlmostEqual (and its negative counterpart): def assertAlmostEqual(self, first, second, places=7, msg=None, delta=None) Note that delta comes after msg, which is different to other assert methods. To put delta before msg would be backwards incompatible with existing uses passing arguments positionally. In Python 3.2 we can make delta a keyword argument. Passing both places and delta would be an error (TypeError). Anyway, unless there are strenuous objections I intend to do this. All the best, Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From cournape at gmail.com Sat Mar 27 06:32:07 2010 From: cournape at gmail.com (David Cournapeau) Date: Sat, 27 Mar 2010 14:32:07 +0900 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net> Message-ID: <5b8d13221003262232m11c30a9ata3b17410f3d7c53@mail.gmail.com> On Sat, Mar 27, 2010 at 8:16 AM, Raymond Hettinger wrote: > > On Mar 26, 2010, at 2:16 PM, Xavier Morel wrote: > > How about raising an exception instead of creating nans in the first place, > except maybe within specific contexts (so that the IEEE-754 minded can get > their nans working as they currently do)? > > -1 > The numeric community uses NaNs as placeholders in vectorized calculations. But is this relevant to python itself ? In Numpy, we indeed do use and support NaN, but we have much more control on what happens compared to python float objects. We can control whether invalid operations raises an exception or not, we had isnan/isfinite for a long time, and the fact that nan != nan has never been a real problem AFAIK. David From dickinsm at gmail.com Sat Mar 27 12:12:02 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 27 Mar 2010 11:12:02 +0000 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: <5d44f72f1003241815g109236fco1ab9607544352cf0@mail.gmail.com> References: <20100323120959.GA486@mail.bytereef.org> <4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org> <5c6f2a5d1003241409g806d79dw4cd245b97ce47ae9@mail.gmail.com> <5d44f72f1003241815g109236fco1ab9607544352cf0@mail.gmail.com> Message-ID: <5c6f2a5d1003270412r2756f4d0ue0e2a4603166a96b@mail.gmail.com> On Thu, Mar 25, 2010 at 1:15 AM, Jeffrey Yasskin wrote: > On Wed, Mar 24, 2010 at 2:09 PM, Mark Dickinson wrote: >> Slight change of topic. ?I've been implementing the extra comparisons >> required for the Decimal type and found an anomaly while testing. >> Currently in py3k, order comparisons (but not ==, !=) between a >> complex number and another complex, float or int raise TypeError: >> >>>>> z = complex(0, 0) >>>>> z < int() >> Traceback (most recent call last): >> ?File "", line 1, in >> TypeError: unorderable types: complex() < int() >>>>> z < float() >> Traceback (most recent call last): >> ?File "", line 1, in >> TypeError: unorderable types: complex() < float() >>>>> z < complex() >> Traceback (most recent call last): >> ?File "", line 1, in >> TypeError: unorderable types: complex() < complex() >> >> But Fraction is the odd man out: ?a comparison between a Fraction and >> a complex raises a TypeError for complex numbers with nonzero >> imaginary component, but returns a boolean value if the complex number >> has zero imaginary component: >> >>>>> z < Fraction() >> False >>>>> complex(0, 1) < Fraction() >> Traceback (most recent call last): >> ?File "", line 1, in >> TypeError: unorderable types: complex() < Fraction() >> >> I'm tempted to call this Fraction behaviour a bug, but maybe it arises >> from the numeric integration themes of PEP 3141. ?Any ideas? > > I'd call it a bug. > Thanks, Jeffrey (and everyone else who answered). Fixed in r79456 (py3k) and r79455 (trunk). Mark From dickinsm at gmail.com Sat Mar 27 13:22:20 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 27 Mar 2010 12:22:20 +0000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net> Message-ID: <5c6f2a5d1003270522p2a84cce5seb4c85e824d14358@mail.gmail.com> On Fri, Mar 26, 2010 at 11:16 PM, Raymond Hettinger wrote: > Of the ideas I've seen in this thread, only two look reasonable: > * Do nothing. ?This is attractive because it doesn't break anything. > * Have float.__eq__(x, y) return True whenever x and y are > ?? the same NaN object. ?This is attractive because it is a > ?? minimal change that provides a little protection for > ?? simple containers. > I support either of those options. Yes; those are the only two options I've seen that seem workable. Of the two, I prefer the first (do nothing), but would be content with second. I'd be interested to know whether there's any real-life code that's suffering as a result of nan != nan. While the nan weirdnesses certainly exist, I'm having difficulty imagining them turning up in real code. Casey Duncan's point that there can't be many good uses for floats as dict keys or set members is compelling, though there may be type-agnostic applications that care (e.g., memoizing). Similarly, putting floats into a list must be very common, but I'd guess that checking whether a given float is in a list doesn't happen that often. I suspect that (nan+container)-related oddities turn up infrequently enough to make it not worth fixing. By the way, for those suggesting that any operation producing a nan raise an exception instead: Python's math module actually does go out of its way to protect naive users from nans. You can't get a nan out of any of the math module functions without having put a nan in in the first place. Invalid operations like math.sqrt(-1), math.log(-1), consistently raise ValueError rather than return a nan. Ideally I'd like to see this behaviour extended to arithmetic as well, so that e.g., float('inf')/float('inf') raises instead of producing float('nan') (and similarly 1e300 * 1e300 raises OverflowError instead of producing an infinity), but there are backwards compatibility concerns. But even then, I'd still want it to be possible to produce nans deliberately when necessary, e.g., by directly calling float('nan'). Python also needs to be able to handle floating-point data generated from other sources; for this alone it should be at least able to read and write infinities and nans. Mark From ncoghlan at gmail.com Sat Mar 27 14:21:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 27 Mar 2010 23:21:39 +1000 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <5c6f2a5d1003270522p2a84cce5seb4c85e824d14358@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net> <5c6f2a5d1003270522p2a84cce5seb4c85e824d14358@mail.gmail.com> Message-ID: <4BAE0663.8060303@gmail.com> Mark Dickinson wrote: > On Fri, Mar 26, 2010 at 11:16 PM, Raymond Hettinger > wrote: >> Of the ideas I've seen in this thread, only two look reasonable: >> * Do nothing. This is attractive because it doesn't break anything. >> * Have float.__eq__(x, y) return True whenever x and y are >> the same NaN object. This is attractive because it is a >> minimal change that provides a little protection for >> simple containers. >> I support either of those options. > > Yes; those are the only two options I've seen that seem workable. Of > the two, I prefer the first (do nothing), but would be content with > second. I've ended up in the same place as Mark: +1 on retaining the status quo (possibly with better warnings about the potential oddities of floating point values being placed in equality-based containers), +0 on changing NaN equality to check identity first in order to provide reflexivity under == for these two types. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From dickinsm at gmail.com Sat Mar 27 14:45:17 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 27 Mar 2010 13:45:17 +0000 Subject: [Python-Dev] Optional delta argument for assertAlmostEqual In-Reply-To: <4BAD5869.6010607@voidspace.org.uk> References: <4BAD5869.6010607@voidspace.org.uk> Message-ID: <5c6f2a5d1003270645v5a673979s63406f7ed9c87ba7@mail.gmail.com> On Sat, Mar 27, 2010 at 12:59 AM, Michael Foord wrote: > Hello all, > > A user has suggested an optional argument to > unittest.TestCase.assertAlmostEqual for specifying a maximum difference > between the expected and actual values, instead of using rounding. +1. Mark From greg at krypto.org Sat Mar 27 18:49:22 2010 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 27 Mar 2010 10:49:22 -0700 Subject: [Python-Dev] Optional delta argument for assertAlmostEqual In-Reply-To: <4BAD5869.6010607@voidspace.org.uk> References: <4BAD5869.6010607@voidspace.org.uk> Message-ID: <52dc1c821003271049t2b7056a0yf97ff0bbc0d875df@mail.gmail.com> On Fri, Mar 26, 2010 at 5:59 PM, Michael Foord wrote: > Hello all, > > A user has suggested an optional argument to > unittest.TestCase.assertAlmostEqual for specifying a maximum difference > between the expected and actual values, instead of using rounding. > > This sounds great to me as the default implementation of assertAlmostEqual > has *never* been useful to me (YMMV). In fact one of the first things I do > on setting up a test suite is provide a TestCase that overrides > assertAlmostEqual with an implementation that uses a delta rather than > rounding. > > The implementation would be effectively: > > assert abs(actual - expected) < delta > > This has the advantage that it allows things like: > > self.assertAlmostEqual(timeStamp, expected, > delta=datetime.timedelta(seconds=5)) > > The issue is this would make the signature of assertAlmostEqual (and its > negative counterpart): > > def assertAlmostEqual(self, first, second, places=7, msg=None, delta=None) > > Note that delta comes after msg, which is different to other assert methods. > To put delta before msg would be backwards incompatible with existing uses > passing arguments positionally. In Python 3.2 we can make delta a keyword > argument. Passing both places and delta would be an error (TypeError). > > Anyway, unless there are strenuous objections I intend to do this. +1 sounds good to me. > > All the best, > > Michael > > -- > http://www.ironpythoninaction.com/ > http://www.voidspace.org.uk/blog > > READ CAREFULLY. By accepting and reading this email you agree, on behalf of > your employer, to release me from all obligations and waivers arising from > any and all NON-NEGOTIATED agreements, licenses, terms-of-service, > shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, > non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have > entered into with your employer, its partners, licensors, agents and > assigns, in perpetuity, without prejudice to my ongoing rights and > privileges. You further represent that you have the authority to release me > from any BOGUS AGREEMENTS on behalf of your employer. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > From rhamph at gmail.com Sat Mar 27 19:36:27 2010 From: rhamph at gmail.com (Adam Olsen) Date: Sat, 27 Mar 2010 12:36:27 -0600 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net> Message-ID: On Fri, Mar 26, 2010 at 17:16, Raymond Hettinger wrote: > Of the ideas I've seen in this thread, only two look reasonable: > * Do nothing. ?This is attractive because it doesn't break anything. > * Have float.__eq__(x, y) return True whenever x and y are > ?? the same NaN object. ?This is attractive because it is a > ?? minimal change that provides a little protection for > ?? simple containers. > I support either of those options. What's the flaw in using isnan()? -- Adam Olsen, aka Rhamphoryncus From robert.kern at gmail.com Sun Mar 28 01:28:03 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 27 Mar 2010 19:28:03 -0500 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <5b8d13221003262232m11c30a9ata3b17410f3d7c53@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net> <5b8d13221003262232m11c30a9ata3b17410f3d7c53@mail.gmail.com> Message-ID: On 2010-03-27 00:32 , David Cournapeau wrote: > On Sat, Mar 27, 2010 at 8:16 AM, Raymond Hettinger > wrote: >> >> On Mar 26, 2010, at 2:16 PM, Xavier Morel wrote: >> >> How about raising an exception instead of creating nans in the first place, >> except maybe within specific contexts (so that the IEEE-754 minded can get >> their nans working as they currently do)? >> >> -1 >> The numeric community uses NaNs as placeholders in vectorized calculations. > > But is this relevant to python itself ? In Numpy, we indeed do use and > support NaN, but we have much more control on what happens compared to > python float objects. We can control whether invalid operations raises > an exception or not, we had isnan/isfinite for a long time, and the > fact that nan != nan has never been a real problem AFAIK. Nonetheless, the closer our float arrays are to Python's float type, the happier I will be. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Sun Mar 28 01:27:57 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 27 Mar 2010 19:27:57 -0500 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net> Message-ID: On 2010-03-27 13:36 , Adam Olsen wrote: > On Fri, Mar 26, 2010 at 17:16, Raymond Hettinger > wrote: >> Of the ideas I've seen in this thread, only two look reasonable: >> * Do nothing. This is attractive because it doesn't break anything. >> * Have float.__eq__(x, y) return True whenever x and y are >> the same NaN object. This is attractive because it is a >> minimal change that provides a little protection for >> simple containers. >> I support either of those options. > > What's the flaw in using isnan()? There are implicit comparisons being done inside list.__contains__() and other such methods. They do not, and should not, know about isnan(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charles.r.mccreary at gmail.com Sun Mar 28 05:54:19 2010 From: charles.r.mccreary at gmail.com (Charles McCreary) Date: Sat, 27 Mar 2010 22:54:19 -0500 Subject: [Python-Dev] Optional delta argument for assertAlmostEqual In-Reply-To: <4BAD5869.6010607@voidspace.org.uk> References: <4BAD5869.6010607@voidspace.org.uk> Message-ID: <8294397f1003272054l21c659e1jdbb02191e86d3f09@mail.gmail.com> Perhaps not just absolute but relative tolerance, e.g.: def isclose(x, y, rtol=1.e-5, atol=1.e-8): return abs(x-y) <= atol + rtol * abs(y) On Fri, Mar 26, 2010 at 7:59 PM, Michael Foord wrote: > Hello all, > > A user has suggested an optional argument to > unittest.TestCase.assertAlmostEqual for specifying a maximum difference > between the expected and actual values, instead of using rounding. > > This sounds great to me as the default implementation of assertAlmostEqual > has *never* been useful to me (YMMV). In fact one of the first things I do > on setting up a test suite is provide a TestCase that overrides > assertAlmostEqual with an implementation that uses a delta rather than > rounding. > > The implementation would be effectively: > > assert abs(actual - expected) < delta > > This has the advantage that it allows things like: > > self.assertAlmostEqual(timeStamp, expected, > delta=datetime.timedelta(seconds=5)) > > The issue is this would make the signature of assertAlmostEqual (and its > negative counterpart): > > def assertAlmostEqual(self, first, second, places=7, msg=None, delta=None) > > Note that delta comes after msg, which is different to other assert > methods. To put delta before msg would be backwards incompatible with > existing uses passing arguments positionally. In Python 3.2 we can make > delta a keyword argument. Passing both places and delta would be an error > (TypeError). > > Anyway, unless there are strenuous objections I intend to do this. > > All the best, > > Michael > > -- > http://www.ironpythoninaction.com/ > http://www.voidspace.org.uk/blog > > READ CAREFULLY. By accepting and reading this email you agree, on behalf of > your employer, to release me from all obligations and waivers arising from > any and all NON-NEGOTIATED agreements, licenses, terms-of-service, > shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, > non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have > entered into with your employer, its partners, licensors, agents and > assigns, in perpetuity, without prejudice to my ongoing rights and > privileges. You further represent that you have the authority to release me > from any BOGUS AGREEMENTS on behalf of your employer. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/charles.r.mccreary%40gmail.com > -- Charles McCreary P.E. CRM Engineering 903.643.3490 - office 903.224.5701 - mobile/GV -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhamph at gmail.com Sun Mar 28 08:32:46 2010 From: rhamph at gmail.com (Adam Olsen) Date: Sun, 28 Mar 2010 00:32:46 -0600 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net> Message-ID: On Sat, Mar 27, 2010 at 18:27, Robert Kern wrote: > On 2010-03-27 13:36 , Adam Olsen wrote: >> What's the flaw in using isnan()? > > There are implicit comparisons being done inside list.__contains__() and > other such methods. They do not, and should not, know about isnan(). Those methods should raise an exception. Conceptually, NaN should contaminate the result and make list.__contains__() return some "unsortable", but we don't want to bend the whole language backwards just for one obscure feature, especially when we have a much better approach most of the time (exceptions). The reason why NaN's current behaviour is so disturbing is that it increases the mental load of everybody dealing with floats. When you write new code or debug a program you have to ask yourself what might happen if a NaN is produced. When maintaining existing code you have to figure out if it's written a specific way to get NaN to work right, or if it's even a fluke that NaN's work right, even if it was never intended for NaNs or never sees them on developer machines. This is all the subtlety we work so hard to avoid normally, so why make an exception here? NaNs themselves have use cases, but their subtlety doesn't. -- Adam Olsen, aka Rhamphoryncus From steve at pearwood.info Sun Mar 28 09:06:06 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 28 Mar 2010 18:06:06 +1100 Subject: [Python-Dev] Optional delta argument for assertAlmostEqual In-Reply-To: <8294397f1003272054l21c659e1jdbb02191e86d3f09@mail.gmail.com> References: <4BAD5869.6010607@voidspace.org.uk> <8294397f1003272054l21c659e1jdbb02191e86d3f09@mail.gmail.com> Message-ID: <201003281806.07478.steve@pearwood.info> On Sun, 28 Mar 2010 02:54:19 pm Charles McCreary wrote: > Perhaps not just absolute but relative tolerance, e.g.: > > def isclose(x, y, rtol=1.e-5, atol=1.e-8): > return abs(x-y) <= atol + rtol * abs(y) I'm not sure why you add the tolerances like that. Surely a more appropriate approach is to compare the values against the tolerances individually, and return True if they meet either condition. For what it's worth, here's a recipe I wrote for approximately equal, following someone's complaint on comp.lang.python that Python has no approximately-equal operator. http://code.activestate.com/recipes/577124-approximately-equal/ Comments and criticism welcome. I've never found unittest.TestCase.assertAlmostEqual to be useful. I generally write my own comparison function, then call: assert_(compare(x, y)) as needed. I'm +0.5 on Michael's suggested change -- it will make assertAlmostEqual marginally more useful, which is a plus, but I'm not sure that absolute tolerances are better than relative. But how general do we want to be? "Almost equal" means something different to everyone. > On Fri, Mar 26, 2010 at 7:59 PM, Michael Foord wrote: > > Hello all, > > > > A user has suggested an optional argument to > > unittest.TestCase.assertAlmostEqual for specifying a maximum > > difference between the expected and actual values, instead of using > > rounding. [snip] -- Steven D'Aprano From charles.r.mccreary at gmail.com Sun Mar 28 16:35:58 2010 From: charles.r.mccreary at gmail.com (Charles McCreary) Date: Sun, 28 Mar 2010 09:35:58 -0500 Subject: [Python-Dev] Optional delta argument for assertAlmostEqual In-Reply-To: <201003281806.07478.steve@pearwood.info> References: <4BAD5869.6010607@voidspace.org.uk> <8294397f1003272054l21c659e1jdbb02191e86d3f09@mail.gmail.com> <201003281806.07478.steve@pearwood.info> Message-ID: <8294397f1003280735x1f6acd04waf63074f6666c5fd@mail.gmail.com> A good summary on floating point comparison algorithms can be found at http://www.boost.org/doc/libs/1_34_0/libs/test/doc/components/test_tools/floating_point_comparison.html I, too, redefine almostEquals so that the comparison is more robust +1 On Sun, Mar 28, 2010 at 2:06 AM, Steven D'Aprano wrote: > On Sun, 28 Mar 2010 02:54:19 pm Charles McCreary wrote: > > Perhaps not just absolute but relative tolerance, e.g.: > > > > def isclose(x, y, rtol=1.e-5, atol=1.e-8): > > return abs(x-y) <= atol + rtol * abs(y) > > I'm not sure why you add the tolerances like that. Surely a more > appropriate approach is to compare the values against the tolerances > individually, and return True if they meet either condition. > > For what it's worth, here's a recipe I wrote for approximately equal, > following someone's complaint on comp.lang.python that Python has no > approximately-equal operator. > > http://code.activestate.com/recipes/577124-approximately-equal/ > > Comments and criticism welcome. > > I've never found unittest.TestCase.assertAlmostEqual to be useful. I > generally write my own comparison function, then call: > > assert_(compare(x, y)) > > as needed. I'm +0.5 on Michael's suggested change -- it will make > assertAlmostEqual marginally more useful, which is a plus, but I'm not > sure that absolute tolerances are better than relative. But how general > do we want to be? "Almost equal" means something different to everyone. > > > > > On Fri, Mar 26, 2010 at 7:59 PM, Michael Foord > wrote: > > > Hello all, > > > > > > A user has suggested an optional argument to > > > unittest.TestCase.assertAlmostEqual for specifying a maximum > > > difference between the expected and actual values, instead of using > > > rounding. > [snip] > > > > -- > Steven D'Aprano > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/charles.r.mccreary%40gmail.com > -- Charles McCreary P.E. CRM Engineering 903.643.3490 - office 903.224.5701 - mobile/GV -------------- next part -------------- An HTML attachment was scrubbed... URL: From debatem1 at gmail.com Sun Mar 28 20:09:39 2010 From: debatem1 at gmail.com (geremy condra) Date: Sun, 28 Mar 2010 14:09:39 -0400 Subject: [Python-Dev] Optional delta argument for assertAlmostEqual In-Reply-To: <4BAD5869.6010607@voidspace.org.uk> References: <4BAD5869.6010607@voidspace.org.uk> Message-ID: On Fri, Mar 26, 2010 at 8:59 PM, Michael Foord wrote: > Hello all, > > A user has suggested an optional argument to > unittest.TestCase.assertAlmostEqual for specifying a maximum difference > between the expected and actual values, instead of using rounding. > > This sounds great to me as the default implementation of assertAlmostEqual > has *never* been useful to me (YMMV). In fact one of the first things I do > on setting up a test suite is provide a TestCase that overrides > assertAlmostEqual with an implementation that uses a delta rather than > rounding. > > The implementation would be effectively: > > assert abs(actual - expected) < delta > > This has the advantage that it allows things like: > > self.assertAlmostEqual(timeStamp, expected, > delta=datetime.timedelta(seconds=5)) > > The issue is this would make the signature of assertAlmostEqual (and its > negative counterpart): > > def assertAlmostEqual(self, first, second, places=7, msg=None, delta=None) > > Note that delta comes after msg, which is different to other assert methods. > To put delta before msg would be backwards incompatible with existing uses > passing arguments positionally. In Python 3.2 we can make delta a keyword > argument. Passing both places and delta would be an error (TypeError). > > Anyway, unless there are strenuous objections I intend to do this. > > All the best, > > Michael +1, this would be very helpful to me. Geremy Condra From larry at hastings.org Sun Mar 28 21:25:37 2010 From: larry at hastings.org (Larry Hastings) Date: Sun, 28 Mar 2010 12:25:37 -0700 Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk: Doc/c-api/capsule.rst Doc/c-api/cobject.rst Doc/c-api/concrete.rst Doc/data/refcounts.dat Doc/extending/extending.rst Include/Python.h Include/cStringIO.h Include/cobject.h Include/datetime.h Include/py_curses.h Include/pycapsule.h Include/pyexpat.h Include/ucnhash.h Lib/test/test_sys.py Makefile.pre.in Misc/NEWS Modules/_ctypes/callproc.c Modules/_ctypes/cfield.c Modules/_ctypes/ctypes.h Modules/_cursesmodule.c Modules/_elementtree.c Modules/_testcapimodule.c Modules/cStringIO.c Modules/cjkcodecs/cjkcodecs.h Modules/cjkcodecs/multibytecodec.c Modules/cjkcodecs/multibytecodec.h Modules/datetimemodule.c Modules/pyexpat.c Modules/socketmodule.c Modules/socketmodule.h Modules/unicodedata.c Objects/capsule.c Objects/object.c Objects/unicodeobject.c PC/VS7.1/pythoncore.vcproj PC/VS8.0/pythoncore.vcproj PC/os2emx/python27.def PC/os2vacpp/python.def Python/compile.c Python/getargs.c In-Reply-To: <4BAC7C48.3080600@egenix.com> References: <20100325005454.B1AC5FCCD@mail.python.org> <4BAB2882.6070508@egenix.com> <4BAB9FAE.1070206@hastings.org> <4BAC7C48.3080600@egenix.com> Message-ID: <4BAFAD31.2050300@hastings.org> Well, it's been a couple of days, and nobody has stepped forward to speak in favor of my aggressive PyCapsule schedule. So I will bow to the wishes of those who have spoken up, and scale it back. Specifically, I propose writing a patch that will: * allow the CObject API to (unsafely) dereference a capsule * add support for the new files added by capsule to the Visual Studio builds (fixes an unrelated second complaint about my checkin) * add a -3 warning for calls to CObject * add a PendingDeprecation warning to CObject I've gotten in the habit of doing code reviews at work, and I now greatly prefer working that way. Can I get a volunteer to review the patch when it's ready? Please email me directly. Thanks! M.-A. Lemburg wrote: > Just as reminder of the process we have in place for such changes: > Please discuss any major breakage on python-dev before checking in > the patch. > I'm aware this is a good idea. I simply didn't consider this a major breakage. Recompiling against the 2.7 header files fixes it for everybody. (Except external users of pyexpat, if any exist. Google doesn't show any, though this is not proof that they don't exist.) If you suggest that any breakage with previous versions is worth mentioning, no matter how small, then I'll remember that in the future. Certainly the Python community has had many thrilling and dynamic conversations over minutae, so I guess it wouldn't be that surprising if this were true. >> 1. The CObject API isn't safe. It's easy to crash Python 2.6 in just a >> few lines by mixing and matching CObjects. Switching Python to capsules >> prevents a class of exploits. I've included a script at the bottom of >> this message that demonstrates three such crashes. The script runs in >> Python 2 and 3, but 3.1 doesn't crash because it's using capsules. >> > > That's good, but then again: deliberate wrong use of APIs will > always cause crashes and at least I don't know of any report about > PyCObjects posing a problem in all their years of existence. > > [...] we've been happy with this vulnerability for years, just > as we've been happy with the fact that it's easy to crash the > VM by passing hand-crafted byte-code to it, or using ctypes to > call an OS function with the wrong parameters, etc. > I'm surprised you find that acceptable. I thought the community was in agreement: our goal is to make the interpreter uncrashable from pure Python. I would describe ctypes as "impure" Python. If you wanted to allow untrusted developers to run code through a Python interpreter, you'd harden it: take away ctypes, and the ability to feed in arbitrary bytecode, and reduce the stack limit. You probably wouldn't take away "datetime", "cStringIO", and "cPickle"--yet I can crash 2.6 using those. >> Third, I've been pondering writing a set of preprocessor macros, shipped >> in their own header file distributed independently of Python and >> released to the public domain, that would make it easy to use either >> CObjects or capsules depending on what version of Python you were >> compiling against. Obviously, using these macros would require a source >> code change in the third-party library. But these macros would make it >> a five-minute change. This could compliment the first or second >> approaches. >> > I'm not sure whether that would worth the effort. 3rd party modules > for Python 2.x are likely not going to use them, since they typically > have to support more than just Python 2.7. > The point of these macros would be to support older versions of Python. The macros would boil down to either PyCapsule or CObject calls, depending on what version of Python you were compiling against. For Python 2.6 and before it would alway use CObjects. One set of macros would switch to PyCapsule when building against 2.7; another would use CObject permanently in 2.x. (Or maybe it should switch in 2.8? 2.9? But now we're in the realm of speculative fiction.) I'd be happy to write these if there was demand, but my sneaking suspicion is that nobody would use them. If you're genuinely interested in these macros please email me privately. /larry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Mar 28 23:17:25 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 29 Mar 2010 08:17:25 +1100 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> Message-ID: <201003290817.25870.steve@pearwood.info> On Sun, 28 Mar 2010 05:32:46 pm Adam Olsen wrote: > On Sat, Mar 27, 2010 at 18:27, Robert Kern wrote: > > On 2010-03-27 13:36 , Adam Olsen wrote: > >> What's the flaw in using isnan()? > > > > There are implicit comparisons being done inside > > list.__contains__() and other such methods. They do not, and should > > not, know about isnan(). > > Those methods should raise an exception. Conceptually, NaN should > contaminate the result and make list.__contains__() return some > "unsortable", but we don't want to bend the whole language backwards > just for one obscure feature, especially when we have a much better > approach most of the time (exceptions). I disagree -- if I ask: 3.0 in [1.0, 2.0, float('nan'), 3.0] I should get True, not an exception. Comparing NANs for equality isn't an error. +1 on leaving the behaviour alone -- the surprising behaviour people have pointed out with NANs in lists, dicts and sets occurs more often in theory than in practice. -- Steven D'Aprano From greg.ewing at canterbury.ac.nz Mon Mar 29 01:55:55 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 29 Mar 2010 11:55:55 +1200 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <201003290817.25870.steve@pearwood.info> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003290817.25870.steve@pearwood.info> Message-ID: <4BAFEC8B.8020009@canterbury.ac.nz> Steven D'Aprano wrote: > I disagree -- if I ask: > > 3.0 in [1.0, 2.0, float('nan'), 3.0] > > I should get True, not an exception. Yes, I don't think anyone would disagree that NaN should compare unequal to anything that isn't a NaN. Problems only arise when comparing two NaNs. -- Greg From rnk at mit.edu Mon Mar 29 04:13:23 2010 From: rnk at mit.edu (Reid Kleckner) Date: Sun, 28 Mar 2010 22:13:23 -0400 Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk: Doc/c-api/capsule.rst Doc/c-api/cobject.rst Doc/c-api/concrete.rst Doc/data/refcounts.dat Doc/extending/extending.rst Include/Python.h Include/cStringIO.h Include/cobject.h Include/datetime Message-ID: <9a9942201003281913u77f7fa31v6ec3b04cfeea5112@mail.gmail.com> On Sun, Mar 28, 2010 at 3:25 PM, Larry Hastings wrote: > M.-A. Lemburg wrote: > > Just as reminder of the process we have in place for such changes: > Please discuss any major breakage on python-dev before checking in > the patch. > > > I'm aware this is a good idea.? I simply didn't consider this a major breakage.? Recompiling against the 2.7 header files fixes it for everybody.? (Except external users of pyexpat, if any exist.? Google doesn't show any, though this is not proof that they don't exist.) > > If you suggest that any breakage with previous versions is worth mentioning, no matter how small, then I'll remember that in the future.? Certainly the Python community has had many thrilling and dynamic conversations over minutae, so I guess it wouldn't be that surprising if this were true. I'm curious what is considered reasonable/unreasonable breakage. If this breakage just requires a recompile, then doesn't it just introduce an ABI incompatibility? Aren't those allowed every minor (point) release? Or do people believe that this is more than an ABI change? Reid From rhamph at gmail.com Mon Mar 29 07:04:33 2010 From: rhamph at gmail.com (Adam Olsen) Date: Sun, 28 Mar 2010 23:04:33 -0600 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <4BAFEC8B.8020009@canterbury.ac.nz> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003290817.25870.steve@pearwood.info> <4BAFEC8B.8020009@canterbury.ac.nz> Message-ID: On Sun, Mar 28, 2010 at 17:55, Greg Ewing wrote: > Steven D'Aprano wrote: > >> I disagree -- if I ask: >> >> 3.0 in [1.0, 2.0, float('nan'), 3.0] >> >> I should get True, not an exception. > > Yes, I don't think anyone would disagree that NaN should compare > unequal to anything that isn't a NaN. Problems only arise when > comparing two NaNs. NaN includes real numbers. Although a NaN is originally produced for results that are not real numbers, further operations could produce a real number; we'd never know as NaN has no precision. Extending with complex numbers instead gives enough precision to show how this can happen. -- Adam Olsen, aka Rhamphoryncus From techtonik at gmail.com Mon Mar 29 07:55:32 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 29 Mar 2010 08:55:32 +0300 Subject: [Python-Dev] Request for commit access In-Reply-To: <4BA94202.6060502@v.loewis.de> References: <20100323152748.50ECE1FD33F@kimball.webabinitio.net> <4BA94202.6060502@v.loewis.de> Message-ID: On Wed, Mar 24, 2010 at 12:34 AM, "Martin v. L?wis" wrote: > > Procedurally, I wonder where people got the notion from that you can or > need to apply for commit access. IIUC, it used to be the case that you > would be recommended for commit access, by some (more or less senior) > fellow committer. That person then would work on actually getting commit > access to the new committer - perhaps by first asking other people in > private, to avoid any public embarrassment if access is ultimately > denied. IMO, that committer should then also mentor the new guy, both by > helping out in difficult cases, and by closely following commits to see > whether (possibly unstated) conventions are being followed. That's an ideal case, but it doesn't work, because more or less senior committers are already too busy. If they do not even have time to review issues, followup on patches - how can they monitor who reached the appropriate karma level? -- anatoly t. From cournape at gmail.com Mon Mar 29 08:17:34 2010 From: cournape at gmail.com (David Cournapeau) Date: Mon, 29 Mar 2010 15:17:34 +0900 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net> <5b8d13221003262232m11c30a9ata3b17410f3d7c53@mail.gmail.com> Message-ID: <5b8d13221003282317h32b2807bm467df310fe56b285@mail.gmail.com> On Sun, Mar 28, 2010 at 9:28 AM, Robert Kern wrote: > On 2010-03-27 00:32 , David Cournapeau wrote: >> >> On Sat, Mar 27, 2010 at 8:16 AM, Raymond Hettinger >> ?wrote: >>> >>> On Mar 26, 2010, at 2:16 PM, Xavier Morel wrote: >>> >>> How about raising an exception instead of creating nans in the first >>> place, >>> except maybe within specific contexts (so that the IEEE-754 minded can >>> get >>> their nans working as they currently do)? >>> >>> -1 >>> The numeric community uses NaNs as placeholders in vectorized >>> calculations. >> >> But is this relevant to python itself ? In Numpy, we indeed do use and >> support NaN, but we have much more control on what happens compared to >> python float objects. We can control whether invalid operations raises >> an exception or not, we had isnan/isfinite for a long time, and the >> fact that nan != nan has never been a real problem AFAIK. > > Nonetheless, the closer our float arrays are to Python's float type, the > happier I will be. Me too, but I don't see how to reconcile this with the intent of simplifying nan handling because they are not intuitive, which seems to be the goal of this discussion. David From martin at v.loewis.de Mon Mar 29 08:19:01 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 29 Mar 2010 08:19:01 +0200 Subject: [Python-Dev] Request for commit access In-Reply-To: References: <20100323152748.50ECE1FD33F@kimball.webabinitio.net> <4BA94202.6060502@v.loewis.de> Message-ID: <4BB04655.6060008@v.loewis.de> > That's an ideal case, but it doesn't work, because more or less senior > committers are already too busy. If they do not even have time to > review issues, followup on patches - how can they monitor who reached > the appropriate karma level? The practice proves to be different. In the recent additions, the new developer had been approached by some committer, suggesting that they apply. So the initiative actually started from a committer who thought that the karma level was appropriate. All I'm asking for is that then this committer takes the issues in his own hands, rather than letting the new developer ask for himself. Regards, Martin From techtonik at gmail.com Mon Mar 29 09:30:03 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 29 Mar 2010 10:30:03 +0300 Subject: [Python-Dev] Bootstrap script for package management tool in Python 2.7 (Was: Re: [Distutils] At least one package management tool for 2.7) Message-ID: So, there won't be any package management tool shipped with Python 2.7 and users will have to download and install `setuptools` manually as before: "search" -> "download" -> "unzip" -> "cmd" -> "cd" -> "python setup.py install" Therefore I still propose shipping bootstrap package that instruct user how to download and install an actual package management tool when users tries to use it. So far I know only one stable tool - `easy_install` - a part of `setuptools` package. The required behavior for very basic user friendliness: 1. user installs Python 2.7 2. user issues `python -m easy_install something` 3. user gets message 'easy_install' tool is not installed on this system. To make it available, download and install `setuptools` package from http://pypi.python.org/pypi/setuptools/ 4. the screen is paused before exit (for windows systems) Other design notes: 1. if package tries to import `easy_install` module used for bootstrap, it gets the same ImportException as if there were no `easy_install` at all 2. bootstrap module is overwritten by actual package when users installs it So, do we need a PEP for that? How else can I know if consensus is reached? Anybody is willing to elaborate on implementation? P.S. Please be careful to reply to relevant lists -- anatoly t. On Mon, Mar 29, 2010 at 9:37 AM, Tarek Ziad? wrote: > 2010/3/29 anatoly techtonik : > [..] >> It is really hard to follow. You should at least change subjects when >> switching topic. > > I was talking about the work going on and the decisions taken lately. > > I never change topics of threads mails when there's less than 100 mails, > because I find it way more confusing :) > >> >> So, what is the verdict? Will there be a package management tool or >> bootstrap package for it shipped with Python 2.7 or not? > > As I said in the mail you've quoted, all improvements are made in > Distutils2. So the answer is no. > Python 2.7b1 is due in less than a week anyways, so any new > development on this topic will happen after. > > Basically Python 2.7 == Python 2.6 in term of packaging. > > Regards > Tarek > > -- > Tarek Ziad? | http://ziade.org > From techtonik at gmail.com Mon Mar 29 09:40:06 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 29 Mar 2010 10:40:06 +0300 Subject: [Python-Dev] GSoC 2010 is on -- projects? In-Reply-To: <20100319023630.GC1617@idyll.org> References: <20100319023630.GC1617@idyll.org> Message-ID: I would vote for allowing student work on community infrastructure tasks. Tracker, Wiki, Web site management tools are all outdated and everybody who cares agrees that they've seen a better tools. -- anatoly t. On Fri, Mar 19, 2010 at 4:36 AM, C. Titus Brown wrote: > Hi all, > > once again, the PSF has been accepted as a mentoring foundation for the Google > Summer of Code! ?This year, we're going to emphasize python 3 porting, so > please think of projects you'd like to see tackled. > > Please submit ideas for projects as soon as possible, as students will be able > to start applying soon. ?You can add them to this page: > > ? http://wiki.python.org/moin/SummerOfCode/2010 > > You can subscribe to the mentors list here, > > ? http://mail.python.org/mailman/listinfo/soc2010-mentors > > and the general discussion list (students included) here: > > ? http://mail.python.org/mailman/listinfo/soc2010-general > > thanks, > --titus > -- > C. Titus Brown, ctb at msu.edu > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/techtonik%40gmail.com > From ziade.tarek at gmail.com Mon Mar 29 09:55:08 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 29 Mar 2010 09:55:08 +0200 Subject: [Python-Dev] Bootstrap script for package management tool in Python 2.7 (Was: Re: [Distutils] At least one package management tool for 2.7) In-Reply-To: References: Message-ID: <94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com> On Mon, Mar 29, 2010 at 9:30 AM, anatoly techtonik wrote: > So, there won't be any package management tool shipped with Python 2.7 > and users will have to download and install `setuptools` manually as > before: > > ?"search" -> "download" -> "unzip" -> "cmd" -> "cd" -> "python > setup.py install" > > > Therefore I still propose shipping bootstrap package that instruct > user how to download and install an actual package ?management tool > when users tries to use it. So far I know only one stable tool - > `easy_install` - a part of `setuptools` package. There are other tools to install something in a vanilla Python : - pip (which also have an uninstall feature) - distutils - distribute > > The required behavior for very basic user friendliness: > 1. user installs Python 2.7 > 2. user issues `python -m easy_install something` > 3. user gets message > 'easy_install' tool is not installed on this system. To make it > available, download and install `setuptools` package from > http://pypi.python.org/pypi/setuptools/ Are you thinking about something generic ? Like, being able to call : $ python -m ANYTHING And get an error message saying that the ANYTHING script is not installed ? Then have a registry somewhere to get the name of the project that owns the ANYTHING script ? [..] > So, do we need a PEP for that? How else can I know if consensus is > reached? Anybody is willing to elaborate on implementation? I see two paths here: 1 - do you want to define a general mechanism for Python to install scripts ? 2 - are you just suggesting to have this mechanism only for Package Managers ? In case of 2), what you would need to do is propose an extension to PEP 376, we are currently working on, then we can add it there once people have discussed it on distutils-SIG, or maybe create a new PEP if the topic is too big to fit in 376. > > > P.S. Please be careful to reply to relevant lists Yes, so if it's 2) let's keep this thread in Distutils-SIG. Cross-posting like this makes it hard to follow. Tarek -- Tarek Ziad? | http://ziade.org From regebro at gmail.com Mon Mar 29 09:58:59 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 29 Mar 2010 09:58:59 +0200 Subject: [Python-Dev] [Distutils] Bootstrap script for package management tool in Python 2.7 (Was: Re: At least one package management tool for 2.7) In-Reply-To: References: Message-ID: <319e029f1003290058tf8f840bm23e454386570ea9a@mail.gmail.com> On Mon, Mar 29, 2010 at 09:30, anatoly techtonik wrote: > Therefore I still propose shipping bootstrap package that instruct > user how to download and install an actual package ?management tool > when users tries to use it. So far I know only one stable tool - > `easy_install` - a part of `setuptools` package. We can't ship *A* bootstrap script until there is *A* package management tool in the Python world. Currently there are three. :) In short: The tools that exist are not ready yet. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From techtonik at gmail.com Mon Mar 29 11:02:10 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 29 Mar 2010 12:02:10 +0300 Subject: [Python-Dev] Bootstrap script for package management tool in Python 2.7 (Was: Re: [Distutils] At least one package management tool for 2.7) In-Reply-To: <94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com> References: <94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com> Message-ID: On Mon, Mar 29, 2010 at 10:55 AM, Tarek Ziad? wrote: >> >> Therefore I still propose shipping bootstrap package that instruct >> user how to download and install an actual package ?management tool >> when users tries to use it. So far I know only one stable tool - >> `easy_install` - a part of `setuptools` package. > > There are other tools to install something in a vanilla Python : > > - pip ?(which also have an uninstall feature) > - distutils > - distribute distutils is not a `package management` tool, because it doesn't know anything even about installed packages, not saying anything about dependencies. `pip` and `distribute` are unknown for a vast majority of Python users, so if you have a perspective replacement for `easy_install` - it can be said in bootstrap package message. There is no problem with packages that require `setuptools` either - they will require `setuptools` as dependency anyway. For now there are two questions: 1. Are they stable enough for the replacement of user command line `easy_install` tool? 2. Which one is the recommended? P.S. Should there be an accessible FAQ in addition to ML? >> >> The required behavior for very basic user friendliness: >> 1. user installs Python 2.7 >> 2. user issues `python -m easy_install something` >> 3. user gets message >> 'easy_install' tool is not installed on this system. To make it >> available, download and install `setuptools` package from >> http://pypi.python.org/pypi/setuptools/ > > Are you thinking about something generic ? > > Like, being able to call : > > $ python -m ANYTHING > > And get an error message saying that the ANYTHING script is not installed ? > Then have a registry somewhere to get the name of the project that > owns the ANYTHING script ? > No. To get something in 2.7 I would refrain from developing into this direction for now. > [..] >> So, do we need a PEP for that? How else can I know if consensus is >> reached? Anybody is willing to elaborate on implementation? > > I see two paths here: > 1 - do you want to define a general mechanism for Python to install scripts ? > 2 - are you just suggesting to have this mechanism only for Package Managers ? 3 - I want current "best practice" for installing Python modules with dependencies from PyPI to be shipped with Python 2.7 by default Answering your questions: 1. I want to have some user-friendly way for installing Python scripts, but I am more concerned that I will miss `easy_install` in Python 2.7 2. Bootstrap script is aimed at users. Package managers are assumed to already have their package at PyPI and provide install instructions that involve `easy_install`, so nothing is required to be done on their part. > > In case of 2), what you would need to do is propose an extension to > PEP 376, we are currently working > on, then we can add it there once people have discussed it on > distutils-SIG, or maybe create a new PEP if the topic is too big to > fit in 376. PEP 376 is completely irrelevant to user side boot package proposal. This proposal is to recommend whatever package managing tool you think user need and instruct how to get the tool. The boot package doesn't know anything at all about how packages are managed. >> P.S. Please be careful to reply to relevant lists > > Yes, so if it's 2) let's keep this thread in Distutils-SIG. > Cross-posting like this makes it hard to follow. My vision is that decision about having bootstrap package or not in 2.7 should be in python-dev and specific packaging, implementation and pip/distutils/distribute questions in distutils-sig. -- anatoly t. From ziade.tarek at gmail.com Mon Mar 29 11:15:06 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 29 Mar 2010 11:15:06 +0200 Subject: [Python-Dev] Bootstrap script for package management tool in Python 2.7 (Was: Re: [Distutils] At least one package management tool for 2.7) In-Reply-To: References: <94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com> Message-ID: <94bdd2611003290215s72ea681dp246505b8894695@mail.gmail.com> On Mon, Mar 29, 2010 at 11:02 AM, anatoly techtonik wrote: [..] > distutils is not a `package management` tool, because it doesn't know > anything even about installed packages, not saying anything about > dependencies. At this point, no one knows anything about installed packages at the Python level. Keeping track of installed projects is a feature done within each package managment system. And the whole purpose of PEP 376 is to define a database of what's installed, for the sake of interoperability. > > `pip` and `distribute` are unknown for a vast majority of Python > users, so if you have a perspective replacement for `easy_install` - Depending on how you call a Python user, I disagree here. Many people use pip and distribute. The first one because it has an uninstall feature among other things. The second one because it fixes some bugs of setuptools and provides Python 3 support > > For now there are two questions: > 1. Are they stable enough for the replacement of user command line > `easy_install` tool? > 2. Which one is the recommended? > > P.S. Should there be an accessible FAQ in addition to ML? This FAQ work has been started in th "HitchHicker's guide to Packaging" you can find here: http://guide.python-distribute.org [..] > No. To get something in 2.7 I would refrain from developing into this > direction for now. > >> [..] >>> So, do we need a PEP for that? How else can I know if consensus is >>> reached? Anybody is willing to elaborate on implementation? >> >> I see two paths here: >> 1 - do you want to define a general mechanism for Python to install scripts ? >> 2 - are you just suggesting to have this mechanism only for Package Managers ? > > 3 - I want current "best practice" for installing Python modules with > dependencies from PyPI to be shipped with Python 2.7 by default Again, any new code work will not happen because 2.7 is due in less than a week. Things are happening in Distutils2. Now, for the "best practice" documentation, I think the guide is the best plce to look at. [..] > PEP 376 is completely irrelevant to user side boot package proposal. It is, since it proposes an uninstall script called via -m. So having a install script called by -m is definitely its business. > My vision is that decision about having bootstrap package or not in > 2.7 should be in python-dev and specific packaging, implementation and > pip/distutils/distribute questions in distutils-sig. If the mentioned bootstrap script is about a package managment system, you should keep the discussion in distutils-SIG I think. Regards Tarek -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Mon Mar 29 11:19:39 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 29 Mar 2010 11:19:39 +0200 Subject: [Python-Dev] Bootstrap script for package management tool in Python 2.7 (Was: Re: [Distutils] At least one package management tool for 2.7) In-Reply-To: <94bdd2611003290215s72ea681dp246505b8894695@mail.gmail.com> References: <94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com> <94bdd2611003290215s72ea681dp246505b8894695@mail.gmail.com> Message-ID: <94bdd2611003290219h11f01c27sf09cd085ea20f7a8@mail.gmail.com> On Mon, Mar 29, 2010 at 11:15 AM, Tarek Ziad? wrote: [..] > Depending on how you call a Python user, I disagree here. Many people > use pip and distribute. s/how/who :) From techtonik at gmail.com Mon Mar 29 11:24:56 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 29 Mar 2010 12:24:56 +0300 Subject: [Python-Dev] [Distutils] Bootstrap script for package management tool in Python 2.7 (Was: Re: At least one package management tool for 2.7) In-Reply-To: <319e029f1003290058tf8f840bm23e454386570ea9a@mail.gmail.com> References: <319e029f1003290058tf8f840bm23e454386570ea9a@mail.gmail.com> Message-ID: 2010/3/29 Lennart Regebro : > On Mon, Mar 29, 2010 at 09:30, anatoly techtonik wrote: >> Therefore I still propose shipping bootstrap package that instruct >> user how to download and install an actual package ?management tool >> when users tries to use it. So far I know only one stable tool - >> `easy_install` - a part of `setuptools` package. > > We can't ship *A* bootstrap script until there is *A* package > management tool in the Python world. Currently there are three. :) > > In short: The tools that exist are not ready yet. Ok. How about shipping bootstrap script only for `easy_install` tool for now? -- anatoly t. From ziade.tarek at gmail.com Mon Mar 29 11:38:46 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 29 Mar 2010 11:38:46 +0200 Subject: [Python-Dev] [Distutils] Bootstrap script for package management tool in Python 2.7 (Was: Re: At least one package management tool for 2.7) In-Reply-To: References: <319e029f1003290058tf8f840bm23e454386570ea9a@mail.gmail.com> Message-ID: <94bdd2611003290238x6455c0d0xefa9afb2d2620064@mail.gmail.com> 2010/3/29 anatoly techtonik : > 2010/3/29 Lennart Regebro : >> On Mon, Mar 29, 2010 at 09:30, anatoly techtonik wrote: >>> Therefore I still propose shipping bootstrap package that instruct >>> user how to download and install an actual package ?management tool >>> when users tries to use it. So far I know only one stable tool - >>> `easy_install` - a part of `setuptools` package. >> >> We can't ship *A* bootstrap script until there is *A* package >> management tool in the Python world. Currently there are three. :) >> >> In short: The tools that exist are not ready yet. > > Ok. How about shipping bootstrap script only for `easy_install` tool for now? -1 I don't see how this will improve the current state. It will make it worse since the future of packaging for the stldib is being built in distutils2, and also in Pip. People that want to use easy_install can install it. It's very simple and doesn't require more work than what you have described through a -m call. The instructions you want to add in that bootstrap script to inform people belong to the documentation imho. And that's one of the goal of the packaging guide we are building. Another suggestion could be to add in the Python installer a notice about the guide, saying : "Hey, we are working on packaging right now. Python currently doesn't ship with a full-featured package manager, you can have a look in this guide though, for guidance" I still think you should remove python-dev from the cc list ;) Tarek -- Tarek Ziad? | http://ziade.org From ncoghlan at gmail.com Mon Mar 29 13:56:27 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 29 Mar 2010 21:56:27 +1000 Subject: [Python-Dev] Request for commit access In-Reply-To: <4BB04655.6060008@v.loewis.de> References: <20100323152748.50ECE1FD33F@kimball.webabinitio.net> <4BA94202.6060502@v.loewis.de> <4BB04655.6060008@v.loewis.de> Message-ID: <4BB0956B.5030904@gmail.com> Martin v. L?wis wrote: >> That's an ideal case, but it doesn't work, because more or less senior >> committers are already too busy. If they do not even have time to >> review issues, followup on patches - how can they monitor who reached >> the appropriate karma level? > > The practice proves to be different. In the recent additions, the new > developer had been approached by some committer, suggesting that they > apply. So the initiative actually started from a committer who thought > that the karma level was appropriate. All I'm asking for is that then > this committer takes the issues in his own hands, rather than letting > the new developer ask for himself. Agreed, since there is a bit of a commitment on the sponsor's part to help coach the new committer in the early days (most new committers don't need much coaching, but it's still good practice to have a specific existing committer looking at their work for a while, in addition to those of us that just try to review everything that goes by on the checkin list). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Mon Mar 29 13:58:41 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 29 Mar 2010 21:58:41 +1000 Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk: Doc/c-api/capsule.rst Doc/c-api/cobject.rst Doc/c-api/concrete.rst Doc/data/refcounts.dat Doc/extending/extending.rst Include/Python.h Include/cStringIO.h Include/cobject.h Include/datetime.h Include/py_curses.h Include/pycapsule.h Include/pyexpat.h Include/ucnhash.h Lib/test/test_sys.py Makefile.pre.in Misc/NEWS Modules/_ctypes/callproc.c Modules/_ctypes/cfield.c Modules/_ctypes/ctypes.h Modules/_cursesmodule.c Modules/_elementtree.c Modules/_testcapimodule.c Modules/cStringIO.c Modules/cjkcodecs/cjkcodecs.h Modules/cjkcodecs/multibytecodec.c Modules/cjkcodecs/multibytecodec.h Modules/datetimemodule.c Modules/pyexpat.c Modules/socketmodule.c Modules/socketmodule.h Modules/unicodedata.c Objects/capsule.c Objects/object.c Objects/unicodeobject.c PC/VS7.1/pythoncore.vcproj PC/VS8.0/pythoncore.vcproj PC/os2emx/python27.def PC/os2vacpp/python.def Python/compile.c Python/getargs.c In-Reply-To: <4BAFAD31.2050300@hastings.org> References: <20100325005454.B1AC5FCCD@mail.python.org> <4BAB2882.6070508@egenix.com> <4BAB9FAE.1070206@hastings.org> <4BAC7C48.3080600@egenix.com> <4BAFAD31.2050300@hastings.org> Message-ID: <4BB095F1.5090005@gmail.com> Larry Hastings wrote: > * allow the CObject API to (unsafely) dereference a capsule > * add support for the new files added by capsule to the Visual > Studio builds (fixes an unrelated second complaint about my checkin) > * add a -3 warning for calls to CObject > * add a PendingDeprecation warning to CObject The last item probably isn't necessary given the -3 warning. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Mon Mar 29 14:10:52 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 29 Mar 2010 22:10:52 +1000 Subject: [Python-Dev] Bootstrap script for package management tool in Python 2.7 (Was: Re: [Distutils] At least one package management tool for 2.7) In-Reply-To: References: Message-ID: <4BB098CC.1000708@gmail.com> anatoly techtonik wrote: > So, there won't be any package management tool shipped with Python 2.7 > and users will have to download and install `setuptools` manually as > before: Until the discussed package management tools support a robust inventory and uninstallation system that plays well with directly installed Python packages, you won't find widespread support on python-dev for endorsing any of them. Yes, the people who use them love them for good and valid reasons, but those who absolutely detest them also do so for good and valid reasons. While this is still the case, it would be highly inappropriate for python-dev to include bootstrap scripts that direct users to these systems, as they're all flawed in their current incarnations (this isn't the packaging systems' fault - the flaws are largely due to a lack of supporting infrastructure in the standard library). The distutils2 work and the various metadata PEPs that have been approved recently are all about addressing those limitations in the infrastructure. With those in place and flowing through the Python package management ecosystem, bootstrapping interoperable package management tools is likely to become a reasonable option in the future. But we aren't there yet, and won't be for 2.7 or 3.2. From an outsider's perspective, the 3.3 time frame appears to be very possible though. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ctb at msu.edu Mon Mar 29 14:30:40 2010 From: ctb at msu.edu (C. Titus Brown) Date: Mon, 29 Mar 2010 05:30:40 -0700 Subject: [Python-Dev] GSoC 2010 is on -- projects? In-Reply-To: References: <20100319023630.GC1617@idyll.org> Message-ID: <20100329123039.GA23643@idyll.org> On Mon, Mar 29, 2010 at 10:40:06AM +0300, anatoly techtonik wrote: > I would vote for allowing student work on community infrastructure > tasks. Tracker, Wiki, Web site management tools are all outdated and > everybody who cares agrees that they've seen a better tools. As long as it's programming, it's allowed by Google. So let's find a good student or two, and outline a few good projects! I do worry that that kind of work is difficult to evaluate, and requires really great communication on both sides... --titus From techtonik at gmail.com Mon Mar 29 15:45:40 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 29 Mar 2010 16:45:40 +0300 Subject: [Python-Dev] Bootstrap script for package management tool in Python 2.7 (Was: Re: [Distutils] At least one package management tool for 2.7) In-Reply-To: <94bdd2611003290215s72ea681dp246505b8894695@mail.gmail.com> References: <94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com> <94bdd2611003290215s72ea681dp246505b8894695@mail.gmail.com> Message-ID: On Mon, Mar 29, 2010 at 12:15 PM, Tarek Ziad? wrote: > [..] >> distutils is not a `package management` tool, because it doesn't know >> anything even about installed packages, not saying anything about >> dependencies. > > At this point, no one knows anything about installed packages at the > Python level. Users do not care about this, and `distutils` doesn't know this even at package level. > Keeping track of installed projects is a feature done within each > package managment system. > > And the whole purpose of PEP 376 is to define a database of what's > installed, for the sake of interoperability. That's great. When it will be ready everybody would be happy to make their package management tool compliant. >> >> `pip` and `distribute` are unknown for a vast majority of Python >> users, so if you have a perspective replacement for `easy_install` - > > Depending on how you call a Python user, I disagree here. Many people > use pip and distribute. > > The first one because it has an uninstall feature among other things. > The second one because it fixes some bugs of setuptools and provides > Python 3 support I do not mind if we can distribute three stubs, they will also serve as pointers for a better way of packaging when an ultimate tool is finally born. Me personally is willing to elaborate for `easy_install` stub in 2.7. >> >> For now there are two questions: >> 1. Are they stable enough for the replacement of user command line >> `easy_install` tool? >> 2. Which one is the recommended? >> >> P.S. Should there be an accessible FAQ in addition to ML? > > This FAQ work has been started in th "HitchHicker's guide to > Packaging" you can find here: > > http://guide.python-distribute.org I can see any FAQ. To me the FAQ is something that could be posted to distutils ML once a month to reflect current state of packaging. It should also carry version number. So anybody can comment on the FAQ, ask another question or ask to make a change. > Again, any new code work will not happen because 2.7 is due in less > than a week. Things are happening in Distutils2. That doesn't solve the problem. Bootstrap script can be written in one day. What we need is a consensus whatever this script is welcomed in 2.7 or not? Who is the person to make the decision? > Now, for the "best practice" documentation, I think the guide is the > best plce to look at. Let's refer to original user story: "I installed Python and need a quick way to install my packages on top of it." "I execute `easy_install something` as said in installation manual, but nothing/error happens." > [..] >> PEP 376 is completely irrelevant to user side boot package proposal. > > It is, since it proposes an uninstall script called via -m. So having > a install script called by -m is definitely its business. The scope of my proposal is a bootstrap script that instructs user how to install the package management tool of user's choice if this tool is not yet installed on user system (this is the case with new Python installation). If you'll come up with a better way of package management - you will update this bootstrap script with relevant information in future Python releases. Right now it is essential to get this _feature_ in Python 2.7 until 2.7 is frozen for new features. The script that shows message upon invocation and is overwritten by the version of the package it proposes to install. Is it hard to do? >> My vision is that decision about having bootstrap package or not in >> 2.7 should be in python-dev and specific packaging, implementation and >> pip/distutils/distribute questions in distutils-sig. > > If the mentioned bootstrap script is about a package managment system, > you should keep the discussion in distutils-SIG I think. If there will be no bootstrap script in 2.7, I won't have any motivation to continue discussion until 2.8 deadline is near. Of course I will be disappointed, because I will have to explain everything to 2.7 users again and again instead of letting them execute one command and follow its instructions. -- anatoly t. From ziade.tarek at gmail.com Mon Mar 29 15:50:18 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 29 Mar 2010 15:50:18 +0200 Subject: [Python-Dev] Bootstrap script for package management tool in Python 2.7 (Was: Re: [Distutils] At least one package management tool for 2.7) In-Reply-To: References: <94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com> <94bdd2611003290215s72ea681dp246505b8894695@mail.gmail.com> Message-ID: <94bdd2611003290650x27a074b5j586fdd1f05beebe2@mail.gmail.com> Anatoly, I am now answering only in Distutils-SIG. On Mon, Mar 29, 2010 at 3:45 PM, anatoly techtonik wrote: [..] From rrr at ronadam.com Mon Mar 29 15:52:39 2010 From: rrr at ronadam.com (Ron Adam) Date: Mon, 29 Mar 2010 08:52:39 -0500 Subject: [Python-Dev] Bootstrap script for package management tool in Python 2.7 (Was: Re: At least one package management tool for 2.7) In-Reply-To: References: Message-ID: <4BB0B0A7.4060405@ronadam.com> anatoly techtonik wrote: > So, there won't be any package management tool shipped with Python 2.7 > and users will have to download and install `setuptools` manually as > before: > > "search" -> "download" -> "unzip" -> "cmd" -> "cd" -> "python > setup.py install" > > > Therefore I still propose shipping bootstrap package that instruct > user how to download and install an actual package management tool > when users tries to use it. So far I know only one stable tool - > `easy_install` - a part of `setuptools` package. > > The required behavior for very basic user friendliness: > 1. user installs Python 2.7 > 2. user issues `python -m easy_install something` > 3. user gets message > 'easy_install' tool is not installed on this system. To make it > available, download and install `setuptools` package from > http://pypi.python.org/pypi/setuptools/ > > 4. the screen is paused before exit (for windows systems) > > Other design notes: > 1. if package tries to import `easy_install` module used for > bootstrap, it gets the same ImportException as if there were no > `easy_install` at all > 2. bootstrap module is overwritten by actual package when users installs it > > > So, do we need a PEP for that? How else can I know if consensus is > reached? Anybody is willing to elaborate on implementation? > > > P.S. Please be careful to reply to relevant lists An even lighter option would be to add an item to pythons 'help' feature. Currently help("PACKAGES") == help("import") It may be enough at this time to add a "PACKAGES" help entry that gives an overview of packages and hints on installing them. Then "import" can be a related help topic for "PACKAGES". Ron From regebro at gmail.com Mon Mar 29 15:57:04 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 29 Mar 2010 15:57:04 +0200 Subject: [Python-Dev] [Distutils] Bootstrap script for package management tool in Python 2.7 (Was: Re: At least one package management tool for 2.7) In-Reply-To: References: <94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com> Message-ID: <319e029f1003290657r2a15dbc7s131b83f51f731218@mail.gmail.com> On Mon, Mar 29, 2010 at 11:02, anatoly techtonik wrote: > distutils is not a `package management` tool, because it doesn't know > anything even about installed packages With that definition, there are no packaga management tools for Python. So it's going to be pretty hard to ship one with 2.7. > `pip` and `distribute` are unknown for a vast majority of Python > users, so if you have a perspective replacement for `easy_install` - > it can be said in bootstrap package message. There is no problem with > packages that require `setuptools` either - they will require > `setuptools` as dependency anyway. > > For now there are two questions: > 1. Are they stable enough for the replacement of user command line > `easy_install` tool? Yes. But that's the wrong question. The correct question is: Are they stable enough to be included in standard library. And the answer is "no". > 2. Which one is the recommended? Both. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From cournape at gmail.com Mon Mar 29 15:58:32 2010 From: cournape at gmail.com (David Cournapeau) Date: Mon, 29 Mar 2010 22:58:32 +0900 Subject: [Python-Dev] [Distutils] Bootstrap script for package management tool in Python 2.7 (Was: Re: At least one package management tool for 2.7) In-Reply-To: References: <94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com> <94bdd2611003290215s72ea681dp246505b8894695@mail.gmail.com> Message-ID: <5b8d13221003290658j4734d46atd373b0d389367d4@mail.gmail.com> On Mon, Mar 29, 2010 at 10:45 PM, anatoly techtonik wrote: > On Mon, Mar 29, 2010 at 12:15 PM, Tarek Ziad? wrote: >> [..] >>> distutils is not a `package management` tool, because it doesn't know >>> anything even about installed packages, not saying anything about >>> dependencies. >> >> At this point, no one knows anything about installed packages at the >> Python level. > > Users do not care about this, and `distutils` doesn't know this even > at package level. > >> Keeping track of installed projects is a feature done within each >> package managment system. >> >> And the whole purpose of PEP 376 is to define a database of what's >> installed, for the sake of interoperability. > > That's great. When it will be ready everybody would be happy to make > their package management tool compliant. > >>> >>> `pip` and `distribute` are unknown for a vast majority of Python >>> users, so if you have a perspective replacement for `easy_install` - >> >> Depending on how you call a Python user, I disagree here. Many people >> use pip and distribute. >> >> The first one because it has an uninstall feature among other things. >> The second one because it fixes some bugs of setuptools and provides >> Python 3 support > > I do not mind if we can distribute three stubs, they will also serve > as pointers for a better way of packaging when an ultimate tool is > finally born. Me personally is willing to elaborate for `easy_install` > stub in 2.7. > >>> >>> For now there are two questions: >>> 1. Are they stable enough for the replacement of user command line >>> `easy_install` tool? >>> 2. Which one is the recommended? >>> >>> P.S. Should there be an accessible FAQ in addition to ML? >> >> This FAQ work has been started in th "HitchHicker's guide to >> Packaging" you can find here: >> >> http://guide.python-distribute.org > > I can see any FAQ. To me the FAQ is something that could be posted to > distutils ML once a month to reflect current state of packaging. It > should also carry version number. So anybody can comment on the FAQ, > ask another question or ask to make a change. > >> Again, any new code work will not happen because 2.7 is due in less >> than a week. Things are happening in Distutils2. > > That doesn't solve the problem. Bootstrap script can be written in one > day. What we need is a consensus whatever this script is welcomed in > 2.7 or not? Who is the person to make the decision? > >> Now, for the "best practice" documentation, I think the guide is the >> best plce to look at. > > Let's refer to original user story: > "I installed Python and need a quick way to install my packages on top of it." python setup.py install works well, and has for almost a decade. If you need setuptools, you can include ez_setup.py, which does exactly what you want, without adding a hugely controversial feature to python proper. You do something like: try: import setuptools except ImportError: print "Run ez_setup.py first" .... And you're done, cheers, David From regebro at gmail.com Mon Mar 29 15:59:47 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 29 Mar 2010 15:59:47 +0200 Subject: [Python-Dev] [Distutils] Bootstrap script for package management tool in Python 2.7 (Was: Re: At least one package management tool for 2.7) In-Reply-To: References: <319e029f1003290058tf8f840bm23e454386570ea9a@mail.gmail.com> Message-ID: <319e029f1003290659j2243ef5dv473805c9df67b250@mail.gmail.com> 2010/3/29 anatoly techtonik : > Ok. How about shipping bootstrap script only for `easy_install` tool for now? Since there are many who are of the opinion that easy_install isn't very good, and pip should be used instead, that would be a bad idea. Just drop it. 2.7 will not have a package management tool or bootstrap script to one, because there is currently not one that is good enough to be the standard. People are working on making the situation good enough, this debate is wasting their time. The end. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From techtonik at gmail.com Mon Mar 29 16:05:33 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 29 Mar 2010 17:05:33 +0300 Subject: [Python-Dev] Google Groups mirror (Was: Bootstrap script for package management tool in Python 2.7) Message-ID: On Mon, Mar 29, 2010 at 4:50 PM, Tarek Ziad? wrote: > Anatoly, I am now answering only in Distutils-SIG. > > On Mon, Mar 29, 2010 at 3:45 PM, anatoly techtonik wrote: > [..] Seems like I start to hate mailing lists even more with all this message duplication and thread following nightmare. Why can't people here create a Google Groups mirror? -- anatoly t. From solipsis at pitrou.net Mon Mar 29 16:15:35 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 29 Mar 2010 14:15:35 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Google_Groups_mirror_=28Was=3A_Bootstrap_s?= =?utf-8?q?cript_for=09package_management_tool_in_Python_2=2E7=29?= References: Message-ID: anatoly techtonik gmail.com> writes: > > Seems like I start to hate mailing lists even more with all this > message duplication and thread following nightmare. Why can't people > here create a Google Groups mirror? There are already gmane mirrors, such as: http://news.gmane.org/gmane.comp.python.devel From techtonik at gmail.com Mon Mar 29 16:21:39 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 29 Mar 2010 17:21:39 +0300 Subject: [Python-Dev] Bootstrap script for package management tool in Python 2.7 (Was: Re: [Distutils] At least one package management tool for 2.7) In-Reply-To: <4BB098CC.1000708@gmail.com> References: <4BB098CC.1000708@gmail.com> Message-ID: 2010/3/29 Nick Coghlan : > anatoly techtonik wrote: >> So, there won't be any package management tool shipped with Python 2.7 >> and users will have to download and install `setuptools` manually as >> before: > > Until the discussed package management tools support a robust inventory > and uninstallation system that plays well with directly installed Python > packages, you won't find widespread support on python-dev for endorsing > any of them. This wasn't the question, but the summary. I would greatly appreciate if you could provide your feedback on the second part of my letter that started with "Therefore..." > Yes, the people who use them love them for good and valid reasons, but > those who absolutely detest them also do so for good and valid reasons. > While this is still the case, it would be highly inappropriate for > python-dev to include bootstrap scripts that direct users to these Scripts do not _direct_ users - they _help_ users, already directed by some installation instruction or tutorial, to find and install package management system they are _trying_ to use. > The distutils2 work and the various metadata PEPs that have been > approved recently are all about addressing those limitations in the > infrastructure. With those in place and flowing through the Python > package management ecosystem, bootstrapping interoperable package > management tools is likely to become a reasonable option in the future. Bootstrap tools evolve together with packaging situation. You may deprecate package management tool in your future bootstrap scripts if it is "incompatible" with this Python release. It is just user message that is a flexible as the mind of its author. > But we aren't there yet, and won't be for 2.7 or 3.2. From an outsider's > perspective, the 3.3 time frame appears to be very possible though. Just a thought about user story my customers would likely write if I shipped Python as a product: "As a user, I think Python is suxx, because it makes its users suffer for a long time from packaging disorder". -- anatoly t. From regebro at gmail.com Mon Mar 29 16:30:36 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 29 Mar 2010 16:30:36 +0200 Subject: [Python-Dev] Bootstrap script for package management tool in Python 2.7 (Was: Re: [Distutils] At least one package management tool for 2.7) In-Reply-To: References: <4BB098CC.1000708@gmail.com> Message-ID: <319e029f1003290730y1a3d3e78s2bf07c284ff2885b@mail.gmail.com> On Mon, Mar 29, 2010 at 16:21, anatoly techtonik wrote: > Just a thought about user story my customers would likely write if I > shipped Python as a product: > "As a user, I think Python is suxx, because it makes its users suffer > for a long time from packaging disorder". So fix it. Python is open source. It's created by it's users. If there is something you think it lacking, you step up and help make it better. But the packaging solution will not be ready for 2.7. It's a really hard problem. Your help is appreciated. Complaints are not. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From techtonik at gmail.com Mon Mar 29 16:32:31 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 29 Mar 2010 17:32:31 +0300 Subject: [Python-Dev] Google Groups mirror (Was: Bootstrap script for package management tool in Python 2.7) In-Reply-To: References: Message-ID: On Mon, Mar 29, 2010 at 5:15 PM, Antoine Pitrou wrote: > anatoly techtonik gmail.com> writes: >> >> Seems like I start to hate mailing lists even more with all this >> message duplication and thread following nightmare. Why can't people >> here create a Google Groups mirror? > > There are already gmane mirrors, such as: > http://news.gmane.org/gmane.comp.python.devel Do you use it yourself for thread subscription? My point is that gmane mouse clicking interface is insane. They can not fold quotations. And if you'll look attentively you won't find most of messages from ongoing discussion about bootstrap script for package management. -- anatoly t. From solipsis at pitrou.net Mon Mar 29 16:43:35 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 29 Mar 2010 16:43:35 +0200 Subject: [Python-Dev] Google Groups mirror (Was: Bootstrap script for package management tool in Python 2.7) In-Reply-To: References: Message-ID: <1269873815.3451.4.camel@localhost> > > There are already gmane mirrors, such as: > > http://news.gmane.org/gmane.comp.python.devel > > Do you use it yourself for thread subscription? I don't think there's any "thread subscription", if you mean subscribing to individual threads. > My point is that gmane mouse clicking interface is insane. They can > not fold quotations. Neither can some e-mail clients. It's up to posters to quote (that is, strip quotes) properly. Gmane also allows you to use NNTP, though, perhaps you can find an NNTP client which has the desired quoting UI :) > And if you'll look attentively you won't find > most of messages from ongoing discussion about bootstrap script for > package management. Yes, gmane seems to have some strange "duplicate suppression" algorithm which doesn't go well with cross-posting. From robert.kern at gmail.com Mon Mar 29 16:41:58 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Mar 2010 09:41:58 -0500 Subject: [Python-Dev] Why is nan != nan? In-Reply-To: <5b8d13221003282317h32b2807bm467df310fe56b285@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net> <5b8d13221003262232m11c30a9ata3b17410f3d7c53@mail.gmail.com> <5b8d13221003282317h32b2807bm467df310fe56b285@mail.gmail.com> Message-ID: On 2010-03-29 01:17 AM, David Cournapeau wrote: > On Sun, Mar 28, 2010 at 9:28 AM, Robert Kern wrote: >> On 2010-03-27 00:32 , David Cournapeau wrote: >>> >>> On Sat, Mar 27, 2010 at 8:16 AM, Raymond Hettinger >>> wrote: >>>> >>>> On Mar 26, 2010, at 2:16 PM, Xavier Morel wrote: >>>> >>>> How about raising an exception instead of creating nans in the first >>>> place, >>>> except maybe within specific contexts (so that the IEEE-754 minded can >>>> get >>>> their nans working as they currently do)? >>>> >>>> -1 >>>> The numeric community uses NaNs as placeholders in vectorized >>>> calculations. >>> >>> But is this relevant to python itself ? In Numpy, we indeed do use and >>> support NaN, but we have much more control on what happens compared to >>> python float objects. We can control whether invalid operations raises >>> an exception or not, we had isnan/isfinite for a long time, and the >>> fact that nan != nan has never been a real problem AFAIK. >> >> Nonetheless, the closer our float arrays are to Python's float type, the >> happier I will be. > > Me too, but I don't see how to reconcile this with the intent of > simplifying nan handling because they are not intuitive, which seems > to be the goal of this discussion. "Do nothing" is still on the table, I think. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Jeffrey.Pry at sig.com Mon Mar 29 17:02:13 2010 From: Jeffrey.Pry at sig.com (Pry, Jeffrey) Date: Mon, 29 Mar 2010 11:02:13 -0400 Subject: [Python-Dev] OOP On Web Message-ID: <26AD5A26C15C3948AEF795C346F7E9ED03EC949DCD@msgbal518.ds.susq.com> Hello, Nice to meet everyone! I have two files (class.py and object.py). Now, within object.py I create two objects from the class.py file. The class.py file contains the necessary code to make a connection to a FTP server. The two objects pass the credentials, etc. My question is, if I wanted to display a dropdown box on a webpage listing all instances of the object (most likely five servers), how would I do that. Would I do it at the class level, the object level, or at some other level. Forgive me if this is a somewhat basic question. Thank you for your help! Jeffrey Kevin Pry ________________________________ IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Mon Mar 29 18:25:08 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 29 Mar 2010 17:25:08 +0100 Subject: [Python-Dev] OOP On Web In-Reply-To: <26AD5A26C15C3948AEF795C346F7E9ED03EC949DCD@msgbal518.ds.susq.com> References: <26AD5A26C15C3948AEF795C346F7E9ED03EC949DCD@msgbal518.ds.susq.com> Message-ID: <4BB0D464.8090305@voidspace.org.uk> On 29/03/2010 16:02, Pry, Jeffrey wrote: > > Hello, > > Nice to meet everyone! > Hello Jeffrey, This mailing list is for the development *of* Python, not for developing *with* Python. More appropriate places to ask are the Python-tutor mailing list or the comp.lang.python newsgroup. Both of these have google and gmane web interfaces as well I believe. You can find more information at: http://www.python.org/community/lists/ All the best, Michael Foord > I have two files (class.py and object.py). Now, within object.py I > create two objects from the class.py file. The class.py file contains > the necessary code to make a connection to a FTP server. The two > objects pass the credentials, etc. My question is, if I wanted to > display a dropdown box on a webpage listing all instances of the > object (most likely five servers), how would I do that. Would I do it > at the class level, the object level, or at some other level. Forgive > me if this is a somewhat basic question. > > Thank you for your help! > > Jeffrey Kevin Pry > > > ------------------------------------------------------------------------ > IMPORTANT: The information contained in this email and/or its > attachments is confidential. If you are not the intended recipient, > please notify the sender immediately by reply and immediately delete > this message and all its attachments. Any review, use, reproduction, > disclosure or dissemination of this message or any attachment by an > unintended recipient is strictly prohibited. Neither this message nor > any attachment is intended as or should be construed as an offer, > solicitation or recommendation to buy or sell any security or other > financial instrument. Neither the sender, his or her employer nor any > of their respective affiliates makes any warranties as to the > completeness or accuracy of any of the information contained herein or > that this message or any of its attachments is free of viruses. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phd.pp.ru Mon Mar 29 18:28:37 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Mon, 29 Mar 2010 20:28:37 +0400 Subject: [Python-Dev] OOP On Web In-Reply-To: <26AD5A26C15C3948AEF795C346F7E9ED03EC949DCD@msgbal518.ds.susq.com> References: <26AD5A26C15C3948AEF795C346F7E9ED03EC949DCD@msgbal518.ds.susq.com> Message-ID: <20100329162837.GA931@phd.pp.ru> Hello. We'are sorry but we cannot help you. This mailing list is to work on developing Python (fixing bugs and adding new features to Python itself); if you're having problems using Python, please find another forum. Probably python-list (comp.lang.python) news group/mailing list is the best place. See http://www.python.org/community/lists/ for other lists/news groups/fora. On Mon, Mar 29, 2010 at 11:02:13AM -0400, Pry, Jeffrey wrote: Thank you for understanding. > Hello, > > Nice to meet everyone! > > I have two files (class.py and object.py). Now, within object.py I create two objects from the class.py file. The class.py file contains the necessary code to make a connection to a FTP server. The two objects pass the credentials, etc. My question is, if I wanted to display a dropdown box on a webpage listing all instances of the object (most likely five servers), how would I do that. Would I do it at the class level, the object level, or at some other level. Forgive me if this is a somewhat basic question. > > Thank you for your help! > > Jeffrey Kevin Pry > > ________________________________ > IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/phd%40phd.pp.ru Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From michael at voidspace.org.uk Mon Mar 29 19:16:50 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Mon, 29 Mar 2010 18:16:50 +0100 Subject: [Python-Dev] WeakSet in Python 2.7 Message-ID: <4BB0E082.8040602@voidspace.org.uk> Hello all, Python 3 includes a WeakSet implementation. Any objections to me backporting this to 2.7? http://docs.python.org/py3k/library/weakref.html#weakref.WeakSet I needed this in unittest and used a WeakKeyDictionary instead. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From larry at hastings.org Mon Mar 29 19:26:05 2010 From: larry at hastings.org (Larry Hastings) Date: Mon, 29 Mar 2010 10:26:05 -0700 Subject: [Python-Dev] Bootstrap script for package management tool in Python 2.7 (Was: Re: [Distutils] At least one package management tool for 2.7) In-Reply-To: References: Message-ID: <4BB0E2AD.6020407@hastings.org> anatoly techtonik wrote: > So, there won't be any package management tool shipped with Python 2.7 > and users will have to download and install `setuptools` manually as > before: > > "search" -> "download" -> "unzip" -> "cmd" -> "cd" -> "python > setup.py install" > > > Therefore I still propose shipping bootstrap package that instruct > user how to download and install an actual package management tool > when users tries to use it. For what it's worth, Guido prototyped something similar in March of 2008, but his was an actual bootstrapping tool for package management: http://mail.python.org/pipermail/python-dev/2008-March/077837.html His tool knew how to download a tar file, untar it, and run "python setup.py install" on it. No version numbers, no dependency management, simple enough that it should be easy to get right. Only appropriate for bootstrapping into a real package management tool. The thread ends with him saying "I don't have time to deal with this further this week", and I dunno, maybe it just fell off the radar? I'd been thinking about resurrecting the discussion but I didn't have time either. I fear it is too late for 2.7, //larry// From skip at pobox.com Mon Mar 29 19:31:17 2010 From: skip at pobox.com (skip at pobox.com) Date: Mon, 29 Mar 2010 12:31:17 -0500 Subject: [Python-Dev] Google Groups mirror (Was: Bootstrap script for package management tool in Python 2.7) In-Reply-To: References: Message-ID: <19376.58341.90376.949602@montanaro.dyndns.org> anatoly> Why can't people here create a Google Groups mirror? Whoever "people here" are could, I suppose. I wouldn't mind a read-only mirror but would not be a fan of a two-way mirror because Google seems to do an awful job of weeding out spam. In any case, the correct place to submit such a request would be postmaster at python.org. Skip From daniel at stutzbachenterprises.com Mon Mar 29 20:14:13 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Mon, 29 Mar 2010 13:14:13 -0500 Subject: [Python-Dev] WeakSet in Python 2.7 In-Reply-To: <4BB0E082.8040602@voidspace.org.uk> References: <4BB0E082.8040602@voidspace.org.uk> Message-ID: On Mon, Mar 29, 2010 at 12:16 PM, Michael Foord wrote: > Python 3 includes a WeakSet implementation. Any objections to me > backporting this to 2.7? > > http://docs.python.org/py3k/library/weakref.html#weakref.WeakSet > Backporting WeakSet would also make it possible to backport the fix for this reference leak: http://bugs.python.org/issue2521 -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Mon Mar 29 21:20:11 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 29 Mar 2010 20:20:11 +0100 Subject: [Python-Dev] WeakSet in Python 2.7 In-Reply-To: <4BB0E082.8040602@voidspace.org.uk> References: <4BB0E082.8040602@voidspace.org.uk> Message-ID: <4BB0FD6B.6080104@voidspace.org.uk> On 29/03/2010 18:16, Michael Foord wrote: > Hello all, > > Python 3 includes a WeakSet implementation. Any objections to me > backporting this to 2.7? > > http://docs.python.org/py3k/library/weakref.html#weakref.WeakSet > > I needed this in unittest and used a WeakKeyDictionary instead. Hmmm... I was under the mistaken impression that this was implemented in pure-Python. It's in C so beyond my threshold for simple porting I'm afraid. Still if someone else is up for it... All the best, Michael > > Michael > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From michael at voidspace.org.uk Mon Mar 29 21:21:04 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Mon, 29 Mar 2010 20:21:04 +0100 Subject: [Python-Dev] WeakSet in Python 2.7 In-Reply-To: References: <4BB0E082.8040602@voidspace.org.uk> Message-ID: <4BB0FDA0.1050207@voidspace.org.uk> On 29/03/2010 19:14, Daniel Stutzbach wrote: > On Mon, Mar 29, 2010 at 12:16 PM, Michael Foord > > wrote: > > Python 3 includes a WeakSet implementation. Any objections to me > backporting this to 2.7? > > http://docs.python.org/py3k/library/weakref.html#weakref.WeakSet > > > Backporting WeakSet would also make it possible to backport the fix > for this reference leak: > > http://bugs.python.org/issue2521 It should be possible to fix it with a WeakKeyDictionary instead of WeakSet. All the best, Michael Foord > -- > Daniel Stutzbach, Ph.D. > President, Stutzbach Enterprises, LLC > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel at stutzbachenterprises.com Mon Mar 29 21:46:10 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Mon, 29 Mar 2010 14:46:10 -0500 Subject: [Python-Dev] WeakSet in Python 2.7 In-Reply-To: <4BB0FDA0.1050207@voidspace.org.uk> References: <4BB0E082.8040602@voidspace.org.uk> <4BB0FDA0.1050207@voidspace.org.uk> Message-ID: On Mon, Mar 29, 2010 at 2:21 PM, Michael Foord wrote: > It should be possible to fix it with a WeakKeyDictionary instead of > WeakSet. > True. I should have said "Backporting WeakSet would make it *easier* to backport the fix ..." :-) -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at voidspace.org.uk Mon Mar 29 21:46:52 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Mon, 29 Mar 2010 20:46:52 +0100 Subject: [Python-Dev] WeakSet in Python 2.7 In-Reply-To: <4BB0FD6B.6080104@voidspace.org.uk> References: <4BB0E082.8040602@voidspace.org.uk> <4BB0FD6B.6080104@voidspace.org.uk> Message-ID: <4BB103AC.7090508@voidspace.org.uk> On 29/03/2010 20:20, Michael Foord wrote: > On 29/03/2010 18:16, Michael Foord wrote: >> Hello all, >> >> Python 3 includes a WeakSet implementation. Any objections to me >> backporting this to 2.7? >> >> http://docs.python.org/py3k/library/weakref.html#weakref.WeakSet >> >> I needed this in unittest and used a WeakKeyDictionary instead. > > Hmmm... I was under the mistaken impression that this was implemented > in pure-Python. It's in C so beyond my threshold for simple porting > I'm afraid. Still if someone else is up for it... No, my mistake - it *is* implemented in Python. I'm porting it now. Michael > > All the best, > > Michael > >> >> Michael >> > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From martin at v.loewis.de Mon Mar 29 22:06:20 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 29 Mar 2010 22:06:20 +0200 Subject: [Python-Dev] [Distutils] Bootstrap script for package management tool in Python 2.7 (Was: Re: At least one package management tool for 2.7) In-Reply-To: References: Message-ID: <4BB1083C.30509@v.loewis.de> anatoly techtonik wrote: > So, there won't be any package management tool shipped with Python 2.7 > and users will have to download and install `setuptools` manually as > before: [...] > So, do we need a PEP for that? How else can I know if consensus is > reached? Anybody is willing to elaborate on implementation? Notice that the first beta release of Python 2.7 will be made on April 3. Any new feature that you propose must be committed by then. I suggest focusing on Python 3.2 instead. Regards, Martin From fuzzyman at voidspace.org.uk Mon Mar 29 22:43:01 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 29 Mar 2010 21:43:01 +0100 Subject: [Python-Dev] WeakSet in Python 2.7 In-Reply-To: References: <4BB0E082.8040602@voidspace.org.uk> <4BB0FDA0.1050207@voidspace.org.uk> Message-ID: <4BB110D5.7090802@voidspace.org.uk> On 29/03/2010 20:46, Daniel Stutzbach wrote: > On Mon, Mar 29, 2010 at 2:21 PM, Michael Foord > > wrote: > > It should be possible to fix it with a WeakKeyDictionary instead > of WeakSet. > > > True. I should have said "Backporting WeakSet would make it *easier* > to backport the fix ..." :-) WeakSet is now in. Michael > -- > Daniel Stutzbach, Ph.D. > President, Stutzbach Enterprises, LLC > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ianb at colorstudy.com Mon Mar 29 23:45:00 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 29 Mar 2010 15:45:00 -0600 Subject: [Python-Dev] Bootstrap script for package management tool in Python 2.7 (Was: Re: [Distutils] At least one package management tool for 2.7) In-Reply-To: <4BB0E2AD.6020407@hastings.org> References: <4BB0E2AD.6020407@hastings.org> Message-ID: On Mon, Mar 29, 2010 at 11:26 AM, Larry Hastings wrote: > anatoly techtonik wrote: > >> So, there won't be any package management tool shipped with Python 2.7 >> and users will have to download and install `setuptools` manually as >> before: >> >> "search" -> "download" -> "unzip" -> "cmd" -> "cd" -> "python >> setup.py install" >> >> >> Therefore I still propose shipping bootstrap package that instruct >> user how to download and install an actual package management tool >> when users tries to use it. >> > > For what it's worth, Guido prototyped something similar in March of 2008, > but his was an actual bootstrapping tool for package management: > > http://mail.python.org/pipermail/python-dev/2008-March/077837.html > > His tool knew how to download a tar file, untar it, and run "python > setup.py install" on it. No version numbers, no dependency management, > simple enough that it should be easy to get right. Only appropriate for > bootstrapping into a real package management tool. > > The thread ends with him saying "I don't have time to deal with this > further this week", and I dunno, maybe it just fell off the radar? I'd been > thinking about resurrecting the discussion but I didn't have time either. > I would consider this bootstrap to be quite workable, though I would add that any extra option to the bootstrap script should be passed to setup.py install, and the download should be cached (so you can do -h and not have to re-download the package once you figure out the extra options -- at least a --user option is reasonable here for people without root). Specifically targeting this bootstrap for tools like pip and virtualenv is no problem. I think looking around PyPI etc is kind of more than I'd bother with. Those things change, this bootstrap code won't change, it could cause unnecessary future pain. Maybe (*maybe*) it could look in http://pypi.python.org/well-known-packages/PACKAGE_NAME and so we can have it install a certain small number of things quickly that way -- if the URL it looks to is targeted only for the bootstrap script itself then we don't have to worry about compatibility problems as much. Oh... then i can think of a half dozen other options it could take, and then it becomes an installer. Blech. OK, I'd be willing to cut off the options at --user (which I think is a minimum... maybe --prefix too), and maybe some simple package detection so people could write "python -m boostrap Setuptools --user" -- entirely based on some well-known URL baked into bootstrap.py, where the URL is independent of any other service (and so is least likely to cause future problems or ambiguities). An advantage to this kind of bootstrapper is that as future packaging systems are developed there's a clear way to get started with them, without prematurely baking anything in to Python. -- Ian Bicking | http://blog.ianbicking.org | http://twitter.com/ianbicking -------------- next part -------------- An HTML attachment was scrubbed... URL: From turturici.valerio at gmail.com Tue Mar 30 01:06:13 2010 From: turturici.valerio at gmail.com (Valerio Turturici) Date: Tue, 30 Mar 2010 01:06:13 +0200 Subject: [Python-Dev] Contribute Message-ID: <4BB13265.9020508@gmail.com> Hello everyone, my name's Valerio, i'm italian and i study computer science at University of Pisa. I'm honored to speak and learn from people like you. My knowledge of C language is not very advanced, but i know Python well enough. I really like to contribute, i can learn and improve a lot. This is the first time that i participate in a serious, large and complex project. Have you any advice for me? :) From benjamin at python.org Tue Mar 30 00:11:06 2010 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 29 Mar 2010 16:11:06 -0600 Subject: [Python-Dev] Contribute In-Reply-To: <4BB13265.9020508@gmail.com> References: <4BB13265.9020508@gmail.com> Message-ID: <1afaf6161003291511u500063f9wc87263610ea90e14@mail.gmail.com> 2010/3/29 Valerio Turturici : > Hello everyone, > my name's Valerio, i'm italian and i study computer science at University of > Pisa. I'm honored to speak and learn from people like you. > My knowledge of C language is not very advanced, but i know Python well > enough. I really like to contribute, i can learn and improve a lot. This is > the first time that i participate in a serious, large and complex project. > > Have you any advice for me? :) Pick a bug and provide a patch. -- Regards, Benjamin From ncoghlan at gmail.com Tue Mar 30 00:24:41 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 30 Mar 2010 08:24:41 +1000 Subject: [Python-Dev] Contribute In-Reply-To: <4BB13265.9020508@gmail.com> References: <4BB13265.9020508@gmail.com> Message-ID: <4BB128A9.6010704@gmail.com> Valerio Turturici wrote: > Hello everyone, > my name's Valerio, i'm italian and i study computer science at > University of Pisa. I'm honored to speak and learn from people like you. > My knowledge of C language is not very advanced, but i know Python well > enough. I really like to contribute, i can learn and improve a lot. This > is the first time that i participate in a serious, large and complex > project. > > Have you any advice for me? :) Head over to http://bugs.python.org and look for items with the "easy" tag. Alternatively, look for items with attached patches and download, apply and test them out to see if they work as advertised. I think there's a page on the wiki with suggestions on how to get started, but I don't have a link handy - hopefully someone else will chime in with it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From fuzzyman at voidspace.org.uk Tue Mar 30 00:27:38 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 29 Mar 2010 23:27:38 +0100 Subject: [Python-Dev] Contribute In-Reply-To: <4BB13265.9020508@gmail.com> References: <4BB13265.9020508@gmail.com> Message-ID: <4BB1295A.7030108@voidspace.org.uk> On 30/03/2010 00:06, Valerio Turturici wrote: > Hello everyone, > my name's Valerio, i'm italian and i study computer science at > University of Pisa. I'm honored to speak and learn from people like you. > My knowledge of C language is not very advanced, but i know Python > well enough. I really like to contribute, i can learn and improve a > lot. This is the first time that i participate in a serious, large and > complex project. > > Have you any advice for me? :) The #python-dev IRC channel on freenode is a useful place to hangout if you have realtime questions. Michael > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From skip at pobox.com Tue Mar 30 00:30:24 2010 From: skip at pobox.com (skip at pobox.com) Date: Mon, 29 Mar 2010 17:30:24 -0500 Subject: [Python-Dev] Contribute In-Reply-To: <4BB128A9.6010704@gmail.com> References: <4BB13265.9020508@gmail.com> <4BB128A9.6010704@gmail.com> Message-ID: <19377.10752.20195.712556@montanaro.dyndns.org> Nick> I think there's a page on the wiki with suggestions on how to get Nick> started, but I don't have a link handy - hopefully someone else Nick> will chime in with it. :ring: :ring: :ring: http://www.python.org/dev/contributing/ Skip From matthew at matthewwilkes.co.uk Tue Mar 30 00:36:23 2010 From: matthew at matthewwilkes.co.uk (Matthew Wilkes) Date: Mon, 29 Mar 2010 23:36:23 +0100 Subject: [Python-Dev] Contribute In-Reply-To: <4BB13265.9020508@gmail.com> References: <4BB13265.9020508@gmail.com> Message-ID: <791240A6-C179-4685-AFE3-6BCFC5276219@matthewwilkes.co.uk> On 2010-03-30, at 0006, Valerio Turturici wrote: > Have you any advice for me? :) Also, have a look at Google's Summer of Code programme. It is a good way to get involved with a large project. Matthw From anand.shashwat at gmail.com Tue Mar 30 00:36:35 2010 From: anand.shashwat at gmail.com (Shashwat Anand) Date: Tue, 30 Mar 2010 04:06:35 +0530 Subject: [Python-Dev] Contribute In-Reply-To: <19377.10752.20195.712556@montanaro.dyndns.org> References: <4BB13265.9020508@gmail.com> <4BB128A9.6010704@gmail.com> <19377.10752.20195.712556@montanaro.dyndns.org> Message-ID: Python should have something like gnome-love ( http://live.gnome.org/GnomeLove ). Some bugs on bugzilla is tagged with gnome-love which are fairly easy and especially created to help dive in new developers. On Tue, Mar 30, 2010 at 4:00 AM, wrote: > > Nick> I think there's a page on the wiki with suggestions on how to get > Nick> started, but I don't have a link handy - hopefully someone else > Nick> will chime in with it. > > :ring: :ring: :ring: > > http://www.python.org/dev/contributing/ > > Skip > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/anand.shashwat%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Tue Mar 30 01:20:01 2010 From: skip at pobox.com (skip at pobox.com) Date: Mon, 29 Mar 2010 18:20:01 -0500 Subject: [Python-Dev] Contribute In-Reply-To: References: <4BB13265.9020508@gmail.com> <4BB128A9.6010704@gmail.com> <19377.10752.20195.712556@montanaro.dyndns.org> Message-ID: <19377.13729.368146.37884@montanaro.dyndns.org> Shashwat> Python should have something like gnome-love ( Shashwat> http://live.gnome.org/GnomeLove ). Some bugs on bugzilla is Shashwat> tagged with gnome-love which are fairly easy and especially Shashwat> created to help dive in new developers. I don't know what gnome love is (and would be hesitant to click on any such links, especially at work) but on bugs.python.org such items are tagged "easy". Right on the front page is a link with the label "Show Easy". Skip From techtonik at gmail.com Wed Mar 31 09:14:56 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 31 Mar 2010 10:14:56 +0300 Subject: [Python-Dev] Contribute In-Reply-To: <19377.13729.368146.37884@montanaro.dyndns.org> References: <4BB13265.9020508@gmail.com> <4BB128A9.6010704@gmail.com> <19377.10752.20195.712556@montanaro.dyndns.org> <19377.13729.368146.37884@montanaro.dyndns.org> Message-ID: On Tue, Mar 30, 2010 at 2:20 AM, wrote: > > ? ?Shashwat> Python should have something like gnome-love ( > ? ?Shashwat> http://live.gnome.org/GnomeLove ). Some bugs on bugzilla is > ? ?Shashwat> tagged with gnome-love which are fairly easy and especially > ? ?Shashwat> created to help dive in new developers. > > I don't know what gnome love is (and would be hesitant to click on any such > links, especially at work) but on bugs.python.org such items are tagged > "easy". ?Right on the front page is a link with the label "Show Easy". The difference between "gnome-love" and "easy" tags is that the latter can only be flagged by code devs (who usually do not have much time to review bugs). -- anatoly t. From techtonik at gmail.com Wed Mar 31 10:06:34 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 31 Mar 2010 11:06:34 +0300 Subject: [Python-Dev] GSoC 2010 is on -- projects? In-Reply-To: <20100329123039.GA23643@idyll.org> References: <20100319023630.GC1617@idyll.org> <20100329123039.GA23643@idyll.org> Message-ID: On Mon, Mar 29, 2010 at 3:30 PM, C. Titus Brown wrote: > >> I would vote for allowing student work on community infrastructure >> tasks. Tracker, Wiki, Web site management tools are all outdated and >> everybody who cares agrees that they've seen a better tools. > > As long as it's programming, it's allowed by Google. ?So let's find a good > student or two, and outline a few good projects! > > I do worry that that kind of work is difficult to evaluate, and requires really > great communication on both sides... First we need to compile a list of things to do into one big list. I see the major problem that there is no "dashboard" that gives an overview of available/supported community services and their status. Services that are parts of python.org and those that linked and often used, but not parts. Status of service is the amount of opened/languishing bugs/enhancements. Some services don't have trackers at all. For example, infrastructure proposals, web site patches is nowhere to track. Service on a dashboard should be accompanied by contact points info. Service should list location of primary repository and mirrors. Feature creep: - Service can be monitored for online/offline status. - Service may have maintenance scripts, which community members can see, inspect scheduled times for the next run and results of execution. - Service may publish stats for further analysis. After dashboard is ready, it would be nice to unify all Services: 1. Add OpenID/Google support 2. Add searchable Google Groups mirrors 3. Add Google search form to Mailman web interface for hosted groups 4. Register services search with Google (see attach) Then we need to find a contact point - a person who could be most helpful for each service and help other students with other specific tasks. I am not such person for neither service, but I am interested to help push progress forward. -- anatoly t. -------------- next part -------------- A non-text attachment was scrubbed... Name: search_search_form.png Type: image/png Size: 16969 bytes Desc: not available URL: From stephen at xemacs.org Wed Mar 31 10:41:47 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 31 Mar 2010 17:41:47 +0900 Subject: [Python-Dev] GSoC 2010 is on -- projects? In-Reply-To: References: <20100319023630.GC1617@idyll.org> Message-ID: <87wrwssx10.fsf@uwakimon.sk.tsukuba.ac.jp> anatoly techtonik writes: > I would vote for allowing student work on community infrastructure > tasks. Tracker, Wiki, Web site management tools are all outdated and > everybody who cares agrees that they've seen a better tools. I've also seen *much* worse tools in actual use. You don't have to look any farther than macports.org and GNU Savannah for examples of implementations of individual tools that are much worse, and overall usability that is clearly worse, than Python's environment. Over and over again I have seen short-term volunteers pick the best- reputed software of the day, run out of steam just getting the old data ported over to the new host software, leaving behind a disgrace. (This is what happened to MacPorts Trac, it would seem -- it took 4 years for them to get an explanation of how to search for bugs on a given port on the top page of the issue tracker, and you still have to type queries like "?PORT=python26" by hand in the browser address field![1]) In other cases, the old data is never successfully ported at all (a common way to migrate from one VCS to another). So while improving the infrastructure is clearly a good thing, it is not a good idea to have people without *long-term* commitment to maintenance *changing the tools themselves*. I would recommend changing the tools only if *current* maintainers are either planning to step down (and so we face the problem of how to support maintenance in the future in any case), or are willing to supervise (ie, the people who will come back and fix problems in the future find the proposed "improvements" to be real improvements). Eg, MvL should be intimately involved in any move to use different software for the tracker. If the GSoC student(s) is (are) willing to work within the constraints of the existing software (ie, incremental improvements), then the constraints on mentors could be substantially relaxed to any of the senior folks who have recently contributed in those areas. Footnotes: [1] Actually, it may be even worse than it looks -- IIUC, macosforge has paid staff! From ctb at msu.edu Wed Mar 31 15:29:11 2010 From: ctb at msu.edu (C. Titus Brown) Date: Wed, 31 Mar 2010 06:29:11 -0700 Subject: [Python-Dev] GSoC 2010 is on -- projects? In-Reply-To: <87wrwssx10.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20100319023630.GC1617@idyll.org> <87wrwssx10.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20100331132911.GA8170@idyll.org> On Wed, Mar 31, 2010 at 05:41:47PM +0900, Stephen J. Turnbull wrote: > anatoly techtonik writes: > > > I would vote for allowing student work on community infrastructure > > tasks. Tracker, Wiki, Web site management tools are all outdated and > > everybody who cares agrees that they've seen a better tools. [ munch ] > I would recommend changing the tools only if *current* maintainers are > either planning to step down (and so we face the problem of how to > support maintenance in the future in any case), or are willing to > supervise (ie, the people who will come back and fix problems in the > future find the proposed "improvements" to be real improvements). Eg, > MvL should be intimately involved in any move to use different > software for the tracker. If the GSoC student(s) is (are) willing to > work within the constraints of the existing software (ie, incremental > improvements), then the constraints on mentors could be substantially > relaxed to any of the senior folks who have recently contributed in > those areas. Agreed. It'd be great to have students work on prototyping improvements, with an eye to making changes that can then be evaluated in terms of maintainability, extensibility, etc. Having them actually change the infrastructure itself seems to me like a bad idea :) --titus -- C. Titus Brown, ctb at msu.edu From ezio.melotti at gmail.com Wed Mar 31 16:58:59 2010 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Wed, 31 Mar 2010 17:58:59 +0300 Subject: [Python-Dev] Add -3 and -Wd to the buildbots Message-ID: <4BB36333.6020806@gmail.com> Hi, now that the py3k warnings are fixed (see http://bugs.python.org/issue7092) we should run the tests on the trunk buildbots with the -3 flags, in order to check if new warnings are introduced and possibly to uncover other ones. It might be a good idea to add the -Wd flag too, both on trunk and py3k. -Ezio From screwdriver at lxnt.info Wed Mar 31 17:23:27 2010 From: screwdriver at lxnt.info (Alexander Sabourenkov) Date: Wed, 31 Mar 2010 19:23:27 +0400 Subject: [Python-Dev] Replacing threads with swapcontext() Message-ID: <4BB368EF.8010206@lxnt.info> Hello. I have went and replaced Python threads with explicitly-scheduled coroutines using swapcontext() family of functions (basically out of frustration with the greenlet). I have two questions. 1. Is it as dangerous to switch C stacks and not switch PyThreadState as I suspect? In particular, how tightly is PyThreadState::frame coupled to current C stack? 2. How do you all look at merging my work as another threading-related configure switch? It's only another thread_something.h file and a small patch to threadmodule.c. Project is at http://code.google.com/p/coev/ -- ./lxnt From brett at python.org Wed Mar 31 20:08:00 2010 From: brett at python.org (Brett Cannon) Date: Wed, 31 Mar 2010 11:08:00 -0700 Subject: [Python-Dev] Add -3 and -Wd to the buildbots In-Reply-To: <4BB36333.6020806@gmail.com> References: <4BB36333.6020806@gmail.com> Message-ID: On Wed, Mar 31, 2010 at 07:58, Ezio Melotti wrote: > Hi, > now that the py3k warnings are fixed (see http://bugs.python.org/issue7092) > we should run the tests on the trunk buildbots with the -3 flags, in order > to check if new warnings are introduced and possibly to uncover other ones. > It might be a good idea to add the -Wd flag too, both on trunk and py3k. > +1 from me. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From florent.xicluna at gmail.com Wed Mar 31 22:03:52 2010 From: florent.xicluna at gmail.com (Florent Xicluna) Date: Wed, 31 Mar 2010 22:03:52 +0200 Subject: [Python-Dev] Add -3 and -Wd to the buildbots In-Reply-To: References: <4BB36333.6020806@gmail.com> Message-ID: 2010/3/31 Brett Cannon: > On Wed, Mar 31, 2010 at 07:58, Ezio Melotti wrote: >> Hi, >> now that the py3k warnings are fixed (see >> http://bugs.python.org/issue7092) we should run the tests on the trunk >> buildbots with the -3 flags, in order to check if new warnings are >> introduced and possibly to uncover other ones. >> It might be a good idea to add the -Wd flag too, both on trunk and py3k. > > +1 from me. I agree, it could help to catch unexpected warnings or bugs earlier in the process. -- Florent From martin at v.loewis.de Wed Mar 31 23:05:56 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 31 Mar 2010 23:05:56 +0200 Subject: [Python-Dev] Add -3 and -Wd to the buildbots In-Reply-To: <4BB36333.6020806@gmail.com> References: <4BB36333.6020806@gmail.com> Message-ID: <4BB3B934.4030003@v.loewis.de> > now that the py3k warnings are fixed (see > http://bugs.python.org/issue7092) we should run the tests on the trunk > buildbots with the -3 flags, in order to check if new warnings are > introduced and possibly to uncover other ones. > It might be a good idea to add the -Wd flag too, both on trunk and py3k. Not sure what you are asking for. If you want confirmation that this change is desirable, it seems you got that already. If you are asking for someone to make that change - go ahead and make it yourself. You just need to change the buildbottest target. Regards, Martin From ncoghlan at gmail.com Wed Mar 31 23:53:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 01 Apr 2010 07:53:50 +1000 Subject: [Python-Dev] Replacing threads with swapcontext() In-Reply-To: <4BB368EF.8010206@lxnt.info> References: <4BB368EF.8010206@lxnt.info> Message-ID: <4BB3C46E.8000507@gmail.com> Alexander Sabourenkov wrote: > Hello. > > I have went and replaced Python threads with explicitly-scheduled > coroutines using swapcontext() family of functions (basically out of > frustration with the greenlet). > > I have two questions. > > 1. Is it as dangerous to switch C stacks and not switch PyThreadState as > I suspect? In particular, how tightly is PyThreadState::frame coupled to > current C stack? Off the top of my head, recursion limit checks will definitely go haywire, and I would expect threading.local() and exception handling to explode at some point (although the latter depends on what you mean by "switching C stacks" - if you really are just fiddling the stack pointer without affecting the thread local heap storage mechanisms, the latter two should be fine. Still doesn't sound like a good idea though). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia ---------------------------------------------------------------