From talin at acm.org Wed Nov 1 04:26:47 2006 From: talin at acm.org (Talin) Date: Tue, 31 Oct 2006 19:26:47 -0800 Subject: [Python-3000] Path Reform: Get the ball rolling Message-ID: <454813F7.8040901@acm.org> The thread on python-dev with respect to PEP 355 seems to have died out, so I thought it might be a good time to move things over here. What I'd like to do is to start by examining the most fundamental assumptions about refactoring os.path and the various other path-related functions. Before we even begin to write a new PEP, I want to at least make sure that consensus is even possible - even though we may not all be on the same page (to use a tired metaphor), we're at least reading from the same book :) Let me break this down into a number of questions: 1) Does os.path need to be refactored at all? I think most people agree that os.path isn't perfect, but IMHO it isn't terrible either - you can get work done with it, and its not really that hard to learn. From my own experience, I've often run into cases where I overlooked some critical feature of the standard library, and ended up looking foolish after posting my question on the python mailing lists, only to discover that the answer was there all along (kind of like Dorothy's red slippers in that way.) But I've not yet had an occasion, as far as I can recall, where the os.path module was the culprit. 2) Putting aside for the moment the issue of syntactic sugar, convenience, code beauty, and ease of learning, is there anything that the existing os.path *won't do* that we desperately need it to do? 3) Assuming that the answer to #1 is "yes", the next question is: "evolution or revolution?" Do we want something that looks 95%, 80%, or 0% the same as the existing library? Is this just a case of moving around a few stray functions, or is are we going to take and axe and give os.path a reprogramming job it will never forget? 4) My third question is: Who are we going to steal our ideas from? Boost, Java, C# and others - all are worthy of being the, ahem, target of our inspiration. Or we have some alternative that's so cool that it makes sense to "Think Different(ly)"? 5) Must there be one ring to rule them all? I suggested earlier that we might have a "low-level" and a "high-level" API, one built on top of the other. Is this a good idea, or a non-starter? OK lets start with that and see what happens. -- Talin From mbk.lists at gmail.com Wed Nov 1 06:38:47 2006 From: mbk.lists at gmail.com (Mike Krell) Date: Tue, 31 Oct 2006 22:38:47 -0700 Subject: [Python-3000] Alternatives to 'outer' In-Reply-To: <00b101c6fb23$ba0bf210$0301010a@VIMES> References: <00b101c6fb23$ba0bf210$0301010a@VIMES> Message-ID: > Is anybody working [on a] PEP? This has gone unanswered for several days, so I wanted to mention that Ka-Ping Yee said in an off-list email from mid-October that he is indeed working on a PEP for this, but it would take a while due to other priorities. Mike From sluggoster at gmail.com Wed Nov 1 07:22:29 2006 From: sluggoster at gmail.com (Mike Orr) Date: Tue, 31 Oct 2006 22:22:29 -0800 Subject: [Python-3000] Path Reform: Get the ball rolling Message-ID: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> Talin wrote: > 1) Does os.path need to be refactored at all? Yes. Functions are scattered arbitrarily across six modules: os, os.path, shutil, stat, glob, fnmatch. You have to search through five scattered doc pages in the Python library to find your function, plus the os module doc is split into five sections. You may think 'shlutil' has to do with shells, not paths. shutil.copy2 is riduculously named: what's so "2" about it? Why is 'split' in os.path but 'stat' and 'mkdir' and 'remove' are in os? Don't they all operate on paths? The lack of method chaning means you have to use nested functions, which must be read "inside out" rather than left-to-right like paths normally go. Say you want to add the absolute path of "../../lib" to the Python path in a platform-independent manner, relative to an absolute path (__file__): # Assume __file__ is "/toplevel/app1/bin/main_program.py". # Result is "/toplevel/app1/lib". p = os.path.join(os.path.dirname(os.path.dirname(__file__)), "lib") PEP 355 proposes a much easier-to-read: # The Path object emulates "/toplevel/app1/bin/main_program.py". p = Path(__file__).parent.parent.join("lib") Noam Raphael's directory-component object would make this even more straightforward: # The Path object emulates ("/", "toplevel", "app1", "bin", "main_program.py") p = Path(__file__)[:-2] + "lib" Stat handling has grown cruft over the years. To check the modify time of a file: os.path.getmtime("/FOO") os.stat("/FOO").st_mtime os.stat("/FOO")[stat.ST_MTIME] # List subscript, deprecated usage. If you want to check whether a file is a type for which there is no os.path.is*() method: stat.S_ISSOCK( os.stat("/FOO").st_mode ) # Is the file a socket? Compare to the directory-component proposal: Path("/foo").stat().issock os.path functions are too low-level. Say you want to recursively delete a path you're about to overwrite, no matter whether it exists or is a file or directory. You can't do it in one line of code, darn, you gotta write this function or inline the code everywhere you use it: def purge(p): if os.path.isdir(p): shutil.rmtree(p) # Raises error if nonexistent or not a directory. elif os.path.exists(): # isfile follows symlinks and returns False for special files, so it's # not a reliable guide of whether we can call os.remove. os.remove(p) # Raises error if nonexistent or a directory. if os.path.isfile(p): # Includes all symlinks. os.remove(p) > 2) is there anything that the existing os.path *won't do* that we desperately need it to do? For filesystem files, no. Though you really mean all six modules above and not just os.path. It has been proposed to support non-filesystem directories (zip files, CSV/Subversion sandboxes, URLs, FTP objects) under a new Path API. > 3) Assuming that the answer to #1 is "yes", the next question is: "evolution or revolution?" Revolution. It needs a clean new API. However, this can live alongside the existing functions if necessary: posixpath.PosixPath, path.Path, etc. > 4) My third question is: Who are we going to steal our ideas from? Boost, Java, C# and others - all are worthy of being the, ahem, target of our inspiration. Or we have some alternative that's so cool that it makes sense to "Think Different(ly)"? Java is the only one I'm familiar with. The existing Python proposals are summarized below. > 5) Must there be one ring to rule them all? I suggested earlier that we might have a "low-level" and a "high-level" API, one built on top of the other. Is this a good idea, or a non-starter? It's worth discussing. One question is whether the dichotomy does anything useful or just adds unnecessary complexity. But that can only be answered for a specific API proposal. Whatever we do will be "low-level" compared to third-party extensions that will be built on top of it, so we should plan for extensibility. * * * * Here's a summary of the existing Python proposals in chronological order. Jason Orendorff's path.py http://www.jorendorff.com/articles/python/path/src/path.py This provides an OO path object encompassing the six modules above. The object is a string subclass. PEP 355 http://www.python.org/dev/peps/pep-0355/ An update to Orendorff's code. This was rejected by the BDFL because: (A) we haven't proven an OO interface is superior to os.path et al, (B) it mixes abstract path operations and filesystem-dependent operations, and (C) it's not radical enough for many OO proponents. However, it does separate filesystem-dependent operations to an extent: they must be methods, while abstract operations may be attributes. Orendorff's module does not separate these at all. Noam Raphael's directory-based class Introduction: http://wiki.python.org/moin/AlternativePathClass Feature discussion: http://wiki.python.org/moin/AlternativePathDiscussion Reference implementation: http://wiki.python.org/moin/AlternativePathModule Noam's emulates a sequence of components (a la os.path.split) rather than a string. This expresses slicing and joining by Python's [] and + operators, eliminating several named methods. Each component emulates a unicode with methods to extract the name and extension(s). The first component of absolute paths is a "root object" representing the Posix root ("/"), a Windows drive-absolute ("C:\"), a Windows drive-relative ("C:"), a network share, etc. This proposal has a reference implementation but a PEP has not been written, and the discussion page has many feature alternatives that aren't coded. A robust reference implementation would have methods for all alternatives so they can be compared in real programs. There have been a few additional ideas on python-dev but no code. Nick Coghlan suggested separate classes for (A) string manipulation, (B) abstract path operations, (C) read-only inspection of filesystem, (D) add/remove files/directories/links. Another suggested four classes for (A) abstract path operations, (B) file, (C) directory, (D) symlink. Talin proposed an elaborate set of classes and functions to do this. I pointed out that each class would need 3+ versions to accomodate platform differences, so somebody wanting to make a generic subclass would potentially have to make 12 versions to accommodate all the superclass possibilities (4 path classes * 3 platforms). Functions would cut down the need for multiple classes and duplicated methods between them, but functions would make "subclassing Path" more difficult. I would like to see one or more implementations tested and widely used as soon as possible, so that we'd have confidence using them in our programs until a general Python solution emerges. But first we need to see if we can achieve a common API, as Talin started this thread saying. -- Mike Orr From fredrik at pythonware.com Wed Nov 1 08:04:37 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 01 Nov 2006 08:04:37 +0100 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> Message-ID: Mike Orr wrote: > You may think 'shutil' has to do with shells, not paths. well, it has; it provides Python implementations of selected shell commands. > Why is 'split' in os.path but 'stat' and 'mkdir' and 'remove' are in > os? Don't they all operate on paths? no. are you saying that you're unable to see the conceptual difference between a name and an object? > There have been a few additional ideas on python-dev but no code. > Nick Coghlan suggested separate classes for (A) string manipulation, > (B) abstract path operations, (C) read-only inspection of filesystem, > (D) add/remove files/directories/links. that's the only remotely sane path-related proposal I've seen this far. here's mine; it's fully backwards compatible, can go right into 2.6, and can be incrementally improved in future releases: 1) add a pathname wrapper to "os.path", which lets you do basic path "algebra". this should probably be a subclass of unicode, and should *only* contain operations on names. 2) make selected "shutil" operations available via the "os" name- space; the old POSIX API vs. POSIX SHELL distinction is pretty irrelevant. also make the os.path predicates available via the "os" namespace. this gives a very simple conceptual model for the user; to manipulate path *names*, use "os.path.(string)" functions or the "" wrapper. to manipulate *objects* identified by a path, given either as a string or a path wrapper, use "os.(path)". this can be taught in less than a minute. From talin at acm.org Wed Nov 1 08:41:41 2006 From: talin at acm.org (Talin) Date: Tue, 31 Oct 2006 23:41:41 -0800 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> Message-ID: <45484FB5.3060608@acm.org> Mike Orr wrote: > Talin wrote: >> 1) Does os.path need to be refactored at all? > > Yes. Functions are scattered arbitrarily across six modules: os, > os.path, shutil, stat, glob, fnmatch. You have to search through five > scattered doc pages in the Python library to find your function, plus > the os module doc is split into five sections. You may think > 'shlutil' has to do with shells, not paths. shutil.copy2 is > riduculously named: what's so "2" about it? Why is 'split' in os.path > but 'stat' and 'mkdir' and 'remove' are in os? Don't they all operate > on paths? 'stat' and 'mkdir' do not operate on paths. To my mind, that's kind of like saying that "toupper" and "encrypt" and "SQLObject.select" should all go together, because they all operate on strings. -- Talin From sluggoster at gmail.com Wed Nov 1 09:04:20 2006 From: sluggoster at gmail.com (Mike Orr) Date: Wed, 1 Nov 2006 00:04:20 -0800 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> Message-ID: <6e9196d20611010004y6ebce483l6f18f6e1bcdd802e@mail.gmail.com> On 10/31/06, Fredrik Lundh wrote: > > Why is 'split' in os.path but 'stat' and 'mkdir' and 'remove' are in > > os? Don't they all operate on paths? > > no. are you saying that you're unable to see the conceptual difference > between a name and an object? I see the difference between modifying a path name (which doesn't depend on the filesystem) vs doing something with it (which requires the file to exist and be the right type). But both os and os.path contain both of these operations. > 1) add a pathname wrapper to "os.path", which lets you do basic > path "algebra". this should probably be a subclass of unicode, > and should *only* contain operations on names. I assume this means the P.parent and P.join("lib") functionality that I and others have been clamoring for. > 2) make selected "shutil" operations available via the "os" name- > space; the old POSIX API vs. POSIX SHELL distinction is pretty > irrelevant. also make the os.path predicates available via the > "os" namespace. I assume this means os.path.is*(), os.path.get*(), and shutil.* would be duplicated in the os module, as well as os.path.exists and os.path.lexists. os.walk() already supercedes os.path.walk(), although I find neither of these as convenient as a generator yielding individual path objects for every file and/or directory. > this gives a very simple conceptual model for the user; to manipulate > path *names*, use "os.path.(string)" functions or the "" > wrapper. to manipulate *objects* identified by a path, given either as > a string or a path wrapper, use "os.(path)". this can be taught in > less than a minute. It would be better than the current situation at least. It's mainly the name-manipulation functions that are nested, and thus in need of an OO paradigm and method chaining (P.basename().splitext()[0]). Object-manipulation functions can't really be nested so OO isn't as critical, although os.stat() is nested sometimes. By the way, the whole stat concept is arbitrary too. Just because C makes some operations dependent on stat() doesn't mean we necessarily should. In that vein, the os.path.is* and os.path.get* functions are an improvement. However, there is one good thing about stat(): P.stat().mtime and P.lstat().mtime look a lot better than P.mtime() and P.lmtime() -- especially considering the two dozen other functions/methods that would accompany the latter. -- Mike Orr From talin at acm.org Wed Nov 1 09:17:25 2006 From: talin at acm.org (Talin) Date: Wed, 01 Nov 2006 00:17:25 -0800 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> Message-ID: <45485815.1030001@acm.org> More comments... Mike Orr wrote: > Talin wrote: >> 1) Does os.path need to be refactored at all? > > Yes. Functions are scattered arbitrarily across six modules: os, > os.path, shutil, stat, glob, fnmatch. You have to search through five > scattered doc pages in the Python library to find your function, plus > the os module doc is split into five sections. You may think > 'shlutil' has to do with shells, not paths. shutil.copy2 is > riduculously named: what's so "2" about it? Why is 'split' in os.path > but 'stat' and 'mkdir' and 'remove' are in os? Don't they all operate > on paths? > > The lack of method chaning means you have to use nested functions, > which must be read "inside out" rather than left-to-right like paths > normally go. Say you want to add the absolute path of "../../lib" to > the Python path in a platform-independent manner, relative to an > absolute path (__file__): > > # Assume __file__ is "/toplevel/app1/bin/main_program.py". > # Result is "/toplevel/app1/lib". > p = os.path.join(os.path.dirname(os.path.dirname(__file__)), "lib") > > PEP 355 proposes a much easier-to-read: > > # The Path object emulates "/toplevel/app1/bin/main_program.py". > p = Path(__file__).parent.parent.join("lib") Actually I generally use: p = os.path.normpath( os.path.join( __file__, "../..", "lib" ) ) or even: p = os.path.normpath( os.path.join( __file__, "../../lib" ) ) ...which isn't quite as concise as what you wrote, but is better than the first example. (The reason this works is because 'normpath' doesn't know whether the last component is a file or a directory -- it simply interprets the ".." as an instruction to strip off the last component.) What I'd like to see is a version of "join" that automatically simplifies as it goes. Lets call it "combine": p = os.path.combine( __file__, "../..", "lib" ) or: p = os.path.combine( __file__, "../../lib" ) That's even easier to read than any of the above versions IMHO. > Noam Raphael's directory-component object would make this even more > straightforward: > > # The Path object emulates ("/", "toplevel", "app1", "bin", > "main_program.py") > p = Path(__file__)[:-2] + "lib" I don't know if I would describe this as 'straightforward'. 'Concise', certainly; 'terse', yes, and 'clever'. But also 'cute', and 'tricky'. I see a couple of problems with it: -- Its only intuitive if you remember that array elements are path components and not strings. In other words, if you attempt to "read" the [:-2] as if the path were a string, you'll get the wrong answer. -- Is path[ 0 ] a string or a path? What if I really do want to get the first two *characters* of the path, and not the first to components? Do I have to say something like: str( path )[ :2 ] > Stat handling has grown cruft over the years. To check the modify time > of a file: > > os.path.getmtime("/FOO") > os.stat("/FOO").st_mtime > os.stat("/FOO")[stat.ST_MTIME] # List subscript, deprecated usage. > > If you want to check whether a file is a type for which there is no > os.path.is*() method: > > stat.S_ISSOCK( os.stat("/FOO").st_mode ) # Is the file a socket? > > Compare to the directory-component proposal: > > Path("/foo").stat().issock This is the part I really don't like. A path is not a file. Imagine that if instead of paths we were doing SQL queries. Take SQLObject for example; say we have a table of addresses: Address.select( query_string ) Now, suppose you say that you want to be able to perform manipulations on the query string, and therefore it should be an object. So we'll define a new class, SQLQuery( string ): Address.select( SQLQuery( string ) ) And we will allow queries to be conjoined using boolean operators: Address.select( SQLQuery( string ) | SQLQuery( string2 ) ) Nothing controversial so far - this is the way many such systems work already. But now you say "Well, since SQLQuery is an object, it would be more elegant to have all of the query-related functions be methods of the query object." So for example, if you wanted to run the query string against the Address table, and see how many records came back, you would have to do something like: SQLQuery( string ).select( Address ).count() ..which is exactly backwards, and here's why: Generally when creating member functions of objects, you arrange them in the form: actor.operation( subject ) Where 'actor' is considered to be the 'active agent' of the operation, while the 'subject' is the passive input parameter. I would argue that both paths and query strings are passive, whereas tables and file systems are, if not exactly lively, at least more 'actor-like' than paths or queries. Now, that being said, I wouldn't have a problem with there being an "abstract filesystem object" that represents an entity on disk (be it file, directory, etc.), which would have a path inside it that would do some of the things you suggest. > os.path functions are too low-level. Say you want to recursively > delete a path you're about to overwrite, no matter whether it exists > or is a file or directory. You can't do it in one line of code, darn, > you gotta write this function or inline the code everywhere you use > it: > > def purge(p): > if os.path.isdir(p): > shutil.rmtree(p) # Raises error if nonexistent or > not a directory. > elif os.path.exists(): > # isfile follows symlinks and returns False for special > files, so it's > # not a reliable guide of whether we can call os.remove. > os.remove(p) # Raises error if nonexistent or a directory. > if os.path.isfile(p): # Includes all symlinks. > os.remove(p) I don't deny that such a function ought to exist. But it shouldn't be a member function on a path object IMHO. >> 2) is there anything that the existing os.path *won't do* that we desperately need it to do? > > For filesystem files, no. Though you really mean all six modules > above and not just os.path. It has been proposed to support > non-filesystem directories (zip files, CSV/Subversion sandboxes, URLs, > FTP objects) under a new Path API. > >> 3) Assuming that the answer to #1 is "yes", the next question is: > "evolution or revolution?" > > Revolution. It needs a clean new API. However, this can live > alongside the existing functions if necessary: posixpath.PosixPath, > path.Path, etc. > >> 4) My third question is: Who are we going to steal our ideas from? > Boost, Java, C# and others - all are worthy of being the, ahem, target > of our inspiration. Or we have some alternative that's so cool that it > makes sense to "Think Different(ly)"? > > Java is the only one I'm familiar with. The existing Python proposals > are summarized below. > >> 5) Must there be one ring to rule them all? I suggested earlier that we > might have a "low-level" and a "high-level" API, one built on top of the > other. Is this a good idea, or a non-starter? > > It's worth discussing. One question is whether the dichotomy does > anything useful or just adds unnecessary complexity. But that can > only be answered for a specific API proposal. Whatever we do will be > "low-level" compared to third-party extensions that will be built on > top of it, so we should plan for extensibility. Actually, I was considering the PEP 355 to be "high-level" and the current os.path to be "low-level". -- Talin From murman at gmail.com Wed Nov 1 15:03:55 2006 From: murman at gmail.com (Michael Urman) Date: Wed, 1 Nov 2006 08:03:55 -0600 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <45485815.1030001@acm.org> References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> Message-ID: > p = os.path.normpath( os.path.join( __file__, "../..", "lib" ) ) Shouldn't an example avoid using the path separator directly within a string literal? p = path.normpath(path.join(__file__, "..", "..", "lib")) -- Michael Urman From phd at phd.pp.ru Wed Nov 1 15:10:11 2006 From: phd at phd.pp.ru (Oleg Broytmann) Date: Wed, 1 Nov 2006 17:10:11 +0300 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> Message-ID: <20061101141011.GA16013@phd.pp.ru> On Wed, Nov 01, 2006 at 08:03:55AM -0600, Michael Urman wrote: > > p = os.path.normpath( os.path.join( __file__, "../..", "lib" ) ) > > Shouldn't an example avoid using the path separator directly within a > string literal? > > p = path.normpath(path.join(__file__, "..", "..", "lib")) It also should use platform-independent constants: p = path.normpath(path.join(__file__, os.pardir, os.pardir, "lib")) Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From jimjjewett at gmail.com Wed Nov 1 15:27:22 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 1 Nov 2006 09:27:22 -0500 Subject: [Python-3000] Alternatives to 'outer' In-Reply-To: <00b101c6fb23$ba0bf210$0301010a@VIMES> References: <00b101c6fb23$ba0bf210$0301010a@VIMES> Message-ID: On 10/29/06, Gary Stephenson wrote: > I would like to suggest the spelling "free", as I believe the term "free > variable" has been widely used to describe the concept (albeit > usually in reference to undeclared variables.) They are currently called free variables (or cell variables). This isn't helpful for the beginner, and was confusing for me. On the other hand, maybe beginners shouldn't be using them anyhow. I'll suggest "reuse" reuse myoutvar A simple "use" is too generic, but "reuse" implies that it already exists elsewhere, which (other than "not global") is the main point. -jJ > > > What this discussion needs now is not further keyword suggestions, but > > a PEP and an implementation. > > Is anybody working on such a PEP? > > gary > > http://www.stephensong.com.au > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/jimjjewett%40gmail.com > From python-dev at zesty.ca Wed Nov 1 16:17:47 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Wed, 1 Nov 2006 09:17:47 -0600 (CST) Subject: [Python-3000] Draft PEP for outer scopes Message-ID: Hi folks, I have finally completed a draft of a PEP on rebinding of names in outer scopes. I've tried to go back and gather all of the (amazingly numerous) proposals -- if i've forgotten or misattributed any, let me know and i'll be happy to correct them. I look forward to your thoughts on it: http://zesty.ca/python/pep-3104.txt (Could i get an official PEP number, please?) -- ?!ng From python at zesty.ca Wed Nov 1 16:27:06 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Wed, 1 Nov 2006 09:27:06 -0600 (CST) Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: Message-ID: I wrote: > I have finally completed a draft of a PEP on rebinding of names [...] > http://zesty.ca/python/pep-3104.txt I've turned this into HTML also, in case you find that easier to read: http://zesty.ca/python/pep-3104.html -- ?!ng From jcarlson at uci.edu Wed Nov 1 17:32:08 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 01 Nov 2006 08:32:08 -0800 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <20061101141011.GA16013@phd.pp.ru> References: <20061101141011.GA16013@phd.pp.ru> Message-ID: <20061101083046.C0E3.JCARLSON@uci.edu> Oleg Broytmann wrote: > On Wed, Nov 01, 2006 at 08:03:55AM -0600, Michael Urman wrote: > > > p = os.path.normpath( os.path.join( __file__, "../..", "lib" ) ) > > > > Shouldn't an example avoid using the path separator directly within a > > string literal? > > > > p = path.normpath(path.join(__file__, "..", "..", "lib")) > > It also should use platform-independent constants: > > p = path.normpath(path.join(__file__, os.pardir, os.pardir, "lib")) What operating systems that Python currently supports doesn't have ".." mean "parent directory"? And/or What systems currently in development don't have ".." meaning "parent directory"? Are they significant enough for us to care? - Josiah From talin at acm.org Wed Nov 1 17:45:41 2006 From: talin at acm.org (Talin) Date: Wed, 01 Nov 2006 08:45:41 -0800 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> Message-ID: <4548CF35.8050000@acm.org> Michael Urman wrote: >> p = os.path.normpath( os.path.join( __file__, "../..", "lib" ) ) > > Shouldn't an example avoid using the path separator directly within a > string literal? > > p = path.normpath(path.join(__file__, "..", "..", "lib")) No need. 'normpath' will automatically convert the forward-slash into a backward-slash on Win32 platforms. More generally: My defacto solution to the problem of path manipulation is that the "path algebra" operators are strings, rather than object methods or functions. When you join two paths, all of the instructions as to how to combine the two paths are contained in the string. -- Talin From rrr at ronadam.com Wed Nov 1 18:42:22 2006 From: rrr at ronadam.com (Ron Adam) Date: Wed, 01 Nov 2006 11:42:22 -0600 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <45485815.1030001@acm.org> References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> Message-ID: Talin wrote: > Now, that being said, I wouldn't have a problem with there being an > "abstract filesystem object" that represents an entity on disk (be it > file, directory, etc.), which would have a path inside it that would do > some of the things you suggest. I think this option should be explored a bit more for python 3000. Is this a matter of just adding name spaces, or do you have something more specific in mind? filesys.path filesys.curdir filesys.pardir filesys.open etc... If this, then what methods and attributes under 'os' would be moved and which would stay? Or is this too big a change? And also if this, would name spaces be used to organize methods to other IO objects such as local or WWW networks, and etc.? (net.path, inet.path) Cheers, Ron From phd at phd.pp.ru Wed Nov 1 19:20:51 2006 From: phd at phd.pp.ru (Oleg Broytmann) Date: Wed, 1 Nov 2006 21:20:51 +0300 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <20061101083046.C0E3.JCARLSON@uci.edu> References: <20061101141011.GA16013@phd.pp.ru> <20061101083046.C0E3.JCARLSON@uci.edu> Message-ID: <20061101182051.GA13184@phd.pp.ru> On Wed, Nov 01, 2006 at 08:32:08AM -0800, Josiah Carlson wrote: > > p = path.normpath(path.join(__file__, os.pardir, os.pardir, "lib")) > > What operating systems that Python currently supports doesn't have ".." > mean "parent directory"? macpath.py defines pardir = '::'. MacOS 9, probably. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From jcarlson at uci.edu Wed Nov 1 20:07:17 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 01 Nov 2006 11:07:17 -0800 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <20061101182051.GA13184@phd.pp.ru> References: <20061101083046.C0E3.JCARLSON@uci.edu> <20061101182051.GA13184@phd.pp.ru> Message-ID: <20061101110513.C0E9.JCARLSON@uci.edu> Oleg Broytmann wrote: > > On Wed, Nov 01, 2006 at 08:32:08AM -0800, Josiah Carlson wrote: > > > p = path.normpath(path.join(__file__, os.pardir, os.pardir, "lib")) > > > > What operating systems that Python currently supports doesn't have ".." > > mean "parent directory"? > > macpath.py defines pardir = '::'. MacOS 9, probably. MacOS 9 isn't a supported platform for Python anymore. The continued existance of macpath.py is either a mistake, or a kindness to those who still use OS 9. - Josiah From mark.russell at zen.co.uk Wed Nov 1 20:05:01 2006 From: mark.russell at zen.co.uk (Mark Russell) Date: Wed, 1 Nov 2006 19:05:01 +0000 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: Message-ID: <3B97E289-3548-4795-84D8-620DEB304B7E@zen.co.uk> On 1 Nov 2006, at 15:17, Ka-Ping Yee wrote: > I have finally completed a draft of a PEP on rebinding of names > in outer scopes. Nice summary - thanks for all the work. > I look forward to your thoughts on it: Minor point: it might be worth mentioning (in the section "Rebinding Operator") that := has been explicitly rejected in PEP 3099. (A shame since IMHO it's the best solution, but there you go). Mark Russell From mcherm at mcherm.com Wed Nov 1 20:28:21 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Wed, 01 Nov 2006 11:28:21 -0800 Subject: [Python-3000] Draft PEP for outer scopes Message-ID: <20061101112821.qd0fq3c7jnk0kows@login.werra.lunarpages.com> ?!ng writes: > I have finally completed a draft of a PEP on rebinding of names > in outer scopes. I've tried to go back and gather all of the > (amazingly numerous) proposals And you've done an astonishingly good job of it. The amount of history in this PEP is impressive, and it demonstrates the degree to which this is seen as a "wart". > I look forward to your thoughts on it I think it's great... let's approve and implement it immediately! -- Michael Chermside From sluggoster at gmail.com Wed Nov 1 21:47:12 2006 From: sluggoster at gmail.com (Mike Orr) Date: Wed, 1 Nov 2006 12:47:12 -0800 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <45485815.1030001@acm.org> References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> Message-ID: <6e9196d20611011247k1b918ba2ga3b12cfca1cb7469@mail.gmail.com> The thread on python-dev has been revived, so those interested in this subject will want to look in both places. On 11/1/06, Talin wrote: > Actually I generally use: > > p = os.path.normpath( os.path.join( __file__, "../..", "lib" ) ) > > or even: > > p = os.path.normpath( os.path.join( __file__, "../../lib" ) ) > > ...which isn't quite as concise as what you wrote, but is better than > the first example. (The reason this works is because 'normpath' doesn't > know whether the last component is a file or a directory -- it simply > interprets the ".." as an instruction to strip off the last component.) This illustrates two problems with os.path. The reason I use all these nested functions instead of a simple normpath/join is one is told "it's bad to use platform-specific separators". Perhaps this disrecommendation should be lifted, especially since both Mac and Windows do the right thing with "/", "..", and "." now. The other thing is, ".." and "." seem to be smarter than os.path.dirname. I can't quite articulate the rules but '.' off a file chops the filename component, while '.' off a directory does nothing. '..' off a file goes to the file's directory, while '..' off a directory goes to the directory's parent. Dirname() just chops the final component without asking what it is, while '..' and '.' do different things depending on whether the final component is a directory. I think a method like .ancestor(N) would be useful, meaning "do '..' N times. /a/b/../c # Previous component is always a directory, so eliminate. /a/b/./c # Swallow the '.'. /a/directory/.. > What I'd like to see is a version of "join" that automatically > simplifies as it goes. Lets call it "combine": > > p = os.path.combine( __file__, "../..", "lib" ) > > or: > > p = os.path.combine( __file__, "../../lib" ) > > That's even easier to read than any of the above versions IMHO. I wouldn't mind that actually. But the feedback I've gotten is the fewer variations from os.path functions, the better. I disagree with that though. [In Noam Raphael's propsal:] > -- Is path[ 0 ] a string or a path? What if I really do want to get the > first two *characters* of the path, and not the first to components? Do > I have to say something like: > > str( path )[ :2 ] That's something we've gone back and forth on: how to add characters to a path, how to add an extension, etc. We decided component-slicing was too important to give up. str(p) or unicode(p) of any Path will give the string representation. Converting to a string and back may not be the most elegant thing in the world but it's more straightforward than having special methods for character slicing. I have also proposed p[0] is a special "root object", which may be '', '/', 'c:\', 'c:', '\\abc' depending on the platform. So when joining there's no separator before the next component. p[1:] individually are each a subclass of unicode, with extra methods to extract basename and extension, delete N extensions from the end, add N extensions, etc. Any slice of a path object is a new path object. If the root is chopped off it becomes a relative path. > I would argue that both paths and query strings are passive, whereas > tables and file systems are, if not exactly lively, at least more > 'actor-like' than paths or queries. I can see your point. The only reason I went with a "monolithic" OO class is because that's what all the proposals have been for the past three years until last month, and I didn't think another way was possible or desirable. -- Mike Orr From qrczak at knm.org.pl Wed Nov 1 22:16:18 2006 From: qrczak at knm.org.pl (Marcin 'Qrczak' Kowalczyk) Date: Wed, 01 Nov 2006 22:16:18 +0100 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <6e9196d20611011247k1b918ba2ga3b12cfca1cb7469@mail.gmail.com> (Mike Orr's message of "Wed, 1 Nov 2006 12:47:12 -0800") References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> <6e9196d20611011247k1b918ba2ga3b12cfca1cb7469@mail.gmail.com> Message-ID: <87velysw9p.fsf@qrnik.zagroda> "Mike Orr" writes: > /a/b/../c # Previous component is always a directory, so eliminate. This is not equivalent to a/c on Unix if b is a symlink to a directory, because .. goes to the parent of the target of the symlink. -- __("< Marcin Kowalczyk \__/ qrczak at knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/ From ntoronto at cs.byu.edu Wed Nov 1 23:29:08 2006 From: ntoronto at cs.byu.edu (Neil Toronto) Date: Wed, 01 Nov 2006 15:29:08 -0700 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: Message-ID: <45491FB4.3000507@cs.byu.edu> Ka-Ping Yee wrote: > Hi folks, > > I have finally completed a draft of a PEP on rebinding of names > in outer scopes. I've tried to go back and gather all of the > (amazingly numerous) proposals -- if i've forgotten or misattributed > any, let me know and i'll be happy to correct them. > > I look forward to your thoughts on it: > It's beautiful. Like Michael said, an impressive amount of history. I wondered about this specifically: > A shorthand form is also permitted, in which nonlocal is prepended to an assignment or augmented assignment statement: > > nonlocal x = 3 Is a similar statement for globals legal in Py3k? It's not in 2.4 (according to my own testing) or 2.5 (according to the grammar). The syntax for 'global' and 'nonlocal' should be almost identical. Neil From guido at python.org Wed Nov 1 23:52:06 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 1 Nov 2006 14:52:06 -0800 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <45491FB4.3000507@cs.byu.edu> References: <45491FB4.3000507@cs.byu.edu> Message-ID: On 11/1/06, Neil Toronto wrote: > Ka-Ping Yee wrote: > > I have finally completed a draft of a PEP on rebinding of names > > in outer scopes. I've tried to go back and gather all of the > > (amazingly numerous) proposals -- if i've forgotten or misattributed > > any, let me know and i'll be happy to correct them. > It's beautiful. Like Michael said, an impressive amount of history. Agreed. > I wondered about this specifically: > > > A shorthand form is also permitted, in which nonlocal is prepended to > > an assignment or augmented assignment statement: > > > > nonlocal x = 3 > > Is a similar statement for globals legal in Py3k? It's not in 2.4 > (according to my own testing) or 2.5 (according to the grammar). The > syntax for 'global' and 'nonlocal' should be almost identical. It's been proposed and I would endorse it. My personal preference is still to abuse 'global' instead of adding a new, ugly keyword. That would make the syntax for global and nonlocal completely identical. :-) But I seem to be alone in this preference. :-( -- --Guido van Rossum (home page: http://www.python.org/~guido/) From brett at python.org Thu Nov 2 00:07:28 2006 From: brett at python.org (Brett Cannon) Date: Wed, 1 Nov 2006 15:07:28 -0800 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> Message-ID: On 11/1/06, Guido van Rossum wrote: > > On 11/1/06, Neil Toronto wrote: > > Ka-Ping Yee wrote: > > > I have finally completed a draft of a PEP on rebinding of names > > > in outer scopes. I've tried to go back and gather all of the > > > (amazingly numerous) proposals -- if i've forgotten or misattributed > > > any, let me know and i'll be happy to correct them. > > > It's beautiful. Like Michael said, an impressive amount of history. > > Agreed. > > > I wondered about this specifically: > > > > > A shorthand form is also permitted, in which nonlocal is prepended to > > > an assignment or augmented assignment statement: > > > > > > nonlocal x = 3 > > > > Is a similar statement for globals legal in Py3k? It's not in 2.4 > > (according to my own testing) or 2.5 (according to the grammar). The > > syntax for 'global' and 'nonlocal' should be almost identical. > > It's been proposed and I would endorse it. > > My personal preference is still to abuse 'global' instead of adding a > new, ugly keyword. That would make the syntax for global and nonlocal > completely identical. :-) But I seem to be alone in this preference. > :-( Seeing Guido have a sad face is enough to force me to have an opinon. I personally always viewed 'global' as "this variable is not local", so making it truly mean that works for me. Otherwise I would like the 'free' keyword (I understand why Ping prefers 'nonlocal', but i just don't like it without a hyphen in it and that isn't about to happen and I have no problem going with the lambda calculus terminology). Then again Guido could still get his way since this could end up being quite the little argument as demonstrated by the list of possible keyword names and thus require a BDFL pronouncement the way Guido wants it. =) -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061101/71c6697a/attachment.html From greg.ewing at canterbury.ac.nz Thu Nov 2 00:14:43 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 02 Nov 2006 12:14:43 +1300 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> Message-ID: <45492A63.9030703@canterbury.ac.nz> Guido van Rossum wrote: > My personal preference is still to abuse 'global' instead of adding a > new, ugly keyword. That would make the syntax for global and nonlocal > completely identical. :-) But I seem to be alone in this preference. You're not alone -- I'd be happy with that, too. I don't even think it's particularly abusive, since I regard "nonlocal" as a legitimate meaning of "global" in a language that can have more than two nested scopes. -- Greg From mahs at telcopartners.com Thu Nov 2 00:28:23 2006 From: mahs at telcopartners.com (Michael Spencer) Date: Wed, 01 Nov 2006 15:28:23 -0800 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: Message-ID: Ka-Ping Yee wrote: > Hi folks, > > I have finally completed a draft of a PEP on rebinding of names > in outer scopes. I've tried to go back and gather all of the > (amazingly numerous) proposals -- if i've forgotten or misattributed > any, let me know and i'll be happy to correct them. > > I look forward to your thoughts on it: > > http://zesty.ca/python/pep-3104.txt > > (Could i get an official PEP number, please?) > > > -- ?!ng Looks good. I tried an experimental implementation using the compiler2 package. (http://groups.google.com/group/comp.lang.python/msg/f10db8a1ca2047b4) compiler2 doesn't do parsing, so instead of `nonlocal a,b` I abuse existing syntax and write `global __nonlocal__,a,b` I did not try to implement `nonlocal a=3` form, since the existing global syntax does not allow it. If it's going to be legal, then will global be similarly adjusted? I assume nonlocal statements should not nest the way global does i.e., a nonlocal statement in one scope has no affect on assignment in an *enclosed* scope. Michael Here it is: """Experimental implementation of 'PEP 3104', using compiler2. Requires: Python 2.5 compiler2 from http://svn.brownspencer.com/pycompiler/branches/new_ast/ See readme.txt for installation This file also available from: http://svn.brownspencer.com/pycompiler/branches/new_ast/sandbox/nonlocal.py """ from compiler2.pyassem import (CompilerUnit, FunctionMixin, ClassMixin, ModuleMixin, SymbolVisitor, CodeGenerator, OrderedSet, SYMBOLS, parse) from _ast import AST, stmt, Global import _ast class NonLocal(stmt): """A new node type to represent the PEP3104 nonlocal statement""" _fields = ('names',) _attributes = ['lineno', 'col_offset'] # push NonLocal into _ast to satisfy SymbolVisitor _ast.NonLocal = NonLocal def convert_nonlocal(tree): """We have no syntax support for 'nonlocal' keyword. So, instead, use global __nonlocal__, target[s]. This conversion function then turns the Global nodes into NonLocal, and removes the __nonlocal__ flag""" stack = [tree] while stack: node = stack.pop() if isinstance(node, Global): if node.names[0] == "__nonlocal__": node.__class__ = NonLocal del node.names[0] else: for attr in node.__dict__.values(): if type(attr) is list: stack.extend(elt for elt in attr if isinstance(elt, AST)) elif isinstance(attr, AST): stack.append(attr) class MySymbolVisitor(SymbolVisitor): def visitNonLocal(self, node, cu): for name in node.names: cu.add_free(name) class MyCompilerUnit(CompilerUnit): def __init__(self, *args): CompilerUnit.__init__(self, *args) self.frees = set() # track names marked explicitly free (via nonlocal) # adjust add_def to ignore frees, like globals def add_def(self, name): m_name = self.mangle(name) if m_name not in self.globals and m_name not in self.frees: if m_name in self.defs: self.redefs.add(m_name) self.defs.add(m_name) self.symbols.add(m_name) self.uses.discard(m_name) # new method - called by MySymbolVisitor.visitNonLocal def add_free(self, name): """Mark a name as explicitly free, i.e., nonlocal""" m_name = self.mangle(name) if m_name in self.uses or m_name in self.defs: pass # Warn about nonlocal following def/use elif m_name in self.params: raise SyntaxError, "Parameter can't be declared nonlocal" elif m_name in self.globals: raise SyntaxError, "Name can't be declared global and nonlocal" self.frees.add(m_name) self.symbols.add(m_name) def _get_frees(self, child_frees, outer_defs): """Helper for finish_symbol_analysis""" # only change from base method is to include self.frees self_frees = (self.frees | self.uses | child_frees) & outer_defs if self_frees: symbols=self.symbols fv = [n for n in symbols if n in self_frees] fv += (n for n in self_frees if n not in symbols) self.freevars = tuple(fv) self.closure.update(fv) return self_frees # Override required to use re-defined FunctionUnit def new_child_function(self, node, name, firstlineno): """Create a new function unit as a child of the current one.""" return FunctionUnit(node, self.filename, name, firstlineno, self) # Override required to use re-defined ClassUnit def new_child_class(self, node, name, firstlineno): """Create a new function unit as a child of the current one.""" return ClassUnit(node, self.filename, name, firstlineno, self) class FunctionUnit(FunctionMixin, MyCompilerUnit): pass class ClassUnit(ClassMixin, MyCompilerUnit): pass class ModuleUnit(ModuleMixin, MyCompilerUnit): # Override to use our re-defined SymbolVisitor def make_scopes(self, visitor = MySymbolVisitor): s = visitor() s.visit(self.node, self) # Adjust pycompile to take non_local flag def pycompile(source, filename, mode, flags=None, dont_inherit=None, non_local=False): """Replacement for builtin compile() function""" tree = parse(source, filename, mode) if non_local: convert_nonlocal(tree) if mode == "exec": gen = ModuleUnit(tree, filename) else: raise ValueError("compile() 3rd arg must be 'exec' [" "'eval' and 'single' not yet Implemented]") return gen.compile() test_src = """\ # Note the behavior of the a=3 assignment depends on whether nonlocal # is enabled. If so, it rebinds f's cellvar. If not, it rebinds # global a a=1 def f(): a=2 def g(): global __nonlocal__, a # means nonlocal a a=3 g() return a f_a = f() """ """ Examples and test: # using the nonlocal feature >>> nsp = {} >>> exec pycompile(test_src, "string", "exec", non_local = True) in nsp >>> assert nsp['a'] == 1 >>> assert nsp['f_a'] == 3 # compare with compiler2, nonlocal not active >>> nsp = {} >>> exec pycompile(test_src, "string", "exec", non_local = False) in nsp >>> assert nsp['a'] == 3 >>> assert nsp['f_a'] == 2 # compare with __builtin__.compile >>> nsp = {} >>> exec compile(test_src, "string", "exec") in nsp >>> assert nsp['a'] == 3 >>> assert nsp['f_a'] == 2 >>> """ def test(): import doctest doctest.testmod() if __name__ == "__main__": test() print 'ok' From python at zesty.ca Thu Nov 2 01:55:26 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Wed, 1 Nov 2006 18:55:26 -0600 (CST) Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <45491FB4.3000507@cs.byu.edu> References: <45491FB4.3000507@cs.byu.edu> Message-ID: Hi again. I've updated the PEP to incorporate some suggestions i received. This revision also adds a Ruby example, a quote from 1994, a security argument, and and an additional error case (attempting to declare a parameter name nonlocal). Thanks, everyone! http://zesty.ca/python/pep-3104.txt http://zesty.ca/python/pep-3104.html -- ?!ng From python at zesty.ca Thu Nov 2 02:01:12 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Wed, 1 Nov 2006 19:01:12 -0600 (CST) Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> Message-ID: Guido wrote: > My personal preference is still to abuse 'global' instead of adding a > new, ugly keyword. That would make the syntax for global and nonlocal > completely identical. :-) But I seem to be alone in this preference. Brett wrote: > Seeing Guido have a sad face is enough to force me to have an opinon. I > personally always viewed 'global' as "this variable is not local", so making > it truly mean that works for me. I'm convinced that "global variable" means top-level for most programmers and so this usage would be confusing -- but i think we're all just repeating what we've said before. Would it help at all to survey some folks to see how many interpret "global variable" to mean "top-level" vs. "anything nonlocal"? -- ?!ng From andrewm at object-craft.com.au Thu Nov 2 02:40:59 2006 From: andrewm at object-craft.com.au (Andrew McNamara) Date: Thu, 02 Nov 2006 12:40:59 +1100 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> Message-ID: <20061102014059.BC4BB6F4198@longblack.object-craft.com.au> >Guido wrote: >> My personal preference is still to abuse 'global' instead of adding a >> new, ugly keyword. That would make the syntax for global and nonlocal >> completely identical. :-) But I seem to be alone in this preference. > >Brett wrote: >> Seeing Guido have a sad face is enough to force me to have an opinon. I >> personally always viewed 'global' as "this variable is not local", so making >> it truly mean that works for me. > >I'm convinced that "global variable" means top-level for most >programmers and so this usage would be confusing -- but i think we're >all just repeating what we've said before. But it doesn't mean "top-level" - it already comes with the qualifier "module". Even after nearly a decade of python use, I still find that slightly unnatural, and changing "global" to mean "enclosing scope" feels only slightly more unnatural. -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/ From tjreedy at udel.edu Thu Nov 2 04:34:44 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 1 Nov 2006 22:34:44 -0500 Subject: [Python-3000] Draft PEP for outer scopes References: <45491FB4.3000507@cs.byu.edu> <20061102014059.BC4BB6F4198@longblack.object-craft.com.au> Message-ID: "Andrew McNamara" wrote in message news:20061102014059.BC4BB6F4198 at longblack.object-craft.com.au... > >Guido wrote: >>> My personal preference is still to abuse 'global' instead of adding a >>> new, ugly keyword. That would make the syntax for global and nonlocal >>> completely identical. :-) But I seem to be alone in this preference. No, I (and others, it seems) just never have reason before to clutter the list with a 'me-too' post. > But it doesn't mean "top-level" - it already comes with the qualifier > "module". Even after nearly a decade of python use, I still find that > slightly unnatural, Me too ;-) > and changing "global" to mean "enclosing scope" > feels only slightly more unnatural. Me too ;-) Actually, I would only want a separate keyword would be if one wanted to be able to write x = 1 def f(): x = 'a' def _f(): global x = {} and have that act differently (as at present) from the inside-out behaviour of newkey x = 'whatever'. But I will not write such code and do not want to see such. With two different but similar keywords, I would expect to see requests to be able to write def _f(): global x = 2 print x nonlocal x = 'b' print x Terry Jan Reedy From steven.bethard at gmail.com Thu Nov 2 04:39:35 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 1 Nov 2006 20:39:35 -0700 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> Message-ID: Guido wrote: > My personal preference is still to abuse 'global' instead of adding a > new, ugly keyword. That would make the syntax for global and nonlocal > completely identical. :-) But I seem to be alone in this preference. Brett wrote: > Seeing Guido have a sad face is enough to force me to have an opinon. I > personally always viewed 'global' as "this variable is not local", so making > it truly mean that works for me. Ka-Ping Yee wrote: > Would it help at all to survey some folks to see how many interpret > "global variable" to mean "top-level" vs. "anything nonlocal"? I don't think that'll really be worth it. I'd be amazed if people didn't expect it to mean "top-level". The real question is, if people see something like this:: def f(): n = 0 def g(i): global n n += i return g func = f() print func(), func() what would they expect it to do? If you need to run a survey, that's probably the one to run. (Of course a different code example could be used, but you get the idea). Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From python at zesty.ca Thu Nov 2 05:28:13 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Wed, 1 Nov 2006 22:28:13 -0600 (CST) Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> Message-ID: Ka-Ping Yee wrote: > Would it help at all to survey some folks to see how many interpret > "global variable" to mean "top-level" vs. "anything nonlocal"? Steven Bethard wrote: > I don't think that'll really be worth it. I'd be amazed if people > didn't expect it to mean "top-level". If that's as obvious to you as it is to me, i don't understand why there's still any question. I've never heard of a programming language or conversation about programming where "global variable" means a variable bound in an outer enclosing function; it always means a variable bound outside of any functions (Wikipedia: "a variable that does not belong to any subroutine or class"). > The real question is, if people see something like this:: [...] > what would they expect it to do? I think a fairer survey example would be something like this: n = 1 def f(): n = 2 def g(): global n = 99 return n g() f() print n Which 'n' do you expect g() to change? (My answer: the global n. That's what it says: "global n". Which n is global? Clearly the first one.) -- ?!ng From steven.bethard at gmail.com Thu Nov 2 05:36:14 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 1 Nov 2006 21:36:14 -0700 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> Message-ID: On 11/1/06, Ka-Ping Yee wrote: > I think a fairer survey example would be something like this: > > n = 1 > def f(): > n = 2 > def g(): > global n = 99 > return n > g() > > f() > print n > > Which 'n' do you expect g() to change? The only reason I didn't use that in the first place is that is seems extremely unlikely in real code. Who uses the same name for a module-level binding and a function-local binding? So yes, that's the corner case, but I contend that the corner case will almost never come up. Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From python at zesty.ca Thu Nov 2 05:52:05 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Wed, 1 Nov 2006 22:52:05 -0600 (CST) Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <20061102014059.BC4BB6F4198@longblack.object-craft.com.au> References: <45491FB4.3000507@cs.byu.edu> <20061102014059.BC4BB6F4198@longblack.object-craft.com.au> Message-ID: I wrote: > I'm convinced that "global variable" means top-level for most > programmers and so this usage would be confusing -- but i think we're > all just repeating what we've said before. Andrew McNamara wrote: > But it doesn't mean "top-level" - it already comes with the qualifier > "module". This is exactly what "global variable" means in C, Perl, JavaScript, Ruby, etc. The global scope is the widest scope in which you can declare a variable. -- ?!ng From greg.ewing at canterbury.ac.nz Thu Nov 2 06:15:18 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 02 Nov 2006 18:15:18 +1300 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <6e9196d20611010004y6ebce483l6f18f6e1bcdd802e@mail.gmail.com> References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <6e9196d20611010004y6ebce483l6f18f6e1bcdd802e@mail.gmail.com> Message-ID: <45497EE6.6080202@canterbury.ac.nz> Mike Orr wrote: > P.stat().mtime and P.lstat().mtime look a lot better than P.mtime() > and P.lmtime() Another thing to consider is that stat() potentially lets you enquire about multiple attributes without making a system call for each one. -- Greg From andrewm at object-craft.com.au Thu Nov 2 06:24:12 2006 From: andrewm at object-craft.com.au (Andrew McNamara) Date: Thu, 02 Nov 2006 16:24:12 +1100 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> <20061102014059.BC4BB6F4198@longblack.object-craft.com.au> Message-ID: <20061102052412.A91B36F4198@longblack.object-craft.com.au> >Andrew McNamara wrote: >> But it doesn't mean "top-level" - it already comes with the qualifier >> "module". > >This is exactly what "global variable" means in C, Perl, JavaScript, >Ruby, etc. > >The global scope is the widest scope in which you can declare a variable. This is inaccurate at best. The reality is: Python's "global" is module-global, yet there's a wider scope that is searched after the module namespace: the __builtin__ module. C has a similar concept module-global (module "static" variables), but if I remember correctly Perl does not (globals are interpreter-global), and neither does Javascript (globals are global to the document, with each document essentially a separate interpreter instance). To be honest, I'm +0 on using "global" in the way GvR proposes. My point is that python's globals are already different from other common languages, and people cope with that. -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/ From greg.ewing at canterbury.ac.nz Thu Nov 2 06:25:08 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 02 Nov 2006 18:25:08 +1300 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> Message-ID: <45498134.1030009@canterbury.ac.nz> Michael Urman wrote: >> p = os.path.normpath( os.path.join( __file__, "../..", "lib" ) ) > > Shouldn't an example avoid using the path separator directly within a > string literal? And avoid using a platform-specific parent directory specifier: p = path.normpath(path.join(__file__, os.pardir, os.pardir, "lib")) -- Greg From greg.ewing at canterbury.ac.nz Thu Nov 2 06:27:47 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 02 Nov 2006 18:27:47 +1300 Subject: [Python-3000] Alternatives to 'outer' In-Reply-To: References: <00b101c6fb23$ba0bf210$0301010a@VIMES> Message-ID: <454981D3.5090909@canterbury.ac.nz> Jim Jewett wrote: > I'll suggest "reuse" > > reuse myoutvar But to me that sounds like you're shadowing myoutvar, which is completely the wrong idea. -- Greg From fredrik at pythonware.com Thu Nov 2 07:23:49 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 02 Nov 2006 07:23:49 +0100 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> Message-ID: Guido van Rossum wrote: >> > A shorthand form is also permitted, in which nonlocal is prepended to >> > an assignment or augmented assignment statement: >> > >> > nonlocal x = 3 >> >> Is a similar statement for globals legal in Py3k? It's not in 2.4 >> (according to my own testing) or 2.5 (according to the grammar). The >> syntax for 'global' and 'nonlocal' should be almost identical. > > It's been proposed and I would endorse it. except that the syntax looks like you're declaring *and* defining something in *this* scope, which isn't what's going on. > My personal preference is still to abuse 'global' instead of adding a > new, ugly keyword. That would make the syntax for global and nonlocal > completely identical. :-) But I seem to be alone in this preference. not necessarily, but that would rule it out for 2.X, I suppose. From sluggoster at gmail.com Thu Nov 2 07:47:54 2006 From: sluggoster at gmail.com (Mike Orr) Date: Wed, 1 Nov 2006 22:47:54 -0800 Subject: [Python-3000] Mini Path object Message-ID: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> Posted to python-dev and python-3000. Follow-ups to python-dev only please. On 10/31/06, Fredrik Lundh wrote: > here's mine; it's fully backwards compatible, can go right into 2.6, > and can be incrementally improved in future releases: > > 1) add a pathname wrapper to "os.path", which lets you do basic > path "algebra". this should probably be a subclass of unicode, > and should *only* contain operations on names. > > 2) make selected "shutil" operations available via the "os" name- > space; the old POSIX API vs. POSIX SHELL distinction is pretty > irrelevant. also make the os.path predicates available via the > "os" namespace. > > this gives a very simple conceptual model for the user; to manipulate > path *names*, use "os.path.(string)" functions or the "" > wrapper. to manipulate *objects* identified by a path, given either as > a string or a path wrapper, use "os.(path)". this can be taught in > less than a minute. Given the widely-diverging views on what, if anything, should be done to os.path, how about we make a PEP and a standalone implementation of (1) for now, and leave (2) and everything else for a later PEP. This will make people who want a reasonably forward-compatable object NOW for their Python 2.4/2.5 programs happy, provide a common seed for more elaborate libraries that may be proposed for the standard library later (and eliminate the possibility of moving the other functions and later deprecating them), and provide a module that will be well tested by the time 2.6 is ready for finalization. There's already a reference implementation in PEP 355, we'd just have to strip out the non-pathname features. There's a copy here (http://wiki.python.org/moin/PathModule) that looks reasonably recent (constructors are self.__class__() to make it subclassable), although I wonder why the class is called path instead of Path. There was another copy in the Python CVS although I can't find it now; was it deleted in the move to Subversion? (I thought it was in /sandbox/trunk/: http://svn.python.org/view/sandbox/trunk/). So, let's say we strip this Path class to: class Path(unicode): Path("foo") Path( Path("directory"), "subdirectory", "file") # Replaces .joinpath(). Path() Path.cwd() Path("ab") + "c" => Path("abc") .abspath() .normcase() .normpath() .realpath() .expanduser() .expandvars() .expand() .parent .name # Full filename without path .namebase # Filename without extension .ext .drive .splitpath() .stripext() .splitunc() .uncshare .splitall() .relpath() .relpathto() Would this offend anyone? Are there any attribute renames or method enhancements people just can't live without? 'namebase' is the only name I hate but I could live with it. The multi-argument constructor is a replacement for joining paths. (The PEP says .joinpath was "problematic" without saying why.) This could theoretically go either way, doing either the same thing as os.path.join, getting a little smarter, or doing "safe" joins by disallowing "/" embedded in string arguments. I would say that a directory-tuple Path object with these features could be maintained in parallel, but since the remaining functions require string arguments you'd have to use unicode() a lot. -- Mike Orr From ronaldoussoren at mac.com Thu Nov 2 08:40:18 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 2 Nov 2006 08:40:18 +0100 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <20061101110513.C0E9.JCARLSON@uci.edu> References: <20061101083046.C0E3.JCARLSON@uci.edu> <20061101182051.GA13184@phd.pp.ru> <20061101110513.C0E9.JCARLSON@uci.edu> Message-ID: <15D45482-38F9-453A-9083-1C9F64C4A7BE@mac.com> On Nov 1, 2006, at 8:07 PM, Josiah Carlson wrote: > > Oleg Broytmann wrote: >> >> On Wed, Nov 01, 2006 at 08:32:08AM -0800, Josiah Carlson wrote: >>>> p = path.normpath(path.join(__file__, os.pardir, os.pardir, "lib")) >>> >>> What operating systems that Python currently supports doesn't >>> have ".." >>> mean "parent directory"? >> >> macpath.py defines pardir = '::'. MacOS 9, probably. > > MacOS 9 isn't a supported platform for Python anymore. The continued > existance of macpath.py is either a mistake, or a kindness to those > who > still use OS 9. Classic MacOS paths are still used by Carbon applications on Mac OS X and by AppleScript, which means there's still a need to use macpath sometimes (especially for the application scripting). Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3562 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20061102/ed3625c9/attachment.bin From p.f.moore at gmail.com Thu Nov 2 09:46:09 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 2 Nov 2006 08:46:09 +0000 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> Message-ID: <79990c6b0611020046j9d95781i378b65a55ea016c3@mail.gmail.com> On 11/2/06, Mike Orr wrote: > Given the widely-diverging views on what, if anything, should be done > to os.path, how about we make a PEP and a standalone implementation of > (1) for now, and leave (2) and everything else for a later PEP. Why write a PEP at this stage? Just release your proposal as a module, and see if people use it. If they do, write a PEP to include it in the stdlib. (That's basically what happened with the original PEP - it started off proposing Jason Orendorff's path module IIRC). >From what you're proposing, I may well use such a module, if it helps :-) (But I'm not sure I'd vote for it in to go the stdlib without having survived as an external module first) Paul. From tanzer at swing.co.at Thu Nov 2 10:03:24 2006 From: tanzer at swing.co.at (Christian Tanzer) Date: Thu, 02 Nov 2006 10:03:24 +0100 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: Your message of "Wed, 01 Nov 2006 19:01:12 CST." Message-ID: Ka-Ping Yee wrote: > Guido wrote: > > My personal preference is still to abuse 'global' instead of adding a > > new, ugly keyword. That would make the syntax for global and nonlocal > > completely identical. :-) But I seem to be alone in this preference. > > Brett wrote: > > Seeing Guido have a sad face is enough to force me to have an opinon. I > > personally always viewed 'global' as "this variable is not local", so making > > it truly mean that works for me. > > I'm convinced that "global variable" means top-level for most > programmers and so this usage would be confusing -- but i think we're > all just repeating what we've said before. Well, for many programmers `global` means visible everywhere in the program, so Python's use of the word is already confusing for some people. But that confusion usually doesn't last long and neither would the proposed reinterpretation's, IMHO. OTOH, having `global` and `nonlocal` mean the same thing for module-bound variables violates TOOWTDI. -- Christian Tanzer http://www.c-tanzer.at/ From tanzer at swing.co.at Thu Nov 2 09:55:41 2006 From: tanzer at swing.co.at (Christian Tanzer) Date: Thu, 02 Nov 2006 09:55:41 +0100 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: Your message of "Wed, 01 Nov 2006 18:55:26 CST." Message-ID: Ka-Ping Yee wrote: > A shorthand form is also permitted, in which ``nonlocal`` is > prepended to an assignment or augmented assignment statement:: > > nonlocal x = 3 > > The above has exactly the same meaning as ``nonlocal x; x = 3``. Nice. > The shorthand form does not allow multiple names. As you allow nonlocal x, y, z and x, y, z = 1, 2, 3 is legal Python (and does the right thing), why not allow nonlocal x, y, z = 1, 2, 3 too? That's one less rule needed to be learned and enforced. -- Christian Tanzer http://www.c-tanzer.at/ From fredrik at pythonware.com Thu Nov 2 10:14:07 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 02 Nov 2006 10:14:07 +0100 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> Message-ID: Steven Bethard wrote: > I don't think that'll really be worth it. I'd be amazed if people > didn't expect it to mean "top-level". as anyone who's spent enough time on c.l.python knows, people expect it to mean "application global". it isn't. alternatively, they expect it to be a scope specifier to be used when creating the variable, rather than when referring to it. it isn't. the exact meaning of "global" is one of the well-known "you only need to learn this once" things in Python. a reinterpreted global would fall in the same category. From jan.grant at bristol.ac.uk Thu Nov 2 10:27:25 2006 From: jan.grant at bristol.ac.uk (Jan Grant) Date: Thu, 2 Nov 2006 09:27:25 +0000 (GMT) Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> Message-ID: <20061102092431.H52313@tribble.ilrt.bris.ac.uk> On Wed, 1 Nov 2006, Steven Bethard wrote: > I don't think that'll really be worth it. I'd be amazed if people > didn't expect it to mean "top-level". The real question is, if people > see something like this:: > > def f(): > n = 0 > def g(i): > global n > n += i > return g > func = f() > print func(), func() > > what would they expect it to do? Set "n" in the scope of "func", assuming func is top-level. That global is already a lie isn't a feature. I'd rather see "outer" than "global" but even that can be understood to mean "outermost" rather than "the next one out". jan PS. If you're hunting for keywords to overload to drag variables into scope, "import" is also available, although would probably be very hard to make work. -- jan grant, ISYS, University of Bristol. http://www.bris.ac.uk/ Tel +44 (0)117 3317661 http://ioctl.org/jan/ Not as randy or clumsom as a blaster. From talin at acm.org Thu Nov 2 10:42:57 2006 From: talin at acm.org (Talin) Date: Thu, 02 Nov 2006 01:42:57 -0800 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> Message-ID: <4549BDA1.4070009@acm.org> Fredrik Lundh wrote: > Steven Bethard wrote: > >> I don't think that'll really be worth it. I'd be amazed if people >> didn't expect it to mean "top-level". > > as anyone who's spent enough time on c.l.python knows, people expect it > to mean "application global". it isn't. > > alternatively, they expect it to be a scope specifier to be used when > creating the variable, rather than when referring to it. it isn't. > > the exact meaning of "global" is one of the well-known "you only need to > learn this once" things in Python. a reinterpreted global would fall in > the same category. One thing I don't understand in this discussion of re-purposing the 'global' keyword is how you handle the case of an inner function that *creates* a global. Right now, you can create a global variable from within a scope, even if that variable hasn't been declared yet: def foo(): global x x = 1 foo() print x However, if 'global' is simply a synonym for 'nonlocal', then how does it know *which* scope to create the variable in? Let me anticipate one possible answer: "Don't do that". My response would be: first, being able to do that is occasionally useful, and second, if we can't do it, then we're back to the notion that global variables have to be "declared" before they can be used in inner functions...not exactly within the theme of Python. However, it just occurred to me that there's another option, one that's not yet included in Ping's PEP: _Named Lexical Scopes_. What if 'global' was not a keyword, but rather a name? So when you say "global x" you really mean that 'x' derives from a lexical scope that has the name 'global'. Inner scopes would be named after the function that defines the scope. So for example: def foo(): def bar(): foo x # means that 'x' is defined in the 'foo' scope x = 1 bar() print x Admittedly, using a bare non-keyword identifier is probably a bit too tricky in terms of parsing; Sure, it could be done, but its likely that it might create some parsing ambiguities that might bite us down the road. So I'd say go ahead and use a keyword to introduce a reference to a named scope, for example: def outer(): def inner1(): def inner2(): scope inner1 x, y = 1, 2 scope outer z = 3 scope global a = "this is a test" This syntax allows us to refer to any arbitrary outer scope, regardless of whether a local variable has been defined in that scope or not. Its also immune to variables changing meaning if the hierarchy of scopes is rearranged - for example, if we remove 'inner2' from inside 'inner1', the the references to 'outer' and 'global' still mean exactly the same thing as they did before. Also, this scheme does not prevent the compiler from knowing at compile time exactly what local variables are defined in which scopes. -- Talin From fredrik at pythonware.com Thu Nov 2 10:59:40 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 02 Nov 2006 10:59:40 +0100 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <4549BDA1.4070009@acm.org> References: <45491FB4.3000507@cs.byu.edu> <4549BDA1.4070009@acm.org> Message-ID: Talin wrote: > One thing I don't understand in this discussion of re-purposing the > 'global' keyword is how you handle the case of an inner function that > *creates* a global. > > Right now, you can create a global variable from within a scope, even if > that variable hasn't been declared yet: > > def foo(): > global x > x = 1 > > foo() > print x > > However, if 'global' is simply a synonym for 'nonlocal', then how does > it know *which* scope to create the variable in? since what's a free variable and not is determined by static analysis, and free variables are "owned" by the innermost scope they're used in, I'm not sure why you even asking that question. From solipsis at pitrou.net Thu Nov 2 17:00:02 2006 From: solipsis at pitrou.net (Antoine) Date: Thu, 2 Nov 2006 17:00:02 +0100 (CET) Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: Your message of "Wed, 01 Nov 2006 19:01:12 CST." Message-ID: <11024.62.39.9.251.1162483202.squirrel@webmail.nerim.net> > OTOH, having `global` and `nonlocal` mean the same thing for > module-bound variables violates TOOWTDI. Then you can disallow "nonlocal" for anything else than closure variables. And disallow "global" for anything else than module-global variables. It is explicit and unambiguous, and reduces the probability of scoping bugs. From g.brandl at gmx.net Thu Nov 2 17:09:26 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 02 Nov 2006 17:09:26 +0100 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <20061102092431.H52313@tribble.ilrt.bris.ac.uk> References: <45491FB4.3000507@cs.byu.edu> <20061102092431.H52313@tribble.ilrt.bris.ac.uk> Message-ID: Jan Grant wrote: > On Wed, 1 Nov 2006, Steven Bethard wrote: > >> I don't think that'll really be worth it. I'd be amazed if people >> didn't expect it to mean "top-level". The real question is, if people >> see something like this:: >> >> def f(): >> n = 0 >> def g(i): >> global n >> n += i >> return g >> func = f() >> print func(), func() >> >> what would they expect it to do? > > Set "n" in the scope of "func", assuming func is top-level. That global > is already a lie isn't a feature. > > I'd rather see "outer" than "global" but even that can be understood to > mean "outermost" rather than "the next one out". > > jan > > PS. If you're hunting for keywords to overload to drag variables into scope, > "import" is also available, although would probably be very hard to make > work. How should that possibly work? Georg From g.brandl at gmx.net Thu Nov 2 17:13:46 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 02 Nov 2006 17:13:46 +0100 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <6e9196d20611011247k1b918ba2ga3b12cfca1cb7469@mail.gmail.com> References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> <6e9196d20611011247k1b918ba2ga3b12cfca1cb7469@mail.gmail.com> Message-ID: Mike Orr wrote: > The thread on python-dev has been revived, so those interested in this > subject will want to look in both places. > > On 11/1/06, Talin wrote: >> Actually I generally use: >> >> p = os.path.normpath( os.path.join( __file__, "../..", "lib" ) ) >> >> or even: >> >> p = os.path.normpath( os.path.join( __file__, "../../lib" ) ) >> >> ...which isn't quite as concise as what you wrote, but is better than >> the first example. (The reason this works is because 'normpath' doesn't >> know whether the last component is a file or a directory -- it simply >> interprets the ".." as an instruction to strip off the last component.) > > This illustrates two problems with os.path. The reason I use all > these nested functions instead of a simple normpath/join is one is > told "it's bad to use platform-specific separators". Perhaps this > disrecommendation should be lifted, especially since both Mac and > Windows do the right thing with "/", "..", and "." now. > > The other thing is, ".." and "." seem to be smarter than > os.path.dirname. I can't quite articulate the rules but '.' off a > file chops the filename component, while '.' off a directory does > nothing. '..' off a file goes to the file's directory, while '..' off > a directory goes to the directory's parent. Dirname() just chops the > final component without asking what it is, while '..' and '.' do > different things depending on whether the final component is a > directory. I think a method like .ancestor(N) would be useful, > meaning "do '..' N times. > > /a/b/../c # Previous component is always a directory, so eliminate. "." and ".." on a file is meaningless on Unix: $ ls /etc/passwd/../ /bin/ls: cannot access /etc/passwd/../: Not a directory Georg From p.f.moore at gmail.com Thu Nov 2 17:21:43 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 2 Nov 2006 16:21:43 +0000 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <6e9196d20611011247k1b918ba2ga3b12cfca1cb7469@mail.gmail.com> References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> <6e9196d20611011247k1b918ba2ga3b12cfca1cb7469@mail.gmail.com> Message-ID: <79990c6b0611020821x6f7fcf12yd1c5d1a846be62ff@mail.gmail.com> On 11/1/06, Mike Orr wrote: > [...] especially since both Mac and > Windows do the right thing with "/", "..", and "." now. Not always: D:\Data>dir C:/ Invalid switch - "". Paul. From ronaldoussoren at mac.com Thu Nov 2 17:25:43 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 2 Nov 2006 17:25:43 +0100 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <79990c6b0611020821x6f7fcf12yd1c5d1a846be62ff@mail.gmail.com> References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> <6e9196d20611011247k1b918ba2ga3b12cfca1cb7469@mail.gmail.com> <79990c6b0611020821x6f7fcf12yd1c5d1a846be62ff@mail.gmail.com> Message-ID: <1E120870-D70C-496F-9960-112A4066D02C@mac.com> On Nov 2, 2006, at 5:21 PM, Paul Moore wrote: > On 11/1/06, Mike Orr wrote: >> [...] especially since both Mac and >> Windows do the right thing with "/", "..", and "." now. > > Not always: > > D:\Data>dir C:/ > Invalid switch - "". And on Mac this depends on the API you use, if you use the Carbon API's you still have to use colons as directory separators (and the OS does the correct translation for you, which is why you can still have a file named "a/b" in the Finder). Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3562 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20061102/023a8e32/attachment.bin From jan.grant at bristol.ac.uk Thu Nov 2 17:46:06 2006 From: jan.grant at bristol.ac.uk (Jan Grant) Date: Thu, 2 Nov 2006 16:46:06 +0000 (GMT) Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> <20061102092431.H52313@tribble.ilrt.bris.ac.uk> Message-ID: <20061102163948.R52313@tribble.ilrt.bris.ac.uk> On Thu, 2 Nov 2006, Georg Brandl wrote: > > PS. If you're hunting for keywords to overload to drag variables into scope, > > "import" is also available, although would probably be very hard to make > > work. > > How should that possibly work? Without treading on the toes of the existing import syntax, god knows (which is what I meant by "very hard"). The docs claim import exists to "(2) define a name or names in the local namespace (of the scope where the import statement occurs)" but since outer variable names may shadow module names, def f(): import x couldn't be used :-( -- jan grant, ISYS, University of Bristol. http://www.bris.ac.uk/ Tel +44 (0)117 3317661 http://ioctl.org/jan/ We thought time travel was impossible. But that was now and this is then. From talin at acm.org Thu Nov 2 18:53:53 2006 From: talin at acm.org (Talin) Date: Thu, 02 Nov 2006 09:53:53 -0800 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <1E120870-D70C-496F-9960-112A4066D02C@mac.com> References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> <6e9196d20611011247k1b918ba2ga3b12cfca1cb7469@mail.gmail.com> <79990c6b0611020821x6f7fcf12yd1c5d1a846be62ff@mail.gmail.com> <1E120870-D70C-496F-9960-112A4066D02C@mac.com> Message-ID: <454A30B1.6060709@acm.org> Ronald Oussoren wrote: > > On Nov 2, 2006, at 5:21 PM, Paul Moore wrote: > >> On 11/1/06, Mike Orr wrote: >>> [...] especially since both Mac and >>> Windows do the right thing with "/", "..", and "." now. >> >> Not always: >> >> D:\Data>dir C:/ >> Invalid switch - "". > > And on Mac this depends on the API you use, if you use the Carbon API's > you still have to use colons as directory separators (and the OS does > the correct translation for you, which is why you can still have a file > named "a/b" in the Finder). I think you folks are missing the point here - of *course* different platforms have different interpretations of path separators. But that's not what we're talking about here - we are talking about the behavior of os.path.normpath(). Look, people have been writing cross-platform Python code for years. How is this possible when different platforms have such a radical interpretation of path strings? The answer is that Python's path manipulation functions are *better* than what the underlying platform supplies. os.path doesn't just slavishly copy the semantics of the local filesystem, it is far more generous (and more useful). Using os.path.normpath(), one is able to manipulate path strings in a cross-platform way. So if say: os.path.normpath( os.path.join( __file__, "../../lib" ) ...what I get on both Win32, Linux and OSX (I can't speak for any other platform) is exactly the same semantics. And as far as things like os.pardir goes - you can't put 'os.pardir' in a configuration file! There are *lots* of applications where you want to share data files across platforms, and not have to revise the path strings every time you migrate between Unix and Win32. In fact, I'll tell you a little secret about Win32 applications - although "officially" the path separator for Win32 is backslash, there are so many Unix and cross-platform programs ported to Win32 that a larger percentage of the programs available on Win32 accept paths in either format. So lets not get bogged down in the strange quirks of how a particular platform interprets paths - I am much more interested in being able to combine paths together in an intuitive way, and have it "just work" on all platforms. And that includes path strings read from config files, not just path objects. -- Talin From talin at acm.org Thu Nov 2 19:00:56 2006 From: talin at acm.org (Talin) Date: Thu, 02 Nov 2006 10:00:56 -0800 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> <4549BDA1.4070009@acm.org> Message-ID: <454A3258.9010706@acm.org> Fredrik Lundh wrote: > Talin wrote: > >> One thing I don't understand in this discussion of re-purposing the >> 'global' keyword is how you handle the case of an inner function that >> *creates* a global. >> >> Right now, you can create a global variable from within a scope, even if >> that variable hasn't been declared yet: >> >> def foo(): >> global x >> x = 1 >> >> foo() >> print x >> >> However, if 'global' is simply a synonym for 'nonlocal', then how does >> it know *which* scope to create the variable in? > > since what's a free variable and not is determined by static analysis, > and free variables are "owned" by the innermost scope they're used in, > I'm not sure why you even asking that question. Right now, 'global' allows you to create a global variable from within a function, even if that global does not yet exist: >>> def foo(): ... global x ... x = 1 ... >>> foo() >>> print x 1 If you change the behavior of 'global' to be the same as 'nonlocal' as has been proposed, then this effectively becomes impossible - you can no longer set any global that hasn't already been pre-declared. -- Talin From ronaldoussoren at mac.com Thu Nov 2 19:12:17 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 2 Nov 2006 19:12:17 +0100 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <454A30B1.6060709@acm.org> References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> <6e9196d20611011247k1b918ba2ga3b12cfca1cb7469@mail.gmail.com> <79990c6b0611020821x6f7fcf12yd1c5d1a846be62ff@mail.gmail.com> <1E120870-D70C-496F-9960-112A4066D02C@mac.com> <454A30B1.6060709@acm.org> Message-ID: On Nov 2, 2006, at 6:53 PM, Talin wrote: > Ronald Oussoren wrote: >> >> On Nov 2, 2006, at 5:21 PM, Paul Moore wrote: >> >>> On 11/1/06, Mike Orr wrote: >>>> [...] especially since both Mac and >>>> Windows do the right thing with "/", "..", and "." now. >>> >>> Not always: >>> >>> D:\Data>dir C:/ >>> Invalid switch - "". >> >> And on Mac this depends on the API you use, if you use the Carbon >> API's >> you still have to use colons as directory separators (and the OS does >> the correct translation for you, which is why you can still have a >> file >> named "a/b" in the Finder). > > I think you folks are missing the point here - of *course* different > platforms have different interpretations of path separators. > > But that's not what we're talking about here - we are talking about > the > behavior of os.path.normpath(). > > Look, people have been writing cross-platform Python code for > years. How > is this possible when different platforms have such a radical > interpretation of path strings? The answer is that Python's path > manipulation functions are *better* than what the underlying platform > supplies. os.path doesn't just slavishly copy the semantics of the > local > filesystem, it is far more generous (and more useful). > > Using os.path.normpath(), one is able to manipulate path strings in a > cross-platform way. So if say: > > os.path.normpath( os.path.join( __file__, "../../lib" ) > > ...what I get on both Win32, Linux and OSX (I can't speak for any > other > platform) is exactly the same semantics. That happens to work because all tree use '..' to represent a parent directory and happen to accept forward slashes as a directory seperator. Your example doesn't work when you use macpath to manipulate classic OS9-style paths. I agree that the functions in os.path are a good abstraction for platform-independent path manipulation, even if your example is not. A better cross-platform way to write the call above is: os.path.join( os.path.dirname( os.path.dirname( os.path.abspath (__file__)))). > > And as far as things like os.pardir goes - you can't put > 'os.pardir' in > a configuration file! There are *lots* of applications where you > want to > share data files across platforms, and not have to revise the path > strings every time you migrate between Unix and Win32. > > In fact, I'll tell you a little secret about Win32 applications - > although "officially" the path separator for Win32 is backslash, there > are so many Unix and cross-platform programs ported to Win32 that a > larger percentage of the programs available on Win32 accept paths in > either format. Most windows APIs accept either separator, but IIRC the actual kernel API uses backslashes only, which leaks through at some points. It has been a while since I last looked at this (and scared collegues who's code I was testing), but I think it was the "really" absolute paths that don't accept forward slashes (paths like \\.\C:\Windows). This was with NT 4 and from the top of my head, so I may well be wrong. > > So lets not get bogged down in the strange quirks of how a particular > platform interprets paths - I am much more interested in being able to > combine paths together in an intuitive way, and have it "just work" on > all platforms. And that includes path strings read from config files, > not just path objects. If you want that you'll have to define some way to convert between a generic path representation to the native one. Ronald, who doesn't really want to get sucked into this discussion. P.S. I agree with the sentiment that several others have raised: please work on a better os.path as a seperate library and come back when that has proven itself in the field. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3562 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20061102/3cbbb84b/attachment.bin From fredrik at pythonware.com Thu Nov 2 19:32:19 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 02 Nov 2006 19:32:19 +0100 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <454A3258.9010706@acm.org> References: <45491FB4.3000507@cs.byu.edu> <4549BDA1.4070009@acm.org> <454A3258.9010706@acm.org> Message-ID: Talin wrote: > If you change the behavior of 'global' to be the same as 'nonlocal' as > has been proposed, then this effectively becomes impossible - you can no > longer set any global that hasn't already been pre-declared. of course you can, as long as it isn't shadowed in an intermediate scope. are you sure you know how free variables are handled in today's Python? From python at zesty.ca Thu Nov 2 21:34:24 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Thu, 2 Nov 2006 14:34:24 -0600 (CST) Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> <4549BDA1.4070009@acm.org> <454A3258.9010706@acm.org> Message-ID: On Thu, 2 Nov 2006, Fredrik Lundh wrote: > Talin wrote: > > > If you change the behavior of 'global' to be the same as 'nonlocal' as > > has been proposed, then this effectively becomes impossible - you can no > > longer set any global that hasn't already been pre-declared. > > of course you can, as long as it isn't shadowed in an intermediate scope. I think the reason you wouldn't be able to create new globals is that Guido wants this to be an error (for good reasons IMHO): http://mail.python.org/pipermail/python-dev/2006-July/066968.html | Perhaps the best solution would be to make it an error if there | wasn't a visible variable named a in an outer scope. -- ?!ng From tdelaney at avaya.com Thu Nov 2 22:18:03 2006 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Fri, 3 Nov 2006 08:18:03 +1100 Subject: [Python-3000] Draft PEP for outer scopes Message-ID: <2773CAC687FD5F4689F526998C7E4E5FF1EB3E@au3010avexu1.global.avaya.com> Steven Bethard wrote: > On 11/1/06, Ka-Ping Yee wrote: >> I think a fairer survey example would be something like this: >> >> n = 1 >> def f(): >> n = 2 >> def g(): >> global n = 99 >> return n >> g() >> >> f() >> print n >> >> Which 'n' do you expect g() to change? > > The only reason I didn't use that in the first place is that is seems > extremely unlikely in real code. Who uses the same name for a > module-level binding and a function-local binding? > > So yes, that's the corner case, but I contend that the corner case > will almost never come up. A much more likely corner case is (using existing syntax): def f(): def g() global n n = 99 return n g() f() print n What does this do? It depends on if f() has a binding of "n". If it does, the above throws a NameError. If it doesn't, it prints 99. For "nonlocal", this is a non-issue - it's a syntax error. But if we reuse "global", the nested function is context-dependent. If "global" were to change in Py3K to require an existing binding at the time the "global" keyword was executed, this would then have the semantics of "nonlocal" and be context-independent again. So I think any proposal to reuse "global" has to include these semantics. Tim Delaney From jcarlson at uci.edu Thu Nov 2 23:19:27 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 02 Nov 2006 14:19:27 -0800 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5FF1EB3E@au3010avexu1.global.avaya.com> References: <2773CAC687FD5F4689F526998C7E4E5FF1EB3E@au3010avexu1.global.avaya.com> Message-ID: <20061102141514.C115.JCARLSON@uci.edu> "Delaney, Timothy (Tim)" wrote: > If "global" were to change in Py3K to require an existing binding at the > time the "global" keyword was executed, this would then have the > semantics of "nonlocal" and be context-independent again. So I think any > proposal to reuse "global" has to include these semantics. Maybe, but it would break currently existing (and working) code like the following... #no foo = declaration def init_foo(): global foo foo = ... I use variants of the above in a few places. Could I initialize foo = None in the module namespace prior to running init_foo()? Sure, but that may or may not clutter up definitions contained in the module. - Josiah From greg.ewing at canterbury.ac.nz Fri Nov 3 00:38:06 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 03 Nov 2006 12:38:06 +1300 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <6e9196d20611011247k1b918ba2ga3b12cfca1cb7469@mail.gmail.com> References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> <6e9196d20611011247k1b918ba2ga3b12cfca1cb7469@mail.gmail.com> Message-ID: <454A815E.80704@canterbury.ac.nz> Mike Orr wrote: > Perhaps this > disrecommendation should be lifted, especially since both Mac and > Windows do the right thing with "/", "..", and "." now. Is anyone still supporting VMS? It has a wildly different pathname syntax. In any case, if we all start writing code that makes assumptions about pathname syntax, and some platform comes along in the future that does things differently, we're hosed. Do we want to take that risk? > The other thing is, ".." and "." seem to be smarter than > os.path.dirname. You can certainly get different results if you interpret a path containing . or .. through the file system than if you treat it as a pure pathname manipulation. But the same will be true of any OO replacement that also restricts itself to pathname manipulation -- it won't be able to treat . and .. any more smartly than os.path does. > '.' off a file chops the filename component, Not sure what you mean by that. On Unix, if you try to open "foo.c/." where "foo.c" is not a directory, you get an exception: >>> open("foo.c/.") Traceback (most recent call last): File "", line 1, in ? IOError: [Errno 20] Not a directory: 'foo.c/.' But os.path has no idea what foo.c is, so it just treats joining "." to a path and then normalising as a no-op. > '..' off a file goes to the file's directory, while '..' off > a directory goes to the directory's parent. These aren't different things -- in both cases you're getting the directory that contains the object in question. > I think a method like .ancestor(N) would be useful, > meaning "do '..' N times. That would be handy, yes. -- Greg From qrczak at knm.org.pl Fri Nov 3 02:03:03 2006 From: qrczak at knm.org.pl (Marcin 'Qrczak' Kowalczyk) Date: Fri, 03 Nov 2006 02:03:03 +0100 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <454A815E.80704@canterbury.ac.nz> (Greg Ewing's message of "Fri, 03 Nov 2006 12:38:06 +1300") References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> <6e9196d20611011247k1b918ba2ga3b12cfca1cb7469@mail.gmail.com> <454A815E.80704@canterbury.ac.nz> Message-ID: <87velxibp4.fsf@qrnik.zagroda> Greg Ewing writes: > In any case, if we all start writing code that makes assumptions > about pathname syntax, and some platform comes along in the future > that does things differently, we're hosed. Some conventions in the world are converging. For example a byte universally has 8 bits, and a machine word is always 8 bits times some power of 2. This used to vary. EBCDIC is nearly dead, everything is a superset of ASCII. Unicode becomes more common. It's very likely that any new OS, if it uses textual filenames at all, will use either the Unix filename convention, or - if it's made by Microsoft - a bastardized Unix convention. -- __("< Marcin Kowalczyk \__/ qrczak at knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/ From greg.ewing at canterbury.ac.nz Fri Nov 3 02:20:47 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 03 Nov 2006 14:20:47 +1300 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> Message-ID: <454A996F.6090400@canterbury.ac.nz> Steven Bethard wrote: > Ka-Ping Yee wrote: > >>Would it help at all to survey some folks to see how many interpret >>"global variable" to mean "top-level" vs. "anything nonlocal"? > > I don't think that'll really be worth it. I'd be amazed if people > didn't expect it to mean "top-level". Another confounding factor could be how familiar the respondent is with languages that have more than two scopes in the first place. If most of their programming experience is with C, they're likely to equate global with top-level simply because there's no other possibility in C. Besides, the BDFL is going to do whatever he wants anyway, so let's just leave him to get on with it. :-) -- Greg From python at zesty.ca Fri Nov 3 02:37:18 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Thu, 2 Nov 2006 19:37:18 -0600 (CST) Subject: [Python-3000] The meaning of "global variable" In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> Message-ID: Before it is reasonable to change the meaning of "global", we would need to have coherent answers to these questions: 1. What is the global namespace? 2. Do global variables belong to the global namespace? 3. When it says "global x" does that mean x is a global variable? I think the folks who support changing "global" to mean "innermost existing nonlocal" really have to provide answers to these. They are bound to become FAQs the moment such a change takes place. Can you imagine trying to teach this? My main problem with changing the meaning of "global" is that doing so causes one of these answers to be nonsense. Maybe i fail to see where our trains of thought diverge. It seems to me, either you have to say (#1) there is no longer such a thing as "the global namespace", or (#2) global variables have nothing to do with the global namespace, or (#3) "global x" has nothing to do with whether x is global. In all cases, well established Python terminology is thrown into confusion. It'll be even worse if we don't all agree on *which* of these three answers to make nonsense -- if we don't all choose the *same* nonsense, no one will know what anyone means by "global variable". The phrase "the global namespace" becomes meaningless or fraught with ambiguity. Or, in a concrete example: def f(): n = 2 def spam(): global n = 99 spam() 1. Is the "n" affected by spam() a global variable? 2. In what namespace does this "n" reside? 3. Is that what you would call the "global namespace"? 4. If not, which namespace is the "global namespace"? (You can't just say "oh, no one will ever write code like that." There have to be sensible answers -- teaching examples look like this, and tests test this, and curious students will want to know.) If the intention is to persuade Python programmers that the word "global" could refer to any of several namespaces, that would break compatibility not just with earlier versions of the language but with all the documentation and educational materials out there that mention "global variable" or "global namespace". Parts of the reference manual and even the codebase would also become confusing or misleading as a result -- identifiers containing the word "global" or comments mentioning the "global namespace" would no longer make sense. That seems like a lot of unnecessary work and a losing battle, IMHO. The battle will only hurt Python users who, henceforth, whenever they hear "global" can no longer tell which namespace you're talking about. And what are we gaining for this cost? So far there's no positive argument that "global" is actually the right word for this purpose -- defenders of "global" only seem to be saying that it's not a totally awful word for this purpose. Why tolerate something that is merely barely understandable (if you try hard enough to ignore what the rest of the world means by "global"), just to save a keyword? This is a new kind of variable access for Python; it deserves a new keyword. (Though "nonlocal" is my preference, i would take just about anything over "global".) I thought Py3K was about a fresh chance to make good choices, instead of getting by with tolerable ones. -- ?!ng From greg.ewing at canterbury.ac.nz Fri Nov 3 02:53:17 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 03 Nov 2006 14:53:17 +1300 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <4549BDA1.4070009@acm.org> References: <45491FB4.3000507@cs.byu.edu> <4549BDA1.4070009@acm.org> Message-ID: <454AA10D.9070903@canterbury.ac.nz> Talin wrote: > Right now, you can create a global variable from within a scope, even if > that variable hasn't been declared yet: That doesn't seem like a good practice from a readability point of view to me, but in any case, there's no reason it couldn't continue to be supported: If you don't find any assigment to the name in an enclosing scope, just assume it's module-level. -- Greg From greg.ewing at canterbury.ac.nz Fri Nov 3 03:00:51 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 03 Nov 2006 15:00:51 +1300 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <454A30B1.6060709@acm.org> References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> <6e9196d20611011247k1b918ba2ga3b12cfca1cb7469@mail.gmail.com> <79990c6b0611020821x6f7fcf12yd1c5d1a846be62ff@mail.gmail.com> <1E120870-D70C-496F-9960-112A4066D02C@mac.com> <454A30B1.6060709@acm.org> Message-ID: <454AA2D3.5020706@canterbury.ac.nz> Talin wrote: > And that includes path strings read from config files, > not just path objects. This is something new. I hadn't realised anyone was considering that to be within the scope of any of the proposed path modules. Obviously this is going to require defining a standard external representation for paths. What do you envisage such a representation being like? -- Greg From greg.ewing at canterbury.ac.nz Fri Nov 3 03:19:33 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 03 Nov 2006 15:19:33 +1300 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> Message-ID: <454AA735.4080406@canterbury.ac.nz> Ka-Ping Yee wrote: > Before it is reasonable to change the meaning of "global", we would > need to have coherent answers to these questions: > > 1. What is the global namespace? Under the proposal, there is no such thing as "the" global namespace, so the question is meaningless. There's no such thing now, either -- there are just (multiple) module-level namespaces. > 2. Do global variables belong to the global namespace? Also meaningless as a corollary of 1. > 3. When it says "global x" does that mean x is a global variable? Yes, from the point of view of the scope containing the "global" declaration, it is. From the point of view of a different scope, it might not be. > Can you imagine trying to teach this? With a few well-chosen examples, yes. > either you have > to say (#1) there is no longer such a thing as "the global namespace", As I said, there isn't one now. Which namespace is "the global namespace" depends on which module you're looking from. Under the proposal, it would depend on which scope you're looking from. -- Greg From talin at acm.org Fri Nov 3 03:42:02 2006 From: talin at acm.org (Talin) Date: Thu, 02 Nov 2006 18:42:02 -0800 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: <454AA735.4080406@canterbury.ac.nz> References: <45491FB4.3000507@cs.byu.edu> <454AA735.4080406@canterbury.ac.nz> Message-ID: <454AAC7A.30906@acm.org> Greg Ewing wrote: > Ka-Ping Yee wrote: >> Before it is reasonable to change the meaning of "global", we would >> need to have coherent answers to these questions: >> >> 1. What is the global namespace? > > Under the proposal, there is no such thing as > "the" global namespace, so the question is > meaningless. > > There's no such thing now, either -- there > are just (multiple) module-level namespaces. I think that the argument that "there's no such thing as a global namespace, either now or in the future" are using an overly pedantic definition of the word "global". In Python "global" means, and has always meant, the namespace of the current module. It is the namespace that is "outside of any function definition", and that is true regardless of which module we are talking about. It is a "special" namespace because it has different semantics than the namespaces that are defined within a function scope. You can't just hand-wave this away - the 'global' namespace is far too important and useful to be just dismissed out of hand. I think the confusion stems from the fact that in many contexts, the word 'global' is a synonym for 'universal'. However, there's more than one planet in the universe...and no one is arguing that global variables in Python are in any way 'universal' variables. -- Talin From talin at acm.org Fri Nov 3 04:09:40 2006 From: talin at acm.org (Talin) Date: Thu, 02 Nov 2006 19:09:40 -0800 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <454AA2D3.5020706@canterbury.ac.nz> References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> <6e9196d20611011247k1b918ba2ga3b12cfca1cb7469@mail.gmail.com> <79990c6b0611020821x6f7fcf12yd1c5d1a846be62ff@mail.gmail.com> <1E120870-D70C-496F-9960-112A4066D02C@mac.com> <454A30B1.6060709@acm.org> <454AA2D3.5020706@canterbury.ac.nz> Message-ID: <454AB2F4.8020108@acm.org> Greg Ewing wrote: > Talin wrote: >> And that includes path strings read from config files, not just path >> objects. > > This is something new. I hadn't realised anyone > was considering that to be within the scope of > any of the proposed path modules. > > Obviously this is going to require defining a > standard external representation for paths. > What do you envisage such a representation > being like? Hmmm, maybe it is new. Let me take a moment to describe my motivations in this discussion. One of the reasons I really enjoy writing in Python is that I don't have to care what platform I am on. I use OS X at home and Win32 at work -- I'd like to be able to write code that works transparently on both systems. It doesn't matter to me whether paths are represented as objects, or as strings. What I'd really like is to be able to represent and manipulate paths in a platform-agnostic way, and do so with code that is both easy to read and concise. But that model can't be restricted to just the objects contained within the Python interpreter, we have to be able to read and write paths that are coming from the external environment. In many instances, those will be paths that are in a platform-specific format; but, given a choice, I am sure that many of us would prefer our data to be platform-neutral. One thing to note is that there doesn't have to be just one single platform neutral representation. For example, lets suppose we're writing a Python script to interpret Perforce configuration files (which happens a lot in my environment.) A typical Perforce mapping has two parts, a depot part and a local part: //depot/projects/metabuilder/src/... //myroot/dev/src/... These 'paths' are the same regardless of what host you happen to be running on. If you plan on doing anything intelligent with them (like scanning for differences with the locally checked-out files or something), you'll probably want to translate them into some form that the local filesystem can interpret. So here we have an example of an already-existing, platform-neutral representation of paths, but I would never suggest that Python adopt wholesale the Perforce representation of paths as the only platform-neutral solution. One might ask - is it the responsibility of the Python standard path module to even deal with such things? Since these are not sanctioned operating system paths, why should Python have a responsibility to support them? That's a good argument, but on the other hand - it's awfully convenient to be able to use the existing os.path to process stuff like this, especially considering the possibility that folks who roll their own Perforce-specific path functionality might 'get it wrong' in some subtle way. One way to solve this would be to define a platform-neutral path representation that can be "rendered" into either os-specific *or* application-specific formats. I could read a Perforce path string, convert that into an abstract path, do some work on it, and then convert it back to a Perforce path - or convert to a local filesystem path if I choose. Basically we'd have importers and exporters (format converters) for paths. On Win32, for example, we could have an "NT" importer for standard NT-style paths; a "Cygwin" importer that was more relaxed about a mixture of forward- and back-slashes; and so on. Of course, there would always be a "shortcut" for the local filesystem format - i.e., if no conversion was specified, then the local host format would be used as the default. How such importers and exporters would actually look syntactically, I am not sure yet. I'm open to ideas... -- Talin From sluggoster at gmail.com Fri Nov 3 04:23:43 2006 From: sluggoster at gmail.com (Mike Orr) Date: Thu, 2 Nov 2006 19:23:43 -0800 Subject: [Python-3000] Path Reform: Get the ball rolling In-Reply-To: <454AB2F4.8020108@acm.org> References: <6e9196d20610312222o1a348eaeo8c41ddba6c0331c9@mail.gmail.com> <45485815.1030001@acm.org> <6e9196d20611011247k1b918ba2ga3b12cfca1cb7469@mail.gmail.com> <79990c6b0611020821x6f7fcf12yd1c5d1a846be62ff@mail.gmail.com> <1E120870-D70C-496F-9960-112A4066D02C@mac.com> <454A30B1.6060709@acm.org> <454AA2D3.5020706@canterbury.ac.nz> <454AB2F4.8020108@acm.org> Message-ID: <6e9196d20611021923l6faf2076u751969871e5eee8f@mail.gmail.com> On 11/2/06, Talin wrote: > One way to solve this would be to define a platform-neutral path > representation that can be "rendered" into either os-specific *or* > application-specific formats. I could read a Perforce path string, > convert that into an abstract path, do some work on it, and then convert > it back to a Perforce path - or convert to a local filesystem path if I > choose. Basically we'd have importers and exporters (format converters) > for paths. On Win32, for example, we could have an "NT" importer for > standard NT-style paths; a "Cygwin" importer that was more relaxed about > a mixture of forward- and back-slashes; and so on. > > Of course, there would always be a "shortcut" for the local filesystem > format - i.e., if no conversion was specified, then the local host > format would be used as the default. > > How such importers and exporters would actually look syntactically, I am > not sure yet. I'm open to ideas... This sounds like a use case for a directory-component path object. If it treats paths as sequences like ['usr', 'local', 'lib', 'python2.4'] and you had to input them that way too, it would be a platform-neutral format. You'd still need to specify absolute/relative path, and in some cases choose the root for an absolute path, which is why Noam's module uses a special "root object" for the first element. This weekend I plan to code up a MiniPath containing the pathname-manipulation features of PEP 355, a MediumPath subclass adding the filesystem-dependent methods that some ppl frown on, and a MaxiPath subclass with my favorite additional methods. I'm thinking of making the path module a class attribute, so that you can subclass any of these and use ntpath or macpath instead of your default if you want. I'll also try my hand at Glyph's "safe join" feature if I can get it to work right, but that will be an optional separate method: either a .child instance method or a class method (not sure which is better). -- Mike Orr From python at zesty.ca Fri Nov 3 08:50:46 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Fri, 3 Nov 2006 01:50:46 -0600 (CST) Subject: [Python-3000] The meaning of "global variable" In-Reply-To: <454AA735.4080406@canterbury.ac.nz> References: <45491FB4.3000507@cs.byu.edu> <454AA735.4080406@canterbury.ac.nz> Message-ID: On Fri, 3 Nov 2006, Greg Ewing wrote: > > Before it is reasonable to change the meaning of "global", we would > > need to have coherent answers to these questions: > > > > 1. What is the global namespace? > > Under the proposal, there is no such thing as > "the" global namespace, so the question is > meaningless. > > There's no such thing now, either -- there > are just (multiple) module-level namespaces. Okay... tell that to all the writers of Python books and webpages that mention "the global namespace". Should we remove any occurrences of this phrase from the official Python documentation as well? > > either you have > > to say (#1) there is no longer such a thing as "the global namespace", > > As I said, there isn't one now. Which namespace > is "the global namespace" depends on which module > you're looking from. Under the proposal, it would > depend on which scope you're looking from. No -- it's much worse than that. That's the problem: even if you know which scope you're looking from, you wouldn't be able to point to anything and say "that's the global namespace". If we repurpose "global" to mean "nonlocal", the declaration "global a, b, c" might mean that a, b, and c all belong to different namespaces. Right now, in any given context, "the global namespace" has a clear meaning: the namespace shared by the whole file. With the new meaning of the "global" keyword, "global namespace" becomes meaningless no matter what the context. In Python, the concepts of "global", "global variable", and "global namespace" are too well established to destroy. -- ?!ng From fredrik at pythonware.com Fri Nov 3 09:53:18 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 03 Nov 2006 09:53:18 +0100 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <454A996F.6090400@canterbury.ac.nz> References: <45491FB4.3000507@cs.byu.edu> <454A996F.6090400@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Besides, the BDFL is going to do whatever he wants > anyway, so let's just leave him to get on with it. :-) but where is he? has anyone seen him lately ? From nnorwitz at gmail.com Fri Nov 3 09:57:18 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Fri, 3 Nov 2006 00:57:18 -0800 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> <454A996F.6090400@canterbury.ac.nz> Message-ID: On 11/3/06, Fredrik Lundh wrote: > Greg Ewing wrote: > > > Besides, the BDFL is going to do whatever he wants > > anyway, so let's just leave him to get on with it. :-) > > but where is he? has anyone seen him lately ? Sure, he's been busy at work and sick recently. He'll probably resurface in the next couple of weeks or so. n From ncoghlan at gmail.com Fri Nov 3 11:12:05 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 03 Nov 2006 20:12:05 +1000 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> <454AA735.4080406@canterbury.ac.nz> Message-ID: <454B15F5.2090303@gmail.com> Ka-Ping Yee wrote: > Right now, in any given context, "the global namespace" has a clear > meaning: the namespace shared by the whole file. With the new meaning > of the "global" keyword, "global namespace" becomes meaningless no > matter what the context. In Python, the concepts of "global", "global > variable", and "global namespace" are too well established to destroy. Exactly. A lookup of a bare name can be resolved in one of the following kinds of namespace: 1. function locals (either the current function or an enclosing function) 2. module globals 3. builtins The new PEP is about being able to better distinguish the case where the name is being resolved in an enclosing function's local namespace instead of the current function's local namespace. Simply moving the ambiguity to the second entry in the list above is NOT an improvement! A new keyword, on the other hand, allows the list to be split relatively cleanly into four possible locations: 1. function locals of the current function (local variables) 2. function locals of an enclosing function (closure variables) 3. module globals (global variables) 4. builtins (builtin names) A name for the second category other than 'nonlocal' would be nice, but I find it infinitely preferable to trying to redefine global. The only requirement I would place on it is that "nonlocal n" when there is no 'n' defined in a surrounding scope should be a SyntaxError. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From talin at acm.org Fri Nov 3 12:17:00 2006 From: talin at acm.org (Talin) Date: Fri, 03 Nov 2006 03:17:00 -0800 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> <454A996F.6090400@canterbury.ac.nz> Message-ID: <454B252C.7060304@acm.org> Fredrik Lundh wrote: > Greg Ewing wrote: > >> Besides, the BDFL is going to do whatever he wants >> anyway, so let's just leave him to get on with it. :-) > > but where is he? has anyone seen him lately ? Also bear in mind that the BDFL doesn't simply make pronouncements in a vacuum...he really does take these arguments into account (when he has time to read them) which influence his decisions. (You see, instead of the classic angel and devil hovering over his shoulders, he has little Fredrik Lundhs and Greg Ewings and Nick Coghlans and all the others, arguing the fine points of pythonicity, each trying to sway him to to their point of view... ...the poor ba***rd.) -- Talin From rrr at ronadam.com Fri Nov 3 20:55:50 2006 From: rrr at ronadam.com (Ron Adam) Date: Fri, 03 Nov 2006 13:55:50 -0600 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> <454AA735.4080406@canterbury.ac.nz> Message-ID: Ka-Ping Yee wrote: > On Fri, 3 Nov 2006, Greg Ewing wrote: >>> Before it is reasonable to change the meaning of "global", we would >>> need to have coherent answers to these questions: >>> >>> 1. What is the global namespace? >> Under the proposal, there is no such thing as >> "the" global namespace, so the question is >> meaningless. >> >> There's no such thing now, either -- there >> are just (multiple) module-level namespaces. > > Okay... tell that to all the writers of Python books and webpages > that mention "the global namespace". > > Should we remove any occurrences of this phrase from the official > Python documentation as well? > >>> either you have >>> to say (#1) there is no longer such a thing as "the global namespace", >> As I said, there isn't one now. Which namespace >> is "the global namespace" depends on which module >> you're looking from. Under the proposal, it would >> depend on which scope you're looking from. > > No -- it's much worse than that. That's the problem: even if you > know which scope you're looking from, you wouldn't be able to point > to anything and say "that's the global namespace". If we repurpose > "global" to mean "nonlocal", the declaration "global a, b, c" might > mean that a, b, and c all belong to different namespaces. > > Right now, in any given context, "the global namespace" has a clear > meaning: the namespace shared by the whole file. With the new meaning > of the "global" keyword, "global namespace" becomes meaningless no > matter what the context. In Python, the concepts of "global", "global > variable", and "global namespace" are too well established to destroy. > > > -- ?!ng I'll put in a 'Me Too' here. (For what it's worth.) You've convinced me that reusing global in verbatim for both global and nested parent scopes would create a confusing difference between 2.x and 3.x versions at least in documentation. How about limiting nonlocal to just the immediate parent scope and using 'parent' as the keyword? Are there many common use cases for accessing grandparent name spaces? That might still be done by repeating 'parent' declarations in nested functions for a name and thereby pulling access to it down the tree. That would be vary explicit and since it's not something you would do very often, it wouldn't be much of a bother. Limiting it to only the parent scope may also limit possible unintended side effects in the case a name is not in the scope that was expected. (Possibly due to omission or deleting it at some point while editing.) Cheers, Ron From mbk.lists at gmail.com Fri Nov 3 21:09:44 2006 From: mbk.lists at gmail.com (Mike Krell) Date: Fri, 3 Nov 2006 13:09:44 -0700 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: References: <454AA735.4080406@canterbury.ac.nz> Message-ID: > How about [...] using 'parent' as the keyword? I once proposed 'parent' in a similar thread a long time ago. It's a non-starter because it's a commonly used variable name. Mike From rrr at ronadam.com Fri Nov 3 21:11:42 2006 From: rrr at ronadam.com (Ron Adam) Date: Fri, 03 Nov 2006 14:11:42 -0600 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: References: <454AA735.4080406@canterbury.ac.nz> Message-ID: Mike Krell wrote: >> How about [...] using 'parent' as the keyword? > > I once proposed 'parent' in a similar thread a long time ago. > > It's a non-starter because it's a commonly used variable name. > > Mike I thought it might be. How about. __parent__ Just kidding, (I think), of course that looks too much like a method name. Ron From ntoronto at cs.byu.edu Fri Nov 3 21:28:07 2006 From: ntoronto at cs.byu.edu (Neil Toronto) Date: Fri, 03 Nov 2006 13:28:07 -0700 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: References: <454AA735.4080406@canterbury.ac.nz> Message-ID: <454BA657.6080404@cs.byu.edu> Ron Adam wrote: > Mike Krell wrote: > >>> How about [...] using 'parent' as the keyword? >>> >> I once proposed 'parent' in a similar thread a long time ago. >> >> It's a non-starter because it's a commonly used variable name. >> >> Mike >> > > > I thought it might be. > > How about. __parent__ > > > Just kidding, (I think), of course that looks too much like a method name. > Heck - just call it mijn_ouder or something. That even *looks* like "outer," and I'll bet it's not in anybody's code, either... Neil From qrczak at knm.org.pl Fri Nov 3 21:33:18 2006 From: qrczak at knm.org.pl (Marcin 'Qrczak' Kowalczyk) Date: Fri, 03 Nov 2006 21:33:18 +0100 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: (Ron Adam's message of "Fri, 03 Nov 2006 13:55:50 -0600") References: <45491FB4.3000507@cs.byu.edu> <454AA735.4080406@canterbury.ac.nz> Message-ID: <877iycp8xd.fsf@qrnik.zagroda> Ron Adam writes: > How about limiting nonlocal to just the immediate parent scope and > using 'parent' as the keyword? Please don't put unnecessary limits. There are two ways to refer to a variable: read it or assign it. Currently reading doesn't create a new variable, while assigning does (at the beginning of the function which assigns it). 'nonlocal' should mean: refer to the same variable as would be referred to if the current function did not assign it, only read it. I don't care much whether 'global' will be retained as is, or whether 'nonlocal' will actually be spelled 'global'. If I had to decide, I would keep 'global' with its current meaning because of the name: "global" means here "a single variable which exists during the whole duration of the program", not "visible from everywhere", which is appropriate for the current meaning but not for the proposed 'nonlocal'. -- __("< Marcin Kowalczyk \__/ qrczak at knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/ From ntoronto at cs.byu.edu Fri Nov 3 21:36:33 2006 From: ntoronto at cs.byu.edu (Neil Toronto) Date: Fri, 03 Nov 2006 13:36:33 -0700 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: Message-ID: <454BA851.4010200@cs.byu.edu> Ka-Ping Yee wrote: > Hi folks, > > I have finally completed a draft of a PEP on rebinding of names > in outer scopes. I've tried to go back and gather all of the > (amazingly numerous) proposals -- if i've forgotten or misattributed > any, let me know and i'll be happy to correct them. > > I look forward to your thoughts on it: Is there any particular reason this *must* be a Py3k PEP? (Aside from the fact that we're discussing it on the Py3k list, of course.) Some of the solutions discussed for historical context would *definitely* be bad ideas for a 2.x (as would changing the semantics of 'global'), but I can't see why the solution proposed in the PEP couldn't be done sooner. Neil From python at zesty.ca Fri Nov 3 21:58:58 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Fri, 3 Nov 2006 14:58:58 -0600 (CST) Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <454BA851.4010200@cs.byu.edu> References: <454BA851.4010200@cs.byu.edu> Message-ID: On Fri, 3 Nov 2006, Neil Toronto wrote: > Is there any particular reason this *must* be a Py3k PEP? (Aside from > the fact that we're discussing it on the Py3k list, of course.) Some of > the solutions discussed for historical context would *definitely* be bad > ideas for a 2.x (as would changing the semantics of 'global'), but I > can't see why the solution proposed in the PEP couldn't be done sooner. Well, i guess that's true. It doesn't seem out of the question for 2.x. -- ?!ng From python at zesty.ca Fri Nov 3 22:19:21 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Fri, 3 Nov 2006 15:19:21 -0600 (CST) Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: Message-ID: On Thu, 2 Nov 2006, Christian Tanzer wrote: > As you allow > > nonlocal x, y, z > > and > > x, y, z = 1, 2, 3 > > is legal Python (and does the right thing), why not allow > > nonlocal x, y, z = 1, 2, 3 > > too? > > That's one less rule needed to be learned and enforced. I suppose that's fine. Also i guess nonlocal x = y = 0 seems okay. We have to be clear that you can't assign to anything that you could in a normal assignment though -- e.g.: nonlocal foo.bar, a[5] = 1, 2 is not allowed. I've updated the PEP to allow multiple assignment and to specify the exact grammar of the "nonlocal" statement. http://zesty.ca/python/pep-3104.html -- ?!ng From solipsis at pitrou.net Fri Nov 3 22:51:56 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 03 Nov 2006 22:51:56 +0100 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: Message-ID: <1162590716.4428.23.camel@fsol> Le vendredi 03 novembre 2006 ? 15:19 -0600, Ka-Ping Yee a ?crit : > I suppose that's fine. Also i guess > > nonlocal x = y = 0 That's ambiguous. Is only "x" nonlocal, or are both "x" and "y"? What's the point anyway? if you write "nonlocal x = 0", it means you aren't reusing the variable value between calls... Of course the variable could be re-used by another inner function, but it's not common to have lots of inner functions sharing variables of a single outer function (people would rather write a class). From python at zesty.ca Fri Nov 3 23:22:42 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Fri, 3 Nov 2006 16:22:42 -0600 (CST) Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <1162590716.4428.23.camel@fsol> References: <1162590716.4428.23.camel@fsol> Message-ID: On Fri, 3 Nov 2006, Antoine Pitrou wrote: > Le vendredi 03 novembre 2006 ? 15:19 -0600, Ka-Ping Yee a ?crit : > > I suppose that's fine. Also i guess > > > > nonlocal x = y = 0 > > That's ambiguous. > Is only "x" nonlocal, or are both "x" and "y"? Only x is nonlocal. nonlocal x = y = 0 is equivalent to nonlocal x; x = y = 0 > What's the point anyway? The point is to make the statement easy to explain. As Christian suggested, prohibiting multiple assignment is an exception: an additional rule to remember. I found his argument convincing, and realized that prohibiting chained assignment is another exception. Consistently allowing these forms of assignment makes the description simple: If you see 'nonlocal' in front of an assignment, it means the same as a 'nonlocal' declaration (read from the beginning up to the assignment operator) followed by the assignment. This tells you both what the statement means and how to tell whether your syntax is valid. No exceptions to memorize. -- ?!ng From solipsis at pitrou.net Fri Nov 3 23:50:37 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 03 Nov 2006 23:50:37 +0100 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <1162590716.4428.23.camel@fsol> Message-ID: <1162594237.4428.37.camel@fsol> Le vendredi 03 novembre 2006 ? 16:22 -0600, Ka-Ping Yee a ?crit : > Only x is nonlocal. > > nonlocal x = y = 0 > > is equivalent to > > nonlocal x; x = y = 0 Perhaps, but the point is that it's non-intuitive from reading. Just like the old "char * x, y" annoyance in C. > The point is to make the statement easy to explain. As Christian > suggested, prohibiting multiple assignment is an exception: an > additional rule to remember. You are just trading one rule for another rule: the rule that only the leftmost variable in the assignment chain is nonlocal. In both cases, you have a non-obvious rule to remember. But in one case, the rule is enforced by syntax, while in the other, it can silently trigger bugs if not respected. OTOH if you forbid assignment altogether, there is nothing special to explain that is not already explained for "global". From greg.ewing at canterbury.ac.nz Sat Nov 4 01:58:07 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 04 Nov 2006 13:58:07 +1300 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: <454B15F5.2090303@gmail.com> References: <45491FB4.3000507@cs.byu.edu> <454AA735.4080406@canterbury.ac.nz> <454B15F5.2090303@gmail.com> Message-ID: <454BE59F.9030301@canterbury.ac.nz> Nick Coghlan wrote: > A new keyword, on the other hand, allows the list to be split relatively > cleanly into four possible locations: > > 1. function locals of the current function (local variables) > 2. function locals of an enclosing function (closure variables) > 3. module globals (global variables) > 4. builtins (builtin names) I don't see what's so important about treating the module namespace differently from other lexically enclosing scopes for the purpose of resolving names. Especially only for *assignment* to names. For read access, we seem to get on just fine without any keywords to give us a hint what level to look in. What's different about assignment that we suddenly feel such a need for them? -- Greg From greg.ewing at canterbury.ac.nz Sat Nov 4 02:07:44 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 04 Nov 2006 14:07:44 +1300 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: Message-ID: <454BE7E0.5010501@canterbury.ac.nz> Ka-Ping Yee wrote: > nonlocal x = y = 0 I don't really like the idea of allowing assignment in the nonlocal/global/whatever statement. It leads to questions like what happens if some assignments in a scope are qualified with nonlocal and others aren't. Certainly answers can be given, but it's more complexity in the rules and more to learn. If declaration and assignment are kept separate, these questions don't arise in the first place. -- Greg From greg.ewing at canterbury.ac.nz Sat Nov 4 02:11:55 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 04 Nov 2006 14:11:55 +1300 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <1162590716.4428.23.camel@fsol> Message-ID: <454BE8DB.20203@canterbury.ac.nz> Ka-Ping Yee wrote: > nonlocal x = y = 0 > > is equivalent to > > nonlocal x; x = y = 0 This is far from obvious. And what happens if you want both x and y to be nonlocal? Can you write nonlocal x = nonlocal y = 0 If so, this is getting even more complicated for very little benefit. > The point is to make the statement easy to explain. It's even easier to explain if assignment isn't allowed at all. -- Greg From ntoronto at cs.byu.edu Sat Nov 4 03:50:41 2006 From: ntoronto at cs.byu.edu (Neil Toronto) Date: Fri, 03 Nov 2006 19:50:41 -0700 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <454BE8DB.20203@canterbury.ac.nz> References: <1162590716.4428.23.camel@fsol> <454BE8DB.20203@canterbury.ac.nz> Message-ID: <454C0001.1080301@cs.byu.edu> Greg Ewing wrote: > Ka-Ping Yee wrote: > > >> nonlocal x = y = 0 >> >> is equivalent to >> >> nonlocal x; x = y = 0 >> > > This is far from obvious. And what happens if > you want both x and y to be nonlocal? Can you > write > > nonlocal x = nonlocal y = 0 > > If so, this is getting even more complicated > for very little benefit. > > >> The point is to make the statement easy to explain. >> > > It's even easier to explain if assignment isn't > allowed at all. > Sorry about the "mee to" post, but me too: +1 on no assignment allowed. I can't think of any way to do it that would be intuitive and obvious to *most* people and wouldn't require a separate FAQ entry besides the one that explains 'global' and 'nonlocal.' Neil From ncoghlan at gmail.com Sat Nov 4 04:23:03 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 04 Nov 2006 13:23:03 +1000 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: <454BE59F.9030301@canterbury.ac.nz> References: <45491FB4.3000507@cs.byu.edu> <454AA735.4080406@canterbury.ac.nz> <454B15F5.2090303@gmail.com> <454BE59F.9030301@canterbury.ac.nz> Message-ID: <454C0797.9010708@gmail.com> Greg Ewing wrote: > I don't see what's so important about treating the > module namespace differently from other lexically > enclosing scopes for the purpose of resolving names. > Especially only for *assignment* to names. > > For read access, we seem to get on just fine without > any keywords to give us a hint what level to look > in. What's different about assignment that we > suddenly feel such a need for them? Because read access doesn't have any non-local effects, but write access does. In most cases, it doesn't matter all that much which namespace a value came from, but where you are putting it can matter a great deal in terms of its visibility and lifecycle. 1. Assigning to x when x is a local variable: - only this function and enclosed functions can see the change - 'x' will now be found in locals() in this function - reference lasts until end of current function execution 2. Assigning to x when x is a global variable: - all code in this module can see the change - every user of the module can see the change (as a module attribute) - 'x' will now be found in globals() in this module - reference lasts until module dictionary is cleared Currently, these are the only 2 possible cases. The PEP proposes adding a 3rd case: 3. Assigning to x when x is a closure variable: - the enclosing function and all enclosed functions can see the change - will not cause 'x' to appear in either globals() or locals() - reference lasts until function object is destroyed The 3rd case is significantly different from the 2nd in terms of both the visibility of any changes (particularly the points about whether or not changes are going to be visible as module attributes and entries in globals()) and the lifecycle of the reference (whether it is released along with the function object or the module dictionary). Obscuring these differences by re-using the keyword global to denote write access to closure variables as well as global variables wouldn't be doing anyone any favours. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Sat Nov 4 04:36:52 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 04 Nov 2006 13:36:52 +1000 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <454C0001.1080301@cs.byu.edu> References: <1162590716.4428.23.camel@fsol> <454BE8DB.20203@canterbury.ac.nz> <454C0001.1080301@cs.byu.edu> Message-ID: <454C0AD4.5090602@gmail.com> Neil Toronto wrote: >>> The point is to make the statement easy to explain. >>> >> It's even easier to explain if assignment isn't >> allowed at all. >> > > Sorry about the "mee to" post, but me too: +1 on no assignment allowed. > I can't think of any way to do it that would be intuitive and obvious to > *most* people and wouldn't require a separate FAQ entry besides the one > that explains 'global' and 'nonlocal.' 'Me too' posts can be useful sometimes :) Count me as another +1 on disallowing the shorthand version. Rationale: 1. We've lived without it in the case of global 2. We can add it later if we decide we really need it, but if we start with it and it turns out to be confusing or error-prone, taking it away would be a serious pain. 3. Leaving it out ducks some hard questions like "what does this do?": def weird(local=True): if local: n = 1 # Does this alter the closure variable? else: nonlocal n += i return n Cheers, Nick. P.S. For those who asked "what's the point of the shorthand?", it's intended more for use with augmented assignment rather than basic assignment: def inc(i=1): nonlocal n += i return n I can see the benefit in shortening toy examples, but I don't think it even comes close to being worth the extra complexity. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Sat Nov 4 04:40:48 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 04 Nov 2006 13:40:48 +1000 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <454BA851.4010200@cs.byu.edu> References: <454BA851.4010200@cs.byu.edu> Message-ID: <454C0BC0.20503@gmail.com> Neil Toronto wrote: > Ka-Ping Yee wrote: >> Hi folks, >> >> I have finally completed a draft of a PEP on rebinding of names >> in outer scopes. I've tried to go back and gather all of the >> (amazingly numerous) proposals -- if i've forgotten or misattributed >> any, let me know and i'll be happy to correct them. >> >> I look forward to your thoughts on it: > > Is there any particular reason this *must* be a Py3k PEP? (Aside from > the fact that we're discussing it on the Py3k list, of course.) Some of > the solutions discussed for historical context would *definitely* be bad > ideas for a 2.x (as would changing the semantics of 'global'), but I > can't see why the solution proposed in the PEP couldn't be done sooner. I think you answered your own question - some of the options up for consideration in the PEP are *not* compatible with 2.x, so targeting Py3k makes for a freer discussion of the possibilities. If the result turns out to be backwards compatible, then we can retarget Python 2.6, but that's actually the case for *all* of the Py3k PEPs (even the backwards incompatible PEPs will likely have elements which can be incorporated into the 2.x series to ease the eventual transition). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From greg.ewing at canterbury.ac.nz Sat Nov 4 01:57:58 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 04 Nov 2006 13:57:58 +1300 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> <454AA735.4080406@canterbury.ac.nz> Message-ID: <454BE596.4020507@canterbury.ac.nz> Ron Adam wrote: > How about limiting nonlocal to just the immediate parent scope and using > 'parent' as the keyword? That could lead to confusing situations. What should the following do: def f(): x = 42 def g(): def h(): parent x x = 88 Should the assignment to x in h() create a name in the scope of g() even though there's no assignment in g() to establish that as its home scope? Should it be an error? -- Greg From python at zesty.ca Sat Nov 4 22:54:03 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Sat, 4 Nov 2006 15:54:03 -0600 (CST) Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <20061102052412.A91B36F4198@longblack.object-craft.com.au> References: <45491FB4.3000507@cs.byu.edu> <20061102014059.BC4BB6F4198@longblack.object-craft.com.au> <20061102052412.A91B36F4198@longblack.object-craft.com.au> Message-ID: On Thu, 2 Nov 2006, Andrew McNamara wrote: > >The global scope is the widest scope in which you can declare a variable. > > This is inaccurate at best. No, my statement is correct. What you are disputing is not what i wrote: The global scope is the widest scope in which you can declare a variable. You cannot declare variables in builtin scope in Python. You can stuff variables into the builtin module with some manipulation, but it is a very rare thing to do, it is not encouraged as a normal style of programming, and it is not supported by the language syntax. Python, C/C++, JavaScript, Ruby, and Perl all have this in common: A "global variable" is visible to the entire file and does not belong to any particular function. > To be honest, I'm +0 on using "global" in the way GvR proposes. My > point is that python's globals are already different from other common > languages, and people cope with that. That does not constitute an argument in favour of anything. For example, Python's integers are already different from other common languages, because they have unlimited range. That is not a reason to make "integers" include, say, floating point numbers. -- ?ing From rrr at ronadam.com Sun Nov 5 05:28:23 2006 From: rrr at ronadam.com (Ron Adam) Date: Sat, 04 Nov 2006 22:28:23 -0600 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: <454BE596.4020507@canterbury.ac.nz> References: <45491FB4.3000507@cs.byu.edu> <454AA735.4080406@canterbury.ac.nz> <454BE596.4020507@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Ron Adam wrote: > >> How about limiting nonlocal to just the immediate parent scope and using >> 'parent' as the keyword? > > That could lead to confusing situations. What should > the following do: > > def f(): > x = 42 > def g(): > def h(): > parent x > x = 88 > > Should the assignment to x in h() create a name in > the scope of g() even though there's no assignment > in g() to establish that as its home scope? Should > it be an error? > > -- > Greg Not confusing at all. It would work just like global does in the same situation. The parent statement here does not create a name, it only designates the scope where a name will be written. I presume you meant: def f(): x = 42 def g(): def h(): parent x x = 88 The x would be a new local x in function g and have no effect on the x in function f. Parent would do exactly what it means in this case, its a reference to a name in only the parent scope. Unless you were to do: def f(): x = 42 def g(): parent x def h(): parent x x = 88 Then all references to x would be the local x in function f. This is what I meant that it might be possibly to pull a reference down to inner scopes. It gives more control on what name space you are writing to and less chance of writing to a grandparent scope unintentionally because of a variable name being left out, having its name changed, or has been deleted in the immediate parent function while editing. I also don't think this would be too much trouble as nested functions don't tend to be more than 2 or 3 levels deep very often and if it becomes a problem of having too many parent statements, it's probably a good sign a class would be better. IMHO, and if a suitable keyword meaning parent could be found. Ron From g.brandl at gmx.net Sun Nov 5 10:49:34 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 05 Nov 2006 10:49:34 +0100 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> <454AA735.4080406@canterbury.ac.nz> <454BE596.4020507@canterbury.ac.nz> Message-ID: Ron Adam wrote: > I presume you meant: > > def f(): > x = 42 > def g(): > def h(): > parent x > x = 88 > > > The x would be a new local x in function g and have no effect on the x in > function f. Parent would do exactly what it means in this case, its a reference > to a name in only the parent scope. This *is* confusing, even if the word "parent" is a slight hint on what's really going on. > Unless you were to do: > > def f(): > x = 42 > def g(): > parent x > def h(): > parent x > x = 88 > > Then all references to x would be the local x in function f. "So why do I have to declare x in g, if I don't use it there?" This is really ugly. Georg From rrr at ronadam.com Sun Nov 5 22:16:34 2006 From: rrr at ronadam.com (Ron Adam) Date: Sun, 05 Nov 2006 15:16:34 -0600 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> <454AA735.4080406@canterbury.ac.nz> <454BE596.4020507@canterbury.ac.nz> Message-ID: Georg Brandl wrote: > Ron Adam wrote: > >> I presume you meant: >> >> def f(): >> x = 42 >> def g(): >> def h(): >> parent x >> x = 88 >> >> >> The x would be a new local x in function g and have no effect on the x in >> function f. Parent would do exactly what it means in this case, its a reference >> to a name in only the parent scope. > > This *is* confusing, even if the word "parent" is a slight hint on what's > really going on. Why is it confusing to you, what would you expect to happen here? def f(): def g(): def h(): parent x x = 42 By not limiting parent to just the parent scope you create exceptions. The rule becomes: The keyword (*)nonlocal designates a name will be written to in the closest enclosing "parent" scope *except* when a pre-existing matching name exists in a scope further up. To me that is more confusing than always referring to the closest enclosing scope without exception. (*) Nonlocal is better a better term for implicit scope selection. Parent is better suited for the explicit scope selection suggested. See below. >> Unless you were to do: >> >> def f(): >> x = 42 >> def g(): >> parent x >> def h(): >> parent x >> x = 88 >> >> Then all references to x would be the local x in function f. > > "So why do I have to declare x in g, if I don't use it there?" This is really > ugly. > > Georg It's a choice. Explicit scope selection: This depends only on the existence of parent statements to determine which scope a name will be bound in. Parent redirects an assignment to the parent scope. Successive parent statements can redirect the binding of names to scopes further up. Implicit scope selection: This depends on the pre-existence of a matching bound name in *any* parent scope to determine where write access will be allowed. If no matching name if found write access will be the nearest enclosing scope. There are more opportunities for silent bugs with the looser implicit scope selection. That was the point I was making. I'll leave it up to the experts to decide which is better. Cheers, Ron From qrczak at knm.org.pl Sun Nov 5 22:46:18 2006 From: qrczak at knm.org.pl (Marcin 'Qrczak' Kowalczyk) Date: Sun, 05 Nov 2006 22:46:18 +0100 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: (Ron Adam's message of "Sun, 05 Nov 2006 15:16:34 -0600") References: <45491FB4.3000507@cs.byu.edu> <454AA735.4080406@canterbury.ac.nz> <454BE596.4020507@canterbury.ac.nz> Message-ID: <87odrla7o5.fsf@qrnik.zagroda> Ron Adam writes: > By not limiting parent to just the parent scope you create exceptions. The rule > becomes: > > The keyword (*)nonlocal designates a name will be written to in the > closest enclosing "parent" scope *except* when a pre-existing matching name > exists in a scope further up. > > To me that is more confusing than always referring to the closest enclosing > scope without exception. The rule should be: The keyword 'nonlocal' causes the lookup to be performed as if there were no assignments to that variable in the scope containing the 'nonlocal' declaration. No exceptions are needed, and no limitation to the immediately surrounding scope is needed. Read accesses already have fine scoping rules, as long as there is something to determine the scope. The only problem with current rules is that assignment is coupled with creating a new variable, and 'nonlocal' just allows to decouple that. -- __("< Marcin Kowalczyk \__/ qrczak at knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/ From mbk.lists at gmail.com Sun Nov 5 23:58:59 2006 From: mbk.lists at gmail.com (Mike Krell) Date: Sun, 5 Nov 2006 15:58:59 -0700 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: <87odrla7o5.fsf@qrnik.zagroda> References: <454AA735.4080406@canterbury.ac.nz> <454BE596.4020507@canterbury.ac.nz> <87odrla7o5.fsf@qrnik.zagroda> Message-ID: > The rule should be: > > The keyword 'nonlocal' causes the lookup to be performed as if there > were no assignments to that variable in the scope containing the > 'nonlocal' declaration. > > No exceptions are needed, and no limitation to the immediately > surrounding scope is needed. Read accesses already have fine scoping > rules, as long as there is something to determine the scope. The only > problem with current rules is that assignment is coupled with creating > a new variable, and 'nonlocal' just allows to decouple that. Exactly. There is no reason to place an artificial limitation on this feature. This is also why I advocate the spelling "use", because ISTM "use x" pretty much means exactly what it says. The main argument against the spelling "use" that I've seen is that it's too generic, i.e, one could "use" anything. This argument doesn't sway me because whatever we call the keyword, I think the programmer will have to look it up in the docs the first time they encounter it. Then they'll say "aha, I get it" and move on -- a minor one-time bump on the learning curve. Once you know what the keyword means, I think "use" is the spelling that makes the most sense, and it's easy to type, too. Mike From g.brandl at gmx.net Sun Nov 5 23:59:35 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 05 Nov 2006 23:59:35 +0100 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: <87odrla7o5.fsf@qrnik.zagroda> References: <45491FB4.3000507@cs.byu.edu> <454AA735.4080406@canterbury.ac.nz> <454BE596.4020507@canterbury.ac.nz> <87odrla7o5.fsf@qrnik.zagroda> Message-ID: Marcin 'Qrczak' Kowalczyk wrote: > Ron Adam writes: > >> By not limiting parent to just the parent scope you create exceptions. The rule >> becomes: >> >> The keyword (*)nonlocal designates a name will be written to in the >> closest enclosing "parent" scope *except* when a pre-existing matching name >> exists in a scope further up. >> >> To me that is more confusing than always referring to the closest enclosing >> scope without exception. > > The rule should be: > > The keyword 'nonlocal' causes the lookup to be performed as if there > were no assignments to that variable in the scope containing the > 'nonlocal' declaration. Plus, if there's no binding in an enclosing scope, an error is raised. (Which brings up the assymetry to today's global again, but is that really a problem?) Georg From greg.ewing at canterbury.ac.nz Mon Nov 6 00:21:26 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 06 Nov 2006 12:21:26 +1300 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> Message-ID: <454E71F6.7090103@canterbury.ac.nz> Mike Orr wrote: > .abspath() > .normpath() > .realpath() > .splitpath() > .relpath() > .relpathto() Seeing as the whole class is about paths, having "path" in the method names seems redundant. I'd prefer to see terser method names without any noise characters in them. -- Greg From python at zesty.ca Mon Nov 6 00:57:14 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Sun, 5 Nov 2006 17:57:14 -0600 (CST) Subject: [Python-3000] Backward compatibility for "nonlocal" Message-ID: As an aside to the discussion about "nonlocal", here are a couple of thoughts on backward compatibility. For some of the proposed keywords, here's the number of occurrences of the keyword in the current standard library (not including comments and docstrings): nonlocal 0 0 use 2 2 using 3 3 reuse 0 4 free 8 8 outer 5 147 global 126 214 (I used 'tokenize' to do this; let me know if you'd like me to count other possibilities.) This may or may not be a good measure of the likely impact of adding a new keyword. At least it's something to think about. It also occurred to me that this new keyword doesn't necessarily have to break anything, if we are willing to tolerate a little hackery in the Python lexer and parser. The situation is a bit reminiscent of 'as' -- we were able to add 'as' to the 'import' statement as though 'as' were a keyword, without actually making 'as' a reserved word in all code. In this case, the usage of "nonlocal" (or whatever we choose) as a keyword never overlaps with its usage as a variable name. The parser could easily tell the difference by checking whether: - The keyword is at the beginning of a statement. - The keyword is immediately followed by an identifier. If both of these things are true, then the keyword is a nonlocal declaration. Otherwise, it can be treated as a variable name. I'm not saying this is necessarily a good idea; i just wanted to note that it was an available option. We might think about a schedule for phasing in the feature: Phase 1. If "nonlocal" is seen in code, issue a warning that it will become a keyword in a future version. Phase 2. Have the parser distinguish when "nonlocal" is intended as a keyword and when as a variable name. Phase 3. Make "nonlocal" an official reserved word. All of these phases are optional -- we could skip straight to 3, or do 1 and then 3, or stop at 2, etc. -- ?!ng From greg.ewing at canterbury.ac.nz Mon Nov 6 01:00:28 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 06 Nov 2006 13:00:28 +1300 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: References: <454AA735.4080406@canterbury.ac.nz> <454BE596.4020507@canterbury.ac.nz> <87odrla7o5.fsf@qrnik.zagroda> Message-ID: <454E7B1C.8040204@canterbury.ac.nz> Mike Krell wrote: > This is also why I advocate the spelling "use", because > ISTM "use x" pretty much means exactly what it says. I beg to differ -- it means nothing at all to me. At least "global" or "nonlocal" sound like they have *something* to do with scope, even if it's not clear exactly what. -- Greg From python at zesty.ca Mon Nov 6 01:17:03 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Sun, 5 Nov 2006 18:17:03 -0600 (CST) Subject: [Python-3000] Backward compatibility for "nonlocal" In-Reply-To: References: Message-ID: On Sun, 5 Nov 2006, Ka-Ping Yee wrote: > nonlocal 0 0 > use 2 2 > using 3 3 > reuse 0 4 > free 8 8 > outer 5 147 > global 126 214 Oops, i forgot to explain: the first column is the count not including tests; the second column is the count in the entire Lib/ directory. -- ?!ng From rrr at ronadam.com Mon Nov 6 02:35:16 2006 From: rrr at ronadam.com (Ron Adam) Date: Sun, 05 Nov 2006 19:35:16 -0600 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> Message-ID: Mike Orr wrote: > The multi-argument constructor is a replacement for joining paths. > (The PEP says .joinpath was "problematic" without saying why.) This > could theoretically go either way, doing either the same thing as > os.path.join, getting a little smarter, or doing "safe" joins by > disallowing "/" embedded in string arguments. Some of this may be due to platform dependency. Maybe adding some equivalent mechanism like unicode.encode() and str.decode() to path objects to enable easier use of foreign or different types of paths in some situations would be good? In this case instead of a string encoding, it would be a path os encoding. Path("ABC").encode('windows') => a windows path string from a path object. Path().decode('ABC', 'windows') => a platform dependant path object Don't take these examples too literally, I'm just pointing out a concept that might be useful where paths from differing platforms may touch each other. You would probably want to use a different spelling for these so they don't get confused with string encodings. Could paths be handled with a string encoding? Just a thought. Cheers, Ron From andrewm at object-craft.com.au Mon Nov 6 02:47:19 2006 From: andrewm at object-craft.com.au (Andrew McNamara) Date: Mon, 06 Nov 2006 12:47:19 +1100 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> <20061102014059.BC4BB6F4198@longblack.object-craft.com.au> <20061102052412.A91B36F4198@longblack.object-craft.com.au> Message-ID: <20061106014719.61D6679419E@longblack.object-craft.com.au> >Python, C/C++, JavaScript, Ruby, and Perl all have this in common: > > A "global variable" is visible to the entire file and does > not belong to any particular function. I note that you didn't say "a global variable is visible to the entire application" - you've deliberately narrowed the definition to suit your argument. >> To be honest, I'm +0 on using "global" in the way GvR proposes. My >> point is that python's globals are already different from other common >> languages, and people cope with that. > >That does not constitute an argument in favour of anything. Sigh. I'll spell it out more fully then: Introducing new keywords is rude. Changing the behaviour of existing language constructs is rude. To implement this proposal, we need to do one or the other. Of the two, I lean very slightly in favour of making the semantics of "global" slightly more complex, on the basis that they're already slightly more complex than most other languages and people cope with that. >For example, Python's integers are already different from other common >languages, because they have unlimited range. That is not a reason to >make "integers" include, say, floating point numbers. I'm disappointed that you're resorting to straw-man arguments. I guess that means we're done now. -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/ From ntoronto at cs.byu.edu Mon Nov 6 03:02:41 2006 From: ntoronto at cs.byu.edu (Neil Toronto) Date: Sun, 05 Nov 2006 19:02:41 -0700 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: <454E7B1C.8040204@canterbury.ac.nz> References: <454AA735.4080406@canterbury.ac.nz> <454BE596.4020507@canterbury.ac.nz> <87odrla7o5.fsf@qrnik.zagroda> <454E7B1C.8040204@canterbury.ac.nz> Message-ID: <454E97C1.5000204@cs.byu.edu> Greg Ewing wrote: >Mike Krell wrote: > > >> This is also why I advocate the spelling "use", because >>ISTM "use x" pretty much means exactly what it says. >> >> > >I beg to differ -- it means nothing at all to >me. At least "global" or "nonlocal" sound like >they have *something* to do with scope, even >if it's not clear exactly what. > > And because it looks like the illegitimate child of a scabby old Pascal keyword, it'll make people look twice. I'm only being half facetious. Another half-facetious (maybe quarter) argument in its favor is that, even though it's short(-ish), it's rare. Any future Google search for "python nonlocal" will find hundreds of relevant documents. Heck, the first seven hits for that *right now* are on subject. This one's starting to grow on me... sort of like a cheese mold, but it's growing nonetheless. Neil From python at zesty.ca Mon Nov 6 05:28:22 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Sun, 5 Nov 2006 22:28:22 -0600 (CST) Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <20061106014719.61D6679419E@longblack.object-craft.com.au> References: <45491FB4.3000507@cs.byu.edu> <20061102014059.BC4BB6F4198@longblack.object-craft.com.au> <20061102052412.A91B36F4198@longblack.object-craft.com.au> <20061106014719.61D6679419E@longblack.object-craft.com.au> Message-ID: On Mon, 6 Nov 2006, Andrew McNamara wrote: > >Python, C/C++, JavaScript, Ruby, and Perl all have this in common: > > > > A "global variable" is visible to the entire file and does > > not belong to any particular function. > > I note that you didn't say "a global variable is visible to the entire > application" - you've deliberately narrowed the definition to suit > your argument. Let's be clear that we're discussing the same topic here. If we're trying to assess whether a particular meaning of "global" in Python will be familiar, surprising, easy to understand, etc. then we have to compare it to what "global" means in other contexts. That requires talking about what is in common among the other contexts. Are you trying to assert that most of the other languages have a common meaning for "global", from which Python has already diverged enough that it doesn't matter for Python to diverge further? But i think we agree it is not always true in programming languages that "global" means "visible to the entire application" -- you provided some good examples, in fact. So this cannot be part of the common meaning. Or are you trying to show that other languages have meanings for "global" that are all so different that there *is* no common meaning for Python to follow? That's the reason i offered, above, a statement of something that is clearly common among them. This is not intended to be the complete definition of "global" in all these contexts, as each language has its own way of integrating multiple modules. But because at least this much is common, this is part of the common meaning that people have come to expect. This is a meaning that Python conforms to now, and that Python would no longer fit if we redefined "global" to mean "belonging to any parent scope of the current scope". > Introducing new keywords is rude. Changing the behaviour of existing > language constructs is rude. To implement this proposal, we need to do one > or the other. Of the two, I lean very slightly in favour of making the > semantics of "global" slightly more complex, on the basis that they're > already slightly more complex than most other languages and people cope > with that. I am trying to warn the participants in this discussion that the cost of redefining "global" in this manner is rather higher than may seem initially apparent. The effect will be to confuse the meaning of "global" not just as a language keyword but as a programming concept: "global namespace" will not just have a different meaning -- it will come to have no meaning at all (or, alternatively, the language itself will be at odds with its own semantics), as i've explained [1]. This would be quite a bit more painful than introducing a keyword in a new version of Python that has been specifically designated as potentially backward incompatible, with time still remaining between now and then for gradual migration. -- ?!ng [1] http://mail.python.org/pipermail/python-3000/2006-November/004215.html From python at zesty.ca Mon Nov 6 06:16:37 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Sun, 5 Nov 2006 23:16:37 -0600 (CST) Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <454C0AD4.5090602@gmail.com> References: <1162590716.4428.23.camel@fsol> <454BE8DB.20203@canterbury.ac.nz> <454C0001.1080301@cs.byu.edu> <454C0AD4.5090602@gmail.com> Message-ID: On Sat, 4 Nov 2006, Nick Coghlan wrote: > Count me as another +1 on disallowing the shorthand version. > > Rationale: > 1. We've lived without it in the case of global > 2. We can add it later if we decide we really need it, but if we > start with it and it turns out to be confusing or error-prone, > taking it away would be a serious pain. That's a good point. We can start without it, or perhaps with just the simple form that Guido said he'd endorse, and find out later if people want to be able to do the other forms of assignment. > 3. Leaving it out ducks some hard questions like "what does this do?": > > def weird(local=True): > if local: > n = 1 # Does this alter the closure variable? > else: > nonlocal n += i > return n Maybe this will confuse people, maybe it won't. I think we do know that so far the possibility of writing: def weird(local=True): if local: n = 1 else: global n n += 1 return n doesn't seem to have caused problems. I can see how the shorthand form makes it more likely that people will want to put "global" in the middle of the function than at the beginning, though. -- ?!ng From talin at acm.org Mon Nov 6 08:39:50 2006 From: talin at acm.org (Talin) Date: Sun, 05 Nov 2006 23:39:50 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> Message-ID: <454EE6C6.4000701@acm.org> Mike Orr wrote: > Posted to python-dev and python-3000. Follow-ups to python-dev only please. > > So, let's say we strip this Path class to: I'm finally taking the time to sit down and go over this in detail. Here are some suggestions. > class Path(unicode): > Path("foo") > Path( Path("directory"), "subdirectory", "file") # Replaces > .joinpath(). > Path() For the constructor, I would write it as: Path( *components ) 'components' is an arbitrary number of path components, either strings or path objects. The components are joined together into a single path. Strings can also be wrapped with an object that indicates that the Path is in a platform- or application-specific format: # Explicitly indicate that the path string is in Windows NTFS format. Path( Path.format.NTFS( "C:\\Program Files" ) ) Note that it's OK to include path separators in the string that's passed to the format wrapper - these will get converted. Not including a format wrapper is equivalent to using the "local" wrapper: Path( Path.format.local( "C:\\Program Files" ) ) Where 'local' is an alias to the native path format for the host's default filesystem. The wrapper objects are themselves classes, and need not be in the "Path" namespace. For example: import p4 Path( p4.path.format( "//depot/files/..." ) ) This makes the set of specific path formats open-ended and extensible. Path format wrappers need not be built into the "Path" module. Each format wrapper will have a "to_path" method, that converts the specific path encoding into the universal path representation. Note that if there are multiple components, they don't have to be wrapped the same way: Path( Path.format.NTFS( "C:\\Program Files" ), Path.format.local( "Gimp" ) ) ...because the conversion to universal representation is done before the components are combined. One question to be asked is whether the path should be simplified or not. There are cases where you *don't* want the path to be simplified, and other cases where you do. Perhaps a keyword argument? Path( "C:\\Program Files", "../../Gimp", normalize = True ) > Path.cwd() No objection here. > Path("ab") + "c" => Path("abc") Wouldn't that be: Path( "ab" ) + "c" => Path( "ab", "c" ) ? It seems that the most common operation is concatenating components, not characters, although both should be easy. > .abspath() I've always thought this was a strange function. To be honest, I'd rather explicitly pass in the cwd(). > .normcase() So the purpose of this function is to get around the fact that on some platforms, comparisons between paths are case-sensitive, and on other platforms not. However, the reason this function seems weird to me is that most case-insensitive filesystems are case-preserving, which makes me thing that the real solution is to fix the comparison functions rather than mangling the string. (Although there's a hitch - its hard to make a case-insensitive dictionary that doesn't require a downcase'd copy of the key; Something I've long wanted was a userdict that allowed both the comparison and hash functions to be replaceable, but that's a different topic.) > .normpath() I'd rename this to "simplify", since it no longer needs to normalize the separator chars. (That's done by the wrappers.) > .realpath() Rename to resolve() or resolvelinks(). > .expanduser() > .expandvars() > .expand() Replace with expand( user=True, vars=True ) > .parent If parent was a function, you could pass in the number of levels go to up, i.e. parent( 2 ) to get the grandparent. > .name # Full filename without path > .namebase # Filename without extension I find the term 'name' ambiguous. How about: .filepart .basepart or: .filename .basename > .ext No problem with this > .drive Do we need to somehow unify the concept of 'drive' and 'unc' part? Maybe '.device' could return the part before the first directory name. > .splitpath() I'd like to replace this with: .component( slice_object ) where the semantics of 'component' are identical to __getitem__ on an array or tuple. So for example: Path( "a", "b" ).component( 0 ) => "a" Path( "a", "b" ).component( 1 ) => "b" Path( "a", "b" ).component( -1 ) => "b" Path( "a", "b" ).component( 0:1 ) => Path( "a", "b" ) Path( "a", "b" ).component( 1: ) => Path( "b" ) This is essentially the same as the "slice notation" proposal given earlier, except that explicitly tell the user that we are dealing with path components, not characters. > .stripext() How about: path.ext = '' > .splitunc() > .uncshare See above - UNC shouldn't be a special case. > .splitall() Something sadly lacking in os.path. > .relpath() Again, I'd rather that they pass in the cwd() explicitly. But I would like to see something like: .relativeto( path ) ...which computes the minimal relative path that goes from 'self' to 'path'. > .relpathto() Not sure what this does, since there's no argument defined. Additional methods: .format( wrapper_class ) ...converts the path into a filesystem-specific format. You can also get the same effect by "wrapping" the path object and calling str() str( Path.format.NTFS( Path( "a", "b", "c" ) ) ) Although it's a bit cumbersome. -- Talin From ncoghlan at gmail.com Mon Nov 6 10:22:25 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 06 Nov 2006 19:22:25 +1000 Subject: [Python-3000] The meaning of "global variable" In-Reply-To: References: <45491FB4.3000507@cs.byu.edu> <454AA735.4080406@canterbury.ac.nz> <454BE596.4020507@canterbury.ac.nz> <87odrla7o5.fsf@qrnik.zagroda> Message-ID: <454EFED1.4080407@gmail.com> Georg Brandl wrote: > Marcin 'Qrczak' Kowalczyk wrote: >> The rule should be: >> >> The keyword 'nonlocal' causes the lookup to be performed as if there >> were no assignments to that variable in the scope containing the >> 'nonlocal' declaration. > > Plus, if there's no binding in an enclosing scope, an error is raised. > > (Which brings up the assymetry to today's global again, but is that really > a problem?) I'd narrow the requirement such that module globals don't count - if you declare a variable nonlocal it *must* be initialised in an enclosing function scope or you will get a syntax error similar to this one: >>> def f(): ... x = 1 ... del x ... def g(): ... return x ... return g ... SyntaxError: can not delete variable 'x' referenced in nested scope Here, the compiler knows that g() references back to 'x' and hence considers the attempt to delete 'x' nonsensical. Attempting to reference 'nonlocal x' when x isn't a closure variable is also nonsensical - if you want a local variable then remove the declaration, if you want a global then declare x as a global. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Mon Nov 6 10:37:36 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 06 Nov 2006 19:37:36 +1000 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <20061106014719.61D6679419E@longblack.object-craft.com.au> References: <45491FB4.3000507@cs.byu.edu> <20061102014059.BC4BB6F4198@longblack.object-craft.com.au> <20061102052412.A91B36F4198@longblack.object-craft.com.au> <20061106014719.61D6679419E@longblack.object-craft.com.au> Message-ID: <454F0260.8020104@gmail.com> Andrew McNamara wrote: >> Python, C/C++, JavaScript, Ruby, and Perl all have this in common: >> >> A "global variable" is visible to the entire file and does >> not belong to any particular function. > > I note that you didn't say "a global variable is visible to the entire > application" - you've deliberately narrowed the definition to suit > your argument. In C/C++ to use another module's 'globals' you must redeclare their names in the current module (usually with a #include of the appropriate header file). The linker then takes care of associating the two declarations with the same defining module. In C, each symbol must be unique. In C++, the global variables may be in different namespaces, requiring either scope qualification or an appropriate using statement to access the names in the namespace. In Python, to use another module's globals you must import the module. You can then accessing the name as an attribute of the imported module (or you can use the alternate form of import and retrieve only the globals you are interested in). So tell me again how Python's globals are different from C++ ones? Sure, the linkage is done at runtime rather than build time, but the module cache means they're still global to the application (accessible from everywhere, always refer to the same object). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From murman at gmail.com Mon Nov 6 15:33:24 2006 From: murman at gmail.com (Michael Urman) Date: Mon, 6 Nov 2006 08:33:24 -0600 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <1162590716.4428.23.camel@fsol> <454BE8DB.20203@canterbury.ac.nz> <454C0001.1080301@cs.byu.edu> <454C0AD4.5090602@gmail.com> Message-ID: > def weird(local=True): > if local: > n = 1 > else: > global n # [ or nonlocal n, in Nick's ] > n += 1 > return n I'm not sure what you're both trying to explain here. First off, the above code yields a SyntaxWarning. Secondly all its accesses to n are global accesses. The global keyword is currently a modifier for the whole function. Thus the local parameter is misleading; it should be called reset. :1: SyntaxWarning: name 'n' is assigned to before global declaration >>> dis.dis(weird) 2 0 LOAD_FAST 0 (local) 3 JUMP_IF_FALSE 10 (to 16) 6 POP_TOP 3 7 LOAD_CONST 1 (1) 10 STORE_GLOBAL 1 (n) 13 JUMP_FORWARD 11 (to 27) >> 16 POP_TOP 6 17 LOAD_GLOBAL 1 (n) 20 LOAD_CONST 1 (1) 23 INPLACE_ADD 24 STORE_GLOBAL 1 (n) 7 >> 27 LOAD_GLOBAL 1 (n) 30 RETURN_VALUE Would you expect the overloaded global keyword to apply differently, or just have slightly different scoping semantics? Would a nonlocal keyword apply differently? Would we allow multiple function-global parameter modifiers? I personally expect that while there is a theoretical clash between variable names in nested scopes, that's already a poor coding decision. The module level globals should not unintentionally collide with function-local non-local access. Thus reusing the global keyword is not a practical limitation. -- Michael Urman http://www.tortall.net/mu/blog From jcarlson at uci.edu Mon Nov 6 17:05:57 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 06 Nov 2006 08:05:57 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <454EE6C6.4000701@acm.org> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> Message-ID: <20061106080304.8229.JCARLSON@uci.edu> Talin wrote: > I'd like to replace this with: > > .component( slice_object ) > > where the semantics of 'component' are identical to __getitem__ on an > array or tuple. So for example: > > Path( "a", "b" ).component( 0 ) => "a" > Path( "a", "b" ).component( 1 ) => "b" > Path( "a", "b" ).component( -1 ) => "b" > Path( "a", "b" ).component( 0:1 ) => Path( "a", "b" ) > Path( "a", "b" ).component( 1: ) => Path( "b" ) Use a property or list or something. Something with __getitem__. In 2.5 it is a syntax error to use slices like that, and it would seem silly to change 3.x syntax to be able to pass slices directly like that. - Josiah From jcarlson at uci.edu Mon Nov 6 17:16:37 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 06 Nov 2006 08:16:37 -0800 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: Message-ID: <20061106080829.822C.JCARLSON@uci.edu> "Michael Urman" wrote: > I personally expect that while there is a theoretical clash between > variable names in nested scopes, that's already a poor coding > decision. The module level globals should not unintentionally collide > with function-local non-local access. Thus reusing the global keyword > is not a practical limitation. Just saying that they shouldn't be colliding doesn't mean that they won't. I've seen horrendous code from some otherwise smart people where locals shadow globals, even in closures. I don't have a link for you because I've tried to block out the experience. However, in my opinion, relying on good coding practices (never shadow variables) to guarantee correctness of the use of the global keyword in Python seems to me to be silly. I hope we can all at least agree that *requiring* good coding practices to make a feature work may be a bit more anal retentive than we want Python to be. While I personally don't want the feature in the first place, if I'm going to be seeing it in code, I would very much rather that a second keyword were introduced. If only because then I wouldn't tear out my hair trying to figure out whether a variable was really in the module globals, or in just another containing scope. - Josiah From talin at acm.org Mon Nov 6 17:55:46 2006 From: talin at acm.org (Talin) Date: Mon, 06 Nov 2006 08:55:46 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <20061106080304.8229.JCARLSON@uci.edu> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <20061106080304.8229.JCARLSON@uci.edu> Message-ID: <454F6912.1030503@acm.org> Josiah Carlson wrote: > Talin wrote: >> I'd like to replace this with: >> >> .component( slice_object ) >> >> where the semantics of 'component' are identical to __getitem__ on an >> array or tuple. So for example: >> >> Path( "a", "b" ).component( 0 ) => "a" >> Path( "a", "b" ).component( 1 ) => "b" >> Path( "a", "b" ).component( -1 ) => "b" >> Path( "a", "b" ).component( 0:1 ) => Path( "a", "b" ) >> Path( "a", "b" ).component( 1: ) => Path( "b" ) > > Use a property or list or something. Something with __getitem__. In > 2.5 it is a syntax error to use slices like that, and it would seem > silly to change 3.x syntax to be able to pass slices directly like that. That's unfortunate. I specifically want to avoid saying something like path[ 1:2 ], because its ambiguous and confusing, especially if "Path" is derived from unicode. I suppose path.components[ 1:2 ] would be all right. > - Josiah From ntoronto at cs.byu.edu Mon Nov 6 19:16:15 2006 From: ntoronto at cs.byu.edu (Neil Toronto) Date: Mon, 06 Nov 2006 11:16:15 -0700 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: <20061106080829.822C.JCARLSON@uci.edu> References: <20061106080829.822C.JCARLSON@uci.edu> Message-ID: <454F7BEF.4080903@cs.byu.edu> Josiah Carlson wrote: >"Michael Urman" wrote: > > >>I personally expect that while there is a theoretical clash between >>variable names in nested scopes, that's already a poor coding >>decision. The module level globals should not unintentionally collide >>with function-local non-local access. Thus reusing the global keyword >>is not a practical limitation. >> >> > >Just saying that they shouldn't be colliding doesn't mean that they >won't. I've seen horrendous code from some otherwise smart people where >locals shadow globals, even in closures. I don't have a link for you >because I've tried to block out the experience. However, in my opinion, >relying on good coding practices (never shadow variables) to guarantee >correctness of the use of the global keyword in Python seems to me to be >silly. > >I hope we can all at least agree that *requiring* good coding practices >to make a feature work may be a bit more anal retentive than we want >Python to be. While I personally don't want the feature in the first >place, if I'm going to be seeing it in code, I would very much rather >that a second keyword were introduced. If only because then I wouldn't >tear out my hair trying to figure out whether a variable was really in >the module globals, or in just another containing scope. > > And that's the most important implication of this decision: the health of Josiah's hair. Um, yeah - I agree, actually. This is my biggest beef with reusing "global" as well: code using it doesn't say what most people would think it says. (My second beef is that there would be no chance of seeing it in 2.x.) There's *always* going to be some mismatch between what the programmer believes the code says and what it actually does say. That doesn't mean we need to exacerbate the problem. The weirdest thing I've seen in this discussion is arguments along the lines of, "Well, it's ambiguous in the first place, so why not make it worse?" I honestly can't fathom that kind of reasoning, especially given that Python is supposed to be a language that's partly designed to be a good user interface. (Michael, I know you were talking about whether there was any technical reason it couldn't be done. My comment is aimed laterally along the discussion tree.) It'd be like saying, "You know, a checkbox is technically a toggle button, so why not call this color wheel a button, too? I mean, it has a discrete state, and you click on it. It's not that different." Neil From brett at python.org Mon Nov 6 20:33:00 2006 From: brett at python.org (Brett Cannon) Date: Mon, 6 Nov 2006 11:33:00 -0800 Subject: [Python-3000] Backward compatibility for "nonlocal" In-Reply-To: References: Message-ID: On 11/5/06, Ka-Ping Yee wrote: > > As an aside to the discussion about "nonlocal", here are a couple of > thoughts on backward compatibility. > > For some of the proposed keywords, here's the number of occurrences > of the keyword in the current standard library (not including comments > and docstrings): > > nonlocal 0 0 > use 2 2 > using 3 3 > reuse 0 4 > free 8 8 > outer 5 147 > global 126 214 > > (I used 'tokenize' to do this; let me know if you'd like me to count > other possibilities.) > > This may or may not be a good measure of the likely impact of adding > a new keyword. At least it's something to think about. > > It also occurred to me that this new keyword doesn't necessarily > have to break anything, if we are willing to tolerate a little > hackery in the Python lexer and parser. The situation is a bit > reminiscent of 'as' -- we were able to add 'as' to the 'import' > statement as though 'as' were a keyword, without actually making > 'as' a reserved word in all code. > > In this case, the usage of "nonlocal" (or whatever we choose) as > a keyword never overlaps with its usage as a variable name. The > parser could easily tell the difference by checking whether: > > - The keyword is at the beginning of a statement. > - The keyword is immediately followed by an identifier. > > If both of these things are true, then the keyword is a nonlocal > declaration. Otherwise, it can be treated as a variable name. > > I'm not saying this is necessarily a good idea; i just wanted to > note that it was an available option. We might think about a > schedule for phasing in the feature: > > Phase 1. If "nonlocal" is seen in code, issue a warning > that it will become a keyword in a future version. > > Phase 2. Have the parser distinguish when "nonlocal" is > intended as a keyword and when as a variable name. > > Phase 3. Make "nonlocal" an official reserved word. > > All of these phases are optional -- we could skip straight to 3, > or do 1 and then 3, or stop at 2, etc. Any new keyword would require a __future__ statement which automatically implies a warning in the version that introduces the __future__ statement. Then the next version makes it a keyword and the __future__ statement redundant. So the transition to a new keyword is already established. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061106/2730b3b4/attachment.html From ncoghlan at gmail.com Mon Nov 6 23:31:11 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 07 Nov 2006 08:31:11 +1000 Subject: [Python-3000] Draft PEP for outer scopes In-Reply-To: References: <1162590716.4428.23.camel@fsol> <454BE8DB.20203@canterbury.ac.nz> <454C0001.1080301@cs.byu.edu> <454C0AD4.5090602@gmail.com> Message-ID: <454FB7AF.3030909@gmail.com> Michael Urman wrote: >> def weird(local=True): >> if local: >> n = 1 >> else: >> global n # [ or nonlocal n, in Nick's ] >> n += 1 >> return n > > I'm not sure what you're both trying to explain here. First off, the > above code yields a SyntaxWarning. Secondly all its accesses to n are > global accesses. The global keyword is currently a modifier for the > whole function. Thus the local parameter is misleading; it should be > called reset. We weren't really talking about global/nonlocal in general - I was merely pointing out the specific problem that permitting the shortcut form (declaration & assignment on the same line) increases the temptation to put the scope declaration somewhere other than at the start of the function. As you point out, the compiler won't care (beyond possibly issuing a warning like the one you mention), but it can be misleading for human readers and writers. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From sluggoster at gmail.com Mon Nov 6 23:37:01 2006 From: sluggoster at gmail.com (Mike Orr) Date: Mon, 6 Nov 2006 14:37:01 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <454EE6C6.4000701@acm.org> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> Message-ID: <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> My latest idea is something like this: #### BEGIN class Path(unicode): """Pathname-manipulation methods.""" pathlib = os.path # Subclass can specify (posix|nt|mac)path. safe_args_only = False # Glyph can set this to True in a subclass. class FSPath(object): """Filesystem-access methods. The constructor takes a path arg and sets self.path to it. It also accepts any other combination of positional args, which are passed to the path constructor to create the path. """ path_class = Path # Subclass can specify an alternate path class. def __init__(klass, *args): if len(args) == 1 and isinstance(args[0], klass.path_class): self.path = args[0] else: self.path = self.path_class(*args) @classmethod def cwd(klass): """Convenience method for this common operation.""" path = klass.path_klass.cwd() rerutn klass(path) #### END This should be versatile enough to handle several of the alternatives that have been proposed. Those who want Path and FSPath separate have it. Those who want them together can use FSPath and FSPath.path. FSPath can itself be broken into two levels: a "medium" level that minimally expresses the os.*, os.path.*, and shutil.* filesystem-access functions, and an "enhanced" level that has all my favorite methods. Those who think one or another level is evil can use just the lower levels. This would satisfy those (like me) who want something we can use now in our Python 2.4/2.5 programs, and there will be working code useful for lobbying that one or more levels should be adopted into the stdlib. If Path contains minimal enhancements, it would have the best chance. Subclassing unicode would be the simplest implementation. The PEP 355 implementation does a unicode-or-str dance in case os.path.supports_unicode_filenames is false. Is it really necessary to support this nowadays? I'd like to promote all str's to unicode in the constructor so that any possible UnicodeDecodeErrors are localized in one place. On 11/5/06, Talin wrote: > Mike Orr wrote: > > Path( Path("directory"), "subdirectory", "file") # Replaces > > .joinpath(). > > For the constructor, I would write it as: > > Path( *components ) Yes. I was just trying to show what the argument objects where. > Strings can also be wrapped with an object that indicates that the Path > is in a platform- or application-specific format: > > # Explicitly indicate that the path string is in Windows NTFS format. > Path( Path.format.NTFS( "C:\\Program Files" ) ) This (and all your other .format examples) sounds a bit complicated. The further we stray from the syntax/semantics of the existing stdlib modules, the harder it will be to achieve consensus, and thus the less chance we'll get anything into the stdlib. So I'm for a scalable solution, where the lower levels are less controversial and well tested, and we can experiment in the higher levels. Can you make your .format proposal into an optional high-level component and we'll see how well it works in practice? > One question to be asked is whether the path should be simplified or > not. There are cases where you *don't* want the path to be simplified, > and other cases where you do. Perhaps a keyword argument? > > Path( "C:\\Program Files", "../../Gimp", normalize = True ) Maybe. I'm inclined to let an .only_safe_args attribute or a SafePath subclass enforce normalizing, and let the main class do whatever os.path.join() does. > > Path("ab") + "c" => Path("abc") > > Wouldn't that be: > > Path( "ab" ) + "c" => Path( "ab", "c" ) If we want string compatibility we can't redefine the '+' operator. If we ditch string compatibility we can't pass Paths to functions expecting a string. We can't have it both ways. This also applies to character slicing vs component slicing. > > .abspath() > > I've always thought this was a strange function. To be honest, I'd > rather explicitly pass in the cwd(). I use it; it's convenient. The method name could be improved. > > .normcase() > > .normpath() ... and other methods. Your proposals are all in the context of "how much do we want to improve os.path's syntax/semantics vs keeping them as-is?" I would certainly like improvement, but improvement works against consensus. Some people don't want anything to change beyond a minimal class wrapper. Others want improvements but have differing views about what the best "improvements" are. So how far do you want to go, and how does this impact your original question, "Is consensus possible?" Then, if consensus is not possible, what do we do? Each go into our corners and make our favorite incompatible module? Or can we come up with a master plan containing alternatives, to bring at least some unity to the differing modules without cramping anybody's style. In this vein, a common utility module with back-end functions would be good. Then we can solve the difficult problems *once* and have a test suite that proves it, and people would have confidence using any OO classes that are built over them. We can start by gathering the existing os.*, os.path.*, and shutil.* functions, and then add whatever other functions our various OO classes might need. However, due to the problem of supporting (posix|nt|mac)path, we may need to express this as a class of classmethods rather than a set of functions, so they can be defined relative to a platform library. > > .realpath() > > Rename to resolve() or resolvelinks(). Good idea. > > .expanduser() > > .expandvars() > > .expand() > > Replace with expand( user=True, vars=True ) Perhaps. There was one guy in the discussion about Noam's path module who didn't like .expand() at all; he thought it did too many things implicitly and was thus too magical. > > .parent > > If parent was a function, you could pass in the number of levels go to > up, i.e. parent( 2 ) to get the grandparent. I'd like .ancestor(N) for that. Parent as a property is nice when it's only one or two levels. > > > .name # Full filename without path > > .namebase # Filename without extension > > I find the term 'name' ambiguous. How about: > > .filepart > .basepart > > or: > > .filename > .basename .name/.namebase isn't great, but nothing else that's been proposed is better. > > .drive > > Do we need to somehow unify the concept of 'drive' and 'unc' part? Maybe > '.device' could return the part before the first directory name. This gets into the "root object" in Noam's proposal. I'd say just read that and the discussion, and see if it approaches what you want. I find this another complicated and potential bog-down point, like .format. http://wiki.python.org/moin/AlternativePathClass http://wiki.python.org/moin/AlternativePathDiscussion > > .splitpath() > > I'd like to replace this with: > > .component( slice_object ) > > where the semantics of 'component' are identical to __getitem__ on an > array or tuple. So for example: > > Path( "a", "b" ).component( 0 ) => "a" > Path( "a", "b" ).component( 1 ) => "b" > Path( "a", "b" ).component( -1 ) => "b" > Path( "a", "b" ).component( 0:1 ) => Path( "a", "b" ) > Path( "a", "b" ).component( 1: ) => Path( "b" ) > > This is essentially the same as the "slice notation" proposal given > earlier, except that explicitly tell the user that we are dealing with > path components, not characters. Path("a/b").components[0:1] => Path("a/b") Is there a problem with .component returning a Path instead of a list of components? In some ways I still like Noam's Path-as-components idea. It eliminates all slicing methods, and '+' does join. The problem is you'd have to explicitly unicode() it when passing it to functions that expect a string. I guess the advantage of Path being unicode still outweigh the disadvantages. Here's one possibility for splitting the absolute/relative part of a path: Path("/a/b").absolute_prefix => "/" relative_start_point = len(Path("/a/b").absolute_prefix) It would have to be defined for all platforms. Or we can have a splitroot method: Path("/a/b").splitroot() => [Path("/"), Path("a/b")] Not sure which way would be most useful overall. > > .stripext() > > How about: > > path.ext = '' The discussion in Noam's proposal has .add_exts(".tar", ".gz") and .del_exts(N). Remember that any component can have extension(s), not just the last. Also, it's up to the user which apparent extensions should be considered extensions. How many extensions does "python-2.4.5-i386.2006-12-12.orig.tar.gz" have? > > .splitall() > > Something sadly lacking in os.path. I thought this was what .splitpath() would do. > > .relpathto() > > Not sure what this does, since there's no argument defined. >From Orendorff's commentary. "The method p1.relpathto(p2) returns a relative path to p2, starting from p1." http://www.jorendorff.com/articles/python/path/ I've always found it confusing to remember which one is 'from' and which one is 'to'? -- Mike Orr From solipsis at pitrou.net Tue Nov 7 00:08:08 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 07 Nov 2006 00:08:08 +0100 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> Message-ID: <1162854488.4384.29.camel@fsol> Le lundi 06 novembre 2006 ? 14:37 -0800, Mike Orr a ?crit : > def __init__(klass, *args): > if len(args) == 1 and isinstance(args[0], klass.path_class): > self.path = args[0] > else: > self.path = self.path_class(*args) s/klass/self/, I suppose ? > Subclassing unicode would be the simplest implementation. The PEP 355 > implementation does a unicode-or-str dance in case > os.path.supports_unicode_filenames is false. Is it really necessary > to support this nowadays? I'm not sure what you mean, but on Mandriva Linux: $ python Python 2.4.3 (#2, Sep 18 2006, 21:07:35) [GCC 4.1.1 20060724 (prerelease) (4.1.1-3mdk)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.path.supports_unicode_filenames False However: >>> os.path.exists(u"/etc/passwd") True >>> os.path.exists("/etc/passwd") True >>> os.path.exists(u"?l?phant") True >>> os.path.exists("?l?phant") True (after having run "touch ?l?phant") > Path("a/b").components[0:1] => Path("a/b") > > Is there a problem with .component returning a Path instead of a list > of components? Having a "list slice" returning a non-sequence result is a bit surprising; the plural "components" is also misleading. Why not simply a method, and why not name it "subpath" ? Path("a/b/c").subpath(0, 2) => Path("a/b") Path("a/b/c").subpath(0, -1) => Path("a/b") By the way, is the absolute root a separate "component"? Path("/a/b/c").subpath(0, 1) => Path("/") Path("/a/b/c").subpath(1, 2) => Path("a") Path("/a/b/c").subpath(1) => Path("a/b/c") From rrr at ronadam.com Tue Nov 7 02:41:06 2006 From: rrr at ronadam.com (Ron Adam) Date: Mon, 06 Nov 2006 19:41:06 -0600 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> Message-ID: > [Mike Orr wrote:] > In this vein, a common utility module with back-end functions would be > good. Then we can solve the difficult problems *once* and have a test > suite that proves it, and people would have confidence using any OO > classes that are built over them. We can start by gathering the > existing os.*, os.path.*, and shutil.* functions, and then add > whatever other functions our various OO classes might need. This really sounds like a situation where writing the test cases first would be worth while. So maybe it would be good to start by gathering the relevant examples and agreeing on the desired behavior in various situations as a doctest suite. Cheers, Ron From talin at acm.org Tue Nov 7 10:15:48 2006 From: talin at acm.org (Talin) Date: Tue, 07 Nov 2006 01:15:48 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> Message-ID: <45504EC4.50609@acm.org> Mike Orr wrote: > My latest idea is something like this: > > #### BEGIN > class Path(unicode): > """Pathname-manipulation methods.""" > pathlib = os.path # Subclass can specify (posix|nt|mac)path. > safe_args_only = False # Glyph can set this to True in a subclass. > I'm a little confused here about the model of how platform-specific and application-specific formats are represented. Is it the case that the creation function converts the platform-specific path into a generic, universal path object, or does it create a platform-specific path object? In the former case -- where you create a platform-agnostic "universal" path object - the question is, how do you then 'render' that into a platform-specific string format? And in the latter case, where the 'path objects' are themselves in platform-specific form, how do you cross-convert from one format to another? One thing I really want to avoid is the situation where the only available formats are those that are built-in to the path module. > On 11/5/06, Talin wrote: >> Mike Orr wrote: >>> Path( Path("directory"), "subdirectory", "file") # Replaces >>> .joinpath(). >> For the constructor, I would write it as: >> >> Path( *components ) > > Yes. I was just trying to show what the argument objects where. > >> Strings can also be wrapped with an object that indicates that the Path >> is in a platform- or application-specific format: >> >> # Explicitly indicate that the path string is in Windows NTFS format. >> Path( Path.format.NTFS( "C:\\Program Files" ) ) > > This (and all your other .format examples) sounds a bit complicated. > The further we stray from the syntax/semantics of the existing stdlib > modules, the harder it will be to achieve consensus, and thus the less > chance we'll get anything into the stdlib. So I'm for a scalable > solution, where the lower levels are less controversial and well > tested, and we can experiment in the higher levels. Can you make your > .format proposal into an optional high-level component and we'll see > how well it works in practice? First, ignore the syntax of my examples - its really the semantics that I want to get across. I don't think you need to follow too closely the syntax of os.path - rather, we should concentrate on the semantics, and even more importantly, the scope of the existing module. In other words don't try to do too much more than os.path did. >> One question to be asked is whether the path should be simplified or >> not. There are cases where you *don't* want the path to be simplified, >> and other cases where you do. Perhaps a keyword argument? >> >> Path( "C:\\Program Files", "../../Gimp", normalize = True ) > > Maybe. I'm inclined to let an .only_safe_args attribute or a SafePath > subclass enforce normalizing, and let the main class do whatever > os.path.join() does. For my own code, I want to simplify 100% of the time. Having to insert the extra call to normpath() everywhere clutters up the code fairly egregiously. The whole issue of path simplification is really an "end user debugging" issue. Since simplified and non-simplified paths are equivalent, the only time it really matters whether or not a path is simplified is when the end-user sees a path; And in my experience, this generally only happens in error messages such as "File xxx not found" or in batch programs that take a list of files as input. Moreover, GUI programs that use a file open dialog to choose files generally don't show the full path when an error occurs. So really we're limited to the case where you have some sort of app that is either script-driven or has a user-edited config file (like Make) in which there are fragments of paths being joined together. As the designer of such an app, the question you want to ask yourself is, will it be easier for the end user to diagnose the error if we show them the simplified result of the concatenation, or the unsimplified, concatenated strings. In my case, I almost always prefer to be shown the simplified path so that I don't have to mentally simplify all of the ../../.. bits before looking at that particular location in the filesystem to see what the problem is. >>> Path("ab") + "c" => Path("abc") >> Wouldn't that be: >> >> Path( "ab" ) + "c" => Path( "ab", "c" ) > > If we want string compatibility we can't redefine the '+' operator. > If we ditch string compatibility we can't pass Paths to functions > expecting a string. We can't have it both ways. This also applies to > character slicing vs component slicing. If the '+' operator can't be used, then we need to specify how paths are joined. Or is it your intent that the only way to concatenate paths be via the constructor? I'm in favor of the verb 'combine' being used to indicate both a joining and a simplification of a path. >>> .abspath() >> I've always thought this was a strange function. To be honest, I'd >> rather explicitly pass in the cwd(). > > I use it; it's convenient. The method name could be improved. The thing that bugs me about it is that it relies on a hidden variable, in this case cwd(). Although that's partly my personal experience speaking, since a lot of the work I do is writing library code on game consoles such as Xbox360 and PS3: These platforms have a filesystem, but no concept of a "current working directory", so if the app wants to have something like a cwd, it has to maintain the variable itself. >>> .normcase() >>> .normpath() > > ... and other methods. Your proposals are all in the context of "how > much do we want to improve os.path's syntax/semantics vs keeping them > as-is?" I would certainly like improvement, but improvement works > against consensus. Some people don't want anything to change beyond a > minimal class wrapper. Others want improvements but have differing > views about what the best "improvements" are. So how far do you want > to go, and how does this impact your original question, "Is consensus > possible?" > > Then, if consensus is not possible, what do we do? Each go into our > corners and make our favorite incompatible module? Or can we come up > with a master plan containing alternatives, to bring at least some > unity to the differing modules without cramping anybody's style. What I'm trying to do is refactor os.path as we go - it seems to me that it would be a shame to simply shovel over the existing functions and simply make them object methods, especially (as several people have pointed out) since many of them can be combined into more powerful and more general methods. As far as consensus goes: Consensus is not hard to get if you can get it piecemeal. People with nitpick at little details of the proposal forever, but that's not a huge obstacle to adoption. However, getting them to swallow one huge lump of a concept is much harder. It seems that once you can get people to accept the basic premise of a path object, you are home free - the rest is niggling over details. So I don't see that this harms the chance of adoption, as long as people are allowed to tweak the details of the proposal. > In this vein, a common utility module with back-end functions would be > good. Then we can solve the difficult problems *once* and have a test > suite that proves it, and people would have confidence using any OO > classes that are built over them. We can start by gathering the > existing os.*, os.path.*, and shutil.* functions, and then add > whatever other functions our various OO classes might need. > > However, due to the problem of supporting (posix|nt|mac)path, we may > need to express this as a class of classmethods rather than a set of > functions, so they can be defined relative to a platform library. > >>> .realpath() >> Rename to resolve() or resolvelinks(). > > Good idea. > >>> .expanduser() >>> .expandvars() >>> .expand() >> Replace with expand( user=True, vars=True ) > > Perhaps. There was one guy in the discussion about Noam's path module > who didn't like .expand() at all; he thought it did too many things > implicitly and was thus too magical. Well I'd say leave it in then, and see who objects. >>> .parent >> If parent was a function, you could pass in the number of levels go to >> up, i.e. parent( 2 ) to get the grandparent. > > I'd like .ancestor(N) for that. Parent as a property is nice when > it's only one or two levels. Reasonable >>> .name # Full filename without path >>> .namebase # Filename without extension >> I find the term 'name' ambiguous. How about: >> >> .filepart >> .basepart >> >> or: >> >> .filename >> .basename > > .name/.namebase isn't great, but nothing else that's been proposed is better. If all else fails, copy what someone else has done: http://msdn2.microsoft.com/en-us/library/system.io.path_members(VS.80).aspx >>> .drive >> Do we need to somehow unify the concept of 'drive' and 'unc' part? Maybe >> '.device' could return the part before the first directory name. > > This gets into the "root object" in Noam's proposal. I'd say just > read that and the discussion, and see if it approaches what you want. > I find this another complicated and potential bog-down point, like > .format. > > http://wiki.python.org/moin/AlternativePathClass > http://wiki.python.org/moin/AlternativePathDiscussion > >>> .splitpath() >> I'd like to replace this with: >> >> .component( slice_object ) >> >> where the semantics of 'component' are identical to __getitem__ on an >> array or tuple. So for example: >> >> Path( "a", "b" ).component( 0 ) => "a" >> Path( "a", "b" ).component( 1 ) => "b" >> Path( "a", "b" ).component( -1 ) => "b" >> Path( "a", "b" ).component( 0:1 ) => Path( "a", "b" ) >> Path( "a", "b" ).component( 1: ) => Path( "b" ) >> >> This is essentially the same as the "slice notation" proposal given >> earlier, except that explicitly tell the user that we are dealing with >> path components, not characters. > > Path("a/b").components[0:1] => Path("a/b") > > Is there a problem with .component returning a Path instead of a list > of components? I don't see why we can't have both: path.component[ a:b ] # returns a list of components path.subpath[ a:b ] # returns a (possibly non-rooted) Path object Note also that this would mean that ".splitall()" is no longer needed, since it can be replaced by "path.component[:]". (More refactoring...woot!) Another thing I like about the slice syntax is that you can also control whether or not the return value is a subsequence or not: path.component[ a ] # Gets the ath component as a string path.component[ a:a+1 ] # Gets a list containing the ath component This distinction is not as easily replicated using a function-call syntax such as path.component( a, a+1 ). Although in the case of subpath, I am not sure what the distinction is. (On the naming of 'component' vs. 'components' - my general naming convention is that array names are plurals - so a table of primes is called 'primes' not 'prime'.) > In some ways I still like Noam's Path-as-components idea. It > eliminates all slicing methods, and '+' does join. The problem is > you'd have to explicitly unicode() it when passing it to functions > that expect a string. I guess the advantage of Path being unicode > still outweigh the disadvantages. > > Here's one possibility for splitting the absolute/relative part of a path: > > Path("/a/b").absolute_prefix => "/" > relative_start_point = len(Path("/a/b").absolute_prefix) > > It would have to be defined for all platforms. Or we can have a > splitroot method: > > Path("/a/b").splitroot() => [Path("/"), Path("a/b")] > > Not sure which way would be most useful overall. > > >>> .stripext() >> How about: >> >> path.ext = '' > > The discussion in Noam's proposal has .add_exts(".tar", ".gz") and > .del_exts(N). Remember that any component can have extension(s), not > just the last. Also, it's up to the user which apparent extensions > should be considered extensions. How many extensions does > "python-2.4.5-i386.2006-12-12.orig.tar.gz" have? A directory name with a '.' in it isn't really an extension, at least not as interpreted by most filesystems. For example, if you create a folder in MSWindows, and name it "foo.bar" and then look at it in windows explorer, it will still say it's a folder; It won't try and display it as a "folder of type 'bar'". Similarly, if you are using LSCOLORS under posix, and you have a directory with a dot in the name, it still shows up in the same color as other dirs. In any case, I really don't think we need to support any special accessors for accessing the part after the '.' in any component but the last. As far as multiple extensions go - the easiest thing would be to simply treat the very last part - '.gz' in your example - as the extension, and let the user worry about the rest. I only know of one program - GNU tar - that attempts to interpret multiple file extensions in this way. (And you'll notice that most of the dots in the example are version number separators, not file extension separators.) I'll go further out on a limb here, and say that interpreting the file extension really isn't the path library's job, and the only reason why this function is here at all is to prevent novice programmers from erroneously calling str.rfind( '.' ) on the path string, which will of course yield the wrong answer if the filename has no dot in it but a directory name does. >>> .splitall() >> Something sadly lacking in os.path. > > I thought this was what .splitpath() would do. .splitpath only splits into two pieces, a dirname and a filename. >>> .relpathto() >> Not sure what this does, since there's no argument defined. > >>From Orendorff's commentary. > "The method p1.relpathto(p2) returns a relative path to p2, starting from p1." > http://www.jorendorff.com/articles/python/path/ > > I've always found it confusing to remember which one is 'from' and > which one is 'to'? From sluggoster at gmail.com Tue Nov 7 23:06:42 2006 From: sluggoster at gmail.com (Mike Orr) Date: Tue, 7 Nov 2006 14:06:42 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <45504EC4.50609@acm.org> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> <45504EC4.50609@acm.org> Message-ID: <6e9196d20611071406j4d98a3ete3b464966d532e1c@mail.gmail.com> On 11/7/06, Talin wrote: > Mike Orr wrote: > > My latest idea is something like this: > > > > #### BEGIN > > class Path(unicode): > > """Pathname-manipulation methods.""" > > pathlib = os.path # Subclass can specify (posix|nt|mac)path. > > safe_args_only = False # Glyph can set this to True in a subclass. > > > > I'm a little confused here about the model of how platform-specific and > application-specific formats are represented. Is it the case that the > creation function converts the platform-specific path into a generic, > universal path object, or does it create a platform-specific path object? > > In the former case -- where you create a platform-agnostic "universal" > path object - the question is, how do you then 'render' that into a > platform-specific string format? > > And in the latter case, where the 'path objects' are themselves in > platform-specific form, how do you cross-convert from one format to another? > > One thing I really want to avoid is the situation where the only > available formats are those that are built-in to the path module. My Path object is platform-specific. By default it uses the "native" path module (os.path). If you need non-native paths, subclass Path and set .pathlib to the other path module (e.g., ntpath, posixpath). To convert paths we can make the constructor smarter; e.g., class NTPath(Path): pathlib = ntpath foo = Path("foo") my_nt_path = NTPath(foo) The constructor might recognize that Path.pathlib != NTPath.pathlib and call foo.to_universal() to get a universal format, which it would then convert. Thinking further, why not use a component tuple as a universal format, with the first element being the root (or '' for a relative path). This is easy to convert to/from any platform string, won't be mistaken for a platform string, and won't give one platform an advantage over another. We can make a pseudo Path class for the universal format, the only restriction being that you can't pass it directly to a filesystem-access function. Actually, you can't pass *any* non-native path to a filesystem-access function anyway. Remember that 99% of Python programmers are concerned only with native paths. I have never used a non-native path or multiple-platform paths in an application. So we need to make the native case easy and clear. For that reason I'd rather keep the non-native cases and conversion code in separate classes. > I don't think you need to follow too closely the syntax of os.path - > rather, we should concentrate on the semantics, and even more > importantly, the scope of the existing module. In other words don't try > to do too much more than os.path did. I think a layered approach is important. Use the code in the existing modules because it's well tested. Add a native-path object on top of that. Put non-native paths and conversions on the side. Then put a filesystem-access class (or functions) on top of the native-path object. Then high-level functions/methods on top of that. Then when we lobby for stdlib inclusion it'll be "one level and everything below it". People can see and test that level, and ignore the (possibly more controversial) levels above it. Of course, the layers can be collapsed at a later date if desired. And the os.path/os/shutil functions can be inlined at some point if it's decided to deprecate them. But that's too much for now. I would really like to improve on os.path and os/shutil. That's a separate issue from the class structure I've proposed. My only reason for making a thin wrapper would be for greater acceptability (lowest common denominator). But I'd rather work on a thick wrapper whose methods corresponds to what the application programer conceptually wants to do, rather than having to specify every piddly step (rmtree if a directory exists, remove if a file exists, nothing if nothing exists). > >> One question to be asked is whether the path should be simplified or > >> not. There are cases where you *don't* want the path to be simplified, > >> and other cases where you do. Perhaps a keyword argument? > >> > >> Path( "C:\\Program Files", "../../Gimp", normalize = True ) > > > > Maybe. I'm inclined to let an .only_safe_args attribute or a SafePath > > subclass enforce normalizing, and let the main class do whatever > > os.path.join() does. > > For my own code, I want to simplify 100% of the time. Having to insert > the extra call to normpath() everywhere clutters up the code fairly > egregiously. That is what I'd prefer. Let the constructor call normpath(). Maybe we can just say we don't support non-normalized paths, and if you need that you can write a NonNormalizedPath subclass or use os.path directly. My pet peeve is paths beginning with "./" . I want that out of error messages and diagnostic messages! > >>> Path("ab") + "c" => Path("abc") > >> Wouldn't that be: > >> > >> Path( "ab" ) + "c" => Path( "ab", "c" ) > > > > If we want string compatibility we can't redefine the '+' operator. > > If we ditch string compatibility we can't pass Paths to functions > > expecting a string. We can't have it both ways. This also applies to > > character slicing vs component slicing. > > If the '+' operator can't be used, then we need to specify how paths are > joined. Or is it your intent that the only way to concatenate paths be > via the constructor? > > I'm in favor of the verb 'combine' being used to indicate both a joining > and a simplification of a path. I like the syntax of a join method. With a multi-arg constructor it's not necessary though. PEP 355 hints at problems with the .joinpath() method though it didn't say what they were. .combine() is an OK name, and there's a precedent in the datetime module. But people are used to calling it "joining paths" so I'm not sure we should rename it so easily. We can't call it .join due to string compatibility. .joinpath is ugly if we delete "path" from the other method names. .pjoin comes to mind but people might find it too cryptic and too similar to .join. By just using the constructor we avoid the debate over the method name. > >>> .abspath() > >> I've always thought this was a strange function. To be honest, I'd > >> rather explicitly pass in the cwd(). > > > > I use it; it's convenient. The method name could be improved. > > The thing that bugs me about it is that it relies on a hidden variable, > in this case cwd(). Although that's partly my personal experience > speaking, since a lot of the work I do is writing library code on game > consoles such as Xbox360 and PS3: These platforms have a filesystem, but > no concept of a "current working directory", so if the app wants to have > something like a cwd, it has to maintain the variable itself. Again, 99.9% of Python users have a functioning cwd, and .abspath() is one of those things people expect in any programming language. Nobody is forcing you to call it on the Xbox. However, I'm not completely opposed to dropping .abspath(). > It seems that once you can get people to accept the basic premise of a > path object, you are home free - the rest is niggling over details. So I > don't see that this harms the chance of adoption, as long as people are > allowed to tweak the details of the proposal. The syntax details do matter. They won't necessarily make or break a proposal but they do make people's support for it stronger or weaker. But flexibility is a good thing. If we have a well-chosen structure but make it easy for people to subclass it and rename methods if they can't stand our choices, they might accept it anyway. This also means not hardwiring class relationships. For instance, Path should create paths via self.__class__ rather than calling Path literally, and FSPath should call self.path_class rather than calling Path directly. This makes it easy for somebody to write a "better" Path class and have FSPath use it. > > In this vein, a common utility module with back-end functions would be > > good. Then we can solve the difficult problems *once* and have a test > > suite that proves it, and people would have confidence using any OO > > classes that are built over them. We can start by gathering the > > existing os.*, os.path.*, and shutil.* functions, and then add > > whatever other functions our various OO classes might need. > > > > However, due to the problem of supporting (posix|nt|mac)path, we may > > need to express this as a class of classmethods rather than a set of > > functions, so they can be defined relative to a platform library. Actually, this may not be a problem after all. The filesystem-access functions are universal; e.g., there's only one os.remove(). And they only make sense with native paths. So we can call os.path.exists() from our purge() or super_duper_copy() function and be happy. > path.component[ a:b ] # returns a list of components > path.subpath[ a:b ] # returns a (possibly non-rooted) Path object Great. I was wondering how to handle the two cases. path.components[a:b] => list of components path.components[a] => one component path.subpath(a, b) => a path (relative if a != 0) Now if Path("/usr/bin/python").components => ["/", "usr", "bin", "python"] then we have the universal format I described above. That's two birds down with one stone. I think .subpath should be a method though. I can't think of another case where slicing returns a non-sequence. If a defaults to 0 and b to -1, it'll be easy for people to get the beginning or end of the path using normal Python subscripts. Or we can even allow None for the endpoinds to make it super easy. > path.component[ a ] # Gets the ath component as a string > path.component[ a:a+1 ] # Gets a list containing the ath component > > This distinction is not as easily replicated using a function-call > syntax such as path.component( a, a+1 ). Although in the case of > subpath, I am not sure what the distinction is. There is no distinction. It's not a sequence in either case. (Or at least it's not *that* kind of sequence. It's a subclass of unicode so it's a sequence of characters.) > (On the naming of 'component' vs. 'components' - my general naming > convention is that array names are plurals - so a table of primes is > called 'primes' not 'prime'.) So we agree that .components is better, if I understand you. > > The discussion in Noam's proposal has .add_exts(".tar", ".gz") and > > .del_exts(N). Remember that any component can have extension(s), not > > just the last. Also, it's up to the user which apparent extensions > > should be considered extensions. How many extensions does > > "python-2.4.5-i386.2006-12-12.orig.tar.gz" have? > > A directory name with a '.' in it isn't really an extension, at least > not as interpreted by most filesystems. For example, if you create a > folder in MSWindows, and name it "foo.bar" and then look at it in > windows explorer, it will still say it's a folder; It won't try and > display it as a "folder of type 'bar'". Similarly, if you are using > LSCOLORS under posix, and you have a directory with a dot in the name, > it still shows up in the same color as other dirs. > > In any case, I really don't think we need to support any special > accessors for accessing the part after the '.' in any component but the > last. > > As far as multiple extensions go - the easiest thing would be to simply > treat the very last part - '.gz' in your example - as the extension, and > let the user worry about the rest. I only know of one program - GNU tar > - that attempts to interpret multiple file extensions in this way. (And > you'll notice that most of the dots in the example are version number > separators, not file extension separators.) > > I'll go further out on a limb here, and say that interpreting the file > extension really isn't the path library's job, and the only reason why > this function is here at all is to prevent novice programmers from > erroneously calling str.rfind( '.' ) on the path string, which will of > course yield the wrong answer if the filename has no dot in it but a > directory name does. People need to add/delete/replace extensions, and they don't want to use character slicing / .rfind / .endswith / len() / + to do it. They expect the library to at least handle extension splitting as well as os.path does. Adding a few convenience methods would be unobtrusive and express people really want to do: p2 = p.add_ext(".tar") p2 = p.del_ext() p2 = Path("foo.gzip").replace_ext(".bz2") But what harm is there in making them scalable to multiple extensions? .add_exts(*exts) .del_exts(N) .replace_exts(N, *exts) You're right that directory extensions are rare. Maybe we should just support extensions on the last path component. -- Mike Orr From sluggoster at gmail.com Tue Nov 7 23:36:38 2006 From: sluggoster at gmail.com (Mike Orr) Date: Tue, 7 Nov 2006 14:36:38 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611071406j4d98a3ete3b464966d532e1c@mail.gmail.com> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> <45504EC4.50609@acm.org> <6e9196d20611071406j4d98a3ete3b464966d532e1c@mail.gmail.com> Message-ID: <6e9196d20611071436hebe17bfl9049f2af45ae3311@mail.gmail.com> How do you convert an absolute path anyway? PosixPath(NTPath("C:\winnt\system")) => ?? NTPath(PosixPath("/mnt/cdrom") => ?? You can convert them to "/winnt/system" and "\mnt\cdrom", but what's the point? They won't exist anyway. -- Mike Orr From greg.ewing at canterbury.ac.nz Wed Nov 8 02:38:23 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 08 Nov 2006 14:38:23 +1300 Subject: [Python-3000] Mini Path object In-Reply-To: <45504EC4.50609@acm.org> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> <45504EC4.50609@acm.org> Message-ID: <4551350F.7070903@canterbury.ac.nz> Talin wrote: > I'm a little confused here about the model of how platform-specific and > application-specific formats are represented. Is it the case that the > creation function converts the platform-specific path into a generic, > universal path object, or does it create a platform-specific path object? I don't think it makes sense to mix different platform's path objects at all. They're not just different ways of representing the same thing, they're actually different things. I would drop any notion of format specifiers when constructing or joining paths. If you really want to do something like that, you can split it into components and construct a new path object of a different type from those components. And then it's up to you to deal with whatever problems arise, e.g. how to get a Windows drive letter into a Unix path object. > The whole issue of path simplification is really an "end user debugging" > issue. Since simplified and non-simplified paths are equivalent Actually, they're not -- it's possible for simplified and non-simplified versions of the same path to refer to different filesystem objects. So I'd say that the path object should never try to simplify on its own initiative, except in ways that can't possibly alter the semantics of the path. >>>> .abspath() >>> >>>I've always thought this was a strange function. To be honest, I'd >>>rather explicitly pass in the cwd(). >> >>I use it; it's convenient. The method name could be improved. It does violate the constraint of the path object being restricted to path algebra operations, though, since the result depends on the cwd of the process. >>Perhaps. There was one guy in the discussion about Noam's path module >>who didn't like .expand() at all; he thought it did too many things >>implicitly and was thus too magical. And again, it strays outside the domain of path algebra operations. >>>Do we need to somehow unify the concept of 'drive' and 'unc' part? Maybe >>>'.device' could return the part before the first directory name. These sound like they belong in the realm of extensions belonging to platform-specific path subclasses. > A directory name with a '.' in it isn't really an extension, at least > not as interpreted by most filesystems. For example, if you create a > folder in MSWindows, and name it "foo.bar" and then look at it in > windows explorer, it will still say it's a folder; It won't try and > display it as a "folder of type 'bar'". Not true on MacOSX, where some kinds of directories do indeed have meaningful extensions -- eg. .app, .framework. > In any case, I really don't think we need to support any special > accessors for accessing the part after the '.' in any component but the > last. That's probably true. -- Greg From sluggoster at gmail.com Wed Nov 8 03:46:59 2006 From: sluggoster at gmail.com (Mike Orr) Date: Tue, 7 Nov 2006 18:46:59 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <4551350F.7070903@canterbury.ac.nz> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> <45504EC4.50609@acm.org> <4551350F.7070903@canterbury.ac.nz> Message-ID: <6e9196d20611071846l1f0b4dddjc689a9991cdab9eb@mail.gmail.com> On 11/7/06, Greg Ewing wrote: > >>>> .abspath() > >>> > >>>I've always thought this was a strange function. To be honest, I'd > >>>rather explicitly pass in the cwd(). > >> > >>I use it; it's convenient. The method name could be improved. > > It does violate the constraint of the path object being > restricted to path algebra operations, though, since the > result depends on the cwd of the process. What do we do with Path.cwd() then? It also violates the path-algrebra-only logic. How else will people get the current directory? We can put it on FSPath, but then it's the only FSPath method that returns a Path, so you'd be passing FSPath to Path to FSPath. > >>Perhaps. There was one guy in the discussion about Noam's path module > >>who didn't like .expand() at all; he thought it did too many things > >>implicitly and was thus too magical. > > And again, it strays outside the domain of path algebra > operations. This is also the same issue. Where do we put the .expand*() methods if not on Path? Is there an actual case where calling normpath() would change which file the path referred to? Any case that's not handled by (posix|nt|mac)path.normpath itself? -- Mike Orr From ncoghlan at gmail.com Wed Nov 8 11:07:10 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 08 Nov 2006 20:07:10 +1000 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611071846l1f0b4dddjc689a9991cdab9eb@mail.gmail.com> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> <45504EC4.50609@acm.org> <4551350F.7070903@canterbury.ac.nz> <6e9196d20611071846l1f0b4dddjc689a9991cdab9eb@mail.gmail.com> Message-ID: <4551AC4E.8090608@gmail.com> Mike Orr wrote: >> And again, it strays outside the domain of path algebra >> operations. > > This is also the same issue. Where do we put the .expand*() methods > if not on Path? For the moment, I'd say that if an operation can raise any flavour of EnvironmentError, don't make it a method of the path algebra. I realise this means normalisation and symbol expansion will be defined only on FSPath's. I'd say that's OK because at least the syntax for environment variable substitution is OS dependent anyway (note that these methods should return the appropriate FSPath objects, rather than the abstract versions). > Is there an actual case where calling normpath() would change which > file the path referred to? Any case that's not handled by > (posix|nt|mac)path.normpath itself? > To quote the docs for os.normpath itself: "It should be understood that this may change the meaning of the path if it contains symbolic links!" Specifically, using path algebra to normalise a path that backtracks after following a symlink will actually give a different answer than asking the filesystem to do it. To steal an explanation from the effbot [1]: """ if BAR is a symbolic link, FOO/BAR/../DIR isn't necessarily the same thing as FOO/DIR. a simple example: $ ln -s /etc etc $ ls -ld etc lrwxrwxrwx 1 fredrik fredrik 4 Dec 2 23:22 etc -> /etc $ etc/../usr/local/bin/python2.4 Python 2.4.1 (#1, Sep 12 2005, 19:35:02) ... >>> import os >>> os.path.normpath("etc/../usr/local/bin/python2.4") 'usr/local/bin/python2.4' >>> $ usr/local/bin/python2.4 -bash: usr/local/bin/python2.4: No such file or directory """ Cheers, Nick. [1] http://mail.python.org/pipermail/python-dev/2005-December/058453.html -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From talin at acm.org Wed Nov 8 11:21:18 2006 From: talin at acm.org (Talin) Date: Wed, 08 Nov 2006 02:21:18 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <4551350F.7070903@canterbury.ac.nz> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> <45504EC4.50609@acm.org> <4551350F.7070903@canterbury.ac.nz> Message-ID: <4551AF9E.9070409@acm.org> Greg Ewing wrote: > Talin wrote: > >> I'm a little confused here about the model of how platform-specific >> and application-specific formats are represented. Is it the case that >> the creation function converts the platform-specific path into a >> generic, universal path object, or does it create a platform-specific >> path object? > > I don't think it makes sense to mix different platform's > path objects at all. They're not just different ways of > representing the same thing, they're actually different > things. Only true if we're talking about fully qualified paths. For relative paths, every file system I know of supports the 'name (sep name)*' syntax to indicate subdirectory structure. If this were not true, 'make' could never work across filesystems. You'd have to have a different makefile for every different filesystem type. -- Talin From talin at acm.org Wed Nov 8 11:21:29 2006 From: talin at acm.org (Talin) Date: Wed, 08 Nov 2006 02:21:29 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611071406j4d98a3ete3b464966d532e1c@mail.gmail.com> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> <45504EC4.50609@acm.org> <6e9196d20611071406j4d98a3ete3b464966d532e1c@mail.gmail.com> Message-ID: <4551AFA9.9090905@acm.org> Mike Orr wrote: > Remember that 99% of Python programmers are concerned only with native > paths. I have never used a non-native path or multiple-platform paths > in an application. So we need to make the native case easy and clear. > For that reason I'd rather keep the non-native cases and conversion > code in separate classes. This is only true if one is speaking of fully-qualified paths. Most of the time, however, we deal with relative paths. Relative paths are, for the most part, universal already - if they weren't, then it would be impossible to create a makefile that works on both Linux and Windows, because the embedded file paths would be incompatible. If relative paths couldn't be used in conjunction with different path types, then there would be no way for make, Apache, Scons, svn, and a whole bunch of other apps I can think of to work cross-platform. What I want is a way to have data files that contain embedded paths encoded in a way that allows those data files to be transported across platform. Generally those embedded paths will be relative to something, rather than fully-qualfied, such as ${root}/something/or/other. What I want to avoid is a situation where I have to edit my config file to switch all the path separators from '/' to '\' when I move my application from OS X to Win32. As a Python programmer, my ideal is to be able to write as if I *didn't know* what the underlying platform was. >> I don't think you need to follow too closely the syntax of os.path - >> rather, we should concentrate on the semantics, and even more >> importantly, the scope of the existing module. In other words don't try >> to do too much more than os.path did. > > I think a layered approach is important. Use the code in the existing > modules because it's well tested. Add a native-path object on top of > that. Put non-native paths and conversions on the side. Then put a > filesystem-access class (or functions) on top of the native-path > object. Then high-level functions/methods on top of that. Then when > we lobby for stdlib inclusion it'll be "one level and everything below > it". People can see and test that level, and ignore the (possibly > more controversial) levels above it. BTW, I don't think that the "filesystem object" is going to fly in the way you describe it. Specifically, it sounds like yet another swiss army knife class. I think that there's a reasonable chance of acceptance for an object that does filesystem-like operations that *doesn't overlap* with what the Path object does. But what you are proposing is a *superset* of what Path does (because you're saying its a subclass), and I don't think that will go over well. The basic rationale is simple: "Things" and "Names for things" are distinct from each other, and shouldn't be conflated into the same object. A path is a name for a thing, but it is not the same as the thing it is naming. >> I'm in favor of the verb 'combine' being used to indicate both a joining >> and a simplification of a path. > > > I like the syntax of a join method. With a multi-arg constructor it's > not necessary though. PEP 355 hints at problems with the .joinpath() > method though it didn't say what they were. .combine() is an OK > name, and there's a precedent in the datetime module. But people are > used to calling it "joining paths" so I'm not sure we should rename it > so easily. We can't call it .join due to string compatibility. > .joinpath is ugly if we delete "path" from the other method names. > .pjoin comes to mind but people might find it too cryptic and too > similar to .join. By just using the constructor we avoid the debate > over the method name. The reason I don't like the word "join" is that it implies some sort of concatenation, like an "append" operation - i.e. the combination of "a" and "b" joined together is "a/b". However, when we combine paths, we're really doing "path algebra", which is a more sophisticated mixing of paths. Thus "a" joined with "/b" is just "/b", not "a/b". > Again, 99.9% of Python users have a functioning cwd, and .abspath() is > one of those things people expect in any programming language. Nobody > is forcing you to call it on the Xbox. However, I'm not completely > opposed to dropping .abspath(). And I'm not completely opposed to keeping it either. I'm just a minimalist :) :) > People need to add/delete/replace extensions, and they don't want to > use character slicing / .rfind / .endswith / len() / + to do it. They > expect the library to at least handle extension splitting as well as > os.path does. Adding a few convenience methods would be unobtrusive > and express people really want to do: > > p2 = p.add_ext(".tar") > p2 = p.del_ext() > p2 = Path("foo.gzip").replace_ext(".bz2") > > But what harm is there in making them scalable to multiple extensions? > > .add_exts(*exts) > .del_exts(N) > .replace_exts(N, *exts) Someone in another message pointed out that paths, being based on strings, are immutable, so this whole handling of extensions will have to be done another way. -- Talin From ncoghlan at gmail.com Wed Nov 8 11:22:43 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 08 Nov 2006 20:22:43 +1000 Subject: [Python-3000] Mini Path object In-Reply-To: <4551AC4E.8090608@gmail.com> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> <45504EC4.50609@acm.org> <4551350F.7070903@canterbury.ac.nz> <6e9196d20611071846l1f0b4dddjc689a9991cdab9eb@mail.gmail.com> <4551AC4E.8090608@gmail.com> Message-ID: <4551AFF3.8020604@gmail.com> Nick Coghlan wrote: > I realise this means normalisation and symbol expansion will be defined only > on FSPath's. Scratch the bit about normalisation not being defined on the path algebra object - as the latter part of the previous email noted, normalisation *is* path algebra based. It just has the problem that whether or not the normalised version means the same as the original on a given filesystem depends on whether or not the original involves backtracking after following a symlink. It's abspath() and cwd() that don't belong on the abstract Path object - the former would be a zero argument instance method of FSPath objects, while the latter would be a class method of FSPath. In both cases, the result would be an FSPath since it relates specifically to the current filesystem (and is OS-dependent, as an absolute path on Windows starts with a drive specifier or UNC name, while one on a *nix system will start at the root directory). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From p.f.moore at gmail.com Wed Nov 8 15:47:19 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Nov 2006 14:47:19 +0000 Subject: [Python-3000] Mini Path object In-Reply-To: <4551AFA9.9090905@acm.org> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> <45504EC4.50609@acm.org> <6e9196d20611071406j4d98a3ete3b464966d532e1c@mail.gmail.com> <4551AFA9.9090905@acm.org> Message-ID: <79990c6b0611080647s3f337c6dl875e02d693fb59a8@mail.gmail.com> On 11/8/06, Talin wrote: > What I want to avoid is a situation where I have to edit my config file > to switch all the path separators from '/' to '\' when I move my > application from OS X to Win32. This requirement conflicts with that of a user who only uses one platform, and wants (expects) to just enter paths in native format in the config file. > As a Python programmer, my ideal is to be able to write as if I *didn't > know* what the underlying platform was. As a user, *my* ideal is to use native format everywhere, and not have to worry about other platforms. I don't think there's a one-size-fits-all answer to this - it's a human issue, not a technical one. Paul. From sluggoster at gmail.com Wed Nov 8 22:21:05 2006 From: sluggoster at gmail.com (Mike Orr) Date: Wed, 8 Nov 2006 13:21:05 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <4551AFA9.9090905@acm.org> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> <45504EC4.50609@acm.org> <6e9196d20611071406j4d98a3ete3b464966d532e1c@mail.gmail.com> <4551AFA9.9090905@acm.org> Message-ID: <6e9196d20611081321u46f0f4baoc42e09c7e464d5b4@mail.gmail.com> On 11/8/06, Talin wrote: > What I want is a way to have data files that contain embedded paths > encoded in a way that allows those data files to be transported across > platform. Generally those embedded paths will be relative to something, > rather than fully-qualfied, such as ${root}/something/or/other. > > What I want to avoid is a situation where I have to edit my config file > to switch all the path separators from '/' to '\' when I move my > application from OS X to Win32. > > As a Python programmer, my ideal is to be able to write as if I *didn't > know* what the underlying platform was. I think my proposal yesterday will do it for you. p = Path( PosixPath("something/or/other") ) This would convert a Posix relative path to the native format if different, using .components as a universal intermediary. I'm inclined to raise an exception if the user tries to convert from an absolute path since there's no one obvious way to do it. Let the user make the path relative and then explicitly attach it to a new base directory. > I think that there's a reasonable chance of acceptance for an object > that does filesystem-like operations that *doesn't overlap* with what > the Path object does. But what you are proposing is a *superset* of what > Path does (because you're saying its a subclass), and I don't think that > will go over well. It's a "contains" relationship, not a subclass. FSPath contains a path in its .path attribute. Path(object): .pathlib = os.path # Class attribute. PosixPath(Path): .pathlib = posixpath FSPath(object): .path = Path("foo") # Instance attribute. FSPath will call the Path constructor whenever it needs to create a path object. That's the only relationship between the two. Three occasions have been mentioned: - When FSPath() is called with any arguments other than a single Path instance. It passes the arguments to Path() and sets self.path to the result. - FSPath.cwd() does 'return FSPath( Path(os.getcwd()) )'. - If .norm and .expand are moved to FSPath, they will return new FSPath objects initialized to new Path objects. (Previously I said you could make FSPath use an alternate *Path class. On reflection that's unnecessary, since FSPath can't do anything with a non-native path anyway.) I would personally like Path and FSPath combined into one class, so this is a compromise on my part. The argument for a "pure path algebra" class seems academic: who really needs it? (Talin's cross-platform example could be handled by a conversion class designed for that purpose.) os.path mixes FS-independent and FS-dependent functions together, and I've never heard anybody say that's a problem. So why is it suddenly a problem now? We've seen that path joining, the current directory, absolute paths, normalization, and FS-sensitive ".." all imply an interaction between Path and FSPath, so it's a messy split. I'm thinking about proposing .absolute() and .relative() to replace .aspath(), where .absolute() prefixes the cwd and .relative() chops off the root only. Although with the Path/FSPath distinction, .absolute() would have to go on FSPath and .relative() belongs on Path. Users might wonder why the two aren't together. > > People need to add/delete/replace extensions, and they don't want to > > use character slicing / .rfind / .endswith / len() / + to do it. They > > expect the library to at least handle extension splitting as well as > > os.path does. Adding a few convenience methods would be unobtrusive > > and express people really want to do: > > > > p2 = p.add_ext(".tar") > > p2 = p.del_ext() > > p2 = Path("foo.gzip").replace_ext(".bz2") > > > > But what harm is there in making them scalable to multiple extensions? > > > > .add_exts(*exts) > > .del_exts(N) > > .replace_exts(N, *exts) > > Someone in another message pointed out that paths, being based on > strings, are immutable, so this whole handling of extensions will have > to be done another way. They would return new Path's, just like str.replace() does. I do want them to be dictionary keys so they'd have to be immutable. I'm veering toward this syntax now: .add_ext(*exts) # Mainly for one but can unobtrusively add more. .del_ext(N=1) # Mainly for one but can unobtrusively delete more. .replace_ext(ext) # Replace one extension only (e.g., ".jpg" to ".gif"). -- Mike Orr From greg.ewing at canterbury.ac.nz Thu Nov 9 00:20:26 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 09 Nov 2006 12:20:26 +1300 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611071846l1f0b4dddjc689a9991cdab9eb@mail.gmail.com> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> <45504EC4.50609@acm.org> <4551350F.7070903@canterbury.ac.nz> <6e9196d20611071846l1f0b4dddjc689a9991cdab9eb@mail.gmail.com> Message-ID: <4552663A.8040002@canterbury.ac.nz> Mike Orr wrote: > What do we do with Path.cwd() then? It also violates the > path-algrebra-only logic. We can put it on FSPath, but then > it's the only FSPath method that returns a Path I don't see why it should be a problem for an FSPath method to return a Path. > Where do we put the .expand*() methods if not on Path? This case might be excusable, since it can require platform-specific knowledge of the path format. > Is there an actual case where calling normpath() would change which > file the path referred to? If the path crosses a symbolic link to a directory, then .. no longer means what it appears to mean from just looking at the path. -- Greg From greg.ewing at canterbury.ac.nz Thu Nov 9 00:43:23 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 09 Nov 2006 12:43:23 +1300 Subject: [Python-3000] Mini Path object In-Reply-To: <4551AF9E.9070409@acm.org> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> <45504EC4.50609@acm.org> <4551350F.7070903@canterbury.ac.nz> <4551AF9E.9070409@acm.org> Message-ID: <45526B9B.2010705@canterbury.ac.nz> Talin wrote: > If this were not true, 'make' could never work across filesystems. You'd > have to have a different makefile for every different filesystem type. Ever tried to use a Unix makefile on classic MacOS or VMS? It wouldn't work very well at all... Even relative pathnames can have semantic differences that don't translate across systems. For example, in VMS, the pathname of a directory looks quite different from the pathname of a file. You couldn't round-trip that to a Unix pathname and back without losing the distinction. -- Greg From greg.ewing at canterbury.ac.nz Thu Nov 9 01:02:31 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 09 Nov 2006 13:02:31 +1300 Subject: [Python-3000] Mini Path object In-Reply-To: <4551AFA9.9090905@acm.org> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> <45504EC4.50609@acm.org> <6e9196d20611071406j4d98a3ete3b464966d532e1c@mail.gmail.com> <4551AFA9.9090905@acm.org> Message-ID: <45527017.4020402@canterbury.ac.nz> Talin wrote: > I think that there's a reasonable chance of acceptance for an object > that does filesystem-like operations that *doesn't overlap* with what > the Path object does. But what you are proposing is a *superset* of what > Path does (because you're saying its a subclass), and I don't think that > will go over well. I agree. File system operations should be functions that operate on paths, not methods of a special kind of path. > Mike Orr wrote: >>I like the syntax of a join method. With a multi-arg constructor it's >>not necessary though. True, but seeing as we're talking about an algebra, it would be nice to have some operators for the most frequent operations. Perhaps p1 / p2 # combine paths p & ext # add extension p | ext # replace last extension p | "" # remove last extension >>But what harm is there in making them scalable to multiple extensions? >> >> .add_exts(*exts) >> .del_exts(N) >> .replace_exts(N, *exts) Not sure if this is worth it, since even when a file has multiple extensions, you're usually dealing with them one at a time. E.g. given foo.tar.gz, you first ungzip it to give foo.tar, and then untar it to give foo (conceptually, at least). > Someone in another message pointed out that paths, being based on > strings, are immutable, so this whole handling of extensions will have > to be done another way. The relevant methods would just have to return new paths instead of modifying in-place. -- Greg From greg.ewing at canterbury.ac.nz Thu Nov 9 01:07:01 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 09 Nov 2006 13:07:01 +1300 Subject: [Python-3000] Mini Path object In-Reply-To: <79990c6b0611080647s3f337c6dl875e02d693fb59a8@mail.gmail.com> References: <6e9196d20611012247w51d740fm68116bd98b6591d9@mail.gmail.com> <454EE6C6.4000701@acm.org> <6e9196d20611061437h35756e6fm1afa267f1048870b@mail.gmail.com> <45504EC4.50609@acm.org> <6e9196d20611071406j4d98a3ete3b464966d532e1c@mail.gmail.com> <4551AFA9.9090905@acm.org> <79990c6b0611080647s3f337c6dl875e02d693fb59a8@mail.gmail.com> Message-ID: <45527125.8060407@canterbury.ac.nz> Paul Moore wrote: > This requirement conflicts with that of a user who only uses one > platform, and wants (expects) to just enter paths in native format in > the config file. If the standard format were designed so as to be unambiguously distinguishable from all native formats, paths in either standard or platform format could be used as desired. -- Greg From janssen at parc.com Thu Nov 9 03:23:30 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 8 Nov 2006 18:23:30 PST Subject: [Python-3000] Mini Path object In-Reply-To: Your message of "Wed, 08 Nov 2006 16:07:01 PST." <45527125.8060407@canterbury.ac.nz> Message-ID: <06Nov8.182336pst."58648"@synergy1.parc.xerox.com> Greg Ewing writes: > If the standard format were designed so as to be > unambiguously distinguishable from all native > formats, ... All native formats both past and future. Bill From talin at acm.org Thu Nov 9 19:14:11 2006 From: talin at acm.org (Talin) Date: Thu, 09 Nov 2006 10:14:11 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <06Nov8.182336pst."58648"@synergy1.parc.xerox.com> References: <06Nov8.182336pst."58648"@synergy1.parc.xerox.com> Message-ID: <45536FF3.4070105@acm.org> Bill Janssen wrote: > Greg Ewing writes: >> If the standard format were designed so as to be >> unambiguously distinguishable from all native >> formats, ... > > All native formats both past and future. That's not difficult. I use 'posix' paths as my universal format. I convert them to native paths just before writing them out or passing to a subsystem that requires native paths. For reading paths, it depends on whether the paths are already in universal (posix) style or native style. If they are native, I use one method for reading them, if they are universal I use a different method (Well, in the current code base that's not true - since os.path always handles paths in posix format as well as native format, even when running on non-posix systems. So I just pass everything to os.path and it just works.) What this means is that universal paths need never be serialized - they are always converted to native form before being written anywhere. Which in turn implies that the 'universal' path format can be an in-memory-only format. And as such, it can express its 'universalness' by additional object properties or object types, rather than via any kind of weird string encoding. Thus, a universal path is nothing more than a posix path, except with a different object type. Representing non-posix concepts such as drive letters is done simply by not interpreting them; Or in some cases, deferring any interpretation until the user specifically calls a function to split off that part. So if you have a path beginning with "C:", it doesn't try to interpret what that means until you ask it to via splitdrive(). (This is a good reason to have paths represented as strings instead of as a tuple, since you can't defer interpretation this way with pre-parsed paths.) -- Talin From python at zesty.ca Thu Nov 9 20:57:52 2006 From: python at zesty.ca (Ka-Ping Yee) Date: Thu, 9 Nov 2006 13:57:52 -0600 (CST) Subject: [Python-3000] PEP 3104 added Message-ID: I've committed PEP 3104 to the SVN repository. It should appear at http://www.python.org/dev/peps/pep-3104/ shortly. The PEP is updated to add a note on the amount of code breakage in the standard library that we might expect for some of the possible keywords we might choose. The updated version is also available at http://zesty.ca/python/pep-3104.html if you want to see it now. -- ?!ng From p.f.moore at gmail.com Thu Nov 9 21:45:10 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 9 Nov 2006 20:45:10 +0000 Subject: [Python-3000] Mini Path object In-Reply-To: <45536FF3.4070105@acm.org> References: <45536FF3.4070105@acm.org> Message-ID: <79990c6b0611091245r2d0251f0q813951d8b94dca1e@mail.gmail.com> On 11/9/06, Talin wrote: > Bill Janssen wrote: > > Greg Ewing writes: > >> If the standard format were designed so as to be > >> unambiguously distinguishable from all native > >> formats, ... > > > > All native formats both past and future. > > That's not difficult. > > I use 'posix' paths as my universal format. I convert them to native > paths just before writing them out or passing to a subsystem that > requires native paths. Not a valid option. C:\Data\a_file.txt is a perfectly valid posix format filename. It's a file in the current directory (with some very strange characters in it). But it's also a native path in Windows. So how do you interpret it? And yes, I have seen precisely this problem occur with an application which took (posix) paths for use on a Unix server, but in a Windows client GUI. So we ended up with some really funky filenames on Unix to sort out :-( I stand by my assertion, it's a human problem, not a technical one. > What this means is that universal paths need never be serialized - they > are always converted to native form before being written anywhere. Again, I don't believe this is possible for all corner cases: what is the drive for /my/file on a Windows system? I'm not at all sure what you're trying to achieve here - if I'm running a program on a Windows system, I want to see Windows native paths for all external information (whether it's input from command line args or ini files, or output in messages etc). If I'm running that program on a Posix system, I want to see and use Posix files. If I want to write an ini file for the program which contains filenames, and which can be used unchanged on a Windows and a Posix system, I assert that I'm misguided (because I can't do it - drive letters will kill me as /data/xxx (or \data\xxx if you prefer) is ambiguous on Windows. If you mandate that all filenames must be relative, you could manage, but enforcing that is probably harder than just using native format everywhere and giving up on portable ini files... (I think this has strayed so far off topic as to be meaningless, so I'll leave it at that, and avoid perpetuating the thread...). Paul. From solipsis at pitrou.net Thu Nov 9 22:05:47 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 09 Nov 2006 22:05:47 +0100 Subject: [Python-3000] Mini Path object In-Reply-To: <79990c6b0611091245r2d0251f0q813951d8b94dca1e@mail.gmail.com> References: <45536FF3.4070105@acm.org> <79990c6b0611091245r2d0251f0q813951d8b94dca1e@mail.gmail.com> Message-ID: <1163106347.5190.20.camel@fsol> Le jeudi 09 novembre 2006 ? 20:45 +0000, Paul Moore a ?crit : > Again, I don't believe this is possible for all corner cases: what is > the drive for /my/file on a Windows system? Why not the current drive? > If you mandate that all filenames must be relative, you could manage, > but enforcing that is probably harder than just using native format > everywhere and giving up on portable ini files... Portable ini files can only use relative paths anyway. If you want to play nice with the OS, you have to follow its existing conventions for file locations, and it isn't the same everywhere (you don't use /usr/share under Windows, or /Programs/Shared Files under Linux). So it's impossible to come up with portable absolute paths. From mbk.lists at gmail.com Thu Nov 9 22:12:48 2006 From: mbk.lists at gmail.com (Mike Krell) Date: Thu, 9 Nov 2006 14:12:48 -0700 Subject: [Python-3000] PEP 3104 added In-Reply-To: References: Message-ID: Ka-Ping, Thanks for all of your hard work on this. Well done! Mike From sluggoster at gmail.com Fri Nov 10 00:41:52 2006 From: sluggoster at gmail.com (Mike Orr) Date: Thu, 9 Nov 2006 15:41:52 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <45536FF3.4070105@acm.org> References: <45536FF3.4070105@acm.org> Message-ID: <6e9196d20611091541y1780accfy82167ae14759fe34@mail.gmail.com> On 11/9/06, Talin wrote: > Bill Janssen wrote: > > Greg Ewing writes: > >> If the standard format were designed so as to be > >> unambiguously distinguishable from all native > >> formats, ... > > > > All native formats both past and future. > > That's not difficult. > > I use 'posix' paths as my universal format. I convert them to native > paths just before writing them out or passing to a subsystem that > requires native paths. > > For reading paths, it depends on whether the paths are already in > universal (posix) style or native style. If they are native, I use one > method for reading them, if they are universal I use a different method > (Well, in the current code base that's not true - since os.path always > handles paths in posix format as well as native format, even when > running on non-posix systems. So I just pass everything to os.path and > it just works.) > > What this means is that universal paths need never be serialized - they > are always converted to native form before being written anywhere. Which > in turn implies that the 'universal' path format can be an > in-memory-only format. And as such, it can express its 'universalness' > by additional object properties or object types, rather than via any > kind of weird string encoding. > > Thus, a universal path is nothing more than a posix path, except with a > different object type. > > Representing non-posix concepts such as drive letters is done simply by > not interpreting them; Or in some cases, deferring any interpretation > until the user specifically calls a function to split off that part. So > if you have a path beginning with "C:", it doesn't try to interpret what > that means until you ask it to via splitdrive(). > > (This is a good reason to have paths represented as strings instead of > as a tuple, since you can't defer interpretation this way with > pre-parsed paths.) Er, is there an API recommendation in there somewhere, or is this just philosophy or usage style? Posix is the lingua franca of the current age, and the non-compliant OSes (Windows and Mac) have various levels of compatibility with it. So using Posix in portable config files is justifiable. However, a list of components works better as an intermediate format for autoconverting. You're going to get lists anyway with .split() and .join(). There's no reason to convert from format A to Posix to format B. If we define the first component as the absolute root or '', the source format knows how to convert it to that, and the destination format will at least recognize it as an even if it doesn't know what it means. That's enough information to raise an exception or convert the path to relative. -- Mike Orr From sluggoster at gmail.com Fri Nov 10 00:44:31 2006 From: sluggoster at gmail.com (Mike Orr) Date: Thu, 9 Nov 2006 15:44:31 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611091541y1780accfy82167ae14759fe34@mail.gmail.com> References: <45536FF3.4070105@acm.org> <6e9196d20611091541y1780accfy82167ae14759fe34@mail.gmail.com> Message-ID: <6e9196d20611091544x44766339h1d091b30ad076f38@mail.gmail.com> On 11/9/06, Mike Orr wrote: > the destination format will at least recognize it as an even if it > doesn't know what it means. ... recognize it as an absolute path ... -- Mike Orr From greg.ewing at canterbury.ac.nz Fri Nov 10 01:18:34 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 10 Nov 2006 13:18:34 +1300 Subject: [Python-3000] Mini Path object In-Reply-To: <45536FF3.4070105@acm.org> References: <06Nov8.182336pst.58648@synergy1.parc.xerox.com> <45536FF3.4070105@acm.org> Message-ID: <4553C55A.5020500@canterbury.ac.nz> Talin wrote: > Bill Janssen wrote: > > > All native formats both past and future. > > That's not difficult. > > I use 'posix' paths as my universal format. That assumes you can always distinguish a posix path from some other kind of path. That's not always the case, e.g. consider /hello:wide/world Is that a posix path or not? It could be, but it could also be a classic MacOS pathname referring to a file called "wide/world" on a volume called "/hello". It's fine if there's a rule that all paths in your config file are in the universal format. But we were talking about allowing the user to use either native or universal format. > What this means is that universal paths need never be serialized - they > are always converted to native form before being written anywhere. Which > in turn implies that the 'universal' path format can be an > in-memory-only format. Hmmm, we seem to have crossed topics again -- I thought we were talking about storing paths in config files. > os.path always > handles paths in posix format as well as native format, even when > running on non-posix systems. Really? I wasn't aware of that. I don't see how it *can*, given what I said above. If you're talking about the way you can use forward or backward slashes on Windows, that's a property of the Windows kernel, not os.path. And it only applies to Unix vs. Windows, not in general. > Representing non-posix concepts such as drive letters is done simply by > not interpreting them; Or in some cases, deferring any interpretation > until the user specifically calls a function to split off that part. So > if you have a path beginning with "C:", it doesn't try to interpret what > that means until you ask it to via splitdrive(). What if you need to know whether it's an absolute or relative path, e.g. so you can join it to another path? You can't do that without knowing something about the semantics. -- Greg From sluggoster at gmail.com Fri Nov 10 03:55:37 2006 From: sluggoster at gmail.com (Mike Orr) Date: Thu, 9 Nov 2006 18:55:37 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <4553C55A.5020500@canterbury.ac.nz> References: <06Nov8.182336pst.58648@synergy1.parc.xerox.com> <45536FF3.4070105@acm.org> <4553C55A.5020500@canterbury.ac.nz> Message-ID: <6e9196d20611091855h53e910c5w3a8e6019c64c79a1@mail.gmail.com> On 11/9/06, Greg Ewing wrote: > Talin wrote: > > I use 'posix' paths as my universal format. > > That assumes you can always distinguish a posix path from > some other kind of path. That's not always the case, > e.g. consider > > /hello:wide/world > > Is that a posix path or not? It could be, but it could > also be a classic MacOS pathname referring to a file > called "wide/world" on a volume called "/hello". The user will tell us what the source format is. If they don't know, they've got bigger problems than we can handle. One can imagine a guesspath() or any2posix() function, but I can't imagine it would be widely used... or 100% correct. -- Mike Orr From talin at acm.org Fri Nov 10 05:49:04 2006 From: talin at acm.org (Talin) Date: Thu, 09 Nov 2006 20:49:04 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611091541y1780accfy82167ae14759fe34@mail.gmail.com> References: <45536FF3.4070105@acm.org> <6e9196d20611091541y1780accfy82167ae14759fe34@mail.gmail.com> Message-ID: <455404C0.70605@acm.org> Mike Orr wrote: > On 11/9/06, Talin wrote: >> Bill Janssen wrote: >>> Greg Ewing writes: >>>> If the standard format were designed so as to be >>>> unambiguously distinguishable from all native >>>> formats, ... >>> All native formats both past and future. >> That's not difficult. >> >> I use 'posix' paths as my universal format. I convert them to native >> paths just before writing them out or passing to a subsystem that >> requires native paths. >> (much useless Talin blathering omitted) >> (This is a good reason to have paths represented as strings instead of >> as a tuple, since you can't defer interpretation this way with >> pre-parsed paths.) > > Er, is there an API recommendation in there somewhere, or is this just > philosophy or usage style? I am indeed trying to describe a usage style, one that is not only used by me but by a large number of pre-existing apps. My goal in describing this is to insure that any new path system allows similar kinds of usage patterns. What I am arguing against is an overly strict and pedantic interpretation of the differences in path representation across platforms. True, in theory, you can't compare a windows path with a posix path, but in practice it generally "just works". I sometimes notice a tendency on this list that anything which is not 100% provably correct should be forbidden, no matter how useful it is. Let me give a concrete example: In my TurboGears project, there is a file called "thes.egg-info/sqlobject.txt" that contains this string: history_dir=$base/thes/sqlobject-history I didn't write this, it was generated by the 'quickstart' script. I fully expect to be able to package up my TG app from my OSX machine, transport it to my Windows box, and have it work. And I would fully expect to be able to do the same in any future version of TurboGears running on a future version of Python. Now, one can argue that this string is meaningless when taken from one platform to another, but the only thing that is accomplished by that argument is to remove functionality that current users find useful and valuable. I could bring up other examples, but do I really need to? (And would you really want to read them?) > Posix is the lingua franca of the current age, and the non-compliant > OSes (Windows and Mac) have various levels of compatibility with it. > So using Posix in portable config files is justifiable. I don't have a problem with this. > However, a list of components works better as an intermediate format > for autoconverting. You're going to get lists anyway with .split() > and .join(). There's no reason to convert from format A to Posix to > format B. If we define the first component as the absolute root or > '', the source format knows how to convert it to that, and the > destination format will at least recognize it as an even if it > doesn't know what it means. That's enough information to raise an > exception or convert the path to relative. The way I tend to deal with these kinds of issues is to let the "absolute" part of the path be an opaque blob that is uninterpreted by me or my code. Code that does path algebra is really only interested in the 'tail' of the path (except in cases where you have an absolute override during a combine). So it really doesn't matter to me what the 'head' part looks like - I just leave it alone and don't worry about it. -- Talin From sluggoster at gmail.com Fri Nov 10 07:11:10 2006 From: sluggoster at gmail.com (Mike Orr) Date: Thu, 9 Nov 2006 22:11:10 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <455404C0.70605@acm.org> References: <45536FF3.4070105@acm.org> <6e9196d20611091541y1780accfy82167ae14759fe34@mail.gmail.com> <455404C0.70605@acm.org> Message-ID: <6e9196d20611092211r2c16c734l52bb6448c69190c7@mail.gmail.com> On 11/9/06, Talin wrote: > Mike Orr wrote: > > On 11/9/06, Talin wrote: > >> (This is a good reason to have paths represented as strings instead of > >> as a tuple, since you can't defer interpretation this way with > >> pre-parsed paths.) > > > > Er, is there an API recommendation in there somewhere, or is this just > > philosophy or usage style? > > I am indeed trying to describe a usage style, one that is not only used > by me but by a large number of pre-existing apps. My goal in describing > this is to insure that any new path system allows similar kinds of usage > patterns. > > What I am arguing against is an overly strict and pedantic > interpretation of the differences in path representation across > platforms. True, in theory, you can't compare a windows path with a > posix path, but in practice it generally "just works". > > I sometimes notice a tendency on this list that anything which is not > 100% provably correct should be forbidden, no matter how useful it is. > > Let me give a concrete example: In my TurboGears project, there is a > file called "thes.egg-info/sqlobject.txt" that contains this string: > > history_dir=$base/thes/sqlobject-history > > I didn't write this, it was generated by the 'quickstart' script. I > fully expect to be able to package up my TG app from my OSX machine, > transport it to my Windows box, and have it work. And I would fully > expect to be able to do the same in any future version of TurboGears > running on a future version of Python. > > Now, one can argue that this string is meaningless when taken from one > platform to another, but the only thing that is accomplished by that > argument is to remove functionality that current users find useful and > valuable. I'm looking for "we need a method named foo with signature bar that does this..." Or "make sure method blah doesn't do this...." Otherwise I don't know what to do with the information. Can you make a use case showing how you'd ideally like to interact with the module (what you'd call) and what it should do, starting wtih a config file fragment and ending with a file operation (mkdir, open, etc). In the case above, the question is how TurboGears would interact with the new path module. First we'd need to know what TG is doing internally now; then we can verify we're not losing any functionality. I would guess TG is doing a text substitution of $base and not even using a join function. That's outside of our control. We know that our join/combine method is tied to a particular platform, and if given two path fragments from that same platform will join them properly. Beyond that all bets are off -- no using the wrong platform class, and no mixing path fragments from different platforms. You can't combine "C:\" and "thes/sqlobject-history" into anything useful. But os.path.join() has the same restrictions. On Windows you're depending on "something below Python" to interpret the forward slashes properly when you do a file operation. This is outside the path module's scope. But it looks like we'll be adding the ability to convert at least some paths from Posix to Windows -- something os.path doesn't do. -- Mike Orr From talin at acm.org Fri Nov 10 08:10:13 2006 From: talin at acm.org (Talin) Date: Thu, 09 Nov 2006 23:10:13 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611092211r2c16c734l52bb6448c69190c7@mail.gmail.com> References: <45536FF3.4070105@acm.org> <6e9196d20611091541y1780accfy82167ae14759fe34@mail.gmail.com> <455404C0.70605@acm.org> <6e9196d20611092211r2c16c734l52bb6448c69190c7@mail.gmail.com> Message-ID: <455425D5.9040002@acm.org> Mike Orr wrote: > I'm looking for "we need a method named foo with signature bar that > does this..." Or "make sure method blah doesn't do this...." > Otherwise I don't know what to do with the information. Can you make > a use case showing how you'd ideally like to interact with the module > (what you'd call) and what it should do, starting wtih a config file > fragment and ending with a file operation (mkdir, open, etc). I could answer this better if there was an actual spec document to look at - right now, there are so many suggestions in play that I've kind of lost track of where we are. -- Talin From p.f.moore at gmail.com Fri Nov 10 09:59:42 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 10 Nov 2006 08:59:42 +0000 Subject: [Python-3000] Mini Path object In-Reply-To: <455404C0.70605@acm.org> References: <45536FF3.4070105@acm.org> <6e9196d20611091541y1780accfy82167ae14759fe34@mail.gmail.com> <455404C0.70605@acm.org> Message-ID: <79990c6b0611100059y7ce30d3exd3886b6d93645cf1@mail.gmail.com> On 11/10/06, Talin wrote: > What I am arguing against is an overly strict and pedantic > interpretation of the differences in path representation across > platforms. True, in theory, you can't compare a windows path with a > posix path, but in practice it generally "just works". > > I sometimes notice a tendency on this list that anything which is not > 100% provably correct should be forbidden, no matter how useful it is. That's a fair point, and you're right that 99% of the time, things work fine. I was reacting against the implication in your comments of "use Posix and you're fine" as if that precluded the need for thinking about the issue. As I've been bitten many, many times by code which assumes Posix is all that matters, and gives unnatural (or indeed simply incorrect) results on Windows, this is a bit of a sore point for me. As long as you think about what you're doing, and don't assume portability is free, then all of the proposed APIs will work fine. Which one is *better* is a different discussion... :-) Paul. From niki.spahiev at gmail.com Fri Nov 10 10:18:28 2006 From: niki.spahiev at gmail.com (Niki Spahiev) Date: Fri, 10 Nov 2006 11:18:28 +0200 Subject: [Python-3000] Mini Path object In-Reply-To: <4553C55A.5020500@canterbury.ac.nz> References: <06Nov8.182336pst.58648@synergy1.parc.xerox.com> <45536FF3.4070105@acm.org> <4553C55A.5020500@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > What if you need to know whether it's an absolute or > relative path, e.g. so you can join it to another path? > You can't do that without knowing something about the > semantics. Maybe use file: URLs for universal format. They become more and more used in internet era. Niki Spahiev From jimjjewett at gmail.com Fri Nov 10 18:37:21 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 10 Nov 2006 12:37:21 -0500 Subject: [Python-3000] Mini Path object In-Reply-To: References: <06Nov8.182336pst.58648@synergy1.parc.xerox.com> <45536FF3.4070105@acm.org> <4553C55A.5020500@canterbury.ac.nz> Message-ID: On 11/10/06, Niki Spahiev wrote: > Greg Ewing wrote: > > What if you need to know whether it's an absolute or > > relative path, e.g. so you can join it to another path? > > You can't do that without knowing something about the > > semantics. > Maybe use file: URLs for universal format. They become more and more > used in internet era. the file: protocol has a long and messy history; it is now probably impossible to use in a consistent way without breaking something. It works fairly well for the tail of the path, but so does an ordinary list of path components. In general practice, even just using a string with a slash in either direction as the separator will work. If that has too many corner cases for your taste, then you should treat "file:" as undefined behavior. -jJ From greg.ewing at canterbury.ac.nz Sat Nov 11 01:20:27 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 11 Nov 2006 13:20:27 +1300 Subject: [Python-3000] Mini Path object In-Reply-To: References: <06Nov8.182336pst.58648@synergy1.parc.xerox.com> <45536FF3.4070105@acm.org> <4553C55A.5020500@canterbury.ac.nz> Message-ID: <4555174B.9020602@canterbury.ac.nz> Niki Spahiev wrote: > Maybe use file: URLs for universal format. They become more and more > used in internet era. You still need some outside way of telling whether you're dealing with a universal format, since "file:" is a valid pathname component in Unix, a valid volume name in classic MacOS, etc. -- Greg From qrczak at knm.org.pl Sun Nov 12 11:58:27 2006 From: qrczak at knm.org.pl (Marcin 'Qrczak' Kowalczyk) Date: Sun, 12 Nov 2006 11:58:27 +0100 Subject: [Python-3000] Mini Path object In-Reply-To: <79990c6b0611091245r2d0251f0q813951d8b94dca1e@mail.gmail.com> (Paul Moore's message of "Thu, 9 Nov 2006 20:45:10 +0000") References: <45536FF3.4070105@acm.org> <79990c6b0611091245r2d0251f0q813951d8b94dca1e@mail.gmail.com> Message-ID: <87ac2x7wz0.fsf@qrnik.zagroda> "Paul Moore" writes: >> I use 'posix' paths as my universal format. I convert them to native >> paths just before writing them out or passing to a subsystem that >> requires native paths. > > Not a valid option. C:\Data\a_file.txt is a perfectly valid posix > format filename. It's a file in the current directory (with some very > strange characters in it). But it's also a native path in Windows. So > how do you interpret it? The Boost Filesystem library has solved this problem by using the POSIX-based syntax by default, with system-dependent modifications possible, and by specifying a portable subset which avoids all the system dependencies (but possibly can't express some filenames). There is a separate constructor which accepts the native path syntax. http://www.boost.org/libs/filesystem/doc/path.htm -- __("< Marcin Kowalczyk \__/ qrczak at knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/ From gsakkis at rutgers.edu Mon Nov 13 19:44:10 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Mon, 13 Nov 2006 13:44:10 -0500 Subject: [Python-3000] Builtin iterator type Message-ID: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> Following up on a recent c.l.py thread (http://groups.google.com/group/comp.lang.python/browse_frm/thread/42818717b400bcd4/#), I'd like to get an idea from python-dev folks on how much of a chance is there for a builtin iterator type. Although there might be a slight possibility to make this proposal backwards compatible, I am mainly targetting Py3K, so compatibility is not a strict requirement. The key points are: 1. Formalize the currently informal concept of iterator by changing iter() from a builtin function to a type. 2. Augment this type with operators to implement operations currently done explicitly by itertools.*. 3. Make this type a base for all other (existing and future) iterator types, e.g. xrange and generator. As a proof of concept, I provide below a sample implementation of how I imagine this type to work (also posted in the original c.l.py. thread): from itertools import chain, tee, islice import __builtin__ _builtin_iter = __builtin__.iter class iter(object): def __init__(self, iterable): self._it = _builtin_iter(iterable) def __iter__(self): return self def next(self): return self._it.next() def __getitem__(self, index): if isinstance(index, int): try: return islice(self._it, index, index+1).next() except StopIteration: raise IndexError('Index %d out of range' % index) else: start,stop,step = index.start, index.stop, index.step if start is None: start = 0 if step is None: step = 1 return islice(self._it, start, stop, step) def __add__(self, other): return chain(self._it, other) def __radd__(self,other): return chain(other, self._it) def __mul__(self, num): return chain(*tee(self._it,num)) __rmul__ = __mul__ __builtin__.iter = iter if __name__ == '__main__': def irange(*args): return iter(xrange(*args)) assert list(irange(5)[:3]) == range(5)[:3] assert list(irange(5)[3:]) == range(5)[3:] assert list(irange(5)[1:3]) == range(5)[1:3] assert list(irange(5)[3:1]) == range(5)[3:1] assert list(irange(5)[:]) == range(5)[:] assert irange(5)[3] == range(5)[3] s = range(5) + range(7,9) assert list(irange(5) + irange(7,9)) == s assert list(irange(5) + range(7,9)) == s assert list(range(5) + irange(7,9)) == s s = range(5) * 3 assert list(irange(5) * 3) == s assert list(3 * irange(5)) == s Thoughts, opinions, suggestions, objections, all welcome. Regards, George From guido at python.org Mon Nov 13 20:31:50 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 13 Nov 2006 11:31:50 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> Message-ID: Hm. Without knowing much of the background, this appears to be a worrysome trend away from duck typing. Why would I have to inherit from a standard class just so that I can implement next()? What's the advantage of the proposed change? Are you going to propose similar changes for all standard de-facto interfaces, like sequences, mappings, files etc.? Unconvinced, --Guido On 11/13/06, George Sakkis wrote: > Following up on a recent c.l.py thread > (http://groups.google.com/group/comp.lang.python/browse_frm/thread/42818717b400bcd4/#), > I'd like to get an idea from python-dev folks on how much of a chance > is there for a builtin iterator type. Although there might be a slight > possibility to make this proposal backwards compatible, I am mainly > targetting Py3K, so compatibility is not a strict requirement. > > The key points are: > 1. Formalize the currently informal concept of iterator by changing > iter() from a builtin function to a type. > 2. Augment this type with operators to implement operations currently > done explicitly by itertools.*. > 3. Make this type a base for all other (existing and future) iterator > types, e.g. xrange and generator. > > As a proof of concept, I provide below a sample implementation of how > I imagine this type to work (also posted in the original c.l.py. > thread): > > > from itertools import chain, tee, islice > > import __builtin__ > _builtin_iter = __builtin__.iter > > class iter(object): > > def __init__(self, iterable): > self._it = _builtin_iter(iterable) > > def __iter__(self): > return self > def next(self): > return self._it.next() > > def __getitem__(self, index): > if isinstance(index, int): > try: return islice(self._it, index, index+1).next() > except StopIteration: > raise IndexError('Index %d out of range' % index) > else: > start,stop,step = index.start, index.stop, index.step > if start is None: start = 0 > if step is None: step = 1 > return islice(self._it, start, stop, step) > > def __add__(self, other): > return chain(self._it, other) > def __radd__(self,other): > return chain(other, self._it) > > def __mul__(self, num): > return chain(*tee(self._it,num)) > > __rmul__ = __mul__ > > __builtin__.iter = iter > > if __name__ == '__main__': > def irange(*args): > return iter(xrange(*args)) > > assert list(irange(5)[:3]) == range(5)[:3] > assert list(irange(5)[3:]) == range(5)[3:] > assert list(irange(5)[1:3]) == range(5)[1:3] > assert list(irange(5)[3:1]) == range(5)[3:1] > assert list(irange(5)[:]) == range(5)[:] > assert irange(5)[3] == range(5)[3] > > s = range(5) + range(7,9) > assert list(irange(5) + irange(7,9)) == s > assert list(irange(5) + range(7,9)) == s > assert list(range(5) + irange(7,9)) == s > > s = range(5) * 3 > assert list(irange(5) * 3) == s > assert list(3 * irange(5)) == s > > > Thoughts, opinions, suggestions, objections, all welcome. > > Regards, > George > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From janssen at parc.com Mon Nov 13 23:51:29 2006 From: janssen at parc.com (Bill Janssen) Date: Mon, 13 Nov 2006 14:51:29 PST Subject: [Python-3000] Builtin iterator type In-Reply-To: Your message of "Mon, 13 Nov 2006 11:31:50 PST." Message-ID: <06Nov13.145133pst."58648"@synergy1.parc.xerox.com> > this appears to be a worrysome trend away from duck typing Duck typing is a seriously bad idea, forced on Python by the now obsolete split between built-in types and user-defined types. > Are you going to propose similar > changes for all standard de-facto interfaces, like sequences, > mappings, files etc.? Good idea. Bill From exarkun at divmod.com Tue Nov 14 00:15:38 2006 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Mon, 13 Nov 2006 18:15:38 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <06Nov13.145133pst."58648"@synergy1.parc.xerox.com> Message-ID: <20061113231538.20948.713332044.divmod.quotient.26448@ohm> On Mon, 13 Nov 2006 14:51:29 PST, Bill Janssen wrote: >> this appears to be a worrysome trend away from duck typing > >Duck typing is a seriously bad idea, forced on Python by the now >obsolete split between built-in types and user-defined types. Wow. Just wow. Anyway, the proposal was in no way suggesting moving away from duck typing. The new iter was to be a _wrapper_ around any existing iterator, not a base class for iterators. Jean-Paul From gsakkis at rutgers.edu Mon Nov 13 21:39:43 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Mon, 13 Nov 2006 15:39:43 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> Message-ID: <91ad5bf80611131239g72afbfaahac86c2387e1f10cf@mail.gmail.com> On 11/13/06, Guido van Rossum wrote: > Hm. Without knowing much of the background, this appears to be a > worrysome trend away from duck typing. Why would I have to inherit > from a standard class just so that I can implement next()? What's the > advantage of the proposed change? Duck typing will still apply of course; an object that just implements next() without inheriting from iter will continue to be iterable. The advantage is the iterator methods and syntax sugar you get for free, instead of importing them explicitly from itertools (I've noticed that most of my modules lately start with a variation of "import sys, itertools as it"). The way I see it, the trend is not away from duck typing but towards iterability (generators, itertools, gen. expressions). A builtin iter type replacing for the most part the need for itertools is a natural next step to this direction. > Are you going to propose similar > changes for all standard de-facto interfaces, like sequences, > mappings, files etc.? No, I won't (at least not for now ;-)). Notice however that most user-defined sequences, mappings, etc. usually don't start from scratch; they either inherit from an existing type or from an appropriate mixin. For a baseline iterator that just implements next() there's no need for a base type or mixin, but if you consider that itertools.* functionality should/would-be-nice to be provided as methods and/or syntax instead of imported functions, the parallel becomes apparent. George > On 11/13/06, George Sakkis wrote: > > Following up on a recent c.l.py thread > > (http://groups.google.com/group/comp.lang.python/browse_frm/thread/42818717b400bcd4/#), > > I'd like to get an idea from python-dev folks on how much of a chance > > is there for a builtin iterator type. Although there might be a slight > > possibility to make this proposal backwards compatible, I am mainly > > targetting Py3K, so compatibility is not a strict requirement. > > > > The key points are: > > 1. Formalize the currently informal concept of iterator by changing > > iter() from a builtin function to a type. > > 2. Augment this type with operators to implement operations currently > > done explicitly by itertools.*. > > 3. Make this type a base for all other (existing and future) iterator > > types, e.g. xrange and generator. > > > > As a proof of concept, I provide below a sample implementation of how > > I imagine this type to work (also posted in the original c.l.py. > > thread): > > > > > > from itertools import chain, tee, islice > > > > import __builtin__ > > _builtin_iter = __builtin__.iter > > > > class iter(object): > > > > def __init__(self, iterable): > > self._it = _builtin_iter(iterable) > > > > def __iter__(self): > > return self > > def next(self): > > return self._it.next() > > > > def __getitem__(self, index): > > if isinstance(index, int): > > try: return islice(self._it, index, index+1).next() > > except StopIteration: > > raise IndexError('Index %d out of range' % index) > > else: > > start,stop,step = index.start, index.stop, index.step > > if start is None: start = 0 > > if step is None: step = 1 > > return islice(self._it, start, stop, step) > > > > def __add__(self, other): > > return chain(self._it, other) > > def __radd__(self,other): > > return chain(other, self._it) > > > > def __mul__(self, num): > > return chain(*tee(self._it,num)) > > > > __rmul__ = __mul__ > > > > __builtin__.iter = iter > > > > if __name__ == '__main__': > > def irange(*args): > > return iter(xrange(*args)) > > > > assert list(irange(5)[:3]) == range(5)[:3] > > assert list(irange(5)[3:]) == range(5)[3:] > > assert list(irange(5)[1:3]) == range(5)[1:3] > > assert list(irange(5)[3:1]) == range(5)[3:1] > > assert list(irange(5)[:]) == range(5)[:] > > assert irange(5)[3] == range(5)[3] > > > > s = range(5) + range(7,9) > > assert list(irange(5) + irange(7,9)) == s > > assert list(irange(5) + range(7,9)) == s > > assert list(range(5) + irange(7,9)) == s > > > > s = range(5) * 3 > > assert list(irange(5) * 3) == s > > assert list(3 * irange(5)) == s > > > > > > Thoughts, opinions, suggestions, objections, all welcome. > > > > Regards, > > George > > _______________________________________________ > > Python-3000 mailing list > > Python-3000 at python.org > > http://mail.python.org/mailman/listinfo/python-3000 > > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > From george.sakkis at gmail.com Tue Nov 14 00:49:49 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Mon, 13 Nov 2006 18:49:49 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> Message-ID: <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> On 11/13/06, Guido van Rossum wrote: > [Adding back python-3000 -- I'm not going to have private conversations] Sorry, I accidentally hit reply instead of reply-all; I realized it a minute later and I sent a cc to the list but it hasn't appeared yet.. strange. > On 11/13/06, George Sakkis wrote: > > On 11/13/06, Guido van Rossum wrote: > > > > > Hm. Without knowing much of the background, this appears to be a > > > worrysome trend away from duck typing. Why would I have to inherit > > > from a standard class just so that I can implement next()? What's the > > > advantage of the proposed change? > > > > Duck typing will still apply of course; an object that just implements > > next() without inheriting from iter will continue to be iterable. The > > advantage is the iterator methods and syntax sugar you get for free, > > instead of importing them explicitly from itertools (I've noticed that > > most of my modules lately start with a variation of "import sys, > > itertools as it"). > > Well, if you move the itertools functions to the base class, without > inheriting from that base class you'd lose the itertools > functionality. (Although many itertools functions take multiple > generator arguments so I'm not sure how e.g. imap or izip can become > methods without losing functionality.) It's actually straightforward; see below. > > 2. Augment this type with operators to implement operations currently > > done explicitly by itertools.*. > > Please go over all of the itertools functions and show how they can be > reimplemented as methods. I think you'll find most of them don't work > this way. There are currently 14 itertools functions; from those, only 2 don't take an iterable argument, count() and repeat(). The rest can be trivially rephrased as methods. Below is the updated iter class (renamed to Iter for backwards compatibility). I didn't add imap, ifilter, ifilterfalse and starmap as I think these should be deprecated in favor of gencomps. Instead I added enumerate and reversed, as I don't see why these are builtins and not itertools in the first place. from itertools import * class Iter(object): def __init__(self, iterable): self._it = iter(iterable) def next(self): return self._it.next() def __iter__(self): return self def __add__(self, other): return Iter(chain(self._it, other)) def __radd__(self,other): return Iter(chain(other, self._it)) def __mul__(self, num): return Iter(chain(*tee(self._it,num))) __rmul__ = __mul__ def __getitem__(self, index): if isinstance(index, int): try: return islice(self._it, index, index+1).next() except StopIteration: raise IndexError('Index %d out of range' % index) else: start,stop,step = index.start, index.stop, index.step if start is None: start = 0 if step is None: step = 1 return Iter(islice(self._it, start, stop, step)) def cycle(self): return Iter(cycle(self._it)) def takewhile(self, predicate): return Iter(takewhile(predicate, self._it)) def dropwhile(self, predicate): return Iter(dropwhile(predicate, self._it)) def groupby(self, keyfunc=None): return Iter(groupby(self._it, keyfunc)) def zip(self, *others): return Iter(izip(self._it, *others)) def enumerate(self): return Iter(enumerate(self._it)) def reversed(self): return Iter(reversed(self._it)) def copy(self): self._it, new = tee(self._it) return Iter(new) Unit tests omitted for brevity. George From guido at python.org Mon Nov 13 23:17:36 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 13 Nov 2006 14:17:36 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> Message-ID: [Adding back python-3000 -- I'm not going to have private conversations] On 11/13/06, George Sakkis wrote: > On 11/13/06, Guido van Rossum wrote: > > > Hm. Without knowing much of the background, this appears to be a > > worrysome trend away from duck typing. Why would I have to inherit > > from a standard class just so that I can implement next()? What's the > > advantage of the proposed change? > > Duck typing will still apply of course; an object that just implements > next() without inheriting from iter will continue to be iterable. The > advantage is the iterator methods and syntax sugar you get for free, > instead of importing them explicitly from itertools (I've noticed that > most of my modules lately start with a variation of "import sys, > itertools as it"). Well, if you move the itertools functions to the base class, without inheriting from that base class you'd lose the itertools functionality. (Although many itertools functions take multiple generator arguments so I'm not sure how e.g. imap or izip can become methods without losing functionality.) > The way I see it, the trend is not away from duck typing but towards > iterability (generators, itertools, gen. expressions). A builtin iter > type replacing for the most part the need for itertools is a natural > next step to this direction. You need to have a better (more technical) argument than that to convince me. Let me go back to your original arguments: > 1. Formalize the currently informal concept of iterator by changing > iter() from a builtin function to a type. That surely sounds like a step away from duck typing. Or what else does "formalize" mean to you? > 2. Augment this type with operators to implement operations currently > done explicitly by itertools.*. Please go over all of the itertools functions and show how they can be reimplemented as methods. I think you'll find most of them don't work this way. > 3. Make this type a base for all other (existing and future) iterator > types, e.g. xrange and generator. Here again you're contradicting your "duck typing isn't dead" claim -- you do say "all other (existing and future) iterator types". > > Are you going to propose similar > > changes for all standard de-facto interfaces, like sequences, > > mappings, files etc.? > > No, I won't (at least not for now ;-)). Notice however that most > user-defined sequences, mappings, etc. usually don't start from > scratch; they either inherit from an existing type or from an > appropriate mixin. For a baseline iterator that just implements next() > there's no need for a base type or mixin, but if you consider that > itertools.* functionality should/would-be-nice to be provided as > methods and/or syntax instead of imported functions, the parallel > becomes apparent. Despite invoking c.l.py I don't think you have broad support for this idea. It's not too late to drop it. > George > > > On 11/13/06, George Sakkis wrote: > > > Following up on a recent c.l.py thread > > > (http://groups.google.com/group/comp.lang.python/browse_frm/thread/42818717b400bcd4/#), > > > I'd like to get an idea from python-dev folks on how much of a chance > > > is there for a builtin iterator type. Although there might be a slight > > > possibility to make this proposal backwards compatible, I am mainly > > > targetting Py3K, so compatibility is not a strict requirement. > > > > > > The key points are: > > > 1. Formalize the currently informal concept of iterator by changing > > > iter() from a builtin function to a type. > > > 2. Augment this type with operators to implement operations currently > > > done explicitly by itertools.*. > > > 3. Make this type a base for all other (existing and future) iterator > > > types, e.g. xrange and generator. > > > > > > As a proof of concept, I provide below a sample implementation of how > > > I imagine this type to work (also posted in the original c.l.py. > > > thread): > > > > > > > > > from itertools import chain, tee, islice > > > > > > import __builtin__ > > > _builtin_iter = __builtin__.iter > > > > > > class iter(object): > > > > > > def __init__(self, iterable): > > > self._it = _builtin_iter(iterable) > > > > > > def __iter__(self): > > > return self > > > def next(self): > > > return self._it.next() > > > > > > def __getitem__(self, index): > > > if isinstance(index, int): > > > try: return islice(self._it, index, index+1).next() > > > except StopIteration: > > > raise IndexError('Index %d out of range' % index) > > > else: > > > start,stop,step = index.start, index.stop, index.step > > > if start is None: start = 0 > > > if step is None: step = 1 > > > return islice(self._it, start, stop, step) > > > > > > def __add__(self, other): > > > return chain(self._it, other) > > > def __radd__(self,other): > > > return chain(other, self._it) > > > > > > def __mul__(self, num): > > > return chain(*tee(self._it,num)) > > > > > > __rmul__ = __mul__ > > > > > > __builtin__.iter = iter > > > > > > if __name__ == '__main__': > > > def irange(*args): > > > return iter(xrange(*args)) > > > > > > assert list(irange(5)[:3]) == range(5)[:3] > > > assert list(irange(5)[3:]) == range(5)[3:] > > > assert list(irange(5)[1:3]) == range(5)[1:3] > > > assert list(irange(5)[3:1]) == range(5)[3:1] > > > assert list(irange(5)[:]) == range(5)[:] > > > assert irange(5)[3] == range(5)[3] > > > > > > s = range(5) + range(7,9) > > > assert list(irange(5) + irange(7,9)) == s > > > assert list(irange(5) + range(7,9)) == s > > > assert list(range(5) + irange(7,9)) == s > > > > > > s = range(5) * 3 > > > assert list(irange(5) * 3) == s > > > assert list(3 * irange(5)) == s > > > > > > > > > Thoughts, opinions, suggestions, objections, all welcome. > > > > > > Regards, > > > George > > > _______________________________________________ > > > Python-3000 mailing list > > > Python-3000 at python.org > > > http://mail.python.org/mailman/listinfo/python-3000 > > > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > > > > > > > > > -- > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ferringb at gmail.com Tue Nov 14 01:52:53 2006 From: ferringb at gmail.com (Brian Harring) Date: Mon, 13 Nov 2006 16:52:53 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> Message-ID: <20061114005252.GA9730@seldon> On Mon, Nov 13, 2006 at 01:44:10PM -0500, George Sakkis wrote: > As a proof of concept, I provide below a sample implementation of how > I imagine this type to work (also posted in the original c.l.py. > thread): > > from itertools import chain, tee, islice > > import __builtin__ > _builtin_iter = __builtin__.iter > > class iter(object): > > def __init__(self, iterable): > self._it = _builtin_iter(iterable) > > def __iter__(self): > return self > def next(self): > return self._it.next() > > def __getitem__(self, index): > if isinstance(index, int): > try: return islice(self._it, index, index+1).next() > except StopIteration: > raise IndexError('Index %d out of range' % index) > else: > start,stop,step = index.start, index.stop, index.step > if start is None: start = 0 > if step is None: step = 1 > return islice(self._it, start, stop, step) Tend to think __getitem__ on base iter is a rather bad idea due to the fact it implies the underlying iter is a mapping; it's not. Additionally... islice *does* have issues when working against an iterator that hasn't been protected via tee; that's not a complaint against islice mind you, it's just reality that if you islice an iterator, make damn sure you pull from that islice before going back to the original iterator; intermixing them is a grand way to shoot yourself in the foot. Current usage, it's easy to spot it (look for islice); overloading getitem however means it gets far harder, thus more likely to trip folks up (in my opinion). > def __add__(self, other): > return chain(self._it, other) The change to the iterator protocol here, that iter1 + iter2 == chain(iter1, iter2) breaks down for self iterating types; trying to shoe horn in add/radd means that for self-iters, add/radd/mul are now taken away from them. Duck typing works (imo) when the hook points all behave similiar; not convinced that abusing __add__ is wise, definitely not convinced __getitem__ is sane, especially since islice doesn't guarntee that it'll return that specific slice- just gurantees that it'll follow a specific pattern for consuming from the wrapped iter. So... nuking those, you wind up with existing iter. Perhaps providing sane examples of __getitem__ usage would help; add sort of makes sense sans it getting whacky for self-iters. ~harring -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20061113/018cee7e/attachment.pgp From guido at python.org Tue Nov 14 02:45:11 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 13 Nov 2006 17:45:11 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> Message-ID: As my final word, I think this is a seriously bad idea, and as you're not answering my challenge about duck typing I don't think you understand your own proposal. On 11/13/06, George Sakkis wrote: > On 11/13/06, Guido van Rossum wrote: > > > [Adding back python-3000 -- I'm not going to have private conversations] > > Sorry, I accidentally hit reply instead of reply-all; I realized it a > minute later and I sent a cc to the list but it hasn't appeared yet.. > strange. > > > On 11/13/06, George Sakkis wrote: > > > On 11/13/06, Guido van Rossum wrote: > > > > > > > Hm. Without knowing much of the background, this appears to be a > > > > worrysome trend away from duck typing. Why would I have to inherit > > > > from a standard class just so that I can implement next()? What's the > > > > advantage of the proposed change? > > > > > > Duck typing will still apply of course; an object that just implements > > > next() without inheriting from iter will continue to be iterable. The > > > advantage is the iterator methods and syntax sugar you get for free, > > > instead of importing them explicitly from itertools (I've noticed that > > > most of my modules lately start with a variation of "import sys, > > > itertools as it"). > > > > Well, if you move the itertools functions to the base class, without > > inheriting from that base class you'd lose the itertools > > functionality. (Although many itertools functions take multiple > > generator arguments so I'm not sure how e.g. imap or izip can become > > methods without losing functionality.) > > It's actually straightforward; see below. > > > > 2. Augment this type with operators to implement operations currently > > > done explicitly by itertools.*. > > > > Please go over all of the itertools functions and show how they can be > > reimplemented as methods. I think you'll find most of them don't work > > this way. > > There are currently 14 itertools functions; from those, only 2 don't > take an iterable argument, count() and repeat(). The rest can be > trivially rephrased as methods. Below is the updated iter class > (renamed to Iter for backwards compatibility). I didn't add imap, > ifilter, ifilterfalse and starmap as I think these should be > deprecated in favor of gencomps. Instead I added enumerate and > reversed, as I don't see why these are builtins and not itertools in > the first place. > > > from itertools import * > > class Iter(object): > > def __init__(self, iterable): self._it = iter(iterable) > def next(self): return self._it.next() > def __iter__(self): return self > def __add__(self, other): return Iter(chain(self._it, other)) > def __radd__(self,other): return Iter(chain(other, self._it)) > def __mul__(self, num): return Iter(chain(*tee(self._it,num))) > __rmul__ = __mul__ > > def __getitem__(self, index): > if isinstance(index, int): > try: return islice(self._it, index, index+1).next() > except StopIteration: > raise IndexError('Index %d out of range' % index) > else: > start,stop,step = index.start, index.stop, index.step > if start is None: start = 0 > if step is None: step = 1 > return Iter(islice(self._it, start, stop, step)) > > def cycle(self): return Iter(cycle(self._it)) > def takewhile(self, predicate): return Iter(takewhile(predicate, self._it)) > def dropwhile(self, predicate): return Iter(dropwhile(predicate, self._it)) > def groupby(self, keyfunc=None): return Iter(groupby(self._it, keyfunc)) > def zip(self, *others): return Iter(izip(self._it, *others)) > def enumerate(self): return Iter(enumerate(self._it)) > def reversed(self): return Iter(reversed(self._it)) > def copy(self): > self._it, new = tee(self._it) > return Iter(new) > > > Unit tests omitted for brevity. > > George > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From george.sakkis at gmail.com Tue Nov 14 04:26:54 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Mon, 13 Nov 2006 22:26:54 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> Message-ID: <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> On 11/13/06, Guido van Rossum wrote: > As my final word, I think this is a seriously bad idea, and as you're > not answering my challenge about duck typing I don't think you > understand your own proposal. I think I do, though I can't tell the same about the reasons of your objections to it. In your previous reply you mentioned that I need to have a more technical argument to convince you, and that's why I chose to reply with the most technical argument, a proof of concept implementation that addresses your concerns about how to incorporate the itertools functions as methods. As for the duck typing, I mentioned already that nobody forces you to extend this type to make some class an iterator, as nobody forces you to extend dict or dictmixin to write a user-defined mapping. You may well start from scratch implementing just next(); if you don't plan to use "+", "*" or any of the itertools operations on this type, extending Iter is useless. If you do plan to provide these operations though, you may either write them from scratch every time, or extend Iter. I honestly fail to understand your current objections. Is my analogy with dictmixin flawed ? Would anything change if I named it "itermixin" instead of iter or Iter ? I'm ok with the idea being rejected, but at least I'd like to understand the reasons. George From alexander.belopolsky at gmail.com Tue Nov 14 05:02:24 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 14 Nov 2006 04:02:24 +0000 (UTC) Subject: [Python-3000] Builtin iterator type References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> Message-ID: George Sakkis gmail.com> writes: ... > As for the duck typing, I mentioned already that nobody forces you to > extend this type to make some class an iterator, as nobody forces you > to extend dict or dictmixin to write a user-defined mapping. You may > well start from scratch implementing just next(); if you don't plan to > use "+", "*" or any of the itertools operations on this type, > extending Iter is useless. If you do plan to provide these operations > though, you may either write them from scratch every time, or extend > Iter. > Unfortunately, an iterator that only defines next() will not be compatible with the functions that would take advantage of your proposal. From gsakkis at rutgers.edu Tue Nov 14 05:19:51 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Mon, 13 Nov 2006 23:19:51 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <20061114005252.GA9730@seldon> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <20061114005252.GA9730@seldon> Message-ID: <91ad5bf80611132019m54de49d2t2ceefbfe181d2aac@mail.gmail.com> On 11/13/06, Brian Harring wrote: > Tend to think __getitem__ on base iter is a rather bad idea due to the > fact it implies the underlying iter is a mapping; it's not. Only mappings define __getitem__ ?!? Now that's a new one. > Additionally... islice *does* have issues when working against an > iterator that hasn't been protected via tee; that's not a complaint > against islice mind you, it's just reality that if you islice an > iterator, make damn sure you pull from that islice before going > back to the original iterator; intermixing them is a grand way to > shoot yourself in the foot. > > Current usage, it's easy to spot it (look for islice); overloading > getitem however means it gets far harder, thus more likely to trip > folks up (in my opinion). That might be a valid point, but according to this argument, Numpy slices are evil too as they return a view and not a copy, like most other sequences. There is a limit to polymorphism and duck typing; you can't always pretend you don't care about the actual type. > > def __add__(self, other): > > return chain(self._it, other) > > The change to the iterator protocol here, that iter1 + iter2 == > chain(iter1, iter2) breaks down for self iterating types; trying to > shoe horn in add/radd means that for self-iters, add/radd/mul are now > taken away from them. Excuse my ignorance, but what's a "self-iter" ? > Duck typing works (imo) when the hook points all behave similiar; not > convinced that abusing __add__ is wise, definitely not convinced > __getitem__ is sane, especially since islice doesn't guarntee that > it'll return that specific slice- just gurantees that it'll follow a > specific pattern for consuming from the wrapped iter. Care to provide an example of what you're talking about ? > So... nuking those, you wind up with existing iter. Perhaps providing > sane examples of __getitem__ usage would help; add sort of makes sense > sans it getting whacky for self-iters. If by sane example you mean expectable, not necessarily useful, here's a simple one off the top of my head: >>> from itertools import count >>> x = Iter(count(10)) >>> y = Iter('abcdefghij') >>> list((x[:3] + y[7:]) * 2) [10, 11, 12, 'h', 'i', 'j', 10, 11, 12, 'h', 'i', 'j'] Try rewriting it using explicit itertools calls and see what looks more readable. George From guido at python.org Tue Nov 14 05:21:16 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 13 Nov 2006 20:21:16 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> Message-ID: On 11/13/06, George Sakkis wrote: > On 11/13/06, Guido van Rossum wrote: > > > As my final word, I think this is a seriously bad idea, and as you're > > not answering my challenge about duck typing I don't think you > > understand your own proposal. > > I think I do, though I can't tell the same about the reasons of your > objections to it. In your previous reply you mentioned that I need to > have a more technical argument to convince you, and that's why I chose > to reply with the most technical argument, a proof of concept > implementation that addresses your concerns about how to incorporate > the itertools functions as methods. > > As for the duck typing, I mentioned already that nobody forces you to > extend this type to make some class an iterator, as nobody forces you > to extend dict or dictmixin to write a user-defined mapping. You may > well start from scratch implementing just next(); if you don't plan to > use "+", "*" or any of the itertools operations on this type, > extending Iter is useless. If you do plan to provide these operations > though, you may either write them from scratch every time, or extend > Iter. > > I honestly fail to understand your current objections. Is my analogy > with dictmixin flawed ? Would anything change if I named it > "itermixin" instead of iter or Iter ? I'm ok with the idea being > rejected, but at least I'd like to understand the reasons. The flaw is that you're creating two categories of iterators: those that support the various methods you're adding, and those that don't. This means that the itertools module can't be discarded, because it is still needed to support those operations for the iterators that don't have them natively. Thus, you're introducing two ways of doing the same thing -- using itertools (works for all iterators) or using the methods (only works for iterators that inherit from your base class). Please stop wasting everybody's time. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ronaldoussoren at mac.com Tue Nov 14 07:46:02 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 14 Nov 2006 07:46:02 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> Message-ID: <8E8031E5-BCE9-4FBF-BA9D-487AA3316E31@mac.com> On 14 Nov 2006, at 4:26 AM, George Sakkis wrote: > > I honestly fail to understand your current objections. Is my analogy > with dictmixin flawed ? Would anything change if I named it > "itermixin" instead of iter or Iter ? I'm ok with the idea being > rejected, but at least I'd like to understand the reasons. You basically propose to extend the iterator interface from 1 method to len(dir(itertools)) methods without any clear benefits. Saving a single import line is not a clear benefit. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3562 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20061114/47b3efb4/attachment-0001.bin From george.sakkis at gmail.com Tue Nov 14 07:59:57 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Tue, 14 Nov 2006 01:59:57 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <8E8031E5-BCE9-4FBF-BA9D-487AA3316E31@mac.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <8E8031E5-BCE9-4FBF-BA9D-487AA3316E31@mac.com> Message-ID: <91ad5bf80611132259v13c7e13ap911d201e189cc4bf@mail.gmail.com> On 11/14/06, Ronald Oussoren wrote: > > On 14 Nov 2006, at 4:26 AM, George Sakkis wrote: > > > > > I honestly fail to understand your current objections. Is my analogy > > with dictmixin flawed ? Would anything change if I named it > > "itermixin" instead of iter or Iter ? I'm ok with the idea being > > rejected, but at least I'd like to understand the reasons. > > You basically propose to extend the iterator interface from 1 method > to len(dir(itertools)) methods without any clear benefits. Saving a > single import line is not a clear benefit. The original motivation is to provide syntax sugar (+, *, indexing, slicing) to iterators, which you can't do with functions. The rest methods that would be spelled out with their current itertools names is a fortunate side-effect, not the primary motivation. George From george.sakkis at gmail.com Tue Nov 14 08:03:08 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Tue, 14 Nov 2006 02:03:08 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> Message-ID: <91ad5bf80611132303y33ecee81k8d7b2093d4b6806d@mail.gmail.com> On 11/13/06, Guido van Rossum wrote: > On 11/13/06, George Sakkis wrote: > > I honestly fail to understand your current objections. Is my analogy > > with dictmixin flawed ? Would anything change if I named it > > "itermixin" instead of iter or Iter ? I'm ok with the idea being > > rejected, but at least I'd like to understand the reasons. > > The flaw is that you're creating two categories of iterators: those > that support the various methods you're adding, and those that don't. > This means that the itertools module can't be discarded, because it is > still needed to support those operations for the iterators that don't > have them natively. Thus, you're introducing two ways of doing the > same thing -- using itertools (works for all iterators) or using the > methods (only works for iterators that inherit from your base class). Understood. Any _technical_ reasons then why shouldn't all iterators inherit from it, or "duck typing is The Right Way" should be taken as an axiom ? Why do we somehow _need_ itertools when we don't need sequencetools, mappingtools, etc ? What's so special about the iterator protocol ? George From jcarlson at uci.edu Tue Nov 14 09:09:31 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 14 Nov 2006 00:09:31 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611132259v13c7e13ap911d201e189cc4bf@mail.gmail.com> References: <8E8031E5-BCE9-4FBF-BA9D-487AA3316E31@mac.com> <91ad5bf80611132259v13c7e13ap911d201e189cc4bf@mail.gmail.com> Message-ID: <20061113235118.82E8.JCARLSON@uci.edu> "George Sakkis" wrote: > The original motivation is to provide syntax sugar (+, *, indexing, > slicing) to iterators, which you can't do with functions. The rest > methods that would be spelled out with their current itertools names > is a fortunate side-effect, not the primary motivation. I've never needed (the equivalent of) +, *, indexing, or slicing on arbitrary iterators (I make them iterators so that I can iterate over them), so I don't see the need to make the current object more cumbersome than necessary. For those who would like the equivalent of +, *, indexing, or slicing arbitrary iterators, there is the itertools module. -1 - Josiah From gsakkis at rutgers.edu Tue Nov 14 05:36:25 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Mon, 13 Nov 2006 23:36:25 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> Message-ID: <91ad5bf80611132036obdb042i1b02a860569d86ac@mail.gmail.com> On 11/13/06, Guido van Rossum wrote: > On 11/13/06, George Sakkis wrote: > > On 11/13/06, Guido van Rossum wrote: > > > > > As my final word, I think this is a seriously bad idea, and as you're > > > not answering my challenge about duck typing I don't think you > > > understand your own proposal. > > > > I think I do, though I can't tell the same about the reasons of your > > objections to it. In your previous reply you mentioned that I need to > > have a more technical argument to convince you, and that's why I chose > > to reply with the most technical argument, a proof of concept > > implementation that addresses your concerns about how to incorporate > > the itertools functions as methods. > > > > As for the duck typing, I mentioned already that nobody forces you to > > extend this type to make some class an iterator, as nobody forces you > > to extend dict or dictmixin to write a user-defined mapping. You may > > well start from scratch implementing just next(); if you don't plan to > > use "+", "*" or any of the itertools operations on this type, > > extending Iter is useless. If you do plan to provide these operations > > though, you may either write them from scratch every time, or extend > > Iter. > > > > I honestly fail to understand your current objections. Is my analogy > > with dictmixin flawed ? Would anything change if I named it > > "itermixin" instead of iter or Iter ? I'm ok with the idea being > > rejected, but at least I'd like to understand the reasons. > > The flaw is that you're creating two categories of iterators: those > that support the various methods you're adding, and those that don't. > This means that the itertools module can't be discarded, because it is > still needed to support those operations for the iterators that don't > have them natively. Thus, you're introducing two ways of doing the > same thing -- using itertools (works for all iterators) or using the > methods (only works for iterators that inherit from your base class). Understood. Any _technical_ reasons then why shouldn't all iterators inherit from it, or "duck typing is The Right Way" should be taken as axiom ? Why do we somehow _need_ itertools when we don't need sequencetools, mappingtools, etc ? What's special about the iterator protocol ? George From bjourne at gmail.com Tue Nov 14 10:21:24 2006 From: bjourne at gmail.com (=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=) Date: Tue, 14 Nov 2006 09:21:24 +0000 Subject: [Python-3000] Builtin iterator type In-Reply-To: <8E8031E5-BCE9-4FBF-BA9D-487AA3316E31@mac.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <8E8031E5-BCE9-4FBF-BA9D-487AA3316E31@mac.com> Message-ID: <740c3aec0611140121t6fac8dfcm4a58f3c56ae9daef@mail.gmail.com> On 11/14/06, Ronald Oussoren wrote: > On 14 Nov 2006, at 4:26 AM, George Sakkis wrote: > > I honestly fail to understand your current objections. Is my analogy > > with dictmixin flawed ? Would anything change if I named it > > "itermixin" instead of iter or Iter ? I'm ok with the idea being > > rejected, but at least I'd like to understand the reasons. > > You basically propose to extend the iterator interface from 1 method > to len(dir(itertools)) methods without any clear benefits. Saving a > single import line is not a clear benefit. But why is both the dict and list protocol so fat then? Is it hard to create your own dict or list-derived types in Python? -- mvh Bj?rn From fredrik at pythonware.com Tue Nov 14 10:59:55 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 14 Nov 2006 10:59:55 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: <740c3aec0611140121t6fac8dfcm4a58f3c56ae9daef@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <8E8031E5-BCE9-4FBF-BA9D-487AA3316E31@mac.com> <740c3aec0611140121t6fac8dfcm4a58f3c56ae9daef@mail.gmail.com> Message-ID: BJ?rn Lindqvist wrote: > But why is both the dict and list protocol so fat then? Is it hard to > create your own dict or list-derived types in Python? don't confuse things like lists and dictionaries with things like sequences and mappings. iterators and iterables belong to the second category. From fredrik at pythonware.com Tue Nov 14 13:36:08 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 14 Nov 2006 13:36:08 +0100 Subject: [Python-3000] Builtin iterator type References: <06Nov13.145133pst."58648"@synergy1.parc.xerox.com> <20061113231538.20948.713332044.divmod.quotient.26448@ohm> Message-ID: Jean-Paul Calderone wrote: > Anyway, the proposal was in no way suggesting moving away from duck > typing. The new iter was to be a _wrapper_ around any existing > iterator, not a base class for iterators. the OP seems to think otherwise: The key points are: /.../ 3. Make this type a base for all other (existing and future) iterator types, e.g. xrange and generator. From ark-mlist at att.net Tue Nov 14 16:31:37 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Tue, 14 Nov 2006 10:31:37 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <06Nov13.145133pst."58648"@synergy1.parc.xerox.com> Message-ID: <002801c70801$fd33d310$6402a8c0@arkdesktop> > Duck typing is a seriously bad idea Why? From ncoghlan at gmail.com Tue Nov 14 16:30:15 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Nov 2006 01:30:15 +1000 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611132303y33ecee81k8d7b2093d4b6806d@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <91ad5bf80611132303y33ecee81k8d7b2093d4b6806d@mail.gmail.com> Message-ID: <4559E107.9090207@gmail.com> George Sakkis wrote: > Understood. Any _technical_ reasons then why shouldn't all iterators > inherit from it, or "duck typing is The Right Way" should be taken as > an axiom ? Why do we somehow _need_ itertools when we don't need > sequencetools, mappingtools, etc ? What's so special about the > iterator protocol ? I have sympathy for the idea of syntactic sugar for working with iterators, I really do (just read the thread I started about iterator slicing early last year [1]). However, since that time we've moved on to the idea of returning richer view objects rather than bare iterators for some of the methods being changed in Py3k to return something other than a list (such as dict.keys() and friends). The underlying realisation behind that change is that an iterable is free to implement whichever aspects of the sequence and mapping interfaces it wants to, depending on what makes sense for the particular iterator. By taking this route the basic iterator protocol is kept simple with minimal assumptions about the underlying data storage (which is exactly what leads to the protocol's wide applicability), while the richer mapping and sequence interfaces are supported as appropriate in order to make the objects easier to work with. The difficult with pulling views out into a generic mixin class such as the one you propose is that they are typically strongly coupled to their associated concrete container class. Arbitrary iterators don't make strong enough guarantees to allow that to work without suprises. For example, it would be somewhat surprising if (a, b, c), (x, y, z) = it[:3], it[:3] puts different results into a & x, but that is exactly what happens if 'it' is a sliceable iterator like the one you suggest. This didn't bother me last year, but I've since grown to dislike it (primarily due to the realisation mentioned above that returning view objects where appropriate is likely to be a better answer, even if it lacks "one idiom to rule them all" status). Regards, Nick. [1] http://mail.python.org/pipermail/python-dev/2005-January/051257.html -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From george.sakkis at gmail.com Tue Nov 14 16:29:42 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Tue, 14 Nov 2006 10:29:42 -0500 Subject: [Python-3000] Python-3000 Digest, Vol 9, Issue 27 In-Reply-To: References: Message-ID: <91ad5bf80611140729o55570666gb8e07c761487c0ce@mail.gmail.com> On 11/14/06, Fredrik Lundh wrote: > BJ?rn Lindqvist wrote: > > > But why is both the dict and list protocol so fat then? Is it hard to > > create your own dict or list-derived types in Python? > > don't confuse things like lists and dictionaries with things like > sequences and mappings. iterators and iterables belong to the second > category. This doesn't answer my last question: why do we need itertools when we can live without sequencetools, mappingtools, fileliketools, etc. ? George From gsakkis at rutgers.edu Tue Nov 14 16:26:03 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Tue, 14 Nov 2006 10:26:03 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <20061114144840.GA24358@panix.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <20061114005252.GA9730@seldon> <91ad5bf80611132019m54de49d2t2ceefbfe181d2aac@mail.gmail.com> <20061114144840.GA24358@panix.com> Message-ID: <91ad5bf80611140726x5ace4eefpe7ae481600417c56@mail.gmail.com> On 11/14/06, Aahz wrote: > [off-list to avoid more CLUTTER] Sorry, I don't consider a well-intentioned discussion to understand how things work at the design level of python CLUTTER. > On Mon, Nov 13, 2006, George Sakkis wrote: > > > > Excuse my ignorance, but what's a "self-iter" ? > > Files # iterate over the first 10 lines of two files f1, f2 = map(Iter, (open('foo.txt'), open('bar.txt')) for line in f1[:10] + f2[:10]: print line, Pretty pythonic if you ask me. George From fredrik at pythonware.com Tue Nov 14 16:54:10 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 14 Nov 2006 16:54:10 +0100 Subject: [Python-3000] Python-3000 Digest, Vol 9, Issue 27 References: <91ad5bf80611140729o55570666gb8e07c761487c0ce@mail.gmail.com> Message-ID: George Sakkis wrote: >> don't confuse things like lists and dictionaries with things like >> sequences and mappings. iterators and iterables belong to the second >> category. > > This doesn't answer my last question: why do we need itertools when we > can live without sequencetools, mappingtools, fileliketools, etc. ? try listing the sequence and mapping and file tools that Python actually provides, and look at how they're used in contemporary Python code. do you see a pattern ? From ncoghlan at gmail.com Tue Nov 14 17:01:51 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Nov 2006 02:01:51 +1000 Subject: [Python-3000] Python-3000 Digest, Vol 9, Issue 27 In-Reply-To: <91ad5bf80611140729o55570666gb8e07c761487c0ce@mail.gmail.com> References: <91ad5bf80611140729o55570666gb8e07c761487c0ce@mail.gmail.com> Message-ID: <4559E86F.4010805@gmail.com> George Sakkis wrote: > On 11/14/06, Fredrik Lundh wrote: > >> BJ?rn Lindqvist wrote: >> >>> But why is both the dict and list protocol so fat then? Is it hard to >>> create your own dict or list-derived types in Python? >> don't confuse things like lists and dictionaries with things like >> sequences and mappings. iterators and iterables belong to the second >> category. > > This doesn't answer my last question: why do we need itertools when we > can live without sequencetools, mappingtools, fileliketools, etc. ? Because the iterator protocol is simple enough to be easily composable - there are a wide variety of things that can be done with an object when all you know about it is that it is some form of iterable. The more assumptions you have to start making about the iterable, the less widely applicable the resulting operation will be. And I think the number one reason we don't see a compelling need for the extra tool libraries you describe is the fact that sequences, mappings and files are themselves all iterables. That said, there are actually a number of modules for working with sequences in the standard library, like bisect and heapq. They just aren't lumped into one place the way itertools and functools are. Having a rich method API vs having a narrow method API and duck-typed support functions is a design trade-off. In the case of sequences and mappings, the trade-off went towards a richer API because the de facto reference implementations were the builtin dict and list classes (which is why DictMixin and ListMixin are so useful when implementing your own containers). In the case of iterables and iterators, the trade-off went towards the narrow API so that the interface could be used in a wide variety of situations (lines in a file, records in a database, characters in a string, bytes from a serial port, frames in a bowling game, active players in a MMORPG, etc, etc, etc). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From george.sakkis at gmail.com Tue Nov 14 17:49:35 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Tue, 14 Nov 2006 11:49:35 -0500 Subject: [Python-3000] Builtin iterator type Message-ID: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> On 11/14/06, Nick Coghlan wrote: > George Sakkis wrote: > > On 11/14/06, Fredrik Lundh wrote: > > > >> BJ?rn Lindqvist wrote: > >> > >>> But why is both the dict and list protocol so fat then? Is it hard to > >>> create your own dict or list-derived types in Python? > >> don't confuse things like lists and dictionaries with things like > >> sequences and mappings. iterators and iterables belong to the second > >> category. > > > > This doesn't answer my last question: why do we need itertools when we > > can live without sequencetools, mappingtools, fileliketools, etc. ? > > Because the iterator protocol is simple enough to be easily composable - there > are a wide variety of things that can be done with an object when all you know > about it is that it is some form of iterable. The more assumptions you have to > start making about the iterable, the less widely applicable the resulting > operation will be. > > And I think the number one reason we don't see a compelling need for the extra > tool libraries you describe is the fact that sequences, mappings and files are > themselves all iterables. > > That said, there are actually a number of modules for working with sequences > in the standard library, like bisect and heapq. They just aren't lumped into > one place the way itertools and functools are. > > Having a rich method API vs having a narrow method API and duck-typed support > functions is a design trade-off. In the case of sequences and mappings, the > trade-off went towards a richer API because the de facto reference > implementations were the builtin dict and list classes (which is why DictMixin > and ListMixin are so useful when implementing your own containers). In the > case of iterables and iterators, the trade-off went towards the narrow API so > that the interface could be used in a wide variety of situations (lines in a > file, records in a database, characters in a string, bytes from a serial port, > frames in a bowling game, active players in a MMORPG, etc, etc, etc). Given the overly negative reaction to a base iterator type, I withdraw the part of my proposal that suggests this type as the base of all iterators. Instead I propose a builtin Iter (or even better iter, if there is no objection to change iter's current behavior) as an OO replacement of the (functional) itertools API. In other words, instead of: from itertools import chain, islice, groupby for k,sub in groupby(chain(islice(it1, 1, None), islice(it2, 5)), key=str.lower): print k, list(sub) you will write: for k,sub in (Iter(it1)[1:] + Iter(it2)[:5]).groupby(str.lower): print k, list(sub) it1, it2 can be arbitrary iterators, no restrictions imposed. Iter() (or iter()) will just return a thin wrapper around them to provide itertools functionality in a less verbose, more pythonic, way. We can decide on the exact API of this wrapper later, but for now I'd like to get a general impression of its acceptance chances. Thoughts ? George From fredrik at pythonware.com Tue Nov 14 18:00:48 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 14 Nov 2006 18:00:48 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> Message-ID: George Sakkis wrote: > Given the overly negative reaction to a base iterator type, I withdraw > the part of my proposal that suggests this type as the base of all > iterators. Instead I propose a builtin Iter (or even better iter, if > there is no objection to change iter's current behavior) as an OO > replacement of the (functional) itertools API. In other words, instead > of: > > from itertools import chain, islice, groupby > for k,sub in groupby(chain(islice(it1, 1, None), islice(it2, 5)), > key=str.lower): > print k, list(sub) > > you will write: > > for k,sub in (Iter(it1)[1:] + Iter(it2)[:5]).groupby(str.lower): > print k, list(sub) Uhuh. And why would this more important than adding, say, Array, AtExit, Bz2, Cgi, CgiTB, Cmath, Code, Codecs, Collections, Copy, Csv, DateTime, Decimal, ElementTree, ErrNo, Ftplib, Gc, GetOpt, GetText, Glob, Gzip, HashLib, HMac, Inspect, Locale, Logging, Marshal, Math, Mmap, Operator, Os, OsPath, Pickle, Platform, Random, Re, Select, ShUtil, SmtpLib, Socket, Struct, SubProcess, Sys, TempFile, Threading, Time, Traceback, UnicodeData, UrlLib, UrlLib2, UrlParse, Warnings, WeakRef, XmlDom, XmlSax, ZipFile, or Zlib builtins? Most of these are definitely used a lot more often than itertools in typical Python programs... From jcarlson at uci.edu Tue Nov 14 18:17:17 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 14 Nov 2006 09:17:17 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> Message-ID: <20061114090956.82EB.JCARLSON@uci.edu> "George Sakkis" wrote: > > On 11/14/06, Nick Coghlan wrote: > > > George Sakkis wrote: > > > On 11/14/06, Fredrik Lundh wrote: > > > > > >> BJ?rn Lindqvist wrote: > > >> > > >>> But why is both the dict and list protocol so fat then? Is it hard to > > >>> create your own dict or list-derived types in Python? > > >> don't confuse things like lists and dictionaries with things like > > >> sequences and mappings. iterators and iterables belong to the second > > >> category. > > > > > > This doesn't answer my last question: why do we need itertools when we > > > can live without sequencetools, mappingtools, fileliketools, etc. ? > > > > Because the iterator protocol is simple enough to be easily composable - there > > are a wide variety of things that can be done with an object when all you know > > about it is that it is some form of iterable. The more assumptions you have to > > start making about the iterable, the less widely applicable the resulting > > operation will be. > > > > And I think the number one reason we don't see a compelling need for the extra > > tool libraries you describe is the fact that sequences, mappings and files are > > themselves all iterables. > > > > That said, there are actually a number of modules for working with sequences > > in the standard library, like bisect and heapq. They just aren't lumped into > > one place the way itertools and functools are. > > > > Having a rich method API vs having a narrow method API and duck-typed support > > functions is a design trade-off. In the case of sequences and mappings, the > > trade-off went towards a richer API because the de facto reference > > implementations were the builtin dict and list classes (which is why DictMixin > > and ListMixin are so useful when implementing your own containers). In the > > case of iterables and iterators, the trade-off went towards the narrow API so > > that the interface could be used in a wide variety of situations (lines in a > > file, records in a database, characters in a string, bytes from a serial port, > > frames in a bowling game, active players in a MMORPG, etc, etc, etc). > > Given the overly negative reaction to a base iterator type, I withdraw > the part of my proposal that suggests this type as the base of all > iterators. Instead I propose a builtin Iter (or even better iter, if > there is no objection to change iter's current behavior) as an OO > replacement of the (functional) itertools API. In other words, instead > of: -1. There is nothing wrong with a functional approach. If you dislike the verbosity of the names of the operations, and/or the arguments they take, you are free to rename them: from itertools import chain as CH ... or write wrappers: def IS(it, start): return islice(it, start, None) You can even have your itertools automatically inserted into __builtins__ on startup with a small manipulation of site.py . Heck, if _you_ want them, _you_ are free to have your Iter object inserted into the __builtins__ namespace by site.py . But due to the overwhelming -1s from _everyone_, _including Guido_, your odds of getting anything like Iter into mainline Python in the next 18 months (when Python 2.6 is expected to be released), is very low. - Josiah From steven.bethard at gmail.com Tue Nov 14 18:16:59 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Tue, 14 Nov 2006 10:16:59 -0700 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> Message-ID: On 11/14/06, George Sakkis wrote: > Given the overly negative reaction to a base iterator type, I withdraw > the part of my proposal that suggests this type as the base of all > iterators. Instead I propose a builtin Iter (or even better iter, if > there is no objection to change iter's current behavior) as an OO > replacement of the (functional) itertools API. [snip] > for k,sub in (Iter(it1)[1:] + Iter(it2)[:5]).groupby(str.lower): > print k, list(sub) Why don't you distribute this class as a third-party library for a while, and see if people use it. If it turns out that everyone loves the functionality that it provides, then you can lobby for its inclusion in the standard library. Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From george.sakkis at gmail.com Tue Nov 14 16:49:54 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Tue, 14 Nov 2006 10:49:54 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <4559E107.9090207@gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <91ad5bf80611132303y33ecee81k8d7b2093d4b6806d@mail.gmail.com> <4559E107.9090207@gmail.com> Message-ID: <91ad5bf80611140749t59fe2067ra5452ac6505ea97b@mail.gmail.com> On 11/14/06, Nick Coghlan wrote: > George Sakkis wrote: > > Understood. Any _technical_ reasons then why shouldn't all iterators > > inherit from it, or "duck typing is The Right Way" should be taken as > > an axiom ? Why do we somehow _need_ itertools when we don't need > > sequencetools, mappingtools, etc ? What's so special about the > > iterator protocol ? > > I have sympathy for the idea of syntactic sugar for working with iterators, I > really do (just read the thread I started about iterator slicing early last > year [1]). > > However, since that time we've moved on to the idea of returning richer view > objects rather than bare iterators for some of the methods being changed in > Py3k to return something other than a list (such as dict.keys() and friends). > > The underlying realisation behind that change is that an iterable is free to > implement whichever aspects of the sequence and mapping interfaces it wants > to, depending on what makes sense for the particular iterator. > > By taking this route the basic iterator protocol is kept simple with minimal > assumptions about the underlying data storage (which is exactly what leads to > the protocol's wide applicability), while the richer mapping and sequence > interfaces are supported as appropriate in order to make the objects easier to > work with. > > The difficult with pulling views out into a generic mixin class such as the > one you propose is that they are typically strongly coupled to their > associated concrete container class. Arbitrary iterators don't make strong > enough guarantees to allow that to work without suprises. > > For example, it would be somewhat surprising if (a, b, c), (x, y, z) = it[:3], > it[:3] puts different results into a & x, but that is exactly what happens if > 'it' is a sliceable iterator like the one you suggest. This didn't bother me > last year, but I've since grown to dislike it (primarily due to the > realisation mentioned above that returning view objects where appropriate is > likely to be a better answer, even if it lacks "one idiom to rule them all" > status). Nick, thank you for the most useful and insightful reply so far. Maybe you are a year ahead of me because the example you showed doesn't bother me either, the same way I'm not bothered by, say, infinite iterators and the risk they introduce when someone writes list(iterator). I'm also not bothered at all with numpy arrays returning views instead of copies. Is it really part of the informal sequence protocol that slicing must return a copy ? That would be new to me. I haven't read the thread you mention about rich views, I probably should do that, but from the sound of it, I think I'll like it. Regards, George From george.sakkis at gmail.com Tue Nov 14 18:55:06 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Tue, 14 Nov 2006 12:55:06 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <20061114090956.82EB.JCARLSON@uci.edu> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> Message-ID: <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> On 11/14/06, Josiah Carlson wrote: > > "George Sakkis" wrote: > > > > On 11/14/06, Nick Coghlan wrote: > > > > > George Sakkis wrote: > > > > On 11/14/06, Fredrik Lundh wrote: > > > > > > > >> BJ?rn Lindqvist wrote: > > > >> > > > >>> But why is both the dict and list protocol so fat then? Is it hard to > > > >>> create your own dict or list-derived types in Python? > > > >> don't confuse things like lists and dictionaries with things like > > > >> sequences and mappings. iterators and iterables belong to the second > > > >> category. > > > > > > > > This doesn't answer my last question: why do we need itertools when we > > > > can live without sequencetools, mappingtools, fileliketools, etc. ? > > > > > > Because the iterator protocol is simple enough to be easily composable - there > > > are a wide variety of things that can be done with an object when all you know > > > about it is that it is some form of iterable. The more assumptions you have to > > > start making about the iterable, the less widely applicable the resulting > > > operation will be. > > > > > > And I think the number one reason we don't see a compelling need for the extra > > > tool libraries you describe is the fact that sequences, mappings and files are > > > themselves all iterables. > > > > > > That said, there are actually a number of modules for working with sequences > > > in the standard library, like bisect and heapq. They just aren't lumped into > > > one place the way itertools and functools are. > > > > > > Having a rich method API vs having a narrow method API and duck-typed support > > > functions is a design trade-off. In the case of sequences and mappings, the > > > trade-off went towards a richer API because the de facto reference > > > implementations were the builtin dict and list classes (which is why DictMixin > > > and ListMixin are so useful when implementing your own containers). In the > > > case of iterables and iterators, the trade-off went towards the narrow API so > > > that the interface could be used in a wide variety of situations (lines in a > > > file, records in a database, characters in a string, bytes from a serial port, > > > frames in a bowling game, active players in a MMORPG, etc, etc, etc). > > > > Given the overly negative reaction to a base iterator type, I withdraw > > the part of my proposal that suggests this type as the base of all > > iterators. Instead I propose a builtin Iter (or even better iter, if > > there is no objection to change iter's current behavior) as an OO > > replacement of the (functional) itertools API. In other words, instead > > of: > > -1. There is nothing wrong with a functional approach. If you dislike > the verbosity of the names of the operations, and/or the arguments they > take, you are free to rename them: > from itertools import chain as CH > ... or write wrappers: > def IS(it, start): > return islice(it, start, None) > > You can even have your itertools automatically inserted into > __builtins__ on startup with a small manipulation of site.py . > > Heck, if _you_ want them, _you_ are free to have your Iter object > inserted into the __builtins__ namespace by site.py . But due to the > overwhelming -1s from _everyone_, _including Guido_, your odds of > getting anything like Iter into mainline Python in the next 18 months > (when Python 2.6 is expected to be released), is very low. This is the python-3000 list; I don't aim at Python 2.x. Given that I propose to *replace* itertools, not add an extra builtin (having them both would be redundant and violates the one obvious way to do it principle), it wouldn't be backwards compatible. As for the -1s, so far most of them seem to oppose the inheritance idea as a worse option than duck typing; let's see if removing this part changes anything. As for your other arguments about renaming the itertools functions to smaller names, let me remind you that in python syntax sugar *does* matter. If I am not mistaken, the list comprehension syntax will not be removed in python-3K, although it is completely redundant with generator comprehensions. Heck, writing list(gencomp) is only 4 characters longer than [gencomp]. My suggested proposal does much better, both in saving keystrokes and readability. George From jimjjewett at gmail.com Tue Nov 14 19:06:12 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 14 Nov 2006 13:06:12 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> Message-ID: On 11/14/06, George Sakkis wrote: > This is the python-3000 list; I don't aim at Python 2.x. Given that I > propose to *replace* itertools, not add an extra builtin (having them > both would be redundant and violates the one obvious way to do it > principle), it wouldn't be backwards compatible. As for the -1s, so > far most of them seem to oppose the inheritance idea as a worse option > than duck typing; let's see if removing this part changes anything. Yes, and at first glance, an IterMixin would be nice. But only at first glance. (1) Things that are already iterators won't implement the entire newly expanded iteration API. I don't want people to start mechanically replacing for var in seq with for var in Iter(seq) just because they can't remember whether or not it matters. (2) Some of the functionality in itertools (particularly islice) is awkward enough that maybe it should be hidden in a library. That way people are less likely to stumble on it without seeing the documentation. (3) IterMixin would be the only mixin exposed as a builtin -- the other mixins must be imported. The builtin type factories are typically the stripped down versions. -jJ From gsakkis at rutgers.edu Tue Nov 14 19:06:33 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Tue, 14 Nov 2006 13:06:33 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> Message-ID: <91ad5bf80611141006k431695e4l1d7752cb1f027aea@mail.gmail.com> On 11/14/06, Fredrik Lundh wrote: > Uhuh. And why would this more important than adding, say, Array, > AtExit, Bz2, Cgi, CgiTB, Cmath, Code, Codecs, Collections, Copy, Csv, > DateTime, Decimal, ElementTree, ErrNo, Ftplib, Gc, GetOpt, GetText, > Glob, Gzip, HashLib, HMac, Inspect, Locale, Logging, Marshal, Math, > Mmap, Operator, Os, OsPath, Pickle, Platform, Random, Re, Select, > ShUtil, SmtpLib, Socket, Struct, SubProcess, Sys, TempFile, Threading, > Time, Traceback, UnicodeData, UrlLib, UrlLib2, UrlParse, Warnings, > WeakRef, XmlDom, XmlSax, ZipFile, or Zlib builtins? Most of these are > definitely used a lot more often than itertools in typical Python > programs... > > I understand you are exaggerating (can't believe you are seriously claiming that cmath or traceback are more frequently used than itertools), but if your objection is on adding yet another builtin, what would be the objection to boosting up the existing iter() to provide this extra functionality ? This might even be backwards compatible (but even if it's not, that's not a main concern for py-3k). George From george.sakkis at gmail.com Tue Nov 14 19:19:17 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Tue, 14 Nov 2006 13:19:17 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> Message-ID: <91ad5bf80611141019i316747b1t45222bf50ecd0446@mail.gmail.com> On 11/14/06, Jim Jewett wrote: > Yes, and at first glance, an IterMixin would be nice. But only at first glance. > > (1) Things that are already iterators won't implement the entire > newly expanded iteration API. I don't want people to start > mechanically replacing > > for var in seq > > with > > for var in Iter(seq) > > just because they can't remember whether or not it matters. Short answer: it doesn't. Only if you actually use the new API it matters, e.g. for var in Iter(it1) + Iter(it2): > (2) Some of the functionality in itertools (particularly islice) is > awkward enough that maybe it should be hidden in a library. That way > people are less likely to stumble on it without seeing the > documentation. You can argue the same for a gazillion other python features. There's no way to know that list slicing returns a copy and not a view, or that numpy arrays do the opposite, without reading the docs. > (3) IterMixin would be the only mixin exposed as a builtin -- the > other mixins must be imported. The builtin type factories are > typically the stripped down versions. That problem is solved if we boost up the existing iter() builtin. George From fdrake at acm.org Tue Nov 14 19:53:18 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue, 14 Nov 2006 13:53:18 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611141006k431695e4l1d7752cb1f027aea@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141006k431695e4l1d7752cb1f027aea@mail.gmail.com> Message-ID: <200611141353.18690.fdrake@acm.org> On Tuesday 14 November 2006 13:06, George Sakkis wrote: > I understand you are exaggerating (can't believe you are seriously > claiming that cmath or traceback are more frequently used than > itertools), I certainly use traceback far more than itertools. I use traceback occaissionally, but I've never actually had reason to use itertools at all. > but if your objection is on adding yet another builtin, > what would be the objection to boosting up the existing iter() to > provide this extra functionality ? This might even be backwards > compatible (but even if it's not, that's not a main concern for > py-3k). The real issue seems to be that there's no benefit. Iterators are nice because they're composable; that doesn't make the compositions part of the iterator, though. -Fred -- Fred L. Drake, Jr. From george.sakkis at gmail.com Tue Nov 14 20:32:22 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Tue, 14 Nov 2006 14:32:22 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <200611141353.18690.fdrake@acm.org> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141006k431695e4l1d7752cb1f027aea@mail.gmail.com> <200611141353.18690.fdrake@acm.org> Message-ID: <91ad5bf80611141132v3cac4035nfa92b52f394ebd1a@mail.gmail.com> On 11/14/06, Fred L. Drake, Jr. wrote: > On Tuesday 14 November 2006 13:06, George Sakkis wrote: > > I understand you are exaggerating (can't believe you are seriously > > claiming that cmath or traceback are more frequently used than > > itertools), > > I certainly use traceback far more than itertools. I use traceback > occaissionally, but I've never actually had reason to use itertools at all. I believe you, but I doubt your usage patterns are close to the average python user. I am sure cmath is invaluable to some people too. There's also a chicken and egg problem; the reason itertools are not used as often as they could is exactly the overheard of importing a module and use a verbose function, rather than having them for free as methods of a builtin object. If instead of for k,v in some_dict.iteritems(): I had to do something like from dictutils import iteritems for k,v in iteritems(some_dict): I'd probably woudn't bother and just use some_dict.items(). > > but if your objection is on adding yet another builtin, > > what would be the objection to boosting up the existing iter() to > > provide this extra functionality ? This might even be backwards > > compatible (but even if it's not, that's not a main concern for > > py-3k). > > The real issue seems to be that there's no benefit. Iterators are nice > because they're composable; that doesn't make the compositions part of the > iterator, though. Seems like a classic case for OOP to me: combine state (iterables) with behavior (iter) in handly little packets (objects). George From gsakkis at rutgers.edu Tue Nov 14 20:44:06 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Tue, 14 Nov 2006 14:44:06 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611141019i316747b1t45222bf50ecd0446@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141019i316747b1t45222bf50ecd0446@mail.gmail.com> Message-ID: <91ad5bf80611141144l58569e26wf5633a849e07c9ad@mail.gmail.com> On 11/14/06, George Sakkis wrote: > On 11/14/06, Jim Jewett wrote: > > (3) IterMixin would be the only mixin exposed as a builtin -- the > > other mixins must be imported. The builtin type factories are > > typically the stripped down versions. > > That problem is solved if we boost up the existing iter() builtin. > > George And for those objecting to touching the existing iter() or bloating the builtin namespace with yet another builtin, let me mention that we can get rid of *two* existing functions which for some reason were promoted to builtin status, although they conceptually belong in itertools: enumerate and reversed. George From guido at python.org Tue Nov 14 20:56:41 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 14 Nov 2006 11:56:41 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611141144l58569e26wf5633a849e07c9ad@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141019i316747b1t45222bf50ecd0446@mail.gmail.com> <91ad5bf80611141144l58569e26wf5633a849e07c9ad@mail.gmail.com> Message-ID: We can all save a lot of time by not continuing this conversation. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From sluggoster at gmail.com Tue Nov 14 22:38:54 2006 From: sluggoster at gmail.com (Mike Orr) Date: Tue, 14 Nov 2006 13:38:54 -0800 Subject: [Python-3000] Python-3000 Digest, Vol 9, Issue 27 In-Reply-To: <4559E86F.4010805@gmail.com> References: <91ad5bf80611140729o55570666gb8e07c761487c0ce@mail.gmail.com> <4559E86F.4010805@gmail.com> Message-ID: <6e9196d20611141338s706a0daao77e5833a22884716@mail.gmail.com> On 11/14/06, Nick Coghlan wrote: > Having a rich method API vs having a narrow method API and duck-typed support > functions is a design trade-off. In the case of sequences and mappings, the > trade-off went towards a richer API because the de facto reference > implementations were the builtin dict and list classes (which is why DictMixin > and ListMixin are so useful when implementing your own containers). In the > case of iterables and iterators, the trade-off went towards the narrow API so > that the interface could be used in a wide variety of situations (lines in a > file, records in a database, characters in a string, bytes from a serial port, > frames in a bowling game, active players in a MMORPG, etc, etc, etc). To augment Nick's excellent explanation, note that you can't restart an iterator or go back ("unyield" a value). There was a conscious decision to limit the iterator interface to make it as easy as possible for some unknown future object to implement an iterator. Now, anything with a compliant .next() method is an iterator. Nothing prevents a particular iterator from having reset or backstepping methods, but these are not required for all iterators. I sense this is the same reason people are opposed to your proposal. I'll not comment on the merits of having iter() return a rich Iterator object because I don't understand well enough what we might lose. The real problem is that iterator is an interface, and there's no formal way to express interfaces in Python; it's all in the documentation. That's why the relationship between dict and "mapping type" seems so nebulous, and also why an iterator looks like the same kind of object as a dict or list but it isn't. The tradeoff between rich objects + methods and minimal objects + functions is pervasive in Python. len(x) is a function because it applies to a wide variety of objects -- not all of them known yet -- rather than to a certain class hierarchy. One major success of the iterator protocol was when file objects became iterable. No more "while loop with break in the middle": now you can just do a simple "for line in file". If the file iterator were required to support "+" and your other rich methods, it may be difficult to implement. What if the file is tied to a socket-like stream instead of a disk file? When you need a mapping, the obvious answer is, "Use a dict!" A dict is an empty container with infrastructure for adding and managing items. When a file object needs to iterate likes in a file, it can't "use an iterator", it has to *be* an iterator. A generic iterator object doesn't know *how* to iterate over a file; that's what your class has to do. But in order to do that it has to implement the iterator interface... and we're right back where we started. A "listtools" package is not a bad idea, actually. A place to gather list-like objects and their functions so they aren't scattered throughout the library. "collections" almost does it, although it's defined a bit too narrowly for function libraries (but that could be changed). But this is a minor issue. The thing with "mappingtools" is, what would such a package contain? I've sometimes wondered why there's only one mapping type after so many years. But the real question is, what kind of "mapping type" functionality is there that a dict doesn't provide? None that I can think of. "ordered dict" and "default dict" can be handled by subclasses. The problem with .__getitem__ is, you can't tell whether an object is a sequence or a mapping. If it has .__getitem__, it's one or the other, but you don't know whether it accepts sequential integers starting from 0, or arbitrary keys. This is not really .__getitem__'s fault since there's only one [] operator to access both. It's the lack of interfaces again: the lack of a universal way to say "this is a sequence type". All the routine can do is document what it expects in arguments, and hope that the caller heeds it. As for itertools, I rarely use it. If so it's one or two functions at a time. And often I'm not actually using the functions, just studying the implementation so I can write a custom function that does what I need. I suppose it would be nice to chain iterators with "+", but I chain iterators so rarely it's no big deal. I suspect many other programmers are the same way. What I'd most like to see in itertools are the functions on the "Recipes" page. Why should everyone have to paste the code for no/quantify/flatten rather than having one central function for them? But I expect this will happen one one by one as certain recipes get popular. -- Mike Orr From mike.klaas at gmail.com Tue Nov 14 22:59:36 2006 From: mike.klaas at gmail.com (Mike Klaas) Date: Tue, 14 Nov 2006 13:59:36 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611141144l58569e26wf5633a849e07c9ad@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141019i316747b1t45222bf50ecd0446@mail.gmail.com> <91ad5bf80611141144l58569e26wf5633a849e07c9ad@mail.gmail.com> Message-ID: <3d2ce8cb0611141359s278c6d4cjd01e1aa6759e6c4@mail.gmail.com> On 11/14/06, George Sakkis wrote: > And for those objecting to touching the existing iter() or bloating > the builtin namespace with yet another builtin, let me mention that we > can get rid of *two* existing functions which for some reason were > promoted to builtin status, although they conceptually belong in > itertools: enumerate and reversed. Well, "conceptually" all itertools are not equal. enumerate and reversed (and let's not forget xrange) are among my most-used "itertools"--enumerate in particular is used is virtually every module. Some tools in itertools also belong to that category--izip in particular is used heavily in my code. Some of the heavily-used itertools are already being pseudo-promoted to builtins: zip, map, (and maybe? filter) are turning into izip, imap, and ifilter. I will admit that itertools are among the few modules that are "first-class" import citizens in my code, that is, I tend to import the symbols directly: from itertools import groupby, chain Another such module is collections. It is possible that once I'm more familiar with functools and contextlib that they will occupy this hallowed place. I don't see the problem of importing important language functionality. Most languages since c have required somthing similar. -Mike From sluggoster at gmail.com Tue Nov 14 23:11:48 2006 From: sluggoster at gmail.com (Mike Orr) Date: Tue, 14 Nov 2006 14:11:48 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <002801c70801$fd33d310$6402a8c0@arkdesktop> References: <002801c70801$fd33d310$6402a8c0@arkdesktop> Message-ID: <6e9196d20611141411x57a4d3f0tdbc81a18b9590889@mail.gmail.com> On 11/14/06, Andrew Koenig wrote: > > Duck typing is a seriously bad idea > > Why? And more importantly, are we all talking about the same thing when we say "duck typing"? Duck typing as I know it means accepting an argument that exhibits certain behaviors rather than being a certain type. It's widely considered to be one of Python's main features because it's flexible for unforeseen circumstances. Sometimes a better implementation is found that can't be expressed as a subclass. Sometimes an object can't be one class because it has to be another, yet it can still "behave like" the other object. -- Mike Orr From george.sakkis at gmail.com Tue Nov 14 23:28:36 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Tue, 14 Nov 2006 17:28:36 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <3d2ce8cb0611141359s278c6d4cjd01e1aa6759e6c4@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141019i316747b1t45222bf50ecd0446@mail.gmail.com> <91ad5bf80611141144l58569e26wf5633a849e07c9ad@mail.gmail.com> <3d2ce8cb0611141359s278c6d4cjd01e1aa6759e6c4@mail.gmail.com> Message-ID: <91ad5bf80611141428r6f9313edq48c95acacf174059@mail.gmail.com> On 11/14/06, Mike Klaas wrote: > On 11/14/06, George Sakkis wrote: > > > And for those objecting to touching the existing iter() or bloating > > the builtin namespace with yet another builtin, let me mention that we > > can get rid of *two* existing functions which for some reason were > > promoted to builtin status, although they conceptually belong in > > itertools: enumerate and reversed. > > Well, "conceptually" all itertools are not equal. enumerate and > reversed (and let's not forget xrange) are among my most-used > "itertools"--enumerate in particular is used is virtually every > module. Some tools in itertools also belong to that category--izip in > particular is used heavily in my code. Some of the heavily-used > itertools are already being pseudo-promoted to builtins: zip, map, > (and maybe? filter) are turning into izip, imap, and ifilter. > > I will admit that itertools are among the few modules that are > "first-class" import citizens in my code, that is, I tend to import > the symbols directly: > > from itertools import groupby, chain I do the same, and that's exactly my point. Iterators, generators, generator comprehensions are among the core python features. Their status is promoted steadily in every new version; the latest 2.5 is no exception. Itertools is not yet another obscure module; it includes functions that provide fundamental operations that apply on any iterable. > I don't see the problem of importing important language functionality. > Most languages since c have required somthing similar. > > -Mike Most languages since C are not as readable and elegant as Python. I think we all agree that list1[5:] + list2 is much better than the hypothetical list1.getslice(5,None).concatenate(list2) or from sequenceutils import concatenate, getslice concatenate(getslice(list1,5,None), list2) Unfortunately, people here seem to consider natural the latter when it comes to itertools. Oh well, whatever. George From mike.klaas at gmail.com Tue Nov 14 23:44:07 2006 From: mike.klaas at gmail.com (Mike Klaas) Date: Tue, 14 Nov 2006 14:44:07 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611141428r6f9313edq48c95acacf174059@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141019i316747b1t45222bf50ecd0446@mail.gmail.com> <91ad5bf80611141144l58569e26wf5633a849e07c9ad@mail.gmail.com> <3d2ce8cb0611141359s278c6d4cjd01e1aa6759e6c4@mail.gmail.com> <91ad5bf80611141428r6f9313edq48c95acacf174059@mail.gmail.com> Message-ID: <3d2ce8cb0611141444oe9284a9w74574e3d12e8c073@mail.gmail.com> On 11/14/06, George Sakkis wrote: > On 11/14/06, Mike Klaas wrote: > > I don't see the problem of importing important language functionality. > > Most languages since c have required somthing similar. > > Most languages since C are not as readable and elegant as Python. I > think we all agree that list1[5:] + list2 is much better than the > hypothetical > list1.getslice(5,None).concatenate(list2) > or > from sequenceutils import concatenate, getslice > concatenate(getslice(list1,5,None), list2) > > Unfortunately, people here seem to consider natural the latter when it > comes to itertools. Oh well, whatever. Two differences: - slicing an iterator is a rare activity; slicing a list is common - I often use .extend rather than += when appending iterables to lists Finally, your analogy between sequences and iterators is flawed. + does not work for sequences, but only for concrete types: >>> [8] + (9,) Traceback (most recent call last): File "", line 1, in TypeError: can only concatenate list (not "tuple") to list In python, only concrete types tend to implement syntatic operator support (another example: sets only support +, -, |, & with other sets, but their methods accept iterables). -Mike From george.sakkis at gmail.com Wed Nov 15 01:23:24 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Tue, 14 Nov 2006 19:23:24 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <3d2ce8cb0611141444oe9284a9w74574e3d12e8c073@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141019i316747b1t45222bf50ecd0446@mail.gmail.com> <91ad5bf80611141144l58569e26wf5633a849e07c9ad@mail.gmail.com> <3d2ce8cb0611141359s278c6d4cjd01e1aa6759e6c4@mail.gmail.com> <91ad5bf80611141428r6f9313edq48c95acacf174059@mail.gmail.com> <3d2ce8cb0611141444oe9284a9w74574e3d12e8c073@mail.gmail.com> Message-ID: <91ad5bf80611141623r6771b14bg5ea55efceb36e820@mail.gmail.com> On 11/14/06, Mike Klaas wrote: > On 11/14/06, George Sakkis wrote: > > On 11/14/06, Mike Klaas wrote: > > > > I don't see the problem of importing important language functionality. > > > Most languages since c have required somthing similar. > > > > Most languages since C are not as readable and elegant as Python. I > > think we all agree that list1[5:] + list2 is much better than the > > hypothetical > > list1.getslice(5,None).concatenate(list2) > > or > > from sequenceutils import concatenate, getslice > > concatenate(getslice(list1,5,None), list2) > > > > Unfortunately, people here seem to consider natural the latter when it > > comes to itertools. Oh well, whatever. > > Two differences: > - slicing an iterator is a rare activity; slicing a list is common Slicing an infinite iterator is *very* common, to which you'll probably reply that infinite iterators are rare. I guess neither of us has numbers to support his claim, so let's leave it at that. > - I often use .extend rather than += when appending iterables to lists > > Finally, your analogy between sequences and iterators is flawed. + > does not work for sequences, but only for concrete types: > > >>> [8] + (9,) > Traceback (most recent call last): > File "", line 1, in > TypeError: can only concatenate list (not "tuple") to list > > In python, only concrete types tend to implement syntatic operator > support (another example: sets only support +, -, |, & with other > sets, but their methods accept iterables). > > -Mike > The TypeError is not raised because the arguments are not of a concrete type (all python types are concrete; interfaces or protocols are informal descriptions) but because they're not the *same* type, and I don't object to this. This is not the case in what I propose: in the expression Iter(file('foo')) + Iter(some_string), both operands have the same type, the Iter wrapper that wraps an underlying iterator. George From greg.ewing at canterbury.ac.nz Wed Nov 15 03:34:06 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 15 Nov 2006 15:34:06 +1300 Subject: [Python-3000] Builtin iterator type In-Reply-To: <06Nov13.145133pst.58648@synergy1.parc.xerox.com> References: <06Nov13.145133pst.58648@synergy1.parc.xerox.com> Message-ID: <455A7C9E.2050904@canterbury.ac.nz> Bill Janssen wrote: > Duck typing is a seriously bad idea, forced on Python by the now > obsolete split between built-in types and user-defined types. Non-duck typing is a seriously bad idea, forced on some other languages by static typing. Python is mercifully free of such constraints. -- Greg From greg.ewing at canterbury.ac.nz Wed Nov 15 03:34:57 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 15 Nov 2006 15:34:57 +1300 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> Message-ID: <455A7CD1.5090503@canterbury.ac.nz> George Sakkis wrote: > I think I do, though I can't tell the same about the reasons of your > objections to it. Perhaps I can fill in some of the things that Guido is not explicitly saying. You've shown that it *can* be done, but you haven't provided a compelling reason why it *should* be done. The main benefit seems to be that itertools operations could then be done using method calls instead of function calls. But that's not automatically better than what we have now. Python does not follow the "everything must be a method of something" ideology found in some other languages such as Java and Ruby. The itertools functions have generic implementations that work with any iterator. They don't need access to the iterator's internals, or need to be dynamically dispatched for any reason, so there is no obvious benefit to making them methods instead of stand-alone functions. The one technical benefit appears to be the ability to use operators instead of named functions. But considering how infrequently itertools operations are used, having to spell them out doesn't seem like a great hardship, so this argument is weak. All other things being equal, it might hold sway, but other things are not equal. You claim that duck typing would not be inhibited. While technically this is true, in practice it would mean that any iterator that didn't inherit from this base class would be a second-class citizen, unable to be used by code that expected it to have all the "standard" itertools methods. Finally, consider that there are infinitely many possible functions of the kind that are found in the itertools module. As stand-alone functions, they all have equal status -- new ones can be added and used in the same way by any code on any iterator. Some good arguments would have to be put forward as to why some particular ones should be picked out and elevated to the status of methods. -- Greg From greg.ewing at canterbury.ac.nz Wed Nov 15 03:35:13 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 15 Nov 2006 15:35:13 +1300 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611141132v3cac4035nfa92b52f394ebd1a@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141006k431695e4l1d7752cb1f027aea@mail.gmail.com> <200611141353.18690.fdrake@acm.org> <91ad5bf80611141132v3cac4035nfa92b52f394ebd1a@mail.gmail.com> Message-ID: <455A7CE1.4020005@canterbury.ac.nz> George Sakkis wrote: > Seems like a classic case for OOP to me: combine state (iterables) > with behavior (iter) in handly little packets (objects). Except that it's not really behaviour. Chaining, e.g., is something you do *to* iterators, not something that the iterators themselves do. -- Greg From greg.ewing at canterbury.ac.nz Wed Nov 15 03:35:21 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 15 Nov 2006 15:35:21 +1300 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611141019i316747b1t45222bf50ecd0446@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141019i316747b1t45222bf50ecd0446@mail.gmail.com> Message-ID: <455A7CE9.5030105@canterbury.ac.nz> George Sakkis wrote: > Short answer: it doesn't. Only if you actually use the new API it matters, e.g. > > for var in Iter(it1) + Iter(it2): Which doesn't look any more readable to me than for var in chain(it1, it2): In fact, it looks considerably *less* readable. -- Greg From greg.ewing at canterbury.ac.nz Wed Nov 15 03:35:26 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 15 Nov 2006 15:35:26 +1300 Subject: [Python-3000] Builtin iterator type In-Reply-To: <740c3aec0611140121t6fac8dfcm4a58f3c56ae9daef@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <8E8031E5-BCE9-4FBF-BA9D-487AA3316E31@mac.com> <740c3aec0611140121t6fac8dfcm4a58f3c56ae9daef@mail.gmail.com> Message-ID: <455A7CEE.9000701@canterbury.ac.nz> BJ?rn Lindqvist wrote: > But why is both the dict and list protocol so fat then? Because there are a variety of things you need to be able to do to sequences and mappings, and the implementations of them are closely tied to the internals of the type concerned. If there were a substantial number of commonly used operations on them that could be implemented efficiently in a generic way, then we probably would have listtools and dicttools modules (or more properly sequencetools and mappingtools). That doesn't seem to have happened, though. -- Greg From greg.ewing at canterbury.ac.nz Wed Nov 15 03:35:33 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 15 Nov 2006 15:35:33 +1300 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611141006k431695e4l1d7752cb1f027aea@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141006k431695e4l1d7752cb1f027aea@mail.gmail.com> Message-ID: <455A7CF5.6000409@canterbury.ac.nz> George Sakkis wrote: > I understand you are exaggerating (can't believe you are seriously > claiming that cmath or traceback are more frequently used than > itertools) I don't think he's exaggerating much -- I'm sure many of those modules are used a lot more than itertools. I don't know about generally, but personally I *do* use traceback more often than itertools. And I can easily see someone who's into number crunching using cmath a *lot* more than itertools. -- Greg From janssen at parc.com Wed Nov 15 04:46:39 2006 From: janssen at parc.com (Bill Janssen) Date: Tue, 14 Nov 2006 19:46:39 PST Subject: [Python-3000] Python-3000 Digest, Vol 9, Issue 27 In-Reply-To: Your message of "Tue, 14 Nov 2006 13:38:54 PST." <6e9196d20611141338s706a0daao77e5833a22884716@mail.gmail.com> Message-ID: <06Nov14.194639pst."58648"@synergy1.parc.xerox.com> > The real > problem is that iterator is an interface, and there's no formal way to > express interfaces in Python; it's all in the documentation. > ... > The problem with .__getitem__ is, you can't tell whether an object is > a sequence or a mapping. If it has .__getitem__, it's one or the > other, but you don't know whether it accepts sequential integers > starting from 0, or arbitrary keys. Actually, Mike, there is a formal way to express interfaces in Python. We just don't use it. The mechanism is to define all interfaces clearly as base types, and use the type mechanism to indicate "typeness" instead of the pervasive use of duck-typing. Then, to indicate that a type "implements" the "foo" interface, just inherit from "foo". But wait, there's more! Since Python usefully supports multiple inheritance in way that Java doesn't, Python "interface" classes can actually provide useful generic implementations of functionality, where possible, or raise NotImplemented where not possible. And that's not all! You could then see if a value implements a particular interface with isinstance(), instead of having to check for the presence of "__getitem__", and wondering if the implementor's "__getitem__" is the same "__getitem__" you were thinking of. Suddenly you can do non-fragile reflective programming. Bill From janssen at parc.com Wed Nov 15 04:53:36 2006 From: janssen at parc.com (Bill Janssen) Date: Tue, 14 Nov 2006 19:53:36 PST Subject: [Python-3000] Builtin iterator type In-Reply-To: Your message of "Tue, 14 Nov 2006 18:34:06 PST." <455A7C9E.2050904@canterbury.ac.nz> Message-ID: <06Nov14.195344pst."58648"@synergy1.parc.xerox.com> Greg Ewing wrote: > Bill Janssen wrote: > > > Duck typing is a seriously bad idea, forced on Python by the now > > obsolete split between built-in types and user-defined types. > > Non-duck typing is a seriously bad idea, forced > on some other languages by static typing. Python > is mercifully free of such constraints. I'm thinking that Mike Orr's question, "are we all talking about the same duck-typing", makes sense. Greg, I'm not suggesting static typing -- I much prefer dynamic strong typing. But what Python has now is dynamic weak typing, which makes programs (particularly frameworks) fragile. And it's mainly due to the historical accident of not being able to inherit from C-based types in Python 1.x. I'd like to be able to look at a value and determine which interfaces I can safely invoke on it, even if I don't understand its full type. I can't (safely) do that by string-matching method names. Bill From guido at python.org Wed Nov 15 07:18:20 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 14 Nov 2006 22:18:20 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <-7764899780500803165@unknownmsgid> References: <455A7C9E.2050904@canterbury.ac.nz> <-7764899780500803165@unknownmsgid> Message-ID: On 11/14/06, Bill Janssen wrote: > Greg Ewing wrote: > > Bill Janssen wrote: > > > > > Duck typing is a seriously bad idea, forced on Python by the now > > > obsolete split between built-in types and user-defined types. > > > > Non-duck typing is a seriously bad idea, forced > > on some other languages by static typing. Python > > is mercifully free of such constraints. > > I'm thinking that Mike Orr's question, "are we all talking about the > same duck-typing", makes sense. Greg, I'm not suggesting static > typing -- I much prefer dynamic strong typing. But what Python has > now is dynamic weak typing, which makes programs (particularly > frameworks) fragile. And it's mainly due to the historical accident > of not being able to inherit from C-based types in Python 1.x. > > I'd like to be able to look at a value and determine which interfaces > I can safely invoke on it, even if I don't understand its full type. > I can't (safely) do that by string-matching method names. I can't completely disagree with everything you say (introspection of interfaces makes sense, and this is proven by that several frameworks have implemented an explicit notion of interfaces). But I object to your claim that it was invented because of the difference between C and python types/classes in Python versions <= 2.1. The real reason was (and is) that there are no type declarations in Python, so there is absolutely no reason why you *couldn't* pass anything with an appropriate append() method to the following function: def foo(x): x.append(42) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From george.sakkis at gmail.com Wed Nov 15 07:33:53 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Wed, 15 Nov 2006 01:33:53 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <455A7CD1.5090503@canterbury.ac.nz> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> Message-ID: <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> On 11/14/06, Greg Ewing wrote: > George Sakkis wrote: > > > I think I do, though I can't tell the same about the reasons of your > > objections to it. > > Perhaps I can fill in some of the things that Guido > is not explicitly saying. Indeed, another high quality reply. Thanks for sharing. > You've shown that it *can* be done, but you haven't > provided a compelling reason why it *should* be done. > > The main benefit seems to be that itertools operations > could then be done using method calls instead of > function calls. But that's not automatically better > than what we have now. Python does not follow the > "everything must be a method of something" ideology > found in some other languages such as Java and Ruby. I won't open another can of worms here, but I'll just say that as much as I hate Java's stubborn insistence on OO purity, I am equally disturbed by Python's arbitrary-looking choices on whether some callable ends up a function or a method. When I talk about or introduce Python to outsiders, one of the toughest questions is often something along the lines of "why len() is a function and not a method?" > The itertools functions have generic implementations > that work with any iterator. They don't need access > to the iterator's internals, or need to be dynamically > dispatched for any reason, so there is no obvious > benefit to making them methods instead of stand-alone > functions. Did you look at my implementation ? The only attribute is a reference to the underlying iterator. No access to its internal is needed. > The one technical benefit appears to be the ability > to use operators instead of named functions. But > considering how infrequently itertools operations > are used, having to spell them out doesn't seem > like a great hardship, so this argument is weak. > All other things being equal, it might hold sway, > but other things are not equal. This seems to be the most central, and at the same time most unexpected for me, point of resistance. Given the prominence of iterators, generators, generator comprehensions and generic iterables in Python, I've been considering itertools the module that makes all these "talk" to each other seamlessly and transparently. I am totally amazed that I have itertools in higher esteem than many/most core Python developers. > You claim that duck typing would not be inhibited. > While technically this is true, in practice it would > mean that any iterator that didn't inherit from this > base class would be a second-class citizen, unable > to be used by code that expected it to have all the > "standard" itertools methods. The promotion from second class to first would require 6 extra characters: Iter(x) (or iter(x) if it's ok to augment the existing iter()). Just spelling "itertools" takes more than that, let alone importing a function or two. The runtime cost would be negligible too since Iter is a thinnest wrapper around x (or it could be x itself if it was already "first-class"). > Finally, consider that there are infinitely many > possible functions of the kind that are found in > the itertools module. As stand-alone functions, > they all have equal status -- new ones can be added > and used in the same way by any code on any > iterator. Some good arguments would have to be > put forward as to why some particular ones should > be picked out and elevated to the status of methods. I'm not sure I understand your point here. If something was deemed useful enough to make it to itertools, it would be a candidate for being a method to a base Iter type. I don't have any strong feelings on the exact set of methods that I want to see included apart from the most basic ones (chain as +, islice as [], izip as zip and enumerate - yes, I count enumerate as an iter tool). Besides, types, like modules, are not carved in stone. It's not as if no builtin type or function in Python has ever grown (in a backwards compatible way) more methods, extra named arguments, etc. I guess the good arguments to be put forward for every candidate method would be similar to the ones that elevated enumerate() and reversed() to builtin status. George From fredrik at pythonware.com Wed Nov 15 08:55:27 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 15 Nov 2006 08:55:27 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: <455A7CEE.9000701@canterbury.ac.nz> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <8E8031E5-BCE9-4FBF-BA9D-487AA3316E31@mac.com> <740c3aec0611140121t6fac8dfcm4a58f3c56ae9daef@mail.gmail.com> <455A7CEE.9000701@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > If there were a substantial number of commonly > used operations on them that could be implemented > efficiently in a generic way, then we probably > would have listtools and dicttools modules > (or more properly sequencetools and mappingtools). > That doesn't seem to have happened, though. oh, of course it has happened. Python's full of sequence tools, and there are plenty of generic operations that can work on arbitrary mappings. and the *very* *first* *statement* a Python newcomer learns is a generic operation on a file-like object. From sluggoster at gmail.com Wed Nov 15 08:56:13 2006 From: sluggoster at gmail.com (Mike Orr) Date: Tue, 14 Nov 2006 23:56:13 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> Message-ID: <6e9196d20611142356m72642bc6qd2821457617285d8@mail.gmail.com> On 11/14/06, George Sakkis wrote: > When I talk about or > introduce Python to outsiders, one of the toughest questions is often > something along the lines of "why len() is a function and not a > method?" I came to Python from Java and originally thought this. The answer I received was as I described: methods are for a certain class hierarchy, while functions are for an unknown number of potential types that exhibit a certain behavior. So you can call len() on both lists and strings. Here Java has an inconsistency: array.length vs String.length(). This in itself illustrates my point. Different types necessarily have different implementations. But if the left hand isn't watching closely what the right hand is doing, you end up with cross-class inconsistencies and incompatibilities. len() here is a unifier: it says "thou shalt do this, .__len__ method, or users will be pissed that len() doesn't work on your type". Python made mistakes along the way in terms of method/function distribution, but these are gradually being corrected. The most notorious were string functions. Who wants to call string.find(s, sub) or string.split(s)? But these have long been replaced by methods. My work on a Path object is along the same lines. The #1 complaint I hear about Python from non-users is indentation, not too many functions. People generally praise Python's clean method interface in its file objects, list/str/dict objects, and its stdlib namespaces. These aren't perfect but they're better than many languages. I was very pleased that typecasting in Python is as straightforward as int() and str(), and calling a function instantiates it. Much more straightforward than " Foo x = new Foo()". > I am totally > amazed that I have itertools in higher esteem than many/most core > Python developers. I esteem itertools, I just don't have much need for those operations. I'm glad it's there because it has better (less memory intensive) implementations than I'd likely make on my own, and it shows ways to use iterators that I'd never considered. > > Finally, consider that there are infinitely many > > possible functions of the kind that are found in > > the itertools module. As stand-alone functions, > > they all have equal status -- new ones can be added > > and used in the same way by any code on any > > iterator. Some good arguments would have to be > > put forward as to why some particular ones should > > be picked out and elevated to the status of methods. > > I'm not sure I understand your point here. If something was deemed > useful enough to make it to itertools, it would be a candidate for > being a method to a base Iter type. That's a good point, the overhead of adding a method is not really worse than adding a function. I wouldn't object to an iter() function that returns a SuperIterator, able to slice tall iterables in a single bound, er, cut. But I'm not a core developer so I don't understand all the ramifications it might have. That's a point worth pressing though: what's the harm in having iter() return an object that's "sma-a-a-rter than your average bear!", er, than your average iterator. > I guess the good arguments to be put > forward for every candidate method would be similar to the ones that > elevated enumerate() and reversed() to builtin status. Actually, it's easier to add a method or package function than to add a builtin. Every additional builtin makes it harder to keep all the builtins in your head, and runs the risk of colliding with a variable in existing code. And as Cheetah discovered, it's not safe to autogenerate Python code based on what the builtins are, because a future version of Python will add ones you can't predict. enumerate() and zip() were added because they solve extremely widespread problems. I can't say I like sorted() and reversed(), but they're there because somebody thought they'd be very widely used. -- Mike Orr From fredrik at pythonware.com Wed Nov 15 08:57:22 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 15 Nov 2006 08:57:22 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> Message-ID: George Sakkis wrote: > I won't open another can of worms here, but I'll just say that as much > as I hate Java's stubborn insistence on OO purity, I am equally > disturbed by Python's arbitrary-looking choices on whether some > callable ends up a function or a method. When I talk about or > introduce Python to outsiders, one of the toughest questions is often > something along the lines of "why len() is a function and not a > method?" I guess those outsiders don't really understand the concept of a "generic operation". are you sure *you* understand what that is? it's not obvious from your posts. From fredrik at pythonware.com Wed Nov 15 09:02:22 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 15 Nov 2006 09:02:22 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: <6e9196d20611142356m72642bc6qd2821457617285d8@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <6e9196d20611142356m72642bc6qd2821457617285d8@mail.gmail.com> Message-ID: Mike Orr wrote: > len() here is a unifier: it says "thou shalt do this, .__len__ method, > or users will be pissed that len() doesn't work on your type". that's an excellent observation. mind if I borrow it for the related FAQ entry? http://tinyurl.com/y6vavp From george.sakkis at gmail.com Wed Nov 15 09:47:22 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Wed, 15 Nov 2006 03:47:22 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> Message-ID: <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> Fredrik Lundh wrote: > George Sakkis wrote: > > > I won't open another can of worms here, but I'll just say that as much > > as I hate Java's stubborn insistence on OO purity, I am equally > > disturbed by Python's arbitrary-looking choices on whether some > > callable ends up a function or a method. When I talk about or > > introduce Python to outsiders, one of the toughest questions is often > > > something along the lines of "why len() is a function and not a > > method?" > > I guess those outsiders don't really understand the concept of a > "generic operation". are you sure *you* understand what that is? > it's not obvious from your your posts. Perhaps I don't, but if you or anyone else wants to enlighten me, please make sure you include in your explanation 1) why having a "generic operation" len() that ends up looking for an ugly special *method* called __len__() makes sense, while calling directly a method len() doesn't, and 2) if "generic operations" are such a cool idea, why there aren't more of them, such as count() or index() (the former for any container type, the latter for sequence types). I don't think that defending a decision by pointing out a different bad decision in another language (Java's inconsistency between string.length and list.length()) is a particularly compelling argument, especially in an FAQ page. The ending sentences though are more convincing: "...but it's a part of Python, and it's too late to make such fundamental changes now. The functions have to remain to avoid massive code breakage". That I can buy (at least for 2.x). George From fredrik at pythonware.com Wed Nov 15 10:12:28 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 15 Nov 2006 10:12:28 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> Message-ID: George Sakkis wrote: > 1) why having a "generic operation" len() that ends up looking for an > ugly special *method* called __len__() makes sense, while calling > directly a method len() doesn't for the very reason Mike explained: keeping implementation interfaces separate from public interfaces has allowed Python to avoid a certain category of design fragmentation that other languages suffer from. > 2) if "generic operations" are such a cool idea, why there aren't more > of them, such as count() or index() (the former for any container > type, the latter for sequence types). count() isn't that common, really (and can be written as sum(1 for item in iterable if condition) if you really want), and things like "in" and "any" and "enumerate" are already built-ins. > I don't think that defending a decision by pointing out a different > bad decision in another language (Java's inconsistency between > string.length and > list.length()) > is a particularly compelling > argument see above. > The ending sentences though are more convincing: "...but it's a part > of Python, and it's too late to make such fundamental changes now. > The functions have to remain to avoid massive code breakage". That > I can buy (at least for 2.x). yeah, it's clear that most of your argumentation is based on a rather common "I haven't thought this through very deeply, but I'm sure I'm smarter than those guys so that won't stop me" approach. trust me, the design of Python is a *lot* more carefully put together than you appear to think. From george.sakkis at gmail.com Wed Nov 15 10:40:12 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Wed, 15 Nov 2006 04:40:12 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> Message-ID: <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> Fredrik Lundh wrote: > George Sakkis wrote: > > > 1) why having a "generic operation" len() that ends up looking for an > > ugly special *method* called __len__() makes sense, while calling > > directly a method len() doesn't > > for the very reason Mike explained: keeping implementation interfaces > separate from public interfaces has allowed Python to avoid a certain > category of design fragmentation that other languages suffer from. Fredrik, I am not arguing for the argument's sake, I just don't get it: Python requires my class to define __len__, otherwise len() fails. Why not require len() as a method instead and forget about __len__ ? Does len() (the function) do anything smarter behind the scenes than just passing the ball to __len__ ? That could justify its role but AFAIK it doesn't. George From solipsis at pitrou.net Wed Nov 15 11:10:44 2006 From: solipsis at pitrou.net (Antoine) Date: Wed, 15 Nov 2006 11:10:44 +0100 (CET) Subject: [Python-3000] duck typing In-Reply-To: <455A7C9E.2050904@canterbury.ac.nz> References: <06Nov13.145133pst.58648@synergy1.parc.xerox.com> <455A7C9E.2050904@canterbury.ac.nz> Message-ID: <49981.62.39.9.251.1163585444.squirrel@webmail.nerim.net> > Non-duck typing is a seriously bad idea, forced > on some other languages by static typing. Static != non-duck. One could imagine static duck typing (is it the same as structural typing?) with type inference. I wonder if some existing languages have static duck typing (boo? haskell?). From fredrik at pythonware.com Wed Nov 15 11:28:54 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 15 Nov 2006 11:28:54 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> Message-ID: George Sakkis wrote: > Fredrik, I am not arguing for the argument's sake, I just don't get > it: Python requires my class to define __len__, otherwise len() fails. > Why not require len() as a method instead and forget about __len__ ? and have people complain about a len/__getitem__ naming inconsistency? (cf. the __iter__/next vs. __iter__/__next__ discussions.) > Does len() (the function) do anything smarter behind the scenes than > just passing the ball to __len__ ? yes. len() can use alternative approaches to determine the length for objects implemented in the host language. this is used by CPython to efficiently implement len() for C-level objects without having to do full Python method resolution and dispatch for each call. you could of course turn len() into a __len__() helper (or even sequencetools.len(), if you prefer) for cases where this matters, or even reserve the "len" method name, but I'm not sure how that would make things *simpler* than they are today. I'm convinced that it's better for people to get over that silly notion that writing x.foo() is somehow always "better" than writing foo(x), in a language that actually supports both forms. (if I'd have to chose between foo(x) and x.foo(), I'd rather get foo(x) with multiple dispatch than today's x.foo() single dispatch, but that's probably too radical for Python 3000 ;-) From ncoghlan at gmail.com Wed Nov 15 11:34:44 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Nov 2006 20:34:44 +1000 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> Message-ID: <455AED44.5090402@gmail.com> George Sakkis wrote: > Fredrik Lundh wrote: > >> George Sakkis wrote: >> >>> 1) why having a "generic operation" len() that ends up looking for an >>> ugly special *method* called __len__() makes sense, while calling >>> directly a method len() doesn't >> for the very reason Mike explained: keeping implementation interfaces >> separate from public interfaces has allowed Python to avoid a certain >> category of design fragmentation that other languages suffer from. > > Fredrik, I am not arguing for the argument's sake, I just don't get > it: Python requires my class to define __len__, otherwise len() fails. > Why not require len() as a method instead and forget about __len__ ? > Does len() (the function) do anything smarter behind the scenes than > just passing the ball to __len__ ? That could justify its role but > AFAIK it doesn't. The benefit of the magic method approach is that it leaves control of the public namespace of the class instance in the hands of the class developer. For a given class, it may make sense for the class API to expose its length under a different name like "size()" or "count()" (e.g. exposing an existing Java or C++ class to Python). However, regardless of what the class API itself looks like, it can be mapped behind the scenes into the standard Python idiom by doing "__len__ = size" (or whatever). If the Python idioms for these protocols were based on public API methods instead of the magic methods then it would greatly interfere with the developer's control over their own class interface. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Wed Nov 15 11:51:04 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Nov 2006 20:51:04 +1000 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611141144l58569e26wf5633a849e07c9ad@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141019i316747b1t45222bf50ecd0446@mail.gmail.com> <91ad5bf80611141144l58569e26wf5633a849e07c9ad@mail.gmail.com> Message-ID: <455AF118.1040903@gmail.com> George Sakkis wrote: > And for those objecting to touching the existing iter() or bloating > the builtin namespace with yet another builtin, let me mention that we > can get rid of *two* existing functions which for some reason were > promoted to builtin status, although they conceptually belong in > itertools: enumerate and reversed. FWIW, reversed isn't based on the iterator protocol - it needs a real sequence (i.e. provides __getitem__ supporting indices from 0 to len(seq)-1), or something that implements the __reversed__ special method. enumerate() was made a builtin as it is a direct replacement for the zip(range(len(x)), x) idiom. As far as the 'flexible iterator' idea goes, I've had a rough implementation of such a beast sitting on my hard drive for the last year and a half (since shortly after that January '05 thread I linked to earlier in this discussion). My conclusion from writing it was that depending on the implementation choices you make in developing the class you will end up in one of two situations: either 1. The entire iterator is soon loaded into memory as you manipulate it or 2. The underlying iterator is consumed unexpectedly as you manipulate it Which lead me to the conclusion that if you want richer access than the iterator protocol provides then your best option is to dump the contents of the iterator into an appropriate container and work with the container instead of the iterator (possible segmenting the retrieval of values if you want to limit peak memory usage). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From murman at gmail.com Wed Nov 15 15:48:33 2006 From: murman at gmail.com (Michael Urman) Date: Wed, 15 Nov 2006 08:48:33 -0600 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> Message-ID: On 11/15/06, George Sakkis wrote: > Why not require len() as a method instead and forget about __len__ ? > Does len() (the function) do anything smarter behind the scenes than > just passing the ball to __len__ ? That could justify its role but > AFAIK it doesn't. It most certainly does. It not only unifies the name, it makes an interface guarantee you couldn't make on a custom method: return an integer or raise an exception. >>> class Len(object): ... def __len__(self): return 'length' ... >>> obj = Len() >>> obj.__len__() 'length' >>> len(obj) Traceback (most recent call last): File "", line 1, in ? TypeError: an integer is required And, if you don't like this interface, it gives you a single point to override it. (Shoving it into your builtins left as an exercise for the interested.) >>> def len(thing): ... try: return int(thing.__len__()) ... except (ValueError, TypeError, AttributeError): return 0 ... >>> len(obj) 0 -- Michael Urman http://www.tortall.net/mu/blog From george.sakkis at gmail.com Wed Nov 15 16:26:48 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Wed, 15 Nov 2006 10:26:48 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <455AED44.5090402@gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> <455AED44.5090402@gmail.com> Message-ID: <91ad5bf80611150726m7d6f187bhf59593aba111e298@mail.gmail.com> On 11/15/06, Nick Coghlan wrote: > George Sakkis wrote: > > Fredrik Lundh wrote: > > > >> George Sakkis wrote: > >> > >>> 1) why having a "generic operation" len() that ends up looking for an > >>> ugly special *method* called __len__() makes sense, while calling > >>> directly a method len() doesn't > >> for the very reason Mike explained: keeping implementation interfaces > >> separate from public interfaces has allowed Python to avoid a certain > >> category of design fragmentation that other languages suffer from. > > > > Fredrik, I am not arguing for the argument's sake, I just don't get > > it: Python requires my class to define __len__, otherwise len() fails. > > Why not require len() as a method instead and forget about __len__ ? > > Does len() (the function) do anything smarter behind the scenes than > > just passing the ball to __len__ ? That could justify its role but > > AFAIK it doesn't. > > The benefit of the magic method approach is that it leaves control of the > public namespace of the class instance in the hands of the class developer. > For a given class, it may make sense for the class API to expose its length > under a different name like "size()" or "count()" (e.g. exposing an existing > Java or C++ class to Python). > > However, regardless of what the class API itself looks like, it can be mapped > behind the scenes into the standard Python idiom by doing "__len__ = size" (or > whatever). If the Python idioms for these protocols were based on public API > methods instead of the magic methods then it would greatly interfere with the > developer's control over their own class interface. On the one hand this makes sense, but on the other it goes back to Bill's reply on that Python should formalize some widely used interfaces (iterators, sequence, mappings, etc.), exactly for extending the "one obvious way to do it" rule to interfaces and prevent anyone who misses Java too much to use getIndex() for his container instead of __getitem__. George From gsakkis at rutgers.edu Wed Nov 15 16:39:26 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Wed, 15 Nov 2006 10:39:26 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <455AF118.1040903@gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141019i316747b1t45222bf50ecd0446@mail.gmail.com> <91ad5bf80611141144l58569e26wf5633a849e07c9ad@mail.gmail.com> <455AF118.1040903@gmail.com> Message-ID: <91ad5bf80611150739h6c19a130ve065a773b0b08855@mail.gmail.com> On 11/15/06, Nick Coghlan wrote: > As far as the 'flexible iterator' idea goes, I've had a rough implementation > of such a beast sitting on my hard drive for the last year and a half (since > shortly after that January '05 thread I linked to earlier in this discussion). > My conclusion from writing it was that depending on the implementation choices > you make in developing the class you will end up in one of two situations: > > either 1. The entire iterator is soon loaded into memory as you manipulate it > > or 2. The underlying iterator is consumed unexpectedly as you manipulate it For (2), I'd say it depends on your expectations. If you expect x[:5] to have always the same semantics you expect from a list, then yes, you will be surprized if x is an iterator, as you may be surprized if x is a numpy array. However, it seems contradictory for Python to insist on specific method semantics, while at the same time is reluctant to enforce APIs for omnipresent interfaces like sequences and iterators. George From gsakkis at rutgers.edu Wed Nov 15 17:08:54 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Wed, 15 Nov 2006 11:08:54 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611150739h6c19a130ve065a773b0b08855@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141019i316747b1t45222bf50ecd0446@mail.gmail.com> <91ad5bf80611141144l58569e26wf5633a849e07c9ad@mail.gmail.com> <455AF118.1040903@gmail.com> <91ad5bf80611150739h6c19a130ve065a773b0b08855@mail.gmail.com> Message-ID: <91ad5bf80611150808m8d72ddanf30bfef495c8475d@mail.gmail.com> Michael Urman wrote: > On 11/15/06, George Sakkis wrote: > > Why not require len() as a method instead and forget about __len__ ? > > Does len() (the function) do anything smarter behind the scenes than > > just passing the ball to __len__ ? That could justify its role but > > AFAIK it doesn't. > > It most certainly does. It not only unifies the name, it makes an > interface guarantee you couldn't make on a custom method: return an > integer or raise an exception. > > >>> class Len(object): > ... def __len__(self): return 'length' > ... > >>> obj = Len() > >>> obj.__len__() > 'length' > >>> len(obj) > Traceback (most recent call last): > File "", line 1, in ? > TypeError: an integer is required > > And, if you don't like this interface, it gives you a single point to > override it. (Shoving it into your builtins left as an exercise for > the interested.) > > >>> def len(thing): > ... try: return int(thing.__len__()) > ... except (ValueError, TypeError, AttributeError): return 0 Thank you for your explanation; I'd rather read this in the FAQ rather than a moot comparison with Java's inconsistent design. As I wrote in my last reply to Nick though, I question Python's right to perform such limited forms of design-by-contract-like assertions ("why not add a precondition on __add__(self,other) to enforce isinstance(other, self.__class__) ?") when it refuses to formalize interfaces for sequences, iterators et al. George From ark-mlist at att.net Wed Nov 15 17:50:47 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Wed, 15 Nov 2006 11:50:47 -0500 Subject: [Python-3000] duck typing In-Reply-To: <49981.62.39.9.251.1163585444.squirrel@webmail.nerim.net> Message-ID: <000001c708d6$39d0fe90$6402a8c0@arkdesktop> > Static != non-duck. > One could imagine static duck typing (is it the same as structural > typing?) with type inference. I wonder if some existing languages have > static duck typing (boo? haskell?). C++ (using templates). From fredrik at pythonware.com Wed Nov 15 17:59:41 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 15 Nov 2006 17:59:41 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611150808m8d72ddanf30bfef495c8475d@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141019i316747b1t45222bf50ecd0446@mail.gmail.com> <91ad5bf80611141144l58569e26wf5633a849e07c9ad@mail.gmail.com> <455AF118.1040903@gmail.com> <91ad5bf80611150739h6c19a130ve065a773b0b08855@mail.gmail.com> <91ad5bf80611150808m8d72ddanf30bfef495c8475d@mail.gmail.com> Message-ID: George Sakkis wrote: > Thank you for your explanation; I'd rather read this in the FAQ rather than a > moot comparison with Java's inconsistent design. if you have comments on the FAQ entries, post them over there. note that the FAQ entry isn't about len(), though; it's about generic functions in general. (and I completely disagree that Mike's observation is moot: consistency is a fundamental part of Python's design, and mechanisms that helps people keep their code consistent without even noticing is an important part of why things are that way.) From gsakkis at rutgers.edu Wed Nov 15 18:39:02 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Wed, 15 Nov 2006 12:39:02 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611150808m8d72ddanf30bfef495c8475d@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141019i316747b1t45222bf50ecd0446@mail.gmail.com> <91ad5bf80611141144l58569e26wf5633a849e07c9ad@mail.gmail.com> <455AF118.1040903@gmail.com> <91ad5bf80611150739h6c19a130ve065a773b0b08855@mail.gmail.com> <91ad5bf80611150808m8d72ddanf30bfef495c8475d@mail.gmail.com> Message-ID: <91ad5bf80611150939s416c7a28n7277cc795e35c82c@mail.gmail.com> Fredrik Lundh wrote: > > Thank you for your explanation; I'd rather read this in the FAQ rather than a > > moot comparison with Java's inconsistent design. > > if you have comments on the FAQ entries, post them over there. > > note that the FAQ entry isn't about len(), though; it's about generic > functions in general. Something along the lines of "a generic function might also do other smart things apart from just calling the appropriate methods on its argument(s), such as ..." would help (rephrased to match the FAQ's style). > (and I completely disagree that Mike's observation is moot: consistency > is a fundamental part of Python's design, and mechanisms that helps > people keep their code consistent without even noticing is an important > part of why things are that way.) Striving for consistency without enforcing interfaces seems self defeating. George From jcarlson at uci.edu Wed Nov 15 18:13:41 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 15 Nov 2006 09:13:41 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611150808m8d72ddanf30bfef495c8475d@mail.gmail.com> References: <91ad5bf80611150739h6c19a130ve065a773b0b08855@mail.gmail.com> <91ad5bf80611150808m8d72ddanf30bfef495c8475d@mail.gmail.com> Message-ID: <20061115090151.82FE.JCARLSON@uci.edu> "George Sakkis" wrote: > As I wrote in my last reply to Nick though, I question Python's right to perform > such limited forms of design-by-contract-like assertions ("why not add > a precondition > on __add__(self,other) to enforce isinstance(other, self.__class__) > ?") when it refuses to formalize interfaces for sequences, iterators > et al. Because 5, 5+0j, 5.0 are different types. To not be able to add any of them because they are different types, would be silly. Note that Decimal is a different beast, which is why I don't include it. There's also string + unicode (at least in 2.x), and if some sort of unicode view makes it into 3.x, view + unicode. - Josiah From fredrik at pythonware.com Wed Nov 15 18:48:29 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 15 Nov 2006 18:48:29 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611150939s416c7a28n7277cc795e35c82c@mail.gmail.com> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <20061114090956.82EB.JCARLSON@uci.edu> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141019i316747b1t45222bf50ecd0446@mail.gmail.com> <91ad5bf80611141144l58569e26wf5633a849e07c9ad@mail.gmail.com> <455AF118.1040903@gmail.com> <91ad5bf80611150739h6c19a130ve065a773b0b08855@mail.gmail.com> <91ad5bf80611150808m8d72ddanf30bfef495c8475d@mail.gmail.com> <91ad5bf80611150939s416c7a28n7277cc795e35c82c@mail.gmail.com> Message-ID: George Sakkis wrote: >> (and I completely disagree that Mike's observation is moot: consistency >> is a fundamental part of Python's design, and mechanisms that helps >> people keep their code consistent without even noticing is an important >> part of why things are that way.) > > Striving for consistency without enforcing interfaces seems self defeating. seems to, perhaps, but Python's been around for a quite some time by now. if the approach didn't work, we'd probably noticed by now. From janssen at parc.com Wed Nov 15 18:54:47 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 15 Nov 2006 09:54:47 PST Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> Message-ID: <06Nov15.095454pst."58648"@synergy1.parc.xerox.com> > Fredrik Lundh wrote: > > > George Sakkis wrote: > > > > > 1) why having a "generic operation" len() that ends up looking for an > > > ugly special *method* called __len__() makes sense, while calling > > > directly a method len() doesn't > > > > for the very reason Mike explained: keeping implementation interfaces > > separate from public interfaces has allowed Python to avoid a certain > > category of design fragmentation that other languages suffer from. > > Fredrik, I am not arguing for the argument's sake, I just don't get > it: Python requires my class to define __len__, otherwise len() fails. > Why not require len() as a method instead and forget about __len__ ? > Does len() (the function) do anything smarter behind the scenes than > just passing the ball to __len__ ? That could justify its role but > AFAIK it doesn't. > > George So you're suggesting something like: class Lengthable: def len(self): raise NotImplemented (presumably defined in some "standard" or "builtins" module) be a mix-in to every type which it currently makes sense to call "len()" on? Bill From george.sakkis at gmail.com Wed Nov 15 19:36:33 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Wed, 15 Nov 2006 13:36:33 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <-9052809295998148604@unknownmsgid> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> <-9052809295998148604@unknownmsgid> Message-ID: <91ad5bf80611151036p248fde9ate83b21f631fe7fc3@mail.gmail.com> On 11/15/06, Bill Janssen wrote: > > Fredrik Lundh wrote: > > > > > George Sakkis wrote: > > > > > > > 1) why having a "generic operation" len() that ends up looking for an > > > > ugly special *method* called __len__() makes sense, while calling > > > > directly a method len() doesn't > > > > > > for the very reason Mike explained: keeping implementation interfaces > > > separate from public interfaces has allowed Python to avoid a certain > > > category of design fragmentation that other languages suffer from. > > > > Fredrik, I am not arguing for the argument's sake, I just don't get > > it: Python requires my class to define __len__, otherwise len() fails. > > Why not require len() as a method instead and forget about __len__ ? > > Does len() (the function) do anything smarter behind the scenes than > > just passing the ball to __len__ ? That could justify its role but > > AFAIK it doesn't. > > > > George > > So you're suggesting something like: > > class Lengthable: > > def len(self): > raise NotImplemented > > (presumably defined in some "standard" or "builtins" module) be a > mix-in to every type which it currently makes sense to call "len()" > on? No, having an interface for every single method is obviously an overkill. Unless I am convinced otherwise, I am thinking something like: class Container(object): def len(self): raise NotImplementedError def __iter__(self): raise NotImplementedError # other generic container methods class Sequence(Container): def __getitem__(self, index_or_slice): raise NotImplementedError # more methods class Mapping(Container): def __getitem__(self, key): raise NotImplementedError # more methods class list(Sequence): # blah blah class tuple(Sequence): # blah blah class MyFancySequence(Sequence): # blah blah Note that this does _not_ disallow duck typing; anyone is perfectly free to define a method len() in a class that does not extend any of the above so that a client can call x.len() successfully. If not, a normal AttributeError will be raised, instead of the TypeError you get today. George From gsakkis at rutgers.edu Wed Nov 15 19:47:50 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Wed, 15 Nov 2006 13:47:50 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <20061115090151.82FE.JCARLSON@uci.edu> References: <91ad5bf80611150739h6c19a130ve065a773b0b08855@mail.gmail.com> <91ad5bf80611150808m8d72ddanf30bfef495c8475d@mail.gmail.com> <20061115090151.82FE.JCARLSON@uci.edu> Message-ID: <91ad5bf80611151047k57046c0ct84ab731b8701c7cd@mail.gmail.com> On 11/15/06, Josiah Carlson wrote: > > "George Sakkis" wrote: > > As I wrote in my last reply to Nick though, I question Python's right to perform > > such limited forms of design-by-contract-like assertions ("why not add > > a precondition > > on __add__(self,other) to enforce isinstance(other, self.__class__) > > ?") when it refuses to formalize interfaces for sequences, iterators > > et al. > > Because 5, 5+0j, 5.0 are different types. To not be able to add any of > them because they are different types, would be silly. Note that > Decimal is a different beast, which is why I don't include it. There's > also string + unicode (at least in 2.x), and if some sort of > unicode view makes it into 3.x, view + unicode. You missed my point; I wasn't seriously suggesting that there should be more checks (hence the quotes), but rather the contrary, that checking whether len() returns a non-negative integer does very little to the overall consistency scheme of things. Pre-conditions, post-conditions, invariants are a stricter form of API conformance than just checking whether a method with a specific name is there, and Python is unwilling to check even that. George From jcarlson at uci.edu Wed Nov 15 20:36:09 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 15 Nov 2006 11:36:09 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611151047k57046c0ct84ab731b8701c7cd@mail.gmail.com> References: <20061115090151.82FE.JCARLSON@uci.edu> <91ad5bf80611151047k57046c0ct84ab731b8701c7cd@mail.gmail.com> Message-ID: <20061115111318.8301.JCARLSON@uci.edu> "George Sakkis" wrote: > You missed my point; I wasn't seriously suggesting that there should > be more checks (hence the quotes), but rather the contrary, that > checking whether len() returns a non-negative integer does very little > to the overall consistency scheme of things. > > Pre-conditions, post-conditions, invariants are a stricter form of API > conformance than just checking whether a method with a specific name > is there, and Python is unwilling to check even that. No, it's not that Python is unwilling, we just haven't found the need. For people who want/need to _enforce_ preconditions, postconditions, invariants, etc., Python 2.4 introduced a decorator syntax that makes it (arguably) trivial to declare such things... @typecheck((int, long), (int, long), (int, long, float)) @precondition(ARG > 0, 0 < ARG < 10, NOTIN(inf, NaN, -inf)) @postcondition(NOTIN(inf, NaN, -inf)) def fcn(a, b, c): ... Given proper definitions of typecheck, precondition, postcondition, ARG and NOTIN. For the rest of us who document, use unit testing (with or without the unittest framework), etc., thinking of development in terms of what we learned in school for developing C, C++, or Java seems backwards (at least in my opinion). I can spend days writing all of that crap, or I can spend a few hours writing the software with unittests in Python. - Josiah From gsakkis at rutgers.edu Wed Nov 15 20:53:48 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Wed, 15 Nov 2006 14:53:48 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <20061115111318.8301.JCARLSON@uci.edu> References: <20061115090151.82FE.JCARLSON@uci.edu> <91ad5bf80611151047k57046c0ct84ab731b8701c7cd@mail.gmail.com> <20061115111318.8301.JCARLSON@uci.edu> Message-ID: <91ad5bf80611151153m6fc76cefu66960901f6fb58c1@mail.gmail.com> On 11/15/06, Josiah Carlson wrote: > > "George Sakkis" wrote: > > You missed my point; I wasn't seriously suggesting that there should > > be more checks (hence the quotes), but rather the contrary, that > > checking whether len() returns a non-negative integer does very little > > to the overall consistency scheme of things. > > > > Pre-conditions, post-conditions, invariants are a stricter form of API > > conformance than just checking whether a method with a specific name > > is there, and Python is unwilling to check even that. > > No, it's not that Python is unwilling, we just haven't found the need. > For people who want/need to _enforce_ preconditions, postconditions, > invariants, etc., Python 2.4 introduced a decorator syntax that makes it > (arguably) trivial to declare such things... > > @typecheck((int, long), (int, long), (int, long, float)) > @precondition(ARG > 0, 0 < ARG < 10, NOTIN(inf, NaN, -inf)) > @postcondition(NOTIN(inf, NaN, -inf)) > def fcn(a, b, c): > ... > > Given proper definitions of typecheck, precondition, postcondition, ARG > and NOTIN. > > For the rest of us who document, use unit testing (with or without the > unittest framework), etc., thinking of development in terms of what we > learned in school for developing C, C++, or Java seems backwards (at > least in my opinion). I can spend days writing all of that crap, or I > can spend a few hours writing the software with unittests in Python. To avoid leading this to a digression about optional typing, my point was that what len() does is a builtin postcondition, and if, for some perverse reason, someone wants to define negative or complex lengths for his perverse class, he'll hit into a wall. Is this yet another case of practicality beating purity hands down ? George From george.sakkis at gmail.com Wed Nov 15 21:34:24 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Wed, 15 Nov 2006 15:34:24 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> <-9052809295998148604@unknownmsgid> <91ad5bf80611151036p248fde9ate83b21f631fe7fc3@mail.gmail.com> Message-ID: <91ad5bf80611151234x7d7f6b27j6c7ef217c197c2dc@mail.gmail.com> On 11/15/06, Jim Jewett wrote: > (list clipped) Sorry, I'd rather not get into private conversations. > On 11/15/06, George Sakkis wrote: > > Note that this does _not_ disallow duck typing; anyone is perfectly > > free to define a method len() in a class that does not extend any of > > the above so that a client can call x.len() successfully. > > > Not if > >>> if instance(x, Sequence): > > becomes a standard idiom It doesn't have to, and it shouldn't, become the standard; after all, whatever you do in the "else:" part of the isinstance check, you can do in a try/except block. This has nothing to do with type checking per se, it's a general LBYL vs EAFP error handling strategy. Python in general encourages the latter and I totally agree, there's no reason for this to change. In the few cases where LBYL is deemed better and you'd use isinstance, you can do the same today with if hasattr(x, '__getitem__'): or whatever attributes you need. George From george.sakkis at gmail.com Wed Nov 15 23:29:05 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Wed, 15 Nov 2006 17:29:05 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <455B7FE7.8040301@ilowe.net> References: <91ad5bf80611140849t25509976q5f396cec9d3c7370@mail.gmail.com> <91ad5bf80611140955s7cd3cfe8qee85b898e98230de@mail.gmail.com> <91ad5bf80611141006k431695e4l1d7752cb1f027aea@mail.gmail.com> <200611141353.18690.fdrake@acm.org> <91ad5bf80611141132v3cac4035nfa92b52f394ebd1a@mail.gmail.com> <455B7FE7.8040301@ilowe.net> Message-ID: <91ad5bf80611151429u3c5153dkeffe7b45be7b8cf0@mail.gmail.com> On 11/15/06, Iain Lowe wrote: > Hi George, > > I've been reading this thread with some interest since you seem to be > really adamant about the "features" you are requesting. I went back into > the c.l.p thread and did a little more searching before finding this > thread: http://shrunklink.com/?nnq Glad you ran a full investigation on me before replying, that's reassuring :-) > I don't really understand how you can be advocating the opposite of what > you explain rather clearly in the referenced thread. Specifically you > say "there's no particular reason this should be a method of class A > since it can be used for any arbitrary object with no extra overhead. > Now, if you intend to use it only for instances of A and its subclasses, > the only difference would be syntactic." > > Now, clearly we are not talking about "any arbitrary object" but rather > "any arbitrary object that implements the iterable protocol" but it > seems to me that there is little difference between what you are > proposing here and the OOP solution the poster in the referenced thread > was proposing. > > In both cases, the implementation of a series of constraints as an > *interface* instead of as a *protocol* un-necessarily limits future > code. It also seems clear to me that you would like to be able to > slice/chain/etc. *any* iterable, not just instances of (subclasses of) > `Iter`. So why advocate the narrowing of the *protocol* into an > *interface* just to add some syntax sugar. Indeed, the two most-quoted > examples so far (slicing and indexing) can be better written as > "list(it)[:3]" and "list(it)[6]". The only possible caveat I can see is > that this will not work for infinite iterables. But *even if* you did > implement this "class-wrapper" for iterables you *still* would be > breaking the implementation of slices (since you would not be able to do > "Iter(infinite_it)[-5]"). > > What am I missing here? Several things: 1. In the thread you refer to, the whole point was *one* callable (listMethods) that takes a single arbitrary object. I wouldn't propose a class for holding a single method of course, but here we're talking about a dozen or so callables that take an iterable (and optionally other parameters). 2. If one wanted to attach listMethods to an existing type, the only obvious choice would be the base object type. That callable isn't so important though to pollute the base object namespace with a new method. OTOH in this thread: (1) I don't propose to pollute object or any other existing type; I propose a brand new "rich iterator" type, and (2) I consider several itertools functions (chain, islice, izip, enumerate) fundamental in getting the most out of iterables. Surprizingly, as it turns out nobody else here seems to share this feeling. I'd say this is the only argument so far I have nothing to respond to. If people consider itertools a small obscure module they rarely need to import anyway, the whole proposal fails flat. 3. I don't know if you've followed the thread for the last day or so, but I've dropped the part that suggested the proposed type as a required base for all iterators. Instead I suggest it as a wrapper (or "adaptor" if you prefer) to provide a rich-iterator API to the code that needs it. The overhead of wrapping an arbitrary iterable is trivial, both in keystrokes ("Iter(x)") and in runtime, so objections of the kind "how should a client know whether he is passed a simple iterator or a rich one?" are easily responded by "never assume you are passed a rich iterator; just wrap it with Iter() yourself if you need it". 4. As for the negative indices that don't work for iterators, let me remind you that indexing and slicing, like pretty much everything else in python, are an informal API. Numpy array slices also have different semantics from list slices, and for good reason. Another hypothetical example would be an automatically extensible sequence type defined so that s[i] for i>=len(s) doesn't raise IndexError, but it extends the sequence with some default value up to i and the returns s[i]. The bottom line is, you can't assume you know what x[:5] does without knowing x's type, the same way you don't if x.shoot() will shoot a photo or a bullet without knowing what x is. > P.S. I notice in a couple of places you ask what the objections are > whereas I think that in this case it's up to the proposal's author to > explain why the community can't live without the feature rather than for > the community to explain why they don't need it. Clearly if they needed > it they would have built it before (or at least you would just get a > stream +1's) Right, and that's the part I mistakenly took for granted, the prevalence of itertools. My bad. George From tomerfiliba at gmail.com Thu Nov 16 00:06:04 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Thu, 16 Nov 2006 01:06:04 +0200 Subject: [Python-3000] yes to class decorators Message-ID: <1d85506f0611151506n48e1ee48q91de76b05754435a@mail.gmail.com> i understand there's a green light for class decorators in py3k, so i wanted to give the issue a slight push. one problem with metaclasses is they are non-nestable, i.e., a function can be wrapped by a number of decorators, while classes can have only one metaclass. not only that, but metaclasses complicate inheritance, because their are not "componentized" the way decorators are (i.e., decorator A does not impose restrictions on how decorator B will work, whereas metaclasses must be subclasses of one another, which means they are not isolated *components*) first i want to differentiate between ``metaclasses`` and ``pseudo metaclasses``. real metaclasses subclass type and change it's behavior. pseudo metaclasses just use the __metaclass__ magic syntax to modify the class being returned, but not it's type (i.e., it's type is //type//) for example -- this is what i call pseudo-metaclass: >>> def singleton(*args): ... return type(*args)() # create the only instance ... >>> class OnlyOne(object): ... __metaclass__ = singleton ... >>> OnlyOne <__main__.OnlyOne object at 0x009F5270> of course we want to keep metaclasses as a facility as well as a concept, but we do not want to have to use the __metaclass__ syntax when it's not about real metaclasses. class decorators -- like function decorators -- wrap the object (class in this case), which makes them stackable and componentized. here are some usecases to class decorators: def singleton(cls): return cls() # create the "single instance" @singleton class OnlyOne(object): pass another example: @staticclass - automatically make all methods staticmethods, or whatever (dotNET has them, for instance). essentially this turns the class into a module. def staticclass(cls): newdict = {} for name, obj in cls.__dict__.iteritems(): if isinstance(getattr(cls, name), MethodType): newdict[name] = staticmethod(obj) else: newdict[name] = obj return type(cls.__name__, cls.__bases__, newdict) @staticclass class Eggs(objects): def foo(): print "foo" def goo(): print "goo" Eggs.foo() e = Eggs() e.goo() i'm not saying this particular example is useful, but i have had many times when i thought "argh, i wish i had class decorators for that". sadly though, i don't remember many of them, since i always had to find workarounds :) ANYWAY, if we went this far already, we might as well just trash the __metaclass__ syntax (*) altogether. we can easily implement a metaclass decorator that does the trick: def metaclass(meta): def wrapper(cls): return meta(cls.__name__, cls.__bases__, dict(cls.__dict__)) return wrapper class FooMeta(type): .... @singleton @metaclass(FooMeta) class Bar(object): .... or even something like @FooMeta class Bar(object): pass although that would require some hackery. forget it, it's better to be explicit. (*) throwing only the syntax. subclassing type is still required of course, but IMHO it's much more clear to understand where the "magic" comes from when a class has a @metaclass decorator instead of a __metaclass__ attribute. moreover, as shown above, we can simplify the way type_new() works, which is a blessing on its own (considering its current complexity) - tomer From steven.bethard at gmail.com Thu Nov 16 00:38:41 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 15 Nov 2006 16:38:41 -0700 Subject: [Python-3000] yes to class decorators In-Reply-To: <1d85506f0611151506n48e1ee48q91de76b05754435a@mail.gmail.com> References: <1d85506f0611151506n48e1ee48q91de76b05754435a@mail.gmail.com> Message-ID: On 11/15/06, tomer filiba wrote: > i understand there's a green light for class decorators in py3k, > so i wanted to give the issue a slight push. If you haven't already, you should look at PEP 359, which tried to address many of the same use-cases and was withdrawn after feedback on python-dev: http://www.python.org/dev/peps/pep-0359/ > def singleton(cls): > return cls() # create the "single instance" > > @singleton > class OnlyOne(object): > pass That would have been something like:: make singleton OnlyOne: pass > @staticclass > class Eggs(objects): > def foo(): > print "foo" > def goo(): > print "goo" And this would have been something like:: make namespace Eggs: def foo(): print "foo" def goo(): print "goo" > ANYWAY, if we went this far already, we might as well just > trash the __metaclass__ syntax (*) altogether. That was one of the open issues in the PEP too. Basically, I think that the use-cases you've given are not a great motivation -- they weren't enough to motivate the "make" statement of PEP 359, so they're probably not enough to motivate class decorators. FWIW, most of the arguments against PEP 359 were along the lines of, "well you can do that with a metaclass already, so we don't really need any new syntax", but you may be able to get around those arguments because the decorator syntax already exists. STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From george.sakkis at gmail.com Thu Nov 16 01:52:57 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Wed, 15 Nov 2006 19:52:57 -0500 Subject: [Python-3000] yes to class decorators Message-ID: <91ad5bf80611151652w51a4446ar1bf1e1307d1c6ba9@mail.gmail.com> tomer filiba wrote: > i'm not saying this particular example is useful, but i have > had many times when i thought "argh, i wish i had class > decorators for that". Same here, but like Steve, I'm not very optimistic this will be greeted with open arms. > sadly though, i don't remember many of them, since i always > had to find workarounds :) Off the top of my head, @abstractclass was one of my workarounds :) George From greg.ewing at canterbury.ac.nz Thu Nov 16 02:09:51 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 16 Nov 2006 14:09:51 +1300 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611151036p248fde9ate83b21f631fe7fc3@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> <-9052809295998148604@unknownmsgid> <91ad5bf80611151036p248fde9ate83b21f631fe7fc3@mail.gmail.com> Message-ID: <455BBA5F.5070106@canterbury.ac.nz> George Sakkis wrote: > class Container(object): > > class Sequence(Container): > class Mapping(Container): > > Note that this does _not_ disallow duck typing; anyone is perfectly > free to define a method len() in a class that does not extend any of > the above so that a client can call x.len() successfully. What's the point, then? What benefit do I get from extending Sequence if I can achieve the same thing without bothering? -- Greg From george.sakkis at gmail.com Thu Nov 16 03:49:12 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Wed, 15 Nov 2006 21:49:12 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <455BBA5F.5070106@canterbury.ac.nz> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> <-9052809295998148604@unknownmsgid> <91ad5bf80611151036p248fde9ate83b21f631fe7fc3@mail.gmail.com> <455BBA5F.5070106@canterbury.ac.nz> Message-ID: <91ad5bf80611151849v70f413e5s9085ae100a3a4741@mail.gmail.com> On 11/15/06, Greg Ewing wrote: > George Sakkis wrote: > > > class Container(object): > > > > class Sequence(Container): > > > class Mapping(Container): > > > > Note that this does _not_ disallow duck typing; anyone is perfectly > > free to define a method len() in a class that does not extend any of > > the above so that a client can call x.len() successfully. > > What's the point, then? What benefit do I get from > extending Sequence if I can achieve the same thing > without bothering? For one thing, the implementations you get for free for the methods that can be implemented without needing to know the specifics of the concrete class (think DictMixin, ListMixin and friends; all these would move there). So yes, for the nitpickers out there, these are not interfaces in the Java sense (or pure virtual classes in C++) but abstract classes. And for two, not everyone feels comfortable with duck typing. People who consider (for better or for worse) isinstance() safer than hasattr()/getattr() would be accomodated too. Why do we have to alienate those folks telling them they are wrong when both approaches can coexist, as it happens for LBYL vs EAFP error strategies, procedural vs OO vs functional paradigms, and other dichotomies and n-otomies that Python has successfully brought together ? George From jcarlson at uci.edu Thu Nov 16 02:43:12 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 15 Nov 2006 17:43:12 -0800 Subject: [Python-3000] yes to class decorators In-Reply-To: References: <1d85506f0611151506n48e1ee48q91de76b05754435a@mail.gmail.com> Message-ID: <20061115172512.8305.JCARLSON@uci.edu> "Steven Bethard" wrote: > On 11/15/06, tomer filiba wrote: > > i understand there's a green light for class decorators in py3k, > > so i wanted to give the issue a slight push. > FWIW, most of the arguments against PEP 359 were along the lines of, > "well you can do that with a metaclass already, so we don't really > need any new syntax", but you may be able to get around those > arguments because the decorator syntax already exists. That's neither here nor there. Here's a post from Guido in response to Phillip and Greg in which he says more or less; someone write a PEP so that we can get them into 2.6 and Py3k... http://mail.python.org/pipermail/python-dev/2006-March/062942.html The use-cases that Tomer brought up were also brought up in various posts within that same thread, and in the 3+ previous threads extending prior to March 2005 in the python-dev list (when I noticed that class decorators didn't make it into 2.4); namespaces, singletons, easy 'metaclass' chaining, whole class manipulations, properties, etc. The syntax already exists and I would imagine that there's a patch around somewhere. If Tomer (or someone else) writes a PEP, I don't see why (the previously overlooked) class decorators shouldn't make it into 2.6 and 3.0 . - Josiah From steven.bethard at gmail.com Thu Nov 16 04:55:56 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 15 Nov 2006 20:55:56 -0700 Subject: [Python-3000] yes to class decorators In-Reply-To: <20061115172512.8305.JCARLSON@uci.edu> References: <1d85506f0611151506n48e1ee48q91de76b05754435a@mail.gmail.com> <20061115172512.8305.JCARLSON@uci.edu> Message-ID: On 11/15/06, tomer filiba wrote: > i understand there's a green light for class decorators in py3k, > so i wanted to give the issue a slight push. "Steven Bethard" wrote: > FWIW, most of the arguments against PEP 359 were along the lines of, > "well you can do that with a metaclass already, so we don't really > need any new syntax", but you may be able to get around those > arguments because the decorator syntax already exists. On 11/15/06, Josiah Carlson wrote: > Here's a post from Guido in response to Phillip and Greg in which he > says more or less; someone write a PEP so that we can get them > into 2.6 and Py3k... > http://mail.python.org/pipermail/python-dev/2006-March/062942.html Thanks for the link. > If Tomer (or someone else) writes a PEP, I don't see why (the > previously overlooked) class decorators shouldn't make it into > 2.6 and 3.0 . So the purpose of this thread then is to write the PEP? If so, my comments about the use cases are still valid. If they weren't convincing use cases before, they're not likely to be convincing use cases for a PEP. Or was there another purpose of the thread? Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From jcarlson at uci.edu Thu Nov 16 05:09:29 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 15 Nov 2006 20:09:29 -0800 Subject: [Python-3000] yes to class decorators In-Reply-To: References: <20061115172512.8305.JCARLSON@uci.edu> Message-ID: <20061115200359.830A.JCARLSON@uci.edu> "Steven Bethard" wrote: > On 11/15/06, tomer filiba wrote: > > i understand there's a green light for class decorators in py3k, > > so i wanted to give the issue a slight push. > > "Steven Bethard" wrote: > > FWIW, most of the arguments against PEP 359 were along the lines of, > > "well you can do that with a metaclass already, so we don't really > > need any new syntax", but you may be able to get around those > > arguments because the decorator syntax already exists. > > On 11/15/06, Josiah Carlson wrote: > > Here's a post from Guido in response to Phillip and Greg in which he > > says more or less; someone write a PEP so that we can get them > > into 2.6 and Py3k... > > http://mail.python.org/pipermail/python-dev/2006-March/062942.html > > Thanks for the link. > > > If Tomer (or someone else) writes a PEP, I don't see why (the > > previously overlooked) class decorators shouldn't make it into > > 2.6 and 3.0 . > > So the purpose of this thread then is to write the PEP? If so, my > comments about the use cases are still valid. If they weren't > convincing use cases before, they're not likely to be convincing use > cases for a PEP. Except that they *were* convincing, which is why Guido green lighted it for 2.6 and 3.0 . One of the purposes of the PEP, like all other PEPs, is so that in the future when someone comes up with some so-called "enhancement" they can't point to a previous feature addition (like class decorators) and say, "but _they_ didn't have to write a PEP to get the feature in, why do _I_?" > Or was there another purpose of the thread? I believe that Tomer just wanted to remind people that class decorators are slated for 2.6 and 3.0 if there was a PEP, and perhaps that he is encouraging someone to write one. - Josiah From talin at acm.org Thu Nov 16 06:41:43 2006 From: talin at acm.org (Talin) Date: Wed, 15 Nov 2006 21:41:43 -0800 Subject: [Python-3000] Class Decorators + ctypes (Was: yes to class decorators) In-Reply-To: <1d85506f0611151506n48e1ee48q91de76b05754435a@mail.gmail.com> References: <1d85506f0611151506n48e1ee48q91de76b05754435a@mail.gmail.com> Message-ID: <455BFA17.1000205@acm.org> tomer filiba wrote: > here are some usecases to class decorators: What I would like is a way to use class decorators, combined with ctypes, to effectively 'declare' a C structure using Python syntax. Right now you can use ctypes to declare a C struct, but the syntax isn't very Python-like: class RECT(Structure): _fields_ = [("upperleft", POINT), ("lowerright", POINT)] Even with class decorators, however, there is still a missing piece - there's no way to decorate a variable declaration. That's because there's no equivalent of 'def' for variables. (Thinking about it, it does seem like there's an asymmetry here - you can define functions using 'def' or using assignment, but you can define variables only using assignment.) Here's a more specific proposal: The token '->' has been proposed (as part of the argument decorator syntax) as a way to indicate the return type of a function definition. What if a similar syntax could be used to define and create a class variable? Here's an example. I realize that the syntax may look strange, but it's all a logical extension of things that have already been proposed and for the most part accepted: @cstruct class CPoint: def x -> float def y -> float Under the hood, here's what's happening: 1) 'def' without an argument list defines a variable instead of a function. 2) The '-> float' is merely advisory - it has no meaning except to the decorator, just as has been proposed with argument decorators. 3) The @cstruct decorator changes the way that the class is evaluated, adding an implicit decorator to each definition. In this case, it evaluates the argument decorators and adds a field definition for each one. One unresolved issue is what exactly gets assigned to 'x' and 'y' at the time the class definition is evaluated. Ordinarily, 'def' adds a named entry to the class dict. In a normal function declaration, the part after the colon defines a suite, which is the value assigned to the name. I first thought about allowing the same colon to be used to define the value to assign to the variable, i.e.: def x -> float: 0 However, this is inconsistent with the use of the colon in other places in Python, which generally introduces a suite. An alternative would be to use an assignment operator: def x -> float = 0 The only difference between this and simply 'x = 0' is that the 'def' statement, unlike an assignment, can be decorated. A different approach would be to say that the class decorator can overload the '__setattr__' method on the class itself during class creation: @cstruct class CPoint: x = FLOAT y = FLOAT I think that you would *only* want to do this override during the initial evaluation of the class. Once the class was fully created, I would think that you'd want to be able to do "CPoint.x = value" without triggering the decorator's override. (As a bit of trivia that may be of interest: The latest proposals for extending the C++ language allow a function return type to be declared using the "(args) -> returntype" syntax. The primary reason for doing this is to avoid certain syntactical ambiguities and to allow support for lambda functions in C++, but there are some other advantages to doing so when using functions as template arguments.) -- Talin From talin at acm.org Thu Nov 16 06:55:15 2006 From: talin at acm.org (Talin) Date: Wed, 15 Nov 2006 21:55:15 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> Message-ID: <455BFD43.6070701@acm.org> Fredrik Lundh wrote: > I'm convinced that it's better for people to get over that silly notion > that writing x.foo() is somehow always "better" than writing foo(x), in > a language that actually supports both forms. > > (if I'd have to chose between foo(x) and x.foo(), I'd rather get foo(x) > with multiple dispatch than today's x.foo() single dispatch, but that's > probably too radical for Python 3000 ;-) I'm in agreement with this. Since OOP was first popularized, there have been various schools of thought which taught that object oriented programming was "good style", and any reversion to a more functional or procedural syntax was "bad style". I really wish that people would get over this. At the moment, there are, by my count, at least four and perhaps as many as a dozen "styles" or program organization, all of which have more or less validity in any given situation. As far as multiple dispatch goes: I would agree here as well, especially when we talk about binary operations. For example, suppose we have two objects, a and b, which are of types A and B respectively. And suppose each of those types overloads both __add__ and __radd__: class A: def __add__( self, other ): ... def __radd__( self, other ): ... class B: def __add__( self, other ): ... def __radd__( self, other ): ... a = A() b = B() print a + b # __add__ wins over __radd__, but should it? print b + a My conclusion: Single dispatch systems are a poor way to specify operator overloading. Compare this to the corresponding generic dispatch case: @generic( A, A ) def __add__( p0, p1 ): ... @generic( A, B ) def __add__( p0, p1 ): ... @generic( B, B ) def __add__( p0, p1 ): ... @generic( B, A ) def __add__( p0, p1 ): ... With multiple dispatch, its easy to spell out what will happen in each possible permutation of arguments - and if you get an ambiguity, the dispatcher will let you know about it, instead of simply choosing an implementation arbitrarily. -- Talin From gsakkis at rutgers.edu Thu Nov 16 07:32:36 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Thu, 16 Nov 2006 01:32:36 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <455BFD43.6070701@acm.org> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> <455BFD43.6070701@acm.org> Message-ID: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> On 11/16/06, Talin wrote: > As far as multiple dispatch goes: I would agree here as well, especially > when we talk about binary operations. For example, suppose we have two > objects, a and b, which are of types A and B respectively. And suppose > each of those types overloads both __add__ and __radd__: > > class A: > def __add__( self, other ): > ... > def __radd__( self, other ): > ... > > class B: > def __add__( self, other ): > ... > def __radd__( self, other ): > ... > > a = A() > b = B() > > print a + b # __add__ wins over __radd__, but should it? > print b + a > > My conclusion: Single dispatch systems are a poor way to specify > operator overloading. Compare this to the corresponding generic dispatch > case: > > @generic( A, A ) > def __add__( p0, p1 ): > ... > > @generic( A, B ) > def __add__( p0, p1 ): > ... > > @generic( B, B ) > def __add__( p0, p1 ): > ... > > @generic( B, A ) > def __add__( p0, p1 ): > ... > > With multiple dispatch, its easy to spell out what will happen in each > possible permutation of arguments - and if you get an ambiguity, the > dispatcher will let you know about it, instead of simply choosing an > implementation arbitrarily. Interesting idea, certainly not mainstream for the moment though. Short question: how do multiple dispatch systems deal with the combinatorial explosion ? I guess there should be a way to avoid having to spell out all combinations every time, or it will soon be unpractical. George From talin at acm.org Thu Nov 16 07:37:31 2006 From: talin at acm.org (Talin) Date: Wed, 15 Nov 2006 22:37:31 -0800 Subject: [Python-3000] A plea for anonymous functions Message-ID: <455C072B.503@acm.org> From the beat-that-dead-horse department: I've never really been satisfied with the outcome of the discussion on lambda and anonymous functions. As much as I love the use of syntactic indentation in Python, it does have one serious drawback, which is that it creates a wall between the realms of 'statements' and 'expressions'. Statements are delimited by line breaks (generally), while expressions are free-format. The result is that while statements can contain expressions, expressions cannot contain statements. None of the scripting languages which are direct competitors to Python (in particular, Ruby, Javascript, Perl, Lua, etc.) have this limitation, and all of them are able to take very powerful advantage of the ability to freely mix statements within expressions. Javascript, for example, doesn't need special syntactical sugar for generator expressions: /* Return a list of the square of numbers. ('map' appears courtesy of Mochikit.js) */ result = map( [1, 2, 3, 4], function (a) { return a * a; }) No, its not quite as concise as a generator expression, but it is far more general in what it can do. Now, there are those who say "Just use a named function", but that argument doesn't hold - because the *same* argument can be levied against generator expressions, the 'with' statement, and a number of other recent syntactical innovations in Python. From my point of view, both 'with' and generator expressions are limited, special-case solutions to a general problem - the desire to be able to use and manipulate unnamed blocks of code as first-class objects. Conversely, the same arguments that make 'with' important enough to change Python syntax - instead of just saying 'use a named function' - apply here as well, only in this case we're not talking about a specific use case, but rather a whole class of use cases which has yet to be fully explored. There are also those who would claim that, given the lack of use of 'lambda' in existing Python code, that there's no compelling use case. To this, I have two counter-arguments: 1) This is IMHO a chicken-and-egg problem. Anonymous functions aren't a solution to an existing problem, so much as they are an opportunity to explore new kinds of solutions to new kinds of problems in Python. As long as functions must be named (or crippled, as in the case of Lambda), people are going to find workarounds. 2) Many of those 'workarounds' are going to come back in the form of PEPs to add new language features. I predict that we'll see a never-ending parade of enhancement proposals - lazy evaluation expressions, asynchronous callback expressions, and so on - all of which are in a sense compensating for the lack of the ability to treat code like data. Instead of attempting to address the problem piecemeal, and thereby obfuscating the clean, elegant minimalism of Python (more than it already has been), I say we cut to the chase and deal with the root problem. Of course, I realize that this isn't an easy problem to solve - its inherent in Python's syntax choices, and certainly I don't want to give those up. But I want to specifically argue against the idea that "it's hard to solve, and it's not important, so let's not bother with it". My argument is that it may be hard, but it *is* important. I'm going to refrain from adding any hare-brained syntax proposals of my own here - I'm more interested in trying to convince people that this is a problem worth bothering with. I would like, however, to be clear in defining the parameters of the problem I would like to see addressed. I am not arguing *specifically* for anonymous functions of the 'lambda' type, although that would be one possible solution to the problem. What I want is "the ability to treat anonymous code blocks as first-class objects", which is a larger space of solutions than just lambda functions. For example, a solution that didn't involve parameter binding would not be a lambda function, but would still meet my criteria. -- Talin From jackdied at jackdied.com Thu Nov 16 07:31:18 2006 From: jackdied at jackdied.com (Jack Diederich) Date: Thu, 16 Nov 2006 01:31:18 -0500 Subject: [Python-3000] yes to class decorators In-Reply-To: <1d85506f0611151506n48e1ee48q91de76b05754435a@mail.gmail.com> References: <1d85506f0611151506n48e1ee48q91de76b05754435a@mail.gmail.com> Message-ID: <20061116063118.GC26748@performancedrivers.com> On Thu, Nov 16, 2006 at 01:06:04AM +0200, tomer filiba wrote: [tomer filiba wrote stuff] I was hoping this thread would go unremarked but it looks like it is picking up some steam. So I'll throw in my two bits: Please for the love of god abandon this thread. It has been done. You weren't there. If you were there you have already said your piece. If you weren't there you are wet behind the ears and won't be contributing anything meaningful. If you really want to post then first: 1) Please go to gmane.org and read all the threads that match "class decorators" in the group "gmane.comp.python.devel" before posting. .. and then .. 2a) Feel free to suggest ideas for a class decorator library (which is what I think Filiba was offering) 2b) If you want to suggest a new language feature like "make" then write a PEP like the "make" folks. 2c) Don't post I realize this is the py3k list and asking people to keep their crazy to themselves is an act of utopian optimism but I'm asking anyway. Don't poison the well for a simple, consistent, and pythonic feature like class decorators by piggy backing your amazing new idea on its adoption. Did I mention I care about this feature and don't want a raft of crazies screwing it up with their own vices? -Jack NB, if this thread is allowed to die peacefully I'll post a patch by the new year (mainly because Christmas is the next time I'll have a couple days free). From fredrik at pythonware.com Thu Nov 16 08:33:50 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 16 Nov 2006 08:33:50 +0100 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <455C072B.503@acm.org> References: <455C072B.503@acm.org> Message-ID: Talin wrote: > Javascript, for example, doesn't need special syntactical sugar for > generator expressions: > > /* Return a list of the square of numbers. > ('map' appears courtesy of Mochikit.js) */ > > result = map( [1, 2, 3, 4], function (a) { > return a * a; > }) > > No, its not quite as concise as a generator expression, but it is far > more general in what it can do. Are you sure you know what a generator expression is? Mochikit's map is just a JavaScript version of Python's function with the same name; it takes an Array or an Array-like object, maps it though a function, and returns an Array. Compare http://mochikit.com/doc/html/MochiKit/Base.html#fn-map with http://effbot.org/pyref/map.htm > Now, there are those who say "Just use a named function", but that > argument doesn't hold - because the *same* argument can be levied > against generator expressions, the 'with' statement, and a number of > other recent syntactical innovations in Python. Oh, please: Generator expressions (which you might need to study a bit further) is generalization of list comprehensions (which is turn is syntactical sugar for a for-in statement); and the the "with" statement was added mostly because it's *not* very practical to wrap try/finally handlers in functions; see http://online.effbot.org/2006_10_01_archive.htm#with for more on this. > From my point of view, both 'with' and generator expressions are > limited, special-case solutions to a general problem - the desire to be > able to use and manipulate unnamed blocks of code as first-class > objects. Python has two full block constructs: "for-in" (which has been in there from the start) and "with" (added in 2.5). The "for-in" block construct is discussed here: http://online.effbot.org/2006_11_01_archive.htm#for As I mention in a comment in the with-article, it's all about use cases: do 'for-in' blocks and 'with' blocks cover most of the use cases for which a full-blown callback isn't a conceptually cleaner thing to use anyway? The current Python hypothesis is 'yes, they do'. To prove that they don't, you *have* to provide use cases, and show how they would look and feel. Hand-waving arguments and vague promises of "once you reach my level of sophistication, you'll understand" isn't good enough. (And frankly, it would be *wonderful* if someone could come up with a new proposal that actually enabled Python programmers to do things they *cannot* do today, instead of just rehashing old "let's move the sofa over there" threads. How about doing something along the lines of LINQ's "give me this expression as an AST so I can optimize it myself" model or looking at COmega's parallelization/synchronization stuff or digging deeper into how PJE's RuleDispatch could be fit into Python or stealing some cool idea from some functional research language that I haven't even heard of. I want to get *new* things when I upgrade from 2.X to 3.0, not just silly syntax tweaks that would only give me the ability to change two lines of Python code to one line of code plus a comment that explains what the heck that line is doing. Let's do some *hard* stuff, for a change.) From krstic at solarsail.hcs.harvard.edu Thu Nov 16 09:33:23 2006 From: krstic at solarsail.hcs.harvard.edu (=?UTF-8?B?SXZhbiBLcnN0acSH?=) Date: Thu, 16 Nov 2006 03:33:23 -0500 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: References: <455C072B.503@acm.org> Message-ID: <455C2253.5090909@solarsail.hcs.harvard.edu> Fredrik Lundh wrote: > And frankly, it would be *wonderful* if someone could come up with a > new proposal that actually enabled Python programmers to do things they > *cannot* do today, instead of just rehashing old "let's move the sofa > over there" threads. Hear, hear. > I want to get *new* things when I upgrade from > 2.X to 3.0, not just silly syntax tweaks that would only give me the > ability to change two lines of Python code to one line of code plus a > comment that explains what the heck that line is doing. >From my understanding of the purpose of 3.0, you're looking at the wrong release. 3.0 is the "accumulated cruft removal" release, with features becoming the focus of the 3.1 cycle and subsequent releases. > Let's do some *hard* stuff, for a change. I couldn't agree more. The fundamental problem that I see, though, is that Guido seems strongly opposed to adding any additional metaprogramming features to the language [0], and solving much of the hard stuff elegantly and without special-cased, one-off solutions, requires these. LINQ-like features certainly require having first-class code blocks. And for a couple of months, I've been jotting down notes, in preparation for a PEP, on how best to add multiprocessing support to Python given the GIL and the fact that it's not going away in the near future. There are a number of attractive solutions -- some of them quite beautiful, and I can provide examples -- that can be used here, but they all make the same requirement as LINQ if they're to not look like utter warts. The bottom line, I think, is that we need generic language support for "hard stuff", and as Lispers figured out a while back, that support /is/ metaprogramming. Guido does not wish to make Python syntax any more programmable, however, while at the same time features like metaclasses tempt us just enough to feel the possibilities are within reach. [0] See second to last paragraph: http://mail.python.org/pipermail/python-3000/2006-April/000286.html -- Ivan Krsti? | GPG: 0x147C722D From fredrik at pythonware.com Thu Nov 16 09:55:11 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 16 Nov 2006 09:55:11 +0100 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <455C2253.5090909@solarsail.hcs.harvard.edu> References: <455C072B.503@acm.org> <455C2253.5090909@solarsail.hcs.harvard.edu> Message-ID: Ivan Krsti? wrote: >> Let's do some *hard* stuff, for a change. > > I couldn't agree more. The fundamental problem that I see, though, is > that Guido seems strongly opposed to adding any additional > metaprogramming features to the language [0], and solving much of the > hard stuff elegantly and without special-cased, one-off solutions, > requires these. > > LINQ-like features certainly require having first-class code blocks. I'm not convinced that they do -- you can emulate LINQ today with generators and iterator tools and good old for-in statements: http://sayspy.blogspot.com/2006/02/why-python-doesnt-need-something-like.html but as noted in the comments, what's missing is a way for the container to do query optimizations based on the *entire* filter expression. if we can come up with a decent mechanism (*) for that, the existing for-in block construct is good enough for the rest. more later. *) there are plenty of hacks to address parts of this; everything from algebra on custom objects (used by various SQL construction kits, RE construction kits, etc), code that analyzes an expression by passing special AST-building objects through it (used in PIL's point method, for example), and code that requires you to pass in the expression as a text string (or a ready-made "compiler" module AST-tree). From talin at acm.org Thu Nov 16 09:58:47 2006 From: talin at acm.org (Talin) Date: Thu, 16 Nov 2006 00:58:47 -0800 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: References: <455C072B.503@acm.org> Message-ID: <455C2847.50903@acm.org> Fredrik Lundh wrote: > Talin wrote: > >> Javascript, for example, doesn't need special syntactical sugar for >> generator expressions: >> >> /* Return a list of the square of numbers. >> ('map' appears courtesy of Mochikit.js) */ >> >> result = map( [1, 2, 3, 4], function (a) { >> return a * a; >> }) >> >> No, its not quite as concise as a generator expression, but it is far >> more general in what it can do. > > Are you sure you know what a generator expression is? Mochikit's map is > just a JavaScript version of Python's function with the same name; it > takes an Array or an Array-like object, maps it though a function, and > returns an Array. Compare You are correct that 'map' isn't equivalent to a generator expression - my mistake. However, replacing 'map' with 'itertools.imap' gives you something roughly equivalent. > To prove that they don't, you *have* to provide use cases, and show how > they would look and feel. Hand-waving arguments and vague promises of > "once you reach my level of sophistication, you'll understand" isn't > good enough. The problem with providing use cases is that each one individually doesn't carry much weight. When I do Javascript/AJAX programming, I use anonymous functions quite a bit, as in the following example: do_command( 'get_synset_links', {}, function( response_data ) { format_expanded( li.content, response_data.related ) }) 'do_command' calls a function on the server side via an async HTTP request, and calls the callback when the data is ready. By using an anonymous function as the callback, I can group the request for the data and the use of that data together in a logical way. Could I do the same thing using a named callback? Sure. Would it be as easy to read and as clear? Not in my opinion. With a named callback, we would have to put the code that uses the data in a different place than the code that fetches the data. We would have to write the code so that the part that executes later appears earlier in the source file. But it doesn't matter, because no matter what cases I come up with, someone will claim "You're just saving 2 lines of code". The fact that it saves 2 lines of code in many different places in my program can't easily be illustrated in an email message. > (And frankly, it would be *wonderful* if someone could come up with a > new proposal that actually enabled Python programmers to do things they > *cannot* do today, instead of just rehashing old "let's move the sofa > over there" threads. How about doing something along the lines of > LINQ's "give me this expression as an AST so I can optimize it myself" > model or looking at COmega's parallelization/synchronization stuff or > digging deeper into how PJE's RuleDispatch could be fit into Python or > stealing some cool idea from some functional research language that I > haven't even heard of. I want to get *new* things when I upgrade from > 2.X to 3.0, not just silly syntax tweaks that would only give me the > ability to change two lines of Python code to one line of code plus a > comment that explains what the heck that line is doing. Let's do some > *hard* stuff, for a change.) I would disagree with one bit: I think most of those things can be done in Python today, they are just complicated and ugly under the current syntax. As much as people decry 'syntactic sugar', I would argue that those things you mentioned (even the AST example, although it's tricky) really just need sugar to make them feasible and useful to ordinary programmers, rather than any fundamental change to the execution model. In my experience, most of the 'hard stuff' is built out of powerful, low-level language primitives - most of which Python already has. Those low-level primitives don't seem particularly powerful until you start combining them in synergetic ways. This is particularly well illustrated in Scheme, where many of the modern programming features - such as OOP-style single method dispatch - can be 'built up' out of more fundamental features like closures. And I would argue that being able to treat 'code as data' is one of those fundamentals. We can already do this to some extent, but there's a long way to go still as far as intuitiveness and ease of use goes. (Although I just read the last paragraph Ivan Krsti?'s response as I was typing this, and I think he says it better than I can, so I'll stop.) -- Talin From krstic at solarsail.hcs.harvard.edu Thu Nov 16 10:35:19 2006 From: krstic at solarsail.hcs.harvard.edu (=?UTF-8?B?SXZhbiBLcnN0acSH?=) Date: Thu, 16 Nov 2006 04:35:19 -0500 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: References: <455C072B.503@acm.org> <455C2253.5090909@solarsail.hcs.harvard.edu> Message-ID: <455C30D7.1020701@solarsail.hcs.harvard.edu> Fredrik Lundh wrote: > I'm not convinced that they do -- you can emulate LINQ today with > generators and iterator tools and good old for-in statements: No, you can't. You can emulate a tiny subset of it, as you yourself note. For people following from home: when you query a database with LINQ, C# can construct the actual SQL *query* by introspecting the AST; you can't do that without first-class code blocks, unless you get ugly. And because the language is aware of what you're querying, it can compile the same LINQ query to SQL, or XQuery, or something else entirely, as needed. > *) there are plenty of hacks to address parts of this; everything from > algebra on custom objects (used by various SQL construction kits, RE > construction kits, etc), code that analyzes an expression by passing > special AST-building objects through it (used in PIL's point method, for > example), and code that requires you to pass in the expression as a text > string (or a ready-made "compiler" module AST-tree). Please distinguish hacks from language support, and note what I said: "... but they all make the same requirement as LINQ if they're to not look like utter warts." So yes, these can be done (to a degree), but they're all nasty solutions, and the fact we have a legitimate need for them -- I brought up SQLObject before in this context -- tells me people are already making Python behave non-obviously (which was Guido's fear), just in exceedingly ugly and arbitrarily limited ways. -- Ivan Krsti? | GPG: 0x147C722D From fredrik at pythonware.com Thu Nov 16 10:50:56 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 16 Nov 2006 10:50:56 +0100 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <455C2847.50903@acm.org> References: <455C072B.503@acm.org> <455C2847.50903@acm.org> Message-ID: Talin wrote: > The problem with providing use cases is that each one individually > doesn't carry much weight. When I do Javascript/AJAX programming, I use > anonymous functions quite a bit, as in the following example: > > do_command( 'get_synset_links', {}, function( response_data ) { > format_expanded( li.content, response_data.related ) > }) > > 'do_command' calls a function on the server side via an async HTTP > request, and calls the callback when the data is ready. By using an > anonymous function as the callback, I can group the request for the data > and the use of that data together in a logical way. Your specific example would probably be written: server.get_synset_links( lambda rec: format_expanded(li.content, rec.related) ) in today's Python, which I find a bit easier to read than your slightly bloated JavaScript example. How do you handle errors, btw? > Could I do the same thing using a named callback? Sure. Would it be as > easy to read and as clear? Well, if you find that easy to read and clear, I suspect you're more of a JavaScript programmer than a Python programmer. Here's how a more general form of the above would look in Python: response = (yield server.get_synset_links()) format_expanded(li.content, response.related) or def handle_response(record): format_expanded(li.content, response.related) server.get_synset_link(handle_response) or perhaps with async server.get_synset_link() as response: format_expanded(li.content, response.related) (all of which are shorter than your JavaScript example, by the way) > But it doesn't matter, because no matter what cases I come up with, > someone will claim "You're just saving 2 lines of code". No, I said that under your proposal, you're trading two lines of code that looks like Python for two (or three) lines of code that looks like something else. > The fact that it saves 2 lines of code in many different places in > my program can't easily be illustrated in an email message. Well, this far, you haven't even been able to illustrate how your unspecified block syntax would have saved you a single line even in your three-line example. That's not a good start, really. > I would disagree with one bit: I think most of those things can be done > in Python today, they are just complicated and ugly under the current > syntax. As much as people decry 'syntactic sugar', I would argue that > those things you mentioned (even the AST example, although it's tricky) > really just need sugar to make them feasible and useful to ordinary > programmers, rather than any fundamental change to the execution model. So show us some sugar, then. After all, Python's design is driven by by an urge to simplify real usage patterns that's been observed in the wild, rather than theoretically-useful-for-everything-maybe language constructs that's been dreamed up in some laboratory. And all the things I've mentioned *are* used in real code. Also see the "Patterns are signs of weakness in programming languages" discussions: http://newbabe.pobox.com/~mjd/blog/prog/design-patterns.html and note that Python's usually aiming for "(nearly) invisible", not "formal", in Norvig's taxonomy. Also see the part about use cases for block constructs in my original post to this thread. From fredrik at pythonware.com Thu Nov 16 11:01:05 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 16 Nov 2006 11:01:05 +0100 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <455C30D7.1020701@solarsail.hcs.harvard.edu> References: <455C072B.503@acm.org> <455C2253.5090909@solarsail.hcs.harvard.edu> <455C30D7.1020701@solarsail.hcs.harvard.edu> Message-ID: Ivan Krsti? wrote: >> I'm not convinced that they do -- you can emulate LINQ today with >> generators and iterator tools and good old for-in statements: > > No, you can't. my sentence didn't end at that colon, though. > For people following from home: when you query a database with > LINQ, C# can construct the actual SQL *query* by introspecting the AST; > you can't do that without first-class code blocks of course you can. all you need is the AST for the *query*, not the rest of the code. for record in (): process record gen = (for record in ()) e.g. something like for record in (select table: ): process record where the table is able to either execute the generator expression on itself, or analyze the AST and do an optimized query. > And because the language is aware of what you're querying, it can > compile the same LINQ query to SQL, or XQuery, or something else > entirely, as needed. well, Python's a dynamic language, so that has to be done by the collection, on the fly. but that's not a problem, really. >> *) there are plenty of hacks to address parts of this; everything from >> algebra on custom objects (used by various SQL construction kits, RE >> construction kits, etc), code that analyzes an expression by passing >> special AST-building objects through it (used in PIL's point method, for >> example), and code that requires you to pass in the expression as a text >> string (or a ready-made "compiler" module AST-tree). > > Please distinguish hacks from language support oh, my use of "decent mechanism" implied that the things I labelled as "hacks" in the footnote weren't decent enough. if you read my post again, you'll find that I said pretty much the same thing as you're saying, minus the "we absolutely must have full support for anonymous blocks to be able to do this" part. From krstic at solarsail.hcs.harvard.edu Thu Nov 16 11:10:40 2006 From: krstic at solarsail.hcs.harvard.edu (=?UTF-8?B?SXZhbiBLcnN0acSH?=) Date: Thu, 16 Nov 2006 05:10:40 -0500 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: References: <455C072B.503@acm.org> <455C2253.5090909@solarsail.hcs.harvard.edu> <455C30D7.1020701@solarsail.hcs.harvard.edu> Message-ID: <455C3920.4030509@solarsail.hcs.harvard.edu> Fredrik Lundh wrote: > for record in (select table: ): > process record This is precisely what I had in mind when I referred to "special-cased, one-off solutions" two posts back. > if you read my post again, you'll find that I said pretty much the same > thing as you're saying, minus the "we absolutely must have full support > for anonymous blocks to be able to do this" part. That's not what I'm saying. What I'm saying is "we absolutely must have full support for anonblocks to be able to do this *elegantly*", i.e. while avoiding special-cased solutions. The LINQ special case given above won't help me in building proper multiprocessing support. Now, maybe special-cased solutions are preferred to more metaprogramming, in which case there's not much point in continuing the discussion. -- Ivan Krsti? | GPG: 0x147C722D From fredrik at pythonware.com Thu Nov 16 11:22:52 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 16 Nov 2006 11:22:52 +0100 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <455C3920.4030509@solarsail.hcs.harvard.edu> References: <455C072B.503@acm.org> <455C2253.5090909@solarsail.hcs.harvard.edu> <455C30D7.1020701@solarsail.hcs.harvard.edu> <455C3920.4030509@solarsail.hcs.harvard.edu> Message-ID: Ivan Krsti? wrote: > That's not what I'm saying. What I'm saying is "we absolutely must have > full support for anonblocks to be able to do this *elegantly*", i.e. > while avoiding special-cased solutions. As I've tried to say several times in my other posts in this thread, from a theoretical computer science perspective, Python's all about "special cases". Python's design values elegant solutions for groups of *practical* problems observed in the wild, over hyper-generalized solutions to problems designed in a laboratory. That's what makes the language so darn practical to use ;-) From talin at acm.org Thu Nov 16 11:50:50 2006 From: talin at acm.org (Talin) Date: Thu, 16 Nov 2006 02:50:50 -0800 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: References: <455C072B.503@acm.org> <455C2847.50903@acm.org> Message-ID: <455C428A.6020207@acm.org> Fredrik Lundh wrote: > Talin wrote: > >> The problem with providing use cases is that each one individually >> doesn't carry much weight. When I do Javascript/AJAX programming, I use >> anonymous functions quite a bit, as in the following example: >> >> do_command( 'get_synset_links', {}, function( response_data ) { >> format_expanded( li.content, response_data.related ) >> }) >> >> 'do_command' calls a function on the server side via an async HTTP >> request, and calls the callback when the data is ready. By using an >> anonymous function as the callback, I can group the request for the data >> and the use of that data together in a logical way. > > Your specific example would probably be written: > > server.get_synset_links( > lambda rec: format_expanded(li.content, rec.related) > ) Only if the code in the block is a one-liner. > in today's Python, which I find a bit easier to read than your slightly > bloated JavaScript example. > > How do you handle errors, btw? For this app, internally in the do_command function. In any case, error handling code can be assumed to have been omitted for clarity. >> Could I do the same thing using a named callback? Sure. Would it be as >> easy to read and as clear? > > Well, if you find that easy to read and clear, I suspect you're more of > a JavaScript programmer than a Python programmer. Here's how a more > general form of the above would look in Python: > > response = (yield server.get_synset_links()) > format_expanded(li.content, response.related) > > or > > def handle_response(record): > format_expanded(li.content, response.related) > server.get_synset_link(handle_response) > > or perhaps > > with async server.get_synset_link() as response: > format_expanded(li.content, response.related) None of these are functionally equivalent to my code example, if I am understanding them correctly. But I guess I need to explain a bit more about my example. First, 'get_synset_links' is a URL, not a method name. (More technically, it's a fragment of a URL). I suppose you could play tricks with __getitem__ to turn a method call into a URL, but I wouldn't do that - for one thing, not all URLs can be represented as method names. More importantly, the flow of execution is different: do_command( stuff ) { // This code executes when the HTTP response from the // server returns. ... code ... } // This code executes immediately after the request has // been sent. ... code ... In other words, 'do_command' sends a request to the server and continues processing after the block. When the response returns, the code inside the block is executed. I don't think you can emulate this behavior with either yield or with - they are both synchronous operations. So I can see how your example #2 might be made to work this way, but not #1 or #3. And I'm not convinced that #2 is really better. More generally, the notion here is to implement a rough kind of continuation, based on external asynchronous events. There are lots and lots of potential use cases for code that handles asynchronous responses to program events. > (all of which are shorter than your JavaScript example, by the way) > >> But it doesn't matter, because no matter what cases I come up with, >> someone will claim "You're just saving 2 lines of code". > > No, I said that under your proposal, you're trading two lines of code > that looks like Python for two (or three) lines of code that looks like > something else. Lets not get confused here - I'm illustrating in Javascript code because this is something that's easy to do in Javascript (or Lua or Perl or whatever) but hard to do in Python. Since I haven't actually proposed what it would look like in Python, I'm not sure that it makes sense to argue that it doesn't "look like" Python. That remains to be seen. > > The fact that it saves 2 lines of code in many different places in > > my program can't easily be illustrated in an email message. > > Well, this far, you haven't even been able to illustrate how your > unspecified block syntax would have saved you a single line even in your > three-line example. That's not a good start, really. Lets try that one again, now that I (hopefully) have given a clearer explanation of what I am trying to do. >> I would disagree with one bit: I think most of those things can be done >> in Python today, they are just complicated and ugly under the current >> syntax. As much as people decry 'syntactic sugar', I would argue that >> those things you mentioned (even the AST example, although it's tricky) >> really just need sugar to make them feasible and useful to ordinary >> programmers, rather than any fundamental change to the execution model. > > So show us some sugar, then. After all, Python's design is driven by by > an urge to simplify real usage patterns that's been observed in the > wild, rather than theoretically-useful-for-everything-maybe language > constructs that's been dreamed up in some laboratory. And all the > things I've mentioned *are* used in real code. Well, I'm not going to give solutions for all of the examples you raised, because (a) its off-topic, and (b) it's not necessary. But I will take two examples from your list: The LINQ example and the RuleDispatch example. For the LINQ example, all I need to do is point to SQLObject, which does something very similar - it builds an SQL query using Python expression syntax. We could improve SQLObjects's syntax quite a bit by allowing overloading of the 'and' and 'or' operator - right now, it has to use '&' and '|', which have the wrong precedence and require extra parens. For RuleDispatch, we already know it works in Python, the only question is how to make it easier to use. And that's been the subject of a number of threads on this list, of which there are two parts: How to write the dispatcher, and how to express the dispatch constrains in syntax. Writing the dispatcher isn't a problem - I wrote one for my embedded mini-language, Saga, in a day or two just by looking up the Chambers/Chen paper on ACM library. So really it all boils down to syntax. > Also see the "Patterns are signs of weakness in programming languages" > discussions: > > http://newbabe.pobox.com/~mjd/blog/prog/design-patterns.html OK I read it - I think the author is overgeneralizing. > and note that Python's usually aiming for "(nearly) invisible", not > "formal", in Norvig's taxonomy. > > Also see the part about use cases for block constructs in my original > post to this thread. -- Talin From fredrik at pythonware.com Thu Nov 16 12:03:05 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 16 Nov 2006 12:03:05 +0100 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <455C428A.6020207@acm.org> References: <455C072B.503@acm.org> <455C2847.50903@acm.org> <455C428A.6020207@acm.org> Message-ID: Talin wrote: > Since I haven't actually proposed what it would look like in Python, I'm > not sure that it makes sense to argue that it doesn't "look like" > Python. That remains to be seen. wow. From tomerfiliba at gmail.com Thu Nov 16 12:24:13 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Thu, 16 Nov 2006 13:24:13 +0200 Subject: [Python-3000] yes to class decorators Message-ID: <1d85506f0611160324x5d70967aha46c3eff47e85a96@mail.gmail.com> several things: [Josiah] > I believe that Tomer just wanted to remind people that class decorators > are slated for 2.6 and 3.0 if there was a PEP, and perhaps that he is > encouraging someone to write one. yeah, i just wanted to bring it up again. however, i do have some spare time at the moment, so i can write the pep myself. but before someone writes a pep, it's better to discuss the issue at hand. currently, to my understanding, this is what we have: ("@" expression "\n")* "class" name ["(" expression "):"] clause we should find some more/better use-cases than what i came up with, and show why class decorators are the right solution, and compare with function decorators. we should also note that it's possible to replace "__metaclass__" by a @metaclass decorator. of course this requires more thinking. - - - - - - - - - [Jack Diederich] > I realize this is the py3k list and asking people to keep their > crazy to themselves is an act of utopian optimism but I'm asking > anyway. MWHAHAAHAHAHAHAHAHA!!!111!!^&%^#%&@ *shoves own hand into the oven* > Did I mention I care about this feature and don't want > a raft of crazies screwing it up with their own vices? i don't get your point. i DID read many posts about class decorators, the most recent of which i've seen were at least half a year old. since we have Guido's blessing on the subject, i wanted to get the wheels moving. of course i couldn't read EVERYTHING that was said about class decorators, so i added a couple of use-cases i found. in fact, the reason why i posted yesterday was because i wrote a singleton class, and wished i had @singleton instead of class Foo(object): pass Foo = Foo() so if we put aside your Stalinistic behavior for the moment, do YOU want to write the pep? please do. i don't see why this thread bothers you so much, but you could have prevented it by writing a pep months ago. now isn't that motivating? :) -tomer From fredrik at pythonware.com Thu Nov 16 12:34:33 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 16 Nov 2006 12:34:33 +0100 Subject: [Python-3000] yes to class decorators In-Reply-To: <1d85506f0611160324x5d70967aha46c3eff47e85a96@mail.gmail.com> References: <1d85506f0611160324x5d70967aha46c3eff47e85a96@mail.gmail.com> Message-ID: tomer filiba wrote: >> Did I mention I care about this feature and don't want >> a raft of crazies screwing it up with their own vices? > > i don't get your point. maybe your news server is dropping posts ? From meyer at acm.org Thu Nov 16 12:36:47 2006 From: meyer at acm.org (Andre Meyer) Date: Thu, 16 Nov 2006 12:36:47 +0100 Subject: [Python-3000] yes to class decorators In-Reply-To: <1d85506f0611160324x5d70967aha46c3eff47e85a96@mail.gmail.com> References: <1d85506f0611160324x5d70967aha46c3eff47e85a96@mail.gmail.com> Message-ID: <7008329d0611160336u5e8e71fex68a9b376ad5fb1af@mail.gmail.com> On 11/16/06, tomer filiba wrote: > > > we should find some more/better use-cases than what i came > up with, and show why class decorators are the right solution, > and compare with function decorators. A while ago I was trying to implement the Singleton or Borg design pattern and thought it could be elegantly made accessible via a class decorator. It proved otherwise. Maybe this is a useful use-case for you. regards Andr? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061116/99f34dde/attachment.htm From fredrik at pythonware.com Thu Nov 16 12:44:30 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 16 Nov 2006 12:44:30 +0100 Subject: [Python-3000] native support for multiple dispatch in 3.0 ? Message-ID: since it seems to be impossible to bring up new ideas in subthreads, I though I'd give this one its own thread: - has there been any discussions on making RuleDispatch-type mechanisms available on a more fundamental level? could someone perhaps come up with a list of real-life uses for PJE's existing dispatcher implementation? or should we go all the way: how about using this mechanism for *all* dispatching, even the usual dispatch-on-instance handler? and poly- morphic builtins too, while we're at it? (the first person to bring up anonymous blocks in this thread will be duly *plonked* ;-) From solipsis at pitrou.net Thu Nov 16 14:49:10 2006 From: solipsis at pitrou.net (Antoine) Date: Thu, 16 Nov 2006 14:49:10 +0100 (CET) Subject: [Python-3000] A plea for anonymous functions In-Reply-To: References: <455C072B.503@acm.org> <455C2847.50903@acm.org> Message-ID: <52661.62.39.9.251.1163684950.squirrel@webmail.nerim.net> Hi, > Well, if you find that easy to read and clear, I suspect you're more of > a JavaScript programmer than a Python programmer. Here's how a more > general form of the above would look in Python: > > response = (yield server.get_synset_links()) > format_expanded(li.content, response.related) This assumes there is an underlying event loop of some sort that accepts and is able to schedule generators. Much more annoyingly, all your "yield"'s must be in the same scope, they can't be spread in various subfunctions called from the generator. It is a massive limitation to writing clean code, unless your generators are very short. > def handle_response(record): > format_expanded(li.content, response.related) > server.get_synset_link(handle_response) As already mentioned, the problem with this idiom is that code that is executed *after* appears *before* the "get_synset_link". It makes the code less straightforward to write and especially to read (and having readable code is important). > with async server.get_synset_link() as response: > format_expanded(li.content, response.related) How is this one supposed to work ? What is "with async blah()" ? > No, I said that under your proposal, you're trading two lines of code > that looks like Python for two (or three) lines of code that looks like > something else. The problem is not merely related to the number of lines of code. It is a readibility problem. Having the callback enclosed in a the function call which registers it makes the code more readable than defining the callback beforehand. Saying this does *not* mean one is a Javascript programmer (I hate Javascript, but this particular point of Javascript is superior to what Python has to offer). > So show us some sugar, then. After all, Python's design is driven by by > an urge to simplify real usage patterns that's been observed in the > wild, Real usage patterns of event-driven code can be seen everywhere from GUI applications to network applications (written with Twisted). In many applications it is arguably much more common than the "try..finally" constructs that "with" attempts to simplify. I'm not sure what more is needed here? Addresses of SVN repositories? > Also see the "Patterns are signs of weakness in programming languages" > discussions: I would agree with this argument if I could understand how it relates to the current discussion. Are you saying anonymous functions are a "design pattern" ? (then what is *not* a design pattern ?). > and note that Python's usually aiming for "(nearly) invisible", not > "formal", in Norvig's taxonomy. What is more "invisible", including the callback straight at the point where it is registered, or defining it beforehand? The only thing I can agree with here is that anonymous functions or code blocks are very hard to design inside the current Python syntax. But saying there is no point in anonymous functions is a plain lie. Regards Antoine. From murman at gmail.com Thu Nov 16 15:06:31 2006 From: murman at gmail.com (Michael Urman) Date: Thu, 16 Nov 2006 08:06:31 -0600 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <52661.62.39.9.251.1163684950.squirrel@webmail.nerim.net> References: <455C072B.503@acm.org> <455C2847.50903@acm.org> <52661.62.39.9.251.1163684950.squirrel@webmail.nerim.net> Message-ID: > The problem is not merely related to the number of lines of code. It is a > readibility problem. Having the callback enclosed in a the function call > which registers it makes the code more readable than defining the callback > beforehand. And, unless I'm missing something, much harder to unit test. Is that really worth encouraging for handlers with complexity beyond the current lambda? -- Michael Urman http://www.tortall.net/mu/blog From fredrik at pythonware.com Thu Nov 16 15:54:20 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 16 Nov 2006 15:54:20 +0100 Subject: [Python-3000] A plea for anonymous functions References: <455C072B.503@acm.org> <455C2847.50903@acm.org> <52661.62.39.9.251.1163684950.squirrel@webmail.nerim.net> Message-ID: Antoine wrote: > > response = (yield server.get_synset_links()) > > format_expanded(li.content, response.related) > > This assumes there is an underlying event loop of some sort that accepts > and is able to schedule generators. no, it only assumes that whoever called your code is set up to schedule communication tasks on your behalf. that's how asyncronous network stuff is usually done, so I'm not sure why you're using this as a counter- argument. > Much more annoyingly, all your "yield"'s must be in the same scope, they > can't be spread in various subfunctions called from the generator dealing with nested generators is simple; a bit tedious, but far from impossible. >> def handle_response(record): >> format_expanded(li.content, response.related) >> server.get_synset_link(handle_response) > > As already mentioned, the problem with this idiom is that code that is > executed *after* appears *before* the "get_synset_link". > It makes the code less straightforward to write and especially to read > (and having readable code is important). why? it *separates* the two things, and lets you concentrate on one thing at a time. I'm far from convinced that that's a drawback, especially when both the request construction and the response handlers gets more complex than what you can fit in a line of code or three. it also makes it a *lot* easier to test and debug the callbacks, makes it easier to refactor both the request-generating code and the response handlers, makes it easier to reuse response handlers for related tasks, is self-documenting, given appropriate callback names, and generally, in my experience, leads to much better code quality. >> with async server.get_synset_link() as response: >> format_expanded(li.content, response.related) > > How is this one supposed to work ? What is "with async blah()" ? a magic thing that runs the code block behind the covers, of course. requires a non-released version of the from __future__ statement, at the time. unlike the following syntax: @server.get_synset_link() def block(response): format_expanded(li.content, response.related) which actually works in today's Python. >> No, I said that under your proposal, you're trading two lines of code >> that looks like Python for two (or three) lines of code that looks like >> something else. > > The problem is not merely related to the number of lines of code. It is a > readibility problem. Having the callback enclosed in a the function call > which registers it makes the code more readable than defining the callback > beforehand. but here we're back to the "it seems to me" kind of argumentation. show us the code, and don't just argue that you really meant to post something else when people point out how lousy your example is. > Saying this does *not* mean one is a Javascript programmer (I hate > Javascript, but this particular point of Javascript is superior to what > Python has to offer). > >> So show us some sugar, then. After all, Python's design is driven by by >> an urge to simplify real usage patterns that's been observed in the >> wild, > > Real usage patterns of event-driven code can be seen everywhere from GUI > applications to network applications (written with Twisted). > In many applications it is arguably much more common than the > "try..finally" constructs that "with" attempts to simplify. yeah, exactly. so what kind of sugar would help event-driven programmers to write better, shorter, more maintainable code ? you and tomer want more onions in your soup; perhaps there are other Twisted hackers out there with other ideas ? >> Also see the "Patterns are signs of weakness in programming languages" >> discussions: > > I would agree with this argument if I could understand how it relates to > the current discussion. Are you saying anonymous functions are a "design > pattern" ? (then what is *not* a design pattern ?). no, the code that you argue would improved by anonymous blocks implement a specific pattern, of course (a handler for an asyncronous event). you're still stuck thinking on mechanisms, I'm trying to get you to talk about use cases. > What is more "invisible", including the callback straight at the point > where it is registered, or defining it beforehand? an invisible solution would be let you write asyncronous event handlers in a way where you weren't even noticing that you were creating callbacks. I have no idea what it would look like, but ideally, it should be a bit like anonymous blocks in for-in loops: they're been in there for ages, everyone is using them, but people still claim that Python doesn't have anonymous blocks at all. From jimjjewett at gmail.com Thu Nov 16 17:37:48 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 16 Nov 2006 11:37:48 -0500 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <455C072B.503@acm.org> References: <455C072B.503@acm.org> Message-ID: On 11/16/06, Talin wrote: > From my point of view, both 'with' and generator expressions are > limited, special-case solutions to a general problem - the desire to be > able to use and manipulate unnamed blocks of code as first-class > objects. Other than your callback of a curried function, I don't think the suites really need to be anonymous; first class is enough. And even that may be overkill; I didn't see any examples that didn't boil down to "If I wrote this normally, it would execute in the wrong namespace." (I may be undervaluing the "but I haven't defined it *yet*" portion of your examples.) > Conversely, the same arguments that make 'with' important > enough to change Python syntax - instead of just saying 'use a named > function' The wrapping portions often are named. The point of "with" is to tie them together as a single named function instead of two paired functions. The center suite (inside the "with") can't always be replaced by a function, and that is because it needs to be evaluated in the same scope as the suites before and after the with block. > 2) Many of those 'workarounds' are going to come back in the form of > PEPs to add new language features. I predict that we'll see a > never-ending parade of enhancement proposals - lazy evaluation > expressions, asynchronous callback expressions, and so on - all of which > are in a sense compensating for the lack of the ability to treat code > like data. If not for the special-case treatment of locals and globals, I can't quite think of an example that couldn't be rewritten as a suite, and called with exec. Just for nostalgia... suite core: stmt1 stmt2 -jJ From gmccaughan at synaptics-uk.com Thu Nov 16 16:39:11 2006 From: gmccaughan at synaptics-uk.com (Gareth McCaughan) Date: Thu, 16 Nov 2006 16:39:11 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <455BFD43.6070701@acm.org> <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> Message-ID: <200611161539.12577.gmccaughan@synaptics-uk.com> [Talin:] > > With multiple dispatch, its easy to spell out what will happen in each > > possible permutation of arguments - and if you get an ambiguity, the > > dispatcher will let you know about it, instead of simply choosing an > > implementation arbitrarily. [George Sakkis:] > Interesting idea, certainly not mainstream for the moment though. (Who cares whether it's "mainstream"? For what it's worth, the idea has been around for decades.) > Short question: how do multiple dispatch systems deal with the > combinatorial explosion ? I guess there should be a way to avoid > having to spell out all combinations every time, or it will soon be > unpractical. What combinatorial explosion? If you restrict yourself to what can be expressed in a traditional single-dispatch OO system, there is no combinatorial explosion; the complexity is no greater than for single dispatch. Multiple dispatch allows you to express much more. Some versions of "much more" would involve defining an enormous number of methods, but that's no more a problem with multiple dispatch than the fact that multidimensional arrays can use a huge amount of memory is a problem with multidimensional arrays. -- g From solipsis at pitrou.net Thu Nov 16 18:21:31 2006 From: solipsis at pitrou.net (Antoine) Date: Thu, 16 Nov 2006 18:21:31 +0100 (CET) Subject: [Python-3000] A plea for anonymous functions In-Reply-To: References: <455C072B.503@acm.org> <455C2847.50903@acm.org> <52661.62.39.9.251.1163684950.squirrel@webmail.nerim.net> Message-ID: <33914.62.39.9.251.1163697691.squirrel@webmail.nerim.net> >> The problem is not merely related to the number of lines of code. It is >> a >> readibility problem. Having the callback enclosed in a the function call >> which registers it makes the code more readable than defining the >> callback >> beforehand. > > And, unless I'm missing something, much harder to unit test. Not harder than any nested function, or am I missing something? From solipsis at pitrou.net Thu Nov 16 18:24:38 2006 From: solipsis at pitrou.net (Antoine) Date: Thu, 16 Nov 2006 18:24:38 +0100 (CET) Subject: [Python-3000] A plea for anonymous functions In-Reply-To: References: <455C072B.503@acm.org> <455C2847.50903@acm.org> <52661.62.39.9.251.1163684950.squirrel@webmail.nerim.net> Message-ID: <43166.62.39.9.251.1163697878.squirrel@webmail.nerim.net> >> How is this one supposed to work ? What is "with async blah()" ? > > a magic thing that runs the code block behind the covers, of course. > requires a > non-released version of the from __future__ statement, at the time. ;-)) > unlike the > following syntax: > > @server.get_synset_link() > def block(response): > format_expanded(li.content, response.related) Ok, that's nearly perfect! Only "nearly" because: 1. There are cases where you want the return value of the "get_synset_link". 2. You sometimes need more than one callback, and you cannot decorate two functions at once. For example, with Twisted you can define a callback (called on success) and an errback (called on error). But I agree that decorators help to solve a lot of problems. Thanks for the answer Antoine. From exarkun at divmod.com Thu Nov 16 18:45:58 2006 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Thu, 16 Nov 2006 12:45:58 -0500 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <43166.62.39.9.251.1163697878.squirrel@webmail.nerim.net> Message-ID: <20061116174558.20948.797356069.divmod.quotient.29882@ohm> On Thu, 16 Nov 2006 18:24:38 +0100 (CET), Antoine wrote: > [snip] > >Ok, that's nearly perfect! >Only "nearly" because: >1. There are cases where you want the return value of the >"get_synset_link". 2. You sometimes need more than one callback, and you >cannot decorate two functions at once. For example, with Twisted you can >define a callback (called on success) and an errback (called on error). > >But I agree that decorators help to solve a lot of problems. > >Thanks for the answer > >Antoine. > >>> @foo().addCallback File "", line 1 @foo().addCallback ^ SyntaxError: invalid syntax >>> Jean-Paul From jason.orendorff at gmail.com Thu Nov 16 19:14:19 2006 From: jason.orendorff at gmail.com (Jason Orendorff) Date: Thu, 16 Nov 2006 13:14:19 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131231x14d85c75jd1daf49dd47cbe58@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> Message-ID: On 11/15/06, Fredrik Lundh wrote: > George Sakkis wrote: > > The ending sentences though are more convincing: "...but it's a part > > of Python, and it's too late to make such fundamental changes now. > > The functions have to remain to avoid massive code breakage". That > > I can buy (at least for 2.x). > > yeah, it's clear that most of your argumentation is based on a rather > common "I haven't thought this through very deeply, but I'm sure I'm > smarter than those guys so that won't stop me" approach. trust me, the > design of Python is a *lot* more carefully put together than you appear > to think. Yikes. Fredrik, if you can't say something nice, please just drop the conversation. (Actually, even if you *can* say something nice, this conversation might be worth dropping. Just a thought.) -j From pje at telecommunity.com Thu Nov 16 19:23:30 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 16 Nov 2006 13:23:30 -0500 Subject: [Python-3000] native support for multiple dispatch in 3.0 ? In-Reply-To: Message-ID: <5.1.1.6.0.20061116123144.03425708@sparrow.telecommunity.com> At 12:44 PM 11/16/2006 +0100, Fredrik Lundh wrote: >since it seems to be impossible to bring up new ideas in subthreads, I >though I'd give this one its own thread: > >- has there been any discussions on making RuleDispatch-type mechanisms > available on a more fundamental level? Only the discussion several months ago, where I tried to show that you could treat all functions as extensible, by using extensible functions to define them. (This makes it possible to keep optimizations like tp_as_sequence->tp_len in place at the C level to support fast builtins.) I was also trying to push for something like a 'defop' statement of the form: defop expr(...): ... Which, instead of binding the resulting function to 'expr', would invoke something like: addmethod(expr, func, containing_class_or_None) Where addmethod is a builtin generic function that can be extended to support user-defined generic function implementation. (Of course, this proposal assumes type information of some kind is available in the function signature, making simple implementations simple.) One purpose of the proposal is to let us get rid of most __special__ names syntactically, even though the implementation would retain them "under the hood". Instead of 'def __iter__(self)', one would 'defop iter(self)', 'defop operator.add(self, other)', etc. >could someone perhaps come up with a list of real-life uses for PJE's >existing dispatcher implementation? http://peak.telecommunity.com/DevCenter/SecurityRules is one; in fact it's the major reason I wrote RuleDispatch in the first place. I know that TurboGears is said to be doing something similar of their own. There are also other people doing enterprisey things that they don't tell me much about. People doing virtual world stuff or classic adventure games have also expressed interest. I myself have found that I don't use full predicate dispatch as often as I use multiple or even single-dispatch generic functions. I even went so far as to create this package so as to be able to use single-dispatch functions in code I'm doing for Chandler: http://cheeseshop.python.org/pypi/simplegeneric And here's what I'm doing with it: http://svn.osafoundation.org/chandler/trunk/chandler/parcels/osaf/sharing/EIM.txt http://svn.osafoundation.org/chandler/trunk/chandler/parcels/osaf/sharing/eim.py The place I'm mostly using multiple dispatch, however, is in the work I'm doing on RuleDispatch's better-documented and better-tested replacement-to-be: http://svn.eby-sarna.com/PEAK-Rules/ The new package uses an enhanced version of Guido's type-tuple dispatching to do multiple dispatch, and I'm using that to bootstrap the more advanced features. For more "real world" usage, however, note that adaptation is a subset of single-dispatch generic functions, so everything in Zope and Twisted that uses adapters would qualify. Zope 3 also does what they call "tuple adapters" which is basically a rather twisted way of implementing multiple dispatch generic functions without actually having a function. :) They also have event adapters that are a lot like the method combination features of RuleDispatch and PEAK-Rules. (And of course, for further single-dispatch use cases, one need look no further than the builtin polymorphics and stdlib stuff like pickle and pprint.) >or should we go all the way: how about using this mechanism for *all* >dispatching, even the usual dispatch-on-instance handler? and poly- >morphic builtins too, while we're at it? I wouldn't go that far. I think we should keep most of the existing C-level dispatch machinery: slots are fast. Instead, I think we should just have One Obvious Way to *add methods* togeneric functions, regardless of how they are implemented. If there is a builtin 'addmethod()' generic, then it can be defined to do the right thing for builtin slot-based generics. And if you implement your own specialty generics (ala RuleDispatch), you would simply: addmethod(addmethod, addmethod_for_my_new_type, my_new_type) Or more succinctly: class my_new_type: ... defop addmethod(self, func, cls=None): # code to add a method to this generic function object Actually, it would be nice if you could subclass 'function', just so that we could have 'inspect' and 'pydoc' functions treat your instances sanely. All of my generic functions go to insane lengths to *be* normal function objects so that inspection and documentation tools will treat them sanely. OTOH, I suppose if the inspect and pydoc code were refactored to generics, that wouldn't be a problem, because you could just 'defop inspect.getdoc(...)' or whatever... From rhamph at gmail.com Thu Nov 16 20:39:59 2006 From: rhamph at gmail.com (Adam Olsen) Date: Thu, 16 Nov 2006 12:39:59 -0700 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <52661.62.39.9.251.1163684950.squirrel@webmail.nerim.net> References: <455C072B.503@acm.org> <455C2847.50903@acm.org> <52661.62.39.9.251.1163684950.squirrel@webmail.nerim.net> Message-ID: On 11/16/06, Antoine wrote: > > Hi, > > > Well, if you find that easy to read and clear, I suspect you're more of > > a JavaScript programmer than a Python programmer. Here's how a more > > general form of the above would look in Python: > > > > response = (yield server.get_synset_links()) > > format_expanded(li.content, response.related) > > This assumes there is an underlying event loop of some sort that accepts > and is able to schedule generators. > Much more annoyingly, all your "yield"'s must be in the same scope, they > can't be spread in various subfunctions called from the generator. It is a > massive limitation to writing clean code, unless your generators are very > short. > > > def handle_response(record): > > format_expanded(li.content, response.related) > > server.get_synset_link(handle_response) > > As already mentioned, the problem with this idiom is that code that is > executed *after* appears *before* the "get_synset_link". > It makes the code less straightforward to write and especially to read > (and having readable code is important). This is a bit biased by my dislike of event-driven programming, but: def x(): response = yield server.do_command('get_synset_links') format_expanded(li.content, response.related) event_loop.add(x()) It may be possible to clean it up further with decorators, anonymous functions, or "suites", but it's interesting to note that the bulk of the code contains no functions (anonymous or named) at all?one is only used to wrap it all in a new "thread". Another benefit is that any error handling can use traditional python methods: a try/except wrapped around the operation that may fail. TOOWTDI. And yes, it does still require an event loop, but so does your original javascript example; what do you think calls your callback? -- Adam Olsen, aka Rhamphoryncus From gsakkis at rutgers.edu Thu Nov 16 20:56:51 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Thu, 16 Nov 2006 14:56:51 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> Message-ID: <91ad5bf80611161156t59857119y90ebab5f966f31b1@mail.gmail.com> On 11/16/06, Jason Orendorff wrote: > (Actually, even if you *can* say something nice, this conversation > might be worth dropping. Just a thought.) I agree. For me, the bottom line and the main argument for dropping the proposal is "itertools is not as big as you think it is to introduce a builtin rich-iterator object". It might be a bad analogy for some, but it reminds me of the "why have a path object, os.path.* is fine" reactions a while back [1]. All other arguments can be rebutted. Perhaps when we start seeing more "from itertools import ..." all over the place, we can reopen it. For now let's leave this thread R.I.P. George [1] I haven't followed the recent path and mini path object discussions to know if things have changed since then. One thing I know though is that Jason's path module has practically replaced os.* and glob.* in my code. From tjreedy at udel.edu Thu Nov 16 23:46:44 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 16 Nov 2006 17:46:44 -0500 Subject: [Python-3000] A plea for anonymous functions References: <455C072B.503@acm.org> <455C2847.50903@acm.org> <52661.62.39.9.251.1163684950.squirrel@webmail.nerim.net> Message-ID: "Antoine" wrote in message news:52661.62.39.9.251.1163684950.squirrel at webmail.nerim.net... >> def handle_response(record): >> format_expanded(li.content, response.related) >> server.get_synset_link(handle_response) > > As already mentioned, the problem with this idiom is that code that is > executed *after* appears *before* the "get_synset_link". *To me* you have this backwards. The function-object creation code is executed *before* the call that passes that object. This is true whether it is placed before, as above, or nested within, as you and Tomer prefer. > It makes the code less straightforward to write *To you*. > and especially to read *To you*. I (and many others, apparently) am less comfortable reading deeply nested code inside out than perhaps you and Tomer. If a function body is more complicated than 'return ', I prefer to have it factored out. > (and having readable code is important). To both of us, but we differ at least slightly in what is most readable. Terry Jan Reedy From mike.klaas at gmail.com Fri Nov 17 00:01:44 2006 From: mike.klaas at gmail.com (Mike Klaas) Date: Thu, 16 Nov 2006 15:01:44 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611161156t59857119y90ebab5f966f31b1@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611161156t59857119y90ebab5f966f31b1@mail.gmail.com> Message-ID: <3d2ce8cb0611161501x64092187u4d867f3c8d1bd987@mail.gmail.com> On 11/16/06, George Sakkis wrote: > > I agree. For me, the bottom line and the main argument for dropping > the proposal is "itertools is not as big as you think it is to > introduce a builtin rich-iterator object". <> You should also consider that some people see that as a false dichotomy. -Mike From greg.ewing at canterbury.ac.nz Fri Nov 17 03:52:52 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 17 Nov 2006 15:52:52 +1300 Subject: [Python-3000] Class Decorators + ctypes (Was: yes to class decorators) In-Reply-To: <455BFA17.1000205@acm.org> References: <1d85506f0611151506n48e1ee48q91de76b05754435a@mail.gmail.com> <455BFA17.1000205@acm.org> Message-ID: <455D2404.6010504@canterbury.ac.nz> Talin wrote: > One unresolved issue is what exactly gets assigned to 'x' and 'y' at the > time the class definition is evaluated. Seems to me it ought to be some kind of descriptor. It might be possible to do this reasonably without any new syntax, e.g. @cstruct class MyStruct: x = cfield("f") y = cfield("f") given a function cfield() that returns a suitable descriptor. > A different approach would be to say that the class decorator can > overload the '__setattr__' method on the class itself during class creation: But it doesn't work that way -- class attributes don't go through __setattr__ during class creation. They go into a dict that is passed to the type object's constructor. The class doesn't even exist while the class body is being evaluated. -- Greg From greg.ewing at canterbury.ac.nz Fri Nov 17 03:53:41 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 17 Nov 2006 15:53:41 +1300 Subject: [Python-3000] yes to class decorators In-Reply-To: <1d85506f0611160324x5d70967aha46c3eff47e85a96@mail.gmail.com> References: <1d85506f0611160324x5d70967aha46c3eff47e85a96@mail.gmail.com> Message-ID: <455D2435.1000704@canterbury.ac.nz> tomer filiba wrote: > we should also note that it's possible to replace "__metaclass__" > by a @metaclass decorator. I don't think so. Your proposed technique for that seems to involve creating an ordinary class first, then taking it apart and recreating it using the specified metaclass. That smells very wrong to me. -- Greg From greg.ewing at canterbury.ac.nz Fri Nov 17 03:54:25 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 17 Nov 2006 15:54:25 +1300 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611151849v70f413e5s9085ae100a3a4741@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> <-9052809295998148604@unknownmsgid> <91ad5bf80611151036p248fde9ate83b21f631fe7fc3@mail.gmail.com> <455BBA5F.5070106@canterbury.ac.nz> <91ad5bf80611151849v70f413e5s9085ae100a3a4741@mail.gmail.com> Message-ID: <455D2461.8000602@canterbury.ac.nz> George Sakkis wrote: > And for two, not everyone feels comfortable with duck typing. People > who consider (for better or for worse) isinstance() safer than > hasattr()/getattr() would be accomodated too. The trouble is that building things into the core to "accommodate" these people ends up making things worse for people who don't subscribe to that school of thought, because they will trip over places where some type won't be acceptable because it doesn't inherit from the right bases, even though it implements all the functionality required for their application. I would suggest that if someone is that uncomfortable with duck typing, then perhaps Python is not the right language for them. Even if the language itself is neutral on the matter, the culture and the body of code libraries that has grown up around it is not. There's a clear preference in favour of duck typing and against LYBL techniques. If you don't go with that flow, your code will have trouble interoperating with that of others. -- Greg From greg.ewing at canterbury.ac.nz Fri Nov 17 03:54:31 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 17 Nov 2006 15:54:31 +1300 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <455C072B.503@acm.org> References: <455C072B.503@acm.org> Message-ID: <455D2467.2030409@canterbury.ac.nz> Talin wrote: > From my point of view, both 'with' and generator expressions are > limited, special-case solutions to a general problem - the desire to be > able to use and manipulate unnamed blocks of code as first-class > objects. I don't think it's as simple as that. Back when the with statement was first being discussed, I suggested that it should be implemented by passing the body as an anonymous function. That would have been a very simple way of doing it, but there were problems, such as what to do if the body contained break or continue statements. In the end, Guido rejected the anonymous-function approach, and we ended up with something that is rather more elaborate. So we have a situation where there was no syntactical barrier to treating a code block as an anonymous function, but we chose not to do so. That seems to cast doubt on the idea that first-class code blocks would solve all the problems you suggest they would solve. -- Greg From guido at python.org Fri Nov 17 05:29:33 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 16 Nov 2006 20:29:33 -0800 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <455D2467.2030409@canterbury.ac.nz> References: <455C072B.503@acm.org> <455D2467.2030409@canterbury.ac.nz> Message-ID: If you want anonymous blocks, by all means use Ruby. Python has first-class callables, Ruby has anonymous blocks. Pick your favorite, but don't whine that neither language has both. It ain't gonna happen. On 11/16/06, Greg Ewing wrote: > Talin wrote: > > > From my point of view, both 'with' and generator expressions are > > limited, special-case solutions to a general problem - the desire to be > > able to use and manipulate unnamed blocks of code as first-class > > objects. > > I don't think it's as simple as that. Back when the with > statement was first being discussed, I suggested that it > should be implemented by passing the body as an anonymous > function. That would have been a very simple way of doing > it, but there were problems, such as what to do if the > body contained break or continue statements. In the end, > Guido rejected the anonymous-function approach, and we > ended up with something that is rather more elaborate. > > So we have a situation where there was no syntactical > barrier to treating a code block as an anonymous function, > but we chose not to do so. That seems to cast doubt on > the idea that first-class code blocks would solve all > the problems you suggest they would solve. > > -- > Greg > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From george.sakkis at gmail.com Fri Nov 17 05:57:30 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Thu, 16 Nov 2006 23:57:30 -0500 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: References: <455C072B.503@acm.org> <455D2467.2030409@canterbury.ac.nz> Message-ID: <91ad5bf80611162057x578fbc56l314d5a8dfd405d81@mail.gmail.com> On 11/16/06, Guido van Rossum wrote: > If you want anonymous blocks, by all means use Ruby. Python has > first-class callables, Ruby has anonymous blocks. Pick your favorite, > but don't whine that neither language has both. It ain't gonna happen. Should there be a new addition for this in PEP 3099 or does the "lambda will not be renamed" entry covers it sufficiently ? George From jcarlson at uci.edu Fri Nov 17 07:33:47 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 16 Nov 2006 22:33:47 -0800 Subject: [Python-3000] yes to class decorators In-Reply-To: <455D2435.1000704@canterbury.ac.nz> References: <1d85506f0611160324x5d70967aha46c3eff47e85a96@mail.gmail.com> <455D2435.1000704@canterbury.ac.nz> Message-ID: <20061116222220.8325.JCARLSON@uci.edu> Greg Ewing wrote: > > tomer filiba wrote: > > > we should also note that it's possible to replace "__metaclass__" > > by a @metaclass decorator. > > I don't think so. Your proposed technique for that seems to > involve creating an ordinary class first, then taking it > apart and recreating it using the specified metaclass. > That smells very wrong to me. I more or less agree, at least on the side of 'creating and destroying a class is a waste', but here's some abuse that I couldn't help chuckling over. No need to even bother constructing a class, but don't forget that 'return locals()' at the end! (not that I'm advocating its [ab]use, just something that made me smile) - Josiah >>> def metaclass(*bases): ... def class_factory(fcn): ... return type(fcn.__name__, bases, fcn()) ... return class_factory ... >>> @metaclass(object) ... def newclass(): ... def foo(self, arg): ... print arg ... return locals() ... >>> newclass().foo('hello!') hello! >>> From talin at acm.org Fri Nov 17 10:00:46 2006 From: talin at acm.org (Talin) Date: Fri, 17 Nov 2006 01:00:46 -0800 Subject: [Python-3000] Class Decorators + ctypes (Was: yes to class decorators) In-Reply-To: <455D2404.6010504@canterbury.ac.nz> References: <1d85506f0611151506n48e1ee48q91de76b05754435a@mail.gmail.com> <455BFA17.1000205@acm.org> <455D2404.6010504@canterbury.ac.nz> Message-ID: <455D7A3E.6030209@acm.org> Greg Ewing wrote: > Talin wrote: >> A different approach would be to say that the class decorator can >> overload the '__setattr__' method on the class itself during class >> creation: > > But it doesn't work that way -- class attributes don't > go through __setattr__ during class creation. They go > into a dict that is passed to the type object's > constructor. The class doesn't even exist while the > class body is being evaluated. I see that as a shortcoming of the current metaclass implementation. It would be much more useful IMHO if the the metaclass had access to the declaration order. Perhaps the class decorator protocol could somehow take this into account? -- Talin From talin at acm.org Fri Nov 17 10:41:53 2006 From: talin at acm.org (Talin) Date: Fri, 17 Nov 2006 01:41:53 -0800 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <455D2467.2030409@canterbury.ac.nz> References: <455C072B.503@acm.org> <455D2467.2030409@canterbury.ac.nz> Message-ID: <455D83E1.4060601@acm.org> Greg Ewing wrote: > Talin wrote: > >> From my point of view, both 'with' and generator expressions are >> limited, special-case solutions to a general problem - the desire to >> be able to use and manipulate unnamed blocks of code as first-class >> objects. > > I don't think it's as simple as that. Back when the with > statement was first being discussed, I suggested that it > should be implemented by passing the body as an anonymous > function. That would have been a very simple way of doing > it, but there were problems, such as what to do if the > body contained break or continue statements. In the end, > Guido rejected the anonymous-function approach, and we > ended up with something that is rather more elaborate. > > So we have a situation where there was no syntactical > barrier to treating a code block as an anonymous function, > but we chose not to do so. That seems to cast doubt on > the idea that first-class code blocks would solve all > the problems you suggest they would solve. Well, it seems that my argument about being able to construct the 'with' statement with anon. code blocks may not be air-tight; However I still stand by my basic argument that anon. code is a fundamental building-block of a number of interesting language extensions, and that I expect to see a series of special-case syntactical work-arounds that compensate for the lack of such a feature. To give a concrete example, suppose we decide to add to Python a special-case syntactical structure to deal with asynchronous events, similar to the code example that I posted earlier. The simplest possible example I can think of is an an alarm-clock function: # Create a callback timer alarm = Timer() alarm.SetDuration( 100, Timer.Milliseconds ) upon alarm: print "It's time to get up, silly-head!" print "Alarm has been set!" In the example, the hypothetical 'upon' statement passes a reference to the following suite to the object resulting from the expression - so in this case, the suite containing the print statement is passed as a callable to the 'alarm' object. Thus it is equivalent to: # Create a callback timer alarm = Timer() alarm.SetDuration( 100, Timer.Milliseconds ) def upon_alarm(): print "It's time to get up, silly-head!" alarm.set_callback( upon_alarm ) print "Alarm has been set!" The latter approach is a common pattern in Python. However, if we take the argument presented in the blog entry cited by Fredrik, which proposes that 'design patterns are symptomatic of a weakness in a programming language' (an argument which I don't entirely buy - absence of a feature is not always a weakness), then in order to make the language 'stronger', we ought to make the design pattern disappear into the language. In this case, we do so by absorbing the setup of the function machinery into the syntax of the 'upon' statement. However, this special-case approach is flawed in this example, because its use is so narrow. If you think about it for a minute, you immediately start to imagine all of the things that 'upon' ought to deal with that it doesn't. For example, it has no way to deal with error callbacks; It has no way to pass arguments from the asynchronous process to the code block. It doesn't let me build a hash table of code snippets (which is my solution to the problem of 'switch' statements in Python.) But even if we spend the next month writing a PEP for an 'upon' statement that overcomes all these limitations, I will still most likely be able to think of some use case that it doesn't handle. No matter what I add to the PEP, there will eventually be another PEP that adds yet another special-case solution to some aspect that I haven't covered. Instead, I'd rather work at a more fundamental level, that solves a larger class of problems. (To the folks that would say that this is only a trivial optimization, I would respond - if that's the case, then why is it used so heavily in languages which do support it? And why does practically every other popular scripting language except Python support it at all? And no, the answer 'you shouldn't be programming in Python' is not an acceptable response, it's a lame cop-out IMHO.) -- Talin From fredrik at pythonware.com Fri Nov 17 10:55:18 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 17 Nov 2006 10:55:18 +0100 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <455D83E1.4060601@acm.org> References: <455C072B.503@acm.org> <455D2467.2030409@canterbury.ac.nz> <455D83E1.4060601@acm.org> Message-ID: Talin wrote: > I expect to see a series of special-case syntactical work-arounds that > compensate for the lack of such a feature. yeah, because the "special-case syntactical work-arounds" are care- fully designed to be *usable* for a well-defined group of *practical* problems. it's about HCI, not CS. please get over this "all I have is a hammer that my CS teacher told me to use" mode of thinking; we're designing for humans, not wannabe language designers who believe in "the one true mechanism". there are plenty of other reli^h^h^h^hlanguages for that kind of thinking. From p.f.moore at gmail.com Fri Nov 17 10:57:05 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 17 Nov 2006 09:57:05 +0000 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <455D83E1.4060601@acm.org> References: <455C072B.503@acm.org> <455D2467.2030409@canterbury.ac.nz> <455D83E1.4060601@acm.org> Message-ID: <79990c6b0611170157w72955f93t988c536cd84e2add@mail.gmail.com> On 11/17/06, Talin wrote: > # Create a callback timer > alarm = Timer() > alarm.SetDuration( 100, Timer.Milliseconds ) > > upon alarm: > print "It's time to get up, silly-head!" > > print "Alarm has been set!" Why invent a new special case syntax - the ones we have work fine. # This goes in a library def upon(event): def wrapper(fn): event.set_callback(fn) return fn return wrapper # Create a callback timer alarm = Timer() alarm.SetDuration( 100, Timer.Milliseconds ) @upon(alarm) def callback(): print "It's time to get up, silly-head!" Looks pretty readable to me... Paul From fredrik at pythonware.com Fri Nov 17 10:59:41 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 17 Nov 2006 10:59:41 +0100 Subject: [Python-3000] native support for multiple dispatch in 3.0 ? In-Reply-To: <5.1.1.6.0.20061116123144.03425708@sparrow.telecommunity.com> References: <5.1.1.6.0.20061116123144.03425708@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > Only the discussion several months ago, where I tried to show that you > could treat all functions as extensible, by using extensible functions to > define them. /snip many useful references/ thanks. more than enough to keep me busy over the weekend ;-) has anyone looked at "unifying" the twisted and zope models, btw? From talin at acm.org Fri Nov 17 11:09:52 2006 From: talin at acm.org (Talin) Date: Fri, 17 Nov 2006 02:09:52 -0800 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: References: <455C072B.503@acm.org> <455D2467.2030409@canterbury.ac.nz> <455D83E1.4060601@acm.org> Message-ID: <455D8A70.8010707@acm.org> Fredrik Lundh wrote: > Talin wrote: > >> I expect to see a series of special-case syntactical work-arounds that >> compensate for the lack of such a feature. > > yeah, because the "special-case syntactical work-arounds" are care- > fully designed to be *usable* for a well-defined group of *practical* > problems. it's about HCI, not CS. > > please get over this "all I have is a hammer that my CS teacher told > me to use" mode of thinking; we're designing for humans, not wannabe > language designers who believe in "the one true mechanism". there are > plenty of other reli^h^h^h^hlanguages for that kind of thinking. I've never taken a CS course BTW - the only formal training I've had was in the Air Force, in 1976, where I learned assembly language, Fortran, and COBOL :) So be careful what assumptions you make here :) -- Talin From solipsis at pitrou.net Fri Nov 17 11:26:59 2006 From: solipsis at pitrou.net (Antoine) Date: Fri, 17 Nov 2006 11:26:59 +0100 (CET) Subject: [Python-3000] upon In-Reply-To: <455D83E1.4060601@acm.org> References: <455C072B.503@acm.org> <455D2467.2030409@canterbury.ac.nz> <455D83E1.4060601@acm.org> Message-ID: <7924.62.39.9.251.1163759219.squirrel@webmail.nerim.net> > However, this special-case approach is flawed in this example, because > its use is so narrow. If you think about it for a minute, you > immediately start to imagine all of the things that 'upon' ought to deal > with that it doesn't. > For example, it has no way to deal with error > callbacks; Arguably you could write : # Create a callback timer alarm = Timer() alarm.SetDuration( 100, Timer.Milliseconds ) upon alarm.success: print "It's time to get up, silly-head!" upon alarm.failure: print "The clock is broken, you can go back to bed" (or @upon(alarm.success) for the suggested decorator equivalent) Which would resolve to something like : def _cb(): print "It's time to get up, silly-head!" alarm.success.__upon__(_cb) def _cb(): print "The clock is broken, you can go back to bed" alarm.failure.__upon__(_cb) From behnel_ml at gkec.informatik.tu-darmstadt.de Fri Nov 17 11:35:34 2006 From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel) Date: Fri, 17 Nov 2006 11:35:34 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611131239g72afbfaahac86c2387e1f10cf@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131239g72afbfaahac86c2387e1f10cf@mail.gmail.com> Message-ID: <455D9076.6010206@gkec.informatik.tu-darmstadt.de> Hi, George Sakkis wrote: > On 11/13/06, Guido van Rossum wrote: >> Are you going to propose similar >> changes for all standard de-facto interfaces, like sequences, >> mappings, files etc.? > > No, I won't (at least not for now ;-)). Notice however that most > user-defined sequences, mappings, etc. usually don't start from > scratch; they either inherit from an existing type or from an > appropriate mixin. I absolutely disagree. One advantage of the way Python handles protocols like sequences, mappings and file-likes is that you do not have to implement the entire protocol if all you want is a thing to be, say, indexable. In most cases I've come across, I did not need to use any special base class or mixin at all, partly because I didn't need the complete protocol anyway, partly for performance reasons, partly because the the behaviour of the class was so different from a list that there was no big gain in having a partial implementation. A superclass like UserDict etc. can be helpful, but requiring you to inherit from it seems plain wrong in my eyes. Stefan From ncoghlan at gmail.com Fri Nov 17 13:35:06 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 17 Nov 2006 22:35:06 +1000 Subject: [Python-3000] Class Decorators + ctypes (Was: yes to class decorators) In-Reply-To: <455D7A3E.6030209@acm.org> References: <1d85506f0611151506n48e1ee48q91de76b05754435a@mail.gmail.com> <455BFA17.1000205@acm.org> <455D2404.6010504@canterbury.ac.nz> <455D7A3E.6030209@acm.org> Message-ID: <455DAC7A.8060601@gmail.com> Talin wrote: > Greg Ewing wrote: >> Talin wrote: >>> A different approach would be to say that the class decorator can >>> overload the '__setattr__' method on the class itself during class >>> creation: >> But it doesn't work that way -- class attributes don't >> go through __setattr__ during class creation. They go >> into a dict that is passed to the type object's >> constructor. The class doesn't even exist while the >> class body is being evaluated. > > I see that as a shortcoming of the current metaclass implementation. It > would be much more useful IMHO if the the metaclass had access to the > declaration order. Perhaps the class decorator protocol could somehow > take this into account? Not really, no: currently, no code other than the compiler has access to the order of declaration of class attributes (and the compiler only has access because it executes the code to populate that namespace). The only way I could see it working is to replace the dictionary used during class instantiation with a dict subclass that kept track of the order that keys were added to the namespace (for example, storing a list of keys in an "ordered_keys" attribute). Adding new keys to and deleting keys from such a dict would be slower than usual, though (as it would have to update the key list as well as the hash map). However, actually doing anything like this would require some mechanism for the metaclass to get class instantiation to use that dict subtype in the first place... Regardless, as Jack pointed out: this has *nothing* to do with class decorators, which operate only on fully constructed classes. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From talin at acm.org Fri Nov 17 14:13:24 2006 From: talin at acm.org (Talin) Date: Fri, 17 Nov 2006 05:13:24 -0800 Subject: [Python-3000] upon In-Reply-To: <7924.62.39.9.251.1163759219.squirrel@webmail.nerim.net> References: <455C072B.503@acm.org> <455D2467.2030409@canterbury.ac.nz> <455D83E1.4060601@acm.org> <7924.62.39.9.251.1163759219.squirrel@webmail.nerim.net> Message-ID: <455DB574.6000207@acm.org> Antoine wrote: >> However, this special-case approach is flawed in this example, because >> its use is so narrow. If you think about it for a minute, you >> immediately start to imagine all of the things that 'upon' ought to deal >> with that it doesn't. >> For example, it has no way to deal with error >> callbacks; > > Arguably you could write : > > # Create a callback timer > alarm = Timer() > alarm.SetDuration( 100, Timer.Milliseconds ) > > upon alarm.success: > print "It's time to get up, silly-head!" > upon alarm.failure: > print "The clock is broken, you can go back to bed" > > (or @upon(alarm.success) for the suggested decorator equivalent) > > Which would resolve to something like : > > def _cb(): > print "It's time to get up, silly-head!" > alarm.success.__upon__(_cb) > def _cb(): > print "The clock is broken, you can go back to bed" > alarm.failure.__upon__(_cb) Interesting. Very interesting. You could even do a switch statement with this idea: # Create the dispatcher s = Switch() # Set up the cases upon s.case(1): ... code for case 1 upon s.case(2): ... code for case 2 upon s.case(3): ... code for case 3 # Invoke the case corresponding to value 'n' s( n ) Where 'case' associates assigns the suite to a dictionary, using the argument (1, 2, ...) as the key. Once the case handlers are set up, you can invoke the 'switch' as many times as you want, without having to re-evaluate the case expressions. You can even pass the dispatcher around to other objects and invoke it remotely. Hmmm, a single syntax that handles both async callbacks, async errors, and switch/case constructs...maybe this would be more useful than I thought ... :) Now, can we figure out how to handle arguments to the code block...? -- Talin From fredrik at pythonware.com Fri Nov 17 14:46:19 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 17 Nov 2006 14:46:19 +0100 Subject: [Python-3000] upon References: <455C072B.503@acm.org><455D2467.2030409@canterbury.ac.nz> <455D83E1.4060601@acm.org><7924.62.39.9.251.1163759219.squirrel@webmail.nerim.net> <455DB574.6000207@acm.org> Message-ID: "Talin" wrote: > Hmmm, a single syntax that handles both async callbacks, async errors, > and switch/case constructs...maybe this would be more useful than I > thought ... :) done much BASIC programming lately ? From bob at redivi.com Fri Nov 17 16:07:39 2006 From: bob at redivi.com (Bob Ippolito) Date: Fri, 17 Nov 2006 07:07:39 -0800 Subject: [Python-3000] native support for multiple dispatch in 3.0 ? In-Reply-To: References: <5.1.1.6.0.20061116123144.03425708@sparrow.telecommunity.com> Message-ID: <6a36e7290611170707q601bcd2eo532ab9b97735c9a4@mail.gmail.com> On 11/17/06, Fredrik Lundh wrote: > Phillip J. Eby wrote: > > > Only the discussion several months ago, where I tried to show that you > > could treat all functions as extensible, by using extensible functions to > > define them. > > /snip many useful references/ > > thanks. more than enough to keep me busy over the weekend ;-) > > has anyone looked at "unifying" the twisted and zope models, btw? Yes. Twisted has been using Zope.Interface for a few years now. -bob From nas at arctrix.com Fri Nov 17 17:35:59 2006 From: nas at arctrix.com (Neil Schemenauer) Date: Fri, 17 Nov 2006 09:35:59 -0700 Subject: [Python-3000] Fwd: Python bytes object Message-ID: <20061117163559.GA13713@mems-exchange.org> ----- Forwarded message from Antti Louko ----- Date: Fri, 17 Nov 2006 08:25:03 +0200 From: Antti Louko Subject: Python bytes object To: nas at arctrix.com Python bytes object is useful. I would add bitwise logical operations. They would be most useful in cryptographic and graphics operations. Also, encoding and decoding (long) integers in binary fromat to and from bytes type would be useful. Currently, there is no way to efficiently convert long integers to binary formats. Regards, Antti Louko ----- End forwarded message ----- From fumanchu at amor.org Fri Nov 17 17:47:04 2006 From: fumanchu at amor.org (Robert Brewer) Date: Fri, 17 Nov 2006 08:47:04 -0800 Subject: [Python-3000] A plea for anonymous functions References: <455C072B.503@acm.org> <455D2467.2030409@canterbury.ac.nz> <455D83E1.4060601@acm.org> Message-ID: <435DF58A933BA74397B42CDEB8145A86224CA8@ex9.hostedexchange.local> Fredrik Lundh wrote: > it's about HCI, not CS. +1 QOTF. Robert Brewer System Architect Amor Ministries fumanchu at amor.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061117/c0552e1b/attachment.htm From guido at python.org Fri Nov 17 18:25:03 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 17 Nov 2006 09:25:03 -0800 Subject: [Python-3000] Fwd: Python bytes object In-Reply-To: <20061117163559.GA13713@mems-exchange.org> References: <20061117163559.GA13713@mems-exchange.org> Message-ID: Sounds good to me -- I don't have this need myself but it seems to fit well. I'm looking forward to a patch. :-) On 11/17/06, Neil Schemenauer wrote: > ----- Forwarded message from Antti Louko ----- > > Date: Fri, 17 Nov 2006 08:25:03 +0200 > From: Antti Louko > Subject: Python bytes object > To: nas at arctrix.com > > Python bytes object is useful. > > I would add bitwise logical operations. They would be most useful in > cryptographic and graphics operations. > > Also, encoding and decoding (long) integers in binary fromat to and from > bytes type would be useful. Currently, there is no way to efficiently > convert long integers to binary formats. > > Regards, > > Antti Louko > > ----- End forwarded message ----- > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From steven.bethard at gmail.com Fri Nov 17 18:39:54 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Fri, 17 Nov 2006 10:39:54 -0700 Subject: [Python-3000] yes to class decorators In-Reply-To: <20061116222220.8325.JCARLSON@uci.edu> References: <1d85506f0611160324x5d70967aha46c3eff47e85a96@mail.gmail.com> <455D2435.1000704@canterbury.ac.nz> <20061116222220.8325.JCARLSON@uci.edu> Message-ID: On 11/16/06, Josiah Carlson wrote: > but here's some abuse that I couldn't help chuckling > over. No need to even bother constructing a class, but don't forget that > 'return locals()' at the end! (not that I'm advocating its [ab]use, > just something that made me smile) > > - Josiah > > >>> def metaclass(*bases): > ... def class_factory(fcn): > ... return type(fcn.__name__, bases, fcn()) > ... return class_factory > ... > >>> @metaclass(object) > ... def newclass(): > ... def foo(self, arg): > ... print arg > ... return locals() > ... > >>> newclass().foo('hello!') > hello! > >>> Of course, this is almost exactly what the class statement does under the covers. It makes a function object for the body of the class statement, inserting a ``return locals()`` at the end. When the class statement is executed, that function is called:: >>> code = compile('class C(object): x = 1', '', 'exec') >>> dis.dis(code) 1 0 LOAD_CONST 0 ('C') 3 LOAD_NAME 0 (object) 6 BUILD_TUPLE 1 9 LOAD_CONST 1 (", line 1>) 12 MAKE_FUNCTION 0 15 CALL_FUNCTION 0 18 BUILD_CLASS 19 STORE_NAME 1 (C) 22 LOAD_CONST 2 (None) 25 RETURN_VALUE >>> dis.dis(code.co_consts[1]) 1 0 LOAD_NAME 0 (__name__) 3 STORE_NAME 1 (__module__) 6 LOAD_CONST 0 (1) 9 STORE_NAME 2 (x) 12 LOAD_LOCALS 13 RETURN_VALUE The important points in the code above are the ``MAKE_FUNCTION`` and ``CALL_FUNCTION`` from the class statement itself, and the ``LOAD_LOCALS`` and ``RETURN VALUE`` that were inserted at the end of the code for the class's body. STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From tjreedy at udel.edu Fri Nov 17 18:46:30 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 17 Nov 2006 12:46:30 -0500 Subject: [Python-3000] A plea for anonymous functions References: <455C072B.503@acm.org><455C2847.50903@acm.org><52661.62.39.9.251.1163684950.squirrel@webmail.nerim.net> Message-ID: "Terry Reedy" wrote in message news:ejipoi$f32$1 at sea.gmane.org... > *To me* you have this backwards. The function-object creation code is > executed *before* the call that passes that object. This is true whether > it is placed before, as above, or nested within, as you and Tomer prefer. Correction: I meant Talin, the OP, not Tomer, who started a different thread on decorators. My apologies to Tomer for putting words in his mouth that were not his. tjr From janssen at parc.com Fri Nov 17 20:22:58 2006 From: janssen at parc.com (Bill Janssen) Date: Fri, 17 Nov 2006 11:22:58 PST Subject: [Python-3000] Builtin iterator type In-Reply-To: <455D2461.8000602@canterbury.ac.nz> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> <-9052809295998148604@unknownmsgid> <91ad5bf80611151036p248fde9ate83b21f631fe7fc3@mail.gmail.com> <455BBA5F.5070106@canterbury.ac.nz> <91ad5bf80611151849v70f413e5s9085ae100a3a4741@mail.gmail.com> <455D2461.8000602@canterbury.ac.nz> Message-ID: <06Nov17.112300pst."58648"@synergy1.parc.xerox.com> Greg Ewing wrote: > George Sakkis wrote: > > > And for two, not everyone feels comfortable with duck typing. People > > who consider (for better or for worse) isinstance() safer than > > hasattr()/getattr() would be accomodated too. > > The trouble is that building things into the core to > "accommodate" these people ends up making things worse > for people who don't subscribe to that school of > thought, because they will trip over places where > some type won't be acceptable because it doesn't > inherit from the right bases, even though it > implements all the functionality required for their > application. I think that the issue is whether the caller can know whether their type "implements all the functionality required for their application". If the library function they are calling is effectively opaque, they somehow have to understand all the checking logic that that function will perform, to build their type and pass their parameters effectively. The use of standard base types (interfaces) is basically a communications mechanism that lets functionality designers easily tell users of their functionality what's needed. Using isinstance() instead of hasattr()/getattr() allows for more and better communication, too. Checking for some textual attributes of a type can't capture semantics not expressed in those textual attributes, which a "type" can capture that kind of intangible attribute. Finally, checking for one type instead of a vast variety of possible syntactic checks is probably usually more efficient, as well. > I would suggest that if someone is that uncomfortable > with duck typing, then perhaps Python is not the > right language for them. Wow. I think that lots of folks aren't exactly uncomfortable with it. It's more that they see it as a blemish on what's otherwise a pretty good language, and what's more, they see it as an unnecessary blemish, since Python's type system is (and this is a rarity among languages) powerful and flexible enough not to *need* duck typing. They contribute suggestions about how to remove the necessity for it in a spirit of public service, to improve the commonweal by improving Python, to improve your life and the lives of all Python users. Much like your work on PyGUI. And the suggestions I've seen aren't about making it *impossible* to do duck typing in Python, but rather about making it *unnecessary* to do duck typing in Python. Bill From talin at acm.org Fri Nov 17 21:34:54 2006 From: talin at acm.org (Talin) Date: Fri, 17 Nov 2006 12:34:54 -0800 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: References: <455C072B.503@acm.org> <455D2467.2030409@canterbury.ac.nz> <455D83E1.4060601@acm.org> Message-ID: <455E1CEE.5030006@acm.org> Fredrik Lundh wrote: > Talin wrote: > >> I expect to see a series of special-case syntactical work-arounds that >> compensate for the lack of such a feature. > > yeah, because the "special-case syntactical work-arounds" are care- > fully designed to be *usable* for a well-defined group of *practical* > problems. it's about HCI, not CS. > > please get over this "all I have is a hammer that my CS teacher told > me to use" mode of thinking; we're designing for humans, not wannabe > language designers who believe in "the one true mechanism". there are > plenty of other reli^h^h^h^hlanguages for that kind of thinking. I don't think that anyone has really answered my fundamental question yet, which is this: Python is my favorite language, but I use a lot of other languages as well, and there's one feature that I use a lot in those other languages which I miss not having in Python. How is it that people are so hostile to something that I, and apparently others, find so useful? -- Talin From brett at python.org Fri Nov 17 22:44:33 2006 From: brett at python.org (Brett Cannon) Date: Fri, 17 Nov 2006 13:44:33 -0800 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <455E1CEE.5030006@acm.org> References: <455C072B.503@acm.org> <455D2467.2030409@canterbury.ac.nz> <455D83E1.4060601@acm.org> <455E1CEE.5030006@acm.org> Message-ID: On 11/17/06, Talin wrote: > > Fredrik Lundh wrote: > > Talin wrote: > > > >> I expect to see a series of special-case syntactical work-arounds that > >> compensate for the lack of such a feature. > > > > yeah, because the "special-case syntactical work-arounds" are care- > > fully designed to be *usable* for a well-defined group of *practical* > > problems. it's about HCI, not CS. > > > > please get over this "all I have is a hammer that my CS teacher told > > me to use" mode of thinking; we're designing for humans, not wannabe > > language designers who believe in "the one true mechanism". there are > > plenty of other reli^h^h^h^hlanguages for that kind of thinking. > > I don't think that anyone has really answered my fundamental question > yet, which is this: Python is my favorite language, but I use a lot of > other languages as well, and there's one feature that I use a lot in > those other languages which I miss not having in Python. How is it that > people are so hostile to something that I, and apparently others, find > so useful? Because we have gone down this road on this topic so many times. This is why Guido declared that lambda would not change; it would neither gain nor lose any abilities (except maybe requiring parentheses around its arguments). No one has come up with a good solution, but plenty of people have said, "this is a marvelous feature and as soon as we have it we can do so much!" Well, that's fine but you need to have the good solution before it will be considered. You need to know what you are going to do with the underpants before you can get rich. As Fredrik said, we worry more about HCI than CS. Stating multi-line anonymous functions are a great solution to all of these issues is fine from a theoretical point of view. But the key point is that Python puts usability/practicality before purity. In terms of this argument, it is manifesting itself as resistance to this idea without a concrete proposal for the syntax that people actually like. And that is going to be damn hard when every proposal so far for multi-line anonymous functions has been ugly and not very usable. So one must worry about the usability aspect of this proposal first instead of what it might buy us because if it ain't usable then it ain't worth anything in Pythonland. You have the rationale behind wanting this, which is great and is a good start on a PEP. But now you need to come up with how this feature will manifest itself in the code and be usable. Once you have that all wrapped up in a PEP then people will probably be more receptive to discussing this whole matter. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061117/38a4ad2e/attachment.html From jcarlson at uci.edu Sat Nov 18 00:20:27 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri, 17 Nov 2006 15:20:27 -0800 Subject: [Python-3000] Fwd: Python bytes object In-Reply-To: References: <20061117163559.GA13713@mems-exchange.org> Message-ID: <20061117151417.832B.JCARLSON@uci.edu> "Guido van Rossum" wrote: > Sounds good to me -- I don't have this need myself but it seems to fit > well. I'm looking forward to a patch. :-) If I remember correctly, 2.5 was supposed to have binascii.long2b() and binascii.b2long() which would have converted a long integer to a 2.x string and vv, offering access to the underlying PyLong_AsString() and PyLong_FromString() functions. According to my Python 2.5 distribution and the documentation, it didn't make it. Note that there exists a patch against some 2.3 for the struct module that implements a typecode for allowing the packing and unpacking of arbitrarily lengthed integers, also offering access to the underlying PyLong functions. This would likely need to be updated for the post-2.5 struct module, if its functionality is desired. - Josiah > On 11/17/06, Neil Schemenauer wrote: > > ----- Forwarded message from Antti Louko ----- > > > > Date: Fri, 17 Nov 2006 08:25:03 +0200 > > From: Antti Louko > > Subject: Python bytes object > > To: nas at arctrix.com > > > > Python bytes object is useful. > > > > I would add bitwise logical operations. They would be most useful in > > cryptographic and graphics operations. > > > > Also, encoding and decoding (long) integers in binary fromat to and from > > bytes type would be useful. Currently, there is no way to efficiently > > convert long integers to binary formats. > > > > Regards, > > > > Antti Louko > > > > ----- End forwarded message ----- > > _______________________________________________ > > Python-3000 mailing list > > Python-3000 at python.org > > http://mail.python.org/mailman/listinfo/python-3000 > > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/jcarlson%40uci.edu From ncoghlan at gmail.com Sat Nov 18 02:18:10 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 18 Nov 2006 11:18:10 +1000 Subject: [Python-3000] yes to class decorators In-Reply-To: References: <1d85506f0611160324x5d70967aha46c3eff47e85a96@mail.gmail.com> <455D2435.1000704@canterbury.ac.nz> <20061116222220.8325.JCARLSON@uci.edu> Message-ID: <455E5F52.1070708@gmail.com> Steven Bethard wrote: > Of course, this is almost exactly what the class statement does under > the covers. That 'almost' you have there covers some pretty significant differences though. > The important points in the code above are the ``MAKE_FUNCTION`` and > ``CALL_FUNCTION`` from the class statement itself, and the > ``LOAD_LOCALS`` and ``RETURN VALUE`` that were inserted at the end of > the code for the class's body. And for anyone tempted to take this parallel too far, the important *difference* is the fact that this scope gets ignored for the purposes of lexical scoping. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From greg.ewing at canterbury.ac.nz Sat Nov 18 02:21:35 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 18 Nov 2006 14:21:35 +1300 Subject: [Python-3000] Builtin iterator type In-Reply-To: <06Nov17.112300pst.58648@synergy1.parc.xerox.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> <-9052809295998148604@unknownmsgid> <91ad5bf80611151036p248fde9ate83b21f631fe7fc3@mail.gmail.com> <455BBA5F.5070106@canterbury.ac.nz> <91ad5bf80611151849v70f413e5s9085ae100a3a4741@mail.gmail.com> <455D2461.8000602@canterbury.ac.nz> <06Nov17.112300pst.58648@synergy1.parc.xerox.com> Message-ID: <455E601F.1090604@canterbury.ac.nz> Bill Janssen wrote: > The use of standard base types (interfaces) > is basically a communications mechanism that lets functionality > designers easily tell users of their functionality what's needed. But it's a blunt tool, because it arbitrarily lumps together sets of functionality that often are not required in their entirety. Unless you break the interfaces down to the level of single methods, in which case you might as well just document which methods are required -- as we do now, and it seems to work fairly well. > Checking for some textual attributes of a > type Now you're talking about LYBL, which is generally considered an anti-pattern in Python. APIs should be designed so that you don't need to test for the presence of features. > I think that lots of folks aren't exactly uncomfortable with it. > It's more that they see it as a blemish on what's otherwise a pretty > good language... They > contribute suggestions about how to remove the necessity for it But this is a fundamental difference of opinion about what is desirable and what is not. From what you've said, they see duck typing as a necessary evil, something to be put up with. Whereas the rest of us see it as a *good* thing that rescues us from the hell of having to tediously declare all our interfaces and variable types. These are opposite and irreconcilable points of view, as far as I can see. -- Greg From greg.ewing at canterbury.ac.nz Sat Nov 18 02:21:42 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 18 Nov 2006 14:21:42 +1300 Subject: [Python-3000] A plea for anonymous functions In-Reply-To: <455E1CEE.5030006@acm.org> References: <455C072B.503@acm.org> <455D2467.2030409@canterbury.ac.nz> <455D83E1.4060601@acm.org> <455E1CEE.5030006@acm.org> Message-ID: <455E6026.1040300@canterbury.ac.nz> Talin wrote: > How is it that > people are so hostile to something that I, and apparently others, find > so useful? I don't think it's hostility so much as a weary feeling that this discussion is revisiting something that's been hashed over many times before with no satisfactory result. If there were a simple and obvious way to fit code blocks into Python's syntax, we'd probably have them already. The reason we don't is not because we don't think they'd be useful, but that we don't think they'd be useful *enough* to warrant the syntactic awkwardness that would be required to get them, unless we gave up on indentation- based block structure, but we like that a *lot*, so it's not really an option. If you want to make any headway on this, you're going to have to come up with an extremely clever syntactic idea. And it would be a good idea to research previous discussions on the topic to find out what sort of ideas were not considered clever enough. -- Greg From ncoghlan at gmail.com Sat Nov 18 02:24:41 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 18 Nov 2006 11:24:41 +1000 Subject: [Python-3000] upon In-Reply-To: <455DB574.6000207@acm.org> References: <455C072B.503@acm.org> <455D2467.2030409@canterbury.ac.nz> <455D83E1.4060601@acm.org> <7924.62.39.9.251.1163759219.squirrel@webmail.nerim.net> <455DB574.6000207@acm.org> Message-ID: <455E60D9.20807@gmail.com> Talin wrote: > Hmmm, a single syntax that handles both async callbacks, async errors, > and switch/case constructs...maybe this would be more useful than I > thought ... :) > > Now, can we figure out how to handle arguments to the code block...? PEP 343 already gives you an answer to this question :) Cheers, Nick. P.S. I'm not saying I agree with the idea. As Paul pointed out, it doesn't let you do anything you can't do with a function definition and an appropriate decorator (particularly once the nonlocal PEP goes through). -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From janssen at parc.com Sat Nov 18 04:51:05 2006 From: janssen at parc.com (Bill Janssen) Date: Fri, 17 Nov 2006 19:51:05 PST Subject: [Python-3000] Builtin iterator type In-Reply-To: <455E601F.1090604@canterbury.ac.nz> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> <-9052809295998148604@unknownmsgid> <91ad5bf80611151036p248fde9ate83b21f631fe7fc3@mail.gmail.com> <455BBA5F.5070106@canterbury.ac.nz> <91ad5bf80611151849v70f413e5s9085ae100a3a4741@mail.gmail.com> <455D2461.8000602@canterbury.ac.nz> <06Nov17.112300pst.58648@synergy1.parc.xerox.com> <455E601F.1090604@canterbury.ac.nz> Message-ID: <06Nov17.195106pst."58648"@synergy1.parc.xerox.com> Greg Ewing writes: > > Checking for some textual attributes of a > > type > > Now you're talking about LYBL, which is generally > considered an anti-pattern in Python. APIs should > be designed so that you don't need to test for the > presence of features. So you join me in wanting to make the use of "hasattr()" unnecessary? Good. But when you do have to test, it would be nice to be able to test for a semantic item rather than a syntactic one. Bill From ronaldoussoren at mac.com Sat Nov 18 16:37:28 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sat, 18 Nov 2006 16:37:28 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: <06Nov17.112300pst.58648@synergy1.parc.xerox.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> <-9052809295998148604@unknownmsgid> <91ad5bf80611151036p248fde9ate83b21f631fe7fc3@mail.gmail.com> <455BBA5F.5070106@canterbury.ac.nz> <91ad5bf80611151849v70f413e5s9085ae100a3a4741@mail.gmail.com> <455D2461.8000602@canterbury.ac.nz> <06Nov17.112300pst.58648@synergy1.parc.xerox.com> Message-ID: <363050DE-3CA0-445E-BB21-52C66406651F@mac.com> On 17 Nov 2006, at 8:22 PM, Bill Janssen wrote: > Greg Ewing wrote: >> George Sakkis wrote: >> >>> And for two, not everyone feels comfortable with duck typing. People >>> who consider (for better or for worse) isinstance() safer than >>> hasattr()/getattr() would be accomodated too. >> >> The trouble is that building things into the core to >> "accommodate" these people ends up making things worse >> for people who don't subscribe to that school of >> thought, because they will trip over places where >> some type won't be acceptable because it doesn't >> inherit from the right bases, even though it >> implements all the functionality required for their >> application. > > I think that the issue is whether the caller can know whether their > type "implements all the functionality required for their > application". If the library function they are calling is effectively > opaque, they somehow have to understand all the checking logic that > that function will perform, to build their type and pass their > parameters effectively. The use of standard base types (interfaces) > is basically a communications mechanism that lets functionality > designers easily tell users of their functionality what's needed. IMHO "checking for functionality" is part of the problem here. Just try to use the argument and let the caller worry about supplying the right kind of object. IMHO there are only two valid reasons to deviate from this: check for optional functionality and some checking if the argument will be used a long time in the future (such as with callbacks). Even in the last case I usually don't bother. BTW. Communication with users of a function is what documentation is for. acronymly-challengedly-yours, Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3562 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20061118/532279dd/attachment.bin From george.sakkis at gmail.com Sat Nov 18 17:18:15 2006 From: george.sakkis at gmail.com (George Sakkis) Date: Sat, 18 Nov 2006 11:18:15 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <455E601F.1090604@canterbury.ac.nz> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> <-9052809295998148604@unknownmsgid> <91ad5bf80611151036p248fde9ate83b21f631fe7fc3@mail.gmail.com> <455BBA5F.5070106@canterbury.ac.nz> <91ad5bf80611151849v70f413e5s9085ae100a3a4741@mail.gmail.com> <455D2461.8000602@canterbury.ac.nz> <06Nov17.112300pst.58648@synergy1.parc.xerox.com> <455E601F.1090604@canterbury.ac.nz> Message-ID: <91ad5bf80611180818l290e0a3j9f7f3f0b0e42007b@mail.gmail.com> On 11/17/06, Greg Ewing wrote: > Bill Janssen wrote: > > Checking for some textual attributes of a > > type > > Now you're talking about LYBL, which is generally > considered an anti-pattern in Python. APIs should > be designed so that you don't need to test for the > presence of features. The end user may not have to explicitly test for it, but it happens anyway behind the scenes; x.foo is functionally equivalent to if hasattr(x,'foo'): return getattr(x,'foo') else: raise AttributeError() I think Bill's point is not whether you have to be explicit about it or the language does it for you, but that the textual check must happen at some point, and that this is inherently unsafer than a semantic check. George From ark-mlist at att.net Sat Nov 18 22:00:40 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Sat, 18 Nov 2006 16:00:40 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> Message-ID: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> I've just come up with an idea that I think may help clarify this discussion in a new way. I'm afraid the description is going to be a little long; but I'd appreciate it if you would bear with me and read the whole thing. I'd like to start by coining a usage, namely the "abilities" of a type. By an ability, I mean a claim such as "Objects of type Foo are iterators" or "Objects of type Foo are copyable." I would like to use the word "ability" to refer to "iterator" and "copyable" in this context. When we say that a an object of a given class (or the class itself) has an ability, we are really saying that objects of that class behave in a particular way when used in a particular way. So, for example, when we say that x is an iterator (or x has the iterator ability), we mean that we can call x.next() to obtain successive values of a sequence, and that doing so for long enough will eventually raise StopIteration. So far, this discussion has been about the idea that a class should be able to acquire an ability by inheriting it. For quite a while, I thought this was an excellent idea; but now I'm not so sure. Here's why. I'll begin by explaining why I think it's an excellent idea. Suppose you have an object x of unknown type, and you want to know whether x is an iterator. Some people in this discussion have suggested that the Pythonic way to answer this question is to try to use it as an iterator and see if it raises an exception. The trouble with this technique is that it is destructive if it succeeds, because it consumes an element of the sequence. That means you can't use the technique until you have something to do with that first element, which rules out some plausible usages. I've raised this issue in discussion before, and gotten the response that the right way to deal with this problem is to use getattr(x,"next"). Doing so is certainly a way of determining whether x.next() is meaningful, but it doesn't offer a general way of telling whether an object has a given ability. For example, there is no analogous technique for determining whether len(x) will work without evaluating it. Nor is there a way to know whether someone just happened to define x.next() without intending to implement the iterator protocol. A good bit of the current discussion is about a way to solve this kind of problem, namely to define types that represent abilities. So, for example, if "iterator" were such a type, we could evaluate isinstance(x,iterator) to determine whether x is of a type that is intended to behave as an iterator. I've proposed this idea myself a while ago, and had been quite fond of it. Yes, there is an objection to this scheme, namely that existing user-defined iterator types would not automatically inherit from the "iterator" type; but maybe there is a way around that problem. However, I want to argue against this idea for a completely different reason: Inheritance should establish an "is-a" relationship, and that's not the right kind of relationship here. To put it bluntly: I can see no reason why the iterator type should have the iterator ability! More generally, I see no reason why a type that signals the presence of a given ability should itself possess that ability. Because if it does, then it becomes difficult to distinguish types that signal abilities from types that do not. In other words: Suppose that we have a family of types with names such as "iterator", "copyable", "equality_comparable", "comparable", and so on. Each of these types signals the corresponding ability. Some of these types may be related by inheritance. For example, I would expect "comparable" to inherit from "equality_comparable", because any type that is comparable had better be comparable for equality. It seems to me to make sense for the notion of "signaling an ability" to be an ability. In other words, the types in the last paragraph can all be used in a particular way, which is the hallmark of an ability. Now, suppose that every type that signals the presence of an ability inherits from "ability". Then we've just broken the whole ability system. Because if "iterator" inherits from "ability" and every iterator type inherits from "iterator", then every iterator is also able to signal the presence of an ability--and we most definitely do not want that! In other words, whether or not we choose to define a family of types that stand for particular abilities, I think we should not use inheritance as the mechanism for specifying that a particular class has a particular ability. I don't know what the mechanism should be, but it shouldn't be inheritance. Am I missing something? --Andrew Koenig From bob at redivi.com Sat Nov 18 22:30:06 2006 From: bob at redivi.com (Bob Ippolito) Date: Sat, 18 Nov 2006 13:30:06 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131549i7ce82745j39fd4313d759f5f4@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> Message-ID: <6a36e7290611181330w728261c3tc148ef2a69f66c56@mail.gmail.com> On 11/15/06, Fredrik Lundh wrote: > George Sakkis wrote: > > > Fredrik, I am not arguing for the argument's sake, I just don't get > > it: Python requires my class to define __len__, otherwise len() fails. > > Why not require len() as a method instead and forget about __len__ ? > > and have people complain about a len/__getitem__ naming inconsistency? > (cf. the __iter__/next vs. __iter__/__next__ discussions.) > > > Does len() (the function) do anything smarter behind the scenes than > > just passing the ball to __len__ ? > > yes. len() can use alternative approaches to determine the length for > objects implemented in the host language. this is used by CPython to > efficiently implement len() for C-level objects without having to do > full Python method resolution and dispatch for each call. > > you could of course turn len() into a __len__() helper (or even > sequencetools.len(), if you prefer) for cases where this matters, or > even reserve the "len" method name, but I'm not sure how that would > make things *simpler* than they are today. > > I'm convinced that it's better for people to get over that silly notion > that writing x.foo() is somehow always "better" than writing foo(x), in > a language that actually supports both forms. > > (if I'd have to chose between foo(x) and x.foo(), I'd rather get foo(x) > with multiple dispatch than today's x.foo() single dispatch, but that's > probably too radical for Python 3000 ;-) Same here. I've tended to write mostly-functions for a long time, and PJE's RuleDispatch made it even more compelling to do things that way. It also made it really easy to learn Erlang and get accustomed to its pattern matching. All they have is functions and modules, and it works out rather well because of pattern matching (essentially generic functions). Instead of classes you just keep the state as a parameter. -bob From sluggoster at gmail.com Sat Nov 18 23:50:17 2006 From: sluggoster at gmail.com (Mike Orr) Date: Sat, 18 Nov 2006 14:50:17 -0800 Subject: [Python-3000] Fwd: Builtin iterator type In-Reply-To: <6e9196d20611181449wbc55cd2j206ae649bcd68c0f@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> <6a36e7290611181330w728261c3tc148ef2a69f66c56@mail.gmail.com> <6e9196d20611181449wbc55cd2j206ae649bcd68c0f@mail.gmail.com> Message-ID: <6e9196d20611181450o6d6ba5d5pfe207dfdf904ce3@mail.gmail.com> On 11/18/06, Bob Ippolito wrote: > On 11/15/06, Fredrik Lundh wrote: > > I'm convinced that it's better for people to get over that silly notion > > that writing x.foo() is somehow always "better" than writing foo(x), in > > a language that actually supports both forms. > > > Instead of classes you just keep the state as a parameter. Classes make nice bundles of methods that you can pass around as a unit. They show that "this goes with that". You can do the same thing with modules but not as flexibly. For instance, you can have several classes in one module, but not several modules in one module. -- Mike Orr From greg.ewing at canterbury.ac.nz Sun Nov 19 02:15:49 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 19 Nov 2006 14:15:49 +1300 Subject: [Python-3000] Builtin iterator type In-Reply-To: <06Nov17.195106pst.58648@synergy1.parc.xerox.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611131926iaef336dk1f2daf5529123a33@mail.gmail.com> <455A7CD1.5090503@canterbury.ac.nz> <91ad5bf80611142233i6b1e60e6i65e79da831e152b9@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> <-9052809295998148604@unknownmsgid> <91ad5bf80611151036p248fde9ate83b21f631fe7fc3@mail.gmail.com> <455BBA5F.5070106@canterbury.ac.nz> <91ad5bf80611151849v70f413e5s9085ae100a3a4741@mail.gmail.com> <455D2461.8000602@canterbury.ac.nz> <06Nov17.112300pst.58648@synergy1.parc.xerox.com> <455E601F.1090604@canterbury.ac.nz> <06Nov17.195106pst.58648@synergy1.parc.xerox.com> Message-ID: <455FB045.6080908@canterbury.ac.nz> Bill Janssen wrote: > But when you do have to test, it would be nice to be able to > test for a semantic item rather than a syntactic one. In your own code, there's nothing to stop you from creating your own interface classes to use in your own tests. But when it comes to interoperating with other people's code, you can't rely on them having inherited from all the classes you want to test for, so the testing approach isn't much use. Unless you require everyone to inherit from a standard set of base classes, and then we're back in Java-land. Anyhow, I've yet to see a case where you really *need* that sort of testing. Whenever someone asks something like "how do I tell whether I've got a sequence or not", usually it's a symptom of poor API design, IMO. -- Greg From greg.ewing at canterbury.ac.nz Sun Nov 19 02:15:55 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 19 Nov 2006 14:15:55 +1300 Subject: [Python-3000] Builtin iterator type In-Reply-To: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> References: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> Message-ID: <455FB04B.3010706@canterbury.ac.nz> Andrew Koenig wrote: > Some people in this discussion have suggested that the Pythonic > way to answer this question is to try to use it as an iterator and see if it > raises an exception. That's not what I would say. I would say that you should design your code so that you don't *need* to find out whether it's an iterator or not. Instead, you just require an iterator, always, and trust that the calling code will give you one. If it doesn't, your unit tests will catch that. (You *do* have unit tests, don't you?-) > I think we should not use inheritance as the > mechanism for specifying that a particular class has a particular ability. > I don't know what the mechanism should be, but it shouldn't be inheritance. Your ability-signallers seem to be something akin to the Haskell notion of typeclasses. A typeclass is not itself a type, but represents a set of characteristics that a type can have. There have been various proposals before for ways of registering interfaces outside of the inheritance hierarchy. It wouldn't be hard to come up with something if it were thought desirable. I'm still not convinced that it solves a problem that really needs solving, however. -- Greg From greg.ewing at canterbury.ac.nz Sun Nov 19 02:15:59 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 19 Nov 2006 14:15:59 +1300 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611180818l290e0a3j9f7f3f0b0e42007b@mail.gmail.com> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <91ad5bf80611150047p73bfbffaqb34d1eafa4618d52@mail.gmail.com> <91ad5bf80611150140r1b1c7d8cwcd8c730ca520d6c@mail.gmail.com> <-9052809295998148604@unknownmsgid> <91ad5bf80611151036p248fde9ate83b21f631fe7fc3@mail.gmail.com> <455BBA5F.5070106@canterbury.ac.nz> <91ad5bf80611151849v70f413e5s9085ae100a3a4741@mail.gmail.com> <455D2461.8000602@canterbury.ac.nz> <06Nov17.112300pst.58648@synergy1.parc.xerox.com> <455E601F.1090604@canterbury.ac.nz> <91ad5bf80611180818l290e0a3j9f7f3f0b0e42007b@mail.gmail.com> Message-ID: <455FB04F.6050105@canterbury.ac.nz> George Sakkis wrote: > The end user may not have to explicitly test for it, but it happens > anyway behind the scenes; x.foo is functionally equivalent to > if hasattr(x,'foo'): return getattr(x,'foo') > else: raise AttributeError() Well, yes, but the point is that since this is being done automatically anyway, doing it yourself as well is redundant. > the textual check must > happen at some point, and that this is inherently unsafer than a > semantic check. But testing for a base class is a rather crude form of semantic check, because it bundles a lot of semantics together that you often don't need bundled. The only completely accurate semantic check is to run the program and see if it produces the right result. In other words, don't LBYL, but Just Do It, and use unit tests. -- Greg From ncoghlan at gmail.com Sun Nov 19 05:00:00 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 19 Nov 2006 14:00:00 +1000 Subject: [Python-3000] Builtin iterator type In-Reply-To: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> References: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> Message-ID: <455FD6C0.7000105@gmail.com> Andrew Koenig wrote: > In other words, whether or not we choose to define a family of types that > stand for particular abilities, I think we should not use inheritance as the > mechanism for specifying that a particular class has a particular ability. > I don't know what the mechanism should be, but it shouldn't be inheritance. > > Am I missing something? It seems to me to be an excellent write-up of why using concrete classes for interface definitions is potentially problematic. (Although I will point out that most protocols for things like len() *do* involve checks for special methods by name, and the check for iterability is typically a non-destructive call to iter(x), rather than a destructive one to x.next()). To make multiple dispatch practical, I suspect we will end up with something more along of the lines of Zope.interface to 'formalise' Python interface definitions. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From guido at python.org Sun Nov 19 05:13:13 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 18 Nov 2006 20:13:13 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <455FD6C0.7000105@gmail.com> References: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FD6C0.7000105@gmail.com> Message-ID: On 11/18/06, Nick Coghlan wrote: > (Although I will point out > that most protocols for things like len() *do* involve checks for special > methods by name, and the check for iterability is typically a non-destructive > call to iter(x), rather than a destructive one to x.next()). Ouch?! I would never check for iterability explicitly. I would just require it, as Greg Ewing says. iter() could be fairly expensive, depending on what is being iterated over. Maybe this is unique to iterators and iterators are a bad example? I don't feel as strongly about this when testing e.g. for string-ness, file-ness or list-ness. While I tend to dislike those tests too, they are sometimes inevitable when we want to overload an API. Especially file/string is often overloaded, and I see no big problem with this (I do dislike overloading list/dict, or string/list-of-strings). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Nov 19 05:28:48 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 18 Nov 2006 20:28:48 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> References: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> <002901c70b54$9b1dd350$6402a8c0@arkdesktop> Message-ID: On 11/18/06, Andrew Koenig wrote: > I've just come up with an idea that I think may help clarify this discussion > in a new way. I'm afraid the description is going to be a little long; but > I'd appreciate it if you would bear with me and read the whole thing. > > I'd like to start by coining a usage, namely the "abilities" of a type. By > an ability, I mean a claim such as "Objects of type Foo are iterators" or > "Objects of type Foo are copyable." I would like to use the word "ability" > to refer to "iterator" and "copyable" in this context. > > When we say that a an object of a given class (or the class itself) has an > ability, we are really saying that objects of that class behave in a > particular way when used in a particular way. So, for example, when we say > that x is an iterator (or x has the iterator ability), we mean that we can > call x.next() to obtain successive values of a sequence, and that doing so > for long enough will eventually raise StopIteration. > > So far, this discussion has been about the idea that a class should be able > to acquire an ability by inheriting it. For quite a while, I thought this > was an excellent idea; but now I'm not so sure. Here's why. > > I'll begin by explaining why I think it's an excellent idea. Suppose you > have an object x of unknown type, and you want to know whether x is an > iterator. Some people in this discussion have suggested that the Pythonic > way to answer this question is to try to use it as an iterator and see if it > raises an exception. The trouble with this technique is that it is > destructive if it succeeds, because it consumes an element of the sequence. > That means you can't use the technique until you have something to do with > that first element, which rules out some plausible usages. > > I've raised this issue in discussion before, and gotten the response that > the right way to deal with this problem is to use getattr(x,"next"). Doing > so is certainly a way of determining whether x.next() is meaningful, but it > doesn't offer a general way of telling whether an object has a given > ability. For example, there is no analogous technique for determining > whether len(x) will work without evaluating it. Nor is there a way to know > whether someone just happened to define x.next() without intending to > implement the iterator protocol. > > A good bit of the current discussion is about a way to solve this kind of > problem, namely to define types that represent abilities. So, for example, > if "iterator" were such a type, we could evaluate isinstance(x,iterator) to > determine whether x is of a type that is intended to behave as an iterator. > I've proposed this idea myself a while ago, and had been quite fond of it. > Yes, there is an objection to this scheme, namely that existing user-defined > iterator types would not automatically inherit from the "iterator" type; but > maybe there is a way around that problem. > > However, I want to argue against this idea for a completely different > reason: Inheritance should establish an "is-a" relationship, and that's not > the right kind of relationship here. To put it bluntly: > > I can see no reason why the iterator type should > have the iterator ability! > > More generally, I see no reason why a type that signals the presence of a > given ability should itself possess that ability. Because if it does, then > it becomes difficult to distinguish types that signal abilities from types > that do not. > > In other words: Suppose that we have a family of types with names such as > "iterator", "copyable", "equality_comparable", "comparable", and so on. > Each of these types signals the corresponding ability. Some of these types > may be related by inheritance. For example, I would expect "comparable" to > inherit from "equality_comparable", because any type that is comparable had > better be comparable for equality. > > It seems to me to make sense for the notion of "signaling an ability" to be > an ability. In other words, the types in the last paragraph can all be used > in a particular way, which is the hallmark of an ability. > > Now, suppose that every type that signals the presence of an ability > inherits from "ability". Then we've just broken the whole ability system. > Because if "iterator" inherits from "ability" and every iterator type > inherits from "iterator", then every iterator is also able to signal the > presence of an ability--and we most definitely do not want that! > > In other words, whether or not we choose to define a family of types that > stand for particular abilities, I think we should not use inheritance as the > mechanism for specifying that a particular class has a particular ability. > I don't know what the mechanism should be, but it shouldn't be inheritance. > > Am I missing something? > > --Andrew Koenig This looks like you're rediscovering or reinventing Java- or Zope-style interfaces: those are not passed on via inheritance but via a separate "implements" clause. I don't remember how it's done in Java (and don't have time to look it up) but it makes sense to me to use something akin to inheritance for constructing interfaces out of other interfaces, like your example of comparable inheriting from equality-comparable. But "being an interface" is not transmitted through this same mechanism -- that's a meta-property. I think that it would be a mistake to us isinstance to test for an interface. I believe Zope already solves most of the deeper problems; the main issue seems to be that the surface syntax for creating and using interfaces isn't very elegant (due to lack of language support) and that the standard types and common de-facto interfaces (e.g. sequence, mapping, iterator, file) don't play along and to some extent defy capturing their semantics in interfaces. I realize that Java- and Zope-style interfaces *seem* to be all about syntax (they define names and signatures of methods but not semantics) but IMO that's only a theoretical objection; in *practice* these interfaces have strong connotations of semantics (albeit written in Enlish rather than using assertions or pre- and post-conditions) and nobody would think of claiming to implement an interface without implementing the proper semantics (or some interpretation thereof :-). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ntoronto at cs.byu.edu Sun Nov 19 06:42:19 2006 From: ntoronto at cs.byu.edu (Neil Toronto) Date: Sat, 18 Nov 2006 22:42:19 -0700 Subject: [Python-3000] Builtin iterator type In-Reply-To: References: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> <002901c70b54$9b1dd350$6402a8c0@arkdesktop> Message-ID: <455FEEBB.9010100@cs.byu.edu> Guido van Rossum wrote: > I realize that Java- and Zope-style interfaces *seem* to be all about > syntax (they define names and signatures of methods but not semantics) > but IMO that's only a theoretical objection; in *practice* these > interfaces have strong connotations of semantics (albeit written in > Enlish rather than using assertions or pre- and post-conditions) and > nobody would think of claiming to implement an interface without > implementing the proper semantics (or some interpretation thereof :-) Actually, plenty of people would dream of it and even do it. I've seen some pretty evil implementations of Java interfaces. All they can enforce is static types and method signatures. My main point is, unless you're willing to go whole-hog with pre- and post-conditions (and even that would be insufficient, as you can't check anything like "this eventually, on some call, responds with x" without spelling it out in temporal logic and running model checking software), no interface can enforce anything *semantically* meaningful. The most useful thing an interface can do is *communicate* semantics to another programmer. I'm not sure I said anything Guido didn't already say, but I wanted to make the distinction between *enforcing* correct behavior and *communicating* correct behavior. Java (and Zope, apparently) simply have a more formal way of doing this. Stock Python doesn't. Neil From ncoghlan at gmail.com Sun Nov 19 07:26:39 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 19 Nov 2006 16:26:39 +1000 Subject: [Python-3000] Builtin iterator type In-Reply-To: References: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FD6C0.7000105@gmail.com> Message-ID: <455FF91F.2000800@gmail.com> Guido van Rossum wrote: > On 11/18/06, Nick Coghlan wrote: >> (Although I will point out >> that most protocols for things like len() *do* involve checks for >> special >> methods by name, and the check for iterability is typically a >> non-destructive >> call to iter(x), rather than a destructive one to x.next()). > > Ouch?! I would never check for iterability explicitly. I would just > require it, as Greg Ewing says. iter() could be fairly expensive, > depending on what is being iterated over. I didn't mean to imply that the call to iter() couldn't be an implicit one in a for loop/genexp/listcomp/itertool. As you say, the need to check explicitly is rare. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From solipsis at pitrou.net Sun Nov 19 11:13:26 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 19 Nov 2006 11:13:26 +0100 Subject: [Python-3000] interfaces In-Reply-To: <455FEEBB.9010100@cs.byu.edu> References: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FEEBB.9010100@cs.byu.edu> Message-ID: <1163931206.4510.30.camel@fsol> Le samedi 18 novembre 2006 ? 22:42 -0700, Neil Toronto a ?crit : > Actually, plenty of people would dream of it and even do it. I've seen > some pretty evil implementations of Java interfaces. All they can > enforce is static types and method signatures. But protecting against erroneous use (rather than deliberate misuse) can be a reasonable goal. There are cases where relying on duck-typing leads to silent bugs instead of nice explicit tracebacks. Then as the implementer of the API it is good to be able to enforce some aspects of the API, so that your users don't lose time finding those bugs. I had the problem recently when I wrote a decorator which took an optional number as an argument. The decorator declaration goes like this: def deferred(timeout=None): def decorate(func): blah blah... return decorate It can be used like this: @deferred() def test_case(): ... The problem was if you forgot the parentheses altogether: @deferred def test_case(): ... In that case, no exception was raised, but the test was silently ignored. Also, "@deferred" doesn't strike at first sight like something is missing (parameterless decorators do exist). So to know if "deferred" was correctly used, I had to devise a test to know if "timeout" is a number (without mandating that it is a e.g. float rather than an int). I ended up with this: try: timeout is None or timeout + 0 except TypeError: raise TypeError("'timeout' argument must be a number or None") Instead of testing "timeout" against an arbitrary arithmetic operation, it would be more robust to test it against an interface (say "Number"). I think that's the kind of use cases which makes people oppose the idea of removing callable(). It feels more robust to test against an interface than against a specific trait of that interface. From jan.grant at bristol.ac.uk Sun Nov 19 12:48:35 2006 From: jan.grant at bristol.ac.uk (Jan Grant) Date: Sun, 19 Nov 2006 11:48:35 +0000 (GMT) Subject: [Python-3000] Builtin iterator type In-Reply-To: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> References: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> Message-ID: <20061119113835.D14417@tribble.ilrt.bris.ac.uk> On Sat, 18 Nov 2006, Andrew Koenig wrote: > Now, suppose that every type that signals the presence of an ability > inherits from "ability". Then we've just broken the whole ability system. > Because if "iterator" inherits from "ability" and every iterator type > inherits from "iterator", then every iterator is also able to signal the > presence of an ability--and we most definitely do not want that! > Am I missing something? I believe you have a type error in this argument. Instance x of class X has ability A, you suggest, should be indicated by having X derive from signalling_class(A). That's is-a: All Xs are signalling_class(A)s. Then sc(A) is the class of objects with ability A. You want to indicate that sc(A) is a class that signals an ability. You suggest you do this by having sc(A) derive from a base class, sc(Signals_Abilities). That is a type error: to indicate that sc(A) is an ability-signalling class, your original convention is that sc(A) should be an _instance_ of sc(Signals_Abilities). That is not represented by an inheritance relationship between sc(A) and sc(Signals_Abiliities), but by a membership relationship. So, sc(Signals_Abilities) is actually a metaclass, I think. jan -- jan grant, ISYS, University of Bristol. http://www.bris.ac.uk/ Tel +44 (0)117 3317661 http://ioctl.org/jan/ Spreadsheet through network. Oh yeah. From jan.grant at bristol.ac.uk Sun Nov 19 12:57:38 2006 From: jan.grant at bristol.ac.uk (Jan Grant) Date: Sun, 19 Nov 2006 11:57:38 +0000 (GMT) Subject: [Python-3000] Builtin iterator type In-Reply-To: <455FEEBB.9010100@cs.byu.edu> References: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FEEBB.9010100@cs.byu.edu> Message-ID: <20061119115009.V14417@tribble.ilrt.bris.ac.uk> On Sat, 18 Nov 2006, Neil Toronto wrote: > Guido van Rossum wrote: > > I realize that Java- and Zope-style interfaces *seem* to be all about > > syntax (they define names and signatures of methods but not semantics) > > but IMO that's only a theoretical objection; in *practice* these > > interfaces have strong connotations of semantics (albeit written in > > Enlish rather than using assertions or pre- and post-conditions) and > > nobody would think of claiming to implement an interface without > > implementing the proper semantics (or some interpretation thereof :-) > > Actually, plenty of people would dream of it and even do it. I've seen > some pretty evil implementations of Java interfaces. All they can > enforce is static types and method signatures. This seems to be an argument that interfaces can be misused by people without taste. That's true; but the same arguments have been levelled against pretty much all language features (operator overloading, asepct-orientation, etc). It strikes me that one aspect of "being Pythonic" is a strong reliance on politeness: that's what duck-typing is all about. Or conventions for private attributes. Such features could potentially be abused, but a Python programmer has the good taste to know what's abuse and what isn't. > My main point is, unless you're willing to go whole-hog with pre- and > post-conditions (and even that would be insufficient, as you can't check > anything like "this eventually, on some call, responds with x" without > spelling it out in temporal logic and running model checking software), > no interface can enforce anything *semantically* meaningful. The most > useful thing an interface can do is *communicate* semantics to another > programmer. Interfaces are more than just language constructs. They come with a semantics. It's true that that semantics may be spelled out in javadoc, but since the consumer of the interface is another programmer, that's ok: we want to communicate effectively to that programmer the contract that they are agreeing to uphold when they implement that interface. It's true that lots of Java programmers may not quite grok this from the word go; but fundamentally an interface should be viewed as a type (or ability marker, as Andrew K called it) and comes with a corresponding set of semantics. That Java doesn't make an attempt to spell out explicitly in code those other semantics isn't necessarily a weakness. Java interfaces are very useful, however. Java programming seems to be less and less about inheritance and more and more about implementing interfaces; at least it does amongst Java programmers with taste :-) Cheers, jan -- jan grant, ISYS, University of Bristol. http://www.bris.ac.uk/ Tel +44 (0)117 3317661 http://ioctl.org/jan/ Whenever I see a dog salivate I get an insatiable urge to ring a bell. From ark-mlist at att.net Sun Nov 19 17:40:46 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Sun, 19 Nov 2006 11:40:46 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <455FB04B.3010706@canterbury.ac.nz> Message-ID: <000501c70bf9$791c2b80$6402a8c0@arkdesktop> > That's not what I would say. I would say that you > should design your code so that you don't *need* > to find out whether it's an iterator or not. Instead, > you just require an iterator, always, and trust that > the calling code will give you one. If it doesn't, > your unit tests will catch that. (You *do* have > unit tests, don't you?-) That's a nice general sentiment, but not always feasible -- especially if you're writing library code. Remember, I'm thinking about abilities in general, not just whether a type is an iterator. For example, testing whether something is an iterator by trying to call its next() method is destructive if it succeeds. So consider the following: def concat(x, y): for a in x: yield a for a in y: yield a Simple as this function may be, it has a drawback: If x is an iterator and y is not a valid sequence, the function consumes x before failing. If it were possible to determine nondestructively whether y is a sequence, the function would not have this drawback. The drawback in this particular example may be trivial; but similar drawbacks can be serious in other contexts, especially where networking is involved. > Your ability-signallers seem to be something akin to > the Haskell notion of typeclasses. A typeclass is not > itself a type, but represents a set of characteristics > that a type can have To the extent that I'm familiar with Haskell, I think you're right. From ark-mlist at att.net Sun Nov 19 17:42:09 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Sun, 19 Nov 2006 11:42:09 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <455FB04F.6050105@canterbury.ac.nz> Message-ID: <000601c70bf9$b015f620$6402a8c0@arkdesktop> > The only completely accurate semantic check is to run > the program and see if it produces the right result. If that were possible, we could solve the halting problem :-) > In other words, don't LBYL, but Just Do It, and use > unit tests. A fine idea when it's possible. Unfortunately, it's not always possible. From ark-mlist at att.net Sun Nov 19 17:56:31 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Sun, 19 Nov 2006 11:56:31 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: Message-ID: <000701c70bfb$a9faef00$6402a8c0@arkdesktop> > This looks like you're rediscovering or reinventing Java- or > Zope-style interfaces: those are not passed on via inheritance but via > a separate "implements" clause. I don't think I'm quite reinventing Java interfaces. In Java, where just about everything is a method, it's temptingly easy to talk about the interface to a type by making claims about its methods and the types they return. But in Python, such claims don't mesh well with the language's dynamic nature. So what I'm really looking for is a formal way of claiming that a type has an informal property, and leaving it up to the programmers (and their unit tests) to enforce the correspondence between the claims and the properties. For iterators, for example, it's not just the presence of the __iter__ and next methods that are relevant; it's also the notion that calling next can raise StopIteration in the ordinary course of events, but if it raises any other exception, it should be considered to be truly exceptional. I don't know how to capture such specifications formally, but I do think it would be useful for a programmer to have a formal way of claiming that a given class meets a given specification. > I don't remember how it's done in Java > (and don't have time to look it up) but it makes sense to me to use > something akin to inheritance for constructing interfaces out of other > interfaces, like your example of comparable inheriting from > equality-comparable. But "being an interface" is not transmitted > through this same mechanism -- that's a meta-property. I think that it > would be a mistake to us isinstance to test for an interface. Yes -- that's it exactly! > I realize that Java- and Zope-style interfaces *seem* to be all about > syntax (they define names and signatures of methods but not semantics) > but IMO that's only a theoretical objection; in *practice* these > interfaces have strong connotations of semantics (albeit written in > Enlish rather than using assertions or pre- and post-conditions) and > nobody would think of claiming to implement an interface without > implementing the proper semantics (or some interpretation thereof :-). I agree almost 100%. Unfortunately, some people claim to implement interfaces without appearing to think at all; but that's another subject entirely :-) From ark-mlist at att.net Sun Nov 19 17:57:50 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Sun, 19 Nov 2006 11:57:50 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <455FEEBB.9010100@cs.byu.edu> Message-ID: <000801c70bfb$d9161b70$6402a8c0@arkdesktop> > I'm not sure I said anything Guido didn't already say, but I wanted to > make the distinction between *enforcing* correct behavior and > *communicating* correct behavior. Java (and Zope, apparently) simply > have a more formal way of doing this. Stock Python doesn't. What he said! From ark-mlist at att.net Sun Nov 19 18:03:28 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Sun, 19 Nov 2006 12:03:28 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <20061119115009.V14417@tribble.ilrt.bris.ac.uk> Message-ID: <000901c70bfc$a8bbe9e0$6402a8c0@arkdesktop> > It strikes me that one aspect of "being Pythonic" is a strong reliance > on politeness: that's what duck-typing is all about. Part of my motivation for entering this discussion is that C++ templates use duck-typing, and the C++ community has for several years been thinking about how to add more explicit constraints to the type system. One motivation is easier diagnostics: If by mistake you hand a non-sequence to something that requires a sequence, it would be nice for the compiler to be able to say "This argument does not satisfy the 'sequence' constraint" rather than spewing forth a page of error messages generated deep in the internals of the library. I understand that this exact motivation doesn't apply in Python because of its dynamic typing. However, I've seen examples in Python that feel to me like that address similar problems to the ones in C++. Of course, such feelings don't prove anything; but sometimes they help point the way to a useful solution. From gsakkis at rutgers.edu Sun Nov 19 18:44:15 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Sun, 19 Nov 2006 12:44:15 -0500 Subject: [Python-3000] interfaces In-Reply-To: <1163931206.4510.30.camel@fsol> References: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FEEBB.9010100@cs.byu.edu> <1163931206.4510.30.camel@fsol> Message-ID: <91ad5bf80611190944y66f4cbedyc4bf324d7dbda028@mail.gmail.com> On 11/19/06, Antoine Pitrou wrote: > I had the problem recently when I wrote a decorator which took an > optional number as an argument. The decorator declaration goes like > this: > > def deferred(timeout=None): > def decorate(func): > blah blah... > return decorate > > It can be used like this: > > @deferred() > def test_case(): > ... > > The problem was if you forgot the parentheses altogether: > > @deferred > def test_case(): > ... > > In that case, no exception was raised, but the test was silently > ignored. Also, "@deferred" doesn't strike at first sight like something > is missing (parameterless decorators do exist). > > So to know if "deferred" was correctly used, I had to devise a test to > know if "timeout" is a number (without mandating that it is a e.g. float > rather than an int). I ended up with this: > try: > timeout is None or timeout + 0 > except TypeError: > raise TypeError("'timeout' argument must be a number or None") I understand this is not the point you're trying to make, but in such cases I usually prefer to make @decorator be equivalent to @decorator() by something like: def deferred(timeout=None): if callable(timeout): f = timeout; timeout = None else: f = None def decorate(func): blah blah... return decorate if f is None else decorate(f) Of course this doesn't work if it's valid for the timeout parameter to be a callable, but usually, like in your example, it's not. > Instead of testing "timeout" against an arbitrary arithmetic operation, > it would be more robust to test it against an interface (say "Number"). > I think that's the kind of use cases which makes people oppose the idea > of removing callable(). It feels more robust to test against an > interface than against a specific trait of that interface. The counter-argument is that instead of picking an arbitrary trait of the interface to check, make sure your unit tests fail if the passed value is not appropriate. The real problem you should look into is why "in that case, no exception was raised, but the test was silently ignored". I don't know how 'timeout' is used in your example, but from a similar bad experience, I'd guess all you are doing with it is compare it with a number ("if timeout < 3") and comparisons between instances of different types don't raise an exception. That's perhaps one of the top gotchas in Python today; thankfully it will change in 3.0. Until then, I can't see how one can avoid an explicit check, either by testing with isinstance() or trying to call an arbitrary method of the expected protocol. Having said that both are necessary evils, I'd go with the isinstance check as the lesser evil. Checking for an arbitrary method is not only an ugly kludge; it is both unnecessary (if the tested method is not used in the actual code) and error-prone (e.g. for instances of a type that just happens to define __add__ so that x + 0 doesn't raise an exception, although x is an unappropriate argument in the specific context). George From python3now at gmail.com Sun Nov 19 19:13:30 2006 From: python3now at gmail.com (James Thiele) Date: Sun, 19 Nov 2006 10:13:30 -0800 Subject: [Python-3000] print() parameters in py3k Message-ID: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> The BDFL has said that print will change from a statement to a function in Python 3. I haven't found anything describing what parameters it will take and what it will return. Has this been decided? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061119/c1a460ca/attachment.htm From sluggoster at gmail.com Sun Nov 19 19:28:42 2006 From: sluggoster at gmail.com (Mike Orr) Date: Sun, 19 Nov 2006 10:28:42 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <455FB04F.6050105@canterbury.ac.nz> References: <91ad5bf80611131044q44fa97eesdf37cdc0bac44fe8@mail.gmail.com> <-9052809295998148604@unknownmsgid> <91ad5bf80611151036p248fde9ate83b21f631fe7fc3@mail.gmail.com> <455BBA5F.5070106@canterbury.ac.nz> <91ad5bf80611151849v70f413e5s9085ae100a3a4741@mail.gmail.com> <455D2461.8000602@canterbury.ac.nz> <06Nov17.112300pst.58648@synergy1.parc.xerox.com> <455E601F.1090604@canterbury.ac.nz> <91ad5bf80611180818l290e0a3j9f7f3f0b0e42007b@mail.gmail.com> <455FB04F.6050105@canterbury.ac.nz> Message-ID: <6e9196d20611191028u4b3ad027kf8e8031bb446cae@mail.gmail.com> On 11/18/06, Greg Ewing wrote: > George Sakkis wrote: > > > The end user may not have to explicitly test for it, but it happens > > anyway behind the scenes; x.foo is functionally equivalent to > > if hasattr(x,'foo'): return getattr(x,'foo') > > else: raise AttributeError() > > Well, yes, but the point is that since this is being > done automatically anyway, doing it yourself as > well is redundant. *Something* is being done automatically but it's not the same thing. The error message may be so unrelated to the actual problem as to give no clue what's wrong. Good code should reflect one's mental model, as close to "one line per step" as feasable. If the language can have features to facilitate this, why not? Many Pythoneers find interfaces useful, or would if the language had them. Making a concrete class with dummy methods is kind of pointless when it'll never be used for anything except isinstance(), and it's even worse at enforcing attribute existence. ("Yippee, I have a default attribute that's None.") There have long been discussions about adding interfaces and/or adapters to Python, so it will probably happen someday. As has been mentioned, Zope has had an interface implementation for years. PEPs 245 and 246 (both rejected) contain specific proposals for interfaces and adapters. The last discussion I saw was in Guido's blog (April 2006): http://www.artima.com/forums/flat.jsp?forum=106&thread=155123 There the discussion was on how generic functions could fulfill both roles and more besides. The discussion seemed to fizzle out at, "Yes, this would be nice to have," without a concrete proposal. Simultaneously there was discussion on this list about it, although I wasn't subscribed then so I haven't fully read it: http://mail.python.org/pipermail/python-3000/2006-April/thread.html -- Mike Orr From sluggoster at gmail.com Sun Nov 19 19:56:01 2006 From: sluggoster at gmail.com (Mike Orr) Date: Sun, 19 Nov 2006 10:56:01 -0800 Subject: [Python-3000] interfaces In-Reply-To: <91ad5bf80611190944y66f4cbedyc4bf324d7dbda028@mail.gmail.com> References: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FEEBB.9010100@cs.byu.edu> <1163931206.4510.30.camel@fsol> <91ad5bf80611190944y66f4cbedyc4bf324d7dbda028@mail.gmail.com> Message-ID: <6e9196d20611191056r6fbe71c0n32c0a740fbc806f6@mail.gmail.com> On 11/19/06, George Sakkis wrote: > That's perhaps one of the top gotchas in Python today; thankfully it > will change in 3.0. Until then, I can't see how one can avoid an > explicit check, either by testing with isinstance() or trying to call > an arbitrary method of the expected protocol. Having said that both > are necessary evils, I'd go with the isinstance check as the lesser > evil. Checking for an arbitrary method is not only an ugly kludge; it > is both unnecessary (if the tested method is not used in the actual > code) and error-prone (e.g. for instances of a type that just happens > to define __add__ so that x + 0 doesn't raise an exception, although x > is an unappropriate argument in the specific context). One of Python's biggest surprises is it has distinct concepts of "sequence" and "mapping", but the API is intertwined so much you can't even test whether an object is one or the other. Say you want to verify an object is a sequence: (1) isinstance(obj, list): fails for UserList et al, but many users have gone this route of requiring a list subclass because it's the most straighforward. (2) hasattr(obj, "__getitem__"): it may have that method, but that doesn't mean it's list-like. (3) Does .__getitem__() accept int arguments? All lists/dicts do. (4) Does .__getitem__() return a value for all ints within range (0, len(obj))? No way to test for this without calling with all possible args and seeing if it raises LookupError. I can't think of any way around this except a 'sequence' interface that 'list' would implement. That in itself wouldn't verify test 4, but the fact that 'implements(obj, sequence) == True' would mean the class author has promised it will. However, requiring objects to be list subclasses hasn't been that much of a burden in practical terms. -- Mike Orr From solipsis at pitrou.net Sun Nov 19 20:18:14 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 19 Nov 2006 20:18:14 +0100 Subject: [Python-3000] interfaces In-Reply-To: <91ad5bf80611190944y66f4cbedyc4bf324d7dbda028@mail.gmail.com> References: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FEEBB.9010100@cs.byu.edu> <1163931206.4510.30.camel@fsol> <91ad5bf80611190944y66f4cbedyc4bf324d7dbda028@mail.gmail.com> Message-ID: <1163963894.4510.87.camel@fsol> Le dimanche 19 novembre 2006 ? 12:44 -0500, George Sakkis a ?crit : > I understand this is not the point you're trying to make, but in such > cases I usually prefer to make @decorator be equivalent to > @decorator() by something like: I could do that, but it's not very clean. Also it doesn't invalidate my point, since what callable() does is precisely to check that the parameter conforms to a given interface (instead of explicitly testing for the underlying implementation e.g. __call__). I'm not an interface fan at all. Simply, in some cases, it would be cleaner to check an object implements, or claims to implement, a conventional interface (which might not even involve any syntactic checks about the presence of a method), rather than testing for a specific implementation trait in the hope that it's both necessary and sufficient. From guido at python.org Sun Nov 19 22:33:46 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 19 Nov 2006 13:33:46 -0800 Subject: [Python-3000] print() parameters in py3k In-Reply-To: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: On 11/19/06, James Thiele wrote: > The BDFL has said that print will change from a statement to a function in > Python 3. > > I haven't found anything describing what parameters it will take and what it > will return. Has this been decided? Not really, I'm hoping someone would write a PEP. But (from memory, I think there are some threads on python-dev or the py3k list about this) here are some suggestions: 1. With only positional parameters (or none at all), print(x, y, z) is equivalent to "print x, y, z" today. So it converts each argument using str(), inserting spaces in between, and appending a newline. 2. A keyword parameter will direct it to a different file. Probably print(x, y, z, file=). 3. Two more keyword parameters can change what gets inserted between items and at the end. Perhaps print(x, y, z, sep=, end=). These default to " " and "\n", respectively. 4. The softspace feature (a semi-secret attribute on files currently used to tell print whether to insert a space before the first item) will be killed. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From g.brandl at gmx.net Sun Nov 19 22:37:06 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 19 Nov 2006 22:37:06 +0100 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: Guido van Rossum wrote: > On 11/19/06, James Thiele wrote: >> The BDFL has said that print will change from a statement to a function in >> Python 3. >> >> I haven't found anything describing what parameters it will take and what it >> will return. Has this been decided? > > Not really, I'm hoping someone would write a PEP. But (from memory, I > think there are some threads on python-dev or the py3k list about > this) here are some suggestions: > > 1. With only positional parameters (or none at all), print(x, y, z) is > equivalent to "print x, y, z" today. So it converts each argument > using str(), inserting spaces in between, and appending a newline. > > 2. A keyword parameter will direct it to a different file. > Probably print(x, y, z, file=). > > 3. Two more keyword parameters can change what gets inserted between > items and at the end. > Perhaps print(x, y, z, sep=, end=). These default to " > " and "\n", respectively. > > 4. The softspace feature (a semi-secret attribute on files currently > used to tell print whether to insert a space before the first item) > will be killed. Do we even need a PEP? This is a good, easy-to-explain interface. Is more functionality needed? Georg From guido at python.org Sun Nov 19 22:55:04 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 19 Nov 2006 13:55:04 -0800 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: On 11/19/06, Georg Brandl wrote: > Guido van Rossum wrote: > > On 11/19/06, James Thiele wrote: > >> The BDFL has said that print will change from a statement to a function in > >> Python 3. > >> > >> I haven't found anything describing what parameters it will take and what it > >> will return. Has this been decided? > > > > Not really, I'm hoping someone would write a PEP. But (from memory, I > > think there are some threads on python-dev or the py3k list about > > this) here are some suggestions: > > > > 1. With only positional parameters (or none at all), print(x, y, z) is > > equivalent to "print x, y, z" today. So it converts each argument > > using str(), inserting spaces in between, and appending a newline. > > > > 2. A keyword parameter will direct it to a different file. > > Probably print(x, y, z, file=). > > > > 3. Two more keyword parameters can change what gets inserted between > > items and at the end. > > Perhaps print(x, y, z, sep=, end=). These default to " > > " and "\n", respectively. > > > > 4. The softspace feature (a semi-secret attribute on files currently > > used to tell print whether to insert a space before the first item) > > will be killed. > > Do we even need a PEP? This is a good, easy-to-explain interface. Is more > functionality needed? PEPs aren't only for difficult discussions. :-) They are also there for reference and to record agreement. Referring to an email isn't really a very good answer when someone asks (as happened here) "what is the spec"? A PEP may also discourage attempts to add more cruft, and encourage someone with a few spare cycles to provide a patch so we can put the whole thing behind it. BTW I forgot to mention that it shouldn't return anything. (So are you going to write that PEP? :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From g.brandl at gmx.net Mon Nov 20 00:18:27 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 20 Nov 2006 00:18:27 +0100 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: Guido van Rossum wrote: > PEPs aren't only for difficult discussions. :-) They are also there > for reference and to record agreement. Referring to an email isn't > really a very good answer when someone asks (as happened here) "what > is the spec"? A PEP may also discourage attempts to add more cruft, > and encourage someone with a few spare cycles to provide a patch so we > can put the whole thing behind it. > > BTW I forgot to mention that it shouldn't return anything. > > (So are you going to write that PEP? :-) It's in Subversion and numbered 3105, should be online with the next website update cycle. cheers, Georg From greg.ewing at canterbury.ac.nz Mon Nov 20 00:33:06 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 20 Nov 2006 12:33:06 +1300 Subject: [Python-3000] interfaces In-Reply-To: <1163931206.4510.30.camel@fsol> References: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FEEBB.9010100@cs.byu.edu> <1163931206.4510.30.camel@fsol> Message-ID: <4560E9B2.70801@canterbury.ac.nz> Antoine Pitrou wrote: > So to know if "deferred" was correctly used, I had to devise a test to > know if "timeout" is a number Another solution might have been to require it to be passed as a keyword (which will become easier if the proposal for keyword-only arguments gets in). -- Greg From greg.ewing at canterbury.ac.nz Mon Nov 20 01:01:10 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 20 Nov 2006 13:01:10 +1300 Subject: [Python-3000] Builtin iterator type In-Reply-To: <000501c70bf9$791c2b80$6402a8c0@arkdesktop> References: <000501c70bf9$791c2b80$6402a8c0@arkdesktop> Message-ID: <4560F046.304@canterbury.ac.nz> Andrew Koenig wrote: > That's a nice general sentiment, but not always feasible -- especially if > you're writing library code. I've written a fair bit of library code, and I've never found a pressing need to test for an abstract interface. In the rare cases where I've used type testing in an API, it's always been for specific well-known concrete types, such as strings and tuples -- never anything so abstract as "iterator", "sequence", etc. -- Greg From janssen at parc.com Mon Nov 20 02:02:40 2006 From: janssen at parc.com (Bill Janssen) Date: Sun, 19 Nov 2006 17:02:40 PST Subject: [Python-3000] Builtin iterator type In-Reply-To: <455FD6C0.7000105@gmail.com> References: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FD6C0.7000105@gmail.com> Message-ID: <06Nov19.170242pst."58648"@synergy1.parc.xerox.com> > Andrew Koenig wrote: > > In other words, whether or not we choose to define a family of types that > > stand for particular abilities, I think we should not use inheritance as the > > mechanism for specifying that a particular class has a particular ability. > > I don't know what the mechanism should be, but it shouldn't be inheritance. > > > > Am I missing something? > > It seems to me to be an excellent write-up of why using concrete classes for > interface definitions is potentially problematic. Huh? Andrew pointed to exactly one "drawback", the inability to disassociate an inherent characteristic of a type, its "typeness" (which he refers to as ability-signalling), from a type. I hardly see that as problematic. Am *I* missing something? Bill From brett at python.org Mon Nov 20 02:04:05 2006 From: brett at python.org (Brett Cannon) Date: Sun, 19 Nov 2006 17:04:05 -0800 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: On 11/19/06, Georg Brandl wrote: > > Guido van Rossum wrote: > > > PEPs aren't only for difficult discussions. :-) They are also there > > for reference and to record agreement. Referring to an email isn't > > really a very good answer when someone asks (as happened here) "what > > is the spec"? A PEP may also discourage attempts to add more cruft, > > and encourage someone with a few spare cycles to provide a patch so we > > can put the whole thing behind it. > > > > BTW I forgot to mention that it shouldn't return anything. > > > > (So are you going to write that PEP? :-) > > It's in Subversion and numbered 3105, should be online with the next > website update cycle. The only thing I would prefer is instead of 'end' have 'newline' and have that be a boolean since I don't see a need to support different line endings. I realize the difference between ``end=''`` and ``newline=False`` is minimal beyond more typing, but for some reason my head just keeps telling me I prefer the latter for clarity reasons. Otherwise +1 from me. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061119/2161efc3/attachment.html From sluggoster at gmail.com Mon Nov 20 02:04:41 2006 From: sluggoster at gmail.com (Mike Orr) Date: Sun, 19 Nov 2006 17:04:41 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <4560F046.304@canterbury.ac.nz> References: <000501c70bf9$791c2b80$6402a8c0@arkdesktop> <4560F046.304@canterbury.ac.nz> Message-ID: <6e9196d20611191704k4a8a8e3dyf225c13d283a4461@mail.gmail.com> On 11/19/06, Greg Ewing wrote: > Andrew Koenig wrote: > > > That's a nice general sentiment, but not always feasible -- especially if > > you're writing library code. > > I've written a fair bit of library code, and I've never > found a pressing need to test for an abstract interface. > > In the rare cases where I've used type testing in an > API, it's always been for specific well-known concrete > types, such as strings and tuples -- never anything so > abstract as "iterator", "sequence", etc. Sometimes people don't realize how much they'd use a feature until it exists. I never thought I'd use generators until I started using them, and then it's "Wow, this is neat!" There have been several cases like this with Python. Another way people often test for sequence-ness: isinstance(obj, (list, tuple)) isinstance(obj, (list, tuple, UserList)) Here one buldes up all the likely classes. That's fine but we left out str because you have to stop somewhere. Usually I forego UserList because that depends on a module that may not be used, and often I forego tuple to make it shorter: "isinstance(obj, list)". That's fine but it's more restrictive than I want to me: my function *could* work with a tuple or list-like object too. Equality for tuples! Down with sequence discrimination! Support interfaces! :) -- Mike Orr From janssen at parc.com Mon Nov 20 02:07:29 2006 From: janssen at parc.com (Bill Janssen) Date: Sun, 19 Nov 2006 17:07:29 PST Subject: [Python-3000] Builtin iterator type In-Reply-To: <20061119115009.V14417@tribble.ilrt.bris.ac.uk> References: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FEEBB.9010100@cs.byu.edu> <20061119115009.V14417@tribble.ilrt.bris.ac.uk> Message-ID: <06Nov19.170737pst."58648"@synergy1.parc.xerox.com> > Java interfaces are very useful, however. Java programming seems to be > less and less about inheritance and more and more about implementing > interfaces; at least it does amongst Java programmers with taste :-) It seems to me that that's where Python has a real advantage. With real support for multiple inheritance, Python "interfaces" could be real classes (either like real Java classes or Java abstract classes), perhaps providing default implementations. You get the goodness of mix-ins, along with interface communication. Bill From guido at python.org Mon Nov 20 03:03:32 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 19 Nov 2006 18:03:32 -0800 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: On 11/19/06, Brett Cannon wrote: > > > On 11/19/06, Georg Brandl wrote: > > Guido van Rossum wrote: > > > > > PEPs aren't only for difficult discussions. :-) They are also there > > > for reference and to record agreement. Referring to an email isn't > > > really a very good answer when someone asks (as happened here) "what > > > is the spec"? A PEP may also discourage attempts to add more cruft, > > > and encourage someone with a few spare cycles to provide a patch so we > > > can put the whole thing behind it. > > > > > > BTW I forgot to mention that it shouldn't return anything. > > > > > > (So are you going to write that PEP? :-) > > > > It's in Subversion and numbered 3105, should be online with the next > > website update cycle. > > The only thing I would prefer is instead of 'end' have 'newline' and have > that be a boolean since I don't see a need to support different line > endings. I realize the difference between ``end=''`` and ``newline=False`` > is minimal beyond more typing, but for some reason my head just keeps > telling me I prefer the latter for clarity reasons. > > Otherwise +1 from me. Oh, but I *do* see a use case for other line endings. E.g. when writing a Windows file or a standards-compliant email from within Unix, end="\r\n" would be helpful. In practice it probably would be a variable set from a config option. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ncoghlan at gmail.com Mon Nov 20 10:23:42 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 20 Nov 2006 19:23:42 +1000 Subject: [Python-3000] Builtin iterator type In-Reply-To: <06Nov19.170242pst."58648"@synergy1.parc.xerox.com> References: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FD6C0.7000105@gmail.com> <06Nov19.170242pst."58648"@synergy1.parc.xerox.com> Message-ID: <4561741E.1040102@gmail.com> Bill Janssen wrote: >> Andrew Koenig wrote: >>> In other words, whether or not we choose to define a family of types that >>> stand for particular abilities, I think we should not use inheritance as the >>> mechanism for specifying that a particular class has a particular ability. >>> I don't know what the mechanism should be, but it shouldn't be inheritance. >>> >>> Am I missing something? >> It seems to me to be an excellent write-up of why using concrete classes for >> interface definitions is potentially problematic. > > Huh? Andrew pointed to exactly one "drawback", the inability to > disassociate an inherent characteristic of a type, its "typeness" > (which he refers to as ability-signalling), from a type. I hardly see > that as problematic. > > Am *I* missing something? Class inheritance implies a transitive relationship: if A is a B and B is a C, then A is also a C. Not all characteristics of a type are necessarily transitive in this fashion, so there are some things which cannot be effectively expressed by means of class inheritance. Consider the classic example of a taxonomy of birds and the expression of the property "this bird can fly" in a world containing emus, penguins and ostriches. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From jan.grant at bristol.ac.uk Mon Nov 20 10:39:37 2006 From: jan.grant at bristol.ac.uk (Jan Grant) Date: Mon, 20 Nov 2006 09:39:37 +0000 (GMT) Subject: [Python-3000] Builtin iterator type In-Reply-To: <06Nov19.170737pst."58648"@synergy1.parc.xerox.com> References: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FEEBB.9010100@cs.byu.edu> <20061119115009.V14417@tribble.ilrt.bris.ac.uk> <06Nov19.170737pst."58648"@synergy1.parc.xerox.com> Message-ID: <20061120093735.Q42166@tribble.ilrt.bris.ac.uk> On Sun, 19 Nov 2006, Bill Janssen wrote: > > Java interfaces are very useful, however. Java programming seems to be > > less and less about inheritance and more and more about implementing > > interfaces; at least it does amongst Java programmers with taste :-) > > It seems to me that that's where Python has a real advantage. With > real support for multiple inheritance, Python "interfaces" could be > real classes (either like real Java classes or Java abstract classes), > perhaps providing default implementations. You get the goodness of > mix-ins, along with interface communication. True, but the enforced separation of implementation and interface is pretty useful. Having said that, nothing to stop people declaring a completely abstract interface class, subclassing that to provide a default implementation. Again, comes down to a question of taste on a case-by-case basis. -- jan grant, ISYS, University of Bristol. http://www.bris.ac.uk/ Tel +44 (0)117 3317661 http://ioctl.org/jan/ Lambda calculus? I hardly know 'er! From bingham at cenix-bioscience.com Mon Nov 20 10:46:16 2006 From: bingham at cenix-bioscience.com (Aaron Bingham) Date: Mon, 20 Nov 2006 10:46:16 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: References: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FD6C0.7000105@gmail.com> Message-ID: <45617968.5050303@cenix-bioscience.com> Guido van Rossum wrote: >On 11/18/06, Nick Coghlan wrote: > > >>(Although I will point out >>that most protocols for things like len() *do* involve checks for special >>methods by name, and the check for iterability is typically a non-destructive >>call to iter(x), rather than a destructive one to x.next()). >> >> > >Ouch?! I would never check for iterability explicitly. I would just >require it, as Greg Ewing says. iter() could be fairly expensive, >depending on what is being iterated over. > >Maybe this is unique to iterators and iterators are a bad example? I >don't feel as strongly about this when testing e.g. for string-ness, >file-ness or list-ness. While I tend to dislike those tests too, they >are sometimes inevitable when we want to overload an API. Especially >file/string is often overloaded, and I see no big problem with this (I >do dislike overloading list/dict, or string/list-of-strings). > Might I point out that with Design by Contract, it is impossible to write a correct and complete precondition if nondestructive tests are not available. Regards, -- -------------------------------------------------------------------- Aaron Bingham Senior Software Engineer Cenix BioScience GmbH -------------------------------------------------------------------- From bingham at cenix-bioscience.com Mon Nov 20 10:47:59 2006 From: bingham at cenix-bioscience.com (Aaron Bingham) Date: Mon, 20 Nov 2006 10:47:59 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: <4561741E.1040102@gmail.com> References: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FD6C0.7000105@gmail.com> <06Nov19.170242pst."58648"@synergy1.parc.xerox.com> <4561741E.1040102@gmail.com> Message-ID: <456179CF.1060201@cenix-bioscience.com> Nick Coghlan wrote: >Bill Janssen wrote: > > >>>Andrew Koenig wrote: >>> >>> >>>>In other words, whether or not we choose to define a family of types that >>>>stand for particular abilities, I think we should not use inheritance as the >>>>mechanism for specifying that a particular class has a particular ability. >>>>I don't know what the mechanism should be, but it shouldn't be inheritance. >>>> >>>>Am I missing something? >>>> >>>> >>>It seems to me to be an excellent write-up of why using concrete classes for >>>interface definitions is potentially problematic. >>> >>> >>Huh? Andrew pointed to exactly one "drawback", the inability to >>disassociate an inherent characteristic of a type, its "typeness" >>(which he refers to as ability-signalling), from a type. I hardly see >>that as problematic. >> >>Am *I* missing something? >> >> > >Class inheritance implies a transitive relationship: if A is a B and B is a C, >then A is also a C. Not all characteristics of a type are necessarily >transitive in this fashion, so there are some things which cannot be >effectively expressed by means of class inheritance. > >Consider the classic example of a taxonomy of birds and the expression of the >property "this bird can fly" in a world containing emus, penguins and ostriches. > You seem to be imagining a class hierarchy like this: class Bird1(object): def fly(self): # default implementation .... class Finch1(Bird1): pass class Ostrich1(Bird1): def fly(self): # oops, this doesn't make sense, and there is nothing we can do # except raise an error which will surprise clients expecting a generic Bird1 raise RuntimeError("Ostriches can't fly") However, of you know ahead of time that not all birds can fly you can design for this. class FlightlessBirdError(Exception): """Raised when a flightless bird attempts to fly""" pass class Bird2(object): def flightless(self): return False def fly(self): """Only valid if not flightless, otherwise raises FlightlessBirdError""" # default implementation if self.flightless(): raise FlightlessBirdError() ... class Finch2(Bird2): pass class Ostrich2(Bird2): def flightless(self): return True Now the possibility of flightlessness has been made explicit in the design and a client will not be surprised by failures caused by attempts to call fly() on a flightless bird. Regards, -- -------------------------------------------------------------------- Aaron Bingham Senior Software Engineer Cenix BioScience GmbH -------------------------------------------------------------------- From bingham at cenix-bioscience.com Mon Nov 20 11:37:46 2006 From: bingham at cenix-bioscience.com (Aaron Bingham) Date: Mon, 20 Nov 2006 11:37:46 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: <06Nov19.170737pst."58648"@synergy1.parc.xerox.com> References: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FEEBB.9010100@cs.byu.edu> <20061119115009.V14417@tribble.ilrt.bris.ac.uk> <06Nov19.170737pst."58648"@synergy1.parc.xerox.com> Message-ID: <4561857A.8080101@cenix-bioscience.com> Bill Janssen wrote: >>Java interfaces are very useful, however. Java programming seems to be >>less and less about inheritance and more and more about implementing >>interfaces; at least it does amongst Java programmers with taste :-) >> >> > >It seems to me that that's where Python has a real advantage. With >real support for multiple inheritance, Python "interfaces" could be >real classes (either like real Java classes or Java abstract classes), >perhaps providing default implementations. You get the goodness of >mix-ins, along with interface communication. > I agree. In Java, interfaces are necessary because multiple inheritance is not supported. I see no good reason to add an additional language mechanism for interfaces when multiple inheritance would do the job, AFAICT. Regards, -- -------------------------------------------------------------------- Aaron Bingham Senior Software Engineer Cenix BioScience GmbH -------------------------------------------------------------------- From and-dev at doxdesk.com Mon Nov 20 11:59:20 2006 From: and-dev at doxdesk.com (Andrew Clover) Date: Mon, 20 Nov 2006 10:59:20 +0000 Subject: [Python-3000] Extending warnings for Py3K - any interest? Message-ID: <45618A88.1040905@doxdesk.com> I've always felt slightly disappointed by the Python warnings interface. It's great for managing language feature deprecation, but I've often wanted something I could also use for recoverable errors in general. I've written multiple application-specific mechanisms for allowing a caller to customise handling of potentially recoverable conditions in the past, so I think it's probably a fairly common use case, and maybe, if others how found the same, worth standardising as an extension of the existing Python warnings system. Here's a first sketch of the sort of thing I'm thinking of: http://doxdesk.com/file/software/py/v/warnings2-0.1.py Features: - warnings as instances with arbitrary properties, rather than being limited to a string message; - different severities with different default actions, so trivial warnings can be ignored by default and things which are by default errors can be recovered if a filter says so; - option to store warning instances that occur for later handling; - switchable warnings-handling contexts (like eg. the decimal module) for temporary changes to filters; - context manager helpers so you can say things like: with ignoring(DeprecationWarning): import oldmodule with storing(InvalidDataWarning) as problems: reghive.scanKeys() for problem in problems: ... Anyone agree/disagree with the idea of adding such functionality to warnings, or what features are worthwhile? -- And Clover mailto:and at doxdesk.com http://www.doxdesk.com/ From murman at gmail.com Mon Nov 20 15:25:27 2006 From: murman at gmail.com (Michael Urman) Date: Mon, 20 Nov 2006 08:25:27 -0600 Subject: [Python-3000] Builtin iterator type In-Reply-To: <456179CF.1060201@cenix-bioscience.com> References: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FD6C0.7000105@gmail.com> <4561741E.1040102@gmail.com> <456179CF.1060201@cenix-bioscience.com> Message-ID: > However, of you know ahead of time that not all birds can fly you can > design for this. To use a more relevant example, how about file-like-objects and nameless files. Any class derived from file can be expected to have the name member. However several files have nonsense names: >>> f = tempfile.TemporaryFile() >>> isinstance(f, file) True >>> f.name, sys.stdin.name, sys.stdout.name ('', '', '') Furthermore arbitrary file-like objects may or may not see it as necessary to provide a name attribute: >>> f = urllib.urlopen('http://www.google.com') >>> isinstance(f, file) False >>> f.read(62) '", line 1, in ? AttributeError: addinfourl instance has no attribute 'name' I'm skeptical that class hierarchy design time is a good time to target. Class hierarchies should be relatively rare in python, maintenance is more common, and developers new to your class hierarchy not knowing the special cases are likely to make incorrect assumptions. Both of the following are wrong, but are easy assumptions to have made: 1) All file-like objects have a usable name 2) All file-like objects have a name attribute To tie this back to the duck vs. lawyer context, if a function created by someone else is going to read data from a file object, which would make it easiest for you to use in legitimate ways the other programmer hadn't planned? 1) It checks for isinstance(obj, file), 2) It opens a copy with file(obj.name, 'r'), 3) It allows filenames by opening file(obj, 'r'), 4) It allows filedata by creating a StringIO(obj), 5) It requires the (fuzzy) right thing and calls obj.read(), or 6) It does 5 with a fallback of 4 or 3 if it can't find obj.read. I've set this scenario to make 6 the obvious answer, but it should be clear that there isn't a single winner between 3 and 4 across all scenarios. -- Michael Urman http://www.tortall.net/mu/blog From rasky at develer.com Mon Nov 20 15:58:44 2006 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 20 Nov 2006 15:58:44 +0100 Subject: [Python-3000] print() parameters in py3k References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: <041601c70cb4$5ff11470$cf09f01b@bagio> Guido van Rossum wrote: >> The only thing I would prefer is instead of 'end' have 'newline' and >> have that be a boolean since I don't see a need to support different >> line endings. I realize the difference between ``end=''`` and >> ``newline=False`` is minimal beyond more typing, but for some reason >> my head just keeps telling me I prefer the latter for clarity >> reasons. >> >> Otherwise +1 from me. > > Oh, but I *do* see a use case for other line endings. E.g. when > writing a Windows file or a standards-compliant email from within > Unix, end="\r\n" would be helpful. In practice it probably would be a > variable set from a config option. Uhm, but then, why not simply: println(x,y,z) -> append "\n" print(x,y,z) -> no terminator print(x,y,z,"\r\n") -> custom terminator Giovanni Bajo From walter at livinglogic.de Mon Nov 20 16:46:20 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 20 Nov 2006 16:46:20 +0100 Subject: [Python-3000] Extending warnings for Py3K - any interest? In-Reply-To: <45618A88.1040905@doxdesk.com> References: <45618A88.1040905@doxdesk.com> Message-ID: <4561CDCC.8050903@livinglogic.de> Andrew Clover wrote: > I've always felt slightly disappointed by the Python warnings interface. > It's great for managing language feature deprecation, but I've often > wanted something I could also use for recoverable errors in general. > > I've written multiple application-specific mechanisms for allowing a > caller to customise handling of potentially recoverable conditions in > the past, so I think it's probably a fairly common use case, and maybe, > if others how found the same, worth standardising as an extension of the > existing Python warnings system. > > Here's a first sketch of the sort of thing I'm thinking of: > > http://doxdesk.com/file/software/py/v/warnings2-0.1.py > > Features: > > - warnings as instances with arbitrary properties, rather than being > limited to a string message; That's already possible: class MyWarning(Warning): ... warnings.warn(MyWarning(...)) > - different severities with different default actions, so trivial > warnings can be ignored by default and things which are by default > errors can be recovered if a filter says so; > > - option to store warning instances that occur for later handling; > > - switchable warnings-handling contexts (like eg. the decimal module) > for temporary changes to filters; > > - context manager helpers so you can say things like: > > with ignoring(DeprecationWarning): > import oldmodule > > with storing(InvalidDataWarning) as problems: > reghive.scanKeys() > for problem in problems: ... +1 > Anyone agree/disagree with the idea of adding such functionality to > warnings, or what features are worthwhile? Another idea: Actions might be callables, so that they can be used directly in handleWarning() instead of dispatching on the action name. BTW, your source code is not PEP 8 compatible. Servus, Walter From barry at python.org Mon Nov 20 16:54:51 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 20 Nov 2006 10:54:51 -0500 Subject: [Python-3000] print() parameters in py3k In-Reply-To: <041601c70cb4$5ff11470$cf09f01b@bagio> References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> <041601c70cb4$5ff11470$cf09f01b@bagio> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Nov 20, 2006, at 9:58 AM, Giovanni Bajo wrote: > Guido van Rossum wrote: > >>> The only thing I would prefer is instead of 'end' have 'newline' and >>> have that be a boolean since I don't see a need to support different >>> line endings. I realize the difference between ``end=''`` and >>> ``newline=False`` is minimal beyond more typing, but for some reason >>> my head just keeps telling me I prefer the latter for clarity >>> reasons. >>> >>> Otherwise +1 from me. >> >> Oh, but I *do* see a use case for other line endings. E.g. when >> writing a Windows file or a standards-compliant email from within >> Unix, end="\r\n" would be helpful. In practice it probably would be a >> variable set from a config option. > > Uhm, but then, why not simply: > > println(x,y,z) -> append "\n" > print(x,y,z) -> no terminator > print(x,y,z,"\r\n") -> custom terminator I'd like to at least see a discussion of a more printf() or logging style for print function. Specifically, I wonder if it wouldn't be useful to require a format string with positional and keyword arguments being used for automatic interpolation. E.g. print('%s: %s', field, value) instead of print('%s: %s' % (field, value)) There are lots of questions to answer, such as whether to use $- strings and require keyword arguments. Another thought: what if 'print' weren't a function but a callable object. exposed in builtins. I'm thinking something roughly parallel to stdout, stderr, stdin, or maybe better cout, cerr, cin. The default 'print' object then would be a "print-to-sys.stdout-with- \n", but you would then be able to do something like: stderr = print.clone(stream=sys.stderr) stderr('here go all my error messages') or smtpout = print.clone(stream=mysock, end='\r\n') smtpout('$code $msg', code=250, msg='Okay') Perhaps you could even hook in i18n by doing something like: print = print.clone(translate=gettext.gettext) print('$who likes to eat $what', person, food) - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRWHP0HEjvBPtnXfVAQJ8HgQAoKfkh94Js4MXMRgjAm3uFHvNxYBrb99k Je8PkICx1QXhkSH9Kq3KJ3AbWXx/EKQFycfXun9ZxXy/hax7RcsDHJRsR9H1moR7 6IiWBEJJ1IqAjCfTNCKhP4VMxQtq6DFb5rRoN9jfe5pAJeA7SqCM0Fqcdf8F0stG OWQtK+UIjLU= =MhoH -----END PGP SIGNATURE----- From fredrik at pythonware.com Mon Nov 20 17:56:03 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Mon, 20 Nov 2006 17:56:03 +0100 Subject: [Python-3000] print() parameters in py3k In-Reply-To: <041601c70cb4$5ff11470$cf09f01b@bagio> References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> <041601c70cb4$5ff11470$cf09f01b@bagio> Message-ID: Giovanni Bajo wrote: > Uhm, but then, why not simply: > > println(x,y,z) -> append "\n" > print(x,y,z) -> no terminator > print(x,y,z,"\r\n") -> custom terminator hey, I proposed that earlier today, but something ate my post ;-) while we're at it, why not add a printf as well, btw? (defined as def printf(fmt, *args): print(fmt % args)). From gsakkis at rutgers.edu Mon Nov 20 18:06:58 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Mon, 20 Nov 2006 12:06:58 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: References: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FD6C0.7000105@gmail.com> <4561741E.1040102@gmail.com> <456179CF.1060201@cenix-bioscience.com> Message-ID: <91ad5bf80611200906y394e6257jd8bd5354d5b98a66@mail.gmail.com> On 11/20/06, Michael Urman wrote: > > However, of you know ahead of time that not all birds can fly you can > > design for this. > > To use a more relevant example, how about file-like-objects and > nameless files. Any class derived from file can be expected to have > the name member. However several files have nonsense names: > > >>> f = tempfile.TemporaryFile() > >>> isinstance(f, file) > True > >>> f.name, sys.stdin.name, sys.stdout.name > ('', '', '') > > Furthermore arbitrary file-like objects may or may not see it as > necessary to provide a name attribute: > > >>> f = urllib.urlopen('http://www.google.com') > >>> isinstance(f, file) > False > >>> f.read(62) > '", line 1, in ? > AttributeError: addinfourl instance has no attribute 'name' > > I'm skeptical that class hierarchy design time is a good time to > target. Class hierarchies should be relatively rare in python, > maintenance is more common, and developers new to your class hierarchy > not knowing the special cases are likely to make incorrect > assumptions. Both of the following are wrong, but are easy assumptions > to have made: > > 1) All file-like objects have a usable name > 2) All file-like objects have a name attribute One issue here is the (informal) name of the protocol "file-like", which although it adds the suffix "like" and gives a clue it's not necessarily about real files, it makes such erroneous assumptions more likely. OTOH if we were talking about, say, "character streams", the expectation of having a name would be less justified. By the way, file-like is a good example of a protocol that would benefit from being less fuzzy and informal. For one, it would give a clear definition of the protocol attributes; you can't know this just by looking at the concrete file type and assume that all file-like objects should support the same attributes ('name' is one example that makes sense to real files mainly). For two, it would give default implementations for methods that can be defined through others, e.g. readline() and readlines() can be implemented using read() (of course a subclass may override these to provide more efficient implementations, just as with dictmixin). Anyone else that would find a 'stream' (or 'charstream' or 'StreamMixin' or whatever) class a good idea? George From jcarlson at uci.edu Mon Nov 20 18:14:55 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 20 Nov 2006 09:14:55 -0800 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <041601c70cb4$5ff11470$cf09f01b@bagio> Message-ID: <20061120082547.8346.JCARLSON@uci.edu> Barry Warsaw wrote: > On Nov 20, 2006, at 9:58 AM, Giovanni Bajo wrote: > > Uhm, but then, why not simply: > > > > println(x,y,z) -> append "\n" > > print(x,y,z) -> no terminator > > print(x,y,z,"\r\n") -> custom terminator I don't like the idea of 2 builtins to print (-1). It doesn't feel right. Then again, I'm not particularly interested in a print function either (-0). > I'd like to at least see a discussion of a more printf() or logging > style for print function. Specifically, I wonder if it wouldn't be > useful to require a format string with positional and keyword > arguments being used for automatic interpolation. E.g. > > print('%s: %s', field, value) > > instead of > > print('%s: %s' % (field, value)) How much print statement rewriting do you want to do? Me, I'd like to get away with as little as possible. Adding parenthesis isn't that bad. Changing everything to use a printf-style would be 1) annoying, 2) unnecessary, and 3) I hate printf. Not to say that #3 is a sufficient reason for everyone else, but when I sit down to write C, among the standard annoyances I wish I didn't have to deal with is that printf _requires_ that damn format string. I understand it is a requirement due to underlying language limitations, but Python has no such limitations, so let us not burdon Python with the braindead nature of C's printf. I would also mention that there would be no way to distinguish between the two cases you specify above (within print()), unless we are going to remove % formatting, which would be a mistake. > There are lots of questions to answer, such as whether to use $- > strings and require keyword arguments. Bad idea. Unless we are changing all of string formatting to use $, then changing the default string formatting of one of the likely most used functions, is at best confusing, and certainly gratuitous breakage. Then again, I never saw that using required parens and a specifier like %(...)s was a burden... > Another thought: what if 'print' weren't a function but a callable > object. exposed in builtins. I'm thinking something roughly parallel > to stdout, stderr, stdin, or maybe better cout, cerr, cin. We still wouldn't be able to get "cin << bar" without frame hacking, and I would be content to never see "cout >> foo" or "cin << bar" ever again. The 2.x "convenience syntax" of print >>foo, ... I think was a decent feature, brought on by the desire to make print usable for non-stdout streams. cout/cerr/cin, almost by definition, are C++ hacks and/or misfeatures, and adding them to Python seems to be silly to me; regardless of the 5 (or fewer) C++ users who don't know how to use C's printf, or who wouldn't be able to understand Python's print(). We've got raw_input() and print(). If users want alternate functionality, they can write their own print function and even assign it to __builtins__.print . > The default 'print' object then would be a "print-to-sys.stdout-with- > \n", but you would then be able to do something like: > > stderr = print.clone(stream=sys.stderr) > stderr('here go all my error messages') > > or > > smtpout = print.clone(stream=mysock, end='\r\n') > smtpout('$code $msg', code=250, msg='Okay') I guess I don't really see either of these as significant improvements, if any, to the guido-defined print function. > Perhaps you could even hook in i18n by doing something like: > > print = print.clone(translate=gettext.gettext) > print('$who likes to eat $what', person, food) Except that would need to be... print('$who likes to eat $what', who=person, what=food) This is the only feature that I raise my eyebrow and say, 'that would be somewhat convenient', but with print as a function, users can _easily_ do that by themselves... _print = print def print_(line, **kwargs): _print(gettext.gettext(line), **kwargs) __builtins__.print = print_ Heck, toss a couple fairly simple examples into the print() documentation, the cookbook, and/or some module, and I think user desire would be satisfied. - Josiah From guido at python.org Mon Nov 20 19:11:11 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 20 Nov 2006 10:11:11 -0800 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> <041601c70cb4$5ff11470$cf09f01b@bagio> Message-ID: On 11/20/06, Fredrik Lundh wrote: > Giovanni Bajo wrote: > > > Uhm, but then, why not simply: > > > > println(x,y,z) -> append "\n" > > print(x,y,z) -> no terminator > > print(x,y,z,"\r\n") -> custom terminator > > hey, I proposed that earlier today, but something ate my post ;-) I don't like that as much; it doesn't have a way to change the separator between items, so if you want your items flush together you'll have to say print(x); print(y); println(z) The third example above has a minor flaw in that it inserts a space before the "\r\n". FWIW, in case anyone proposes to "solve" this by not automatically inserting spaces between items by default, that was quickly voted out of the race the first time we went through this. > while we're at it, why not add a printf as well, btw? (defined as > def printf(fmt, *args): print(fmt % args)). Fine with me, although print-hating folks will probably hate it more. BTW printf() should not automatically append a newline (can't tell whether you intended it to do or not -- that depends on whose definition of print() you were referring, mind or Giovanni's). Also, it should have a file=... keyword parameter. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Nov 20 19:21:34 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 20 Nov 2006 10:21:34 -0800 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> <041601c70cb4$5ff11470$cf09f01b@bagio> Message-ID: On 11/20/06, Barry Warsaw wrote: > I'd like to at least see a discussion of a more printf() or logging > style for print function. Agreed, as long as it isn't presented as an alternative to the more basic print() function. We don't want to force students making their first forays into programming to have to learn format strings. Having printf() for more advanced use together with print() as defined in PEP 3105 makes sense to me. > Specifically, I wonder if it wouldn't be > useful to require a format string with positional and keyword > arguments being used for automatic interpolation. E.g. > > print('%s: %s', field, value) > > instead of > > print('%s: %s' % (field, value)) Before going too far down this route, first read PEP 3101, which proposes a different way out of the issues around the % operator. (Not saying we couldn't adopt PEP 3101 *and* a printf() function. > There are lots of questions to answer, such as whether to use $- > strings and require keyword arguments. IMO it should follow PEP 3101 exactly; I think that answers any questions you might have in this realm. > Another thought: what if 'print' weren't a function but a callable > object. exposed in builtins. I'm thinking something roughly parallel > to stdout, stderr, stdin, or maybe better cout, cerr, cin. I'm not sure I follow the difference. It was quite clear from earlier discussions that we *don't* want print to be a bound method of a hypothetical print method on sys.stdout; that's much less flexible, and makes it much more work to implement a file-like type. print() should dynamically look up sys.stdout each time it is called (and exactly once per call). (BTW I'm not sure why you like cin/cout/cerr better than stdin/stdout/stderr?) > The default 'print' object then would be a "print-to-sys.stdout-with- > \n", but you would then be able to do something like: > > stderr = print.clone(stream=sys.stderr) > stderr('here go all my error messages') > > or > > smtpout = print.clone(stream=mysock, end='\r\n') > smtpout('$code $msg', code=250, msg='Okay') I'd like to firmly resist adding more mechanism for somethign that can be addressed so simply by a tiny wrapper function: def smtpout(*args): print(*args, file=mysock, end="\r\n") > Perhaps you could even hook in i18n by doing something like: > > print = print.clone(translate=gettext.gettext) > print('$who likes to eat $what', person, food) Ditto**2. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Nov 20 19:27:05 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 20 Nov 2006 10:27:05 -0800 Subject: [Python-3000] Abilities / Interfaces Message-ID: I'd like to have a discussion of abilities and interfaces but right now I don't have time -- maybe tomorrow. I see Andrew's proposal as mostly isomorphic to Zope's notion of interfaces, but the new terminology may make it possible to clarify certain distinctions better, e.g. the distinction between a class that provides an ability to its instances, and the abilities that an instance has. Feel free to riff on this theme here. I'll check in within 24 hours or so. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From brett at python.org Mon Nov 20 19:42:58 2006 From: brett at python.org (Brett Cannon) Date: Mon, 20 Nov 2006 10:42:58 -0800 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: On 11/19/06, Guido van Rossum wrote: > > On 11/19/06, Brett Cannon wrote: > > > > > > On 11/19/06, Georg Brandl wrote: > > > Guido van Rossum wrote: > > > > > > > PEPs aren't only for difficult discussions. :-) They are also there > > > > for reference and to record agreement. Referring to an email isn't > > > > really a very good answer when someone asks (as happened here) "what > > > > is the spec"? A PEP may also discourage attempts to add more cruft, > > > > and encourage someone with a few spare cycles to provide a patch so > we > > > > can put the whole thing behind it. > > > > > > > > BTW I forgot to mention that it shouldn't return anything. > > > > > > > > (So are you going to write that PEP? :-) > > > > > > It's in Subversion and numbered 3105, should be online with the next > > > website update cycle. > > > > The only thing I would prefer is instead of 'end' have 'newline' and > have > > that be a boolean since I don't see a need to support different line > > endings. I realize the difference between ``end=''`` and > ``newline=False`` > > is minimal beyond more typing, but for some reason my head just keeps > > telling me I prefer the latter for clarity reasons. > > > > Otherwise +1 from me. > > Oh, but I *do* see a use case for other line endings. E.g. when > writing a Windows file or a standards-compliant email from within > Unix, end="\r\n" would be helpful. In practice it probably would be a > variable set from a config option. Right, I assumed it would always get set by some config option that was set during compile. But whatever, as I said it is not a big deal to me and I can definitely be happy with the existing proposal. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061120/f068d41c/attachment.htm From barry at python.org Mon Nov 20 20:03:47 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 20 Nov 2006 14:03:47 -0500 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> <041601c70cb4$5ff11470$cf09f01b@bagio> Message-ID: <64AE2078-2ABF-4975-95B8-5698EED1EEFC@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Nov 20, 2006, at 1:21 PM, Guido van Rossum wrote: > On 11/20/06, Barry Warsaw wrote: >> I'd like to at least see a discussion of a more printf() or logging >> style for print function. > > Agreed, as long as it isn't presented as an alternative to the more > basic print() function. We don't want to force students making their > first forays into programming to have to learn format strings. Having > printf() for more advanced use together with print() as defined in PEP > 3105 makes sense to me. That seems reasonable to me too (separate print() and printf() thingies). >> Specifically, I wonder if it wouldn't be >> useful to require a format string with positional and keyword >> arguments being used for automatic interpolation. E.g. >> >> print('%s: %s', field, value) >> >> instead of >> >> print('%s: %s' % (field, value)) > > Before going too far down this route, first read PEP 3101, which > proposes a different way out of the issues around the % operator. (Not > saying we couldn't adopt PEP 3101 *and* a printf() function. Thanks for the pointer. I agree that if we adopt a companion printf () function it should definitely require (only) PEP 3101 format strings. >> Another thought: what if 'print' weren't a function but a callable >> object. exposed in builtins. I'm thinking something roughly parallel >> to stdout, stderr, stdin, or maybe better cout, cerr, cin. > > I'm not sure I follow the difference. It was quite clear from earlier > discussions that we *don't* want print to be a bound method of a > hypothetical print method on sys.stdout; that's much less flexible, > and makes it much more work to implement a file-like type. print() > should dynamically look up sys.stdout each time it is called (and > exactly once per call). > > (BTW I'm not sure why you like cin/cout/cerr better than stdin/ > stdout/stderr?) I was trying to make an analogy (probably poorly so) that 'print' would be bound to an instance as opposed to a function (current proposal) or structure-like thingie without methods as in std*. >> The default 'print' object then would be a "print-to-sys.stdout-with- >> \n", but you would then be able to do something like: >> >> stderr = print.clone(stream=sys.stderr) >> stderr('here go all my error messages') >> >> or >> >> smtpout = print.clone(stream=mysock, end='\r\n') >> smtpout('$code $msg', code=250, msg='Okay') > > I'd like to firmly resist adding more mechanism for somethign that can > be addressed so simply by a tiny wrapper function: > > def smtpout(*args): > print(*args, file=mysock, end="\r\n") What I was trying to address was the namespace collisions you'd get between printf-style keyword arguments and 'special' arguments that the print function would accept. The thing is that if we were to add both a print and a printf, you'd like for their interfaces to be as similar as possible except that the latter takes a format string as its first argument. Clearly 'special' arguments like 'file' and 'end' will prohibit their use as keyword arguments for printf, and I don't like that (this isn't anything new -- the problem crops up occasionally in other places, see the email package API). One way out would be to use special argument names, e.g. prepending and or appending some number of underscores. This doesn't eliminate the problem, but it reduces its likelihood. Another approach would for 'print' to be an object with attributes that would change behavior, such as the stream to write to or line-ending characters. A possible third approach might be some special singleton objects that, when seen positionally would modify the state of the underlying print object. Of the three (maybe fourth: "sorry you can't use 'end' or 'file'"), it seems preferable to use a callable print object. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRWH8GXEjvBPtnXfVAQL0BQP/SoF2ETUnUlept5pPwzzeAQdOJjcKuEjL 3saKhz0QAVfCktxTTFK4hZQGeYHKLTSWemr3A+zl82lc58rmSGr8OemXDLXy1J7l PplFkp4iBrT8mLKSFL2c397IKg0M/hbfb3Iw2NqNBu8h2pA1d1bAozBR+nNClUGP z6vCi1zfYvk= =7G56 -----END PGP SIGNATURE----- From jimjjewett at gmail.com Mon Nov 20 20:05:21 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 20 Nov 2006 14:05:21 -0500 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: On 11/19/06, Guido van Rossum wrote: > On 11/19/06, Georg Brandl wrote: > > Guido van Rossum wrote: > > > 2. A keyword parameter will direct it to a different file. > > > Probably print(x, y, z, file=). > > > 3. Two more keyword parameters can change what gets inserted between > > > items and at the end. > > > Perhaps print(x, y, z, sep=, end=). These default to " > > > " and "\n", respectively. > A PEP may also discourage attempts to add more cruft, (1) Is this an explicit rejection of a keyword for a format string, such as print (x, y, z, fmt="My X: %s, Your Y: %s, His Z: %s") (2) What happens to additional keyword arguments that do get passed? (a) A StandardError of some sort is raised. (b) print is treated as though it had (but ignored) a **kwargs argument, to support extensions in print-replacements. (c) The extra arguments are printed at the end, perhaps with a label. -jJ From barry at python.org Mon Nov 20 20:11:34 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 20 Nov 2006 14:11:34 -0500 Subject: [Python-3000] print() parameters in py3k In-Reply-To: <20061120082547.8346.JCARLSON@uci.edu> References: <041601c70cb4$5ff11470$cf09f01b@bagio> <20061120082547.8346.JCARLSON@uci.edu> Message-ID: <46B9A351-2664-4B58-86C6-BB2D9A8DDF7F@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Nov 20, 2006, at 12:14 PM, Josiah Carlson wrote: > How much print statement rewriting do you want to do? Me, I'd like to > get away with as little as possible. Adding parenthesis isn't that > bad. > Changing everything to use a printf-style would be 1) annoying, 2) > unnecessary, and 3) I hate printf. Are we going to include a flawless transformation script for Python 3k? If not, then I'm pretty much resigned to rewriting them by hand anyway. But I think this would be solved by adopting separate print() and printf() thingies. >> There are lots of questions to answer, such as whether to use $- >> strings and require keyword arguments. > > Bad idea. Unless we are changing all of string formatting to use $, > then changing the default string formatting of one of the likely most > used functions, is at best confusing, and certainly gratuitous > breakage. > Then again, I never saw that using required parens and a specifier > like > %(...)s was a burden... I withdraw the suggestion to use $-strings in the format string. If we get printf, it should only use PEP 3101 format strings. I still believe that $-strings will be useful because they target a different audience. The motivating use case for PEP 292 was that translators are not (always) programmers. Programmers can handle %()s substitutions or PEP 3101 format strings, but translators have lots of problems with them. It makes sense to keep the uber-powerful PEP 3101 format strings for programmers and simple PEP 292 strings for non-programmers. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRWH953EjvBPtnXfVAQIA2QQAtSF0djvOIJQCUegtifmIn3w+SG6epykC BDL6mI0T7TbSKyQLQxgKBkqTzGnJCcCa4eWRi0ATOZyPMTU8YmZBB70SLzLOMaqT LAYl77i+PCRxO6DCS2PQuCltv1LuMi6x36KtJ4BM0sMXDK51f/uykz6z8F+qfQpO Okt98c0EiqY= =eYRm -----END PGP SIGNATURE----- From brett at python.org Mon Nov 20 20:30:31 2006 From: brett at python.org (Brett Cannon) Date: Mon, 20 Nov 2006 11:30:31 -0800 Subject: [Python-3000] print() parameters in py3k In-Reply-To: <64AE2078-2ABF-4975-95B8-5698EED1EEFC@python.org> References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> <041601c70cb4$5ff11470$cf09f01b@bagio> <64AE2078-2ABF-4975-95B8-5698EED1EEFC@python.org> Message-ID: On 11/20/06, Barry Warsaw wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Nov 20, 2006, at 1:21 PM, Guido van Rossum wrote: > > > On 11/20/06, Barry Warsaw wrote: > >> I'd like to at least see a discussion of a more printf() or logging > >> style for print function. > > > > Agreed, as long as it isn't presented as an alternative to the more > > basic print() function. We don't want to force students making their > > first forays into programming to have to learn format strings. Having > > printf() for more advanced use together with print() as defined in PEP > > 3105 makes sense to me. > > That seems reasonable to me too (separate print() and printf() > thingies). > > >> Specifically, I wonder if it wouldn't be > >> useful to require a format string with positional and keyword > >> arguments being used for automatic interpolation. E.g. > >> > >> print('%s: %s', field, value) > >> > >> instead of > >> > >> print('%s: %s' % (field, value)) > > > > Before going too far down this route, first read PEP 3101, which > > proposes a different way out of the issues around the % operator. (Not > > saying we couldn't adopt PEP 3101 *and* a printf() function. > > Thanks for the pointer. I agree that if we adopt a companion printf > () function it should definitely require (only) PEP 3101 format strings. > > >> Another thought: what if 'print' weren't a function but a callable > >> object. exposed in builtins. I'm thinking something roughly parallel > >> to stdout, stderr, stdin, or maybe better cout, cerr, cin. > > > > I'm not sure I follow the difference. It was quite clear from earlier > > discussions that we *don't* want print to be a bound method of a > > hypothetical print method on sys.stdout; that's much less flexible, > > and makes it much more work to implement a file-like type. print() > > should dynamically look up sys.stdout each time it is called (and > > exactly once per call). > > > > (BTW I'm not sure why you like cin/cout/cerr better than stdin/ > > stdout/stderr?) > > I was trying to make an analogy (probably poorly so) that 'print' > would be bound to an instance as opposed to a function (current > proposal) or structure-like thingie without methods as in std*. > > >> The default 'print' object then would be a "print-to-sys.stdout-with- > >> \n", but you would then be able to do something like: > >> > >> stderr = print.clone(stream=sys.stderr) > >> stderr('here go all my error messages') > >> > >> or > >> > >> smtpout = print.clone(stream=mysock, end='\r\n') > >> smtpout('$code $msg', code=250, msg='Okay') > > > > I'd like to firmly resist adding more mechanism for somethign that can > > be addressed so simply by a tiny wrapper function: > > > > def smtpout(*args): > > print(*args, file=mysock, end="\r\n") > > What I was trying to address was the namespace collisions you'd get > between printf-style keyword arguments and 'special' arguments that > the print function would accept. The thing is that if we were to add > both a print and a printf, you'd like for their interfaces to be as > similar as possible except that the latter takes a format string as > its first argument. Clearly 'special' arguments like 'file' and > 'end' will prohibit their use as keyword arguments for printf, and I > don't like that (this isn't anything new -- the problem crops up > occasionally in other places, see the email package API). Why do they need to be similar? print() is just a really simple, quick output function. printf() seems to be proposed for advanced formatting needs for sending to stdout. As Guido said, the newline should be explicit in printf(), so that already begins to deviate from print(). And 'sep' would be useless. So all that leaves is 'file', and that is just convenience for passing a string to a 'write' method. And for that I think people can just learn not to use 'file' as a name of a variable. Or we can just lose that as well and have printf() be nothing more than a convenience function that avoids having to specify the method name and sending it to print with the desired arguments. Then again, this to me seems to make printf() not that grand to lead to its own built-in. But I have been fine with the way things have been so far anyway with having a separate line that stores some string I created before calling 'print'. If you don't have any keyword arguments then printf() is just:: def printf(format_string, *args, **kwargs): output = format_string.format(*args, **kwargs) print(output, end='') One way out would be to use special argument names, e.g. prepending > and or appending some number of underscores. This doesn't eliminate > the problem, but it reduces its likelihood. I don't see this being a big problem if the only argument that is common is 'file'. Another approach would > for 'print' to be an object with attributes that would change > behavior, such as the stream to write to or line-ending characters. Yuck. Why would you want to suddenly set global semantics in a built-in instead of a module? That's a total shift in how Python does things at a global level. Plus I don't want to have to worry about setting 'sep' or 'end' for every function that uses print() just because some other function decided that it didn't like single spaces but wanted to use tabs for a separator. A possible third approach might be some special singleton objects > that, when seen positionally would modify the state of the underlying > print object. Yuck again. This seems to be getting rather specific just for printing when we have already lived without all of this functionality for so long with minor issue. Of the three (maybe fourth: "sorry you can't use 'end' > or 'file'"), I like this option. =) -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061120/781abb99/attachment.htm From solipsis at pitrou.net Mon Nov 20 20:39:55 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 20 Nov 2006 20:39:55 +0100 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: Message-ID: <1164051595.4864.25.camel@fsol> Le lundi 20 novembre 2006 ? 10:27 -0800, Guido van Rossum a ?crit : > I'd like to have a discussion of abilities and interfaces but right > now I don't have time -- maybe tomorrow. I see Andrew's proposal as > mostly isomorphic to Zope's notion of interfaces, but the new > terminology may make it possible to clarify certain distinctions > better, e.g. the distinction between a class that provides an ability > to its instances, and the abilities that an instance has. > > Feel free to riff on this theme here. I'll check in within 24 hours or so. It doesn't directly answer your post, but I think the way an object/class is queried to know whether it implements a given interface should be quite free-form, which means it's up to the interface creator to decide what really he wants to check. For example, the creator of the Number interface may just check that Number is declared as an interface, without going any further. So, the author of Complex class just has to write: class Complex: __implements__ = (Number,) An interface creator who has a taste for "stricter" typing may on the other hand mandate that certain or attributes methods are defined, etc. He could then raise an error if a class claiming to implement his interface doesn't provide the required methods/attributes. So to sum it up, to know whether an object-or-class implements a given interface, you could call a standard function implements(obj, interface), which in turn would call interface.__implemented_by__(obj). It's up to each interface to decide what breadth of tests it wants to perform, although it would be practical to fallback on a default implementation (in the Interface base class common to all interfaces) which just checks for presence of the interface in obj.__implements__. (I don't know if this post is useful. Sorry if it's not) Antoine. From sluggoster at gmail.com Mon Nov 20 20:42:34 2006 From: sluggoster at gmail.com (Mike Orr) Date: Mon, 20 Nov 2006 11:42:34 -0800 Subject: [Python-3000] Builtin iterator type In-Reply-To: <91ad5bf80611200906y394e6257jd8bd5354d5b98a66@mail.gmail.com> References: <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FD6C0.7000105@gmail.com> <4561741E.1040102@gmail.com> <456179CF.1060201@cenix-bioscience.com> <91ad5bf80611200906y394e6257jd8bd5354d5b98a66@mail.gmail.com> Message-ID: <6e9196d20611201142q7bd65ca8l622ba42e9b9c0fa5@mail.gmail.com> On 11/20/06, George Sakkis wrote: > One issue here is the (informal) name of the protocol "file-like", > which although it adds the suffix "like" and gives a clue it's not > necessarily about real files, it makes such erroneous assumptions more > likely. This is an issue througout Python, one that people have just learned to live with. Generally it works because routines use only the minimum characteristic feature of the original type (file.read() or file.write(), dict.__getitem__()), and the user has assumed correctly which features are required (or has looked at the routine's source to verify what it calls). But obviously this fails if the routine calls dict.keys() or dict.pop() and the object doesn't have 100% dict compatibility. Older parts of the Python docs describe exactly which methods are called and with which arguments. For instance, "a writelines method that will be called with one argument, and a read method that will be called with no arguments". (Because some file-like objects leave out the optional numeric argument to .read().) But this is so wordy to document that most programmers don't go to this level. I can see the use for MinimalFileLikeForReading, MinimalFileLikeForWriting, and CompletelyFileLike interfaces. > Anyone else that would find > a 'stream' (or 'charstream' or 'StreamMixin' or whatever) class a good > idea? Possibly, although we should distinguish between reading and writing. The traceback module is a good example. It accepts file-like objects and calls only file.write. interface IReadStream: def read(self, num_bytes=None) => bytes object interface IWriteStream: def write(self, bytes) => None # Writes the bytes to the stream. ('bytes object' being the successor to 'str' strings in Python 3000.) The word "stream" may connote more than this to C/Java programmers -- we'd have to check. -- Mike Orr From g.brandl at gmx.net Mon Nov 20 20:42:28 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 20 Nov 2006 20:42:28 +0100 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: Jim Jewett wrote: > On 11/19/06, Guido van Rossum wrote: >> On 11/19/06, Georg Brandl wrote: >> > Guido van Rossum wrote: > >> > > 2. A keyword parameter will direct it to a different file. >> > > Probably print(x, y, z, file=). > >> > > 3. Two more keyword parameters can change what gets inserted between >> > > items and at the end. >> > > Perhaps print(x, y, z, sep=, end=). These default to " >> > > " and "\n", respectively. > >> A PEP may also discourage attempts to add more cruft, > > (1) Is this an explicit rejection of a keyword for a format string, such as > > print (x, y, z, fmt="My X: %s, Your Y: %s, His Z: %s") -1. This is confusing, as normally the format string comes first, and then the arguments to be formatted. > (2) What happens to additional keyword arguments that do get passed? > (a) A StandardError of some sort is raised. Why not the standard TypeError: "xyz" is an invalid keyword arg for this function? > (b) print is treated as though it had (but ignored) a **kwargs > argument, to support extensions in print-replacements. > (c) The extra arguments are printed at the end, perhaps with a label. What label do you have in mind? Georg From guido at python.org Mon Nov 20 20:52:28 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 20 Nov 2006 11:52:28 -0800 Subject: [Python-3000] print() parameters in py3k In-Reply-To: <64AE2078-2ABF-4975-95B8-5698EED1EEFC@python.org> References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> <041601c70cb4$5ff11470$cf09f01b@bagio> <64AE2078-2ABF-4975-95B8-5698EED1EEFC@python.org> Message-ID: On 11/20/06, Barry Warsaw wrote: > >> Another thought: what if 'print' weren't a function but a callable > >> object. exposed in builtins. I'm thinking something roughly parallel > >> to stdout, stderr, stdin, or maybe better cout, cerr, cin. > > > > I'm not sure I follow the difference. It was quite clear from earlier > > discussions that we *don't* want print to be a bound method of a > > hypothetical print method on sys.stdout; that's much less flexible, > > and makes it much more work to implement a file-like type. print() > > should dynamically look up sys.stdout each time it is called (and > > exactly once per call). > > > > (BTW I'm not sure why you like cin/cout/cerr better than stdin/ > > stdout/stderr?) > > I was trying to make an analogy (probably poorly so) that 'print' > would be bound to an instance as opposed to a function (current > proposal) or structure-like thingie without methods as in std*. Sorry, I still don't understand your proposal. > >> The default 'print' object then would be a "print-to-sys.stdout-with- > >> \n", but you would then be able to do something like: > >> > >> stderr = print.clone(stream=sys.stderr) > >> stderr('here go all my error messages') > >> > >> or > >> > >> smtpout = print.clone(stream=mysock, end='\r\n') > >> smtpout('$code $msg', code=250, msg='Okay') > > > > I'd like to firmly resist adding more mechanism for somethign that can > > be addressed so simply by a tiny wrapper function: > > > > def smtpout(*args): > > print(*args, file=mysock, end="\r\n") > > What I was trying to address was the namespace collisions you'd get > between printf-style keyword arguments and 'special' arguments that > the print function would accept. The thing is that if we were to add > both a print and a printf, you'd like for their interfaces to be as > similar as possible except that the latter takes a format string as > its first argument. Clearly 'special' arguments like 'file' and > 'end' will prohibit their use as keyword arguments for printf, and I > don't like that (this isn't anything new -- the problem crops up > occasionally in other places, see the email package API). > > One way out would be to use special argument names, e.g. prepending > and or appending some number of underscores. This doesn't eliminate > the problem, but it reduces its likelihood. Another approach would > for 'print' to be an object with attributes that would change > behavior, such as the stream to write to or line-ending characters. > A possible third approach might be some special singleton objects > that, when seen positionally would modify the state of the underlying > print object. Of the three (maybe fourth: "sorry you can't use 'end' > or 'file'"), it seems preferable to use a callable print object. I still don't quite see how making print a callable object solves that issue. But personally would be totally happy with "You can't use 'file' as a variable name in your format". ('end' and 'sep' should not be supported by printf(), these should just be made part of the format string.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Mon Nov 20 21:13:58 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 20 Nov 2006 15:13:58 -0500 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> <041601c70cb4$5ff11470$cf09f01b@bagio> <64AE2078-2ABF-4975-95B8-5698EED1EEFC@python.org> Message-ID: <46A6517D-6EA1-4BD2-94F7-02D3C7A4AB64@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Nov 20, 2006, at 2:52 PM, Guido van Rossum wrote: > I still don't quite see how making print a callable object solves that > issue. It solves it by making the stream an attribute on the callable instead of an argument to the call. > But personally would be totally happy with "You can't use > 'file' as a variable name in your format". ('end' and 'sep' should not > be supported by printf(), these should just be made part of the format > string.) If 'file' is the only special keyword argument to printf, then let's call it '_file' or some such. There's probably less chance that a newbie will want to change the output stream than they'll want to use a file variable in their format string. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRWIMi3EjvBPtnXfVAQJ2aQP/Qcq+J5ZN0p6aH0QJSO5W7HpJ/MpBQCWu vRQTu13NHFscPNIC3SFvRIg/QPZdpMYSpEUZSl4BUuQMp+MEGSSKKFbq14DyDTfV O7VZ8QjN94EL410cZAHS2PZSZzGo0eG96zwSiGkEK+pjuKVwML+nTSq0xQqJF8Yu w7btiFT/ThY= =SxCt -----END PGP SIGNATURE----- From exarkun at divmod.com Mon Nov 20 21:48:46 2006 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Mon, 20 Nov 2006 15:48:46 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: Message-ID: <20061120204846.20948.1503749203.divmod.quotient.36290@ohm> On Mon, 20 Nov 2006 10:27:05 -0800, Guido van Rossum wrote: >I'd like to have a discussion of abilities and interfaces but right >now I don't have time -- maybe tomorrow. I see Andrew's proposal as >mostly isomorphic to Zope's notion of interfaces, but the new >terminology may make it possible to clarify certain distinctions >better, e.g. the distinction between a class that provides an ability >to its instances, and the abilities that an instance has. Zope Interface makes this distinction very clear. It has separate terminology for each thing, and separate APIs for dealing with each. Jean-Paul From jimjjewett at gmail.com Mon Nov 20 21:57:25 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 20 Nov 2006 15:57:25 -0500 Subject: [Python-3000] print() parameters in py3k In-Reply-To: <46B9A351-2664-4B58-86C6-BB2D9A8DDF7F@python.org> References: <041601c70cb4$5ff11470$cf09f01b@bagio> <20061120082547.8346.JCARLSON@uci.edu> <46B9A351-2664-4B58-86C6-BB2D9A8DDF7F@python.org> Message-ID: On 11/20/06, Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Nov 20, 2006, at 12:14 PM, Josiah Carlson wrote: > > > How much print statement rewriting do you want to do? Me, I'd like to > > get away with as little as possible. Adding parenthesis isn't that > > bad. > > Changing everything to use a printf-style would be 1) annoying, 2) > > unnecessary, and 3) I hate printf. > Are we going to include a flawless transformation script for Python > 3k? If not, then I'm pretty much resigned to rewriting them by hand > anyway. Assuming no further changes to PEP 3105, then it should be possible, at least for the print statement. Just parse the thing into statements; when you get to a print statement make the string replacement before writing it back out. The "print" literal translates into "print(" ... ")" ">>" file_expression moves to the end and translates into "file="file_expression the printable_expression stays the same a trailing comma translates into "end=''" (an empty string) (legacy code never uses the sep keyword) -jJ From tomerfiliba at gmail.com Mon Nov 20 22:17:52 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Mon, 20 Nov 2006 23:17:52 +0200 Subject: [Python-3000] __nonzero__ vs. __bool__ Message-ID: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com> how about changing the name of __nonzero__ to __bool__? today it looks like: * int(x) --> x.__int__() * long(x) --> x.__long__() * float(x) --> x.__float__() * str(x) --> x.__str__() but for bool it's x.__nonzero__() i'd guess __nonzero__ existed even before the bool type was introduced (2.2?), but in the cleanup-old-design-issues spirit, perhaps it can be made symmetrical to the other builtin types. -tomer -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061120/38523dd6/attachment.htm From greg.ewing at canterbury.ac.nz Tue Nov 21 01:23:45 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 21 Nov 2006 13:23:45 +1300 Subject: [Python-3000] print() parameters in py3k In-Reply-To: <041601c70cb4$5ff11470$cf09f01b@bagio> References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> <041601c70cb4$5ff11470$cf09f01b@bagio> Message-ID: <45624711.2060806@canterbury.ac.nz> Giovanni Bajo wrote: > println(x,y,z) -> append "\n" > print(x,y,z) -> no terminator > print(x,y,z,"\r\n") -> custom terminator The third one won't work, because you'll get a space before the terminator. -- Greg From guido at python.org Tue Nov 21 02:21:10 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 20 Nov 2006 17:21:10 -0800 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com> References: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com> Message-ID: Looks about right. Patch? On 11/20/06, tomer filiba wrote: > how about changing the name of __nonzero__ to __bool__? > > today it looks like: > * int(x) --> x.__int__() > * long(x) --> x.__long__() > * float(x) --> x.__float__() > * str(x) --> x.__str__() > > but for bool it's x.__nonzero__() > > i'd guess __nonzero__ existed even before the bool type was introduced > (2.2?), > but in the cleanup-old-design-issues spirit, perhaps it can be made > symmetrical > to the other builtin types. > > > -tomer > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: > http://mail.python.org/mailman/options/python-3000/guido%40python.org > > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Nov 21 02:35:41 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 20 Nov 2006 17:35:41 -0800 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <041601c70cb4$5ff11470$cf09f01b@bagio> <20061120082547.8346.JCARLSON@uci.edu> <46B9A351-2664-4B58-86C6-BB2D9A8DDF7F@python.org> Message-ID: On 11/20/06, Jim Jewett wrote: > On 11/20/06, Barry Warsaw wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > On Nov 20, 2006, at 12:14 PM, Josiah Carlson wrote: > > > > > How much print statement rewriting do you want to do? Me, I'd like to > > > get away with as little as possible. Adding parenthesis isn't that > > > bad. > > > Changing everything to use a printf-style would be 1) annoying, 2) > > > unnecessary, and 3) I hate printf. > > > Are we going to include a flawless transformation script for Python > > 3k? If not, then I'm pretty much resigned to rewriting them by hand > > anyway. > > Assuming no further changes to PEP 3105, then it should be possible, > at least for the print statement. > > Just parse the thing into statements; when you get to a print > statement make the string replacement before writing it back out. > > The "print" literal translates into "print(" ... ")" > > ">>" file_expression moves to the end and translates into > "file="file_expression > > the printable_expression stays the same > > a trailing comma translates into "end=''" (an empty string) No, end=" " (i.e. a single space). And the translation can't be perfect, due to the dropping of softspace. But IMO for code using print statements few people will mind occasionally seeing a line ending with a space before the newline. > (legacy code never uses the sep keyword) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Nov 21 02:36:53 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 20 Nov 2006 17:36:53 -0800 Subject: [Python-3000] print() parameters in py3k In-Reply-To: <46A6517D-6EA1-4BD2-94F7-02D3C7A4AB64@python.org> References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> <041601c70cb4$5ff11470$cf09f01b@bagio> <64AE2078-2ABF-4975-95B8-5698EED1EEFC@python.org> <46A6517D-6EA1-4BD2-94F7-02D3C7A4AB64@python.org> Message-ID: On 11/20/06, Barry Warsaw wrote: > On Nov 20, 2006, at 2:52 PM, Guido van Rossum wrote: > > I still don't quite see how making print a callable object solves that > > issue. > > It solves it by making the stream an attribute on the callable > instead of an argument to the call. I'm still confused. Are you proposing that in order to print to a different file we do save_file = print.file try: print.file = open("log.txt", "a") print(x, y, z) etc.? > > But personally would be totally happy with "You can't use > > 'file' as a variable name in your format". ('end' and 'sep' should not > > be supported by printf(), these should just be made part of the format > > string.) > > If 'file' is the only special keyword argument to printf, then let's > call it '_file' or some such. There's probably less chance that a > newbie will want to change the output stream than they'll want to use > a file variable in their format string. > > - -Barry > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.5 (Darwin) > > iQCVAwUBRWIMi3EjvBPtnXfVAQJ2aQP/Qcq+J5ZN0p6aH0QJSO5W7HpJ/MpBQCWu > vRQTu13NHFscPNIC3SFvRIg/QPZdpMYSpEUZSl4BUuQMp+MEGSSKKFbq14DyDTfV > O7VZ8QjN94EL410cZAHS2PZSZzGo0eG96zwSiGkEK+pjuKVwML+nTSq0xQqJF8Yu > w7btiFT/ThY= > =SxCt > -----END PGP SIGNATURE----- > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Nov 21 02:38:20 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 20 Nov 2006 17:38:20 -0800 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> <041601c70cb4$5ff11470$cf09f01b@bagio> <64AE2078-2ABF-4975-95B8-5698EED1EEFC@python.org> <46A6517D-6EA1-4BD2-94F7-02D3C7A4AB64@python.org> Message-ID: [Sorry, sent before its time] On 11/20/06, Guido van Rossum wrote: > On 11/20/06, Barry Warsaw wrote: > > On Nov 20, 2006, at 2:52 PM, Guido van Rossum wrote: > > > I still don't quite see how making print a callable object solves that > > > issue. > > > > It solves it by making the stream an attribute on the callable > > instead of an argument to the call. > I'm still confused. Are you proposing that in order to print to a different file we do save_file = print.file try: print.file = open("log.txt", "a") print(x, y, z) finally: print.file = save_file etc.? > > > > But personally would be totally happy with "You can't use > > > 'file' as a variable name in your format". ('end' and 'sep' should not > > > be supported by printf(), these should just be made part of the format > > > string.) > > > > If 'file' is the only special keyword argument to printf, then let's > > call it '_file' or some such. There's probably less chance that a > > newbie will want to change the output stream than they'll want to use > > a file variable in their format string. -0. I really don't want it to be _file for the basic print(); and making them different is asking for trouble. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From rhamph at gmail.com Tue Nov 21 10:50:25 2006 From: rhamph at gmail.com (Adam Olsen) Date: Tue, 21 Nov 2006 02:50:25 -0700 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: On 11/20/06, Jim Jewett wrote: > On 11/19/06, Guido van Rossum wrote: > > A PEP may also discourage attempts to add more cruft, > > (1) Is this an explicit rejection of a keyword for a format string, such as > > print (x, y, z, fmt="My X: %s, Your Y: %s, His Z: %s") Although odd, this appeals to me somehow. It avoids the risk of passing an untrusted string as the format (even if that's much less dangerous than in C). -- Adam Olsen, aka Rhamphoryncus From fredrik at pythonware.com Tue Nov 21 12:05:28 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 21 Nov 2006 12:05:28 +0100 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: Adam Olsen wrote: >> (1) Is this an explicit rejection of a keyword for a format string, such as >> >> print (x, y, z, fmt="My X: %s, Your Y: %s, His Z: %s") > > Although odd, this appeals to me somehow. It avoids the risk of > passing an untrusted string as the format (even if that's much less > dangerous than in C). it doesn't read well, though. but if we take the print, printf, and str.format trojka, the rather common mistake you mention, and my preference for designing API:s that have room for performance optimizations, and refactor things a bit, we can do as follows: 1) move formatting to a separate object. str.format(args) becomes one or more of format(str).format(args) # object style format(str)(args) # partial application style format(str) % (args) # old-style this allows the formatter object to parse the formatting string (and possibly also cache it), and also opens up for alternative target API:s (e.g. tostringlist, etc). and the format object should somewhat modular and fully subclassable, of course, so Barry can prototype all his ideas all on his own ;-) 2) for convenience, extend print to treat a format object as a format specifier for the following arguments: print (x, y, z, fmt="My X: %s, Your Y: %s, His Z: %s") becomes print(format("My X: %s, Your Y: %s, His Z: %s"), x, y ,z) 3) get rid of printf. 4) go to Iceland and optimize the heck out of format/print. From tomerfiliba at gmail.com Tue Nov 21 12:59:16 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Tue, 21 Nov 2006 13:59:16 +0200 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: References: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com> Message-ID: <1d85506f0611210359m7625c01ava3b2530465d4c8a4@mail.gmail.com> http://sourceforge.net/tracker/index.php?func=detail&aid=1600346&group_id=5470&atid=305470 """some notes: * the latex docs have been updated * the slot name was changed from nb_nonzero to nb_bool * all XXX_nonzero methods where changed to XXX_bool * all the comments relating to nb_nonzero have been updated * stdlib was updated * all the test suites were updated otoh, i couldn't get it to compile (MSVC++2003), because of a strange bug in ceval.c (none of my code). seems the GETLOCAL macro causes syntactic problems, but i didn't have time to check it thoroughly.""" -tomer On 11/21/06, Guido van Rossum wrote: > > Looks about right. Patch? > > On 11/20/06, tomer filiba wrote: > > how about changing the name of __nonzero__ to __bool__? > > > > today it looks like: > > * int(x) --> x.__int__() > > * long(x) --> x.__long__() > > * float(x) --> x.__float__() > > * str(x) --> x.__str__() > > > > but for bool it's x.__nonzero__() > > > > i'd guess __nonzero__ existed even before the bool type was introduced > > (2.2?), > > but in the cleanup-old-design-issues spirit, perhaps it can be made > > symmetrical > > to the other builtin types. > > > > > > -tomer > > _______________________________________________ > > Python-3000 mailing list > > Python-3000 at python.org > > http://mail.python.org/mailman/listinfo/python-3000 > > Unsubscribe: > > http://mail.python.org/mailman/options/python-3000/guido%40python.org > > > > > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061121/aa6d3280/attachment.htm From tomerfiliba at gmail.com Tue Nov 21 13:23:24 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Tue, 21 Nov 2006 14:23:24 +0200 Subject: [Python-3000] Abilities / Interfaces: why not adapt()? Message-ID: <1d85506f0611210423u7386f622ta8ace619509add30@mail.gmail.com> [GvR] > I'd like to have a discussion of abilities and interfaces but right > now I don't have time -- maybe tomorrow. I see Andrew's proposal as > mostly isomorphic to Zope's notion of interfaces, but the new > terminology may make it possible to clarify certain distinctions > better, e.g. the distinction between a class that provides an ability > to its instances, and the abilities that an instance has. [Antoine Pitrou] > class Complex: > __implements__ = (Number,) ... > So to sum it up, to know whether an object-or-class implements a given > interface, you could call a standard function > implements(obj, interface), which in turn would call > interface.__implemented_by__(obj). It's up to each interface to decide > what breadth of tests it wants to perform, although it would be > practical to fallback on a default implementation (in the Interface base > class common to all interfaces) which just checks for presence of the > interface in obj.__implements__. ------------------------------------------------------------------------------- it reminds me greatly of the old (rejected) mechanism suggested by the adaptation PEP... only weaker. http://www.python.org/dev/peps/pep-0246/ these approaches only *check* a type for conformance with some interface, but the adaptation introduced a new way to look at interfaces/protocol/conformance at all: instead of asking "does object X supports interface Y?", adapt() attempts to *give* you an object X that supports interface Y. adaptation is much more capable ("don't complain, do something about it") and reflective (both the protocol and the instance can perform the adaptation) so you don't have to explicitly inherit from many base classes or interfaces. i'm very +1 on reviving adapt() and dropping all those "i want strongly-duck-typed interfaces in python" talks. give power to the duck -- adapt can do it all. -tomer From barry at python.org Tue Nov 21 14:28:32 2006 From: barry at python.org (Barry Warsaw) Date: Tue, 21 Nov 2006 08:28:32 -0500 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> <041601c70cb4$5ff11470$cf09f01b@bagio> <64AE2078-2ABF-4975-95B8-5698EED1EEFC@python.org> <46A6517D-6EA1-4BD2-94F7-02D3C7A4AB64@python.org> Message-ID: <34A5E882-228F-4CB9-ACBB-278394EB4585@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Nov 20, 2006, at 8:38 PM, Guido van Rossum wrote: > I'm still confused. Are you proposing that in order to print to a > different file we do > > save_file = print.file > try: > print.file = open("log.txt", "a") > print(x, y, z) > finally: > print.file = save_file You could do that, but to make it more convenient I was thinking about adding class methods, such as the clone() I used earlier. Let's rename that method though so that it's hopefully clearer: lprint = print.open('log.txt', 'a') lprint(x, y, z) lprint.close() - -or- lprint.file.close() lprint is just another instance of whatever class 'print' is, except that its 'file' attribute is set to the opened log.txt file. Regardless of whether you like this or not , does that make sense? - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRWL/AHEjvBPtnXfVAQJ1TQQAltC6uJEpzkYXyb4Zr6hRrWsPmYfOyJnp EOLieDvn/pTOVgA+iAuccEtac+Yp5Kl8bR+5s4B8XSLoD8gbKt7aMwmwBdy7fIOc 2OvH+pb2rn8laMAzp/OTzg7OF+ptJFXKdfOx8+pZdy41htV+6U8rsHWUCo2F59Sj q3XzVDbMF44= =DycV -----END PGP SIGNATURE----- From barry at python.org Tue Nov 21 14:37:49 2006 From: barry at python.org (Barry Warsaw) Date: Tue, 21 Nov 2006 08:37:49 -0500 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: <55DAAEFE-373C-4799-9C8E-5A48FABD63C1@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Nov 21, 2006, at 6:05 AM, Fredrik Lundh wrote: > 2) for convenience, extend print to treat a format object as a format > specifier for the following arguments: > > print (x, y, z, fmt="My X: %s, Your Y: %s, His Z: %s") > > becomes > > print(format("My X: %s, Your Y: %s, His Z: %s"), x, y ,z) > > 3) get rid of printf. That's not bad. Is the format object allowed in any positional argument, or just the first? - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRWMBLnEjvBPtnXfVAQIPkQQAo9LHQIzIJnQ1AK7b2OqWKSEfReDQbaEd 7rVVjaYQEb9yuMmAdBEH1HzgA/llSEt4hwY/A7QM079rXTvikcNDQZOc6ATzOGIw mBkRDizn2Z7XKluODe+zr0NOk74Phiv03CYBvlEoYVLUQSNBKI4EFKBmclmx+YsC LR2FUmV5enU= =f6RC -----END PGP SIGNATURE----- From p.f.moore at gmail.com Tue Nov 21 15:10:47 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 21 Nov 2006 14:10:47 +0000 Subject: [Python-3000] Abilities / Interfaces: why not adapt()? In-Reply-To: <1d85506f0611210423u7386f622ta8ace619509add30@mail.gmail.com> References: <1d85506f0611210423u7386f622ta8ace619509add30@mail.gmail.com> Message-ID: <79990c6b0611210610k57d5cda4n1a16fb53dfb3ae59@mail.gmail.com> On 11/21/06, tomer filiba wrote: > adaptation is much more capable ("don't complain, do something > about it") and reflective (both the protocol and the instance can > perform the adaptation) so you don't have to explicitly inherit > from many base classes or interfaces. > > i'm very +1 on reviving adapt() and dropping all those "i want > strongly-duck-typed interfaces in python" talks. To some extent I agree, but I also think that generic functions are even more flexible. (As per the previous discussion which started covering adaptation and moved on to generic functions). I'd like to see some clear evaluation of the pros and cons of the 3 - interfaces, adaptation, and generic functions. My feeling is that they are to some extent "competing" proposals, and we need agreement (or BDFL ruling!) on which is the way forward, before we start discussing specifics. Paul. From gsakkis at rutgers.edu Tue Nov 21 16:10:43 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Tue, 21 Nov 2006 10:10:43 -0500 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: <91ad5bf80611210710m36a56959m552aea57a0c88354@mail.gmail.com> On 11/21/06, Fredrik Lundh wrote: > Adam Olsen wrote: > > >> (1) Is this an explicit rejection of a keyword for a format string, such as > >> > >> print (x, y, z, fmt="My X: %s, Your Y: %s, His Z: %s") > > > > Although odd, this appeals to me somehow. It avoids the risk of > > passing an untrusted string as the format (even if that's much less > > dangerous than in C). > > it doesn't read well, though. > > but if we take the print, printf, and str.format trojka, the rather > common mistake you mention, and my preference for designing API:s that > have room for performance optimizations, and refactor things a bit, we > can do as follows: > > 1) move formatting to a separate object. > > str.format(args) > > becomes one or more of > > format(str).format(args) # object style > format(str)(args) # partial application style > format(str) % (args) # old-style > > this allows the formatter object to parse the formatting string (and > possibly also cache it), and also opens up for alternative target API:s > (e.g. tostringlist, etc). > > and the format object should somewhat modular and fully subclassable, of > course, so Barry can prototype all his ideas all on his own ;-) > > 2) for convenience, extend print to treat a format object as a format > specifier for the following arguments: > > print (x, y, z, fmt="My X: %s, Your Y: %s, His Z: %s") > > becomes > > print(format("My X: %s, Your Y: %s, His Z: %s"), x, y ,z) > > 3) get rid of printf. > > 4) go to Iceland and optimize the heck out of format/print. I like the idea of a format object and (1),(3) and (4), but (2) doesn't feel "right": a. Will print have to do an isinstance(args[0], format) to decide what to do ? If so, don't the usual arguments for duck typing and against type checking apply here ? Even if print uses duck typing (e.g. tries to call a format() method), it still doesn't feel right (wouldn't like to pass some HardDrive object as first argument ). b. print(a,b,c) is no longer obvious whether it prints 3 unformatted objects or 2 formatted, if you don't know what a is. b. What if for some reason one actually wants to print the format object instead of interpreting as a format specifier ? I think format should be either a positional argument at a fixed position or a keyword argument. Between the two there is a tradeoff; the former is more readable but requires a separate printf() function, the latter is less readable but we can get away with just print(). I'm not strongly inclined for either, but they both look better than an optional positional format argument. George From ncoghlan at gmail.com Tue Nov 21 16:25:19 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 22 Nov 2006 01:25:19 +1000 Subject: [Python-3000] Builtin iterator type In-Reply-To: <4561857A.8080101@cenix-bioscience.com> References: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FEEBB.9010100@cs.byu.edu> <20061119115009.V14417@tribble.ilrt.bris.ac.uk> <06Nov19.170737pst."58648"@synergy1.parc.xerox.com> <4561857A.8080101@cenix-bioscience.com> Message-ID: <45631A5F.5080501@gmail.com> Aaron Bingham wrote: > Bill Janssen wrote: > >>> Java interfaces are very useful, however. Java programming seems to be >>> less and less about inheritance and more and more about implementing >>> interfaces; at least it does amongst Java programmers with taste :-) >>> >> It seems to me that that's where Python has a real advantage. With >> real support for multiple inheritance, Python "interfaces" could be >> real classes (either like real Java classes or Java abstract classes), >> perhaps providing default implementations. You get the goodness of >> mix-ins, along with interface communication. >> > I agree. In Java, interfaces are necessary because multiple inheritance > is not supported. I see no good reason to add an additional language > mechanism for interfaces when multiple inheritance would do the job, AFAICT. Just because multiple inheritance is *possible* in Python, don't make the mistake of thinking that it is straight forward. Aside from mixin classes that themselves inherit directly from object, multiple inheritance can get very messy outside of tightly controlled type hierarchies (cooperative calls in particular can become a nightmare). This flies in the face of Python's use as a glue language to tie multiple components together (particularly given the problem that some of the elements being integrated may come from environments that *don't* support multiple inheritance, like the JVM). The idea of a formal interface mechanism is to have something which conveys the desired information *without* all of the additional issues that come with using multiple inheritance. It's similar to the rationale behind adding class decorators, even though everything that a class decorator can do could be done with a metaclass instead: getting a metaclass right is *hard*, but getting a class decorator right is relatively easy and permits many of the things that programmers would currently have to use a metaclass for. Similarly, it's obviously possible to do interfaces in current Python (Zope, PEAK and Twisted all use them, after all), but the question is whether or not it can be made *easier* (both to write, and, more importantly, to read). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From rrr at ronadam.com Tue Nov 21 16:31:24 2006 From: rrr at ronadam.com (Ron Adam) Date: Tue, 21 Nov 2006 09:31:24 -0600 Subject: [Python-3000] print() parameters in py3k In-Reply-To: <34A5E882-228F-4CB9-ACBB-278394EB4585@python.org> References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> <041601c70cb4$5ff11470$cf09f01b@bagio> <64AE2078-2ABF-4975-95B8-5698EED1EEFC@python.org> <46A6517D-6EA1-4BD2-94F7-02D3C7A4AB64@python.org> <34A5E882-228F-4CB9-ACBB-278394EB4585@python.org> Message-ID: Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Nov 20, 2006, at 8:38 PM, Guido van Rossum wrote: > >> I'm still confused. Are you proposing that in order to print to a >> different file we do >> >> save_file = print.file >> try: >> print.file = open("log.txt", "a") >> print(x, y, z) >> finally: >> print.file = save_file > > You could do that, but to make it more convenient I was thinking > about adding class methods, such as the clone() I used earlier. > Let's rename that method though so that it's hopefully clearer: > > lprint = print.open('log.txt', 'a') > lprint(x, y, z) > lprint.close() > > - -or- > > lprint.file.close() > > lprint is just another instance of whatever class 'print' is, except > that its 'file' attribute is set to the opened log.txt file. > > Regardless of whether you like this or not , does that make sense? I'll add 2 cents here... (or possibly less) I agree with others here the print function probably should be as simple and easy to use as possible, basically nearly identical to the current print use, but as a function. It's easy and generalized enough to work in most situations and it fits the use case of an easy convenient built-in way to output basic text. I think maybe there could be more specialized print methods on possibly a console (and printer?) object in the library. That would then match the file objects use of write and writeline. So you then might have a console.print() and console.printline(), and printer.print() and printer.printline(), along with the built in file.write() and file.writeline(). But I think having a generic print object somehow doesn't fit this pattern well. And weather or not these types of objects ever get added to the library, I can't see how it would effect the print function use in any way. Does this make more (or less) sense? Cheers, Ron From ncoghlan at gmail.com Tue Nov 21 16:28:02 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 22 Nov 2006 01:28:02 +1000 Subject: [Python-3000] Abilities / Interfaces: why not adapt()? In-Reply-To: <79990c6b0611210610k57d5cda4n1a16fb53dfb3ae59@mail.gmail.com> References: <1d85506f0611210423u7386f622ta8ace619509add30@mail.gmail.com> <79990c6b0611210610k57d5cda4n1a16fb53dfb3ae59@mail.gmail.com> Message-ID: <45631B02.8050005@gmail.com> Paul Moore wrote: > On 11/21/06, tomer filiba wrote: >> adaptation is much more capable ("don't complain, do something >> about it") and reflective (both the protocol and the instance can >> perform the adaptation) so you don't have to explicitly inherit >> from many base classes or interfaces. >> >> i'm very +1 on reviving adapt() and dropping all those "i want >> strongly-duck-typed interfaces in python" talks. > > To some extent I agree, but I also think that generic functions are > even more flexible. (As per the previous discussion which started > covering adaptation and moved on to generic functions). > > I'd like to see some clear evaluation of the pros and cons of the 3 - > interfaces, adaptation, and generic functions. My feeling is that they > are to some extent "competing" proposals, and we need agreement (or > BDFL ruling!) on which is the way forward, before we start discussing > specifics. The BDFL has favoured function overloading (aka generic functions) pretty heavily in previous rounds of this discussion, but more formal interfaces may still form a significant part of that as an efficient means of dispatching to the different overloaded operations. Multi-argument dispatch plus interfaces is likely to be cleaner than trying to do everything based on concrete types. But this really needs more input from the enterprise framework folks (PEAK, Zope, etc), as that's the level this feature is really aimed at. Quick & dirty scripts are unlikely to need it any more than they do now :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From guido at python.org Tue Nov 21 17:09:07 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Nov 2006 08:09:07 -0800 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: <1d85506f0611210359m7625c01ava3b2530465d4c8a4@mail.gmail.com> References: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com> <1d85506f0611210359m7625c01ava3b2530465d4c8a4@mail.gmail.com> Message-ID: Can anyone help out evaluating this patch? If it has to wait for me it's gonna be a looooooong wait... On 11/21/06, tomer filiba wrote: > http://sourceforge.net/tracker/index.php?func=detail&aid=1600346&group_id=5470&atid=305470 > > """some notes: > * the latex docs have been updated > * the slot name was changed from nb_nonzero to nb_bool > * all XXX_nonzero methods where changed to XXX_bool > * all the comments relating to nb_nonzero have been updated > * stdlib was updated > * all the test suites were updated > > otoh, i couldn't get it to compile (MSVC++2003), because of a strange > bug in ceval.c (none of my code). seems the GETLOCAL macro causes > syntactic problems, but i didn't have time to check it thoroughly.""" > > > -tomer > > > On 11/21/06, Guido van Rossum < guido at python.org> wrote: > > Looks about right. Patch? > > > > On 11/20/06, tomer filiba < tomerfiliba at gmail.com> wrote: > > > how about changing the name of __nonzero__ to __bool__? > > > > > > today it looks like: > > > * int(x) --> x.__int__() > > > * long(x) --> x.__long__() > > > * float(x) --> x.__float__() > > > * str(x) --> x.__str__() > > > > > > but for bool it's x.__nonzero__() > > > > > > i'd guess __nonzero__ existed even before the bool type was introduced > > > ( 2.2?), > > > but in the cleanup-old-design-issues spirit, perhaps it can be made > > > symmetrical > > > to the other builtin types. > > > > > > > > > -tomer > > > _______________________________________________ > > > Python-3000 mailing list > > > Python-3000 at python.org > > > http://mail.python.org/mailman/listinfo/python-3000 > > > Unsubscribe: > > > > http://mail.python.org/mailman/options/python-3000/guido%40python.org > > > > > > > > > > > > > > > -- > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tomerfiliba at gmail.com Tue Nov 21 17:12:19 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Tue, 21 Nov 2006 18:12:19 +0200 Subject: [Python-3000] Abilities / Interfaces: why not adapt()? In-Reply-To: <79990c6b0611210610k57d5cda4n1a16fb53dfb3ae59@mail.gmail.com> References: <1d85506f0611210423u7386f622ta8ace619509add30@mail.gmail.com> <79990c6b0611210610k57d5cda4n1a16fb53dfb3ae59@mail.gmail.com> Message-ID: <1d85506f0611210812i2fcb14b1h9440ae530c3d30c5@mail.gmail.com> adapt() is a "casting" mechanism, whereas generic functions are dispatch mechanism (multimethods)... i'd guess there are some overlapping characteristics, but they are different by concept. [1] we can say interfaces are a subset of adaption (adaption > interfaces): * interfaces are mostly static, where as adaption is dynamic * interfaces only indicate compliance, adaptation resolves mismatches * you can implement interfaces using adaptation (__adapt__ just checks for isinstance or some mechanism) but comparing adaption to generic functions is apples to oranges. suppose i have a function that logs text to a file: def log(text, fileobj = sys.stdout): fileobj.write(text + "\n") and for reasons like computation with side-effects, i want *early* detection of errors (before side effects occur): i want to make sure fileobj has is a file-like object. [2] interfaces and adaptation solve this issue, but generic functions are not applicable really. i mean, if i had a version of log() for files, and another for lists as the backing store, i'd say generic functions are the solution. but all i asked for is an object that has a write() method, and for that i don't want to require any multiple dispatch mechanisms. besides, the use case of multiple dispatch is by far less common than "i need to make sure the object i'm dealing with is what i expect" = = = = = = = = = as a bonus, type annotations go very well with adaption: @adapting def log(text : str, fileobj : file = sys.stdout): fileobj.write(text) where the `adapting` deco takes the function's __signature__, adapts all the arguments accordingly, and invokes the original function with the adapted arguments. this way, all mismatching types are detected prior to executing any code with side effects. adaptions also doesn't require any best-fit algorithms, and is easily extensible, as every argument can theoretically provide its own "how-to-use-me" adapter. and last but not least -- adaption is well suited for object proxies, which would make RPyC's job easier :) -tomer [1] i know this pep has been killed, but i wanted to point out the differences. if it's meant to stay dead, i won't bring it up anymore. [2] i may also define a WritableFileProtocol for adapt, that makes sure the file's open-mode allows writing. i mean, the object's interface may provide a write() method (hasattr(fileobj, "write")), but i want to make sure the file is open properly, etc. On 11/21/06, Paul Moore wrote: > On 11/21/06, tomer filiba wrote: > > adaptation is much more capable ("don't complain, do something > > about it") and reflective (both the protocol and the instance can > > perform the adaptation) so you don't have to explicitly inherit > > from many base classes or interfaces. > > > > i'm very +1 on reviving adapt() and dropping all those "i want > > strongly-duck-typed interfaces in python" talks. > > To some extent I agree, but I also think that generic functions are > even more flexible. (As per the previous discussion which started > covering adaptation and moved on to generic functions). > > I'd like to see some clear evaluation of the pros and cons of the 3 - > interfaces, adaptation, and generic functions. My feeling is that they > are to some extent "competing" proposals, and we need agreement (or > BDFL ruling!) on which is the way forward, before we start discussing > specifics. > > Paul. > From guido at python.org Tue Nov 21 17:14:21 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Nov 2006 08:14:21 -0800 Subject: [Python-3000] print() parameters in py3k In-Reply-To: <34A5E882-228F-4CB9-ACBB-278394EB4585@python.org> References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> <041601c70cb4$5ff11470$cf09f01b@bagio> <64AE2078-2ABF-4975-95B8-5698EED1EEFC@python.org> <46A6517D-6EA1-4BD2-94F7-02D3C7A4AB64@python.org> <34A5E882-228F-4CB9-ACBB-278394EB4585@python.org> Message-ID: OK. I think this proposal is dead, if only because I still don't understand it (or at least I don't understand why anyone would propose this particular API). Everything about it seems backwards. Even if you could explain it, that doesn't bode well for how future generations of Python users will struggle with it. So let's stop wasting our breath and bury it. Summary: -1. --Guido On 11/21/06, Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Nov 20, 2006, at 8:38 PM, Guido van Rossum wrote: > > > I'm still confused. Are you proposing that in order to print to a > > different file we do > > > > save_file = print.file > > try: > > print.file = open("log.txt", "a") > > print(x, y, z) > > finally: > > print.file = save_file > > You could do that, but to make it more convenient I was thinking > about adding class methods, such as the clone() I used earlier. > Let's rename that method though so that it's hopefully clearer: > > lprint = print.open('log.txt', 'a') > lprint(x, y, z) > lprint.close() > > - -or- > > lprint.file.close() > > lprint is just another instance of whatever class 'print' is, except > that its 'file' attribute is set to the opened log.txt file. > > Regardless of whether you like this or not , does that make sense? > > - -Barry > > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.5 (Darwin) > > iQCVAwUBRWL/AHEjvBPtnXfVAQJ1TQQAltC6uJEpzkYXyb4Zr6hRrWsPmYfOyJnp > EOLieDvn/pTOVgA+iAuccEtac+Yp5Kl8bR+5s4B8XSLoD8gbKt7aMwmwBdy7fIOc > 2OvH+pb2rn8laMAzp/OTzg7OF+ptJFXKdfOx8+pZdy41htV+6U8rsHWUCo2F59Sj > q3XzVDbMF44= > =DycV > -----END PGP SIGNATURE----- > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Nov 21 17:18:34 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Nov 2006 08:18:34 -0800 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: On 11/21/06, Fredrik Lundh wrote: > Adam Olsen wrote: > > >> (1) Is this an explicit rejection of a keyword for a format string, such as > >> > >> print (x, y, z, fmt="My X: %s, Your Y: %s, His Z: %s") > > > > Although odd, this appeals to me somehow. It avoids the risk of > > passing an untrusted string as the format (even if that's much less > > dangerous than in C). > > it doesn't read well, though. > > but if we take the print, printf, and str.format trojka, the rather > common mistake you mention, and my preference for designing API:s that > have room for performance optimizations, and refactor things a bit, we > can do as follows: > > 1) move formatting to a separate object. > > str.format(args) > > becomes one or more of > > format(str).format(args) # object style > format(str)(args) # partial application style > format(str) % (args) # old-style > > this allows the formatter object to parse the formatting string (and > possibly also cache it), and also opens up for alternative target API:s > (e.g. tostringlist, etc). > > and the format object should somewhat modular and fully subclassable, of > course, so Barry can prototype all his ideas all on his own ;-) > > 2) for convenience, extend print to treat a format object as a format > specifier for the following arguments: > > print (x, y, z, fmt="My X: %s, Your Y: %s, His Z: %s") > > becomes > > print(format("My X: %s, Your Y: %s, His Z: %s"), x, y ,z) > > 3) get rid of printf. > > 4) go to Iceland and optimize the heck out of format/print. Hm. While not as obviously from a different universe as Barry's proposal, this still pretty weird, probably at least from a different planet (much farther than Iceland anyway :-). Treating the first argument differently based on its being of a specific type doesn't sound right to me; what if you are handed an object x and you decide to print it using print(x), but surreptitiously (or by mistake) they hand you a format object? I do support adding printf(), or printf()-like capabilities. But I suggest to leave this for a separate PEP. After all, one of the arguments PEP 3105 mentions for making print() a function is that it opens the door for printf()... -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Nov 21 17:24:11 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Nov 2006 08:24:11 -0800 Subject: [Python-3000] Abilities / Interfaces: why not adapt()? In-Reply-To: <1d85506f0611210812i2fcb14b1h9440ae530c3d30c5@mail.gmail.com> References: <1d85506f0611210423u7386f622ta8ace619509add30@mail.gmail.com> <79990c6b0611210610k57d5cda4n1a16fb53dfb3ae59@mail.gmail.com> <1d85506f0611210812i2fcb14b1h9440ae530c3d30c5@mail.gmail.com> Message-ID: On 11/21/06, tomer filiba wrote: > adapt() is a "casting" mechanism, whereas generic functions are > dispatch mechanism (multimethods)... i'd guess there are some > overlapping characteristics, but they are different by concept. [1] You're wasting everybody's time. We went over this last time. Please read the archives. Quick summary: using generic functions, you can trivially implement adapt() or anything like it -- and lots of other things. Either way, interfaces or abilities add hugely to the usability -- without them, you're bound to dispatch only on concrete classes, which have severe limitations for this purpose. There's a reason Zope has interfaces *and* adapt -- they go together. So do generic functions and interfaces. References (these don't mention interfaces yet, but they explain how having generic functions makes it unnecessary to have adapt as a central mechanism): http://www.artima.com/weblogs/viewpost.jsp?thread=155123 http://www.artima.com/weblogs/viewpost.jsp?thread=155514 -- --Guido van Rossum (home page: http://www.python.org/~guido/) From janssen at parc.com Tue Nov 21 17:47:09 2006 From: janssen at parc.com (Bill Janssen) Date: Tue, 21 Nov 2006 08:47:09 PST Subject: [Python-3000] Abilities / Interfaces: why not adapt()? In-Reply-To: <79990c6b0611210610k57d5cda4n1a16fb53dfb3ae59@mail.gmail.com> References: <1d85506f0611210423u7386f622ta8ace619509add30@mail.gmail.com> <79990c6b0611210610k57d5cda4n1a16fb53dfb3ae59@mail.gmail.com> Message-ID: <06Nov21.084719pst."58648"@synergy1.parc.xerox.com> > To some extent I agree, but I also think that generic functions are > even more flexible. (As per the previous discussion which started > covering adaptation and moved on to generic functions). While I like generic functions, they only solve part of the problem. There are cases other than calling a function when a program needs to be able to look at a type and understand what to do with it. I'd favor promoting an inheritance-based interface system for type understanding, then move on to generic functions based on that infrastructure. Bill From krstic at solarsail.hcs.harvard.edu Tue Nov 21 17:49:38 2006 From: krstic at solarsail.hcs.harvard.edu (=?UTF-8?B?SXZhbiBLcnN0acSH?=) Date: Tue, 21 Nov 2006 11:49:38 -0500 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: References: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com> <1d85506f0611210359m7625c01ava3b2530465d4c8a4@mail.gmail.com> Message-ID: <45632E22.2030705@solarsail.hcs.harvard.edu> Guido van Rossum wrote: > Can anyone help out evaluating this patch? If it has to wait for me > it's gonna be a looooooong wait... Looks fine to me functionally, although it seems to gratuitously retab some code that used to be aligned with tabstop 8 to a tabstop of 4. -- Ivan Krsti? | GPG: 0x147C722D From janssen at parc.com Tue Nov 21 17:49:49 2006 From: janssen at parc.com (Bill Janssen) Date: Tue, 21 Nov 2006 08:49:49 PST Subject: [Python-3000] Builtin iterator type In-Reply-To: <45631A5F.5080501@gmail.com> References: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FEEBB.9010100@cs.byu.edu> <20061119115009.V14417@tribble.ilrt.bris.ac.uk> <06Nov19.170737pst."58648"@synergy1.parc.xerox.com> <4561857A.8080101@cenix-bioscience.com> <45631A5F.5080501@gmail.com> Message-ID: <06Nov21.084954pst."58648"@synergy1.parc.xerox.com> > The idea of a formal interface mechanism is to have something which conveys > the desired information *without* all of the additional issues that come with > using multiple inheritance. I hear what you're saying here, and I'm sympathetic to this. My question is whether it's practically possible to add another "type" system on top of the existing one. Bill From jimjjewett at gmail.com Tue Nov 21 18:07:35 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 21 Nov 2006 12:07:35 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: <45631A5F.5080501@gmail.com> References: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FEEBB.9010100@cs.byu.edu> <20061119115009.V14417@tribble.ilrt.bris.ac.uk> <4561857A.8080101@cenix-bioscience.com> <45631A5F.5080501@gmail.com> Message-ID: On 11/21/06, Nick Coghlan wrote: > Aaron Bingham wrote: > > I agree. In Java, interfaces are necessary because multiple inheritance > > is not supported. I see no good reason to add an additional language > > mechanism for interfaces when multiple inheritance would do the job, AFAICT. > Just because multiple inheritance is *possible* in Python, don't make the > mistake of thinking that it is straight forward. Aside from mixin classes that > themselves inherit directly from object, multiple inheritance can get very > messy outside of tightly controlled type hierarchies Interfaces have almost exactly the same limits. A mixin class is basically the equivalent of (a) A Java interface, plus (b) A matching default (possibly abstract) base class, plus (c) Automatic delegation (to the mixin), even for classes that need to inherit from something else. > (cooperative calls in particular can become a nightmare). If the method name is reused with different meanings, you have a nightmare no matter how you work it. If the mixin knows that it is the the most base provider of a method, you have no problem. Normally, this can be arranged by putting mixins at the end of the class statement. If you have multiple mixins each of which may or may not implement methods with the same name (such as "save") *and* that name won't be found on object ... then it gets a bit ugly. IMHO, it still isn't any uglier than the Java equivalent of sequentially calling the method for each interface separately. -jJ From exarkun at divmod.com Tue Nov 21 18:22:31 2006 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Tue, 21 Nov 2006 12:22:31 -0500 Subject: [Python-3000] Builtin iterator type In-Reply-To: Message-ID: <20061121172231.20948.1479947618.divmod.quotient.37702@ohm> On Tue, 21 Nov 2006 12:07:35 -0500, Jim Jewett wrote: >On 11/21/06, Nick Coghlan wrote: >> Aaron Bingham wrote: > >> > I agree. In Java, interfaces are necessary because multiple inheritance >> > is not supported. I see no good reason to add an additional language >> > mechanism for interfaces when multiple inheritance would do the job, AFAICT. > >> Just because multiple inheritance is *possible* in Python, don't make the >> mistake of thinking that it is straight forward. Aside from mixin classes that >> themselves inherit directly from object, multiple inheritance can get very >> messy outside of tightly controlled type hierarchies > >Interfaces have almost exactly the same limits. Having developed _extremely_ complex systems (often, the same system repeatedly, sometimes using inheritance, sometimes using interface) with each, I have to strongly disagree here. > >A mixin class is basically the equivalent of > > (a) A Java interface, plus > (b) A matching default (possibly abstract) base class, plus > (c) Automatic delegation (to the mixin), even for classes that >need to inherit from something else. > >> (cooperative calls in particular can become a nightmare). > >If the method name is reused with different meanings, you have a >nightmare no matter how you work it. Not so. Composition works much better than inheritance when this case comes up, and composition and interfaces tend to go hand in hand. > >If the mixin knows that it is the the most base provider of a method, >you have no problem. Normally, this can be arranged by putting mixins >at the end of the class statement. It is often impossible to know this, except when every aspect of the inheritance hierarchy is tightly controlled. This is rarely, if ever, the case for a class exposed by a library as part of its public API. Jean-Paul From guido at python.org Tue Nov 21 18:35:02 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Nov 2006 09:35:02 -0800 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: <45632E22.2030705@solarsail.hcs.harvard.edu> References: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com> <1d85506f0611210359m7625c01ava3b2530465d4c8a4@mail.gmail.com> <45632E22.2030705@solarsail.hcs.harvard.edu> Message-ID: On 11/21/06, Ivan Krsti? wrote: > Guido van Rossum wrote: > > Can anyone help out evaluating this patch? If it has to wait for me > > it's gonna be a looooooong wait... > > Looks fine to me functionally, although it seems to gratuitously retab > some code that used to be aligned with tabstop 8 to a tabstop of 4. Thanks! I hope you meant "indentation level" instead of "tabstop". Tomer, can you fix the indentation to be whatever's prevailing in the file you're editing? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jimjjewett at gmail.com Tue Nov 21 18:42:25 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 21 Nov 2006 12:42:25 -0500 Subject: [Python-3000] Abilities / Interfaces: why not adapt()? In-Reply-To: <79990c6b0611210610k57d5cda4n1a16fb53dfb3ae59@mail.gmail.com> References: <1d85506f0611210423u7386f622ta8ace619509add30@mail.gmail.com> <79990c6b0611210610k57d5cda4n1a16fb53dfb3ae59@mail.gmail.com> Message-ID: On 11/21/06, Paul Moore wrote: > I'd like to see some clear evaluation of the pros and cons of the 3 - > interfaces, adaptation, and generic functions. My feeling is that they > are to some extent "competing" proposals, and we need agreement (or > BDFL ruling!) on which is the way forward, before we start discussing > specifics. Interfaces ========== An interface is the equivalent of >>> obj.magic_name = magic_value Many function decorators do no more than this. Function (or method, or class) annotations can be very useful. They do rely on everyone following the same convention, so they end up with most of the same drawbacks as an isinstance check. (If you implement them by inheriting from a specified otherwise empty class, then they are equivalent to isinstance.) Adaptations =========== An adaptation is the equivalent of >>> obj = MagicAdapter(obj) Adaptation is strictly more powerful than interfaces. It may be simpler to understand/write/use if it already has interfaces to build on. Think of it as dynamic typecasting. Instead of saying "Is this an X?", you say "I want this to be an X". If the object is already an X, then you lose only the time spent to verify that (similar to an assert). If it isn't an X, then the adapter can turn it into an X, or return a new (equivalent) object that is an X. __index__ is an example -- if you have an int or a long, you get it back. Otherwise, you get the equivalent int or long. Generic Functions ================= Generic functions are the most powerful, but they are also the hardest to get your head around. They make metaclasses seem straightforward. They are less difficult if you have Adaptation and Interfaces around to build on. I cannot come up with a one-line equivalent. defop foo(arg1:Int, arg2:Seq): ... This changes what foo(1, []) will do, but not what foo(1.0, 7) would do. -jJ From pje at telecommunity.com Tue Nov 21 18:43:27 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 21 Nov 2006 12:43:27 -0500 Subject: [Python-3000] Abilities / Interfaces: why not adapt()? In-Reply-To: Message-ID: <5.1.1.6.0.20061121120913.03fb6738@sparrow.telecommunity.com> At 08:24 PM 11/21/2006 -0800, "Guido van Rossum" wrote: >Quick summary: using generic functions, you can trivially implement >adapt() or anything like it -- and lots of other things. Either way, >interfaces or abilities add hugely to the usability -- without them, >you're bound to dispatch only on concrete classes, which have severe >limitations for this purpose. There's a reason Zope has interfaces >*and* adapt -- they go together. So do generic functions and >interfaces. Let's ponder that a moment. Do you really need interfaces? Suppose that I want to write some code that behaves differently for string-like things and list-like things. In the current case, I could write code that does 'if isinstance()' for the two behaviors, and anybody using my code is screwed. In the generic function case, I write a generic function that subsumes the 'if', and anybody can define their type as string-like or list-like by reusing my existing 'str' and 'list' methods in the generic function. Using a hypothetical generic function API, let's suppose my generic function is "flatten", and I want to add UserList to it, I might do something like: flatten[UserList] = flatten[list] And that's my "interface" declaration, at least for that one operation. Okay, so maybe you're thinking, "well, we'd want Userlist to be usable for every operation I write that wants to accept 'list-like' objects." Okay, so I write something like this to cover all the generic operations I'm providing in my package: def like(newtype, oldtype): flatten[newtype] = flatten[oldtype] foo[newtype] = foo[oldtype] ... And then call 'like(UserList, list)'. So now you ask, "well, I want it to be listlike *globally*", and I say, "Really?" What does that mean, exactly? Well, consider len(). Isn't it just another generic function? I mean, if you write code that simply uses Python's builtin and stdlib generics, then it suffices to make your new type work with those generics in the first place. If my default "flatten()" method simply uses those generic operations, then any type that isn't declared to do something different, will "just work" if it supports the operations, and won't otherwise. We don't really want to encourage introspectable interfaces, because that leads to unnecessary LBYL-ing. Generic functions provide a kind of "type-safe" duck typing, because they do not rely on similarity of method names, but rather on explicit registration with a known generic function. Thus, explicit introspection just duplicates the work of the generic function *while making it impossible for anybody to override the behavior*. (Because you've gone back to using an if-then, which can't be extended by outside code!) Therefore, if we do have an interface mechanism for use with generic functions, it should merely be a convenient way of *copying methods*; a more formal way of doing the 'like()' operation I described above. We should avoid providing direct introspection of interfaces, because it is always possible to introspect for the specific operations you need (e.g. iterability, length-ability) by querying the relevant generic functions. For example, one could have a 'hasattr' analog 'hasop(f,t)', such that 'hasop(iter,list)' returns True. If hasop() is a generic function, then user-defined generic function types (e.g. RuleDispatch-based ones) are equally usable. (Some of what I've described of course would need more fleshing out for multiple-dispatch use cases, but single-dispatch is by far and away the most common use case, as can be seen from Python's existing generics.) From walter at livinglogic.de Tue Nov 21 18:43:20 2006 From: walter at livinglogic.de (=?ISO-8859-2?Q?Walter_D=F6rwald?=) Date: Tue, 21 Nov 2006 18:43:20 +0100 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: References: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com> <1d85506f0611210359m7625c01ava3b2530465d4c8a4@mail.gmail.com> <45632E22.2030705@solarsail.hcs.harvard.edu> Message-ID: <45633AB8.9010306@livinglogic.de> Guido van Rossum wrote: > On 11/21/06, Ivan Krsti? wrote: >> Guido van Rossum wrote: >>> Can anyone help out evaluating this patch? If it has to wait for me >>> it's gonna be a looooooong wait... >> Looks fine to me functionally, although it seems to gratuitously retab >> some code that used to be aligned with tabstop 8 to a tabstop of 4. > > Thanks! I hope you meant "indentation level" instead of "tabstop". > Tomer, can you fix the indentation to be whatever's prevailing in the > file you're editing? Another question is whether __bool__() should be allowed to return an int (or any other object), or if it *must* be a bool. (The patch currently allows ints). What if __bool__() returns subclasses of int, that overwrite __bool__() themself? Servus, Walter From guido at python.org Tue Nov 21 18:59:41 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Nov 2006 09:59:41 -0800 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: <45633AB8.9010306@livinglogic.de> References: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com> <1d85506f0611210359m7625c01ava3b2530465d4c8a4@mail.gmail.com> <45632E22.2030705@solarsail.hcs.harvard.edu> <45633AB8.9010306@livinglogic.de> Message-ID: I think it would set a good example if __bool__ was *required* to return exactly True or False and no other object, not even int instances equal to 0 or 1. Surely that can't be much of a burden. Feel free to upload an improved patch. On 11/21/06, Walter D?rwald wrote: > Guido van Rossum wrote: > > On 11/21/06, Ivan Krsti? wrote: > >> Guido van Rossum wrote: > >>> Can anyone help out evaluating this patch? If it has to wait for me > >>> it's gonna be a looooooong wait... > >> Looks fine to me functionally, although it seems to gratuitously retab > >> some code that used to be aligned with tabstop 8 to a tabstop of 4. > > > > Thanks! I hope you meant "indentation level" instead of "tabstop". > > Tomer, can you fix the indentation to be whatever's prevailing in the > > file you're editing? > > Another question is whether __bool__() should be allowed to return an > int (or any other object), or if it *must* be a bool. (The patch > currently allows ints). What if __bool__() returns subclasses of int, > that overwrite __bool__() themself? > > Servus, > Walter > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tomerfiliba at gmail.com Tue Nov 21 19:29:33 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Tue, 21 Nov 2006 20:29:33 +0200 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: References: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com> <1d85506f0611210359m7625c01ava3b2530465d4c8a4@mail.gmail.com> <45632E22.2030705@solarsail.hcs.harvard.edu> <45633AB8.9010306@livinglogic.de> Message-ID: <1d85506f0611211029i3969a91cw1ad621d15d4374e7@mail.gmail.com> patch updated -tomer On 11/21/06, Guido van Rossum wrote: > > I think it would set a good example if __bool__ was *required* to > return exactly True or False and no other object, not even int > instances equal to 0 or 1. Surely that can't be much of a burden. Feel > free to upload an improved patch. > > On 11/21/06, Walter D?rwald wrote: > > Guido van Rossum wrote: > > > On 11/21/06, Ivan Krsti? wrote: > > >> Guido van Rossum wrote: > > >>> Can anyone help out evaluating this patch? If it has to wait for me > > >>> it's gonna be a looooooong wait... > > >> Looks fine to me functionally, although it seems to gratuitously > retab > > >> some code that used to be aligned with tabstop 8 to a tabstop of 4. > > > > > > Thanks! I hope you meant "indentation level" instead of "tabstop". > > > Tomer, can you fix the indentation to be whatever's prevailing in the > > > file you're editing? > > > > Another question is whether __bool__() should be allowed to return an > > int (or any other object), or if it *must* be a bool. (The patch > > currently allows ints). What if __bool__() returns subclasses of int, > > that overwrite __bool__() themself? > > > > Servus, > > Walter > > > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061121/d5a1704a/attachment.html From g.brandl at gmx.net Tue Nov 21 20:08:11 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 21 Nov 2006 20:08:11 +0100 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: <1d85506f0611211029i3969a91cw1ad621d15d4374e7@mail.gmail.com> References: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com> <1d85506f0611210359m7625c01ava3b2530465d4c8a4@mail.gmail.com> <45632E22.2030705@solarsail.hcs.harvard.edu> <45633AB8.9010306@livinglogic.de> <1d85506f0611211029i3969a91cw1ad621d15d4374e7@mail.gmail.com> Message-ID: I applied the new patch. Aside from a syntax error on line 4241 in typeobject.c, it compiles. However, test_decimal fails (before the make test run bails out at test_hotshot, which is unrelated...) Georg tomer filiba schrieb: > patch updated > > > -tomer > > On 11/21/06, *Guido van Rossum* > wrote: > > I think it would set a good example if __bool__ was *required* to > return exactly True or False and no other object, not even int > instances equal to 0 or 1. Surely that can't be much of a burden. Feel > free to upload an improved patch. > > On 11/21/06, Walter D?rwald > wrote: > > Guido van Rossum wrote: > > > On 11/21/06, Ivan Krsti? < krstic at solarsail.hcs.harvard.edu > > wrote: > > >> Guido van Rossum wrote: > > >>> Can anyone help out evaluating this patch? If it has to wait > for me > > >>> it's gonna be a looooooong wait... > > >> Looks fine to me functionally, although it seems to > gratuitously retab > > >> some code that used to be aligned with tabstop 8 to a tabstop > of 4. > > > > > > Thanks! I hope you meant "indentation level" instead of "tabstop". > > > Tomer, can you fix the indentation to be whatever's prevailing > in the > > > file you're editing? > > > > Another question is whether __bool__() should be allowed to return an > > int (or any other object), or if it *must* be a bool. (The patch > > currently allows ints). What if __bool__() returns subclasses of int, > > that overwrite __bool__() themself? > > > > Servus, > > Walter > > > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > > From guido at python.org Tue Nov 21 20:16:20 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Nov 2006 11:16:20 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: Message-ID: On 11/20/06, Guido van Rossum wrote: > I'd like to have a discussion of abilities and interfaces but right > now I don't have time -- maybe tomorrow. I see Andrew's proposal as > mostly isomorphic to Zope's notion of interfaces, but the new > terminology may make it possible to clarify certain distinctions > better, e.g. the distinction between a class that provides an ability > to its instances, and the abilities that an instance has. > > Feel free to riff on this theme here. I'll check in within 24 hours or so. OK, I have some more time for this now. I'm hoping Andrew, Bill and Phillip will join the discussion here (instead of various other threads). A. Preliminaries. There are some open issues that we should probably address before diving too deep into the specification of abilities or interfaces. 1. Abstract Base Classes (ABCs)? Are we sure that we have passed the station of ABCs, or should we revisit why those aren't sufficient for our purposes? Bill Janssen seems to be in favor of just using ABCs. Pro: less new infrastructure. Con: hard to add a new ABC on the fly to an existing 3rd party class (mucking with __bases__ is too much of a hack to seriously propose). I'd like to explore this some more before committing to anything. 2. Do we need interfaces if we have generic functions? Phillip Eby argues that we don't need interfaces, because the right API for generic functions makes it easy to do what we do with interfaces. Let me paraphrase his argument. Suppose we have a generic function F with implementations F1, F2, ... (generally: Fi for 1 <= i <= n) suitable for different kinds of arguments. (Concrete example: flatten() implemented one way for strings, another way for lists.) Suppose we have a class C and we want to make it acceptable to F, using some implementation Fi. Using interfaces, each Fi is declared to take an argument that has a particular interface, and we bind C to a certain Fi by claiming that C provides the interface that Fi takes. This also makes C instances behave in a certain way for all other generic functions that mention the same interface. Using Phillip's proposal, we select the right Fi and bind it to the class C using the assignment F[C] = Fi. This doesn't do anything for other generic functions that might also conceivably work with C. Phillip addresses that by proposing help functions that do a bunch of these bindings at once, and by noticing that if you just use the standard library and generic functions, the default implementation might just work. Phillip then argues that he doesn't want to encourage introspectable interfaces. I think others have use cases for those though. It seems that Phillip's approach only hanges well together if everybody totally adopts the generic functions religion; I think that's an unlikely (since too radical) change of course for Python 3.0. It also doesn't seem to work for abilities that are tied to an instance instead of to a class as Zope allows (see below). Assuming we go with abilities (of some sort), we have more concrete things to argue over: B. Developing a spec. I see this as a number of steps, although there's feedback possible between them. We can borrow heavily from Zope, and perhaps somewhat from Java. 1. Naming and general ideas. Should we call these Abilities or Interfaces? Abilities, the term proposed by Andrew Koenig, have less baggage, but possibly they're just an isomorphic relabeling of interfaces. From Zope comes the important question: Do we want to discuss the interfaces/abilities of an instance separately from those provided by its class? Zope's answer is Yes, reminds Jean-Paul Calderone, and I tend to like this (having been exposed to it in the past). 2. More precise concepts. I'd like to distinguish between checking for abilities (which should be quick as it may be used many times during execution, e.g. whenever a generic function is called) and verifying an ability, which would entail checking the class to see whether it actually implements the API (and possibly maintains the invariants, if we go that route) stipulated by a claimed ability. It would be a performance nightmare if during he normal course of an application we would be *verifying* abilities over and over -- this is equivalent to a structural type check and can be very expensive. (Even more so if we have optional argument type declarations.) 3. API design -- how do we spell the various concepts? E.g. has_ability(x, A) asks whether object x has the ability A; provides_ability(C, A) asks whether class C provides the ability A to its instances. We could state that provides_ability(C, A) and isinstance(x, C) implies has_ability(x, A). (A corollary would be that a subclass can't take away an ability provided by a base class!) I think it would be handy to have APIs that ask a class or instance for the set of abilities it provides or has, e.g. get_abilities(x) or get_provisions(C). Should And probably also for APIs to add/remove abilities dynamically (if not constrained by base classes). The verification API is separate also. Some other questions: - We need to come up with syntax for the common case, declaring a class that claims to provide some abilities. Some thoughts: putting the interfaces in the list of base classes might work -- the metaclass can easily tell whether a particular base is a real class or an ability. Another idea might be to add keyword parameters to the class declaration, so we could write e.g. class C(B1, B2, abilities=(A1, A2)): ... - Should has_ability(x, A) pass the responsibility of computing the outcome to the object A, which could implement an arbitrarily complex algorithm to decide? Antoine Pitrou proposes this. The downside is that we won't able to implement get_abilities(x) properly, or efficiently. I'm also not sure of the use case. - If has_ability(x, A) is true, should isinstance(x, C) be true? - How does this interact with generic functions? I think the generic function mechanism (if we introduce one) will have to be able to use either classes or abilities to trigger behavior; while abilities are nice and general, one shouldn't be forced to declare a new ability when all one cares about is one particular concrete class (this happens often enough in applications I develop). Obviously this is an area where Bill's ABC's score better. C. A library of standard abilities and mapping of abilities to standard objects (Sequence to list, etc.). It's easy to come up with a strawman set of interfaces, e.g. Iterable, Iterator, Collection, Set, Sequence, Mapping, String, MutableSet, MutableSequence, MutableMapping, MutableString, and inheritance relations between them, e.g. Iterable - Collection - Sequence - MutableSequence - list. But it's not so easy to define which APIs must be present to for verification of a particular interface to pass. A Mapping must implement __getitem__() in a certain way. But does it need to implement __len__? get()? keys()? values() and items()? Is it okay for a particular mapping implementation to allow only one iterator to exist (per instance) at any time, like dbm files do? Does a String need to implement lower()? Must a Sequence implement __add__() as concatenation? Answering these and similar questions probably requires a standardization committed. (Any volunteers?) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Nov 21 20:18:20 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Nov 2006 11:18:20 -0800 Subject: [Python-3000] Abilities / Interfaces: why not adapt()? In-Reply-To: <5.1.1.6.0.20061121120913.03fb6738@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121120913.03fb6738@sparrow.telecommunity.com> Message-ID: On 11/21/06, Phillip J. Eby wrote: > Let's ponder that a moment. Do you really need interfaces? I responded to this in the parent thread (Abilities / Interfaces). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From walter at livinglogic.de Tue Nov 21 21:18:30 2006 From: walter at livinglogic.de (=?ISO-8859-2?Q?Walter_D=F6rwald?=) Date: Tue, 21 Nov 2006 21:18:30 +0100 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: <1d85506f0611211029i3969a91cw1ad621d15d4374e7@mail.gmail.com> References: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com> <1d85506f0611210359m7625c01ava3b2530465d4c8a4@mail.gmail.com> <45632E22.2030705@solarsail.hcs.harvard.edu> <45633AB8.9010306@livinglogic.de> <1d85506f0611211029i3969a91cw1ad621d15d4374e7@mail.gmail.com> Message-ID: <45635F16.3060404@livinglogic.de> tomer filiba wrote: > patch updated You're fast! ;) AFAICT now we have the following problem: If Objects/typeobject.c::slot_nb_bool() falls back to using __len__(), the return value of __len__() must be a bool. Servus, Walter > On 11/21/06, *Guido van Rossum* > wrote: > > I think it would set a good example if __bool__ was *required* to > return exactly True or False and no other object, not even int > instances equal to 0 or 1. Surely that can't be much of a burden. Feel > free to upload an improved patch. > > On 11/21/06, Walter D?rwald > wrote: > > Guido van Rossum wrote: > > > On 11/21/06, Ivan Krsti? < krstic at solarsail.hcs.harvard.edu > > wrote: > > >> Guido van Rossum wrote: > > >>> Can anyone help out evaluating this patch? If it has to wait > for me > > >>> it's gonna be a looooooong wait... > > >> Looks fine to me functionally, although it seems to > gratuitously retab > > >> some code that used to be aligned with tabstop 8 to a tabstop of 4. > > > > > > Thanks! I hope you meant "indentation level" instead of "tabstop". > > > Tomer, can you fix the indentation to be whatever's prevailing > in the > > > file you're editing? > > > > Another question is whether __bool__() should be allowed to return an > > int (or any other object), or if it *must* be a bool. (The patch > > currently allows ints). What if __bool__() returns subclasses of int, > > that overwrite __bool__() themself? > > > > Servus, > > Walter > > > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > > From sluggoster at gmail.com Tue Nov 21 21:15:37 2006 From: sluggoster at gmail.com (Mike Orr) Date: Tue, 21 Nov 2006 12:15:37 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: Message-ID: <6e9196d20611211215j782fd682s64681e1e6357960@mail.gmail.com> On 11/21/06, Guido van Rossum wrote: > 2. More precise concepts. I'd like to distinguish between checking for > abilities (which should be quick as it may be used many times during > execution, e.g. whenever a generic function is called) and verifying > an ability, which would entail checking the class to see whether it > actually implements the API (and possibly maintains the invariants, if > we go that route) stipulated by a claimed ability. It would be a > performance nightmare if during he normal course of an application we > would be *verifying* abilities over and over -- this is equivalent to > a structural type check and can be very expensive. (Even more so if we > have optional argument type declarations.) If I understand correctly, checking for an ability means "Does class C claim to have ability A?", analagous to isinstance(), and verifying an ability means introspecting the class and inspecting its methods. The distinction is similar to checking whether an XML document claims to match a DTD (or just assuming it does), vs checking all the tags and attributes against the DTD. The latter is definitely a lot of overhead you don't need... except when you do. > A Mapping must > implement __getitem__() in a certain way. But does it need to > implement __len__? get()? keys()? values() and items()? Yes. And no. In other words, there should be at least two choices. When most people think of "mapping" they expect x[key] to work. When they think of "dict like" they may mean this, or they may want to also call x.keys/values/items, and less commonly x.update and x.popitem. Having two levels (Mapping and DictLike) would solve 90% of the problem. Having three levels may be overkill. Same for lists, etc. I wouldn't expect len(x) to work on every mapping. Maybe the mapping models an external resource like a filesystem, and calculating the length would be expensive. Maybe the object is volatile and doesn't allow ascertaining all keys but only lookup of a particular key. An example would be Cheetah's NameMapper. It looks up a variable against a list of objects, examining each object's attributes and keys at the current moment. Although it would be theoretically possible to list all keys and calculate their number, this is not a supported operation, and when you're recursing through 'self' and '__builtins__' the number would become very large. The correct response to "Does variable x exist?" is not "Is it in the list of existing keys?" but "Does the NameMapper succeed in finding it?" (As an aside, I got a momentary power outage while typing this, which made me restart my computer. It reminded me that these things can happen at any time.) -- Mike Orr From pje at telecommunity.com Tue Nov 21 21:26:50 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 21 Nov 2006 15:26:50 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: Message-ID: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> At 11:16 AM 11/21/2006 -0800, Guido van Rossum wrote: >Phillip then argues that he doesn't want to encourage introspectable >interfaces. I think others have use cases for those though. Examples? > It seems >that Phillip's approach only hanges well together if everybody totally >adopts the generic functions religion; I think that's an unlikely >(since too radical) change of course for Python 3.0. Is it really radical? Consider the fact that single-dispatch generic functions are ubiquitous in the language and the stdlib, and have been pretty much forever. They just vary in how they're implemented, and there's no One Obvious Way for developers to implement new ones, nor is there a standard way to add a method to an existing one. That's because some generic functions use if-isinstance checks (bad), while others use custom __special__ methods (not bad, but not great), registries (good), adaptation (okay), or generic function libraries. So, all I am proposing is that we: * provide a standard GF implementation for the simple cases * ...that is extensible to allow others to handle the more complex cases, by having a generic API for manipulating generic functions So that the "one obvious way" to create new generics is to use the standard GF implementation unless you need something else. And, there would be a standard way to add new methods to existing generic functions, perhaps using an addmethod() builtin or decorator, and it could be designed so that e.g.: addmethod(iter, somefunc, sometype) would actually work by doing 'sometype.__iter__ = somefunc' under the hood. This allows us not to have to change any builtin generics that are based on special method names. In other words, the mechanism isn't radical, nor is the idea of having generics. It's just an improvement in ease-of-use: an HCI change, not a comp-sci change! And, with this relatively simple mechanism, all the fancier forms of generic functions (or interfaces and adapters) can be implemented via user libraries. (More on how, below.) > It also doesn't >seem to work for abilities that are tied to an instance instead of to >a class as Zope allows (see below). Actually, it still allows for that, as long as the base generic function system is extensible. For example, RuleDispatch generic functions can choose implementations based on conditions such as whether an object has a particular attribute. Zope could easily add a generic function type that uses their instance-based interface stuff to do the same thing. I am merely proposing that Python not provide this sort of introspection-oriented stuff in the core or stdlib, not that nobody should be *allowed* to have it. Indeed, I explicitly want the Python generic function API (e.g. addmethod, hasmethod, or whatever we call them) to *itself* be generic, so that users can create their own types that work with the Python-provided operations for generic function manipulation. That is, I should be able to call: addmethod(addmethod, adder_for_my_gf_type, my_gf_type) So that others can then call: # this calls adder_for_my_gf_type under the hood: addmethod(some_gf, some_method, some_cls) where 'some_gf' is an instance of 'my_gf_type'. This allows us not to keep the core or stdlib implementation of generic functions quite simple to handle the 80% (single dispatch) or 90% (concrete type-based multiple dispatch) with ease. The final 10% (predicate dispatch, fancy interfaces, etc.) can then be done by outside libraries, since the use cases in that final 10% are more likely to vary than the base 80-90%. Using the 'simplegeneric' library from PyPI (easy_install simplegeneric): from simplegeneric import generic @generic def addmethod(gf, method, cls): """Add a method to a generic function""" raise TypeError("Unknown generic function type", gf) @addmethod.when_object(iter) def add_to_iter(gf, method, cls): # XXX should have more error checking here, e.g. # check for a local __iter__ first cls.__iter__ = method # ... similar declarations for other builtin generics @addmethod.when_type(FunctionType) def add_to_function(gf, method, cls): if hasattr(gf, 'when_type'): gf.when_type(cls)(method) else: raise TypeError("Not a generic function", gf) And there you are: an extensible way to add new extensible function types, while using the same API for all of them. (If you need multiple or predicate dispatch or method combining, it's easy to have decorators that apply to the method itself, so that the same three-argument addmethod() can still be used.) >3. API design -- how do we spell the various concepts? E.g. >has_ability(x, A) asks whether object x has the ability A; >provides_ability(C, A) asks whether class C provides the ability A to >its instances. We could state that provides_ability(C, A) and >isinstance(x, C) implies has_ability(x, A). Even if we explicitly have some type of "ability" object, I'd like them to be defined in terms of generic functions, so that I could effectively say something like: sequence_like = (iter & len) (or perhaps spelled out in some other fashion) to create an "ability" representing iter-ability and len-ability. One of the biggest conceptual/modeling issues with Zope-style interfaces is that they don't allow this kind of fine-grained protocol combination. See, for example, zope.interface's long history of trying to nail down various Python protocols through elaborate interface inheritance hierarchies. Defining interfaces strictly in terms of operations eliminates this issue. >- How does this interact with generic functions? The advanced abilities (per-instance/on-the-fly and multi-operation tests) will likely affect performance and/or simplicity of the default implementation. A purely type-based system can be implemented efficiently, because declaring an interface can simultaneously add a concrete type registration for all affected generics. (And registering an interface-linked method in a generic function can pull in all the concrete classes.) Per-instance tests, however, increase code complexity. Generally speaking, RuleDispatch does per-instance tests after class tests, but the way it gets reasonable performance is by building decision trees beforehand to manage the possible tests and avoid overlap. If it actually had to call individual methods, or call a method more than once (ob.has_ability(Foo), ob.has_ability(Bar), etc.) it would be considerably slower to select an option. Also -- and this is the real kicker -- what do you do if more than one ability applies? Now we have to have precedence rules for what's more specific. The advantage of defining an ability as a set of one or more applicable generic functions is that precedence is comparatively straightforward: supersets take precedence over subsets, and overlaps are ambiguous. You also have some possibility of being able to implement this by registration-time checks, without needing to build a dispatch tree at all. I don't know of anybody who really uses per-instance interface declarations except for Zope. I used them for a little while with PEAK, and decided they were the wrong idea for me; it made more sense to adapt to an interface that provides the metadata, or to use a RuleDispatch function that just directly introspected for whatever was needed. I can't think of anything in the core or stdlib that would even need to go that far. Again, my argument is that anything other than isinstance-based single and multiple-dispatch is too framework-dependent to be part of the language. Simple things should be simple, complex things should be possible. >Answering these and similar questions probably requires a >standardization committed. (Any volunteers?) Note that Zope has already tried for many years to do exactly this, and there is no indication that they have actually succeeded. When I first mentioned PyProtocols on Python-Dev many years ago, Samuele Pedroni argued that individual operations (and groups thereof) should be the currency of interfaces, and after playing with them a bit, I agreed. PyProtocols defines such interfaces as e.g.: protocolForType(file, ['read', 'close']) to mean "file-like object with 'read' and 'close' methods". If I define a class as supporting that protocol, and somebody wants to adapt to 'protocolForType(file, ["read"])', then my type will work. See http://peak.telecommunity.com/protocol_ref/protocols-generated-type.html for a more complete explanation, including how to get the system to actually check for the method names (so as not to require explicit declarations). I think Samuele's idea (i.e., this) works a lot better than trying to define a rigid interface hierarchy for builtin-protocols, although I'd just as soon be able to do something like: file.read & file.close rather than having to use strings. But all this assumes that we are providing a builtin way to spell abilities and introspect them, which I still think is overkill for what Python actually needs to provide as a base. In short, I think it's adding interface introspection that's the radical move, where generic functions are just a bit of polish that brings a bit more order to what Python already has. In contrast, interfaces are a foreign religion imported from other languages, while the roots of the generic function faith (len, iter, etc.) have already been in place since the dawn of time! :) From brett at python.org Tue Nov 21 21:32:05 2006 From: brett at python.org (Brett Cannon) Date: Tue, 21 Nov 2006 12:32:05 -0800 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: <45635F16.3060404@livinglogic.de> References: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com> <1d85506f0611210359m7625c01ava3b2530465d4c8a4@mail.gmail.com> <45632E22.2030705@solarsail.hcs.harvard.edu> <45633AB8.9010306@livinglogic.de> <1d85506f0611211029i3969a91cw1ad621d15d4374e7@mail.gmail.com> <45635F16.3060404@livinglogic.de> Message-ID: On 11/21/06, Walter D?rwald wrote: > > tomer filiba wrote: > > patch updated > > You're fast! ;) > > AFAICT now we have the following problem: If > Objects/typeobject.c::slot_nb_bool() falls back to using __len__(), the > return value of __len__() must be a bool. Why can't the fallback usage just pass the return value from __len__ to bool() (forget the C function name) and return that result? It's just like doing:: def bool(obj): try: return obj.__bool__() except AttributeError: return bool(len(obj)) -Brett Servus, > Walter > > > On 11/21/06, *Guido van Rossum* > > wrote: > > > > I think it would set a good example if __bool__ was *required* to > > return exactly True or False and no other object, not even int > > instances equal to 0 or 1. Surely that can't be much of a burden. > Feel > > free to upload an improved patch. > > > > On 11/21/06, Walter D?rwald > > wrote: > > > Guido van Rossum wrote: > > > > On 11/21/06, Ivan Krsti? < krstic at solarsail.hcs.harvard.edu > > > wrote: > > > >> Guido van Rossum wrote: > > > >>> Can anyone help out evaluating this patch? If it has to wait > > for me > > > >>> it's gonna be a looooooong wait... > > > >> Looks fine to me functionally, although it seems to > > gratuitously retab > > > >> some code that used to be aligned with tabstop 8 to a tabstop > of 4. > > > > > > > > Thanks! I hope you meant "indentation level" instead of > "tabstop". > > > > Tomer, can you fix the indentation to be whatever's prevailing > > in the > > > > file you're editing? > > > > > > Another question is whether __bool__() should be allowed to return > an > > > int (or any other object), or if it *must* be a bool. (The patch > > > currently allows ints). What if __bool__() returns subclasses of > int, > > > that overwrite __bool__() themself? > > > > > > Servus, > > > Walter > > > > > > > > > > > > -- > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > > > > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: > http://mail.python.org/mailman/options/python-3000/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061121/3e851bf4/attachment.html From fumanchu at amor.org Tue Nov 21 21:50:47 2006 From: fumanchu at amor.org (Robert Brewer) Date: Tue, 21 Nov 2006 12:50:47 -0800 Subject: [Python-3000] Abilities / Interfaces Message-ID: <435DF58A933BA74397B42CDEB8145A8606E5D65D@ex9.hostedexchange.local> Guido van Rossum wrote: > But it's not so easy to define which APIs must be present to for > verification of a particular interface to pass. A Mapping must > implement __getitem__() in a certain way. > But does it need to implement __len__? The Ref Manual seems to think so: http://docs.python.org/ref/types.html#l2h-68: "The subscript notation a[k] selects the item indexed by k from the mapping a; this can be used in expressions and as the target of assignments or del statements. The built-in function len() returns the number of items in a mapping." > get()? keys()? values() and items()? http://docs.python.org/ref/sequence-types.html implies all other methods are optional (even if heavily encouraged): "It is also recommended that mappings provide the methods keys(), values(), items(), has_key(), get(), clear(), setdefault(), iterkeys(), itervalues(), iteritems(), pop(), popitem(), copy(), and update() behaving similar to those for Python's standard dictionary objects." I'd be happy starting with the requirements as stated in the ref (__getitem__, __setitem__, __delitem__, and __len__, as I read it), and make any additions or other changes to that contract a separate discussion. (But then, I'm personally happy with having this API specified in the ref and don't see a need to codify it with interfaces.) > Is it okay for a particular mapping implementation to allow > only one iterator to exist (per instance) at any time, > like dbm files do? Sure, because the current spec doesn't deny it. If we have to have interfaces, I think we'd be far better off implementing the spec as-is first, and worry about changing the spec later (even if if only seems to be an "implied spec" at the moment). I didn't write the Ref Manual, of course, sir ;) so I can't assert its intent unequivocally, but as a consumer of it, I expect "it is recommended" to mean "not an interface requirement". Robert Brewer System Architect Amor Ministries fumanchu at amor.org From guido at python.org Tue Nov 21 22:27:17 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Nov 2006 13:27:17 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <435DF58A933BA74397B42CDEB8145A8606E5D65D@ex9.hostedexchange.local> References: <435DF58A933BA74397B42CDEB8145A8606E5D65D@ex9.hostedexchange.local> Message-ID: On 11/21/06, Robert Brewer wrote: > Guido van Rossum wrote: > > But it's not so easy to define which APIs must be present to for > > verification of a particular interface to pass. A Mapping must > > implement __getitem__() in a certain way. > > But does it need to implement __len__? > > The Ref Manual seems to think so: > http://docs.python.org/ref/types.html#l2h-68: > > "The subscript notation a[k] selects the item indexed by k from the > mapping a; this can be used in expressions and as the target of > assignments or del statements. The built-in function len() returns the > number of items in a mapping." > > > get()? keys()? values() and items()? > > http://docs.python.org/ref/sequence-types.html implies all other methods > are optional (even if heavily encouraged): > > "It is also recommended that mappings provide the methods keys(), > values(), items(), has_key(), get(), clear(), setdefault(), iterkeys(), > itervalues(), iteritems(), pop(), popitem(), copy(), and update() > behaving similar to those for Python's standard dictionary objects." > > I'd be happy starting with the requirements as stated in the ref > (__getitem__, __setitem__, __delitem__, and __len__, as I read it), and > make any additions or other changes to that contract a separate > discussion. (But then, I'm personally happy with having this API > specified in the ref and don't see a need to codify it with interfaces.) For Py3k we certainly have the freedom (not to mention the obligation :-) to rewrite the reference manual, so this isn't dogma for me. > > Is it okay for a particular mapping implementation to allow > > only one iterator to exist (per instance) at any time, > > like dbm files do? > > Sure, because the current spec doesn't deny it. > > If we have to have interfaces, I think we'd be far better off > implementing the spec as-is first, and worry about changing the spec > later (even if if only seems to be an "implied spec" at the moment). I > didn't write the Ref Manual, of course, sir ;) so I can't assert its > intent unequivocally, but as a consumer of it, I expect "it is > recommended" to mean "not an interface requirement". IMO the situation with respect to the current "de-facto" interfaces and standard types is sufficiently messy that we will be *forced* to change the spec. In fact I'd be disappointed if some good deep thinking about the ontology of interfaces wouldn't lead us to some welcome changes to the specs. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tomerfiliba at gmail.com Tue Nov 21 22:34:50 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Tue, 21 Nov 2006 23:34:50 +0200 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: References: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com> <1d85506f0611210359m7625c01ava3b2530465d4c8a4@mail.gmail.com> <45632E22.2030705@solarsail.hcs.harvard.edu> <45633AB8.9010306@livinglogic.de> <1d85506f0611211029i3969a91cw1ad621d15d4374e7@mail.gmail.com> <45635F16.3060404@livinglogic.de> Message-ID: <1d85506f0611211334h1182c73cj6aac9d9b40bef99b@mail.gmail.com> patch updated (hopefully for the last time. it got me all bored :) On 11/21/06, Brett Cannon wrote: > > > > On 11/21/06, Walter D?rwald wrote: > > > > tomer filiba wrote: > > > patch updated > > > > You're fast! ;) > > > > AFAICT now we have the following problem: If > > Objects/typeobject.c::slot_nb_bool() falls back to using __len__(), the > > return value of __len__() must be a bool. > > > Why can't the fallback usage just pass the return value from __len__ to > bool() (forget the C function name) and return that result? It's just like > doing:: > > def bool(obj): > try: > return obj.__bool__() > except AttributeError: > return bool(len(obj)) > > -Brett > > Servus, > > Walter > > > > > On 11/21/06, *Guido van Rossum* > > > wrote: > > > > > > I think it would set a good example if __bool__ was *required* to > > > return exactly True or False and no other object, not even int > > > instances equal to 0 or 1. Surely that can't be much of a burden. > > Feel > > > free to upload an improved patch. > > > > > > On 11/21/06, Walter D?rwald > > > wrote: > > > > Guido van Rossum wrote: > > > > > On 11/21/06, Ivan Krsti? < krstic at solarsail.hcs.harvard.edu > > > > wrote: > > > > >> Guido van Rossum wrote: > > > > >>> Can anyone help out evaluating this patch? If it has to wait > > > > > for me > > > > >>> it's gonna be a looooooong wait... > > > > >> Looks fine to me functionally, although it seems to > > > gratuitously retab > > > > >> some code that used to be aligned with tabstop 8 to a tabstop > > of 4. > > > > > > > > > > Thanks! I hope you meant "indentation level" instead of > > "tabstop". > > > > > Tomer, can you fix the indentation to be whatever's prevailing > > > in the > > > > > file you're editing? > > > > > > > > Another question is whether __bool__() should be allowed to > > return an > > > > int (or any other object), or if it *must* be a bool. (The patch > > > > > > currently allows ints). What if __bool__() returns subclasses of > > int, > > > > that overwrite __bool__() themself? > > > > > > > > Servus, > > > > Walter > > > > > > > > > > > > > > > > > -- > > > --Guido van Rossum (home page: http://www.python.org/~guido/ > > ) > > > > > > > > > > _______________________________________________ > > Python-3000 mailing list > > Python-3000 at python.org > > http://mail.python.org/mailman/listinfo/python-3000 > > Unsubscribe: > > http://mail.python.org/mailman/options/python-3000/brett%40python.org > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061121/374bc5be/attachment.html From sluggoster at gmail.com Tue Nov 21 22:37:57 2006 From: sluggoster at gmail.com (Mike Orr) Date: Tue, 21 Nov 2006 13:37:57 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <435DF58A933BA74397B42CDEB8145A8606E5D65D@ex9.hostedexchange.local> References: <435DF58A933BA74397B42CDEB8145A8606E5D65D@ex9.hostedexchange.local> Message-ID: <6e9196d20611211337k6bb50b77y30f1921a79131637@mail.gmail.com> On 11/21/06, Robert Brewer wrote: > If we have to have interfaces, I think we'd be far better off > implementing the spec as-is first, and worry about changing the spec > later (even if if only seems to be an "implied spec" at the moment). Well, except that we have a lot of real-world experience since the spec was written. I'm not opposed to doing it this way, I'm just afraid the first will happen and the second won't, and then the argument will be, "We can't do it because it's against the spec." We're doing something new here; we're adding something to Python, not taking anything away. So this is a perfect time to ponder whether the spec was optimal in all its details; e.g., whether requiring all mappings to have a length is necessary or desirable. The real question is, who will be using these interfaces, what groups of capabilities will they most likely be looking for, and are these adequately addressed in the spec? -- Mike Orr From rrr at ronadam.com Tue Nov 21 22:56:03 2006 From: rrr at ronadam.com (Ron Adam) Date: Tue, 21 Nov 2006 15:56:03 -0600 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: Message-ID: Guido van Rossum wrote: > On 11/20/06, Guido van Rossum wrote: >> I'd like to have a discussion of abilities and interfaces but right >> now I don't have time -- maybe tomorrow. I see Andrew's proposal as >> mostly isomorphic to Zope's notion of interfaces, but the new >> terminology may make it possible to clarify certain distinctions >> better, e.g. the distinction between a class that provides an ability >> to its instances, and the abilities that an instance has. >> >> Feel free to riff on this theme here. I'll check in within 24 hours or so. > > OK, I have some more time for this now. I'm hoping Andrew, Bill and > Phillip will join the discussion here (instead of various other > threads). > I'd like to explore this some more before committing to anything. Here's a thought: For identifying abilities, it almost seems like instead of hard coding a library of this is that relationships, which will always be incomplete and possibly easy to fool, an ability might be described as a small (minimal) test that a interface can pass. Registering an ability relationship could require running it against an ability test prior to use. It doesn't have to be done just prior to use. It could be done at import time, or even earlier with the result restored at application run time so it can be a one time cost for most applications. This may make it more open ended and add a level of actual testing that may be useful for debugging as well. Ron From guido at python.org Tue Nov 21 23:05:07 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Nov 2006 14:05:07 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: Message-ID: I thought I addressed that (search my mail for "Antoine Pitrou"); I expect that this will be too slow for use by the dispatch mechanism for generic functions (or adapt, or anything else that makes use of interfaces assuming the declared and actual interface match). On 11/21/06, Ron Adam wrote: > Guido van Rossum wrote: > > On 11/20/06, Guido van Rossum wrote: > >> I'd like to have a discussion of abilities and interfaces but right > >> now I don't have time -- maybe tomorrow. I see Andrew's proposal as > >> mostly isomorphic to Zope's notion of interfaces, but the new > >> terminology may make it possible to clarify certain distinctions > >> better, e.g. the distinction between a class that provides an ability > >> to its instances, and the abilities that an instance has. > >> > >> Feel free to riff on this theme here. I'll check in within 24 hours or so. > > > > OK, I have some more time for this now. I'm hoping Andrew, Bill and > > Phillip will join the discussion here (instead of various other > > threads). > > > > I'd like to explore this some more before committing to anything. > > > Here's a thought: > > For identifying abilities, it almost seems like instead of hard coding a library > of this is that relationships, which will always be incomplete and possibly easy > to fool, an ability might be described as a small (minimal) test that a > interface can pass. Registering an ability relationship could require running > it against an ability test prior to use. > > It doesn't have to be done just prior to use. It could be done at import time, > or even earlier with the result restored at application run time so it can be a > one time cost for most applications. > > This may make it more open ended and add a level of actual testing that may be > useful for debugging as well. > > Ron > > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gsakkis at rutgers.edu Tue Nov 21 23:18:31 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Tue, 21 Nov 2006 17:18:31 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: Message-ID: <91ad5bf80611211418o76941e13l6864e97e2df23154@mail.gmail.com> On 11/21/06, Guido van Rossum wrote: > A. Preliminaries. > > There are some open issues that we should probably address before > diving too deep into the specification of abilities or interfaces. > > 1. Abstract Base Classes (ABCs)? > > Are we sure that we have passed the station of ABCs, or should we > revisit why those aren't sufficient for our purposes? > > Bill Janssen seems to be in favor of just using ABCs. Pro: less new > infrastructure. Con: hard to add a new ABC on the fly to an existing > 3rd party class (mucking with __bases__ is too much of a hack to > seriously propose). > > I'd like to explore this some more before committing to anything. I'd like that too, with some concrete examples if possible. I'm still unsure what interfaces buy us that ABCs (are these the same/equivalent to mixins?) don't, especially with an adaptation/generic function mechanism in place. George From fdrake at acm.org Tue Nov 21 23:25:06 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue, 21 Nov 2006 17:25:06 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <91ad5bf80611211418o76941e13l6864e97e2df23154@mail.gmail.com> References: <91ad5bf80611211418o76941e13l6864e97e2df23154@mail.gmail.com> Message-ID: <200611211725.06887.fdrake@acm.org> On Tuesday 21 November 2006 17:18, George Sakkis wrote: > I'd like that too, with some concrete examples if possible. I'm still > unsure what interfaces buy us that ABCs (are these the same/equivalent > to mixins?) don't, especially with an adaptation/generic function > mechanism in place. One thing that mixins/ABCs can't support that we use quite frequently in Zope 3 applications is stamping additional interfaces on instances. This is often done to influence adaptation. -Fred -- Fred L. Drake, Jr. From ark-mlist at att.net Tue Nov 21 23:42:32 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Tue, 21 Nov 2006 17:42:32 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: Message-ID: <006301c70dbe$553962c0$6402a8c0@arkdesktop> > In fact I'd be disappointed if some good deep > thinking about the ontology of interfaces wouldn't lead us to some > welcome changes to the specs. I don't know where this idea falls on the continuum between deep and silly, but... Suppose we stick for the moment with the notion that types (and objects of those types) can have abilities, and that those abilities are represented by other objects. So an object of type Foo might have ability Bar, and we would like to have some way of testing whether Foo has that ability. In other words, we would like a way to determine whether the author of Foo claims that Foo has ability Bar. Nothing new so far. But it just occurred to me that part of an ability (i.e. one of the methods of an ability) might be code that performs compliance tests on an object that claims to have that ability. This idea is very sketchy, but it suggests that perhaps abilities could be parking places for generic unit tests for interfaces, rather than tests of specific types or pieces of code. From pje at telecommunity.com Tue Nov 21 23:59:40 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 21 Nov 2006 17:59:40 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <200611211725.06887.fdrake@acm.org> References: <91ad5bf80611211418o76941e13l6864e97e2df23154@mail.gmail.com> <91ad5bf80611211418o76941e13l6864e97e2df23154@mail.gmail.com> Message-ID: <5.1.1.6.0.20061121174806.01f19c50@sparrow.telecommunity.com> At 05:25 PM 11/21/2006 -0500, Fred L. Drake, Jr. wrote: >On Tuesday 21 November 2006 17:18, George Sakkis wrote: > > I'd like that too, with some concrete examples if possible. I'm still > > unsure what interfaces buy us that ABCs (are these the same/equivalent > > to mixins?) don't, especially with an adaptation/generic function > > mechanism in place. > >One thing that mixins/ABCs can't support that we use quite frequently in Zope >3 applications is stamping additional interfaces on instances. This is often >done to influence adaptation. I was under the impression that Zope's actual use cases for instance-specific interfaces have mainly to do with *view* lookups and other types of lookups that don't really come into the current discussion scope as I understand it. That is, I thought that instance interfaces were used primarily for "n-tuple adaptation" (the adaptation equivalent of multiple and/or predicate dispatch)? If that's the case, then ISTM that as long as we make the core mechanism extensible, Zope's use cases should be handle-able. I certainly wouldn't be in favor of a proposal that would marginalize Zope's paradigm, any more than I'd be in favor of one that blessed it. :) I just want a minimal core that blesses what the language and stdlib *already* do (special methods and single-dispatch __mro__ lookups), while still allowing "advanced" paradigms (Zope, RuleDispatch, etc.) to "play on a level playing field" with each other. From guido at python.org Tue Nov 21 23:59:36 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Nov 2006 14:59:36 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <006301c70dbe$553962c0$6402a8c0@arkdesktop> References: <006301c70dbe$553962c0$6402a8c0@arkdesktop> Message-ID: On 11/21/06, Andrew Koenig wrote: > > In fact I'd be disappointed if some good deep > > thinking about the ontology of interfaces wouldn't lead us to some > > welcome changes to the specs. > > I don't know where this idea falls on the continuum between deep and silly, > but... > > Suppose we stick for the moment with the notion that types (and objects of > those types) can have abilities, and that those abilities are represented by > other objects. So an object of type Foo might have ability Bar, and we > would like to have some way of testing whether Foo has that ability. In > other words, we would like a way to determine whether the author of Foo > claims that Foo has ability Bar. > > Nothing new so far. But it just occurred to me that part of an ability > (i.e. one of the methods of an ability) might be code that performs > compliance tests on an object that claims to have that ability. This idea > is very sketchy, but it suggests that perhaps abilities could be parking > places for generic unit tests for interfaces, rather than tests of specific > types or pieces of code. I believe Ka-Ping once proposed something similar. This also jives nicely with the "verify" functionality that I mentioned. However, I don't know how easy it will be to write such compliance tests given that typically the constructor signature is not part of the ability. It may be more pragmatic to put the tests in a separate unit testing module that can be subclassed by unit testing code for classes that want specific claims to be verified. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tomerfiliba at gmail.com Wed Nov 22 00:07:06 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Wed, 22 Nov 2006 01:07:06 +0200 Subject: [Python-3000] LINQ Message-ID: <1d85506f0611211507n54f22ed0l9654f0d02442221d@mail.gmail.com> i read the references fredrik posted several days back, and it got me thinking: why not add a "co_ast" attribute to code objects? i.e., since (virtually) all code objects are constructed from source code by compilation, the compiler could just emit the AST as an attribute to the generated code object. yes, there's an issue of memory consumption, but bear with me for a moment. also, maybe co_ast could be a lazy attribute, reading the ast from the pyc file on demand. having the AST as part of the code object would allow third-party tools, like a LINQ library, to take a normal code object and walk it's AST, generating the relevant SQL text or what not. for those who are not familiar with LINQ, it's an integrated query mechanism, not unlike list comprehensions: from X where Y select Z --> [Z for Z in X if Y] but (and that's a big but) since it's integrated into the language through some special interfaces, classes can implement custom "iterables", i.e., a database. instead of going over the DB in a O(N) fashion (as list-comp does), the LINQ expression can generate SQL text, send it to the DB, and yield the results one at a time. if we combine co_ast with the frame info (or the function's globals), we can do really impressive things: db = sql.connection(...) MIN_AGE = 17 rows = query(firstname for firstname, age in db if age > MIN_AGE) instead of the cumbersome and error-prone version of db.query("select ... from ... where age > %s" % (MIN_AGE,)) we can extract the SELECT and WHERE lines from co_ast, and take the value of MIN_AGE from the frame's globals, thus generating SQL text from an expression, just like LINQ. my intuition tells me there are many possible paths to explore using this method... ================================ [Fredrik Lundh] > (And frankly, it would be *wonderful* if someone could come up with a > new proposal that actually enabled Python programmers to do things they > *cannot* do today, instead of just rehashing old "let's move the sofa > over there" threads. How about doing something along the lines of > LINQ's "give me this expression as an AST so I can optimize it myself" > model or looking at COmega's parallelization/synchronization stuff or > digging deeper into how PJE's RuleDispatch could be fit into Python or > stealing some cool idea from some functional research language that I > haven't even heard of. I want to get *new* things when I upgrade from > 2.X to 3.0, not just silly syntax tweaks that would only give me the > ability to change two lines of Python code to one line of code plus a > comment that explains what the heck that line is doing. Let's do some > *hard* stuff, for a change.) but then again, if we go this far, why not just introduce real, first class code objects? first class code is something LISP has got right from the very beginning (although i hate LISP :) i know metasyntax is considered a taboo here, but i thought i'd share this idea i had for an extensible language: * the compiler's AST could be altered at runtime (a la boo, only better) * bytecode instructions could be added at runtime to show it off a little: with_statement = Statement( Literal("with"), Expression, Literal("as"), Name, Suite ) # also define the code generating visitors for this node ... import compiler compiler.add_statement(with_statement) # now add a new instruction for the WITH statement import sys def enter_with(stack, obj): obj = stack.pop() stack.push( obj.__enter__() ) bytecode = sys.add_instruction(enter_with, 0) --- this way, you could really do from "__future__ import X"... the language can be self-extending, which means we can try new features out before adding them to the C implementation (kinda like PyPy) but i guess it's too far fetched for py3k... maybe i should aim it at py4k :) g'night. -tomer From krstic at solarsail.hcs.harvard.edu Wed Nov 22 00:11:40 2006 From: krstic at solarsail.hcs.harvard.edu (=?UTF-8?B?SXZhbiBLcnN0acSH?=) Date: Tue, 21 Nov 2006 18:11:40 -0500 Subject: [Python-3000] LINQ In-Reply-To: <1d85506f0611211507n54f22ed0l9654f0d02442221d@mail.gmail.com> References: <1d85506f0611211507n54f22ed0l9654f0d02442221d@mail.gmail.com> Message-ID: <456387AC.5060808@solarsail.hcs.harvard.edu> tomer filiba wrote: > but then again, if we go this far, why not just introduce real, first > class code objects? Because Guido has pronounced it's not happening. -- Ivan Krsti? | GPG: 0x147C722D From fumanchu at amor.org Wed Nov 22 00:24:09 2006 From: fumanchu at amor.org (Robert Brewer) Date: Tue, 21 Nov 2006 15:24:09 -0800 Subject: [Python-3000] LINQ Message-ID: <435DF58A933BA74397B42CDEB8145A8606EDBC5E@ex9.hostedexchange.local> tomer filiba wrote: > i read the references fredrik posted several days back, and it got > me thinking: why not add a "co_ast" attribute to code objects? > > i.e., since (virtually) all code objects are constructed from source > code by compilation, the compiler could just emit the AST as an > attribute to the generated code object. > > yes, there's an issue of memory consumption, but bear with me > for a moment. also, maybe co_ast could be a lazy attribute, reading > the ast from the pyc file on demand. > > having the AST as part of the code object would allow > third-party tools, > like a LINQ library, to take a normal code object and walk it's AST, > generating the relevant SQL text or what not. > > for those who are not familiar with LINQ, it's an integrated query > mechanism, not unlike list comprehensions: > from X where Y select Z --> [Z for Z in X if Y] > > but (and that's a big but) since it's integrated into the language > through some special interfaces, classes can implement custom > "iterables", i.e., a database. > > instead of going over the DB in a O(N) fashion (as list-comp does), > the LINQ expression can generate SQL text, send it to the DB, > and yield the results one at a time. > > if we combine co_ast with the frame info (or the function's globals), > we can do really impressive things: > > db = sql.connection(...) > MIN_AGE = 17 > rows = query(firstname for firstname, age in db if age > MIN_AGE) > > instead of the cumbersome and error-prone version of > > db.query("select ... from ... where age > %s" % (MIN_AGE,)) > > we can extract the SELECT and WHERE lines from co_ast, > and take the value of MIN_AGE from the frame's globals, > thus generating SQL text from an expression, just like LINQ. This is exactly what I do in Dejavu [1], only using the co_code instead, and first-class expression objects [2] instead of full-blown code objects. If co_ast were present, I'd probably use it instead; the only reason I use bytecode now (Python 2.4 and under) is because it's faster. > my intuition tells me there are many possible paths to explore > using this method... No intuition necessary; I've already explored them in Dejavu. ;) Read through the short Querying doc [3] for a summary. I have a proposal in for PyCon 2007 to explain the first-class expression objects in detail, and how they enable LINQ-style code. Robert Brewer System Architect Amor Ministries fumanchu at amor.org [1] http://projects.amor.org/dejavu [2] http://projects.amor.org/dejavu/browser/trunk/logic.py [3] http://projects.amor.org/docs/dejavu/managing.html#Querying From nestornissen at gmail.com Wed Nov 22 00:59:00 2006 From: nestornissen at gmail.com (Nestor) Date: Tue, 21 Nov 2006 18:59:00 -0500 Subject: [Python-3000] Abilities / Interfaces Message-ID: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> Short question. Do these aproaches allow to do something like the java.util.RandomAccess interface that is just used as a marker? From rrr at ronadam.com Wed Nov 22 01:02:28 2006 From: rrr at ronadam.com (Ron Adam) Date: Tue, 21 Nov 2006 18:02:28 -0600 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: Message-ID: Guido van Rossum wrote: > I thought I addressed that (search my mail for "Antoine Pitrou"); I > expect that this will be too slow for use by the dispatch mechanism > for generic functions (or adapt, or anything else that makes use of > interfaces assuming the declared and actual interface match). I wasn't able to find the referenced message in this news group or on gmane, but Andrew Koenig's recent post in this thread is along the lines of what I was thinking. > On 11/21/06, Ron Adam wrote: >> Guido van Rossum wrote: >>> On 11/20/06, Guido van Rossum wrote: >>>> I'd like to have a discussion of abilities and interfaces but right >>>> now I don't have time -- maybe tomorrow. I see Andrew's proposal as >>>> mostly isomorphic to Zope's notion of interfaces, but the new >>>> terminology may make it possible to clarify certain distinctions >>>> better, e.g. the distinction between a class that provides an ability >>>> to its instances, and the abilities that an instance has. >>>> >>>> Feel free to riff on this theme here. I'll check in within 24 hours or so. >>> OK, I have some more time for this now. I'm hoping Andrew, Bill and >>> Phillip will join the discussion here (instead of various other >>> threads). >> >>> I'd like to explore this some more before committing to anything. >> >> Here's a thought: >> >> For identifying abilities, it almost seems like instead of hard coding a library >> of this is that relationships, which will always be incomplete and possibly easy >> to fool, an ability might be described as a small (minimal) test that a >> interface can pass. >> Registering an ability relationship could require running >> it against an ability test prior to use. To be a bit clearer, I meant here, that running an ability test-unit could be part of the pre-registration (or other setup) process and needn't be done at a function's or method's call time. >> It doesn't have to be done just prior to use. It could be done at import time, >> or even earlier with the result restored at application run time so it can be a >> one time cost for most applications. >> >> This may make it more open ended and add a level of actual testing that may be >> useful for debugging as well. >> >> Ron From guido at python.org Wed Nov 22 01:06:52 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Nov 2006 16:06:52 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> References: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> Message-ID: On 11/21/06, Nestor wrote: > Do these aproaches allow to do something like the > java.util.RandomAccess interface that is just used as a marker? Zope interfaces certainly do. I don't know about Phillip's counter-proposal. With ABCs it would seem that one has to plan this ahead of time (i.e., at class definition time), which isn't always reasonable. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Nov 22 02:12:58 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Nov 2006 17:12:58 -0800 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: <1d85506f0611211334h1182c73cj6aac9d9b40bef99b@mail.gmail.com> References: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com> <45632E22.2030705@solarsail.hcs.harvard.edu> <45633AB8.9010306@livinglogic.de> <1d85506f0611211029i3969a91cw1ad621d15d4374e7@mail.gmail.com> <45635F16.3060404@livinglogic.de> <1d85506f0611211334h1182c73cj6aac9d9b40bef99b@mail.gmail.com> Message-ID: Someone with commit privs please give this one more look over and submit... On 11/21/06, tomer filiba wrote: > patch updated (hopefully for the last time. it got me all bored :) > > > > On 11/21/06, Brett Cannon wrote: > > > > > > > > On 11/21/06, Walter D?rwald wrote: > > > tomer filiba wrote: > > > > patch updated > > > > > > You're fast! ;) > > > > > > AFAICT now we have the following problem: If > > > Objects/typeobject.c::slot_nb_bool() falls back to > using __len__(), the > > > return value of __len__() must be a bool. > > > > > > Why can't the fallback usage just pass the return value from __len__ to > bool() (forget the C function name) and return that result? It's just like > doing:: > > > > def bool(obj): > > try: > > return obj.__bool__() > > except AttributeError: > > return bool(len(obj)) > > > > -Brett > > > > > > > > > > Servus, > > > Walter > > > > > > > On 11/21/06, *Guido van Rossum* > > > > wrote: > > > > > > > > I think it would set a good example if __bool__ was *required* to > > > > return exactly True or False and no other object, not even int > > > > instances equal to 0 or 1. Surely that can't be much of a burden. > Feel > > > > free to upload an improved patch. > > > > > > > > On 11/21/06, Walter D?rwald > > > > wrote: > > > > > Guido van Rossum wrote: > > > > > > On 11/21/06, Ivan Krsti? < > krstic at solarsail.hcs.harvard.edu > > > > > wrote: > > > > > >> Guido van Rossum wrote: > > > > > >>> Can anyone help out evaluating this patch? If it has to wait > > > > for me > > > > > >>> it's gonna be a looooooong wait... > > > > > >> Looks fine to me functionally, although it seems to > > > > gratuitously retab > > > > > >> some code that used to be aligned with tabstop 8 to a tabstop > of 4. > > > > > > > > > > > > Thanks! I hope you meant "indentation level" instead of > "tabstop". > > > > > > Tomer, can you fix the indentation to be whatever's prevailing > > > > in the > > > > > > file you're editing? > > > > > > > > > > Another question is whether __bool__() should be allowed to > return an > > > > > int (or any other object), or if it *must* be a bool. (The patch > > > > > currently allows ints). What if __bool__() returns subclasses of > int, > > > > > that overwrite __bool__() themself? > > > > > > > > > > Servus, > > > > > Walter > > > > > > > > > > > > > > > > > > > > > > -- > > > > --Guido van Rossum (home page: http://www.python.org/~guido/ ) > > > > > > > > > > > > > > _______________________________________________ > > > Python-3000 mailing list > > > Python-3000 at python.org > > > http://mail.python.org/mailman/listinfo/python-3000 > > > Unsubscribe: > http://mail.python.org/mailman/options/python-3000/brett%40python.org > > > > > > > > > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: > http://mail.python.org/mailman/options/python-3000/guido%40python.org > > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Wed Nov 22 02:20:54 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 21 Nov 2006 20:20:54 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> Message-ID: <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> At 04:06 PM 11/21/2006 -0800, Guido van Rossum wrote: >On 11/21/06, Nestor wrote: > > Do these aproaches allow to do something like the > > java.util.RandomAccess interface that is just used as a marker? > >Zope interfaces certainly do. I don't know about Phillip's >counter-proposal. If we're basing GF's on types, then certainly marker mixins are feasible to *use*. I only contend that mixin markers (and other pure interfaces devoid of execution effect) are unnecessary and shouldn't be encouraged, not that they are or should be impossible. > With ABCs it would seem that one has to plan this >ahead of time (i.e., at class definition time), which isn't always >reasonable. For this reason, I would be against allowing *only* definition-time interface declarations. A major benefit of both adaptation and generic functions is that they allow retrofitting or gluing of "oblivious" code; i.e., code that wasn't originally designed to work with the thing you're gluing it to. Having only interfaces and introspection would also just mean that we are adding Java-like declarations and LBYLisms, without any improvements in expressive power or code clarity. Declaring an interface or adding a mixin should *do* something, instead of simply being a comment in the form of code, that other code is able to read. :) From tjreedy at udel.edu Wed Nov 22 02:36:32 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 21 Nov 2006 20:36:32 -0500 Subject: [Python-3000] __nonzero__ vs. __bool__ References: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com><1d85506f0611210359m7625c01ava3b2530465d4c8a4@mail.gmail.com><45632E22.2030705@solarsail.hcs.harvard.edu><45633AB8.9010306@livinglogic.de><1d85506f0611211029i3969a91cw1ad621d15d4374e7@mail.gmail.com><45635F16.3060404@livinglogic.de> Message-ID: "Brett Cannon" wrote in message news:bbaeab100611211232r53de3c33p915bee7b49dbdf9a at mail.gmail.com... Why can't the fallback usage just pass the return value from __len__ to bool() (forget the C function name) and return that result? It's just like doing:: def bool(obj): try: return obj.__bool__() except AttributeError: return bool(len(obj)) ------------ If an object without __bool__ returned itself as its length, this would be an infinite loop, at least in this Python version. Do we worry about something so crazy? tjr From nestornissen at gmail.com Wed Nov 22 02:38:11 2006 From: nestornissen at gmail.com (Nestor) Date: Tue, 21 Nov 2006 20:38:11 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> References: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> Message-ID: <4b57b0700611211738o3e952f35pcc2ac91dcf1160c3@mail.gmail.com> On 11/21/06, Phillip J. Eby wrote: > At 04:06 PM 11/21/2006 -0800, Guido van Rossum wrote: > >On 11/21/06, Nestor wrote: > > > Do these aproaches allow to do something like the > > > java.util.RandomAccess interface that is just used as a marker? > > > >Zope interfaces certainly do. I don't know about Phillip's > >counter-proposal. > > If we're basing GF's on types, then certainly marker mixins are feasible to > *use*. I only contend that mixin markers (and other pure interfaces devoid > of execution effect) are unnecessary and shouldn't be encouraged, not that > they are or should be impossible. > > > > With ABCs it would seem that one has to plan this > >ahead of time (i.e., at class definition time), which isn't always > >reasonable. > > For this reason, I would be against allowing *only* definition-time > interface declarations. A major benefit of both adaptation and generic > functions is that they allow retrofitting or gluing of "oblivious" code; > i.e., code that wasn't originally designed to work with the thing you're > gluing it to. > > Having only interfaces and introspection would also just mean that we are > adding Java-like declarations and LBYLisms, without any improvements in > expressive power or code clarity. Declaring an interface or adding a mixin > should *do* something, instead of simply being a comment in the form of > code, that other code is able to read. :) > > I agree that interfaces instead of functions adds another level of indirection for a mechanism that most of the time is used to signify the presence of one or two methods. If the functionality of dispatching on characteristics other than methods is worth adding to Python I don't feel experienced enough to answer. From tdelaney at avaya.com Wed Nov 22 03:05:26 2006 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Wed, 22 Nov 2006 13:05:26 +1100 Subject: [Python-3000] __nonzero__ vs. __bool__ Message-ID: <2773CAC687FD5F4689F526998C7E4E5FF1EBD1@au3010avexu1.global.avaya.com> Terry Reedy wrote: > If an object without __bool__ returned itself as its length, this > would be > an infinite loop, at least in this Python version. Do we worry about > something so crazy? Doesn't len() have a requirement that __len__ return an integer? If so, perhaps it would be better if this were: def bool(obj): try: return obj.__bool__() except AttributeError: return len(obj) > 0 If I'm wrong (don't have the source available right now) then perhaps len() *should* have this requirement. Tim Delaney From guido at python.org Wed Nov 22 03:07:56 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Nov 2006 18:07:56 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> Message-ID: On 11/21/06, Phillip J. Eby wrote: > At 11:16 AM 11/21/2006 -0800, Guido van Rossum wrote: > >Phillip then argues that he doesn't want to encourage introspectable > >interfaces. I think others have use cases for those though. > > Examples? I'll wait for others to speak up, but I'd think that automatically generating appropriate docs for interfaces qualifies. Python in general has a very proactive approach to introspection -- everythign is introspectable unless it would violate safety. > > It seems > >that Phillip's approach only hanges well together if everybody totally > >adopts the generic functions religion; I think that's an unlikely > >(since too radical) change of course for Python 3.0. > > Is it really radical? Given some of your examples, I'd say so. Python users are used to providing functionality in classes by defining special methods. Java users (and users of many other languages) are used to interfaces that describe contracts (even if they *appear* only to be syntactic) and ways to claim that a class fulfills the contract, as part of the class definition syntax. Optional argument type declarations in Python would be happy to use such a mechanism too. These are both quite differently than binding a class and a specific operation (e.g. flatten) through some call to something named addmethod(). While the approaches are probably provably equivalent in expressive power, IMO doing common things is a lot more intuitive using interfaces. (Always accepting that "intuitive" isn't -- it just means "familiar from a prevous context".) While we're discussing familiarity, if you want to continue the discussion, may I remind you that most people (including myself) aren't familiar with RuleDispatch, and examples like addmethod(addmethod, adder_for_my_gf_type, my_gf_type) or @addmethod.when_object(iter) are downright off-putting? What on earth is "when_object(iter)" supposed to mean? HCI indeed! :-) > Consider the fact that single-dispatch generic functions are ubiquitous in > the language and the stdlib, and have been pretty much forever. They just > vary in how they're implemented, and there's no One Obvious Way for > developers to implement new ones, nor is there a standard way to add a > method to an existing one. Actually the standard way is to have a __special__ method and define that in your class. We've recently discussed doing that for dir(), for example (on python-dev, as it can safely go into 2.6). > That's because some generic functions use if-isinstance checks (bad), while > others use custom __special__ methods (not bad, but not great), registries > (good), adaptation (okay), or generic function libraries. __special__ methods are great for many purposes because they are well understood. Registries are mostly good when there's a need for an independent third party to add an ability; IMO this isn't as common in most places as you seem to think. > So, all I am proposing is that we: > > * provide a standard GF implementation for the simple cases > * ...that is extensible to allow others to handle the more complex > cases, by having a generic API for manipulating generic functions I'm all for this. I think it would be more successful if it had support for interfaces. > So that the "one obvious way" to create new generics is to use the standard > GF implementation unless you need something else. I'd rather not try to teach people that they can't define a __len__ method any more but must update half a dozen generic functions in order to create a new sequence type. Overriding __special__ methods works fine for the most part. The one exception I've seen is binary operations where some types are too aggressive in rejecting unacceptable 'other' operands, raising TypeError where they should return NotImplemented. I believe datetime does this, but I also believe this is mostly to work around issues with comparisons in 2.x that have been fixed in 3.0 (and datetime has been taught to be less rude). Sure, I see a use case for defining an operation that the class author(s) did *not* foresee, but that's the exception rather than the rule. > And, there would be a > standard way to add new methods to existing generic functions, perhaps > using an addmethod() builtin or decorator, and it could be designed so that > e.g.: > > addmethod(iter, somefunc, sometype) I don't actually understand what you want this example to mean. What is 'iter' supposed to be? A generic function? > would actually work by doing 'sometype.__iter__ = somefunc' under the > hood. How would addmethod know this? > This allows us not to have to change any builtin generics that are > based on special method names. In other words, the mechanism isn't > radical, nor is the idea of having generics. It's just an improvement in > ease-of-use: an HCI change, not a comp-sci change! You can repeat that as many times as you want but that doesn't make it so. At least one poster has already remarked that the head-exploding capacity of generic functions is greater than that of metaclasses. I think that's a gross exaggeration, but calling it "simple" and "easy to use" is an exaggeration too. It's quite deep (otherwise we'd seen it in Java already ;-). > And, with this relatively simple mechanism, all the fancier forms of > generic functions (or interfaces and adapters) can be implemented via user > libraries. (More on how, below.) > > > > It also doesn't > >seem to work for abilities that are tied to an instance instead of to > >a class as Zope allows (see below). > > Actually, it still allows for that, as long as the base generic function > system is extensible. How should it be extensible? Please explain this without making any use of examples from RuleDispatch. > For example, RuleDispatch generic functions can > choose implementations based on conditions such as whether an object has a > particular attribute. Zope could easily add a generic function type that > uses their instance-based interface stuff to do the same thing. What's a generic function type? How does one create one? > I am merely proposing that Python not provide this sort of > introspection-oriented stuff in the core or stdlib, not that nobody should > be *allowed* to have it. And I think introspection is too important to leave out of the core. People will want to introspect and find a way to do it by referring to implementation details -- just like there's currently code in inspect.py to reconstruct a function's signature from random attributes of the code object, including flag bits. > Indeed, I explicitly want the Python generic function API (e.g. addmethod, > hasmethod, or whatever we call them) to *itself* be generic, so that users > can create their own types that work with the Python-provided operations > for generic function manipulation. That is, I should be able to call: > > addmethod(addmethod, adder_for_my_gf_type, my_gf_type) OK, I take it back. My brain just exploded. it *is* worse than metaclasses. Please find enclosed the bits of brain that I scraped of my monitors. :-) > So that others can then call: > > # this calls adder_for_my_gf_type under the hood: > addmethod(some_gf, some_method, some_cls) > > where 'some_gf' is an instance of 'my_gf_type'. This allows us not to keep > the core or stdlib implementation of generic functions quite simple to > handle the 80% (single dispatch) or 90% (concrete type-based multiple > dispatch) with ease. The final 10% (predicate dispatch, fancy interfaces, > etc.) can then be done by outside libraries, since the use cases in that > final 10% are more likely to vary than the base 80-90%. > > Using the 'simplegeneric' library from PyPI (easy_install simplegeneric): Hey, another example referring to technology so advanced that for me and other readers it is indistinguishable from magic. ;-) > from simplegeneric import generic > > @generic > def addmethod(gf, method, cls): > """Add a method to a generic function""" > raise TypeError("Unknown generic function type", gf) > > @addmethod.when_object(iter) > def add_to_iter(gf, method, cls): > # XXX should have more error checking here, e.g. > # check for a local __iter__ first > cls.__iter__ = method > > # ... similar declarations for other builtin generics > > @addmethod.when_type(FunctionType) > def add_to_function(gf, method, cls): > if hasattr(gf, 'when_type'): > gf.when_type(cls)(method) > else: > raise TypeError("Not a generic function", gf) > > And there you are: an extensible way to add new extensible function types, > while using the same API for all of them. Poof. You disappear in a cloud of orange smoke. > (If you need multiple or predicate dispatch or method combining, it's easy > to have decorators that apply to the method itself, so that the same > three-argument addmethod() can still be used.) > > > >3. API design -- how do we spell the various concepts? E.g. > >has_ability(x, A) asks whether object x has the ability A; > >provides_ability(C, A) asks whether class C provides the ability A to > >its instances. We could state that provides_ability(C, A) and > >isinstance(x, C) implies has_ability(x, A). > > Even if we explicitly have some type of "ability" object, I'd like them to > be defined in terms of generic functions, so that I could effectively say > something like: > > sequence_like = (iter & len) > > (or perhaps spelled out in some other fashion) to create an "ability" > representing iter-ability and len-ability. That's an interesting thought but it works with or without generic functions -- it's just "ability algebra". I think I've played with "type algebra" using similar ideas in a blog entry once. > One of the biggest conceptual/modeling issues with Zope-style interfaces is > that they don't allow this kind of fine-grained protocol combination. See, > for example, zope.interface's long history of trying to nail down various > Python protocols through elaborate interface inheritance > hierarchies. Defining interfaces strictly in terms of operations > eliminates this issue. Not really, IMO. The underlying problem is that standard type hierarchy defies capturing it in interfaces *BECAUSE INTERFACES DIDN'T EXIST WHEN THEY WERE CREATED*. > >- How does this interact with generic functions? > > The advanced abilities (per-instance/on-the-fly and multi-operation tests) > will likely affect performance and/or simplicity of the default > implementation. A purely type-based system can be implemented efficiently, > because declaring an interface can simultaneously add a concrete type > registration for all affected generics. (And registering an > interface-linked method in a generic function can pull in all the concrete > classes.) This seems to argue *for* interfaces? > Per-instance tests, however, increase code complexity. Generally speaking, > RuleDispatch does per-instance tests after class tests, but the way it gets > reasonable performance is by building decision trees beforehand to manage > the possible tests and avoid overlap. If it actually had to call > individual methods, or call a method more than once (ob.has_ability(Foo), > ob.has_ability(Bar), etc.) it would be considerably slower to select an > option. That would make more sense as Foo.implemented_by(ob), right? > Also -- and this is the real kicker -- what do you do if more than one > ability applies? Now we have to have precedence rules for what's more > specific. The advantage of defining an ability as a set of one or more > applicable generic functions is that precedence is comparatively > straightforward: supersets take precedence over subsets, and overlaps are > ambiguous. You also have some possibility of being able to implement this > by registration-time checks, without needing to build a dispatch tree at all. Couldn't ability inheritance serve the same purpose? > I don't know of anybody who really uses per-instance interface declarations > except for Zope. I used them for a little while with PEAK, and decided > they were the wrong idea for me; it made more sense to adapt to an > interface that provides the metadata, or to use a RuleDispatch function > that just directly introspected for whatever was needed. I can't think of > anything in the core or stdlib that would even need to go that far. > > Again, my argument is that anything other than isinstance-based single and > multiple-dispatch is too framework-dependent to be part of the > language. Simple things should be simple, complex things should be possible. Sure, I can dig that. I'd like to have separate APIs for querying instances and classes, but I don't mind if the built-in facilities only let you define abilities for classes. As long as it's possible to add an ability to a class dynamically (to deal with the problem of pre-existing 3rd party classes that implement de-facto interfaces but don't declare them). > >Answering these and similar questions probably requires a > >standardization committed. (Any volunteers?) > > Note that Zope has already tried for many years to do exactly this, and > there is no indication that they have actually succeeded. I tend to blame that on their inability to change the language and the standard library though. Py3k has that option, and I'm not afraid to exercise it if it would help. > When I first > mentioned PyProtocols on Python-Dev many years ago, Samuele Pedroni argued > that individual operations (and groups thereof) should be the currency of > interfaces, and after playing with them a bit, I agreed. PyProtocols > defines such interfaces as e.g.: > > protocolForType(file, ['read', 'close']) > > to mean "file-like object with 'read' and 'close' methods". If I define a > class as supporting that protocol, and somebody wants to adapt to > 'protocolForType(file, ["read"])', then my type will work. > > See > http://peak.telecommunity.com/protocol_ref/protocols-generated-type.html > for a more complete explanation, including how to get the system to > actually check for the method names (so as not to require explicit > declarations). > > I think Samuele's idea (i.e., this) works a lot better than trying to > define a rigid interface hierarchy for builtin-protocols, although I'd just > as soon be able to do something like: > > file.read & file.close > > rather than having to use strings. But all this assumes that we are > providing a builtin way to spell abilities and introspect them, which I > still think is overkill for what Python actually needs to provide as a base. Well, I think it would be useful and I'm sure it would be popular. I'm asking for a committee to lay this to rest by defining the one true hierarchy for containers. Other languages have succeeded in doing so (even Java). > In short, I think it's adding interface introspection that's the radical > move, where generic functions are just a bit of polish that brings a bit > more order to what Python already has. In contrast, interfaces are a > foreign religion imported from other languages, while the roots of the > generic function faith (len, iter, etc.) have already been in place since > the dawn of time! :) Python has a long tradition of borrowing ideas from other languages, and mostly very successful. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From janssen at parc.com Wed Nov 22 04:53:29 2006 From: janssen at parc.com (Bill Janssen) Date: Tue, 21 Nov 2006 19:53:29 PST Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: Message-ID: <06Nov21.195340pst."58648"@synergy1.parc.xerox.com> > Bill Janssen seems to be in favor of just using ABCs. Pro: less new > infrastructure. Con: hard to add a new ABC on the fly to an existing > 3rd party class (mucking with __bases__ is too much of a hack to > seriously propose). Don't get me wrong, I think having optional interface-based method dispatch would be an additional goodness, if only for the documentation aspect of it. But yes, I think adding ABC to define the interfaces, and using inheritance for aggregation, is the way to go about it. I think an additional benefit would be an increased emphasis on mixins as a simple way to add functionality. About that con -- perhaps a new bit of machinery could be added to muck with __bases__ in a principled fashion? But isn't this a bit like the argument about anonymous code blocks? If someone wants to add a new ABC to an existing third-party class, can't they just write class NewClass(ExistingThirdParty, NewABC): pass and use an instance of that new class? Then we get into the argument about factories that only produce instances of ExistingThirdParty, and so on... > 2. Do we need interfaces if we have generic functions? I don't mind having both, but generic functions (interface-based method dispatch) is really only one aspect of the issue. Bill From janssen at parc.com Wed Nov 22 04:55:10 2006 From: janssen at parc.com (Bill Janssen) Date: Tue, 21 Nov 2006 19:55:10 PST Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <435DF58A933BA74397B42CDEB8145A8606E5D65D@ex9.hostedexchange.local> References: <435DF58A933BA74397B42CDEB8145A8606E5D65D@ex9.hostedexchange.local> Message-ID: <06Nov21.195515pst."58648"@synergy1.parc.xerox.com> > If we have to have interfaces, I think we'd be far better off > implementing the spec as-is first, and worry about changing the spec > later (even if if only seems to be an "implied spec" at the moment). Sure. But we could factor the present spec into pieces. Bill From janssen at parc.com Wed Nov 22 04:58:33 2006 From: janssen at parc.com (Bill Janssen) Date: Tue, 21 Nov 2006 19:58:33 PST Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061121174806.01f19c50@sparrow.telecommunity.com> References: <91ad5bf80611211418o76941e13l6864e97e2df23154@mail.gmail.com> <91ad5bf80611211418o76941e13l6864e97e2df23154@mail.gmail.com> <5.1.1.6.0.20061121174806.01f19c50@sparrow.telecommunity.com> Message-ID: <06Nov21.195840pst."58648"@synergy1.parc.xerox.com> > I just want a minimal > core that blesses what the language and stdlib *already* do (special > methods and single-dispatch __mro__ lookups) Well, this *is* Py3K. We can fix things that the language doesn't do but should. Bill From janssen at parc.com Wed Nov 22 05:06:23 2006 From: janssen at parc.com (Bill Janssen) Date: Tue, 21 Nov 2006 20:06:23 PST Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> Message-ID: <06Nov21.200625pst."58648"@synergy1.parc.xerox.com> > I'm asking for a committee to lay this to rest by defining the one > true hierarchy for containers. Other languages have succeeded in doing > so (even Java). Well, I don't think Java has really done this yet. But it's a good idea. Someone should try to re-describe the standard set of Python built-in types (list, tuple, file, string, dict, etc.) as a set of ABCs (not sure they have to be all that abstract, either), using inheritance to describe abilities. If it can't be done, that would be a strike against ABC, I'd think :-). Bill From pje at telecommunity.com Wed Nov 22 05:07:13 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 21 Nov 2006 23:07:13 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> Before delving into any specific points, let me attempt to explain my assumptions, briefly, because I suspect that you and I do not mean the same thing by the words "generic function", and thus you perceive me to be proposing to change things that I am in fact proposing we *not* change. To me, Python already has generic functions, including len(), iter(), getattr(), hasattr(), operator.getitem(), etc. etc. Anything based on __special__ methods is in my view, *already* a "generic function", because it can take various types, and new types can be created that have specialized behavior. The implementation of 'len()' is *open to extension for new types*, and therefore it is "generic". That being the case, my position is that we should provide language support for defining other kinds of generic functions that are on an equal footing with the builtins in terms of language support. __special__-based generics are adequate for quite a lot of purposes, registry-based ones are useful for many others, and advanced special cases have their uses as well. This doesn't seem particularly radical to me, no more so than say, type/class unification. And it's far less ambitious than type/class unification was, from an implementation point of view, because I don't propose to change the implementation of anything that already exists! *Nothing* in this position relies upon RuleDispatch or even the simple multiple-dispatch generic function implementation you prototyped earlier this year. Nor is there anything head-exploding about __special__ methods, as far as I am aware. :) So I'm at a loss to see any radicalness here. What I have said that seems to be controversial, is the idea that we could do without interfaces. However, we actually *already have interfaces based on generics*. Consider 'iter()', for example, which can be viewed as adapting an object to the "iteration interface" and returning an object supporting iteration. This is an *example* of what I am proposing: i.e., using generic operations to define interfaces. And again, it's already supported by Python! So how is this radical? Everything I've stated up to this point in this message, is *exactly what we have today*! The only things I propose to *add*, would be: 1. functions like 'addmethod()' and 'hasmethod()', themselves to be generics in the style of iter() or len(). Whether they are implemented using registries or __special__ methods is of no consequence. 2. a 'defop' syntax as shorthand for 'addmethod()', such that e.g.: import operator class Foo: defop iter(self): ... defop len(self): ... defop operator.getitem(self, key): ... would produce exactly the same results as the same code using __special__ methods, except that it could be expanded to add methods for arbitrary *new* operations not defined by the language. With these two things, Python would have enough foundation to allow any number of interface, adaptation, or generic function-based frameworks to run on top of it. And, instead of each framework having to teach people new __special__ methods or provide different decorators, they could all simply say, "here are the generic functions or interfaces we use". I submit that this is more than sufficient to cover the 80-90% of common use cases that don't need abstract interfaces at all, or which can simply use a generic function to stand in as an interface, the way 'iter' can stand in for the "iteration interface". And, in the same way that 'def __iter__(self): return self' means "I implement iteration", doing "defop IFoo(self): return self" would mean, "I implement IFoo". (Note, by the way, that common usage of interface adaptation in Python is already to call IFoo(anObject), so this is *also* no change to current usage!) So, from my POV, this is actually a very modest proposal thus far. Now I'll answer those specific points you brought up that aren't covered in the above. At 06:07 PM 11/21/2006 -0800, Guido van Rossum wrote: >are downright off-putting? What on earth is "when_object(iter)" >supposed to mean? HCI indeed! :-) Simplegeneric allows you to define methods for either types or instances. The doc is at http://cheeseshop.python.org/pypi/simplegeneric > > So, all I am proposing is that we: > > > > * provide a standard GF implementation for the simple cases > > * ...that is extensible to allow others to handle the more complex > > cases, by having a generic API for manipulating generic functions > >I'm all for this. I think it would be more successful if it had >support for interfaces. Sure. See my explanation above for why I think that this *does* support interfaces. > > So that the "one obvious way" to create new generics is to use the standard > > GF implementation unless you need something else. > >I'd rather not try to teach people that they can't define a __len__ >method any more but must update half a dozen generic functions in >order to create a new sequence type. Implementing __len__ (or invoking 'addmethod(len,...)', or 'defop len(self):...') will still be sufficient to make a type len-able. So code that uses len() will "just work" in that case. Same thing for operator.getitem() or its [] shorthand syntax. >Overriding __special__ methods works fine for the most part. We can certainly keep the implementation of built-in generics based on __special__ methods. My example code was intended to demonstrate that it's possible even in today's Python to have a uniform interface to defining such methods -- not that it's necessarily a desirable way to do it, given the absence of dedicated syntax like 'defop' for operator (operation?) overloading. Even *with* a 'defop' syntax, I would still not propose we eliminate __special__ methods, as they are very fast when implemented as C slots. I would just say that, like 'apply()', they would no longer be the optimum way to do something for which there is a syntactical shortcut. Note that __special__ methods can be misspelled, and the resulting error can be hard to find. Misspell a 'defop', and the failure is immediate. (Of course, you could still defop the *wrong* function, but it's still an improvement.) >Sure, I see a use case for defining an operation that the class >author(s) did *not* foresee, but that's the exception rather than the >rule. Right, at least in the sense that team development is exceptional for Python. In "enterprisey" and other team development scenarios (e.g. Zope, Chandler, etc.), it's valuable to have separation between domain model code and presentation code. IIRC, the driving use cases for adaptation in Zope 3 were to achieve this separation. In other words, even if you *can* define the methods, that doesn't mean it's a good idea to, if you are trying to maximize reuse. But I freely admit that Zope and PEAK at least are exceptional situations: Zope Corp. (and my group at Verio during the time I initially created PEAK), were both effectively consulting organizations gaining cost-effectiveness through reuse of framework code. This is not exactly the most common use of Python... unless of course you're in an "enterprise" shop or consulting organization. > > addmethod(iter, somefunc, sometype) > >I don't actually understand what you want this example to mean. What >is 'iter' supposed to be? A generic function? The iter builtin, as an example of the uniform addmethod() being applicable to any generic function, including existing builtin ones. Although I guess you already know that from what you read beyond that point. > > would actually work by doing 'sometype.__iter__ = somefunc' under the > > hood. > >How would addmethod know this? By addmethod itself being a generic function. >At least one poster has already remarked that the head-exploding >capacity of generic functions is greater than that of metaclasses. I >think that's a gross exaggeration, but calling it "simple" and "easy >to use" is an exaggeration too. It's quite deep (otherwise we'd seen >it in Java already ;-). We've already seen it in Python, actually. Generic operations of this nature are at the very soul of the language; we simply provide syntactic shorthand for many of them, and builtins for the rest. So, adding an explicit GF API seems more like type/class unification to me (or adding the callable __class__ hook), than a sea change in the nature of Python itself. That is, a nice improvement in uniformity and user-extensibility, rather than any actual change. >How should it be extensible? Please explain this without making any >use of examples from RuleDispatch. Implement the API using generic functions. If you have an addmethod(), make it a generic. If you have a 'hasmethod()', make it generic. (Please note that 'generic' here refers to the extensibility of the function, not to a requirement that it be implemented via a registry! I don't care if we have to give generic functions __addmethod__ and __hasmethod__ specials. The point is merely that there should be *some* way to extend the core API.) >What's a generic function type? How does one create one? Implement a callable object that can be passed to addmethod(), hasmethod(), etc. >OK, I take it back. My brain just exploded. it *is* worse than >metaclasses. Please find enclosed the bits of brain that I scraped of >my monitors. :-) Ironically, it's *you* who gave me this idea, although I don't think you realized it at the time. It was back when you were prototyping the multiple-dispatch implementation using a tuple of types -- something you said made me realize that we could have a generic API for manipulating generics, and thus allow hypothetical Zope and PEAK generics to live alongside Python-provided generics as equal citizens. I then went away and made a proof of concept, before coming back to explode your head. >Poof. You disappear in a cloud of orange smoke. I guess now it's my turn not to understand what something means. :) I was merely trying to show that my idea is trivially implementable. simplegeneric is only about 100 lines of Python added to what I wrote. Throw in a 'defop' syntax that calls addmethod(), and we're done. What I suppose is not obvious to anyone else from what I've written, is that this would: 1. allow competing type systems to coexist and even interoperate with a minimum of fuss 2. provide a *definition-time checkable* alternative to __special__ method names 3. ...while still allowing __specials__ to be used for fast lookup under the hood 4. allow for a "blessed" interface mechanism to either be chosen based on actual uses, OR 5. avoid the need for having a "blessed" interface mechanism at all, if GF introspection or adaptation suffices >That's an interesting thought but it works with or without generic >functions -- it's just "ability algebra". I think I've played with >"type algebra" using similar ideas in a blog entry once. Sure -- the difference is that I'm suggesting using the operation objects themselves as the basic unit of that algebra. >Not really, IMO. The underlying problem is that standard type >hierarchy defies capturing it in interfaces *BECAUSE INTERFACES DIDN'T >EXIST WHEN THEY WERE CREATED*. That's a reasonable hypothesis. You might find it works out okay for things with few operations (like sequences and "mappings") but not so well for things with lots of utility methods ("list-like", "file-like", "dict-like" etc.). I guess we'll see how it works out in practice. > > The advanced abilities (per-instance/on-the-fly and multi-operation tests) > > will likely affect performance and/or simplicity of the default > > implementation. A purely type-based system can be implemented efficiently, > > because declaring an interface can simultaneously add a concrete type > > registration for all affected generics. (And registering an > > interface-linked method in a generic function can pull in all the concrete > > classes.) > >This seems to argue *for* interfaces? I was pointing out that if you really need interfaces, you can have them, without needing to create a distinct interface object. As you yourself pointed out in your blog, an "interface" can just be a generic function that returns an object supporting that interface, for a given input object. In this sense, "iter" is an interface, because when you call it on an object, you get back an object that supports the "iter" interface. Thus, I don't see much need to create a special notion of interfaces as a first-class citizen. In the common case, a generic "adapting" function (like iter) is sufficient, even if the "interface" includes multiple methods (like __iter__ and next()). > > Per-instance tests, however, increase code complexity. Generally speaking, > > RuleDispatch does per-instance tests after class tests, but the way it gets > > reasonable performance is by building decision trees beforehand to manage > > the possible tests and avoid overlap. If it actually had to call > > individual methods, or call a method more than once (ob.has_ability(Foo), > > ob.has_ability(Bar), etc.) it would be considerably slower to select an > > option. > >That would make more sense as Foo.implemented_by(ob), right? Um, I guess. I was emphasizing the dynamic/per-instance aspect of the issue. Whichever way you do it, unless you have a well-defined implication or inheritance hierarchy between interfaces, you have the problem of ambiguity between opaque interfaces. > > Also -- and this is the real kicker -- what do you do if more than one > > ability applies? Now we have to have precedence rules for what's more > > specific. The advantage of defining an ability as a set of one or more > > applicable generic functions is that precedence is comparatively > > straightforward: supersets take precedence over subsets, and overlaps are > > ambiguous. You also have some possibility of being able to implement this > > by registration-time checks, without needing to build a dispatch tree > at all. > >Couldn't ability inheritance serve the same purpose? Sure. The above was part of my argument against *dynamically-determined* abilities as a core feature. > > In short, I think it's adding interface introspection that's the radical > > move, where generic functions are just a bit of polish that brings a bit > > more order to what Python already has. In contrast, interfaces are a > > foreign religion imported from other languages, while the roots of the > > generic function faith (len, iter, etc.) have already been in place since > > the dawn of time! :) > >Python has a long tradition of borrowing ideas from other languages, >and mostly very successful. Of course -- and generic functions have a successful tradition in many other languages too, from Common Lisp to Haskell, that IIUC goes back much further than "interfaces" as such do. I was pointing out that you had *already* borrowed this tradition many years ago, whether you realized it or not. You just "hid" it by using __special__ method names instead of registries. ;-) From guido at python.org Wed Nov 22 05:57:53 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Nov 2006 20:57:53 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> Message-ID: Phillip, please shorten your posts. You're hogging all the bandwidth I have for thinking about this. Please! On 11/21/06, Phillip J. Eby wrote: > The only things I propose to *add*, would be: > > 1. functions like 'addmethod()' and 'hasmethod()', themselves to be > generics in the style of iter() or len(). Whether they are implemented > using registries or __special__ methods is of no consequence. Please write a spec for these rather than assuming the reader already knows what they are supposed to do. If you could present simplified versions of these that were *not* generic themselves first that would be a huge boon -- the self-referentiality is what's head-exploding. > 2. a 'defop' syntax as shorthand for 'addmethod()', such that e.g.: > > import operator > > class Foo: > defop iter(self): > ... > defop len(self): > ... > defop operator.getitem(self, key): > ... > > would produce exactly the same results as the same code using __special__ > methods, New syntax is radical. Especialy when it's such superficial syntactic sugar. > except that it could be expanded to add methods for arbitrary > *new* operations not defined by the language. I fail to see how we need new syntax to define new __special__ operations. There are plenty of new __special__ methods being introduced (e.g. by Zope, or by some library modules, like pickle, at least originally) without new syntax. Please spec this without new syntax first. The lack of new syntax is not what blocks the proposal's acceptance. > With these two things, Python would have enough foundation to allow any > number of interface, adaptation, or generic function-based frameworks to > run on top of it. And, instead of each framework having to teach people > new __special__ methods or provide different decorators, they could all > simply say, "here are the generic functions or interfaces we use". Again, that sounds mostly like a very thin layer of syntactic sugar. Let's pretend we don't need that sugar -- at least not in the first round. > I submit that this is more than sufficient to cover the 80-90% of common > use cases that don't need abstract interfaces at all, or which can simply > use a generic function to stand in as an interface, the way 'iter' can > stand in for the "iteration interface". I'm still at a conceptual loss about this particular example. I fail to see how 'iter' means 'iteration interface'. iter is a callable that can do two very different things based on how it's called. Pretending that it means "an object that has an __iter__ method" is too far a stretch for me, since there is no way to find out that that is its meaning except by knowing it. This is the problem I have with non-introspectable interfaces. I don't understand why you find that an advantage. > And, in the same way that 'def __iter__(self): return self' means "I > implement iteration", doing "defop IFoo(self): return self" would mean, "I > implement IFoo". Why the 'return self'? Did you mean '...'? Or did you really mean that the 'return self' is part of the statement "I implement XXX"? > (Note, by the way, that common usage of interface > adaptation in Python is already to call IFoo(anObject), so this is *also* > no change to current usage!) You're taking some liberties by calling that current usage. What does "interface adaptation" even mean? It's not part of the current Python vocabulary. > So, from my POV, this is actually a very modest proposal thus far. > > Now I'll answer those specific points you brought up that aren't covered in > the above. Maybe I'll have time to read and respond to those tomorrow. > > At 06:07 PM 11/21/2006 -0800, Guido van Rossum wrote: > >are downright off-putting? What on earth is "when_object(iter)" > >supposed to mean? HCI indeed! :-) > > Simplegeneric allows you to define methods for either types or > instances. The doc is at http://cheeseshop.python.org/pypi/simplegeneric > > > > > So, all I am proposing is that we: > > > > > > * provide a standard GF implementation for the simple cases > > > * ...that is extensible to allow others to handle the more complex > > > cases, by having a generic API for manipulating generic functions > > > >I'm all for this. I think it would be more successful if it had > >support for interfaces. > > Sure. See my explanation above for why I think that this *does* support > interfaces. > > > > > So that the "one obvious way" to create new generics is to use the standard > > > GF implementation unless you need something else. > > > >I'd rather not try to teach people that they can't define a __len__ > >method any more but must update half a dozen generic functions in > >order to create a new sequence type. > > Implementing __len__ (or invoking 'addmethod(len,...)', or 'defop > len(self):...') will still be sufficient to make a type len-able. So code > that uses len() will "just work" in that case. Same thing for > operator.getitem() or its [] shorthand syntax. > > > >Overriding __special__ methods works fine for the most part. > > We can certainly keep the implementation of built-in generics based on > __special__ methods. My example code was intended to demonstrate that it's > possible even in today's Python to have a uniform interface to defining > such methods -- not that it's necessarily a desirable way to do it, given > the absence of dedicated syntax like 'defop' for operator (operation?) > overloading. > > Even *with* a 'defop' syntax, I would still not propose we eliminate > __special__ methods, as they are very fast when implemented as C slots. I > would just say that, like 'apply()', they would no longer be the optimum > way to do something for which there is a syntactical shortcut. Note that > __special__ methods can be misspelled, and the resulting error can be hard > to find. Misspell a 'defop', and the failure is immediate. (Of course, > you could still defop the *wrong* function, but it's still an improvement.) > > > >Sure, I see a use case for defining an operation that the class > >author(s) did *not* foresee, but that's the exception rather than the > >rule. > > Right, at least in the sense that team development is exceptional for > Python. In "enterprisey" and other team development scenarios (e.g. Zope, > Chandler, etc.), it's valuable to have separation between domain model code > and presentation code. IIRC, the driving use cases for adaptation in Zope > 3 were to achieve this separation. > > In other words, even if you *can* define the methods, that doesn't mean > it's a good idea to, if you are trying to maximize reuse. But I freely > admit that Zope and PEAK at least are exceptional situations: Zope Corp. > (and my group at Verio during the time I initially created PEAK), were both > effectively consulting organizations gaining cost-effectiveness through > reuse of framework code. This is not exactly the most common use of > Python... unless of course you're in an "enterprise" shop or consulting > organization. > > > > > addmethod(iter, somefunc, sometype) > > > >I don't actually understand what you want this example to mean. What > >is 'iter' supposed to be? A generic function? > > The iter builtin, as an example of the uniform addmethod() being applicable > to any generic function, including existing builtin ones. Although I guess > you already know that from what you read beyond that point. > > > > > would actually work by doing 'sometype.__iter__ = somefunc' under the > > > hood. > > > >How would addmethod know this? > > By addmethod itself being a generic function. > > > >At least one poster has already remarked that the head-exploding > >capacity of generic functions is greater than that of metaclasses. I > >think that's a gross exaggeration, but calling it "simple" and "easy > >to use" is an exaggeration too. It's quite deep (otherwise we'd seen > >it in Java already ;-). > > We've already seen it in Python, actually. Generic operations of this > nature are at the very soul of the language; we simply provide syntactic > shorthand for many of them, and builtins for the rest. > > So, adding an explicit GF API seems more like type/class unification to me > (or adding the callable __class__ hook), than a sea change in the nature of > Python itself. That is, a nice improvement in uniformity and > user-extensibility, rather than any actual change. > > > >How should it be extensible? Please explain this without making any > >use of examples from RuleDispatch. > > Implement the API using generic functions. If you have an addmethod(), > make it a generic. If you have a 'hasmethod()', make it generic. > > (Please note that 'generic' here refers to the extensibility of the > function, not to a requirement that it be implemented via a registry! I > don't care if we have to give generic functions __addmethod__ and > __hasmethod__ specials. The point is merely that there should be *some* > way to extend the core API.) > > > >What's a generic function type? How does one create one? > > Implement a callable object that can be passed to addmethod(), hasmethod(), > etc. > > > >OK, I take it back. My brain just exploded. it *is* worse than > >metaclasses. Please find enclosed the bits of brain that I scraped of > >my monitors. :-) > > Ironically, it's *you* who gave me this idea, although I don't think you > realized it at the time. It was back when you were prototyping the > multiple-dispatch implementation using a tuple of types -- something you > said made me realize that we could have a generic API for manipulating > generics, and thus allow hypothetical Zope and PEAK generics to live > alongside Python-provided generics as equal citizens. I then went away and > made a proof of concept, before coming back to explode your head. > > > >Poof. You disappear in a cloud of orange smoke. > > I guess now it's my turn not to understand what something means. :) > > I was merely trying to show that my idea is trivially > implementable. simplegeneric is only about 100 lines of Python added to > what I wrote. Throw in a 'defop' syntax that calls addmethod(), and we're > done. > > What I suppose is not obvious to anyone else from what I've written, is > that this would: > > 1. allow competing type systems to coexist and even interoperate with a > minimum of fuss > 2. provide a *definition-time checkable* alternative to __special__ method > names > 3. ...while still allowing __specials__ to be used for fast lookup under > the hood > 4. allow for a "blessed" interface mechanism to either be chosen based on > actual uses, OR > 5. avoid the need for having a "blessed" interface mechanism at all, if GF > introspection or adaptation suffices > > > >That's an interesting thought but it works with or without generic > >functions -- it's just "ability algebra". I think I've played with > >"type algebra" using similar ideas in a blog entry once. > > Sure -- the difference is that I'm suggesting using the operation objects > themselves as the basic unit of that algebra. > > >Not really, IMO. The underlying problem is that standard type > >hierarchy defies capturing it in interfaces *BECAUSE INTERFACES DIDN'T > >EXIST WHEN THEY WERE CREATED*. > > That's a reasonable hypothesis. You might find it works out okay for > things with few operations (like sequences and "mappings") but not so well > for things with lots of utility methods ("list-like", "file-like", > "dict-like" etc.). I guess we'll see how it works out in practice. > > > > > The advanced abilities (per-instance/on-the-fly and multi-operation tests) > > > will likely affect performance and/or simplicity of the default > > > implementation. A purely type-based system can be implemented efficiently, > > > because declaring an interface can simultaneously add a concrete type > > > registration for all affected generics. (And registering an > > > interface-linked method in a generic function can pull in all the concrete > > > classes.) > > > >This seems to argue *for* interfaces? > > I was pointing out that if you really need interfaces, you can have them, > without needing to create a distinct interface object. As you yourself > pointed out in your blog, an "interface" can just be a generic function > that returns an object supporting that interface, for a given input > object. In this sense, "iter" is an interface, because when you call it on > an object, you get back an object that supports the "iter" interface. > > Thus, I don't see much need to create a special notion of interfaces as a > first-class citizen. In the common case, a generic "adapting" function > (like iter) is sufficient, even if the "interface" includes multiple > methods (like __iter__ and next()). > > > > > Per-instance tests, however, increase code complexity. Generally speaking, > > > RuleDispatch does per-instance tests after class tests, but the way it gets > > > reasonable performance is by building decision trees beforehand to manage > > > the possible tests and avoid overlap. If it actually had to call > > > individual methods, or call a method more than once (ob.has_ability(Foo), > > > ob.has_ability(Bar), etc.) it would be considerably slower to select an > > > option. > > > >That would make more sense as Foo.implemented_by(ob), right? > > Um, I guess. I was emphasizing the dynamic/per-instance aspect of the > issue. Whichever way you do it, unless you have a well-defined implication > or inheritance hierarchy between interfaces, you have the problem of > ambiguity between opaque interfaces. > > > > > Also -- and this is the real kicker -- what do you do if more than one > > > ability applies? Now we have to have precedence rules for what's more > > > specific. The advantage of defining an ability as a set of one or more > > > applicable generic functions is that precedence is comparatively > > > straightforward: supersets take precedence over subsets, and overlaps are > > > ambiguous. You also have some possibility of being able to implement this > > > by registration-time checks, without needing to build a dispatch tree > > at all. > > > >Couldn't ability inheritance serve the same purpose? > > Sure. The above was part of my argument against *dynamically-determined* > abilities as a core feature. > > > > > In short, I think it's adding interface introspection that's the radical > > > move, where generic functions are just a bit of polish that brings a bit > > > more order to what Python already has. In contrast, interfaces are a > > > foreign religion imported from other languages, while the roots of the > > > generic function faith (len, iter, etc.) have already been in place since > > > the dawn of time! :) > > > >Python has a long tradition of borrowing ideas from other languages, > >and mostly very successful. > > Of course -- and generic functions have a successful tradition in many > other languages too, from Common Lisp to Haskell, that IIUC goes back much > further than "interfaces" as such do. I was pointing out that you had > *already* borrowed this tradition many years ago, whether you realized it > or not. You just "hid" it by using __special__ method names instead of > registries. ;-) > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From talin at acm.org Wed Nov 22 07:15:56 2006 From: talin at acm.org (Talin) Date: Tue, 21 Nov 2006 22:15:56 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: Message-ID: <4563EB1C.5040408@acm.org> Guido van Rossum wrote: > 1. Naming and general ideas. Should we call these Abilities or > Interfaces? Abilities, the term proposed by Andrew Koenig, have less > baggage, but possibly they're just an isomorphic relabeling of > interfaces. From Zope comes the important question: Do we want to > discuss the interfaces/abilities of an instance separately from those > provided by its class? Zope's answer is Yes, reminds Jean-Paul > Calderone, and I tend to like this (having been exposed to it in the > past). In C++ template-land these are called "concepts". Thus, a vector embodies the 'sequential container' concept. Other terms that one can use are: features, affordances, characteristics, facet, quality, etc. Both 'ability' and 'interface' imply (to me, anyway) that the class being inspected is an actor, that it 'does something' rather than being operated on. I kind of like 'feature', although that too has baggage. (I'm thinking of the word 'feature' in the sense of having some property or characteristic, rather than the sense of having positive benefit.) -- Talin From pje at telecommunity.com Wed Nov 22 08:24:35 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 02:24:35 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> At 08:57 PM 11/21/2006 -0800, Guido van Rossum wrote: >Phillip, please shorten your posts. You're hogging all the bandwidth I >have for thinking about this. Please! Er, that *was* the shortened version; I actually spent about an hour *just* on shortening it. I tried to write a short reply to your message, but with little success. So, I'll just rate limit myself by refraining from posting for a couple of days. Will that work? From pje at telecommunity.com Wed Nov 22 08:26:43 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 02:26:43 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <06Nov21.195340pst."58648"@synergy1.parc.xerox.com> References: Message-ID: <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> At 07:53 PM 11/21/2006 -0800, Bill Janssen wrote: >I don't mind having both, but generic functions (interface-based >method dispatch) is really only one aspect of the issue. How so? From g.brandl at gmx.net Wed Nov 22 09:36:09 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 22 Nov 2006 09:36:09 +0100 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: References: <1d85506f0611201317m3f1697d7rd9d3e2fff1124209@mail.gmail.com><1d85506f0611210359m7625c01ava3b2530465d4c8a4@mail.gmail.com><45632E22.2030705@solarsail.hcs.harvard.edu><45633AB8.9010306@livinglogic.de><1d85506f0611211029i3969a91cw1ad621d15d4374e7@mail.gmail.com><45635F16.3060404@livinglogic.de> Message-ID: Terry Reedy schrieb: > "Brett Cannon" wrote in message > news:bbaeab100611211232r53de3c33p915bee7b49dbdf9a at mail.gmail.com... > > Why can't the fallback usage just pass the return value from __len__ to > bool() (forget the C function name) and return that result? It's just like > doing:: > > def bool(obj): > try: > return obj.__bool__() > except AttributeError: > return bool(len(obj)) > ------------ > > If an object without __bool__ returned itself as its length, this would be > an infinite loop, at least in this Python version. Do we worry about > something so crazy? The length would have to be an integer, and this would have to be checked. Georg From walter at livinglogic.de Wed Nov 22 09:45:46 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 22 Nov 2006 09:45:46 +0100 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> Message-ID: <45640E3A.7010307@livinglogic.de> Guido van Rossum wrote: > On 11/21/06, Phillip J. Eby wrote: > [...] >> That's because some generic functions use if-isinstance checks (bad), while >> others use custom __special__ methods (not bad, but not great), registries >> (good), adaptation (okay), or generic function libraries. > > __special__ methods are great for many purposes because they are well > understood. Registries are mostly good when there's a need for an > independent third party to add an ability; IMO this isn't as common in > most places as you seem to think. Another effect of __special__ methods is that they divide the class namespace into two parts: The __special__ names are those that have to be implemented to support core Python interfaces, and the methods without underscores are those that implement the "main aspect" of the class. However there might be other aspects worth supporting (fancy repr, pickling, GUI output, etc.). The methods needed for that support either have to share the special namespace or the main namespace. In Java there are interfaces, but there's only one namespace in which support for an interface can be implemented, so method names have to be fairly long to minimize collisions. With generic functions that problem goes away. It's no longer the name of the method that links the implementation to the protocol, but an explicit registration of the implementing function. > [...] Servus, Walter From p.f.moore at gmail.com Wed Nov 22 10:22:11 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 22 Nov 2006 09:22:11 +0000 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> Message-ID: <79990c6b0611220122y49d27d67hf719670119d715c8@mail.gmail.com> On 11/22/06, Phillip J. Eby wrote: > At 08:57 PM 11/21/2006 -0800, Guido van Rossum wrote: > >Phillip, please shorten your posts. You're hogging all the bandwidth I > >have for thinking about this. Please! > > Er, that *was* the shortened version; I actually spent about an hour *just* > on shortening it. > > I tried to write a short reply to your message, but with little > success. So, I'll just rate limit myself by refraining from posting for a > couple of days. Will that work? I'm also getting overloaded by the flood of information in the longer postings in this thread, compounded by the relatively abstract nature of the discussion. Phillip, as you seem to be of the view that your proposal is relatively small in scope, but wider in implications, maybe you could write a (pre-)PEP? Formally describing the problem, why the current solution(s) aren't sufficient, the precise implementation details and some simple but complete examples, may help people like me understand the scope of what you're proposing. But *please* keep it concrete - it's the abstract descriptions that are losing me! Paul. From gustavo at niemeyer.net Wed Nov 22 14:16:56 2006 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Wed, 22 Nov 2006 11:16:56 -0200 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> References: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> Message-ID: <20061122131656.GA8659@niemeyer.net> Hello Phillip, > If we're basing GF's on types, then certainly marker mixins are > feasible to *use*. I only contend that mixin markers (and other pure > interfaces devoid of execution effect) are unnecessary and shouldn't > be encouraged, not that they are or should be impossible. I don't see a reason to discourage them. It seems like a useful pattern used in a few places in Zope 3, genuinely providing additional value to the adaptation mechanism. > For this reason, I would be against allowing *only* definition-time > interface declarations. A major benefit of both adaptation and > generic functions is that they allow retrofitting or gluing of > "oblivious" code; i.e., code that wasn't originally designed to work > with the thing you're gluing it to. Yes, late binding of interfaces to classes and instances is also of real usage when using interface as markers, as it allows changing the adaptation effect on uncontrolled classes and specific object instances. > Having only interfaces and introspection would also just mean that we > are adding Java-like declarations and LBYLisms, without any > improvements in expressive power or code clarity. Declaring an > interface or adding a mixin should *do* something, instead of simply > being a comment in the form of code, that other code is able to read. > :) I've lost you here. As I understand interfaces, their most important feature is indeed being a "comment in the form of code", and I hope they continue to be that way. What you do with them, being an adaptation system like the Zope 3 component architecture, or even generic functions with interface capabilities, is up to the framework using them. Limiting a standard interface package to a given behavior without these introspection features will only ensure that more variants continue to exist and multiply. Having a standard interface package will of course allow the interpreter itself to do decisions about them ("doing something"), but it would be just one more user of their intrinsic expressiveness. -- Gustavo Niemeyer http://niemeyer.net From solipsis at pitrou.net Wed Nov 22 14:23:39 2006 From: solipsis at pitrou.net (Antoine) Date: Wed, 22 Nov 2006 14:23:39 +0100 (CET) Subject: [Python-3000] defop ? In-Reply-To: <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> Message-ID: <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> > 2. a 'defop' syntax as shorthand for 'addmethod()', such that e.g.: > > import operator > > class Foo: > defop iter(self): > ... > defop len(self): > ... > defop operator.getitem(self, key): > ... Is "defop" useful? Why wouldn't it be possible to enable special method lookup with this kind of syntax: iter = generic_function() iter.lookup_special_method("__iter__") And then the classical: class Foo: def __iter__(self): pass From ncoghlan at gmail.com Wed Nov 22 14:55:27 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 22 Nov 2006 23:55:27 +1000 Subject: [Python-3000] defop ? In-Reply-To: <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> Message-ID: <456456CF.8000005@gmail.com> Antoine wrote: > Is "defop" useful? > > Why wouldn't it be possible to enable special method lookup with this kind > of syntax: > > iter = generic_function() > iter.lookup_special_method("__iter__") > > And then the classical: > > class Foo: > def __iter__(self): > pass A major benefit of the defop syntax lies in the fact that the full Python module namespace is available to you for disambiguation of which operation you are referring to. So if module A defines an overloaded operation "snap" and module B define an overloaded operation "snap", that's OK, because "defop A.snap" and "defop B.snap" can be used to make it clear which one you mean. The magic method namespace, on the other hand, is completely flat: there can be only one __snap__ magic method in a given Python application. If two different modules both want to use the same magic method name, you're out of luck because you can't use the module namespace to disambiguate which one you actually want to support (heck, you might even want to support both of them). A second benefit of defop is that if you have a reference to a generic function then you can overload it. With magic methods, you can only overload it if you know the appropriate incantation (see the thread about correcting the fact that the incantation to overload bool in 2.x is __nonzero__ instead of __bool__ - with defop, it could be written simply as "defop bool(self):", and issue wouldn't need to rise). Finally, the defop syntax means a typo will likely result in an exception when the class definition is executed instead of at some random later time. For example, the typo "def __getiten__(self, key):" will go completely undetected when a class definition is executed, but "defop operator.getiten(self, key):" would fail with an AttributeError. That said, it doesn't *have* to be new syntax - an appropriate function decorator and a class decorator or metaclass to register the appropriate overloads also support this. defop just makes it easier to read and write. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From benji at benjiyork.com Wed Nov 22 14:36:18 2006 From: benji at benjiyork.com (Benji York) Date: Wed, 22 Nov 2006 08:36:18 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: Message-ID: <45645252.6010101@benjiyork.com> Guido van Rossum wrote: > 1. Naming and general ideas. Should we call these Abilities or > Interfaces? I'll try to make an argument for "Interface" over "Ability" using examples from Zope. It seems to me that not all interfaces coincide with something the object can _do_. Some speak to what can be done _to_ an object: class IAttributeAnnotatable(IAnnotatable): """Marker indicating that annotations can be stored on an attribute. This is a marker interface giving permission for an `IAnnotations` adapter to store data in an attribute named `__annotations__`. """ Some give describe how the object will _act_: Persistent. Some are application-level statements about how what is allowed to be _done_with_ the object: ISharable (the user can "share" this object with other users in various ways; read, write, etc.). These (especially the marker interfaces) don't seem to be statements about what the objects can do (abilities), but about other characteristics, therefore a more general name like "interface" seems appropriate. -- Benji York From ark-mlist at att.net Wed Nov 22 15:51:27 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Wed, 22 Nov 2006 09:51:27 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: Message-ID: <002101c70e45$b0888de0$6402a8c0@arkdesktop> > I believe Ka-Ping once proposed something similar. This also jives > nicely with the "verify" functionality that I mentioned. However, I > don't know how easy it will be to write such compliance tests given > that typically the constructor signature is not part of the ability. > It may be more pragmatic to put the tests in a separate unit testing > module that can be subclassed by unit testing code for classes that > want specific claims to be verified. Well, suppose we invent an ability that indicates compliance with a unit-test protocol. Objects that have this ability have a method or methods that can be used to test the object, to the extend that the author wants to render it testable. Then the verify functionality could, as a matter of convention, self-test all objects that claim to be auditable, without the verifier having to know the details. From ark-mlist at att.net Wed Nov 22 15:55:57 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Wed, 22 Nov 2006 09:55:57 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <4563EB1C.5040408@acm.org> Message-ID: <002201c70e46$5149f750$6402a8c0@arkdesktop> > Both 'ability' and 'interface' imply (to me, anyway) that the class > being inspected is an actor, that it 'does something' rather than being > operated on. I chose 'ability' because to me it doesn't require that the class being inspected is active by itself. For example, it feels natural to me to speak of a class as "having the totally ordered ability". The problem I have with 'feature' is that it's already in widespread use without a formal meaning. From ark-mlist at att.net Wed Nov 22 15:59:14 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Wed, 22 Nov 2006 09:59:14 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <45645252.6010101@benjiyork.com> Message-ID: <002301c70e46$c7473260$6402a8c0@arkdesktop> > I'll try to make an argument for "Interface" over "Ability" using > examples from Zope. > It seems to me that not all interfaces coincide with something the > object can _do_. Some speak to what can be done _to_ an object: When I see the word "interface", I think of the collection of method calls that a type supports, because the word is used that way in other well-known programming languages. I am suggesting the word "ability" because it covers more than just method calls. From guido at python.org Wed Nov 22 16:01:42 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 07:01:42 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <002301c70e46$c7473260$6402a8c0@arkdesktop> References: <45645252.6010101@benjiyork.com> <002301c70e46$c7473260$6402a8c0@arkdesktop> Message-ID: On 11/22/06, Andrew Koenig wrote: > > I'll try to make an argument for "Interface" over "Ability" using > > examples from Zope. > > > It seems to me that not all interfaces coincide with something the > > object can _do_. Some speak to what can be done _to_ an object: > > When I see the word "interface", I think of the collection of method calls > that a type supports, because the word is used that way in other well-known > programming languages. I am suggesting the word "ability" because it covers > more than just method calls. Hm, I would think it extends very naturally to instance variables, and I see no reason why it's a particularly bad idea to also extend it (mostly in our minds anyway, I expect) to describe certain behaviors that cannot be expressed (easily) in code. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Nov 22 16:08:21 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 07:08:21 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <45640E3A.7010307@livinglogic.de> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <45640E3A.7010307@livinglogic.de> Message-ID: On 11/22/06, Walter D?rwald wrote: > Another effect of __special__ methods is that they divide the class > namespace into two parts: The __special__ names are those that have to > be implemented to support core Python interfaces, and the methods > without underscores are those that implement the "main aspect" of the > class. Well, there are plenty of examples of __special__ methods for other purposes than core Python interfaces, and plenty of core Python interfaces that don't use __special__ (dict.get(), list.append(), iter.next(), str.lower(), you get the idea.) The *original* idea was for __special__ names to apply to anything that wasn't invoked using regular method notation: x+y, a[i], o.foobar, f(args), len(a), that sort of thing. By extension or expedience it also became used for meta-data (e.g. Zope's __implements__) and for generic functions like pickling (__reduce__). They do have the downside that there's only one __special__ namespace. But that can be mitigated by using naming conventions. IMO it remains to be seen whether there will be too many generic functions to make this work sensibly. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gsakkis at rutgers.edu Wed Nov 22 16:28:47 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Wed, 22 Nov 2006 10:28:47 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <002201c70e46$5149f750$6402a8c0@arkdesktop> References: <4563EB1C.5040408@acm.org> <002201c70e46$5149f750$6402a8c0@arkdesktop> Message-ID: <91ad5bf80611220728p55dc2220q8673cd6858fcf60e@mail.gmail.com> On 11/22/06, Andrew Koenig wrote: > > Both 'ability' and 'interface' imply (to me, anyway) that the class > > being inspected is an actor, that it 'does something' rather than being > > operated on. > > I chose 'ability' because to me it doesn't require that the class being > inspected is active by itself. For example, it feels natural to me to speak > of a class as "having the totally ordered ability". >From a linguistic point of view, "having the totally ordered property" seems more natural and, more generally, captures better both active ("does something") and passive ("is [a description of] something") concepts, but... > The problem I have with 'feature' is that it's already in widespread use > without a formal meaning. Even worse, "property" has already a (different) formal meaning in python, so I guess that's not an option (unless someone can come up with a better name for existing properties _and_ convince that the backwards incompatible name change is not a show-stopper). George From ark-mlist at att.net Wed Nov 22 16:29:22 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Wed, 22 Nov 2006 10:29:22 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: Message-ID: <000501c70e4a$fcb29800$6402a8c0@arkdesktop> > Hm, I would think it extends very naturally to instance variables, and > I see no reason why it's a particularly bad idea to also extend it > (mostly in our minds anyway, I expect) to describe certain behaviors > that cannot be expressed (easily) in code. I think of an ability as a property of an interface rather than an interface itself. For example, I can imagine a single interface having multiple abilities. From pje at telecommunity.com Wed Nov 22 16:45:40 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 10:45:40 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <20061122131656.GA8659@niemeyer.net> References: <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> At 11:16 AM 11/22/2006 -0200, Gustavo Niemeyer wrote: >I've lost you here. As I understand interfaces, their most important >feature is indeed being a "comment in the form of code", and I hope >they continue to be that way. What does it add to have to declare a class as being "Iterable", if it already implements __iter__? What does the notion of "Iterable" add to the execution *or* understanding of the code? If you say that it is because having "Iterable" is needed to distinguish different possible interpretations of the same method name(s), then why not just have method namespaces to begin with? "defop" is one way to do that, of course, but there could easily be others, like a "namespace" statement inside of classes that would contain the method definitions. I'd like to try to meet the needs, without assuming that "interfaces" are the right answer. Indeed, if interfaces are considered the answer, I want to know what the question is. :) Many years ago, I helped to create Zope's interface adaptation system, and I built PyProtocols as well, using interfaces quite extensively in PEAK. So, I am familiar with the presumed benefits of interfaces as well as their drawbacks -- especially their drawbacks. And chief amongst those drawbacks are complexity and duplication. In essence, interfaces turn libraries into "frameworks", but generic functions turn frameworks into libraries. I didn't really discover this until January of last year, though, when I did some experiments in replacing various PEAK interfaces with generic functions: http://dirtsimple.org/2005/01/like-acid-for-frameworks.html The experience of refactoring was what made me realize that generic functions weren't just for obscure fancy use cases like security rules and binary math operators. They *gutted* my interface-based code down to nothing -- hence my description of them as being "like acid for frameworks". From guido at python.org Wed Nov 22 16:53:08 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 07:53:08 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> Message-ID: On 11/21/06, Phillip J. Eby wrote: > At 08:57 PM 11/21/2006 -0800, Guido van Rossum wrote: > >Phillip, please shorten your posts. You're hogging all the bandwidth I > >have for thinking about this. Please! > > Er, that *was* the shortened version; I actually spent about an hour *just* > on shortening it. Ouch. > I tried to write a short reply to your message, but with little > success. So, I'll just rate limit myself by refraining from posting for a > couple of days. Will that work? Sure. Or you could take up my suggestion and spec out addmethod etc., trying to do without defop for now. Or answer the email below in less space than the original. Here are a few more things to ponder. 1. While I write a lot of code, I don't write much code that could benefit from interfaces or generic functions much. What I write is typically a very specific application or a library that speaks a very specific protocol, not a framework. There's one kind of interface that could benefit me, and that's a quick way to distinguish between certain standard types. Examples that keep recurring are things like the smtplib.SMTP.sendmail() method, whose second argument is either a string or a list of strings. The optimal way of writing this, IMO, is a single method that typechecks its 2nd argument. It currently uses "if isinstance(x, basestring)" but I'd be happier if this could be written as some kind of check for implementing the String interface. (If it's neither a string nor a list, I don't really care what happens, it will just raise an exception a few lines later and that's perfectly fine for diagnosing the problem). Rewriting this as a generic method (how do I even create generic methods? I'm sure it's easy) sounds like a total waste of time and code. Expecting that authors of alternative String implementations would care enough about this obscure and super-specialized method to add a binding for their type (smtplib.SMTP.sendmail[MyStringType] = smtplib.SMTP.sendmail[str]) would be naive. So every application that wants to pass a MyStringType to sendmail() needs to do this assignment itself -- a serious reduction in convenience. If there were an interface that means "Behaves like a string" then those few brave souls who write their own string implementation from scratch could just claim to implement it and their string objects would be welcome everywhere. (Sure, there might be a few places where they weren't welcome due to very subtle differences from 'str' -- but I don't expect those to be simply the lack of a certain method, rather some invariant involving several objects.) Another example is the many "open" functions (for opening e.g. tar or zip files, or compressed or encrypted streams, or parsers) that take either a filename or a file objects. It is much more convenient to write a single function that checks if the argument is a string and in that case opens it than to have to use a generic function. I'm foreseeing that you'll propose having a standard generic function that takes a string or a stream and returns a stream -- that is indeed a handy thing to have, except we still have to know whether the argument was a string or not, because if we open it, we want to close it, but if we were passed a stream, we *don't* want to close it. 2. Numbers are another example. They are very common arguments to functions and methods. Very few people write their own number implementations, because the built-in ones are so good. But there *are* people who do, for very specialized reasons (numpy has a whole slew of new number types). I don't think it makes sense to turn everything that takes a number as argument into a generic function. Now, due to duck typing, if you just *use* numbers, and the numpy folks have done their job, things will just work -- this is your argument about using generic operations. But as soon as any type testing happens (e.g. when writing an API that takes either a number of a string or a list of numbers), I'd much rather have a few "big" interfaces (Number, RealNumber, Integer) than lots of small ones Pedroni-style (number-that-support-exponentiation, number-that-supports-bitwise-and, number-that-supports-left-shift, ...?). I guess I either don't understand or don't completely buy the "Pedroni" argument that interfaces should focus on single methods. Or perhaps numbers and strings (types with rich APIs that are only rarely re-implemented from scratch) are special cases? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Nov 22 16:55:25 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 07:55:25 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <000501c70e4a$fcb29800$6402a8c0@arkdesktop> References: <000501c70e4a$fcb29800$6402a8c0@arkdesktop> Message-ID: On 11/22/06, Andrew Koenig wrote: > > Hm, I would think it extends very naturally to instance variables, and > > I see no reason why it's a particularly bad idea to also extend it > > (mostly in our minds anyway, I expect) to describe certain behaviors > > that cannot be expressed (easily) in code. > > I think of an ability as a property of an interface rather than an interface > itself. For example, I can imagine a single interface having multiple > abilities. Hm, that sounds like a case for inheritance. The abilities would be the bases and the interface would extend all of them. They could all be called abilities or they could all be called interfaces. The linguistic metaphor only goes so far... -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Wed Nov 22 17:04:31 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 11:04:31 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <45645252.6010101@benjiyork.com> References: Message-ID: <5.1.1.6.0.20061122104602.02e32618@sparrow.telecommunity.com> At 08:36 AM 11/22/2006 -0500, Benji York wrote: >It seems to me that not all interfaces coincide with something the object >can _do_. Some speak to what can be done _to_ an object: > >class IAttributeAnnotatable(IAnnotatable): > """Marker indicating that annotations can be stored on an attribute. > > This is a marker interface giving permission for an `IAnnotations` > adapter to store data in an attribute named `__annotations__`. > > """ Notice that this can be defined in terms of a "get_annotation_map(ob)" generic function. Classes registered for that function can simply return self.__annotations__ (after setting it to a dictionary, if not already initialized). In fact, it seems to me that such would be the *default* implementation of such a function, eliminating the need to declare anything except for classes that need to implement annotation storage differently. Or am I misunderstanding the intended use? >Some give describe how the object will _act_: Persistent. Which indicates what outside operations can be performed on the object, such as "load", "save", etc. > Some are application-level statements about how what is allowed to be > _done_with_ the object: ISharable (the user can "share" this object with > other users in various ways; read, write, etc.). Aren't these also *operations* being performed? >These (especially the marker interfaces) don't seem to be statements about >what the objects can do (abilities), "Ability" doesn't imply that the object itself has to do the thing. > but about other characteristics, >therefore a more general name like "interface" seems appropriate. Actually "ability" is more general than "interface", because interface implies two things talking to one another. Some Zope uses of "interfaces" don't include any actual "interfacing" at all, so to me your examples actually support calling them "abilities" rather than "interfaces". From guido at python.org Wed Nov 22 17:23:51 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 08:23:51 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122104602.02e32618@sparrow.telecommunity.com> References: <45645252.6010101@benjiyork.com> <5.1.1.6.0.20061122104602.02e32618@sparrow.telecommunity.com> Message-ID: On 11/22/06, Phillip J. Eby wrote: > Actually "ability" is more general than "interface", because interface > implies two things talking to one another. Some Zope uses of "interfaces" > don't include any actual "interfacing" at all, so to me your examples > actually support calling them "abilities" rather than "interfaces". I like this line of reasoning. It makes a lot of sense. I'm still torn because I also like using familiar terminology or syntax even if I have to give it new semantics; Python is all about that! It can get tedious to have to explain to new users "well, an assignment is known as PUT, and a variable is called a location, and subroutines we call a HOW'TO, and strings we call texts" (as was the case for ABC, Python's ill-fated predecessor which invented "optimal" terminology in a vacuum). But for the time being I'd like to try out this new word and see how it fits. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ark-mlist at att.net Wed Nov 22 17:41:10 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Wed, 22 Nov 2006 11:41:10 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> Message-ID: <000801c70e55$043464a0$6402a8c0@arkdesktop> > What does it add to have to declare a class as being "Iterable", if it > already implements __iter__? What does the notion of "Iterable" add to > the execution *or* understanding of the code? Let's suppose for the sake of argument that declaring a class as being "Iterable" adds nothing. What does that say about whether the ability to declare other abilities is useful? I think it doesn't say much. From steven.bethard at gmail.com Wed Nov 22 17:42:52 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 22 Nov 2006 09:42:52 -0700 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> Message-ID: On 11/22/06, Guido van Rossum wrote: > Examples that keep recurring are things like the > smtplib.SMTP.sendmail() method, whose second argument > is either a string or a list of strings. [snip] > Rewriting this as a generic method [snip] > sounds like a total waste of time and code. Expecting that > authors of alternative String implementations would care enough about > this obscure and super-specialized method to add a binding for their > type (smtplib.SMTP.sendmail[MyStringType] = > smtplib.SMTP.sendmail[str]) would be naive. I'm probably totally off base here, but I thought Phillip's suggestion would be not for people to be registering their types for functions, but to be registering their type's methods with the classes they're supposed to be emulating. So I would have expected something like: str.lower[MyStringType] = MyStringType.lower str.split[MyStringType] = MyStringType.split ... And then they wouldn't register themselves for smtplib.SMTP.sendmail; they'd just rely on that function asking for a ``str.lower & str.split`` or whatever as its second argument. (Does seem like there needs to be some sort of shorthand though for "all the string methods".) > 2. Numbers are another example. They are very common arguments to > functions and methods. Very few people write their own number > implementations, because the built-in ones are so good. But there > *are* people who do, for very specialized reasons (numpy has a whole > slew of new number types). I don't think it makes sense to turn > everything that takes a number as argument into a generic function. Again, I may have misunderstood the proposal, but I didn't think you'd be changing the functions that took numbers, I thought you'd be registering the new number type's methods with the builtin methods they were intended to emulate, e.g.:: int.__add__[MyIntegerType] = MyIntegerType.__add__ int.__mul__[MyIntegerType] = MyIntegerType.__mul__ (or however + and * are supposed to be registered). > Now, due to duck typing, if you just *use* numbers, and the numpy > folks have done their job, things will just work -- this is your > argument about using generic operations. But as soon as any type > testing happens (e.g. when writing an API that takes either a number > of a string or a list of numbers), I'd much rather have a few "big" > interfaces (Number, RealNumber, Integer) Yeah, it certainly seems like in both this case and the string case, there needs to be some shorthand for saying "implements all the basic String/Number/whatever methods". STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From pje at telecommunity.com Wed Nov 22 18:12:16 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 12:12:16 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <000801c70e55$043464a0$6402a8c0@arkdesktop> References: <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122120602.041d2af0@sparrow.telecommunity.com> At 11:41 AM 11/22/2006 -0500, Andrew Koenig wrote: > > What does it add to have to declare a class as being "Iterable", if it > > already implements __iter__? What does the notion of "Iterable" add to > > the execution *or* understanding of the code? > >Let's suppose for the sake of argument that declaring a class as being >"Iterable" adds nothing. What does that say about whether the ability to >declare other abilities is useful? I think it doesn't say much. I submit that every other ability can be similarly defined, because the code to *implement* that ability must exist in one of three places: 1. a method of the object 2. a method of an adapter 3. a method registered somewhere, e.g. with a generic function Neither #1 nor #3 require a separate interface declaration, and #2 can be subsumed into an adaptation generic function -- which is actually what iter() does, anyway. Note that all three cases offer potential for directly verifying implementation of the ability, whereas *declaring* an ability merely adds a distinct *claim* that it has been implemented -- which then may need to be separately verified. From pje at telecommunity.com Wed Nov 22 18:02:18 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 12:02:18 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122111942.041e6c30@sparrow.telecommunity.com> At 07:53 AM 11/22/2006 -0800, Guido van Rossum wrote: > If there were an >interface that means "Behaves like a string" then those few brave >souls who write their own string implementation from scratch could >just claim to implement it and their string objects would be welcome >everywhere. Okay, make an 'as_string()' generic function that returns a string (or subclass thereof), or fails. Then, allow the use of one generic function as a type specifier for another, so that if I want to make a 'flatten()' function, I can 'addmethod(flatten, lambda ob: ob, as_string)' to mean "the flatten() of something that can be passed to as_string is the object itself". >having a standard generic function that takes a string or a stream and >returns a stream -- that is indeed a handy thing to have, except we >still have to know whether the argument was a string or not, because >if we open it, we want to close it, but if we were passed a stream, we >*don't* want to close it. Actually, I'd say you'd have a generic function that returns either an adapter or a context manager. The stream adapter would have a no-op close(), or the stream context manager would do nothing on exit. (e.g. "with stream_for(string_or_stream) as f:".) >Now, due to duck typing, if you just *use* numbers, and the numpy >folks have done their job, things will just work -- this is your >argument about using generic operations. But as soon as any type >testing happens (e.g. when writing an API that takes either a number >of a string or a list of numbers), Isn't this basically just another flatten() example? The only time you'd need special casing for numbers would be if you had an iterable number type; otherwise, why even bother checking if they're numbers? >perhaps numbers and strings (types with rich APIs that are only rarely >re-implemented from scratch) are special cases? The Pedroni argument is oriented towards types like file, list, and dict, that have a rich API but are more frequently re-implemented from scratch and more likely to be missing methods. From gustavo at niemeyer.net Wed Nov 22 17:55:13 2006 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Wed, 22 Nov 2006 14:55:13 -0200 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> Message-ID: <20061122165513.GA13795@niemeyer.net> > What does it add to have to declare a class as being "Iterable", if it > already implements __iter__? What does the notion of "Iterable" add > to the execution *or* understanding of the code? If you take a standard method which has a well documented meaning, it's of course non-useful. Now, let's say that you create a library named libfruit, and that it implements a very nice IFruit interface. This library is so nice and shiny that it becomes widely used, and thus many ways to handle a fruit are developed by third parties. Now, someone else has a libapple library, which unfortunately is a bit outdated, and wasn't designed to work with the shiny libfruit library. With interfaces, all you need to do is define an adapter from Apple to IFruit, and instantly all the widely available frameworks will be handling apples just fine. Even though silly, this example describes some of the issues that are common when handling libraries from several vendors. Can you describe how that'd work under your ideas? > If you say that it is because having "Iterable" is needed to > distinguish different possible interpretations of the same method > name(s), then why not just have method namespaces to begin with? > > "defop" is one way to do that, of course, but there could easily be > others, like a "namespace" statement inside of classes that would > contain the method definitions. (...) I'm sure there are many ways to get similar semantics to that offered by interfaces. I can argue that a namespace like this would just take ambiguity to a different level since it needs a name, or that it'd be worse in terms of validation, or even that it'd take away some of the benefits of introspection, and so on. But I don't see value in moving on that kind of discussion here comparing a well established practice and implementation with an idea which isn't clearly defined. > In essence, interfaces turn libraries into "frameworks", but generic > functions turn frameworks into libraries. I didn't really discover this > until January of last year, though, when I did some experiments in > replacing various PEAK interfaces with generic functions: I'm sure these sentences have a well defined meaning in your mind, but they seem extremmely vague from my perspective. > The experience of refactoring was what made me realize that generic > functions weren't just for obscure fancy use cases like security rules > and binary math operators. They *gutted* my interface-based code down > to nothing -- hence my description of them as being "like acid for > frameworks". You seem to compare interfaces to generic functions. As you must be aware of, they're really orthogonal concepts. You can do adaptation and generic function-based code with or without interfaces. -- Gustavo Niemeyer http://niemeyer.net From janssen at parc.com Wed Nov 22 18:20:05 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 22 Nov 2006 09:20:05 PST Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <000501c70e4a$fcb29800$6402a8c0@arkdesktop> References: <000501c70e4a$fcb29800$6402a8c0@arkdesktop> Message-ID: <06Nov22.092011pst."58648"@synergy1.parc.xerox.com> Andrew Koenig writes: > For example, I can imagine a single interface having multiple > abilities. Perhaps because it inherits from multiple sub-interfaces? Bill From janssen at parc.com Wed Nov 22 18:24:26 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 22 Nov 2006 09:24:26 PST Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> References: <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> Message-ID: <06Nov22.092436pst."58648"@synergy1.parc.xerox.com> > At 07:53 PM 11/21/2006 -0800, Bill Janssen wrote: > >I don't mind having both, but generic functions (interface-based > >method dispatch) is really only one aspect of the issue. > > How so? > I think Guido answered this better than I could in . I seem to spend a lot of time inside functions looking over values and figuring out what to do do with them, actions which don't involve invoking further functions, but rather operating directly on those values in different ways. That's probably why the lack of interface-based design in the Python fundamentals seems such a problem to me (and not to others). Bill From janssen at parc.com Wed Nov 22 18:32:46 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 22 Nov 2006 09:32:46 PST Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <45645252.6010101@benjiyork.com> <5.1.1.6.0.20061122104602.02e32618@sparrow.telecommunity.com> Message-ID: <06Nov22.093251pst."58648"@synergy1.parc.xerox.com> Guido wrote: > On 11/22/06, Phillip J. Eby wrote: > > Actually "ability" is more general than "interface", because interface > > implies two things talking to one another. Some Zope uses of "interfaces" > > don't include any actual "interfacing" at all, so to me your examples > > actually support calling them "abilities" rather than "interfaces". > > I like this line of reasoning. It makes a lot of sense. > > I'm still torn because I also like using familiar terminology or > syntax even if I have to give it new semantics; Python is all about > that! It can get tedious to have to explain to new users "well, an > assignment is known as PUT, and a variable is called a location, and > subroutines we call a HOW'TO, and strings we call texts" (as was the > case for ABC, Python's ill-fated predecessor which invented "optimal" > terminology in a vacuum). > > But for the time being I'd like to try out this new word and see how it fits. > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) I see nothing wrong with defining "empty" ABCs to indicate abilities rather than interfaces. class WellTested (Ability): """Indicates that the type has passed the QA process""" pass class TestedNumber (Number, WellTested): ... The whole point of using classes instead of just looking at the methods is to be able to indicate semantics that the programming language can't express. (Well, along with improved simplicity and improved communication and faster type-checking :-). Bill From pje at telecommunity.com Wed Nov 22 18:59:38 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 12:59:38 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122123304.03a6af48@sparrow.telecommunity.com> At 09:42 AM 11/22/2006 -0700, Steven Bethard wrote: >Yeah, it certainly seems like in both this case and the string case, >there needs to be some shorthand for saying "implements all the basic >String/Number/whatever methods". But we are probably better off composing such a concept *from* the methods, rather than having the concept be merely a container *of* methods. This is basically the difference between Java interfaces and Haskell's notion of "typeclasses". You can define typeclasses like "ordered" or "four-function arithmetic" and it's okay if they overlap, because they're defined in terms of the operations you can perform. So nothing prevents somebody else from making "six-function arithmetic", or "reversible strings" or whatever. And nobody has to *re*define their type as supporting these new "interfaces", as long as they have the requisite operations. So, if we have to have something like an interface, let's steal Haskell's version of the idea instead of Java's, because it involves a lot less typing. :) From brett at python.org Wed Nov 22 19:05:36 2006 From: brett at python.org (Brett Cannon) Date: Wed, 22 Nov 2006 10:05:36 -0800 Subject: [Python-3000] LINQ In-Reply-To: <1d85506f0611211507n54f22ed0l9654f0d02442221d@mail.gmail.com> References: <1d85506f0611211507n54f22ed0l9654f0d02442221d@mail.gmail.com> Message-ID: On 11/21/06, tomer filiba wrote: > > i read the references fredrik posted several days back, and it got > me thinking: why not add a "co_ast" attribute to code objects? > > i.e., since (virtually) all code objects are constructed from source > code by compilation, the compiler could just emit the AST as an > attribute to the generated code object. > > yes, there's an issue of memory consumption, but bear with me > for a moment. also, maybe co_ast could be a lazy attribute, reading > the ast from the pyc file on demand. Because the .pyc file does not store enough information for an easy bytecode->AST transformation. You would need to read from the source code and and reconstruct the AST. This is why, as of now, you need to go through compile() to get an AST. I personally would rather wait to see how much the AST is actually used before trying to give it an attribute on code objects. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061122/4734ad25/attachment.html From steven.bethard at gmail.com Wed Nov 22 19:08:20 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 22 Nov 2006 11:08:20 -0700 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122123304.03a6af48@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> <5.1.1.6.0.20061122123304.03a6af48@sparrow.telecommunity.com> Message-ID: On 11/22/06, Phillip J. Eby wrote: > At 09:42 AM 11/22/2006 -0700, Steven Bethard wrote: > >Yeah, it certainly seems like in both this case and the string case, > >there needs to be some shorthand for saying "implements all the basic > >String/Number/whatever methods". > > But we are probably better off composing such a concept *from* the methods, > rather than having the concept be merely a container *of* methods. Yeah, I kinda assumed that the shorthand would be to define something like: IString = str.lower & str.upper & str.split & ... But I'm still a little vague on how this supposed to be implemented. This should definitely be elaborated in the PEP. STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From pje at telecommunity.com Wed Nov 22 19:23:42 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 13:23:42 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <06Nov22.092436pst."58648"@synergy1.parc.xerox.com> References: <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122130031.041f4960@sparrow.telecommunity.com> At 09:24 AM 11/22/2006 -0800, Bill Janssen wrote: >I seem to spend a lot of time inside functions looking over values and >figuring out what to do do with them, actions which don't involve >invoking further functions, but rather operating directly on those >values in different ways. Whereas to me, that's an antipattern because that makes your code closed to extension. What if somebody has a new kind of object that you didn't anticipate? Interfaces don't solve the problem, they just kick it up to another level of abstraction, like hiding the dirty laundry under the rug. The real problem here is that type inspection violates encapsulation and the whole point of being "object-oriented" in the first place. The point of OO is that it lets you *hide* the implementation. If you put the implementation in an if-then branch, you've just lost your OO-ness. The reason it's *tempting* to do this in Python, is because we have so many convenient built-in data types that don't have application-specific methods. In Java, where you have to create new types every time you turn around, and have method overloading anyway, you don't *do* this kind of type inspection. It gets handled by type dispatching -- a primitive sort of compile-time only generic function! It's considered bad style to write a huge block of 'ob instanceof IFoo' tests in Java, when you can just write overloaded methods. In Java, SMTP.sendmail would be something like this (using Python-like syntax 'cause my Java is rusty): def sendmail(self, from, to_addrs:str, msg, ...): return self.sendmail(from_addr, [to_addrs], msg, ...) def sendmail(self, from, to_addrs:list[str], msg, ...): # main implementation So, what I've been trying to say is that I don't think it's a good idea to copy Java interfaces without also copying the machinery that allows you to avoid 'instanceof' spaghetti. From ark-mlist at att.net Wed Nov 22 19:27:07 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Wed, 22 Nov 2006 13:27:07 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <06Nov22.092011pst."58648"@synergy1.parc.xerox.com> Message-ID: <001801c70e63$d172eeb0$6402a8c0@arkdesktop> > Andrew Koenig writes: > > For example, I can imagine a single interface having multiple > > abilities. > Perhaps because it inherits from multiple sub-interfaces? Or perhaps because after the interface was defined, someone noticed that it happened to have those abilities and wanted to be able to say to after the fact. From guido at python.org Wed Nov 22 19:31:38 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 10:31:38 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122123304.03a6af48@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> <5.1.1.6.0.20061122123304.03a6af48@sparrow.telecommunity.com> Message-ID: On 11/22/06, Phillip J. Eby wrote: > At 09:42 AM 11/22/2006 -0700, Steven Bethard wrote: > >Yeah, it certainly seems like in both this case and the string case, > >there needs to be some shorthand for saying "implements all the basic > >String/Number/whatever methods". > > But we are probably better off composing such a concept *from* the methods, > rather than having the concept be merely a container *of* methods. > > This is basically the difference between Java interfaces and Haskell's > notion of "typeclasses". You can define typeclasses like "ordered" or > "four-function arithmetic" and it's okay if they overlap, because they're > defined in terms of the operations you can perform. So nothing prevents > somebody else from making "six-function arithmetic", or "reversible > strings" or whatever. And nobody has to *re*define their type as > supporting these new "interfaces", as long as they have the requisite > operations. > > So, if we have to have something like an interface, let's steal Haskell's > version of the idea instead of Java's, because it involves a lot less > typing. :) But the main thing that stands in the way of doing the right thing in Java is that all interfaces have to be declared when the class is defined. In *my* notion of abilities, this part being inspired by Zope interfaces, you can state after the fact that a particular class (which you didn't write) provides a given ability (which you didn't write either). I think it would also be great if we had "ability algebra" whereby you could state that a given ability A is composed of existing abilities A1 and A2, and every object or class that already has A1 and A2 will automatically be considered to have A. However, I do *not* want to go the way of duck typing here -- if an ability A claims that it is implemented by method foo(), and an object x has a foo() method but doesn't explicitly claim to have ability A (and neither does its class), then testing x for A should return false. I think this deviates from Haskell, which seems to be big on structural type equivalence (?). So I think Python needs to do its own thing here, to some extent. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jackdied at jackdied.com Wed Nov 22 18:04:32 2006 From: jackdied at jackdied.com (Jack Diederich) Date: Wed, 22 Nov 2006 12:04:32 -0500 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: References: Message-ID: <20061122170432.GA21751@performancedrivers.com> On Wed, Nov 22, 2006 at 09:36:09AM +0100, Georg Brandl wrote: > Terry Reedy schrieb: > > "Brett Cannon" wrote in message > > news:bbaeab100611211232r53de3c33p915bee7b49dbdf9a at mail.gmail.com... > > > > Why can't the fallback usage just pass the return value from __len__ to > > bool() (forget the C function name) and return that result? It's just like > > doing:: > > > > def bool(obj): > > try: > > return obj.__bool__() > > except AttributeError: > > return bool(len(obj)) > > ------------ > > > > If an object without __bool__ returned itself as its length, this would be > > an infinite loop, at least in this Python version. Do we worry about > > something so crazy? > > The length would have to be an integer, and this would have to be checked. > It looks like the regular checks are happening on __len__ methods anyway so the explicit int check in slot_nb_bool is redundant. This is the first time I've looked at the new slots in py3k so feel free to correct. (using bool4.patch) sprat:~/src/py3k-rw> ./python Python 3.0x (p3yk:52823M, Nov 22 2006, 11:57:34) [GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> class A(object): def __len__(self): return -1 a = A() print bool(a) ... ... ... >>> >>> Traceback (most recent call last): File "", line 1, in ValueError: __len__() should return >= 0 >>> -Jack From ark-mlist at att.net Wed Nov 22 19:37:06 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Wed, 22 Nov 2006 13:37:06 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: Message-ID: <001901c70e65$3a40a7b0$6402a8c0@arkdesktop> > I think it would also be great if we had "ability > algebra" whereby you could state that a given ability A is composed of > existing abilities A1 and A2, and every object or class that already > has A1 and A2 will automatically be considered to have A. Yes! > However, I do *not* want to go the way of duck typing here -- if an > ability A claims that it is implemented by method foo(), and an object > x has a foo() method but doesn't explicitly claim to have ability A > (and neither does its class), then testing x for A should return > false. Also yes. > I think this deviates from Haskell, which seems to be big on > structural type equivalence (?). How so? > So I think Python needs to do its own thing here, to some extent. Yup. That's part of why I like the term "ability" -- one isn't tempted to think it means the same thing as the corresponding term in another language. From guido at python.org Wed Nov 22 19:43:41 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 10:43:41 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122130031.041f4960@sparrow.telecommunity.com> References: <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> <5.1.1.6.0.20061122130031.041f4960@sparrow.telecommunity.com> Message-ID: On 11/22/06, Phillip J. Eby wrote: > At 09:24 AM 11/22/2006 -0800, Bill Janssen wrote: > >I seem to spend a lot of time inside functions looking over values and > >figuring out what to do do with them, actions which don't involve > >invoking further functions, but rather operating directly on those > >values in different ways. > > Whereas to me, that's an antipattern because that makes your code closed to > extension. What if somebody has a new kind of object that you didn't > anticipate? I'm sorry Phillip, but not all code needs to be open to extension. There's an awful lot of bread-and-butter coding (including some of my own :-) that really doesn't benefit from being open to extension. > Interfaces don't solve the problem, they just kick it up to > another level of abstraction, like hiding the dirty laundry under the rug. There is a dangerous amount of content-free rhetoric in this thread. (And I apogize from my own contributions.) > The real problem here is that type inspection violates encapsulation and > the whole point of being "object-oriented" in the first place. The point > of OO is that it lets you *hide* the implementation. If you put the > implementation in an if-then branch, you've just lost your OO-ness. OO is not religion, it's a tool, and it is not the only hammer we have. As I've tried to mention through examples, there are plenty of situations where a carefully placed if-then is a lot more practical than adding a new method to some remote class (or several). > The reason it's *tempting* to do this in Python, is because we have so many > convenient built-in data types that don't have application-specific > methods. In Java, where you have to create new types every time you turn > around, and have method overloading anyway, you don't *do* this kind of > type inspection. It gets handled by type dispatching -- a primitive sort > of compile-time only generic function! It's considered bad style to write > a huge block of 'ob instanceof IFoo' tests in Java, when you can just write > overloaded methods. True, some type testing code in Python would be done using overloading in Java. But it doesn't always apply, for example when you're examining objects that weren't passed as arguments but were maybe gotten by accessing an argument's attribute, or pulled out of a container, or perhaps produced by some factory function. > In Java, SMTP.sendmail would be something like this (using Python-like > syntax 'cause my Java is rusty): > > def sendmail(self, from, to_addrs:str, msg, ...): > return self.sendmail(from_addr, [to_addrs], msg, ...) > > def sendmail(self, from, to_addrs:list[str], msg, ...): > # main implementation Right. If this syntax was possible in Python lots of people would be very happy. But even the best generic function API I've seen is a lot more verbose than this -- there seems to be a separate set-up involved. A single if-then test does the same job with less fuss, and when you need to support more types, it's easy to refactor to use the GF approach. > So, what I've been trying to say is that I don't think it's a good idea to > copy Java interfaces without also copying the machinery that allows you to > avoid 'instanceof' spaghetti. It's not always automatically spaghetti. Often a single carefully placed test is all you need. sendmail() grew that test long ago when it was noticed that passing a string instead of a list was a common mistake (a certain Googler with a single-letter username gets a lot of junk mail occasionally :-). Nobody has felt the need to add more branches to it in many years. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From janssen at parc.com Wed Nov 22 19:45:21 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 22 Nov 2006 10:45:21 PST Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <001801c70e63$d172eeb0$6402a8c0@arkdesktop> References: <001801c70e63$d172eeb0$6402a8c0@arkdesktop> Message-ID: <06Nov22.104526pst."58648"@synergy1.parc.xerox.com> > > Andrew Koenig writes: > > > For example, I can imagine a single interface having multiple > > > abilities. > > > Perhaps because it inherits from multiple sub-interfaces? > > Or perhaps because after the interface was defined, someone noticed that it > happened to have those abilities and wanted to be able to say to after the > fact. More of a tagging approach, then? Something like class MyNewClass (ExistingClass, OtherInterfaceIJustNoticedExistingClassImplements): pass ? Bill From guido at python.org Wed Nov 22 19:46:42 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 10:46:42 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <001901c70e65$3a40a7b0$6402a8c0@arkdesktop> References: <001901c70e65$3a40a7b0$6402a8c0@arkdesktop> Message-ID: On 11/22/06, Andrew Koenig wrote: > > I think this deviates from Haskell, which seems to be big on > > structural type equivalence (?). I may be mistaken, but I thought that in Haskell, if there's a typeclass composed of methods X and Y (with appropriate signatures), every object that implements X and Y (with those signatures) is *automatically* assumed to be a member of that typeclass? Since I've never used Haskell and all my knowledge of it is from reading about it in Phillip's posts (I've never even followed any of the links he gave) I may well be mistaken. Anyway, I don't care so much about Haskell or whether we copy it; I care about the specific rules/ideas I posted. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ark-mlist at att.net Wed Nov 22 19:48:24 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Wed, 22 Nov 2006 13:48:24 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <06Nov22.104526pst."58648"@synergy1.parc.xerox.com> Message-ID: <001b01c70e66$ca43ea10$6402a8c0@arkdesktop> > More of a tagging approach, then? > > Something like > > class MyNewClass (ExistingClass, > OtherInterfaceIJustNoticedExistingClassImplements): > pass > > ? Maybe, maybe not. I'm not sure. I'm thinking that it may be useful to be able somehow to assert that pre-existing class C has pre-existing ability A, because the authors of C and A weren't talking to each other at the time they wrote them. From guido at python.org Wed Nov 22 19:49:24 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 10:49:24 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <3218934778639261534@unknownmsgid> References: <001801c70e63$d172eeb0$6402a8c0@arkdesktop> <3218934778639261534@unknownmsgid> Message-ID: On 11/22/06, Bill Janssen wrote: > Something like > > class MyNewClass (ExistingClass, OtherInterfaceIJustNoticedExistingClassImplements): > pass No, unless you can also patch up all code that creates ExistingClass instances. You alluded to this before and it sounded like you didn't think it was an issue. Why do you think that? I see it as a huge problem when retrofitting pre-existing third-party code. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From janssen at parc.com Wed Nov 22 19:55:28 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 22 Nov 2006 10:55:28 PST Subject: [Python-3000] Special methods and interface-based type system Message-ID: <06Nov22.105531pst."58648"@synergy1.parc.xerox.com> Talking about the Abilities/Interfaces made me think about some of our "rogue" special method names. In the Language Reference, it says, "A class can implement certain operations that are invoked by special syntax (such as arithmetic operations or subscripting and slicing) by defining methods with special names." But there are all these methods with special names like __len__ or __unicode__ which seem to be provided for the benefit of built-in functions, rather than for support of syntax. Presumably in an interface-based Python, these methods would turn into regularly-named methods on an ABC, so that __len__ would become class container: ... def len(self): raise NotImplemented Though, thinking about it some more, I don't see why *all* syntactic operations wouldn't just invoke the appropriate normally-named method on a specific ABC. "<", for instance, would presumably invoke "object.lessthan" (or perhaps "comparable.lessthan"). So another benefit would be the ability to wean Python away from this mangled-name oddness, which seems to me an HCI improvement. Bill From walter at livinglogic.de Wed Nov 22 19:56:08 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 22 Nov 2006 19:56:08 +0100 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: <20061122170432.GA21751@performancedrivers.com> References: <20061122170432.GA21751@performancedrivers.com> Message-ID: <45649D48.3090703@livinglogic.de> Jack Diederich wrote: > On Wed, Nov 22, 2006 at 09:36:09AM +0100, Georg Brandl wrote: >> Terry Reedy schrieb: >>> "Brett Cannon" wrote in message >>> news:bbaeab100611211232r53de3c33p915bee7b49dbdf9a at mail.gmail.com... >>> >>> Why can't the fallback usage just pass the return value from __len__ to >>> bool() (forget the C function name) and return that result? It's just like >>> doing:: >>> >>> def bool(obj): >>> try: >>> return obj.__bool__() >>> except AttributeError: >>> return bool(len(obj)) >>> ------------ >>> >>> If an object without __bool__ returned itself as its length, this would be >>> an infinite loop, at least in this Python version. Do we worry about >>> something so crazy? >> The length would have to be an integer, and this would have to be checked. >> > > It looks like the regular checks are happening on __len__ methods > anyway so the explicit int check in slot_nb_bool is redundant. > This is the first time I've looked at the new slots in py3k so > feel free to correct. (using bool4.patch) > > sprat:~/src/py3k-rw> ./python > Python 3.0x (p3yk:52823M, Nov 22 2006, 11:57:34) > [GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> class A(object): > def __len__(self): > return -1 > > a = A() > print bool(a) > ... ... ... >>> >>> Traceback (most recent call last): > File "", line 1, in > ValueError: __len__() should return >= 0 OK, I've updated the patch: bugs.python.org/1600346 Servus, Walter From fumanchu at amor.org Wed Nov 22 20:16:02 2006 From: fumanchu at amor.org (Robert Brewer) Date: Wed, 22 Nov 2006 11:16:02 -0800 Subject: [Python-3000] Abilities / Interfaces Message-ID: <435DF58A933BA74397B42CDEB8145A8606F87058@ex9.hostedexchange.local> Phillip J. Eby wrote: > The real problem here is that type inspection violates > encapsulation and the whole point of being "object-oriented" > in the first place. The point of OO is that it lets you > *hide* the implementation. If you put the implementation > in an if-then branch, you've just lost your OO-ness. > > The reason it's *tempting* to do this in Python, is because > we have so many convenient built-in data types that don't > have application-specific methods. Grrmbl. It's such a strong temptation, I'm not sure it's worth fighting in the interests of OO purity. I find myself constantly pushing teammates to favor builtins over custom types for exactly this reason; it reduces the overall complexity budget of any application, thus freeing brain cells to add value (rather than "waste" them striving for pure OO). And given the ease of subclassing builtin types, I'd rather standardize on built-in methods than "application-specific" methods; they meet 99+% of my needs. > In Java, SMTP.sendmail would be something like this (using > Python-like syntax 'cause my Java is rusty): > > def sendmail(self, from, to_addrs:str, msg, ...): > return self.sendmail(from_addr, [to_addrs], msg, ...) > > def sendmail(self, from, to_addrs:list[str], msg, ...): > # main implementation I have a feeling that the current Python-function-call performance overhead would outweigh the benefits of this approach in my own code (which rarely uses more than one isinstance check now). Robert Brewer System Architect Amor Ministries fumanchu at amor.org From pje at telecommunity.com Wed Nov 22 20:32:58 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 14:32:58 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061122123304.03a6af48@sparrow.telecommunity.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> <5.1.1.6.0.20061122123304.03a6af48@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122141542.03790d10@sparrow.telecommunity.com> At 10:31 AM 11/22/2006 -0800, Guido van Rossum wrote: >In *my* notion of abilities, this part being inspired by Zope >interfaces, you can state after the fact that a particular class >(which you didn't write) provides a given ability (which you didn't >write either). I think it would also be great if we had "ability >algebra" whereby you could state that a given ability A is composed of >existing abilities A1 and A2, and every object or class that already >has A1 and A2 will automatically be considered to have A. > >However, I do *not* want to go the way of duck typing here -- if an >ability A claims that it is implemented by method foo(), and an object >x has a foo() method but doesn't explicitly claim to have ability A >(and neither does its class), then testing x for A should return >false. I think this deviates from Haskell, which seems to be big on >structural type equivalence (?). > >So I think Python needs to do its own thing here, to some extent. Actually, you just described Haskell typeclasses. :) That is, they are: 1. Able to be specified after the fact 2. Are algebraically defined as sets of operations 3. Strongly typed based on functions With respect to #3, I think you are confusing Haskell function names with Python method names. Typeclasses also have one more ability that you didn't include above: 4. Able to provide implementations after the fact, not merely declarations This is basically the same concept as adaptation and generic functions, so that you can not only *declare* that a type meets an interface, but also provide a way to make it work. So, we are in agreement on some basics, I think. I am mainly concerned that we do not treat interfaces as collections of method names, rather than collections of operations. (Attributes can be considered operations to read, write, or delete them, and in many cases you will want to specify those operations individually in an interface/typeclass). In Haskell, however, it is not that common for ordinary code to define or use typeclasses, since direct use of the operations (e.g. len()/iter() equivalents) normally suffices to imply the typeclass(es) involved. Also, like Java, Haskell selects overloaded implementations based on their argument type, so you don't write "if isinstance" tests either. Generics reduce the need for type testing and explicit interfaces, since you can just use an operation rather than needing a method (as in Haskell) or you can define overloads to select an implementation (as in Java). In Python, however, interface libraries have historically treated their operations as simply a bag of method names and call signatures, which doesn't allow algebraic operations and introduces the need for explicit adapter types. In contrast, an operation-based approach (where interfaces are collections of first-class "operations" of some kind, that can be recombined to create new interfaces) allows both interface algebra and *automatic* adapter generation (as described in my "monkey typing" proposal early last year). From pje at telecommunity.com Wed Nov 22 20:50:00 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 14:50:00 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <435DF58A933BA74397B42CDEB8145A8606F87058@ex9.hostedexchang e.local> Message-ID: <5.1.1.6.0.20061122143522.03be6308@sparrow.telecommunity.com> At 11:16 AM 11/22/2006 -0800, Robert Brewer wrote: >Phillip J. Eby wrote: > > The real problem here is that type inspection violates > > encapsulation and the whole point of being "object-oriented" > > in the first place. The point of OO is that it lets you > > *hide* the implementation. If you put the implementation > > in an if-then branch, you've just lost your OO-ness. > > > > The reason it's *tempting* to do this in Python, is because > > we have so many convenient built-in data types that don't > > have application-specific methods. > >Grrmbl. It's such a strong temptation, I'm not sure it's worth fighting >in the interests of OO purity. I find myself constantly pushing >teammates to favor builtins over custom types for exactly this reason; I wasn't saying that doing it was a *bad* thing, I suggested we need things to help us do it *better* as a matter of course. For me, if I'm not writing "throwaway" code, I pull type-testing operations out into separate functions, because more often than not, the *same* type testing operation will be used many times in the overall body of code. This is just ordinary DRY to me, not any special "OO purity". To be clear: OO purity isn't my goal. Maintainability, reusability, and incremental implementation are the point. Generic operations win out over interfaces as a fundamental concept because if you need the interface you can always make one *from* a set of operations. And a lot of times, you only care about the one operation anyway. Anyway, I'm beginning to see that the real problem in this discussion is that I tend to phrase "analog" ideas in a way that make other people think they're "digital" rules. I've been mostly trying to say that the path of implementation (and perhaps teaching) should be: generics -> interfaces/typeclasses -> inspection ability In other words, my references to "necessary" and "unnecessary" are not categorical declarations about the nature of programming, but rather are references to how "fundamental" or "incremental" the concept is in a constructive sense. You can achieve all the use cases people have presented using generics only, without any formal notion of interfaces or the ability to do inspection. (As Guido points out, this may not always be *convenient*, but it can be done.) But the reverse isn't true: you can't really do what generics do, using only interfaces. The generics are a more general solution, with interfaces and inspection being merely *convenience* features layered atop the more fundamental base. And since generics are useful to have in their own right, it makes more sense (I argue) to start there. In other words, this isn't some sort of moral judgment regarding the "purity" of interfaces or inspection, it's merely a comment about how best to *construct* them. Building interfaces in the Java/Zope style (as mere collections of method names) loses you things that you can't just put back in later. From guido at python.org Wed Nov 22 20:52:16 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 11:52:16 -0800 Subject: [Python-3000] defop ? In-Reply-To: <456456CF.8000005@gmail.com> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> <456456CF.8000005@gmail.com> Message-ID: [Side remark: I'm sure we would reach understanding and agreement much quicker if we were all in the same room for a day. Maybe we could try to have a day at the next PyCon where we hash this out?] One of the problems that I still have with defop is that it is presented as pure magic. On the one hand, it looks like defop len(self): return 42 is just syntactic sugar for def __len__(self): return 42 OTOH it seems that it is expected to do something completely different when someone write defop foobar(self): ... where foobar is some user-defined generic function. Can someone who understands this provide exact syntax and semantics? Do I understand correctly that the syntax is more like 'defop' '(' ')' ':' than like 'defop' NAME '(' ')' ':' ? (That would need some kind of limitation on the syntax for so that it can't contain a call, or else it would be syntactically ambiguous.) What kind of object should the expression produce? What methods on it are we going to call, with what arguments? I'd love to see an explanation that showed how instead of defop we could also use a certain decorator. Perhaps defop ( ) : is equivalent to @defop() def _ ( ) : # The method name is immaterial here ? Then what goes into the defop decorator? And how does this jive with it being inside a class? (And what would it do if it wasn't?) I'm assuming that "defop foo.bar (...)" only works if foo is imported or defined as a global and has a bar attribute. I'm grasping for understanding here. I don't think Phillip explained this part of his proposal -- all I can find is examples using built-in operations (e.g. "defop len" or "defop operator.getitem") and the handwaving gesture "it could be expanded to add methods for arbitrary *new* operations not defined by the language". My lack of understanding is directly related to this support for new operations. Suppose I wanted to create a flatten operation. What do I put in the module that defines the 'flatten' generic function? Perhaps # This is flattening.py from import generic @generic def flatten(obj): "Generator that flattens a possibly nested sequence." # Default implementation assumes the argument is atomic. yield obj @flatten.when(list) # Oh how I hate "when" but that's not the point here def flatten_list(obj): for value in obj: for v in flatten(value): yield v @flatten[tuple] = flatten_list # or flatten[list] Now I have another module where I define a class implementing a binary tree. I want it to be flattened as the flattening of its items in in-order. import flattening class BinaryTree: def __init__(self, value, left=None, right=None): # Let's pretend the labels must always be strings assert isinstance(value, str) # I really want this to be homogeneous, i.e. all nodes are BinaryTrees or None assert left is None or isinstance(left, BinaryTree) assert right is None or isinstance(right, BinaryTree) self.value = value self.left = left self.right = right def __iter__(self): # Recursive in-order iterator if self.left is not None: for value in self.left: yield value yield self.value if self.right is not None: for value in self.right: yield value defop flattening.flatten(self): for value in self: yield value I realize that I could also have written flattening.flatten[BinaryTree] = flattening.flatten[list] but I'd like to assume for the sake of argument that the BinaryTree class wants to define its own flattening code -- for example, since we know its values are always strings (see the first assert), we don't need the recursive call to flatten() that's present in flatten_list, and we really care about performance. Anyway, the point is to show a class that has its own implementation of some user-defined generic function. So I'm still at a loss about how defop works here. I could understand a different way of defining the flattening of a binary tree: class BinaryTree: ... # as before, but without the defop # now, outside the class: import flattening @flattening.flatten.when(BinaryTree) def flatten_binary_tree(bt): for value in bt: yield value but how on earth is the defop syntax of the @defop decorator going to generate this? I can't see any way to implement it without having access to the class object, but that doesn't exist yet at the time defop or @defop executes. The best I can think of is for @defop to use sys._getframe() to access the dict in which the class body is being evaluated, find or create a list __defop_deferred__ there, and append the tuple(, ) to it. Then the metaclass (i.e., type) should look for this list, and if it exists, do the registrations. I'm guessing you had something like that in mind. I find it pretty fragile though -- what if defop is used in a different context (not directly inside a class)? Is that a syntactic error or does it just modify a random frame's globals, effectively becoming an expensive no-op? A different line of questioning would be to try to understand how Phillip's addmethod and hasmethod are supposed to work. I still hope someone will explain it to me. --Guido On 11/22/06, Nick Coghlan wrote: > Antoine wrote: > > Is "defop" useful? > > > > Why wouldn't it be possible to enable special method lookup with this kind > > of syntax: > > > > iter = generic_function() > > iter.lookup_special_method("__iter__") > > > > And then the classical: > > > > class Foo: > > def __iter__(self): > > pass > > A major benefit of the defop syntax lies in the fact that the full Python > module namespace is available to you for disambiguation of which operation you > are referring to. So if module A defines an overloaded operation "snap" and > module B define an overloaded operation "snap", that's OK, because "defop > A.snap" and "defop B.snap" can be used to make it clear which one you mean. > > The magic method namespace, on the other hand, is completely flat: there can > be only one __snap__ magic method in a given Python application. If two > different modules both want to use the same magic method name, you're out of > luck because you can't use the module namespace to disambiguate which one you > actually want to support (heck, you might even want to support both of them). > > A second benefit of defop is that if you have a reference to a generic > function then you can overload it. With magic methods, you can only overload > it if you know the appropriate incantation (see the thread about correcting > the fact that the incantation to overload bool in 2.x is __nonzero__ instead > of __bool__ - with defop, it could be written simply as "defop bool(self):", > and issue wouldn't need to rise). > > Finally, the defop syntax means a typo will likely result in an exception when > the class definition is executed instead of at some random later time. For > example, the typo "def __getiten__(self, key):" will go completely undetected > when a class definition is executed, but "defop operator.getiten(self, key):" > would fail with an AttributeError. > > That said, it doesn't *have* to be new syntax - an appropriate function > decorator and a class decorator or metaclass to register the appropriate > overloads also support this. defop just makes it easier to read and write. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From krstic at solarsail.hcs.harvard.edu Wed Nov 22 21:07:50 2006 From: krstic at solarsail.hcs.harvard.edu (=?UTF-8?B?SXZhbiBLcnN0acSH?=) Date: Wed, 22 Nov 2006 15:07:50 -0500 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <06Nov22.105531pst."58648"@synergy1.parc.xerox.com> References: <06Nov22.105531pst."58648"@synergy1.parc.xerox.com> Message-ID: <4564AE16.5040409@solarsail.hcs.harvard.edu> Bill Janssen wrote: > Though, thinking about it some more, I don't see why *all* syntactic > operations wouldn't just invoke the appropriate normally-named method > on a specific ABC. "<", for instance, would presumably invoke > "object.lessthan" (or perhaps "comparable.lessthan"). Because it's odd to have these "plumbing" methods in the same namespace as the "porcelain" methods (the ones that implement the exposed functionality of the class.) The desire to keep this separation of namespaces is one of the reasons I understand Phillip is championing `defop`, though other approaches would for example include explicit method namespaces. -- Ivan Krsti? | GPG: 0x147C722D From guido at python.org Wed Nov 22 21:09:31 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 12:09:31 -0800 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <-6683921950610559197@unknownmsgid> References: <-6683921950610559197@unknownmsgid> Message-ID: On 11/22/06, Bill Janssen wrote: > Talking about the Abilities/Interfaces made me think about some of our > "rogue" special method names. In the Language Reference, it says, "A > class can implement certain operations that are invoked by special > syntax (such as arithmetic operations or subscripting and slicing) by > defining methods with special names." But there are all these methods > with special names like __len__ or __unicode__ which seem to be > provided for the benefit of built-in functions, rather than for > support of syntax. Presumably in an interface-based Python, these > methods would turn into regularly-named methods on an ABC, so that > __len__ would become > > class container: > ... > def len(self): > raise NotImplemented > > Though, thinking about it some more, I don't see why *all* syntactic > operations wouldn't just invoke the appropriate normally-named method > on a specific ABC. "<", for instance, would presumably invoke > "object.lessthan" (or perhaps "comparable.lessthan"). So another > benefit would be the ability to wean Python away from this > mangled-name oddness, which seems to me an HCI improvement. Hm. I'm not sure I agree (figure that :-). There are two bits of "Python rationale" that I'd like to explain first. First of all, I chose len(x) over x.len() for HCI reasons (def __len__() came much later). There are two intertwined reasons actually, both HCI: (a) For some operations, prefix notation just reads better than postfix -- prefix (and infix!) operations have a long tradition in mathematics which likes notations where the visuals help the mathematician thinking about a problem. Compare the easy with which we rewrite a formula like x*(a+b) into x*a + x*b to the clumsiness of doing the same thing using a raw OO notation. (b) When I read code that says len(x) I *know* that it is asking for the length of something. This tells me two things: the result is an integer, and the argument is some kind of container. To the contrary, when I read x.len(), I have to already know that x is some kind of container implementing an interface or inheriting from a class that has a standard len(). Witness the confusion we occasionally have when a class that is not implementing a mapping has a get() or keys() method, or something that isn't a file has a write() method. Saying the same thing in another way, I see 'len' as a built-in *operation*. I'd hate to lose that. I can't say for sure whether you meant that or not, but 'def len(self): ...' certainly sounds like you want to demote it to an ordinary method. I'm strongly -1 on that. The second bit of Python rationale I promised to explain is the reason why I chose special methods to look __special__ and not merely special. I was anticipating lots of operations that classes might want to override, some standard (e.g. __add__ or __getitem__), some not so standard (e.g. pickle's __reduce__ for a long time had no support in C code at all). I didn't want these special operations to use ordinary method names, because then pre-existing classes, or classes written by users without an encyclopedic memory for all the special methods, would be liable to accidentally define operations they didn't mean to implement, with possibly disastrous consequences. Ivan Krsti? explained this more concise in his message, which arrived after I'd written all this up. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Nov 22 21:16:49 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 12:16:49 -0800 Subject: [Python-3000] __nonzero__ vs. __bool__ In-Reply-To: <45649D48.3090703@livinglogic.de> References: <20061122170432.GA21751@performancedrivers.com> <45649D48.3090703@livinglogic.de> Message-ID: Once all who care agree on the patch, feel free to check it in without my review. On 11/22/06, Walter D?rwald wrote: > Jack Diederich wrote: > > > On Wed, Nov 22, 2006 at 09:36:09AM +0100, Georg Brandl wrote: > >> Terry Reedy schrieb: > >>> "Brett Cannon" wrote in message > >>> news:bbaeab100611211232r53de3c33p915bee7b49dbdf9a at mail.gmail.com... > >>> > >>> Why can't the fallback usage just pass the return value from __len__ to > >>> bool() (forget the C function name) and return that result? It's just like > >>> doing:: > >>> > >>> def bool(obj): > >>> try: > >>> return obj.__bool__() > >>> except AttributeError: > >>> return bool(len(obj)) > >>> ------------ > >>> > >>> If an object without __bool__ returned itself as its length, this would be > >>> an infinite loop, at least in this Python version. Do we worry about > >>> something so crazy? > >> The length would have to be an integer, and this would have to be checked. > >> > > > > It looks like the regular checks are happening on __len__ methods > > anyway so the explicit int check in slot_nb_bool is redundant. > > This is the first time I've looked at the new slots in py3k so > > feel free to correct. (using bool4.patch) > > > > sprat:~/src/py3k-rw> ./python > > Python 3.0x (p3yk:52823M, Nov 22 2006, 11:57:34) > > [GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2 > > Type "help", "copyright", "credits" or "license" for more information. > >>>> class A(object): > > def __len__(self): > > return -1 > > > > a = A() > > print bool(a) > > ... ... ... >>> >>> Traceback (most recent call last): > > File "", line 1, in > > ValueError: __len__() should return >= 0 > > OK, I've updated the patch: bugs.python.org/1600346 > > Servus, > Walter > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Nov 22 21:28:36 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 12:28:36 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122143522.03be6308@sparrow.telecommunity.com> References: <5.1.1.6.0.20061122143522.03be6308@sparrow.telecommunity.com> Message-ID: On 11/22/06, Phillip J. Eby wrote: > In other words, this isn't some sort of moral judgment regarding the > "purity" of interfaces or inspection, it's merely a comment about how best > to *construct* them. Building interfaces in the Java/Zope style (as mere > collections of method names) loses you things that you can't just put back > in later. I'm not sure that it's right to describe Java/Zope interfaces as "mere collections of method names". For example, Java's collections framework (http://java.sun.com/j2se/1.4.2/docs/guide/collections/) ascribes a lot of semantics to its interfaces that aren't expressible by method signatures alone, and strongly urges implementations to implement the right semantics (of course some folks cheat, but that doesn't invalidate the point). So maybe there are even less differences between the different POVs. I'm not arguing that generics aren't strictly more powerful than interfaces -- I just think that interfaces don't necessarily deserve the bad rep you're creating for them, and I'd like to have interfaces prominently in Py3k's toolbox (as well as generics) rather than having to hide them in the closet. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gustavo at niemeyer.net Wed Nov 22 21:29:24 2006 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Wed, 22 Nov 2006 18:29:24 -0200 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: Message-ID: <20061122202924.GA21361@niemeyer.net> > 2. Do we need interfaces if we have generic functions? > > Phillip Eby argues that we don't need interfaces, because the right > API for generic functions makes it easy to do what we do with > interfaces. Let me paraphrase his argument. I agree that we don't *need* interfaces to have generic functions, just like we don't need them to have adaptation. Using your example, consider Flatten as a base class, and F as an adapter registry containing adapters Fi from C to Flatten. It's feasible. Interfaces provide greater value for both approaches. A short usage case, shortened from the other thread. """ Let's say that you create a library named libfruit, and that it implements a very nice IFruit interface. This library is so nice and shiny that it becomes widely used, and thus many ways to handle a fruit are developed by third parties. Now, someone else has a libapple library, which unfortunately is a bit outdated, and wasn't designed to work with the shiny libfruit library. With interfaces, all you need to do is define an adapter from Apple to IFruit, and instantly all the widely available frameworks will be handling apples just fine. """ Without interfaces, we won't be able to say "in all places, use the generic method registered for an IFruit whenever an Apple is seen". Using generic functions, the same is true. The only difference is that with generic functions each "adapter registry" is capable of adapting only to a single "interface", thus removing the need to explicitly declare it. > 3. API design -- how do we spell the various concepts? E.g. > has_ability(x, A) asks whether object x has the ability A; > provides_ability(C, A) asks whether class C provides the ability A to > its instances. I'm sure you're aware about it, but in Zope 3 terminology, these are 'provide' and 'implement', respectively. > - We need to come up with syntax for the common case, declaring a > class that claims to provide some abilities. Some thoughts: putting (...) Have you considered class decorators? @implements(FooBar) class Baz: ... -- Gustavo Niemeyer http://niemeyer.net From fredrik at pythonware.com Wed Nov 22 21:38:00 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 22 Nov 2006 21:38:00 +0100 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> Message-ID: Guido van Rossum wrote: > Ivan Krsti? explained this more concise in his message, which arrived > after I'd written all this up. which is a good thing, because your post is a much better version of the corresponding FAQ entry than what we came up with when this was last discussed in this thread (last week, I think?). I hope you don't mind me stealing it ;-) cheers /F From guido at python.org Wed Nov 22 21:38:45 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 12:38:45 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <20061122202924.GA21361@niemeyer.net> References: <20061122202924.GA21361@niemeyer.net> Message-ID: On 11/22/06, Gustavo Niemeyer wrote: > > 2. Do we need interfaces if we have generic functions? > > > > Phillip Eby argues that we don't need interfaces, because the right > > API for generic functions makes it easy to do what we do with > > interfaces. Let me paraphrase his argument. > > I agree that we don't *need* interfaces to have generic functions, > just like we don't need them to have adaptation. Using your > example, consider Flatten as a base class, and F as an adapter > registry containing adapters Fi from C to Flatten. It's feasible. > > Interfaces provide greater value for both approaches. A short > usage case, shortened from the other thread. Right. > """ > Let's say that you create a library named libfruit, and that it > implements a very nice IFruit interface. This library is so nice > and shiny that it becomes widely used, and thus many ways to handle > a fruit are developed by third parties. Now, someone else has a > libapple library, which unfortunately is a bit outdated, and wasn't > designed to work with the shiny libfruit library. > > With interfaces, all you need to do is define an adapter from Apple > to IFruit, and instantly all the widely available frameworks will > be handling apples just fine. > """ > > Without interfaces, we won't be able to say "in all places, use the > generic method registered for an IFruit whenever an Apple is seen". Well, this assumes automatic adaptation (which is off the table), *or* everything that uses IFruits must always explicitly invoke some adapter, rather than just assuming that it is passed an IFruit. Or it only works for generic functions that are declared as taking IFruit -- there are many other situations where an IFruit could pop up without directly being the argument of some (generic) function. Consider container types, which haven't figured prominently in this thread yet. Suppose I have a function that expects a sequence of IFruits. Now I have a sequence of Apples. I don't think there's a reasonable mechanism that lets me pass this to that function; I will need to map it to a sequence of IFruits explicitly. > Using generic functions, the same is true. The only difference is > that with generic functions each "adapter registry" is capable of > adapting only to a single "interface", thus removing the need to > explicitly declare it. You've lost me here. A less abstract example might work better. > > 3. API design -- how do we spell the various concepts? E.g. > > has_ability(x, A) asks whether object x has the ability A; > > provides_ability(C, A) asks whether class C provides the ability A to > > its instances. > > I'm sure you're aware about it, but in Zope 3 terminology, these > are 'provide' and 'implement', respectively. I like provide, but I'm not so sure about implement, since it is awfully ambiguous -- most of the time it is the class that does the implementing. That's why I settled for "has". > > - We need to come up with syntax for the common case, declaring a > > class that claims to provide some abilities. Some thoughts: putting > (...) > > Have you considered class decorators? > > @implements(FooBar) > class Baz: > ... No, I'm still waiting for someone to write a PEP for those. But as far as ideas go it's a fine idea. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From solipsis at pitrou.net Wed Nov 22 21:40:53 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 22 Nov 2006 21:40:53 +0100 Subject: [Python-3000] defop ? In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> <456456CF.8000005@gmail.com> Message-ID: <1164228053.4674.54.camel@fsol> Le mercredi 22 novembre 2006 ? 11:52 -0800, Guido van Rossum a ?crit : > but how on earth is the defop syntax of the @defop decorator going to > generate this? I can't see any way to implement it without having > access to the class object, but that doesn't exist yet at the time > defop or @defop executes. The best I can think of is for @defop to use > sys._getframe() to access the dict in which the class body is being > evaluated, find or create a list __defop_deferred__ there, and append > the tuple(, ) to it. Then the metaclass (i.e., type) > should look for this list, and if it exists, do the registrations. Let's say @defop is implemented as follows: class DefOpDescriptor: def __init__(self, genfunc, implfunc): self.genfunc = genfunc self.implfunc = implfunc def __call__(self, *args, **kargs): # Directly call the implementation (why not) self.implfunc(*args, **kargs) def __with_class__(self, cls): """ Register self.implfunc as an implementation of generic function self.genfunc for class cls. """ self.genfunc.when(cls)(self.implfunc) def defop(genfunc): def decorate(implfunc): return DefOpDescriptor(genfunc, implfunc) return decorate And then in the metaclass (i.e., type), if an attribute "attr" has a __with_class__ method, execute attr.__with_class__(cls). (this could perhaps be used for other things than defop) From guido at python.org Wed Nov 22 21:42:53 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 12:42:53 -0800 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> Message-ID: You're welcome. It did occur to me that writing up that particular piece of rationale would be of value outside this thread -- I had no idea it would be picked up so quickly. :-) On 11/22/06, Fredrik Lundh wrote: > Guido van Rossum wrote: > > > Ivan Krsti? explained this more concise in his message, which arrived > > after I'd written all this up. > > which is a good thing, because your post is a much better version of > the corresponding FAQ entry than what we came up with when this was last > discussed in this thread (last week, I think?). I hope you don't mind > me stealing it ;-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Nov 22 21:47:29 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 12:47:29 -0800 Subject: [Python-3000] defop ? In-Reply-To: <1164228053.4674.54.camel@fsol> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> <456456CF.8000005@gmail.com> <1164228053.4674.54.camel@fsol> Message-ID: On 11/22/06, Antoine Pitrou wrote: > Le mercredi 22 novembre 2006 ? 11:52 -0800, Guido van Rossum a ?crit : > > but how on earth is the defop syntax of the @defop decorator going to > > generate this? I can't see any way to implement it without having > > access to the class object, but that doesn't exist yet at the time > > defop or @defop executes. The best I can think of is for @defop to use > > sys._getframe() to access the dict in which the class body is being > > evaluated, find or create a list __defop_deferred__ there, and append > > the tuple(, ) to it. Then the metaclass (i.e., type) > > should look for this list, and if it exists, do the registrations. > > Let's say @defop is implemented as follows: > > class DefOpDescriptor: > def __init__(self, genfunc, implfunc): > self.genfunc = genfunc > self.implfunc = implfunc > def __call__(self, *args, **kargs): > # Directly call the implementation (why not) > self.implfunc(*args, **kargs) I'm not sure how/when __call__ would be used. Shouldn't you implement a __get__ descriptor instead? > def __with_class__(self, cls): > """ Register self.implfunc as an implementation > of generic function self.genfunc for class cls. """ > self.genfunc.when(cls)(self.implfunc) > > def defop(genfunc): > def decorate(implfunc): > return DefOpDescriptor(genfunc, implfunc) > return decorate > > > And then in the metaclass (i.e., type), if an attribute "attr" has a > __with_class__ method, execute attr.__with_class__(cls). > > (this could perhaps be used for other things than defop) Aha, cool. You forgot to add that now the method name chosen must be unique. This works fine to describe @defop, and I like it better than using sys._getframe(). But it doesn't really work for the defop syntax, unless that makes up a unique (random?) name for each occurrence. I guess there's also the issue of invoking it in the wrong context. Maybe that should be a pychecker issue. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Wed Nov 22 21:15:11 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 15:15:11 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061122130031.041f4960@sparrow.telecommunity.com> <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> <5.1.1.6.0.20061122130031.041f4960@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122145257.03be18b8@sparrow.telecommunity.com> At 10:43 AM 11/22/2006 -0800, Guido van Rossum wrote: >There is a dangerous amount of content-free rhetoric in this thread. Actually, I think that some of my statements are being misunderstood as such, when I'm intending only to describe *how* interfaces (or inspection) can be *derived* from generic functions. When I say that these other concepts are "unnecessary", I mean that they can be derived from generics, and that the reverse is not true. This isn't rhetoric, IMO, and it's not about religion or purity, but about practicalities. Often, an interface consists of just one operation, so treating interfaces as fundamental introduces additional "unnecessary" complexity in those cases, whereas being able to treat that one-operation interface as a single generic function removes the need for interfaces -- even as a *concept* to be understood, let alone implemented. That's the power of Python's existing generics: they don't rely on you "implementing the iterable interface", but just on defining the iter() *operation*. >As I've tried to mention through examples, there are plenty of >situations where a carefully placed if-then is a lot more practical >than adding a new method to some remote class (or several). Absolutely. I am saying only that the inspection can be implemented *in terms of* generic functions. >But even the best generic function API I've seen is a lot >more verbose than this -- there seems to be a separate set-up >involved. Actually, my 'defop' proposal would allow you to do this: def sendmail(...): # default case defop sendmail(...): # special case, w/additional type info in arg signature The first statement creates a function called 'sendmail', and the subsequent statement overloads it. Of course, I suppose 'defop' could also be spelled differently to emphasize the difference, e.g.: def sendmail(...): # default case overload sendmail(...): # special case, w/additional type info in arg signature From pje at telecommunity.com Wed Nov 22 21:52:21 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 15:52:21 -0500 Subject: [Python-3000] defop ? In-Reply-To: References: <456456CF.8000005@gmail.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> <456456CF.8000005@gmail.com> Message-ID: <5.1.1.6.0.20061122151242.03f68a88@sparrow.telecommunity.com> At 11:52 AM 11/22/2006 -0800, Guido van Rossum wrote: >Do I understand correctly that the syntax is more like > > 'defop' '(' ')' ':' > >than like > > 'defop' NAME '(' ')' ':' > >? (That would need some kind of limitation on the syntax for so >that it can't contain a call, or else it would be syntactically >ambiguous.) Yes - it should be the same as for decorators or imports, i.e. a possibly dotted name. >What kind of object should the expression produce? What methods on it >are we going to call, with what arguments? The defop statement (and its decorators if any) would translate to this code: @ def (): addmethod(, , enclosing_class_or_None) In other words, addmethod() is invoked following completion of the class suite (if there is one), and passed the decorated anonymous function object. Now, we could say that addmethod() just calls expr.__addmethod__. This would work as long as each builtin had its own __addmethod__() that did the 'cls.__special__ = method' work. However, I personally find it easier to just think of addmethod() as a generic function, so that you can define methods for it that recognize the builtin generics and do the right thing. This also doesn't require the builtins to grow new methods, and thus can be implemented in today's Python using only Python code. (Like the example that you described as brain-exploding orange smoke, which was an actual implementation of addmethod that would work in today's Python 2.x) But if it's easier for you to conceptualize, let's just say it expects expr to have an __addmethod__ method. > I'd love to see an >explanation that showed how instead of defop we could also use a >certain decorator. Perhaps > > defop ( ) : > >is equivalent to > > @defop() > def _ ( ) : # The method name is immaterial here > >? Then what goes into the defop decorator? And how does this jive with >it being inside a class? (And what would it do if it wasn't?) An implementation for today's Python: from peak.util.decorators import decorate_class def defop(expr): def decorate(func): def do_add(cls): addmethod(expr, func, cls) return cls try: decorate_class(do_add) except SyntaxError: # not in a class do_add(None) # handle top-level function return decorate (The 'decorate_class' function applies its argument as a class decorator for the class its caller is invoked from. This is the same mechanism (and actual code!) that Zope uses to do in-body class decorators like 'implements()'.) decorate_class raises a SyntaxError if its caller is not being run directly inside a class body. And yes, this uses frame magic, and no, it's not fragile, as it knows the difference between class and non-class frames. The implementation has been in Zope for a good few years now, likewise Twisted and PEAK. It's also available as a standalone package from: http://cheeseshop.python.org/pypi/DecoratorTools >but how on earth is the defop syntax of the @defop decorator going to >generate this? @defop(flattening.flatten) def flatten_btree(bt:BinaryTree): # ... do stuff to bt Of course, this assumes that flattening.flatten.__addmethod__() is smart enough to pull the type information off the first argument, if the third addmethod() argument is None. >A different line of questioning would be to try to understand how >Phillip's addmethod and hasmethod are supposed to work. I still hope >someone will explain it to me. addmethod registers a method with a generic function. The only reason we even need it as a separate function (as opposed to just assuming a __hasmethod__) is to allow retrofitting of existing Python objects (such as iter/len/etc.) as generic functions via __special__ method setting. The same applies to hasmethod vs. just using __contains__ or something of that sort. If you want to treat it as a Py3K only thing and have __hasmethod__ and __addmethod__ slots for builtin functions, then there's no need to have actual addmethod/hasmethod functions. But that's sort of like saying we don't need getattr() any more because we can just use type(ob).__getattribute__() now! :) From pje at telecommunity.com Wed Nov 22 21:54:20 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 15:54:20 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061122143522.03be6308@sparrow.telecommunity.com> <5.1.1.6.0.20061122143522.03be6308@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122154127.03f77708@sparrow.telecommunity.com> At 12:28 PM 11/22/2006 -0800, Guido van Rossum wrote: >So maybe there are even less differences between the different POVs. Yep. >I'm not arguing that generics aren't strictly more powerful than >interfaces -- I just think that interfaces don't necessarily deserve >the bad rep you're creating for them, and I'd like to have interfaces >prominently in Py3k's toolbox (as well as generics) rather than having >to hide them in the closet. My point is that I'm not trying to create a "bad rep", I'm just emphasizing that it's easier/more useful to define interfaces as collections of generic operations, because it makes the simple cases simple and powerful, and the complex ones possible. Doing it the other way around makes all of the cases more complex, and loses other capabilities. My use of the word "unnecessary" isn't a moral judgment of interfaces, it's an argument about how best to layer the various concepts we want to have. Interfaces aren't necessary for simple things, and it's better (IMO) if we define them *in terms of* simpler things, because then people don't have to learn about them unless they need something more complex. And if they need something more complex, it is also more likely (I believe) that they will have more specialized needs. So, I intended to suggest that if we just do the basics (operations) and allow others to define interface frameworks atop that, then we can avoid overgeneralizing from insufficient experience. However, in the expanded discussion it seems to me that you have goals for additional abilities you'd like to see sooner rather than later. What worries me about Java-style interfaces is that you can't split and recombine them in arbitrary ways, without effectively decomposing them into single-operation interfaces using multiple inheritance. At which point, you just might as well have treated them as operations to start with! Thus, taking an operation-based approach means there need not be a concept of the "one true sequence interface", just the *default* sequence interface. If my needs are more or less expansive than the default "sequence" interface, I should be able to do something like (sequence.len, sequence.iter) and recombine a new interface. This would also make the interface committee's job at least somewhat easier. :) From pje at telecommunity.com Wed Nov 22 21:59:32 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 15:59:32 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <20061122202924.GA21361@niemeyer.net> References: Message-ID: <5.1.1.6.0.20061122155546.03f6c7c0@sparrow.telecommunity.com> At 06:29 PM 11/22/2006 -0200, Gustavo Niemeyer wrote: >Without interfaces, we won't be able to say "in all places, use the >generic method registered for an IFruit whenever an Apple is seen". ...for some definition of "interface". :) Dylan has "protocols", Haskell has "typeclasses", and they do what you're describing. The term "interface" in this discussion has been heavily linked to Java/Zope-style interfaces, however, as opposed to these other styles of "interface". To be clear, I'm opposing a particular style most commonly associated with the word "interface", *not* to the idea of having some way to describe a multitude of types. From guido at python.org Wed Nov 22 22:00:52 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 13:00:52 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122145257.03be18b8@sparrow.telecommunity.com> References: <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> <5.1.1.6.0.20061122130031.041f4960@sparrow.telecommunity.com> <5.1.1.6.0.20061122145257.03be18b8@sparrow.telecommunity.com> Message-ID: On 11/22/06, Phillip J. Eby wrote: > At 10:43 AM 11/22/2006 -0800, Guido van Rossum wrote: > >There is a dangerous amount of content-free rhetoric in this thread. > > Actually, I think that some of my statements are being misunderstood as > such, when I'm intending only to describe *how* interfaces (or inspection) > can be *derived* from generic functions. When I say that these other > concepts are "unnecessary", I mean that they can be derived from generics, > and that the reverse is not true. > > This isn't rhetoric, IMO, and it's not about religion or purity, but about > practicalities. Often, an interface consists of just one operation, so > treating interfaces as fundamental introduces additional "unnecessary" > complexity in those cases, whereas being able to treat that one-operation > interface as a single generic function removes the need for interfaces -- > even as a *concept* to be understood, let alone implemented. That's the > power of Python's existing generics: they don't rely on you "implementing > the iterable interface", but just on defining the iter() *operation*. > > > >As I've tried to mention through examples, there are plenty of > >situations where a carefully placed if-then is a lot more practical > >than adding a new method to some remote class (or several). > > Absolutely. I am saying only that the inspection can be implemented *in > terms of* generic functions. > > > >But even the best generic function API I've seen is a lot > >more verbose than this -- there seems to be a separate set-up > >involved. > > Actually, my 'defop' proposal would allow you to do this: > > def sendmail(...): > # default case > > defop sendmail(...): > # special case, w/additional type info in arg signature > > The first statement creates a function called 'sendmail', and the > subsequent statement overloads it. Of course, I suppose 'defop' could also > be spelled differently to emphasize the difference, e.g.: > > def sendmail(...): > # default case > > overload sendmail(...): > # special case, w/additional type info in arg signature It's important for my understanding to show what's on the dots and also how defop is implemented. See another thread. Right now, while not rhetoric, the above examples are indistinguishable from magic to me (I hope you recognize the quote). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jan.grant at bristol.ac.uk Wed Nov 22 22:14:42 2006 From: jan.grant at bristol.ac.uk (Jan Grant) Date: Wed, 22 Nov 2006 21:14:42 +0000 (GMT) Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <000501c70e4a$fcb29800$6402a8c0@arkdesktop> Message-ID: <20061122210226.O12950@tribble.ilrt.bris.ac.uk> On Wed, 22 Nov 2006, Guido van Rossum wrote: > On 11/22/06, Andrew Koenig wrote: > > > Hm, I would think it extends very naturally to instance variables, and > > > I see no reason why it's a particularly bad idea to also extend it > > > (mostly in our minds anyway, I expect) to describe certain behaviors > > > that cannot be expressed (easily) in code. > > > > I think of an ability as a property of an interface rather than an interface > > itself. For example, I can imagine a single interface having multiple > > abilities. > > Hm, that sounds like a case for inheritance. The abilities would be > the bases and the interface would extend all of them. They could all > be called abilities or they could all be called interfaces. The > linguistic metaphor only goes so far... I don't think it's quite inheritance (except in the set-theoretic sense), unless you can define new superclasses to existing interfaces. To illustrate, let's say that python can take "typed" arguments. You write a function, foo, that has one argument. That argument must have two separate abilities: let's say (for argument's sake) that we want to iterate over its contents (bad choice, perhaps, but bear with me) and also persist it. How do we declare the function? interface interable_and_serialisable(iterable, serialisable): pass def foo(x : iterable_and_serialisable): etc. That's great, but if we want to call foo(x) for an x that comes from a third-party library? Perhaps the library owner even was kind enough to declare that x implemented "iterable" and "serialisable". We've invented "iterable_and_serialisable" after the library was written. Unless we can go back and declare that the class of x implements that interface, there's an impasse. Fine-grained "abilities" lead to some kind of simple ability algebra or other hoop-jumping if you want to do typing of arguments. Of course, without typed arguments we can simply check that x implements iterable and serialisable inside the body of "foo". (Perhaps this is a non-problem: it certainly doesn't appear to crop up much in the Java world.) jan -- jan grant, ISYS, University of Bristol. http://www.bris.ac.uk/ Tel +44 (0)117 3317661 http://ioctl.org/jan/ Hang on, wasn't he holding a wooden parrot? No! It was a porcelain owl. From pje at telecommunity.com Wed Nov 22 22:25:03 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 16:25:03 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061122145257.03be18b8@sparrow.telecommunity.com> <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> <5.1.1.6.0.20061122130031.041f4960@sparrow.telecommunity.com> <5.1.1.6.0.20061122145257.03be18b8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122160753.03cd51d0@sparrow.telecommunity.com> At 01:00 PM 11/22/2006 -0800, Guido van Rossum wrote: >On 11/22/06, Phillip J. Eby wrote: >>The first statement creates a function called 'sendmail', and the >>subsequent statement overloads it. Of course, I suppose 'defop' could also >>be spelled differently to emphasize the difference, e.g.: >> >> def sendmail(...): >> # default case >> >> overload sendmail(...): >> # special case, w/additional type info in arg signature > >It's important for my understanding to show what's on the dots and >also how defop is implemented. See another thread. The defop questions are now answered in that thread. Regarding the dots, I meant them to refer to the Java-style overload example that came previously, assuming the use of a multi-dispatch overload implementation like your type-tuple prototype of earlier this year, using the optional type declaration syntax to describe the overloaded properties. IOW: def sendmail(self, from_addr, to_addrs, msg, mail_options=[], rcpt_options=[]): # default implementation overload sendmail(self, from_addr, to_addrs:str, msg, mail_options=[], rcpt_options=[]): self.sendmail(from_addr, [to_addrs], msg, mail_options, rcpt_options) While this is more verbose than sticking an 'if' into the original sendmail(), note that it can also be added by a third party, via: overload smtpblib.SMTP.sendmail( self, from_addr, to_addrs:str, msg, mail_options=[], rcpt_options=[] ): self.sendmail(from_addr, [to_addrs], msg, mail_options, rcpt_options) So, someone who didn't want to wait for the patch to add your "if" could just add their own fix on the fly. :) > Right now, while >not rhetoric, the above examples are indistinguishable from magic to >me (I hope you recognize the quote). The problem I have sometimes in making my proposals clear to you is that, at the level of abstraction I think on, there are often multiple ways to carry out a useful/usable implementation, but you seem to want to understand one *particular* implementation, even though I'm trying to propose an approach to the solution, rather than a particular solution. However, when I present you with a particular implementation, you then pick holes in the implementation itself, or the tradeoffs made to render the example simple. If I knew what tradeoffs you would prefer ahead of time, then I could make my examples match the desired tradeoffs, but in the absence of that information I can only fall back to describing a general concept. So we tend to go back and forth between abstractions rejected for lack of specificity, and specific things rejected for their particular nature. :) Perhaps I should try giving implementations explicitly labelled as prototypes and stating that there are other ways to do the same thing? From gsakkis at rutgers.edu Wed Nov 22 22:48:11 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Wed, 22 Nov 2006 16:48:11 -0500 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> Message-ID: <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> On 11/22/06, Fredrik Lundh wrote: > Guido van Rossum wrote: > > > Ivan Krsti? explained this more concise in his message, which arrived > > after I'd written all this up. > > which is a good thing, because your post is a much better version of > the corresponding FAQ entry than what we came up with when this was last > discussed in this thread (last week, I think?). I hope you don't mind > me stealing it ;-) > > cheers /F Although I don't necessarily agree with the arguments from a puristic point of view ("then why not make keys(), values(), items(), read(), write(), ... builtin GFs ?"), they are indeed a better answer to the FAQ entry. After all, HCI-based arguments are usually a fancier way of saying "it's a matter of taste" and arguing against them is like arguing that green is a better colour than red. George From janssen at parc.com Wed Nov 22 23:05:09 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 22 Nov 2006 14:05:09 PST Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> Message-ID: <06Nov22.140512pst."58648"@synergy1.parc.xerox.com> > Saying the same thing in another way, I see 'len' as a built-in > *operation*. I'd hate to lose that. I can't say for sure whether you > meant that or not, but 'def len(self): ...' certainly sounds like you > want to demote it to an ordinary method. I'm strongly -1 on that. OK, I can see that. So the built-in function len() and its cousins should be thought of as special Python syntax. I see no real reason to remove that; if "<" just invokes a standard method "orderable.lessthan", the built-in len can just invoke "container.len". But foo.len() would also be available for use. > I didn't want these special operations to use ordinary > method names, because then pre-existing classes, or classes written by > users without an encyclopedic memory for all the special methods, > would be liable to accidentally define operations they didn't mean to > implement, with possibly disastrous consequences. Hmmm. This seems kind of a weak rationale to me, but OK. Bill From guido at python.org Wed Nov 22 23:06:33 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 14:06:33 -0800 Subject: [Python-3000] defop ? In-Reply-To: <5.1.1.6.0.20061122151242.03f68a88@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> <456456CF.8000005@gmail.com> <5.1.1.6.0.20061122151242.03f68a88@sparrow.telecommunity.com> Message-ID: On 11/22/06, Phillip J. Eby wrote: > The defop statement (and its decorators if any) would translate to this code: > > @ > def (): > > > addmethod(, , enclosing_class_or_None) Where is it gonna get the enclosing class? That object hasn't been created yet at the time defop executes. > In other words, addmethod() is invoked following completion of the class > suite (if there is one), and passed the decorated anonymous function object. > Now, we could say that addmethod() just calls expr.__addmethod__. This > would work as long as each builtin had its own __addmethod__() that did the > 'cls.__special__ = method' work. OK, this I can understand. Thanks for being concrete. > However, I personally find it easier to just think of addmethod() as a > generic function, yes, but that's you've had generic functions on your brain for a year or more now. For the rest of us, saying "it calls expr.__addmethod__" is a lot easier to process. And you've already shown earlier that these two are pretty much equivalent. :-) > so that you can define methods for it that recognize the > builtin generics and do the right thing. This also doesn't require the > builtins to grow new methods, and thus can be implemented in today's Python > using only Python code. (Like the example that you described as > brain-exploding orange smoke, which was an actual implementation of > addmethod that would work in today's Python 2.x) Working code can explode heads too. :-) > But if it's easier for you to conceptualize, let's just say it expects expr > to have an __addmethod__ method. Yes, thank you, this helps. > An implementation for today's Python: > > from peak.util.decorators import decorate_class > > def defop(expr): > def decorate(func): > def do_add(cls): > addmethod(expr, func, cls) > return cls > try: > decorate_class(do_add) > except SyntaxError: # not in a class > do_add(None) # handle top-level function > return decorate Alas, this is all smoke and mirrors again for me, since I don't know what decorate_class does, or why on earth it would raise SyntaxError. How does this address the problem that the class doesn't yet exist? > (The 'decorate_class' function applies its argument as a class decorator > for the class its caller is invoked from. This is the same mechanism (and > actual code!) that Zope uses to do in-body class decorators like > 'implements()'.) decorate_class raises a SyntaxError if its caller is not > being run directly inside a class body. That's a bizarre use of that exception. I'm guessing that decorate_class squirrels away some info for later? Everything here seems designed to hide the implementation details, which is great for actual use, but kills understanding of the mechanism. :-( > And yes, this uses frame magic, and no, it's not fragile, as it knows the > difference between class and non-class frames. Wow. How can you tell? Does it work with Jython, IronPython and PyPy? I still find it fragile somehow -- or at least severely hackish. I like the other proposal (where it puts a special attribute on the function object which is then found by the type) better, it doesn't require sys._getframe(). > The implementation has been > in Zope for a good few years now, likewise Twisted and PEAK. It's also > available as a standalone package from: > > http://cheeseshop.python.org/pypi/DecoratorTools > > > >but how on earth is the defop syntax of the @defop decorator going to > >generate this? > > @defop(flattening.flatten) > def flatten_btree(bt:BinaryTree): > # ... do stuff to bt I'm assuming that's outside the context of the BinaryTree class -- since inside it, Binarytree is undefined. > Of course, this assumes that flattening.flatten.__addmethod__() is smart > enough to pull the type information off the first argument, if the third > addmethod() argument is None. Assuming we have optional argument annotations that would be nice. > >A different line of questioning would be to try to understand how > >Phillip's addmethod and hasmethod are supposed to work. I still hope > >someone will explain it to me. > > addmethod registers a method with a generic function. The only reason we > even need it as a separate function (as opposed to just assuming a > __hasmethod__) is to allow retrofitting of existing Python objects (such as > iter/len/etc.) as generic functions via __special__ method setting. The > same applies to hasmethod vs. just using __contains__ or something of that > sort. Hm, I think this is the first time in this thread that you meant to use 'iter', 'len' etc. as "tokens". I think I'd rather change them a bit so that they can be objects that actually understand __addmethod__ and __hasmethod__. > If you want to treat it as a Py3K only thing I do, that's why we're discussing it here. > and have __hasmethod__ and > __addmethod__ slots for builtin functions, I have no problem with making those builtins that are generic functions into instances of some other class. I don't think all 50+ builtins should become generic methods. > then there's no need to have > actual addmethod/hasmethod functions. But that's sort of like saying we > don't need getattr() any more because we can just use > type(ob).__getattribute__() now! :) But I think you're really proposing an addmethod builtin that calls an __addmethod__ method, right? That sounds fine to me if we can work out the rest of the details. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik at pythonware.com Wed Nov 22 23:22:08 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 22 Nov 2006 23:22:08 +0100 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <06Nov22.140512pst."58648"@synergy1.parc.xerox.com> References: <-6683921950610559197@unknownmsgid> <06Nov22.140512pst."58648"@synergy1.parc.xerox.com> Message-ID: Bill Janssen wrote: > This seems kind of a weak rationale to me, but OK. yeah, namespaces are such a useless thing. let's put everything in one big flat namespace, so we don't have to wonder where things are. From fredrik at pythonware.com Wed Nov 22 23:26:33 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 22 Nov 2006 23:26:33 +0100 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> References: <-6683921950610559197@unknownmsgid> <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> Message-ID: George Sakkis wrote: > After all, HCI-based arguments are usually a fancier way of > saying "it's a matter of taste" It would be easier to take you seriously if you gave up that "I don't understand their arguments; therefore I'm right and they're stupid" style of argumentation. Mike and others said *exactly* the same thing as Guido, but that time, you just didn't want to listen. From gustavo at niemeyer.net Wed Nov 22 23:33:56 2006 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Wed, 22 Nov 2006 20:33:56 -0200 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <20061122202924.GA21361@niemeyer.net> Message-ID: <20061122223356.GB22089@niemeyer.net> > Well, this assumes automatic adaptation (which is off the table), *or* > everything that uses IFruits must always explicitly invoke some (...) > You've lost me here. A less abstract example might work better. Sorry.. I should have provided a good concrete example in the first place. Here it goes. implements(Apple, IFruit) implements(Orange, IFruit) With adapters: registry = AdapterRegistry() registry.register(IFruit, IJuice, fruit_to_juice) registry.register(IFruit, IJelly, fruit_to_jelly) apple_juice = registry.query(apple, IJuice) orange_jelly = registry.query(orange, IJelly) With generic functions: @generic juice(obj): return None juice.register(IFruit)(fruit_to_juice) @generic jelly(obj): return None jelly.register(IFruit)(fruit_to_jelly) apple_juice = juice(apple) orange_jelly = jelly(orange) Notice the subtle difference between them. The "target" interface is gone with generic functions. Think of how both would be handled without interfaces. It can be done for sure, but interfaces do provide additional value. -- Gustavo Niemeyer http://niemeyer.net From fredrik at pythonware.com Wed Nov 22 23:50:57 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 22 Nov 2006 23:50:57 +0100 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122160753.03cd51d0@sparrow.telecommunity.com> References: <5.1.1.6.0.20061122145257.03be18b8@sparrow.telecommunity.com> <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> <5.1.1.6.0.20061122130031.041f4960@sparrow.telecommunity.com> <5.1.1.6.0.20061122145257.03be18b8@sparrow.telecommunity.com> <5.1.1.6.0.20061122160753.03cd51d0@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > Perhaps I should try giving implementations explicitly labelled as > prototypes tends to work. > and stating that there are other ways to do the same thing? that's kind of implied in any proposal posted to this list, isn't it? From janssen at parc.com Wed Nov 22 23:51:37 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 22 Nov 2006 14:51:37 PST Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> <06Nov22.140512pst."58648"@synergy1.parc.xerox.com> Message-ID: <06Nov22.145147pst."58648"@synergy1.parc.xerox.com> > Bill Janssen wrote: > > > This seems kind of a weak rationale to me, but OK. > > yeah, namespaces are such a useless thing. let's put everything in one > big flat namespace, so we don't have to wonder where things are. > > Pardon me? That's what we've got now, isn't it? That's why we have to do that ugly and hard-to-explain name mangling for "special" methods. Here's why I think it's weak: 1) Mainly, all methods are special to someone. Having only one hard-to-differentiate syntactic mechanism to identify them really doesn't work very well. 2) But also, if Python was built with interfaces, "special" methods would each be in their own namespace. That is, a user wouldn't have to worry about accidentally overriding a method; they'd have to explicitly override the "len" method inherited from the "container" interface; just defining a method called "len" in their subclass wouldn't do it. CLOS is rather elegant in this respect. Bill From solipsis at pitrou.net Wed Nov 22 23:56:03 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 22 Nov 2006 23:56:03 +0100 Subject: [Python-3000] defop ? In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> <456456CF.8000005@gmail.com> <1164228053.4674.54.camel@fsol> Message-ID: <1164236163.4674.125.camel@fsol> Le mercredi 22 novembre 2006 ? 12:47 -0800, Guido van Rossum a ?crit : > > class DefOpDescriptor: > > def __init__(self, genfunc, implfunc): > > self.genfunc = genfunc > > self.implfunc = implfunc > > def __call__(self, *args, **kargs): > > # Directly call the implementation (why not) > > self.implfunc(*args, **kargs) > > I'm not sure how/when __call__ would be used. > > Shouldn't you implement a __get__ descriptor instead? Hmmm yes, thanks for pointing that out :-o. It will (or should) work better like this: def __get__(self, obj, objtype): return types.MethodType(self.implfunc, obj, objtype) Besides, if we don't want to allow direct calls of the implementation without going through generic method dispath (see below), we don't need __get__ at all. > Aha, cool. You forgot to add that now the method name chosen must be > unique. This works fine to describe @defop, and I like it better than > using sys._getframe(). But it doesn't really work for the defop > syntax, unless that makes up a unique (random?) name for each > occurrence. Indeed it could mangle a unique name, like __private does. Also, DefOpDescriptors could be removed from the class dictionary at class construction, and appended to a __generic_overloads__ attribute. This would keep the class dict clean, and produce better documentation by separating generic function implementations from other methods in the documentation. (it probably still sounds a bit hackish) From fredrik at pythonware.com Wed Nov 22 23:55:45 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 22 Nov 2006 23:55:45 +0100 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <45645252.6010101@benjiyork.com> <5.1.1.6.0.20061122104602.02e32618@sparrow.telecommunity.com> Message-ID: Guido van Rossum wrote: > as was the case for ABC, Python's ill-fated predecessor which invented > "optimal" terminology in a vacuum optimal? I've always thought it was just odd translations from dutch? (but please, cannot we get the trains back in Python 3.0? my son would *love* that ;-) From guido at python.org Thu Nov 23 00:06:26 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 15:06:26 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <45645252.6010101@benjiyork.com> <5.1.1.6.0.20061122104602.02e32618@sparrow.telecommunity.com> Message-ID: On 11/22/06, Fredrik Lundh wrote: > Guido van Rossum wrote: > > > as was the case for ABC, Python's ill-fated predecessor which invented > > "optimal" terminology in a vacuum > > optimal? I've always thought it was just odd translations from dutch? No, dutch was not involved. Lambert's English is impeccable as far as I can recall and Steven Pemberton's British (although he tries to pass for a typical Amsterdammer :-). > (but please, cannot we get the trains back in Python 3.0? my son would > *love* that ;-) Mine would love it too, as long as the trains were called Thomas. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik at pythonware.com Thu Nov 23 00:05:49 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 23 Nov 2006 00:05:49 +0100 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <06Nov22.145147pst."58648"@synergy1.parc.xerox.com> References: <-6683921950610559197@unknownmsgid> <06Nov22.140512pst."58648"@synergy1.parc.xerox.com> <06Nov22.145147pst."58648"@synergy1.parc.xerox.com> Message-ID: Bill Janssen wrote: > Pardon me? That's what we've got now, isn't it? no. names of the form "__name__" are distinctly different from names of the form "name". From fredrik at pythonware.com Thu Nov 23 00:09:41 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 23 Nov 2006 00:09:41 +0100 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> Message-ID: Guido van Rossum wrote: > Examples that keep recurring are things like > the smtplib.SMTP.sendmail() method, whose second argument is either a > string or a list of strings. The optimal way of writing this, IMO, is > a single method that typechecks its 2nd argument. It currently uses > "if isinstance(x, basestring)" but I'd be happier if this could be > written as some kind of check for implementing the String interface. > (If it's neither a string nor a list, I don't really care what > happens, it will just raise an exception a few lines later and that's > perfectly fine for diagnosing the problem). Rewriting this as a > generic method (how do I even create generic methods? I'm sure it's > easy) sounds like a total waste of time and code. I'm not sure I see how changing def sendmail(self, from_addr, to_addr, ...): if isinstance(to_addr, basestring): to_addr = [to_addr] ... to, say def sendmail(self, from_addr, to_addr is a string, ...): # specific self.sendmail(from_addr, [to_addr], ...) def sendmail(self, from_addr, to_addr, ...): # general ... would be a *total* waste of time. but maybe I've just done too much C# programming ;-) From baptiste13 at altern.org Thu Nov 23 00:15:18 2006 From: baptiste13 at altern.org (Baptiste Carvello) Date: Thu, 23 Nov 2006 00:15:18 +0100 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <06Nov22.093251pst."58648"@synergy1.parc.xerox.com> References: <45645252.6010101@benjiyork.com> <5.1.1.6.0.20061122104602.02e32618@sparrow.telecommunity.com> <06Nov22.093251pst."58648"@synergy1.parc.xerox.com> Message-ID: Bill Janssen a ?crit : > > I see nothing wrong with defining "empty" ABCs to indicate abilities > rather than interfaces. > > class WellTested (Ability): > """Indicates that the type has passed the QA process""" > pass > > class TestedNumber (Number, WellTested): > ... > And then if people want to add some tests to check if TestedNumber really implements the Ability, they can do it using docstring syntax. class WellTested (Ability): """Indicates that the type has passed the QA process >>> myinstance=get_test_object() >>> hasattr(myinstance, 'attribute') True >>> myinstance.method(1,2) 3 """ class TestedNumber (Number, WellTested): ... check_ability(TestedNumber, WellTested) The check_ability function would just call the doctest with the appropriate test object (here a TestedNumber instance). pros: - mixin class has no behavior, hence no multiple inheritance problems - doctest syntax allows to describe complex semantic properties without reinventing an ad-hoc language like Zope interfaces do - could be done already in python 2.X cons: - we need another mechanism for tagging instances. Perhaps a WeakKeyDictionary mapping the instances to their added abilities can do. Cheers, BC From guido at python.org Thu Nov 23 00:19:26 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 15:19:26 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> Message-ID: I consider it a waste of time since the if-version works and there's no reason for it to be extensible. There's only so much time we can spend on each program before we move on; perfection isn't always desirable, just working well enough. Now if we had the nice syntax you propose I probably would have used it. But I don't think Phillips' proposal is quite that smooth just yet. On 11/22/06, Fredrik Lundh wrote: > Guido van Rossum wrote: > > > Examples that keep recurring are things like > > the smtplib.SMTP.sendmail() method, whose second argument is either a > > string or a list of strings. The optimal way of writing this, IMO, is > > a single method that typechecks its 2nd argument. It currently uses > > "if isinstance(x, basestring)" but I'd be happier if this could be > > written as some kind of check for implementing the String interface. > > (If it's neither a string nor a list, I don't really care what > > happens, it will just raise an exception a few lines later and that's > > perfectly fine for diagnosing the problem). Rewriting this as a > > generic method (how do I even create generic methods? I'm sure it's > > easy) sounds like a total waste of time and code. > > I'm not sure I see how changing > > def sendmail(self, from_addr, to_addr, ...): > if isinstance(to_addr, basestring): > to_addr = [to_addr] > ... > > to, say > > def sendmail(self, from_addr, to_addr is a string, ...): # specific > self.sendmail(from_addr, [to_addr], ...) > > def sendmail(self, from_addr, to_addr, ...): # general > ... > > would be a *total* waste of time. but maybe I've just done too much C# > programming ;-) > > > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From janssen at parc.com Thu Nov 23 00:21:03 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 22 Nov 2006 15:21:03 PST Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <001801c70e63$d172eeb0$6402a8c0@arkdesktop> <3218934778639261534@unknownmsgid> Message-ID: <06Nov22.152112pst."58648"@synergy1.parc.xerox.com> > On 11/22/06, Bill Janssen wrote: > > Something like > > > > class MyNewClass (ExistingClass, OtherInterfaceIJustNoticedExistingClassImplements): > > pass > > No, unless you can also patch up all code that creates ExistingClass > instances. You alluded to this before and it sounded like you didn't > think it was an issue. Why do you think that? I see it as a huge > problem when retrofitting pre-existing third-party code. I don't think the specific issue raised above is very important (wow, I just noticed that type A also implements the equivalent of interface B, so I wish I could treat A as a B, but darn it, I can't since it didn't inherit from B). I think that ad-hoc adapter classes work OK for handling that rare and incidental case, in the extremely rare case that it *has* to be handled. Not to say that there aren't interesting problems, though I'm not sure just how "big" they are. The biggest issue I see is the factory problem. The other is the existing value problem, which is a variant on either the above or the factory problem. Factory problem: This is a known OO problem. You're given a factory, which produces instances of type A. You've got B, which is a subtype of A, and you wish the factory would return that, since nobody who knows anything uses A instead of B anymore. You've got a number of options: (1) change the factory to produce B instead of A; (2) wrap each instance of A in a B which forwards A's methods to the original instance; (3) somehow change the instance of A into an instance of B. (1) is often practically not possible, though it's probably the right thing to do. In fact, "factories" should always be built to be extensible (that is, they should be built so that user code can register a new factory if desired). (2) is always possible, but takes some extra coding. It would be nifty if Python could have a standard way of doing (3). That's what I'd aim for. That would solve all three of these related problems. Notice, however, that (2) and (3) would probably look pretty much the same for the practicing programmer: myB = coerce_to_B(factory.produce_A()); (3) is practically speaking only possible for types which have relatively independent initializers. Is munging __bases__ so terrible? Isn't it just: def ensure_base (i, clss): if not isinstance(i, clss): class X (i.__class__, clss): pass i.__class__ = X clss.__init__(i) return i Would be nice to have anonymous classes :-). Bill From guido at python.org Thu Nov 23 00:30:33 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 15:30:33 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122160753.03cd51d0@sparrow.telecommunity.com> References: <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> <5.1.1.6.0.20061122130031.041f4960@sparrow.telecommunity.com> <5.1.1.6.0.20061122145257.03be18b8@sparrow.telecommunity.com> <5.1.1.6.0.20061122160753.03cd51d0@sparrow.telecommunity.com> Message-ID: On 11/22/06, Phillip J. Eby wrote: > The problem I have sometimes in making my proposals clear to you is that, > at the level of abstraction I think on, there are often multiple ways to > carry out a useful/usable implementation, but you seem to want to > understand one *particular* implementation, even though I'm trying to > propose an approach to the solution, rather than a particular solution. Oh, it would all be so much easier if we could all be in the same room for a day... The problem often seems to be that I don't have the imagination to come up with a suitable implementation for the abstraction you are describing, and at that point it's no longer an abstraction but an arbitrary string of words to me. Those words may brim over with meaning, but there are so many choices that it's hard to guess which one is meant. [[As an analogy, someone with a lot of HCI experience once pointed out to me that when you show a code fragment to someone who isn't used to programming, they tend to only see the words and lose the punctuation, so for example they would see "self.__getattr__(x)+y" as "self getattr x y" which loses a lot of meaning. Maybe I have the same problem with some of your examples.]] I guess an abstraction ideally needs to be described by formal semantics, not by a hypothetical usage example. Since formal semantics are often hard to come by, I'll settle for an implementation sketch. For example, I really couldn't fathom what you meant for addmethod(iter, ...) to do. It suddenly became completely clear once you said "addmethod(x, ...) calls x.__addmethod__(...)". > However, when I present you with a particular implementation, you then pick > holes in the implementation itself, or the tradeoffs made to render the > example simple. That's unfortunate but can't always be avoided. I'm thinking that I criticize the implementation in order to understand the abstraction better. Since the implementation is often the only clue I have for understanding the abstraction, if the implementation oversimplifies things or has an obvious hole in it this distracts from the understanding. > If I knew what tradeoffs you would prefer ahead of time, > then I could make my examples match the desired tradeoffs, but in the > absence of that information I can only fall back to describing a general > concept. That's why I wish we had more direct face-to-face interaction. > So we tend to go back and forth between abstractions rejected for > lack of specificity, and specific things rejected for their particular > nature. :) Perhaps I should try giving implementations explicitly > labelled as prototypes and stating that there are other ways to do the same > thing? I don't know. In general, I find implementations that draw too strongly on PEAK or RuleDispatch distracting because I'm not familiar with their semantics or terminology and I'm in the "self getattr x y" situation again. At this point I think I understand the defop proposal and I understand addmethod. But I think I could use some help with hasmethod. Suppose I wanted to write the sendmail example using hasmethod. (I know it can be written differently, but other examples may need hasmethod, and I'm familiar with the sendmail example.) Could you should (a) what the example would look like and (b) what would / could go on behind the scenes to make it work? Bonus points for an attempt to show that it can be implemented efficiently. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Thu Nov 23 00:32:38 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 18:32:38 -0500 Subject: [Python-3000] defop ? In-Reply-To: References: <5.1.1.6.0.20061122151242.03f68a88@sparrow.telecommunity.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> <456456CF.8000005@gmail.com> <5.1.1.6.0.20061122151242.03f68a88@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122180342.03ce3008@sparrow.telecommunity.com> At 02:06 PM 11/22/2006 -0800, Guido van Rossum wrote: > > (The 'decorate_class' function applies its argument as a class decorator > > for the class its caller is invoked from. This is the same mechanism (and > > actual code!) that Zope uses to do in-body class decorators like > > 'implements()'.) decorate_class raises a SyntaxError if its caller is not > > being run directly inside a class body. > >That's a bizarre use of that exception. Well, the idea was that a class decorator can only be invoked inside a class, so it's a syntax error to invoke it elsewhere. And, there's little chance of confusing the SyntaxError with some error generated by *running* the decorator itself, as opposed to merely its positioning. Anyway, it seemed like a good idea at the time. :) >I'm guessing that decorate_class squirrels away some info for later? It introduces a stealth metaclass to the class frame, that will call the decorator. If there's an existing metaclass in the frame, it invokes it first to create the class. The devil is in the details, of course, and it does require that you define any explicit metaclass *before* using any class decorators, but this is a documented limitation of the DecoratorTools library. >Everything here seems designed to hide the implementation details, >which is great for actual use, but kills understanding of the >mechanism. :-( > > > And yes, this uses frame magic, and no, it's not fragile, as it knows the > > difference between class and non-class frames. > >Wow. How can you tell? Basically, if f_locals contains a '__module__' key that's equal to the '__name__' in f_globals, although there are some interesting corner cases involving 'exec' -- the details are in the implementation. The relevant function is the 'frameinfo(frame)' function which tells you what "kind" of frame you have. > Does it work with Jython, IronPython and PyPy? PyPy should be fine, since it uses the same code objects and bytecode. IIRC, the others emulate enough of the frame protocol that the '__module__' and '__name__' checks should be no problem. However, I have not tried it in any of these. Since this hack requires metaclass support, there was no point to attempting it in Jython. And when the code was written, neither IronPython nor PyPy existed yet. >I still find it fragile somehow -- or at least severely hackish. I >like the other proposal (where it puts a special attribute on the >function object which is then found by the type) better, it doesn't >require sys._getframe(). Sure, there are certainly lots of other ways to implement it! What I showed was just to demonstrate the *possibility*. If there were defop syntax, then of course the compiler could simply generate the appropriate bytecode following the MAKE_CLASS operation. Or MAKE_CLASS could look for some data inserted into the source dictionary. decorate_class() is just what I had handy to answer your question - it should not be confused with being the One Obvious Way to do it is. It's just proof of possibility (and practical workability, in that it has been long and widely used successfully with at least CPython). But, there are still other ways. The compiler could stick "__owning_class__" attributes on any function defined in a class, and __addmethod__ could defer actually registering the functions until the generic function was invoked, thus ensuring that __owning_class__ would be set first. In general, if we're allowed to change the implementation of Python, it's easy to find some way to do it. But you asked for a way to do it in (I thought) today's Python, so that's what I gave you. >I'm assuming that's outside the context of the BinaryTree class -- >since inside it, Binarytree is undefined. Correct. >Hm, I think this is the first time in this thread that you meant to >use 'iter', 'len' etc. as "tokens". I don't understand what you mean. They're objects, and so usable as tokens. That's why I've also been saying they can also represent "abilities", with defop being a way to unify their use as generic operation (e.g. iter()), implementation (i.e. defop iter), and "interface" or ability (e.g. 'hasmethod(iter,ob)' or something of that sort). >I think I'd rather change them a >bit so that they can be objects that actually understand __addmethod__ >and __hasmethod__. Okay. That's an example of the kind of preference I have no way to divine ahead of time. If I knew you preferred function attributes for deferred registration, and to make these objects of a special type, I'd have proposed it that way. :) My assumption was that changing the type of Python builtin functions would be considered more "radical" than the idea of defop, since it requires changing something, rather than just adding. > > then there's no need to have > > actual addmethod/hasmethod functions. But that's sort of like saying we > > don't need getattr() any more because we can just use > > type(ob).__getattribute__() now! :) > >But I think you're really proposing an addmethod builtin that calls an >__addmethod__ method, right? Yes, assuming that we can call do this with regular functions too. That is, as long as there's some way to retroactively overload a regular function. Other than that, I'm not worried about the need to retrofit any other existing generic function types; their authors will have to change them for Py3K anyway. :) From fredrik at pythonware.com Thu Nov 23 00:36:11 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 23 Nov 2006 00:36:11 +0100 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> Message-ID: Guido van Rossum wrote: > Now if we had the nice syntax you propose I probably would have used > it. well, if it looks nice, what's keeping us from having that? ;-) (but I'm suspect that my view of "generics" is somewhat different from Phillips, so let's focus on fully understanding his proposal first) From pje at telecommunity.com Thu Nov 23 00:37:59 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 18:37:59 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122183441.0403dbf8@sparrow.telecommunity.com> At 03:19 PM 11/22/2006 -0800, Guido van Rossum wrote: >I consider it a waste of time since the if-version works and there's >no reason for it to be extensible. But if that's true, then there's no reason for it to use interfaces, either, is there? :) It seems like a double standard to say, "well, sometimes you have quick-and-dirty code that doesn't need extensibility", and then later complain that it should be able to check for an interface, instead of using either a concrete type or an overload or generic. That is, I don't think you can use the quick'n'dirty case simultaneously as an argument against overloads and *for* interfaces. :) From guido at python.org Thu Nov 23 00:38:19 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 15:38:19 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <-3135130819962384877@unknownmsgid> References: <001801c70e63$d172eeb0$6402a8c0@arkdesktop> <3218934778639261534@unknownmsgid> <-3135130819962384877@unknownmsgid> Message-ID: I'm concerned about the additional wrinkle where there is a library that calls the factory to produce A's and then turns around and passes them on to *another* library. Now the other can deal with A's but has recently been upgraded so that it works much better with B's; but it doesn't know that A's are fine B's already. ideally, there'd be an indirection in there somewhere that would let us insert a conversion from A to B; but if there isn't, it would be nice to be able to poke into A a marker indicating that it also supports B. Somehow that looks nicer with interfaces than with ABCs. On 11/22/06, Bill Janssen wrote: > > On 11/22/06, Bill Janssen wrote: > > > Something like > > > > > > class MyNewClass (ExistingClass, OtherInterfaceIJustNoticedExistingClassImplements): > > > pass > > > > No, unless you can also patch up all code that creates ExistingClass > > instances. You alluded to this before and it sounded like you didn't > > think it was an issue. Why do you think that? I see it as a huge > > problem when retrofitting pre-existing third-party code. > > I don't think the specific issue raised above is very important (wow, > I just noticed that type A also implements the equivalent of interface > B, so I wish I could treat A as a B, but darn it, I can't since it > didn't inherit from B). I think that ad-hoc adapter classes work OK > for handling that rare and incidental case, in the extremely rare case > that it *has* to be handled. > > Not to say that there aren't interesting problems, though I'm not sure > just how "big" they are. The biggest issue I see is the factory > problem. The other is the existing value problem, which is a variant > on either the above or the factory problem. > > Factory problem: This is a known OO problem. You're given a factory, > which produces instances of type A. You've got B, which is a subtype > of A, and you wish the factory would return that, since nobody who > knows anything uses A instead of B anymore. You've got a number of > options: (1) change the factory to produce B instead of A; (2) wrap > each instance of A in a B which forwards A's methods to the original > instance; (3) somehow change the instance of A into an instance of B. > > (1) is often practically not possible, though it's probably the right > thing to do. In fact, "factories" should always be built to be > extensible (that is, they should be built so that user code can > register a new factory if desired). (2) is always possible, but takes > some extra coding. It would be nifty if Python could have a standard > way of doing (3). That's what I'd aim for. That would solve all > three of these related problems. Notice, however, that (2) and (3) > would probably look pretty much the same for the practicing > programmer: > > myB = coerce_to_B(factory.produce_A()); > > (3) is practically speaking only possible for types which have > relatively independent initializers. > > Is munging __bases__ so terrible? Isn't it just: > > def ensure_base (i, clss): > if not isinstance(i, clss): > class X (i.__class__, clss): > pass > i.__class__ = X > clss.__init__(i) > return i > > Would be nice to have anonymous classes :-). > > > Bill > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Nov 23 00:44:23 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 15:44:23 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122183441.0403dbf8@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> <5.1.1.6.0.20061122183441.0403dbf8@sparrow.telecommunity.com> Message-ID: Touche. I think what I was trying to argue is that I probably wouldn't have written it using function overloading, but I would have liked to write the if-test using has_ability(to_addrs, String) instead of isinstance(to_addrs, basestring). (The fact that I even wrote basestring suggests that I was feeling a *little* guilty about the isinstance() call. :-) I also believe that a has_ability() test applies to places where function overloading doesn't, e.g. when the object that could be one of two kinds is not directly an argument but retrieved from a container or returned by some function. Having to introduce an overloaded helper just to avoid a single has_ability() call is a lot of distraction. On 11/22/06, Phillip J. Eby wrote: > At 03:19 PM 11/22/2006 -0800, Guido van Rossum wrote: > >I consider it a waste of time since the if-version works and there's > >no reason for it to be extensible. > > But if that's true, then there's no reason for it to use interfaces, > either, is there? :) It seems like a double standard to say, "well, > sometimes you have quick-and-dirty code that doesn't need extensibility", > and then later complain that it should be able to check for an interface, > instead of using either a concrete type or an overload or generic. > > That is, I don't think you can use the quick'n'dirty case simultaneously as > an argument against overloads and *for* interfaces. :) > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Thu Nov 23 00:45:29 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 18:45:29 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122184343.02871500@sparrow.telecommunity.com> At 12:36 AM 11/23/2006 +0100, Fredrik Lundh wrote: >well, if it looks nice, what's keeping us from having that? ;-) Hear hear. ;) >(but I'm suspect that my view of "generics" is somewhat different from >Phillips, so let's focus on fully understanding his proposal first) So now I'm curious. What *is* your view? Maybe it's better. From guido at python.org Thu Nov 23 00:48:53 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 15:48:53 -0800 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> References: <-6683921950610559197@unknownmsgid> <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> Message-ID: On 11/22/06, George Sakkis wrote: > Although I don't necessarily agree with the arguments from a puristic > point of view ("then why not make keys(), values(), items(), read(), > write(), ... builtin GFs ?"), they are indeed a better answer to the > FAQ entry. After all, HCI-based arguments are usually a fancier way of > saying "it's a matter of taste" and arguing against them is like > arguing that green is a better colour than red. I'm so glad you're not trying to hide that you're a troll. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Nov 23 00:54:28 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 15:54:28 -0800 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <8853879015409334245@unknownmsgid> References: <-6683921950610559197@unknownmsgid> <8853879015409334245@unknownmsgid> Message-ID: On 11/22/06, Bill Janssen wrote: > 2) But also, if Python was built with interfaces, "special" methods > would each be in their own namespace. That is, a user wouldn't have > to worry about accidentally overriding a method; they'd have to > explicitly override the "len" method inherited from the "container" > interface; just defining a method called "len" in their subclass > wouldn't do it. CLOS is rather elegant in this respect. Ow, that's the first time you describe this particular wrinkle. I don't know anything by CLOS. Python has a rather much simpler approach to attribute namespaces for class instances; you can have only one attribute named "len" (since it's contained within a class, most folks consider this sufficient localization to not have to worry about conflicts). I don't think that anyone else here thought of ABCs or interfaces introducing separate namespaces yet. The __special__ namespace is effectively a second namespace, orthogonal to the regular namespace, because there's an explicit convention that you shouldn't define __special__ attributes unless you are implementing whatever semantics a particular __special__ name has. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From janssen at parc.com Thu Nov 23 00:59:55 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 22 Nov 2006 15:59:55 PST Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <001801c70e63$d172eeb0$6402a8c0@arkdesktop> <3218934778639261534@unknownmsgid> <-3135130819962384877@unknownmsgid> Message-ID: <06Nov22.160002pst."58648"@synergy1.parc.xerox.com> > I'm concerned about the additional wrinkle where there is a library > that calls the factory to produce A's and then turns around and passes > them on to *another* library. Now the other can deal with A's but has > recently been upgraded so that it works much better with B's; but it > doesn't know that A's are fine B's already. ideally, there'd be an > indirection in there somewhere that would let us insert a conversion > from A to B; but if there isn't, it would be nice to be able to poke > into A a marker indicating that it also supports B. There's a dangling "it" in there which I'm not sure I have the right antecedent for. Let's see if I've got this straight :-). class B (A): ... def library_1_code(): ... value_A = factory_which_really_produces_Bs() library_2_which_works_much_better_with_Bs(value_A) ... Isn't the marker already implicit? Does it matter that the first library only thinks of the A interface for the value? Or are you just looking for a new built-in like def markas (i, interface_class): if not isinstance(i, interface_class): class X (i.__class__, interface_class): pass i.__class__ = X interface_class.__init__(i) return i ? > Somehow that looks nicer with interfaces than with ABCs. I think I've lost track of the difference. Bill From guido at python.org Thu Nov 23 01:14:16 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 16:14:16 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122154127.03f77708@sparrow.telecommunity.com> References: <5.1.1.6.0.20061122143522.03be6308@sparrow.telecommunity.com> <5.1.1.6.0.20061122154127.03f77708@sparrow.telecommunity.com> Message-ID: On 11/22/06, Phillip J. Eby wrote: > My point is that I'm not trying to create a "bad rep", I'm just emphasizing > that it's easier/more useful to define interfaces as collections of generic > operations, because it makes the simple cases simple and powerful, and the > complex ones possible. Doing it the other way around makes all of the > cases more complex, and loses other capabilities. > > My use of the word "unnecessary" isn't a moral judgment of interfaces, it's > an argument about how best to layer the various concepts we want to > have. Interfaces aren't necessary for simple things, and it's better (IMO) > if we define them *in terms of* simpler things, because then people don't > have to learn about them unless they need something more complex. > > And if they need something more complex, it is also more likely (I believe) > that they will have more specialized needs. So, I intended to suggest that > if we just do the basics (operations) and allow others to define interface > frameworks atop that, then we can avoid overgeneralizing from insufficient > experience. However, in the expanded discussion it seems to me that you > have goals for additional abilities you'd like to see sooner rather than later. > > What worries me about Java-style interfaces is that you can't split and > recombine them in arbitrary ways, without effectively decomposing them into > single-operation interfaces using multiple inheritance. But that's a limitation of Java interfaces (and perhaps of Zope interfaces); it doesn't have to be a limitation of Python 3000 abilities. > At which point, > you just might as well have treated them as operations to start > with! Thus, taking an operation-based approach means there need not be a > concept of the "one true sequence interface", just the *default* sequence > interface. If my needs are more or less expansive than the default > "sequence" interface, I should be able to do something like (sequence.len, > sequence.iter) and recombine a new interface. I'm not sure I understand the significance of prefixing len and iter with "sequence." in your example. After all len and iter are globals. And a default interface will surely tend to become a de-facto standard. What's your view on the interfaces for numbers and strings? I really don't think it's very useful to care much about objects that implement + and * but not / or ** -- while mathematicians have tons of use cases for those, programmers rarely do. As long as those mathematicians can create their own interfaces and apply them to the standard types they should be happy; the rest of us will be served well by a simple hierarchy e.g. Number, Complex, Real, Integer. Even more so for strings; I expect that a single String interface would cover most uses. Well, perhaps there ought to be something more basic representing the shared operations of strings and bytes. But that's about it. Container and file types are a different story; there are many partial reimplementations of the basic patterns and it is useful to be able to describe fairly minimal custom interfaces composed of a small number of operations. User-defined types are different again. I've recently had the opportunity to think a bit about refactoring a library that works with Perforce changelists (CLs; Perforce is a commercial source control system) to be generalized into supporting Subversion revisions. There are a lot of operations one can do on a Perforce CL objects. There is a lot of code that receives, produces or passes around Perforce CL objects. Some of this generalizes to Subversion; some of it doesn't. But I don't expect I'll be needing more than three interfaces to describe the eventual refactoring: an abstract source control revision class as the base interface, and two subclasses specializing for Subversion and Perforce. Sure, eventually someone will want to support CVS, or RCS, or (more likely) git or Mercurial or what have you. The code will continue to evolve and be refactored. But I don't think at any point I'll need to break my interfaces up into individual operations. > This would also make the interface committee's job at least somewhat > easier. :) Anything to make their job easier. I guess we'll have to lock them (us? :-) up in a room for several days before/after PyCon. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From krstic at solarsail.hcs.harvard.edu Thu Nov 23 01:19:03 2006 From: krstic at solarsail.hcs.harvard.edu (=?UTF-8?B?SXZhbiBLcnN0acSH?=) Date: Wed, 22 Nov 2006 19:19:03 -0500 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <06Nov22.145147pst."58648"@synergy1.parc.xerox.com> References: <-6683921950610559197@unknownmsgid> <06Nov22.140512pst."58648"@synergy1.parc.xerox.com> <06Nov22.145147pst."58648"@synergy1.parc.xerox.com> Message-ID: <4564E8F7.6060605@solarsail.hcs.harvard.edu> Bill Janssen wrote: > 1) Mainly, all methods are special to someone. But some, such as those that have special meaning to the language itself, are more special than others. __methods__ are not just plain old methods with funny names. You write regular methods to express functionality you wish your class to have, but you write __methods__ to provide low-level glue tying your class in with the interpreter. This lets the interpreter maintain a kind of pleasant fiction in which user code can depend on system methods, like len() or attribute access, returning sensible results and performing sensible actions even for objects the user code knows nothing about. This difference is substantial enough that a different terminology has been quietly tossed around: the name `defop` implicitly proposes that __methods__ are really "operations". I'm not sure if I like the term yet, but in the meantime, thinking about these methods as system glue, staples, paperclips, or a different fastener term of your choice, might illuminate the difference further. :) -- Ivan Krsti? | GPG: 0x147C722D From pje at telecommunity.com Thu Nov 23 01:20:22 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 19:20:22 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061122160753.03cd51d0@sparrow.telecommunity.com> <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> <5.1.1.6.0.20061122130031.041f4960@sparrow.telecommunity.com> <5.1.1.6.0.20061122145257.03be18b8@sparrow.telecommunity.com> <5.1.1.6.0.20061122160753.03cd51d0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122185122.03e49500@sparrow.telecommunity.com> At 03:30 PM 11/22/2006 -0800, Guido van Rossum wrote: >Oh, it would all be so much easier if we could all be in the same room >for a day... Well, there's always Skype/Gtalk et al... :) >The problem often seems to be that I don't have the imagination to >come up with a suitable implementation for the abstraction you are >describing, and at that point it's no longer an abstraction but an >arbitrary string of words to me. This, and the rest of your message helps a lot, at least in terms of me getting past the "what the heck does he *want* from me?" question. :) >I criticize the implementation in order to understand the abstraction >better. Since the implementation is often the only clue I have for >understanding the abstraction, if the implementation oversimplifies >things or has an obvious hole in it this distracts from the >understanding. Ah. Here, I usually resist working out all the details of a production-quality implementation, because then it seems to make it harder to talk about the *ideas* and select a *good* implementation approach. But I will do my best to accommodate this now that I understand more specifically what you need. >I don't know. In general, I find implementations that draw too >strongly on PEAK or RuleDispatch distracting because I'm not familiar >with their semantics or terminology and I'm in the "self getattr x y" >situation again. Fair enough, except that I haven't drawn on them as much as you think I have; in this discussion I've only referred to the 'simplegeneric' and 'DecoratorTools' libraries, both of which have usage examples on their CheeseShop pages, and both of which are only a few hundred lines of code (~100 for simplegeneric, ~350 for all of DecoratorTools including many things besides the comparatively brief decorate_class feature). What has happened instead, is that when I have introduced hypothetical constructs like 'addmethod', you've assumed that I was referring to something that existed elsewhere and were then saying "but I don't know what that is". But I was saying, "here is a thing I'm *proposing*, and here is how it would work". >But I think I could use some help with hasmethod. Suppose I >wanted to write the sendmail example using hasmethod. (I know it can >be written differently, but other examples may need hasmethod, and I'm >familiar with the sendmail example.) Could you should (a) what the >example would look like and Assuming the existence of an 'as_string()' generic function, either built-in or in the stdlib, one could write it as: if hasmethod(as_string, to_addrs): # it's a string-like thing to_addrs = [to_addrs] The assumption here is that we've chosen to have an 'as_string' generic to denote a general notion of "stringness". The semantics of 'as_string' is that it returns a string of the same "value" as its input, or raises an error if its input isn't "really" a string. In other words, this function would be to strings what __index__ is to integers, if that makes sense. It doesn't convert non-strings to strings, it only converts conceptual strings to concrete strings, the way __index__ converts conceptual integers to concrete integers. (Assuming I understand __index__ correctly, of course, because I tuned out after some of the earliest discussions about what it should do.) So, the generic function 'as_string' here functions as an "interface" or "ability". Of course, instead of being called 'as_string', it could live as an attribute of the 'str' type, e.g. 'str.cast': if hasmethod(str.cast, to_addrs): # it's a string-like thing to_addrs = [to_addrs] I make the assumption, by the way, that once the string-like thing is passed to something that really needs a concrete, C-compatible string (like open() or socket.send()), that it will actually *call* 'str.cast' to get that string. >(b) what would / could go on behind the >scenes to make it work? The generic function's __hasmethod__ checks for an implementation. This can happen via __special__ attribute check (for built-in generics like len/iter/etc.) or via registry lookup (other kinds of generics). >Bonus points for an attempt to show that it >can be implemented efficiently. Registry lookup is a dict lookup against MRO -- i.e., it's like an attribute access check. Slot checks are O(1) if there's a C-level slot, or equivalent to an attribute access check otherwise. From guido at python.org Thu Nov 23 01:32:51 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 16:32:51 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <6961093367361956666@unknownmsgid> References: <001801c70e63$d172eeb0$6402a8c0@arkdesktop> <3218934778639261534@unknownmsgid> <-3135130819962384877@unknownmsgid> <6961093367361956666@unknownmsgid> Message-ID: I was trying to make it so that the factory is part of library 1 and really returns As, which however can be turned into Bs without adding new functionality, merely by marking them as such -- but the assumption is that we don't have a convenient way to modify the factory so that it marks the objects it returns as Bs. IMO the difference between ABCs and interfaces/abilities is that inserting a new ABC into an existing class hierarchy without modifying its source code is considered a hack, and must be done by a helper routine that *actually* mucks with __bases__; while (Zope-style) interfaces use an external registry or a separate class attribute (__implements__) that is intended to be mucked with. So perhaps it really just boils down to mucking with __bases__ or with __implements__. I happen to know that there are cases where Python prevents you from modifying __bases__ in certain ways because the C part of the implementation's safety depends on what's in __bases__. Also modifying __bases__ is expensive, involving a walk of all subclasses. The code that implements safely setting __bases__ is a single long function at lines 184-322 of Objects/typeobject.c in Python (svn head). OTOH, since __implements__ has no special meaning to Python, setting or modifying __implements__ is a simple class attribute assignment. On 11/22/06, Bill Janssen wrote: > > I'm concerned about the additional wrinkle where there is a library > > that calls the factory to produce A's and then turns around and passes > > them on to *another* library. Now the other can deal with A's but has > > recently been upgraded so that it works much better with B's; but it > > doesn't know that A's are fine B's already. ideally, there'd be an > > indirection in there somewhere that would let us insert a conversion > > from A to B; but if there isn't, it would be nice to be able to poke > > into A a marker indicating that it also supports B. > > There's a dangling "it" in there which I'm not sure I have the right > antecedent for. Let's see if I've got this straight :-). > > class B (A): > ... > > def library_1_code(): > ... > value_A = factory_which_really_produces_Bs() > library_2_which_works_much_better_with_Bs(value_A) > ... > > Isn't the marker already implicit? Does it matter that the first > library only thinks of the A interface for the value? Or are you just > looking for a new built-in like > > def markas (i, interface_class): > if not isinstance(i, interface_class): > class X (i.__class__, interface_class): > pass > i.__class__ = X > interface_class.__init__(i) > return i > > ? > > > Somehow that looks nicer with interfaces than with ABCs. > > I think I've lost track of the difference. > > Bill > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Thu Nov 23 01:37:58 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 19:37:58 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061122154127.03f77708@sparrow.telecommunity.com> <5.1.1.6.0.20061122143522.03be6308@sparrow.telecommunity.com> <5.1.1.6.0.20061122154127.03f77708@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122192155.03e3a780@sparrow.telecommunity.com> At 04:14 PM 11/22/2006 -0800, Guido van Rossum wrote: >But that's a limitation of Java interfaces (and perhaps of Zope >interfaces); it doesn't have to be a limitation of Python 3000 >abilities. Agreed -- but some people made statements that implied this was their prototypical vision of "interface", and I was concerned that you considered it the canonical model as well, since you referenced both Java and Zope. > > At which point, > > you just might as well have treated them as operations to start > > with! Thus, taking an operation-based approach means there need not be a > > concept of the "one true sequence interface", just the *default* sequence > > interface. If my needs are more or less expansive than the default > > "sequence" interface, I should be able to do something like (sequence.len, > > sequence.iter) and recombine a new interface. > >I'm not sure I understand the significance of prefixing len and iter >with "sequence." in your example. After all len and iter are globals. There was no significance, it was because I had the Zope/Java interface model in my head at the time and I spoke through that paradigm. IOW, just a brain fart. :) >And a default interface will surely tend to become a de-facto >standard. Sure... except that as long as the operations define compliance, then the "interface" is just an explanation of which operations are required. You can always require a lesser interface (such as iter) rather than a more complex one. >What's your view on the interfaces for numbers and strings? I really >don't think it's very useful to care much about objects that implement >+ and * but not / or ** -- while mathematicians have tons of use cases >for those, programmers rarely do. As long as those mathematicians can >create their own interfaces and apply them to the standard types they >should be happy; the rest of us will be served well by a simple >hierarchy e.g. Number, Complex, Real, Integer. Even more so for >strings; I expect that a single String interface would cover most >uses. Well, perhaps there ought to be something more basic >representing the shared operations of strings and bytes. But that's >about it. Yep, pretty much. However, note that all of these "interfaces" are actually more like *data types*. For immutable data types, I would argue that what you care about is the ability to treat an item "as a" string or whatever. That is, "can I cast this thing to a string without data loss or noise?" "Can I cast to some concrete integral type without loss of precision?" Note too that these casts are *operations* -- perfect for representation as generic operations like int.cast and str.cast. Number, Complex and Real are a little harder to place, since there are no concrete types for those, but there's no reason not to just have the three casting generic functions, and boom, you're done. Now, when you move back up to "protocols" like sequence or arithmetic, we're again talking about single operations (or sets thereof), because rarely does any one piece of code require *all* sequence or arithmetic operations. Ergo, little reason to have a single fixed "interface" spelling out every aspect of the protocol, versus using generics like operator.getitem et al to denote the required capabilities. (Note that in a 'defop' world, it may make sense to put 'operator' in the builtin namespace rather than requiring it to be imported. Or maybe it doesn't. I haven't thought about that one deeply yet.) >Container and file types are a different story; there are many partial >reimplementations of the basic patterns and it is useful to be able to >describe fairly minimal custom interfaces composed of a small number >of operations. Yep. >The code will continue to evolve and be refactored. But I don't think at >any point I'll need to break my interfaces up into individual >operations. True enough, however if the concept of "interface" is implemented in terms of "set of operations", this allows for such things as automatically generated adapters, and partial implementations ala some of Ping and Alex's past proposals, where UserList and UserDict would actually be "interfaces" rather than concrete objects. From greg.ewing at canterbury.ac.nz Thu Nov 23 01:50:38 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 23 Nov 2006 13:50:38 +1300 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <002201c70e46$5149f750$6402a8c0@arkdesktop> References: <002201c70e46$5149f750$6402a8c0@arkdesktop> Message-ID: <4564F05E.7090504@canterbury.ac.nz> Andrew Koenig wrote: > it feels natural to me to speak > of a class as "having the totally ordered ability". That doesn't sound natural to me at all. If you were speaking about a person with brown eyes, would you say that they "had the brown eyes ability"? -- Greg From greg.ewing at canterbury.ac.nz Thu Nov 23 02:26:15 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 23 Nov 2006 14:26:15 +1300 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <20061122165513.GA13795@niemeyer.net> References: <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <20061122165513.GA13795@niemeyer.net> Message-ID: <4564F8B7.5030007@canterbury.ac.nz> Gustavo Niemeyer wrote: > With interfaces, all you need to do is define an adapter from Apple > to IFruit, and instantly all the widely available frameworks will > be handling apples just fine. I don't quite see how that works. Where exactly is the adaptation supposed to be done? It can't be done in libapple, because libfruit didn't exist when it was written. Is libfruit supposed to adapt everything it's given to IFruit just in case it's necessary? That seems like a lot of extra work to go to, both at coding time and run time, for no immediate benefit. -- Greg From greg.ewing at canterbury.ac.nz Thu Nov 23 02:34:49 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 23 Nov 2006 14:34:49 +1300 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <06Nov22.092436pst.58648@synergy1.parc.xerox.com> References: <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> <06Nov22.092436pst.58648@synergy1.parc.xerox.com> Message-ID: <4564FAB9.1010902@canterbury.ac.nz> Bill Janssen wrote: > I seem to spend a lot of time inside functions looking over values and > figuring out what to do do with them, If you're really spending that much time doing things like that, I'd strongly suggest you take a step back and re-think your APIs. -- Greg From gustavo at niemeyer.net Thu Nov 23 03:00:54 2006 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Thu, 23 Nov 2006 00:00:54 -0200 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <4564F8B7.5030007@canterbury.ac.nz> References: <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <20061122165513.GA13795@niemeyer.net> <4564F8B7.5030007@canterbury.ac.nz> Message-ID: <20061123020054.GA31618@niemeyer.net> > I don't quite see how that works. Where exactly is > the adaptation supposed to be done? It can't be done > in libapple, because libfruit didn't exist when it > was written. Is libfruit supposed to adapt everything > it's given to IFruit just in case it's necessary? > That seems like a lot of extra work to go to, both > at coding time and run time, for no immediate benefit. The user of both libraries may define how Apple adapts to IFruit. Being able to define adapters for things that weren't previously prepared for it is one of the usage cases for the component architecture in Zope 3. The mentioned example is a bad one for generic functions, specifically. Look at the more recent mail for a more concrete example than this one, which validates the same point of view for both generic functions and adaptation. -- Gustavo Niemeyer http://niemeyer.net From janssen at parc.com Thu Nov 23 03:14:37 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 22 Nov 2006 18:14:37 PST Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> <8853879015409334245@unknownmsgid> Message-ID: <06Nov22.181438pst."58648"@synergy1.parc.xerox.com> > Ow, that's the first time you describe this particular wrinkle. Yep. I hesitated to bring it up, but if there's a need for separate namespaces for method names, why not do it right? Bill From janssen at parc.com Thu Nov 23 03:20:18 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 22 Nov 2006 18:20:18 PST Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <4564FAB9.1010902@canterbury.ac.nz> References: <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> <06Nov22.092436pst.58648@synergy1.parc.xerox.com> <4564FAB9.1010902@canterbury.ac.nz> Message-ID: <06Nov22.182023pst."58648"@synergy1.parc.xerox.com> > Bill Janssen wrote: > > > I seem to spend a lot of time inside functions looking over values and > > figuring out what to do do with them, > > If you're really spending that much time doing > things like that, I'd strongly suggest you take > a step back and re-think your APIs. > > -- > Greg Typically, they're values that I've gotten back from other people's APIs (another reason not generic functions alone don't solve enough of the problem). I seem to like to do things that they hadn't thought of :-). Bill From jonathan-lists at cleverdevil.org Thu Nov 23 03:30:01 2006 From: jonathan-lists at cleverdevil.org (Jonathan LaCour) Date: Wed, 22 Nov 2006 21:30:01 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> Message-ID: <02AFCDEB-25D2-4D10-9BE4-A2773CCED65C@cleverdevil.org> Phillip J. Eby wrote: > In essence, interfaces turn libraries into "frameworks", but generic > functions turn frameworks into libraries. I didn't really discover > this until January of last year, though, when I did some experiments > in replacing various PEAK interfaces with generic functions. This is easily the most important part of the conversation, IMO, and I would hate for it to be missed. "Interfaces" in the Java universe are a hard contract that everyone has to agree upon up-front, and if you don't agree, you have to create your own framework that does the very same things in a different way. What you end up with is a world of many competing frameworks, none of which interoperate. The alternative may not be as *familiar* but it certainly paints a much brighter picture. If you have two different frameworks that take slightly different approaches (or even _radically_ different approaches) to the same problems, you can make them work together as one, rather than having to pick one or the other. Its like framework detente! Adopting Java-like interfaces just because they are "familiar" would be a huge mistake, as it would condemn Python to the limitations of such an implementation unnecessarily. This is Python 3000, not Java, and while I certainly understand the argument of familiarity, I won't let something that seems a tad magical at first scare me into a comfortable decision. I have actually used generic functions (using RuleDispatch) in practice and find them to not only be hugely powerful, but much more pragmatic. Am I the only one out there who sees it this way? -- Jonathan LaCour http://cleverdevil.org From janssen at parc.com Thu Nov 23 03:30:32 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 22 Nov 2006 18:30:32 PST Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <4564E8F7.6060605@solarsail.hcs.harvard.edu> References: <-6683921950610559197@unknownmsgid> <06Nov22.140512pst."58648"@synergy1.parc.xerox.com> <06Nov22.145147pst."58648"@synergy1.parc.xerox.com> <4564E8F7.6060605@solarsail.hcs.harvard.edu> Message-ID: <06Nov22.183038pst."58648"@synergy1.parc.xerox.com> > > 1) Mainly, all methods are special to someone. > > But some, such as those that have special meaning to the language > itself, are more special than others. > > __methods__ are not just plain old methods with funny names. You write > regular methods to express functionality you wish your class to have, > but you write __methods__ to provide low-level glue tying your class in > with the interpreter. That's a somewhat different rationale than Guido provided, thanks. Or maybe I just understand it better. Still seems an odd way to differentiate a namespace. Bill From janssen at parc.com Thu Nov 23 03:31:32 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 22 Nov 2006 18:31:32 PST Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> <8853879015409334245@unknownmsgid> Message-ID: <06Nov22.183133pst."58648"@synergy1.parc.xerox.com> > I don't know anything by CLOS. I'll drop off a copy of the book on Monday. Lord knows we have zillions of extra copies floating around PARC. Bill From guido at python.org Thu Nov 23 04:03:57 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 19:03:57 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <02AFCDEB-25D2-4D10-9BE4-A2773CCED65C@cleverdevil.org> References: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <02AFCDEB-25D2-4D10-9BE4-A2773CCED65C@cleverdevil.org> Message-ID: On 11/22/06, Jonathan LaCour wrote: > Phillip J. Eby wrote: > > > In essence, interfaces turn libraries into "frameworks", but generic > > functions turn frameworks into libraries. I didn't really discover > > this until January of last year, though, when I did some experiments > > in replacing various PEAK interfaces with generic functions. > > This is easily the most important part of the conversation, IMO, and I > would hate for it to be missed. "Interfaces" in the Java universe are a > hard contract that everyone has to agree upon up-front, and if you don't > agree, you have to create your own framework that does the very same > things in a different way. What you end up with is a world of many > competing frameworks, none of which interoperate. > > The alternative may not be as *familiar* but it certainly paints a much > brighter picture. If you have two different frameworks that take > slightly different approaches (or even _radically_ different approaches) > to the same problems, you can make them work together as one, rather > than having to pick one or the other. Its like framework detente! > > Adopting Java-like interfaces just because they are "familiar" would be > a huge mistake, as it would condemn Python to the limitations of such an > implementation unnecessarily. This is Python 3000, not Java, and while I > certainly understand the argument of familiarity, I won't let something > that seems a tad magical at first scare me into a comfortable decision. > > I have actually used generic functions (using RuleDispatch) in practice > and find them to not only be hugely powerful, but much more pragmatic. > Am I the only one out there who sees it this way? Sorry, but that was about as informative as a "+1" vote. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gsakkis at rutgers.edu Thu Nov 23 04:08:47 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Wed, 22 Nov 2006 22:08:47 -0500 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> Message-ID: <91ad5bf80611221908t4e1ed686yb94d700d1bbc0cb5@mail.gmail.com> On 11/22/06, Fredrik Lundh wrote: > George Sakkis wrote: > > > After all, HCI-based arguments are usually a fancier way of > > saying "it's a matter of taste" > > It would be easier to take you seriously if you gave up that "I don't > understand their arguments; therefore I'm right and they're stupid" > style of argumentation. Mike and others said *exactly* the same thing > as Guido, but that time, you just didn't want to listen. First off, I never implied someone's stupid just because we don't happen to agree on everything. As for Mike's answer, I went back and read it again; please do the same. He doesn't address HCI reasons at all, it's a purely technical argument by pointing out an inconsistency of Java (which kinda distracts from the main point but that's ok). George From greg.ewing at canterbury.ac.nz Thu Nov 23 04:12:28 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 23 Nov 2006 16:12:28 +1300 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <06Nov22.182023pst.58648@synergy1.parc.xerox.com> References: <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> <06Nov22.092436pst.58648@synergy1.parc.xerox.com> <4564FAB9.1010902@canterbury.ac.nz> <06Nov22.182023pst.58648@synergy1.parc.xerox.com> Message-ID: <4565119C.6060700@canterbury.ac.nz> Bill Janssen wrote: > Typically, they're values that I've gotten back from other people's > APIs Can you give an example? I'm having a hard time trying to think of an API that I've used which returns unpredictable types. Maybe it's those other APIs that need to be redesigned? -- Greg From greg.ewing at canterbury.ac.nz Thu Nov 23 04:19:49 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 23 Nov 2006 16:19:49 +1300 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <20061123020054.GA31618@niemeyer.net> References: <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <20061122165513.GA13795@niemeyer.net> <4564F8B7.5030007@canterbury.ac.nz> <20061123020054.GA31618@niemeyer.net> Message-ID: <45651355.7090605@canterbury.ac.nz> Gustavo Niemeyer wrote: > The user of both libraries may define how Apple adapts to > IFruit. Sure, but then there's no need for a formal framwork for defining interfaces and adaptation -- by making use of duck typing, the joint user of the two libraries can wrap things as needed to make it all work. The proponents of adaptation seem to be making a stronger claim, however -- that somehow you can just register an adaptor and have all users of IFruits automatically understand Apples. That's the part that I fail to understand. -- Greg From greg.ewing at canterbury.ac.nz Thu Nov 23 04:19:53 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 23 Nov 2006 16:19:53 +1300 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> Message-ID: <45651359.8020800@canterbury.ac.nz> Steven Bethard wrote: > So I would have expected something like: > > str.lower[MyStringType] = MyStringType.lower > str.split[MyStringType] = MyStringType.split But that would only work if everyone switched to writing str.lower(s) everywhere instead of s.lower(), etc. In other words, abolish method call notation completely and use generic function calls instead. I would say that *is* a radical proposal... And it still wouldn't help SMTP.sendmail to tell whether it was dealing with a string or a list of strings. -- Greg From greg.ewing at canterbury.ac.nz Thu Nov 23 04:23:55 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 23 Nov 2006 16:23:55 +1300 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> Message-ID: <4565144B.5060109@canterbury.ac.nz> Guido van Rossum wrote: > I chose len(x) over x.len() for HCI reasons > > (a) For some operations, prefix notation just reads better than > postfix > (b) When I read code that says len(x) I *know* that it is asking for > the length of something. Just as a matter of interest, how far do you think these principles apply or don't apply to more recent things like iter.next() that don't follow this pattern? -- Greg From gsakkis at rutgers.edu Thu Nov 23 04:29:15 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Wed, 22 Nov 2006 22:29:15 -0500 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> Message-ID: <91ad5bf80611221929o4230fccfh663bf7a5bc5564d@mail.gmail.com> On 11/22/06, Guido van Rossum wrote: > On 11/22/06, George Sakkis wrote: > > Although I don't necessarily agree with the arguments from a puristic > > point of view ("then why not make keys(), values(), items(), read(), > > write(), ... builtin GFs ?"), they are indeed a better answer to the > > FAQ entry. After all, HCI-based arguments are usually a fancier way of > > saying "it's a matter of taste" and arguing against them is like > > arguing that green is a better colour than red. > > I'm so glad you're not trying to hide that you're a troll. I see how my reply could be misinterpreted as sarcastic but that was not at all my intention. If I were a troll,that would be pretty obvious after three years in c.l.py. I've supported the vast majority of language changes since 2.2-2.3 that I first met Python (even controversial ones such as decorators and conditional expressions) as well as pretty much all goals listed in PEP 3100. Still, we don't have to agree on everything, do we ? After all, what I said essentially was that beauty is in the eye of the beholder. That hardly justifies the troll labeling. George From greg.ewing at canterbury.ac.nz Thu Nov 23 04:33:34 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 23 Nov 2006 16:33:34 +1300 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> Message-ID: <4565168E.8000404@canterbury.ac.nz> Phillip J. Eby wrote: > Consider 'iter()', for example, which can be viewed as adapting an object > to the "iteration interface" and returning an object supporting iteration. An architecture astronaut might view it that way, but I don't. To my way of thinking, iter(x) creates a new object that iterates over x. Calling it a form of adaptation just muddies things with uneccessary words. (The fact that iter(x) returns x when it's already an iterator is a distraction. It's really just a kludge so that the same for-statement syntax can be used on both iterators and iterables.) -- Greg From guido at python.org Thu Nov 23 04:56:23 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 19:56:23 -0800 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <91ad5bf80611221929o4230fccfh663bf7a5bc5564d@mail.gmail.com> References: <-6683921950610559197@unknownmsgid> <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> <91ad5bf80611221929o4230fccfh663bf7a5bc5564d@mail.gmail.com> Message-ID: Your presence here has been entirely a distraction. That's a troll to me. On 11/22/06, George Sakkis wrote: > On 11/22/06, Guido van Rossum wrote: > > On 11/22/06, George Sakkis wrote: > > > Although I don't necessarily agree with the arguments from a puristic > > > point of view ("then why not make keys(), values(), items(), read(), > > > write(), ... builtin GFs ?"), they are indeed a better answer to the > > > FAQ entry. After all, HCI-based arguments are usually a fancier way of > > > saying "it's a matter of taste" and arguing against them is like > > > arguing that green is a better colour than red. > > > > I'm so glad you're not trying to hide that you're a troll. > > I see how my reply could be misinterpreted as sarcastic but that was > not at all my intention. If I were a troll,that would be pretty > obvious after three years in c.l.py. I've supported the vast majority > of language changes since 2.2-2.3 that I first met Python (even > controversial ones such as decorators and conditional expressions) as > well as pretty much all goals listed in PEP 3100. Still, we don't have > to agree on everything, do we ? After all, what I said essentially was > that beauty is in the eye of the beholder. That hardly justifies the > troll labeling. > > George > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ironfroggy at gmail.com Thu Nov 23 04:59:15 2006 From: ironfroggy at gmail.com (Calvin Spealman) Date: Wed, 22 Nov 2006 22:59:15 -0500 Subject: [Python-3000] defop ? In-Reply-To: <5.1.1.6.0.20061122180342.03ce3008@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> <456456CF.8000005@gmail.com> <5.1.1.6.0.20061122151242.03f68a88@sparrow.telecommunity.com> <5.1.1.6.0.20061122180342.03ce3008@sparrow.telecommunity.com> Message-ID: <76fd5acf0611221959g74e8d7a9o36723f63814aa950@mail.gmail.com> This whole thing seems a bit off from start to finish. A seperate definition syntax with a special name/expression weirdo thingy, etc. Anyway, we need something that handles this but isnt specific to this. There are other situations it may be useful. Essentially we want to have names that are known to be in some namespace, even within another. Does that make sense? Perhaps we could allow a special type of key in __dict__'s (and __slots__'s ?) that was more than a simple string name but included a namespace or context in which the name was to be understood. Even a simple 2-tuple would be enough. (operator, 'len') might be the entry for a method that the len builtin called, for example. A simple syntax would preceed the . operator to create names with this convention, maybe even using the familiar :: from other languages. class myclass(object): def operator::len(self): return self.length ... Very simple, can have more uses than the single, limited use case being discussed. On 11/22/06, Phillip J. Eby wrote: > At 02:06 PM 11/22/2006 -0800, Guido van Rossum wrote: > > > (The 'decorate_class' function applies its argument as a class decorator > > > for the class its caller is invoked from. This is the same mechanism (and > > > actual code!) that Zope uses to do in-body class decorators like > > > 'implements()'.) decorate_class raises a SyntaxError if its caller is not > > > being run directly inside a class body. > > > >That's a bizarre use of that exception. > > Well, the idea was that a class decorator can only be invoked inside a > class, so it's a syntax error to invoke it elsewhere. And, there's little > chance of confusing the SyntaxError with some error generated by *running* > the decorator itself, as opposed to merely its positioning. > > Anyway, it seemed like a good idea at the time. :) > > > >I'm guessing that decorate_class squirrels away some info for later? > > It introduces a stealth metaclass to the class frame, that will call the > decorator. If there's an existing metaclass in the frame, it invokes it > first to create the class. The devil is in the details, of course, and it > does require that you define any explicit metaclass *before* using any > class decorators, but this is a documented limitation of the DecoratorTools > library. > > > >Everything here seems designed to hide the implementation details, > >which is great for actual use, but kills understanding of the > >mechanism. :-( > > > > > And yes, this uses frame magic, and no, it's not fragile, as it knows the > > > difference between class and non-class frames. > > > >Wow. How can you tell? > > Basically, if f_locals contains a '__module__' key that's equal to the > '__name__' in f_globals, although there are some interesting corner cases > involving 'exec' -- the details are in the implementation. The relevant > function is the 'frameinfo(frame)' function which tells you what "kind" of > frame you have. > > > > Does it work with Jython, IronPython and PyPy? > > PyPy should be fine, since it uses the same code objects and > bytecode. IIRC, the others emulate enough of the frame protocol that the > '__module__' and '__name__' checks should be no problem. However, I have > not tried it in any of these. Since this hack requires metaclass support, > there was no point to attempting it in Jython. And when the code was > written, neither IronPython nor PyPy existed yet. > > > >I still find it fragile somehow -- or at least severely hackish. I > >like the other proposal (where it puts a special attribute on the > >function object which is then found by the type) better, it doesn't > >require sys._getframe(). > > Sure, there are certainly lots of other ways to implement it! What I > showed was just to demonstrate the *possibility*. If there were defop > syntax, then of course the compiler could simply generate the appropriate > bytecode following the MAKE_CLASS operation. Or MAKE_CLASS could look for > some data inserted into the source dictionary. > > decorate_class() is just what I had handy to answer your question - it > should not be confused with being the One Obvious Way to do it is. It's > just proof of possibility (and practical workability, in that it has been > long and widely used successfully with at least CPython). > > But, there are still other ways. The compiler could stick > "__owning_class__" attributes on any function defined in a class, and > __addmethod__ could defer actually registering the functions until the > generic function was invoked, thus ensuring that __owning_class__ would be > set first. > > In general, if we're allowed to change the implementation of Python, it's > easy to find some way to do it. But you asked for a way to do it in (I > thought) today's Python, so that's what I gave you. > > > >I'm assuming that's outside the context of the BinaryTree class -- > >since inside it, Binarytree is undefined. > > Correct. > > > >Hm, I think this is the first time in this thread that you meant to > >use 'iter', 'len' etc. as "tokens". > > I don't understand what you mean. They're objects, and so usable as > tokens. That's why I've also been saying they can also represent > "abilities", with defop being a way to unify their use as generic operation > (e.g. iter()), implementation (i.e. defop iter), and "interface" or ability > (e.g. 'hasmethod(iter,ob)' or something of that sort). > > > >I think I'd rather change them a > >bit so that they can be objects that actually understand __addmethod__ > >and __hasmethod__. > > Okay. That's an example of the kind of preference I have no way to divine > ahead of time. If I knew you preferred function attributes for deferred > registration, and to make these objects of a special type, I'd have > proposed it that way. :) My assumption was that changing the type of > Python builtin functions would be considered more "radical" than the idea > of defop, since it requires changing something, rather than just adding. > > > > > then there's no need to have > > > actual addmethod/hasmethod functions. But that's sort of like saying we > > > don't need getattr() any more because we can just use > > > type(ob).__getattribute__() now! :) > > > >But I think you're really proposing an addmethod builtin that calls an > >__addmethod__ method, right? > > Yes, assuming that we can call do this with regular functions too. That > is, as long as there's some way to retroactively overload a regular > function. Other than that, I'm not worried about the need to retrofit any > other existing generic function types; their authors will have to change > them for Py3K anyway. :) > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/ironfroggy%40gmail.com > -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/ From gsakkis at rutgers.edu Thu Nov 23 05:11:30 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Wed, 22 Nov 2006 23:11:30 -0500 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> <91ad5bf80611221929o4230fccfh663bf7a5bc5564d@mail.gmail.com> Message-ID: <91ad5bf80611222011j1ec3a872v478ccf6d1923f706@mail.gmail.com> If you consider distraction a thread that drew more than 100 replies, many of them very insightful (including Andrew Koenig's about abilities that has spawned 3-4 new threads), please allow me to disagree. On 11/22/06, Guido van Rossum wrote: > Your presence here has been entirely a distraction. That's a troll to me. > > On 11/22/06, George Sakkis wrote: > > On 11/22/06, Guido van Rossum wrote: > > > On 11/22/06, George Sakkis wrote: > > > > Although I don't necessarily agree with the arguments from a puristic > > > > point of view ("then why not make keys(), values(), items(), read(), > > > > write(), ... builtin GFs ?"), they are indeed a better answer to the > > > > FAQ entry. After all, HCI-based arguments are usually a fancier way of > > > > saying "it's a matter of taste" and arguing against them is like > > > > arguing that green is a better colour than red. > > > > > > I'm so glad you're not trying to hide that you're a troll. > > > > I see how my reply could be misinterpreted as sarcastic but that was > > not at all my intention. If I were a troll,that would be pretty > > obvious after three years in c.l.py. I've supported the vast majority > > of language changes since 2.2-2.3 that I first met Python (even > > controversial ones such as decorators and conditional expressions) as > > well as pretty much all goals listed in PEP 3100. Still, we don't have > > to agree on everything, do we ? After all, what I said essentially was > > that beauty is in the eye of the beholder. That hardly justifies the > > troll labeling. > > > > George > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/george.sakkis%40gmail.com > From guido at python.org Thu Nov 23 05:12:33 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 20:12:33 -0800 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <91ad5bf80611222011j1ec3a872v478ccf6d1923f706@mail.gmail.com> References: <-6683921950610559197@unknownmsgid> <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> <91ad5bf80611221929o4230fccfh663bf7a5bc5564d@mail.gmail.com> <91ad5bf80611222011j1ec3a872v478ccf6d1923f706@mail.gmail.com> Message-ID: It could have been a much shorter thread without you. On 11/22/06, George Sakkis wrote: > If you consider distraction a thread that drew more than 100 replies, > many of them very insightful (including Andrew Koenig's about > abilities that has spawned 3-4 new threads), please allow me to > disagree. > > On 11/22/06, Guido van Rossum wrote: > > > Your presence here has been entirely a distraction. That's a troll to me. > > > > On 11/22/06, George Sakkis wrote: > > > On 11/22/06, Guido van Rossum wrote: > > > > On 11/22/06, George Sakkis wrote: > > > > > Although I don't necessarily agree with the arguments from a puristic > > > > > point of view ("then why not make keys(), values(), items(), read(), > > > > > write(), ... builtin GFs ?"), they are indeed a better answer to the > > > > > FAQ entry. After all, HCI-based arguments are usually a fancier way of > > > > > saying "it's a matter of taste" and arguing against them is like > > > > > arguing that green is a better colour than red. > > > > > > > > I'm so glad you're not trying to hide that you're a troll. > > > > > > I see how my reply could be misinterpreted as sarcastic but that was > > > not at all my intention. If I were a troll,that would be pretty > > > obvious after three years in c.l.py. I've supported the vast majority > > > of language changes since 2.2-2.3 that I first met Python (even > > > controversial ones such as decorators and conditional expressions) as > > > well as pretty much all goals listed in PEP 3100. Still, we don't have > > > to agree on everything, do we ? After all, what I said essentially was > > > that beauty is in the eye of the beholder. That hardly justifies the > > > troll labeling. > > > > > > George > > > > > > > > > -- > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > _______________________________________________ > > Python-3000 mailing list > > Python-3000 at python.org > > http://mail.python.org/mailman/listinfo/python-3000 > > Unsubscribe: http://mail.python.org/mailman/options/python-3000/george.sakkis%40gmail.com > > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gsakkis at rutgers.edu Thu Nov 23 05:15:44 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Wed, 22 Nov 2006 23:15:44 -0500 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> <91ad5bf80611221929o4230fccfh663bf7a5bc5564d@mail.gmail.com> <91ad5bf80611222011j1ec3a872v478ccf6d1923f706@mail.gmail.com> Message-ID: <91ad5bf80611222015s6d41c612x7035169154283558@mail.gmail.com> Feel free to ignore it if you find it trivial/stupid/irrelevant; many other apparently didn't. On 11/22/06, Guido van Rossum wrote: > It could have been a much shorter thread without you. > > On 11/22/06, George Sakkis wrote: > > If you consider distraction a thread that drew more than 100 replies, > > many of them very insightful (including Andrew Koenig's about > > abilities that has spawned 3-4 new threads), please allow me to > > disagree. > > > > On 11/22/06, Guido van Rossum wrote: > > > > > Your presence here has been entirely a distraction. That's a troll to me. > > > > > > On 11/22/06, George Sakkis wrote: > > > > On 11/22/06, Guido van Rossum wrote: > > > > > On 11/22/06, George Sakkis wrote: > > > > > > Although I don't necessarily agree with the arguments from a puristic > > > > > > point of view ("then why not make keys(), values(), items(), read(), > > > > > > write(), ... builtin GFs ?"), they are indeed a better answer to the > > > > > > FAQ entry. After all, HCI-based arguments are usually a fancier way of > > > > > > saying "it's a matter of taste" and arguing against them is like > > > > > > arguing that green is a better colour than red. > > > > > > > > > > I'm so glad you're not trying to hide that you're a troll. > > > > > > > > I see how my reply could be misinterpreted as sarcastic but that was > > > > not at all my intention. If I were a troll,that would be pretty > > > > obvious after three years in c.l.py. I've supported the vast majority > > > > of language changes since 2.2-2.3 that I first met Python (even > > > > controversial ones such as decorators and conditional expressions) as > > > > well as pretty much all goals listed in PEP 3100. Still, we don't have > > > > to agree on everything, do we ? After all, what I said essentially was > > > > that beauty is in the eye of the beholder. That hardly justifies the > > > > troll labeling. > > > > > > > > George > > > > > > > > > > > > > -- > > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > > _______________________________________________ > > > Python-3000 mailing list > > > Python-3000 at python.org > > > http://mail.python.org/mailman/listinfo/python-3000 > > > Unsubscribe: http://mail.python.org/mailman/options/python-3000/george.sakkis%40gmail.com > > > > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/george.sakkis%40gmail.com > From guido at python.org Thu Nov 23 05:29:54 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 20:29:54 -0800 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <4565144B.5060109@canterbury.ac.nz> References: <-6683921950610559197@unknownmsgid> <4565144B.5060109@canterbury.ac.nz> Message-ID: On 11/22/06, Greg Ewing wrote: > Guido van Rossum wrote: > > I chose len(x) over x.len() for HCI reasons > > > > (a) For some operations, prefix notation just reads better than > > postfix > > (b) When I read code that says len(x) I *know* that it is asking for > > the length of something. > > Just as a matter of interest, how far do you > think these principles apply or don't apply > to more recent things like iter.next() that > don't follow this pattern? it.next() isn't something one does commonly; it's mostly implied by the for-loop. iter(seq) which was invented at the same time *is* a prefix op. There's probably a somewhat better rule hiding in all this; something that explains why keys() is a method. Perhaps it depends on the generality of the operation -- keys() is pretty unique to mappings, while len() and iter() apply to all containers. Similarly, next() only applies to iterators (even if there are about as many iterators as their are containers, they all have the same API). I guess the chances that something becomes an "operation" are better if it applies to a wide variety of types (that have otherwise different interfaces, unlike iterators which all have the same interface). Interestingly, keys() being a method makes it pretty much impossible to turn it into a generic function. This doesn't seem a big loss, it's unlikely one could implement it easily for a mapping type that doesn't provide keys() natively. OTOH, values() and items() could be implemented easily for anything that has keys() and __getitem__() (or iterkeys() and __getitem__()). But it's also easy to write an app without using values() and items() (using keys() and __getitem__()) so this is still not a big deal. One thing that rubs me the wrong way about generic functions is that it appears to go against OO. Now I'm not someone to take OO as religion, but there's something uncomfortable (for me) about how, in Phillip's world, many things become functions instead of methods, which brings along concerns about the global namespace filling up, and also about functionality being spread randomly across too many modules. I fear I will miss the class as a convenient focus for related functionality. While [@]defop moves the operation definitions back into the class, the invocations still takes the form of a function call. We could end up with two different mainstream notations: foo(x) and x.foo(), and which one is used for a particular operation is the result of a very subtle decision process that takes into account different evolution and/or usage patterns for the classes involved (as is shown by the difficulty we've had explaining why len() and iter() are functions but next() and keys() are methods, for example). Of course Python always had functions and methods, but the choice used to be so much simpler before generic functions: a function has a single implementation, a method can be overridden. I will miss that simplicity. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Thu Nov 23 05:34:25 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 23:34:25 -0500 Subject: [Python-3000] defop ? In-Reply-To: <76fd5acf0611221959g74e8d7a9o36723f63814aa950@mail.gmail.co m> References: <5.1.1.6.0.20061122180342.03ce3008@sparrow.telecommunity.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> <456456CF.8000005@gmail.com> <5.1.1.6.0.20061122151242.03f68a88@sparrow.telecommunity.com> <5.1.1.6.0.20061122180342.03ce3008@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122233205.01f446a8@sparrow.telecommunity.com> At 10:59 PM 11/22/2006 -0500, Calvin Spealman wrote: >Perhaps we could allow a special type of key in __dict__'s (and >__slots__'s ?) that was more than a simple string name but included a >namespace or context in which the name was to be understood. Even a >simple 2-tuple would be enough. (operator, 'len') might be the entry >for a method that the len builtin called, for example. A simple syntax >would preceed the . operator to create names with this convention, >maybe even using the familiar :: from other languages. > >class myclass(object): > def operator::len(self): > return self.length > ... > >Very simple, can have more uses than the single, limited use case >being discussed. So, how does that work for adding methods after the fact, *outside* of a class? (See Guido's BinaryTree flattening example.) From steven.bethard at gmail.com Thu Nov 23 05:38:09 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 22 Nov 2006 21:38:09 -0700 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <45651359.8020800@canterbury.ac.nz> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> <45651359.8020800@canterbury.ac.nz> Message-ID: On 11/22/06, Greg Ewing wrote: > Steven Bethard wrote: > > So I would have expected something like: > > > > str.lower[MyStringType] = MyStringType.lower > > str.split[MyStringType] = MyStringType.split > > But that would only work if everyone switched to > writing str.lower(s) everywhere instead of s.lower(), I don't think so, but perhaps I misunderstood the proposal. I thought it would just rely on someone defining a String Interface as the union of ``str.lower``, ``str.split``, etc. Then to figure out what to do with your ``string_or_list`` argument, you'd test it against this String Interface. You could then freely make calls like ``s.lower()``, knowing that they were calling the equivalent of ``str.lower()``. Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From guido at python.org Thu Nov 23 05:42:34 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Nov 2006 20:42:34 -0800 Subject: [Python-3000] defop ? In-Reply-To: <76fd5acf0611221959g74e8d7a9o36723f63814aa950@mail.gmail.com> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> <456456CF.8000005@gmail.com> <5.1.1.6.0.20061122151242.03f68a88@sparrow.telecommunity.com> <5.1.1.6.0.20061122180342.03ce3008@sparrow.telecommunity.com> <76fd5acf0611221959g74e8d7a9o36723f63814aa950@mail.gmail.com> Message-ID: On 11/22/06, Calvin Spealman wrote: > This whole thing seems a bit off from start to finish. A seperate > definition syntax with a special name/expression weirdo thingy, etc. I have the same gut feelings but find it hard to explain why. But I've learned to trust my gut -- eventually it will come to me. Perhaps it's okay for the operation definitions to be outside the class for which they apply? That at least clarifies that their *invocation* also doesn't involve method notation. > Anyway, we need something that handles this but isnt specific to this. > There are other situations it may be useful. Essentially we want to > have names that are known to be in some namespace, even within > another. Does that make sense? > > Perhaps we could allow a special type of key in __dict__'s (and > __slots__'s ?) that was more than a simple string name but included a > namespace or context in which the name was to be understood. Even a > simple 2-tuple would be enough. (operator, 'len') might be the entry > for a method that the len builtin called, for example. A simple syntax > would preceed the . operator to create names with this convention, > maybe even using the familiar :: from other languages. > > class myclass(object): > def operator::len(self): > return self.length > ... > > Very simple, can have more uses than the single, limited use case > being discussed. Not sure I like this better. My spelling of "operator::XXX" is "__XXX__" and I like that just fine -- no new syntax needed. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Thu Nov 23 05:47:12 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Nov 2006 23:47:12 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <4565168E.8000404@canterbury.ac.nz> References: <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061122233433.028ea230@sparrow.telecommunity.com> At 04:33 PM 11/23/2006 +1300, Greg Ewing wrote: >Phillip J. Eby wrote: > > > Consider 'iter()', for example, which can be viewed as adapting an object > > to the "iteration interface" and returning an object supporting iteration. > >An architecture astronaut might view it that way, but Calling people names isn't particularly conducive to a technical discussion... or any other kind of discussion, for that matter. :) >I don't. To my way of thinking, iter(x) creates a new >object that iterates over x. Calling it a form of >adaptation just muddies things with uneccessary words. You're right. We should never reuse words to describe similar things, because then people would be confused. We should only ever invent *new* words, because everything is different from everything else. ;-) More seriously, the purpose of descriptive words like "adapting" is to allow comparison and the discovery of patterns. If iter() were unique in all the world, it would make no sense to have a special word that applied only to it. But it's far from unique... >(The fact that iter(x) returns x when it's already an >iterator is a distraction. It's really just a kludge >so that the same for-statement syntax can be used on >both iterators and iterables.) That kludge, as you call it, is something called idempo... whoops, I almost used an architecture word there! Whew, that was close! :) Suffice to say, however, that it is a property shared by other builtins such as str, int, tuple, and bool, and that many perverted Python developers actually believe this unspeakable idea is a highly desirable property in a builtin function or type! Let's not tell them it's really a "kludge", though, 'cause it might hurt their feelings. :) crudely-attempting-Tim-Peters-style-humor'ly yours, --PJE From pje at telecommunity.com Thu Nov 23 07:01:18 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 23 Nov 2006 01:01:18 -0500 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <4565144B.5060109@canterbury.ac.nz> <-6683921950610559197@unknownmsgid> <4565144B.5060109@canterbury.ac.nz> Message-ID: <5.1.1.6.0.20061122234926.02d249e0@sparrow.telecommunity.com> At 08:29 PM 11/22/2006 -0800, Guido van Rossum wrote: >One thing that rubs me the wrong way about generic functions is that >it appears to go against OO. Now I'm not someone to take OO as >religion, but there's something uncomfortable (for me) about how, in >Phillip's world, many things become functions instead of methods, >which brings along concerns about the global namespace filling up, and >also about functionality being spread randomly across too many >modules. I fear I will miss the class as a convenient focus for >related functionality. I originally proposed a solution for this back in January '05, but it was too premature. But since you have now stated the problem that the proposal was intended to solve, perhaps the solution has a chance now. :) I will try to be as concrete as possible. Let's start with an actual, hopefully non-exploding 'Interface' implementation, based on an assumption that we have generic functions available: class InterfaceClass(type): def __init__(cls, name, bases, cdict): for k,v in cdict.items(): # XXX this should probably skip at least __slots__, # __metaclass__, and __module__, but oh well cdict[k] = AdaptingDescriptor(v) class Interface: __metaclass__ = InterfaceClass __slots__ = '__self__' def __init__(self, subject): # this isinstance() check should be replaced by an # 'unwrap()' generic function, so other adapter types # will work, but this is just an example, so... if isinstance(subject, Interface): subject = subject.__self__ self.__self__ = subject class AdaptingDescriptor: def __init__(self, descriptor): self.wrapped = descriptor def __get__(self, ob, typ=None): if ob is None: return self return self.wrapped.__get__(ob.__self__, typ) Now, using this new "interface framework", let's implement a small "mapping" typeclas... er, interface. class Mapping(Interface): def keys(self): return [k for k,v in self.items()] def items(self): return [k,self[k] for k in self.keys()] # ... other self-recursive definitions What does this do? Well, we can now call Mapping(foo) to turn an arbitrary object into something that has Mapping's generic functions as its methods, and invokes them on foo! (I am assuming here that normal functions are implicitly overloadable, even if that means they change type at runtime to do so.) We could even use interfaces for argument type declarations, to automatically put things in the "right namespace" for what the code expects to use. That is, if you declare an argument to be a Mapping, then that's what you get. If you call .keys() on the resulting adapted object and the type doesn't support the operation, you get an error. Too late a form of error checking you say? Well, make a more sophisticated factory mechanism in Interface.__new__ that actually creates (and caches) different adapter types based on the type of object being adapted, so that hasattr() tests will work on the wrapped type, or so that you can get an early error if none of the wrapped generic functions has a method defined for the target type. A few important points here: 1. A basic interface mechanism is extemely simple to implement, given generic functions 2. It is highly customizable with respect to error checking and other features, even on a per-user basis, because there doesn't have to be only one "true" Interface type to rule them all (or one true generic function type either, but that's a separate discussion). 3. It allows interfaces to include partial implementations, ala Ping and Alex's past proposals, thus allowing you to implement partial mapping or "file" objects and have the rest of the interface's implementation filled in for you 4. It allows you to hide the very existence of the notion of a "generic function", if you prefer not to think about such things 5. It even supports interface inheritance and interface algebra: subclassing an interface allows adding new operations, and simple assignment suffices to compose new interfaces, e.g.: class MappingItems(Interface): items = Mapping.items Notice that nothing special is required, this "just works" as a natural consequence of the rest of the implementation shown. Okay, so now you want to know how to *implement* a "Mapping". Well, simplest but most tedious, you can just register operations directly, e.g.: class MyMapping: def __init__(self, data): self.data = dict(data) defop operator.getitem(self, key): return self.data[key] defop Mapping.items(self): return self.data.items() But as you can imagine, this would probably get a bit tedious if you're implementing lots of methods. So, we can add metaclasses or class decorators here to say, "I implement these interfaces, so any methods I have whose names match the method names in the interfaces, please hook 'em up for me." I'm going to leave out the implementation, as it should be a straightforward exercise for the reader to come up with many ways by which it can be accomplished. The spelling might be something like: class MyMapping: implements(Mapping) def items(self): ... #etc. At which point, we have now come full circle to being able to provide all of the features of interfaces, adaptation, and generic functions, without forcing anyone to give up the tasty OO flavor of method calls. Heck, they can even keep the way they spell existing adaptation calls (e.g. IFoo(bar) to adapt bar to IFoo) in PEAK, Twisted, and Zope! And finally, note that if you only want to perform one method call on a given object, you can also use the generics directly, e.g. Mapping.items(foo) instead of Mapping(foo).items(). Voila -- generic goodness and classic OO method-calling simplicity, all in one simple to implement package. It should now be apparent why I said that interfaces are trivial to implement if you define them as namespaces for generic functions, rather than as namespaces for methods. There are many spinoffs possible, too. For example, you could have a factory function that turns an existing class's public operations into an interface object. There are also probably also some dark corners of the idea that haven't been explored, because when I first proposed basically this idea in '05, nobody was ready for it. Now maybe we can actually talk about the implications. From ironfroggy at gmail.com Thu Nov 23 08:10:57 2006 From: ironfroggy at gmail.com (Calvin Spealman) Date: Thu, 23 Nov 2006 02:10:57 -0500 Subject: [Python-3000] Fwd: defop ? In-Reply-To: <76fd5acf0611222310x64f03aa3i3d516c30452b1d79@mail.gmail.com> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> <456456CF.8000005@gmail.com> <5.1.1.6.0.20061122151242.03f68a88@sparrow.telecommunity.com> <5.1.1.6.0.20061122180342.03ce3008@sparrow.telecommunity.com> <76fd5acf0611221959g74e8d7a9o36723f63814aa950@mail.gmail.com> <76fd5acf0611222310x64f03aa3i3d516c30452b1d79@mail.gmail.com> Message-ID: <76fd5acf0611222310h1f6e4137r1dee4afa7a747ef0@mail.gmail.com> I need to move away from the GMail client for lists or they need to fix it. Sorry Guido, forwarding to the list as GMail should be doing in the first place! ---------- Forwarded message ---------- From: Calvin Spealman Date: Nov 23, 2006 2:10 AM Subject: Re: [Python-3000] defop ? To: Guido van Rossum On 11/22/06, Guido van Rossum wrote: > Not sure I like this better. My spelling of "operator::XXX" is > "__XXX__" and I like that just fine -- no new syntax needed. Well, that might be a spelling of `operator::XXX` but what about other use cases like `third_party_interface_system::adapt` or `anything_at_all::XXX`? Thats what I mean with not being too limiting and solving the problem in a way that opens up solutions to other problems. I get the opposition to it, but it seems reasonable, nonetheless. On 11/22/06, Phillip J. Eby wrote: > So, how does that work for adding methods after the fact, *outside* of a > class? (See Guido's BinaryTree flattening example.) class MyClass(object): pass MyClass.operator::iter = lambda self: iter((self.a, self.b, self.c)) Or, possibly: def MyClass.operator::iter(self): yield a yield b yield c But, I understand if that would get even more opposition, so I won't push it, but its on the table. Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/ -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/ From fredrik at pythonware.com Thu Nov 23 08:23:40 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 23 Nov 2006 08:23:40 +0100 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <91ad5bf80611221908t4e1ed686yb94d700d1bbc0cb5@mail.gmail.com> References: <-6683921950610559197@unknownmsgid> <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> <91ad5bf80611221908t4e1ed686yb94d700d1bbc0cb5@mail.gmail.com> Message-ID: George Sakkis wrote: > First off, I never implied someone's stupid just because we don't > happen to agree on everything. brushing off a carefully thought out design and the process that led up to it with "it's just like picking between two random colors" is a pretty direct way of saying that you consider yourself so much smarter than the people involved that you don't even care about what they're doing. > As for Mike's answer, I went back and > read it again; please do the same. He doesn't address HCI reasons at > all, it's a purely technical argument by pointing out an inconsistency > of Java (which kinda distracts from the main point but that's ok). HCI is all about picking the *right* technical solutions, though. Python's designed for humans, and humans aren't robots; the solutions you choose and the way you integrate them affect how people understands them and use them, on many different levels. what Mike wrote about was one such case (which follows from 1b in GvR's post), where Python's way of doing things helped people avoid a certain real-life problem, without even noticing. seriously, if you think that HCI (or for that matter, design) is only about what color you use for a technical solution designed by real engineers, you've pretty much missed the point of the whole field. and you've pretty much missed the point of Python's design. From talin at acm.org Thu Nov 23 08:44:02 2006 From: talin at acm.org (Talin) Date: Wed, 22 Nov 2006 23:44:02 -0800 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> <91ad5bf80611221908t4e1ed686yb94d700d1bbc0cb5@mail.gmail.com> Message-ID: <45655142.7050800@acm.org> Fredrik Lundh wrote: > George Sakkis wrote: > >> First off, I never implied someone's stupid just because we don't >> happen to agree on everything. > > brushing off a carefully thought out design and the process that led > up to it with "it's just like picking between two random colors" is a > pretty direct way of saying that you consider yourself so much smarter > than the people involved that you don't even care about what they're doing. That's not what he said though. If you go back and read his original posting, he was commenting on the quality of *most arguments* about HCI, not HCI itself, or all HCI-related discussions. While I wouldn't say it quite so strongly, I do think that there is a great deal of subjectivity in HCI discussions. One person's 'intuitive' may be another person's 'orange smoke', as we've seen in the current discussion. I'm not saying that such discussion don't have merit, but we must be careful not to take our own sense of 'obviousness' too seriously. And that's what I think George was trying to say. -- Talin From fredrik at pythonware.com Thu Nov 23 08:52:04 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 23 Nov 2006 08:52:04 +0100 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122233433.028ea230@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <4565168E.8000404@canterbury.ac.nz> <5.1.1.6.0.20061122233433.028ea230@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > That kludge, as you call it, is something called idempo... whoops, I > almost used an architecture word there! No, that's a mathematical term. And while Greg wasn't using that word, he was actually talking about its CS form, where it means something similar, but not entirely equivalent. In math, something is idempotent if you can change f(x) to f(f(x)) and still get the same result; in CS, something is idempotent if you can do f(x) twice on the same x, and get the same result both times. Python's len() function is idempotent in the CS sense, but not in the math sense. And Python's iter() is not necessarily idempotent in the CS sense when used on an *iterable*, but it is, in both senses, when used on an iterator. Which was Greg's point, I think. > We should never reuse words to describe similar things, because then > people would be confused. Well, I'd say you just illustrated the danger of using the same word to described two similar but yet distinctly different concepts. From bingham at cenix-bioscience.com Thu Nov 23 08:58:38 2006 From: bingham at cenix-bioscience.com (Aaron Bingham) Date: Thu, 23 Nov 2006 08:58:38 +0100 Subject: [Python-3000] Builtin iterator type In-Reply-To: <45631A5F.5080501@gmail.com> References: <91ad5bf80611152232m2e375225vcc5dea6665effb1@mail.gmail.com> <002901c70b54$9b1dd350$6402a8c0@arkdesktop> <455FEEBB.9010100@cs.byu.edu> <20061119115009.V14417@tribble.ilrt.bris.ac.uk> <06Nov19.170737pst."58648"@synergy1.parc.xerox.com> <4561857A.8080101@cenix-bioscience.com> <45631A5F.5080501@gmail.com> Message-ID: <456554AE.1050803@cenix-bioscience.com> Nick Coghlan wrote: > Aaron Bingham wrote: > >> Bill Janssen wrote: >> >>>> Java interfaces are very useful, however. Java programming seems to >>>> be less and less about inheritance and more and more about >>>> implementing interfaces; at least it does amongst Java programmers >>>> with taste :-) >>>> >>> >>> It seems to me that that's where Python has a real advantage. With >>> real support for multiple inheritance, Python "interfaces" could be >>> real classes (either like real Java classes or Java abstract classes), >>> perhaps providing default implementations. You get the goodness of >>> mix-ins, along with interface communication. >>> >> I agree. In Java, interfaces are necessary because multiple >> inheritance is not supported. I see no good reason to add an >> additional language mechanism for interfaces when multiple >> inheritance would do the job, AFAICT. > > > Just because multiple inheritance is *possible* in Python, don't make > the mistake of thinking that it is straight forward. Aside from mixin > classes that themselves inherit directly from object, multiple > inheritance can get very messy outside of tightly controlled type > hierarchies (cooperative calls in particular can become a nightmare). > This flies in the face of Python's use as a glue language to tie > multiple components together (particularly given the problem that some > of the elements being integrated may come from environments that > *don't* support multiple inheritance, like the JVM). It's true that multiple inheritance in Python can get quite messy. Python lacks two mechanisms to control this complexity that are, AFAIK, only available in Eiffel: renaming and hiding of superclass attributes and methods. Implementing these mechanisms in a dynamically typed languages is AFAICT impossible. I won't go into a detailed explanation here unless someone requests it. Regards, -- -------------------------------------------------------------------- Aaron Bingham Senior Software Engineer Cenix BioScience GmbH -------------------------------------------------------------------- From fredrik at pythonware.com Thu Nov 23 08:59:39 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 23 Nov 2006 08:59:39 +0100 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <45655142.7050800@acm.org> References: <-6683921950610559197@unknownmsgid> <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> <91ad5bf80611221908t4e1ed686yb94d700d1bbc0cb5@mail.gmail.com> <45655142.7050800@acm.org> Message-ID: Talin wrote: >> brushing off a carefully thought out design and the process that led >> up to it with "it's just like picking between two random colors" is a >> pretty direct way of saying that you consider yourself so much smarter >> than the people involved that you don't even care about what they're doing. > > That's not what he said though. If you go back and read his original > posting, he was commenting on the quality of *most arguments* about HCI, > not HCI itself, or all HCI-related discussions. same thing. he didn't get the argument, therefore it didn't have the right quality, and therefore it's not very important, and both the argument and the person making it can be ignored. it's a circular argument. and the only way to snap out of it is to start asking yourself "I don't get his argument, but he definitely has a track record; maybe that guy knows something that I don't". From fredrik at pythonware.com Thu Nov 23 09:04:12 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 23 Nov 2006 09:04:12 +0100 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122184343.02871500@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> <5.1.1.6.0.20061122184343.02871500@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: >> (but I'm suspect that my view of "generics" is somewhat different from >> Phillips, so let's focus on fully understanding his proposal first) > > So now I'm curious. Me too. Let me see if I can figure out what it is, well enough to write it down. > Maybe it's better. I'm pretty sure we're close from a *technological* perspective (at least on the "internal mechanism" level. But I'm not sure we're close from a *conceptual* perspective. Or something. I'll think about it. From thomas at thomas-lotze.de Thu Nov 23 08:47:35 2006 From: thomas at thomas-lotze.de (Thomas Lotze) Date: Thu, 23 Nov 2006 08:47:35 +0100 Subject: [Python-3000] Abilities / Interfaces References: <20061122202924.GA21361@niemeyer.net> Message-ID: Guido van Rossum wrote: >> I'm sure you're aware about it, but in Zope 3 terminology, these are >> 'provide' and 'implement', respectively. > > I like provide, but I'm not so sure about implement, since it is awfully > ambiguous -- most of the time it is the class that does the implementing. > That's why I settled for "has". JFTR: In Zope3, classes "implement" interfaces while instances "provide" them. -- Thomas From fredrik at pythonware.com Thu Nov 23 09:05:59 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 23 Nov 2006 09:05:59 +0100 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <06Nov22.181438pst."58648"@synergy1.parc.xerox.com> References: <-6683921950610559197@unknownmsgid> <8853879015409334245@unknownmsgid> <06Nov22.181438pst."58648"@synergy1.parc.xerox.com> Message-ID: Bill Janssen wrote: >> Ow, that's the first time you describe this particular wrinkle. > > Yep. I hesitated to bring it up, but if there's a need for separate > namespaces for method names, why not do it right? what makes you so sure that Python's current design isn't "right" ? From fredrik at pythonware.com Thu Nov 23 09:09:06 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 23 Nov 2006 09:09:06 +0100 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <4565144B.5060109@canterbury.ac.nz> References: <-6683921950610559197@unknownmsgid> <4565144B.5060109@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Just as a matter of interest, how far do you > think these principles apply or don't apply > to more recent things like iter.next() that > don't follow this pattern? this has been discussed before, I think. googling for __next__ might bring up some python-dev discussions. From talin at acm.org Thu Nov 23 09:35:02 2006 From: talin at acm.org (Talin) Date: Thu, 23 Nov 2006 00:35:02 -0800 Subject: [Python-3000] defop ? In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> <456456CF.8000005@gmail.com> <5.1.1.6.0.20061122151242.03f68a88@sparrow.telecommunity.com> <5.1.1.6.0.20061122180342.03ce3008@sparrow.telecommunity.com> <76fd5acf0611221959g74e8d7a9o36723f63814aa950@mail.gmail.com> Message-ID: <45655D36.9060809@acm.org> Guido van Rossum wrote: > On 11/22/06, Calvin Spealman wrote: >> This whole thing seems a bit off from start to finish. A seperate >> definition syntax with a special name/expression weirdo thingy, etc. > > I have the same gut feelings but find it hard to explain why. > > But I've learned to trust my gut -- eventually it will come to me. Here's my understanding of the rationale for defop: From an implementation standpoint, everything that defop does can be emulated by an appropriate set of function decorators. However, when we start getting into things like namespaces and operators, some of those decorators start to look pretty ugly and have unwanted side-effects. If all we wanted to do was to be able to create generic functions in the global namespace, the decorator syntax could be fairly simple: @overload def len( x : list ): ... The decorator can inspect the function object to discover the name of the function - so there's no need to pass a separate argument to the decorator telling it what function to overload: @overload( len ) def len( x : list ): ... This syntax is less desirable since it violates DRY. In either case, what the decorator returns is not the reference to the function object, but rather the reference to the generic dispatcher object, 'len'. The result of the decorator will then be bound to the name 'len' in that scope. (It means that for every overload, the name 'len' gets rebound to the same dispatcher object over and over.) Now, all that's fine and dandy as long as we limit ourselves to only creating global functions with names that conform to the syntax of Python identifiers. But what if we don't want to clutter up the global namespace? Suppose we want to create a generic function in some other namespace, such as an attribute of an object. Now in order to define the overload, we have to do something like this: @overload( sequtils.len ) def len( x : list ): ... The problem here is that the function object is going to get bound to the name 'len' no matter what - the decorator can't control that. Even if we go ahead and add the function object as a method of 'sequtils.len', we still, as a side-effect, end up binding the result of the decorator to 'len', which will at best create a spurious symbol, and at worse overwrite something we actually wanted to keep. Taking this one step further, suppose we wanted to get rid of the __special__ names for overloaded operators, and instead be able to use the actual symbol for the operator. Suppose there was a built-in dictionary of operators, where the keys of this dictionary was the actual operators themselves. So 'infix['+']' would be the expression for the generic infix addition operator. You could easily create a decorator that takes the operator name as an argumemt, similar to @overload above: @overload_infix('+') def add_bool_to_bool( a : bool, b : bool ): ... But again, you have to deal with the unwanted function name. And you can't embed the '+' symbol in the function name itself, since '+' isn't allowed in Python identifiers. (Maybe not the best of examples, but it illustrates the idea that perhaps not all generic functions will have simple names.) Now, if we had anonymous functions (hint, hint :) ) we could say something along the lines of: infix( '+' ) += def( a : bool, b: bool ): ... (Not the prettiest syntax I know...but at least it gets around the 'tyranny of naming') I suppose you could do that with lambda now, if your implementation was limited to a single expression with no statements (although that's a question - will lambda support decorated arguments?) So my understanding is that 'defop' is a way to get around all of these issues. 'defop' is nothing more than 'def', except that it has a different rule as to how the function object is bound to a name. (PJE, do I have this right?) All that being said - I'm not sure that 'defop' is the right name for it, or that it's even the right way to solve it. But I hope that my explanation (if it's correct) may help clear up the discussion a bit. -- Talin From p.f.moore at gmail.com Thu Nov 23 09:45:14 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 23 Nov 2006 08:45:14 +0000 Subject: [Python-3000] defop ? In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> <456456CF.8000005@gmail.com> <5.1.1.6.0.20061122151242.03f68a88@sparrow.telecommunity.com> <5.1.1.6.0.20061122180342.03ce3008@sparrow.telecommunity.com> <76fd5acf0611221959g74e8d7a9o36723f63814aa950@mail.gmail.com> Message-ID: <79990c6b0611230045w5c2c95e6l97e51cb7ae1c230@mail.gmail.com> On 11/23/06, Guido van Rossum wrote: > On 11/22/06, Calvin Spealman wrote: > > This whole thing seems a bit off from start to finish. A seperate > > definition syntax with a special name/expression weirdo thingy, etc. > > I have the same gut feelings but find it hard to explain why. > > But I've learned to trust my gut -- eventually it will come to me. I agree entirely. The whole defop thing seems odd (and I *hate* the name "defop"). I fail to see what it gains over a decorator syntax such as is already available in the 3 generic function packages available (RuleDispatch/PEAK.Rules, simplegeneric, and Guido's version in the sandbox). Without defop, the proposal seems to boil down to sanctioning a standard library addition which provides a generic function package - *possibly* with some modification of existing standard library and builtin functions to become generics. I'm sure I'm missing something, because the proposal feels alternately overcomplex and trivial to me... Paul. From p.f.moore at gmail.com Thu Nov 23 09:56:22 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 23 Nov 2006 08:56:22 +0000 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122185122.03e49500@sparrow.telecommunity.com> References: <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> <5.1.1.6.0.20061122130031.041f4960@sparrow.telecommunity.com> <5.1.1.6.0.20061122145257.03be18b8@sparrow.telecommunity.com> <5.1.1.6.0.20061122160753.03cd51d0@sparrow.telecommunity.com> <5.1.1.6.0.20061122185122.03e49500@sparrow.telecommunity.com> Message-ID: <79990c6b0611230056i5fcf0ac3td46e8df7f2d73f12@mail.gmail.com> On 11/23/06, Phillip J. Eby wrote: > So, the generic function 'as_string' here functions as an "interface" or > "ability". Of course, instead of being called 'as_string', it could live > as an attribute of the 'str' type, e.g. 'str.cast': OK, I think I just "clicked" with what defop is for. It's to allow you to define additional behaviour for methods, etc, where the normal def syntax won't allow something like str.cast? Sorry if that description is a bit vague - it reflects a feeling of "why bother?" Is defop str.cast(s: MyClass): ... that much better than @str.cast.when(MyClass) def _(s): ... given that Guido doesn't like "when", so a better name may end up being used, and the slight namespace pollution of having to come up with a name to use for the function (I used _ here, maybe there's a case for allowing def without a name which passes the anonymous function to the decorator but does no assignment to names...) I could quite happily live with the decorator form as it stands though - new syntax seems like overkill. Paul From gsakkis at rutgers.edu Thu Nov 23 11:38:53 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Thu, 23 Nov 2006 05:38:53 -0500 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <45655142.7050800@acm.org> References: <-6683921950610559197@unknownmsgid> <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> <91ad5bf80611221908t4e1ed686yb94d700d1bbc0cb5@mail.gmail.com> <45655142.7050800@acm.org> Message-ID: <91ad5bf80611230238u50c770a1kaba0f2431318f941@mail.gmail.com> On 11/23/06, Talin wrote: > Fredrik Lundh wrote: > > George Sakkis wrote: > > > >> First off, I never implied someone's stupid just because we don't > >> happen to agree on everything. > > > > brushing off a carefully thought out design and the process that led > > up to it with "it's just like picking between two random colors" is a > > pretty direct way of saying that you consider yourself so much smarter > > than the people involved that you don't even care about what they're doing. > > That's not what he said though. If you go back and read his original > posting, he was commenting on the quality of *most arguments* about HCI, > not HCI itself, or all HCI-related discussions. > > While I wouldn't say it quite so strongly, I do think that there is a > great deal of subjectivity in HCI discussions. One person's 'intuitive' > may be another person's 'orange smoke', as we've seen in the current > discussion. > > I'm not saying that such discussion don't have merit, but we must be > careful not to take our own sense of 'obviousness' too seriously. And > that's what I think George was trying to say. Exactly. My point was not against HCI in general, or Python's in specific; most probably it wouldn't be my primary language for the last three years if it didn't "feel right" in so many ways. What I'm saying is that I can't claim it *is* right because it *feels* right to *me*. And to come back to len(), many people value uniformity and consistency in a language. While I am not an absolute purist, I don't see the gain in practicality and/or elegance of having len() a function instead of method to justify the special case (at least in today's python; generic functions may change this). Guido's point about the generality of the operation (keys() applies to mappings, len() and iter() apply to containers) raises the question of where to set the "generality threshold" between functions and methods. Is it absolutely wrong to set it higher than the container level (e.g. at the object level) or lower (e.g. at sequence or mapping) ? I don't think it's either wrong or right; it's subjective. That's all I am saying. George From gustavo at niemeyer.net Thu Nov 23 13:14:17 2006 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Thu, 23 Nov 2006 10:14:17 -0200 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <45651355.7090605@canterbury.ac.nz> References: <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <20061122165513.GA13795@niemeyer.net> <4564F8B7.5030007@canterbury.ac.nz> <20061123020054.GA31618@niemeyer.net> <45651355.7090605@canterbury.ac.nz> Message-ID: <20061123121416.GA6072@niemeyer.net> > Sure, but then there's no need for a formal framwork > for defining interfaces and adaptation -- by making > use of duck typing, the joint user of the two libraries > can wrap things as needed to make it all work. You can always do everything in another way, just like you don't need OO to be able to develop something. > The proponents of adaptation seem to be making a > stronger claim, however -- that somehow you can just > register an adaptor and have all users of IFruits > automatically understand Apples. That's the part that > I fail to understand. If you have an adapter from Apple to IFruit, any user requiring an IFruit interface for an object will be able to obtain it from Apple instances. The "automatically" part is not quite true, as adaptation must be explicitly requested. This is true for generic functions as well, except that instead of using an IFruit interface, you have a fruit() generic function which handles the adaptation. Unfortunately, this model doesn't work with more complex hierachies. For instance: >>> from zope import interface, component >>> class Apple(object): pass >>> class IFruit(Interface): pass >>> class ITastyFruit(IFruit): pass >>> def to_tasty_fruit(obj): return "I'm actually a tasty fruit" >>> component.provideAdapter(to_tasty_fruit, (Apple,), ITastyFruit) >>> IFruit(Apple()) "I'm actually a tasty fruit" -- Gustavo Niemeyer http://niemeyer.net From gustavo at niemeyer.net Thu Nov 23 13:45:48 2006 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Thu, 23 Nov 2006 10:45:48 -0200 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <4565168E.8000404@canterbury.ac.nz> References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <4565168E.8000404@canterbury.ac.nz> Message-ID: <20061123124548.GB6072@niemeyer.net> > > Consider 'iter()', for example, which can be viewed as adapting an > > object to the "iteration interface" and returning an object > > supporting iteration. > > An architecture astronaut might view it that way, but > I don't. To my way of thinking, iter(x) creates a new > object that iterates over x. Calling it a form of > adaptation just muddies things with uneccessary words. FWIW, Phillip was pointing out the following similarity: >>> class IIterable(Interface): pass ... >>> class IIter(Interface): pass ... >>> class C(object): ... interface.implements(IIterable) ... def __iter__(self): return iter("foo") ... >>> def adapt_to_iter(obj): ... return obj.__iter__() ... >>> component.provideAdapter(adapt_to_iter, (IIterable,), IIter) >>> >>> iter(C()).next() 'f' >>> IIter(C()).next() 'f' As mentioned before, the similarity doesn't hold for more complex interface hierarchies. -- Gustavo Niemeyer http://niemeyer.net From gustavo at niemeyer.net Thu Nov 23 14:02:43 2006 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Thu, 23 Nov 2006 11:02:43 -0200 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061121174806.01f19c50@sparrow.telecommunity.com> References: <91ad5bf80611211418o76941e13l6864e97e2df23154@mail.gmail.com> <91ad5bf80611211418o76941e13l6864e97e2df23154@mail.gmail.com> <5.1.1.6.0.20061121174806.01f19c50@sparrow.telecommunity.com> Message-ID: <20061123130243.GC6072@niemeyer.net> > I was under the impression that Zope's actual use cases for > instance-specific interfaces have mainly to do with *view* lookups and > other types of lookups that don't really come into the current > discussion scope as I understand it. That is, I thought that instance > interfaces were used primarily for "n-tuple adaptation" (the > adaptation equivalent of multiple and/or predicate dispatch)? While both, instance-specific interfaces and multiple adaptation, are used in Zope 3 and considered very useful, their usefulness is not dependent on each other. The whole concept surely becomes more interesting with both in place, of course. (...) > :) I just want a minimal core that blesses what the language and > stdlib *already* do (special methods and single-dispatch __mro__ > lookups), while still allowing "advanced" paradigms (Zope, > RuleDispatch, etc.) to "play on a level playing field" with each > other. What the standard library is already doing doesn't need blessing, and neither is blessing needed for allowing advanced paradigms elsewhere, since they're already in place right now. -- Gustavo Niemeyer http://niemeyer.net From python3000 at davious.org Thu Nov 23 13:14:59 2006 From: python3000 at davious.org (Dave Anderson) Date: Thu, 23 Nov 2006 07:14:59 -0500 Subject: [Python-3000] Abilities / Interfaces Message-ID: <456590C3.4040801@davious.org> Proposal Summary ================ Add 2 new keywords: implements, implement (alternatively, implements and implement can be implemented as object level functions) Add new built-in function: interface Add new object level function: does_implement Interfaces ========== There is a fight between using classes/types for dispatch verses class methods for dispatch. There is a fight between declaring class compatibility and declaring method compatibility. We see this explicit declaration in Phillip's defop iter(self): versus method checking We should unify these concepts by allowing them to mix in an interface definition: Interfaces, here, are a set of class types, class methods/attributes, and strings * Class/Type: Can be built-in or user-defined when Class2 declares it implements Class1, we will presume the class will support anything Class1 will support, but will only crash on a call to a method Class1 has, but Class2 does not * Class Method/Attribute: Can be from a built-in object or be user-defined class when Class2 declares it implements a Class1.amethod, we will presume the class as a method amethod, but will only crash on a call to Class2.amethod if Class2.amethod is not defined. * String: Indicates a method or attribute with that name, introspection is used to see if a Class implements this iter_interface = interface(iter) # class-based interface file_open_close_inter = interface(file.open, file.close) # method interface # here is an interface looking for both a class and some methods to be 'implemented' iter_file_open_close_inter = interface(iter, file.open, file.close) Implementation of Interfaces (Implements and Implement) ======================================================= In the common case, a class already implements another class behavior and we just need to say that. Other times, we want to adapt or use a special method in a certain context. Implement and Implement: * Implements: Declares pre-existing computability It is a list of classes/types and class methods/attributes when Class2 declares it implements Class1, we will presume the class will support anything Class1 will support, but will only crash on a call to a method Class1 has, but Class2 does not * Implement: Indicates special method-help via adaptation or a different method. The keyword is used to define a function that acts as a class/type adapter or special function. when Class2 declares it implements a Class1.amethod, we will presume the class as a method amethod, but will only crash on a call to Class2.amethod if Class2.amethod is not defined. Class's declared to be implemented are presumed to have the full method set of the class, those reducing the need for extensive method checking at the adult-allowable risk of someone not completely implementing the classes methods. (Of course, we now have implementation declaration inspection of a sort.) Dispatching by Interface and Declared Implementation ===================================================== the example Bar class (following these dispatching examples) should be able to be dispatched by * methods looking for a iter, Foo, SomeOtherClass, YetAnotherClass instances example: def funct(iter_var: iter): * methods looking for a method of those classes example: def funct(var_that_implements_foo_or_declares_a_foo_method: Foo.method): * methods looking for methods file.open, file.close, AnotherClass.close example: def funct(var_with_a_method_like_file_open: file.open): * methods looking for just method of with a particular name example: def funct(var_with_a_method_called_open: "open"): the dispatch need not be based on a singular type or method (see below for those multi-condition interface examples) also, these dispatch functions can be class functions example def funct(self, var_with_a_file_open_meth: file.open): Example of Using Implements and Implement ========================================= class Bar(Foo): """based on the declarations that follow class is taken as iter, Foo, and SomeOtherClass without method introspection (return of self for these is implicit) methods open and close are understood to be file like methods. An adaption functionality is available in the declaration implement AnotherClass and special method as a particular class is available in the declaration implement YetAnotherClass.method.""" # new keyword implements implements iter, SomeOtherClass, file.open, file.close # new keyword implement: for class adaptation implement AnotherClass(self): return transform_to_yac(self) implement YetAnotherClass.open(self, var): return different_open_function(self, var) # presumed for file.open def open(self): # presumed for file.close def close(self): [implementing iter functions not shown] Interface objects, including multiple object apis =================================================== * use of new standard interface function * allows multiple interface apis to be combined foo_interface = interface(Foo) iter_interface = interface(iter) file_open_close_interface = interface(file.open, file.close) any_open_close_interface = interface("open", "close") iter_file_open_close_inter = interface(iter, file_open_close_interface) def funct(foo_var: Foo): .. equivalent to .. def funct(foo_var: foo_interface): def funct(iter_var: iter): .. equivalent to .. def funct(iter_var: iter_interface): def funct(file_like_var: file.open, file.close): .. equivalent to .. def funct(file_like_var: file_open_close_interface): def funct(method_abilities_var: "open", "close"): .. equivalent to .. def funct(method_abilities_var: just_open_close_methods_interface): def funct(method_abilities_var: iter, file.open, file.close): .. equivalent to .. def funct(method_abilities_var: iter_file_open_close_inter): does_implement ============== * does_implement: a boolean function that at the object level that returns true if the object has declared that it implements an interface object if obj.does_implement(iter): if obj.does_implement(Class1): if obj.does_implement(interface_obj): Implementing Interfaces without the keywords Implements, Implement ================================================================== Class2.implements(iter) Class2.implements(Class1) Class2.implements(interface_obj) Class2.implements(file.open) Class2.implements(iter, Class1, interface_obj, file.open) Class2.implement(Class1, transform_function_def) Class2.implement(Class1.method, function_def) Dispatching Issues ================== There will have to be some way to decide which declarations are chosen ahead of others, but I haven't thought about that too much.. we'd probably look at the order of the objects in the implements statement. The last issue I have in my head is Guido's issue > This doesn't do anything for other generic functions that might also > conceivably work with C. Phillip addresses that by proposing help > functions that do a bunch of these bindings at once, and by noticing > that if you just use the standard library and generic functions, the > default implementation might just work. But, it think those bindings could be generated with some *implements* declarations inside of standard objects like Iterable - Collection - Sequence - MutableSequence - list to each other. These declarations will be inherited by those object which declare they implement them. With this syntax, dispatch functions would seamlessly integrate with the current python language, the absence of any interface in a function def would be the python language as it works now, by defaulting to the lowest and most-likely only "dispatching" function. From python3000 at davious.org Thu Nov 23 13:46:34 2006 From: python3000 at davious.org (Dave Anderson) Date: Thu, 23 Nov 2006 07:46:34 -0500 Subject: [Python-3000] Abilities / Interfaces Message-ID: <4565982A.9030803@davious.org> Proposal Summary ================ Add 2 new keywords: implements, implement (alternatively, implements and implement can be implemented as object level functions) Add new built-in function: interface Add new object level function: does_implement (with minor corrections in Implementation of Interfaces (Implements and Implement) section) Interfaces ========== There is a fight between using classes/types for dispatch verses class methods for dispatch. There is a fight between declaring class compatibility and declaring method compatibility. We see this explicit declaration in Phillip's defop iter(self): versus method checking We should unify these concepts by allowing them to mix in an interface definition: Interfaces, here, are a set of class types, class methods/attributes, and strings * Class/Type: Can be built-in or user-defined when Class2 declares it implements Class1, we will presume the class will support anything Class1 will support, but will only crash on a call to a method Class1 has, but Class2 does not * Class Method/Attribute: Can be from a built-in object or be user-defined class when Class2 declares it implements a Class1.amethod, we will presume the class as a method amethod, but will only crash on a call to Class2.amethod if Class2.amethod is not defined. * String: Indicates a method or attribute with that name, introspection is used to see if a Class implements this iter_interface = interface(iter) # class-based interface file_open_close_inter = interface(file.open, file.close) # method interface # here is an interface looking for both a class and some methods to be 'implemented' iter_file_open_close_inter = interface(iter, file.open, file.close) Implementation of Interfaces (Implements and Implement) ======================================================= In the common case, a class already implements another class behavior and we just need to say that. Other times, we want to adapt or use a special method in a certain context. Implement and Implement: * Implements: Declares pre-existing compatability It is a list of classes/types and class methods/attributes when Class2 declares it implements Class1, we will presume the class will support anything Class1 will support, but will only crash on a call to a method Class1 has, but Class2 does not when Class2 declares it implements a Class1.amethod, we will presume the class as a method amethod, but will only crash on a call to Class2.amethod if Class2.amethod is not defined. * Implement: Indicates special method-help via adaptation or a different method. The keyword is used to define a function that acts as a class/type adapter or special function. When Class2 uses an implement to associate a class/type or method/attribute with a function definition, that function should called to which effectively adapts the Class2 obj when the context is 'implement Class' or runs a specially defined method when the context is 'implement method'. Class's declared to be implemented are presumed to have the full method set of the class, those reducing the need for extensive method checking at the adult-allowable risk of someone not completely implementing the classes methods. (Of course, we now have implementation declaration inspection of a sort.) Dispatching by Interface and Declared Implementation ===================================================== the example Bar class (following these dispatching examples) should be able to be dispatched by * methods looking for a iter, Foo, SomeOtherClass, YetAnotherClass instances example: def funct(iter_var: iter): * methods looking for a method of those classes example: def funct(var_that_implements_foo_or_declares_a_foo_method: Foo.method): * methods looking for methods file.open, file.close, AnotherClass.close example: def funct(var_with_a_method_like_file_open: file.open): * methods looking for just method of with a particular name example: def funct(var_with_a_method_called_open: "open"): the dispatch need not be based on a singular type or method (see below for those multi-condition interface examples) also, these dispatch functions can be class functions example def funct(self, var_with_a_file_open_meth: file.open): Example of Using Implements and Implement ========================================= class Bar(Foo): """based on the declarations that follow class is taken as iter, Foo, and SomeOtherClass without method introspection (return of self for these is implicit) methods open and close are understood to be file like methods. An adaption functionality is available in the declaration implement AnotherClass and special method as a particular class is available in the declaration implement YetAnotherClass.method.""" # new keyword implements implements iter, SomeOtherClass, file.open, file.close # new keyword implement: for class adaptation implement AnotherClass(self): return transform_to_yac(self) implement YetAnotherClass.open(self, var): return different_open_function(self, var) # presumed for file.open def open(self): # presumed for file.close def close(self): [implementing iter functions not shown] Interface objects, including multiple object apis =================================================== * use of new standard interface function * allows multiple interface apis to be combined foo_interface = interface(Foo) iter_interface = interface(iter) file_open_close_interface = interface(file.open, file.close) any_open_close_interface = interface("open", "close") iter_file_open_close_inter = interface(iter, file_open_close_interface) def funct(foo_var: Foo): .. equivalent to .. def funct(foo_var: foo_interface): def funct(iter_var: iter): .. equivalent to .. def funct(iter_var: iter_interface): def funct(file_like_var: file.open, file.close): .. equivalent to .. def funct(file_like_var: file_open_close_interface): def funct(method_abilities_var: "open", "close"): .. equivalent to .. def funct(method_abilities_var: just_open_close_methods_interface): def funct(method_abilities_var: iter, file.open, file.close): .. equivalent to .. def funct(method_abilities_var: iter_file_open_close_inter): does_implement ============== * does_implement: a boolean function that at the object level that returns true if the object has declared that it implements an interface object if obj.does_implement(iter): if obj.does_implement(Class1): if obj.does_implement(interface_obj): Implementing Interfaces without the keywords Implements, Implement ================================================================== Class2.implements(iter) Class2.implements(Class1) Class2.implements(interface_obj) Class2.implements(file.open) Class2.implements(iter, Class1, interface_obj, file.open) Class2.implement(Class1, transform_function_def) Class2.implement(Class1.method, function_def) Dispatching Issues ================== There will have to be some way to decide which declarations are chosen ahead of others, but I haven't thought about that too much.. we'd probably look at the order of the objects in the implements statement. The last issue I have in my head is Guido's issue > This doesn't do anything for other generic functions that might also > conceivably work with C. Phillip addresses that by proposing help > functions that do a bunch of these bindings at once, and by noticing > that if you just use the standard library and generic functions, the > default implementation might just work. But, it think those bindings could be generated with some *implements* declarations inside of standard objects like Iterable - Collection - Sequence - MutableSequence - list to each other. These declarations will be inherited by those object which declare they implement them. With this syntax, dispatch functions would seamlessly integrate with the current python language, the absence of any interface in a function def would be the python language as it works now, by defaulting to the lowest and most-likely only "dispatching" function. From gustavo at niemeyer.net Thu Nov 23 14:20:22 2006 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Thu, 23 Nov 2006 11:20:22 -0200 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <001801c70e63$d172eeb0$6402a8c0@arkdesktop> <3218934778639261534@unknownmsgid> <-3135130819962384877@unknownmsgid> <6961093367361956666@unknownmsgid> Message-ID: <20061123132022.GD6072@niemeyer.net> > IMO the difference between ABCs and interfaces/abilities is that > inserting a new ABC into an existing class hierarchy without modifying > its source code is considered a hack, and must be done by a helper > routine that *actually* mucks with __bases__; while (Zope-style) > interfaces use an external registry or a separate class attribute > (__implements__) that is intended to be mucked with. Right. That and also being able to assign interfaces to individual instances, being able to split and merge interfaces in a level that doesn't need changing the implementation itself, and so on. -- Gustavo Niemeyer http://niemeyer.net From fredrik at pythonware.com Thu Nov 23 14:53:33 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 23 Nov 2006 14:53:33 +0100 Subject: [Python-3000] print() parameters in py3k References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> <55DAAEFE-373C-4799-9C8E-5A48FABD63C1@python.org> Message-ID: Barry Warsaw wrote: >> 2) for convenience, extend print to treat a format object as a format >> specifier for the following arguments: >> >> print (x, y, z, fmt="My X: %s, Your Y: %s, His Z: %s") >> >> becomes >> >> print(format("My X: %s, Your Y: %s, His Z: %s"), x, y ,z) >> >> 3) get rid of printf. > > That's not bad. Is the format object allowed in any positional > argument, or just the first? any, I think. you can either just let print collect all none-format objects and pass it to the preceeding format object, or (better) ask the format object for how many arguments it expects, and use a default formatter (i.e. str) on the rest. From fredrik at pythonware.com Thu Nov 23 15:10:27 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 23 Nov 2006 15:10:27 +0100 Subject: [Python-3000] print() parameters in py3k References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: Guido van Rossum wrote: > Hm. While not as obviously from a different universe as Barry's > proposal, this still pretty weird, probably at least from a different > planet (much farther than Iceland anyway :-) it's inspired by http://online.effbot.org/2006_11_01_archive.htm#et-builder which owes a lot to http://www.tothink.com/python/ElementBuilder/ which is inspired by http://effbot.org/zone/idea-xml-literal.htm which you found "interesting in a futuristic kind of way" at the time ;-), and which is probably influenced by http://okmij.org/ftp/Scheme/SXML.html whether or not I was aware of that at the time, and google gives me around 150,000 hits for "John McCarthy ufo", so I guess that settles it. > Treating the first argument differently based on its being of a specific > type doesn't sound right to me; what if you are handed an object x > and you decide to print it using print(x), but surreptitiously (or by > mistake) they hand you a format object? it'll fail in some way (exactly how remains to be defined), but at least it'll fail more consistently and reliably than if you accidentally do printf(value) on a string value (see Adam's post at the top of this subthread). a failure caused by the wrong type is easier to handle than a failure caused by using the right type, but the wrong value. (functions that needs to print arbitrary objects in a safe fashion cannot really use a bare "print" anyway, of course.) From fredrik at pythonware.com Thu Nov 23 15:13:43 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 23 Nov 2006 15:13:43 +0100 Subject: [Python-3000] print() parameters in py3k References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> <91ad5bf80611210710m36a56959m552aea57a0c88354@mail.gmail.com> Message-ID: George Sakkis wrote: > a. Will print have to do an isinstance(args[0], format) to decide what > to do ? If so, don't the usual arguments for duck typing and against > type checking apply here ? isinstance (or an equivalent mechanism) would be more robust, in this specific case. > b. print(a,b,c) is no longer obvious whether it prints 3 unformatted > objects or 2 formatted, if you don't know what a is. nope, but on the other hand, you don't know what it prints today either (see my other post). > b. What if for some reason one actually wants to print the format > object instead of interpreting as a format specifier ? repr() is your friend. From aahz at pythoncraft.com Thu Nov 23 16:52:32 2006 From: aahz at pythoncraft.com (Aahz) Date: Thu, 23 Nov 2006 07:52:32 -0800 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <45655142.7050800@acm.org> References: <-6683921950610559197@unknownmsgid> <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> <91ad5bf80611221908t4e1ed686yb94d700d1bbc0cb5@mail.gmail.com> <45655142.7050800@acm.org> Message-ID: <20061123155232.GA13376@panix.com> On Wed, Nov 22, 2006, Talin wrote: > > While I wouldn't say it quite so strongly, I do think that there is a > great deal of subjectivity in HCI discussions. One person's 'intuitive' > may be another person's 'orange smoke', as we've seen in the current > discussion. But the whole point of making Guido BDFL was to codify his intuition WRT language decisions. There's a reason why we here are using Python instead of other languages, and as Guido himself has noted, often he knows what the right decisions is without being able to explain it. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "In many ways, it's a dull language, borrowing solid old concepts from many other languages & styles: boring syntax, unsurprising semantics, few automatic coercions, etc etc. But that's one of the things I like about it." --Tim Peters on Python, 16 Sep 1993 From gsakkis at rutgers.edu Thu Nov 23 17:26:36 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Thu, 23 Nov 2006 11:26:36 -0500 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <20061123155232.GA13376@panix.com> References: <-6683921950610559197@unknownmsgid> <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> <91ad5bf80611221908t4e1ed686yb94d700d1bbc0cb5@mail.gmail.com> <45655142.7050800@acm.org> <20061123155232.GA13376@panix.com> Message-ID: <91ad5bf80611230826g37a29e0fx46414562bb537fe1@mail.gmail.com> On 11/23/06, Aahz wrote: > On Wed, Nov 22, 2006, Talin wrote: > > > > While I wouldn't say it quite so strongly, I do think that there is a > > great deal of subjectivity in HCI discussions. One person's 'intuitive' > > may be another person's 'orange smoke', as we've seen in the current > > discussion. > > But the whole point of making Guido BDFL was to codify his intuition WRT > language decisions. There's a reason why we here are using Python > instead of other languages, and as Guido himself has noted, often he > knows what the right decisions is without being able to explain it. No, the reason we're here is that we share the same notion of rightness in most, or at least the most important, things. It's not an all-or-nothing choice though. Making general claims about rightness is some absolute way (and indirectly implying that Rubyists, Lispers and all other folks are wrong) should better remain a priviledge of religion, not language design. George From guido at python.org Thu Nov 23 17:38:46 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Nov 2006 08:38:46 -0800 Subject: [Python-3000] Fwd: defop ? In-Reply-To: <76fd5acf0611222310h1f6e4137r1dee4afa7a747ef0@mail.gmail.com> References: <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> <456456CF.8000005@gmail.com> <5.1.1.6.0.20061122151242.03f68a88@sparrow.telecommunity.com> <5.1.1.6.0.20061122180342.03ce3008@sparrow.telecommunity.com> <76fd5acf0611221959g74e8d7a9o36723f63814aa950@mail.gmail.com> <76fd5acf0611222310x64f03aa3i3d516c30452b1d79@mail.gmail.com> <76fd5acf0611222310h1f6e4137r1dee4afa7a747ef0@mail.gmail.com> Message-ID: On 11/22/06, Calvin Spealman wrote: > On 11/22/06, Guido van Rossum wrote: > > Not sure I like this better. My spelling of "operator::XXX" is > > "__XXX__" and I like that just fine -- no new syntax needed. > > Well, that might be a spelling of `operator::XXX` but what about other > use cases like `third_party_interface_system::adapt` or > `anything_at_all::XXX`? Thats what I mean with not being too limiting > and solving the problem in a way that opens up solutions to other > problems. I get the opposition to it, but it seems reasonable, > nonetheless. How was I to generalize from a single example what you meant? I thought you were using 'operator' as a fixed keyword like it is in C++, and I was wondering what you meant by the double colon. I still believe it is easy enough to solve this using a naming convention. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Nov 23 17:50:08 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Nov 2006 08:50:08 -0800 Subject: [Python-3000] print() parameters in py3k In-Reply-To: References: <8f01efd00611191013k5fa1118br2daa99d8817d975d@mail.gmail.com> Message-ID: Let me just say -1 on this idea so we can close the thread. None of that convinces me. I don't think that print(a, b) failing when b has a certain unexpected type can be considered consistent. The ElementBuilder example doesn't apply because it takes very limited argument types (string, dict, E; everything else raises an exception). print is carefully designed to be able to print *anything*. Not being able to print a format object strikes me as wrong. Apart from that, I'm not sure that having formats be a separate type is a good idea either; but even if it were a good idea, the rest of the reason for the -1 still stands. --Guido On 11/23/06, Fredrik Lundh wrote: > Guido van Rossum wrote: > > > Hm. While not as obviously from a different universe as Barry's > > proposal, this still pretty weird, probably at least from a different > > planet (much farther than Iceland anyway :-) > > it's inspired by > > http://online.effbot.org/2006_11_01_archive.htm#et-builder > > which owes a lot to > > http://www.tothink.com/python/ElementBuilder/ > > which is inspired by > > http://effbot.org/zone/idea-xml-literal.htm > > which you found "interesting in a futuristic kind of way" at the time ;-), > and which is probably influenced by > > http://okmij.org/ftp/Scheme/SXML.html > > whether or not I was aware of that at the time, and google gives me > around 150,000 hits for "John McCarthy ufo", so I guess that settles > it. > > > Treating the first argument differently based on its being of a specific > > type doesn't sound right to me; what if you are handed an object x > > and you decide to print it using print(x), but surreptitiously (or by > > mistake) they hand you a format object? > > it'll fail in some way (exactly how remains to be defined), but at least it'll fail > more consistently and reliably than if you accidentally do > > printf(value) > > on a string value (see Adam's post at the top of this subthread). a failure caused > by the wrong type is easier to handle than a failure caused by using the right type, > but the wrong value. > > (functions that needs to print arbitrary objects in a safe fashion cannot really use > a bare "print" anyway, of course.) > > > > > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Nov 23 18:00:54 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Nov 2006 09:00:54 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <456590C3.4040801@davious.org> References: <456590C3.4040801@davious.org> Message-ID: Smells a bit like the rejected PEP 245 doesn't it? You should at least explain how it differs in a summary or rationale. Anyway, it's Thanksgiving day here, and I'm unlikely to be able to spend much time on this collection of threads until Monday or so... And even then I'll have other work to do -- it was a rare opportunity that I could spend pretty much the entire day reading, thinking and writing about one topic (if perhaps a little too quick of the trigger). If you want to save me time, a personal email from Phillip, Fredrik, Bill or Andrew with a pointer to a proto-PEP that summarizes a spec that might emerge from the discussion would carry a lot of weight in my inbox (even if there were 4 competing proto-PEPs :-). (I'm not suggesting they are the only ones who can write the PEP, but without an endorsement from a heavyweight I would treat it as just another post.) --Guido On 11/23/06, Dave Anderson wrote: > Proposal Summary > ================ > > Add 2 new keywords: implements, implement > (alternatively, implements and implement can be implemented as object > level functions) > Add new built-in function: interface > Add new object level function: does_implement > > > Interfaces > ========== > > There is a fight between using classes/types for dispatch verses class > methods for dispatch. There is a fight between declaring class > compatibility and declaring method compatibility. > > We see this explicit declaration in Phillip's > defop iter(self): > > versus method checking > > We should unify these concepts by allowing them to mix in an interface > definition: > > Interfaces, here, are a set of class types, class methods/attributes, > and strings > > * Class/Type: Can be built-in or user-defined > when Class2 declares it implements Class1, we will presume > the class will support anything Class1 will support, but will only > crash on a call to a method Class1 has, but Class2 does not > * Class Method/Attribute: Can be from a built-in object or be > user-defined class > when Class2 declares it implements a Class1.amethod, we will > presume the class as a method amethod, but will only crash on a call > to Class2.amethod if Class2.amethod is not defined. > * String: Indicates a method or attribute with that name, introspection > is used to see if a Class implements this > > iter_interface = interface(iter) # class-based interface > file_open_close_inter = interface(file.open, file.close) # method interface > # here is an interface looking for both a class and some methods to be > 'implemented' > iter_file_open_close_inter = interface(iter, file.open, file.close) > > Implementation of Interfaces (Implements and Implement) > ======================================================= > > In the common case, a class already implements another class behavior > and we just need to say that. > Other times, we want to adapt or use a special method in a certain context. > > Implement and Implement: > * Implements: Declares pre-existing computability > It is a list of classes/types and class methods/attributes > when Class2 declares it implements Class1, we will presume > the class will support anything Class1 will support, but will > only crash on a call to a method Class1 has, but Class2 does not > * Implement: Indicates special method-help via adaptation or a different > method. > The keyword is used to define a function that acts as a class/type > adapter or special function. > when Class2 declares it implements a Class1.amethod, we will > presume the class as a method amethod, but will only crash > on a call to Class2.amethod if Class2.amethod is not defined. > > Class's declared to be implemented are presumed to have the full method > set of the class, those reducing the need for extensive method checking > at the adult-allowable risk of someone not completely implementing the > classes methods. (Of course, we now have implementation declaration > inspection of a sort.) > > Dispatching by Interface and Declared Implementation > ===================================================== > > the example Bar class (following these dispatching examples) should be > able to be dispatched by > * methods looking for a iter, Foo, SomeOtherClass, YetAnotherClass instances > example: def funct(iter_var: iter): > > * methods looking for a method of those classes > example: def funct(var_that_implements_foo_or_declares_a_foo_method: > Foo.method): > > * methods looking for methods file.open, file.close, AnotherClass.close > example: def funct(var_with_a_method_like_file_open: file.open): > > * methods looking for just method of with a particular name > example: def funct(var_with_a_method_called_open: "open"): > > the dispatch need not be based on a singular type or method > (see below for those multi-condition interface examples) > > also, these dispatch functions can be class functions > example def funct(self, var_with_a_file_open_meth: file.open): > > > Example of Using Implements and Implement > ========================================= > > class Bar(Foo): > """based on the declarations that follow class is taken as iter, Foo, > and SomeOtherClass without method introspection (return of self for > these is implicit) methods open and close are understood to be file like > methods. An adaption functionality is available in the declaration > implement AnotherClass and special method as a particular class is > available in the declaration implement > YetAnotherClass.method.""" > # new keyword implements > implements iter, SomeOtherClass, file.open, file.close > > # new keyword implement: for class adaptation > implement AnotherClass(self): > return transform_to_yac(self) > > implement YetAnotherClass.open(self, var): > return different_open_function(self, var) > > # presumed for file.open > def open(self): > > # presumed for file.close > def close(self): > > [implementing iter functions not shown] > > > Interface objects, including multiple object apis > =================================================== > * use of new standard interface function > * allows multiple interface apis to be combined > > foo_interface = interface(Foo) > iter_interface = interface(iter) > file_open_close_interface = interface(file.open, file.close) > any_open_close_interface = interface("open", "close") > iter_file_open_close_inter = interface(iter, file_open_close_interface) > > def funct(foo_var: Foo): > .. equivalent to .. > def funct(foo_var: foo_interface): > > def funct(iter_var: iter): > .. equivalent to .. > def funct(iter_var: iter_interface): > > def funct(file_like_var: file.open, file.close): > .. equivalent to .. > def funct(file_like_var: file_open_close_interface): > > def funct(method_abilities_var: "open", "close"): > .. equivalent to .. > def funct(method_abilities_var: just_open_close_methods_interface): > > def funct(method_abilities_var: iter, file.open, file.close): > .. equivalent to .. > def funct(method_abilities_var: iter_file_open_close_inter): > > > does_implement > ============== > > * does_implement: a boolean function that at the object level that > returns true if the object has declared that it implements an > interface object > > if obj.does_implement(iter): > > if obj.does_implement(Class1): > > if obj.does_implement(interface_obj): > > Implementing Interfaces without the keywords Implements, Implement > ================================================================== > Class2.implements(iter) > > Class2.implements(Class1) > > Class2.implements(interface_obj) > > Class2.implements(file.open) > > Class2.implements(iter, Class1, interface_obj, file.open) > > Class2.implement(Class1, transform_function_def) > > Class2.implement(Class1.method, function_def) > > Dispatching Issues > ================== > > There will have to be some way to decide which declarations are chosen > ahead of others, but I haven't thought about that too much.. we'd > probably look at the order of the objects in the implements statement. > > The last issue I have in my head is Guido's issue > > This doesn't do anything for other generic functions that might also > > conceivably work with C. Phillip addresses that by proposing help > > functions that do a bunch of these bindings at once, and by noticing > > that if you just use the standard library and generic functions, the > > default implementation might just work. > > But, it think those bindings could be generated with some *implements* > declarations inside of standard objects like Iterable - Collection - > Sequence - MutableSequence - list to each other. These declarations > will be inherited by those object which declare they implement them. > > With this syntax, dispatch functions would seamlessly integrate with the > current python language, the absence of any interface in a function def > would be the python language as it works now, by defaulting to the > lowest and most-likely only "dispatching" function. > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Nov 23 18:37:07 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Nov 2006 09:37:07 -0800 Subject: [Python-3000] Generic functions vs. OO Message-ID: I'm reposting this under a different subject because the other subject seems to have gone off on a tangent. The rest are Phillip's words. --Guido ---------- Forwarded message ---------- From: Phillip J. Eby Date: Nov 22, 2006 10:01 PM Subject: Re: [Python-3000] Special methods and interface-based type system To: Guido van Rossum Cc: Python 3000 At 08:29 PM 11/22/2006 -0800, Guido van Rossum wrote: >One thing that rubs me the wrong way about generic functions is that >it appears to go against OO. Now I'm not someone to take OO as >religion, but there's something uncomfortable (for me) about how, in >Phillip's world, many things become functions instead of methods, >which brings along concerns about the global namespace filling up, and >also about functionality being spread randomly across too many >modules. I fear I will miss the class as a convenient focus for >related functionality. I originally proposed a solution for this back in January '05, but it was too premature. But since you have now stated the problem that the proposal was intended to solve, perhaps the solution has a chance now. :) I will try to be as concrete as possible. Let's start with an actual, hopefully non-exploding 'Interface' implementation, based on an assumption that we have generic functions available: class InterfaceClass(type): def __init__(cls, name, bases, cdict): for k,v in cdict.items(): # XXX this should probably skip at least __slots__, # __metaclass__, and __module__, but oh well cdict[k] = AdaptingDescriptor(v) class Interface: __metaclass__ = InterfaceClass __slots__ = '__self__' def __init__(self, subject): # this isinstance() check should be replaced by an # 'unwrap()' generic function, so other adapter types # will work, but this is just an example, so... if isinstance(subject, Interface): subject = subject.__self__ self.__self__ = subject class AdaptingDescriptor: def __init__(self, descriptor): self.wrapped = descriptor def __get__(self, ob, typ=None): if ob is None: return self return self.wrapped.__get__(ob.__self__, typ) Now, using this new "interface framework", let's implement a small "mapping" typeclas... er, interface. class Mapping(Interface): def keys(self): return [k for k,v in self.items()] def items(self): return [k,self[k] for k in self.keys()] # ... other self-recursive definitions What does this do? Well, we can now call Mapping(foo) to turn an arbitrary object into something that has Mapping's generic functions as its methods, and invokes them on foo! (I am assuming here that normal functions are implicitly overloadable, even if that means they change type at runtime to do so.) We could even use interfaces for argument type declarations, to automatically put things in the "right namespace" for what the code expects to use. That is, if you declare an argument to be a Mapping, then that's what you get. If you call .keys() on the resulting adapted object and the type doesn't support the operation, you get an error. Too late a form of error checking you say? Well, make a more sophisticated factory mechanism in Interface.__new__ that actually creates (and caches) different adapter types based on the type of object being adapted, so that hasattr() tests will work on the wrapped type, or so that you can get an early error if none of the wrapped generic functions has a method defined for the target type. A few important points here: 1. A basic interface mechanism is extemely simple to implement, given generic functions 2. It is highly customizable with respect to error checking and other features, even on a per-user basis, because there doesn't have to be only one "true" Interface type to rule them all (or one true generic function type either, but that's a separate discussion). 3. It allows interfaces to include partial implementations, ala Ping and Alex's past proposals, thus allowing you to implement partial mapping or "file" objects and have the rest of the interface's implementation filled in for you 4. It allows you to hide the very existence of the notion of a "generic function", if you prefer not to think about such things 5. It even supports interface inheritance and interface algebra: subclassing an interface allows adding new operations, and simple assignment suffices to compose new interfaces, e.g.: class MappingItems(Interface): items = Mapping.items Notice that nothing special is required, this "just works" as a natural consequence of the rest of the implementation shown. Okay, so now you want to know how to *implement* a "Mapping". Well, simplest but most tedious, you can just register operations directly, e.g.: class MyMapping: def __init__(self, data): self.data = dict(data) defop operator.getitem(self, key): return self.data[key] defop Mapping.items(self): return self.data.items() But as you can imagine, this would probably get a bit tedious if you're implementing lots of methods. So, we can add metaclasses or class decorators here to say, "I implement these interfaces, so any methods I have whose names match the method names in the interfaces, please hook 'em up for me." I'm going to leave out the implementation, as it should be a straightforward exercise for the reader to come up with many ways by which it can be accomplished. The spelling might be something like: class MyMapping: implements(Mapping) def items(self): ... #etc. At which point, we have now come full circle to being able to provide all of the features of interfaces, adaptation, and generic functions, without forcing anyone to give up the tasty OO flavor of method calls. Heck, they can even keep the way they spell existing adaptation calls (e.g. IFoo(bar) to adapt bar to IFoo) in PEAK, Twisted, and Zope! And finally, note that if you only want to perform one method call on a given object, you can also use the generics directly, e.g. Mapping.items(foo) instead of Mapping(foo).items(). Voila -- generic goodness and classic OO method-calling simplicity, all in one simple to implement package. It should now be apparent why I said that interfaces are trivial to implement if you define them as namespaces for generic functions, rather than as namespaces for methods. There are many spinoffs possible, too. For example, you could have a factory function that turns an existing class's public operations into an interface object. There are also probably also some dark corners of the idea that haven't been explored, because when I first proposed basically this idea in '05, nobody was ready for it. Now maybe we can actually talk about the implications. From p.f.moore at gmail.com Thu Nov 23 18:55:20 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 23 Nov 2006 17:55:20 +0000 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: References: Message-ID: <79990c6b0611230955j57bd3c34y60a35d2b95ea1dfa@mail.gmail.com> On 11/23/06, Guido van Rossum wrote: > I'm reposting this under a different subject because the other subject > seems to have gone off on a tangent. The rest are Phillip's words. [...] OK, I've read and tried to digest this. It looks good. The one thing I'm still not getting, at a very concrete level, is precisely what changes are required to Python to make it work. The code is clear enough, but it's subtly not Python... For example, the comment "(I am assuming here that normal functions are implicitly overloadable, even if that means they change type at runtime to do so.)" - I don't understand what, if anything this implies about the semantics of the "def" statement (assuming that's the statement involved). If there's a pointer to a previous posting that would clarify, I'd appreciate it - although maybe it's another thing that would benefit from being restated in the light of the new context. Thanks, and apologies if I'm being dumb. Paul. From janssen at parc.com Thu Nov 23 19:17:11 2006 From: janssen at parc.com (Bill Janssen) Date: Thu, 23 Nov 2006 10:17:11 PST Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <06Nov22.183133pst."58648"@synergy1.parc.xerox.com> References: <-6683921950610559197@unknownmsgid> <8853879015409334245@unknownmsgid> <06Nov22.183133pst."58648"@synergy1.parc.xerox.com> Message-ID: <06Nov23.101719pst."58648"@synergy1.parc.xerox.com> > > I don't know anything by CLOS. > > I'll drop off a copy of the book on Monday. Lord knows we have > zillions of extra copies floating around PARC. I'll still drop off a copy (of Common Lisp the Language, version 2), but there's no need to wait. It's on the Web at http://www.supelec.fr/docs/cltl/clm/node260.html. See in particular the "change-class" operation at http://www.supelec.fr/docs/cltl/clm/node305.html. I think I'm still confused (happens a lot :-) about our method namespace discussion. It seems to me that Python's method namespaces work pretty much the same way that CLOS's do, already. That is, you don't "clobber" an existing method in a base class when you define a new method by the same name in a derived class; you just mask it. The base class' method is still there, and can still be called explicitly. Bill From pje at telecommunity.com Thu Nov 23 19:19:07 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 23 Nov 2006 13:19:07 -0500 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: <79990c6b0611230955j57bd3c34y60a35d2b95ea1dfa@mail.gmail.co m> References: Message-ID: <5.1.1.6.0.20061123130727.01f42838@sparrow.telecommunity.com> At 05:55 PM 11/23/2006 +0000, Paul Moore wrote: >On 11/23/06, Guido van Rossum wrote: > > I'm reposting this under a different subject because the other subject > > seems to have gone off on a tangent. The rest are Phillip's words. >[...] > >OK, I've read and tried to digest this. It looks good. The one thing >I'm still not getting, at a very concrete level, is precisely what >changes are required to Python to make it work. No changes as such, just additions: 1. defop, addmethod/__addmethod__, maybe hasmethod/__hasmethod__ 2. some generic function implementation that can be applied to "normal" Python functions > The code is clear >enough, but it's subtly not Python... For example, the comment "(I am >assuming here that normal functions are implicitly overloadable, even >if that means they change type at runtime to do so.)" - I don't >understand what, if anything this implies about the semantics of the >"def" statement (assuming that's the statement involved). No changes in semantics to "def". I'm just saying that you have to be able to call: addmethod(a_function, methodfunc, a_class) And have the existing simple Python function 'a_function' be overloaded as a result. RuleDispatch and PEAK-Rules do this by replacing the function's func_code with some newly-generated dispatching code, but if this is to become a feature of the Python language, it might be better to simply allow subclassing FunctionType, and let the addmethod operation change the function's __class__ on the fly. This would allow generic function implementations to be written in C without additional calling overhead, but still just modify the original object in-place instead of having to replace it. (You can't just replace the function with a *different* object, because by the time you overload it, there may be lots of references to it already.) It may be that if a_function doesn't have an __addmethod__, then addmethod() should try to call some special method (__overload__?) on methodfunc (in case it's been decorated with something that knows how to turn a_function into a generic function. Finally, if neither side knows what to do, addmethod() should default to using some built-in or stdlib generic function implementation. From pje at telecommunity.com Thu Nov 23 19:31:56 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 23 Nov 2006 13:31:56 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <20061123121416.GA6072@niemeyer.net> References: <45651355.7090605@canterbury.ac.nz> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <20061122165513.GA13795@niemeyer.net> <4564F8B7.5030007@canterbury.ac.nz> <20061123020054.GA31618@niemeyer.net> <45651355.7090605@canterbury.ac.nz> Message-ID: <5.1.1.6.0.20061123132551.04454ae8@sparrow.telecommunity.com> At 10:14 AM 11/23/2006 -0200, Gustavo Niemeyer wrote: >This is true for generic functions as well, except that >instead of using an IFruit interface, you have a fruit() >generic function which handles the adaptation. Unfortunately, >this model doesn't work with more complex hierachies. You've got that backwards, actually; generic functions can work with "recombinant" hierarchies, where traditional interfaces can't. See Guido's repost of my explanation under the "Generic functions vs OO" thread. In it, I present a short "Interface" implementation based on generic functions that provides universal adaptation for arbitrary interface hierarchies, including subset interfaces -- *without having to write adapter classes*: http://mail.python.org/pipermail/python-3000/2006-November/004746.html From janssen at parc.com Thu Nov 23 19:39:24 2006 From: janssen at parc.com (Bill Janssen) Date: Thu, 23 Nov 2006 10:39:24 PST Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> <8853879015409334245@unknownmsgid> <06Nov22.181438pst."58648"@synergy1.parc.xerox.com> Message-ID: <06Nov23.103924pst."58648"@synergy1.parc.xerox.com> > Bill Janssen wrote: > > >> Ow, that's the first time you describe this particular wrinkle. > > > > Yep. I hesitated to bring it up, but if there's a need for separate > > namespaces for method names, why not do it right? > > what makes you so sure that Python's current design isn't "right" ? Well, "right" is a highly subjective term, and I think it's a bit foolish to say that a language as widely used, and widely-useful, as Python, isn't "right". But in Py3K, we're talking about what could be improved, and there are *facets* of Python which could better support its use. I think method namespaces are already OK; the question is whether to move these specially-named (for whatever reason) methods to a real namespace (a dict somewhere), rather than using this name-mangling kludge to introduce one. If they're special methods for the VM, as Ivan implied, let's put them in "PythonVM". If they're what operations and built-in functions call, let's use the "operation" namespace suggested by Calvin Speakman. Bill From guido at python.org Thu Nov 23 19:43:43 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Nov 2006 10:43:43 -0800 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: <5.1.1.6.0.20061123130727.01f42838@sparrow.telecommunity.com> References: <5.1.1.6.0.20061123130727.01f42838@sparrow.telecommunity.com> Message-ID: On 11/23/06, Phillip J. Eby wrote: > At 05:55 PM 11/23/2006 +0000, Paul Moore wrote: > >On 11/23/06, Guido van Rossum wrote: > > > I'm reposting this under a different subject because the other subject > > > seems to have gone off on a tangent. The rest are Phillip's words. > >[...] > > > >OK, I've read and tried to digest this. It looks good. The one thing > >I'm still not getting, at a very concrete level, is precisely what > >changes are required to Python to make it work. > > No changes as such, just additions: > > 1. defop, addmethod/__addmethod__, maybe hasmethod/__hasmethod__ > 2. some generic function implementation that can be applied to "normal" > Python functions > > > > The code is clear > >enough, but it's subtly not Python... For example, the comment "(I am > >assuming here that normal functions are implicitly overloadable, even > >if that means they change type at runtime to do so.)" - I don't > >understand what, if anything this implies about the semantics of the > >"def" statement (assuming that's the statement involved). > > No changes in semantics to "def". I'm just saying that you have to be able > to call: > > addmethod(a_function, methodfunc, a_class) > > And have the existing simple Python function 'a_function' be overloaded as > a result. RuleDispatch and PEAK-Rules do this by replacing the function's > func_code with some newly-generated dispatching code, but if this is to > become a feature of the Python language, it might be better to simply allow > subclassing FunctionType, and let the addmethod operation change the > function's __class__ on the fly. This would allow generic function > implementations to be written in C without additional calling overhead, but > still just modify the original object in-place instead of having to replace it. > > (You can't just replace the function with a *different* object, because by > the time you overload it, there may be lots of references to it already.) > > It may be that if a_function doesn't have an __addmethod__, then > addmethod() should try to call some special method (__overload__?) on > methodfunc (in case it's been decorated with something that knows how to > turn a_function into a generic function. Finally, if neither side knows > what to do, addmethod() should default to using some built-in or stdlib > generic function implementation. Changing the class can't add fields to the C struct that represent the object (since that would mean realloc()'ing the object and hence potentially changing its address). So why not make this a capability of the base function type, rather than messing with __class__? (This is not an endorsement of the whole thing, which I haven't read yet.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tony at PageDNA.com Thu Nov 23 19:00:01 2006 From: tony at PageDNA.com (Tony Lownds) Date: Thu, 23 Nov 2006 10:00:01 -0800 Subject: [Python-3000] optional argument annotations Message-ID: <33D8CBFC-9B3B-4FDE-927F-4B6D6B11C434@PageDNA.com> I have a working optional argument syntax implementation, I'm hoping to get some direction on the implementation decisions so far.... please see below. Python 3.0x (p3yk:52824M, Nov 23 2006, 09:22:23) [GCC 3.4.4 20050721 (Red Hat 3.4.4-2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> def f()-> 1: pass ... >>> f.func_returns 1 >>> def f(): pass ... >>> f.func_annotations >>> f.func_returns Traceback (most recent call last): File "", line 1, in AttributeError: 'function' object has no attribute 'func_returns' >>> def f(x:1): pass ... >>> f.func_annotations {'x': 1} >>> >>> import dis >>> dis.dis(compile("def f(x:1, y:2=3, z=4)->5:pass", "", "exec")) 1 0 LOAD_CONST 0 (3) # default for y 3 LOAD_CONST 1 (4) # default for z 6 LOAD_CONST 2 (1) # annotation for x 9 LOAD_CONST 3 (2) # annotation for y 12 LOAD_CONST 4 (6) # bitmask for annotations... 0b110 15 LOAD_CONST 5 (5) # func_returns 18 LOAD_CONST 6 (", line 1>) 21 MAKE_FUNCTION 49154 # numdefaults(2) + kwdefaults(0) << 8 + has_annotations(1) << 14 + has_returns(1) << 15 24 STORE_NAME 0 (f) 27 LOAD_CONST 7 (None) 30 RETURN_VALUE Index: Parser/Python.asdl =================================================================== --- Parser/Python.asdl (revision 52824) +++ Parser/Python.asdl (working copy) @@ -9,8 +9,8 @@ -- not really an actual node but useful in Jython's typesystem. | Suite(stmt* body) - stmt = FunctionDef(identifier name, arguments args, - stmt* body, expr* decorators) + stmt = FunctionDef(identifier name, typedarguments args, + stmt* body, expr* decorators, expr? returns) | ClassDef(identifier name, expr* bases, stmt* body) | Return(expr? value) @@ -102,6 +102,9 @@ arguments = (expr* args, identifier? vararg, expr* kwonlyargs, identifier? kwarg, expr* defaults, expr* kw_defaults) + typedarguments = (annotatedarg* args, identifier? vararg, expr* kwonlyargs, + identifier? kwarg, expr* defaults, expr* kw_defaults) + annotatedarg = (expr arg, expr? annotation) -- keyword arguments supplied to call keyword = (identifier arg, expr value) From guido at python.org Thu Nov 23 19:53:19 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Nov 2006 10:53:19 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <20061122202924.GA21361@niemeyer.net> Message-ID: On 11/22/06, Thomas Lotze wrote: > Guido van Rossum wrote: > > >> I'm sure you're aware about it, but in Zope 3 terminology, these are > >> 'provide' and 'implement', respectively. > > > > I like provide, but I'm not so sure about implement, since it is awfully > > ambiguous -- most of the time it is the class that does the implementing. > > That's why I settled for "has". > > JFTR: In Zope3, classes "implement" interfaces while instances "provide" > them. Well, that pretty much proves their terminology is confusing. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Thu Nov 23 20:12:50 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 23 Nov 2006 14:12:50 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <20061122202924.GA21361@niemeyer.net> Message-ID: <5.1.1.6.0.20061123140510.044eb4e8@sparrow.telecommunity.com> At 10:53 AM 11/23/2006 -0800, Guido van Rossum wrote: >On 11/22/06, Thomas Lotze wrote: > > Guido van Rossum wrote: > > > > >> I'm sure you're aware about it, but in Zope 3 terminology, these are > > >> 'provide' and 'implement', respectively. > > > > > > I like provide, but I'm not so sure about implement, since it is awfully > > > ambiguous -- most of the time it is the class that does the implementing. > > > That's why I settled for "has". > > > > JFTR: In Zope3, classes "implement" interfaces while instances "provide" > > them. > >Well, that pretty much proves their terminology is confusing. :-) You can actually blame that one on me; I'm the one who proposed it. :) At the time, mailing list discussions were hard to follow as to whether we were talking about instances or classes when we were talking about "implements". So I suggested that we add "provide" to mean that you could actually *use* the interface on the object, and leave "implements" to mean that your *instances* provide the interface. The distinction helped the immediate discussion, but later, when I went on to create PyProtocols, I realized it wasn't as clear outside the original context, so I stopped talking about 'implements' entirely and just always used 'provides' instead. So, instead of saying that a class "implements" X, PyProtocols says that a class' *instances* provide X, and you declare this with 'instancesProvide'. If the class *itself* provides the interface (e.g. via classmethods), you declare it with 'classProvides'. So, I think the "provide" term was a good idea, it's just that having a different word to mean "my instances provide", didn't work out for me. From pje at telecommunity.com Thu Nov 23 20:21:23 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 23 Nov 2006 14:21:23 -0500 Subject: [Python-3000] optional argument annotations In-Reply-To: <33D8CBFC-9B3B-4FDE-927F-4B6D6B11C434@PageDNA.com> Message-ID: <5.1.1.6.0.20061123141550.04c672e8@sparrow.telecommunity.com> At 10:00 AM 11/23/2006 -0800, Tony Lownds wrote: >I have a working optional argument syntax implementation, I'm hoping >to get some direction on >the implementation decisions so far.... please see below. Why not just have the generated bytecode construct the annotation dictionary directly and assign it to the returned function's func_annotations, and the same for func_return if needed? Then there'd be no need to change the MAKE_FUNCTION opcode. Mightn't that make it easier to e.g. backport to the 2.x line, since it'd only be a compiler change, not an interpreter or function type change? From pje at telecommunity.com Thu Nov 23 20:24:04 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 23 Nov 2006 14:24:04 -0500 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: References: <5.1.1.6.0.20061123130727.01f42838@sparrow.telecommunity.com> <5.1.1.6.0.20061123130727.01f42838@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061123135021.04c2ecf8@sparrow.telecommunity.com> At 10:43 AM 11/23/2006 -0800, Guido van Rossum wrote: >Changing the class can't add fields to the C struct that represent the >object (since that would mean realloc()'ing the object and hence >potentially changing its address). So why not make this a capability >of the base function type, rather than messing with __class__? Because it would allow generic function implementations written in Python (i.e. using __dict__ for any additional data) to do the same thing without needing to much with func_code, the way some of my implementations do now. The use case for changing the type is allowing functions to only become generic once they're *actually* overloaded, so you don't have to go around decorating every function as generic "just in case" you want to overload it later. Meanwhile, it would interfere with the "fast track" calling path in the interpreter to have generic functions be the same type as regular functions. We'd have to add another check there, like a flag or something, so swapping out the type would make it just work with the existing eval loop. (If we later want to have a fast-track for generics, we can always check for that new type, too.) Anyway, all of this implementation discussion is probably premature optimization. My main purpose in allowing delayed overloading is to support CLOS-style method combination (or other specialized generic function features) on normal functions, without having to predefine the original function as being generic. Of course, changing func_code will presumably still work for that, but it precludes the possibility of using a C implementation in that case. But of course all of this is moot unless/until we're actually discussing a "real" implementation instead of the proof-of-concept. From gustavo at niemeyer.net Thu Nov 23 21:02:11 2006 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Thu, 23 Nov 2006 18:02:11 -0200 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: References: Message-ID: <20061123200211.GA22490@niemeyer.net> > At 10:14 AM 11/23/2006 -0200, Gustavo Niemeyer wrote: > >This is true for generic functions as well, except that > >instead of using an IFruit interface, you have a fruit() > >generic function which handles the adaptation. Unfortunately, > >this model doesn't work with more complex hierachies. > > You've got that backwards, actually; generic functions can work with > "recombinant" hierarchies, where traditional interfaces can't. See > Guido's repost of my explanation under the "Generic functions vs OO" > thread. In it, I present a short "Interface" implementation based on Nonsense. Coming up with an interface implementation that includes generic functions makes the point that interfaces are important stronger, and thus reinforce what I just said. I never said generic functions are bad or non-useful, I said that they're limited and won't work in more complex cases without additional infrastructure. -- Gustavo Niemeyer http://niemeyer.net From gustavo at niemeyer.net Thu Nov 23 21:14:13 2006 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Thu, 23 Nov 2006 18:14:13 -0200 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: References: Message-ID: <20061123201413.GB22490@niemeyer.net> (...) > A few important points here: > > 1. A basic interface mechanism is extemely simple to implement, given > generic functions (...) All of these points are completely true for interfaces implemented without generic functions. In fact, I fail to recognize in your mail a reason why generic functions improve the developer's experience at all. Would you be able to expose a concrete case where this system would present advantages? -- Gustavo Niemeyer http://niemeyer.net From tony at PageDNA.com Thu Nov 23 21:27:48 2006 From: tony at PageDNA.com (Tony Lownds) Date: Thu, 23 Nov 2006 12:27:48 -0800 Subject: [Python-3000] optional argument annotations In-Reply-To: <5.1.1.6.0.20061123141550.04c672e8@sparrow.telecommunity.com> References: <5.1.1.6.0.20061123141550.04c672e8@sparrow.telecommunity.com> Message-ID: <73F95868-C85C-4AD7-ACA7-E969D641BB67@PageDNA.com> On Nov 23, 2006, at 11:21 AM, Phillip J. Eby wrote: > Why not just have the generated bytecode construct the annotation > dictionary directly and assign it to the returned function's > func_annotations, and the same for func_return if needed? Then > there'd be no need to change the MAKE_FUNCTION opcode. Mightn't > that make it easier to e.g. backport to the 2.x line, since it'd > only be a compiler change, not an interpreter or function type change? > > It's more trips around the ceval loop and more constants on the code object with the MAKE_FUNCTION opcode. OTOH the mask stuff is pretty ugly. As a bonus, solving this problem is easier: >>> def f((x,y):1): pass ... >>> f.func_annotations {'.0': 1} I'm not sure how backwards compatibility is affected. MAKE_FUNCTION has already changed in p3yk. It's worth a try though, thanks! -Tony From greg.ewing at canterbury.ac.nz Thu Nov 23 21:27:48 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 24 Nov 2006 09:27:48 +1300 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> <91ad5bf80611221929o4230fccfh663bf7a5bc5564d@mail.gmail.com> Message-ID: <45660444.4090204@canterbury.ac.nz> Guido van Rossum wrote: > Your presence here has been entirely a distraction. That's a troll to me. Steady on, Guido. Trolling is making deliberately inflammatory statements with the intention of stirring up controversy. That's a pretty heavy accusation to make. You're effectively saying he's setting out to be obnoxious, and I don't see that in any of his postings. -- Greg From pje at telecommunity.com Thu Nov 23 21:54:23 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 23 Nov 2006 15:54:23 -0500 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: <20061123201413.GB22490@niemeyer.net> References: Message-ID: <5.1.1.6.0.20061123153733.01f418e8@sparrow.telecommunity.com> At 06:14 PM 11/23/2006 -0200, Gustavo Niemeyer wrote: >(...) > > A few important points here: > > > > 1. A basic interface mechanism is extemely simple to implement, given > > generic functions >(...) > >All of these points are completely true for interfaces implemented >without generic functions. Let's see your implementation. Zope's interface system fails on at least points 3 and 5. And regarding #1, Zope's implementation of the basic features I described is an order of magnitude more complex, even if you compare *just* its code that implements adaptation and interface inheritance, with *all* of my code plus Guido's generic function implementation. Indeed, even PyProtocols' implementation of similar features is an order of magnitude more complex than this. PyProtocols at least has a kludgy way of doing #5, but it fails #3 along with Zope. My point is that no interface system for Python that I'm aware of can do anywhere *near* as much with so little code or complexity. >In fact, I fail to recognize in your mail a reason why generic >functions improve the developer's experience at all. Would you be >able to expose a concrete case where this system would present >advantages? In no particular order: 1. "Recombinant" interfaces 2. Absence of adapter classes for "stateless" adapters 3. No need to adapt for single-operation interfaces (or to define interfaces at all in that case) 4. No need to understand interface or adaptation theory to be able to use it 5. Generic functions are easily grasped as a dynamic version of overloading as seen in other languages (including C#, Java, C++, etc., where there's no built-in notion of "adaptation") From pje at telecommunity.com Thu Nov 23 22:05:15 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 23 Nov 2006 16:05:15 -0500 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: <5.1.1.6.0.20061123153733.01f418e8@sparrow.telecommunity.co m> References: <20061123201413.GB22490@niemeyer.net> Message-ID: <5.1.1.6.0.20061123155720.01f40858@sparrow.telecommunity.com> At 03:54 PM 11/23/2006 -0500, Phillip J. Eby wrote: >5. Generic functions are easily grasped as a dynamic version of overloading >as seen in other languages (including C#, Java, C++, etc., where there's no >built-in notion of "adaptation") Oh, and I forgot: they're also easily understood as generic or polymorphic functions by people who use them in languages like CLOS, Dylan, Haskell, etc. Not that that's necessarily a big population compared to the above languages, but my point here is that even these less-popular languages don't really have adaptation as such. (Haskell's typeclasses are also very close in spirit to the interface approach I propose, but that's neither an advantage nor disadvantage, just a data point.) Also, COM and .NET *do* have a notion of interfaces, but it's actually more similar to what I'm proposing than to what Zope or PyProtocols interfaces do. They are like typeclasses in the sense that an interface on a COM object is normally just a stateless adapter providing an effective "method namespace" on the object. Anyway, the point was just that one advantage of my proposal is that it's easier to teach, because it has many things it can be compared to, depending on the audience: dynamic overloading, generic functions, typeclasses, COM "query-interface", etc. It can even be compared to existing Python interface implementations, in terms of what things it can do that they can't, as well as the things they have in common. :) From p.f.moore at gmail.com Thu Nov 23 22:53:57 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 23 Nov 2006 21:53:57 +0000 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: <5.1.1.6.0.20061123130727.01f42838@sparrow.telecommunity.com> References: <5.1.1.6.0.20061123130727.01f42838@sparrow.telecommunity.com> Message-ID: <79990c6b0611231353o70059496w8e2e04aea2786421@mail.gmail.com> On 11/23/06, Phillip J. Eby wrote: > At 05:55 PM 11/23/2006 +0000, Paul Moore wrote: > >On 11/23/06, Guido van Rossum wrote: > > > I'm reposting this under a different subject because the other subject > > > seems to have gone off on a tangent. The rest are Phillip's words. > >[...] > > > >OK, I've read and tried to digest this. It looks good. The one thing > >I'm still not getting, at a very concrete level, is precisely what > >changes are required to Python to make it work. > > No changes as such, just additions: Hmm, I'm not getting my question across (but that's OK, you answered anyway). From my POV, additions are changes - if you like, I'm trying to imagine what a diff between current Python and your proposal would look like. > 1. defop, addmethod/__addmethod__, maybe hasmethod/__hasmethod__ > 2. some generic function implementation that can be applied to "normal" > Python functions Um, isn't (1) a generic function implementation that can be applied to Python functions? > No changes in semantics to "def". I'm just saying that you have to be able > to call: > > addmethod(a_function, methodfunc, a_class) OK. So under your proposal, all Python functions are (potentially) generic, and addmethod is how you add extra overloads? (And hasmethod is pretty obvious, and defop AFAICT is a syntax for addmethod, is that right?) Did I get that right? Paul From greg.ewing at canterbury.ac.nz Thu Nov 23 23:03:15 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 24 Nov 2006 11:03:15 +1300 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <20061123121416.GA6072@niemeyer.net> References: <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <20061122165513.GA13795@niemeyer.net> <4564F8B7.5030007@canterbury.ac.nz> <20061123020054.GA31618@niemeyer.net> <45651355.7090605@canterbury.ac.nz> <20061123121416.GA6072@niemeyer.net> Message-ID: <45661AA3.9070803@canterbury.ac.nz> Gustavo Niemeyer wrote: > If you have an adapter from Apple to IFruit, any user > requiring an IFruit interface for an object will be > able to obtain it from Apple instances. The "automatically" > part is not quite true, as adaptation must be explicitly > requested. Yes, and that's where it falls down, as far as I can see. It's not just that it's "not quite" automatic, it's almost completely *non*-automatic. Given that, I see little benefit in having a formal mechanism for it. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From pje at telecommunity.com Thu Nov 23 23:27:05 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 23 Nov 2006 17:27:05 -0500 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: <79990c6b0611231353o70059496w8e2e04aea2786421@mail.gmail.co m> References: <5.1.1.6.0.20061123130727.01f42838@sparrow.telecommunity.com> <5.1.1.6.0.20061123130727.01f42838@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061123172607.01f443f0@sparrow.telecommunity.com> At 09:53 PM 11/23/2006 +0000, Paul Moore wrote: >OK. So under your proposal, all Python functions are (potentially) >generic, and addmethod is how you add extra overloads? (And hasmethod >is pretty obvious, and defop AFAICT is a syntax for addmethod, is that >right?) > >Did I get that right? Yes, that's exactly it. Thanks for the succinct summary of the entire thing (minus the interface example). As you may have noticed, I have a very hard time doing "succinct". :) From gustavo at niemeyer.net Thu Nov 23 23:47:59 2006 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Thu, 23 Nov 2006 20:47:59 -0200 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <45661AA3.9070803@canterbury.ac.nz> References: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <20061122165513.GA13795@niemeyer.net> <4564F8B7.5030007@canterbury.ac.nz> <20061123020054.GA31618@niemeyer.net> <45651355.7090605@canterbury.ac.nz> <20061123121416.GA6072@niemeyer.net> <45661AA3.9070803@canterbury.ac.nz> Message-ID: <20061123224759.GA27828@niemeyer.net> > Yes, and that's where it falls down, as far as I can > see. It's not just that it's "not quite" automatic, > it's almost completely *non*-automatic. Given that, > I see little benefit in having a formal mechanism > for it. I don't see your point here. Are you arguing that pretty much every approach being discussed is bad and we should instead implement something magically transforming types? I'm strongly -1 if that's the case. All proposals so far do some kind of conditional jumping only in specific entry points depending on features (class, interface, ability, whatever) declared for the object. -- Gustavo Niemeyer http://niemeyer.net From greg.ewing at canterbury.ac.nz Fri Nov 24 00:41:35 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 24 Nov 2006 12:41:35 +1300 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <20061123224759.GA27828@niemeyer.net> References: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <20061122165513.GA13795@niemeyer.net> <4564F8B7.5030007@canterbury.ac.nz> <20061123020054.GA31618@niemeyer.net> <45651355.7090605@canterbury.ac.nz> <20061123121416.GA6072@niemeyer.net> <45661AA3.9070803@canterbury.ac.nz> <20061123224759.GA27828@niemeyer.net> Message-ID: <456631AF.1020208@canterbury.ac.nz> Gustavo Niemeyer wrote: > Are you arguing that > pretty much every approach being discussed is bad and > we should instead implement something magically > transforming types? No, I'm questioning the need to do anything at all. I don't see a big problem that merits a heavyweight solution like a formal interface or adaptation framework. -- Greg From greg.ewing at canterbury.ac.nz Fri Nov 24 01:02:17 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 24 Nov 2006 13:02:17 +1300 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> <4565144B.5060109@canterbury.ac.nz> Message-ID: <45663689.5030006@canterbury.ac.nz> Guido van Rossum wrote: > I fear I will miss the class as a convenient focus for > related functionality. I have the same feeling. I get the impression that generic functions work okay in a language designed around them from the beginning. But adding them belatedly to a language built around classes and methods would give Too Many Ways To Do It. Another thing I don't like about generic functions is their comefromish nature. You know that an implementation of a method for some given type is going to be defined in one of a few easily-found places, but an implementation of a generic function for that type could be anywhere in the program. Or maybe that's just another way of stating your "classes as a focus of functionality" argument. In any case, from a software engineering viewpoint it seems bad, for the same reasons that monkeypatching is generally considered undesirable. -- Greg From greg.ewing at canterbury.ac.nz Fri Nov 24 01:09:44 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 24 Nov 2006 13:09:44 +1300 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122233433.028ea230@sparrow.telecommunity.com> References: <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122233433.028ea230@sparrow.telecommunity.com> Message-ID: <45663848.1000508@canterbury.ac.nz> Phillip J. Eby wrote: > Calling people names isn't particularly conducive to a technical > discussion... Sorry, I didn't mean it to sound that way. I was just making reference to a concept elucidated elsewhere -- invoking an "argument pattern", so to speak.-) What I'm saying is that, because the object returned by iter() not only has a different interface but a different *function*, it doesn't seem to fit the notion implied to me by "adaptation". And that to stretch the definition of adaptation to include it would be to make the term so broad it would lose much of its usefulness. -- Greg From guido at python.org Fri Nov 24 03:55:33 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Nov 2006 18:55:33 -0800 Subject: [Python-3000] optional argument annotations In-Reply-To: <33D8CBFC-9B3B-4FDE-927F-4B6D6B11C434@PageDNA.com> References: <33D8CBFC-9B3B-4FDE-927F-4B6D6B11C434@PageDNA.com> Message-ID: On 11/23/06, Tony Lownds wrote: > I have a working optional argument syntax implementation, I'm hoping > to get some direction on > the implementation decisions so far.... please see below. Wow! > Python 3.0x (p3yk:52824M, Nov 23 2006, 09:22:23) > [GCC 3.4.4 20050721 (Red Hat 3.4.4-2)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> def f()-> 1: pass > ... > >>> f.func_returns > 1 > >>> def f(): pass > ... > >>> f.func_annotations It would be ok if this returned {} too. (But None is fine too I think.) > >>> f.func_returns > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'function' object has no attribute 'func_returns' I would prefer this to be None. Attributes that don't always exist are a pain to use. > >>> def f(x:1): pass > ... > >>> f.func_annotations > {'x': 1} Very cool! I agree that Phillip's idea for simplification is worth a try. We're generally not too concerned over the cost of function declarations since they typically execute only once per program. As long as it's really cheap when no annotations are present (which would suggest that func_annotations should be None in that case since an empty dict is kind of expensive, at least in memory). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Nov 24 05:10:41 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Nov 2006 20:10:41 -0800 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <-4940442411288794438@unknownmsgid> References: <-6683921950610559197@unknownmsgid> <8853879015409334245@unknownmsgid> <-4940442411288794438@unknownmsgid> Message-ID: On 11/23/06, Bill Janssen wrote: > > > I don't know anything by CLOS. > > > > I'll drop off a copy of the book on Monday. Lord knows we have > > zillions of extra copies floating around PARC. > > I'll still drop off a copy (of Common Lisp the Language, version 2), > but there's no need to wait. It's on the Web at > http://www.supelec.fr/docs/cltl/clm/node260.html. See in particular > the "change-class" operation at > http://www.supelec.fr/docs/cltl/clm/node305.html. > > I think I'm still confused (happens a lot :-) about our method > namespace discussion. It seems to me that Python's method namespaces > work pretty much the same way that CLOS's do, already. That is, you > don't "clobber" an existing method in a base class when you define a > new method by the same name in a derived class; you just mask it. The > base class' method is still there, and can still be called explicitly. OK, maybe I misunderstood what you wrote. I thought I heard you say that "len" isn't just "len" -- it's the "len" defined by some interface (and presumably implemented in a base class), and if one defined a new "len" it wouldn't override the "len" defined by that interface (unless one explicitly stated that it did), it would just add a different method named "len". That would fly in the face of Python's lookup algorithm for methods (where the first "len" you find is the one you get). If that's not what you meant, all is probably well. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Nov 24 05:13:38 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Nov 2006 20:13:38 -0800 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <45660444.4090204@canterbury.ac.nz> References: <-6683921950610559197@unknownmsgid> <91ad5bf80611221348n59ecbfb6w930046b1378d49ca@mail.gmail.com> <91ad5bf80611221929o4230fccfh663bf7a5bc5564d@mail.gmail.com> <45660444.4090204@canterbury.ac.nz> Message-ID: I'll give you that it may not have been intentional. But the effect is the same. And I felt rather offended by his offhand rejection of "most HCI arguments" -- in my view sarcasm is quite a bit more acceptable than insult. I stick to my point that he is lacking essential social skills for effectively participating in an on-line discussion. On 11/23/06, Greg Ewing wrote: > Guido van Rossum wrote: > > Your presence here has been entirely a distraction. That's a troll to me. > > Steady on, Guido. Trolling is making deliberately > inflammatory statements with the intention of > stirring up controversy. That's a pretty heavy > accusation to make. You're effectively saying > he's setting out to be obnoxious, and I don't > see that in any of his postings. > > -- > Greg > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Nov 24 06:34:49 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Nov 2006 21:34:49 -0800 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: References: Message-ID: Phillip J. Eby wrote: > At 08:29 PM 11/22/2006 -0800, Guido van Rossum wrote: > >One thing that rubs me the wrong way about generic functions is that > >it appears to go against OO. Now I'm not someone to take OO as > >religion, but there's something uncomfortable (for me) about how, in > >Phillip's world, many things become functions instead of methods, > >which brings along concerns about the global namespace filling up, and > >also about functionality being spread randomly across too many > >modules. I fear I will miss the class as a convenient focus for > >related functionality. > > I originally proposed a solution for this back in January '05, but it was > too premature. But since you have now stated the problem that the proposal > was intended to solve, perhaps the solution has a chance now. :) > > I will try to be as concrete as possible. Let's start with an actual, > hopefully non-exploding 'Interface' implementation, based on an assumption > that we have generic functions available: Could you point out where the example uses generic functions? > class InterfaceClass(type): > def __init__(cls, name, bases, cdict): > for k,v in cdict.items(): > # XXX this should probably skip at least __slots__, > # __metaclass__, and __module__, but oh well > cdict[k] = AdaptingDescriptor(v) > > class Interface: > __metaclass__ = InterfaceClass > __slots__ = '__self__' > > def __init__(self, subject): > # this isinstance() check should be replaced by an > # 'unwrap()' generic function, so other adapter types > # will work, but this is just an example, so... > if isinstance(subject, Interface): > subject = subject.__self__ > self.__self__ = subject > > class AdaptingDescriptor: > def __init__(self, descriptor): > self.wrapped = descriptor > def __get__(self, ob, typ=None): > if ob is None: > return self > return self.wrapped.__get__(ob.__self__, typ) OK, that didn't explode my head, but it went straight over it. It may only be 20 or so lines of code, but it's so concentrated that I can't really understand it. All I get is that there's a class named Interface that does some magic. (I'm sure I could understand it, but it would take me a whiteboard and half an hour close reading of the code, and I just don't want to spend that time right now.) > Now, using this new "interface framework", let's implement a small > "mapping" typeclas... er, interface. > > class Mapping(Interface): > def keys(self): > return [k for k,v in self.items()] > def items(self): > return [k,self[k] for k in self.keys()] > # ... other self-recursive definitions > > What does this do? Well, we can now call Mapping(foo) to turn an arbitrary > object into something that has Mapping's generic functions as its methods, What are Mapping's generic functions? keys and items? How can I tell they are generic? What made them generic? > and invokes them on foo! (I am assuming here that normal functions are > implicitly overloadable, even if that means they change type at runtime to > do so.) I read your explanation of that assumption before but I still don't like it. I'd much rather have a slot where the overloading gets stored that's NULL if there's no overloading. We can optimize the snot out of it some other way. (I guess my problem with objects that change type is that it's not a common thing to happen in Python, even though it's explicitly possible, and I expect that it will cause all sorts of bizarre surprises for code that thinks it understands Python's object model. Anyway, it's a distraction. Maybe we can put the class-changing idea aside as a premature optimization.) > We could even use interfaces for argument type declarations, to > automatically put things in the "right namespace" for what the code expects > to use. What do you mean by "the right namespace"? What things are put there? > That is, if you declare an argument to be a Mapping, then that's > what you get. Sounds like implied adaptation to me, which was already explicitly rejected long ago. > If you call .keys() on the resulting adapted object and the > type doesn't support the operation, you get an error. How could it not support it if Mapping(x) succeeded? How has keys() suddenly turned from a method into an operation (assuming "operation" is a technical term and not just a different word for method)? > Too late a form of error checking you say? Well, make a more sophisticated > factory mechanism in Interface.__new__ that actually creates (and caches) > different adapter types based on the type of object being adapted, so that > hasattr() tests will work on the wrapped type, or so that you can get an > early error if none of the wrapped generic functions has a method defined > for the target type. > > A few important points here: > > 1. A basic interface mechanism is extemely simple to implement, given > generic functions Given my inability to understand it I can't yet agree with this claim. > 2. It is highly customizable with respect to error checking and other > features, even on a per-user basis, because there doesn't have to be only > one "true" Interface type to rule them all (or one true generic function > type either, but that's a separate discussion). I would think there are few approaches (in Python) that would require "one true interface type". Maybe Zope/Twisted do, but probably more for expedience than because it's the only way it could be done. > 3. It allows interfaces to include partial implementations, ala Ping and > Alex's past proposals, thus allowing you to implement partial mapping or > "file" objects and have the rest of the interface's implementation filled > in for you I'm not even sure I like that. It also seems to hinge on adaptation. I'd rather *not* make adaptation a key concept; to me adaptation and interfaces are completely orthogonal. I think of interfaces as ways to *talk* about sets of methods or operations or abilities, and I like that interfaces are objects so you can inspect what you're talking about, but I don't like the interface to play a role in the actual *use* of an object. Perhaps as analogy, the use of hasattr in current Python will help: I can test whether an object has an attribute, and if it does, use the attribute. But the attribute test is not used as an intermediary for using the attribute. Similar with isinstance -- I can test whether an object is a basestring, and if it is, use its lower() method. But the basestring object isn't involved in using the lower() method. This is somewhat in contrast to Java and other statically typed languages with dynamic typecasts -- there you might have an Object x that you believe is really a String, so you write "String s = (String)x" (I believe that's the syntax -- I'm a bit rusty) and then you can use String methods on s -- but not on x. I find that aspect unPythonic. > 4. It allows you to hide the very existence of the notion of a "generic > function", if you prefer not to think about such things You've hidden them so well that I can't find them in your example. :-) > 5. It even supports interface inheritance and interface algebra: > subclassing an interface allows adding new operations, and simple > assignment suffices to compose new interfaces, e.g.: > > class MappingItems(Interface): > items = Mapping.items > > Notice that nothing special is required, this "just works" as a natural > consequence of the rest of the implementation shown. This I believe I actually understand. > Okay, so now you want to know how to *implement* a "Mapping". Well, > simplest but most tedious, you can just register operations directly, e.g.: > > class MyMapping: > def __init__(self, data): > self.data = dict(data) > defop operator.getitem(self, key): > return self.data[key] > defop Mapping.items(self): > return self.data.items() Looks like you're being inconsistent with your example above, which has keys() and items() but not __getitem__(). This confuses me. I'm also concerned that this looks to me like a big step back from simply writing class MyMapping: def __init__(self, data): self.data = dict(data) def __getitem__(self, key): return self.data[key] def items(self): return self.data.items() and then (either inside the class or external to it) adding some statement that claims that MyMapping implements Mapping. > But as you can imagine, this would probably get a bit tedious if you're > implementing lots of methods. So, we can add metaclasses or class > decorators here to say, "I implement these interfaces, so any methods I > have whose names match the method names in the interfaces, please hook 'em > up for me." I'm going to leave out the implementation, as it should be a > straightforward exercise for the reader to come up with many ways by which > it can be accomplished. The spelling might be something like: > > class MyMapping: > implements(Mapping) > > def items(self): > ... > > #etc. Sure, that's what I was after above. But how would you do this after the fact, when you find that some 3rd party module already has the right methods but didn't bother to add the correct implements() clause? > At which point, we have now come full circle to being able to provide all > of the features of interfaces, adaptation, and generic functions, without > forcing anyone to give up the tasty OO flavor of method calls. Heck, they > can even keep the way they spell existing adaptation calls (e.g. IFoo(bar) > to adapt bar to IFoo) in PEAK, Twisted, and Zope! > > And finally, note that if you only want to perform one method call on a > given object, you can also use the generics directly, e.g. > Mapping.items(foo) instead of Mapping(foo).items(). But I want to write foo.items()!!! > Voila -- generic goodness and classic OO method-calling simplicity, all in > one simple to implement package. It should now be apparent why I said that > interfaces are trivial to implement if you define them as namespaces for > generic functions, rather than as namespaces for methods. > > There are many spinoffs possible, too. For example, you could have a > factory function that turns an existing class's public operations into an > interface object. There are also probably also some dark corners of the > idea that haven't been explored, because when I first proposed basically > this idea in '05, nobody was ready for it. Now maybe we can actually talk > about the implications. Well, okay, I have to admit that I only half "get" what that was all about. But let me end on a more positive note. I find the idea very attractive that many builtin operations (from len to iter to + to <= to __getitem__) can be considered to be generic functions. In my head I worked out how you could take the built-in set type and the (ancient) set-of-integers implementation hiding in mhlib.py (mhlib.IntSet), and add the various set-theoretical operations (union, intersection, differences, and containment relationships) on mixed operands without modifying the code of the IntSet class. This is pretty neat. You could even add those operations for two IntSet instances without modifying that class. I am envisioning that most of these operations would have a default implementation that does what those operations currently do; e.g. the generic function __add__ would be something like @generic def __add__(a, b): # default implementation if hasattr(a, "__add__"): r = a.__add__(b) if r is not NotImplemented: return r if hasattr(b, "__radd__"): r = b.__radd__(a) if r is not NotImplemented: return r raise TypeError(...) I also like the idea that we can associate an "atomic" interface with a generic function, so that an object for which that GF has an implementation is automatically said to implement that interface. And I like the idea that we can compose new interfaces out of such atomic interfaces, such that testing for whether an object implements such a composite interface is automatically turned into testing whether it implements each of the atomic interfaces. And I also like being able to take arbitrary subsets of composite interfaces as the logical consequence of this. (I don't think the name of the interface should be the same as the name of the generic function. This has caused me enough confusion already. I'd rather be able to say e.g. that there's an interface "Sizeable" which means that the operation "len" is provided.) I think there are still some important holes in this idea (at least in my understanding). For example, if all generic functions have a default implementation, then essentially all generic functions are implemented for all objects, and all objects have all interfaces! That's not right. Of course, the default implementation may well raise TypeError for some arguments (like my __add__ example). But that's not helpful in deciding whether an object has an interface -- invoking the operation is way too heavy-handed. Maybe we should provide a separate testing function. (Although that doesn't seem quite sufficient for the __add__ example, since it really *does* need to execute the operation in order to determine whether it is implemented. I'll have to come back to this problem later.) Another issue is GFs taking multiple arguments. At least for binary operators (like __add__ above) it may make sense to say that a class C implements the operator's interface (e.g. Addable in this case) if the operator is defined for operands that are both C instances. Or maybe that interface should be called SelfAddable, and we should reserve Addable for anything that can occur on the lhs of a + operator? I think we can probably come up with a reasonable rule for this; right now I'm too tired to think of one. But the biggest hole (to me) is being able to talk about regular methods in an interface. I would like to be able to say that e.g. the interface StandardMapping implements the operations getitem, len, iter and contains, and the methods get, keys, values and items. I have no problem with the operations, but I don't quite get how we can add the methods to the interface. I understand that your explanation above provides ways to talk about those, I just don't quite see the light yet. I would like to end up in a world where, in order to claim that a class is a standard mapping class, the class definition would (normally) explicitly claim to implement StandardMapping (using as yet unspecified syntax) and that there would be a way (again I'm not yet specifying syntax for this) to efficiently test any given class or instance for that interface, returning True or False. The standard dict implementation would claim to implement StandardMapping (and probably also something that we could call StandardMutableMapping). Someone could define an interface MinimalMapping as the interface formed of StandardMapping.getitem, StandardMapping.contains and StandardMapping.keys, and testing a dict for MinimalMapping would return True. But if someone created ExtendedMapping as StandardMapping plus the copy method and the eq operation, testing a dict for ExtendedMapping would return False; however there should be something one could execute to explicitly mark dict as implementing ExtendedMapping, using some kind of registry. Reiterating, I wonder if perhaps a major disconnect has to do with the difference between "testing" for an interface vs. "casting" to an interface. I don't like casting, since it smells like adaptation. (There's also an extension of these ideas I'd like to explore where full method signatures are possible in an interface, in an introspectable way. This would probably require parameterizable types, so that one could talk about StandardMapping[int, str] (a standard mapping from ints to strings), and one could derive that the argument to getitem is an int and its return value is a str, and so on; keys() would return an Iterable of ints (or an IterableSet of ints, or whatever), for example. But that's also for a later post.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik at pythonware.com Fri Nov 24 07:49:03 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 24 Nov 2006 07:49:03 +0100 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <06Nov23.103924pst."58648"@synergy1.parc.xerox.com> References: <-6683921950610559197@unknownmsgid> <8853879015409334245@unknownmsgid> <06Nov22.181438pst."58648"@synergy1.parc.xerox.com> <06Nov23.103924pst."58648"@synergy1.parc.xerox.com> Message-ID: Bill Janssen wrote: > I think method namespaces are already OK; the question is > whether to move these specially-named (for whatever reason) > methods to a real namespace (a dict somewhere) where "somewhere" is exactly where? how do I add things to that namespace? how do I specify what namespace to look in when accessing methods or attributes? From rrr at ronadam.com Fri Nov 24 08:00:05 2006 From: rrr at ronadam.com (Ron Adam) Date: Fri, 24 Nov 2006 01:00:05 -0600 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <456631AF.1020208@canterbury.ac.nz> References: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <20061122165513.GA13795@niemeyer.net> <4564F8B7.5030007@canterbury.ac.nz> <20061123020054.GA31618@niemeyer.net> <45651355.7090605@canterbury.ac.nz> <20061123121416.GA6072@niemeyer.net> <45661AA3.9070803@canterbury.ac.nz> <20061123224759.GA27828@niemeyer.net> <456631AF.1020208@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Gustavo Niemeyer wrote: >> Are you arguing that >> pretty much every approach being discussed is bad and >> we should instead implement something magically >> transforming types? > > No, I'm questioning the need to do anything at all. > I don't see a big problem that merits a heavyweight > solution like a formal interface or adaptation > framework. > > -- > Greg I think I agree with you Greg, although it is still early and something simpler and possibly more direct may yet come from the discussion. It does look like it is starting to be a bit more focused. It might be nice to have some selected examples of current library code that may benefit from interfaces/abilities, with a brief comment of where it currently falls short for each example. These may be useful later in a PEP. Earlier messages here had sometimes vague suggestions of what interfaces would do, but it seems different people have different ideas of that. Some of those implied ideas are: [A probably incomplete list with letter for referencing] (A.) Better introspection to use in situations where type(), isinstance(), or hasattribute(), is perceived as being less than optimal. (B.) Adding an additional or alternate method of auto-dispatching either through generic functions or though the class structure. (C.) Selecting an Adapter, or using adapters. (D.) Verifying or Testing a function/object/or method to determine it's ability or signature. (Or some other variation of pragmatically determining a signature.) I believe Guido already set aside discussion of "C" and "D", as they are implementation details that can be examined separately and added later if it they seem to be promising. (Guido may have other reasons.) So that leaves "A" and "B". Most of the discussion is focused on "B" at the moment. The optional signatures proposed in a different thread may serve for "A". I don't know how much "A" and "B" overlap, possibly only a little or not at all depending on how "B" is implemented, or it may be that a good solution to "A" is needed in order for "B" to work well enough to bother with. It may also be that only a good unified solution to "A" is needed in order to allow library and user code to be written "easier" to address "B","C" and "D". It may even be just doing "A" is a good enough 95% solution (works well enough for 95% of actual use cases) and nothing more needs to be done. There has also been put forth various, (also sometimes vague), suggestion of just what item "A" is. Does "A" return information of "is_a", "has_a", or "does_a" relationships? A "does_a" relationship may be difficult to define. Mixing these in a signature may not be as straight forward as it may seem. The optional signatures addresses "is_a" relationships fairly well, but not "has_a" or "does_a" relationships. Anyway, I'm waiting on a PEP and hoping I will be able to understand it. A difficult to understand and use 99% solution is probably not better than what we currently have I think. Cheers, Ron From jimjjewett at gmail.com Fri Nov 24 09:11:50 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 24 Nov 2006 03:11:50 -0500 Subject: [Python-3000] optional argument annotations In-Reply-To: References: <33D8CBFC-9B3B-4FDE-927F-4B6D6B11C434@PageDNA.com> Message-ID: On 11/23/06, Guido van Rossum wrote: > On 11/23/06, Tony Lownds wrote: > > I have a working optional argument syntax implementation, I'm hoping > > to get some direction on > > the implementation decisions so far.... please see below. I would rather see it integrated into a Signature object (Brett had an implementation), instead of showing up as two separate attributes. > > >>> f.func_returns > > Traceback (most recent call last): > > File "", line 1, in > > AttributeError: 'function' object has no attribute 'func_returns' > I would prefer this to be None. Attributes that don't always exist are > a pain to use. I suspect he was trying to distinguish "returns None" from "Doesn't say what it returns". There is no good way to do this, but a separate flag on a Signature object seems the least bad to me. -jJ From gustavo at niemeyer.net Fri Nov 24 15:00:37 2006 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Fri, 24 Nov 2006 12:00:37 -0200 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: <5.1.1.6.0.20061123153733.01f418e8@sparrow.telecommunity.com> References: <5.1.1.6.0.20061123153733.01f418e8@sparrow.telecommunity.com> Message-ID: <20061124140037.GA10468@niemeyer.net> > Let's see your implementation. Zope's interface system fails on at > least points 3 and 5. And regarding #1, Zope's implementation of the > basic features I described is an order of magnitude more complex, even > if you compare *just* its code that implements adaptation and > interface inheritance, with *all* of my code plus Guido's generic > function implementation. Comparing complexity of implementations which offer completely different end results isn't of great value. Copying methods from the interface to the class is trivial, as you're aware of (it doesn't mean it's a good idea). Support for partial implementations may also be easily achieved, but its drawbacks must be considered as well (how to deal with marker interfaces, etc). In any case, here is a silly implementation showing that it's easily feasible without generic functions. It's slow, since it does linear searching, but there are several paths for speed improvements. class InterfaceClass(type): def __init__(cls, name, bases, attrs): signature = set() for base in bases: signature.update(base.__signature__) for name, attr in attrs.items(): if name not in ("__module__", "__metaclass__"): if isinstance(attr, UnboundMethodType): attr = attr.im_func signature.add((name, attr)) cls.__signature__ = frozenset(signature) class Interface: __metaclass__ = InterfaceClass class AdapterRegistry(object): def __init__(self): self._adapters = [] def register(self, adapter, iorigin, itarget): self._adapters.append((adapter, iorigin.__signature__, itarget.__signature__)) def adapt(self, obj, iface): origin = set() for obj_iface in obj.__interfaces__: origin.update(obj_iface.__signature__) target = iface.__signature__ if origin.issuperset(target): return obj for adapter, aorigin, atarget in self._adapters: if origin.issuperset(aorigin) and target.issubset(atarget): return adapter(obj) return None That's it. Here is a tested example: class C(Interface): def m1(): pass class D(Interface): m1 = C.m1 def m2(): pass class E(Interface): def m3(): pass class F(E): def m4(): pass registry = AdapterRegistry() class Foo(object): __interfaces__ = [D] registry.register(lambda x: "Adapted!", C, E) foo = Foo() print registry.adapt(foo, E) # => "Adapted!" print registry.adapt(foo, D) # => foo print registry.adapt(foo, F) # => None > 4. No need to understand interface or adaptation theory to be able > to use it Sure.. no need to understand concepts you're not using. You have to understand what you *are* using instead. -- Gustavo Niemeyer http://niemeyer.net From python3000 at davious.org Fri Nov 24 15:58:50 2006 From: python3000 at davious.org (Dave Anderson) Date: Fri, 24 Nov 2006 09:58:50 -0500 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: References: Message-ID: <456708AA.7040907@davious.org> on 11/24/2006 12:34 AM Guido van Rossum wrote: > I would like to end up in a world where, in order to claim that a > class is a standard mapping class, the class definition would > (normally) explicitly claim to implement StandardMapping (using as yet > unspecified syntax) Guido, In my world http://mail.python.org/pipermail/python-3000/2006-November/004736.html I feel have a usable syntax for you class StandardMapping: ... class Foo: implements StandardMapping ... > and that there would be a way (again I'm not yet > specifying syntax for this) to efficiently test any given class or > instance for that interface, returning True or False. Foo.does_implement(StandardMapping) -> True # based solely on the declaration of Foo implements StandardMapping > The standard > dict implementation would claim to implement StandardMapping (and > probably also something that we could call StandardMutableMapping). class StandardMutableMapping: implements StandardMapping ... dict.implements(StandardMutableMapping) dict.does_implement(StandardMapping) -> True # based on dict's declaration of implementing StandardMutableMapping # and StandardMutableMapping's declaration of implementing # StandardMapping > Someone could define an interface MinimalMapping as the interface > formed of StandardMapping.getitem, StandardMapping.contains and > StandardMapping.keys, MinimalMapping = interface(StandardMapping.getitem, StandardMapping.contains, StandardMapping.keys) > and testing a dict for MinimalMapping would > return True. dict.does_implement(MinimalMapping) -> True # Since dict has declared that it implements StandardMutableMapping # and StandardMutableMapping has declared it implements StandardMapping # so, introspection would stop there, since a class implementation # declaration is pledging that all StandardMapping's methods are # implemented > But if someone created ExtendedMapping as StandardMapping > plus the copy method and the eq operation, testing a dict for > ExtendedMapping would return False; ExtendedMapping = interface(StandardMapping, "copy", "__eq__") dict.does_implement(ExtendedMapping) -> True # According to the current implementation of dict # "copy", "__eq__" are implemented by dict, looking at dir(dict) # As for StandardMapping, the above declaration chain of dict implements # StandardMutableMapping and then StandardMutableMapping implements # StandardMapping then # dict implements all of ExtendedMapping > however there should be something > one could execute to explicitly mark dict as implementing > ExtendedMapping, using some kind of registry. dict.implements(ExtendedMapping) dict.does_implement(ExtendedMapping) -> True # Now, since I've explicitly claimed that dict implements # ExtendedMapping, my introspection stops there. # the previous introspection described just above has been made # unnecessary From tomerfiliba at gmail.com Fri Nov 24 16:18:24 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Fri, 24 Nov 2006 17:18:24 +0200 Subject: [Python-3000] iostack and Oh Oh Message-ID: <1d85506f0611240718s364f83c2ya3e9e12e2ca7354c@mail.gmail.com> recap: we agreed py3k needs a unified, better organized and unbuffered new io stack. i began working on the subject several months back, and we had long discussions on what it's design. you can see where we got at the following page (although not up to date): http://sebulba.wikispaces.com/project+iostack+v2 i mostly neglected it in the past months, and got back to only now. the problem is, i got confused with all the current OO/generics/interfaces talks. back in the day, we said all stream objects would support read() and write(), and streams with limited functionality (readonly, writeonly) would be "queriable" using the following properties: * writable * readable * seekable i.e., >>> f = FileStream("somefile.txt", "r") >>> f.readable True >>> f.writable False but in light of the current discussion, maybe we'd prefer better a java-like approach, separating the stream into readers and writers over some fundamental "source". i.e., a source would be a socket, file, StringIO, whatever, over which we'll put a reader/writer layer. it may work well for uni-directional stream, but for sockets it causes some overhead: >>> s = socket.socket(...) >>> sr = SocketReader(s) >>> sw = SocketWriter(s) >>> sr.read(5) >>> sw.write("hello") you have to construct both a writer an a reader to work properly with a socket, and use a different object for different operations... doesn't feel right to me. also, it would be a huge headache to keep in sync the buffers of both reader/writer, when a source is opened for both read and write... - - - - - - - - - we had another problem, iirc, namely "seeking-decoding problem": what is the meaning of seeking in a character-stream over a file-stream? the underlying file-stream works with bytes, while the text stream works with characters of mixed widths... * how do we handle unaligned-seeking? * should tell() return "opaque cookies", instead of ints, so we could seek only to locations that tell() returned? * still, if we modify the stream, these cookies need to be invalidated... here's an example: >>> f = TextAdapter(FileStream("somefile.txt", "w+"), "utf16") >>> p0 = f.tell() # filepos = 0 >>> f.write("ABC") >>> p1 = f.tell() # filepos = 3 >>> f.seek(p0) >>> f.write(u"\uffff") # takes 4 bytes >>> f.seek(p1) >>> f.read(1) # in the middle of a character now... there was a suggestion back in the day, to make streams sequential by definition (i.e., no seek/tell) -- mostly like java. a StreamReader can skip() a number of characters (essentially equivalent to reading and discarding the result) instead of seeking at the stream level, the "sources" would grow indexing (getitem/setitem) for that. a seekable source (only files actually) could be accessed directly: >>> f = FileSource("somefile.txt", "r") >>> f[10:20] = "hello worl" i.e., the source acts like a sequence of bytes. but of course sources should be unbuffered, which may prove problematic when we have buffering added by streams on top of the source... bladt - - - - - - - - - any suggestions? i'd guess the interfaces-supporters (Bill?) would prefer a java-like stack (where subclasses of StreamReader provide reading and subclasses of StreamWriter provide writing); duck-typing-supporters and conservatives would probably prefer the succinct queiable approach (streams support both read and write, but can be checked using the relevant properties) we ought to agree on the design here, if this is supposed to replace the entire io stack... it may also be a good starting place for the supporters of each camp to demonstrate how their approach "does it better". tata, - tomer From guido at python.org Fri Nov 24 17:30:12 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 24 Nov 2006 08:30:12 -0800 Subject: [Python-3000] optional argument annotations In-Reply-To: References: <33D8CBFC-9B3B-4FDE-927F-4B6D6B11C434@PageDNA.com> Message-ID: On 11/24/06, Jim Jewett wrote: > On 11/23/06, Guido van Rossum wrote: > > On 11/23/06, Tony Lownds wrote: > > > I have a working optional argument syntax implementation, I'm hoping > > > to get some direction on > > > the implementation decisions so far.... please see below. > > I would rather see it integrated into a Signature object (Brett had an > implementation), instead of showing up as two separate attributes. Yes, you're right; I forgot about that. > > > >>> f.func_returns > > > Traceback (most recent call last): > > > File "", line 1, in > > > AttributeError: 'function' object has no attribute 'func_returns' > > > I would prefer this to be None. Attributes that don't always exist are > > a pain to use. > > I suspect he was trying to distinguish "returns None" from "Doesn't > say what it returns". There is no good way to do this, but a separate > flag on a Signature object seems the least bad to me. Hm, I think it would be fine if there *was* no distinction. IOW if def foo(a: None) -> None: pass was indistinguishable from def foo(a): pass In fact I think I'd prefer it that way. Having an explicit way to say "no type here, move along" sounds fine. It's different from defaults, whose absence translates in different behavior (i.e., requiring that a value be passed in). (It does imply that we can't use "-> None" to indicate "returns no value". We'd have to spell that as "-> type(None)" or find some other way. I think I don't care. Do others? Collin?) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Nov 24 18:19:16 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 24 Nov 2006 09:19:16 -0800 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: References: Message-ID: I'd like to take the opportunity to present another example of what I'd like to be able to do with abilities and generic functions. It's short on implementation and I don't want to emphasize syntax, but it's high on concepts. The baseline assumption is that most built-in operations, especially binary operators, are implemented as generic functions. Let's suppose we didn't have a set type yet, and we wanted to introduce it. I'd start by defining three abilities: MinimalSet which only provides the 'in' operator; IterableSet which also provides iteration (but not len(); it may aply to infinite sets); and StandardSet which adds most standard operations on sets, but still not len(). The operations on standard sets are: & (intersection), | (union), ^ (symmetric difference), - (asymmetric difference), <= (is subset), >= (is superset), < (strict subset), and > (strict superset). Sets also implement == and !=, as these are inherited from an even more basic ability (say, Object). Mutable operations are not part of this ability. One reason why I'm saying ability instead of interface is that not every object that implements the above operations should automatically be considered a set! If a class claims to be a set, it should conform to a much richer contract, which guarantees the basic properties of mathematical sets. For example, a&b == b&a, a&a == a, and so on. The contract implies the existence of an empty set, and all empty sets should compare equal (even if they have different underlying implementations). So StandardSet implies more than a bunch of operations! In particular, subclassing int and adding implementations for __contains__ and __iter__ should *not* result in that subclass having the StandardSet ability (unless I explicitly declare it). Note: while this appears incompatible with Phillip's proposal "interfaces are just collections of operations", I think this can be consolidated easily by allowing some kind of "marker" operation. Now I'd like to introduce a standard set implementation. It implements all those operations. But I'd like it to automatically accept operands of other set implementations, as long as they claim the relevant ability. For example, in Python 2.4, the built-in set implementation's __and__ is defined roughly like this (ignoring some optimizations, like that it's written in C :-): # This a method of class 'set', a concrete set implementation def __and__(self, other): if not isinstance(other, baseset): return NotImplemented result = set() for elem in self: if elem in other: result.add(elem) return result But I'd like to be able to write it like this instead (still inside the 'set' class): def __and__(self, other: MinimalSet): result = set() for elem in self: if elem in other: result.add(elem) return result which means that it automatically interoperates with other set implementations. The header "def __and__(self, other: MinimalSet)" is meant to be enough syntax to declare an implementation of the pre-existing __and__ operations for the pair (set, MinimalSet) -- the first being a concrete type (class), the second being an ability. Actually, I might want to define three implementations of __and__: one for the pair (set, set), which could be optimized based on knowledge of the internals; one for (set, MinimalSet); and one for (MinimalSet, set). The exact syntax for all this is not that important; the latter one may have to be defined outside the class as a true generic function; or defining a method __radd__ could do the trick; or even aliasing __radd__ to __add__ (like we would do today). I believe this example captures an important requirement that has been expressed both by Bill Janssen and by Andrew Koenig (needless to say I agree): abilities should be able to imply contracts that go beyond a collection of methods (without necessarily being able to express those contracts in code). The example doesn't show a way to test whether an object has an ability, but I certainly want to make that possible and even efficient; maybe isinstance(x, A) could be made to work (it's already pretty flexible under the hood, and could easily be turned into a GF itself), or maybe you'll have to write has_ability(x, A). Note that I haven't shown syntax for declaring the various abilities. That is because I don't have a concrete proposal yet. However, I would prefer syntax that looks declarative (similar to a class declaration) for the normal case. Syntax that looks procedural (e.g. "IterableSet = MinimalSet + iter") might be available as an alternative way to create abilities more dynamically, just like in a pinch we can create a new class today by calling type(name, bases, classdict). I still don't have new ideas for how to add regular-looking methods (like keys()) to an ability. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ark-mlist at att.net Fri Nov 24 18:53:07 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Fri, 24 Nov 2006 12:53:07 -0500 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: Message-ID: <001601c70ff1$663f54f0$6402a8c0@arkdesktop> > I believe this example captures an important requirement that has been > expressed both by Bill Janssen and by Andrew Koenig (needless to say I > agree): abilities should be able to imply contracts that go beyond a > collection of methods (without necessarily being able to express those > contracts in code). Yes. For that matter, I would like an ability to be able to say things about a + b regardless of whether it happens to be implemented through __add__ or __radd__ -- which is yet another reason why abilities need to be more than just collections of methods. > Note that I haven't shown syntax for declaring the various abilities. > That is because I don't have a concrete proposal yet. However, I would > prefer syntax that looks declarative (similar to a class declaration) > for the normal case. A thought borrowed from ML: You wrote def __and__(self, other: MinimalSet): and that reminds me of ML's syntax: fun f(x, y: MininmalSet) = (* whatever *) In the ML version, we're saying that f's first argument can be of any type (that meets whatever other type constraints are implied by f's body, and the second argument must be of type MinimalSet in addition to those constraints. Seeing this analogy, I am tempted to carry it further. In ML, I can write val foo = bar: MinimalSet which binds foo to the value of bar, and also implies a (compile-time) type check that bar has type MinimalSet. So consider extending that analogy to Python: foo = bar: MinimalSet which would verify (at run time) that bar has the MinimalSet ability, and then bind foo to the same object as bar. If bar doesn't have the MinimalSet ability, this would do whatever would have happened if you tried to call the __and__ function above -- probably raise an exception. From tony at pagedna.com Fri Nov 24 18:59:03 2006 From: tony at pagedna.com (Tony Lownds) Date: Fri, 24 Nov 2006 09:59:03 -0800 Subject: [Python-3000] optional argument annotations In-Reply-To: References: <33D8CBFC-9B3B-4FDE-927F-4B6D6B11C434@PageDNA.com> Message-ID: <99B2A3E4-1226-45D5-BC0D-DF72890A8D54@pagedna.com> On Nov 24, 2006, at 8:30 AM, Guido van Rossum wrote: >> I would rather see it integrated into a Signature object (Brett >> had an >> implementation), instead of showing up as two separate attributes. > > Yes, you're right; I forgot about that. > I'd imagine that the Signature object would live on top of this change. >>>>>>> f.func_returns >>>> Traceback (most recent call last): >>>> File "", line 1, in >>>> AttributeError: 'function' object has no attribute 'func_returns' >> >>> I would prefer this to be None. Attributes that don't always >>> exist are >>> a pain to use. >> >> I suspect he was trying to distinguish "returns None" from "Doesn't >> say what it returns". There is no good way to do this, but a >> separate >> flag on a Signature object seems the least bad to me. > > Hm, I think it would be fine if there *was* no distinction. IOW if > > def foo(a: None) -> None: pass > > was indistinguishable from > > def foo(a): pass > > In fact I think I'd prefer it that way. Having an explicit way to say > "no type here, move along" sounds fine. It's different from defaults, > whose absence translates in different behavior (i.e., requiring that a > value be passed in). > > (It does imply that we can't use "-> None" to indicate "returns no > value". We'd have to spell that as "-> type(None)" or find some other > way. I think I don't care. Do others? Collin?) > That's fine with me, but just to toss out another idea: we could put the return annotation in func_annotations['return']. -Tony From tony at pagedna.com Fri Nov 24 19:00:48 2006 From: tony at pagedna.com (Tony Lownds) Date: Fri, 24 Nov 2006 10:00:48 -0800 Subject: [Python-3000] optional argument annotations In-Reply-To: References: <33D8CBFC-9B3B-4FDE-927F-4B6D6B11C434@PageDNA.com> Message-ID: <523FAB43-7859-437B-9A80-167F14A5291D@pagedna.com> >>>>> f.func_annotations >> {'x': 1} > > Very cool! I agree that Phillip's idea for simplification is worth a > try. We're generally not too concerned over the cost of function > declarations since they typically execute only once per program. As > long as it's really cheap when no annotations are present (which would > suggest that func_annotations should be None in that case since an > empty dict is kind of expensive, at least in memory). > > func_annotations could create the empty dict just in time. Building the annotations dict in bytecode actually takes slightly more C code and quite a bit more bytecode. Maybe using dict(zip(, )) would be more efficient. I'll probably keep both the MAKE_FUNCTION change and the bytecode version for a while. The next step is changing the syntax for nested parameters, eg def f((x:1, y:2)) >>> dis.dis(compile("def f(x:1, y:2=3, z=4)->5:pass", "", "exec")) 1 0 LOAD_CONST 0 (3) 3 LOAD_CONST 1 (4) 6 LOAD_CONST 2 (", line 1>) 9 MAKE_FUNCTION 2 12 DUP_TOP 13 BUILD_MAP 0 16 DUP_TOP 17 LOAD_CONST 3 (1) 20 ROT_TWO 21 LOAD_CONST 4 ('x') 24 STORE_SUBSCR 25 DUP_TOP 26 LOAD_CONST 5 (2) 29 ROT_TWO 30 LOAD_CONST 6 ('y') 33 STORE_SUBSCR 34 ROT_TWO 35 STORE_ATTR 0 (func_annotations) 38 DUP_TOP 39 LOAD_CONST 7 (5) 42 ROT_TWO 43 STORE_ATTR 1 (func_returns) 46 STORE_NAME 2 (f) 49 LOAD_CONST 8 (None) 52 RETURN_VALUE >>> dis.dis(compile("def f(x:1, y:2=3, z=4)->5:pass", "", "exec")) >>> 1 0 LOAD_CONST 0 (3) # default for y 3 LOAD_CONST 1 (4) # default for z 6 LOAD_CONST 2 (1) # annotation for x 9 LOAD_CONST 3 (2) # annotation for y 12 LOAD_CONST 4 (6) # bitmask for annotations... 0b110 15 LOAD_CONST 5 (5) # func_returns 18 LOAD_CONST 6 (", line 1>) 21 MAKE_FUNCTION 49154 # numdefaults(2) + kwdefaults(0) << 8 + has_annotations(1) << 14 + has_returns(1) << 15 24 STORE_NAME 0 (f) 27 LOAD_CONST 7 (None) 30 RETURN_VALUE From brett at python.org Fri Nov 24 20:20:39 2006 From: brett at python.org (Brett Cannon) Date: Fri, 24 Nov 2006 11:20:39 -0800 Subject: [Python-3000] optional argument annotations In-Reply-To: References: <33D8CBFC-9B3B-4FDE-927F-4B6D6B11C434@PageDNA.com> Message-ID: On 11/24/06, Guido van Rossum wrote: > On 11/24/06, Jim Jewett wrote: > > On 11/23/06, Guido van Rossum wrote: > > > On 11/23/06, Tony Lownds wrote: > > > > I have a working optional argument syntax implementation, I'm hoping > > > > to get some direction on > > > > the implementation decisions so far.... please see below. > > > > I would rather see it integrated into a Signature object (Brett had an > > implementation), instead of showing up as two separate attributes. > > Yes, you're right; I forgot about that. > Guess I need to make sure to bug you about pronouncing on the PEP once I get keyword-only arguments support added before you forget again. =) Obviously signature objects would grow support for annotations, but I still need the information to be carried on the code object to incorporate into signature objects. Another option for how to handle annotations is to take the route of having a tuple listing all of the strings representing the annotations, using None for arguments that have no annotation. That way there is no issue distinguishing whether an annotation exists or not plus it is a cheap operation since you are then just storing strings. This also allows the last value in the tuple to imply the return annotation and thus only have a single attribute for annotations. Then you can have signature objects worry about pulling out the info to tie into specific parameters to work with the annotations. > > > > >>> f.func_returns > > > > Traceback (most recent call last): > > > > File "", line 1, in > > > > AttributeError: 'function' object has no attribute 'func_returns' > > > > > I would prefer this to be None. Attributes that don't always exist are > > > a pain to use. > > > > I suspect he was trying to distinguish "returns None" from "Doesn't > > say what it returns". There is no good way to do this, but a separate > > flag on a Signature object seems the least bad to me. > > Hm, I think it would be fine if there *was* no distinction. IOW if > > def foo(a: None) -> None: pass > > was indistinguishable from > > def foo(a): pass > > In fact I think I'd prefer it that way. Having an explicit way to say > "no type here, move along" sounds fine. It's different from defaults, > whose absence translates in different behavior (i.e., requiring that a > value be passed in). > This also mirrors the default way that functions work since they return None if you don't bother to specify somethng. > (It does imply that we can't use "-> None" to indicate "returns no > value". We'd have to spell that as "-> type(None)" or find some other > way. I think I don't care. Do others? Collin?) > I always figured we would return type. -Brett From ark-mlist at att.net Fri Nov 24 20:42:31 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Fri, 24 Nov 2006 14:42:31 -0500 Subject: [Python-3000] optional argument annotations In-Reply-To: <99B2A3E4-1226-45D5-BC0D-DF72890A8D54@pagedna.com> Message-ID: <000a01c71000$ae9a7130$6402a8c0@arkdesktop> > Hm, I think it would be fine if there *was* no distinction. IOW if > > def foo(a: None) -> None: pass > > was indistinguishable from > > def foo(a): pass > > In fact I think I'd prefer it that way. Having an explicit way to say > "no type here, move along" sounds fine. I'd like to urge against making None magical -- I don't see any reason for it. Instead, consider that if I say def foo(a: T1) -> T2: I am presumably going to accept an argument of type T1 or a type derived from T1, and return a result of type T2 or a type derived from T2. In which case, def foo(a): pass should be equivalent to def foo(a: object) -> object: pass and we don't need to recycle None for the purpose -- especially as ininstance(None,object) yields True and I presume that isn't going to change. From python3000 at davious.org Fri Nov 24 22:15:24 2006 From: python3000 at davious.org (Dave Anderson) Date: Fri, 24 Nov 2006 16:15:24 -0500 Subject: [Python-3000] My poor-man's non-oo implementation of my proposed interface and implements/implement syntax In-Reply-To: References: Message-ID: <456760EC.6070503@davious.org> class Interface(object): """ A hashable, type-identifiable 'list' The list should contain only other Interface objects , Classes/Types, or Class Methods/Attributes, and strings Interface objects can be used in specifying dispatch rules and function parameter "types" """ def __init__(self, *args): self.interfacelist = list(args) class Implementation(dict): """This is used just to indicate that implementation sets are dicts Reasoning is to allow for adaptation on the class or method level by a class reference pointing to a transform method or a class method reference pointing to a method definition (this is not implemented, though, and would be dependent on smart dispatching) """ pass def implements(class_, *interface_objects): """Declares that a class implements one or more interface objects See Interface class's description for allowed items """ for interface in interface_objects: # the implementation slot in a real implementation # would allows be an object attribute if not class_.__dict__.has_key('implementation'): class_.implementation = Implementation() # the class vouches to implement an Interface list if isinstance(interface, Interface): class_.implementation[interface] = class_ # the class vouches to implement a class/type if isinstance(interface, type): class_.implementation[interface] = class_ # the class vouchers to implement a # non-class-specific method/attribute elif isinstance(interface, str): class_.implementation[interface] = interface # the class vouchers to implement a # class-specific method/attribute elif "__name__" in dir(interface): # this is a class-specific method: class_.implementation[interface] = \ interface.__name__ else: raise TypeError, interface, \ "must be an Interface, Class/Type, Class Method/Attribute, or string" def implement(class_, obj, adaptor_or_method): """a special adaption case for declaring an implementation upon dispatch the adaptor_or_method value would be used depending on context class context: apply adaptor_or_method to class instance to adapt method context: use adaptor_or_method in the call """ if not class_.__dict__.has_key('implementation'): class_.implementation = Implementation() class_.implementation[obj] = adaptor_or_method def does_implement(class_or_instance, *interface_objects): """returns True or False depending on whether the class implements the list of interface objects """ # interface_objects can be an Interface object, a class, or # a class method/attribute class_ = class_or_instance if not isinstance(class_, type) and \ not isinstance(class_, Interface): class_ = class_.__class__ for interface in interface_objects: # interface is either a class/type or class method/attribute # or a string representation some name of a method or attribute # self or inherited classes are implemented if isinstance(interface, type) and \ isinstance(class_, interface): continue # interface is declared to be implemented # (interface is either a class or class # method/attribute) elif class_.__dict__.has_key('implementation') \ and interface in class_.implementation: continue # if this is a real interface # check each of the interfaces specified in it elif isinstance(interface, Interface): # check the items within the interface if does_implement(class_, *interface.interfacelist): continue # check non-class-specific method/attribute interfaces elif isinstance(interface, str) and \ interface in dir(class_): continue # check to see if the class is implemented # if a class specific the methods has not been declared elif 'im_class' in dir(interface) \ and does_implement(class_, interface.im_class): continue # recursive base implementations check elif class_.__dict__.has_key('implementation'): base_implementation = True for implementation in class_.implementation: if does_implement(implementation, interface): break else: base_implementation = False if base_implementation: continue return False return True def test_prototype(): """Tests implementation of Guido's desired goals per bottom of http://mail.python.org/pipermail/python-3000/2006-November/004774.html """ class StandardMapping(object): def __getitem__(self): pass def __contains__(self): pass def keys(self): pass class Foo(object): pass implements(Foo, StandardMapping) print "does_implement(Foo, StandardMapping):", print does_implement(Foo, StandardMapping) print class StandardMutableMapping(object): pass implements(StandardMutableMapping, StandardMapping) class dict_(dict): pass implements(dict_, StandardMutableMapping) print "does_implement(dict_, StandardMapping):", print does_implement(dict_, StandardMapping) print MinimalMapping = Interface(StandardMapping.__getitem__, StandardMapping.__contains__, StandardMapping.keys) print "does_implement(dict_, MinimalMapping):", print does_implement(dict_, MinimalMapping) print ExtendedMapping = Interface(StandardMapping, "copy", "__eq__") print "does_implement(dict_, ExtendedMapping):", print does_implement(dict_, ExtendedMapping) print implements(dict_, ExtendedMapping) print "does_implement(dict_, ExtendedMapping):", print does_implement(dict_, ExtendedMapping) print From greg.ewing at canterbury.ac.nz Fri Nov 24 23:47:46 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 25 Nov 2006 11:47:46 +1300 Subject: [Python-3000] optional argument annotations In-Reply-To: <000a01c71000$ae9a7130$6402a8c0@arkdesktop> References: <000a01c71000$ae9a7130$6402a8c0@arkdesktop> Message-ID: <45677692.103@canterbury.ac.nz> Andrew Koenig wrote: > def foo(a: T1) -> T2: > > I am presumably going to accept an argument of type T1 or a type derived > from T1, You're making assumptions about the semantics of the annotations here, which I thought wasn't going to be done. -- Greg From greg.ewing at canterbury.ac.nz Fri Nov 24 23:48:54 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 25 Nov 2006 11:48:54 +1300 Subject: [Python-3000] iostack and Oh Oh In-Reply-To: <1d85506f0611240718s364f83c2ya3e9e12e2ca7354c@mail.gmail.com> References: <1d85506f0611240718s364f83c2ya3e9e12e2ca7354c@mail.gmail.com> Message-ID: <456776D6.8020909@canterbury.ac.nz> tomer filiba wrote: > back in the day, we said all stream objects would support read() and > write(), and streams with limited functionality (readonly, writeonly) would > be "queriable" using the following properties: > * writable > * readable > * seekable Is that really necessary? I can't remember ever writing code that didn't already know what kind of stream it was expecting. -- Greg From guido at python.org Fri Nov 24 23:55:31 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 24 Nov 2006 14:55:31 -0800 Subject: [Python-3000] iostack and Oh Oh In-Reply-To: <456776D6.8020909@canterbury.ac.nz> References: <1d85506f0611240718s364f83c2ya3e9e12e2ca7354c@mail.gmail.com> <456776D6.8020909@canterbury.ac.nz> Message-ID: On 11/24/06, Greg Ewing wrote: > tomer filiba wrote: > > > back in the day, we said all stream objects would support read() and > > write(), and streams with limited functionality (readonly, writeonly) would > > be "queriable" using the following properties: > > * writable > > * readable > > * seekable > > Is that really necessary? I can't remember ever writing > code that didn't already know what kind of stream it > was expecting. Agreed that for the distinction between readable/writable it's pretty silly, and probably just encourages LBYL code ("if not f.readable: raise IOError;; f.read(...)" :-). But I've written plenty of code that could make good use of seekability but was able to do without it if necessary, and I suspect I'm not the only one. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sat Nov 25 00:18:27 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 24 Nov 2006 15:18:27 -0800 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: References: Message-ID: [With apologies for having a monologue with myself here.] On 11/24/06, I wrote: > I'd like to take the opportunity to present another example of what > I'd like to be able to do with abilities and generic functions. [...] Thinking about my own example some more, I think it would actually work just as well using ABCs instead of abilities. (The GFs are still needed.) It seems the use case for making abilities/interfaces separate from ABCs hinges on the assumption that there are classes that implement certain protocols without realizing it, IOW emergent protocols, plus that it's not practical to change __bases__. This seems to be exacerbated by there not being any standard abilities/interfaces defined, while some of the standard types are just *begging* to be classified by a container hierarchy. Plus, the standard types won't let you alter their __bases__ at all. I wonder if a bunch of well thought-out standard ABCs, applied to the standard data types, and perhaps more support for setting __bases__, wouldn't address most concerns. Note: I still like GFs for operations, especially binary operators, because they solve the problem of two-sided dispatch much better than __add__ and __radd__ ever can. (For unary operators I'm not sure it buys us that much; I'm pretty happy with __len__ and __iter__.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tony at pagedna.com Sat Nov 25 03:19:40 2006 From: tony at pagedna.com (Tony Lownds) Date: Fri, 24 Nov 2006 18:19:40 -0800 Subject: [Python-3000] optional argument annotations In-Reply-To: References: <33D8CBFC-9B3B-4FDE-927F-4B6D6B11C434@PageDNA.com> Message-ID: > Obviously signature objects would grow support for annotations, but I > still need the information to be carried on the code object to > incorporate into signature objects. > Signature objects still need a way to know the nested parameters, right? How about a co_argnames attribute? eg for def f((x, y), z): pass f.func_code.co_argnames would be (('x', 'y'), 'z') I need to implement something like this to properly build func_annotations inside MAKE_FUNCTION. -Tony From ironfroggy at gmail.com Sat Nov 25 03:35:24 2006 From: ironfroggy at gmail.com (Calvin Spealman) Date: Fri, 24 Nov 2006 21:35:24 -0500 Subject: [Python-3000] Fwd: defop ? In-Reply-To: References: <55071.62.39.9.251.1164201819.squirrel@webmail.nerim.net> <456456CF.8000005@gmail.com> <5.1.1.6.0.20061122151242.03f68a88@sparrow.telecommunity.com> <5.1.1.6.0.20061122180342.03ce3008@sparrow.telecommunity.com> <76fd5acf0611221959g74e8d7a9o36723f63814aa950@mail.gmail.com> <76fd5acf0611222310x64f03aa3i3d516c30452b1d79@mail.gmail.com> <76fd5acf0611222310h1f6e4137r1dee4afa7a747ef0@mail.gmail.com> Message-ID: <76fd5acf0611241835m73186e80h7eb0e9eb3fa878ff@mail.gmail.com> On 11/23/06, Guido van Rossum wrote: > On 11/22/06, Calvin Spealman wrote: > > On 11/22/06, Guido van Rossum wrote: > > > Not sure I like this better. My spelling of "operator::XXX" is > > > "__XXX__" and I like that just fine -- no new syntax needed. > > > > Well, that might be a spelling of `operator::XXX` but what about other > > use cases like `third_party_interface_system::adapt` or > > `anything_at_all::XXX`? Thats what I mean with not being too limiting > > and solving the problem in a way that opens up solutions to other > > problems. I get the opposition to it, but it seems reasonable, > > nonetheless. > > How was I to generalize from a single example what you meant? I > thought you were using 'operator' as a fixed keyword like it is in > C++, and I was wondering what you meant by the double colon. > > I still believe it is easy enough to solve this using a naming convention. Sorry, I meant it in the context of this thread, where the previous examples where using the operator module (which, by virtue of existing, should suggest that I wasn't implying its name for use in some other way). Naming conventions will only take us so far before we run into conflicts more often than we can offord. We only have one convention for special names, really, and it will run up its invitation with or without a suitable solution around its limitations. I understand the associtiation of this with "Too Much Magic" or simply the idea that its more than is needed, overkill, etc. and I share that warriness. But its something to consider, I suppose. I guess the thing to consider is simply, "What's in a name?" Another suggestion for this proposal: class MyList(object): def __builtins__::len(self): return 0 This would basically say that MyList has a method identified by the len attribute of __builtins__. Does that make sense? Maybe the syntax isn't enjoyable but the effect is more the value I see, so are there other ways it could be expressed? From collinw at gmail.com Sat Nov 25 04:12:35 2006 From: collinw at gmail.com (Collin Winter) Date: Fri, 24 Nov 2006 21:12:35 -0600 Subject: [Python-3000] optional argument annotations In-Reply-To: <99B2A3E4-1226-45D5-BC0D-DF72890A8D54@pagedna.com> References: <33D8CBFC-9B3B-4FDE-927F-4B6D6B11C434@PageDNA.com> <99B2A3E4-1226-45D5-BC0D-DF72890A8D54@pagedna.com> Message-ID: <43aa6ff70611241912j2afa8894g1d583aaca287f75b@mail.gmail.com> On 11/24/06, Tony Lownds wrote: > On Nov 24, 2006, at 8:30 AM, Guido van Rossum wrote: > > (It does imply that we can't use "-> None" to indicate "returns no > > value". We'd have to spell that as "-> type(None)" or find some other > > way. I think I don't care. Do others? Collin?) > > That's fine with me, but just to toss out another idea: we could put the > return annotation in func_annotations['return']. This is the solution I had initially envisioned, for exactly the reason Guido outlines. Collin Winter From guido at python.org Sat Nov 25 06:29:49 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 24 Nov 2006 21:29:49 -0800 Subject: [Python-3000] Fwd: defop ? In-Reply-To: <76fd5acf0611241835m73186e80h7eb0e9eb3fa878ff@mail.gmail.com> References: <456456CF.8000005@gmail.com> <5.1.1.6.0.20061122151242.03f68a88@sparrow.telecommunity.com> <5.1.1.6.0.20061122180342.03ce3008@sparrow.telecommunity.com> <76fd5acf0611221959g74e8d7a9o36723f63814aa950@mail.gmail.com> <76fd5acf0611222310x64f03aa3i3d516c30452b1d79@mail.gmail.com> <76fd5acf0611222310h1f6e4137r1dee4afa7a747ef0@mail.gmail.com> <76fd5acf0611241835m73186e80h7eb0e9eb3fa878ff@mail.gmail.com> Message-ID: Hm. The double colon rubs me the wrong way (Perl and/or C++). But apart from that, if this is the solution, I'm not sure the problem you're trying to solve is really worth solving. I just don't expect there will be all that many generic operations that need to be stuffed into arbitrary classes. Maybe I'll just take back what I said about wanting all such operations to be inside the class. Or maybe I'd be happier if there was a decorator indicating the name of the operation. Also, I think you're overloading 'def' in a non-obvious way. Currently, "def foo..." means an assignment to the local variable foo. I would expect that if we extend the syntax for the thing between 'def' and the argument list to be more than just a name, it should still be considered an assignment target. But that's clearly not what you're after. From that POV, I find defop (while still unpleasant for other reasons) more "honest" than your overloading of def -- at least defop says upfront that it's not just an assignment. (Although the similarity with def is still confusing IMO.) Still not convinced? Focus on other problems first. This isn't the most important problem we're trying to solve. PS, It's __builtin__, not __builtins__ -- the latter is an unfortunately named but ultimately unimportant implementation detail (unimportant unless you're implemented restricted python, that is); the former is the module that is implied at the end of every free name search. --Guido On 11/24/06, Calvin Spealman wrote: > On 11/23/06, Guido van Rossum wrote: > > On 11/22/06, Calvin Spealman wrote: > > > On 11/22/06, Guido van Rossum wrote: > > > > Not sure I like this better. My spelling of "operator::XXX" is > > > > "__XXX__" and I like that just fine -- no new syntax needed. > > > > > > Well, that might be a spelling of `operator::XXX` but what about other > > > use cases like `third_party_interface_system::adapt` or > > > `anything_at_all::XXX`? Thats what I mean with not being too limiting > > > and solving the problem in a way that opens up solutions to other > > > problems. I get the opposition to it, but it seems reasonable, > > > nonetheless. > > > > How was I to generalize from a single example what you meant? I > > thought you were using 'operator' as a fixed keyword like it is in > > C++, and I was wondering what you meant by the double colon. > > > > I still believe it is easy enough to solve this using a naming convention. > > Sorry, I meant it in the context of this thread, where the previous > examples where using the operator module (which, by virtue of > existing, should suggest that I wasn't implying its name for use in > some other way). > > Naming conventions will only take us so far before we run into > conflicts more often than we can offord. We only have one convention > for special names, really, and it will run up its invitation with or > without a suitable solution around its limitations. I understand the > associtiation of this with "Too Much Magic" or simply the idea that > its more than is needed, overkill, etc. and I share that warriness. > But its something to consider, I suppose. I guess the thing to > consider is simply, "What's in a name?" > > Another suggestion for this proposal: > > class MyList(object): > def __builtins__::len(self): > return 0 > > This would basically say that MyList has a method identified by the > len attribute of __builtins__. Does that make sense? Maybe the syntax > isn't enjoyable but the effect is more the value I see, so are there > other ways it could be expressed? > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From kay.schluehr at gmx.net Sat Nov 25 19:56:31 2006 From: kay.schluehr at gmx.net (Kay Schluehr) Date: Sat, 25 Nov 2006 19:56:31 +0100 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <456631AF.1020208@canterbury.ac.nz> References: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <20061122165513.GA13795@niemeyer.net> <4564F8B7.5030007@canterbury.ac.nz> <20061123020054.GA31618@niemeyer.net> <45651355.7090605@canterbury.ac.nz> <20061123121416.GA6072@niemeyer.net> <45661AA3.9070803@canterbury.ac.nz> <20061123224759.GA27828@niemeyer.net> <456631AF.1020208@canterbury.ac.nz> Message-ID: <456891DF.3060609@gmx.net> Greg Ewing schrieb: >Gustavo Niemeyer wrote: > > >>Are you arguing that >>pretty much every approach being discussed is bad and >>we should instead implement something magically >>transforming types? >> >> > >No, I'm questioning the need to do anything at all. >I don't see a big problem that merits a heavyweight >solution like a formal interface or adaptation >framework. > >-- >Greg >_______________________________________________ >Python-3000 mailing list >Python-3000 at python.org >http://mail.python.org/mailman/listinfo/python-3000 >Unsubscribe: http://mail.python.org/mailman/options/python-3000/kay.schluehr%40gmx.net > > Seems like we need more formal procedures for writing test than application code. This also solves the problem of expense of "verification" of typeclasses/concepts/abilities that need to be done just once for a class with a fixed interface and not each time when the class gets loaded from a module. The value of "test time" is increased to a higher status not unlike the role of "compile time" for static analysis in languages with default type systems. But maybe my head is just too far in the clouds when I'm thinking about the end of the static/dynamic language distinction using type reconstruction at "test time". Heavyweight frameworks naturally emerge also in Python but being latent. Reconstructing them from runtime information is a challenge for dynamically typed languages. A good enough approximation can be turned into a set of prescriptions. This is not much different from the way natural scientists frame reality according to Karl Poppers "logic of progress" ( hypothetical deductive method ). Regular enough data are used to form a hypothesis about the future behaviour of a system. This hypothesis is used to frame reality in a mathematical description - just like type systems. Making new experiments / measurents can falsify a yet established deductive system which leads to improved frames ( reframing ) and progressive understanding of systems properties. The role of tests in this approach is twofold: they check application code and keep information about the "true nature" of the programs behaviour using measurements which leads to automatical generation of a formal hypothesis of a systems behaviour that may become prescriptive i.e. forms a type system: Note that I've not yet strong indications that a Popperian methodology really works for any kind of language including Python i.e. I have neither worked out a formal proof nor do I have a working system ( not to mention treatments of "singularities" ) But I believe enough in this idea for questioning the value of making "Python more Java like" which seems to be the main intention of the interface/ability/optional type declaration proposals discussed here. Kay From tomerfiliba at gmail.com Sat Nov 25 20:32:44 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Sat, 25 Nov 2006 21:32:44 +0200 Subject: [Python-3000] two things Message-ID: <1d85506f0611251132q5325b36cn4570f5bbe403484@mail.gmail.com> i'd suggest using the term "contract" instead of abilities or interfaces. they way BDFL described it [1], abilities specify more than mere method signatures -- they go as deep as how the implementation should implement the desired behavior. http://mail.python.org/pipermail/python-3000/2006-November/004782.html > I'd start by defining three abilities: MinimalSet which only provides > the 'in' operator; IterableSet which also provides iteration (but not > len(); it may aply to infinite sets); and StandardSet which adds most > standard operations on sets [...] > If a class claims to be a set, it should conform > to a much richer contract, which guarantees the basic properties of > mathematical sets. For example, a&b == b&a, a&a == a, and so on. "contract" is a better term, IMO, since it's already used in CS (as in Eiffel), and describes the situation more correctly: *behavior* rather than *signature*. "ability" just doesn't seem right to me: my class is not *able* to be a set, it *behaves* like a set. it follows the set contract, not "ability" - - - - - instead of the suggested /defop/ keyword, which seems very awkward to me, i'd prefer an extension to the current function syntax: def __contains__(self) of MinimalSet: pass if we don't want to make "of" a keyword, we can use "in", with a similar meaning. this syntax is inspired by (god forbid) visual basic .net: http://msdn2.microsoft.com/en-us/library/sect4ck6.aspx Class Spam Implements IFooable Sub Foo(ByVal x as Integer) As String Implements IFooable.SomeMethod ... End Sub End Class the suggested syntax of defop MinimalSet::__contains___(self, other): pass starts to look like C++ to me (yuch), and it's not necessarily an "operator", as *defop* implies. -tomer From solipsis at pitrou.net Sat Nov 25 20:56:32 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 25 Nov 2006 20:56:32 +0100 Subject: [Python-3000] two things In-Reply-To: <1d85506f0611251132q5325b36cn4570f5bbe403484@mail.gmail.com> References: <1d85506f0611251132q5325b36cn4570f5bbe403484@mail.gmail.com> Message-ID: <1164484592.5176.27.camel@fsol> Le samedi 25 novembre 2006 ? 21:32 +0200, tomer filiba a ?crit : > "contract" is a better term, IMO, since it's already used in CS (as in Eiffel), > and describes the situation more correctly: *behavior* rather than *signature*. > "ability" just doesn't seem right to me: my class is not *able* to be a set, > it *behaves* like a set. You can simply call it a behaviour rather than a contract. Contract-based programming is usually defined as involving pre- and post-conditions on method calls, to check that the method behaves as expected. The underlying idea is that the terms of the contract are programmatically enforced rather than simply declared. Wikipedia mentions those attempts at contract-based programming with Python: http://www.nongnu.org/pydbc/ http://www.wayforward.net/pycontract/ http://www.targeted.org/python/recipes/ipdbc.py > instead of the suggested /defop/ keyword, which seems very awkward to me, > i'd prefer an extension to the current function syntax: > > def __contains__(self) of MinimalSet: > pass Other suggestion: from MinimalSet def __contains__(self): pass The "origin" of the method is then clearly visible instead of being relegated at the end of the declaration (it works better with multi-line declarations too). Also the class declaration look nicer when you have several of these methods: class BazContainer: # Implement methods from MinimalSet from MinimalSet def __contains__(self): pass from MinimalSet def add(self): pass from MinimalSet def remove(self): pass # Other methods def to_bicycle(self): """ Turn this container into a bicycle """ (etc.) From tjreedy at udel.edu Sat Nov 25 20:59:10 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 25 Nov 2006 14:59:10 -0500 Subject: [Python-3000] Abilities / Interfaces References: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <20061122165513.GA13795@niemeyer.net> <4564F8B7.5030007@canterbury.ac.nz> <20061123020054.GA31618@niemeyer.net> <45651355.7090605@canterbury.ac.nz> <20061123121416.GA6072@niemeyer.net> <45661AA3.9070803@canterbury.ac.nz> <20061123224759.GA27828@niemeyer.net><456631AF.1020208@canterbury.ac.nz> <456891DF.3060609@gmx.net> Message-ID: "Kay Schluehr" wrote in message news:456891DF.3060609 at gmx.net... > Seems like we need more formal procedures for writing test than > application code. This also solves the problem of expense of > "verification" of typeclasses/concepts/abilities that need to be done > just once for a class with a fixed interface and not each time when the > class gets loaded from a module. The value of "test time" is increased > to a higher status not unlike the role of "compile time" for static > analysis in languages with default type systems. But maybe my head is > just too far in the clouds when I'm thinking about the end of the > static/dynamic language distinction using type reconstruction at "test > time". Heavyweight frameworks naturally emerge also in Python but being > latent. Reconstructing them from runtime information is a challenge for > dynamically typed languages. A good enough approximation can be turned > into a set of prescriptions. This is not much different from the way > natural scientists frame reality according to Karl Poppers "logic of > progress" ( hypothetical deductive method ). Regular enough data are > used to form a hypothesis about the future behaviour of a system. This > hypothesis is used to frame reality in a mathematical description - just > like type systems. Making new experiments / measurents can falsify a yet > established deductive system which leads to improved frames ( reframing > ) and progressive understanding of systems properties. The role of tests > in this approach is twofold: they check application code and keep > information about the "true nature" of the programs behaviour using > measurements which leads to automatical generation of a formal > hypothesis of a systems behaviour that may become prescriptive i.e. > forms a type system: To me, run-on paragraphs (and the above seems to me to be at least three) have the same comprehension problems as run-on sentences ;-) "In the clouds" indeed. Maybe try again? Terry Jan Reedy From ark-mlist at att.net Sat Nov 25 21:17:58 2006 From: ark-mlist at att.net (Andrew Koenig) Date: Sat, 25 Nov 2006 15:17:58 -0500 Subject: [Python-3000] two things In-Reply-To: <1d85506f0611251132q5325b36cn4570f5bbe403484@mail.gmail.com> Message-ID: <001c01c710ce$d3000610$6402a8c0@arkdesktop> > "contract" is a better term, IMO, since it's already used in CS (as in > Eiffel), and describes the situation more correctly: *behavior* rather > than *signature*. > "ability" just doesn't seem right to me: my class is not *able* to be a > set, > it *behaves* like a set. it follows the set contract, not "ability" I avoided suggesting the word "contract" instead of "ability" because I do not want people to think that the notion is somehow connected to Eiffel contracts. As Guido (I think) pointed out, it's closer to Haskell typeclasses. From jimjjewett at gmail.com Sun Nov 26 01:03:16 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Sat, 25 Nov 2006 19:03:16 -0500 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <3497511880590744734@unknownmsgid> References: <-6683921950610559197@unknownmsgid> <3497511880590744734@unknownmsgid> Message-ID: On 11/22/06, Bill Janssen wrote: > > yeah, namespaces are such a useless thing. let's put everything in one > > big flat namespace, so we don't have to wonder where things are. > Pardon me? That's what we've got now, isn't it? That's why we have > to do that ugly and hard-to-explain name mangling for "special" > methods. Not quite. The documentation (http://docs.python.org/ref/id-classes.html) states that __special__ methods are reserved to the language. This isn't strictly enforced, but it does add a second namespace. There isn't usually any need for a third namespace, unless you are using another huge framework, such as Zope or Peak. If so, the frameworks have their own conventions. I have no opinion on whether their conventions happen to be good, safe from false alarms, or even frequently used; I am stating only that the frameworks themselves are responsible for carving out their pseudo-namespaces. That said, I think the situation would change (for the worse) if we started to see heavy use of generic functions or interfaces. These should not use the __special__ convention (they aren't part of the system, and won't all even be widespread). Using a designated name in the main namespace will lead to clashes. (Not every "next" attribute is an iterator-related function). I had thought that you (Bill), (along with PJE) were assuming that the generic function or interface itself would serve to distinguish the namespace. Instead of class A(object): def magic_meth_foo(self, ...): I would write class A(object): defop foo.magic_meth(self, ...) This does work, but ... code gets just a little bit longer, with the extra going to dots and boilerplate. The boilerplate won't hide the way "__" does. It probably shouldn't always hide, since it won't even be boilerplate in some cases. But it will be boilerplate so often that readers will start to skim, even when they shouldn't. In the end, that extra almost-boilerplate strikes me as a very high price to pay, and I'm not sure it can be avoided unless Interfaces and Generic Functions stay relatively rare. -jJ From greg.ewing at canterbury.ac.nz Sun Nov 26 01:26:45 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 26 Nov 2006 13:26:45 +1300 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> <3497511880590744734@unknownmsgid> Message-ID: <4568DF45.1050200@canterbury.ac.nz> Jim Jewett wrote: > There isn't usually any need for a third namespace, unless you are > using another huge framework, such as Zope or Peak. Even then, there isn't a problem unless you want to multiply inherit from two classes that clash. I can't remember ever encountering such a problem with __special__ methods. -- Greg From pje at telecommunity.com Sun Nov 26 02:52:06 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 25 Nov 2006 20:52:06 -0500 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <3497511880590744734@unknownmsgid> <-6683921950610559197@unknownmsgid> <3497511880590744734@unknownmsgid> Message-ID: <5.1.1.6.0.20061125204355.027f7be8@sparrow.telecommunity.com> At 07:03 PM 11/25/2006 -0500, Jim Jewett wrote: >I had thought that you (Bill), (along with PJE) were assuming that the >generic function or interface itself would serve to distinguish the >namespace. Instead of > > class A(object): > def magic_meth_foo(self, ...): > >I would write > > class A(object): > defop foo.magic_meth(self, ...) > >This does work, but ... code gets just a little bit longer, with the >extra going to dots and boilerplate. The boilerplate won't hide the >way "__" does. It probably shouldn't always hide, since it won't even >be boilerplate in some cases. But it will be boilerplate so often >that readers will start to skim, even when they shouldn't. > >In the end, that extra almost-boilerplate strikes me as a very high >price to pay, and I'm not sure it can be avoided unless Interfaces and >Generic Functions stay relatively rare. This isn't quite correct. First, in the interface API I proposed, you could use regular methods, if you were putting them directly in the object class. However, in the normal use cases for generic functions, you don't often put the methods directly in the classes! As Guido mentions for e.g. __add__ and __radd__, you are probably defining them in some kind of glue code that is not of either type. Second, for "visitor" patterns, you aren't going to be modifying the base types to add new methods to generic functions. Consider, for example, the compiler.ast package, that has a variety of objects that represent AST nodes. If you are creating a new compiler or refactoring tool or "lint" checker with generic functions, you'd write the functions in a new module for the function to be performed, and then define methods for the function by AST class. You would not modify the AST classes to add "lint_me" methods! And so it is for any other case where you are retrofitting or extending the behavior of existing types - making it on the whole relatively infrequent that you'll want to put most methods inside class bodies. This is especially true in cases where you want to support your operation on built-in types: there's no place for you to add the code in that case. Finally, in situations where you *do* put the methods directly on the class, chances are good that you'll either be adding a single operation or will want to just have a generic factory function to just produce an object that has all the desired methods in the first place. In none of these cases will you have many generic function methods in a single class body. The only case left, really, where you'd put many generic function methods in a class body is when they're builtins like len(), iter() etc... which don't have the "namespace noise" you're worried about. From jimjjewett at gmail.com Sun Nov 26 03:17:36 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Sat, 25 Nov 2006 21:17:36 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <456590C3.4040801@davious.org> References: <456590C3.4040801@davious.org> Message-ID: On 11/23/06, Dave Anderson wrote: > Interfaces > ========== > iter_interface = interface(iter) # class-based interface > file_open_close_inter = interface(file.open, file.close) # method interface [ strings were also treated differently ] Are you keying off the dot in the name, or the type of the named object? SomeClass.name is typically a method, but may be a string, or even another class. -jJ From jimjjewett at gmail.com Sun Nov 26 03:56:06 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Sat, 25 Nov 2006 21:56:06 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <02AFCDEB-25D2-4D10-9BE4-A2773CCED65C@cleverdevil.org> Message-ID: On 11/22/06, Guido van Rossum wrote: > On 11/22/06, Jonathan LaCour wrote: > > Phillip J. Eby wrote: > > > In essence, interfaces turn libraries into "frameworks", but generic > > > functions turn frameworks into libraries. I didn't really discover > > > this until January of last year ... > > This is easily the most important part of the conversation, IMO, and I > > would hate for it to be missed. "Interfaces" in the Java universe are a > > hard contract that everyone has to agree upon up-front, ... you end up > > with is a world of many competing frameworks, none of which interoperate. > > ... something that seems a tad magical at first > > I have actually used generic functions (using RuleDispatch) in practice > > and find them to not only be hugely powerful, but much more pragmatic. > Sorry, but that was about as informative as a "+1" vote. I found it a bit more specific on two points: What I (mis?)read was: (1) PJE: Interfaces rely on buyin; they end up accidentally creating walled gardens. Generic Functions make fewer demands, so they start to work sooner, and integrate better. [The down side is that finding the relevant code -- or even being sure that it *is* is the relevant code, and isn't overridden elsewhere -- gets more difficult. -jJ] Jonathan: Phillip has written a lot of long messages; this one point is particularly important for the summary. It captures a (the?) key advantage of generic functions. (2) Generic Functions are harder to get your head around, but once you do, they're worth it. Phillip has been saying this already, but it matters that someone else has gotten to the point where they are useful and seem simple. Personally, I stand by my assessment that they can make metaclasses look straightforward -- but that is partly because metaclasses have had their roughest edges worn off already. (For example, they no longer require compiling a C class so odd that the pattern gets named after someone.) If GF explanations start to come from several different people, the same process should eventually simplify them. If that happened quickly enough, it would change the tradeoffs. -jJ From ironfroggy at gmail.com Sun Nov 26 05:28:47 2006 From: ironfroggy at gmail.com (Calvin Spealman) Date: Sat, 25 Nov 2006 23:28:47 -0500 Subject: [Python-3000] Fwd: defop ? In-Reply-To: References: <5.1.1.6.0.20061122151242.03f68a88@sparrow.telecommunity.com> <5.1.1.6.0.20061122180342.03ce3008@sparrow.telecommunity.com> <76fd5acf0611221959g74e8d7a9o36723f63814aa950@mail.gmail.com> <76fd5acf0611222310x64f03aa3i3d516c30452b1d79@mail.gmail.com> <76fd5acf0611222310h1f6e4137r1dee4afa7a747ef0@mail.gmail.com> <76fd5acf0611241835m73186e80h7eb0e9eb3fa878ff@mail.gmail.com> Message-ID: <76fd5acf0611252028i496ef6d7r9b23674bcb38046e@mail.gmail.com> I will make only one more comment and then ill drop my comments without direct questions. On 11/25/06, Guido van Rossum wrote: > Hm. The double colon rubs me the wrong way (Perl and/or C++). But > apart from that, if this is the solution, I'm not sure the problem > you're trying to solve is really worth solving. I just don't expect > there will be all that many generic operations that need to be stuffed > into arbitrary classes. Maybe I'll just take back what I said about > wanting all such operations to be inside the class. Or maybe I'd be > happier if there was a decorator indicating the name of the operation. I don't care about the syntax. anything that can denote an expression (the left of the ::) and a name (the right of the ::) is OK and I dont care how its denoted. > Also, I think you're overloading 'def' in a non-obvious way. > Currently, "def foo..." means an assignment to the local variable foo. > I would expect that if we extend the syntax for the thing between > 'def' and the argument list to be more than just a name, it should > still be considered an assignment target. But that's clearly not what > you're after. From that POV, I find defop (while still unpleasant for > other reasons) more "honest" than your overloading of def -- at least > defop says upfront that it's not just an assignment. (Although the > similarity with def is still confusing IMO.) You must be misunderstanding me. I am not saying that its not an assignment. It would not change what def really means. operator::len would be the actual name of the function to be created and the name of the global, local, or class attribute it is bound to. I am saying operator::len would become something like MyClass.__dict__[operator::len] and what operator::len evaluates to, i dont know. something that represents what it is. Maybe just a tuple. I would expect it also exist for any assignment. special casing being bad and all. > Still not convinced? Focus on other problems first. This isn't the > most important problem we're trying to solve. > > PS, It's __builtin__, not __builtins__ -- the latter is an > unfortunately named but ultimately unimportant implementation detail > (unimportant unless you're implemented restricted python, that is); > the former is the module that is implied at the end of every free name > search. > > > --Guido -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/ From jonathan-lists at cleverdevil.org Sun Nov 26 05:54:36 2006 From: jonathan-lists at cleverdevil.org (Jonathan LaCour) Date: Sat, 25 Nov 2006 23:54:36 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <02AFCDEB-25D2-4D10-9BE4-A2773CCED65C@cleverdevil.org> Message-ID: Jim Jewett wrote: >> Sorry, but that was about as informative as a "+1" vote. > > I found it a bit more specific on two points: > > What I (mis?)read was: > > (1) > PJE: Interfaces rely on buyin; they end up accidentally creating > walled gardens. Generic Functions make fewer demands, so they start > to work sooner, and integrate better. [The down side is that finding > the relevant code -- or even being sure that it *is* is the relevant > code, and isn't overridden elsewhere -- gets more difficult. -jJ] > > Jonathan: Phillip has written a lot of long messages; this one point > is particularly important for the summary. It captures a (the?) key > advantage of generic functions. I am glad that *someone* got what I was trying to say. Philip indeed writes long, informative messages. Its easy for little gems like this one go by unnoticed. I had not seen this relative merit of generic functions (and disadvantage of interfaces) discussed, so I responded with my agreement that this was very true for me in actual practice (not just in theory). > (2) > Generic Functions are harder to get your head around, but once you do, > they're worth it. Phillip has been saying this already, but it > matters that someone else has gotten to the point where they are > useful and seem simple. In fact, I used Philip's wonderful RuleDispatch module again just today to create something quite useful. I personally found generic functions to be quite easy to understand, but didn't truly begin to appreciate them until I started to use them. So, sure, it might have been a +1 vote, but it comes with the context that I have actually *used* both generic functions and interfaces in practice, and have found Philip's statement above to ring very true. -- Jonathan LaCour http://cleverdevil.org From sluggoster at gmail.com Sun Nov 26 10:55:44 2006 From: sluggoster at gmail.com (Mike Orr) Date: Sun, 26 Nov 2006 01:55:44 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: <87ac2x7wz0.fsf@qrnik.zagroda> References: <45536FF3.4070105@acm.org> <79990c6b0611091245r2d0251f0q813951d8b94dca1e@mail.gmail.com> <87ac2x7wz0.fsf@qrnik.zagroda> Message-ID: <6e9196d20611260155h3e476474oe9e0beff8c9bbb0d@mail.gmail.com> Status update and questions about root splitting. I've got a path module written (called unipath) and am in the middle of the unit tests. When they're done I'll release it for review. It's been slow coz I was setting up two computers at the same time. I tried to make a separate PathAlgebra class and FSPath class, but it got so unweildly to use I made the latter a subclass. They're now called PathName and Path. The main issues are: - Should FSPath.cwd(), .walk(), .rel_path_to(), etc return a PathAlgebraObject or an FSPath? The use cases seem evenly divided. - It gets ugly when you have to deconstruct an FSPath to modify it, then call the constructor again to create a new FSPath to use it. Users would complain at: my_directory = FSPath(PathName("...")).mkdir() or: my_directory = FSPath("...") FSPath(my_directory.path, "subdir").mkdir() Since I really don't think I want to use the non-subclass version in my own applications and thus have little incentive to work on it, maybe someone else who really wants that model for the stdlib can take over implementing that part. * * * But the main question I wanted to ask about was root splitting. I'm going for the following API: p = PathName(*joinable_components) p.parent p.name p.ext p.name_without_ext p.drive p.unc p.split_root() -> (root, rest) p.components() -> [root, component1, component2, ...] The idea being that you'll often want to make a path relative and then attach it to a new parent, add/remove components, etc. My idea, based on Noam's proposal, was to consider the root: / \ c:\ C: \\unc_share\ or "" for a relative path. That way, root + rest would always be concatenation without a separator. But os.path.split* don't work like this. >>> ntpath.splitdrive(r"C:\dir\subdir") ('C:', '\\dir\\subdir') >>> ntpath.splitunc(r"\\unc_share\foo\bar\baz") ('\\\\unc_share\\foo', '\\bar\\baz') There is no method to split a slash root (/ or \) so I have to do that manually. To realize my original idea, in the drive and UNC cases I'd have to move the intervening slash from the remainder to the root. But this would make the root different that what p.drive and p.unc return -- if that matters. Do we even need p.drive and p.unc if we have p.split_root() that handles all kinds of roots? Do we need the .split* methods if the properties/methods above handle all the use cases? It would be cleaner not to include them but somebody will whine "where is my method?" It's also slightly less efficient to do (p.parent, p.name) or (p.name_without_ext, p.ext) because you're implicitly doing the same split twice -- if that matters. On the other hand, os.path.join() handles all this, adding a separator when necessary, and even handling "" as the first argument. So maybe I should just let os.path.join() do its magic and be happy. Noam distinguished between a drive-relative (r"C:foo") and drive-absolute (r"C:\foo"). I'm treating the former as absolute because I don't think it matters in this context. os.path.join() does not add a slash with drive letters: >>> ntpath.join(r"C:", "foo") 'C:foo' >>> ntpath.join(r"C:\ "[:-1], "foo") 'C:\\foo' (The latter funny syntax is to get around the fact that you can't have a backslash at the end of a raw string literal.) So with drive letters it's vital not to add a backslash, and to trust the root portion ends with a backslash if necessary. * * * I've put some work into converting paths: NTPath(PosixPath("foo/bar.py"), using the .components() format as a universal intermediary, and refusing to convert absolute paths. This has taken some work to implement and several paragraphs to document, and I still haven't tested to see if it's correct. I'm wondering if it's becoming overkill for the original goal and anticipated use cases. We thought, "Yeah, it would be a good idea to support non-native paths and to convert paths." Then Talin, who wanted it, said he was mainly concerned about putting Posix paths in config files and having them work on other platforms (Windows on embedded systems that don't have a current directory), without having to manually convert them. But os.path.normpath() and os.path.normcase() already convert Posix slashes to Windows backslashes. Maybe that's enough? Is there a need to go the other way or do other conversions? * * * Path APIs are really like web frameworks in Python: there's no way everybody's going to agree on one, and it's so easy to write a different one. So I'm factoring the more complex code into a function library that can be used as a back end to other path APIs. -- Mike Orr From jimjjewett at gmail.com Sun Nov 26 18:25:17 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Sun, 26 Nov 2006 12:25:17 -0500 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <5.1.1.6.0.20061125204355.027f7be8@sparrow.telecommunity.com> References: <-6683921950610559197@unknownmsgid> <3497511880590744734@unknownmsgid> <5.1.1.6.0.20061125204355.027f7be8@sparrow.telecommunity.com> Message-ID: On 11/25/06, Phillip J. Eby wrote: > However, in the normal use cases for generic functions, > you don't often put the methods directly in the classes! I think this gets to the core of my biggest worry. When I'm doing sysadmin-type maintenance (which at my employer does not mean python, but rather nested *sh scripts, some sourced and some called), my biggest problem is figuring out where things *really* came from. Ahh ... it comes from this variable, which was set ... where? And reset where? Oh, at that point the path was this, so it was really using *that* version. I have resisted using twisted because (at least at first glance) it seems to factor classes so small that I end up with the same feeling of magic -- I couldn't easily find where things were defined, or get the full API and documents in one place. I fear that generic functions would encourage this sort of action at a distance. I would probably use it myself, because I know it can be very helpful when I'm initially writing something. But it can also make life terribly hard for the next person. One reason I like python is that it doesn't usually encourage that sort of tradeoff. -jJ From jimjjewett at gmail.com Sun Nov 26 18:54:53 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Sun, 26 Nov 2006 12:54:53 -0500 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611260155h3e476474oe9e0beff8c9bbb0d@mail.gmail.com> References: <45536FF3.4070105@acm.org> <79990c6b0611091245r2d0251f0q813951d8b94dca1e@mail.gmail.com> <87ac2x7wz0.fsf@qrnik.zagroda> <6e9196d20611260155h3e476474oe9e0beff8c9bbb0d@mail.gmail.com> Message-ID: On 11/26/06, Mike Orr wrote: > I tried to make a separate PathAlgebra class and FSPath class, but it > got so unweildly to use I made the latter a subclass. They're now > called PathName and Path. This makes sense to me. An FSPath without path algebra is basically a "directory listing" That might be a useful abstraction on its own, or it might be too much purity. > My idea, based on Noam's proposal, was to consider the root: > / > \ > c:\ > C: > \\unc_share\ > or "" for a relative path. That way, root + rest would always be > concatenation without a separator. But os.path.split* don't work like > this. [ They put the C: or such as the root, and the first \ in the path] > Do we even need p.drive and p.unc if we have > p.split_root() that handles all kinds of roots? I don't think so. Maybe on a WindowsFSPath subclass, for compatibility. But taking them out of the generic path is a good thing. Most people don't need to worry about them, and shouldn't need to wonder whether or not they do. > Noam distinguished between a drive-relative (r"C:foo") and > drive-absolute (r"C:\foo"). I'm treating the former as absolute > because I don't think it matters in this context. I think you are correct, though I'm not sure I would have thought of it. C: without a slash is effectively a mount point into the current directory. This particular mount point might shift around unexpectedly, but ... someone could rewrite the regular files on you too. It does make a leading ".." meaningful though; I'm not sure if that matters. > >>> ntpath.join(r"C:\ "[:-1], "foo") > 'C:\\foo' > (The latter funny syntax is to get around the fact that you can't have > a backslash at the end of a raw string literal.) You have to double it. One way to make lemonade of this lemon is to add a class attribute P.sep equivalent to os.path.sep. (And, realistically, the separator is more tied to the File System than it is to the Operating System.) If might encourage people to use path.sep instead of just assuming the locally correct character. -jJ From sluggoster at gmail.com Sun Nov 26 19:42:43 2006 From: sluggoster at gmail.com (Mike Orr) Date: Sun, 26 Nov 2006 10:42:43 -0800 Subject: [Python-3000] Mini Path object In-Reply-To: References: <45536FF3.4070105@acm.org> <79990c6b0611091245r2d0251f0q813951d8b94dca1e@mail.gmail.com> <87ac2x7wz0.fsf@qrnik.zagroda> <6e9196d20611260155h3e476474oe9e0beff8c9bbb0d@mail.gmail.com> Message-ID: <6e9196d20611261042y561a9a92s83f71291e5ef87b9@mail.gmail.com> On 11/26/06, Jim Jewett wrote: > On 11/26/06, Mike Orr wrote: > > I tried to make a separate PathAlgebra class and FSPath class, but it > > got so unweildly to use I made the latter a subclass. They're now > > called PathName and Path. > > This makes sense to me. An FSPath without path algebra is basically a > "directory listing" That might be a useful abstraction on its own, or > it might be too much purity. It's a single path, actually, upon which you can call filesystem operations. p.mkdir(), p.copytree(), p.read_file(), p.readlink(), p.listdir() What you *can't* do is extract/alter the pathname itself using FSPath operations. p.ext, p + ".gz", p.expand_user() # Error! Use p.path.ext instead. Nobody has specifically asked for a FSPath class. But some people are adamant they want a PathAlgebra class, a clear separation between pathname vs filesystem operations. Otherwise we'd just use the earlier implementations which combine both in one class. One proposal is to put PathAlgebra in os.path and not have a FSPath class in the stdlib at all. That's not my ideal but it may be small enough and non-controversial enough to get into Python 2.6, and then we can build a more elaborate FSPath on top of it for 3.0 (or not). > C: without a slash is effectively a mount point into the current > directory. That's what I always thought "C:foo" is. But Glyph said it's more complicated than that: '"C:blah" does not mean what you think it means on Windows. Regardless of what you think it means, it is not that. I thought I understood it once as the current process having a current directory on every mapped drive, but then I had to learn about UNC paths of network mapped drives and it stopped making sense again.' [python-dev message dated 2006-Nov-1, subject: Path object design, from glyph at divmod.com] -- Mike Orr From jimjjewett at gmail.com Sun Nov 26 20:13:57 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Sun, 26 Nov 2006 14:13:57 -0500 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611261042y561a9a92s83f71291e5ef87b9@mail.gmail.com> References: <45536FF3.4070105@acm.org> <79990c6b0611091245r2d0251f0q813951d8b94dca1e@mail.gmail.com> <87ac2x7wz0.fsf@qrnik.zagroda> <6e9196d20611260155h3e476474oe9e0beff8c9bbb0d@mail.gmail.com> <6e9196d20611261042y561a9a92s83f71291e5ef87b9@mail.gmail.com> Message-ID: On 11/26/06, Mike Orr wrote: > On 11/26/06, Jim Jewett wrote: > > On 11/26/06, Mike Orr wrote: > > > I tried to make a separate PathAlgebra class and > > > FSPath class, but it got so unweildly to use I made > > > the latter a subclass. They're now called > > > PathName and Path. > > This makes sense to me. An FSPath without path > > algebra is basically a "directory listing" > It's a single path, actually, Fair enough. > upon which you can call filesystem operations. > p.mkdir(), p.copytree(), p.read_file(), p.readlink(), p.listdir() So if it doesn't happen to point to a directory, you can either ask it for a different Path (the result of readlink) or open it (read_file)... > What you *can't* do is extract/alter the pathname > itself using FSPath operations. > p.ext, p + ".gz", p.expand_user() # Error! Use p.path.ext instead. I had thought that Path objects were immutable, and so these would produce new isntances. > > C: without a slash is effectively a mount point into > > the current directory. > That's what I always thought "C:foo" is. But Glyph > said it's more complicated than that: Most of the exceptions were things like the CON special file. These make the windows (or DOS) filesystem wierd, but they aren't really affected by C: vs C:\ CON, C:\dir1\CON, C:CON, and .\con all refer to the same special file. -jJ From p.f.moore at gmail.com Sun Nov 26 20:48:55 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 26 Nov 2006 19:48:55 +0000 Subject: [Python-3000] Mini Path object In-Reply-To: References: <45536FF3.4070105@acm.org> <79990c6b0611091245r2d0251f0q813951d8b94dca1e@mail.gmail.com> <87ac2x7wz0.fsf@qrnik.zagroda> <6e9196d20611260155h3e476474oe9e0beff8c9bbb0d@mail.gmail.com> <6e9196d20611261042y561a9a92s83f71291e5ef87b9@mail.gmail.com> Message-ID: <79990c6b0611261148v5d95dabfnecf00d5dc73b8c62@mail.gmail.com> On 11/26/06, Jim Jewett wrote: > On 11/26/06, Mike Orr wrote: > > On 11/26/06, Jim Jewett wrote: > > > C: without a slash is effectively a mount point into > > > the current directory. > > > That's what I always thought "C:foo" is. But Glyph > > said it's more complicated than that: > > Most of the exceptions were things like the CON special file. These > make the windows (or DOS) filesystem wierd, but they aren't really > affected by C: vs C:\ > > CON, C:\dir1\CON, C:CON, and .\con all refer to the same special file. Technically, every drive has a "current path". You can see this by doing the following on a Windows system (here, the prompt shows the current path) C:\Data>cd F:\Software *** Note from the following prompt that this does *not* change the current directory - because the current drive is C:, and we changed the directory on F: C:\Data>dir F:4nt.zip Volume in drive F is Data Volume Serial Number is 7043-3187 Directory of F:\Software [...] *** But note that F:4nt.zip is located in F:\Software, which is where we made the current directory on F: C:\Data>f: F:\Software> *** And if we go to the F: drive (unspecified directory) we get to \Software, the current directory on F:. Basically, Windows maintains a full set of current directories, one for each drive (actually stored as hidden environment variables named something like "=C:", "=D:" etc.) and D:foo refers to the file foo in the current directory on the D: drive. This is obscure to the point where I doubt anyone relies on it, but it's the sort of edge case that you need to get right or someone will complain in the end... :-) Nitpicking-ly y'rs Paul. From brett at python.org Sun Nov 26 22:10:24 2006 From: brett at python.org (Brett Cannon) Date: Sun, 26 Nov 2006 13:10:24 -0800 Subject: [Python-3000] optional argument annotations In-Reply-To: References: <33D8CBFC-9B3B-4FDE-927F-4B6D6B11C434@PageDNA.com> Message-ID: On 11/24/06, Tony Lownds wrote: > > > Obviously signature objects would grow support for annotations, but I > > still need the information to be carried on the code object to > > incorporate into signature objects. > > > > Signature objects still need a way to know the nested parameters, right? They already handle them. How about a co_argnames attribute? eg for > > def f((x, y), z): pass > > f.func_code.co_argnames would be (('x', 'y'), 'z') That's fine, but the compiler would need to change if it were to use this. Plus I am still hoping to make nested parameters disappear in Py3K (I won't be pushing for it any sooner than PyCon, though). I need to implement something like this to properly build > func_annotations > inside MAKE_FUNCTION. I don't quite follow. Don't you already have support in MAKE_FUNCTION when the object is created? -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061126/fc3c1711/attachment.htm From greg.ewing at canterbury.ac.nz Sun Nov 26 21:57:50 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 27 Nov 2006 09:57:50 +1300 Subject: [Python-3000] Mini Path object In-Reply-To: <6e9196d20611260155h3e476474oe9e0beff8c9bbb0d@mail.gmail.com> References: <45536FF3.4070105@acm.org> <79990c6b0611091245r2d0251f0q813951d8b94dca1e@mail.gmail.com> <87ac2x7wz0.fsf@qrnik.zagroda> <6e9196d20611260155h3e476474oe9e0beff8c9bbb0d@mail.gmail.com> Message-ID: <4569FFCE.9050507@canterbury.ac.nz> Mike Orr wrote: > They're now called PathName and Path. Not a sane choice of names, since in this context, "path" and "pathname" are synonymous in ordinary language. -- Greg From jimjjewett at gmail.com Sun Nov 26 22:47:02 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Sun, 26 Nov 2006 16:47:02 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <45640E3A.7010307@livinglogic.de> Message-ID: On 11/22/06, Guido van Rossum wrote: > On 11/22/06, Walter D?rwald wrote: > > Another effect of __special__ methods is that they divide > > the class namespace into two parts: The __special__ names > > are those that have to be implemented to support core > > Python interfaces, and the methods without underscores > > are those that implement the "main aspect" of the > > class. > Well, there are plenty of examples of __special__ methods > for other purposes than core Python interfaces, The documentation explicitly reserves __*__ to the language. You can argue that some uses aren't core, but that still means __*__ isn't appropriate for a random user-supplied aspect. (And if it were, that would lead to conflicts over attractive names like __log__. __package_module__name__ might work, but ... starts to get hackish.) > and plenty of core Python > interfaces that don't use __special__ (dict.get(), list.append(), > iter.next(), str.lower(), you get the idea.) ermm... I'm not seeing any counterexamples just yet. get, append, and lower apply only to specific classes, and from there they get generalized to the informal interfaces that those classes prototype. Less obviously, this is true even of next. __iter__ is the special method, and after you have called it, you are operating on the iterable it returns (which may be self). next isn't generic to all objects; only to those which implement iteration (result from a previous call to __iter__). > The *original* idea was for __special__ names to apply to > anything that wasn't invoked using regular method notation: By this original intent, next would indeed be special, as would implementation of any generic functions -- but if 3rd-party generic/extensible functions become common they probably will need a way to carve out their own namespaces. -jJ From jimjjewett at gmail.com Sun Nov 26 22:52:55 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Sun, 26 Nov 2006 16:52:55 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: <5.1.1.6.0.20061122104602.02e32618@sparrow.telecommunity.com> References: <45645252.6010101@benjiyork.com> <5.1.1.6.0.20061122104602.02e32618@sparrow.telecommunity.com> Message-ID: On 11/22/06, Phillip J. Eby wrote: > At 08:36 AM 11/22/2006 -0500, Benji York wrote: > >It seems to me that not all interfaces coincide with > >something the object can _do_. ... > Aren't these also *operations* being performed? ISafeForThreading may boil down to certain operations not being dangerous, but I don't think the operations are the primary meaning. -jJ From pje at telecommunity.com Sun Nov 26 23:01:48 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 26 Nov 2006 17:01:48 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061122104602.02e32618@sparrow.telecommunity.com> <45645252.6010101@benjiyork.com> <5.1.1.6.0.20061122104602.02e32618@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061126165856.03ef19b0@sparrow.telecommunity.com> At 04:52 PM 11/26/2006 -0500, Jim Jewett wrote: >On 11/22/06, Phillip J. Eby wrote: > > At 08:36 AM 11/22/2006 -0500, Benji York wrote: > > >It seems to me that not all interfaces coincide with > > >something the object can _do_. ... > > > Aren't these also *operations* being performed? > >ISafeForThreading may boil down to certain operations not being >dangerous, but I don't think the operations are the primary meaning. I don't see how you can specify that something is thread-safe without explicit reference to what *operations* are thread-safe, and in what way. Indeed, it's the *operations* that one wishes to confirm are thread-safe; a data structure in and of itself has no "thread safety" per se. From tony at pagedna.com Sun Nov 26 23:14:25 2006 From: tony at pagedna.com (Tony Lownds) Date: Sun, 26 Nov 2006 14:14:25 -0800 Subject: [Python-3000] optional argument annotations In-Reply-To: References: <33D8CBFC-9B3B-4FDE-927F-4B6D6B11C434@PageDNA.com> Message-ID: On Nov 26, 2006, at 1:10 PM, Brett Cannon wrote: > On 11/24/06, Tony Lownds wrote: > > Obviously signature objects would grow support for annotations, > but I > > still need the information to be carried on the code object to > > incorporate into signature objects. > > > > Signature objects still need a way to know the nested parameters, > right? > > They already handle them. > I see, through the bytecode inspection that inspect.py does. > How about a co_argnames attribute? eg for > > def f((x, y), z): pass > > f.func_code.co_argnames would be (('x', 'y'), 'z') > > That's fine, but the compiler would need to change if it were to > use this. Plus I am still hoping to make nested parameters > disappear in Py3K (I won't be pushing for it any sooner than PyCon, > though). > Yes, it is the compiler that would implement this. I have it implemented as follows. >>> def f(a, (b, c), d=1, *e, f, g=1, **h): pass ... >>> f.func_code.co_argnames ('a', ('b', 'c'), 'd', '*e', 'f', 'g', '**h') However since inspect.py doesn't need this and neither does my code, I'll drop it. > I need to implement something like this to properly build > func_annotations > inside MAKE_FUNCTION. > > > I don't quite follow. Don't you already have support in > MAKE_FUNCTION when the object is created? > The support available is the code object and anything pushed onto the stack. Just looking at the code object, the information is simply not available outside of reverse- engineering co_code (as inspect does). I ended up pushing a tuple of argument names onto the stack. eg, for def f((x:1, y))->3: pass the bytecode is... 1 0 LOAD_CONST 0 (1) 3 LOAD_CONST 1 (3) 6 LOAD_CONST 2 (('x', 'return')) 9 LOAD_CONST 3 (", line 1>) 12 EXTENDED_ARG 3 15 MAKE_FUNCTION 196608 18 STORE_NAME 0 (f) 21 LOAD_CONST 4 (None) 24 RETURN_VALUE (the argument is so big because I need to pass the # of annotations in the argument of MAKE_FUNCTION so that the stack effect can be easily calculated) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061126/cd7596f8/attachment.html From jimjjewett at gmail.com Mon Nov 27 00:27:15 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Sun, 26 Nov 2006 18:27:15 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061121142441.03fb64e8@sparrow.telecommunity.com> <5.1.1.6.0.20061121211338.01f35718@sparrow.telecommunity.com> <5.1.1.6.0.20061122022306.03828420@sparrow.telecommunity.com> Message-ID: On 11/22/06, Guido van Rossum wrote: > I'd much rather have a few "big" interfaces (Number, > RealNumber, Integer) than lots of small ones > Pedroni-style (number-that-support-exponentiation, > number-that-supports-bitwise-and, > number-that-supports-left-shift, ...?). My understanding is that the Pedroni-style interfaces are big; they are just decomposable. File would be a (large) interface, but you could also specify that you only cared about (or only implemented) the subset consisting of File.close and File.write, in case you wanted something sort-of-like an existing interface. -jJ From guido at python.org Mon Nov 27 02:36:53 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 26 Nov 2006 17:36:53 -0800 Subject: [Python-3000] Fwd: defop ? In-Reply-To: <76fd5acf0611252028i496ef6d7r9b23674bcb38046e@mail.gmail.com> References: <5.1.1.6.0.20061122180342.03ce3008@sparrow.telecommunity.com> <76fd5acf0611221959g74e8d7a9o36723f63814aa950@mail.gmail.com> <76fd5acf0611222310x64f03aa3i3d516c30452b1d79@mail.gmail.com> <76fd5acf0611222310h1f6e4137r1dee4afa7a747ef0@mail.gmail.com> <76fd5acf0611241835m73186e80h7eb0e9eb3fa878ff@mail.gmail.com> <76fd5acf0611252028i496ef6d7r9b23674bcb38046e@mail.gmail.com> Message-ID: On 11/25/06, Calvin Spealman wrote: > I will make only one more comment and then ill drop my comments > without direct questions. > > On 11/25/06, Guido van Rossum wrote: > > Hm. The double colon rubs me the wrong way (Perl and/or C++). But > > apart from that, if this is the solution, I'm not sure the problem > > you're trying to solve is really worth solving. I just don't expect > > there will be all that many generic operations that need to be stuffed > > into arbitrary classes. Maybe I'll just take back what I said about > > wanting all such operations to be inside the class. Or maybe I'd be > > happier if there was a decorator indicating the name of the operation. > > I don't care about the syntax. anything that can denote an expression > (the left of the ::) and a name (the right of the ::) is OK and I dont > care how its denoted. > > > Also, I think you're overloading 'def' in a non-obvious way. > > Currently, "def foo..." means an assignment to the local variable foo. > > I would expect that if we extend the syntax for the thing between > > 'def' and the argument list to be more than just a name, it should > > still be considered an assignment target. But that's clearly not what > > you're after. From that POV, I find defop (while still unpleasant for > > other reasons) more "honest" than your overloading of def -- at least > > defop says upfront that it's not just an assignment. (Although the > > similarity with def is still confusing IMO.) > > You must be misunderstanding me. I am not saying that its not an > assignment. It would not change what def really means. operator::len > would be the actual name of the function to be created and the name of > the global, local, or class attribute it is bound to. I am saying > operator::len would become something like > MyClass.__dict__[operator::len] and what operator::len evaluates to, i > dont know. something that represents what it is. Maybe just a tuple. I > would expect it also exist for any assignment. special casing being > bad and all. Do you realize that expr[expr::expr] already has a meaning? You seem to be doing an exceptionally poor job explaining your proposal; you are introducing a new token "::" without ever explaining what you propose it should mean. How does operator::len differ from operator.len? > > Still not convinced? Focus on other problems first. This isn't the > > most important problem we're trying to solve. > > > > PS, It's __builtin__, not __builtins__ -- the latter is an > > unfortunately named but ultimately unimportant implementation detail > > (unimportant unless you're implemented restricted python, that is); > > the former is the module that is implied at the end of every free name > > search. > > > > > > --Guido > > -- > Read my blog! I depend on your acceptance of my opinion! I am interesting! > http://ironfroggy-code.blogspot.com/ > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Nov 27 02:40:03 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 26 Nov 2006 17:40:03 -0800 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <02AFCDEB-25D2-4D10-9BE4-A2773CCED65C@cleverdevil.org> Message-ID: On 11/25/06, Jim Jewett wrote: > Personally, I stand by my assessment that they can make metaclasses > look straightforward -- but that is partly because metaclasses have > had their roughest edges worn off already. (For example, they no > longer require compiling a C class so odd that the pattern gets named > after someone.) If GF explanations start to come from several > different people, the same process should eventually simplify them. > If that happened quickly enough, it would change the tradeoffs. Did you read my blog post where I give my own interpretation and implementation? http://www.artima.com/weblogs/viewpost.jsp?thread=155514 -- --Guido van Rossum (home page: http://www.python.org/~guido/) From janssen at parc.com Mon Nov 27 02:54:10 2006 From: janssen at parc.com (Bill Janssen) Date: Sun, 26 Nov 2006 17:54:10 PST Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> <8853879015409334245@unknownmsgid> <-4940442411288794438@unknownmsgid> Message-ID: <06Nov26.175416pst."58648"@synergy1.parc.xerox.com> > > I'll still drop off a copy (of Common Lisp the Language, version 2), > > but there's no need to wait. It's on the Web at > > http://www.supelec.fr/docs/cltl/clm/node260.html. See in particular > > the "change-class" operation at > > http://www.supelec.fr/docs/cltl/clm/node305.html. > > > > I think I'm still confused (happens a lot :-) about our method > > namespace discussion. It seems to me that Python's method namespaces > > work pretty much the same way that CLOS's do, already. That is, you > > don't "clobber" an existing method in a base class when you define a > > new method by the same name in a derived class; you just mask it. The > > base class' method is still there, and can still be called explicitly. > > OK, maybe I misunderstood what you wrote. I thought I heard you say > that "len" isn't just "len" -- it's the "len" defined by some > interface (and presumably implemented in a base class), and if one > defined a new "len" it wouldn't override the "len" defined by that > interface (unless one explicitly stated that it did), it would just > add a different method named "len". That would fly in the face of > Python's lookup algorithm for methods (where the first "len" you find > is the one you get). If that's not what you meant, all is probably > well. Actually, Python does both: it overrides the base class' "len", and adds a new "len" of its own. class Base: def len(self): ... class Derived(Base): def len(self): ... Given an instance of Derived, I can make a call on Derived.len, Base.len, or just let the default method lookup work, in which case I get Derived.len(). So even if Derived.len() is overridden, Base.len() is available. The question then becomes more subtle: is the data model maintained by the derived class consistent with that of the base class? But this is essentially off-topic: I still think the concern about "accidentally" overriding methods inherited from some base class is misplaced, but still minor compared to the other issue of not being able to figure out whether a value has a particular type. By the way, it's interesting to look at the distinction made in Guy Steele's (and Sun's) new research language, Fortress, between "traits" (base classes), and "objects" (final classes). Values are instances of "object" types, but "object" types may inherit code and interfaces from traits. Only "object" types can define instance variables, which means that much of the code in non-abstract "trait" methods is eventually based on abstract method calls (and property getter/setters). Bill From janssen at parc.com Mon Nov 27 02:59:35 2006 From: janssen at parc.com (Bill Janssen) Date: Sun, 26 Nov 2006 17:59:35 PST Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <20061122165513.GA13795@niemeyer.net> <4564F8B7.5030007@canterbury.ac.nz> <20061123020054.GA31618@niemeyer.net> <45651355.7090605@canterbury.ac.nz> <20061123121416.GA6072@niemeyer.net> <45661AA3.9070803@canterbury.ac.nz> <20061123224759.GA27828@niemeyer.net> <456631AF.1020208@canterbury.ac.nz> Message-ID: <06Nov26.175941pst."58648"@synergy1.parc.xerox.com> > It may also be that only a good unified solution to "A" is needed in order to > allow library and user code to be written "easier" to address "B","C" and "D". > It may even be just doing "A" is a good enough 95% solution (works well enough > for 95% of actual use cases) and nothing more needs to be done. to tell> I guess this is sort of what I think. And I hate to add yet another framework/system to the Python system, thus my proposal to handle it all using the existing type system, by re-factoring the various base classes (file, sequence, number, list, mapping, string, etc.) into a set of interface types, and re-defining the base types as various combinations of these interface types (which may be abstract, or concrete -- we haven't really talked enough about specifics to know yet). This would give us a decent base to use later (or now) for adding a multi-method dispatching system, as well. And optional type annotations, etc. Bill From janssen at parc.com Mon Nov 27 03:16:10 2006 From: janssen at parc.com (Bill Janssen) Date: Sun, 26 Nov 2006 18:16:10 PST Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> <3497511880590744734@unknownmsgid> Message-ID: <06Nov26.181616pst."58648"@synergy1.parc.xerox.com> > I had thought that you (Bill), (along with PJE) were assuming that the > generic function or interface itself would serve to distinguish the > namespace. Instead of > > class A(object): > def magic_meth_foo(self, ...): > > I would write > > class A(object): > defop foo.magic_meth(self, ...) > There seem to be two justifications for "special" methods that have been adduced (though I see no real justification for the name-mangling scheme). One is that they serve as a way of specifying which code operators and their cousins, the "built-in" generic functions, call. The other is that they somehow serve as connections to the underlying VM implementation. Now, I don't see why the connection to the underlying VM implementation needs name-mangling -- it would be reasonable just to define a distinguished module (call it, say, for tradition's sake, "__VM__" :-), and put them in that module. But there does need to be some way to connect operator syntax to code, and we could talk about how to do that. Defop is one solution; you could also imagine an ABC for each operator, and classes which wish to use that operator, or overload it, would inherit from that ABC and re-define the methods. The question is, would those methods, bound to specific operators, be so pervasive and so common that their logical method names would, if not mangled, occupy too much of the available naming space? Maybe. I think it's a judgement call. Bill From ross at sourcelabs.com Mon Nov 27 06:09:29 2006 From: ross at sourcelabs.com (Ross Jekel) Date: Sun, 26 Nov 2006 21:09:29 -0800 (PST) Subject: [Python-3000] Generic function queries In-Reply-To: <238202539.33881164602206006.JavaMail.root@mail-2.colo.sourcelabs.com> Message-ID: <1017234058.33901164604169480.JavaMail.root@mail-2.colo.sourcelabs.com> Hi, I've been reading the discussions on interfaces and generic functions with much interest. Some people have expressed concerns that readability and code discovery (where is the code that this call is going to run) may suffer. Could someone please address some simple questions for me about this as I'm not familiar with using generics in large projects? For a reusable library of code that includes generics, does the library writer typically try to limit exception types that can pass out of the generic function? In other words, I guess what I'm asking is if there is way a for a library writer to register an "around" rule that would always be outermost so that one could wrap everything including later overrides in a try: except: or a transaction or something like that. If this is a *common* pattern would it be better handled by registering one or more context managers for the generic function? I haven't fully thought this out yet, but it seems like if context managers were useful for a suite of code using the "with" statement, that they could be just as applicable for managing the context of the suites of code reachable through the generic function dispatch. I leave syntax proposals for later if this is a useful idea, but some obvious ones come to mind. I understand addmethod and hasmethod as have been proposed, but is there a need (for unit tests or other situations) to have a function that would return the concrete function object or list of function objects (if there is a chain) for a particular set of arguments without actually calling them (a getmethod or querymethod operation)? If I had a generic function f, could I interactively do: dir(f) to begin discovering concrete functions associated with the generic, or there be a separate inspection mechanism? What does help(f) do? I think Phillip said something about this, but I don't understand it from the usability aspect (i.e. Does help merge all docstrings from known implementations, provide the docstring of the default?) Ross From ironfroggy at gmail.com Mon Nov 27 06:51:39 2006 From: ironfroggy at gmail.com (Calvin Spealman) Date: Mon, 27 Nov 2006 00:51:39 -0500 Subject: [Python-3000] Fwd: defop ? In-Reply-To: References: <76fd5acf0611221959g74e8d7a9o36723f63814aa950@mail.gmail.com> <76fd5acf0611222310x64f03aa3i3d516c30452b1d79@mail.gmail.com> <76fd5acf0611222310h1f6e4137r1dee4afa7a747ef0@mail.gmail.com> <76fd5acf0611241835m73186e80h7eb0e9eb3fa878ff@mail.gmail.com> <76fd5acf0611252028i496ef6d7r9b23674bcb38046e@mail.gmail.com> Message-ID: <76fd5acf0611262151l7abc50bekd5bef41ca9311d0e@mail.gmail.com> On 11/26/06, Guido van Rossum wrote: > On 11/25/06, Calvin Spealman wrote: > > I will make only one more comment and then ill drop my comments > > without direct questions. > > > > On 11/25/06, Guido van Rossum wrote: > > > Hm. The double colon rubs me the wrong way (Perl and/or C++). But > > > apart from that, if this is the solution, I'm not sure the problem > > > you're trying to solve is really worth solving. I just don't expect > > > there will be all that many generic operations that need to be stuffed > > > into arbitrary classes. Maybe I'll just take back what I said about > > > wanting all such operations to be inside the class. Or maybe I'd be > > > happier if there was a decorator indicating the name of the operation. > > > > I don't care about the syntax. anything that can denote an expression > > (the left of the ::) and a name (the right of the ::) is OK and I dont > > care how its denoted. > > > > > Also, I think you're overloading 'def' in a non-obvious way. > > > Currently, "def foo..." means an assignment to the local variable foo. > > > I would expect that if we extend the syntax for the thing between > > > 'def' and the argument list to be more than just a name, it should > > > still be considered an assignment target. But that's clearly not what > > > you're after. From that POV, I find defop (while still unpleasant for > > > other reasons) more "honest" than your overloading of def -- at least > > > defop says upfront that it's not just an assignment. (Although the > > > similarity with def is still confusing IMO.) > > > > You must be misunderstanding me. I am not saying that its not an > > assignment. It would not change what def really means. operator::len > > would be the actual name of the function to be created and the name of > > the global, local, or class attribute it is bound to. I am saying > > operator::len would become something like > > MyClass.__dict__[operator::len] and what operator::len evaluates to, i > > dont know. something that represents what it is. Maybe just a tuple. I > > would expect it also exist for any assignment. special casing being > > bad and all. > > Do you realize that expr[expr::expr] already has a meaning? Yes. As I've stated repeatedly, I don't care for or encourage the a::b syntax itself, but am using it as a representation of whatever might become used. I don't know what, at this point. > You seem to be doing an exceptionally poor job explaining your > proposal; you are introducing a new token "::" without ever explaining > what you propose it should mean. How does operator::len differ from > operator.len? A.b evaluates to the value of attribute 'b' of some object bound to name 'A', of course. A::b (or whatever syntax might be used) would evaluate to some value that actually represents "The attribute 'b' of this object". This "attribute object" (poor term, yes) could have other uses, but thats not completely relevent now. For the case at hand, basically we would allow the syntax for these objects to be used as legal names (keys in the __dict__ or elements of the __slots__). Thus, the way in `A.__len__` the string '__len__' is created and looked up in the dictionary, with `A.operator::len` (or whatever syntax is used) would be the key in the __dict__, the actual name of the attribute of the A class. Basically, it allows for using any object in combination with a name to have identifiers that do not conflict. Thus we can have all the operator methods we want and never conflict with names others might use. The interface libraries can operate without worry. Lots of problems solved. I am sorry if my thoughts are not conveyed as clear as they sound in my mind. > > > Still not convinced? Focus on other problems first. This isn't the > > > most important problem we're trying to solve. > > > > > > PS, It's __builtin__, not __builtins__ -- the latter is an > > > unfortunately named but ultimately unimportant implementation detail > > > (unimportant unless you're implemented restricted python, that is); > > > the former is the module that is implied at the end of every free name > > > search. > > > > > > > > > --Guido > > > > -- > > Read my blog! I depend on your acceptance of my opinion! I am interesting! > > http://ironfroggy-code.blogspot.com/ > > _______________________________________________ > > Python-3000 mailing list > > Python-3000 at python.org > > http://mail.python.org/mailman/listinfo/python-3000 > > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/ From fredrik at pythonware.com Mon Nov 27 08:01:26 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Mon, 27 Nov 2006 08:01:26 +0100 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: <06Nov26.181616pst."58648"@synergy1.parc.xerox.com> References: <-6683921950610559197@unknownmsgid> <3497511880590744734@unknownmsgid> <06Nov26.181616pst."58648"@synergy1.parc.xerox.com> Message-ID: Bill Janssen wrote: > Now, I don't see why the connection to the > underlying VM implementation needs name-mangling -- it would be > reasonable just to define a distinguished module (call it, say, for > tradition's sake, "__VM__" :-), and put them in that module. putting methods in a module? how would that work? From jimjjewett at gmail.com Mon Nov 27 14:30:07 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 27 Nov 2006 08:30:07 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <4b57b0700611211559y20e84b5fp2198a15d9f75e59f@mail.gmail.com> <5.1.1.6.0.20061121201100.038cf008@sparrow.telecommunity.com> <5.1.1.6.0.20061122102152.01f2f350@sparrow.telecommunity.com> <02AFCDEB-25D2-4D10-9BE4-A2773CCED65C@cleverdevil.org> Message-ID: On 11/26/06, Guido van Rossum wrote: > On 11/25/06, Jim Jewett wrote: > > Personally, I stand by my assessment that they can make metaclasses > > look straightforward -- but that is partly because metaclasses have > > had their roughest edges worn off already. (For example, they no > > longer require compiling a C class so odd that the pattern gets named > > after someone.) If GF explanations start to come from several > > different people, the same process should eventually simplify them. > > If that happened quickly enough, it would change the tradeoffs. > > Did you read my blog post where I give my own interpretation and implementation? > > http://www.artima.com/weblogs/viewpost.jsp?thread=155514 Yes, and also the comments. If overloaded functions were proposed as an optional 3rd party module (a possibility you did mention), they would be interesting. When they are proposed as a fundamental part of the language itself, it is crucial that there be a quick, easy explanation available as soon as people need to work with them. For metaclasses, people don't need to work with them for quite a while. For something like print or len ... they need it sooner. (And 3rd party registration means that someone else's misunderstanding can cause bugs in your code.) I felt that there was still too much confusion for something that is proposed as a fundamental building block. I expect that the confusion would get worse once you have to start debugging multiple 3rd-party overloads. The "TypeError: hex() argument can't be converted to hex" is a hint at the problems, but I expect them to be worse when the chosen function is a match -- just not the one you expected. -jJ From gsakkis at rutgers.edu Mon Nov 27 15:21:29 2006 From: gsakkis at rutgers.edu (George Sakkis) Date: Mon, 27 Nov 2006 09:21:29 -0500 Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> <3497511880590744734@unknownmsgid> Message-ID: <91ad5bf80611270621v193cc81cv86b4b5577eef6bfc@mail.gmail.com> On 11/27/06, Fredrik Lundh wrote: > Bill Janssen wrote: > > > Now, I don't see why the connection to the > > underlying VM implementation needs name-mangling -- it would be > > reasonable just to define a distinguished module (call it, say, for > > tradition's sake, "__VM__" :-), and put them in that module. > > putting methods in a module? how would that work? > > How about a nested class, like Meta in Django ? George From janssen at parc.com Mon Nov 27 18:07:37 2006 From: janssen at parc.com (Bill Janssen) Date: Mon, 27 Nov 2006 09:07:37 PST Subject: [Python-3000] Special methods and interface-based type system In-Reply-To: References: <-6683921950610559197@unknownmsgid> <3497511880590744734@unknownmsgid> <06Nov26.181616pst."58648"@synergy1.parc.xerox.com> Message-ID: <06Nov27.090742pst."58648"@synergy1.parc.xerox.com> > putting methods in a module? how would that work? You'd define classes and functions intended for VM coupling in that module's namespace. They might (or might not, depending on the VM) actually be implemented by the VM, but to Python code they would appear in that namespace. Bill From guido at python.org Mon Nov 27 18:41:26 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 27 Nov 2006 09:41:26 -0800 Subject: [Python-3000] Fwd: defop ? In-Reply-To: <76fd5acf0611262151l7abc50bekd5bef41ca9311d0e@mail.gmail.com> References: <76fd5acf0611222310x64f03aa3i3d516c30452b1d79@mail.gmail.com> <76fd5acf0611222310h1f6e4137r1dee4afa7a747ef0@mail.gmail.com> <76fd5acf0611241835m73186e80h7eb0e9eb3fa878ff@mail.gmail.com> <76fd5acf0611252028i496ef6d7r9b23674bcb38046e@mail.gmail.com> <76fd5acf0611262151l7abc50bekd5bef41ca9311d0e@mail.gmail.com> Message-ID: On 11/26/06, Calvin Spealman wrote: > A.b evaluates to the value of attribute 'b' of some object bound to > name 'A', of course. > A::b (or whatever syntax might be used) would evaluate to some value > that actually represents "The attribute 'b' of this object". This > "attribute object" (poor term, yes) could have other uses, but thats > not completely relevent now. For the case at hand, basically we would > allow the syntax for these objects to be used as legal names (keys in > the __dict__ or elements of the __slots__). Thus, the way in > `A.__len__` the string '__len__' is created and looked up in the > dictionary, with `A.operator::len` (or whatever syntax is used) > would be the key in > the __dict__, the actual name of the attribute of the A class. > > Basically, it allows for using any object in combination with a name > to have identifiers that do not conflict. Thus we can have all the > operator methods we want and never conflict with names others might > use. The interface libraries can operate without worry. Lots of > problems solved. > > I am sorry if my thoughts are not conveyed as clear as they sound in my mind. No worry. It is now completely clear. The idea is interesting and it's a useful one to have handy, even without a concrete syntax proposal. Thanks for hanging in there! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jimjjewett at gmail.com Mon Nov 27 20:01:55 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 27 Nov 2006 14:01:55 -0500 Subject: [Python-3000] Abilities / Interfaces In-Reply-To: References: <5.1.1.6.0.20061122022615.037de350@sparrow.telecommunity.com> <5.1.1.6.0.20061122130031.041f4960@sparrow.telecommunity.com> Message-ID: On 11/22/06, Guido van Rossum wrote: > On 11/22/06, Phillip J. Eby wrote: > > At 09:24 AM 11/22/2006 -0800, Bill Janssen wrote: > > In Java, SMTP.sendmail would be something like this (using Python-like > > syntax 'cause my Java is rusty): > > def sendmail(self, from, to_addrs:str, msg, ...): > > return self.sendmail(from_addr, [to_addrs], msg, ...) > > def sendmail(self, from, to_addrs:list[str], msg, ...): > > # main implementation > Right. If this syntax was possible in Python lots of people would be > very happy. But even the best generic function API I've seen is a lot > more verbose than this -- there seems to be a separate set-up > involved. 3rd-party registration is fundamental to extensible functions. The best you can do is to minimize it. def sendmail(self, from, to_addrs, msg, ...): raise TypeError(to_addrs must be a string or list) overload sendmail(self, from, to_addrs:str, msg, ...): return self.sendmail(from_addr, [to_addrs], msg, ...) overload sendmail(self, from, to_addrs:list[str], msg, ...): # main implementation And then someone else could write overload smtplib.SMTP.sendmail(self, from, to_addrs:Seq[my_str], msg, ...): to_addrs = [str(addr) for addr in to_addrs] return smtplib.SMTP.sendmail(self, from, to_addrs:list[str], msg, ...) From jimjjewett at gmail.com Mon Nov 27 20:19:22 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 27 Nov 2006 14:19:22 -0500 Subject: [Python-3000] Abilities / Interfaces and security Message-ID: On 11/22/06, Phillip J. Eby wrote: > While this is more verbose than sticking an 'if' into the original > sendmail(), note that it can also be added by a third party, via: > overload smtpblib.SMTP.sendmail( > self, from_addr, to_addrs:str, msg, > mail_options=[], rcpt_options=[] > ): > self.sendmail(from_addr, [to_addrs], msg, mail_options, rcpt_options) > So, someone who didn't want to wait for the patch to add your "if" could > just add their own fix on the fly. :) Hey, cool, can I also do overload smtplib.SMTP.sendmail(self, from_addr:str, to_addrs:str, msg, ...) if stalkee == from_addr: archive_and_randomly_rewrite_before_sending(...) Since this is more specific on the from_addr, it should always be called. Also, can I overload __builtins__.str(arg:object) to effectively extend (or replace) the builtin type? There is certainly value in open classes, and it can make writing code easier. But it can also make maintaining code much harder. Today, almost all classes are open, but those coded in C are typically closed -- and Brett is relying on this for security. How-it-happened-to-be-implemented may not be the best way to decide open/closed, but I think there needs to be *some* way of keeping a class closed. -jJ From jimjjewett at gmail.com Mon Nov 27 20:34:33 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 27 Nov 2006 14:34:33 -0500 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: References: Message-ID: On 11/24/06, Guido van Rossum wrote: > Thinking about my own example some more, I think it would actually > work just as well using ABCs instead of abilities. ... It seems the use > case for making abilities/interfaces separate from ABCs hinges on > the assumption that there are classes that implement certain > protocols without realizing it, IOW emergent > protocols, plus that it's not practical to change __bases__. In some cases, the interface might require slight adaptation, such as aliasing a function. But that is something which can be handled better by adding a new base class. > I wonder if a bunch of well thought-out standard ABCs, applied to the > standard data types, and perhaps more support for setting __bases__, > wouldn't address most concerns. I think so. A couple thoughts: The problem with setting bases (other than extension classes) is largely a style issue -- it can be done now. The code for setting bases even has comments suggesting how to speed it up, but saying that it isn't worth it because bases shouldn't change much. It might be worth adding an __interfaces__ that gets searched after __bases__. The distinction would indicate that an interface is lightweight, may change, and (for security and efficiency) cannot override existing methods. I couldn't change the builtin str.__contains__, but I could add myproto to str.__interfaces__, and then I could make calls to "asdf".__interfaces__.myproto.__contains__ for arbitrary strings. I'm not sure that adding it to the builtin (but in a new name) would really buy much over an adapter (x in MyProto("asdf") ), but the interface fans may feel differently. -jJ From tim.hochberg at ieee.org Mon Nov 27 20:53:29 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Mon, 27 Nov 2006 12:53:29 -0700 Subject: [Python-3000] Abilities / Interfaces and security In-Reply-To: References: Message-ID: Jim Jewett wrote: > On 11/22/06, Phillip J. Eby wrote: > >> While this is more verbose than sticking an 'if' into the original >> sendmail(), note that it can also be added by a third party, via: > >> overload smtpblib.SMTP.sendmail( >> self, from_addr, to_addrs:str, msg, >> mail_options=[], rcpt_options=[] >> ): >> self.sendmail(from_addr, [to_addrs], msg, mail_options, rcpt_options) > >> So, someone who didn't want to wait for the patch to add your "if" could >> just add their own fix on the fly. :) > > Hey, cool, can I also do > > overload smtplib.SMTP.sendmail(self, from_addr:str, to_addrs:str, msg, ...) > if stalkee == from_addr: > archive_and_randomly_rewrite_before_sending(...) > > Since this is more specific on the from_addr, it should always be called. I suppose this issue may be real, but this example seems like FUD; for most classes I can completely override the behavior today simply by monkey patching the method and this isn't typeically a problem that we see: smtplib.SMTP.sendmail = my_bogus_sendmail_function > > Also, can I > > overload __builtins__.str(arg:object) > > to effectively extend (or replace) the builtin type? > > There is certainly value in open classes, and it can make writing code > easier. This isn't about open/closed classes except incidentally is it? It's really about open/closed functions and methods. But it can also make maintaining code much harder. Today, > almost all classes are open, but those coded in C are typically closed > -- and Brett is relying on this for security. > How-it-happened-to-be-implemented may not be the best way to decide > open/closed, but I think there needs to be *some* way of keeping a > class closed. I would expect that either functions would default to closed and you would get a generic function with an appropriate decorator, or functions would default to open and you could instead close them, again with an appropriate decorator. To be concrete, either: def my_closed_function(arg): #.... @overloadable def my_open_function(arg): #.... overload my_open_function(arg:str): #.... or: @closed #final? nonoverloadable? def my_closed_function(arg): #.... def my_open_function(arg): #.... overload my_open_function(arg:str): #.... Either way, this doesn't look like a big deal. I'm more interested in having, for debugging purposes, a way to do some introspection on generic functions. For instance, finding out what function would be called for a given set of arguments and perhaps some information on why that match was chosen. Also, what overloads are set up and with what signatures. -tim From pje at telecommunity.com Mon Nov 27 21:25:14 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 27 Nov 2006 15:25:14 -0500 Subject: [Python-3000] Generic functions vs. OO In-Reply-To: References: Message-ID: <5.1.1.6.0.20061127151739.028664a8@sparrow.telecommunity.com> At 02:34 PM 11/27/2006 -0500, Jim Jewett wrote: >I'm not sure that adding it to the builtin (but in a new name) would >really buy much over an adapter (x in MyProto("asdf") ), but the >interface fans may feel differently. For what it's worth, in Chandler there is a system called Annotations that allows third-party packages to add private attributes to existing (persistent) data types. It's used to implement "stamping", which is a way that you can (for example) dynamically change a note or photo into a task or a calendar event. So, if I want to be able to add calendar information to "stampable" objects, I can do something like: class CalendarEvent(Stamp): # field definitions here # method definitions And then I can access fields on stamped items by using e.g. CalendarEvent(someItem).startTime. So, effectively CalendarEvent is a kind of adapter class, but we don't use the word adapter, simply because it isn't relevant to the application domain. It's simply a way of having an open system for adding additional data to existing object types. There are no interfaces or generic functions involved, either, it's just a way of separating namespaces belonging to independent developers. I just mention it because the reaction to it among developers with no previous exposure to Python adaptation has been fairly positive. From brett at python.org Mon Nov 27 21:35:20 2006 From: brett at python.org (Brett Cannon) Date: Mon, 27 Nov 2006 12:35:20 -0800 Subject: [Python-3000] optional argument annotations In-Reply-To: References: <33D8CBFC-9B3B-4FDE-927F-4B6D6B11C434@PageDNA.com> Message-ID: On 11/26/06, Tony Lownds wrote: > > > On Nov 26, 2006, at 1:10 PM, Brett Cannon wrote: > > On 11/24/06, Tony Lownds wrote: > > > > > Obviously signature objects would grow support for annotations, but I > > > still need the information to be carried on the code object to > > > incorporate into signature objects. > > > > > > > Signature objects still need a way to know the nested parameters, right? > > > > > They already handle them. > > > I see, through the bytecode inspection that inspect.py does. > Yep. How about a co_argnames attribute? eg for > > > > def f((x, y), z): pass > > > > f.func_code.co_argnames would be (('x', 'y'), 'z') > > > That's fine, but the compiler would need to change if it were to use > this. Plus I am still hoping to make nested parameters disappear in Py3K (I > won't be pushing for it any sooner than PyCon, though). > > > Yes, it is the compiler that would implement this. I have it implemented > as follows. > > >>> def f(a, (b, c), d=1, *e, f, g=1, **h): pass > ... > >>> f.func_code.co_argnames > ('a', ('b', 'c'), 'd', '*e', 'f', 'g', '**h') > > However since inspect.py doesn't need this and neither does my code, I'll > drop it. > OK. I need to implement something like this to properly build > > func_annotations > > inside MAKE_FUNCTION. > > > > I don't quite follow. Don't you already have support in MAKE_FUNCTION > when the object is created? > > > The support available is the code object and anything pushed onto the > stack. Just looking at the code object, > the information is simply not available outside of reverse-engineering > co_code (as inspect does). > I ended up pushing a tuple of argument names onto the stack. > > eg, for > > def f((x:1, y))->3: pass > > the bytecode is... > > 1 0 LOAD_CONST 0 (1) > 3 LOAD_CONST 1 (3) > 6 LOAD_CONST 2 (('x', 'return')) > 9 LOAD_CONST 3 ( file "", line 1>) > 12 EXTENDED_ARG 3 > 15 MAKE_FUNCTION 196608 > 18 STORE_NAME 0 (f) > 21 LOAD_CONST 4 (None) > 24 RETURN_VALUE > > (the argument is so big because I need to pass the # of annotations in the > argument of MAKE_FUNCTION so > that the stack effect can be easily calculated) > Ah, I see how you are doing it. Well, as I said, as long as the info is available in some way that can be used to infer what the type annotations are per argument then I should be fine. It doesn't have to be overly clean just for signature objects. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061127/eb33f641/attachment.htm From talin at acm.org Tue Nov 28 10:22:07 2006 From: talin at acm.org (Talin) Date: Tue, 28 Nov 2006 01:22:07 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject Message-ID: <456BFFBF.9000208@acm.org> More on the 'cruft removal' topic: I notice that a number of fields in PyTypeObject are deprecated; the question is, how to get rid of them without gratuitously breaking everything? Even if there's zero code out there that still uses tp_getattr instead of tp_getattro, you can't simply remove the field because it takes up a placeholder slot in the C initializer list. Has anyone proposed the notion of getting away from C-style initializer lists, at least for the case of PyTypeObject? For example, consider the case for tp_init and tp_new. Currently these are filled in with the C initializer list for PyTypeObject. Because these are the 38th and 40th fields in PyTypeObject, you have to put 37 initializer values before them, whether you use them or not. Instead of doing that, however, you could put them in the method table, tp_methods: static PyMethodDef Noddy_methods[] = { { SPECMETH_NEW, (PyCFunction)Noddy_new, METH_VARARGS, "..." }, { SPECMETH_INIT, (PyCFunction)Noddy_init, METH_VARARGS, "..." }, { NULL } /* Sentinel */ }; 'SPECMETH_NEW' and 'SPECMETH_INIT' are sentinel values indicating that these are 'special' methods, which are ignored during the normal scanning of this table. Instead, when the type is initialized, the method table is scanned for these special methods, and the corresponding field in PyTypeObject is filled in with the method pointer. The same holds true for all of the other special method pointers. Now, you still have the problem that 'tp_methods' itself is something like the 28th field in the struct, which means you still have all of those empty fields to consider. So instead, make a new version of PyType_Ready which takes an optional parameter of the method table for the type, which is filled in as part of the initialization. You could add another parameter for filling of data fields, which would point to a similar table that would hold values for non-function initializers. What you end up with is code that looks like this: PyTypeObject myType = { PyObject_HEAD_INIT(NULL) 0, "myType", sizeof(myInstance) } void init() { if (PyType_ReadyInit( &myType, myTypeMethods, myTypeData ) < 0) return; } Note that this requires no change to the PyTypeObject struct itself, or of any code that currently uses it - the only thing that changes is the way that the struct's data fields are filled in. What this would give you is a way to construct types that would be immune to future revisions of the PyTypeObject struct; You could add, remove, rename, or re-order the fields in the struct without breaking existing code. (If there's an existing proposal for this, feel free to post a reference. I did look :) -- Talin From fredrik at pythonware.com Tue Nov 28 10:35:48 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 28 Nov 2006 10:35:48 +0100 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456BFFBF.9000208@acm.org> References: <456BFFBF.9000208@acm.org> Message-ID: Talin wrote: > Has anyone proposed the notion of getting away from C-style initializer > lists, at least for the case of PyTypeObject? yes, and it was mentioned in the python-dev summaries that were mailed out a couple of days ago: http://effbot.org/zone/idea-register-type.htm Larry Hastings has prepared a patch for this; check the patch tracker or the python-dev archives for details. From greg.ewing at canterbury.ac.nz Tue Nov 28 11:42:04 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 28 Nov 2006 23:42:04 +1300 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456BFFBF.9000208@acm.org> References: <456BFFBF.9000208@acm.org> Message-ID: <456C127C.8080502@canterbury.ac.nz> Talin wrote: > What you end up with is code that looks like this: > > PyTypeObject myType = { > PyObject_HEAD_INIT(NULL) > 0, > "myType", > sizeof(myInstance) > } > > void init() { > if (PyType_ReadyInit( &myType, myTypeMethods, myTypeData ) < 0) > return; > } If you're going that far, why not go a step further and do away with the statically-declared type object altogether? PyTypeObject *myType; myType = PyType_Create(sizeof(myInstance), myTypeMethods, myTypeData); -- Greg From talin at acm.org Tue Nov 28 11:53:51 2006 From: talin at acm.org (Talin) Date: Tue, 28 Nov 2006 02:53:51 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456C127C.8080502@canterbury.ac.nz> References: <456BFFBF.9000208@acm.org> <456C127C.8080502@canterbury.ac.nz> Message-ID: <456C153F.4070108@acm.org> Greg Ewing wrote: > Talin wrote: > >> What you end up with is code that looks like this: >> >> PyTypeObject myType = { >> PyObject_HEAD_INIT(NULL) >> 0, >> "myType", >> sizeof(myInstance) >> } >> >> void init() { >> if (PyType_ReadyInit( &myType, myTypeMethods, myTypeData ) < 0) >> return; >> } > > If you're going that far, why not go a step further and do > away with the statically-declared type object altogether? > > PyTypeObject *myType; > > myType = PyType_Create(sizeof(myInstance), myTypeMethods, myTypeData); That makes sense - I was trying to avoid allocations (my day-job habits leaking through again), but this is Python after all, and avoiding a single memory allocation per type is silly. > -- > Greg > From guido at python.org Tue Nov 28 16:39:11 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 28 Nov 2006 07:39:11 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: References: <456BFFBF.9000208@acm.org> Message-ID: Some comments: - I'm all for trying something new here; the existing approach is definitely aged. - Fredrik's solution makes one call per registered method. (I don't know if the patch he refers to follows that model.) That seems a fair amount of code for an average type -- I'm wondering if it's too early to worry about code bloat (I don't think the speed is going to matter). - Talin's solution seems to require the definition of an awful lot of new constants -- one per slot. And a lot of special-casing in the type initialization code to handle them because there are so many different signatures. - Both solutions proposed require rewriting *all* type initialization. This is likely to require a tool that can do 99% of the work automatically (or else extension writers will truly hate us). But then maybe we could write a tool instead that can automatically rewrite a type struct declaration to follow the new lay-out. Since almost all type declarations are cloned from the examples in early Python, they are very regular -- basically it's an optional case, a function name or zero, and a comment with the member name. - Can't we require a C99 compiler and use C99 struct initialization? Then the table lines could look like tp_new = Noddy_new, tp_init = Noddy_init, This probably means the conversion tool would be even simpler (a couple of lines of sed would do). It has my vote if C99 is available on Windows (GCC covers all other platforms -- vendors that don't have a C99 compiler yet lose, it's 2006 now for X sake). --Guido On 11/28/06, Fredrik Lundh wrote: > Talin wrote: > > > Has anyone proposed the notion of getting away from C-style initializer > > lists, at least for the case of PyTypeObject? > > yes, and it was mentioned in the python-dev summaries that were mailed > out a couple of days ago: > > http://effbot.org/zone/idea-register-type.htm > > Larry Hastings has prepared a patch for this; check the patch tracker or > the python-dev archives for details. > > > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik at pythonware.com Tue Nov 28 17:19:54 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 28 Nov 2006 17:19:54 +0100 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: References: <456BFFBF.9000208@acm.org> Message-ID: Guido van Rossum wrote: > - Fredrik's solution makes one call per registered method. (I don't > know if the patch he refers to follows that model.) That seems a fair > amount of code for an average type -- I'm wondering if it's too early > to worry about code bloat (I don't think the speed is going to > matter). too early, I think. and memory is relatively cheap, compare to the costs of upgrade pain, programmer time, and lack of optimization opportunities due to "bare data structures". > - Both solutions proposed require rewriting *all* type initialization. > This is likely to require a tool that can do 99% of the work > automatically (or else extension writers will truly hate us). yup. I think a tool that generates cut-that-and-paste-this instructions for the developer should be good enough, though, and fairly easy to write, for the reasons you give. > Can't we require a C99 compiler and use C99 struct initialization? that won't address the binary compatibility and optimization issues that are the main rationales for my proposal, though. From Brent.Benson at oracle.com Tue Nov 28 17:10:19 2006 From: Brent.Benson at oracle.com (Brent Benson) Date: Tue, 28 Nov 2006 11:10:19 -0500 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: References: <456BFFBF.9000208@acm.org> Message-ID: <456C5F6B.9090009@oracle.com> Guido van Rossum wrote: > - Can't we require a C99 compiler and use C99 struct initialization? > Then the table lines could look like > > tp_new = Noddy_new, > tp_init = Noddy_init, > > This probably means the conversion tool would be even simpler (a > couple of lines of sed would do). It has my vote if C99 is available > on Windows (GCC covers all other platforms -- vendors that don't have > a C99 compiler yet lose, it's 2006 now for X sake). My reading of the Microsoft Visual C++ documentation indicates that they do not support or plan to support C99. On the other hand, it looks like the Intel V8 compiler for Windows supports most of C99 including the new style struct initialization. -Brent From jimjjewett at gmail.com Tue Nov 28 17:59:56 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 28 Nov 2006 11:59:56 -0500 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456BFFBF.9000208@acm.org> References: <456BFFBF.9000208@acm.org> Message-ID: On 11/28/06, Talin wrote: > For example, consider the case for tp_init and tp_new. ... you have > to put 37 initializer values before them, whether you use them or not. > Now, you still have the problem that 'tp_methods' itself is something > like the 28th field in the struct, which means you still have all of > those empty fields to consider. > Note that this requires no change to the PyTypeObject struct itself, or > of any code that currently uses it - the only thing that changes is the > way that the struct's data fields are filled in. > What this would give you is a way to construct types that would be > immune to future revisions of the PyTypeObject struct; You could add, > remove, rename, or re-order the fields in the struct without breaking > existing code. Am I understanding this correctly? (1) The change is that instead of a header structure filled in with zeros, you have a collection of function calls. (And this secretly makes that zero-filled structure for you.) Since they're each a function call, it may take just as much space to write out, but there won't be any (source code) place to read the physical layout, and it won't be as obvious that you've left a method out. With C99 initializers, you at least get named (rather than comment-named) fields, but I don't see any advantage to using function calls if the type slots don't change. (2) In theory, it would be easier to add/remove/reorder slots between major versions, but you wouldn't actually do this because of backwards compatibility. So then why make a change at all? -jJ From guido at python.org Tue Nov 28 18:34:33 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 28 Nov 2006 09:34:33 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: References: <456BFFBF.9000208@acm.org> Message-ID: On 11/28/06, Fredrik Lundh wrote: > Guido van Rossum wrote: > > > - Fredrik's solution makes one call per registered method. (I don't > > know if the patch he refers to follows that model.) That seems a fair > > amount of code for an average type -- I'm wondering if it's too early > > to worry about code bloat (I don't think the speed is going to > > matter). > > too early, I think. > > and memory is relatively cheap, compare to the costs of upgrade pain, > programmer time, and lack of optimization opportunities due to "bare > data structures". > > > - Both solutions proposed require rewriting *all* type initialization. > > This is likely to require a tool that can do 99% of the work > > automatically (or else extension writers will truly hate us). > > yup. I think a tool that generates cut-that-and-paste-this instructions > for the developer should be good enough, though, and fairly easy to > write, for the reasons you give. > > > Can't we require a C99 compiler and use C99 struct initialization? > > that won't address the binary compatibility and optimization issues that > are the main rationales for my proposal, though. Why not? This is the Py3k list -- there is no hope for binary compatibility with 2.x. AFAIU the C99 approach can be easily binary compatible between 3.0, 3.1 and beyond -- please explain if this is not so. And what's the optimization issue? Do you mean the ccasional NULL poitner check? Once we make PyType_Ready() obligatory, it can do the same thing. Or whatever else you are thinking of. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik at pythonware.com Tue Nov 28 18:48:01 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 28 Nov 2006 18:48:01 +0100 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: References: <456BFFBF.9000208@acm.org> Message-ID: Guido van Rossum wrote: >>> Can't we require a C99 compiler and use C99 struct initialization? >> that won't address the binary compatibility and optimization issues that >> are the main rationales for my proposal, though. > > Why not? This is the Py3k list -- there is no hope for binary > compatibility with 2.x. AFAIU the C99 approach can be easily binary > compatible between 3.0, 3.1 and beyond -- please explain if this is > not so. how can it be? isn't the C99 syntax just a way to avoid filling all the zeros? you'll still end up with a fixed-size structure in memory, with no way to move things around, and nowhere to put any auxiliary data. > And what's the optimization issue? Do you mean the ccasional NULL > poitner check? Once we make PyType_Ready() obligatory, it can do the > same thing. Or whatever else you are thinking of. null pointer checks, using different structures depending on object types, different ways to represent method lists (linear lists for short method lists, custom dictionaries for longer lists), and what else we might come with during our next trip to Reykjavik. "naked structures" are a lot more limiting than data structures hidden behind simple API:s. From talin at acm.org Tue Nov 28 18:58:06 2006 From: talin at acm.org (Talin) Date: Tue, 28 Nov 2006 09:58:06 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: References: <456BFFBF.9000208@acm.org> Message-ID: <456C78AE.1080601@acm.org> Fredrik Lundh wrote: > Guido van Rossum wrote: > >> - Fredrik's solution makes one call per registered method. (I don't >> know if the patch he refers to follows that model.) That seems a fair >> amount of code for an average type -- I'm wondering if it's too early >> to worry about code bloat (I don't think the speed is going to >> matter). > > too early, I think. I agree with this, especially given that I think that this should be *immediately* backported to 2.6, so that developers can gradually transition over; So that by the time Py3K comes out, there will be fewer existing libraries to convert. > and memory is relatively cheap, compare to the costs of upgrade pain, > programmer time, and lack of optimization opportunities due to "bare > data structures". > >> - Both solutions proposed require rewriting *all* type initialization. >> This is likely to require a tool that can do 99% of the work >> automatically (or else extension writers will truly hate us). > > yup. I think a tool that generates cut-that-and-paste-this instructions > for the developer should be good enough, though, and fairly easy to > write, for the reasons you give. > >> Can't we require a C99 compiler and use C99 struct initialization? > > that won't address the binary compatibility and optimization issues that > are the main rationales for my proposal, though. > > > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/talin%40acm.org > From janssen at parc.com Tue Nov 28 19:03:24 2006 From: janssen at parc.com (Bill Janssen) Date: Tue, 28 Nov 2006 10:03:24 PST Subject: [Python-3000] Generic functions vs. OO In-Reply-To: References: Message-ID: <06Nov28.100326pst."58648"@synergy1.parc.xerox.com> Guido van Rossum wrote: > I wonder if a bunch of well thought-out standard ABCs, applied to the > standard data types, and perhaps more support for setting __bases__, > wouldn't address most concerns. I think it would, but I don't know if we're going to get anywhere without getting more concrete. Let's tap "the wisdom of crowds". I've set up a Wiki page at http://wiki.python.org/moin/AbstractBaseClasses, and spent a little time defining some ABCs and concrete classes. I'll work on it more over the week, but feel free to pitch in and expand it. Or create alternative versions (create a new Wiki page, and put a link to it at the top of the existing page). Bill From guido at python.org Tue Nov 28 19:12:51 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 28 Nov 2006 10:12:51 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: References: <456BFFBF.9000208@acm.org> Message-ID: On 11/28/06, Fredrik Lundh wrote: > Guido van Rossum wrote: > > >>> Can't we require a C99 compiler and use C99 struct initialization? > >> that won't address the binary compatibility and optimization issues that > >> are the main rationales for my proposal, though. > > > > Why not? This is the Py3k list -- there is no hope for binary > > compatibility with 2.x. AFAIU the C99 approach can be easily binary > > compatible between 3.0, 3.1 and beyond -- please explain if this is > > not so. > > how can it be? isn't the C99 syntax just a way to avoid filling all the > zeros? you'll still end up with a fixed-size structure in memory, with > no way to move things around, and nowhere to put any auxiliary data. We can use the same approach to extending the structure in a binary compatible way as today. You don't need to move things around since the initialization order doesn't need to match the order of the members in the struct, so strictly adding to the end isn't a problem. > > And what's the optimization issue? Do you mean the ccasional NULL > > poitner check? Once we make PyType_Ready() obligatory, it can do the > > same thing. Or whatever else you are thinking of. > > null pointer checks, using different structures depending on object > types, different ways to represent method lists (linear lists for short > method lists, custom dictionaries for longer lists), and what else we > might come with during our next trip to Reykjavik. That could all be handled by PyType_Ready(), once we allow for some moderate extensibility in the 3.0 base struct. > "naked structures" are a lot more limiting than data structures hidden > behind simple API:s. OTOH I like a data-driven approach for initialization better than a code-driven approach. Perhaps Talin's way is a reasonable compromise? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Nov 28 19:14:43 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 28 Nov 2006 10:14:43 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456C78AE.1080601@acm.org> References: <456BFFBF.9000208@acm.org> <456C78AE.1080601@acm.org> Message-ID: On 11/28/06, Talin wrote: > Fredrik Lundh wrote: > > Guido van Rossum wrote: > > > >> - Fredrik's solution makes one call per registered method. (I don't > >> know if the patch he refers to follows that model.) That seems a fair > >> amount of code for an average type -- I'm wondering if it's too early > >> to worry about code bloat (I don't think the speed is going to > >> matter). > > > > too early, I think. > > I agree with this, Sure, forget I mentioned the efficiency thing. > especially given that I think that this should be > *immediately* backported to 2.6, so that developers can gradually > transition over; So that by the time Py3K comes out, there will be fewer > existing libraries to convert. Not *immediately* -- only after a lot of people agree and a working patch and conversion strategy has been defined. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From talin at acm.org Tue Nov 28 19:21:47 2006 From: talin at acm.org (Talin) Date: Tue, 28 Nov 2006 10:21:47 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: References: <456BFFBF.9000208@acm.org> Message-ID: <456C7E3B.7090601@acm.org> Guido van Rossum wrote: > Some comments: > > - Talin's solution seems to require the definition of an awful lot of > new constants -- one per slot. And a lot of special-casing in the type > initialization code to handle them because there are so many different > signatures. Actually, I was thinking more on this, and I have a couple ideas on how to avoid this, at least the part about the number of constants. The first idea is that instead of having a constant per special method, simply use the regular name of the method, but have a method flag that indicates that the method is "special": static PyMethodDef Noddy_methods[] = { { "new", (PyCFunction)Noddy_new, METH_VARARGS | METH_SPECIAL, "..." }, { "__init__", (PyCFunction)Noddy_init, METH_VARARGS | METH_SPECIAL, "..." }, { NULL } /* Sentinel */ }; One drawback here is that not all of the fields in PyTypeObject have equivalent names; But there's no reason why we couldn't give them names, especially if the names were not valid Python identifiers. The other drawback is that there's a greater chance of a misspelling, but I don't see that as a serious problem - even the most cursory testing of the class should discover this, it's not like the resulting bugs will be subtle and hard to find, IMHO. You could go even further, and drop the "special" flag entirely, and there's a compelling reason why you might want to do this: It means that now the VM gets to decide what methods are special and what methods aren't. So if, for example, we decide in the future that '__unicode__' needs to be sped up by putting it directly in the PyTypeObject, we can add a slot for it, and have all existing code magically work. It does mean that the interpretation of the method table is a bit more complex, but as long as you have a fast way of looking up the method names, and determining whether to put each method into the class dict or to fill in a PyTypeObject slot, I don't think it would be too bad. This is what Fredrik is saying about the advantage over C99 initialization, which only allows us to change the order of initializers, not their names or meanings. Now, I tend to prefer using a static table vs. the function-call-per-method simply because of my habits, which tend to be overly parsimonious with code size and such (It's a side-effect of working on embedded systems, which is what game consoles effectively are.) I wouldn't have strong objections to doing it the other way. -- Talin From talin at acm.org Tue Nov 28 19:54:33 2006 From: talin at acm.org (Talin) Date: Tue, 28 Nov 2006 10:54:33 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: References: <456BFFBF.9000208@acm.org> Message-ID: <456C85E9.8000905@acm.org> Guido van Rossum wrote: > Some comments: > > - Fredrik's solution makes one call per registered method. (I don't > know if the patch he refers to follows that model.) That seems a fair > amount of code for an average type -- I'm wondering if it's too early > to worry about code bloat (I don't think the speed is going to > matter). One other thought: The special constants could themselves be nothing more than the offset into the PyTypeObject struct, i.e.: #define SPECMETHOD_NEW ((const char*)offsetof(PyTypeObject,tp_new)) In the PyType_Ready code, you would see if the method name had a value of less than sizeof(PyTypeObject); If so, then it's a special method name, and you fill in the struct at the specified offset. So the interpretation of the table could be very simple and fast. It has a slight disadvantage from the approach of using actual string names for special methods, in that it doesn't allow the VM to silently promote/demote methods to 'special' status. -- Talin From brett at python.org Tue Nov 28 21:19:59 2006 From: brett at python.org (Brett Cannon) Date: Tue, 28 Nov 2006 12:19:59 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456C7E3B.7090601@acm.org> References: <456BFFBF.9000208@acm.org> <456C7E3B.7090601@acm.org> Message-ID: On 11/28/06, Talin wrote: > > Guido van Rossum wrote: > > Some comments: > > > > - Talin's solution seems to require the definition of an awful lot of > > new constants -- one per slot. And a lot of special-casing in the type > > initialization code to handle them because there are so many different > > signatures. > > Actually, I was thinking more on this, and I have a couple ideas on how > to avoid this, at least the part about the number of constants. > > The first idea is that instead of having a constant per special method, > simply use the regular name of the method, but have a method flag that > indicates that the method is "special": > > static PyMethodDef Noddy_methods[] = { > { "new", (PyCFunction)Noddy_new, > METH_VARARGS | METH_SPECIAL, "..." }, > { "__init__", (PyCFunction)Noddy_init, > METH_VARARGS | METH_SPECIAL, "..." }, > { NULL } /* Sentinel */ > }; > > One drawback here is that not all of the fields in PyTypeObject have > equivalent names; But there's no reason why we couldn't give them names, > especially if the names were not valid Python identifiers. Coming up with equivalent names should definitely not be difficult. I honestly can't think of any that don't have a matching name off the top of my head short of tp_free/tp_dealloc. One issue with this, though, will be the shifting of list and dict interfaces. They both have a __getitem__ method, but one expects only integers and one expects anything. Are you not going to optimize for that and just force lists to handle the type check and cast for every call? The other drawback is that there's a greater chance of a misspelling, > but I don't see that as a serious problem - even the most cursory > testing of the class should discover this, it's not like the resulting > bugs will be subtle and hard to find, IMHO. > > You could go even further, and drop the "special" flag entirely, and > there's a compelling reason why you might want to do this: It means that > now the VM gets to decide what methods are special and what methods > aren't. So if, for example, we decide in the future that '__unicode__' > needs to be sped up by putting it directly in the PyTypeObject, we can > add a slot for it, and have all existing code magically work. That does help minimize backwards-compatibility issues. But it does make things implicit. It does mean that the interpretation of the method table is a bit more > complex, but as long as you have a fast way of looking up the method > names, and determining whether to put each method into the class dict or > to fill in a PyTypeObject slot, I don't think it would be too bad. > > This is what Fredrik is saying about the advantage over C99 > initialization, which only allows us to change the order of > initializers, not their names or meanings. Ah, that is what you guys are getting at ! That makes sense and is a compelling argument. I think with this cleanup and a standardization on calling conventions (caller or callee checks for NULL arguments, who handles conversion to specific types, error return values, specifying ref stealing, etc.) this should go a long way to make extension modules much easier to work with. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061128/a63e0793/attachment.htm From guido at python.org Tue Nov 28 22:09:03 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 28 Nov 2006 13:09:03 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456C85E9.8000905@acm.org> References: <456BFFBF.9000208@acm.org> <456C85E9.8000905@acm.org> Message-ID: On 11/28/06, Talin wrote: > Guido van Rossum wrote: > > Some comments: > > > > - Fredrik's solution makes one call per registered method. (I don't > > know if the patch he refers to follows that model.) That seems a fair > > amount of code for an average type -- I'm wondering if it's too early > > to worry about code bloat (I don't think the speed is going to > > matter). > > One other thought: The special constants could themselves be nothing > more than the offset into the PyTypeObject struct, i.e.: > > #define SPECMETHOD_NEW ((const char*)offsetof(PyTypeObject,tp_new)) I think this would cause too many issues with backwards compatibility. I like the idea much better to use special names (e.g. starting with a "."). > In the PyType_Ready code, you would see if the method name had a value > of less than sizeof(PyTypeObject); If so, then it's a special method > name, and you fill in the struct at the specified offset. > > So the interpretation of the table could be very simple and fast. It has > a slight disadvantage from the approach of using actual string names for > special methods, in that it doesn't allow the VM to silently > promote/demote methods to 'special' status. I think the interpretation will be fast enough (or else what you said about premature optimization earlier wouldn't be correct. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Wed Nov 29 01:04:10 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 29 Nov 2006 13:04:10 +1300 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456C7E3B.7090601@acm.org> References: <456BFFBF.9000208@acm.org> <456C7E3B.7090601@acm.org> Message-ID: <456CCE7A.6020109@canterbury.ac.nz> Talin wrote: > The other drawback is that there's a greater chance of a misspelling, I don't think there is, really. It wouldn't be caught at compile time, but it would be caught very quickly at run time if you tried to initialise a type with a method flagged as "special" whose name wasn't one of the valid special method names. > You could go even further, and drop the "special" flag entirely, and > there's a compelling reason why you might want to do this: It means that > now the VM gets to decide what methods are special and what methods > aren't. No, that would be a bad idea. It would mean, for example, that *any* method called "next" would have to be assumed to potentially be the next() method of an iterator and forced to have the corresponding signature. (This is why I changed its name to __next__ in Pyrex. I suppose the same solution could be used here, if you were willing to accept the name that you put in the method table not necessarily being the name you use from Python.) -- Greg From greg.ewing at canterbury.ac.nz Wed Nov 29 01:10:41 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 29 Nov 2006 13:10:41 +1300 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456C7E3B.7090601@acm.org> References: <456BFFBF.9000208@acm.org> <456C7E3B.7090601@acm.org> Message-ID: <456CD001.4000907@canterbury.ac.nz> Talin wrote: > It means that > now the VM gets to decide what methods are special and what methods > aren't. Further to that, the VM still gets to decide whether any given special method gets a type slot. But the SPECIAL flag is needed to say that you intend the method to be *used* as a special method, which is something you can't tell just from the name (as long as there exist special methods like next() that don't use __xxx__ names). BTW, another advantage of all this is that it provides a lot more flexibility in the overall approach to implementing the type object. For example, we might decide to move the type slots into a separate memory block, so that the type struct could be extended by subclasses in the same way as other types. -- Greg From ronaldoussoren at mac.com Wed Nov 29 06:55:55 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 29 Nov 2006 06:55:55 +0100 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456CD001.4000907@canterbury.ac.nz> References: <456BFFBF.9000208@acm.org> <456C7E3B.7090601@acm.org> <456CD001.4000907@canterbury.ac.nz> Message-ID: On 29 Nov 2006, at 1:10 AM, Greg Ewing wrote: > > BTW, another advantage of all this is that it provides a lot > more flexibility in the overall approach to implementing > the type object. For example, we might decide to move the > type slots into a separate memory block, so that the > type struct could be extended by subclasses in the same > way as other types. You underestimate the amount of effort that was spend on the 2.x PyTypeObject implementation :-). It is already possibly to extend the type struct in Python 2.3 and later (at least of heap allocated types). One example use of this is PyObjCClassObject in PyObjC. This won't work in Python 2.2, the necessary machinery was added to the type implementation in 2.3. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3562 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20061129/c0638310/attachment.bin From talin at acm.org Wed Nov 29 08:23:38 2006 From: talin at acm.org (Talin) Date: Tue, 28 Nov 2006 23:23:38 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: References: <456BFFBF.9000208@acm.org> <456C85E9.8000905@acm.org> Message-ID: <456D357A.8030707@acm.org> Guido van Rossum wrote: > On 11/28/06, Talin wrote: >> Guido van Rossum wrote: >> > Some comments: >> > >> > - Fredrik's solution makes one call per registered method. (I don't >> > know if the patch he refers to follows that model.) That seems a fair >> > amount of code for an average type -- I'm wondering if it's too early >> > to worry about code bloat (I don't think the speed is going to >> > matter). >> >> One other thought: The special constants could themselves be nothing >> more than the offset into the PyTypeObject struct, i.e.: >> >> #define SPECMETHOD_NEW ((const char*)offsetof(PyTypeObject,tp_new)) > > I think this would cause too many issues with backwards compatibility. > > I like the idea much better to use special names (e.g. starting with a > "."). > >> In the PyType_Ready code, you would see if the method name had a value >> of less than sizeof(PyTypeObject); If so, then it's a special method >> name, and you fill in the struct at the specified offset. >> >> So the interpretation of the table could be very simple and fast. It has >> a slight disadvantage from the approach of using actual string names for >> special methods, in that it doesn't allow the VM to silently >> promote/demote methods to 'special' status. > > I think the interpretation will be fast enough (or else what you said > about premature optimization earlier wouldn't be correct. :-) OK, based on these comments and the other feedback from this thread, here's a more concrete proposal: == Method Table == Method definitions are stored in a static table, identical in format to the existing PyMethodDef table. For non-method initializers, the most commonly-used ones will be passed in as parameters to the type creation function. Those that are less commonly used can be written in as a secondary step after the type has been created, or in some cases represented in the tp_members table. == Method Names == As suggested by Guido, we use a naming convention to determine how a method in the method table is handled. I propose that methods be divided into three categories, which are "Normal", "Special", and "Internal" methods, and which are interpreted slightly differently at type initialization time. * Internal methods are those that have no equivalent Python name, such as tp_free/tp_alloc. Internal methods names start with a dot ("."), so tp_alloc would be represented by the string ".tp_alloc". Internal methods are always stored into a slot in the PyTypeObject. If there is no corresponding slot for a given name, that is a runtime error. * Special methods have the double-underscore (__special__) naming convention. A special method may or may not have a slot definition in PyTypeObject. If there is such a slot, the method pointer will be stored into it; If there is no such slot, then the method pointer is stored into the class dict just like a normal method. Because the decision whether to put the method into a slot is made by the VM, the set of available slots can be modified in future Python releases without breaking existing code. * Normal methods are any methods that are neither special or internal. They are not placed in a slot, but are simply stored in the class dict. Brett Cannon brought up the point about __getitem__ being ambiguous, since there are two slots, one for lists and one for mappings. This is handled as follows: The "mapping" version of __getitem__ is a special method, named "__getitem__". The "list" version, however, is considered an internal method (since it's more specialized), and has the name ".tp_getitem". Greg Ewing's point about "next" is handled as follows: A function named "next" will never be treated as a special method name, since it does not follow the naming convention of either internal or special names. However, if you want to fill in the "tp_next" slot of the PyTypeObject, you can use the string ".tp_next" rather than "next". == Type Creation == For backwards compatibility, the existing PyType_Ready function will continue to work on statically-declared PyTypeObject structures. A new function, 'PyType_Create' will be added that creates a new type from the input parameters and the method initialization tables as described previously. The actual type record may be allocated dynamically, as suggested by Greg Ewing. Structures such as tp_as_sequence which extend the PyTypeObject will be created as needed, if there are any methods that require those extension structures. == Backwards compatibility == The existing PyType_Ready and C-style static initialization mechanism will continue to work - the new method for type creation will coexist alongside the old. It is an open question as to whether PyType_Ready should attempt to interpret the special method names and fill in the PyTypeObject slots. If it does, then PyType_Create can all PyType_Ready as a subroutine during the type creation process. Otherwise, the only modifications to the interpreter will be the creation of the new PyType_Create function and any required subroutines. Existing code should be unaffected. -- Talin From guido at python.org Wed Nov 29 16:52:03 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 29 Nov 2006 07:52:03 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456D357A.8030707@acm.org> References: <456BFFBF.9000208@acm.org> <456C85E9.8000905@acm.org> <456D357A.8030707@acm.org> Message-ID: Have you thought much about the issue of different signature? The regular method table only has functions taking one or more objects and returning an object. (There are a few flags to indicate variations on the call args.) The special slots have all sorts of other signatures, some returning int, some returning void, some taking ints or C strings, etc. (And changing this would be a huge inefficiency in some cases.) Not using some variant of C (or C99) struct initialization means more chance for undetected errors since you will necessarily have to cast everything to a standard type and then the compiler can't type-check that the function signature matches the slot's needs. Nevertheless, I kind of like this, and would like someone to produce a patch for this approach. It is my opinion that without working code it is too easy to overlook design flaws. Since you are keen to have this in 2.6, the solution would have to be backwards compatible with 2.5, so the patch wouldn't have to change every type initializer in the system -- it would only have to provide the complete machinery and convert a few example types to prove it works. Any volunteers? Without code this proposal is dead in the water. --Guido On 11/28/06, Talin wrote: > Guido van Rossum wrote: > > On 11/28/06, Talin wrote: > >> Guido van Rossum wrote: > >> > Some comments: > >> > > >> > - Fredrik's solution makes one call per registered method. (I don't > >> > know if the patch he refers to follows that model.) That seems a fair > >> > amount of code for an average type -- I'm wondering if it's too early > >> > to worry about code bloat (I don't think the speed is going to > >> > matter). > >> > >> One other thought: The special constants could themselves be nothing > >> more than the offset into the PyTypeObject struct, i.e.: > >> > >> #define SPECMETHOD_NEW ((const char*)offsetof(PyTypeObject,tp_new)) > > > > I think this would cause too many issues with backwards compatibility. > > > > I like the idea much better to use special names (e.g. starting with a > > "."). > > > >> In the PyType_Ready code, you would see if the method name had a value > >> of less than sizeof(PyTypeObject); If so, then it's a special method > >> name, and you fill in the struct at the specified offset. > >> > >> So the interpretation of the table could be very simple and fast. It has > >> a slight disadvantage from the approach of using actual string names for > >> special methods, in that it doesn't allow the VM to silently > >> promote/demote methods to 'special' status. > > > > I think the interpretation will be fast enough (or else what you said > > about premature optimization earlier wouldn't be correct. :-) > > OK, based on these comments and the other feedback from this thread, > here's a more concrete proposal: > > == Method Table == > > Method definitions are stored in a static table, identical in format to > the existing PyMethodDef table. > > For non-method initializers, the most commonly-used ones will be passed > in as parameters to the type creation function. Those that are less > commonly used can be written in as a secondary step after the type has > been created, or in some cases represented in the tp_members table. > > == Method Names == > > As suggested by Guido, we use a naming convention to determine how a > method in the method table is handled. I propose that methods be divided > into three categories, which are "Normal", "Special", and "Internal" > methods, and which are interpreted slightly differently at type > initialization time. > > * Internal methods are those that have no equivalent Python name, such > as tp_free/tp_alloc. Internal methods names start with a dot ("."), so > tp_alloc would be represented by the string ".tp_alloc". > > Internal methods are always stored into a slot in the PyTypeObject. If > there is no corresponding slot for a given name, that is a runtime error. > > * Special methods have the double-underscore (__special__) naming > convention. A special method may or may not have a slot definition in > PyTypeObject. If there is such a slot, the method pointer will be stored > into it; If there is no such slot, then the method pointer is stored > into the class dict just like a normal method. > > Because the decision whether to put the method into a slot is made by > the VM, the set of available slots can be modified in future Python > releases without breaking existing code. > > * Normal methods are any methods that are neither special or internal. > They are not placed in a slot, but are simply stored in the class dict. > > Brett Cannon brought up the point about __getitem__ being ambiguous, > since there are two slots, one for lists and one for mappings. This is > handled as follows: > > The "mapping" version of __getitem__ is a special method, named > "__getitem__". The "list" version, however, is considered an internal > method (since it's more specialized), and has the name ".tp_getitem". > > Greg Ewing's point about "next" is handled as follows: A function named > "next" will never be treated as a special method name, since it does not > follow the naming convention of either internal or special names. > However, if you want to fill in the "tp_next" slot of the PyTypeObject, > you can use the string ".tp_next" rather than "next". > > == Type Creation == > > For backwards compatibility, the existing PyType_Ready function will > continue to work on statically-declared PyTypeObject structures. A new > function, 'PyType_Create' will be added that creates a new type from the > input parameters and the method initialization tables as described > previously. The actual type record may be allocated dynamically, as > suggested by Greg Ewing. > > Structures such as tp_as_sequence which extend the PyTypeObject will be > created as needed, if there are any methods that require those extension > structures. > > == Backwards compatibility == > > The existing PyType_Ready and C-style static initialization mechanism > will continue to work - the new method for type creation will coexist > alongside the old. > > It is an open question as to whether PyType_Ready should attempt to > interpret the special method names and fill in the PyTypeObject slots. > If it does, then PyType_Create can all PyType_Ready as a subroutine > during the type creation process. > > Otherwise, the only modifications to the interpreter will be the > creation of the new PyType_Create function and any required subroutines. > Existing code should be unaffected. > > -- Talin > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From janssen at parc.com Wed Nov 29 18:07:08 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 29 Nov 2006 09:07:08 PST Subject: [Python-3000] Generic functions vs. OO In-Reply-To: <06Nov28.100326pst."58648"@synergy1.parc.xerox.com> References: <06Nov28.100326pst."58648"@synergy1.parc.xerox.com> Message-ID: <06Nov29.090713pst."58648"@synergy1.parc.xerox.com> I've updated the page to answer a few questions Jim Jewett posed, and to expand a few more interfaces. I haven't tried to explain the implementation types, like Code or Frame. I'm hoping someone else will do that. I've broken up the File type into a set of types which attempt to capture (1) the difference between an open file-system file, and a file-like object, and (2) the difference between files opened in binary mode and files opened in text mode. Going through the type system again is interesting, and raises a few questions about some of the interfaces. For instance, should files opened in binary mode support iteration? If so, what's the unit of iteration? See http://wiki.python.org/moin/AbstractBaseClasses for more. Bill > Guido van Rossum wrote: > > I wonder if a bunch of well thought-out standard ABCs, applied to the > > standard data types, and perhaps more support for setting __bases__, > > wouldn't address most concerns. > > I think it would, but I don't know if we're going to get anywhere > without getting more concrete. Let's tap "the wisdom of crowds". I've > set up a Wiki page at http://wiki.python.org/moin/AbstractBaseClasses, > and spent a little time defining some ABCs and concrete classes. I'll > work on it more over the week, but feel free to pitch in and expand it. > Or create alternative versions (create a new Wiki page, and put a link > to it at the top of the existing page). > > Bill From tomerfiliba at gmail.com Wed Nov 29 18:41:19 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Wed, 29 Nov 2006 19:41:19 +0200 Subject: [Python-3000] iostack and Oh Oh In-Reply-To: References: <1d85506f0611240718s364f83c2ya3e9e12e2ca7354c@mail.gmail.com> <456776D6.8020909@canterbury.ac.nz> Message-ID: <1d85506f0611290941g1aac361ib23247058bbb8223@mail.gmail.com> Guido: > Agreed that for the distinction between readable/writable it's pretty > silly, and probably just encourages LBYL code no, the point was -- should we use separate StreamReader/StreamWriter classes, we wouldn't need this querying, the object will fail with AttributeError/TypeError if we attempt to perform an invalid operation. for example: f = file("...", "w") f.read() # IOError fr = FileWriter(file("...", "w")) fr.read() # AttributeError with all that stricter-interfaces talk, which is likely to enter py3k, the question we should ask is do we want the new IO stack to make use of those (or at least be designed with that in mind)? if we do want stricter interfaces in the IO stack too, then we should follow the java-style path (separate readers and writers, etc.). for example, do you think users of the IO stack would want multiple dispatch based on different streams? i'm well aware that GFs are not yet finalized material, but do we want to take them into account? or keep the plain old duck-typed io stack of today? -tomer From talin at acm.org Wed Nov 29 19:19:29 2006 From: talin at acm.org (Talin) Date: Wed, 29 Nov 2006 10:19:29 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: References: <456BFFBF.9000208@acm.org> <456C85E9.8000905@acm.org> <456D357A.8030707@acm.org> Message-ID: <456DCF31.50301@acm.org> Guido van Rossum wrote: > Have you thought much about the issue of different signature? The > regular method table only has functions taking one or more objects and > returning an object. (There are a few flags to indicate variations on > the call args.) The special slots have all sorts of other signatures, > some returning int, some returning void, some taking ints or C > strings, etc. (And changing this would be a huge inefficiency in some > cases.) > > Not using some variant of C (or C99) struct initialization means more > chance for undetected errors since you will necessarily have to cast > everything to a standard type and then the compiler can't type-check > that the function signature matches the slot's needs. Yes, there will be some loss of type safety. I'm not sure what to say about that. (Although, there is one benefit that you will at least be able to declare the "self" argument as its concrete type, rather than as PyObject *.) > Nevertheless, I kind of like this, and would like someone to produce a > patch for this approach. It is my opinion that without working code it > is too easy to overlook design flaws. Since you are keen to have this > in 2.6, the solution would have to be backwards compatible with 2.5, > so the patch wouldn't have to change every type initializer in the > system -- it would only have to provide the complete machinery and > convert a few example types to prove it works. > > Any volunteers? Without code this proposal is dead in the water. A patch won't be difficult to produce; It all depends on the availability of round 'tuits. Probably not this week at the very least. I'm curious as to what folks would recommend as the best way to do the mapping of slot names to slot addresses as a static table in the VM. I'm assuming that the result of the mapping would be a "slot descriptor" containing the offset of the pointer, and a code indicating whether it goes into the main type object or one of the extension structures. (Oh, and off topic, I have a procedural question about the list: Is it considered good etiquette to only reply to the list, and not the individuals whom you are replying to? If you "reply all", then folks might get two copies of the message, but if you reply to only the list, then non-subscribers posting via gmane might not get replies.) > --Guido > > On 11/28/06, Talin wrote: >> Guido van Rossum wrote: >> > On 11/28/06, Talin wrote: >> >> Guido van Rossum wrote: >> >> > Some comments: >> >> > >> >> > - Fredrik's solution makes one call per registered method. (I don't >> >> > know if the patch he refers to follows that model.) That seems a >> fair >> >> > amount of code for an average type -- I'm wondering if it's too >> early >> >> > to worry about code bloat (I don't think the speed is going to >> >> > matter). >> >> >> >> One other thought: The special constants could themselves be nothing >> >> more than the offset into the PyTypeObject struct, i.e.: >> >> >> >> #define SPECMETHOD_NEW ((const >> char*)offsetof(PyTypeObject,tp_new)) >> > >> > I think this would cause too many issues with backwards compatibility. >> > >> > I like the idea much better to use special names (e.g. starting with a >> > "."). >> > >> >> In the PyType_Ready code, you would see if the method name had a value >> >> of less than sizeof(PyTypeObject); If so, then it's a special method >> >> name, and you fill in the struct at the specified offset. >> >> >> >> So the interpretation of the table could be very simple and fast. >> It has >> >> a slight disadvantage from the approach of using actual string >> names for >> >> special methods, in that it doesn't allow the VM to silently >> >> promote/demote methods to 'special' status. >> > >> > I think the interpretation will be fast enough (or else what you said >> > about premature optimization earlier wouldn't be correct. :-) >> >> OK, based on these comments and the other feedback from this thread, >> here's a more concrete proposal: >> >> == Method Table == >> >> Method definitions are stored in a static table, identical in format to >> the existing PyMethodDef table. >> >> For non-method initializers, the most commonly-used ones will be passed >> in as parameters to the type creation function. Those that are less >> commonly used can be written in as a secondary step after the type has >> been created, or in some cases represented in the tp_members table. >> >> == Method Names == >> >> As suggested by Guido, we use a naming convention to determine how a >> method in the method table is handled. I propose that methods be divided >> into three categories, which are "Normal", "Special", and "Internal" >> methods, and which are interpreted slightly differently at type >> initialization time. >> >> * Internal methods are those that have no equivalent Python name, such >> as tp_free/tp_alloc. Internal methods names start with a dot ("."), so >> tp_alloc would be represented by the string ".tp_alloc". >> >> Internal methods are always stored into a slot in the PyTypeObject. If >> there is no corresponding slot for a given name, that is a runtime error. >> >> * Special methods have the double-underscore (__special__) naming >> convention. A special method may or may not have a slot definition in >> PyTypeObject. If there is such a slot, the method pointer will be stored >> into it; If there is no such slot, then the method pointer is stored >> into the class dict just like a normal method. >> >> Because the decision whether to put the method into a slot is made by >> the VM, the set of available slots can be modified in future Python >> releases without breaking existing code. >> >> * Normal methods are any methods that are neither special or internal. >> They are not placed in a slot, but are simply stored in the class dict. >> >> Brett Cannon brought up the point about __getitem__ being ambiguous, >> since there are two slots, one for lists and one for mappings. This is >> handled as follows: >> >> The "mapping" version of __getitem__ is a special method, named >> "__getitem__". The "list" version, however, is considered an internal >> method (since it's more specialized), and has the name ".tp_getitem". >> >> Greg Ewing's point about "next" is handled as follows: A function named >> "next" will never be treated as a special method name, since it does not >> follow the naming convention of either internal or special names. >> However, if you want to fill in the "tp_next" slot of the PyTypeObject, >> you can use the string ".tp_next" rather than "next". >> >> == Type Creation == >> >> For backwards compatibility, the existing PyType_Ready function will >> continue to work on statically-declared PyTypeObject structures. A new >> function, 'PyType_Create' will be added that creates a new type from the >> input parameters and the method initialization tables as described >> previously. The actual type record may be allocated dynamically, as >> suggested by Greg Ewing. >> >> Structures such as tp_as_sequence which extend the PyTypeObject will be >> created as needed, if there are any methods that require those extension >> structures. >> >> == Backwards compatibility == >> >> The existing PyType_Ready and C-style static initialization mechanism >> will continue to work - the new method for type creation will coexist >> alongside the old. >> >> It is an open question as to whether PyType_Ready should attempt to >> interpret the special method names and fill in the PyTypeObject slots. >> If it does, then PyType_Create can all PyType_Ready as a subroutine >> during the type creation process. >> >> Otherwise, the only modifications to the interpreter will be the >> creation of the new PyType_Create function and any required subroutines. >> Existing code should be unaffected. >> >> -- Talin >> _______________________________________________ >> Python-3000 mailing list >> Python-3000 at python.org >> http://mail.python.org/mailman/listinfo/python-3000 >> Unsubscribe: >> http://mail.python.org/mailman/options/python-3000/guido%40python.org >> > > From guido at python.org Wed Nov 29 19:27:06 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 29 Nov 2006 10:27:06 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456DCF31.50301@acm.org> References: <456BFFBF.9000208@acm.org> <456C85E9.8000905@acm.org> <456D357A.8030707@acm.org> <456DCF31.50301@acm.org> Message-ID: On 11/29/06, Talin wrote: > (Oh, and off topic, I have a procedural question about the list: Is it > considered good etiquette to only reply to the list, and not the > individuals whom you are replying to? If you "reply all", then folks > might get two copies of the message, but if you reply to only the list, > then non-subscribers posting via gmane might not get replies.) By all means use reply-all which CC's the author (and others). This seems to be what most people do, so folks who aren't using gmail yet have presumably developed other strategies to cope. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From benji at benjiyork.com Wed Nov 29 19:44:56 2006 From: benji at benjiyork.com (Benji York) Date: Wed, 29 Nov 2006 13:44:56 -0500 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: References: <456BFFBF.9000208@acm.org> <456C85E9.8000905@acm.org> <456D357A.8030707@acm.org> <456DCF31.50301@acm.org> Message-ID: <456DD528.2090802@benjiyork.com> Guido van Rossum wrote: > By all means use reply-all which CC's the author (and others). This > seems to be what most people do, so folks who aren't using gmail yet > have presumably developed other strategies to cope. One such coping mechanism is to configure mailman to not send you copies of messages you were sent directly via the "Avoid duplicate copies of messages?" option. -- Benji York From brett at python.org Wed Nov 29 20:18:35 2006 From: brett at python.org (Brett Cannon) Date: Wed, 29 Nov 2006 11:18:35 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456DCF31.50301@acm.org> References: <456BFFBF.9000208@acm.org> <456C85E9.8000905@acm.org> <456D357A.8030707@acm.org> <456DCF31.50301@acm.org> Message-ID: On 11/29/06, Talin wrote: > > Guido van Rossum wrote: > > Have you thought much about the issue of different signature? The > > regular method table only has functions taking one or more objects and > > returning an object. (There are a few flags to indicate variations on > > the call args.) The special slots have all sorts of other signatures, > > some returning int, some returning void, some taking ints or C > > strings, etc. (And changing this would be a huge inefficiency in some > > cases.) > > > > Not using some variant of C (or C99) struct initialization means more > > chance for undetected errors since you will necessarily have to cast > > everything to a standard type and then the compiler can't type-check > > that the function signature matches the slot's needs. > > Yes, there will be some loss of type safety. I'm not sure what to say > about that. (Although, there is one benefit that you will at least be > able to declare the "self" argument as its concrete type, rather than as > PyObject *.) I must be missing something, why is this any different than what you can do now? You either define the function with the exact call signature as typed for the struct field, or you define how you want the arguments to be treated and you cast to the call signature that is expected for the struct field. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061129/b7064aad/attachment.htm From brett at python.org Wed Nov 29 20:35:17 2006 From: brett at python.org (Brett Cannon) Date: Wed, 29 Nov 2006 11:35:17 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456D357A.8030707@acm.org> References: <456BFFBF.9000208@acm.org> <456C85E9.8000905@acm.org> <456D357A.8030707@acm.org> Message-ID: On 11/28/06, Talin wrote: > > Guido van Rossum wrote: > > On 11/28/06, Talin wrote: > >> Guido van Rossum wrote: > >> > Some comments: > >> > > >> > - Fredrik's solution makes one call per registered method. (I don't > >> > know if the patch he refers to follows that model.) That seems a fair > >> > amount of code for an average type -- I'm wondering if it's too early > >> > to worry about code bloat (I don't think the speed is going to > >> > matter). > >> > >> One other thought: The special constants could themselves be nothing > >> more than the offset into the PyTypeObject struct, i.e.: > >> > >> #define SPECMETHOD_NEW ((const char*)offsetof(PyTypeObject,tp_new)) > > > > I think this would cause too many issues with backwards compatibility. > > > > I like the idea much better to use special names (e.g. starting with a > > "."). > > > >> In the PyType_Ready code, you would see if the method name had a value > >> of less than sizeof(PyTypeObject); If so, then it's a special method > >> name, and you fill in the struct at the specified offset. > >> > >> So the interpretation of the table could be very simple and fast. It > has > >> a slight disadvantage from the approach of using actual string names > for > >> special methods, in that it doesn't allow the VM to silently > >> promote/demote methods to 'special' status. > > > > I think the interpretation will be fast enough (or else what you said > > about premature optimization earlier wouldn't be correct. :-) > > OK, based on these comments and the other feedback from this thread, > here's a more concrete proposal: > > == Method Table == > > Method definitions are stored in a static table, identical in format to > the existing PyMethodDef table. > > For non-method initializers, the most commonly-used ones will be passed > in as parameters to the type creation function. Those that are less > commonly used can be written in as a secondary step after the type has > been created, or in some cases represented in the tp_members table. > > == Method Names == > > As suggested by Guido, we use a naming convention to determine how a > method in the method table is handled. I propose that methods be divided > into three categories, which are "Normal", "Special", and "Internal" > methods, and which are interpreted slightly differently at type > initialization time. > > * Internal methods are those that have no equivalent Python name, such > as tp_free/tp_alloc. Internal methods names start with a dot ("."), so > tp_alloc would be represented by the string ".tp_alloc". Haven't we had various arguments about how it's bad to use a leading dot to have a special meaning? I understand why we need some way to flag internal methods on a type and I support going with an explicit way of specifying, but is a dot really the best solution? I mean something like INTERNAL_METH "tp_alloc" would even work using C's automatic string concatentation and doing:: #define INTERNAL_METH "." or whatever string we wanted that was not valid in a method name. I don't think this would lead us down the road of tons of macros and it makes things very visible. Internal methods are always stored into a slot in the PyTypeObject. If > there is no corresponding slot for a given name, that is a runtime error. > > * Special methods have the double-underscore (__special__) naming > convention. A special method may or may not have a slot definition in > PyTypeObject. If there is such a slot, the method pointer will be stored > into it; If there is no such slot, then the method pointer is stored > into the class dict just like a normal method. > > Because the decision whether to put the method into a slot is made by > the VM, the set of available slots can be modified in future Python > releases without breaking existing code. > > * Normal methods are any methods that are neither special or internal. > They are not placed in a slot, but are simply stored in the class dict. > > Brett Cannon brought up the point about __getitem__ being ambiguous, > since there are two slots, one for lists and one for mappings. This is > handled as follows: > > The "mapping" version of __getitem__ is a special method, named > "__getitem__". The "list" version, however, is considered an internal > method (since it's more specialized), and has the name ".tp_getitem". Or the other option is that in the future we just don't have the distinction and make sure that the __getitem__ methods do the requisite type checks. The type check is done at some point in the C code anyway so it isn't like there is a performance reason for the different slots. And as for providing a C-level function that provides a __getitem__ that takes Py_ssize_t, that can still be provided, it just isn't what is put into the struct. The one problem this does cause is testing for the interface support at the C level. But that could be a C function that looks for specific defined functions. Plus this would help make the C code less distinct from the way things expose themselves at the Python level (which I personally think is a good thing). Greg Ewing's point about "next" is handled as follows: A function named > "next" will never be treated as a special method name, since it does not > follow the naming convention of either internal or special names. > However, if you want to fill in the "tp_next" slot of the PyTypeObject, > you can use the string ".tp_next" rather than "next". > > == Type Creation == > > For backwards compatibility, the existing PyType_Ready function will > continue to work on statically-declared PyTypeObject structures. A new > function, 'PyType_Create' will be added that creates a new type from the > input parameters and the method initialization tables as described > previously. The actual type record may be allocated dynamically, as > suggested by Greg Ewing. > > Structures such as tp_as_sequence which extend the PyTypeObject will be > created as needed, if there are any methods that require those extension > structures. > > == Backwards compatibility == > > The existing PyType_Ready and C-style static initialization mechanism > will continue to work - the new method for type creation will coexist > alongside the old. > > It is an open question as to whether PyType_Ready should attempt to > interpret the special method names and fill in the PyTypeObject slots. > If it does, then PyType_Create can all PyType_Ready as a subroutine > during the type creation process. > > Otherwise, the only modifications to the interpreter will be the > creation of the new PyType_Create function and any required subroutines. > Existing code should be unaffected. Overall sounds good to me! -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20061129/7e47b095/attachment.html From fredrik at pythonware.com Wed Nov 29 21:32:17 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 29 Nov 2006 21:32:17 +0100 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456C7E3B.7090601@acm.org> References: <456BFFBF.9000208@acm.org> <456C7E3B.7090601@acm.org> Message-ID: Talin wrote: > Now, I tend to prefer using a static table vs. the > function-call-per-method simply because of my habits, which tend to be > overly parsimonious with code size and such (It's a side-effect of > working on embedded systems, which is what game consoles effectively > are.) I wouldn't have strong objections to doing it the other way. Umm. AFAICT, a call with one variable and two constant arguments typically needs 16-20 bytes on x86; a four-item struct with a pointer to a string literal needs 16 bytes plus the string literal, which means that you're restricted to 3-character names if you don't want to waste memory... ...and the decoder will of course be a lot larger, uglier, and slower if it has to test for a whole bunch of strings constants instead just switching on an integer value. Bloat alert. From guido at python.org Wed Nov 29 22:45:05 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 29 Nov 2006 13:45:05 -0800 Subject: [Python-3000] iostack and Oh Oh In-Reply-To: <1d85506f0611290941g1aac361ib23247058bbb8223@mail.gmail.com> References: <1d85506f0611240718s364f83c2ya3e9e12e2ca7354c@mail.gmail.com> <456776D6.8020909@canterbury.ac.nz> <1d85506f0611290941g1aac361ib23247058bbb8223@mail.gmail.com> Message-ID: I would not count on GFs, but I would suggest to count on ABCs and a better hierarchy. E.g. look at what Bill Janssen came up with (clearly incomplete): http://wiki.python.org/moin/AbstractBaseClasses I think that seekability should be a dynamically determined property though. On 11/29/06, tomer filiba wrote: > Guido: > > Agreed that for the distinction between readable/writable it's pretty > > silly, and probably just encourages LBYL code > > no, the point was -- should we use separate StreamReader/StreamWriter > classes, we wouldn't need this querying, the object will fail with > AttributeError/TypeError if we attempt to perform an invalid operation. > > for example: > f = file("...", "w") > f.read() # IOError > > fr = FileWriter(file("...", "w")) > fr.read() # AttributeError > > with all that stricter-interfaces talk, which is likely to enter py3k, > the question we should ask is do we want the new IO stack to make > use of those (or at least be designed with that in mind)? > > if we do want stricter interfaces in the IO stack too, then we > should follow the java-style path (separate readers and writers, > etc.). > > for example, do you think users of the IO stack would want multiple > dispatch based on different streams? i'm well aware that GFs > are not yet finalized material, but do we want to take them into > account? or keep the plain old duck-typed io stack of today? > > > -tomer > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From janssen at parc.com Thu Nov 30 00:45:35 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 29 Nov 2006 15:45:35 PST Subject: [Python-3000] iostack and Oh Oh In-Reply-To: References: <1d85506f0611240718s364f83c2ya3e9e12e2ca7354c@mail.gmail.com> <456776D6.8020909@canterbury.ac.nz> <1d85506f0611290941g1aac361ib23247058bbb8223@mail.gmail.com> Message-ID: <06Nov29.154543pst."58648"@synergy1.parc.xerox.com> Though I'm not happy with the current factoring of the "file" type at http://wiki.python.org/moin/AbstractBaseClasses. I'm thinking this afternoon that there should be an InputStream type and an OutputStream type, and that the particular mix of interfaces you get back from "open" should depend on what kind of file it is (tty or disk, etc.) and the mode that you used to open it. The actual file classes might be hidden; you would look at which interfaces you got to see what you could do with it. Of course, you should get back the right mix for the specified mode and file kind! Incidentally, what kind of iteration should apply to files opened in "binary" mode (which I continue to believe should be the default for Python)? for line in open("foo.jpg", "rb"): ... just seems wrong to me. Should be an iteration over bytes of the file, or over blocks. How about 80-byte "records" :-? Bill > I would not count on GFs, but I would suggest to count on ABCs and a > better hierarchy. > > E.g. look at what Bill Janssen came up with (clearly incomplete): > http://wiki.python.org/moin/AbstractBaseClasses > > I think that seekability should be a dynamically determined property though. > > On 11/29/06, tomer filiba wrote: > > Guido: > > > Agreed that for the distinction between readable/writable it's pretty > > > silly, and probably just encourages LBYL code > > > > no, the point was -- should we use separate StreamReader/StreamWriter > > classes, we wouldn't need this querying, the object will fail with > > AttributeError/TypeError if we attempt to perform an invalid operation. > > > > for example: > > f = file("...", "w") > > f.read() # IOError > > > > fr = FileWriter(file("...", "w")) > > fr.read() # AttributeError > > > > with all that stricter-interfaces talk, which is likely to enter py3k, > > the question we should ask is do we want the new IO stack to make > > use of those (or at least be designed with that in mind)? > > > > if we do want stricter interfaces in the IO stack too, then we > > should follow the java-style path (separate readers and writers, > > etc.). > > > > for example, do you think users of the IO stack would want multiple > > dispatch based on different streams? i'm well aware that GFs > > are not yet finalized material, but do we want to take them into > > account? or keep the plain old duck-typed io stack of today? From guido at python.org Thu Nov 30 01:30:21 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 29 Nov 2006 16:30:21 -0800 Subject: [Python-3000] iostack and Oh Oh In-Reply-To: <3152438538160113168@unknownmsgid> References: <1d85506f0611240718s364f83c2ya3e9e12e2ca7354c@mail.gmail.com> <456776D6.8020909@canterbury.ac.nz> <1d85506f0611290941g1aac361ib23247058bbb8223@mail.gmail.com> <3152438538160113168@unknownmsgid> Message-ID: First choice: binary files should not be iterable at all. I've never felt the need for this feature. Second choice: binary files should iterate given the buffer size (explicit or default) passed to open(). For unbuffered files they should iterate over bytes. On 11/29/06, Bill Janssen wrote: > Though I'm not happy with the current factoring of the "file" type at > http://wiki.python.org/moin/AbstractBaseClasses. I'm thinking this > afternoon that there should be an InputStream type and an OutputStream > type, and that the particular mix of interfaces you get back from > "open" should depend on what kind of file it is (tty or disk, etc.) > and the mode that you used to open it. The actual file classes might > be hidden; you would look at which interfaces you got to see what you > could do with it. Of course, you should get back the right mix for > the specified mode and file kind! > > Incidentally, what kind of iteration should apply to files opened in > "binary" mode (which I continue to believe should be the default for > Python)? > > for line in open("foo.jpg", "rb"): > ... > > just seems wrong to me. Should be an iteration over bytes of the > file, or over blocks. How about 80-byte "records" :-? > > Bill > > > I would not count on GFs, but I would suggest to count on ABCs and a > > better hierarchy. > > > > E.g. look at what Bill Janssen came up with (clearly incomplete): > > http://wiki.python.org/moin/AbstractBaseClasses > > > > I think that seekability should be a dynamically determined property though. > > > > On 11/29/06, tomer filiba wrote: > > > Guido: > > > > Agreed that for the distinction between readable/writable it's pretty > > > > silly, and probably just encourages LBYL code > > > > > > no, the point was -- should we use separate StreamReader/StreamWriter > > > classes, we wouldn't need this querying, the object will fail with > > > AttributeError/TypeError if we attempt to perform an invalid operation. > > > > > > for example: > > > f = file("...", "w") > > > f.read() # IOError > > > > > > fr = FileWriter(file("...", "w")) > > > fr.read() # AttributeError > > > > > > with all that stricter-interfaces talk, which is likely to enter py3k, > > > the question we should ask is do we want the new IO stack to make > > > use of those (or at least be designed with that in mind)? > > > > > > if we do want stricter interfaces in the IO stack too, then we > > > should follow the java-style path (separate readers and writers, > > > etc.). > > > > > > for example, do you think users of the IO stack would want multiple > > > dispatch based on different streams? i'm well aware that GFs > > > are not yet finalized material, but do we want to take them into > > > account? or keep the plain old duck-typed io stack of today? > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Thu Nov 30 01:34:16 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 30 Nov 2006 13:34:16 +1300 Subject: [Python-3000] iostack and Oh Oh In-Reply-To: <06Nov29.154543pst.58648@synergy1.parc.xerox.com> References: <1d85506f0611240718s364f83c2ya3e9e12e2ca7354c@mail.gmail.com> <456776D6.8020909@canterbury.ac.nz> <1d85506f0611290941g1aac361ib23247058bbb8223@mail.gmail.com> <06Nov29.154543pst.58648@synergy1.parc.xerox.com> Message-ID: <456E2708.6070205@canterbury.ac.nz> Bill Janssen wrote: > Incidentally, what kind of iteration should apply to files opened in > "binary" mode I don't think binary files should directly support iteration at all. Wrap it in an object that iterates the way you want. for b in readbytes(f): ... for blk in readblocks(f, 1024): ... I'm inclined to think that text files shouldn't be directly iterable either, and you should use for line in readlines(f): ... I can appreciate that some people might consider practicality to beat purity here, though. > How about 80-byte "records" :-? No problem: for card in readblocks(f, 80): fortran_compiler.eat(card) :-) -- Greg From greg.ewing at canterbury.ac.nz Thu Nov 30 01:34:37 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 30 Nov 2006 13:34:37 +1300 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456D357A.8030707@acm.org> References: <456BFFBF.9000208@acm.org> <456C85E9.8000905@acm.org> <456D357A.8030707@acm.org> Message-ID: <456E271D.4070701@canterbury.ac.nz> Talin wrote: > The "list" version, however, is considered an internal > method (since it's more specialized), and has the name ".tp_getitem". Is there really a need for both the dot and the tp_ prefix? Just ".getitem" etc. ought to be sufficient. -- Greg From greg.ewing at canterbury.ac.nz Thu Nov 30 01:34:44 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 30 Nov 2006 13:34:44 +1300 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: References: <456BFFBF.9000208@acm.org> <456C7E3B.7090601@acm.org> <456CD001.4000907@canterbury.ac.nz> Message-ID: <456E2724.6090702@canterbury.ac.nz> Ronald Oussoren wrote: > It is already possibly to extend the > type struct in Python 2.3 and later From Python? Python 2.3 (#1, Aug 5 2003, 15:52:30) [GCC 3.1 20020420 (prerelease)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> class C(type): ... __slots__ = ['foo'] ... Traceback (most recent call last): File "", line 1, in ? TypeError: nonempty __slots__ not supported for subtype of 'type' That looks to me like the type object is not a full member of the Python community just yet. -- Greg From greg.ewing at canterbury.ac.nz Thu Nov 30 01:38:53 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 30 Nov 2006 13:38:53 +1300 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: References: <456BFFBF.9000208@acm.org> <456C85E9.8000905@acm.org> <456D357A.8030707@acm.org> Message-ID: <456E281D.3090606@canterbury.ac.nz> Brett Cannon wrote: > Or the other option is that in the future we just don't have the > distinction and make sure that the __getitem__ methods do the requisite > type checks. The type check is done at some point in the C code anyway > so it isn't like there is a performance reason for the different slots. The performance reason is that C code can call the sq_getitem slot with a C integer directly, without having to convert to a Python integer and back again. Has anyone ever tested whether this is an important enough gain to be worth keeping the dual slots? -- Greg From greg.ewing at canterbury.ac.nz Thu Nov 30 01:47:44 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 30 Nov 2006 13:47:44 +1300 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456DD528.2090802@benjiyork.com> References: <456BFFBF.9000208@acm.org> <456C85E9.8000905@acm.org> <456D357A.8030707@acm.org> <456DCF31.50301@acm.org> <456DD528.2090802@benjiyork.com> Message-ID: <456E2A30.5080003@canterbury.ac.nz> Benji York wrote: > One such coping mechanism is to configure mailman to not send you copies > of messages you were sent directly via the "Avoid duplicate copies of > messages?" option. That's what I do, and it seems to work well enough. When replying, sometimes I'll trim the address list down to just the group, or just the group plus the person I'm immediately replying to, depending on whether I'm making comments to a particular participant or about the discussion generally. Sometimes I don't bother. I'm not all that worried about what happens for people not following the list. If they want to participate in a discussion taking place on the list, reading the list doesn't seem like an unreasonable requirement. Hopefully one day the writers of popular email software will do the screamingly obvious thing and add a "Reply to List" button... -- Greg From rasky at develer.com Thu Nov 30 01:51:56 2006 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 30 Nov 2006 01:51:56 +0100 Subject: [Python-3000] Low-hanging fruit: change interpreter prompt? Message-ID: Hello, it's a minor issue, but I was wondering if the interpreter prompt could be changed in Py3K. The current prompt is ">>>" which happens to match the standard character used for quotes in e-mails. As a result, intepreter sessions might look funky when copy & pasted inside mails. Given that showing interactive sessions in e-mails is *very* common and encouraged, it might be worthy to change the prompt to avoid this conflict. Something like "-->" or "---" or "%%%" or really *anything* else but sequences of ">" and "|" would do. Giovanni Bajo From guido at python.org Thu Nov 30 02:45:16 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 29 Nov 2006 17:45:16 -0800 Subject: [Python-3000] Low-hanging fruit: change interpreter prompt? In-Reply-To: References: Message-ID: But the >>> prompt is Python's trademark! I always get a warm fuzzy feeling when I see it, e.g. in a corner of a slide in a NASA presentation. Just set sys.ps1 = "-->" if you want something different. You can even set PYTHONSTARTUP= in your environment so you won't hve to think about it. --Guido On 11/29/06, Giovanni Bajo wrote: > Hello, > > it's a minor issue, but I was wondering if the interpreter prompt could be > changed in Py3K. The current prompt is ">>>" which happens to match the > standard character used for quotes in e-mails. As a result, intepreter > sessions might look funky when copy & pasted inside mails. > > Given that showing interactive sessions in e-mails is *very* common and > encouraged, it might be worthy to change the prompt to avoid this conflict. > Something like "-->" or "---" or "%%%" or really *anything* else but sequences > of ">" and "|" would do. > > Giovanni Bajo > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Nov 30 02:51:58 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 29 Nov 2006 17:51:58 -0800 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456E281D.3090606@canterbury.ac.nz> References: <456BFFBF.9000208@acm.org> <456C85E9.8000905@acm.org> <456D357A.8030707@acm.org> <456E281D.3090606@canterbury.ac.nz> Message-ID: I don't know about this one. But I'm sure that the CVS history of typeobject.c (in the 2.2a through 2.3 era, probably) would show that serious hacks were necessary to make certain slots perform well enough. I remember doing quite a bit of performance tuning with some incredible results -- all to make it competitive with classic classes. On 11/29/06, Greg Ewing wrote: > Brett Cannon wrote: > > > Or the other option is that in the future we just don't have the > > distinction and make sure that the __getitem__ methods do the requisite > > type checks. The type check is done at some point in the C code anyway > > so it isn't like there is a performance reason for the different slots. > > The performance reason is that C code can call the > sq_getitem slot with a C integer directly, without > having to convert to a Python integer and back > again. > > Has anyone ever tested whether this is an important > enough gain to be worth keeping the dual slots? > > -- > Greg > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Thu Nov 30 02:56:56 2006 From: aahz at pythoncraft.com (Aahz) Date: Wed, 29 Nov 2006 17:56:56 -0800 Subject: [Python-3000] Low-hanging fruit: change interpreter prompt? In-Reply-To: References: Message-ID: <20061130015656.GB18780@panix.com> On Thu, Nov 30, 2006, Giovanni Bajo wrote: > > it's a minor issue, but I was wondering if the interpreter prompt could be > changed in Py3K. The current prompt is ">>>" which happens to match the > standard character used for quotes in e-mails. As a result, intepreter > sessions might look funky when copy & pasted inside mails. While I feel your pain, I loathe the idea of updating all my doctests... -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ Usenet is not a democracy. It is a weird cross between an anarchy and a dictatorship. From janssen at parc.com Thu Nov 30 03:38:18 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 29 Nov 2006 18:38:18 PST Subject: [Python-3000] iostack and Oh Oh In-Reply-To: <456E2708.6070205@canterbury.ac.nz> References: <1d85506f0611240718s364f83c2ya3e9e12e2ca7354c@mail.gmail.com> <456776D6.8020909@canterbury.ac.nz> <1d85506f0611290941g1aac361ib23247058bbb8223@mail.gmail.com> <06Nov29.154543pst.58648@synergy1.parc.xerox.com> <456E2708.6070205@canterbury.ac.nz> Message-ID: <06Nov29.183824pst."58648"@synergy1.parc.xerox.com> Greg Ewing wrote: > Bill Janssen wrote: > > > Incidentally, what kind of iteration should apply to files opened in > > "binary" mode > > I don't think binary files should directly support > iteration at all. Wrap it in an object that iterates > the way you want. > > for b in readbytes(f): > ... > > for blk in readblocks(f, 1024): > ... > > I'm inclined to think that text files shouldn't be > directly iterable either, and you should use > > for line in readlines(f): > ... I like both these ideas. So file.readlines() would return an iterable, then? Bill From janssen at parc.com Thu Nov 30 04:21:10 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 29 Nov 2006 19:21:10 PST Subject: [Python-3000] iostack and Oh Oh In-Reply-To: References: <1d85506f0611240718s364f83c2ya3e9e12e2ca7354c@mail.gmail.com> <456776D6.8020909@canterbury.ac.nz> <1d85506f0611290941g1aac361ib23247058bbb8223@mail.gmail.com> Message-ID: <06Nov29.192119pst."58648"@synergy1.parc.xerox.com> I've re-factored the file interfaces to reflect the current discussion. I've also redone the MutableSequence and Mapping interfaces to use default parameters, which makes them much more comprehensible. See what you think. > http://wiki.python.org/moin/AbstractBaseClasses Bill From barry at python.org Thu Nov 30 05:39:02 2006 From: barry at python.org (Barry Warsaw) Date: Wed, 29 Nov 2006 23:39:02 -0500 Subject: [Python-3000] Low-hanging fruit: change interpreter prompt? In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Nov 29, 2006, at 7:51 PM, Giovanni Bajo wrote: > it's a minor issue, but I was wondering if the interpreter prompt > could be > changed in Py3K. The current prompt is ">>>" which happens to match > the > standard character used for quotes in e-mails. As a result, intepreter > sessions might look funky when copy & pasted inside mails. "Low hanging" as in peaches rotting on the ground? :) - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRW5gZnEjvBPtnXfVAQKDIgP/UgIVI94cW8cQuvcU/cbT3L1zTsbW8r0Q EPWsJkDSxmV/H+9WDCNhIN6RfwozPtOSln2zL6K8BIZzBuZJ8qM7GyxHG+6knGr8 l61X7IXBOKboQSnN9+Mhhx7TKb9wnm4ME08GtAX5BqwaJ8KjuVj/+sNkMcIQSTPP xdAhVFIylhY= =flic -----END PGP SIGNATURE----- From ronaldoussoren at mac.com Thu Nov 30 07:19:43 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 30 Nov 2006 07:19:43 +0100 Subject: [Python-3000] A better way to initialize PyTypeObject In-Reply-To: <456E2724.6090702@canterbury.ac.nz> References: <456BFFBF.9000208@acm.org> <456C7E3B.7090601@acm.org> <456CD001.4000907@canterbury.ac.nz> <456E2724.6090702@canterbury.ac.nz> Message-ID: <0106D5AC-C38D-4BF6-AD6D-73CE8C1258C0@mac.com> On 30 Nov 2006, at 1:34 AM, Greg Ewing wrote: > Ronald Oussoren wrote: >> It is already possibly to extend the type struct in Python 2.3 >> and later > > From Python? No, from C. > > Python 2.3 (#1, Aug 5 2003, 15:52:30) > [GCC 3.1 20020420 (prerelease)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> class C(type): > ... __slots__ = ['foo'] > ... > Traceback (most recent call last): > File "", line 1, in ? > TypeError: nonempty __slots__ not supported for subtype of 'type' It should be easy enough to add support for this in Python 2.6, all required machinery already seems to be present. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3562 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20061130/ba65482c/attachment.bin From rasky at develer.com Thu Nov 30 09:11:44 2006 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 30 Nov 2006 09:11:44 +0100 Subject: [Python-3000] Low-hanging fruit: change interpreter prompt? In-Reply-To: References: Message-ID: <456E9240.5010302@develer.com> Guido van Rossum wrote: > But the >>> prompt is Python's trademark! I always get a warm fuzzy > feeling when I see it, e.g. in a corner of a slide in a NASA > presentation. In fact, there would be another reason to change the prompt in Py3k: to identify Py3k snippets and tell them from Python 2.x snippets. When people will really start using Py3k, and will start sending mails with interpreter sessions in them, or publish blog entries or whatnot, I think it would be useful if you could immediately, no-brainy tell whether it's Python 2.x or Py3k. > Just set sys.ps1 = "-->" if you want something different. You can even > set PYTHONSTARTUP= in your environment so > you won't hve to think about it. The problem is not *my* environment, but when I read mails posted by others. -- Giovanni Bajo Develer S.r.l. http://www.develer.com From greg.ewing at canterbury.ac.nz Thu Nov 30 10:13:02 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 30 Nov 2006 22:13:02 +1300 Subject: [Python-3000] Low-hanging fruit: change interpreter prompt? In-Reply-To: References: Message-ID: <456EA09E.8030908@canterbury.ac.nz> Guido van Rossum wrote: > But the >>> prompt is Python's trademark! Maybe you could change it to something like _/\_/\_P (That's meant to be an ascii-art python.) -- Greg From rasky at develer.com Thu Nov 30 10:16:18 2006 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 30 Nov 2006 10:16:18 +0100 Subject: [Python-3000] Low-hanging fruit: change interpreter prompt? In-Reply-To: <20061130015656.GB18780@panix.com> References: <20061130015656.GB18780@panix.com> Message-ID: Aahz wrote: >> it's a minor issue, but I was wondering if the interpreter prompt could be >> changed in Py3K. The current prompt is ">>>" which happens to match the >> standard character used for quotes in e-mails. As a result, intepreter >> sessions might look funky when copy & pasted inside mails. > > While I feel your pain, I loathe the idea of updating all my doctests... doctest could easily be updated in the Py3k so to support also the old prompt syntax to ease the transition (either by default, or with doctest.testmod(old_syntax=True)). Anyway, you could search & replace and... uhm... run the tests to see if you missed something :) -- Giovanni Bajo Develer S.r.l. http://www.develer.com From steven.bethard at gmail.com Thu Nov 30 15:46:26 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Thu, 30 Nov 2006 07:46:26 -0700 Subject: [Python-3000] Low-hanging fruit: change interpreter prompt? In-Reply-To: References: Message-ID: On 11/29/06, Guido van Rossum wrote: > But the >>> prompt is Python's trademark! I always get a warm fuzzy > feeling when I see it, e.g. in a corner of a slide in a NASA > presentation. I was using ``py>`` for a while. That still gave me a warm fuzzy feeling ;-) and seemed to work with most newsreaders. STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From guido at python.org Thu Nov 30 16:43:07 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 30 Nov 2006 07:43:07 -0800 Subject: [Python-3000] Low-hanging fruit: change interpreter prompt? In-Reply-To: References: Message-ID: OK, let's stop this. It's not going to change. The deficiencies of email shouldn't be dictating the language. If I hear more pleading I'll have to add it to PEP 3099. Please read http://yellow.bikeshed.com. --Guido On 11/30/06, Steven Bethard wrote: > On 11/29/06, Guido van Rossum wrote: > > But the >>> prompt is Python's trademark! I always get a warm fuzzy > > feeling when I see it, e.g. in a corner of a slide in a NASA > > presentation. > > I was using ``py>`` for a while. That still gave me a warm fuzzy > feeling ;-) and seemed to work with most newsreaders. > > STeVe > -- > I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a > tiny blip on the distant coast of sanity. > --- Bucky Katt, Get Fuzzy > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik at pythonware.com Thu Nov 30 17:37:05 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 30 Nov 2006 17:37:05 +0100 Subject: [Python-3000] Low-hanging fruit: change interpreter prompt? In-Reply-To: References: Message-ID: Guido van Rossum wrote: > Please read http://yellow.bikeshed.com. make that http://gold.bikeshed.com/ From ldlandis at gmail.com Thu Nov 30 18:28:36 2006 From: ldlandis at gmail.com (LD 'Gus' Landis) Date: Thu, 30 Nov 2006 11:28:36 -0600 Subject: [Python-3000] Low-hanging fruit: change interpreter prompt? In-Reply-To: References: Message-ID: NEITHER, both hard to read due to color intensity... http://white.bikeshed.com/ ;-O see >Fredrik Lundh wrote:http://gold.bikeshed.com/ >Guido van Rossum wrote: http://yellow.bikeshed.com/ From tomerfiliba at gmail.com Thu Nov 30 21:45:03 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Thu, 30 Nov 2006 22:45:03 +0200 Subject: [Python-3000] iostack and Oh Oh In-Reply-To: <8979205067466383560@unknownmsgid> References: <1d85506f0611240718s364f83c2ya3e9e12e2ca7354c@mail.gmail.com> <456776D6.8020909@canterbury.ac.nz> <1d85506f0611290941g1aac361ib23247058bbb8223@mail.gmail.com> <8979205067466383560@unknownmsgid> Message-ID: <1d85506f0611301245x357bcf86w24b6338dd8bde716@mail.gmail.com> > http://wiki.python.org/moin/AbstractBaseClasses NOOOOOOOO! so many baseclasses, who can remember all that?! now whenever i wish to implement, say, MyFunnyDict, i'll have to go over the manual, see all the standard interfaces (which may reach hundreds of classes), and think which interfaces i desire for my class to "adhere to". that's very bad programming. they have a name for that in CS: java. you have to manually declare things, before doing them. with these ABCs, instead of a simple "def __something__", you first have to declare everything you'll possibly want to do (either by explicitly inheriting from, or another syntax such as __implements__ = ...). and with all that reliance on the *type* of objects, how would proxies work? for example, in RPyC i use local proxy objects that refer to remote objects (in another process/machine). just like weakref.proxy, you get another object that behaves exactly the same as the real object, but your code needn't be aware of that: if the object's has all the attributes, it's a duck. now, the proxy object will have to "inherit" all of the interfaces the real object has, in order for dispatching/isinstance to work. and that's bad. for weakrefs that may be simple, but in RPyC we are talking about objects from different processes, which is not simple at all (types compare by object id, for once). to sum it up -- if it looks like java and limps like java, it's java! -tomer P.S, i took a look over my post, and it's seems rather angry. i'm not. sorry if it does, but i'm too tired to rewrite it. it's been a long day and i'm a moment from bed, so no offense in advance. From janssen at parc.com Thu Nov 30 22:47:25 2006 From: janssen at parc.com (Bill Janssen) Date: Thu, 30 Nov 2006 13:47:25 PST Subject: [Python-3000] iostack and Oh Oh In-Reply-To: <1d85506f0611301245x357bcf86w24b6338dd8bde716@mail.gmail.com> References: <1d85506f0611240718s364f83c2ya3e9e12e2ca7354c@mail.gmail.com> <456776D6.8020909@canterbury.ac.nz> <1d85506f0611290941g1aac361ib23247058bbb8223@mail.gmail.com> <8979205067466383560@unknownmsgid> <1d85506f0611301245x357bcf86w24b6338dd8bde716@mail.gmail.com> Message-ID: <06Nov30.134726pst."58648"@synergy1.parc.xerox.com> > so no offense in advance. Sure, no offense taken. I've seen comments like this before on this list (recently :-). I think both approaches (interface types and duck typing) are complicated in different ways. But interface types seem less so, because they provide a place where the user can stop thinking about what's behind a function or method, instead of having to understand the code "all the way down". And it doesn't preclude the use of "just implement the method" for code that you understand "all the way down" already, where you know that the only use of a particular value is to call the "foo" method on it. But it also ensures that when someone gets a Mapping, they can call "values" on it (to list one mis-example of partial implementation that I've tripped over in the past). Bill From greg.ewing at canterbury.ac.nz Thu Nov 30 23:42:23 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 01 Dec 2006 11:42:23 +1300 Subject: [Python-3000] iostack and Oh Oh In-Reply-To: <06Nov29.183824pst.58648@synergy1.parc.xerox.com> References: <1d85506f0611240718s364f83c2ya3e9e12e2ca7354c@mail.gmail.com> <456776D6.8020909@canterbury.ac.nz> <1d85506f0611290941g1aac361ib23247058bbb8223@mail.gmail.com> <06Nov29.154543pst.58648@synergy1.parc.xerox.com> <456E2708.6070205@canterbury.ac.nz> <06Nov29.183824pst.58648@synergy1.parc.xerox.com> Message-ID: <456F5E4F.6090705@canterbury.ac.nz> Bill Janssen wrote: > So file.readlines() would return an iterable, then? I wasn't thinking of having a file.readlines(), just a stand-alone function. I'd like to see the file object's interface made quite minimal in Py3k. -- Greg