From j.wielicki at sotecware.net Mon Dec 1 08:05:11 2014 From: j.wielicki at sotecware.net (Jonas Wielicki) Date: Mon, 01 Dec 2014 08:05:11 +0100 Subject: [Python-ideas] Fast context creation In-Reply-To: References: <20141127120934.GA2433@ferrara.linux.it> <20141127142413.GA5579@ferrara.linux.it> Message-ID: <547C1327.1070607@sotecware.net> Hi Clint, (this is a bit of nit-picking, but kind of supports the OPs concerns about intent) On 27.11.2014 17:01, Clint Hepner wrote: > On Thu, Nov 27, 2014 at 9:24 AM, Stefano Borini < > stefano.borini at ferrara.linux.it> wrote: > >> On Thu, Nov 27, 2014 at 11:30:40PM +1100, Chris Angelico wrote: >>> You could abuse try/finally for this purpose. >>> >>> try: >>> whatever you need to do >>> finally: >>> my_widget.setFocus() >> >> I thought about this solution, but I am concerned about communication of >> intent. Using the try >> communicates to a code reader that the code inside is expected to throw, >> which is not the case. >> >> If there is absolutely no possibility of an exception, you don't need any > special construct. Just use > > whatever you need to do > my_widget.setFocus() > > Otherwise, you are tacitly agreeing that an exception *might* occur, so > try/finally is not misleading. Not neccessarily. try/finally can also be used to do something in all return paths: def foo(): try: return True finally: print("foo") will print foo and return True. There is no way this can throw, nevertheless try/finally can be useful if you have to do some clean up and have several return paths. regards, jwi > > >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From elfring at users.sourceforge.net Mon Dec 1 13:13:41 2014 From: elfring at users.sourceforge.net (SF Markus Elfring) Date: Mon, 01 Dec 2014 13:13:41 +0100 Subject: [Python-ideas] Multi-index Containers Library Message-ID: <547C5B75.4000001@users.sourceforge.net> Hello, I find a data structure like it is provided by the "Boost Multi-index Containers Library" interesting for efficient data processing. http://www.boost.org/doc/libs/1_57_0/libs/multi_index/doc/index.html How are the chances to integrate a class library with similar functionality into the Python software infrastructure? https://bugs.python.org/issue22957 Regards, Markus From elfring at users.sourceforge.net Mon Dec 1 13:01:26 2014 From: elfring at users.sourceforge.net (SF Markus Elfring) Date: Mon, 01 Dec 2014 13:01:26 +0100 Subject: [Python-ideas] Improved support for prepared SQL statements Message-ID: <547C5896.3010808@users.sourceforge.net> Hello, An interface for parameterised SQL statements (working with placeholders) is provided by the execute() method from the Cursor class at the moment. https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.execute I assume that the "SQL Statement Object" from the SQLite C interface is reused there already. http://sqlite.org/c3ref/stmt.html I imagine that it will be more efficient occasionally to offer also a base class like "prepared_statement" so that the parameter specification does not need to be parsed for every passed command. I suggest to improve corresponding preparation and compilation possibilities. https://bugs.python.org/issue22956 Regards, Markus From mal at egenix.com Mon Dec 1 13:24:57 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 01 Dec 2014 13:24:57 +0100 Subject: [Python-ideas] Improved support for prepared SQL statements In-Reply-To: <547C5896.3010808@users.sourceforge.net> References: <547C5896.3010808@users.sourceforge.net> Message-ID: <547C5E19.5000902@egenix.com> On 01.12.2014 13:01, SF Markus Elfring wrote: > Hello, > > An interface for parameterised SQL statements (working with placeholders) is provided by the execute() method from the Cursor class at the moment. > https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.execute > > I assume that the "SQL Statement Object" from the SQLite C interface is reused there already. > http://sqlite.org/c3ref/stmt.html > > I imagine that it will be more efficient occasionally to offer also a base class like "prepared_statement" so that the parameter specification does not need to be parsed for every passed command. > I suggest to improve corresponding preparation and compilation possibilities. > https://bugs.python.org/issue22956 For database questions, please use the DB-SIG: https://mail.python.org/mailman/listinfo/db-sig FWIW: The DB-API which the sqlite3 module implements, supports implicit and explicit reuse of statements: https://www.python.org/dev/peps/pep-0249/ -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 01 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-11-19: Released mxODBC Connect 2.1.1 ... http://egenix.com/go65 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dw+python-ideas at hmmz.org Mon Dec 1 15:24:52 2014 From: dw+python-ideas at hmmz.org (David Wilson) Date: Mon, 1 Dec 2014 14:24:52 +0000 Subject: [Python-ideas] Multi-index Containers Library In-Reply-To: <547C5B75.4000001@users.sourceforge.net> References: <547C5B75.4000001@users.sourceforge.net> Message-ID: <20141201142452.GA11467@k3> On Mon, Dec 01, 2014 at 01:13:41PM +0100, SF Markus Elfring wrote: > Hello, > > I find a data structure like it is provided by the "Boost Multi-index Containers Library" interesting for efficient data processing. > http://www.boost.org/doc/libs/1_57_0/libs/multi_index/doc/index.html This came up on #python just last week, and I've personally encountered a desire for something similar a bunch of times. Probably, though, the ideal interface to something like this would need some time to evolve before it would be useful to include in the stdlib, along with some motivating use cases -- at least, it would be nice if there were one or two third party implementations to compare before settling on an interface. I can't think of any existing implementations that operate on purely in-memory structures. Boost::multiindex is pretty awesome, but its interface is also somewhat complicated, and a 1:1 translation wouldn't make for a pretty Python experience. I thought about this a little bit last week, initially starting with a "MultiSet" type interface, where the "default" interface for the container emulated a regular set object. But that would prevent having sets of things unless they were hashable (instead of, say, merely having properties that were orderable). In applications where I'd like to use a structure like this, often the element type is mutable. My second idea was for something like an "IndexedList" type, where the default interface mimicked the built-in list. This would allow generic use of the container with code that wasn't expecting it, and avoid the hashability issue from before. Perhaps something like: lst = IndexedList() lst.add_unique_index('id', lambda o: o['id']) lst.add_index('age', lambda o: o['age']) lst.append({'id': 1, 'age': 50}) lst.append({'id': 1, 'age': 50}) # KeyError lst.append({'id': 2, 'age': 51}) assert lst.indices['id'][1]['age'] == 50 slice = lst.indices['age'][40:50] # No idea what would be returned here, some kind of view perhaps. assert len(slice) == 2 Following on from the blist thread a few months back, this is another example of a container that would be easy to construct if Python had some kind of built-in ordered mapping type. David > > How are the chances to integrate a class library with similar functionality into the Python software infrastructure? > https://bugs.python.org/issue22957 > > Regards, > Markus > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From dw+python-ideas at hmmz.org Mon Dec 1 16:03:28 2014 From: dw+python-ideas at hmmz.org (David Wilson) Date: Mon, 1 Dec 2014 15:03:28 +0000 Subject: [Python-ideas] Multi-index Containers Library In-Reply-To: <20141201142452.GA11467@k3> References: <547C5B75.4000001@users.sourceforge.net> <20141201142452.GA11467@k3> Message-ID: <20141201150328.GA20114@k3> > before it would be useful to include in the stdlib, along with some > motivating use cases One example would be a continuous auction, like a stock exchange order book. In that case, efficient enumeration is desirable by all of account ID, order ID, or (price, time). book = IndexedList() book.add_unique_index('order_id', lambda o: o.id) book.add_index('account_id', lambda o: o.account_id) book.add_index('price_time', lambda o: (o.price, o.time)) def add_order(order): book.append(order) def del_order(order): del book.indices['order_id'][order.id] def close_account(account_id): del book.indices['account_id'][account_id] def iter_matches(limit_price): for (price, time), order in book.indices['price_time']: if price > limit_price: break yield order Again it's not very clear how the magical 'indices' object slices would work. David From elfring at users.sourceforge.net Mon Dec 1 16:12:12 2014 From: elfring at users.sourceforge.net (SF Markus Elfring) Date: Mon, 01 Dec 2014 16:12:12 +0100 Subject: [Python-ideas] Multi-index Containers Library In-Reply-To: <20141201142452.GA11467@k3> References: <547C5B75.4000001@users.sourceforge.net> <20141201142452.GA11467@k3> Message-ID: <547C854C.2000205@users.sourceforge.net> >> http://www.boost.org/doc/libs/1_57_0/libs/multi_index/doc/index.html > > This came up on #python just last week, and I've personally encountered > a desire for something similar a bunch of times. How challenging is it to map powerful Boost classes to the Python programming language? > Probably, though, the ideal interface to something like this would need > some time to evolve before it would be useful to include in the stdlib, I am curious on how this could happen ... > along with some motivating use cases Would you like to look at any more examples I could show? > -- at least, it would be nice if there were one or two third party > implementations to compare before settling on an interface. > I can't think of any existing implementations that operate on purely > in-memory structures. Thanks for your feedback. > Boost::multiindex is pretty awesome, but its interface is also > somewhat complicated, and a 1:1 translation wouldn't make for a pretty > Python experience. Will this story become another promising software development adventure? Regards, Markus From rosuav at gmail.com Mon Dec 1 16:13:02 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 2 Dec 2014 02:13:02 +1100 Subject: [Python-ideas] Multi-index Containers Library In-Reply-To: <20141201150328.GA20114@k3> References: <547C5B75.4000001@users.sourceforge.net> <20141201142452.GA11467@k3> <20141201150328.GA20114@k3> Message-ID: On Tue, Dec 2, 2014 at 2:03 AM, David Wilson wrote: >> before it would be useful to include in the stdlib, along with some >> motivating use cases > > One example would be a continuous auction, like a stock exchange order > book. In that case, efficient enumeration is desirable by all of > account ID, order ID, or (price, time). For small numbers of entries, it'd be simpler to just sort and filter on demand; for large numbers of entries, you should probably be using a database, which will have these sorts of facilities. Is there a mid-range where it's better to keep it all in memory, but it's too slow to sort on demand? ChrisA From wes.turner at gmail.com Mon Dec 1 16:39:05 2014 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 1 Dec 2014 09:39:05 -0600 Subject: [Python-ideas] Multi-index Containers Library In-Reply-To: <20141201142452.GA11467@k3> References: <547C5B75.4000001@users.sourceforge.net> <20141201142452.GA11467@k3> Message-ID: Is this what pandas MultiIndex does (in Cython)? * http://pandas.pydata.org/pandas-docs/dev/api.html#reshaping-sorting * http://toolz.readthedocs.org/en/latest/api.html On Mon, Dec 1, 2014 at 8:24 AM, David Wilson wrote: > On Mon, Dec 01, 2014 at 01:13:41PM +0100, SF Markus Elfring wrote: > > Hello, > > > > I find a data structure like it is provided by the "Boost Multi-index > Containers Library" interesting for efficient data processing. > > http://www.boost.org/doc/libs/1_57_0/libs/multi_index/doc/index.html > > This came up on #python just last week, and I've personally encountered > a desire for something similar a bunch of times. Probably, though, the > ideal interface to something like this would need some time to evolve > before it would be useful to include in the stdlib, along with some > motivating use cases -- at least, it would be nice if there were one or > two third party implementations to compare before settling on an > interface. I can't think of any existing implementations that operate on > purely in-memory structures. > > Boost::multiindex is pretty awesome, but its interface is also > somewhat complicated, and a 1:1 translation wouldn't make for a pretty > Python experience. > > I thought about this a little bit last week, initially starting with a > "MultiSet" type interface, where the "default" interface for the > container emulated a regular set object. But that would prevent having > sets of things unless they were hashable (instead of, say, merely having > properties that were orderable). In applications where I'd like to use a > structure like this, often the element type is mutable. > > My second idea was for something like an "IndexedList" type, where the > default interface mimicked the built-in list. This would allow generic > use of the container with code that wasn't expecting it, and avoid the > hashability issue from before. > > Perhaps something like: > > lst = IndexedList() > lst.add_unique_index('id', lambda o: o['id']) > lst.add_index('age', lambda o: o['age']) > > lst.append({'id': 1, 'age': 50}) > lst.append({'id': 1, 'age': 50}) # KeyError > > lst.append({'id': 2, 'age': 51}) > > assert lst.indices['id'][1]['age'] == 50 > > slice = lst.indices['age'][40:50] > # No idea what would be returned here, some kind of view perhaps. > assert len(slice) == 2 > > Following on from the blist thread a few months back, this is another > example of a container that would be easy to construct if Python had > some kind of built-in ordered mapping type. > > > David > > > > > > How are the chances to integrate a class library with similar > functionality into the Python software infrastructure? > > https://bugs.python.org/issue22957 > > > > Regards, > > Markus > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Dec 1 16:40:20 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 1 Dec 2014 07:40:20 -0800 Subject: [Python-ideas] Multi-index Containers Library In-Reply-To: <547C5B75.4000001@users.sourceforge.net> References: <547C5B75.4000001@users.sourceforge.net> Message-ID: I don't know the first thing about Boost, but this sounds like a database. Perhaps you can use sqlite? On Mon, Dec 1, 2014 at 4:13 AM, SF Markus Elfring < elfring at users.sourceforge.net> wrote: > Hello, > > I find a data structure like it is provided by the "Boost Multi-index > Containers Library" interesting for efficient data processing. > http://www.boost.org/doc/libs/1_57_0/libs/multi_index/doc/index.html > > How are the chances to integrate a class library with similar > functionality into the Python software infrastructure? > https://bugs.python.org/issue22957 > > Regards, > Markus > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Mon Dec 1 16:49:52 2014 From: toddrjen at gmail.com (Todd) Date: Mon, 1 Dec 2014 16:49:52 +0100 Subject: [Python-ideas] Multi-index Containers Library In-Reply-To: <20141201150328.GA20114@k3> References: <547C5B75.4000001@users.sourceforge.net> <20141201142452.GA11467@k3> <20141201150328.GA20114@k3> Message-ID: On Dec 1, 2014 4:03 PM, "David Wilson" wrote: > > > before it would be useful to include in the stdlib, along with some > > motivating use cases > > One example would be a continuous auction, like a stock exchange order > book. In that case, efficient enumeration is desirable by all of > account ID, order ID, or (price, time). > > book = IndexedList() > book.add_unique_index('order_id', lambda o: o.id) > book.add_index('account_id', lambda o: o.account_id) > book.add_index('price_time', lambda o: (o.price, o.time)) > > def add_order(order): > book.append(order) > > def del_order(order): > del book.indices['order_id'][order.id] > > def close_account(account_id): > del book.indices['account_id'][account_id] > > def iter_matches(limit_price): > for (price, time), order in book.indices['price_time']: > if price > limit_price: > break > yield order > > Again it's not very clear how the magical 'indices' object slices would > work. You can also use pandas for this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Mon Dec 1 16:52:41 2014 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 1 Dec 2014 09:52:41 -0600 Subject: [Python-ideas] A service to crawl +1s and URLs out of mailman archives Message-ID: In context to building a PEP or similar, I don't know how many times I've trawled looking for: * Docs links * Source links * Patch links * THREAD POST LINKS * Consensus A tool to crawl structued and natural language data from the forums could be very useful for preparing PEPs. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Mon Dec 1 16:53:51 2014 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 1 Dec 2014 09:53:51 -0600 Subject: [Python-ideas] A service to crawl +1s and URLs out of mailman archives In-Reply-To: References: Message-ID: * CI Build Links On Mon, Dec 1, 2014 at 9:52 AM, Wes Turner wrote: > In context to building a PEP or similar, I don't know how many times I've > trawled looking for: > > * Docs links > * Source links > * Patch links > * THREAD POST LINKS > * Consensus > > A tool to crawl structued and natural language data from the forums could > be very useful for preparing PEPs. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From constantin at exxxtremesys.lu Mon Dec 1 17:34:16 2014 From: constantin at exxxtremesys.lu (Constantin Berhard) Date: Mon, 01 Dec 2014 17:34:16 +0100 Subject: [Python-ideas] functools.lru_cache manual cache modification Message-ID: <547C9888.6010705@exxxtremesys.lu> Hi, I've written a tail call optimization lib for python3. It works fine. I use a custom return, which throws the next function arguments as an exception and a decorator, which handles the exception. One strength of the functools.lru_cache lies in caching results of calls initiated by the function itself (i.e. recursive call results). However because of the exception, the intermediate results from the tail recursion don't end up in the cache, if my tail call optimization is used together with lru_cache. So I would like to be able to manually add argument-result pairs in the cache. A manual lookup is not needed for my purpose, but I propose that there should be methods to 1. add argument-result pairs to the cache 2. lookup if there is a result for given arguments 3. lookup the result for given arguments if it exists (exception thrown otherwise) 4. invalidate specific cache entries (at the moment you can only invalidate the cache as a whole through func.cache_clear()) What do you think? Best Regards, Constantin -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From elfring at users.sourceforge.net Mon Dec 1 17:50:44 2014 From: elfring at users.sourceforge.net (SF Markus Elfring) Date: Mon, 01 Dec 2014 17:50:44 +0100 Subject: [Python-ideas] Multi-index Containers Library In-Reply-To: References: <547C5B75.4000001@users.sourceforge.net> Message-ID: <547C9C64.8050501@users.sourceforge.net> > I don't know the first thing about Boost, but this sounds like a database. The summary mentions it also. http://www.boost.org/doc/libs/1_57_0/libs/multi_index/doc/index.html "... The concept of multi-indexing over the same collection of elements is borrowed from relational database terminology and allows for the specification of complex data structures in the spirit of multiply indexed relational tables where simple sets and maps are not enough. ..." > Perhaps you can use sqlite? I am reusing this software already as you can see here: Determination for the number of named function parameters (with SmPL) https://lkml.org/lkml/2014/12/1/184 Regards, Markus From steve at pearwood.info Mon Dec 1 19:51:58 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 2 Dec 2014 05:51:58 +1100 Subject: [Python-ideas] A service to crawl +1s and URLs out of mailman archives In-Reply-To: References: Message-ID: <20141201185158.GJ11424@ando.pearwood.info> On Mon, Dec 01, 2014 at 09:52:41AM -0600, Wes Turner wrote: > In context to building a PEP or similar, I don't know how many times I've > trawled looking for: > > * Docs links > * Source links > * Patch links > * THREAD POST LINKS > * Consensus > > A tool to crawl structued and natural language data from the forums could > be very useful for preparing PEPs. Yes it would be. Do you have any idea how to write such a tool? Do you think suh a tool would be of enough interest to enough people that it should be distributed in the Python standard library? I think that this would make a great project on PyPI, especially since it make take a long, long time for it to develop enough intelligence to be able to do the job you're suggesting. Finding links to documentation and source code is fairly straightforward, but building in the intelligence to find "consensus" is a non-trivial application of natural language processing and an impressive feat of artificial intelligence. It certainly doesn't sound like something that somebody could write over a weekend and add to the 3.5 standard library, it's more like an on-going project that will see continual development for many years. -- Steven From grant.jenks at gmail.com Mon Dec 1 21:14:15 2014 From: grant.jenks at gmail.com (Grant Jenks) Date: Mon, 1 Dec 2014 12:14:15 -0800 Subject: [Python-ideas] Multi-index Containers Library In-Reply-To: <547C854C.2000205@users.sourceforge.net> References: <547C5B75.4000001@users.sourceforge.net> <20141201142452.GA11467@k3> <547C854C.2000205@users.sourceforge.net> Message-ID: As the author of the sortedcontainers project (similar to blist, but pure-Python and often faster), I thought this problem would be an interesting application of the library so here goes. I'll define an IndexedList as a container of (pk, item) pairs with at least one unique index mapping pk -> pair like so: from sortedcontainers import SortedListWithKey, SortedDict from operator import itemgetter class IndexedList(object): def __init__(self, *args, **kwargs): self.pk = 0 self.indices = {} self.uniques = {'_pk': SortedDict()} self.keys = {'_pk': itemgetter(0)} Different types of indices are used based on the unique requirement. In general, I'd use a list since an index might contain duplicates but with the unique constraint, I can use a dict: def add_unique_index(self, name, key): key = lambda pair: key(pair[1]) self.uniques[name] = SortedDict(zip(map(key, self), self)) self.keys[name] = key When we have to use a list, we'll store (key, pair) tuples and keep them sorted by the key. def add_index(self, name, key): key = lambda pair: key(pair[1]) items = zip(map(key, self), self) self.indices[name] = SortedListWithKey(items, key=itemgetter(0)) When we iterate, we'll return the pairs: def __iter__(self): return self.uniques['_pk'].values() The append operation means creating a new pair and updating the indices: def append(self, item): self.pk += 1 pair = (self.pk, item) for name in self.uniques: self.uniques[name][self.keys[name](pair)] = pair for name in self.indices: self.indices[name].add(pair) If we want to remove a unique key, we first find the item and then update the indices: def remove_unique(self, name, key): item = self.uniques[name][key] for name in self.uniques: del self.uniques[name][self.keys[name](item)] for name in self.indices: self.indices[name].remove(item) Removing a key from a general index is a little more difficult because there may be many matching items. Because we stored tuples in the index, we can bisect to find the range of items with matching keys and then remove them: def remove_index(self, name, key): start = self.indices[name].bisect_left((key,)) end = self.indices[name].bisect_right((key,)) items = map(itemgetter(1), self.indices[name][start:end]) for item in items: self.remove_unique('_pk', item[0]) Finally, David's original api needs only a bit of modification, everything else remains the same: def del_order(order): book.remove_unique('order_id', order.id) def close_account(account_id): book.remove_index('account_id', account_id) Here's the whole recipe on pastebin*: http://pastebin.com/8S71W3Ue And a link to the SortedContainers project: http://www.grantjenks.com/docs/sortedcontainers/ Grant * Beware, I haven't benchmarked or tested this code. On Mon, Dec 1, 2014 at 7:12 AM, SF Markus Elfring < elfring at users.sourceforge.net> wrote: > >> http://www.boost.org/doc/libs/1_57_0/libs/multi_index/doc/index.html > > > > This came up on #python just last week, and I've personally encountered > > a desire for something similar a bunch of times. > > How challenging is it to map powerful Boost classes to the Python > programming language? > > > > Probably, though, the ideal interface to something like this would need > > some time to evolve before it would be useful to include in the stdlib, > > I am curious on how this could happen ... > > > > along with some motivating use cases > > Would you like to look at any more examples I could show? > > > > -- at least, it would be nice if there were one or two third party > > implementations to compare before settling on an interface. > > I can't think of any existing implementations that operate on purely > > in-memory structures. > > Thanks for your feedback. > > > > Boost::multiindex is pretty awesome, but its interface is also > > somewhat complicated, and a 1:1 translation wouldn't make for a pretty > > Python experience. > > Will this story become another promising software development adventure? > > Regards, > Markus > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Dec 1 22:36:56 2014 From: barry at python.org (Barry Warsaw) Date: Mon, 1 Dec 2014 16:36:56 -0500 Subject: [Python-ideas] A service to crawl +1s and URLs out of mailman archives References: Message-ID: <20141201163656.2654897d@anarchist.wooz.org> On Dec 01, 2014, at 09:52 AM, Wes Turner wrote: >In context to building a PEP or similar, I don't know how many times I've >trawled looking for: > >* Docs links >* Source links >* Patch links >* THREAD POST LINKS >* Consensus > >A tool to crawl structued and natural language data from the forums could >be very useful for preparing PEPs. You should come talk to mailman-developers at python.org. There are lots of interesting possibilities for such things in Mailman 3, and probably decent hacks for Mailman 2. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From cs at zip.com.au Mon Dec 1 23:41:40 2014 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 2 Dec 2014 09:41:40 +1100 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: <547C9888.6010705@exxxtremesys.lu> References: <547C9888.6010705@exxxtremesys.lu> Message-ID: <20141201224140.GA3814@cskk.homeip.net> On 01Dec2014 17:34, Constantin Berhard wrote: >One strength of the functools.lru_cache lies in caching results of calls >initiated by the function itself (i.e. recursive call results). > >However because of the exception, the intermediate results from the tail >recursion don't end up in the cache, if my tail call optimization is >used together with lru_cache. > >So I would like to be able to manually add argument-result pairs in the >cache. A manual lookup is not needed for my purpose, but I propose that >there should be methods to >1. add argument-result pairs to the cache >2. lookup if there is a result for given arguments >3. lookup the result for given arguments if it exists (exception thrown >otherwise) >4. invalidate specific cache entries (at the moment you can only >invalidate the cache as a whole through func.cache_clear()) > >What do you think? I'm kind of +1 on this, but I'd be for a different exposure. Let lru_cache expose a mapping of the cache. Once you have that mapping, expecially if it is just a dict, everything else comes for free. In fact, I'd advocate moving the LRU cache supporting object off into collections and building functools.lru_cache on top of that. Use case: I had to write my own LRU cache bcause I wasn't just wrapping a function. Having that available as a standalone object from collections which looked like a lossy mapping (i.e. things vanish from the mapping when the cache overflows) would be immensely useful. Cheers, Cameron Simpson The first ninety percent of the task takes ninety percent of the time, and the last ten percent takes the other ninety percent. From jdhardy at gmail.com Tue Dec 2 10:59:40 2014 From: jdhardy at gmail.com (Jeff Hardy) Date: Tue, 2 Dec 2014 09:59:40 +0000 Subject: [Python-ideas] Multi-index Containers Library In-Reply-To: References: <547C5B75.4000001@users.sourceforge.net> <20141201142452.GA11467@k3> <20141201150328.GA20114@k3> Message-ID: On Mon, Dec 1, 2014 at 3:13 PM, Chris Angelico wrote: > On Tue, Dec 2, 2014 at 2:03 AM, David Wilson > wrote: > >> before it would be useful to include in the stdlib, along with some > >> motivating use cases > > > > One example would be a continuous auction, like a stock exchange order > > book. In that case, efficient enumeration is desirable by all of > > account ID, order ID, or (price, time). > > For small numbers of entries, it'd be simpler to just sort and filter > on demand; for large numbers of entries, you should probably be using > a database, which will have these sorts of facilities. Is there a > mid-range where it's better to keep it all in memory, but it's too > slow to sort on demand? > My current work project keeps ~8 GB of data in RAM (and looking at 64-128GB servers to get us through the next 3 years). Sorting on demand would be way too slow but it doesn't need to be in a database either - it can be reconstructed from an event stream, and running a DB server is extra ops. Using an in-memory DB like SQLite is an unnecessary extra layer. Currently it just has maps as needed to speed up queries, and I don't think it fits the use case for an MI container, but other projects might. With the amount of RAM in modern machines "keep it all in memory" is viable for lots of use cases with otherwise large values of N. - Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Tue Dec 2 11:51:30 2014 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 2 Dec 2014 04:51:30 -0600 Subject: [Python-ideas] A service to crawl +1s and URLs out of mailman archives In-Reply-To: <20141201185158.GJ11424@ando.pearwood.info> References: <20141201185158.GJ11424@ando.pearwood.info> Message-ID: On Mon, Dec 1, 2014 at 12:51 PM, Steven D'Aprano wrote: > On Mon, Dec 01, 2014 at 09:52:41AM -0600, Wes Turner wrote: > > > In context to building a PEP or similar, I don't know how many times I've > > trawled looking for: > > > > * Docs links > > * Source links > > * Patch links > > * THREAD POST LINKS > > * Consensus > > > > A tool to crawl structued and natural language data from the forums could > > be very useful for preparing PEPs. > > Yes it would be. Do you have any idea how to write such a tool? > > Do you think suh a tool would be of enough interest to enough people > that it should be distributed in the Python standard library? > Such a module would undoubtedly rely upon external libraries like: requests, celery, beautifulsoup, and NLTK. And whatever is necessary to poll mailman without asyncio (e.g. channels, websockets). This is sort of in scope for python-ideas, as a general observation that *linked* development artifacts are traceable, reproducible, and task focused on: docs, code, and tests (the build). > I think that this would make a great project on PyPI, especially since > it make take a long, long time for it to develop enough intelligence to > be able to do the job you're suggesting. Finding links to documentation > and source code is fairly straightforward, but building in the > intelligence to find "consensus" is a non-trivial application of natural > language processing and an impressive feat of artificial intelligence. > It certainly doesn't sound like something that somebody could write over > a weekend and add to the 3.5 standard library, it's more like an > on-going project that will see continual development for many years. > > https://github.com/wrdrd/docs/blob/master/wrdrd/tools/crawl.py Issues: * Too many HTTP requests * Inefficient * A real live queue could be helpful Thank you for your feedback! -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Tue Dec 2 11:53:39 2014 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 2 Dec 2014 04:53:39 -0600 Subject: [Python-ideas] A service to crawl +1s and URLs out of mailman archives In-Reply-To: <20141201163656.2654897d@anarchist.wooz.org> References: <20141201163656.2654897d@anarchist.wooz.org> Message-ID: On Mon, Dec 1, 2014 at 3:36 PM, Barry Warsaw wrote: > On Dec 01, 2014, at 09:52 AM, Wes Turner wrote: > > >In context to building a PEP or similar, I don't know how many times I've > >trawled looking for: > > > >* Docs links > >* Source links > >* Patch links > >* THREAD POST LINKS > >* Consensus > > > >A tool to crawl structued and natural language data from the forums could > >be very useful for preparing PEPs. > > You should come talk to mailman-developers at python.org. There are lots of > interesting possibilities for such things in Mailman 3, and probably decent > hacks for Mailman 2. > Good call, thanks. Are the Python mailman instances upgraded to Mailman 3, with the Django GUI? Is there any way to recognize a Lightweight Markup Language doctype declaration, in email? ```restructuredtext ... ``` ```python #!/usr/bin/env python ``` -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Dec 2 13:57:07 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 2 Dec 2014 22:57:07 +1000 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: <20141201224140.GA3814@cskk.homeip.net> References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> Message-ID: On 2 December 2014 at 08:41, Cameron Simpson wrote: > On 01Dec2014 17:34, Constantin Berhard wrote: >> >> One strength of the functools.lru_cache lies in caching results of calls >> initiated by the function itself (i.e. recursive call results). >> >> However because of the exception, the intermediate results from the tail >> recursion don't end up in the cache, if my tail call optimization is >> used together with lru_cache. >> >> So I would like to be able to manually add argument-result pairs in the >> cache. A manual lookup is not needed for my purpose, but I propose that >> there should be methods to >> 1. add argument-result pairs to the cache >> 2. lookup if there is a result for given arguments >> 3. lookup the result for given arguments if it exists (exception thrown >> otherwise) >> 4. invalidate specific cache entries (at the moment you can only >> invalidate the cache as a whole through func.cache_clear()) >> >> What do you think? > > > I'm kind of +1 on this, but I'd be for a different exposure. Let lru_cache > expose a mapping of the cache. Once you have that mapping, expecially if it > is just a dict, everything else comes for free. > > In fact, I'd advocate moving the LRU cache supporting object off into > collections and building functools.lru_cache on top of that. Use case: I had > to write my own LRU cache bcause I wasn't just wrapping a function. Having > that available as a standalone object from collections which looked like a > lossy mapping (i.e. things vanish from the mapping when the cache > overflows) would be immensely useful. As far as I'm aware, this is actually a deliberate design decision. There are so many degrees of freedom in designing a cache API that without constrainting the usage model it's really quite difficult to come up with a flexible abstraction that's easier to use than just building your own custom caching class. And once you expose the underlying mapping in functools.lru_cache itself, it hugely constraints the internal implementation of that cache (since you just made a whole lot of things that are currently implementation details part of the public API). So collections.OrderedDict provides the raw building block needed to implement an LRU cache, while functools.lru_cache applies that building block to a particular use case. It's OK if folks with needs that don't quite fit the standard idiom write their own custom class to handle it - that makes it possible to keep the standard tools simple to handle the standard cases, while folks with different needs can just write something specifically tailored to their situation, rather than trying to learn and configure a more generic API. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Dec 2 14:02:16 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 2 Dec 2014 23:02:16 +1000 Subject: [Python-ideas] A service to crawl +1s and URLs out of mailman archives In-Reply-To: References: <20141201163656.2654897d@anarchist.wooz.org> Message-ID: On 2 December 2014 at 20:53, Wes Turner wrote: > > On Mon, Dec 1, 2014 at 3:36 PM, Barry Warsaw wrote: >> You should come talk to mailman-developers at python.org. There are lots of >> interesting possibilities for such things in Mailman 3, and probably >> decent >> hacks for Mailman 2. > > Good call, thanks. > > Are the Python mailman instances upgraded to Mailman 3, with the Django GUI? We'd like them to be, but there's no ETA yet (for either the final MM3 release or the subsequent upgrade of the python.org mailing lists). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From constantin at exxxtremesys.lu Tue Dec 2 14:37:14 2014 From: constantin at exxxtremesys.lu (Constantin Berhard) Date: Tue, 02 Dec 2014 14:37:14 +0100 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> Message-ID: <547DC08A.2060609@exxxtremesys.lu> On 02.12.2014 13:57, Nick Coghlan wrote: > As far as I'm aware, this is actually a deliberate design decision. > There are so many degrees of freedom in designing a cache API that > without constrainting the usage model it's really quite difficult to > come up with a flexible abstraction that's easier to use than just > building your own custom caching class. Then couldn't we just create a functools.cache, which takes as argument a data structure with dict-like actions. Then there is a collections.LruCache class, which is the data structure currently used in functools.lru_cache. Then @functools.lru_cache would become @functools.cache(collections.LruCache(maxsize=128)) Where LruCache would be quite trivially based on collections.OrderedDict > And once you expose the underlying mapping in functools.lru_cache > itself, it hugely constraints the internal implementation of that > cache (since you just made a whole lot of things that are currently > implementation details part of the public API). With the above approach, exposing the underlying cache structure would be on purpose, s.t. the user can freely chose the kind of cache to use. This can then be done more explicitly by exposing the underlying cache as decorated_function.cache or something like that. > It's OK if folks with needs that don't quite fit the standard idiom > write their own custom class to handle it - that makes it possible to > keep the standard tools simple to handle the standard cases, while > folks with different needs can just write something specifically > tailored to their situation, rather than trying to learn and configure > a more generic API. In this case I need the standard tools for the standard cases to expose more functionality, s.t. I can make my lib [1] interoperate with the standard lib. I.e. I need to manually add entries to the cache because there are function calls that the cache decorator can't get directly notified of. Best Regards, Constantin [1] my tail call optimization lib: From apalala at gmail.com Tue Dec 2 15:29:30 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Tue, 2 Dec 2014 09:59:30 -0430 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: <547DC08A.2060609@exxxtremesys.lu> References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> <547DC08A.2060609@exxxtremesys.lu> Message-ID: On Tue, Dec 2, 2014 at 9:07 AM, Constantin Berhard < constantin at exxxtremesys.lu> wrote: > Then > > @functools.lru_cache > > would become > > @functools.cache(collections.LruCache(maxsize=128)) > +1 -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Dec 2 18:39:17 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 2 Dec 2014 09:39:17 -0800 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> Message-ID: On Tue, Dec 2, 2014 at 4:57 AM, Nick Coghlan wrote: > On 2 December 2014 at 08:41, Cameron Simpson wrote: > > On 01Dec2014 17:34, Constantin Berhard > wrote: > > In fact, I'd advocate moving the LRU cache supporting object off into > > collections and building functools.lru_cache on top of that. Use case: I > had > > to write my own LRU cache bcause I wasn't just wrapping a function. > Having > > that available as a standalone object from collections which looked like > a > > lossy mapping (i.e. things vanish from the mapping when the cache > > overflows) would be immensely useful. > > As far as I'm aware, this is actually a deliberate design decision. > There are so many degrees of freedom in designing a cache API that > without constrainting the usage model it's really quite difficult to > come up with a flexible abstraction that's easier to use than just > building your own custom caching class. > > And once you expose the underlying mapping in functools.lru_cache > itself, it hugely constraints the internal implementation of that > cache (since you just made a whole lot of things that are currently > implementation details part of the public API). > > So collections.OrderedDict provides the raw building block needed to > implement an LRU cache, while functools.lru_cache applies that > building block to a particular use case. > But, oddly enough, functools.lru_cache doesn't use collections.OrderedDict -- it uses its own linked list, which strikes me as slow. Or what am I missing? > It's OK if folks with needs that don't quite fit the standard idiom > write their own custom class to handle it - that makes it possible to > keep the standard tools simple to handle the standard cases, while > folks with different needs can just write something specifically > tailored to their situation, rather than trying to learn and configure > a more generic API. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Dec 2 18:50:26 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 2 Dec 2014 18:50:26 +0100 Subject: [Python-ideas] functools.lru_cache manual cache modification References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> Message-ID: <20141202185026.77765332@fsol> On Tue, 2 Dec 2014 09:39:17 -0800 Guido van Rossum wrote: > > But, oddly enough, functools.lru_cache doesn't use collections.OrderedDict > -- it uses its own linked list, which strikes me as slow. Or what am I > missing? collections.OrderedDict uses its own slow linked list. I suppose lru_cache is micro-optimized; also, it's thread-safe. Regards Antoine. From barry at python.org Tue Dec 2 20:32:16 2014 From: barry at python.org (Barry Warsaw) Date: Tue, 2 Dec 2014 14:32:16 -0500 Subject: [Python-ideas] A service to crawl +1s and URLs out of mailman archives References: <20141201163656.2654897d@anarchist.wooz.org> Message-ID: <20141202143216.170c13cd@anarchist.wooz.org> On Dec 02, 2014, at 04:53 AM, Wes Turner wrote: >Are the Python mailman instances upgraded to Mailman 3, with the Django GUI? No, but we know there are people using it in production. I expect we'll get another beta before the end of the year and then I think it's time to start planning an experimental deployment on pdo. We have a few experimental lists on mpo that we can convert and start playing with. >Is there any way to recognize a Lightweight Markup Language doctype >declaration, in email? Maybe a Content-Type? But can you elaborate one what you're thinking about? Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From apalala at gmail.com Tue Dec 2 21:48:43 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Tue, 2 Dec 2014 16:18:43 -0430 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: <20141202185026.77765332@fsol> References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> <20141202185026.77765332@fsol> Message-ID: On Tue, Dec 2, 2014 at 1:20 PM, Antoine Pitrou wrote: > collections.OrderedDict uses its own slow linked list. > I suppose lru_cache is micro-optimized; also, it's thread-safe. > One of the propositions is to expose these implementations in a way that: 1. The implementation used for functools.lru_cache can be chosen/switched. 2. The current implementation in lru_cache can be used for other purposes. Grako parsers, for example don't use lru_cache in memoization because they need finer control about what gets cached. -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From constantin at exxxtremesys.lu Tue Dec 2 23:55:34 2014 From: constantin at exxxtremesys.lu (Constantin Berhard) Date: Tue, 02 Dec 2014 23:55:34 +0100 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> <20141202185026.77765332@fsol> Message-ID: <547E4366.3040001@exxxtremesys.lu> On 02.12.2014 21:48, Juancarlo A?ez wrote: > One of the propositions is to expose these implementations in a way that: > > 1. The implementation used for functools.lru_cache can be chosen/switched. > 2. The current implementation in lru_cache can be used for other purposes. Thank you for summing it up. This is really what I want. Cheers, Constantin From elfring at users.sourceforge.net Wed Dec 3 14:10:16 2014 From: elfring at users.sourceforge.net (SF Markus Elfring) Date: Wed, 03 Dec 2014 14:10:16 +0100 Subject: [Python-ideas] Multi-index Containers Library In-Reply-To: References: <547C5B75.4000001@users.sourceforge.net> <20141201142452.GA11467@k3> <20141201150328.GA20114@k3> Message-ID: <547F0BB8.60501@users.sourceforge.net> > Sorting on demand would be way too slow but it doesn't need to be in > a database either - it can be reconstructed from an event stream, > and running a DB server is extra ops. I am looking also for ways to omit persistent storage for a specific data processing task. How often will simple index management be sufficient for some software applications? > Using an in-memory DB like SQLite is an unnecessary extra layer. How many in-memory data bases can be reused by the Python software infrastructure? Would you like to point any more class libraries out? Regards, Markus From raymond.hettinger at gmail.com Wed Dec 3 18:11:20 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 3 Dec 2014 09:11:20 -0800 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> Message-ID: > On Dec 2, 2014, at 4:57 AM, Nick Coghlan wrote: > > And once you expose the underlying mapping in functools.lru_cache > itself, it hugely constraints the internal implementation of that > cache (since you just made a whole lot of things that are currently > implementation details part of the public API). Yes, that is exactly correct. For example, there is a proposed C implementation that has different internal details that it surely won't want to or be able to expose. > > So collections.OrderedDict provides the raw building block needed to > implement an LRU cache, while functools.lru_cache applies that > building block to a particular use case. Right. The OrderedDict makes it really easy to roll your own variants. And that is what the LRU cache used originally. Later, it evolved to use its own linked list so that it could have only one internal dictionary instead of the two used by the OrderedDict. This cut the space overhead almost in half and sped-up cache lookups considerably. Having its own structure also lets the LRU cache neatly deal with multi-threading and re-entrancy so that it does fail in odd ways when used in complex environments. > It's OK if folks with needs that don't quite fit the standard idiom > write their own custom class to handle it - that makes it possible to > keep the standard tools simple to handle the standard cases, while > folks with different needs can just write something specifically > tailored to their situation, rather than trying to learn and configure > a more generic API. I concur. Raymond From ethan at stoneleaf.us Wed Dec 3 19:22:40 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 03 Dec 2014 10:22:40 -0800 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> Message-ID: <547F54F0.8060601@stoneleaf.us> On 12/03/2014 09:11 AM, Raymond Hettinger wrote: > > Having its own structure also lets the LRU cache neatly deal with > multi-threading and re-entrancy so that it does fail in odd ways > when used in complex environments. Hopefully you meant "does _not_ fail in odd ways". ;) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From raymond.hettinger at gmail.com Wed Dec 3 21:24:30 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 3 Dec 2014 12:24:30 -0800 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: <547F54F0.8060601@stoneleaf.us> References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> <547F54F0.8060601@stoneleaf.us> Message-ID: <88095F8E-D708-4575-AD70-4B8E461CB822@gmail.com> > On Dec 3, 2014, at 10:22 AM, Ethan Furman wrote: > > On 12/03/2014 09:11 AM, Raymond Hettinger wrote: >> >> Having its own structure also lets the LRU cache neatly deal with >> multi-threading and re-entrancy so that it does fail in odd ways >> when used in complex environments. > > Hopefully you meant "does _not_ fail in odd ways". ;) Yes :-) Raymond From abarnert at yahoo.com Wed Dec 3 22:22:47 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 3 Dec 2014 13:22:47 -0800 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> Message-ID: <322236CA-F314-4FEC-8C89-153F1BD67702@yahoo.com> On Dec 3, 2014, at 9:11, Raymond Hettinger wrote: > >> On Dec 2, 2014, at 4:57 AM, Nick Coghlan wrote: >> >> And once you expose the underlying mapping in functools.lru_cache >> itself, it hugely constraints the internal implementation of that >> cache (since you just made a whole lot of things that are currently >> implementation details part of the public API). > > Yes, that is exactly correct. For example, there is a proposed C > implementation that has different internal details that it surely > won't want to or be able to expose. > >> >> So collections.OrderedDict provides the raw building block needed to >> implement an LRU cache, while functools.lru_cache applies that >> building block to a particular use case. > > Right. The OrderedDict makes it really easy to roll your own variants. > > And that is what the LRU cache used originally. Later, it evolved to use > its own linked list so that it could have only one internal dictionary > instead of the two used by the OrderedDict. Maybe just providing the original version as a recipe in functools, as an "equivalent to..." block to show people how to roll their own, would solve the problem well enough for this thread? (Especially if 3.5 or 3.6 is going to get a C implementation of OrderedDict, which would presumably help the single-threaded speed problem, if not the space and coarse locking problems, for people who wanted to build something between the odict and the cache?) the 3.2 version is under 100 lines, so it doesn't seem too horrible to copy into the docs. > This cut the space overhead > almost in half and sped-up cache lookups considerably. > > Having its own structure also lets the LRU cache neatly deal with > multi-threading and re-entrancy so that it does fail in odd ways > when used in complex environments. > >> It's OK if folks with needs that don't quite fit the standard idiom >> write their own custom class to handle it - that makes it possible to >> keep the standard tools simple to handle the standard cases, while >> folks with different needs can just write something specifically >> tailored to their situation, rather than trying to learn and configure >> a more generic API. > > > I concur. > > > > Raymond > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From constantin at exxxtremesys.lu Wed Dec 3 23:24:57 2014 From: constantin at exxxtremesys.lu (Constantin Berhard) Date: Wed, 03 Dec 2014 23:24:57 +0100 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: <322236CA-F314-4FEC-8C89-153F1BD67702@yahoo.com> References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> <322236CA-F314-4FEC-8C89-153F1BD67702@yahoo.com> Message-ID: <547F8DB9.1010804@exxxtremesys.lu> On 03.12.2014 22:22, Andrew Barnert wrote: > Maybe just providing the original version as a recipe in functools, as an "equivalent to..." block to show people how to roll their own, would solve the problem well enough for this thread? Actually no, this doesn't solve my problem at all. Independent of the inner implementation of lru_cache I need to be able to manually add and lookup key-value pairs from the outside. I need this to make my decorator compatible with lru_cache. It's only 27 lines of code, you may read it if you want to understand my problem. (additionally 62 lines of example code) The problem is that the exception carrying the arguments->return_data information is thrown past the lru_cache decorator because it can't catch it. Best Regards, Constantin From ethan at stoneleaf.us Wed Dec 3 23:33:14 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 03 Dec 2014 14:33:14 -0800 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: <547F8DB9.1010804@exxxtremesys.lu> References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> <322236CA-F314-4FEC-8C89-153F1BD67702@yahoo.com> <547F8DB9.1010804@exxxtremesys.lu> Message-ID: <547F8FAA.2050004@stoneleaf.us> On 12/03/2014 02:24 PM, Constantin Berhard wrote: > On 03.12.2014 22:22, Andrew Barnert wrote: > > I need this to make my decorator compatible with lru_cache. > > It's only 27 lines of code, you may read it if you want to understand my > problem. (additionally 62 lines of example code) I think you are missing the point: the internals are not going be made public; you need to write your own custom lru_cache. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From constantin at exxxtremesys.lu Wed Dec 3 23:41:36 2014 From: constantin at exxxtremesys.lu (Constantin Berhard) Date: Wed, 03 Dec 2014 23:41:36 +0100 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: <547F8FAA.2050004@stoneleaf.us> References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> <322236CA-F314-4FEC-8C89-153F1BD67702@yahoo.com> <547F8DB9.1010804@exxxtremesys.lu> <547F8FAA.2050004@stoneleaf.us> Message-ID: <547F91A0.9050801@exxxtremesys.lu> On 03.12.2014 23:33, Ethan Furman wrote: > On 12/03/2014 02:24 PM, Constantin Berhard wrote: >> On 03.12.2014 22:22, Andrew Barnert wrote: >> >> I need this to make my decorator compatible with lru_cache. >> >> It's only 27 lines of code, you may read it if you want to understand my >> problem. (additionally 62 lines of example code) > > I think you are missing the point: the internals are not going be made public; you need to write your own custom lru_cache. I don't need or want an lru_cache. I want people, who want to use my library, to be able to use the standard functools together with my lib. So I want to become compatible with lru_cache. Of course, I could provide a special version of an lru cache, which is tuned to work well with my library, but I regard that solution as unelegant. Constantin From ethan at stoneleaf.us Wed Dec 3 23:57:13 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 03 Dec 2014 14:57:13 -0800 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: <547F91A0.9050801@exxxtremesys.lu> References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> <322236CA-F314-4FEC-8C89-153F1BD67702@yahoo.com> <547F8DB9.1010804@exxxtremesys.lu> <547F8FAA.2050004@stoneleaf.us> <547F91A0.9050801@exxxtremesys.lu> Message-ID: <547F9549.4020901@stoneleaf.us> On 12/03/2014 02:41 PM, Constantin Berhard wrote: > On 03.12.2014 23:33, Ethan Furman wrote: >> >> I think you are missing the point: the internals are not going be made public; you need to write your own custom lru_cache. > > I don't need or want an lru_cache. I want people, who want to use my > library, to be able to use the standard functools together with my lib. > So I want to become compatible with lru_cache. How would your library know an lru_cache was being used? Can you provide an example of what the code looked like if you had what you needed? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ncoghlan at gmail.com Thu Dec 4 15:02:04 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 5 Dec 2014 00:02:04 +1000 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: <547F91A0.9050801@exxxtremesys.lu> References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> <322236CA-F314-4FEC-8C89-153F1BD67702@yahoo.com> <547F8DB9.1010804@exxxtremesys.lu> <547F8FAA.2050004@stoneleaf.us> <547F91A0.9050801@exxxtremesys.lu> Message-ID: On 4 December 2014 at 08:41, Constantin Berhard wrote: > On 03.12.2014 23:33, Ethan Furman wrote: >> On 12/03/2014 02:24 PM, Constantin Berhard wrote: >>> On 03.12.2014 22:22, Andrew Barnert wrote: >>> >>> I need this to make my decorator compatible with lru_cache. >>> >>> It's only 27 lines of code, you may read it if you want to understand my >>> problem. (additionally 62 lines of example code) >> >> I think you are missing the point: the internals are not going be made public; you need to write your own custom lru_cache. > > I don't need or want an lru_cache. I want people, who want to use my > library, to be able to use the standard functools together with my lib. > So I want to become compatible with lru_cache. There are *a lot* of memoisation decorators out there - functools.lru_cache is only one of them. It's highly unlikely *any* of them are going to trigger if the call throws an exception. However, it looks as if your library will add the appropriate cache entry on the next pass through the iterative loop, so it isn't clear what additional values you would like to cache. As Ethan suggests, a clearer explanation of the current incompatibility, and what additional cache entries you would like to be able to inject may help us better understand the suggestion. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From constantin at exxxtremesys.lu Fri Dec 5 01:49:36 2014 From: constantin at exxxtremesys.lu (Constantin Berhard) Date: Fri, 05 Dec 2014 01:49:36 +0100 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> <322236CA-F314-4FEC-8C89-153F1BD67702@yahoo.com> <547F8DB9.1010804@exxxtremesys.lu> <547F8FAA.2050004@stoneleaf.us> <547F91A0.9050801@exxxtremesys.lu> Message-ID: <54810120.4060709@exxxtremesys.lu> On 04.12.2014 15:02, Nick Coghlan wrote: > As Ethan suggests, a clearer explanation of the current > incompatibility, and what additional cache entries you would like to > be able to inject may help us better understand the suggestion. You're right. I'm sorry for being unclear. I now have written code that compares a) a stupid custom cache, which offers the interfaces I need ("f") b) the most stupid way to combine libtco with lru_cache ("f2") c) the slightly less stupid way to combine libtco with lru_cache ("f3") A test run computes in this order: f(6), f(6), f(4) in a) only the first computation has to be done, the other two are in cache afterwards in b) nothing can be cached in c) only exact arguments can be cached, i.e. f(6) is only calculated once, but the intermediate result that was calculated on the way is not used to have f(4) in the cache, so f(4) has to be completely recomputed. The code is here: It's a single python 3 script ready for execution. These changes, when properly split up will also find their way into the libtco git. As you can see from the code, the interface I propose just adds two functions to the cache: cache_get and cache_put. I think that any cache should be able to provide these functions, no matter how its cache strategy or how it is implemented. I'm sorry if my code is tldr, but I really can't think of an easy way of explaining it right now. I'll be happy to answer further questions though. Best Regards, Constantin From abarnert at yahoo.com Fri Dec 5 17:32:08 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 5 Dec 2014 08:32:08 -0800 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: <54810120.4060709@exxxtremesys.lu> References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> <322236CA-F314-4FEC-8C89-153F1BD67702@yahoo.com> <547F8DB9.1010804@exxxtremesys.lu> <547F8FAA.2050004@stoneleaf.us> <547F91A0.9050801@exxxtremesys.lu> <54810120.4060709@exxxtremesys.lu> Message-ID: <5146D2EF-E82E-43F5-8345-4DF7B21F5DCF@yahoo.com> On Dec 4, 2014, at 16:49, Constantin Berhard wrote: > On 04.12.2014 15:02, Nick Coghlan wrote: >> As Ethan suggests, a clearer explanation of the current >> incompatibility, and what additional cache entries you would like to >> be able to inject may help us better understand the suggestion. > > You're right. I'm sorry for being unclear. > > I now have written code that compares > a) a stupid custom cache, which offers the interfaces I need ("f") > b) the most stupid way to combine libtco with lru_cache ("f2") > c) the slightly less stupid way to combine libtco with lru_cache ("f3") > > A test run computes in this order: f(6), f(6), f(4) > in a) only the first computation has to be done, the other two are in > cache afterwards > in b) nothing can be cached > in c) only exact arguments can be cached, i.e. f(6) is only calculated > once, but the intermediate result that was calculated on the way is not > used to have f(4) in the cache, so f(4) has to be completely recomputed. > > The code is here: > > It's a single python 3 script ready for execution. These changes, when > properly split up will also find their way into the libtco git. > > As you can see from the code, the interface I propose just adds two > functions to the cache: cache_get and cache_put. I think that any cache > should be able to provide these functions, no matter how its cache > strategy or how it is implemented. You realize that, if this change gets into the 3.5 stdlib, it's still not going to work with any of the many third-party memoization decorators that work today that Nick mentioned, or in 3.4 or earlier, right? So it seems like you're still going to need to provide a workaround--even if that's just including a backport of the current lru_cache in your module or recommending a third-party backport--unless you just want to document that your module requires 3.5+ if people want to also use a memoization cache. Also, it seems like you could have already written a patch for 3.5, and also put your patched version up on PyPI or included it in your module; since you're going to need to do those things even if you convince everyone, why not do it now? (This would also give you a chance to write up initial wording for the docs patch to explain what these new methods are for, which might help explain your problem, and which would also let people critique the wording early.) > I'm sorry if my code is tldr, but I really can't think of an easy way of > explaining it right now. I'll be happy to answer further questions though. > > Best Regards, > Constantin > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ethan at stoneleaf.us Fri Dec 5 20:35:31 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 05 Dec 2014 11:35:31 -0800 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: <54810120.4060709@exxxtremesys.lu> References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> <322236CA-F314-4FEC-8C89-153F1BD67702@yahoo.com> <547F8DB9.1010804@exxxtremesys.lu> <547F8FAA.2050004@stoneleaf.us> <547F91A0.9050801@exxxtremesys.lu> <54810120.4060709@exxxtremesys.lu> Message-ID: <54820903.1090506@stoneleaf.us> A couple comments: You need better names. f? _f? fun? which of those two are the same? Have you tried it without the 'return_from' function? It might be interesting to have this be a more general 'recurse' decorator. You could have your own basic cache, and specify which methods you need for somebody to substitute in a different cache (which is a better option for all previous Pythons than making a change to 3.5). -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From sturla.molden at gmail.com Sat Dec 6 09:43:11 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sat, 6 Dec 2014 08:43:11 +0000 (UTC) Subject: [Python-ideas] Asynchronous IO ideas for Python References: Message-ID: <1658769301439546842.073127sturla.molden-gmail.com@news.gmane.org> Paul Colomiets wrote: > I've written an article about how I perceive the future of > asynchronous I/O in Python. It's not something that should directly be > incorporated into python now, but I believe it's useful for > python-ideas list. > > https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-ce200536d847 This approximately how asynchronous I/O is implemented on Windows (IOCP) and Mac and FreeBSD (GCD): You have a thread-pool that operates independent of the main program thread, and then you can enqueue I/O operations as work tasks to the thread pool. You suggest to have a singleton "I/O kernel" thread in Python, but it is actually not that different from having a pool of worker threads. In a very minimalistic way, one could implement something similar to the set of I/O functions present in GCD. This API is already designed to have the smallest possible complexity, and yet it is as powerful as Windows' IOCP for most practical purposes. https://developer.apple.com/library/mac/documentation/Performance/Reference/GCD_libdispatch_Ref/index.html On Windows it could run on top of IOCP, on Mac and FreeBSD it could run on top of GCD and thus use kqueue under the hood. Linux would actually be the hardest platform, but IOCPs have been implemented with epoll and a threadpool in Wine. Sturla From constantin at exxxtremesys.lu Mon Dec 8 01:10:41 2014 From: constantin at exxxtremesys.lu (Constantin Berhard) Date: Mon, 08 Dec 2014 01:10:41 +0100 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: <54820903.1090506@stoneleaf.us> References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> <322236CA-F314-4FEC-8C89-153F1BD67702@yahoo.com> <547F8DB9.1010804@exxxtremesys.lu> <547F8FAA.2050004@stoneleaf.us> <547F91A0.9050801@exxxtremesys.lu> <54810120.4060709@exxxtremesys.lu> <54820903.1090506@stoneleaf.us> Message-ID: <5484EC81.1080100@exxxtremesys.lu> On 05.12.2014 20:35, Ethan Furman wrote: > A couple comments: > > You need better names. f? _f? fun? which of those two are the same? You are right. I will fix this. > Have you tried it without the 'return_from' function? Sometimes you want to mix tail recursion and non-tail recursion. I see no way to automatically determine, which one to use (apart from changing CPython). So I think there must be two different returns, in this case the normal return and the return_from. > It might be interesting to have this be a more general 'recurse' decorator. What would it do? > You could have your own basic cache, and specify which methods you need for somebody to substitute in a different cache > (which is a better option for all previous Pythons than making a change to 3.5). I didn't really intend to write a cache. @Andrew Barnert: Thank you for the pointers. I will follow them as soon as I have some time. Best Regards, Constantin From mrts.pydev at gmail.com Mon Dec 8 12:13:37 2014 From: mrts.pydev at gmail.com (=?UTF-8?Q?Mart_S=C3=B5mermaa?=) Date: Mon, 8 Dec 2014 13:13:37 +0200 Subject: [Python-ideas] Restitute imp.load_source or equivalent in Python 3? Message-ID: http://bugs.python.org/issue14551 discusses the deprecation and removal of imp.load_source. In the discussion, there seems to be a general consensus that (1) a convenience function for loading modules from an arbitrary file is a good thing and (2) that many people use it. I can only concur, the need for loading a module from an arbitrary file without having to worry about sys.path or __init__.py is useful for supporting plugin-like extensibility/configurability and I personally use imp.load_source in a couple of projects. However, the discussions in http://bugs.python.org/issue14551 seemed to have died off without any clear decision for action. Can we consider adding such a convenience function to importlib in Python 3? Best, Mart S?mermaa -------------- next part -------------- An HTML attachment was scrubbed... URL: From aquavitae69 at gmail.com Mon Dec 8 13:18:02 2014 From: aquavitae69 at gmail.com (David Townshend) Date: Mon, 8 Dec 2014 14:18:02 +0200 Subject: [Python-ideas] Multi-index Containers Library In-Reply-To: <547F0BB8.60501@users.sourceforge.net> References: <547C5B75.4000001@users.sourceforge.net> <20141201142452.GA11467@k3> <20141201150328.GA20114@k3> <547F0BB8.60501@users.sourceforge.net> Message-ID: FWIW, a couple of years ago I wrote an implementation of a multi-index container with an ORM-ish interface ( http://norman.readthedocs.org/en/latest/), more for my own amusement than any practical purposes. However, I found that in my work the applications were actually rather limited and generally better served by sqlalchemy + appropriate db, or pandas (depending on use case). I haven't touched the code in quite a while and there may be some bugs that I never discovered, but it might still be of some use. David On Wed, Dec 3, 2014 at 3:10 PM, SF Markus Elfring < elfring at users.sourceforge.net> wrote: > > Sorting on demand would be way too slow but it doesn't need to be in > > a database either - it can be reconstructed from an event stream, > > and running a DB server is extra ops. > > I am looking also for ways to omit persistent storage for a specific > data processing task. How often will simple index management be sufficient > for some software applications? > > > > Using an in-memory DB like SQLite is an unnecessary extra layer. > > How many in-memory data bases can be reused by the Python software > infrastructure? > Would you like to point any more class libraries out? > > Regards, > Markus > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Dec 8 17:10:42 2014 From: brett at python.org (Brett Cannon) Date: Mon, 08 Dec 2014 16:10:42 +0000 Subject: [Python-ideas] Restitute imp.load_source or equivalent in Python 3? References: Message-ID: On Mon Dec 08 2014 at 6:13:45 AM Mart S?mermaa wrote: > http://bugs.python.org/issue14551 discusses the deprecation and removal > of imp.load_source. In the discussion, there seems to be a general > consensus that (1) a convenience function for loading modules from an > arbitrary file is a good thing and (2) that many people use it. > > I can only concur, the need for loading a module from an arbitrary file > without having to worry about sys.path or __init__.py is useful for > supporting plugin-like extensibility/configurability and I personally use > imp.load_source in a couple of projects. > > However, the discussions in http://bugs.python.org/issue14551 seemed to > have died off without any clear decision for action. > > Can we consider adding such a convenience function to importlib in Python > 3? > You have to realize that imp.load_source() has been fundamentally broken since PEP 302 and just has gotten more broken since Python 3 and importlib. A correct replacement for imp.load_source() in Python 3.5 would be:: # Normally a finder would get you the loader and spec. loader = importlib.machinery.SourceFileLoader(module_name, path) spec = importlib.machinery.ModuleSpec(module_name, loader, origin=path) # Basically what import does when there is no loader.create_module(). module = importlib.util.module_from_spec(spec) # Now is the time to put the module in sys.modules if you want. # How import initializes the module. loader.exec_module(module) As you can see it's not as simple as "read some bytes from a file and make an initialized module" which is why imp.load_source() is deprecated. Plus this side-steps any and all finder and loaders that may be installed which people want to have executed instead of this recipe. So I don't really want to promote people side-stepping all of this through the stdlib. Now if people want to do this in there own code then that's fine and I'm sure someone has/will create a project on PyPI to implement their own solution. But at least for importlib I would rather keep it as-is and lower-level and let the community come up with their own solutions for simplifying the loading of source code dynamically if they are going to side-step a huge chunk of the import machinery to do it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Dec 9 11:50:43 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 9 Dec 2014 20:50:43 +1000 Subject: [Python-ideas] Restitute imp.load_source or equivalent in Python 3? In-Reply-To: References: Message-ID: On 9 December 2014 at 02:10, Brett Cannon wrote: > On Mon Dec 08 2014 at 6:13:45 AM Mart S?mermaa wrote: >> >> http://bugs.python.org/issue14551 discusses the deprecation and removal of >> imp.load_source. In the discussion, there seems to be a general consensus >> that (1) a convenience function for loading modules from an arbitrary file >> is a good thing and (2) that many people use it. >> >> I can only concur, the need for loading a module from an arbitrary file >> without having to worry about sys.path or __init__.py is useful for >> supporting plugin-like extensibility/configurability and I personally use >> imp.load_source in a couple of projects. >> >> However, the discussions in http://bugs.python.org/issue14551 seemed to >> have died off without any clear decision for action. >> >> Can we consider adding such a convenience function to importlib in Python >> 3? > > > You have to realize that imp.load_source() has been fundamentally broken > since PEP 302 and just has gotten more broken since Python 3 and importlib. > A correct replacement for imp.load_source() in Python 3.5 would be:: > > # Normally a finder would get you the loader and spec. > loader = importlib.machinery.SourceFileLoader(module_name, path) > spec = importlib.machinery.ModuleSpec(module_name, loader, origin=path) > # Basically what import does when there is no loader.create_module(). > module = importlib.util.module_from_spec(spec) > # Now is the time to put the module in sys.modules if you want. > # How import initializes the module. > loader.exec_module(module) > > As you can see it's not as simple as "read some bytes from a file and make > an initialized module" which is why imp.load_source() is deprecated. Plus > this side-steps any and all finder and loaders that may be installed which > people want to have executed instead of this recipe. So I don't really want > to promote people side-stepping all of this through the stdlib. Now if > people want to do this in there own code then that's fine and I'm sure > someone has/will create a project on PyPI to implement their own solution. > But at least for importlib I would rather keep it as-is and lower-level and > let the community come up with their own solutions for simplifying the > loading of source code dynamically if they are going to side-step a huge > chunk of the import machinery to do it. And if the goal is "run this Python code to populate a namespace" rather than specifically needing a module as the output, then that's what the runpy module is intended to handle, rather than using the import system. runpy could use updating to be able to target an *existing* namespace rather than a new one (http://bugs.python.org/issue19982), but that shouldn't affect the "possible alternative to imp.load_source()" case. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From oscar.j.benjamin at gmail.com Wed Dec 10 13:05:06 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 10 Dec 2014 12:05:06 +0000 Subject: [Python-ideas] PEP 479 and take() Message-ID: I've been away from Python-related mailing lists for almost a year now and only a few days ago started following them again just in time to see the "hopefully final" text of PEP 479 posted on python-dev. I had been intending to just lurk around for a while but felt compelled to post something after seeing that. It's taken a while for me to find the time to read through what's already been written but I see that PEP 479 is apparently a done deal so I won't try to argue with it (except to register a quick -1 here). Somehow the discussion of PEP 479 became obsessed with the idea that leaked StopIteration is a problem for generators when it is a problem for all iterators. I haven't seen this pointed out yet so I'll demonstrate that map() is susceptible to the same problem: $ cat tmp.py people = [ ['John Cleese', 1, 0, 1], ['Michael Palin', 123, 123], [], # Whoops! ['Terry Gilliam', 12, False, ''], ] def first_name(person): return next(iter(person)).split()[0] for name in map(first_name, people): print(name) $ python3.4 tmp.py John Michael (i.e. Terry was not printed and no error was raised.) There's nothing hacky about the use of map above: the mistake is just the bare next call. The same thing happens with filter, takewhile, etc. Essentially any of the itertools style functions that takes a user-defined function allows StopIteration to pass from the user function to the parent iterator-consumer. I believe this is by design since apparently the author Raymond Hettinger (like me) considered StopIteration fall-through a deliberate design feature of the iterator protocol. Fixing this problem so that a leaked StopIteration turns into a loud error message has been deemed important enough that a partial fix (applying only to generators) warrants breaking the backward compatibility of the core language in a minor release. So what should happen with all the other places that are susceptible? Is StopIteration fall-through to be considered an anti-pattern that anyone implementing the iterator protocol should avoid? With or without PEP 479 the root of the problem is simply in the careless use of next(). The PEP proposes to make this easier to track down but once located the problem will be an unguarded next call that needs to be fixed. (It will also force people to "fix" other things like "raise StopIteration" but these were not actually problematic before). Clearly people want a function like next() that isn't susceptible to this problem or they wouldn't use next in this way and the problem wouldn't exist. So I propose a new function called take() with the following semantics: class TakeError(Exception): pass def take(iterator, n=None): if n is None: try: return next(iterator) except StopIteration: raise TakeError else: return tuple(take(iterator) for _ in range(n)) The idea is that take(iterator) is the generic way to get the next item from the iterator and assert that the item should exist. When you use take(iterator) your intention that the item should exist is self documenting whereas a bare next() is ambiguous without a comment: x = next(iterator) # Never raises StopIteration x = next(iterator) # Propagate StopIteration x = next(iterator) # Haven't considered StopIteration This gives users a clear and un-ugly fix to any code that uses next inappropriately: s/next/take so that there is no excuse for not fixing that code to: x = take(iterator) # Either I get an item or a proper Error is raised. Similarly take(iterator, n) is like islice except that it immediately advances the iterator and raises if the required number of items was not found. Essentially this is a safer version of: firstn = [next(iterator) for _ in range(n)] # Leaks StopIteration firstn = tuple(next(iterator) for _ in range(n)) # Terminates silently firstn = list(islice(iterator, n)) # Terminates silently (Actually the second example would raise RuntimeError with PEP 479) With take it becomes: firstn = take(iterator, n) # n items returned or an Error Oscar From oscar.j.benjamin at gmail.com Wed Dec 10 13:36:59 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 10 Dec 2014 12:36:59 +0000 Subject: [Python-ideas] Generators are iterators Message-ID: Somehow during the discussion of PEP 479 it was concluded that generators will become distinct from iterators. A few people have pointed out that this is a harmful distinction (complicates the language hard to explain etc). I think it is simply an invalid distinction and should be abolished. PEP 479 proposes to change the way that generators handle StopIteration fall-through by default. However the iterator protocol has never formally specified that an iterator should allow StopIteration to fall through to a grandparent consumer. I considered the fact that StopIteration fall-through was possible to be a convenient design feature but it was never a required part of the definition of an iterator. Many iterators perform other kinds of exception handling without allowing that to fall through and many iterators catch StopIteration from child iterators without reraising it. This has never lead anyone to suggest that such iterators are not true iterators in any way. Under PEP 479 generators will handle a StopIteration arising from the executing frame differently. The generator will still be an object with __iter__ and __next__ that exposes StopIteration at the appropriate time to its parent iterator-consumer. In other words it will still satisfy the definition of an iterator. The PEP says: """ Under this proposal, generators and iterators would be distinct, but related, concepts. Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, but proper separation will make bugs more visible. """" This is just plain false. Under the proposal generators will still be iterators. Mixing generators and iterators is nothing like mixing text and bytes and never has been. Mixing iterators and iterables is a bit like mixing text and the bytes in the sense that it can seem to work but then sometimes silently do the wrong thing. Mixing generators and iterators is like mixing sets and containers: there is no mixing since it is simply a false distinction. AFAICT from reading through the discussions this idea has (implicitly) followed from the following fallacious argument: 1) Bare next creates a problem for generators. 2) Therefore we fix it by changing generators. 3) Therefore generators are not iterators any more. Anyone with experience in propositional logic can see multiple fallacies in that argument but actually the biggest mistake is simply in the opening premise: bare next is a problem for all iterators. Generators are affected precisely because they are iterators. Generators were introduced as "a kind of Python iterator, but of an especially powerful kind" (PEP 255). The coroutine idea has since turned generators into something of a Frankenstein concept but the fundamental fact that they are iterators remains unchanged. And regardless of how much discussion coroutines generate the fact remains that 99% of generators are used purely for iteration. I propose to abolish this notion that generators are not iterators and to amend the text of the PEP to unambiguously state that generators are iterators regardless of any changes to the way they propagate StopIteration from the executing frame. Oscar From apalala at gmail.com Wed Dec 10 15:10:11 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Wed, 10 Dec 2014 09:40:11 -0430 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: Message-ID: +1 On Wed, Dec 10, 2014 at 8:06 AM, Oscar Benjamin wrote: > Somehow during the discussion of PEP 479 it was concluded that > generators will become distinct from iterators. A few people have > pointed out that this is a harmful distinction (complicates the > language hard to explain etc). I think it is simply an invalid > distinction and should be abolished. > > PEP 479 proposes to change the way that generators handle > StopIteration fall-through by default. However the iterator protocol > has never formally specified that an iterator should allow > StopIteration to fall through to a grandparent consumer. I considered > the fact that StopIteration fall-through was possible to be a > convenient design feature but it was never a required part of the > definition of an iterator. > > Many iterators perform other kinds of exception handling without > allowing that to fall through and many iterators catch StopIteration > from child iterators without reraising it. This has never lead anyone > to suggest that such iterators are not true iterators in any way. > > Under PEP 479 generators will handle a StopIteration arising from the > executing frame differently. The generator will still be an object > with __iter__ and __next__ that exposes StopIteration at the > appropriate time to its parent iterator-consumer. In other words it > will still satisfy the definition of an iterator. > > The PEP says: > > """ > Under this proposal, generators and iterators would be distinct, but > related, concepts. Like the mixing of text and bytes in Python 2, the > mixing of generators and iterators has resulted in certain perceived > conveniences, but proper separation will make bugs more visible. > """" > > This is just plain false. Under the proposal generators will still be > iterators. Mixing generators and iterators is nothing like mixing text > and bytes and never has been. Mixing iterators and iterables is a bit > like mixing text and the bytes in the sense that it can seem to work > but then sometimes silently do the wrong thing. Mixing generators and > iterators is like mixing sets and containers: there is no mixing since > it is simply a false distinction. > > AFAICT from reading through the discussions this idea has (implicitly) > followed from the following fallacious argument: > > 1) Bare next creates a problem for generators. > 2) Therefore we fix it by changing generators. > 3) Therefore generators are not iterators any more. > > Anyone with experience in propositional logic can see multiple > fallacies in that argument but actually the biggest mistake is simply > in the opening premise: bare next is a problem for all iterators. > Generators are affected precisely because they are iterators. > > Generators were introduced as "a kind of Python iterator, but of an > especially powerful kind" (PEP 255). The coroutine idea has since > turned generators into something of a Frankenstein concept but the > fundamental fact that they are iterators remains unchanged. And > regardless of how much discussion coroutines generate the fact remains > that 99% of generators are used purely for iteration. > > I propose to abolish this notion that generators are not iterators and > to amend the text of the PEP to unambiguously state that generators > are iterators regardless of any changes to the way they propagate > StopIteration from the executing frame. > > > Oscar > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Dec 10 15:18:59 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 11 Dec 2014 01:18:59 +1100 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: Message-ID: <20141210141858.GA20332@ando.pearwood.info> On Wed, Dec 10, 2014 at 12:36:59PM +0000, Oscar Benjamin wrote: > The PEP says: > > """ > Under this proposal, generators and iterators would be distinct, but > related, concepts. Like the mixing of text and bytes in Python 2, the > mixing of generators and iterators has resulted in certain perceived > conveniences, but proper separation will make bugs more visible. > """" > > This is just plain false. I certainly hope so. Currently, generators are iterators: they obey the iterator protocol and the Iterator ABC correctly registers them as instances. py> def gen(): ... yield 1 ... py> it = gen() py> iter(it) is it # Iterator protocol is obeyed. True py> hasattr(it, '__next__') True py> from collections.abc import Iterator py> isinstance(it, Iterator) # And registered with the ABC. True Surely this isn't going to change? If it does, I expect that's going to break an enormous amount of code. If generators are to cease to be iterators, what will they be? > I propose to abolish this notion that generators are not iterators and > to amend the text of the PEP to unambiguously state that generators > are iterators regardless of any changes to the way they propagate > StopIteration from the executing frame. Sounds reasonable to me. -- Steven From ncoghlan at gmail.com Wed Dec 10 17:21:07 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 11 Dec 2014 02:21:07 +1000 Subject: [Python-ideas] Generators are iterators In-Reply-To: <20141210141858.GA20332@ando.pearwood.info> References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: On 11 December 2014 at 00:18, Steven D'Aprano wrote: > > If generators are to cease to be iterators, what will they be? What the PEP is trying to say is that generators are not the same thing as __next__ method implementations (and I agree that shortening the latter to "iterators" is incorrect). __next__ method paths for leaving the frame: * "return value" = produce next value * "raise StopIteration" = end of iteration Current generator paths for leaving the frame: * "yield value" = produce next value * "return" = end of iteration * "raise StopIteration" = end of iteration PEP 479 generator paths for leaving the frame: * "yield value" = produce next value * "return" = end of iteration * "raise StopIteration" = RuntimeError The only change is to eliminate the redundant method of leaving the frame, as that behaviour means that adding a "yield" to a function *implicitly* suppresses StopIteration exceptions raised elsewhere in that frame. That implicit side effect on exception handling behaviour doesn't exist for __next__ method implementations, or for ordinary functions used in iteration operations that accept arbitrary callables, as ordinary functions won't include any yield expressions (by definition). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Wed Dec 10 17:35:04 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Dec 2014 03:35:04 +1100 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: On Thu, Dec 11, 2014 at 3:21 AM, Nick Coghlan wrote: > On 11 December 2014 at 00:18, Steven D'Aprano wrote: >> >> If generators are to cease to be iterators, what will they be? > > What the PEP is trying to say is that generators are not the same > thing as __next__ method implementations (and I agree that shortening > the latter to "iterators" is incorrect). Current wording: (at least, I think this is the only bit that's questionable) """ Under this proposal, generators and iterators would be distinct, but related, concepts. Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, but proper separation will make bugs more visible. """ Would it be better to clarify that as "generator functions"? Maybe something like this: """ Under this proposal, generator functions and iterators would be distinct, but related, concepts. Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, but proper separation will make bugs more visible. The distinction is simple: A generator function returns a generator object. The latter is an iterator, having proper __iter__ and __next__ methods, while the former has neither and does not follow iterator protocol. """ (Most of the text ignores __iter__, even though technically it's an important part of iterator protocol for it to exist and return self. I don't think anyone will be confused by the omission, as it perfectly follows __next__, but I could add text if people think it's important.) ChrisA From njs at pobox.com Wed Dec 10 18:37:24 2014 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 10 Dec 2014 17:37:24 +0000 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: On 10 Dec 2014 16:38, "Chris Angelico" wrote: > > Would it be better to clarify that as "generator functions"? Maybe > something like this: > > """ > Under this proposal, generator functions and iterators would be > distinct, but related, concepts. Like the mixing of text and bytes in > Python 2, the mixing of generators and iterators has resulted in > certain perceived conveniences, but proper separation will make bugs > more visible. The distinction is simple: A generator function returns > a generator object. The latter is an iterator, having proper __iter__ > and __next__ methods, while the former has neither and does not follow > iterator protocol. > """ I find this more confusing than the original, actually, because now it sounds like you're saying that the distinction between a generator function and a generator instance is something new that this PEP is adding, in order to fix all the problems that are being caused by people writing 'for x in genfunc: ...'. Which doesn't make much sense. Like Nick said, it seems to me that the key distinction to emphasize is the distinction between generator function bodies and __next__ method bodies. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Dec 10 19:13:56 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 10 Dec 2014 10:13:56 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: Everyone (including Chris), hold your horses. I will come up with better words for the PEP. I apologize for the confusion. On Wed, Dec 10, 2014 at 9:37 AM, Nathaniel Smith wrote: > On 10 Dec 2014 16:38, "Chris Angelico" wrote: > > > > Would it be better to clarify that as "generator functions"? Maybe > > something like this: > > > > """ > > Under this proposal, generator functions and iterators would be > > distinct, but related, concepts. Like the mixing of text and bytes in > > Python 2, the mixing of generators and iterators has resulted in > > certain perceived conveniences, but proper separation will make bugs > > more visible. The distinction is simple: A generator function returns > > a generator object. The latter is an iterator, having proper __iter__ > > and __next__ methods, while the former has neither and does not follow > > iterator protocol. > > """ > > I find this more confusing than the original, actually, because now it > sounds like you're saying that the distinction between a generator function > and a generator instance is something new that this PEP is adding, in order > to fix all the problems that are being caused by people writing 'for x in > genfunc: ...'. Which doesn't make much sense. Like Nick said, it seems to > me that the key distinction to emphasize is the distinction between > generator function bodies and __next__ method bodies. > > -n > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Dec 10 19:35:35 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 10 Dec 2014 10:35:35 -0800 Subject: [Python-ideas] PEP 479 and take() In-Reply-To: References: Message-ID: Hm... The type of take() seems schizophrenic: take() with one argument returns a single item, while take() with a count returns a tuple of that many items. It would be better if these were two separate functions. Other than that, it's a simple function that people can easily code up themselves (or perhaps there's already a variant in itertools :-). BTW did you know that next(iterator, default) returns default if the iterator is exhausted? IOW this will never raise StopIteration. It's similar to dict.get(key, default) or getattr(obj, attrname, default). On Wed, Dec 10, 2014 at 4:05 AM, Oscar Benjamin wrote: > I've been away from Python-related mailing lists for almost a year now > and only a few days ago started following them again just in time to > see the "hopefully final" text of PEP 479 posted on python-dev. I had > been intending to just lurk around for a while but felt compelled to > post something after seeing that. It's taken a while for me to find > the time to read through what's already been written but I see that > PEP 479 is apparently a done deal so I won't try to argue with it > (except to register a quick -1 here). > > Somehow the discussion of PEP 479 became obsessed with the idea that > leaked StopIteration is a problem for generators when it is a problem > for all iterators. I haven't seen this pointed out yet so I'll > demonstrate that map() is susceptible to the same problem: > > $ cat tmp.py > > people = [ > ['John Cleese', 1, 0, 1], > ['Michael Palin', 123, 123], > [], # Whoops! > ['Terry Gilliam', 12, False, ''], > ] > > def first_name(person): > return next(iter(person)).split()[0] > > for name in map(first_name, people): > print(name) > $ python3.4 tmp.py > John > Michael > > (i.e. Terry was not printed and no error was raised.) > > There's nothing hacky about the use of map above: the mistake is just > the bare next call. The same thing happens with filter, takewhile, > etc. Essentially any of the itertools style functions that takes a > user-defined function allows StopIteration to pass from the user > function to the parent iterator-consumer. I believe this is by design > since apparently the author Raymond Hettinger (like me) considered > StopIteration fall-through a deliberate design feature of the iterator > protocol. > > Fixing this problem so that a leaked StopIteration turns into a loud > error message has been deemed important enough that a partial fix > (applying only to generators) warrants breaking the backward > compatibility of the core language in a minor release. So what should > happen with all the other places that are susceptible? Is > StopIteration fall-through to be considered an anti-pattern that > anyone implementing the iterator protocol should avoid? > > With or without PEP 479 the root of the problem is simply in the > careless use of next(). The PEP proposes to make this easier to track > down but once located the problem will be an unguarded next call that > needs to be fixed. (It will also force people to "fix" other things > like "raise StopIteration" but these were not actually problematic > before). > > Clearly people want a function like next() that isn't susceptible to > this problem or they wouldn't use next in this way and the problem > wouldn't exist. So I propose a new function called take() with the > following semantics: > > class TakeError(Exception): > pass > > def take(iterator, n=None): > if n is None: > try: > return next(iterator) > except StopIteration: > raise TakeError > else: > return tuple(take(iterator) for _ in range(n)) > > The idea is that take(iterator) is the generic way to get the next > item from the iterator and assert that the item should exist. When you > use take(iterator) your intention that the item should exist is self > documenting whereas a bare next() is ambiguous without a comment: > > x = next(iterator) # Never raises StopIteration > x = next(iterator) # Propagate StopIteration > x = next(iterator) # Haven't considered StopIteration > > This gives users a clear and un-ugly fix to any code that uses next > inappropriately: s/next/take so that there is no excuse for not fixing > that code to: > > x = take(iterator) # Either I get an item or a proper Error is raised. > > Similarly take(iterator, n) is like islice except that it immediately > advances the iterator and raises if the required number of items was > not found. Essentially this is a safer version of: > > firstn = [next(iterator) for _ in range(n)] # Leaks StopIteration > firstn = tuple(next(iterator) for _ in range(n)) # Terminates silently > firstn = list(islice(iterator, n)) # Terminates silently > > (Actually the second example would raise RuntimeError with PEP 479) > With take it becomes: > > firstn = take(iterator, n) # n items returned or an Error > > > Oscar > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Wed Dec 10 20:14:21 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Wed, 10 Dec 2014 14:14:21 -0500 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: Message-ID: <1418238861.485297.201337453.691624F4@webmail.messagingengine.com> On Wed, Dec 10, 2014, at 07:36, Oscar Benjamin wrote: > Many iterators perform other kinds of exception handling without > allowing that to fall through and many iterators catch StopIteration > from child iterators without reraising it. This has never lead anyone > to suggest that such iterators are not true iterators in any way. The problem isn't what they do when the child iterator raises StopIteration. That's always been well-defined as "treat that particular child iterator as having ended" and behave however the function wants to treat that case (e.g. chain moves on to the next argument). The question is what they should do when *one of their other inputs* (such as a map or filter function) raises StopIteration. From guido at python.org Wed Dec 10 21:46:11 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 10 Dec 2014 12:46:11 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: I have replaced the confusing paragraph with this: """ The proposal does not change the relationship between generators and iterators: a generator object is still an iterator, and not all iterators are generators. Generators have additional methods that iterators don't have, like ``send`` and ``throw``. All this is unchanged. Nothing changes for generator users -- only authors of generator functions may have to learn something new. """ Let us know if there is still confusion about what the PEP means. --Guido On Wed, Dec 10, 2014 at 10:13 AM, Guido van Rossum wrote: > Everyone (including Chris), hold your horses. I will come up with better > words for the PEP. I apologize for the confusion. > > On Wed, Dec 10, 2014 at 9:37 AM, Nathaniel Smith wrote: > >> On 10 Dec 2014 16:38, "Chris Angelico" wrote: >> > >> > Would it be better to clarify that as "generator functions"? Maybe >> > something like this: >> > >> > """ >> > Under this proposal, generator functions and iterators would be >> > distinct, but related, concepts. Like the mixing of text and bytes in >> > Python 2, the mixing of generators and iterators has resulted in >> > certain perceived conveniences, but proper separation will make bugs >> > more visible. The distinction is simple: A generator function returns >> > a generator object. The latter is an iterator, having proper __iter__ >> > and __next__ methods, while the former has neither and does not follow >> > iterator protocol. >> > """ >> >> I find this more confusing than the original, actually, because now it >> sounds like you're saying that the distinction between a generator function >> and a generator instance is something new that this PEP is adding, in order >> to fix all the problems that are being caused by people writing 'for x in >> genfunc: ...'. Which doesn't make much sense. Like Nick said, it seems to >> me that the key distinction to emphasize is the distinction between >> generator function bodies and __next__ method bodies. >> >> -n >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Dec 10 22:34:03 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 11 Dec 2014 10:34:03 +1300 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: <5488BC4B.7050200@canterbury.ac.nz> Chris Angelico wrote: > """ > Under this proposal, generator functions and iterators would be > distinct, but related, concepts. Like the mixing of text and bytes in > Python 2, the mixing of generators and iterators has resulted in > certain perceived conveniences, but proper separation will make bugs > more visible. The distinction is simple: A generator function returns > a generator object. The latter is an iterator, having proper __iter__ > and __next__ methods, while the former has neither and does not follow > iterator protocol. > """ No, that's still too confused -- all of that was true before as well. Generator functions have never been iterators themselves. I think Nick has it nailed -- the difference is that code inside a generator function implementing a __next__ method will behave differently from that inside an ordinary function implementing a __next__ method. -- Greg From guido at python.org Wed Dec 10 22:37:42 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 10 Dec 2014 13:37:42 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: <5488BC4B.7050200@canterbury.ac.nz> References: <20141210141858.GA20332@ando.pearwood.info> <5488BC4B.7050200@canterbury.ac.nz> Message-ID: On Wed, Dec 10, 2014 at 1:34 PM, Greg Ewing wrote: > Chris Angelico wrote: > >> """ >> Under this proposal, generator functions and iterators would be >> distinct, but related, concepts. Like the mixing of text and bytes in >> Python 2, the mixing of generators and iterators has resulted in >> certain perceived conveniences, but proper separation will make bugs >> more visible. The distinction is simple: A generator function returns >> a generator object. The latter is an iterator, having proper __iter__ >> and __next__ methods, while the former has neither and does not follow >> iterator protocol. >> """ >> > > No, that's still too confused -- all of that was true before as > well. Generator functions have never been iterators themselves. > > I think Nick has it nailed -- the difference is that code inside > a generator function implementing a __next__ method will behave > differently from that inside an ordinary function implementing > a __next__ method. > That's hardly news though. *Except* for the possibility of raising StopIteration to end the iteration, the inside of a generator has always been completely different from the inside of a __next__ method implementation. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Dec 10 18:55:44 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 10 Dec 2014 09:55:44 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: On Wed, Dec 10, 2014 at 8:35 AM, Chris Angelico wrote: > > What the PEP is trying to say is that generators are not the same > > thing as __next__ method implementations (and I agree that shortening > > the latter to "iterators" is incorrect). > agreed that this is moistly semantics, so yes, "generator functions" is probably better. Also, IIUC, there is no such thing (or at least not one thing) as an "iterator" -- rather there are various things that conform to the iterator protocol. The most common of these is a class that implements __iter__ and __next__ methods (that behave the way specified) Should we call that an "iterator class"? Whereas there IS such a thing as a "generator" -- it is the type that is returned by a generator function or generator expression (comprehension?) And generators are one of the types that conform to the iterator protocol So, and here is where I am a bit confused, does this PEP change somethign about the generator type, or something about the syntax of for writing generator functions? (or both?) If it's really changing the behavior of the type, then the PEP should use "generator" > Would it be better to clarify that as "generator functions"? Maybe > something like this: > > """ > Under this proposal, generator functions and iterators would be > distinct, but related, concepts. I think this confuses things, too: generator functions and iterators have always been completely different -- again, there is no such thing as an iterator at all, there are only things that conform to the protocol, and as I think Oscar pointed out, the behavior of StopIteration _inside_ such a type is not specified by the protocol. We could define an "iterator" as anything that conforms to the protocol -- probably how it is being used for the most part already. In which case. generators are a type of iterator custom classes with __iter__ and __next__ are another type of iterator some build-in objects are also iterators -- range objects, files, what have you. So how to write the above paragraph? How about something like: """ Under this proposal, generator functions and custom iterator classes would be more distinct than they currently are. They are two ways of writing an iterator, but with not only different syntax but different behavior with respect to the handling of StopIteration within the object itself. But generator functions have always been a compact way to write an iterator that "does some of the book keeping for you" -- with this PEP, they will do a bit more of the book keeping. """ """ Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, """ I don't know that generators and iterators have ever been "mixed" in the way that text and bytes were. So I think this analogy just confused things. """ The distinction is simple: A generator function returns generator object. The latter is an iterator, having proper __iter__ and __next__ methods, while the former has neither and does not follow iterator protocol. """ I'm not sure there is confusion about that either -- at least not the confusion that is being addressed here. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Dec 10 22:42:44 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 11 Dec 2014 10:42:44 +1300 Subject: [Python-ideas] PEP 479 and take() In-Reply-To: References: Message-ID: <5488BE54.8040809@canterbury.ac.nz> > On Wed, Dec 10, 2014 at 4:05 AM, Oscar Benjamin > > wrote: > > class TakeError(Exception): > pass > > def take(iterator, n=None): > if n is None: > try: > return next(iterator) > except StopIteration: > raise TakeError > else: > return tuple(take(iterator) for _ in range(n)) Won't that just shift the problem from leaked StopIterations to leaked TakeErrors? -- Greg From rosuav at gmail.com Wed Dec 10 22:52:09 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Dec 2014 08:52:09 +1100 Subject: [Python-ideas] Generators are iterators In-Reply-To: <5488BC4B.7050200@canterbury.ac.nz> References: <20141210141858.GA20332@ando.pearwood.info> <5488BC4B.7050200@canterbury.ac.nz> Message-ID: On Thu, Dec 11, 2014 at 8:34 AM, Greg Ewing wrote: > I think Nick has it nailed -- the difference is that code inside > a generator function implementing a __next__ method will behave > differently from that inside an ordinary function implementing > a __next__ method. Hang on. When would code inside a generator function implement __next__? A generator function is a valid way of implementing __iter__, not __next__. ChrisA From greg.ewing at canterbury.ac.nz Wed Dec 10 23:00:32 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 11 Dec 2014 11:00:32 +1300 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> <5488BC4B.7050200@canterbury.ac.nz> Message-ID: <5488C280.3030204@canterbury.ac.nz> Chris Angelico wrote: > Hang on. When would code inside a generator function implement > __next__? A generator function is a valid way of implementing > __iter__, not __next__. Well, it kind of does both at once. -- Greg From alexander.belopolsky at gmail.com Wed Dec 10 23:11:43 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 10 Dec 2014 17:11:43 -0500 Subject: [Python-ideas] Generators are iterators In-Reply-To: <5488BC4B.7050200@canterbury.ac.nz> References: <20141210141858.GA20332@ando.pearwood.info> <5488BC4B.7050200@canterbury.ac.nz> Message-ID: On Wed, Dec 10, 2014 at 4:34 PM, Greg Ewing wrote: > Chris Angelico wrote: > >> """ >> Under this proposal, generator functions and iterators would be >> distinct, but related, concepts. Like the mixing of text and bytes in >> Python 2, the mixing of generators and iterators has resulted in >> certain perceived conveniences, but proper separation will make bugs >> more visible. The distinction is simple: A generator function returns >> a generator object. The latter is an iterator, having proper __iter__ >> and __next__ methods, while the former has neither and does not follow >> iterator protocol. >> """ >> > > No, that's still too confused I think the problem is with the term "generator function" because what we call "generator function" is neither a generator nor a function. It is a callable that returns a generator. To our excuse, this abuse of language is not uncommon. People often say "integer function" when they mean a function with integer domain and range (or even just range). -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Dec 10 23:22:27 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Dec 2014 09:22:27 +1100 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> <5488BC4B.7050200@canterbury.ac.nz> Message-ID: On Thu, Dec 11, 2014 at 9:11 AM, Alexander Belopolsky wrote: > I think the problem is with the term "generator function" because what we > call "generator function" is neither a generator nor a function. It is a > callable that returns a generator. To our excuse, this abuse of language > is not uncommon. People often say "integer function" when they mean a > function with integer domain and range (or even just range). Interesting. Are you saying the 'def' statement may produce a thing that isn't a function, even though it calls itself a function? >>> def gen(): ... yield 1 ... >>> type(gen) >>> type(gen()) It's a function that returns an iterator, same as this is: >>> def nongen(): ... return iter((1,2,3)) ... >>> type(nongen) >>> type(nongen()) A generator function is a function that returns a generator object. Is this problematic? ChrisA From greg.ewing at canterbury.ac.nz Wed Dec 10 22:26:34 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 11 Dec 2014 10:26:34 +1300 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: <5488BA8A.40700@canterbury.ac.nz> Nick Coghlan wrote: > What the PEP is trying to say is that generators are not the same > thing as __next__ method implementations The wording of that part definitely needs improving, then. -- Greg From abarnert at yahoo.com Thu Dec 11 00:58:54 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 10 Dec 2014 15:58:54 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> <5488BC4B.7050200@canterbury.ac.nz> Message-ID: <9433503D-75A0-4A9A-86F2-C2D7415A0F5C@yahoo.com> On Dec 10, 2014, at 14:22, Chris Angelico wrote: > On Thu, Dec 11, 2014 at 9:11 AM, Alexander Belopolsky > wrote: >> I think the problem is with the term "generator function" because what we >> call "generator function" is neither a generator nor a function. It is a >> callable that returns a generator. To our excuse, this abuse of language >> is not uncommon. People often say "integer function" when they mean a >> function with integer domain and range (or even just range). > > Interesting. Are you saying the 'def' statement may produce a thing > that isn't a function, even though it calls itself a function? > >>>> def gen(): > ... yield 1 > ... >>>> type(gen) > >>>> type(gen()) > > > It's a function that returns an iterator, same as this is: > >>>> def nongen(): > ... return iter((1,2,3)) > ... >>>> type(nongen) > >>>> type(nongen()) > > > A generator function is a function that returns a generator object. Is > this problematic? def whatisthis(): return gen() That's not a generator function, but it is a function that returns a generator. (That's a bit of a silly case, but there's plenty of real-life code like this. In fact, IIRC, a couple of the functions in itertools are generator functions in the pure Python implementation but regular functions that return iterators in the C implementation.) > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From steve at pearwood.info Thu Dec 11 01:30:32 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 11 Dec 2014 11:30:32 +1100 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> <5488BC4B.7050200@canterbury.ac.nz> Message-ID: <20141211003031.GC20332@ando.pearwood.info> On Wed, Dec 10, 2014 at 05:11:43PM -0500, Alexander Belopolsky wrote: > I think the problem is with the term "generator function" because what we > call "generator function" is neither a generator nor a function. I'm afraid that is just wrong. "Generator functions" are functions. py> def gen(): ... yield 1 ... py> type(gen) They are distinguishable from other functions by the presence of a flag on the __code__ object. The ``isgeneratorfunction`` function in the inspect module is short enough to reproduce here: def isgeneratorfunction(object): """Return true if the object is a user-defined generator function. Generator function objects provides same attributes as functions. See help(isfunction) for attributes listing.""" return bool((isfunction(object) or ismethod(object)) and object.__code__.co_flags & CO_GENERATOR) Generator functions are essentially syntactic sugar for building iterators from coroutines. The specific type of iterator they return is a generator, which is a built-in type (but not a built-in name). -- Steven From tjreedy at udel.edu Thu Dec 11 10:49:24 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 11 Dec 2014 04:49:24 -0500 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> <5488BC4B.7050200@canterbury.ac.nz> Message-ID: On 12/10/2014 4:37 PM, Guido van Rossum wrote: > On Wed, Dec 10, 2014 at 1:34 PM, Greg Ewing > wrote: > > Chris Angelico wrote: > > """ > Under this proposal, generator functions and iterators would be > distinct, but related, concepts. Like the mixing of text and > bytes in > Python 2, the mixing of generators and iterators has resulted in > certain perceived conveniences, but proper separation will make bugs > more visible. The distinction is simple: A generator function > returns > a generator object. The latter is an iterator, having proper > __iter__ > and __next__ methods, while the former has neither and does not > follow > iterator protocol. > """ > > > No, that's still too confused -- all of that was true before as > well. Generator functions have never been iterators themselves. > > I think Nick has it nailed -- the difference is that code inside > a generator function implementing a __next__ method will behave > differently from that inside an ordinary function implementing > a __next__ method. > > > That's hardly news though. *Except* for the possibility of raising > StopIteration to end the iteration, the inside of a generator has always > been completely different from the inside of a __next__ method > implementation. Perhaps some Python equivalent of the C code will make it clearer what will and (mostly) will not change. Correct me if I am wrong, but I believe the function __call__ method must do something more or less like the following. class function: ... def __call__(self, *arg, **kwargs): "Setup execution frame and either run it or pass to generator." frame = execution_frame(self, args, kwargs) # several attributes of self are used to setup the frame if no_yield(self.code): run(frame) # until it stops return top(frame.stack) else: return generator(self.code, frame) The generator must look something like the following. class generator: def __iter__(self): return self def __init__(self, code, frame): self.gi_code = code self.gi_frame = frame self.gi_running = False def __next__(self): "Adapt gen-func yield, return to iterator return, raise S.I." self.gi_running = True run(self.gi_frame) if yielded(): return top(self.gi_frame.stack) else: # returned raise StopIteration() self.gi_running = False ... (close, send, and throw methods, not relevant here) (I probably have some details of gi_running wrong, and ignored how exhausted generator calls are handled.) Ignoring gi_code, which I believe is only there for introspection (such as gi_code.__name__, which I use for error-reporting), the link between the generator function and the generator's pre-written __next__ method is the suspended generator-function frame, stored as gi_frame. This is what make one generator instance different from another. Currently, a StopIteration raised during 'run(self.gi_frame)' is passed on like any other exception, but treated specially by the generator caller. The change is to wrap the frame execution or do the equivalent thereof. try: run(self.gi_frame) except StopIteration() raise RuntimeError All else remains the same. Do I have this right? The rationale for the change is the bulk of the PEP. However, the PEP rationale and its acceptance are based on the existing relationship between the execution frame of the gf body and the generator.__next__ method that translates the 'gf protocol' to the iterator protocol. Guido, Nick, and Greg have all pointed to this, but for me, thinking in terms of Python equivalents makes it clearer. -- Terry Jan Reedy From rosuav at gmail.com Thu Dec 11 11:05:36 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Dec 2014 21:05:36 +1100 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> <5488BC4B.7050200@canterbury.ac.nz> Message-ID: On Thu, Dec 11, 2014 at 8:49 PM, Terry Reedy wrote: > Currently, a StopIteration raised during 'run(self.gi_frame)' is passed on > like any other exception, but treated specially by the generator caller. > The change is to wrap the frame execution or do the equivalent thereof. > > try: > run(self.gi_frame) > except StopIteration() > raise RuntimeError That's pretty much how the proof-of-concept patch works, yes. (Though the parentheses after StopIteration seem wrong I think?) In theory, it should chain the exceptions, though that's not always currently working. Improvements to the patch welcomed :) ChrisA From constantin at exxxtremesys.lu Thu Dec 11 15:01:46 2014 From: constantin at exxxtremesys.lu (Constantin Berhard) Date: Thu, 11 Dec 2014 15:01:46 +0100 Subject: [Python-ideas] functools.lru_cache manual cache modification In-Reply-To: <5484EC81.1080100@exxxtremesys.lu> References: <547C9888.6010705@exxxtremesys.lu> <20141201224140.GA3814@cskk.homeip.net> <322236CA-F314-4FEC-8C89-153F1BD67702@yahoo.com> <547F8DB9.1010804@exxxtremesys.lu> <547F8FAA.2050004@stoneleaf.us> <547F91A0.9050801@exxxtremesys.lu> <54810120.4060709@exxxtremesys.lu> <54820903.1090506@stoneleaf.us> <5484EC81.1080100@exxxtremesys.lu> Message-ID: <5489A3CA.3030902@exxxtremesys.lu> There is a ticket with a patch now: From hernan.grecco at gmail.com Thu Dec 11 16:06:20 2014 From: hernan.grecco at gmail.com (Hernan Grecco) Date: Thu, 11 Dec 2014 15:06:20 +0000 Subject: [Python-ideas] [Python-Dev] Python 2.x and 3.x use survey, 2014 edition References: Message-ID: On Thu Dec 11 2014 at 11:48:22 AM Giampaolo Rodola' wrote: > On Wed, Dec 10, 2014 at 5:59 PM, Bruno Cauet wrote: > >> Hi all, >> Last year a survey was conducted on python 2 and 3 usage. >> Here is the 2014 edition, slightly updated (from 9 to 11 questions). >> It should not take you more than 1 minute to fill. I would be pleased if >> you took that time. >> > It would be really nice to complement this survey with information gathered from PyPI. I think the python version of the client is not being recorded now, right? I think this could be very useful not only for the Python devs and the community, but particularly for package maintainers. In this way we could find out not only the number of times a given package was downloaded, but also for which python version. I understand that this information might not be entirely accurate (i.e. people downloading the file from the browser or from github) or using other repositories (like the ones provided by anaconda), but still I think it could be useful. And with pip becoming the standard tool, it might be a good moment to do it. cheers, Hern?n -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Dec 11 16:14:38 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 11 Dec 2014 10:14:38 -0500 Subject: [Python-ideas] [Python-Dev] Python 2.x and 3.x use survey, 2014 edition In-Reply-To: References: Message-ID: <56D9BB6C-8792-41EF-85F4-DCB029C72820@stufft.io> > On Dec 11, 2014, at 10:06 AM, Hernan Grecco wrote: > > > On Thu Dec 11 2014 at 11:48:22 AM Giampaolo Rodola' > wrote: > On Wed, Dec 10, 2014 at 5:59 PM, Bruno Cauet > wrote: > Hi all, > Last year a survey was conducted on python 2 and 3 usage. > Here is the 2014 edition, slightly updated (from 9 to 11 questions). > It should not take you more than 1 minute to fill. I would be pleased if you took that time. > > It would be really nice to complement this survey with information gathered from PyPI. I think the python version of the client is not being recorded now, right? > > I think this could be very useful not only for the Python devs and the community, but particularly for package maintainers. In this way we could find out not only the number of times a given package was downloaded, but also for which python version. > > I understand that this information might not be entirely accurate (i.e. people downloading the file from the browser or from github) or using other repositories (like the ones provided by anaconda), but still I think it could be useful. And with pip becoming the standard tool, it might be a good moment to do it. This information is a few months old mostly because I?m lazy and creating the information is a pain in the ass. Total Downloads (For reference): http://d.stufft.io/image/2N293l3v2S1c % Downloads for Python Version: http://d.stufft.io/image/2g1T2U140h1O % Downloads for Python Version (Zoomed to Py3): http://d.stufft.io/image/0B233A151k1k Total Downloads for Python Version: http://d.stufft.io/image/3f3f3g3P181M Bonus - OS Downloads: http://d.stufft.io/image/021v383I0O2c All of the above filter out anything that has an extremely small number of downloads so as not to overwhelm the graphs with a ton of small barely used things. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Thu Dec 11 16:14:58 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 11 Dec 2014 15:14:58 +0000 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: On 10 December 2014 at 20:46, Guido van Rossum wrote: > I have replaced the confusing paragraph with this: > > """ > The proposal does not change the relationship between generators and > iterators: a generator object is still an iterator, and not all > iterators are generators. Generators have additional methods that > iterators don't have, like ``send`` and ``throw``. All this is > unchanged. Nothing changes for generator users -- only authors of > generator functions may have to learn something new. > """ Generators are affected and this therefore affects authors of both generator functions and generator expressions. In the case of a generator function you can achieve the old behaviour post-PEP 479 simply by wrapping the entire body of the function with: try: # body except StopIteration as e: return e.value The only difference is that the generated StopIteration has a different traceback attached so you can't see where it was originally "raised". Generator expressions would need to be converted to generator functions to achieve the same effect. > Let us know if there is still confusion about what the PEP means. The PEP is still very confused. Primarily the fact is that the PEP is attempting to address an issue which affects all iterators but proposing a solution which is only about generators. The PEP seems to suggest that there is something special about generators in this respect when there really isn't. For example: """ The interaction of generators and StopIteration is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, StopIteration can be absorbed by the generator construct. """ There is no interaction between generators and StopIteration. The issue isn't about generators it is about the iterator protocol. StopIteration cannot be absorbed by the generator construct. I think the PEP would be clearer if it properly acknowledged that the problem is a problem for all iterators. The question then is why the fix is only targeted at generators and what should be done about the same problem that occurs in many other forms. The PEP rationale avoids these issues by falsely claiming that generators are special. From donald at stufft.io Thu Dec 11 16:16:50 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 11 Dec 2014 10:16:50 -0500 Subject: [Python-ideas] [Python-Dev] Python 2.x and 3.x use survey, 2014 edition In-Reply-To: <56D9BB6C-8792-41EF-85F4-DCB029C72820@stufft.io> References: <56D9BB6C-8792-41EF-85F4-DCB029C72820@stufft.io> Message-ID: > On Dec 11, 2014, at 10:14 AM, Donald Stufft wrote: > >> >> On Dec 11, 2014, at 10:06 AM, Hernan Grecco > wrote: >> >> >> On Thu Dec 11 2014 at 11:48:22 AM Giampaolo Rodola' > wrote: >> On Wed, Dec 10, 2014 at 5:59 PM, Bruno Cauet > wrote: >> Hi all, >> Last year a survey was conducted on python 2 and 3 usage. >> Here is the 2014 edition, slightly updated (from 9 to 11 questions). >> It should not take you more than 1 minute to fill. I would be pleased if you took that time. >> >> It would be really nice to complement this survey with information gathered from PyPI. I think the python version of the client is not being recorded now, right? >> >> I think this could be very useful not only for the Python devs and the community, but particularly for package maintainers. In this way we could find out not only the number of times a given package was downloaded, but also for which python version. >> >> I understand that this information might not be entirely accurate (i.e. people downloading the file from the browser or from github) or using other repositories (like the ones provided by anaconda), but still I think it could be useful. And with pip becoming the standard tool, it might be a good moment to do it. > > This information is a few months old mostly because I?m lazy and creating the information is a pain in the ass. > > Total Downloads (For reference): http://d.stufft.io/image/2N293l3v2S1c > % Downloads for Python Version: http://d.stufft.io/image/2g1T2U140h1O > % Downloads for Python Version (Zoomed to Py3): http://d.stufft.io/image/0B233A151k1k > Total Downloads for Python Version: http://d.stufft.io/image/3f3f3g3P181M > Bonus - OS Downloads: http://d.stufft.io/image/021v383I0O2c > > All of the above filter out anything that has an extremely small number of downloads so as not to overwhelm the graphs with a ton of small barely used things. Oops, because it?s not clear: These are numbers from PyPI. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From hernan.grecco at gmail.com Thu Dec 11 16:34:01 2014 From: hernan.grecco at gmail.com (Hernan Grecco) Date: Thu, 11 Dec 2014 15:34:01 +0000 Subject: [Python-ideas] [Python-Dev] Python 2.x and 3.x use survey, 2014 edition References: <56D9BB6C-8792-41EF-85F4-DCB029C72820@stufft.io> Message-ID: Hi Donald, I was not aware of this? It looks really useful. How do you access the raw information? cheers, Hern?n On Thu Dec 11 2014 at 12:16:57 PM Donald Stufft wrote: > > On Dec 11, 2014, at 10:14 AM, Donald Stufft wrote: > > > On Dec 11, 2014, at 10:06 AM, Hernan Grecco > wrote: > > > On Thu Dec 11 2014 at 11:48:22 AM Giampaolo Rodola' > wrote: > >> On Wed, Dec 10, 2014 at 5:59 PM, Bruno Cauet >> wrote: >> >>> Hi all, >>> Last year a survey was conducted on python 2 and 3 usage. >>> Here is the 2014 edition, slightly updated (from 9 to 11 questions). >>> It should not take you more than 1 minute to fill. I would be pleased if >>> you took that time. >>> >> > It would be really nice to complement this survey with information > gathered from PyPI. I think the python version of the client is not being > recorded now, right? > > I think this could be very useful not only for the Python devs and the > community, but particularly for package maintainers. In this way we could > find out not only the number of times a given package was downloaded, but > also for which python version. > > I understand that this information might not be entirely accurate (i.e. > people downloading the file from the browser or from github) or using other > repositories (like the ones provided by anaconda), but still I think it > could be useful. And with pip becoming the standard tool, it might be a > good moment to do it. > > > This information is a few months old mostly because I?m lazy and creating > the information is a pain in the ass. > > Total Downloads (For reference): http://d.stufft.io/image/2N293l3v2S1c > % Downloads for Python Version: http://d.stufft.io/image/2g1T2U140h1O > % Downloads for Python Version (Zoomed to Py3): > http://d.stufft.io/image/0B233A151k1k > Total Downloads for Python Version: http://d.stufft.io/image/3f3f3g3P181M > Bonus - OS Downloads: http://d.stufft.io/image/021v383I0O2c > > All of the above filter out anything that has an extremely small number of > downloads so as not to overwhelm the graphs with a ton of small barely used > things. > > > Oops, because it?s not clear: These are numbers from PyPI. > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Dec 11 16:34:34 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 11 Dec 2014 10:34:34 -0500 Subject: [Python-ideas] [Python-Dev] Python 2.x and 3.x use survey, 2014 edition In-Reply-To: References: <56D9BB6C-8792-41EF-85F4-DCB029C72820@stufft.io> Message-ID: <500FCD2E-FC72-44E0-A42E-1F46715D9D85@stufft.io> > On Dec 11, 2014, at 10:34 AM, Hernan Grecco wrote: > > Hi Donald, > > I was not aware of this? It looks really useful. How do you access the raw information? > I?m one of the PyPI administrators and I compile it from the raw logs. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From hernan.grecco at gmail.com Thu Dec 11 16:41:50 2014 From: hernan.grecco at gmail.com (Hernan Grecco) Date: Thu, 11 Dec 2014 15:41:50 +0000 Subject: [Python-ideas] [Python-Dev] Python 2.x and 3.x use survey, 2014 edition References: <56D9BB6C-8792-41EF-85F4-DCB029C72820@stufft.io> <500FCD2E-FC72-44E0-A42E-1F46715D9D85@stufft.io> Message-ID: On Thu Dec 11 2014 at 12:34:41 PM Donald Stufft wrote: > > I?m one of the PyPI administrators and I compile it from the raw logs. > > It would be nice to expose an API for this. I imagine having a tool like vanity[0] but also providing per Python version/OS information. Anyway we could help? cheers, Hernan [0] https://pypi.python.org/pypi/vanity/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Thu Dec 11 21:39:56 2014 From: ron3200 at gmail.com (Ron Adam) Date: Thu, 11 Dec 2014 14:39:56 -0600 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: On 12/11/2014 09:14 AM, Oscar Benjamin wrote: > I think the PEP would be clearer if it properly acknowledged that the > problem is a problem for all iterators. The question then is why the > fix is only targeted at generators and what should be done about the > same problem that occurs in many other forms. The PEP rationale avoids > these issues by falsely claiming that generators are special. It seems to me, the main benefit is for writers of nested generators and co-routines. It makes those cases easier to debug, and may help generators in general be more reusable, and reliable, in the context of nested generators and co-routines. Which if true could contribute to better, and more dependable, multi-thread, multi-process, or even multi-core, software designs. That is the general impression I get from the conversations about the changes being made. But I'm not sure I can point to any one message stating it like that. Possibly someone can confirm if any of those points are valid and why. Or it could be I'm just a bit too optimistic. ;-) Cheers, Ron From luca.sbardella at gmail.com Thu Dec 11 23:38:48 2014 From: luca.sbardella at gmail.com (Luca Sbardella) Date: Thu, 11 Dec 2014 22:38:48 +0000 Subject: [Python-ideas] [Python-Dev] Python 2.x and 3.x use survey, 2014 edition In-Reply-To: References: <56D9BB6C-8792-41EF-85F4-DCB029C72820@stufft.io> <500FCD2E-FC72-44E0-A42E-1F46715D9D85@stufft.io> Message-ID: On 11 December 2014 at 15:41, Hernan Grecco wrote: > > > > On Thu Dec 11 2014 at 12:34:41 PM Donald Stufft wrote: >> >> I?m one of the PyPI administrators and I compile it from the raw logs. >> >> > It would be nice to expose an API for this. I imagine having a tool like > vanity[0] but also providing per Python version/OS information. > > It would be awesome to have these stats for CPython too, possible? Very happy to put up a website with nice visualizations if I can access the raw data. In any case, useful survey, and for me the only blocker for ditching py2 is the google app engine. Regards Luca -- http://lucasbardella.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Dec 12 05:44:45 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 11 Dec 2014 23:44:45 -0500 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: On 12/11/2014 10:14 AM, Oscar Benjamin wrote: > On 10 December 2014 at 20:46, Guido van Rossum wrote: > The PEP is still very confused. Primarily the fact is that the PEP is > attempting to address an issue which affects all iterators but > proposing a solution which is only about generators. The PEP seems to > suggest that there is something special about generators in this > respect when there really isn't. For example: > > """ > The interaction of generators and StopIteration is currently somewhat > surprising, and can conceal obscure bugs. An unexpected exception > should not result in subtly altered behaviour, but should cause a > noisy and easily-debugged traceback. Currently, StopIteration can be > absorbed by the generator construct. > """ > > There is no interaction between generators and StopIteration. The > issue isn't about generators it is about the iterator protocol. > StopIteration cannot be absorbed by the generator construct. What is very confusing is the ambiguous use of 'generator' to mean either 'generator function' or 'generator class instance'. Since I do not know what either of you mean by 'generator' in each instance above, I do not know exactly what either of you meant. If the PEP and discussants consistency used 'generator' to mean "instance of the internal generator class, initialized with a suspended instance of a *user-written* generator function body", then I think some of the confusion would dissipate. The PEP is about the conjunction of the following facts (and a few more, but I think these are the most salient): a) generator function bodies follow a different protocol than iterator __next__ methods, one that does not require them to *ever* raise StopIteration; b) if a g.f. body does raise StopIteration, it might be a substitute for 'return', but it might be a bug -- and apparently bugs do happen in real code; and c) generator.__next__ currently trusts that *user-written* g.f.s are never buggy. The PEP proposal is, either literally or in effect, that generator.__next should stop trusting StopIteration from a g.f., thereby disallowing the sometimes convenient but optional substitution. The justification is that bugs should not pass silently, and now they do. > I think the PEP would be clearer if it properly acknowledged that the > problem is a problem for all iterators. The question then is why the > fix is only targeted at generators and what should be done about the > same problem that occurs in many other forms. The PEP rationale avoids > these issues by falsely claiming that generators are special. Guido today add the following, apparently in response to the above. +When implementing a regular ``__next__()`` method, the only way to +indicate the end of the iteration is to raise ``StopIteration``. So +catching ``StopIteration`` here and converting it to ``RuntimeError`` +would defeat the purpose. This is a reminder of the special status of +generator functions: in a generator function, raising +``StopIteration`` is redundant since the iteration can be terminated +by a simple ``return``. I would put this answer slightly differently. A __next__ method is supposed to raise StopIteration if *and only if* there are no more items to return. The __next__ method author is responsible for fulfilling the contract. Core developers are responsible for generator.__next__; we are not responsible for iterators that others write. Anyone who writes an iterator class whose instances are initialized with 3rd party non-iterator code that is executed in .__next__, should think about what to do if that code raises StopIteration. It would be possible for the PEP to recommend that other .__next__ methods executing external code might do something similar to what generator.__next__ will do. try: execute(self.external_code) except StopIteration as e: raise RuntimeError() -- Terry Jan Reedy From tjreedy at udel.edu Fri Dec 12 05:50:42 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 11 Dec 2014 23:50:42 -0500 Subject: [Python-ideas] [Python-Dev] Python 2.x and 3.x use survey, 2014 edition In-Reply-To: <56D9BB6C-8792-41EF-85F4-DCB029C72820@stufft.io> References: <56D9BB6C-8792-41EF-85F4-DCB029C72820@stufft.io> Message-ID: On 12/11/2014 10:14 AM, Donald Stufft wrote: > This information is a few months old mostly because I?m lazy and > creating the information is a pain in the ass. > > Total Downloads (For reference): http://d.stufft.io/image/2N293l3v2S1c > % Downloads for Python Version: http://d.stufft.io/image/2g1T2U140h1O > % Downloads for Python Version (Zoomed to Py3): > http://d.stufft.io/image/0B233A151k1k > Total Downloads for Python Version: http://d.stufft.io/image/3f3f3g3P181M > Bonus - OS Downloads: http://d.stufft.io/image/021v383I0O2c My interpretation: 1. Dropping 2.4, 2.5 support to support 3.x can easily be justified. 2. Starting with 3.3 (for instance, to use u'abc') can easily be justified. -- Terry Jan Reedy From tjreedy at udel.edu Fri Dec 12 05:55:55 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 11 Dec 2014 23:55:55 -0500 Subject: [Python-ideas] [Python-Dev] Python 2.x and 3.x use survey, 2014 edition In-Reply-To: References: <56D9BB6C-8792-41EF-85F4-DCB029C72820@stufft.io> <500FCD2E-FC72-44E0-A42E-1F46715D9D85@stufft.io> Message-ID: On 12/11/2014 5:38 PM, Luca Sbardella wrote: > It would be awesome to have these stats for CPython too, possible? The last report anyone posted (pydev, maybe a year ago) stats for CPython downloads from python.org, I think 2.x and 3.x were about equal and Windows dominated. I am curious too on data since. -- Terry Jan Reedy From njs at pobox.com Fri Dec 12 06:41:21 2014 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 12 Dec 2014 05:41:21 +0000 Subject: [Python-ideas] [Python-Dev] Python 2.x and 3.x use survey, 2014 edition In-Reply-To: <56D9BB6C-8792-41EF-85F4-DCB029C72820@stufft.io> References: <56D9BB6C-8792-41EF-85F4-DCB029C72820@stufft.io> Message-ID: On 11 Dec 2014 15:14, "Donald Stufft" wrote: > > > This information is a few months old mostly because I?m lazy and creating the information is a pain in the ass. > > Total Downloads (For reference): http://d.stufft.io/image/2N293l3v2S1c > % Downloads for Python Version: http://d.stufft.io/image/2g1T2U140h1O > % Downloads for Python Version (Zoomed to Py3): http://d.stufft.io/image/0B233A151k1k > Total Downloads for Python Version: http://d.stufft.io/image/3f3f3g3P181M > Bonus - OS Downloads: http://d.stufft.io/image/021v383I0O2c > > All of the above filter out anything that has an extremely small number of downloads so as not to overwhelm the graphs with a ton of small barely used things. Neat data, thanks for sharing! I do wonder how meaningful it is, though, because my impression is that PyPI download numbers are overwhelmingly driven by automated test and deployment systems (e.g. Travis-CI) that end up downloading the same dependencies dozens of times a day. Among other things this would explain how it could be that Linux downloads appear to outnumber Windows downloads by an unbelievable factor of ~30x (!). This doesn't invalidate the numbers, of course, but it does mean they may only represent one specific slice of Python's userbase. Another way to get a sense of py2 versus py3 usage is to look at download counts for version-specific wheels on non-linux systems. Some quick playing with vanity suggests that lxml windows downloads are about 10% py3 (even though the only py3 builds they offer are for 32-bit py3.2!), and numpy osx downloads are about 19% py3. I don't know how representative these numbers are either, but they're dramatically higher than what you found. If someone's curious it might be worth trying this approach more systematically. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Dec 12 06:54:38 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 12 Dec 2014 00:54:38 -0500 Subject: [Python-ideas] [Python-Dev] Python 2.x and 3.x use survey, 2014 edition In-Reply-To: References: <56D9BB6C-8792-41EF-85F4-DCB029C72820@stufft.io> Message-ID: > On Dec 12, 2014, at 12:41 AM, Nathaniel Smith wrote: > > On 11 Dec 2014 15:14, "Donald Stufft" > wrote: > > > > > > This information is a few months old mostly because I?m lazy and creating the information is a pain in the ass. > > > > Total Downloads (For reference): http://d.stufft.io/image/2N293l3v2S1c > > % Downloads for Python Version: http://d.stufft.io/image/2g1T2U140h1O > > % Downloads for Python Version (Zoomed to Py3): http://d.stufft.io/image/0B233A151k1k > > Total Downloads for Python Version: http://d.stufft.io/image/3f3f3g3P181M > > Bonus - OS Downloads: http://d.stufft.io/image/021v383I0O2c > > > > All of the above filter out anything that has an extremely small number of downloads so as not to overwhelm the graphs with a ton of small barely used things. > > Neat data, thanks for sharing! > > I do wonder how meaningful it is, though, because my impression is that PyPI download numbers are overwhelmingly driven by automated test and deployment systems (e.g. Travis-CI) that end up downloading the same dependencies dozens of times a day. Among other things this would explain how it could be that Linux downloads appear to outnumber Windows downloads by an unbelievable factor of ~30x (!). This doesn't invalidate the numbers, of course, but it does mean they may only represent one specific slice of Python's userbase. > > Another way to get a sense of py2 versus py3 usage is to look at download counts for version-specific wheels on non-linux systems. Some quick playing with vanity suggests that lxml windows downloads are about 10% py3 (even though the only py3 builds they offer are for 32-bit py3.2!), and numpy osx downloads are about 19% py3. I don't know how representative these numbers are either, but they're dramatically higher than what you found. If someone's curious it might be worth trying this approach more systematically. > > -n > Certainly. Also things like people who use the pip download cache (currently off by default, will be on by default in the next version) will only get a download count triggered for the first download of a particular file so something that is not Python specific if someone ran pip install twice in a row, with 2.7 first and 3.4 second _with_ the download cache on, would only register as a single 2.7 download. I have some other numbers that are specific to certain packages too. I don?t know what conclusion can be drawn from them, but here?s those too: Django: http://d.stufft.io/image/0Q3M2q1M070z cryptography: http://d.stufft.io/image/2p0f1F1O3D3P requests: http://d.stufft.io/image/2c2R2f043W10 pip: http://d.stufft.io/image/3l3M2d2U343C Twisted: http://d.stufft.io/image/031u0x2d1A3v cffi: http://d.stufft.io/image/2H0I2p1A0M2a It might be possible to make some conclusions about different ?slices? looking at these. Unfortunately currently computing this is a fairly intensive task, I have to load months worth of raw logs into a PostgreSQL server, which needs to have a ton of RAM and some SSDs to make the process not take _forever_. That takes a few days normally when I do it (that?s not being super efficient, but it?s a fairly steady pace). The server it resides on is something like $1600 a month when I spin it up (though that doesn?t cost anything since it?s on an OSS Rackspace account) and querying takes 10-15 minutes or so. Luckily! There is an effort under way to move this into Google?s BigQuery system and doing it on a daily basis. This will hopefully lead to both being able to get whatever random one off queries people like as well as make it possible to integrate with PyPI itself to put graphs like these on PyPI. Bonus Because I have them laying around: What kind of distribution is being downloaded: http://d.stufft.io/image/2F0N0f3V3E0V What is actually downloading things: http://d.stufft.io/image/1o0D2g2D2N3D --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Fri Dec 12 10:48:05 2014 From: toddrjen at gmail.com (Todd) Date: Fri, 12 Dec 2014 10:48:05 +0100 Subject: [Python-ideas] [Python-Dev] Python 2.x and 3.x use survey, 2014 edition In-Reply-To: References: <56D9BB6C-8792-41EF-85F4-DCB029C72820@stufft.io> Message-ID: On Dec 12, 2014 6:41 AM, "Nathaniel Smith" wrote: > > On 11 Dec 2014 15:14, "Donald Stufft" wrote: > > > > > > This information is a few months old mostly because I?m lazy and creating the information is a pain in the ass. > > > > Total Downloads (For reference): http://d.stufft.io/image/2N293l3v2S1c > > % Downloads for Python Version: http://d.stufft.io/image/2g1T2U140h1O > > % Downloads for Python Version (Zoomed to Py3): http://d.stufft.io/image/0B233A151k1k > > Total Downloads for Python Version: http://d.stufft.io/image/3f3f3g3P181M > > Bonus - OS Downloads: http://d.stufft.io/image/021v383I0O2c > > > > All of the above filter out anything that has an extremely small number of downloads so as not to overwhelm the graphs with a ton of small barely used things. > > Neat data, thanks for sharing! > > I do wonder how meaningful it is, though, because my impression is that PyPI download numbers are overwhelmingly driven by automated test and deployment systems (e.g. Travis-CI) that end up downloading the same dependencies dozens of times a day. Among other things this would explain how it could be that Linux downloads appear to outnumber Windows downloads by an unbelievable factor of ~30x (!). This doesn't invalidate the numbers, of course, but it does mean they may only represent one specific slice of Python's userbase. Translating these numbers to actual usage in general is hard. The way in which python and packages are distributed on different platforms is very different. How comfortable the average user is with using the command-line vs. just relying on pre-built packages or installers is different. > Another way to get a sense of py2 versus py3 usage is to look at download counts for version-specific wheels on non-linux systems. Some quick playing with vanity suggests that lxml windows downloads are about 10% py3 (even though the only py3 builds they offer are for 32-bit py3.2!), and numpy osx downloads are about 19% py3. I don't know how representative these numbers are either, but they're dramatically higher than what you found. If someone's curious it might be worth trying this approach more systematically. Would it be possible to add an API, flag, or argument to pypi that lets automated services like Travis and py2pack to identify themselves as not being ordinary downloads? Of course this would depend on the services making use of it, but they seem to be trying to be good members of the ecosystem so I would like to think they would. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Dec 12 11:34:09 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 12 Dec 2014 20:34:09 +1000 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: On 12 December 2014 at 01:14, Oscar Benjamin wrote: > > I think the PEP would be clearer if it properly acknowledged that the > problem is a problem for all iterators. The question then is why the > fix is only targeted at generators and what should be done about the > same problem that occurs in many other forms. The PEP rationale avoids > these issues by falsely claiming that generators are special. I believe you're misunderstanding the problem being solved. The specific problem deemed worthy of being fixed is that the presence of "yield" in a function body can implicitly suppress StopIteration exceptions raised elsewhere in that function body (or in functions it calls). The difference in behaviour between comprehensions and generator expressions when it comes to embedded function calls that trigger StopIteration is a special case of that more general difference. This is a problem unique to generators, it does not affect any other iterator (since explicit __next__ method implementations do not use yield). The change in the PEP is to change that side effect such that those exceptions are converted to RuntimeError rather than silently suppressed - making generator function bodies behave less like __next__ method implementations. For special methods themselves, the problem of unexpected signalling exceptions from other called functions being interpreted according to their defined meaning in the relevant protocol is inherent in their design - they only have two "normal" outcomes (returning from the frame, or raising the signalling exception). That limitation applies to StopIteration & __next__ in the same way that it applies to KeyError/IndexError and __getitem__, or AttributeError and the various methods in the attribute access protocols. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From oscar.j.benjamin at gmail.com Fri Dec 12 13:42:55 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 12 Dec 2014 12:42:55 +0000 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: On 12 December 2014 at 10:34, Nick Coghlan wrote: > On 12 December 2014 at 01:14, Oscar Benjamin wrote: >> >> I think the PEP would be clearer if it properly acknowledged that the >> problem is a problem for all iterators. The question then is why the >> fix is only targeted at generators and what should be done about the >> same problem that occurs in many other forms. The PEP rationale avoids >> these issues by falsely claiming that generators are special. > > I believe you're misunderstanding the problem being solved. And I believe you are too. :) > The > specific problem deemed worthy of being fixed is that the presence of > "yield" in a function body can implicitly suppress StopIteration > exceptions raised elsewhere in that function body (or in functions it > calls). The yield causes the function to become a generator function. The frame for a generator function (like for any other function) will allow uncaught exceptions to propagate to the frame above. The difference between generator functions and other functions is that the code in the body of a generator function is executed when someone calls the generator's __next__ method. Since the caller (the iterator consumer) is expecting StopIteration it is treated as the signalling the end of iteration. The yield suppresses nothing; it is the iterator consumer e.g. the for-loop or the list() function etc. which catches the StopIteration and treats it as termination. > The difference in behaviour between comprehensions and > generator expressions when it comes to embedded function calls that > trigger StopIteration is a special case of that more general > difference. I don't know what you mean by this. > This is a problem unique to generators, it does not affect any other > iterator (since explicit __next__ method implementations do not use > yield). Incorrect. The problem is not unique to generators and the yield is irrelevant. The problem (insofar as it is) is a problem for all iterators since all iterators interact with iterator-consumers and it is the iterator-consumer that catches the StopIteration. Here is an example using map: >>> def func(x): ... if x < 0: ... raise StopIteration ... return x ** 2 ... >>> it = map(func, [1, 2, 3, -1, 2]) >>> list(it) [1, 4, 9] >>> list(it) # map continues to yield values... [4] Ordinarily map should terminate when the underlying iterator raises StopIteration. Since it simply allows a StopIteration raised anywhere to bubble up it will also allow the StopIteration raised by func to bubble up. Implicitly the effect of that is that map agrees to cooperatively terminate iteration when any part of the code that it executes raises StopIteration. Note that it is the list() function (the iterator-consumer) that actually catches the StopIteration and terminates just as it would be if map were a generator: >>> def map(func, iterable): ... for item in iterable: ... yield func(item) # func could raise StopIteration ... >>> it = map(func, [1, 2, 3, -1, 2]) >>> list(it) [1, 4, 9] >>> list(it) # The exception terminates the frame in this case [] In principle that could be a useful feature in some cases. In practise it is more likely to be something that masks a bug manifesting in a stray StopIteration e.g.: >>> def first_line(filename): ... with open(filename) as fin: ... return next(fin) # What happens if the file is empty ... >>> for line in map(first_line, filenames): ... print(line) ... In this case it is the for-loop (the iterator-consumer) that catches the StopIteration. Whether or not map is a generator or some other iterator is irrelevant. > The change in the PEP is to change that side effect such that > those exceptions are converted to RuntimeError rather than silently > suppressed - making generator function bodies behave less like > __next__ method implementations. Fine but generator function bodies are __next__ method implementations. You can claim that the generator function implements __iter__ but its body implements the __next__ method of the generator. It is precisely because it is a __next__ method that it finds itself being called by an iterator consumer and so finds that a loose StopIteration is suppressed by the consumer. This is the same for all iterators and there is nothing special about generators in this regard (although the PEP will change that a little). To summarise succinctly: Loose StopIteration can have an unintended interaction with the iterator protocol and more specifically with iterator-consumers. The effect is as if the iterator had been stopped since the consumer will stop iteration. Because generators are iterators they (like all iterators) are affected by this. Oscar From solipsis at pitrou.net Fri Dec 12 13:45:26 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 12 Dec 2014 13:45:26 +0100 Subject: [Python-ideas] [Python-Dev] Python 2.x and 3.x use survey, 2014 edition References: <56D9BB6C-8792-41EF-85F4-DCB029C72820@stufft.io> Message-ID: <20141212134526.101ae5c5@fsol> On Fri, 12 Dec 2014 10:48:05 +0100 Todd wrote: > > Would it be possible to add an API, flag, or argument to pypi that lets > automated services like Travis and py2pack to identify themselves as not > being ordinary downloads? Of course this would depend on the services > making use of it, but they seem to be trying to be good members of the > ecosystem so I would like to think they would. Those services (or at least Travis) usually invoke hand-written scripts, so this would rely on every developer being a "good citizen" and updating their scripts to use that option. Perhaps filtering by source IP would work better, if you could identify the IPs used by Travis VMs and the like. Regards Antoine. From ncoghlan at gmail.com Fri Dec 12 14:14:56 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 12 Dec 2014 23:14:56 +1000 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: On 12 December 2014 at 22:42, Oscar Benjamin wrote: > On 12 December 2014 at 10:34, Nick Coghlan wrote: >> On 12 December 2014 at 01:14, Oscar Benjamin wrote: >>> >>> I think the PEP would be clearer if it properly acknowledged that the >>> problem is a problem for all iterators. The question then is why the >>> fix is only targeted at generators and what should be done about the >>> same problem that occurs in many other forms. The PEP rationale avoids >>> these issues by falsely claiming that generators are special. >> >> I believe you're misunderstanding the problem being solved. > > And I believe you are too. :) > >> The >> specific problem deemed worthy of being fixed is that the presence of >> "yield" in a function body can implicitly suppress StopIteration >> exceptions raised elsewhere in that function body (or in functions it >> calls). > > The yield causes the function to become a generator function. The > frame for a generator function (like for any other function) will > allow uncaught exceptions to propagate to the frame above. The > difference between generator functions and other functions is that the > code in the body of a generator function is executed when someone > calls the generator's __next__ method. Since the caller (the iterator > consumer) is expecting StopIteration it is treated as the signalling > the end of iteration. The yield suppresses nothing; it is the iterator > consumer e.g. the for-loop or the list() function etc. which catches > the StopIteration and treats it as termination. Oscar, this is where you're *just plain wrong* from a technical perspective. The conversion to a generator function also introduces an additional layer of indirection at iteration time through the generator-iterator's __next__ method implementation. That indirection layer is the one that converts an ordinary return from the generator frame into a StopIteration exception. Historically, it also allowed StopIteration from within the frame to propagate out, make it impossible to tell from outside the generator iterator whether the termination was due to the generator returning normally, or due to StopIteraton being raised. The introduction of "yield from" in Python 3.3, and its use in the coroutine design for the asyncio library and similar contexts *makes that distinction more important than it used to be*. It even managed to introduce obscure bugs around the use of generators to write context managers, where factoring out part of the context manager to a subgenerator may implicitly suppress a StopIteration exception raised in the body of the with statement. The problem you think PEP 479 is trying to solve *is* the one where the discussion started, but it is *not* the one where it ended when Guido accepted PEP 479. The problems PEP 479 solves are generator specific - they have nothing to do with the iterator protocol in general. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From oscar.j.benjamin at gmail.com Fri Dec 12 14:58:22 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 12 Dec 2014 13:58:22 +0000 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: On 12 December 2014 at 13:14, Nick Coghlan wrote: > On 12 December 2014 at 22:42, Oscar Benjamin wrote: >> On 12 December 2014 at 10:34, Nick Coghlan wrote: >>> The >>> specific problem deemed worthy of being fixed is that the presence of >>> "yield" in a function body can implicitly suppress StopIteration >>> exceptions raised elsewhere in that function body (or in functions it >>> calls). >> >> The yield causes the function to become a generator function. The >> frame for a generator function (like for any other function) will >> allow uncaught exceptions to propagate to the frame above. The >> difference between generator functions and other functions is that the >> code in the body of a generator function is executed when someone >> calls the generator's __next__ method. Since the caller (the iterator >> consumer) is expecting StopIteration it is treated as the signalling >> the end of iteration. The yield suppresses nothing; it is the iterator >> consumer e.g. the for-loop or the list() function etc. which catches >> the StopIteration and treats it as termination. > > Oscar, this is where you're *just plain wrong* from a technical perspective. Which of the above sentences is technically incorrect? > The conversion to a generator function also introduces an additional > layer of indirection at iteration time through the > generator-iterator's __next__ method implementation. > > That indirection layer is the one that converts an ordinary return > from the generator frame into a StopIteration exception. Historically, > it also allowed StopIteration from within the frame to propagate out, > make it impossible to tell from outside the generator iterator whether > the termination was due to the generator returning normally, or due to > StopIteraton being raised. I understand all of this. > The introduction of "yield from" in Python 3.3, and its use in the > coroutine design for the asyncio library and similar contexts *makes > that distinction more important than it used to be*. It even managed > to introduce obscure bugs around the use of generators to write > context managers, where factoring out part of the context manager to a > subgenerator may implicitly suppress a StopIteration exception raised > in the body of the with statement. > > The problem you think PEP 479 is trying to solve *is* the one where > the discussion started, but it is *not* the one where it ended when > Guido accepted PEP 479. The problems PEP 479 solves are generator > specific - they have nothing to do with the iterator protocol in > general. Okay well this is not clear from the PEP (and bear in mind that I did take the time to read through all of the lengthy threads on this subject before posting here). I haven't read the most recent changes but when I did read the PEP the justification appeared to stem from claims that were confusing at best and plain false at worst e.g. "generators are not iterators". Oscar From chris.barker at noaa.gov Fri Dec 12 18:16:38 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 12 Dec 2014 09:16:38 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: On Fri, Dec 12, 2014 at 5:58 AM, Oscar Benjamin wrote: > Okay well this is not clear from the PEP (and bear in mind that I did > take the time to read through all of the lengthy threads on this > subject before posting here). I haven't read the most recent changes > but when I did read the PEP the justification appeared to stem from > claims that were confusing at best and plain false at worst e.g. > "generators are not iterators". > I don't think there is much technical confusion here, but there seems to be, so, so that highlights that it's very important to write vary carefully about all this. I"m going to try this: 1) It's not clear what is meant by an "iterator" -- when we use the term is seems like we are talking about a type, when we are really talking about "anything that conforms to the iterator protocol", so I'm going to introduce a new term: TIIP (Type Implementing the Iterator Protocol) -- maybe that's too cute, but role with it for now. 2) there are a number of TIPPS in the standard library: things like tupleiterator and friends, the range object, etc. 3) there are essentially two (and a half) ways for python code authors to write their own, custom TIIP: 1) write a class that has a __iter__ and __next__ methods that "do the right thing" 2a) write a generator function 2b) write a generator expression The latter create a generator type, which is a specific TIIP that come built in to the standard library. Generator functions are a particularly nifty syntax for creating TIIPs -- i.e. I'm pretty sure that there is nothing you can do with a generator function that you can't do by writing a custom TIIP class. But generator functions provide really nifty syntax that "does a lot of the bookeeping for you" This is pretty much how I've taught this stuff in the past. But what this PEP does is change generators a bit, so that they do even more of the book keeping for you -- that sure seems reasonable enough when phrased that way. So I agree with Oscar (and others) here the issue of what happens to a StopIteration that is raised inside a TIIP be another call is an issue that effects not only generators, but also custom TIIP classes. I think is wold be nice if this were addressed in both cases the same way, but that has been proposed and rejected for what I, at least, think are sound technical reasons. And the PEP has been accepted so the only thing left to bike-shed about is how it discussed. So why it it worth changing the behavior of generators? 1) we can -- custom classes are, well, custom, so are free to do what author wants them to do. Generators are built-in classes and can therefor be controlled by python itself. 2) generator function have a different, cleaner, more compact way to express intent. while they create generators that conform to the iteration protocol -- i.e. raise a StopIteration when done, the author does not have to explicitly do that -- you can write a generator function without any awareness of StopIteration ( I might argue that you won't get far if you don't understand all that, but...) So having generators automatically "do the right thing" when they call another TIIP inside themselves conforms well to the spirit of generator functions: clean compact code that does the book keeping for you. 3) when you write a custom TIIP class, you have complete control over what it does, and have to be thinking about when StopIteration is raise. So you are more likely to think about how you want to handle a StopIteration that may be raised by another call to a TIP inside your code. You also want to be able to control that depending on circumstances, so probably wouldn't want magic Exception mangling anyway. Not sure if this helps the PEP writing, but I've got some good notes for the next time I teach this... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Dec 12 20:28:49 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 12 Dec 2014 11:28:49 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: <548B41F1.3040005@stoneleaf.us> On 12/12/2014 09:16 AM, Chris Barker wrote: > > 3) there are essentially two (and a half) ways for python code authors to write their own, custom TIIP: > 1) write a class that has a __iter__ and __next__ methods that "do the right thing" > 2a) write a generator function > 2b) write a generator expression 3) write a class with __getitem__ that supports integer indexing -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From alexander.belopolsky at gmail.com Fri Dec 12 20:41:15 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 12 Dec 2014 14:41:15 -0500 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: On Fri, Dec 12, 2014 at 12:16 PM, Chris Barker wrote: > 2) there are a number of TIPPS in the standard library: things like > tupleiterator and friends, the range object, etc. The range object is not an iterator >>> isinstance(range(1), collections.Iterator) False and does not implement the iterator protocol >>> range(1).__next__ Traceback (most recent call last): File "", line 1, in AttributeError: 'range' object has no attribute '__next__' Please don't confuse iterators and iterables: >>> isinstance(range(1), collections.Iterable) True -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Dec 12 21:06:14 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 12 Dec 2014 15:06:14 -0500 Subject: [Python-ideas] Generators are iterators In-Reply-To: <548B41F1.3040005@stoneleaf.us> References: <20141210141858.GA20332@ando.pearwood.info> <548B41F1.3040005@stoneleaf.us> Message-ID: On Fri, Dec 12, 2014 at 2:28 PM, Ethan Furman wrote: > > 3) there are essentially two (and a half) ways for python code authors > to write their own, custom TIIP: > > 1) write a class that has a __iter__ and __next__ methods that "do the > right thing" > > 2a) write a generator function > > 2b) write a generator expression > > 3) write a class with __getitem__ that supports integer indexing > This gets you a custom iterable. The corresponding iterator will be the buitin iterator type: >>> class X: ... __getitem__ = None ... >>> iter(X()) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Dec 12 21:06:11 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 12 Dec 2014 15:06:11 -0500 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: <548B4AB3.4030408@udel.edu> On 12/12/2014 7:42 AM, Oscar Benjamin wrote:> On 12 December 2014 at 10:34, Nick Coghlan wrote: >> specific problem deemed worthy of being fixed is that the presence of >> "yield" in a function body can implicitly suppress StopIteration >> exceptions raised elsewhere in that function body (or in functions it >> calls). This makes no sense to me either. > The yield causes the function to become a generator function. The > frame for a generator function (like for any other function) will > allow uncaught exceptions to propagate to the frame above. The > difference between generator functions and other functions is that the > code in the body of a generator function is executed when someone > calls the generator's __next__ method. Since the caller (the iterator > consumer) is expecting StopIteration it is treated as the signalling > the end of iteration. The yield suppresses nothing; it is the iterator > consumer e.g. the for-loop or the list() function etc. which catches > the StopIteration and treats it as termination. > >> The difference in behaviour between comprehensions and >> generator expressions when it comes to embedded function calls that >> trigger StopIteration is a special case of that more general >> difference. > > I don't know what you mean by this. Neither do I >> This is a problem unique to generators, it does not affect any other >> iterator (since explicit __next__ method implementations do not use >> yield). The parenthetical comment is true. However, I think part of the problem is unique to generator functions, and part is more general -- that loose usage of StopIteration creates problems when a .__next__ method calls external code. > Incorrect. The problem is not unique to generators and the yield is > irrelevant. Nick (and Guido) are looking at the unique part and Oscar is looking at he general part. > The problem (insofar as it is) is a problem for all > iterators since all iterators interact with iterator-consumers and it > is the iterator-consumer that catches the StopIteration. Here is an > example using map: > >>>> def func(x): > ... if x < 0: > ... raise StopIteration > ... return x ** 2 This function violates the implicit guideline, which I think should be more explicit in the doc, that the only functions that should expose StopIteration to the outside world are iterator.__next__ functions and the corresponding builtin next(). It is not surprising that this violation causes map to misbehave. However, changing the .__next__ methods of map and filter iterators should be a new thread, which I will write after posting this. The g.f. PEP says that generator functions should also follow this guideline and not raise StopIteration (but should always return when not yielding). ... >> The change in the PEP is to change that side effect such that >> those exceptions are converted to RuntimeError rather than silently >> suppressed - making generator function bodies behave less like >> __next__ method implementations. > > Fine but generator function bodies are __next__ method > implementations. This is about as wrong, or at least as confusing, as saying that the function passed to map is a __next__ method implementation. There is only one generator.__next__, just as there is only one map.__next__. Both .__next__ methods execute the user function (or function frame) passed in to the corresponding .__init__ to calculate the next value to return. So the external functions assist the __next__ methods but are not __next__ methods themselves. In particular, they should not raise StopIteration. -- Terry Jan Reedy From alexander.belopolsky at gmail.com Fri Dec 12 21:14:44 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 12 Dec 2014 15:14:44 -0500 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> <548B41F1.3040005@stoneleaf.us> Message-ID: On Fri, Dec 12, 2014 at 3:06 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > On Fri, Dec 12, 2014 at 2:28 PM, Ethan Furman wrote: > >> > 3) there are essentially two (and a half) ways for python code authors >> to write their own, custom TIIP: >> > 1) write a class that has a __iter__ and __next__ methods that "do >> the right thing" >> > 2a) write a generator function >> > 2b) write a generator expression >> >> 3) write a class with __getitem__ that supports integer indexing >> > > This gets you a custom iterable. The corresponding iterator will be the > buitin iterator type: > > >>> class X: > ... __getitem__ = None > ... > >>> iter(X()) > > The same logic applies to generators and I think this misconception is at the root of the confusion in this thread. When you are writing a generator function, you are not creating a custom TIIP. You are writing a code object that Python knows how to convert into an instance of a built-in TIIP (the generator type). -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Dec 12 21:41:28 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 12 Dec 2014 12:41:28 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> <548B41F1.3040005@stoneleaf.us> Message-ID: <548B52F8.2000109@stoneleaf.us> On 12/12/2014 12:06 PM, Alexander Belopolsky wrote: > On Fri, Dec 12, 2014 at 2:28 PM, Ethan Furman wrote: >> >> 3) write a class with __getitem__ that supports integer indexing > > This gets you a custom iterable. The corresponding iterator will be the buitin iterator type: > >>>> class X: > ... __getitem__ = None > ... >>>> iter(X()) > Good catch, thanks! -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From oscar.j.benjamin at gmail.com Fri Dec 12 22:10:00 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 12 Dec 2014 21:10:00 +0000 Subject: [Python-ideas] Generators are iterators In-Reply-To: <548B4AB3.4030408@udel.edu> References: <20141210141858.GA20332@ando.pearwood.info> <548B4AB3.4030408@udel.edu> Message-ID: On 12 December 2014 at 20:06, Terry Reedy wrote: > > > On 12/12/2014 7:42 AM, Oscar Benjamin wrote:> On 12 December 2014 at 10:34, > Nick Coghlan wrote: > >>> specific problem deemed worthy of being fixed is that the presence of >>> "yield" in a function body can implicitly suppress StopIteration >>> exceptions raised elsewhere in that function body (or in functions it >>> calls). > > This makes no sense to me either. I'm wondering if Nick is referring to two different things at once: On the one hand "yield from" which is an iterator-consumer suppresses StopIteration from a child iterator and on the other hand every generator interacts with a parent iterator-consumer that will catch any StopIteration it raises. When generator function f "yield from"s to generator function g both things occur in tandem in two different frames. > >> The problem (insofar as it is) is a problem for all >> iterators since all iterators interact with iterator-consumers and it >> is the iterator-consumer that catches the StopIteration. Here is an >> example using map: >> >>>>> def func(x): >> ... if x < 0: >> ... raise StopIteration >> ... return x ** 2 > > This function violates the implicit guideline, which I think should be more > explicit in the doc, that the only functions that should expose > StopIteration to the outside world are iterator.__next__ functions and the > corresponding builtin next(). It is not surprising that this violation > causes map to misbehave. However, changing the .__next__ methods of map and > filter iterators should be a new thread, which I will write after posting > this. The function was just supposed to generate the effect. More plausible examples can be made. I tripped over this myself once when doing something with iterators that required me to extract the first item before processing the rest. A simple demonstration: def sum(iterable): iterator = iter(iterable) total = next(iterator) for item in iterator: total += item return total results = list(map(sum, data_sources)) > The g.f. PEP says that generator functions should also follow this guideline > and not raise StopIteration (but should always return when not yielding). > ... >>> The change in the PEP is to change that side effect such that >>> those exceptions are converted to RuntimeError rather than silently >>> suppressed - making generator function bodies behave less like >>> __next__ method implementations. >> >> Fine but generator function bodies are __next__ method >> implementations. > > This is about as wrong, or at least as confusing, as saying that the > function passed to map is a __next__ method implementation. There is only > one generator.__next__, just as there is only one map.__next__. Both > .__next__ methods execute the user function (or function frame) passed in to > the corresponding .__init__ to calculate the next value to return. So the > external functions assist the __next__ methods but are not __next__ methods > themselves. In particular, they should not raise StopIteration. The idea is that it happens by accident. Oscar From tjreedy at udel.edu Fri Dec 12 22:14:07 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 12 Dec 2014 16:14:07 -0500 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError Message-ID: " exception StopIteration Raised by built-in function next() and an iterator?s __next__() method to signal that there are no further items produced by the iterator." I have always taken this to mean that these are the only functions that should raise StopIteration, but other have not, and indeed, StopIteration in generator functions, which are not __next__ methods, has been accepted and passed on by generator.__next__. PEP 479 reverses this acceptance by having generator.__next turn StopIteration raised in a user-written generator function body into a RuntimeError. I propose that other builtin iterator.__next__ methods that execute a passed in function do the same. This proposal comes from Oscar Benjamin's comments on the PEP in the 'Generator are iterators' thread. In one post, he gave this example. >>> def func(x): ... if x < 0: ... raise StopIteration ... return x ** 2 ... >>> it = map(func, [1, 2, 3, -1, 2]) >>> list(it) [1, 4, 9] >>> list(it) # map continues to yield values... [4] Function func violates what I think should be the guideline. When passed to map, the result is a silently buggy iterator. The doc says that map will "Return an iterator that applies function to every item of iterable, yielding the results.". If map cannot do that, because func raises for some item in iterable, map should also raise, but the exception should be something other than StopIteration, which signals normal non-buggy completion of its task. I propose that map.__next__ convert StopIteration raised by func to RuntimeError, just as generator.__next__ now does for StopIteration raised by executing a generator function frame. (And same for filter().) class newmap: "Simplified version allowing just one input iterable." def __iter__(self): return self def __init__(self, func, iterable): self.func = func self.argit = iter(iterable) def __next__(self): func = self.func args = next(self.argit) # pass on expected StopIteration if func is None: return args else: try: # new wrapper return func(args) except StopIteration: raise RuntimeError('func raised StopIteration') -- Terry Jan Reedy From rosuav at gmail.com Fri Dec 12 22:21:10 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 13 Dec 2014 08:21:10 +1100 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: Message-ID: On Sat, Dec 13, 2014 at 8:14 AM, Terry Reedy wrote: > I propose that map.__next__ convert StopIteration raised by func to > RuntimeError, just as generator.__next__ now does for StopIteration raised > by executing a generator function frame. (And same for filter().) > > class newmap: > "Simplified version allowing just one input iterable." > def __iter__(self): > return self > def __init__(self, func, iterable): > self.func = func > > self.argit = iter(iterable) > def __next__(self): > func = self.func > args = next(self.argit) # pass on expected StopIteration > if func is None: > return args > else: > try: # new wrapper > return func(args) > except StopIteration: > raise RuntimeError('func raised StopIteration') (I'm not sure why you have the "if func is None" check. Currently map() doesn't accept None as its function. But that could equally be implemented below if you wish.) I propose a much MUCH simpler version of map, then: def newmap(func, iterable): """Simplified version allowing just one input iterable.""" for val in iterable: yield func(val) Et voila! Conversion of StopIteration into RuntimeError. ChrisA From alexander.belopolsky at gmail.com Fri Dec 12 22:24:21 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 12 Dec 2014 16:24:21 -0500 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: Message-ID: On Fri, Dec 12, 2014 at 4:14 PM, Terry Reedy wrote: > I propose that map.__next__ convert StopIteration raised by func to > RuntimeError Aren't we stepping on a slippery slope here? What about say >>> def f(x, y): ... if x > 2: ... raise StopIteration ... return x + y ... >>> import itertools >>> itertools.accumulate(range(10), f) >>> list(_) [0, 1, 3] -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Fri Dec 12 22:44:16 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 12 Dec 2014 21:44:16 +0000 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: Message-ID: On 12 December 2014 at 21:24, Alexander Belopolsky wrote: > > On Fri, Dec 12, 2014 at 4:14 PM, Terry Reedy wrote: >> >> I propose that map.__next__ convert StopIteration raised by func to >> RuntimeError > > Aren't we stepping on a slippery slope here? What about say > >>>> def f(x, y): > ... if x > 2: > ... raise StopIteration > ... return x + y > ... >>>> import itertools >>>> itertools.accumulate(range(10), f) > >>>> list(_) > [0, 1, 3] It applies to map, filter, and 5 or 6 things in itertools and certainly many other things outside stdlib. I guess the question is: should it be considered a bug for an iterator to leak a StopIteration outside of its "normal" exit condition? Apparently it was for generators which represent the majority of all iterators and it was considered sufficiently important to change the core language in a backward incompatible way. Oscar From chris.barker at noaa.gov Fri Dec 12 22:59:15 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 12 Dec 2014 13:59:15 -0800 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: Message-ID: On Fri, Dec 12, 2014 at 1:44 PM, Oscar Benjamin wrote: > > I guess the question is: should it be considered a bug for an iterator > to leak a StopIteration outside of its "normal" exit condition? > Apparently it was for generators I don't think anyone aid it was a bug -- often it is a bug, but that's use-case dependent. which represent the majority of all > iterators and it was considered sufficiently important to change the > core language in a backward incompatible way. It is considered prone to hard to find and understand bugs -- which was the motivation for the PEP. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Fri Dec 12 23:11:25 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 12 Dec 2014 22:11:25 +0000 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: Message-ID: On 12 December 2014 at 21:59, Chris Barker wrote: > On Fri, Dec 12, 2014 at 1:44 PM, Oscar Benjamin > wrote: >> >> I guess the question is: should it be considered a bug for an iterator >> to leak a StopIteration outside of its "normal" exit condition? >> Apparently it was for generators > > I don't think anyone aid it was a bug -- often it is a bug, but that's > use-case dependent. I don't think that the positive use-cases of this feature are common. >> which represent the majority of all >> iterators and it was considered sufficiently important to change the >> core language in a backward incompatible way. > > It is considered prone to hard to find and understand bugs -- which was the > motivation for the PEP. The same reasoning applies here. I had a bug like this and it was data-dependent, hard to reproduce, and there was no exception to hook a debugger onto or stack trace to give information. Since the full program was quite slow and I had to keep running it in full it turned what would have been a 1 minute fix into a half of a day of debugging. Since then I've simply been wary about next() and haven't suffered the problem again. I suspect that in the same situation now it wouldn't take me so long because at the time I wasn't really aware of the possibility that StopIteration could get "caught" by a for-loop several frames up. From chris.barker at noaa.gov Fri Dec 12 22:51:39 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 12 Dec 2014 13:51:39 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> <548B41F1.3040005@stoneleaf.us> Message-ID: On Fri, Dec 12, 2014 at 12:14 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > This gets you a custom iterable. The corresponding iterator will be the >> buitin iterator type: >> >> >>> class X: >> ... __getitem__ = None >> ... >> >>> iter(X()) >> >> > > The same logic applies to generators and I think this misconception is at > the root of the confusion in this thread. When you are writing a > generator function, you are not creating a custom TIIP. You are writing a > code object that Python knows how to convert into an instance of a built-in > TIIP (the generator type). > The point was supposed to be that you want a TIIP with particular behavior that is not built-in (hence custom). Yes, in terms of type, a generator function, when called, will return the a built-in type, but its behavior is customized. Indeed the same as writing a class with __getitem__ and no __iter__ or __next__ will give you an instance a built-in iterator type with customized behavior. I guess my take is that the user shouldn't have to know exactly what the type of TIIP is produced is -- only what its behavior will be. But maybe that IS the source of much of this confusion -- the PEP authors make the point that generators are something special. I want tend to think of them as an implementation detail. This PEP makes them more special, and that's really the point of the contentious part of the PEP. So maybe that's the the point we are trying to make -- while we can argue about whether generators ARE iterators, or whether they have been "conflated" -- generators are special, and might as well be more special ;-) -Chris > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Dec 12 23:33:42 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 12 Dec 2014 14:33:42 -0800 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: Message-ID: <548B6D46.4000404@stoneleaf.us> On 12/12/2014 02:11 PM, Oscar Benjamin wrote: > On 12 December 2014 at 21:59, Chris Barker wrote: >> >> It is considered prone to hard to find and understand bugs -- which was the >> motivation for the PEP. > > The same reasoning applies here. I had a bug like this and it was > data-dependent, hard to reproduce, and there was no exception to hook > a debugger onto or stack trace to give information. And now it will raise an exception at the point where the error actually occurred. Having gone through that experience I would have thought you would be more in favor of the change. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From oscar.j.benjamin at gmail.com Fri Dec 12 23:41:06 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 12 Dec 2014 22:41:06 +0000 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: <548B6D46.4000404@stoneleaf.us> References: <548B6D46.4000404@stoneleaf.us> Message-ID: On 12 December 2014 at 22:33, Ethan Furman wrote: > On 12/12/2014 02:11 PM, Oscar Benjamin wrote: >> On 12 December 2014 at 21:59, Chris Barker wrote: >>> >>> It is considered prone to hard to find and understand bugs -- which was the >>> motivation for the PEP. >> >> The same reasoning applies here. I had a bug like this and it was >> data-dependent, hard to reproduce, and there was no exception to hook >> a debugger onto or stack trace to give information. > > And now it will raise an exception at the point where the error actually occurred. It won't because I wasn't using generators. The point II have been trying to make is that this isn't just about generators. > Having gone through that experience I would have thought you would be more in favor of the change. On further reflection I am more in favour of the change. My initial thought was that it places the emphasis in the wrong place. I had concluded that the focus should be on the inappropriateness of next() and I still think that but that doesn't mean that the PEP isn't a good thing in and of itself. From tjreedy at udel.edu Fri Dec 12 23:54:01 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 12 Dec 2014 17:54:01 -0500 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> <548B4AB3.4030408@udel.edu> Message-ID: On 12/12/2014 4:10 PM, Oscar Benjamin wrote: > The function was just supposed to generate the effect. More plausible > examples can be made. I tripped over this myself once when doing > something with iterators that required me to extract the first item > before processing the rest. A simple demonstration: > > def sum(iterable): > iterator = iter(iterable) > total = next(iterator) > for item in iterator: > total += item > return total Examples like this has been posted on python-list, probably more than once. The answer has been to catch StopIteration (which should only come from next/__next__) and either return a default value or raise ValueError('cannot sum an empty iterable'). > results = list(map(sum, data_sources)) -- Terry Jan Reedy From tjreedy at udel.edu Sat Dec 13 00:12:41 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 12 Dec 2014 18:12:41 -0500 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: Message-ID: On 12/12/2014 4:21 PM, Chris Angelico wrote: > On Sat, Dec 13, 2014 at 8:14 AM, Terry Reedy wrote: >> I propose that map.__next__ convert StopIteration raised by func to >> RuntimeError, just as generator.__next__ now does for StopIteration raised >> by executing a generator function frame. (And same for filter().) >> >> class newmap: >> "Simplified version allowing just one input iterable." >> def __iter__(self): >> return self >> def __init__(self, func, iterable): >> self.func = func >> >> self.argit = iter(iterable) >> def __next__(self): >> func = self.func >> args = next(self.argit) # pass on expected StopIteration >> if func is None: >> return args >> else: >> try: # new wrapper >> return func(args) >> except StopIteration: >> raise RuntimeError('func raised StopIteration') > > (I'm not sure why you have the "if func is None" check. Because I initially tried to implement newmap with multiple iterators, I looked at the 2.7 itertools.imap equivalent Python code (removed from the doc in 3.x) and I forgot that None no longer works. The check and return should be removed. > Currently map() doesn't accept None as its function. map.__init__ accepts None, but map.__next__ croaks trying to call it. > I propose a much MUCH simpler version of map, then: > > def newmap(func, iterable): > """Simplified version allowing just one input iterable.""" > for val in iterable: > yield func(val) > > Et voila! Conversion of StopIteration into RuntimeError. I wrote a class because map is a class with .__next__. A python implementation would better be a generator function, as with examples in the itertools doc. -- Terry Jan Reedy From oscar.j.benjamin at gmail.com Sat Dec 13 00:15:33 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 12 Dec 2014 23:15:33 +0000 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> <548B4AB3.4030408@udel.edu> Message-ID: On 12 December 2014 at 22:54, Terry Reedy wrote: > On 12/12/2014 4:10 PM, Oscar Benjamin wrote: > >> The function was just supposed to generate the effect. More plausible >> examples can be made. I tripped over this myself once when doing >> something with iterators that required me to extract the first item >> before processing the rest. A simple demonstration: >> >> def sum(iterable): >> iterator = iter(iterable) >> total = next(iterator) >> for item in iterator: >> total += item >> return total > > > Examples like this has been posted on python-list, probably more than once. > The answer has been to catch StopIteration (which should only come from > next/__next__) and either return a default value or raise ValueError('cannot > sum an empty iterable'). I know what the solution is. For my purposes simply raising an exception (any exception that wasn't StopIteration) would have been sufficient. I'm just saying that it's a mistake that can easily be made. Since I made it I'm now super-suspicious of next() so I probably wouldn't make it again. This mistake is the root of all of the problems that PEP 479 attempts to solve. If you don't believe it is a problem that someone might make this mistake then it follows that PEP 479 is a massive mistake. From ethan at stoneleaf.us Sat Dec 13 00:19:38 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 12 Dec 2014 15:19:38 -0800 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: <548B6D46.4000404@stoneleaf.us> Message-ID: <548B780A.2030701@stoneleaf.us> On 12/12/2014 02:41 PM, Oscar Benjamin wrote: > On 12 December 2014 at 22:33, Ethan Furman wrote: >> >> And now it will raise an exception at the point where the error actually occurred. > > It won't because I wasn't using generators. The point I have been > trying to make is that this isn't just about generators. FWIW I agree that the real culprit is next() -- just about any other function that we call will raise an error exception if something goes wrong, but in 'next's case, asking for the next item when there isn't one raises a flow-control exception instead of an EmptyIterable exception. Happily, we can write our own next() for our own modules (or even for built-ins if we're really adventurous!): #untested unsafe_next = builtins.next _unset_ = object() def next(iterator, default=_unset_): try: return unsafe_next(iterator) except StopIteration: if default is _unset_: raise EmptyIterable return default -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From abarnert at yahoo.com Sat Dec 13 00:22:32 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 12 Dec 2014 15:22:32 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> On Dec 12, 2014, at 9:16, Chris Barker wrote: > On Fri, Dec 12, 2014 at 5:58 AM, Oscar Benjamin wrote: >> Okay well this is not clear from the PEP (and bear in mind that I did >> take the time to read through all of the lengthy threads on this >> subject before posting here). I haven't read the most recent changes >> but when I did read the PEP the justification appeared to stem from >> claims that were confusing at best and plain false at worst e.g. >> "generators are not iterators". > > I don't think there is much technical confusion here, but there seems to be, so, so that highlights that it's very important to write vary carefully about all this. I"m going to try this: > > 1) It's not clear what is meant by an "iterator" -- when we use the term is seems like we are talking about a type, when we are really talking about "anything that conforms to the iterator protocol", so I'm going to introduce a new term: TIIP (Type Implementing the Iterator Protocol) -- maybe that's too cute, but role with it for now. When you're writing actual code, you can test this with isinstance(collections.abc.Iterator). Of course ABCs don't _perfectly_ cover their related protocols (in this case, if you create a type that has __iter__ and __next__ but __iter__ doesn't return self--or, of course, if you just register some random type with Iterator explicitly--you've created a subclass of Iterator that isn't a subtype of the iterator protocol). But I think it still makes sense to use Iterator as shorthand for "type that subclasses Iterator" with the implied "is an actual subtype of the semantic type intended by the Iterator class". This kind of shorthand is used regularly in discussing types in every OO language, not just Python. You don't see people writing about "an instance of a Type Implementing IEnumerable [or NSFastEnumeration or random_access_iterator]). You just see "an IEnumerable". And to discuss the type itself, you just say "an IEnumerable type". The fact that IEnumerable is itself defined as a type is rarely a source of confusion--and when it is, you say "an IEnumerable subtype". In Python, the term "Iterator" is just as consistent and meaningful as in all these other languages. The fact that some people confuse iterables and iterators isn't a reason to abandon this simplicity. Especially since it doesn't even help to solve the problem--as we've just seen in this very message I'm replying to, it's just as easy for someone to mistakenly use TIIP to describe "range" as it is for them to mistakenly use "Iterator". (In fact, it's arguably worse, because what would you call an Iterable under the same naming scheme but Type Implementing Iterable Protocol, or TIIP?) The things we need to be clear about here are the things that _dont't_ have an official name. In particular, the thing that's built from calling a generator function or evaluating a generator expression and used by the generator.__next__ method is not a generator, a generator function, a generator function body, an iterator, or anything else with a name in the language. And that's what's confusing people. They want to call it one of the things that has a name. Renaming any of those things that it's not doesn't help to give the thing that it is a name. > 2) there are a number of TIPPS in the standard library: things like tupleiterator and friends, the range object, etc. > > 3) there are essentially two (and a half) ways for python code authors to write their own, custom TIIP: > 1) write a class that has a __iter__ and __next__ methods that "do the right thing" > 2a) write a generator function > 2b) write a generator expression > > The latter create a generator type, which is a specific TIIP that come built in to the standard library. Generator functions are a particularly nifty syntax for creating TIIPs -- i.e. I'm pretty sure that there is nothing you can do with a generator function that you can't do by writing a custom TIIP class. But generator functions provide really nifty syntax that "does a lot of the bookeeping for you" > > This is pretty much how I've taught this stuff in the past. But what this PEP does is change generators a bit, so that they do even more of the book keeping for you -- that sure seems reasonable enough when phrased that way. > > So I agree with Oscar (and others) here the issue of what happens to a StopIteration that is raised inside a TIIP be another call is an issue that effects not only generators, but also custom TIIP classes. I think is wold be nice if this were addressed in both cases the same way, but that has been proposed and rejected for what I, at least, think are sound technical reasons. > > And the PEP has been accepted so the only thing left to bike-shed about is how it discussed. > > So why it it worth changing the behavior of generators? > > 1) we can -- custom classes are, well, custom, so are free to do what author wants them to do. Generators are built-in classes and can therefor be controlled by python itself. > > 2) generator function have a different, cleaner, more compact way to express intent. while they create generators that conform to the iteration protocol -- i.e. raise a StopIteration when done, the author does not have to explicitly do that -- you can write a generator function without any awareness of StopIteration ( I might argue that you won't get far if you don't understand all that, but...) So having generators automatically "do the right thing" when they call another TIIP inside themselves conforms well to the spirit of generator functions: clean compact code that does the book keeping for you. > > 3) when you write a custom TIIP class, you have complete control over what it does, and have to be thinking about when StopIteration is raise. So you are more likely to think about how you want to handle a StopIteration that may be raised by another call to a TIP inside your code. You also want to be able to control that depending on circumstances, so probably wouldn't want magic Exception mangling anyway. > > Not sure if this helps the PEP writing, but I've got some good notes for the next time I teach this... > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Dec 12 20:45:24 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 12 Dec 2014 11:45:24 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> Message-ID: On Fri, Dec 12, 2014 at 11:41 AM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > > On Fri, Dec 12, 2014 at 12:16 PM, Chris Barker > wrote: > >> 2) there are a number of TIPPS in the standard library: things like >> tupleiterator and friends, the range object, etc. > > > The range object is not an iterator > indeed -- sorry about that -- the "rangeiterator" object, that is. The point here is that there are a bunch of builtin types that are TIIPs > Please don't confuse iterators and iterables: > indeed, that is a distinction that should be kept clear. Thanks -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Dec 13 01:09:13 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 12 Dec 2014 19:09:13 -0500 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: Message-ID: On 12/12/2014 4:24 PM, Alexander Belopolsky wrote: > > On Fri, Dec 12, 2014 at 4:14 PM, Terry Reedy > > wrote: > > I propose that map.__next__ convert StopIteration raised by func to > RuntimeError > > > Aren't we stepping on a slippery slope here? What about say > > >>> def f(x, y): > ... if x > 2: > ... raise StopIteration > ... return x + y I consider this a wretched 'function'. If an input value is outside the function's domain, making 'return value' impossible, the function should raise TypeError or ValueError. As it is, it mixes a function with a hard-coded takewhile. (A similar comment applies to the example I copied from Oscar.) The two 'functions' should be kept separate. >>> import itertools as its >>> list(its.accumulate(its.takewhile(lambda x: x <= 2, range(10)), lambda x,y: x+y)) [0, 1, 3] > >>> import itertools > >>> itertools.accumulate(range(10), f) > > >>> list(_) > [0, 1, 3] To me, this is a bug. accumulate should accumulate until the iterable is exhauted. (Or the doc should change to say that it accumulates until iterable is exhausted or func raises StopIteration.) If accumulate cannot continue, I think it should raise something other than StopIteration. The doc says that accumulate is equivalent to def accumulate(iterable, func=operator.add): it = iter(iterable) total = next(it) yield total for element in it: yield func(total, element) In 3.5, the StopIteration raised by f above will become RuntimeError. If accumulate.__next__ is not changed to match, then the equivalent would have to be given as def accumulate(iterable, func=operator.add): it = iter(iterable) total = next(it) yield total for element in it: try: yield func(total, element) except StopIteration: return Yes, similar considerations apply to all the itertools classes that call user functions: dropwhile, filterfalse, groupby, starmap, and takewhile. "itertools.dropwhile(predicate, iterable) Make an iterator that drops elements from the iterable as long as the predicate is true; afterwards, returns every element." If predicate raises before becoming false, dropwhile cannot 'return every element after'. Should dropwhile simply pass through StopIteration, falsely saying that there are no more elements, or should it convert StopIteration to an exception that says 'something is wrong'? (It could also ignore exceptions from predicate, but I do not suggest that.) -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Sat Dec 13 01:36:55 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 13 Dec 2014 13:36:55 +1300 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: Message-ID: <548B8A27.8030504@canterbury.ac.nz> Terry Reedy wrote: > PEP 479 reverses > this acceptance by having generator.__next turn StopIteration raised in > a user-written generator function body into a RuntimeError. I propose > that other builtin iterator.__next__ methods that execute a passed in > function do the same. Maybe also any user-written function whose name isn't "__next__"? -- Greg From chris.barker at noaa.gov Sat Dec 13 01:40:14 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 12 Dec 2014 16:40:14 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> References: <20141210141858.GA20332@ando.pearwood.info> <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> Message-ID: On Fri, Dec 12, 2014 at 3:22 PM, Andrew Barnert wrote: > > 1) It's not clear what is meant by an "iterator" -- when we use the term > is seems like we are talking about a type, when we are really talking about > "anything that conforms to the iterator protocol", so I'm going to > introduce a new term: TIIP (Type Implementing the Iterator Protocol) -- > maybe that's too cute, but role with it for now. > > > When you're writing actual code, you can test this with > isinstance(collections.abc.Iterator). Of course ABCs don't _perfectly_ > cover their related protocols (in this case, if you create a type that has > __iter__ and __next__ but __iter__ doesn't return self--or, of course, if > you just register some random type with Iterator explicitly--you've created > a subclass of Iterator that isn't a subtype of the iterator protocol). But > I think it still makes sense to use Iterator as shorthand for "type that > subclasses Iterator" with the implied "is an actual subtype of the semantic > type intended by the Iterator class". > this issue here is that inteh PEP, and in this list, it seems "iterator" was used to describe classes with __init__ and __next__ methods, AS APPOSED to the things returned from functions with a yield in them... and: In [12]: g = (i for i in range(10)) In [13]: isinstance(it, collections.abc.Iterator) Out[13]: True and In [15]: def gf(): ....: yield 1 ....: yield 2 In [16]: gen = gf() In [17]: isinstance(gen, collections.abc.Iterator) Out[17]: True So generators ARE iterators, but his definition and yet the contrast was being made.... > This kind of shorthand is used regularly in discussing types in every OO > language, not just Python. > sure, but Python is dynamically types, so "conforms to a protocol" not at all teh same is "isinstance", even though ABCS have made this closer... > In Python, the term "Iterator" is just as consistent and meaningful as in > all these other languages. The fact that some people confuse iterables and > iterators isn't a reason to abandon this simplicity. > actually, the confusion between iterables and iterators want the issue I as trying to address -- but the confusion with a sentence like "Under this proposal, generators and iterators would be distinct, but related, concepts" That has been removed from the PEP -- I think it's more clean now in any case. But yes, I could have written all that and stuck with plain old "iterator" instead of making up "TIIP" Especially since it doesn't even help to solve the problem--as we've just > seen in this very message I'm replying to, it's just as easy for someone to > mistakenly use TIIP to describe "range" as it is for them to mistakenly use > "Iterator". > My bad -- typing too fast! And this was never about the itterable vs iterator distinction anyway. > (In fact, it's arguably worse, because what would you call an Iterable > under the same naming scheme but Type Implementing Iterable Protocol, or > TIIP?) > I was not trying to mingle iterable and iterator -- really I wan't :-) > The things we need to be clear about here are the things that _dont't_ > have an official name. In particular, the thing that's built from calling a > generator function or evaluating a generator expression and used by the > generator. > now I am confused: In [18]: g = ( i for i in range(10) ) In [19]: type(g) Out[19]: generator how is that not a "generator"? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Dec 13 02:22:15 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 12 Dec 2014 17:22:15 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> Message-ID: <2BA09641-4643-4F4C-A114-5B369C607C6A@yahoo.com> On Dec 12, 2014, at 16:40, Chris Barker wrote: > On Fri, Dec 12, 2014 at 3:22 PM, Andrew Barnert wrote: >> >>> 1) It's not clear what is meant by an "iterator" -- when we use the term is seems like we are talking about a type, when we are really talking about "anything that conforms to the iterator protocol", so I'm going to introduce a new term: TIIP (Type Implementing the Iterator Protocol) -- maybe that's too cute, but role with it for now. >> >> When you're writing actual code, you can test this with isinstance(collections.abc.Iterator). Of course ABCs don't _perfectly_ cover their related protocols (in this case, if you create a type that has __iter__ and __next__ but __iter__ doesn't return self--or, of course, if you just register some random type with Iterator explicitly--you've created a subclass of Iterator that isn't a subtype of the iterator protocol). But I think it still makes sense to use Iterator as shorthand for "type that subclasses Iterator" with the implied "is an actual subtype of the semantic type intended by the Iterator class". > > this issue here is that inteh PEP, and in this list, it seems "iterator" was used to describe classes with __init__ and __next__ methods, AS APPOSED to the things returned from functions with a yield in them... But the thing returned by a function without yield in it is an instance of the built-in class generator, which is a subtype of Iterator. Are you trying to make a distinction between iterator _types_ and iterator type _instances_? If so, TIIP doesn't seem to help very much. > and: > > In [12]: g = (i for i in range(10)) > > In [13]: isinstance(it, collections.abc.Iterator) > Out[13]: True > > and > > In [15]: def gf(): > ....: yield 1 > ....: yield 2 > > In [16]: gen = gf() > > In [17]: isinstance(gen, collections.abc.Iterator) > Out[17]: True > > So generators ARE iterators, but his definition and yet the contrast was being made.... Yes, there was some unfortunate wording in the earlier version that Guido already acknowledged and promised to fix well before your message. >> This kind of shorthand is used regularly in discussing types in every OO language, not just Python. > > sure, but Python is dynamically types, so "conforms to a protocol" not at all teh same is "isinstance", even though ABCS have made this closer... The whole point of ABCs is to allow you to define protocols in-language. Other dynamic OO languages like Smalltalk and ObjC have similar concepts. >> In Python, the term "Iterator" is just as consistent and meaningful as in all these other languages. The fact that some people confuse iterables and iterators isn't a reason to abandon this simplicity. > > actually, the confusion between iterables and iterators want the issue I as trying to address -- but the confusion with a sentence like > > "Under this proposal, generators and iterators would be distinct, but related, concepts" > > That has been removed from the PEP -- I think it's more clean now in any case. > > But yes, I could have written all that and stuck with plain old "iterator" instead of making up "TIIP" > >> Especially since it doesn't even help to solve the problem--as we've just seen in this very message I'm replying to, it's just as easy for someone to mistakenly use TIIP to describe "range" as it is for them to mistakenly use "Iterator". > > My bad -- typing too fast! And this was never about the itterable vs iterator distinction anyway. > >> (In fact, it's arguably worse, because what would you call an Iterable under the same naming scheme but Type Implementing Iterable Protocol, or TIIP?) > > I was not trying to mingle iterable and iterator -- really I wan't :-) > > >> The things we need to be clear about here are the things that _dont't_ have an official name. In particular, the thing that's built from calling a generator function or evaluating a generator expression and used by the generator. > > now I am confused: > > In [18]: g = ( i for i in range(10) ) > > In [19]: type(g) > Out[19]: generator > > how is that not a "generator"? You've skipped the "used by the generator" part of that sentence. Of course g is an instance of generator (and therefore also an iterator). But you're not defining the generator type here--that already exists as a builtin type. What you're defining is a thing made up of a code object and a frame, which is used by that generator instance to do its generating. That thing is not a generator, or a generator type (or an iterator instance or type), or a __next__ method, or anything else with a name. (I don't know if that's the thing Guido was trying to describe as not an iterator, but I don't think that's too important, since he's already acknowledged that the distinction he was trying to make isn't important anyway.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Dec 13 02:32:31 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 12 Dec 2014 17:32:31 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: <2BA09641-4643-4F4C-A114-5B369C607C6A@yahoo.com> References: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> <2BA09641-4643-4F4C-A114-5B369C607C6A@yahoo.com> Message-ID: <548B972F.90607@stoneleaf.us> On 12/12/2014 05:22 PM, Andrew Barnert wrote: > On Dec 12, 2014, at 16:40, Chris Barker wrote: >> >> this issue here is that inteh PEP, and in this list, it seems "iterator" was used to describe classes with __init__ >> and __next__ methods, AS APPOSED to the things returned from functions with a yield in them... > > But the thing returned by a function without yield in it is an instance of the built-in class generator, which is a > subtype of Iterator. Um, what? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From tjreedy at udel.edu Sat Dec 13 03:35:03 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 12 Dec 2014 21:35:03 -0500 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: <548B8A27.8030504@canterbury.ac.nz> References: <548B8A27.8030504@canterbury.ac.nz> Message-ID: On 12/12/2014 7:36 PM, Greg Ewing wrote: > Terry Reedy wrote: "I have always taken this to mean that [next and __next__] are the only functions that should raise StopIteration," ... >> PEP 479 reverses this acceptance by having generator.__next turn >> StopIteration raised in a user-written generator function body into a >> RuntimeError. I propose that other builtin iterator.__next__ methods >> that execute a passed in function do the same. > > Maybe also any user-written function whose > name isn't "__next__"? That such functions should catch StopIteration (most likely from next) is implied in the paragraph's opening sentence, repeated above. What the replacement should be is a different matter. There are often better choices than RuntimeError. -- Terry Jan Reedy From raymond.hettinger at gmail.com Sat Dec 13 07:10:58 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 12 Dec 2014 22:10:58 -0800 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: <548B8A27.8030504@canterbury.ac.nz> References: <548B8A27.8030504@canterbury.ac.nz> Message-ID: <91250724-CB2F-43B9-8727-53EA4E6D8914@gmail.com> > On Dec 12, 2014, at 4:36 PM, Greg Ewing wrote: > > Terry Reedy wrote: >> PEP 479 reverses this acceptance by having generator.__next turn StopIteration raised in a user-written generator function body into a RuntimeError. I propose that other builtin iterator.__next__ methods that execute a passed in function do the same. > > Maybe also any user-written function whose > name isn't "__next__"? That does seem to be where this is headed :-( I'm -1 on this proposal. Generators are part of the language internals and language spec, so Guido can reasonably decide to take this in whatever direction he wants. In contrast, now you're moving on to parts of the language library that merely call functions and return results. It is not the responsibility of accumulate(), filter(), map(), or any other higher-order functions to impose rules about what those functions are allowed to do -- they can take as much time as they want, they can hold the GIL, they can manipulate signals, they can use tons of memory, they can return scalar values or complex objects, and they can raise any exception, but now StopIteration would become an event that gets special treatment. Further, this new made-up rule (for which there is zero demonstrated need in any language I know) would have to be applied throughout the standard library and possibly be extended to third-party code. It would be yet another language idiosyncrasy that would have to be learned, remembered, and StackOverflowed about for an eternity. In the case of generators, the PEP 479 rule is easily applied (not hard to search for or to mitigate). In contrast, the "fix" in this case would need to be applied to the *called* functions or their callees, possibly far removed from the map/filter/accumulate call. If that function is in third-party code or in a C-library, then the mitigation would require redesigning the call logic completely (not fun) or to wrap the function in something transforms a StopIteration into a custom exception and then re-catches the custom exception upstream from the higher-order function (also not fun). For anyone who has tested code that is currently working correctly but becomes broken by this proposal, the simplest mitigation will be for them to write their own variants of map/filter/accumulate/dropwhile/takewhile/groupby/starmap that just ignore this proposal and restore the behavior that has been happily in place for ages. Raymond From ethan at stoneleaf.us Sat Dec 13 07:46:16 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 12 Dec 2014 22:46:16 -0800 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: <91250724-CB2F-43B9-8727-53EA4E6D8914@gmail.com> References: <548B8A27.8030504@canterbury.ac.nz> <91250724-CB2F-43B9-8727-53EA4E6D8914@gmail.com> Message-ID: <548BE0B8.8080400@stoneleaf.us> On 12/12/2014 10:10 PM, Raymond Hettinger wrote: > I'm -1 on this proposal. > > Generators are part of the language internals and language spec, so Guido > can reasonably decide to take this in whatever direction he wants. In > contrast, now you're moving on to parts of the language library that merely > call functions and return results. > > It is not the responsibility of accumulate(), filter(), map(), or any other > higher-order functions to impose rules about what those functions are allowed > to do [...] Completely agreed. While the problem (at least in some views, my own included) is really about how next() treats an empty iterable, the biggest reason against changing how next() works is because of all the correctly written code that would be broken (by "correct" I mean code that already guards against next() raising a StopIteration). Conversely, the biggest reason for changing just generators (again, at least in my view ;) is that when writing a generator it's easy to forget that you are actually mucking about with internals and should be guarding against an unexpected flow control exception occuring, as it were, out of the blue. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From abarnert at yahoo.com Sat Dec 13 09:23:03 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 13 Dec 2014 00:23:03 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: <548B972F.90607@stoneleaf.us> References: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> <2BA09641-4643-4F4C-A114-5B369C607C6A@yahoo.com> <548B972F.90607@stoneleaf.us> Message-ID: <504016C2-B7DA-4359-85EF-F43F5331B5B9@yahoo.com> On Dec 12, 2014, at 17:32, Ethan Furman wrote: > On 12/12/2014 05:22 PM, Andrew Barnert wrote: >> On Dec 12, 2014, at 16:40, Chris Barker wrote: >>> >>> this issue here is that inteh PEP, and in this list, it seems "iterator" was used to describe classes with __init__ >>> and __next__ methods, AS APPOSED to the things returned from functions with a yield in them... >> >> But the thing returned by a function without yield in it is an instance of the built-in class generator, which is a >> subtype of Iterator. > > Um, what? Sorry, autocorrect typo. The thing returned by a function *with a* (not *without*) yield in it is a generator. And generator is a class with __iter__ and __next__. Of course an individual generator *instance* isn't such a type, but that's no different from the fact that 2 isn't a number type. From oscar.j.benjamin at gmail.com Sat Dec 13 13:23:29 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sat, 13 Dec 2014 12:23:29 +0000 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: <91250724-CB2F-43B9-8727-53EA4E6D8914@gmail.com> References: <548B8A27.8030504@canterbury.ac.nz> <91250724-CB2F-43B9-8727-53EA4E6D8914@gmail.com> Message-ID: On 13 December 2014 at 06:10, Raymond Hettinger wrote: > > In contrast, the "fix" in this case would need to be applied to the *called* functions or their callees, possibly far removed from the map/filter/accumulate call. If that function is in third-party code or in a C-library, then the mitigation would require redesigning the call logic completely (not fun) or to wrap the function in something transforms a StopIteration into a custom exception and then re-catches the custom exception upstream from the higher-order function (also not fun). Is it a common pattern to raise StopIteration from some deeply nested function (that isn't itself a __next__ method) in order to terminate iteration at some far removed level? I think that PEP 479 largely breaks that pattern if it is since it will no longer work if any generators are involved. Oscar From oscar.j.benjamin at gmail.com Sat Dec 13 13:27:45 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sat, 13 Dec 2014 12:27:45 +0000 Subject: [Python-ideas] PEP 479 and take() In-Reply-To: <5488BE54.8040809@canterbury.ac.nz> References: <5488BE54.8040809@canterbury.ac.nz> Message-ID: On 10 December 2014 at 21:42, Greg Ewing wrote: >> On Wed, Dec 10, 2014 at 4:05 AM, Oscar Benjamin >> > wrote: > >> >> >> class TakeError(Exception): >> pass >> >> def take(iterator, n=None): >> if n is None: >> try: >> return next(iterator) >> except StopIteration: >> raise TakeError >> else: >> return tuple(take(iterator) for _ in range(n)) > > > Won't that just shift the problem from leaked StopIterations > to leaked TakeErrors? No because TakeError is an Error whereas StopIteration as a flow control construct (as Ethan pointed out in another thread). On one level it really doesn't matter what exception it is as long as it's not StopIteration since StopIteration is caught in unexpected places e.g. a for-loop. Of course introducing a completely new error mitigates the possibility of masking some other error from the underlying iterator. Oscar From oscar.j.benjamin at gmail.com Sat Dec 13 13:45:36 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sat, 13 Dec 2014 12:45:36 +0000 Subject: [Python-ideas] PEP 479 and take() In-Reply-To: References: Message-ID: On 10 December 2014 at 18:35, Guido van Rossum wrote: > Hm... The type of take() seems schizophrenic: take() with one argument > returns a single item, while take() with a count returns a tuple of that > many items. It would be better if these were two separate functions. Fair enough. It seemed like a natural extension to me but since it can easily be done with a list comprehension it's not really necessary. > Other > than that, it's a simple function that people can easily code up themselves > (or perhaps there's already a variant in itertools :-). It's not hard to write your own take(). It's also not hard to call next() properly and catch the StopIteration provided you're aware of the danger that a loose StopIteration poses (and never make any mistakes!). I've made the mistake before in real code and I've seen plenty of experienced pythoneers making it when posting snippets on this list. I've tried to point it out where possible e.g.: https://mail.python.org/pipermail/python-ideas/2013-February/019627.html I guess it's just easy to reach into builtins looking for take() and then pull out next() because take() isn't there. The point is that while next() makes perfect sense as the canonical way to call __next__() it is often inappropriate as an end user function where the user simply wants to get items from an iterator. i.e. when you're just using iterators and not consciously implementing them next() does the wrong thing. Consider the different uses of next(): 1) You want to leak StopIteration. 2) You want to catch StopIteration. 3) You know that StopIteration will never be raised. 4) You haven't considered the StopIteration. If you had both next() and take() to choose from then the only time next() would be preferable is when you want to leak StopIteration (a pattern that is now largely broken by PEP 479). > BTW did you know that next(iterator, default) returns default if the > iterator is exhausted? IOW this will never raise StopIteration. It's similar > to dict.get(key, default) or getattr(obj, attrname, default). I more often find that I want an error than a default value and of course being able to detect the default requires knowing that it cannot be returned by the iterator. You can't always use next() this way but when you do of course there is no problem. I don't really need the dict.get method if I want an error since I can simply write dict[key]. The point of PEP 479 is about silently masking an error where loudly reporting it is considered preferable: in this case the error is easy to avoid for someone who understands everything perfectly and is explicitly thinking about it. Oscar From steve at pearwood.info Sat Dec 13 14:36:22 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 14 Dec 2014 00:36:22 +1100 Subject: [Python-ideas] Generators are iterators In-Reply-To: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> References: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> Message-ID: <20141213133622.GH20332@ando.pearwood.info> On Fri, Dec 12, 2014 at 03:22:32PM -0800, Andrew Barnert wrote: > In Python, the term "Iterator" is just as consistent and meaningful as > in all these other languages. The fact that some people confuse > iterables and iterators isn't a reason to abandon this simplicity. [...] +1 to this. But: > The things we need to be clear about here are the things that _dont't_ > have an official name. In particular, the thing that's built from > calling a generator function or evaluating a generator expression and > used by the generator.__next__ method is not a generator, a generator > function, a generator function body, an iterator, or anything else > with a name in the language. And that's what's confusing people. I don't know what thing you are referring to. If I write this: py> def gen(): ... yield 1 ... py> it = gen() then `gen` is a function. It's a specific kind of function that uses "yield", and the name for that is a generator function. Informally, sometimes people call it a "generator", which is a bad habit due to the possibility of confusing `gen` with `it`, but since it is quite rare to be in a position where such confusion can occur (PEP 479 and related discussions not withstanding), we can often get away with such sloppy terminology. Also, `it` is an iterator. It's also a generator: py> type(it) The same applies to generator expressions: py> type(c for c in "aaa") So I'm not sure which mysterious object you are referring to that doesn't have a standard name. Can you give a concrete example? -- Steven From steve at pearwood.info Sat Dec 13 15:45:08 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 14 Dec 2014 01:45:08 +1100 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: Message-ID: <20141213144508.GJ20332@ando.pearwood.info> On Fri, Dec 12, 2014 at 04:14:07PM -0500, Terry Reedy wrote: > This proposal comes from Oscar Benjamin's comments on the PEP in the > 'Generator are iterators' thread. In one post, he gave this example. > > >>> def func(x): > ... if x < 0: > ... raise StopIteration > ... return x ** 2 > ... > >>> it = map(func, [1, 2, 3, -1, 2]) > >>> list(it) > [1, 4, 9] > >>> list(it) # map continues to yield values... > [4] > > Function func violates what I think should be the guideline. The term used by the docs is that it is "broken", which in this context has a technical meaning. An iterator is broken if it fails the rule that once it raises StopIteration, it will continue to always raise StopIteration. Whether or not map() and filter() return a "broken iterator" is completely at the mercy of the user's function. func() can silently break the iterator in many different ways: - it may call os._exit() or os.abort() - it may call threaded code which deadlocks - it may call time.sleep(2**10000) - it may enter an infinite loop to mention just a few. The documentation's promise that map() will return an iterator that "applies function to every item of iterable" is not an unconditional promise. It cannot possibly be, and neither map() nor Python can detect every possible failure condition in advance. It is not Python's responsibility to police that all iterators are non-broken. That is the responsibility of the coder. If you write func() such that it "breaks" map(), then you either have a good reason for doing so, or you are responsible for your own actions. We are all consenting adults here. Let's not complicate things in a futile attempt to prevent people from shooting themselves in the foot. There's a thread on python-dev at the moment decrying the difficulty of writing polylingual Python 2 + 3 code and how some people find it sucks all the joy out of writing Python code. Every backwards incompatible change we add just makes it more difficult to deal with the 2/3 transition. New features are an incentive to upgrade to 3.x. This is not an incentive to upgrade, but it will make it harder to write and understand polylingual iterator code. Gratuitously fixing perceived weaknesses in the iterator protocol which have been there since it was first introduced will, in my opinion, cause more pain than benefit. In another thread, Nick wrote to Oscar: The problem you think PEP 479 is trying to solve *is* the one where the discussion started, but it is *not* the one where it ended when Guido accepted PEP 479. The problems PEP 479 solves are generator specific - they have nothing to do with the iterator protocol in general. Let's leave the iterator protocol alone. -- Steven From jeanpierreda at gmail.com Sat Dec 13 20:02:35 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sat, 13 Dec 2014 11:02:35 -0800 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: <20141213144508.GJ20332@ando.pearwood.info> References: <20141213144508.GJ20332@ando.pearwood.info> Message-ID: On Sat, Dec 13, 2014 at 6:45 AM, Steven D'Aprano wrote: > It is not Python's responsibility to police that all iterators are > non-broken. [...] > > Let's leave the iterator protocol alone. Terry was suggesting fixing builtin iterators, not fixing all iterators or the iterator protocol. It *is* Python's responsibility to make sure the builtin iterators are not broken. It's fine and to be expected if user code raises StopIteration, but map and filter don't handle it correctly and therefore disobey the iterator protocol, which is a bug in map and filter -- not the callback. -- Devin From abarnert at yahoo.com Sat Dec 13 20:37:23 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 13 Dec 2014 11:37:23 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: <20141213133622.GH20332@ando.pearwood.info> References: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> <20141213133622.GH20332@ando.pearwood.info> Message-ID: On Dec 13, 2014, at 5:36, Steven D'Aprano wrote: > On Fri, Dec 12, 2014 at 03:22:32PM -0800, Andrew Barnert wrote: > >> In Python, the term "Iterator" is just as consistent and meaningful as >> in all these other languages. The fact that some people confuse >> iterables and iterators isn't a reason to abandon this simplicity. > [...] > > +1 to this. > > > But: > >> The things we need to be clear about here are the things that _dont't_ >> have an official name. In particular, the thing that's built from >> calling a generator function or evaluating a generator expression and >> used by the generator.__next__ method is not a generator, a generator >> function, a generator function body, an iterator, or anything else >> with a name in the language. And that's what's confusing people. > > I don't know what thing you are referring to. If I write this: > > py> def gen(): > ... yield 1 > ... > py> it = gen() > > then `gen` is a function. Chris Barker just asked exactly the same question yesterday, except that he trimmed the quote more and used "g" instead of "it" for his example. So I'll just paste the same reply, and you can manually s/g/it/. Of course g is an instance of generator (and therefore also an iterator). But you're not defining the generator type here--that already exists as a builtin type. What you're defining is a thing made up of a code object and a frame, which is used by that generator instance to do its generating. That thing is not a generator, or a generator type (or an iterator instance or type), or a __next__ method, or anything else with a name. (I don't know if that's the thing Guido was trying to describe as not an iterator, but I don't think that's too important, since he's already acknowledged that the distinction he was trying to make isn't important anyway.) > It's a specific kind of function that uses > "yield", and the name for that is a generator function. Informally, > sometimes people call it a "generator", which is a bad habit due to the > possibility of confusing `gen` with `it`, but since it is quite rare to > be in a position where such confusion can occur (PEP 479 and related > discussions not withstanding), we can often get away with such sloppy > terminology. > > Also, `it` is an iterator. It's also a generator: > > py> type(it) > > > The same applies to generator expressions: > > py> type(c for c in "aaa") > > > So I'm not sure which mysterious object you are referring to that > doesn't have a standard name. Can you give a concrete example? > > > > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rosuav at gmail.com Sat Dec 13 20:51:23 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 14 Dec 2014 06:51:23 +1100 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: <20141213144508.GJ20332@ando.pearwood.info> Message-ID: On Sun, Dec 14, 2014 at 6:02 AM, Devin Jeanpierre wrote: > Terry was suggesting fixing builtin iterators, not fixing all iterators or the > iterator protocol. It *is* Python's responsibility to make sure the > builtin iterators are not broken. It's fine and to be expected if user > code raises StopIteration, but map and filter don't handle it > correctly and therefore disobey the iterator protocol, which is a bug > in map and filter -- not the callback. I'm not sure that "fine and to be expected" is necessarily true. If the function given to map() raises StopIteration, I'd say any of these approaches is justifiable: 1) It's an error. Raise something other than StopIteration, so the caller knows something bad happened. (A generator version of map() will do this post-479.) 2) It's to be expected. Consider map() to have now terminated. (In other words, catch it and return.) 3) It's a signal that this one element should not be returned. Suppress it, yield nothing, and go get the next value from the input iterator. Personally, I'd rank them in this order of preference. Consistency with generators is a plus (especially in terms of citing a pure-Python equivalent), and the second and especially third options risk opening up a misfeature - using StopIteration as a weird form of flow control/signalling. But your description implies that option 2 is inherently obvious, which I'm not entirely convinced of; unless this is actually implemented as a generic "this iterator is now exhausted" flag, and next() itself sets and checks this flag, thus automatically enforcing that one StopIteration will result in an infinite stream of StopIterations. ChrisA From ron3200 at gmail.com Sat Dec 13 21:19:55 2014 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 13 Dec 2014 14:19:55 -0600 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> <20141213133622.GH20332@ando.pearwood.info> Message-ID: On 12/13/2014 01:37 PM, Andrew Barnert wrote: >> >I don't know what thing you are referring to. If I write this: >> > >> >py> def gen(): >> >... yield 1 >> >... >> >py> it = gen() >> > >> >then `gen` is a function. > Chris Barker just asked exactly the same question yesterday, except that he trimmed the quote more and used "g" instead of "it" for his example. So I'll just paste the same reply, and you can manually s/g/it/. > > Of course g is an instance of generator (and therefore also an iterator). > > But you're not defining the generator type here--that already exists as a builtin type. What you're defining is a thing made up of a code object and a frame, which is used by that generator instance to do its generating. That thing is not a generator, or a generator type (or an iterator instance or type), or a __next__ method, or anything else with a name. > > (I don't know if that's the thing Guido was trying to describe as not an iterator, but I don't think that's too important, since he's already acknowledged that the distinction he was trying to make isn't important anyway.) >>> def gen(): ... yield 1 ... >>> type(gen) >>> gen It's a function with a flag set. >>> gen.__code__.co_flags 99 >>> dis(gen) 2 0 LOAD_CONST 1 (1) 3 YIELD_VALUE 4 POP_TOP 5 LOAD_CONST 0 (None) 8 RETURN_VALUE Where a function's flags would be this... >>> def foo(): ... return 1 ... >>> foo.__code__.co_flags 67 So, it the answer is yes and no. It's a function object, with an altered behaviour. It seems like the flag may be checked at call time, and then a new function generator instance is returned.. >>> gi = gen() >>> type(gi) >>> gi And the original function body is moved to it's gi_code attribute. >>> dis(gi.gi_code) 2 0 LOAD_CONST 1 (1) 3 YIELD_VALUE 4 POP_TOP 5 LOAD_CONST 0 (None) 8 RETURN_VALUE I think the send() method, a builtin C routine, handles all the particulars of running gi_code with gi_frame. (I'd have to look to be sure.) I think if function definitons with yield in them created generator_instance_factory objects instead of function objects, things would be easier to explain. What it comes down to, is they aren't normal functions and it's a bit confusing to think of them that way. Cheers, Ron From guido at python.org Sat Dec 13 21:48:38 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 13 Dec 2014 12:48:38 -0800 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: <91250724-CB2F-43B9-8727-53EA4E6D8914@gmail.com> References: <548B8A27.8030504@canterbury.ac.nz> <91250724-CB2F-43B9-8727-53EA4E6D8914@gmail.com> Message-ID: On Fri, Dec 12, 2014 at 10:10 PM, Raymond Hettinger < raymond.hettinger at gmail.com> wrote: > > > > On Dec 12, 2014, at 4:36 PM, Greg Ewing > wrote: > > > > Terry Reedy wrote: > >> PEP 479 reverses this acceptance by having generator.__next turn > StopIteration raised in a user-written generator function body into a > RuntimeError. I propose that other builtin iterator.__next__ methods that > execute a passed in function do the same. > > > > Maybe also any user-written function whose > > name isn't "__next__"? > > That does seem to be where this is headed :-( > > I'm -1 on this proposal. > Me too. PEP 479 was a clear win. __next__ methods are a much murkier area and we should not mess with them. > Generators are part of the language internals and language spec, so Guido > can reasonably decide to take this in whatever direction he wants. In > contrast, now you're moving on to parts of the language library that merely > call functions and return results. > > It is not the responsibility of accumulate(), filter(), map(), or any > other higher-order functions to impose rules about what those functions are > allowed to do -- they can take as much time as they want, they can hold > the GIL, they can manipulate signals, they can use tons of memory, they can > return scalar values or complex objects, and they can raise any exception, > but now StopIteration would become an event that gets special treatment. > > Further, this new made-up rule (for which there is zero demonstrated need > in any language I know) would have to be applied throughout the standard > library and possibly be extended to third-party code. It would be yet > another language idiosyncrasy that would have to be learned, remembered, > and StackOverflowed about for an eternity. > > In the case of generators, the PEP 479 rule is easily applied (not hard to > search for or to mitigate). In contrast, the "fix" in this case would need > to be applied to the *called* functions or their callees, possibly far > removed from the map/filter/accumulate call. If that function is in > third-party code or in a C-library, then the mitigation would require > redesigning the call logic completely (not fun) or to wrap the function in > something transforms a StopIteration into a custom exception and then > re-catches the custom exception upstream from the higher-order function > (also not fun). > > For anyone who has tested code that is currently working correctly but > becomes broken by this proposal, the simplest mitigation will be for them > to write their own variants of > map/filter/accumulate/dropwhile/takewhile/groupby/starmap that just ignore > this proposal and restore the behavior that has been happily in place for > ages. > When it comes to map() and filter() and friends the situation is murkier yet. In Python 2, raising StopIteration in the function passed to map/filter was just bubbled out. In Python 3, where these have become lazy, a StopIteration raised by the function terminates the iteration. Was that intentional? I doubt it. Is it now a feature? Maybe (though it would make more sense to use something like takewhile). Could it mask a subtle bug? Probably (but it doesn't sound like a common situation -- map/filter functions are usually small and simple). Should we "fix" it? I don't think so. It's not sufficiently broken, the fix would lead us onto a slippery slope. Enough is enough. > Raymond > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Dec 13 22:27:56 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 14 Dec 2014 08:27:56 +1100 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> <20141213133622.GH20332@ando.pearwood.info> Message-ID: On Sun, Dec 14, 2014 at 7:19 AM, Ron Adam wrote: > I think if function definitons with yield in them created > generator_instance_factory objects instead of function objects, things would > be easier to explain. What it comes down to, is they aren't normal > functions and it's a bit confusing to think of them that way. They're no more different than, say, a decorated function that returns a different function. But if you want a different way to explain them, here's a similar way to craft an iterator: class seq_iter: def __init__(self, func, *args, **kw): self.func = func self.args = args self.kw = kw self.state = 0 def __iter__(self): return self def __next__(self): if self.state == -1: raise StopIteration ret = self.func(self.state, self, *self.args, **self.kw) self.state += 1 if ret is not self: return ret self.state = -1 raise StopIteration def sequential_iterator(func): @functools.wraps(func) def inner(*args, **kw): return seq_iter(func, *args, **kw) return inner ##### Example usage ##### @sequential_iterator def ten_squares(idx, finished): if idx == 10: return finished return idx*idx The function returned by the decorator isn't the one decorated, but it's clearly still a function. No reason for it to be a unique type. This is similar to the relationship between generator functions and the iterators returned when you call them. It's an alternative way to write an iterator class, dealing with lots of the bookkeeping for you. ChrisA From tjreedy at udel.edu Sat Dec 13 23:31:36 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 13 Dec 2014 17:31:36 -0500 Subject: [Python-ideas] Add stop=exception option to next() (was Re: PEP 479 and take()) In-Reply-To: References: Message-ID: On 12/13/2014 7:45 AM, Oscar Benjamin wrote: > On 10 December 2014 at 18:35, Guido van Rossum wrote: > If you had both next() and take() to choose from then the only time > next() would be preferable is when you want to leak StopIteration (a > pattern that is now largely broken by PEP 479). > >> BTW did you know that next(iterator, default) returns default if the >> iterator is exhausted? IOW this will never raise StopIteration. It's similar >> to dict.get(key, default) or getattr(obj, attrname, default). > > I more often find that I want an error than a default value and of I once proposed, either here or on python-list, and propose again, that the signature of next be expanded so that the user could specify the ending exception. If possible, the stop object could either be an exception class, which would be called with a generic message, or an exception instance. Then the awkward try: item = next(it) except StopIteration: raise ValueError('iterable must not be empty') from None would simply be, for instance, item = next(it, stop=ValueError('iterable must not be empty')) The doc for next could say that, except in __next__ methods, the default stop exception, StopIteration, should be either overriden or caught, to prevent it from leaking out of the containing function. No need for a new builtin to change the exception. -- Terry Jan Reedy From solipsis at pitrou.net Sat Dec 13 23:36:33 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 13 Dec 2014 23:36:33 +0100 Subject: [Python-ideas] Add stop=exception option to next() (was Re: PEP 479 and take()) References: Message-ID: <20141213233633.058b66f9@fsol> On Sat, 13 Dec 2014 17:31:36 -0500 Terry Reedy wrote: > On 12/13/2014 7:45 AM, Oscar Benjamin wrote: > > On 10 December 2014 at 18:35, Guido van Rossum wrote: > > > If you had both next() and take() to choose from then the only time > > next() would be preferable is when you want to leak StopIteration (a > > pattern that is now largely broken by PEP 479). > > > >> BTW did you know that next(iterator, default) returns default if the > >> iterator is exhausted? IOW this will never raise StopIteration. It's similar > >> to dict.get(key, default) or getattr(obj, attrname, default). > > > > I more often find that I want an error than a default value and of > > I once proposed, either here or on python-list, and propose again, > that the signature of next be expanded so that the user could specify > the ending exception. If possible, the stop object could either be an > exception class, which would be called with a generic message, or an > exception instance. > > Then the awkward > > try: > item = next(it) > except StopIteration: > raise ValueError('iterable must not be empty') from None I don't remember ever needing to write such code. Regards Antoine. From tjreedy at udel.edu Sun Dec 14 00:37:16 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 13 Dec 2014 18:37:16 -0500 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: <548B8A27.8030504@canterbury.ac.nz> <91250724-CB2F-43B9-8727-53EA4E6D8914@gmail.com> Message-ID: On 12/13/2014 3:48 PM, Guido van Rossum wrote: > On Fri, Dec 12, 2014 at 10:10 PM, Raymond Hettinger > > wrote: > > On Dec 12, 2014, at 4:36 PM, Greg Ewing > wrote: > > > > Terry Reedy wrote: > >> PEP 479 reverses this acceptance by having generator.__next turn StopIteration raised in a user-written generator function body into a RuntimeError. I propose that other builtin iterator.__next__ methods that execute a passed in function do the same. > > > > Maybe also any user-written function whose > > name isn't "__next__"? > > That does seem to be where this is headed :-( > > I'm -1 on this proposal. > > Me too. PEP 479 was a clear win. __next__ methods are a much murkier > area and we should not mess with them. After reading other responses, even before reading this one, I decided that Alexander Belopolsky's initial comment "Aren't we stepping on a slippery slope here?" was correct, that generator.__next__ and generator functions are a sufficiently unique case to get unique treatment, and that I would withdraw my proposal. [The rest of Guido's post, which I have snipped, further explicates the slipperness.] I am also cognizant that creating incompatibilities between even 3.4 and 3.5 requires strong justification. So I consider this trial balloon popped. My proposal was a response to the problem of *silent breakage* of 'transform iterators' that take as arguments an iterator and transform function and produce a new, transformed sequence. While breaking functions with bad input generally falls under 'consenting adults', a transform function raising StopIteration is special in that it can easily lead to silent rather than noisy breakage. I have two alternate proposals. 1. In the doc entry for StopIteration, explicitly recommend against raising StopIteration outside of __next__ methods on the basis that doing so can lead to silent breakage of iterators. Then writing functions that do so anyway or passing them into iterators will clearly be a 'consenting adults' issue. 2. Make it easier to avoid accidentally leaking StopIteration when using next() by adding an option to have it raise a different exception. I already posted this idea to a different thread. -- Terry Jan Reedy From raymond.hettinger at gmail.com Sun Dec 14 00:56:15 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 13 Dec 2014 15:56:15 -0800 Subject: [Python-ideas] Add stop=exception option to next() (was Re: PEP 479 and take()) In-Reply-To: <20141213233633.058b66f9@fsol> References: <20141213233633.058b66f9@fsol> Message-ID: > On Dec 13, 2014, at 2:36 PM, Antoine Pitrou wrote: > > On Sat, 13 Dec 2014 17:31:36 -0500 > Terry Reedy wrote: > >> I once proposed, either here or on python-list, and propose again, >> that the signature of next be expanded so that the user could specify >> the ending exception. If possible, the stop object could either be an >> exception class, which would be called with a generic message, or an >> exception instance. >> >> Then the awkward >> >> try: >> item = next(it) >> except StopIteration: >> raise ValueError('iterable must not be empty') from None > > I don't remember ever needing to write such code. I've also never needed or desired this. Several people on the list have become obsessed with the notion of "leaking StopIteration" and there seem to be no end of proposals that fight the basic mechanics of "exceptions propagate upwards until caught". Everyone, please, please stop directing so much effort to revise the fundamentals of the language -- it's NOT broken. The iterator protocol and exceptions have been around for a long time. They have been one of Python's greatest success stories (not just in Python, but in many other languages as well). Raymond From tjreedy at udel.edu Sun Dec 14 01:39:25 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 13 Dec 2014 19:39:25 -0500 Subject: [Python-ideas] Add stop=exception option to next() (was Re: PEP 479 and take()) In-Reply-To: <20141213233633.058b66f9@fsol> References: <20141213233633.058b66f9@fsol> Message-ID: On 12/13/2014 5:36 PM, Antoine Pitrou wrote: > On Sat, 13 Dec 2014 17:31:36 -0500 > Terry Reedy wrote: > >> On 12/13/2014 7:45 AM, Oscar Benjamin wrote: >>> On 10 December 2014 at 18:35, Guido van Rossum wrote: >> >>> If you had both next() and take() to choose from then the only time >>> next() would be preferable is when you want to leak StopIteration (a >>> pattern that is now largely broken by PEP 479). >>> >>>> BTW did you know that next(iterator, default) returns default if the >>>> iterator is exhausted? IOW this will never raise StopIteration. It's similar >>>> to dict.get(key, default) or getattr(obj, attrname, default). >>> >>> I more often find that I want an error than a default value and of >> >> I once proposed, either here or on python-list, and propose again, >> that the signature of next be expanded so that the user could specify >> the ending exception. If possible, the stop object could either be an >> exception class, which would be called with a generic message, or an >> exception instance. >> >> Then the awkward >> >> try: >> item = next(it) >> except StopIteration: >> raise ValueError('iterable must not be empty') from None > > I don't remember ever needing to write such code. Others do, whenever the first item of an iterable needs special treatment. And others *have* forgotten to catch StopIteration when calling next(it). The code for reduce *is* written with such code. >>> from functools import reduce as r >>> r(lambda a, b: a+b, []) Traceback (most recent call last): File "", line 1, in r(lambda a, b: a+b, []) TypeError: reduce() of empty sequence with no initial value But the current equivalent code in the doc is buggy because it was not. def reduce(function, iterable, initializer=None): it = iter(iterable) if initializer is None: value = next(it) else: value = initializer for element in it: value = function(value, element) return value >>> reduce(lambda a, b: a+b, []) Traceback (most recent call last): File "C:\Programs\Python34\tem.py", line 11, in reduce(lambda a, b: a+b, []) File "C:\Programs\Python34\tem.py", line 4, in reduce value = next(it) StopIteration The equivalent code now would be try: value = next(it) except StopIteration: raise TypeError("reduce() of empty sequence with no initial value") from None http://bugs.python.org/issue23049? That a core developer would miss this illustrates to me why the addition is needed. With this proposal, the correct equivalent would be value = next(it. stop=TypeError( "reduce() of empty sequence with no initial value")) -- Terry Jan Reedy From steve at pearwood.info Sun Dec 14 02:17:35 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 14 Dec 2014 12:17:35 +1100 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> <20141213133622.GH20332@ando.pearwood.info> Message-ID: <20141214011735.GK20332@ando.pearwood.info> On Sat, Dec 13, 2014 at 11:37:23AM -0800, Andrew Barnert wrote: > On Dec 13, 2014, at 5:36, Steven D'Aprano wrote: > > > On Fri, Dec 12, 2014 at 03:22:32PM -0800, Andrew Barnert wrote: > > > >> In Python, the term "Iterator" is just as consistent and meaningful as > >> in all these other languages. The fact that some people confuse > >> iterables and iterators isn't a reason to abandon this simplicity. > > [...] > > > > +1 to this. > > > > > > But: > > > >> The things we need to be clear about here are the things that _dont't_ > >> have an official name. In particular, the thing that's built from > >> calling a generator function or evaluating a generator expression and > >> used by the generator.__next__ method is not a generator, a generator > >> function, a generator function body, an iterator, or anything else > >> with a name in the language. And that's what's confusing people. > > > > I don't know what thing you are referring to. If I write this: > > > > py> def gen(): > > ... yield 1 > > ... > > py> it = gen() > > > > then `gen` is a function. > > Chris Barker just asked exactly the same question yesterday, except > that he trimmed the quote more and used "g" instead of "it" for his > example. So I'll just paste the same reply, and you can manually > s/g/it/. I read your reply to Chris last night, and it didn't make sense to me then and it still doesn't make sense to me now :-( Hence my request for a concrete example. I am still unclear as to what this mystery unnamed thing is. It's not a generator function, because that is called a generator function. It's not the thing you get when you call a generator function, because that is called a generator or an iterator, depending on context. Surely it's not the generator type, because that has a name: "generator". We always use the same name for the class and its instances, e.g. we talk about "int" the class as well as saying that 123 is an int. The class "int" is not itself an int instance, but we still call it "int". And likewise for "generator". > Of course g is an instance of generator (and therefore also an > iterator). Agreed. But giving examples of things which aren't the mystery entity doesn't really help explain which mystery entity you are talking about :-) > But you're not defining the generator type here--that already exists > as a builtin type. What you're defining is a thing made up of a code > object and a frame, which is used by that generator instance to do its > generating. That thing is not a generator, or a generator type (or an > iterator instance or type), or a __next__ method, or anything else > with a name. Sure the thing you are defining when you write "def gen() ..." is a function? I thought we were in agreement about that. It's a function with a flag set, a.k.a. a generator function. If you're referring to something else, a concrete example will help. Because as we stand now, I don't know whether we're talking about something important or not, or if we are in agreement, disagreement, or merely confusion :-) -- Steven From ron3200 at gmail.com Sun Dec 14 02:33:14 2014 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 13 Dec 2014 19:33:14 -0600 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> <20141213133622.GH20332@ando.pearwood.info> Message-ID: On 12/13/2014 03:27 PM, Chris Angelico wrote: > On Sun, Dec 14, 2014 at 7:19 AM, Ron Adam wrote: >> >I think if function definitons with yield in them created >> >generator_instance_factory objects instead of function objects, things would >> >be easier to explain. What it comes down to, is they aren't normal >> >functions and it's a bit confusing to think of them that way. > They're no more different than, say, a decorated function that returns > a different function. But that is different, because the decorator isn't visible. You can also offer examples of other objects in terms of other objects, they are still other objects. So it raises questions like this ... When should we make a distinction of one object from another? And when shouldn't we? Usually it has more to do with weather or not it makes writing some program easier. I'm not sure changing the type in this case would be useful in that way. > But if you want a different way to explain them, > here's a similar way to craft an iterator: I think it's what the average python user expects that matters. The term generator_function was use quite a lot in the PEP discussion in order to distinguish it from a regular function, and also from the generator returned from it. So it seems to me, that maybe there is something to that. Currently (especially when discussing things here), I think it's important to try to have an accurate model of what is being discussed. Comparisons to similar code is also useful, but for checking the implementation works correctly and consistently. They don't replace the accurate model. That model could be pseudo-code, or python, or a diagram, but it helps to keep in in mind. So, we have a rule we need to remember: If a function has "yield" in it, it will return a generator-object. Ok. So a generator is: a function + that rule. We have a case where we differentiate it in our minds, but the code, type(g) indicates it is not different, but the code actually does do something a bit magical. Yes we can visually examine the body of the function and tell that it's different, but that isn't full proof. That is some of what leads me to think it should maybe be a different type. Possibly a sub-class of a function class, would be the only option. Fortunately generators are not new, so most everyone does understands how to define and use them, just not how the implementation is actually done. But maybe that's ok. Cheers, Ron From steve at pearwood.info Sun Dec 14 02:34:18 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 14 Dec 2014 12:34:18 +1100 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: <548B8A27.8030504@canterbury.ac.nz> <91250724-CB2F-43B9-8727-53EA4E6D8914@gmail.com> Message-ID: <20141214013418.GL20332@ando.pearwood.info> On Sat, Dec 13, 2014 at 06:37:16PM -0500, Terry Reedy wrote: > I have two alternate proposals. > > 1. In the doc entry for StopIteration, explicitly recommend against > raising StopIteration outside of __next__ methods on the basis that > doing so can lead to silent breakage of iterators. Then writing > functions that do so anyway or passing them into iterators will clearly > be a 'consenting adults' issue. I think that is reasonable. > 2. Make it easier to avoid accidentally leaking StopIteration when using > next() by adding an option to have it raise a different exception. I > already posted this idea to a different thread. I think that is adding complexity that isn't needed. -- Steven From rosuav at gmail.com Sun Dec 14 02:38:25 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 14 Dec 2014 12:38:25 +1100 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> <20141213133622.GH20332@ando.pearwood.info> Message-ID: On Sun, Dec 14, 2014 at 12:33 PM, Ron Adam wrote: > The term generator_function was use quite a lot in the PEP discussion in > order to distinguish it from a regular function, and also from the generator > returned from it. So it seems to me, that maybe there is something to that. Many things can be distinguished that are not distinct types. A generator function is a function that has a 'yield' in it. A factory function is a function that constructs and returns a new object. A recursive function is a function which calls itself. All three are functions in every way. ChrisA From steve at pearwood.info Sun Dec 14 02:55:27 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 14 Dec 2014 12:55:27 +1100 Subject: [Python-ideas] Add stop=exception option to next() (was Re: PEP 479 and take()) In-Reply-To: References: <20141213233633.058b66f9@fsol> Message-ID: <20141214015526.GM20332@ando.pearwood.info> On Sat, Dec 13, 2014 at 07:39:25PM -0500, Terry Reedy wrote: [...] > >>try: > >> item = next(it) > >>except StopIteration: > >> raise ValueError('iterable must not be empty') from None > > > >I don't remember ever needing to write such code. > > Others do, whenever the first item of an iterable needs special > treatment. And others *have* forgotten to catch StopIteration when > calling next(it). And if they forget that, what makes you think they will remember to write next(it, stop=ValueError)? > The code for reduce *is* written with such code. > > >>> from functools import reduce as r > >>> r(lambda a, b: a+b, []) > Traceback (most recent call last): > File "", line 1, in > r(lambda a, b: a+b, []) > TypeError: reduce() of empty sequence with no initial value > > But the current equivalent code in the doc is buggy because it was not. One has to expect that pedagogical examples often avoid complicating the code with error handling, and consequently are buggy compared to the real production code. It's almost a truism that an algorithm that takes 5 lines in text books might take 50 lines in production code :-) > def reduce(function, iterable, initializer=None): > it = iter(iterable) > if initializer is None: > value = next(it) > else: > value = initializer > for element in it: > value = function(value, element) > return value > > >>> reduce(lambda a, b: a+b, []) > Traceback (most recent call last): > File "C:\Programs\Python34\tem.py", line 11, in > reduce(lambda a, b: a+b, []) > File "C:\Programs\Python34\tem.py", line 4, in reduce > value = next(it) > StopIteration Seems perfectly reasonable to me. Many user-written functions don't bother to catch exceptions and convert them to a standard exception, instead they just let them bubble up. That's not a bug, it's a feature of Python. Generators are rather special, but for non-generator functions like reduce, StopIteration is just another exception, no different from ValueError, IndexError, UnicodeDecodeError, etc. > The equivalent code now would be > > try: > value = next(it) > except StopIteration: > raise TypeError("reduce() of empty sequence with no initial value") > from None > > http://bugs.python.org/issue23049? I don't think that this example needs to be fixed, but I don't object if you do fix it. > That a core developer would miss this [...] You're assuming that it was an oversight. It's more likely that the author of that code snippet simply didn't care to complicate the code just for the sake of raising the same exception type as the built-in reduce. If you read the docs for reduce, it doesn't make any promises about what exceptions it will raise. Raising ValueError is not a documented part of the API. https://docs.python.org/3/library/functools.html#functools.reduce So the Devil's Advocate might argue that the bug is not in the pure Python code but in the built-in reduce, since it wrongly raises ValueError when it should raise StopIteration. -- Steven From raymond.hettinger at gmail.com Sun Dec 14 03:08:55 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 13 Dec 2014 18:08:55 -0800 Subject: [Python-ideas] Add stop=exception option to next() (was Re: PEP 479 and take()) In-Reply-To: References: <20141213233633.058b66f9@fsol> Message-ID: <91653699-CF7A-4F52-862C-569BCE93AD83@gmail.com> > On Dec 13, 2014, at 4:39 PM, Terry Reedy wrote: > > But the current equivalent code in the doc is buggy because it was not. > > def reduce(function, iterable, initializer=None): > it = iter(iterable) > if initializer is None: > value = next(it) ... > That a core developer would miss this illustrates to me why the addition is needed. With this proposal, the correct equivalent would be A number of people seem to take the "python equivalent" code far too literally. Its purpose is primarily documentation to show the overall logic of reducing. In some cases you can make the "equivalent" more exact, but make the documentation worse because it starts to lose its explanatory value. I'm feeling upset by the never ending flurry of proposals on python-ideas. At their root is doing battle the with the core concept that StopIteration is an exception that can propagate upward like all other exceptions. There seems to be a not-so-secret wish that StopIteration was a sentinel value that can only be checked immediately and can't be propagated. In order words, the proponents seem to reject a core idea in the iterator protocol as implemented in Python for over a decade and as implemented in a number of other languages (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Iterators_and_Generators) or as specified in Design Patterns. In rejecting the notion of StopIteration for control flow, a number of people on this list are going nuts and coming with one random suggestion after another, ignoring the years of success or the experiences of devs in other languages. You guys seem to be just making up new rules about how other people should write programs to express their reasoning. Some of the discussed proposals seem like no more than stylistic pedantry. I wish more people on this list had experience teaching Python. It would give greater appreciation for language simplicity and not introducing twists in the road. At one time, all we had was it.next() which either returned a value or raised StopIteration. Then, next() was added as shortcut for it.__next__(). The relationship was no more special than the relation between len(s) and s.__len__(). Then, next() grew a default argument. This wasn't bad because it paralleled default arguments in dict.get() and dict.pop() -- also it simplified a common pattern for supplying a default. But now, you want to add "stop=TypeError("reduce() of empty sequence with no initial value") which has no precedent and must be accompanied by explanation your admonitions about how you want people to always catch StopIteration immediately. It stops being easily teachable and becomes one more piece of arcana different from the rest of Python, different from the last 15 years of Python iterator thinking, and different from every other iterator protocol implementation in other languages. I do not look forward to teaching this. For Christmas, it would be a nice gift if Guido could put an end to this line of thinking. The conversations are difficult because several of the participants are deeply convinced that there is something fundamentally wrong with StopIteration. I don't think any of them can easily be convinced to the leave the iterator protocol alone and to continue enjoying one of our best success stories. sometimes-the-cure-is-worse-than-the-disease-ly, Raymond P.S. "Leaking StopIteration" is a phrase that I've only heard here -- this seems to be a villain of legend rather than something that arises in the lives of ordinary Python programmers. From guido at python.org Sun Dec 14 03:31:11 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 13 Dec 2014 18:31:11 -0800 Subject: [Python-ideas] Add stop=exception option to next() (was Re: PEP 479 and take()) In-Reply-To: <91653699-CF7A-4F52-862C-569BCE93AD83@gmail.com> References: <20141213233633.058b66f9@fsol> <91653699-CF7A-4F52-862C-569BCE93AD83@gmail.com> Message-ID: On Sat, Dec 13, 2014 at 6:08 PM, Raymond Hettinger < raymond.hettinger at gmail.com> wrote: > For Christmas, it would be a nice gift if Guido could put an end to this > line of thinking. The conversations are difficult because several of the > participants are deeply convinced that there is something fundamentally > wrong with StopIteration. I don't think any of them can easily be > convinced to the leave the iterator protocol alone and to continue enjoying > one of our best success stories. > It ends with PEP 479. I have no intention to let this go any further. I am now going to mute any further threads on the topic, and I recommend you do the same, Raymond. Happy holidays! -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Sun Dec 14 03:46:43 2014 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 13 Dec 2014 20:46:43 -0600 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> <20141213133622.GH20332@ando.pearwood.info> Message-ID: On 12/13/2014 07:38 PM, Chris Angelico wrote: > On Sun, Dec 14, 2014 at 12:33 PM, Ron Adam wrote: >> >The term generator_function was use quite a lot in the PEP discussion in >> >order to distinguish it from a regular function, and also from the generator >> >returned from it. So it seems to me, that maybe there is something to that. > Many things can be distinguished that are not distinct types. A > generator function is a function that has a 'yield' in it. A factory > function is a function that constructs and returns a new object. A > recursive function is a function which calls itself. All three are > functions in every way. I see only two things... functions that do exactly what's in the defined body when called, and generator_functions which does something other than what is defined in the body when called. That seems like quite a difference to me. Cheers, Ron From abarnert at yahoo.com Sun Dec 14 06:43:02 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 13 Dec 2014 21:43:02 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: <20141214011735.GK20332@ando.pearwood.info> References: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> <20141213133622.GH20332@ando.pearwood.info> <20141214011735.GK20332@ando.pearwood.info> Message-ID: <6902865E-BEF3-4DBB-AABA-5260BD54647F@yahoo.com> On Dec 13, 2014, at 17:17, Steven D'Aprano wrote: > On Sat, Dec 13, 2014 at 11:37:23AM -0800, Andrew Barnert wrote: >> On Dec 13, 2014, at 5:36, Steven D'Aprano wrote: >> >>> On Fri, Dec 12, 2014 at 03:22:32PM -0800, Andrew Barnert wrote: >>> >>>> In Python, the term "Iterator" is just as consistent and meaningful as >>>> in all these other languages. The fact that some people confuse >>>> iterables and iterators isn't a reason to abandon this simplicity. >>> [...] >>> >>> +1 to this. >>> >>> >>> But: >>> >>>> The things we need to be clear about here are the things that _dont't_ >>>> have an official name. In particular, the thing that's built from >>>> calling a generator function or evaluating a generator expression and >>>> used by the generator.__next__ method is not a generator, a generator >>>> function, a generator function body, an iterator, or anything else >>>> with a name in the language. And that's what's confusing people. >>> >>> I don't know what thing you are referring to. If I write this: >>> >>> py> def gen(): >>> ... yield 1 >>> ... >>> py> it = gen() >>> >>> then `gen` is a function. >> >> Chris Barker just asked exactly the same question yesterday, except >> that he trimmed the quote more and used "g" instead of "it" for his >> example. So I'll just paste the same reply, and you can manually >> s/g/it/. > > I read your reply to Chris last night, and it didn't make sense to me > then and it still doesn't make sense to me now :-( Hence my request for > a concrete example. You didn't ask for a concrete example, you provided the exact same answer as Chris but with different names. > I am still unclear as to what this mystery unnamed thing is. It's not a > generator function, because that is called a generator function. It's > not the thing you get when you call a generator function, because that > is called a generator or an iterator, depending on context. Again, it's the thing used by the generator instance's __next__ method: the frame and code objects. >> But you're not defining the generator type here--that already exists >> as a builtin type. What you're defining is a thing made up of a code >> object and a frame, which is used by that generator instance to do its >> generating. That thing is not a generator, or a generator type (or an >> iterator instance or type), or a __next__ method, or anything else >> with a name. > > Sure the thing you are defining when you write "def gen() ..." is a > function? I thought we were in agreement about that. It's a function > with a flag set, a.k.a. a generator function. > > If you're referring to something else, a concrete example will help. I don't know how else to explain it. Look at the members stored by the generator type: a frame and a code object. So, going back to the original point: Generator instances are not different from iterators (except that it's a subtype, of course). The builtin generator type is likewise no different than a custom iterator class. The generator body is a normal function body. The only thing that isn't like an iterator is the frame. Which is not an important distinction. I think Chris was making a different distinction, claiming that generators are not an Iterator type (which they aren't, but only because they're _instances_ of an Iterator type, the builtin type generator), and I was pointing out that this can't be the distinction Guido was trying to make. But, again, since Guido has already said that sentence was mistaken, this really isn't important. > Because as we stand now, I don't know whether we're talking about > something important or not, or if we are in agreement, disagreement, or > merely confusion :-) I already said from the start that this is almost certainly not important. From steve at pearwood.info Sun Dec 14 09:58:14 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 14 Dec 2014 19:58:14 +1100 Subject: [Python-ideas] Generators are iterators In-Reply-To: <6902865E-BEF3-4DBB-AABA-5260BD54647F@yahoo.com> References: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> <20141213133622.GH20332@ando.pearwood.info> <20141214011735.GK20332@ando.pearwood.info> <6902865E-BEF3-4DBB-AABA-5260BD54647F@yahoo.com> Message-ID: <20141214085814.GO20332@ando.pearwood.info> In an effort to try to keep this already too-long thread a bit shorter, I'm going to trim drastically. On Sat, Dec 13, 2014 at 09:43:02PM -0800, Andrew Barnert wrote: > On Dec 13, 2014, at 17:17, Steven D'Aprano wrote: > > I read your reply to Chris last night, and it didn't make sense to me > > then and it still doesn't make sense to me now :-( Hence my request for > > a concrete example. > > You didn't ask for a concrete example, you provided the exact same > answer as Chris but with different names. My first post in this sub-thread ended with: "So I'm not sure which mysterious object you are referring to that doesn't have a standard name. Can you give a concrete example?" But moving along: > > I am still unclear as to what this mystery unnamed thing is. It's not a > > generator function, because that is called a generator function. It's > > not the thing you get when you call a generator function, because that > > is called a generator or an iterator, depending on context. > > Again, it's the thing used by the generator instance's __next__ > method: the frame and code objects. Aha, now the penny drops. Given a generator object, is has attributes holding a frame object and a code object (which it shares with the generator function that created it): py> def gen(): yield 1 ... py> it = gen() py> it.gi_frame py> it.gi_code ", line 1> py> it.gi_code is gen.__code__ True Both have standard names: "frame object" and "code object" respectively. > I don't know how else to explain it. Look at the members stored by the > generator type: a frame and a code object. That's exactly the concrete example I was looking for. > So, going back to the original point: Generator instances are not > different from iterators (except that it's a subtype, of course). The > builtin generator type is likewise no different than a custom iterator > class. The generator body is a normal function body. The only thing > that isn't like an iterator is the frame. Which is not an important > distinction. Well, there are a few other differences as well. `generator` is a concrete type, `Iterator` is an abstract type, and `iterator` can refer to either a specific iterator type: py> class X: ... def __getitem__(self, i): pass ... py> iter(X()) or speaking more generically, to anything which obeys the iterator protocol. More importantly, generators have abilities that other iterators don't have: you can send data into a generator, not just get data out of them. > I think Chris was making a different distinction, claiming that > generators are not an Iterator type (which they aren't, but only > because they're _instances_ of an Iterator type, the builtin type > generator), I'm not sure what Chris was trying to say, but I've seen various other people maintain that generators aren't iterators. As you say, that is wrong: py> from collections.abc import Iterator py> isinstance(gen(), Iterator) True The *class* `generator` is an Iterator (sub-)type, but not an Iterator itself: py> issubclass(type(gen()), Iterator) True py> isinstance(type(gen()), Iterator) False which is no different from instances of int being numbers, but int itself (the class) not being a number. Thank you for helping me understand what you were trying to say. -- Steven From jeanpierreda at gmail.com Sun Dec 14 13:47:59 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sun, 14 Dec 2014 04:47:59 -0800 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: <20141213144508.GJ20332@ando.pearwood.info> Message-ID: On Sat, Dec 13, 2014 at 11:51 AM, Chris Angelico wrote: > Personally, I'd rank them in this order of preference. Consistency > with generators is a plus (especially in terms of citing a pure-Python > equivalent), and the second and especially third options risk opening > up a misfeature - using StopIteration as a weird form of flow > control/signalling. But your description implies that option 2 is > inherently obvious, which I'm not entirely convinced of; unless this > is actually implemented as a generic "this iterator is now exhausted" > flag, and next() itself sets and checks this flag, thus automatically > enforcing that one StopIteration will result in an infinite stream of > StopIterations. I worded myself poorly. It's not fine, it probably signals a bug in the callback (which is what makes #1 a reasonable option). What I mean is that callbacks can return any return value and raise any exception. Doing so shouldn't cause you to break the documented API of your type, including breaking its conformance to the iterator protocol. Anyway, I think we're on the same page, since I agree that your suggestions are justifiable, and you seem to agree that the status quo is not. This is not a "consenting adults"-type issue. map() and filter() don't correctly handle a perfectly plausible case, and as a result they violate their documented API. That is a bug. I'd rather fix the code than fix the docs. I can write a patch implementing Chris's #1, if it would be accepted. I don't think it needs a PEP. -- Devin From ncoghlan at gmail.com Sun Dec 14 16:02:40 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Dec 2014 01:02:40 +1000 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> <548B4AB3.4030408@udel.edu> Message-ID: On 13 December 2014 at 09:15, Oscar Benjamin wrote: > I'm just saying that it's a mistake that can easily be made. Since I > made it I'm now super-suspicious of next() so I probably wouldn't make > it again. This mistake is the root of all of the problems that PEP 479 > attempts to solve. If you don't believe it is a problem that someone > might make this mistake then it follows that PEP 479 is a massive > mistake. Oscar, StopIteration unexpectedly escaping from __next__ methods is *not* the problem PEP 479 is trying to solve, and that misunderstanding appears to be the fundamental error causing your concern. PEP 479 is aiming to solve a specific problem with *writing generator functions*, not the more general problem with invoking iterators that bothers you. There is nothing to be done about StopIteration escaping from __next__ methods in general - it's an inherent limitation of the iterator protocol. StopIteration escaping from generator frames is a different story, as the generator-iterator __next__ method implementation has the chance to intercept them and convert them to something else. So let me try again from the pedagogical angle in a post PEP 380 Python world. If you are teaching someone to write explicit __next__ methods, here are the things you *absolutely* need to teach them: 1. Values returned from the method are produced as values in the iterator 2. Raising StopIteration terminates the iterator 3. If invoked via "yield from", the argument to StopIteration is the result of the "yield from" expression (default: None) 4. If delegating to a function via a function call, points 2 & 3 also apply to that subfunction (this includes the builtin next() function) If you are instead teaching someone to write generator functions, here are the things you *absolutely* need to teach them: 1. Values passed to yield expressions are produced as values in the iterator 2. Returning from the frame terminates the iterator 3. If invoked via "yield from", the return value of the generator frame is the result of the "yield from" expression (default: None) 4. If delegating to a subgenerator (rather than to an arbitrary iterator) via a "yield from" expression, then next(), send() and throw() are passed to that subgenerator until it terminates, at which point execution resumes in the parent generator frame (either producing a value from the yield from expression, or raising an exception if the subgenerator failed) With just those 8 points (and their existing knowledge regarding how to *use* iterators and generators), a developer is now equipped to write full featured __next__ method and generator function implementations, including delegation to subiterators and subgenerators via "yield from" in the latter case. Unfortunately, in both Python 3.3 and 3.4, there's a problem with these instructions: they leave something out. The thing they leave out is the fact that points 2-4 from the section on __next__ methods *also* apply to writing generator functions. Even though it's completely unnecessary, and usually not the right approach if only supporting modern versions of Python, new users still need to be taught it, because it can have unexpected side effects on the way their generators work. This redundancy increases the cognitive complexity of generator functions: new users need to be taught *two* ways of doing something, and then told "don't use the second way, we're only teaching it to you because someone else might use it in code you end up reading or maintaining, or because you might hit it accidentally and be struggling to figure out why your generator based iteration is unexpectedly finishing early". *That's* the problem PEP 479 solves: it's takes those 3 bullet points regarding the behaviour of StopIteration inside __next__ method implementations and makes them *no longer apply* to writing generator functions. If folks do run into them, they'll get a RuntimError and have the opportunity to debug it, and figure out where that came from. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Dec 14 16:17:24 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Dec 2014 01:17:24 +1000 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: Message-ID: On 13 December 2014 at 07:14, Terry Reedy wrote: > " exception StopIteration > Raised by built-in function next() and an iterator?s __next__() method > to signal that there are no further items produced by the iterator." > > I have always taken this to mean that these are the only functions that > should raise StopIteration, but other have not, and indeed, StopIteration in > generator functions, which are not __next__ methods, has been accepted and > passed on by generator.__next__. PEP 479 reverses this acceptance by having > generator.__next turn StopIteration raised in a user-written generator > function body into a RuntimeError. I propose that other builtin > iterator.__next__ methods that execute a passed in function do the same. No, Oscar's proposal is based on a fundamental misunderstanding of the problem PEP 479 is solving. The PEP solves a generator specific problem with having "Two Ways to Do It" when it comes to terminating generator functions. It does this by way of the relatively straightforward expedient of having the PEP 380 approach completely replace the legacy approach, rather than retaining the status quo of offering both in parallel (my attempt at explaining this perspective in more detail: https://mail.python.org/pipermail/python-ideas/2014-December/030371.html). __next__ methods in general don't have this problem - there's only one way to signal termination from a __next__ method implementation, and that's raising StopIteration. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Dec 14 16:35:43 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Dec 2014 01:35:43 +1000 Subject: [Python-ideas] Add stop=exception option to next() (was Re: PEP 479 and take()) In-Reply-To: References: <20141213233633.058b66f9@fsol> <91653699-CF7A-4F52-862C-569BCE93AD83@gmail.com> Message-ID: On 14 December 2014 at 12:31, Guido van Rossum wrote: > On Sat, Dec 13, 2014 at 6:08 PM, Raymond Hettinger > wrote: >> >> For Christmas, it would be a nice gift if Guido could put an end to this >> line of thinking. The conversations are difficult because several of the >> participants are deeply convinced that there is something fundamentally >> wrong with StopIteration. I don't think any of them can easily be convinced >> to the leave the iterator protocol alone and to continue enjoying one of our >> best success stories. > > It ends with PEP 479. I have no intention to let this go any further. I am > now going to mute any further threads on the topic, and I recommend you do > the same, Raymond. Happy holidays! Hear, hear! For myself, I've made one last contribution to a couple of the threads (in an attempt to convey how PEP 479 manages to make the language *smaller* rather than larger when it comes to *writing* generator functions). I'll monitor replies to those to see if further clarifications may be useful, but that's specifically aimed at helping folks to understand the design decision, rather than encouraging further debate about it. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Sun Dec 14 18:48:22 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 14 Dec 2014 12:48:22 -0500 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> <548B4AB3.4030408@udel.edu> Message-ID: On 12/14/2014 10:02 AM, Nick Coghlan wrote: > If you are teaching someone to write explicit __next__ methods, here > are the things you *absolutely* need to teach them: > > 1. Values returned from the method are produced as values in the iterator > 2. Raising StopIteration terminates the iterator > 3. If invoked via "yield from", the argument to StopIteration is the > result of the "yield from" expression (default: None) I had forgotten this apparently important point. After re-reading the somewhat confusing 'Yield expressions' doc, I came up with this example. class subc: def __iter__(self): return self def __init__(self): self.i, self.it = -1, (0,1,2) def __next__(self): if self.i < 2: self.i += 1 return self.it[self.i] else: raise StopIteration('subc done') def subg(): for i in range(3): yield i return 'sub done' def sup(sub): print((yield from sub())) print(list(sup(subc))) print(list(sup(subg))) it = subg() result = [] try: while True: result.append(next(it)) except StopIteration as e: print(e.args[0]) print(result) >>> sub done [0, 1, 2] sub done [0, 1, 2] sub done [0, 1, 2] Pep 380 not only adds 'yield from', but also a mechanism for *any* iterator to optionally pass a terminal message back to *any* direct user that looks for the message. I don't know how I might use this feature, which is why I more or less ignored it at the time it was introduced, but it is part of asyncio's use of 'yield from'. For statements ignore the return message, so this is one possible reason to use while and explicit next inside try. > 4. If delegating to a function via a function call, points 2 & 3 also > apply to that subfunction (this includes the builtin next() function) > > If you are instead teaching someone to write generator functions, here > are the things you *absolutely* need to teach them: > > 1. Values passed to yield expressions are produced as values in the iterator > 2. Returning from the frame terminates the iterator > 3. If invoked via "yield from", the return value of the generator > frame is the result of the "yield from" expression (default: None) Or the return value can be accessed by catching the resulting StopIteration of next(). > 4. If delegating to a subgenerator (rather than to an arbitrary > iterator) via a "yield from" expression, then next(), send() and > throw() are passed to that subgenerator until it terminates, at which > point execution resumes in the parent generator frame (either > producing a value from the yield from expression, or raising an > exception if the subgenerator failed) -- Terry Jan Reedy From rosuav at gmail.com Sun Dec 14 20:29:33 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 15 Dec 2014 06:29:33 +1100 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: <20141213144508.GJ20332@ando.pearwood.info> Message-ID: On Sun, Dec 14, 2014 at 11:47 PM, Devin Jeanpierre wrote: > I worded myself poorly. It's not fine, it probably signals a bug in > the callback (which is what makes #1 a reasonable option). What I mean > is that callbacks can return any return value and raise any exception. > Doing so shouldn't cause you to break the documented API of your type, > including breaking its conformance to the iterator protocol. Anyway, I > think we're on the same page, since I agree that your suggestions are > justifiable, and you seem to agree that the status quo is not. Sure. Then yes, I think we are broadly in agreement. > This is not a "consenting adults"-type issue. map() and filter() don't > correctly handle a perfectly plausible case, and as a result they > violate their documented API. That is a bug. I'd rather fix the code > than fix the docs. I can write a patch implementing Chris's #1, if it > would be accepted. I don't think it needs a PEP. Sounds fine to me, although without concrete use-cases I can't say it would be a huge practical improvement. It's still a nice theoretical improvement, tightening up a promise. ChrisA From abarnert at yahoo.com Mon Dec 15 01:28:58 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 14 Dec 2014 16:28:58 -0800 Subject: [Python-ideas] Generators are iterators In-Reply-To: <20141214085814.GO20332@ando.pearwood.info> References: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> <20141213133622.GH20332@ando.pearwood.info> <20141214011735.GK20332@ando.pearwood.info> <6902865E-BEF3-4DBB-AABA-5260BD54647F@yahoo.com> <20141214085814.GO20332@ando.pearwood.info> Message-ID: <6EEBC35D-A452-4B22-B80D-F6F1A0063EC0@yahoo.com> On Dec 14, 2014, at 0:58, Steven D'Aprano wrote: > In an effort to try to keep this already too-long thread a bit > shorter, I'm going to trim drastically. > > On Sat, Dec 13, 2014 at 09:43:02PM -0800, Andrew Barnert wrote: >> On Dec 13, 2014, at 17:17, Steven D'Aprano wrote: > >>> I am still unclear as to what this mystery unnamed thing is. It's not a >>> generator function, because that is called a generator function. It's >>> not the thing you get when you call a generator function, because that >>> is called a generator or an iterator, depending on context. >> >> Again, it's the thing used by the generator instance's __next__ >> method: the frame and code objects. > > Aha, now the penny drops. Given a generator object, is has attributes > holding a frame object and a code object (which it shares with the > generator function that created it): > > py> def gen(): yield 1 > ... > py> it = gen() > py> it.gi_frame > > py> it.gi_code > ", line 1> > py> it.gi_code is gen.__code__ > True > > Both have standard names: "frame object" and "code object" respectively. Sure, but the problem is that people have been writing things that come "from the generator's internals", and then trying to explain how those things "make generators different from iterators" (or "... from explicit __next__ methods"). Because of the wording that was being used, people were looking for some way in which generators, or the generator type, or generator functions, are different from iterators, from other iterator types, or whatever, when none of that is at issue; it's the suspended generator frame, which doesn't have any parallel in other iterator implementations. I was thinking that if we had some term like "generator state" to refer to the frame-and-code (or just the frame, because the code isn't really necessary), the confusion wouldn't have arisen. Now I notice that Nick Coghlan has been referring to "the generator frame", which is a nice way to avoid that confusion. What I was basically suggesting is that someone should rewrite the explanations exactly the way Nick already had been rewriting them. :) >> I don't know how else to explain it. Look at the members stored by the >> generator type: a frame and a code object. > > That's exactly the concrete example I was looking for. That was in the message you replied to. I think the problem isn't that I didn't give the example you were looking for, it's that (at least in that message) I didn't explain why you (or Chris or anyone else) should care about it, so you passed right over it. Sorry for not making that part clear. From random832 at fastmail.us Mon Dec 15 18:37:26 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Mon, 15 Dec 2014 12:37:26 -0500 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: References: <20141213144508.GJ20332@ando.pearwood.info> Message-ID: <1418665046.2948059.203141813.5723B4F6@webmail.messagingengine.com> On Sat, Dec 13, 2014, at 14:51, Chris Angelico wrote: > I'm not sure that "fine and to be expected" is necessarily true. If > the function given to map() raises StopIteration, I'd say any of these > approaches is justifiable: > 1) It's an error. Raise something other than StopIteration, so the > caller knows something bad happened. (A generator version of map() > will do this post-479.) > 2) It's to be expected. Consider map() to have now terminated. (In > other words, catch it and return.) > 3) It's a signal that this one element should not be returned. > Suppress it, yield nothing, and go get the next value from the input > iterator. > > Personally, I'd rank them in this order of preference. Consistency > with generators is a plus (especially in terms of citing a pure-Python > equivalent), It's worth mentioning that #2 is, more or less,* precisely how both map and a pure-python equivalent behaved before PEP 479. Some people in this discussion considered this expected. Obviously that was not the end of the discussion. *Assuming that the code consuming the outer iterator will not attempt to read it again after a single StopIteration has been raised. A well-behaved iterator will continue to raise StopIteration forever after it has raised it once - an iterator whose StopIteration has been caused by some inner function will not. From rosuav at gmail.com Tue Dec 16 00:05:07 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 16 Dec 2014 10:05:07 +1100 Subject: [Python-ideas] Map and filter should also convert StopIteration to RuntimeError In-Reply-To: <1418665046.2948059.203141813.5723B4F6@webmail.messagingengine.com> References: <20141213144508.GJ20332@ando.pearwood.info> <1418665046.2948059.203141813.5723B4F6@webmail.messagingengine.com> Message-ID: On Tue, Dec 16, 2014 at 4:37 AM, wrote: > It's worth mentioning that #2 is, more or less,* precisely how both map > and a pure-python equivalent behaved before PEP 479. Some people in this > discussion considered this expected. Obviously that was not the end of > the discussion. > > *Assuming that the code consuming the outer iterator will not attempt to > read it again after a single StopIteration has been raised. A > well-behaved iterator will continue to raise StopIteration forever after > it has raised it once - an iterator whose StopIteration has been caused > by some inner function will not. Right, and it's that assumption that's the entire question here. This is buggy behaviour; the question is, is the bug map's problem (because it's failing to behave the way an iterator should), or is it simply a bug in the called function, which shouldn't be raising StopIteration? ChrisA From antony.lee at berkeley.edu Tue Dec 16 04:47:31 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Mon, 15 Dec 2014 19:47:31 -0800 Subject: [Python-ideas] Less tight validation in namedtuple input validation, and usefulness of issue 21832. Message-ID: While namedtuples have a good reason to not support field names starting with an underscore (namely, to prevent name collision), it would be nice to still be able to define some sort of "private" attribute using (manual) double-underscore name mangling, i.e. allow "namedtuple('Foo', 'bar baz _Foo__qux')". It used to be possible to bypass the validation by passing a carefully crafted subclass of str to the factory function, but this is no longer an option after the fix for issue 21832 (which, by the way, seems a complete overkill - there are a lot of other casts-to-str in the stdlib which do not check that the result is exactly of type str (e.g., ConfigParser.read_dict), and if I want to execute untrusted code by passing a subclass of str I can always put that code in the __str__ method of the class anyways). Antony -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Tue Dec 16 04:50:25 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Mon, 15 Dec 2014 23:20:25 -0430 Subject: [Python-ideas] Generators are iterators In-Reply-To: <6EEBC35D-A452-4B22-B80D-F6F1A0063EC0@yahoo.com> References: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> <20141213133622.GH20332@ando.pearwood.info> <20141214011735.GK20332@ando.pearwood.info> <6902865E-BEF3-4DBB-AABA-5260BD54647F@yahoo.com> <20141214085814.GO20332@ando.pearwood.info> <6EEBC35D-A452-4B22-B80D-F6F1A0063EC0@yahoo.com> Message-ID: On Sun, Dec 14, 2014 at 7:58 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > Sure, but the problem is that people have been writing things that come > "from the generator's internals", and then trying to explain how those > things "make generators different from iterators" (or "... from explicit > __next__ methods"). Because of the wording that was being used, people were > looking for some way in which generators, or the generator type, or > generator functions, are different from iterators, from other iterator > types, or whatever, when none of that is at issue; it's the suspended > generator frame, which doesn't have any parallel in other iterator > implementations. I was thinking that if we had some term like "generator > state" to refer to the frame-and-code (or just the frame, because the code > isn't really necessary), the confusion wouldn't have arisen. I never was too much into using an exception to signal than an iterator had finished its job. Either intuition or experience said that it would be a source of obscure problems. I'm not proposing a solution. Not even proposing a change. I do propose that since this thread has been long and going long, we take the opportunity to consider the base assumptions. Most newcomers to programming take a while to grok complex control flows, and what's happening here is too complex even for Python contributors. I suggest we revisit "import this" for a little while, just in case we made a mistake along the way. Sorry for the rambling! Regards, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Tue Dec 16 06:48:23 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 16 Dec 2014 14:48:23 +0900 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <614EDB99-CD88-420D-8734-3F958328D07B@yahoo.com> <20141213133622.GH20332@ando.pearwood.info> <20141214011735.GK20332@ando.pearwood.info> <6902865E-BEF3-4DBB-AABA-5260BD54647F@yahoo.com> <20141214085814.GO20332@ando.pearwood.info> <6EEBC35D-A452-4B22-B80D-F6F1A0063EC0@yahoo.com> Message-ID: <87siggdu1k.fsf@uwakimon.sk.tsukuba.ac.jp> Juancarlo A?ez writes: > Most newcomers to programming take a while to grok complex control > flows, and what's happening here is too complex even for Python > contributors. As I understand Raymond's point, what is happening here is *not* particularly complex (I don't see it as complex myself, anyway). It's simply an obscure (because rare) and obvious (once you know about it) consequence of two simple design decisions: "exceptions bubble" and "iterator termination is signaled with an exception". That obscurity is compounded by the fact that a programmer error will pass silently. (There must be a theorem that says that in any Turing complete language some errors will pass silently!) In the case of *generators* silently ignoring an error is unnecessary, and PEP 479 fixes that one case, at a cost of breaking a useful idiom in some functions. For iterators in general, I suppose we could add a flag to checkers to look for function calls inside iterators, and warn the user to check for the potential for spuriously raising StopIteration. (I doubt that's a great idea, but it could be done.) From oscar.j.benjamin at gmail.com Tue Dec 16 15:48:28 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 16 Dec 2014 14:48:28 +0000 Subject: [Python-ideas] Generators are iterators In-Reply-To: References: <20141210141858.GA20332@ando.pearwood.info> <548B4AB3.4030408@udel.edu> Message-ID: On 14 December 2014 at 15:02, Nick Coghlan wrote: > On 13 December 2014 at 09:15, Oscar Benjamin wrote: >> I'm just saying that it's a mistake that can easily be made. Since I >> made it I'm now super-suspicious of next() so I probably wouldn't make >> it again. This mistake is the root of all of the problems that PEP 479 >> attempts to solve. If you don't believe it is a problem that someone >> might make this mistake then it follows that PEP 479 is a massive >> mistake. > > Oscar, StopIteration unexpectedly escaping from __next__ methods is > *not* the problem PEP 479 is trying to solve, and that > misunderstanding appears to be the fundamental error causing your > concern. PEP 479 is aiming to solve a specific problem with *writing > generator functions*, not the more general problem with invoking > iterators that bothers you. There is nothing to be done about > StopIteration escaping from __next__ methods in general - it's an > inherent limitation of the iterator protocol. StopIteration escaping > from generator frames is a different story, as the generator-iterator > __next__ method implementation has the chance to intercept them and > convert them to something else. Fair enough. I understand your point that the intention of the PEP is only to solve this for generators. I also understand that these threads are not working towards anything so this is my last post on the subject. I see nothing wrong with the rationale that "it's easier to to address this problem for generators" but that's very different from the "this problem is unique to generators" argument. Since the start of this thread the PEP is now modified so that it no longer explicitly makes the latter argument (although it still does not explicitly state the former). > So let me try again from the pedagogical angle in a post PEP 380 Python world. > > If you are teaching someone to write explicit __next__ methods, here > are the things you *absolutely* need to teach them: > > 1. Values returned from the method are produced as values in the iterator > 2. Raising StopIteration terminates the iterator > 3. If invoked via "yield from", the argument to StopIteration is the > result of the "yield from" expression (default: None) > 4. If delegating to a function via a function call, points 2 & 3 also > apply to that subfunction (this includes the builtin next() function) > > If you are instead teaching someone to write generator functions, here > are the things you *absolutely* need to teach them: > > 1. Values passed to yield expressions are produced as values in the iterator > 2. Returning from the frame terminates the iterator > 3. If invoked via "yield from", the return value of the generator > frame is the result of the "yield from" expression (default: None) > 4. If delegating to a subgenerator (rather than to an arbitrary > iterator) via a "yield from" expression, then next(), send() and > throw() are passed to that subgenerator until it terminates, at which > point execution resumes in the parent generator frame (either > producing a value from the yield from expression, or raising an > exception if the subgenerator failed) > > With just those 8 points (and their existing knowledge regarding how > to *use* iterators and generators), a developer is now equipped to > write full featured __next__ method and generator function > implementations, including delegation to subiterators and > subgenerators via "yield from" in the latter case. > > Unfortunately, in both Python 3.3 and 3.4, there's a problem with > these instructions: they leave something out. The thing they leave out > is the fact that points 2-4 from the section on __next__ methods > *also* apply to writing generator functions. Even though it's > completely unnecessary, and usually not the right approach if only > supporting modern versions of Python, new users still need to be > taught it, because it can have unexpected side effects on the way > their generators work. I'm not sure why you single out 3.3 and 3.4 as it happens without yield from. In 2.7 (and anything since 2.2): >>> def f(): ... yield 1 ... yield 2 ... raise StopIteration ... yield 3 ... >>> list(f()) [1, 2] I will leave this subject now. Oscar From antony.lee at berkeley.edu Wed Dec 17 02:24:37 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Tue, 16 Dec 2014 17:24:37 -0800 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? Message-ID: Currently, BoundArguments stores the results of signature.bind (and bind_partial) in an OrderedDict. Given that there is no way to retrieve the order in which keyword arguments were passed, it is not surprising that the dict ordering is based on the order of parameters in the signature, not the order in which arguments are passed (although this point could certainly be made clearer in the docs). But this also means that that ordering could easily be retrieved by iterating over the parameters ("[(name, ba.arguments[name]) for name in ba.signature.parameters if name in ba.arguments]"), and signature.bind now has to pay for the cost of using a Python-implemented OrderedDict. Obviously, one solution would be to switch to a C-implemented OrderedDict, but another simpler one would simply to make BoundArguments.arguments a regular dict rather than an OrderedDict (which is a one-line patch, changing the definition of "arguments" in Signature._bind). A simple test on the example below show approximately a two-fold faster "bind". from inspect import signature s = signature(lambda x, y=1, *, z: None) for _ in range(30000): s.bind(1, z=2) s.bind(1, 3, z=2) s.bind(x=1, z=2, y=3) try: s.bind(1) except TypeError: pass try: s.bind() except TypeError: pass try: s.bind(1, y=2) except TypeError: pass Antony -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Wed Dec 17 03:27:09 2014 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 16 Dec 2014 20:27:09 -0600 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: Message-ID: It really would be helpful to have a C-implemented OrderedDict in stdlib. On Tue, Dec 16, 2014 at 7:24 PM, Antony Lee wrote: > > Currently, BoundArguments stores the results of signature.bind (and > bind_partial) in an OrderedDict. Given that there is no way to retrieve > the order in which keyword arguments were passed, it is not surprising that > the dict ordering is based on the order of parameters in the signature, not > the order in which arguments are passed (although this point could > certainly be made clearer in the docs). But this also means that that > ordering could easily be retrieved by iterating over the parameters > ("[(name, ba.arguments[name]) for name in ba.signature.parameters if name > in ba.arguments]"), and signature.bind now has to pay for the cost of using > a Python-implemented OrderedDict. > > Obviously, one solution would be to switch to a C-implemented OrderedDict, > but another simpler one would simply to make BoundArguments.arguments a > regular dict rather than an OrderedDict (which is a one-line patch, > changing the definition of "arguments" in Signature._bind). A simple test > on the example below show approximately a two-fold faster "bind". > > from inspect import signature > s = signature(lambda x, y=1, *, z: None) > for _ in range(30000): > s.bind(1, z=2) > s.bind(1, 3, z=2) > s.bind(x=1, z=2, y=3) > try: s.bind(1) > except TypeError: pass > try: s.bind() > except TypeError: pass > try: s.bind(1, y=2) > except TypeError: pass > > Antony > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Dec 17 03:45:14 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 16 Dec 2014 18:45:14 -0800 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: Message-ID: On Tue, Dec 16, 2014 at 6:27 PM, Wes Turner wrote: > > It really would be helpful to have a C-implemented OrderedDict in stdlib. > +1 -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Wed Dec 17 07:18:55 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 16 Dec 2014 23:18:55 -0700 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: Message-ID: On Tue, Dec 16, 2014 at 7:45 PM, Guido van Rossum wrote: > On Tue, Dec 16, 2014 at 6:27 PM, Wes Turner wrote: >> >> It really would be helpful to have a C-implemented OrderedDict in stdlib. > > > +1 http://bugs.python.org/issue16991 Reviews welcome! :) The patch is undoubtedly stale (after a year and a half). It definitely could use some trimming (basically +2500 lines), but a lot of the bulk comes from the boilerplate involved in writing iterators (and the odict views) at the C/C-API level. -eric From ericsnowcurrently at gmail.com Wed Dec 17 07:25:04 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 16 Dec 2014 23:25:04 -0700 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: Message-ID: On Tue, Dec 16, 2014 at 11:18 PM, Eric Snow wrote: > The patch is undoubtedly stale (after a year and a half). FWIW, I'm finally starting to have some spare time so I may be able to refresh the patch. However, if I can't get reviews I imagine it will just get stale again, so I don't know if I can devote the time in that case. If anyone has suggestions on how I can split the patch up to make it more easily reviewed, or on any other ways to help reviewers of the patch, please leave a comment on the issue (#16991). I'd love to get the C-OrderedDict landed! -eric From guido at python.org Wed Dec 17 17:34:22 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 17 Dec 2014 08:34:22 -0800 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: Message-ID: Maybe this would be an opportunity for you to mentor someone who wants to get into serious C-level Python maintenance? I think that would qualify as a code review, if your student is serious about learning how the C code you've written works. Perhaps an ad on core-mentorship might turn up such a person. On Tue, Dec 16, 2014 at 10:25 PM, Eric Snow wrote: > > On Tue, Dec 16, 2014 at 11:18 PM, Eric Snow > wrote: > > The patch is undoubtedly stale (after a year and a half). > > FWIW, I'm finally starting to have some spare time so I may be able to > refresh the patch. However, if I can't get reviews I imagine it will > just get stale again, so I don't know if I can devote the time in that > case. If anyone has suggestions on how I can split the patch up to > make it more easily reviewed, or on any other ways to help reviewers > of the patch, please leave a comment on the issue (#16991). I'd love > to get the C-OrderedDict landed! > > -eric > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Dec 17 19:11:39 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 17 Dec 2014 19:11:39 +0100 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? References: Message-ID: <20141217191139.0da7463c@fsol> On Tue, 16 Dec 2014 23:18:55 -0700 Eric Snow wrote: > On Tue, Dec 16, 2014 at 7:45 PM, Guido van Rossum wrote: > > On Tue, Dec 16, 2014 at 6:27 PM, Wes Turner wrote: > >> > >> It really would be helpful to have a C-implemented OrderedDict in stdlib. > > > > > > +1 > > http://bugs.python.org/issue16991 > > Reviews welcome! :) > > The patch is undoubtedly stale (after a year and a half). It > definitely could use some trimming (basically +2500 lines), but a lot > of the bulk comes from the boilerplate involved in writing iterators > (and the odict views) at the C/C-API level. I have reservations about the overall approach. Should it really be a builtin and expose a public C API? It sounds like putting it in the _collections module should be sufficient. Regards Antoine. From guido at python.org Wed Dec 17 19:14:25 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 17 Dec 2014 10:14:25 -0800 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: <20141217191139.0da7463c@fsol> References: <20141217191139.0da7463c@fsol> Message-ID: Yes, it should stay in collections. On Dec 17, 2014 10:12 AM, "Antoine Pitrou" wrote: > On Tue, 16 Dec 2014 23:18:55 -0700 > Eric Snow > wrote: > > On Tue, Dec 16, 2014 at 7:45 PM, Guido van Rossum > wrote: > > > On Tue, Dec 16, 2014 at 6:27 PM, Wes Turner > wrote: > > >> > > >> It really would be helpful to have a C-implemented OrderedDict in > stdlib. > > > > > > > > > +1 > > > > http://bugs.python.org/issue16991 > > > > Reviews welcome! :) > > > > The patch is undoubtedly stale (after a year and a half). It > > definitely could use some trimming (basically +2500 lines), but a lot > > of the bulk comes from the boilerplate involved in writing iterators > > (and the odict views) at the C/C-API level. > > I have reservations about the overall approach. Should it really be a > builtin and expose a public C API? It sounds like putting it in the > _collections module should be sufficient. > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Wed Dec 17 19:22:17 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 17 Dec 2014 13:22:17 -0500 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: <20141217191139.0da7463c@fsol> Message-ID: On Wed, Dec 17, 2014 at 1:14 PM, Guido van Rossum wrote: > > Yes, it should stay in collections. It would be nice to keep public C API, though. We have a precedent in datetime of an stdlib module that exposes public C API. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Dec 17 19:30:56 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 17 Dec 2014 19:30:56 +0100 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? References: Message-ID: <20141217193056.46664962@fsol> On Tue, 16 Dec 2014 23:18:55 -0700 Eric Snow wrote: > On Tue, Dec 16, 2014 at 7:45 PM, Guido van Rossum wrote: > > On Tue, Dec 16, 2014 at 6:27 PM, Wes Turner wrote: > >> > >> It really would be helpful to have a C-implemented OrderedDict in stdlib. > > > > > > +1 > > http://bugs.python.org/issue16991 > > Reviews welcome! :) Other reservations: - why aren't the linked list pointers not included in the hash entries? that should simplify some logic quite a bit - is it useful to keep the shared / split keys mechanism? shared keys are mostly useful for instance dicts - does it have to inherit from dict? that looks like a potential can of worms Regards Antoine. From ericsnowcurrently at gmail.com Wed Dec 17 22:11:23 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 17 Dec 2014 14:11:23 -0700 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: <20141217191139.0da7463c@fsol> References: <20141217191139.0da7463c@fsol> Message-ID: On Wed, Dec 17, 2014 at 11:11 AM, Antoine Pitrou wrote: > I have reservations about the overall approach. Should it really be a > builtin The patch doesn't add it to bltinmodule.c. Perhaps you mean the additions to Python.h, object.c, and the Makefile. In that case then I actually agree. I was probably just anticipating the use of OrderedDict by the interpreter. However, that can wait until needed. > and expose a public C API? Yeah, it really doesn't need that either (if I remember right). If a C-API becomes desirable then it would be easy to add later. > It sounds like putting it in the > _collections module should be sufficient. Agreed. -eric From ericsnowcurrently at gmail.com Wed Dec 17 22:11:57 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 17 Dec 2014 14:11:57 -0700 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: Message-ID: On Wed, Dec 17, 2014 at 9:34 AM, Guido van Rossum wrote: > Maybe this would be an opportunity for you to mentor someone who wants to > get into serious C-level Python maintenance? I think that would qualify as a > code review, if your student is serious about learning how the C code you've > written works. Perhaps an ad on core-mentorship might turn up such a person. Good idea. I'll look into that. -eric From ericsnowcurrently at gmail.com Wed Dec 17 22:26:25 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 17 Dec 2014 14:26:25 -0700 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: <20141217193056.46664962@fsol> References: <20141217193056.46664962@fsol> Message-ID: BTW, thanks for the feedback. :) On Wed, Dec 17, 2014 at 11:30 AM, Antoine Pitrou wrote: > Other reservations: > > - why aren't the linked list pointers not included in the hash entries? > that should simplify some logic quite a bit I'll look into that. If I recall correctly this came up in earlier discussion and there was a justification. > - is it useful to keep the shared / split keys mechanism? shared keys > are mostly useful for instance dicts I'm pretty sure this is an artifact of subclassing dict. > - does it have to inherit from dict? that looks like a potential can of > worms One of the constraints here is to make the C implementation of OrderedDict match the API of the Python type exactly and the underlying implementation as closely as reasonable. The Python type subclasses dict so the C implementation does as well. FWIW (and not directed to Antoine specifically), the implementation up for review may not be ideal, but it exists and is complete. :) Barring any substantial concerns during review (and the points Antoine has brought up), I would rather the patch landed than wait indefinitely for a more ideal implementation. That could happen afterward, though I'm fairly confident in the correctness and efficiency of the implementation. -eric From ncoghlan at gmail.com Thu Dec 18 01:37:00 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 18 Dec 2014 10:37:00 +1000 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: <20141217193056.46664962@fsol> Message-ID: On 18 December 2014 at 07:26, Eric Snow wrote: >> - does it have to inherit from dict? that looks like a potential can of >> worms > > One of the constraints here is to make the C implementation of > OrderedDict match the API of the Python type exactly and the > underlying implementation as closely as reasonable. The Python type > subclasses dict so the C implementation does as well. This kind of design rationale is potentially useful to include inline as a block comment. The kinds of questions reviewers have are often going to be the same kinds of questions future maintainers have, and "why" data regarding core architectural decisions is less likely to go out of date as future maintainers make changes. > FWIW (and not directed to Antoine specifically), the implementation up > for review may not be ideal, but it exists and is complete. :) > Barring any substantial concerns during review (and the points Antoine > has brought up), I would rather the patch landed than wait > indefinitely for a more ideal implementation. That could happen > afterward, though I'm fairly confident in the correctness and > efficiency of the implementation. Aye, a definite +1 for "better" over "perfect". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From antony.lee at berkeley.edu Thu Dec 18 03:15:37 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Wed, 17 Dec 2014 18:15:37 -0800 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: <20141217193056.46664962@fsol> Message-ID: The discussion has drifted towards improving OrderedDict (something I certainly approve), but the semantic question is still there: why should BoundArguments.arguments be ordered by the parameter order? For example, the recipe just below in the docs, for filling in the remaining default arguments, breaks that ordering, without even mentioning that. Antony 2014-12-17 16:37 GMT-08:00 Nick Coghlan : > > On 18 December 2014 at 07:26, Eric Snow > wrote: > >> - does it have to inherit from dict? that looks like a potential can of > >> worms > > > > One of the constraints here is to make the C implementation of > > OrderedDict match the API of the Python type exactly and the > > underlying implementation as closely as reasonable. The Python type > > subclasses dict so the C implementation does as well. > > This kind of design rationale is potentially useful to include inline > as a block comment. The kinds of questions reviewers have are often > going to be the same kinds of questions future maintainers have, and > "why" data regarding core architectural decisions is less likely to go > out of date as future maintainers make changes. > > > FWIW (and not directed to Antoine specifically), the implementation up > > for review may not be ideal, but it exists and is complete. :) > > Barring any substantial concerns during review (and the points Antoine > > has brought up), I would rather the patch landed than wait > > indefinitely for a more ideal implementation. That could happen > > afterward, though I'm fairly confident in the correctness and > > efficiency of the implementation. > > Aye, a definite +1 for "better" over "perfect". > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Dec 18 03:57:53 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 17 Dec 2014 18:57:53 -0800 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: <20141217193056.46664962@fsol> Message-ID: On Wed, Dec 17, 2014 at 6:15 PM, Antony Lee wrote: > > The discussion has drifted towards improving OrderedDict (something I > certainly approve), but the semantic question is still there: why should > BoundArguments.arguments be ordered by the parameter order? For example, > the recipe just below in the docs, for filling in the remaining default > arguments, breaks that ordering, without even mentioning that. > Given that the answer hasn't been answered yet, perhaps nobody remembers the reason any more. But why does it bother you so much? In my experience the inspect module wasn't written for performance but for functionality. If it really hurts your ability to do something that you need to be fast, perhaps you can supply a patch? Make sure to run the tests and update the docs as needed. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Dec 18 08:05:57 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 18 Dec 2014 17:05:57 +1000 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: <20141217193056.46664962@fsol> Message-ID: On 18 December 2014 at 12:57, Guido van Rossum wrote: > On Wed, Dec 17, 2014 at 6:15 PM, Antony Lee wrote: >> >> The discussion has drifted towards improving OrderedDict (something I >> certainly approve), but the semantic question is still there: why should >> BoundArguments.arguments be ordered by the parameter order? For example, >> the recipe just below in the docs, for filling in the remaining default >> arguments, breaks that ordering, without even mentioning that. > > > Given that the answer hasn't been answered yet, perhaps nobody remembers the > reason any more. But why does it bother you so much? In my experience the > inspect module wasn't written for performance but for functionality. If it > really hurts your ability to do something that you need to be fast, perhaps > you can supply a patch? Make sure to run the tests and update the docs as > needed. As far as I'm aware, it's an ordered dictionary because that makes the default repr() predictable when binding arguments for a given function (in the absence of after-the-fact manipulation like the example in the docs that injects the default values as explicitly bound arguments). The inspect.signature() machinery includes quite a few things like that where the serialisation as a human readable string is considered as important then the programmatic representation. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From antony.lee at berkeley.edu Thu Dec 18 13:45:22 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Thu, 18 Dec 2014 04:45:22 -0800 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: <20141217193056.46664962@fsol> Message-ID: 2014-12-17 23:05 GMT-08:00 Nick Coghlan : > > On 18 December 2014 at 12:57, Guido van Rossum wrote: > > On Wed, Dec 17, 2014 at 6:15 PM, Antony Lee > wrote: > >> > >> The discussion has drifted towards improving OrderedDict (something I > >> certainly approve), but the semantic question is still there: why should > >> BoundArguments.arguments be ordered by the parameter order? For > example, > >> the recipe just below in the docs, for filling in the remaining default > >> arguments, breaks that ordering, without even mentioning that. > > > > > > Given that the answer hasn't been answered yet, perhaps nobody remembers > the > > reason any more. But why does it bother you so much? In my experience the > > inspect module wasn't written for performance but for functionality. If > it > > really hurts your ability to do something that you need to be fast, > perhaps > > you can supply a patch? Make sure to run the tests and update the docs as > > needed. > Actually the second example in PEP362 uses bind as an argument type-checker, which should arguably be as fast as possible. I'm opening an issue with a patch right now. > > As far as I'm aware, it's an ordered dictionary because that makes the > default repr() predictable when binding arguments for a given function > (in the absence of after-the-fact manipulation like the example in the > docs that injects the default values as explicitly bound arguments). > That can't really be the case as BoundArguments doesn't define a __repr__... (and OrderedDict's __repr__ is a bit too verbose IMO, but that's another issue and little can be done about it anyways). Now that I delved into the details of BoundArgument's implementation, I'll also note that __eq__ also checks the arguments order, and again additional binding will mess that up: s = signature(lambda x=None, y=None: None) ba0 = s.bind() ba1 = s.bind(x=None) ba2 = s.bind(y=None) Should ba0, ba1 and ba2 compare now equal or not? As it is, we'll have ba0 == ba1 != ba2, which doesn't strike me as particularly reasonable. Antony > The inspect.signature() machinery includes quite a few things like > that where the serialisation as a human readable string is considered > as important then the programmatic representation. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Thu Dec 18 17:09:23 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 18 Dec 2014 11:09:23 -0500 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: <20141217193056.46664962@fsol> Message-ID: <1418918963.18586.204483089.2CF3A673@webmail.messagingengine.com> On Thu, Dec 18, 2014, at 02:05, Nick Coghlan wrote: > As far as I'm aware, it's an ordered dictionary because that makes the > default repr() predictable when binding arguments for a given function > (in the absence of after-the-fact manipulation like the example in the > docs that injects the default values as explicitly bound arguments). > > The inspect.signature() machinery includes quite a few things like > that where the serialisation as a human readable string is considered > as important then the programmatic representation. Would it be reasonable to make a lightweight "predictable dict" class that makes a weaker guarantee, e.g. that the enumeration order will match the insertion order in the case where it is filled from empty with no intervening deletions and not guaranteed in any other cases? From wes.turner at gmail.com Thu Dec 18 17:18:59 2014 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 18 Dec 2014 10:18:59 -0600 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: <1418918963.18586.204483089.2CF3A673@webmail.messagingengine.com> References: <20141217193056.46664962@fsol> <1418918963.18586.204483089.2CF3A673@webmail.messagingengine.com> Message-ID: > Would it be reasonable to make a lightweight "predictable dict" class > that makes a weaker guarantee, e.g. that the enumeration order will > match the insertion order in the case where it is filled from empty with > no intervening deletions and not guaranteed in any other cases? Something like DefaultOrderedDict (like defaultdict)? >From http://bugs.python.org/issue16991#msg232825 : > Would this make it easy/faster to also have a DefaultOrderedDict > (which can/could also be accomplished with > .get(attr, []) and .setdefault(attr, [])? On Thu, Dec 18, 2014 at 10:09 AM, wrote: > > On Thu, Dec 18, 2014, at 02:05, Nick Coghlan wrote: > > As far as I'm aware, it's an ordered dictionary because that makes the > > default repr() predictable when binding arguments for a given function > > (in the absence of after-the-fact manipulation like the example in the > > docs that injects the default values as explicitly bound arguments). > > > > The inspect.signature() machinery includes quite a few things like > > that where the serialisation as a human readable string is considered > > as important then the programmatic representation. > > Would it be reasonable to make a lightweight "predictable dict" class > that makes a weaker guarantee, e.g. that the enumeration order will > match the insertion order in the case where it is filled from empty with > no intervening deletions and not guaranteed in any other cases? > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Thu Dec 18 17:27:59 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 18 Dec 2014 11:27:59 -0500 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: <20141217193056.46664962@fsol> <1418918963.18586.204483089.2CF3A673@webmail.messagingengine.com> Message-ID: <1418920079.23228.204489681.02FF696E@webmail.messagingengine.com> On Thu, Dec 18, 2014, at 11:18, Wes Turner wrote: > > Would it be reasonable to make a lightweight "predictable dict" class > > that makes a weaker guarantee, e.g. that the enumeration order will > > match the insertion order in the case where it is filled from empty with > > no intervening deletions and not guaranteed in any other cases? > > Something like DefaultOrderedDict (like defaultdict)? > > From http://bugs.python.org/issue16991#msg232825 : > > > Would this make it easy/faster to also have a DefaultOrderedDict > > (which can/could also be accomplished with > > .get(attr, []) and .setdefault(attr, [])? Really, I'm thinking more along the lines of a dict that only guarantees enumeration order (no integer indexed access) in the specific cases where people "need" it (ordered kwargs for e.g. the OrderedDict constructor would be another example), while being 'lightweight' enough (in terms of not having a lot of extra machinery dedicated to maintaining the order) to ultimately be used as the real dict implementation (and therefore usable for kwargs, class dictionaries, etc) Your comment's "DefaultOrderedDict" seems to be more about combining DefaultDict (default values) and OrderedDict (ordered all the time, indexed access) rather than anything like having a "default order". From wes.turner at gmail.com Thu Dec 18 17:47:35 2014 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 18 Dec 2014 10:47:35 -0600 Subject: [Python-ideas] A service to crawl +1s and URLs out of mailman archives In-Reply-To: <20141202143216.170c13cd@anarchist.wooz.org> References: <20141201163656.2654897d@anarchist.wooz.org> <20141202143216.170c13cd@anarchist.wooz.org> Message-ID: >>Is there any way to recognize a Lightweight Markup Language doctype >>declaration, in email? > Maybe a Content-Type? But can you elaborate one what you're thinking about? ```restructuredtext Often, mailing list and issue text gets *worked into* docs. ``` ReST in email may ultimately be a bit noisy and ultimately an unproductive feature. On Tue, Dec 2, 2014 at 1:32 PM, Barry Warsaw wrote: > > On Dec 02, 2014, at 04:53 AM, Wes Turner wrote: > > >Are the Python mailman instances upgraded to Mailman 3, with the Django > GUI? > > No, but we know there are people using it in production. I expect we'll > get > another beta before the end of the year and then I think it's time to start > planning an experimental deployment on pdo. We have a few experimental > lists > on mpo that we can convert and start playing with. > > >Is there any way to recognize a Lightweight Markup Language doctype > >declaration, in email? > > Maybe a Content-Type? But can you elaborate one what you're thinking > about? > > Cheers, > -Barry > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Dec 18 17:52:46 2014 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 18 Dec 2014 10:52:46 -0600 Subject: [Python-ideas] A service to crawl +1s and URLs out of mailman archives In-Reply-To: References: <20141201163656.2654897d@anarchist.wooz.org> <20141202143216.170c13cd@anarchist.wooz.org> Message-ID: cc'd here from https://westurner.github.io/wiki/ideas#open-source-mailing-list-extractor Open Source Mailing List Extractor ---------------------------------- Use cases: * https://mail.python.org/pipermail/python-ideas/2014-December/030228.html * incentivization of actionable crossreferences * PEP research * "is this actionable?" * "are we voting?" * Crawl/parse/extract links and +1 from given thread(s) * Detect a few standard link types: * Issue * Src * Doc * Ref * x-link * +1s with expandable snippets? (like ``grep -C``) * There could be configurable per-list link heuristics: * http[s] * Issue: https://bugs.python.org/issue(\d+) * Src: https://hg.python.org// * Src: https://github.com/// * Src: https://bitbucket.org/// * Patch/Attachment: http[s]://bugs.python.org/(file[\d]+)/ * Doc: https://docs.python.org// * Wiki: https://wiki.python.org/moin/ * Homepage: https://www.python.org/ * PyPI pkg: https://pypi.python.org/pypi/ * Warehouse pkg: https://warehouse.python.org/project/ * Wikipedia: https://[lang].wikipedia.org/wiki/ --> (dbpedia:) * Build: http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.4/builds/771 * ... JSON-LD RDF * This could - most efficiently - be added to mailman (e.g. in Postorious and/or HyperKitty) * http://mailman-bundler.readthedocs.org/en/latest/ * http://pythonhosted.org//mailman/ * https://mail.python.org/mailman/listinfo/mailman-developers ... looking forward Mailman3. On Thu, Dec 18, 2014 at 10:47 AM, Wes Turner wrote: > > >>Is there any way to recognize a Lightweight Markup Language doctype > >>declaration, in email? > > > Maybe a Content-Type? But can you elaborate one what you're thinking > about? > > ```restructuredtext > Often, mailing list and issue text gets *worked into* docs. > ``` > > ReST in email may ultimately be a bit noisy and ultimately an unproductive > feature. > > > On Tue, Dec 2, 2014 at 1:32 PM, Barry Warsaw wrote: > >> On Dec 02, 2014, at 04:53 AM, Wes Turner wrote: >> >> >Are the Python mailman instances upgraded to Mailman 3, with the Django >> GUI? >> >> No, but we know there are people using it in production. I expect we'll >> get >> another beta before the end of the year and then I think it's time to >> start >> planning an experimental deployment on pdo. We have a few experimental >> lists >> on mpo that we can convert and start playing with. >> >> >Is there any way to recognize a Lightweight Markup Language doctype >> >declaration, in email? >> >> Maybe a Content-Type? But can you elaborate one what you're thinking >> about? >> >> Cheers, >> -Barry >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Dec 18 18:01:00 2014 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 18 Dec 2014 11:01:00 -0600 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: <1418920079.23228.204489681.02FF696E@webmail.messagingengine.com> References: <20141217193056.46664962@fsol> <1418918963.18586.204483089.2CF3A673@webmail.messagingengine.com> <1418920079.23228.204489681.02FF696E@webmail.messagingengine.com> Message-ID: On Thu, Dec 18, 2014 at 10:27 AM, wrote: > > > > On Thu, Dec 18, 2014, at 11:18, Wes Turner wrote: > > > Would it be reasonable to make a lightweight "predictable dict" class > > > that makes a weaker guarantee, e.g. that the enumeration order will > > > match the insertion order in the case where it is filled from empty > with > > > no intervening deletions and not guaranteed in any other cases? > > > > Something like DefaultOrderedDict (like defaultdict)? > > > > From http://bugs.python.org/issue16991#msg232825 : > > > > > Would this make it easy/faster to also have a DefaultOrderedDict > > > (which can/could also be accomplished with > > > .get(attr, []) and .setdefault(attr, [])? > > Really, I'm thinking more along the lines of a dict that only guarantees > enumeration order (no integer indexed access) in the specific cases > where people "need" it (ordered kwargs for e.g. the OrderedDict > constructor would be another example), while being 'lightweight' enough > (in terms of not having a lot of extra machinery dedicated to > maintaining the order) to ultimately be used as the real dict > implementation (and therefore usable for kwargs, class dictionaries, > etc) I understand. More like the function of DEFAULT_PREDICATE_ORDERING here: https://github.com/westurner/healthref/blob/gh-pages/healthref.py#L100 . That would indeed be useful. What would you call it? -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Thu Dec 18 18:42:43 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 18 Dec 2014 12:42:43 -0500 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: <20141217193056.46664962@fsol> <1418918963.18586.204483089.2CF3A673@webmail.messagingengine.com> <1418920079.23228.204489681.02FF696E@webmail.messagingengine.com> Message-ID: <1418924563.41241.204521381.0D5E3278@webmail.messagingengine.com> On Thu, Dec 18, 2014, at 12:01, Wes Turner wrote: > I understand. More like the function of DEFAULT_PREDICATE_ORDERING > here: I have no idea what that does or why you think it is similar to what I said. From timothy.c.delaney at gmail.com Thu Dec 18 21:18:31 2014 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Fri, 19 Dec 2014 07:18:31 +1100 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: <1418918963.18586.204483089.2CF3A673@webmail.messagingengine.com> References: <20141217193056.46664962@fsol> <1418918963.18586.204483089.2CF3A673@webmail.messagingengine.com> Message-ID: On 19 December 2014 at 03:09, wrote: > > On Thu, Dec 18, 2014, at 02:05, Nick Coghlan wrote: > > As far as I'm aware, it's an ordered dictionary because that makes the > > default repr() predictable when binding arguments for a given function > > (in the absence of after-the-fact manipulation like the example in the > > docs that injects the default values as explicitly bound arguments). > > > > The inspect.signature() machinery includes quite a few things like > > that where the serialisation as a human readable string is considered > > as important then the programmatic representation. > > Would it be reasonable to make a lightweight "predictable dict" class > that makes a weaker guarantee, e.g. that the enumeration order will > match the insertion order in the case where it is filled from empty with > no intervening deletions and not guaranteed in any other cases? My understanding is that Raymond's alternative dict implementation works exactly like this, and is noted as an alternative for PEP 468: https://www.python.org/dev/peps/pep-0468/ . Discussion starts at: https://mail.python.org/pipermail/python-dev/2012-December/123028.html Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Dec 19 01:23:05 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 19 Dec 2014 10:23:05 +1000 Subject: [Python-ideas] A service to crawl +1s and URLs out of mailman archives In-Reply-To: References: <20141201163656.2654897d@anarchist.wooz.org> <20141202143216.170c13cd@anarchist.wooz.org> Message-ID: On 19 Dec 2014 02:53, "Wes Turner" wrote: > > > * This could - most efficiently - be added to mailman > (e.g. in Postorious and/or HyperKitty) > > * http://mailman-bundler.readthedocs.org/en/latest/ > * http://pythonhosted.org//mailman/ > * https://mail.python.org/mailman/listinfo/mailman-developers > > ... looking forward Mailman3. Indeed - I'm also looking forward to that on Fedora & Red Hat side of things, in addition to for the Python mailing lists :) I was actually thinking about this kind of feature in a HyperKitty context yesterday, after Marc-Andre Lemburg pointed out a new service called Loomio (www.loomio.org). That's essentially a web discussion forum that allows the use of Apache style voting to come to a consensus. It calls the four levels Agree, Abstain, Disagree, Block, and presents a summary of "current position statements" alongside the discussion. With a little bit of lightweight mark up to reduce false positives, I could see such a feature being a valuable addition to HyperKitty. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cfkaran2 at gmail.com Fri Dec 19 05:27:51 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Thu, 18 Dec 2014 23:27:51 -0500 Subject: [Python-ideas] Bug? Feature? setattr(foo, '3', 4) works! Message-ID: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> I accidentally discovered that the following works, at least in Python 3.4.2: >>> class foo(object): ... pass ... >>> setattr(foo, '3', 4) >>> dir(foo) ['3', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__'] >>> getattr(foo, '3') 4 >>> bar = foo() >>> dir(bar) ['3', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__'] >>> getattr(bar, '3') 4 >>> hasattr(foo, '3') True >>> hasattr(bar, '3') True However, the following doesn't work: >>> foo.3 File "", line 1 foo.3 ^ SyntaxError: invalid syntax >>> bar.3 File "", line 1 bar.3 ^ SyntaxError: invalid syntax I'd like to suggest that getattr(), setattr(), and hasattr() all be modified so that syntactically invalid statements raise SyntaxErrors. On a related note, are numbers defined in a module, or they are part of interpreter? Just to see how awful I could make things, I tried to extend the above to redefine 3 to be 4. Fortunately (unfortunately?) I couldn't find a way, but I'm curious if there is a pythonic way of making this happen, and if there is, I'd like to suggest making that impossible ASAP. Thanks, Cem Karan From rosuav at gmail.com Fri Dec 19 06:15:13 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 19 Dec 2014 16:15:13 +1100 Subject: [Python-ideas] Bug? Feature? setattr(foo, '3', 4) works! In-Reply-To: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> References: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> Message-ID: On Fri, Dec 19, 2014 at 3:27 PM, Cem Karan wrote: > I'd like to suggest that getattr(), setattr(), and hasattr() all be modified so that syntactically invalid statements raise SyntaxErrors. Is there actually a need to have them check like that? :) After all, attributes are just stored in a dictionary. You want syntactically-invalid attributes? Sure, no problem. > On a related note, are numbers defined in a module, or they are part of interpreter? Just to see how awful I could make things, I tried to extend the above to redefine 3 to be 4. Fortunately (unfortunately?) I couldn't find a way, but I'm curious if there is a pythonic way of making this happen, and if there is, I'd like to suggest making that impossible ASAP. > Syntax comes first. If you want to play with redefinitions like that, it'd be simpler to stuff stuff into globals(), which is also simply a dictionary. >>> globals()["3"]=4 >>> 3 3 >>> globals()["3"] 4 When you type 3, Python compiles that into the integer 3, not into a name lookup. Now, you *can* fiddle with this kind of thing using ctypes, though I don't know how many other interpreters (PyPy, Jython, etc) will let you shoot yourself in that particular foot. But if you'd like to play around with this kind of thing, check out the 'ast' module and try compiling code to AST, then fiddling with it a bit, then finishing compilation and execution. And python-list will probably enjoy discussing the random things you can accomplish with that :) It's good fun. ChrisA From encukou at gmail.com Fri Dec 19 10:07:58 2014 From: encukou at gmail.com (Petr Viktorin) Date: Fri, 19 Dec 2014 10:07:58 +0100 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: <20141217193056.46664962@fsol> <1418918963.18586.204483089.2CF3A673@webmail.messagingengine.com> Message-ID: On Thu, Dec 18, 2014 at 9:18 PM, Tim Delaney wrote: > On 19 December 2014 at 03:09, wrote: >> >> On Thu, Dec 18, 2014, at 02:05, Nick Coghlan wrote: >> > As far as I'm aware, it's an ordered dictionary because that makes the >> > default repr() predictable when binding arguments for a given function >> > (in the absence of after-the-fact manipulation like the example in the >> > docs that injects the default values as explicitly bound arguments). >> > >> > The inspect.signature() machinery includes quite a few things like >> > that where the serialisation as a human readable string is considered >> > as important then the programmatic representation. >> >> Would it be reasonable to make a lightweight "predictable dict" class >> that makes a weaker guarantee, e.g. that the enumeration order will >> match the insertion order in the case where it is filled from empty with >> no intervening deletions and not guaranteed in any other cases? I believe that if there's a new predictabledict type, and deleting from it would invalidate the order, then predictabledict should not support deletions. Perhaps it should be all-out immutable to prevent surprises. It would break backwards compatibility if that was used for **kwargs, though. > My understanding is that Raymond's alternative dict implementation works > exactly like this, and is noted as an alternative for PEP 468: > https://www.python.org/dev/peps/pep-0468/ . > > Discussion starts at: > https://mail.python.org/pipermail/python-dev/2012-December/123028.html > From wes.turner at gmail.com Fri Dec 19 10:50:34 2014 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 19 Dec 2014 03:50:34 -0600 Subject: [Python-ideas] A service to crawl +1s and URLs out of mailman archives In-Reply-To: References: <20141201163656.2654897d@anarchist.wooz.org> <20141202143216.170c13cd@anarchist.wooz.org> Message-ID: On Thu, Dec 18, 2014 at 6:23 PM, Nick Coghlan wrote: > > [...] Loomio (www.loomio.org). > > That's essentially a web discussion forum that allows the use of Apache > style voting to come to a consensus. It calls the four levels Agree, > Abstain, Disagree, Block, and presents a summary of "current position > statements" alongside the discussion. > What a great idea! > You?ll be prompted to make a short statement about the reason for your > decision. This makes it easy to see a summary of what everyone thinks and > why. You can change your mind and edit your decision freely until the > proposal closes I took a course in "Collaboration" offered through a local university in which one of our applied objectives was to collaboratively write a free ebook (now on LuLu) about collaboration; it was a great way to study for the final, in regards to Collaborative Engineering. "6 Patterns of Collaboration": Generate, Reduce, Clarify, Organize, Evaluate, Build Consensus. In band with email and mailing lists would be great. Recently, I made the mistake of trying to approximate the https://github.com/twitter/twitter-text JS code which matches @username and #\w (#RTLunicodew\). Syntax highlighting could be useful. TL;DR I really like the "Agree, Abstain, Disagree, Block" distinction; it would be great to somehow integrate with email, somehow. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Fri Dec 19 11:39:33 2014 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 19 Dec 2014 04:39:33 -0600 Subject: [Python-ideas] A service to crawl +1s and URLs out of mailman archives In-Reply-To: References: <20141201163656.2654897d@anarchist.wooz.org> <20141202143216.170c13cd@anarchist.wooz.org> Message-ID: > TL;DR I really like the "Agree, Abstain, Disagree, Block" distinction; it would be great to somehow integrate with email, somehow. I can't help but wonder whether this could be implemented by extending OpenAnnotation [1][2][3] (JSON-LD RDF) for **any URI**, like the Hypothesis [4] OpenAnnotation web service and Annotator JS / browser extension. [5] [1] http://openannotation.org/spec/core/ [2] http://openannotation.org/spec/core/specific.html#Selectors [3] http://www.w3.org/annotation/ [4] https://github.com/hypothesis/h (Pyramid) [5] https://github.com/hypothesis/annotator (OKFN) Alas, mailing list posts do not have easily-gettable URIs (like Reddit comment permalinks). On Fri, Dec 19, 2014 at 3:50 AM, Wes Turner wrote: > > > > On Thu, Dec 18, 2014 at 6:23 PM, Nick Coghlan wrote: >> >> [...] Loomio (www.loomio.org). >> >> That's essentially a web discussion forum that allows the use of Apache >> style voting to come to a consensus. It calls the four levels Agree, >> Abstain, Disagree, Block, and presents a summary of "current position >> statements" alongside the discussion. >> > What a great idea! > > > You?ll be prompted to make a short statement about the reason for your >> decision. This makes it easy to see a summary of what everyone thinks and >> why. You can change your mind and edit your decision freely until the >> proposal closes > > > I took a course in "Collaboration" offered through a local university in > which one of our applied objectives was to collaboratively write a free > ebook (now on LuLu) about collaboration; it was a great way to study for > the final, in regards to Collaborative Engineering. > > "6 Patterns of Collaboration": Generate, Reduce, Clarify, Organize, > Evaluate, Build Consensus. > > In band with email and mailing lists would be great. Recently, I made the > mistake of trying to approximate the > https://github.com/twitter/twitter-text JS code which matches @username > and #\w (#RTLunicodew\). Syntax highlighting could be useful. > > TL;DR I really like the "Agree, Abstain, Disagree, Block" distinction; it > would be great to somehow integrate with email, somehow. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cfkaran2 at gmail.com Fri Dec 19 11:57:43 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Fri, 19 Dec 2014 05:57:43 -0500 Subject: [Python-ideas] Bug? Feature? setattr(foo, '3', 4) works! In-Reply-To: References: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> Message-ID: <5DB1B6A2-6982-47F8-8A89-17A705F3290B@gmail.com> On Dec 19, 2014, at 12:15 AM, Chris Angelico wrote: > On Fri, Dec 19, 2014 at 3:27 PM, Cem Karan wrote: >> I'd like to suggest that getattr(), setattr(), and hasattr() all be modified so that syntactically invalid statements raise SyntaxErrors. > > Is there actually a need to have them check like that? :) After all, > attributes are just stored in a dictionary. You want > syntactically-invalid attributes? Sure, no problem. Its only needed from a 'Principle of Least Astonishment' point of view. When I realized I could work my way around the syntax by using getattr()/setattr(), I was astonished. I really didn't expect it to work, and I would MUCH prefer that trying to use getattr()/setattr() in this way would have the exact same results as normal attribute access. >> On a related note, are numbers defined in a module, or they are part of interpreter? Just to see how awful I could make things, I tried to extend the above to redefine 3 to be 4. Fortunately (unfortunately?) I couldn't find a way, but I'm curious if there is a pythonic way of making this happen, and if there is, I'd like to suggest making that impossible ASAP. >> > > Syntax comes first. If you want to play with redefinitions like that, > it'd be simpler to stuff stuff into globals(), which is also simply a > dictionary. > >>>> globals()["3"]=4 >>>> 3 > 3 >>>> globals()["3"] > 4 > > When you type 3, Python compiles that into the integer 3, not into a > name lookup. Now, you *can* fiddle with this kind of thing using > ctypes, though I don't know how many other interpreters (PyPy, Jython, > etc) will let you shoot yourself in that particular foot. But if you'd > like to play around with this kind of thing, check out the 'ast' > module and try compiling code to AST, then fiddling with it a bit, > then finishing compilation and execution. And python-list will > probably enjoy discussing the random things you can accomplish with > that :) It's good fun. I feel a wave of morbid curiosity coming over me... I think I'm going to have to try that, just to see what I can do. I'm terrified to think that I might actually be able to make it happen... O_o. But, getting back to the main question, is this a bug or a feature? I personally feel like this is a bug, and I'd like to both clarify it in the language spec, and have cpython modified to enforce the syntax, regardless of how you try to mess with an attribute. How does everyone else feel about this? Thanks, Cem Karan From ncoghlan at gmail.com Fri Dec 19 12:00:17 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 19 Dec 2014 21:00:17 +1000 Subject: [Python-ideas] Bug? Feature? setattr(foo, '3', 4) works! In-Reply-To: References: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> Message-ID: On 19 December 2014 at 15:15, Chris Angelico wrote: > On Fri, Dec 19, 2014 at 3:27 PM, Cem Karan wrote: > > I'd like to suggest that getattr(), setattr(), and hasattr() all be > modified so that syntactically invalid statements raise SyntaxErrors. > > Is there actually a need to have them check like that? :) After all, > attributes are just stored in a dictionary. You want > syntactically-invalid attributes? Sure, no problem. > As far as I'm aware, it's actually an implementation defined behaviour - implementations are permitted, but not required, to use string-only dictionaries for namespaces. (Although I can't currently lay my hands on anything in the language reference that actually *says* that...) Most implementations will go with whichever is faster in their particular case (for CPython, that means using the standard permissive dict type, for Jython, it means only allowing strings, for other implementations, I'm not sure). CPython is highly unlikely to ever break compatibility over this, but it *is* a potential cross-implementation portability problem when folks rely on it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Dec 19 12:03:10 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 19 Dec 2014 21:03:10 +1000 Subject: [Python-ideas] Bug? Feature? setattr(foo, '3', 4) works! In-Reply-To: <5DB1B6A2-6982-47F8-8A89-17A705F3290B@gmail.com> References: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> <5DB1B6A2-6982-47F8-8A89-17A705F3290B@gmail.com> Message-ID: On 19 December 2014 at 20:57, Cem Karan wrote: > But, getting back to the main question, is this a bug or a feature? I > personally feel like this is a bug, and I'd like to both clarify it in the > language spec, and have cpython modified to enforce the syntax, regardless > of how you try to mess with an attribute. How does everyone else feel > about this? > Namespaces are just dictionaries. The one thing implementations may do is to place a type restriction on the keys, so they throw TypeError if you try to use something other than str (although *CPython* doesn't do that). getattr/setattr/delattr would never throw SyntaxError though - that's only thrown by the compiler. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Dec 19 12:10:17 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 19 Dec 2014 21:10:17 +1000 Subject: [Python-ideas] A service to crawl +1s and URLs out of mailman archives In-Reply-To: References: <20141201163656.2654897d@anarchist.wooz.org> <20141202143216.170c13cd@anarchist.wooz.org> Message-ID: On 19 December 2014 at 20:39, Wes Turner wrote: > > TL;DR I really like the "Agree, Abstain, Disagree, Block" distinction; > it would be great to somehow integrate with email, somehow. > > I can't help but wonder whether this could be implemented by extending > OpenAnnotation [1][2][3] (JSON-LD RDF) for **any URI**, like the Hypothesis > [4] OpenAnnotation web service and Annotator JS / browser extension. [5] > > [1] http://openannotation.org/spec/core/ > [2] http://openannotation.org/spec/core/specific.html#Selectors > [3] http://www.w3.org/annotation/ > [4] https://github.com/hypothesis/h (Pyramid) > [5] https://github.com/hypothesis/annotator (OKFN) > > Alas, mailing list posts do not have easily-gettable URIs (like Reddit > comment permalinks). > > We're getting fairly well off-topic for python-ideas now (since anything we do in this area will likely only be by way of upgrading to Mailman 3 and HyperKitty), but I do highly recommend getting in touch with the HyperKitty devs on their mailing list. It takes advantage of the new archiving design in Mailman 3 to add the archive server permalink for the post to the footer before it gets sent to the list members. Cheers, Nick. P.S. Barry's write-up of the Mailman 3 architecture is also well worth a read in general: http://www.aosabook.org/en/mailman.html -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Dec 19 12:11:39 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 19 Dec 2014 22:11:39 +1100 Subject: [Python-ideas] Bug? Feature? setattr(foo, '3', 4) works! In-Reply-To: References: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> Message-ID: On Fri, Dec 19, 2014 at 10:00 PM, Nick Coghlan wrote: > Most implementations will go with whichever is faster in their particular > case (for CPython, that means using the standard permissive dict type, for > Jython, it means only allowing strings, for other implementations, I'm not > sure). What do you mean by "only allowing strings"? CPython doesn't allow non-string attributes, and Jython does allow non-identifier attributes: Jython 2.5.3 (, Oct 8 2014, 03:39:09) [OpenJDK 64-Bit Server VM (Oracle Corporation)] on java1.7.0_65 Type "help", "copyright", "credits" or "license" for more information. >>> class x(object): pass ... >>> x=x() >>> setattr(x,"1","2") >>> getattr(x,"1") '2' >>> setattr(x,1,2) Traceback (most recent call last): File "", line 1, in TypeError: setattr(): attribute name must be string Python 3.5.0a0 (default:64e45efdc3e2, Dec 11 2014, 11:52:01) [GCC 4.7.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> class x: pass ... >>> x=x() >>> setattr(x,"1",2) >>> getattr(x,"1") 2 >>> setattr(x,1,2) Traceback (most recent call last): File "", line 1, in TypeError: attribute name must be string, not 'int' (I'm not sure why there's the empty string in the Jython version tag; this is what I got by apt-getting jython on Debian Jessie.) In any case, it's probably time this moved off python-ideas. ChrisA From cfkaran2 at gmail.com Fri Dec 19 12:17:18 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Fri, 19 Dec 2014 06:17:18 -0500 Subject: [Python-ideas] Bug? Feature? setattr(foo, '3', 4) works! In-Reply-To: References: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> <5DB1B6A2-6982-47F8-8A89-17A705F3290B@gmail.com> Message-ID: On Dec 19, 2014, at 6:03 AM, Nick Coghlan wrote: > On 19 December 2014 at 20:57, Cem Karan wrote: > But, getting back to the main question, is this a bug or a feature? I personally feel like this is a bug, and I'd like to both clarify it in the language spec, and have cpython modified to enforce the syntax, regardless of how you try to mess with an attribute. How does everyone else feel about this? > > Namespaces are just dictionaries. The one thing implementations may do is to place a type restriction on the keys, so they throw TypeError if you try to use something other than str (although *CPython* doesn't do that). > > getattr/setattr/delattr would never throw SyntaxError though - that's only thrown by the compiler. I see what you're saying, but I feel like having (get|set|has)attr throw TypeError instead of SyntaxError would itself be astonishing in this case. The feature I'm after is that "foo.3" results in exactly the same behavior as "getattr(foo, '3')". Thanks, Cem Karan From cfkaran2 at gmail.com Fri Dec 19 12:20:10 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Fri, 19 Dec 2014 06:20:10 -0500 Subject: [Python-ideas] Bug? Feature? setattr(foo, '3', 4) works! In-Reply-To: References: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> Message-ID: <8B2923C2-5028-44C0-A1EF-90A111AC52E6@gmail.com> On Dec 19, 2014, at 6:11 AM, Chris Angelico wrote: > On Fri, Dec 19, 2014 at 10:00 PM, Nick Coghlan wrote: >> Most implementations will go with whichever is faster in their particular >> case (for CPython, that means using the standard permissive dict type, for >> Jython, it means only allowing strings, for other implementations, I'm not >> sure). > > What do you mean by "only allowing strings"? CPython doesn't allow > non-string attributes, and Jython does allow non-identifier > attributes: > > Jython 2.5.3 (, Oct 8 2014, 03:39:09) > [OpenJDK 64-Bit Server VM (Oracle Corporation)] on java1.7.0_65 > Type "help", "copyright", "credits" or "license" for more information. >>>> class x(object): pass > ... >>>> x=x() >>>> setattr(x,"1","2") >>>> getattr(x,"1") > '2' >>>> setattr(x,1,2) > Traceback (most recent call last): > File "", line 1, in > TypeError: setattr(): attribute name must be string > > Python 3.5.0a0 (default:64e45efdc3e2, Dec 11 2014, 11:52:01) > [GCC 4.7.2] on linux > Type "help", "copyright", "credits" or "license" for more information. >>>> class x: pass > ... >>>> x=x() >>>> setattr(x,"1",2) >>>> getattr(x,"1") > 2 >>>> setattr(x,1,2) > Traceback (most recent call last): > File "", line 1, in > TypeError: attribute name must be string, not 'int' > > > (I'm not sure why there's the empty string in the Jython version tag; > this is what I got by apt-getting jython on Debian Jessie.) > > In any case, it's probably time this moved off python-ideas. What's the best place to move it to? If this is a bug, I'll write up a bug report, but if this is a change in the language definition, then it might be best to keep it here. Thanks, Cem Karan From rosuav at gmail.com Fri Dec 19 12:23:33 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 19 Dec 2014 22:23:33 +1100 Subject: [Python-ideas] Bug? Feature? setattr(foo, '3', 4) works! In-Reply-To: <8B2923C2-5028-44C0-A1EF-90A111AC52E6@gmail.com> References: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> <8B2923C2-5028-44C0-A1EF-90A111AC52E6@gmail.com> Message-ID: On Fri, Dec 19, 2014 at 10:20 PM, Cem Karan wrote: > What's the best place to move it to? If this is a bug, I'll write up a bug report, but if this is a change in the language definition, then it might be best to keep it here. python-list at python.org I think; basically, it's now chatting about the language, rather than dealing with a known and recognized bug. If, as a result of a python-list discussion, a language change is recommended, you can bring a post-discussion proposal back to python-ideas, but more likely we'll just chat happily and decide that no change is needed. :) ChrisA From cfkaran2 at gmail.com Fri Dec 19 12:38:11 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Fri, 19 Dec 2014 06:38:11 -0500 Subject: [Python-ideas] Bug? Feature? setattr(foo, '3', 4) works! In-Reply-To: <8B2923C2-5028-44C0-A1EF-90A111AC52E6@gmail.com> References: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> <8B2923C2-5028-44C0-A1EF-90A111AC52E6@gmail.com> Message-ID: On Dec 19, 2014, at 6:20 AM, Cem Karan wrote: > On Dec 19, 2014, at 6:11 AM, Chris Angelico wrote: >> >> >> In any case, it's probably time this moved off python-ideas. > > What's the best place to move it to? If this is a bug, I'll write up a bug report, but if this is a change in the language definition, then it might be best to keep it here. Per Chris's suggestion, I'm moving this to python-list, and if any strong preferences come up I'll bring them back here. Thanks, Cem Karan From ncoghlan at gmail.com Fri Dec 19 14:46:38 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 19 Dec 2014 23:46:38 +1000 Subject: [Python-ideas] Bug? Feature? setattr(foo, '3', 4) works! In-Reply-To: References: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> <5DB1B6A2-6982-47F8-8A89-17A705F3290B@gmail.com> Message-ID: On 19 December 2014 at 21:17, Cem Karan wrote: > > On Dec 19, 2014, at 6:03 AM, Nick Coghlan wrote: > > > On 19 December 2014 at 20:57, Cem Karan wrote: > > But, getting back to the main question, is this a bug or a feature? I > personally feel like this is a bug, and I'd like to both clarify it in the > language spec, and have cpython modified to enforce the syntax, regardless > of how you try to mess with an attribute. How does everyone else feel > about this? > > > > Namespaces are just dictionaries. The one thing implementations may do > is to place a type restriction on the keys, so they throw TypeError if you > try to use something other than str (although *CPython* doesn't do that). > > > > getattr/setattr/delattr would never throw SyntaxError though - that's > only thrown by the compiler. > > I see what you're saying, but I feel like having (get|set|has)attr throw > TypeError instead of SyntaxError would itself be astonishing in this case. > The feature I'm after is that "foo.3" results in exactly the same behavior > as "getattr(foo, '3')". > I can probably save you some time: not going to happen :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Dec 19 14:54:25 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 19 Dec 2014 23:54:25 +1000 Subject: [Python-ideas] Bug? Feature? setattr(foo, '3', 4) works! In-Reply-To: References: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> <5DB1B6A2-6982-47F8-8A89-17A705F3290B@gmail.com> Message-ID: On 19 December 2014 at 23:46, Nick Coghlan wrote: > On 19 December 2014 at 21:17, Cem Karan wrote: > >> >> On Dec 19, 2014, at 6:03 AM, Nick Coghlan wrote: >> >> > On 19 December 2014 at 20:57, Cem Karan wrote: >> > But, getting back to the main question, is this a bug or a feature? I >> personally feel like this is a bug, and I'd like to both clarify it in the >> language spec, and have cpython modified to enforce the syntax, regardless >> of how you try to mess with an attribute. How does everyone else feel >> about this? >> > >> > Namespaces are just dictionaries. The one thing implementations may do >> is to place a type restriction on the keys, so they throw TypeError if you >> try to use something other than str (although *CPython* doesn't do that). >> > >> > getattr/setattr/delattr would never throw SyntaxError though - that's >> only thrown by the compiler. >> >> I see what you're saying, but I feel like having (get|set|has)attr throw >> TypeError instead of SyntaxError would itself be astonishing in this case. >> The feature I'm after is that "foo.3" results in exactly the same behavior >> as "getattr(foo, '3')". >> > > I can probably save you some time: not going to happen :) > To elaborate on that a bit: >>> x.a 1 >>> getattr(x, 'a') 1 >>> x.__dict__['a'] 1 The ".a" attribute access syntax is ultimately just syntactic sugar for a dictionary lookup with the string 'a'. That syntactic sugar is what applies the restriction to being a valid identifier, not the underlying namespace object. Even for string values, you can do syntactically unacceptable lookups via the underlying namespace mapping: >>> setattr(x, 'not a legal identifier', 2) >>> getattr(x, 'not a legal identifier') 2 >>> x.__dict__['not a legal identifier'] 2 You'll find that kind of behaviour in a few different areas - syntactic restrictions aren't actually present in the underlying runtime machinery, they only exist at compile time. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Fri Dec 19 16:26:46 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Fri, 19 Dec 2014 10:26:46 -0500 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? In-Reply-To: References: <20141217193056.46664962@fsol> <1418918963.18586.204483089.2CF3A673@webmail.messagingengine.com> Message-ID: <1419002806.434290.204882797.51C73C21@webmail.messagingengine.com> On Fri, Dec 19, 2014, at 04:07, Petr Viktorin wrote: > I believe that if there's a new predictabledict type, and deleting > from it would invalidate the order, then predictabledict should not > support deletions. This would defeat the broader idea of using an implementation with such properties as the standard dict (which would allow its use for kwargs, class dictionaries, etc). What about adding a flag to indicate whether it is in an ordered state (i.e. either there have been no deletions since it was last empty, or there have been no insertions since the first deletion after it was last empty, or whatever other status applies to the implementation). From barry at python.org Fri Dec 19 17:25:17 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 19 Dec 2014 11:25:17 -0500 Subject: [Python-ideas] A service to crawl +1s and URLs out of mailman archives In-Reply-To: References: <20141201163656.2654897d@anarchist.wooz.org> <20141202143216.170c13cd@anarchist.wooz.org> Message-ID: <20141219112517.2ce1dc83@limelight.wooz.org> On Dec 19, 2014, at 04:39 AM, Wes Turner wrote: >Alas, mailing list posts do not have easily-gettable URIs (like Reddit >comment permalinks). Actually, we have a proposal for that, and yes it's implemented in MM3: http://wiki.list.org/display/DEV/Stable+URLs Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From brett at python.org Fri Dec 19 18:25:03 2014 From: brett at python.org (Brett Cannon) Date: Fri, 19 Dec 2014 17:25:03 +0000 Subject: [Python-ideas] Why does BoundArguments use an OrderedDict? References: <20141217193056.46664962@fsol> Message-ID: On Thu, Dec 18, 2014, 04:46 Antony Lee wrote: 2014-12-17 23:05 GMT-08:00 Nick Coghlan : On 18 December 2014 at 12:57, Guido van Rossum wrote: > On Wed, Dec 17, 2014 at 6:15 PM, Antony Lee wrote: >> >> The discussion has drifted towards improving OrderedDict (something I >> certainly approve), but the semantic question is still there: why should >> BoundArguments.arguments be ordered by the parameter order? For example, >> the recipe just below in the docs, for filling in the remaining default >> arguments, breaks that ordering, without even mentioning that. > > > Given that the answer hasn't been answered yet, perhaps nobody remembers the > reason any more. But why does it bother you so much? In my experience the > inspect module wasn't written for performance but for functionality. If it > really hurts your ability to do something that you need to be fast, perhaps > you can supply a patch? Make sure to run the tests and update the docs as > needed. Actually the second example in PEP362 uses bind as an argument type-checker, which should arguably be as fast as possible. I'm opening an issue with a patch right now. As the author of PEP 362 I can tell you that was an example and not something to optimize for. If you want to type check your code then you should do it offline or only during debugging in which case performance is not an issue. -Brett As far as I'm aware, it's an ordered dictionary because that makes the default repr() predictable when binding arguments for a given function (in the absence of after-the-fact manipulation like the example in the docs that injects the default values as explicitly bound arguments). That can't really be the case as BoundArguments doesn't define a __repr__... (and OrderedDict's __repr__ is a bit too verbose IMO, but that's another issue and little can be done about it anyways). Now that I delved into the details of BoundArgument's implementation, I'll also note that __eq__ also checks the arguments order, and again additional binding will mess that up: s = signature(lambda x=None, y=None: None) ba0 = s.bind() ba1 = s.bind(x=None) ba2 = s.bind(y=None) Should ba0, ba1 and ba2 compare now equal or not? As it is, we'll have ba0 == ba1 != ba2, which doesn't strike me as particularly reasonable. Antony The inspect.signature() machinery includes quite a few things like that where the serialisation as a human readable string is considered as important then the programmatic representation. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sat Dec 20 00:53:18 2014 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 19 Dec 2014 17:53:18 -0600 Subject: [Python-ideas] A service to crawl +1s and URLs out of mailman archives In-Reply-To: <20141219112517.2ce1dc83@limelight.wooz.org> References: <20141201163656.2654897d@anarchist.wooz.org> <20141202143216.170c13cd@anarchist.wooz.org> <20141219112517.2ce1dc83@limelight.wooz.org> Message-ID: > P.S. Barry's write-up of the Mailman 3 architecture is also well worth a read in general: http://www.aosabook.org/en/mailman.html Thanks! On Fri, Dec 19, 2014 at 10:25 AM, Barry Warsaw wrote: > On Dec 19, 2014, at 04:39 AM, Wes Turner wrote: > > >Alas, mailing list posts do not have easily-gettable URIs (like Reddit > >comment permalinks). > > Actually, we have a proposal for that, and yes it's implemented in MM3: > > http://wiki.list.org/display/DEV/Stable+URLs #Stable URL calculation : Subject: An important message Date: Wed, 04 Jul 2007 16:49:58 +0900 Message-ID: <87myycy5eh.fsf at uwakimon.sk.tsukuba.ac.jp> X-Message-ID-Hash: JJIGKPKB6CVDX6B2CUG4IHAJRIQIOUTP List-Archive: http://mail.python.org/archives/mailman-developers Archived-At: > http://mail.python.org/archives/mailman-developers/JJIGKPKB6CVDX6B2CUG4IHAJRIQIOUTP -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Dec 20 01:55:37 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 19 Dec 2014 16:55:37 -0800 Subject: [Python-ideas] Type Hinting Kick-off Message-ID: A few months ago we had a long discussion about type hinting. I've thought a lot more about this. I've written up what I think is a decent "theory" document -- writing it down like this certainly helped *me* get a lot of clarity about some of the important issues. https://quip.com/r69HA9GhGa7J I should thank Jeremy Siek for his blog post about Gradual Typing, Jukka Lehtosalo for mypy (whose notation I am mostly borrowing), and Jim Baker for pushing for an in-person meeting where we all got a better understanding of several issues. There's also a PEP draft, written by ?ukasz Langa and revised by him based on notes from the above-mentioned in-person meeting; unfortunately it is still a bit out of date and I didn't have time to update it yet. Instead of working on the PEP, I tried to implement a conforming version of typing.py, for which I also ran out of time -- then I decided to just write up an explanation of the theory. I am still hoping to get a PEP out for discussion in early January, and I am aiming for provisional acceptance by PyCon Montr?al, which should allow a first version of typing.py to be included with Python 3.5 alpha 4. If you are wondering how I can possibly meet that schedule: (a) the entire runtime component proposal can be implemented as a single pure-Python module: hence the use of square brackets for generic types; (b) a static type checker is not part of the proposal: you can use mypy, or write your own. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Dec 20 02:36:56 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 20 Dec 2014 11:36:56 +1000 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: Message-ID: On 20 December 2014 at 10:55, Guido van Rossum wrote: > A few months ago we had a long discussion about type hinting. I've thought > a lot more about this. I've written up what I think is a decent "theory" > document -- writing it down like this certainly helped *me* get a lot of > clarity about some of the important issues. > > https://quip.com/r69HA9GhGa7J > > I should thank Jeremy Siek for his blog post about Gradual Typing, Jukka > Lehtosalo for mypy (whose notation I am mostly borrowing), and Jim Baker > for pushing for an in-person meeting where we all got a better > understanding of several issues. > > There's also a PEP draft, written by ?ukasz Langa and revised by him based > on notes from the above-mentioned in-person meeting; unfortunately it is > still a bit out of date and I didn't have time to update it yet. Instead of > working on the PEP, I tried to implement a conforming version of typing.py, > for which I also ran out of time -- then I decided to just write up an > explanation of the theory. > This looks like a great direction to me. While I know it's not the primary purpose, a multidispatch library built on top of it could potentially do wonders for cleaning up some of the ugliness in the URL parsing libraries (which have quite a few of those "all str, or all bytes, but not a mixture" style interfaces). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Dec 20 02:59:52 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 19 Dec 2014 17:59:52 -0800 Subject: [Python-ideas] Type Hinting Kick-off -- overloading Message-ID: On Fri, Dec 19, 2014 at 5:36 PM, Nick Coghlan wrote: > > > On 20 December 2014 at 10:55, Guido van Rossum wrote: > >> A few months ago we had a long discussion about type hinting. I've >> thought a lot more about this. I've written up what I think is a decent >> "theory" document -- writing it down like this certainly helped *me* get a >> lot of clarity about some of the important issues. >> >> https://quip.com/r69HA9GhGa7J >> [...] >> > > This looks like a great direction to me. While I know it's not the primary > purpose, a multidispatch library built on top of it could potentially do > wonders for cleaning up some of the ugliness in the URL parsing libraries > (which have quite a few of those "all str, or all bytes, but not a mixture" > style interfaces). > Right. Unfortunately we don't have a great proposal for a multidispatch *syntax*. Here are the docs for mypy's @overload decorator http://mypy.readthedocs.org/en/latest/function_overloading.html but the implementation requires using peeking into the call frame's locals via sys._getframe(). (Don't be distracted by the fact that mypy currently doesn't record the parameter value for parameterized types, e.g. Sequence[int] -- I have a simple enough solution for that, but it doesn't help the overload syntax.) The requirements for an overloading syntax are complex; it must be possible to implement it efficiently for simple cases, but it must also support complex overloading schemes (not everybody needs to dispatch just on str vs. bytes :-), it must diagnose ambiguities clearly (e.g. showing near misses if no exact match can be found, and showing what went wrong if multiple matches apply). It must also be easy to parse for a static checker like mypy. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Dec 20 03:19:54 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 20 Dec 2014 12:19:54 +1000 Subject: [Python-ideas] Type Hinting Kick-off -- overloading In-Reply-To: References: Message-ID: On 20 December 2014 at 11:59, Guido van Rossum wrote: > On Fri, Dec 19, 2014 at 5:36 PM, Nick Coghlan wrote: >> >> >> On 20 December 2014 at 10:55, Guido van Rossum wrote: >> >>> A few months ago we had a long discussion about type hinting. I've >>> thought a lot more about this. I've written up what I think is a decent >>> "theory" document -- writing it down like this certainly helped *me* get a >>> lot of clarity about some of the important issues. >>> >>> https://quip.com/r69HA9GhGa7J >>> [...] >>> >> >> This looks like a great direction to me. While I know it's not the >> primary purpose, a multidispatch library built on top of it could >> potentially do wonders for cleaning up some of the ugliness in the URL >> parsing libraries (which have quite a few of those "all str, or all bytes, >> but not a mixture" style interfaces). >> > > Right. Unfortunately we don't have a great proposal for a multidispatch > *syntax*. Here are the docs for mypy's @overload decorator > http://mypy.readthedocs.org/en/latest/function_overloading.html but the > implementation requires using peeking into the call frame's locals via > sys._getframe(). (Don't be distracted by the fact that mypy currently > doesn't record the parameter value for parameterized types, e.g. > Sequence[int] -- I have a simple enough solution for that, but it doesn't > help the overload syntax.) > Ugh, I hadn't fully thought through the consequences of the type annotations being tracked separately from the assigned values when it came to actually doing the dispatch. > The requirements for an overloading syntax are complex; it must be > possible to implement it efficiently for simple cases, but it must also > support complex overloading schemes (not everybody needs to dispatch just > on str vs. bytes :-), it must diagnose ambiguities clearly (e.g. showing > near misses if no exact match can be found, and showing what went wrong if > multiple matches apply). It must also be easy to parse for a static checker > like mypy. > I personally like the explicit registration style used for functools.singledispatch, but that still doesn't help with the problem of getting access to type annotation data when *doing* the dispatch. Perhaps it's another case like the introduction of annotations themselves, where it may be better to leave that higher level capability out of the initial iteration of the stdlib proposal, and instead look to standardise it later after we have a broader base of experience with the underlying gradual typing support? Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Dec 20 06:33:06 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 20 Dec 2014 16:33:06 +1100 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: Message-ID: <20141220053306.GE20332@ando.pearwood.info> On Fri, Dec 19, 2014 at 04:55:37PM -0800, Guido van Rossum wrote: > A few months ago we had a long discussion about type hinting. I've thought > a lot more about this. I've written up what I think is a decent "theory" > document -- writing it down like this certainly helped *me* get a lot of > clarity about some of the important issues. > > https://quip.com/r69HA9GhGa7J If anyone else is having trouble reading the page in their browser, this seems to work perfectly for me under Linux: wget --no-check-certificate https://quip.com/r69HA9GhGa7J mv r69HA9GhGa7J r69HA9GhGa7J.html links r69HA9GhGa7J.html Using links https://quip.com/r69HA9GhGa7J also works, if you don't mind reading grey text on a light-grey background. -- Steven From steve at pearwood.info Sat Dec 20 08:59:45 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 20 Dec 2014 18:59:45 +1100 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: Message-ID: <20141220075945.GF20332@ando.pearwood.info> On Fri, Dec 19, 2014 at 04:55:37PM -0800, Guido van Rossum wrote: > A few months ago we had a long discussion about type hinting. I've thought > a lot more about this. I've written up what I think is a decent "theory" > document -- writing it down like this certainly helped *me* get a lot of > clarity about some of the important issues. > > https://quip.com/r69HA9GhGa7J Very interesting indeed. Some questions, which you may not have answers to yet :-) (1) Under "General rules", you state "No type can be subclassed unless stated." That's not completely clear to me. I presume you are talking about the special types like Union, Generic, Sequence, Tuple, Any etc. Is that correct? (2) Under "Types", you give an example Tuple[t1, t2, ...], a tuple whose items are instances of t1 etc. To be more concrete, a declaration of Tuple[int, float, str] will mean "a tuple with exactly three items, the first item must be an int, the second item must be a float, the third item must be a string." Correct? (3) But there's no way of declaring "a tuple of any length, which each item is of type t". We can declare it as Sequence[t], but cannot specify that it must be a tuple, not a list. Example: class MyStr(str): def startswith(self, prefix:Union[str, ???])->bool: pass There's nothing I can use instead of ??? to capture the current behaviour of str.startswith. Correct? (4) Under "Pragmatics", you say "Don't use dynamic type expressions; use builtins and imported types only. No 'if'." What's the reason for this rule? Will it be enforced by the compiler? (5) I presume (4) above also applies to things like this: if condition(): X = Var('X', str) else: X = Var('X', bytes) # Later on def spam(arg: X)-> X: ... How about this? try: from condition_is_true import X # str except ImportError: from condition_is_false import X # bytes (6) Under "Generic types", you have: X = Var('X'). Declares a unique type variable. The name must match the variable name. To be clear, X is a type used only for declarations, right? By (1) above, it cannot be instantiated? But doesn't that make it impossible to satisfy the declaration? I must be misunderstanding something. I imagine the declaration X = Var("X") to be something equivalent to: class X(type): pass except that X cannot be instantiated. (7) You have an example: AnyStr = Var('AnyStr', str, bytes) def longest(a: AnyStr, b: AnyStr) -> AnyStr: Would this be an acceptable syntax? def longest(a:str|bytes, b:str|bytes) -> str|bytes It seems a shame to have to invent a name for something you might only use once or twice. -- Steve From toddrjen at gmail.com Sat Dec 20 09:28:23 2014 From: toddrjen at gmail.com (Todd) Date: Sat, 20 Dec 2014 09:28:23 +0100 Subject: [Python-ideas] Line-by-line profiling Message-ID: Currently, the python profiler works at the function or method level. This is good for identifying slow functions, but not so good for identifying things like slow for loops inside larger functions. You can use timers, but putting a lot of times in its cumbersome and makes direct comparison between potential hotspots difficult. And although we should, in practice it isn't airways feasible to refactor or code so there is only one major hotspot per function or method. A solution to this is to have a profiler that shows the execution time for each line (cumulative and per-call). Currently there is the third-party line-profiler, but at least to me this seems like the sort of common functionality that belongs in the standard library. line-profiler also requires the use of a decorator since it has a large performance penalty, and perhaps a more deeply integrated line-by-line profiler could avoid this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Dec 20 11:08:22 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 20 Dec 2014 20:08:22 +1000 Subject: [Python-ideas] Line-by-line profiling In-Reply-To: References: Message-ID: On 20 December 2014 at 18:28, Todd wrote: > Currently, the python profiler works at the function or method level. > This is good for identifying slow functions, but not so good for > identifying things like slow for loops inside larger functions. You can > use timers, but putting a lot of times in its cumbersome and makes direct > comparison between potential hotspots difficult. And although we should, > in practice it isn't airways feasible to refactor or code so there is only > one major hotspot per function or method. > > A solution to this is to have a profiler that shows the execution time for > each line (cumulative and per-call). Currently there is the third-party > line-profiler, but at least to me this seems like the sort of common > functionality that belongs in the standard library. line-profiler also > requires the use of a decorator since it has a large performance penalty, > and perhaps a more deeply integrated line-by-line profiler could avoid this. > It's worth noting that since Python 3.4 and 2.7.9, the PyPI line_profiler module is only a "pip install line_profiler" away for most users. While bringing line_profiler into the standard library would technically be feasible, it creates some additional challenges: * line_profiler uses Cython for its build process, so including it would mean being able to bootstrap Cython in a CPython source checkout in order to completely build the standard library * a line_profiler module would still be needed on PyPI for older Python versions, so this would involve either forking a separately maintained copy for the standard library, or else persuading Robert Kern to accept the additional maintenance burden associated with inclusion in the standard library as well as being published on PyPI A potentially lower cost alternative may be to add a profiling HOWTO guide, or even just some "See Also" links in the documentation for the profile module, and point out the available of the line_profiler and kernprof utilities. (The SnakeViz visualiser I was recently introduced to at the Brisbane Python User Group is another useful utility that could be worth making more readily discoverable: https://jiffyclub.github.io/snakeviz/) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From cfkaran2 at gmail.com Sat Dec 20 14:34:10 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Sat, 20 Dec 2014 08:34:10 -0500 Subject: [Python-ideas] Bug? Feature? setattr(foo, '3', 4) works! In-Reply-To: References: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> <5DB1B6A2-6982-47F8-8A89-17A705F3290B@gmail.com> Message-ID: <3EA96BCE-0A5E-495F-B942-8ED8A744A573@gmail.com> On Dec 19, 2014, at 8:54 AM, Nick Coghlan wrote: > On 19 December 2014 at 23:46, Nick Coghlan wrote: > On 19 December 2014 at 21:17, Cem Karan wrote: > > On Dec 19, 2014, at 6:03 AM, Nick Coghlan wrote: > > > On 19 December 2014 at 20:57, Cem Karan wrote: > > But, getting back to the main question, is this a bug or a feature? I personally feel like this is a bug, and I'd like to both clarify it in the language spec, and have cpython modified to enforce the syntax, regardless of how you try to mess with an attribute. How does everyone else feel about this? > > > > Namespaces are just dictionaries. The one thing implementations may do is to place a type restriction on the keys, so they throw TypeError if you try to use something other than str (although *CPython* doesn't do that). > > > > getattr/setattr/delattr would never throw SyntaxError though - that's only thrown by the compiler. > > I see what you're saying, but I feel like having (get|set|has)attr throw TypeError instead of SyntaxError would itself be astonishing in this case. The feature I'm after is that "foo.3" results in exactly the same behavior as "getattr(foo, '3')". > > I can probably save you some time: not going to happen :) > > To elaborate on that a bit: > > >>> x.a > 1 > >>> getattr(x, 'a') > 1 > >>> x.__dict__['a'] > 1 > > The ".a" attribute access syntax is ultimately just syntactic sugar for a dictionary lookup with the string 'a'. That syntactic sugar is what applies the restriction to being a valid identifier, not the underlying namespace object. > > Even for string values, you can do syntactically unacceptable lookups via the underlying namespace mapping: > > >>> setattr(x, 'not a legal identifier', 2) > >>> getattr(x, 'not a legal identifier') > 2 > >>> x.__dict__['not a legal identifier'] > 2 > > You'll find that kind of behaviour in a few different areas - syntactic restrictions aren't actually present in the underlying runtime machinery, they only exist at compile time. Got it. On python-list, someone pointed out that ctypes requires this behavior to work correctly, so regardless of what I think, it can't be changed. I'm dropping the suggestion. Thanks, Cem Karan From guido at python.org Sat Dec 20 19:08:01 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 20 Dec 2014 10:08:01 -0800 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: <20141220075945.GF20332@ando.pearwood.info> References: <20141220075945.GF20332@ando.pearwood.info> Message-ID: On Fri, Dec 19, 2014 at 11:59 PM, Steven D'Aprano wrote: > > On Fri, Dec 19, 2014 at 04:55:37PM -0800, Guido van Rossum wrote: > > A few months ago we had a long discussion about type hinting. I've > thought > > a lot more about this. I've written up what I think is a decent "theory" > > document -- writing it down like this certainly helped *me* get a lot of > > clarity about some of the important issues. > > > > https://quip.com/r69HA9GhGa7J > > Very interesting indeed. > > Some questions, which you may not have answers to yet :-) > > (1) Under "General rules", you state "No type can be subclassed unless > stated." That's not completely clear to me. I presume you are talking > about the special types like Union, Generic, Sequence, Tuple, Any etc. > Is that correct? > Yes. (Having to answer this is the price I pay for attempting brevity.) > > > (2) Under "Types", you give an example Tuple[t1, t2, ...], a tuple > whose items are instances of t1 etc. To be more concrete, a declaration > of Tuple[int, float, str] will mean "a tuple with exactly three items, > the first item must be an int, the second item must be a float, the > third item must be a string." Correct? > Yes. > > > (3) But there's no way of declaring "a tuple of any length, which each > item is of type t". We can declare it as Sequence[t], but cannot specify > that it must be a tuple, not a list. Example: > > class MyStr(str): > def startswith(self, prefix:Union[str, ???])->bool: > pass > > There's nothing I can use instead of ??? to capture the current > behaviour of str.startswith. Correct? > Yes, though there's a proposal to let you write Union[str, Tuple[str, ...]] -- the ... are literally that (i.e. Ellipsis). > > > (4) Under "Pragmatics", you say "Don't use dynamic type expressions; use > builtins and imported types only. No 'if'." What's the reason for this > rule? Will it be enforced by the compiler? > No, but it will get you on the static checker's nasty list. > > > (5) I presume (4) above also applies to things like this: > > if condition(): > X = Var('X', str) > else: > X = Var('X', bytes) > > # Later on > def spam(arg: X)-> X: > ... > Yes. > > How about this? > > try: > from condition_is_true import X # str > except ImportError: > from condition_is_false import X # bytes > Probably also. In mypy there is limited support for a few specific tests, IIRC it has PY3 and PY2 conditions built in. In any case this is all just to make the static checker's life easier (since it won't know whether the condition is true or false at runtime). > > > (6) Under "Generic types", you have: > > X = Var('X'). Declares a unique type variable. > The name must match the variable name. > > To be clear, X is a type used only for declarations, right? By (1) > above, it cannot be instantiated? But doesn't that make it impossible to > satisfy the declaration? I must be misunderstanding something. > It's in support of generic types. Read up on them in the mypy docs about generics. > > I imagine the declaration X = Var("X") to be something equivalent to: > > class X(type): > pass > > except that X cannot be instantiated. > No. See above. > > > (7) You have an example: > > AnyStr = Var('AnyStr', str, bytes) > def longest(a: AnyStr, b: AnyStr) -> AnyStr: > > > Would this be an acceptable syntax? > > def longest(a:str|bytes, b:str|bytes) -> str|bytes > > It seems a shame to have to invent a name for something you might only > use once or twice. > That was proposed and rejected (though for Union, which is slightly different) because it would require changes to the builtin types to support that operator at runtime. Please do read up on generic types in mypy. http://mypy.readthedocs.org/en/latest/generics.html -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim.baker at python.org Sat Dec 20 19:29:55 2014 From: jim.baker at python.org (Jim Baker) Date: Sat, 20 Dec 2014 11:29:55 -0700 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: <20141220075945.GF20332@ando.pearwood.info> Message-ID: On Sat, Dec 20, 2014 at 11:08 AM, Guido van Rossum wrote: > On Fri, Dec 19, 2014 at 11:59 PM, Steven D'Aprano > wrote: >> >> >> >> (4) Under "Pragmatics", you say "Don't use dynamic type expressions; use >> builtins and imported types only. No 'if'." What's the reason for this >> rule? Will it be enforced by the compiler? >> > > No, but it will get you on the static checker's nasty list. > For a very good reason - static type checking means we are evaluating types *statically*, before a program is run. (Otherwise we are doing dynamic type checking. Python already does a good job of that!) The Python compiler is completely oblivious to any of this of course - nothing is changing with it in this proposal for 3.4. Only mypy and other possible static type checkers and related tools like IDEs will complain. Or you might be able to get this information at runtime, which can help support gradual typing. > > >> >> How about this? >> >> try: >> from condition_is_true import X # str >> except ImportError: >> from condition_is_false import X # bytes >> > > Probably also. In mypy there is limited support for a few specific tests, > IIRC it has PY3 and PY2 conditions built in. In any case this is all just > to make the static checker's life easier (since it won't know whether the > condition is true or false at runtime). > Note that in theory it's possible to define compile-time expressions, as has been done in C++ 11. ( http://en.wikipedia.org/wiki/Compile_time_function_execution is a good starting point.) Except for possibly the simplest cases, such as baked-in support for Py3/Py2, let's not do that. > (7) You have an example: >> >> AnyStr = Var('AnyStr', str, bytes) >> def longest(a: AnyStr, b: AnyStr) -> AnyStr: >> >> >> Would this be an acceptable syntax? >> >> def longest(a:str|bytes, b:str|bytes) -> str|bytes >> >> It seems a shame to have to invent a name for something you might only >> use once or twice. >> > > That was proposed and rejected (though for Union, which is slightly > different) because it would require changes to the builtin types to support > that operator at runtime. > The other thing is AnyStr is a type variable, not a type definition to shorten writing out types. So AnyStr acts as a constraint that must be satisfied for the program to type check as valid. (A very simple constraint.) You can think of it as supporting two possible signatures for the longest function from the caller's perspective: def longest(a: str, b: str) -> str def longest(b: bytes, b: bytes) -> bytes But not some mix, this would be invalid if the caller of longest try to use it in this way: def longest(a: str, b: str) -> bytes > Please do read up on generic types in mypy. > http://mypy.readthedocs.org/en/latest/generics.html > The linked example of a Stack class demonstrates how the type variable can work as a constraint for a class, across its methods, similar to what one might see in Java or Scala. -- - Jim jim.baker@{colorado.edu|python.org|rackspace.com|zyasoft.com} -------------- next part -------------- An HTML attachment was scrubbed... URL: From brakhane at googlemail.com Sat Dec 20 23:00:51 2014 From: brakhane at googlemail.com (Dennis Brakhane) Date: Sat, 20 Dec 2014 23:00:51 +0100 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: Message-ID: <5495F193.9000402@gmail.com> Maybe I'm missing something, but wouldn't type hinting as it's defined now break "virtual subclassing" of ABC? For example, given the following code: from collections import Sequence class MySequence: ... Sequence.register(MySequence) it seems to me like the following would work: def foo(bar): if not isinstance(bar, Sequence): raise RuntimeError("Foo can only work with sequences") ... but when rewritten for static type checking def foo(bar: Sequence): .... it would cease to work. At least I don't see a way a static type checker could handle this realiably (the register call might occur anywhere, after all) Is this intentional? Even if this might be more a mypy/implementation question, it should be clear to users of typing.py if they should expect ABCs to break or not From gmludo at gmail.com Sun Dec 21 00:55:51 2014 From: gmludo at gmail.com (Ludovic Gasc) Date: Sun, 21 Dec 2014 00:55:51 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? Message-ID: Hi, I'm only a simple Python developer, not a Type Hinting expert and I don't know if you already discuss about that somewhere, but: With the future official support of Type Hinting in Python, is it means that CPython could use this pieces of information to store variables with more efficient data structures, not only check types ? It could possible to have better performance one day with Type Hinting like you have with Cython (explicit types declaration) or PyPy (guessing types) ? Is it realistic to except that one day, or I've missed some mountains ;-) ? If this is correct, better performances will be a great consequence for Type Hinting, more people will be interested in by this feature, as we have with AsyncIO (BTW, I'm working to publish benchmarks on this, I'll publish that on AsyncIO ML). Regards. -- Ludovic Gasc -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Dec 21 02:15:17 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 21 Dec 2014 11:15:17 +1000 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: Message-ID: On 21 December 2014 at 09:55, Ludovic Gasc wrote: > Hi, > > I'm only a simple Python developer, not a Type Hinting expert and I don't > know if you already discuss about that somewhere, but: > > With the future official support of Type Hinting in Python, is it means > that CPython could use this pieces of information to store variables with > more efficient data structures, not only check types ? > > It could possible to have better performance one day with Type Hinting > like you have with Cython (explicit types declaration) or PyPy (guessing > types) ? > The primary goals of the type hinting standardisation effort are improving program correctness (through enhanced static analysis) and API documentation (through clearer communication of expectations for input and output, both in the code, in documentation, and in IDEs). It should also allow the development of more advanced techniques for function signature based dispatch and other forms of structured pattern matching. >From a performance perspective though, CPython already has a wide array of fast paths for particular builtin types - being able to communicate those assumptions more clearly won't automatically make it easier to speed them up further (or add more in a maintainable fashion). There's no *philosophical* objection to such changes (other than "simplicity is a virtue"), there's just a long track record of previous attempts like psyco and Unladen Swallow that make it clear that the problem is genuinely *hard*. There's also the fact that with both Numba and PyPy now supporting selective JIT acceleration of decorated functions within the context of a larger CPython application, as well as Cython's existing support for precompilation as a C extension, the pattern of profiling to find performance critical areas, and finding ways to optimise those, now seems well established. (Hence my suggestion the other day that we could likely use an introductory how to guide on performance profiling, which could also provide suggestions for optimisation tools to explore once the hot spots have been found). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From liam.marsh.home at gmail.com Sun Dec 21 11:33:35 2014 From: liam.marsh.home at gmail.com (Liam Marsh) Date: Sun, 21 Dec 2014 11:33:35 +0100 Subject: [Python-ideas] Fwd: improve compatibility In-Reply-To: References: <00DC45BE-A51D-4EB7-BF90-D4BA2F4B3189@yahoo.com> Message-ID: Hello again a few days later, i wad a new idea: the user who wants the 2.X modules, the 3.2 and 3.4 modules to work will probably install the three versions. so, is it possible to have the \Python\python34; \Python\python32; and \Python\python27 dirs, and the \Python\PythonAutoVersion program, useing the comment "#py ver XX" to chose which version will take care of it? (same thing with the IDLE) how about that? have a nice day/evening! -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Dec 21 15:31:17 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 22 Dec 2014 00:31:17 +1000 Subject: [Python-ideas] Fwd: improve compatibility In-Reply-To: References: <00DC45BE-A51D-4EB7-BF90-D4BA2F4B3189@yahoo.com> Message-ID: On 21 December 2014 at 20:33, Liam Marsh wrote: > Hello again > > a few days later, i wad a new idea: > the user who wants the 2.X modules, the 3.2 and 3.4 modules to work will > probably install the three versions. > so, is it possible to have the \Python\python34; > \Python\python32; and \Python\python27 dirs, and the > \Python\PythonAutoVersion program, useing the comment "#py ver > XX" to chose which version will take care of it? > (same thing with the IDLE) > You're exploring a space which already has a large array of tools available to address it. In particular, selective application upgrades is one of the features provided by virtual environments. If someone so desires, *.pth files can be used to share modules between different virtual environments (although this isn't a common practice - virtual environments are generally aimed at providing near total isolation of an application's dependency stack from other virtual environments on the system). You can also already manipulate sys.path freely at runtime, and do even more through custom import hooks. While I don't actually think the particular approach you suggest here is a good idea (unlike the explicit requirements.txt files of the PyPA ecosystem, you'll likely end up with an utterly unmaintainable system, where nobody has any idea which applications are depending on what), the tools to implement it exist at least as far back as Python 2.7 (by way of the importlib2 backport of the Python 3.4+ import implementation) and even further if you're prepared to work within the constraints of the old C based import implementation, rather than the newer primarily Python one. So if you'd like to explore this idea further, I'd suggest trying to implement it. Even if you end up agreeing with me that it's a bad idea, you'll likely learn a *lot* about the import system and the full extent of its capabilities in the process (and if you have a blog, others may also be able to learn from the exercise, especially if it's linked from Planet Python). Another alternative would be to look at ways to layout a directory structure such that API compatible upgrades of dependencies (in a semantic versioning sense) can be shared between different virtual environments. That's a topic for distutils-sig rather than python-ideas, but it's certainly one where a working proof of concept would be a *very* interesting thing to see. (We know it should work in theory, we just haven't had anyone sit down and figure out what the technical details might look like) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Dec 21 17:53:40 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 21 Dec 2014 08:53:40 -0800 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: <5495F193.9000402@gmail.com> References: <5495F193.9000402@gmail.com> Message-ID: On Sat, Dec 20, 2014 at 2:00 PM, Dennis Brakhane wrote: > > Maybe I'm missing something, but wouldn't type hinting as it's defined > now break "virtual subclassing" of ABC? > > For example, given the following code: > > from collections import Sequence > > class MySequence: > ... > > Sequence.register(MySequence) > > it seems to me like the following would work: > > def foo(bar): > if not isinstance(bar, Sequence): > raise RuntimeError("Foo can only work with sequences") > ... > > but when rewritten for static type checking > > def foo(bar: Sequence): > .... > > it would cease to work. At least I don't see a way a static type checker > could handle this relaiably (the register call might occur anywhere, > after all) > > Is this intentional? > > Even if this might be more a mypy/implementation question, it should be > clear to users of typing.py if they should expect ABCs to break or not > Well, the program will still run fine with CPython, assuming you keep the isinstance() check in your program. The argument declaration does not cause calls with non-Sequence arguments to be rejected, it just guides the static checker (which is a program that you must run separately, in a similar way as a linter -- it is not built into CPython). The static checker may request the call foo(MySequence()) if it cannot see that MySequence is a Sequence. You have two options there: if you can change the code of MySequence, you should just inherit from Sequence rather than using the register() call; if you cannot change the code of MySequence, you should use a *stub* module which declares that MySequence is a Sequence. You can read up on stub modules in the mypy docs: http://mypy.readthedocs.org/en/latest/basics.html#library-stubs -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Dec 21 17:57:03 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 21 Dec 2014 08:57:03 -0800 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: Message-ID: What Nick is trying to say is that mypy and other optimizers have been quite successful without type hints. I believe that Jim Baker is hopeful that he will be able to do some interesting things for Jython with type hints though. --Guido On Sat, Dec 20, 2014 at 5:15 PM, Nick Coghlan wrote: > > On 21 December 2014 at 09:55, Ludovic Gasc wrote: > >> Hi, >> >> I'm only a simple Python developer, not a Type Hinting expert and I don't >> know if you already discuss about that somewhere, but: >> >> With the future official support of Type Hinting in Python, is it means >> that CPython could use this pieces of information to store variables with >> more efficient data structures, not only check types ? >> >> It could possible to have better performance one day with Type Hinting >> like you have with Cython (explicit types declaration) or PyPy (guessing >> types) ? >> > > The primary goals of the type hinting standardisation effort are improving > program correctness (through enhanced static analysis) and API > documentation (through clearer communication of expectations for input and > output, both in the code, in documentation, and in IDEs). It should also > allow the development of more advanced techniques for function signature > based dispatch and other forms of structured pattern matching. > > From a performance perspective though, CPython already has a wide array of > fast paths for particular builtin types - being able to communicate those > assumptions more clearly won't automatically make it easier to speed them > up further (or add more in a maintainable fashion). There's no > *philosophical* objection to such changes (other than "simplicity is a > virtue"), there's just a long track record of previous attempts like psyco > and Unladen Swallow that make it clear that the problem is genuinely *hard*. > > There's also the fact that with both Numba and PyPy now supporting > selective JIT acceleration of decorated functions within the context of a > larger CPython application, as well as Cython's existing support for > precompilation as a C extension, the pattern of profiling to find > performance critical areas, and finding ways to optimise those, now seems > well established. (Hence my suggestion the other day that we could likely > use an introductory how to guide on performance profiling, which could also > provide suggestions for optimisation tools to explore once the hot spots > have been found). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Sun Dec 21 20:55:14 2014 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sun, 21 Dec 2014 21:55:14 +0200 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: <5495F193.9000402@gmail.com> Message-ID: The silly question. I use python 3 annotations for argument checking. My assumption is very simple, for def f(arg: checker): pass the checker will raise ValueError or TypeError if arg is not correct. I do it by `checker(arg)` call. I use this in aiozmq rpc (http://aiozmq.readthedocs.org/en/0.5/rpc.html#signature-validation) for example and checkers from trafaret (https://github.com/Deepwalker/trafaret) works fine. Will proposed change break my workflow? On Sun, Dec 21, 2014 at 6:53 PM, Guido van Rossum wrote: > On Sat, Dec 20, 2014 at 2:00 PM, Dennis Brakhane > wrote: >> >> Maybe I'm missing something, but wouldn't type hinting as it's defined >> now break "virtual subclassing" of ABC? >> >> For example, given the following code: >> >> from collections import Sequence >> >> class MySequence: >> ... >> >> Sequence.register(MySequence) >> >> it seems to me like the following would work: >> >> def foo(bar): >> if not isinstance(bar, Sequence): >> raise RuntimeError("Foo can only work with sequences") >> ... >> >> but when rewritten for static type checking >> >> def foo(bar: Sequence): >> .... >> >> it would cease to work. At least I don't see a way a static type checker >> could handle this relaiably (the register call might occur anywhere, >> after all) >> >> Is this intentional? >> >> Even if this might be more a mypy/implementation question, it should be >> clear to users of typing.py if they should expect ABCs to break or not > > > Well, the program will still run fine with CPython, assuming you keep the > isinstance() check in your program. The argument declaration does not cause > calls with non-Sequence arguments to be rejected, it just guides the static > checker (which is a program that you must run separately, in a similar way > as a linter -- it is not built into CPython). > > The static checker may request the call foo(MySequence()) if it cannot see > that MySequence is a Sequence. You have two options there: if you can change > the code of MySequence, you should just inherit from Sequence rather than > using the register() call; if you cannot change the code of MySequence, you > should use a *stub* module which declares that MySequence is a Sequence. You > can read up on stub modules in the mypy docs: > http://mypy.readthedocs.org/en/latest/basics.html#library-stubs > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Thanks, Andrew Svetlov From gmludo at gmail.com Sun Dec 21 20:23:50 2014 From: gmludo at gmail.com (Ludovic Gasc) Date: Sun, 21 Dec 2014 20:23:50 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: Message-ID: Hi Nick, Thanks for your answer. I understand primary goal, I'm not completely naive on this question: A long time ago, I used a lot Type Hinting with PHP 5. Nevertheless, in Python community, you can find a lot of libraries to improve performance based on types handling with different optimization strategies (numba, pypy, pythran, cython, llvm-py, shedskin...). To my knowledge, you don't have the same number of libraries to do that with another dynamic language. It means that in Python community we have this problematic. It's like with async pattern, in Python you have a plenty of libraries (Twisted, eventlet, gevent, stackless, tornado...) and now, with AsyncIO, the community should converge on it. And yes, I understand that it's almost impossible to create a silver bullet to improve automagically performance, but, as with my simple dev eyes, the common pattern I see with pythran, cython... is the type handling. They don't use only this strategy to improve performance, but it's the biggest visible part in example codes I've seen. Guido: " optimizers have been quite successful without type hints" <= Certainly, but instead of to loose time to try to guess the right data structure, maybe it could be faster that the developer gives directly what he wants. To be honest, I'm a little bit tired to listen some bias like "Python is slow", "not good for performance", "you must use C/Java/Erlang/Go..." For me, Python has the right compromise to write quickly readable source code and performance possibilities to speed up your code. More we have primitives in CPython to build performant applications, more it will be easier to convince people to use Python. Regards. -- Ludovic Gasc On 21 Dec 2014 02:15, "Nick Coghlan" wrote: > On 21 December 2014 at 09:55, Ludovic Gasc wrote: > >> Hi, >> >> I'm only a simple Python developer, not a Type Hinting expert and I don't >> know if you already discuss about that somewhere, but: >> >> With the future official support of Type Hinting in Python, is it means >> that CPython could use this pieces of information to store variables with >> more efficient data structures, not only check types ? >> >> It could possible to have better performance one day with Type Hinting >> like you have with Cython (explicit types declaration) or PyPy (guessing >> types) ? >> > > The primary goals of the type hinting standardisation effort are improving > program correctness (through enhanced static analysis) and API > documentation (through clearer communication of expectations for input and > output, both in the code, in documentation, and in IDEs). It should also > allow the development of more advanced techniques for function signature > based dispatch and other forms of structured pattern matching. > > From a performance perspective though, CPython already has a wide array of > fast paths for particular builtin types - being able to communicate those > assumptions more clearly won't automatically make it easier to speed them > up further (or add more in a maintainable fashion). There's no > *philosophical* objection to such changes (other than "simplicity is a > virtue"), there's just a long track record of previous attempts like psyco > and Unladen Swallow that make it clear that the problem is genuinely *hard*. > > There's also the fact that with both Numba and PyPy now supporting > selective JIT acceleration of decorated functions within the context of a > larger CPython application, as well as Cython's existing support for > precompilation as a C extension, the pattern of profiling to find > performance critical areas, and finding ways to optimise those, now seems > well established. (Hence my suggestion the other day that we could likely > use an introductory how to guide on performance profiling, which could also > provide suggestions for optimisation tools to explore once the hot spots > have been found). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Dec 21 21:11:23 2014 From: brett at python.org (Brett Cannon) Date: Sun, 21 Dec 2014 20:11:23 +0000 Subject: [Python-ideas] Type Hinting Kick-off References: <5495F193.9000402@gmail.com> Message-ID: You're workflow won't necessarily break. You need to use a tool which expects type hints for function annotations to cause you any problems. If you simply don't use such a tool then you will have no problems. On Sun, Dec 21, 2014, 11:55 Andrew Svetlov wrote: > The silly question. > > I use python 3 annotations for argument checking. > > My assumption is very simple, for > > def f(arg: checker): pass > > the checker will raise ValueError or TypeError if arg is not correct. > I do it by `checker(arg)` call. > > I use this in aiozmq rpc > (http://aiozmq.readthedocs.org/en/0.5/rpc.html#signature-validation) > for example and checkers from trafaret > (https://github.com/Deepwalker/trafaret) works fine. > > Will proposed change break my workflow? > > On Sun, Dec 21, 2014 at 6:53 PM, Guido van Rossum > wrote: > > On Sat, Dec 20, 2014 at 2:00 PM, Dennis Brakhane < > brakhane at googlemail.com> > > wrote: > >> > >> Maybe I'm missing something, but wouldn't type hinting as it's defined > >> now break "virtual subclassing" of ABC? > >> > >> For example, given the following code: > >> > >> from collections import Sequence > >> > >> class MySequence: > >> ... > >> > >> Sequence.register(MySequence) > >> > >> it seems to me like the following would work: > >> > >> def foo(bar): > >> if not isinstance(bar, Sequence): > >> raise RuntimeError("Foo can only work with sequences") > >> ... > >> > >> but when rewritten for static type checking > >> > >> def foo(bar: Sequence): > >> .... > >> > >> it would cease to work. At least I don't see a way a static type checker > >> could handle this relaiably (the register call might occur anywhere, > >> after all) > >> > >> Is this intentional? > >> > >> Even if this might be more a mypy/implementation question, it should be > >> clear to users of typing.py if they should expect ABCs to break or not > > > > > > Well, the program will still run fine with CPython, assuming you keep the > > isinstance() check in your program. The argument declaration does not > cause > > calls with non-Sequence arguments to be rejected, it just guides the > static > > checker (which is a program that you must run separately, in a similar > way > > as a linter -- it is not built into CPython). > > > > The static checker may request the call foo(MySequence()) if it cannot > see > > that MySequence is a Sequence. You have two options there: if you can > change > > the code of MySequence, you should just inherit from Sequence rather than > > using the register() call; if you cannot change the code of MySequence, > you > > should use a *stub* module which declares that MySequence is a Sequence. > You > > can read up on stub modules in the mypy docs: > > http://mypy.readthedocs.org/en/latest/basics.html#library-stubs > > > > -- > > --Guido van Rossum (python.org/~guido) > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > Thanks, > Andrew Svetlov > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Sun Dec 21 21:32:50 2014 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sun, 21 Dec 2014 22:32:50 +0200 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: <5495F193.9000402@gmail.com> Message-ID: Sorry, I want to ask again. The proposal is for static checks only? My expectations for processing annotations in runtime as-is (just a mark without any restrictions) will not changed? On Sun, Dec 21, 2014 at 10:11 PM, Brett Cannon wrote: > You're workflow won't necessarily break. You need to use a tool which > expects type hints for function annotations to cause you any problems. If > you simply don't use such a tool then you will have no problems. > > > On Sun, Dec 21, 2014, 11:55 Andrew Svetlov wrote: >> >> The silly question. >> >> I use python 3 annotations for argument checking. >> >> My assumption is very simple, for >> >> def f(arg: checker): pass >> >> the checker will raise ValueError or TypeError if arg is not correct. >> I do it by `checker(arg)` call. >> >> I use this in aiozmq rpc >> (http://aiozmq.readthedocs.org/en/0.5/rpc.html#signature-validation) >> for example and checkers from trafaret >> (https://github.com/Deepwalker/trafaret) works fine. >> >> Will proposed change break my workflow? >> >> On Sun, Dec 21, 2014 at 6:53 PM, Guido van Rossum >> wrote: >> > On Sat, Dec 20, 2014 at 2:00 PM, Dennis Brakhane >> > >> > wrote: >> >> >> >> Maybe I'm missing something, but wouldn't type hinting as it's defined >> >> now break "virtual subclassing" of ABC? >> >> >> >> For example, given the following code: >> >> >> >> from collections import Sequence >> >> >> >> class MySequence: >> >> ... >> >> >> >> Sequence.register(MySequence) >> >> >> >> it seems to me like the following would work: >> >> >> >> def foo(bar): >> >> if not isinstance(bar, Sequence): >> >> raise RuntimeError("Foo can only work with sequences") >> >> ... >> >> >> >> but when rewritten for static type checking >> >> >> >> def foo(bar: Sequence): >> >> .... >> >> >> >> it would cease to work. At least I don't see a way a static type >> >> checker >> >> could handle this relaiably (the register call might occur anywhere, >> >> after all) >> >> >> >> Is this intentional? >> >> >> >> Even if this might be more a mypy/implementation question, it should be >> >> clear to users of typing.py if they should expect ABCs to break or not >> > >> > >> > Well, the program will still run fine with CPython, assuming you keep >> > the >> > isinstance() check in your program. The argument declaration does not >> > cause >> > calls with non-Sequence arguments to be rejected, it just guides the >> > static >> > checker (which is a program that you must run separately, in a similar >> > way >> > as a linter -- it is not built into CPython). >> > >> > The static checker may request the call foo(MySequence()) if it cannot >> > see >> > that MySequence is a Sequence. You have two options there: if you can >> > change >> > the code of MySequence, you should just inherit from Sequence rather >> > than >> > using the register() call; if you cannot change the code of MySequence, >> > you >> > should use a *stub* module which declares that MySequence is a Sequence. >> > You >> > can read up on stub modules in the mypy docs: >> > http://mypy.readthedocs.org/en/latest/basics.html#library-stubs >> > >> > -- >> > --Guido van Rossum (python.org/~guido) >> > >> > _______________________________________________ >> > Python-ideas mailing list >> > Python-ideas at python.org >> > https://mail.python.org/mailman/listinfo/python-ideas >> > Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> >> -- >> Thanks, >> Andrew Svetlov >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ -- Thanks, Andrew Svetlov From brett at python.org Sun Dec 21 21:44:33 2014 From: brett at python.org (Brett Cannon) Date: Sun, 21 Dec 2014 20:44:33 +0000 Subject: [Python-ideas] Type Hinting Kick-off References: <5495F193.9000402@gmail.com> Message-ID: The proposal is to standardize how to specify type hints. The *assumption* is that static type checkers will use the type hints. Nothing is being forced to occur at runtime; Guido's proposal is outlining how static tools that consume the type hints and people that use them should expect things to work. On Sun, Dec 21, 2014, 12:32 Andrew Svetlov wrote: > Sorry, I want to ask again. > The proposal is for static checks only? > My expectations for processing annotations in runtime as-is (just a > mark without any restrictions) will not changed? > > On Sun, Dec 21, 2014 at 10:11 PM, Brett Cannon wrote: > > You're workflow won't necessarily break. You need to use a tool which > > expects type hints for function annotations to cause you any problems. If > > you simply don't use such a tool then you will have no problems. > > > > > > On Sun, Dec 21, 2014, 11:55 Andrew Svetlov > wrote: > >> > >> The silly question. > >> > >> I use python 3 annotations for argument checking. > >> > >> My assumption is very simple, for > >> > >> def f(arg: checker): pass > >> > >> the checker will raise ValueError or TypeError if arg is not correct. > >> I do it by `checker(arg)` call. > >> > >> I use this in aiozmq rpc > >> (http://aiozmq.readthedocs.org/en/0.5/rpc.html#signature-validation) > >> for example and checkers from trafaret > >> (https://github.com/Deepwalker/trafaret) works fine. > >> > >> Will proposed change break my workflow? > >> > >> On Sun, Dec 21, 2014 at 6:53 PM, Guido van Rossum > >> wrote: > >> > On Sat, Dec 20, 2014 at 2:00 PM, Dennis Brakhane > >> > > >> > wrote: > >> >> > >> >> Maybe I'm missing something, but wouldn't type hinting as it's > defined > >> >> now break "virtual subclassing" of ABC? > >> >> > >> >> For example, given the following code: > >> >> > >> >> from collections import Sequence > >> >> > >> >> class MySequence: > >> >> ... > >> >> > >> >> Sequence.register(MySequence) > >> >> > >> >> it seems to me like the following would work: > >> >> > >> >> def foo(bar): > >> >> if not isinstance(bar, Sequence): > >> >> raise RuntimeError("Foo can only work with sequences") > >> >> ... > >> >> > >> >> but when rewritten for static type checking > >> >> > >> >> def foo(bar: Sequence): > >> >> .... > >> >> > >> >> it would cease to work. At least I don't see a way a static type > >> >> checker > >> >> could handle this relaiably (the register call might occur anywhere, > >> >> after all) > >> >> > >> >> Is this intentional? > >> >> > >> >> Even if this might be more a mypy/implementation question, it should > be > >> >> clear to users of typing.py if they should expect ABCs to break or > not > >> > > >> > > >> > Well, the program will still run fine with CPython, assuming you keep > >> > the > >> > isinstance() check in your program. The argument declaration does not > >> > cause > >> > calls with non-Sequence arguments to be rejected, it just guides the > >> > static > >> > checker (which is a program that you must run separately, in a similar > >> > way > >> > as a linter -- it is not built into CPython). > >> > > >> > The static checker may request the call foo(MySequence()) if it cannot > >> > see > >> > that MySequence is a Sequence. You have two options there: if you can > >> > change > >> > the code of MySequence, you should just inherit from Sequence rather > >> > than > >> > using the register() call; if you cannot change the code of > MySequence, > >> > you > >> > should use a *stub* module which declares that MySequence is a > Sequence. > >> > You > >> > can read up on stub modules in the mypy docs: > >> > http://mypy.readthedocs.org/en/latest/basics.html#library-stubs > >> > > >> > -- > >> > --Guido van Rossum (python.org/~guido) > >> > > >> > _______________________________________________ > >> > Python-ideas mailing list > >> > Python-ideas at python.org > >> > https://mail.python.org/mailman/listinfo/python-ideas > >> > Code of Conduct: http://python.org/psf/codeofconduct/ > >> > >> > >> > >> -- > >> Thanks, > >> Andrew Svetlov > >> _______________________________________________ > >> Python-ideas mailing list > >> Python-ideas at python.org > >> https://mail.python.org/mailman/listinfo/python-ideas > >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > Thanks, > Andrew Svetlov > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Dec 22 00:20:38 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 22 Dec 2014 10:20:38 +1100 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: Message-ID: On Mon, Dec 22, 2014 at 6:23 AM, Ludovic Gasc wrote: > To be honest, I'm a little bit tired to listen some bias like "Python is > slow", "not good for performance", "you must use C/Java/Erlang/Go..." > For me, Python has the right compromise to write quickly readable source > code and performance possibilities to speed up your code. > > More we have primitives in CPython to build performant applications, more it > will be easier to convince people to use Python. There may be something to that. Most of the people I've heard moaning that "Python is slow" aren't backing that up with any actual facts. Telling people "Add these type hints to your code if you feel like it" might satisfy their need for red tape and the gut feeling that it's speeding up the code. ChrisA From ncoghlan at gmail.com Mon Dec 22 00:38:17 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 22 Dec 2014 09:38:17 +1000 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: Message-ID: On 22 December 2014 at 05:23, Ludovic Gasc wrote: > Hi Nick, > > Thanks for your answer. I understand primary goal, I'm not completely > naive on this question: A long time ago, I used a lot Type Hinting with PHP > 5. > > Nevertheless, in Python community, you can find a lot of libraries to > improve performance based on types handling with different optimization > strategies (numba, pypy, pythran, cython, llvm-py, shedskin...). > To my knowledge, you don't have the same number of libraries to do that > with another dynamic language. > It means that in Python community we have this problematic. > > It's like with async pattern, in Python you have a plenty of libraries > (Twisted, eventlet, gevent, stackless, tornado...) and now, with AsyncIO, > the community should converge on it. > > And yes, I understand that it's almost impossible to create a silver > bullet to improve automagically performance, but, as with my simple dev > eyes, the common pattern I see with pythran, cython... is the type handling. > They don't use only this strategy to improve performance, but it's the > biggest visible part in example codes I've seen. > > Guido: " optimizers have been quite successful without type hints" <= > Certainly, but instead of to loose time to try to guess the right data > structure, maybe it could be faster that the developer gives directly what > he wants. > > To be honest, I'm a little bit tired to listen some bias like "Python is > slow", "not good for performance", "you must use C/Java/Erlang/Go..." > For me, Python has the right compromise to write quickly readable source > code and performance possibilities to speed up your code. > > More we have primitives in CPython to build performant applications, more > it will be easier to convince people to use Python. > Perhaps the most effective thing anyone could do to make significant progress in the area of CPython performance is to actually get CodeSpeed working properly with anything other than PyPy, as automated creation of clear metrics like that can be incredibly powerful as a motivational tool (think about the competition on JavaScript benchmarks between different browser vendors, or the way the PyPy team use speed.pypy.org as a measure of their success in making new versions faster). Work on speed.python.org (as a cross-implementation counterpart to speed.pypy.org) was started years ago, but no leader ever emerged to drive the effort to completion (and even a funded development effort by the PSF failed to produce a running instance of the service). Another possible approach would be to create a JavaScript front end for PyPy (along the lines of the PyPy-based Topaz interpreter for Ruby, or the HippyVM interpreter for PHP), and make a serious attempt at displacing V8 at the heart of Node.js. (The Node.js build system for binary extensions is already written in Python, so why not the core interpreter as well? There's also the fact that Node.js regularly ends up running on no longer supported versions of V8, as V8 is written to meet the needs of Chrome, not those of the server-side JavaScript community). One key advantage of the latter approach is that the more general purpose PyPy infrastructure being competitive with the heavily optimised JavaScript interpreters created by the browser vendors on a set of industry standard performance benchmarks is much, much stronger evidence of PyPy's raw speed than being faster than the not-known-for-its-speed CPython interpreter on a set of benchmarks originally chosen specifically by Google for the Unladen Swallow project. Even with Topaz being one of the fastest Ruby interpreters, to the point of Oracle Labs using it as a relative benchmark for comparison of JRuby's performance in http://www.slideshare.net/ThomasWuerthinger/graal-truffle-ethdec2013, that's still relatively weak evidence for raw speed, since Ruby in general is also not well known for being fast. (Likewise, HippyVM being faster than Facebook's HHVM is impressive, but vulnerable to the same counter-argument that people make for Python and Ruby, "If you care about raw speed, why are you still using PHP?") Objective benchmarks and real world success stories are the kinds of things that people find genuinely persuasive - otherwise we're just yet another programming language community making self-promoting claims on the internet without adequate supporting evidence. ( http://economics.sas.upenn.edu/~jesusfv/comparison_languages.pdf is an example of providing good supporting evidence that compares Numba and Cython, amongst many other alternatives, to the speed of raw C++ and FORTRAN for evaluation of a particular numeric model - given the benefits they offer in maintainability relative to the lower level languages, they fare extremely well on the speed front) As things stand, we have lots of folks wanting *someone else* to do the work to counter the inaccurate logic of "CPython-the-implementation tends to prioritise portability and maintainability over raw speed, therefore Python-the-language is inherently slow", yet very few volunteering to actually do the work needed to counter it effectively in a global sense (rather than within the specific niches currently targeted by the PyPy, Numba, and Cython development teams - those teams do some extraordinarily fine work that doesn't get the credit it deserves due to a mindset amongst many users that only CPython performance counts in cross-language comparisons). Regards, Nick. P.S. As noted earlier, a profiling and optimising HOWTO in the standard documentation set would also make a lot of sense as a way of making these alternatives more discoverable, but again, it needs a volunteer to write it (or at least an initial draft which then be polished in review on Reitveld). -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Dec 22 00:47:18 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 22 Dec 2014 09:47:18 +1000 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: <5495F193.9000402@gmail.com> Message-ID: On 22 December 2014 at 06:32, Andrew Svetlov wrote: > Sorry, I want to ask again. > The proposal is for static checks only? > My expectations for processing annotations in runtime as-is (just a > mark without any restrictions) will not changed? > Correct, there are no changes being proposed to the runtime semantics of annotations. The type hinting proposal describes a conventional use for them that will be of benefit to static type checking systems and integrated development environments, but it will be exactly that: a convention, not an enforced behaviour. The convention of treating "_" prefixed methods and other attributes as private to the implementation of a class or module is a good example of a similar approach. While some things (like pydoc and wildcard imports) will respect the convention, it's not enforced at the core language level - if a developer decides they're prepared to accept the compatibility risk, then they're free to use the "private" attribute if they choose to do so. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Mon Dec 22 01:17:42 2014 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 21 Dec 2014 19:17:42 -0500 Subject: [Python-ideas] Fwd: improve compatibility In-Reply-To: References: <00DC45BE-A51D-4EB7-BF90-D4BA2F4B3189@yahoo.com> Message-ID: <54976326.4090707@trueblade.com> On 12/21/2014 9:31 AM, Nick Coghlan wrote: > On 21 December 2014 at 20:33, Liam Marsh > wrote: > > Hello again > > a few days later, i wad a new idea: > the user who wants the 2.X modules, the 3.2 and 3.4 modules to work > will probably install the three versions. > so, is it possible to have the \Python\python34; > \Python\python32; and \Python\python27 dirs, and > the \Python\PythonAutoVersion program, useing the comment > "#py ver XX" to chose which version will take care of it? > (same thing with the IDLE) > > > You're exploring a space which already has a large array of tools > available to address it. In particular, selective application upgrades > is one of the features provided by virtual environments. If someone so > desires, *.pth files can be used to share modules between different > virtual environments (although this isn't a common practice - virtual > environments are generally aimed at providing near total isolation of an > application's dependency stack from other virtual environments on the > system). I thought he was looking for something like the py launcher, implemented for Windows as PEP 397, and discussed for Linux here: https://mail.python.org/pipermail/python-ideas/2014-April/027633.html -- Eric. From ncoghlan at gmail.com Mon Dec 22 01:27:10 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 22 Dec 2014 10:27:10 +1000 Subject: [Python-ideas] Fwd: improve compatibility In-Reply-To: <54976326.4090707@trueblade.com> References: <00DC45BE-A51D-4EB7-BF90-D4BA2F4B3189@yahoo.com> <54976326.4090707@trueblade.com> Message-ID: On 22 December 2014 at 10:17, Eric V. Smith wrote: > I thought he was looking for something like the py launcher, implemented > for Windows as PEP 397, and discussed for Linux here: > https://mail.python.org/pipermail/python-ideas/2014-April/027633.html > I interpreted the second suggestion in light of the first one, which included the point "-any imported py file will be able to choose which version it wants". That's suggesting mixing and matching between versions within the same process space, which is highly unlikely to work correctly (except when modules are specifically designed for that, as indicated by the availability of backported versions on PyPI). Either way, there's a lot of prior art in this space that actually works pretty well once you're aware of its existence, which means future improvements are likely to come in the form of availability, usability, and discoverability improvements for already popular approaches, rather than introducing radical new concepts. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Dec 22 05:05:04 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 21 Dec 2014 20:05:04 -0800 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: <5495F193.9000402@gmail.com> Message-ID: On Sun, Dec 21, 2014 at 3:47 PM, Nick Coghlan wrote: > > > On 22 December 2014 at 06:32, Andrew Svetlov > wrote: > >> Sorry, I want to ask again. >> The proposal is for static checks only? >> My expectations for processing annotations in runtime as-is (just a >> mark without any restrictions) will not changed? >> > > Correct, there are no changes being proposed to the runtime semantics of > annotations. The type hinting proposal describes a conventional use for > them that will be of benefit to static type checking systems and integrated > development environments, but it will be exactly that: a convention, not an > enforced behaviour. > Well... The elephant in the room is that *eventually* other uses of annotations *may* be frowned upon, or may need to be marked by some decorator. But I promise that in Python 3.5 your code will not break -- it just might not be very useful to run a static checker like mypy on it. (IIRC mypy used to only typecheck code that imports the typing.py module, but this seems to have changed.) When we discussed this earlier this year, a few other uses of annotations were brought up, and some have proposed that static type annotations would need to be marked by a decorator. There is even a proposed syntax that allows multiple annotations to coexist for the same argument (a dict with fixed keys -- to me it looks pretty ugly though). I would really like to wait and see how this plays out -- the proposal I'm working on is careful not to have any effect at runtime (as long as the typing.py module can be imported and as long as the annotation expressions don't raise exceptions), and use of a static checker is entirely optional and voluntary. Perhaps the PEP should define some way to tell the type checker not to follow certain imports? That would be useful in case you have a program that tries to follow the annotation conventions for static checking but imports some library that uses annotations for a different purpose. You can probably do this already for mypy by writing a stub module. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Dec 22 05:39:48 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 21 Dec 2014 20:39:48 -0800 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: Message-ID: Ludovic, If I understand you correctly you would like type annotations to be used to speed up Python programs. You are not alone, but it's a really hard problem, because nobody wants their program to break. I think that eventually this will be possible, but in the short term, it would be a much, much bigger effort to create such a compiler than it is just to define (and reach agreement on) the syntax for annotations. I like to take smaller steps first, especially when it comes to defining syntax. Maybe once we have a standard type hinting notation someone will write a compiler. In the meantime, let me give you an example of why it's such a hard problem. Suppose I have created a library that contains a function that finds the longest of two strings: def longest(a, b): return a if len(a) >= len(b) else b Now I start adding annotations, and I change this to: def longest(a: str, b: str) -> str: return a if len(a) >= len(b) else b But suppose someone else is using the first version of my library and they have found it useful for other sequences besides strings, e.g. they call longest() with two lists. That wasn't what I had written the function for. But their program works. Should it be broken when the annotations are added? Who gets the blame? How should it be fixed? Now, I intentionally chose this example to be quite controversial, and there are different possible answers to these questions. But the bottom line is that this kind of situation (and others that can be summarized as "duck typing") make it hard to create a Python compiler -- so hard that nobody has yet cracked the problem completely. Maybe once we have agreed on type hinting annotations someone will figure it out. In the meantime, I think "offline" static type checking such as done by mypy is quite useful in and of itself. Also, IDEs will be able to use the information conveyed by type hinting annotations (especially in stub files). So I am content with limiting the scope of the proposal to these use cases. Salut, --Guido On Sun, Dec 21, 2014 at 11:23 AM, Ludovic Gasc wrote: > > Hi Nick, > > Thanks for your answer. I understand primary goal, I'm not completely > naive on this question: A long time ago, I used a lot Type Hinting with PHP > 5. > > Nevertheless, in Python community, you can find a lot of libraries to > improve performance based on types handling with different optimization > strategies (numba, pypy, pythran, cython, llvm-py, shedskin...). > To my knowledge, you don't have the same number of libraries to do that > with another dynamic language. > It means that in Python community we have this problematic. > > It's like with async pattern, in Python you have a plenty of libraries > (Twisted, eventlet, gevent, stackless, tornado...) and now, with AsyncIO, > the community should converge on it. > > And yes, I understand that it's almost impossible to create a silver > bullet to improve automagically performance, but, as with my simple dev > eyes, the common pattern I see with pythran, cython... is the type handling. > They don't use only this strategy to improve performance, but it's the > biggest visible part in example codes I've seen. > > Guido: " optimizers have been quite successful without type hints" <= > Certainly, but instead of to loose time to try to guess the right data > structure, maybe it could be faster that the developer gives directly what > he wants. > > To be honest, I'm a little bit tired to listen some bias like "Python is > slow", "not good for performance", "you must use C/Java/Erlang/Go..." > For me, Python has the right compromise to write quickly readable source > code and performance possibilities to speed up your code. > > More we have primitives in CPython to build performant applications, more > it will be easier to convince people to use Python. > > Regards. > -- > Ludovic Gasc > On 21 Dec 2014 02:15, "Nick Coghlan" wrote: > >> On 21 December 2014 at 09:55, Ludovic Gasc wrote: >> >>> Hi, >>> >>> I'm only a simple Python developer, not a Type Hinting expert and I >>> don't know if you already discuss about that somewhere, but: >>> >>> With the future official support of Type Hinting in Python, is it means >>> that CPython could use this pieces of information to store variables with >>> more efficient data structures, not only check types ? >>> >>> It could possible to have better performance one day with Type Hinting >>> like you have with Cython (explicit types declaration) or PyPy (guessing >>> types) ? >>> >> >> The primary goals of the type hinting standardisation effort are >> improving program correctness (through enhanced static analysis) and API >> documentation (through clearer communication of expectations for input and >> output, both in the code, in documentation, and in IDEs). It should also >> allow the development of more advanced techniques for function signature >> based dispatch and other forms of structured pattern matching. >> >> From a performance perspective though, CPython already has a wide array >> of fast paths for particular builtin types - being able to communicate >> those assumptions more clearly won't automatically make it easier to speed >> them up further (or add more in a maintainable fashion). There's no >> *philosophical* objection to such changes (other than "simplicity is a >> virtue"), there's just a long track record of previous attempts like psyco >> and Unladen Swallow that make it clear that the problem is genuinely *hard*. >> >> There's also the fact that with both Numba and PyPy now supporting >> selective JIT acceleration of decorated functions within the context of a >> larger CPython application, as well as Cython's existing support for >> precompilation as a C extension, the pattern of profiling to find >> performance critical areas, and finding ways to optimise those, now seems >> well established. (Hence my suggestion the other day that we could likely >> use an introductory how to guide on performance profiling, which could also >> provide suggestions for optimisation tools to explore once the hot spots >> have been found). >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Dec 22 06:35:24 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 22 Dec 2014 15:35:24 +1000 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: <5495F193.9000402@gmail.com> Message-ID: On 22 December 2014 at 14:05, Guido van Rossum wrote: > I would really like to wait and see how this plays out -- the proposal I'm > working on is careful not to have any effect at runtime (as long as the > typing.py module can be imported and as long as the annotation expressions > don't raise exceptions), and use of a static checker is entirely optional > and voluntary. > Agreed - I see this as a good, incremental evolution from the increased formality and flexibility in the type system that was introduced with ABCs, and I think that's been successful in letting folks largely not need to worry about them unless they actually need to deal with the problems they solve (like figuring out whether a container is a Sequence or Mapping). Ideally we'll get the same result here - folks that have problems where type annotations can help will benefit, while those that don't need them won't need to worry about them. > Perhaps the PEP should define some way to tell the type checker not to > follow certain imports? That would be useful in case you have a program > that tries to follow the annotation conventions for static checking but > imports some library that uses annotations for a different purpose. You can > probably do this already for mypy by writing a stub module. > For inline use, it may be worth defining a module level equivalent to pylint's "#pylint: skip-file" comments (rather than having each static checker come up with its own way of spelling that). Aside from that, it may also be worth recommending that static type checkers provide clear ways to control the scope of scanning (e.g. by allowing particular directories and files to be excluded, or limit scans to particular directories. In both cases, the PEP would likely need to define the implied annotations to be assumed for excluded modules. I realised in trying to write this email that I don't currently understand the consequences of not having annotation data available for a module in terms of the ripple effects that may have on what scanners can check - from the end user perspective, I believe that's something I'd need to know, even though I wouldn't necessarily need to know *why* those were the default assumptions for unannotated operations. (I'm curious about the latter from a language *design* perspective, but I think I'd be able to use the feature effectively just by knowing the practical consequences without necessarily understanding the theory) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Dec 22 07:25:51 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 22 Dec 2014 16:25:51 +1000 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: Message-ID: On 22 December 2014 at 14:39, Guido van Rossum wrote: > Ludovic, > > If I understand you correctly you would like type annotations to be used > to speed up Python programs. You are not alone, but it's a really hard > problem, because nobody wants their program to break. I think that > eventually this will be possible, but in the short term, it would be a > much, much bigger effort to create such a compiler than it is just to > define (and reach agreement on) the syntax for annotations. I like to take > smaller steps first, especially when it comes to defining syntax. Maybe > once we have a standard type hinting notation someone will write a compiler. > I'd like to echo this particular sentiment, since my other replies could easily give the impression I disagreed. I'm actually optimistic that once this kind of annotation becomes popular, folks will find ways to make use of it for performance improvements (even if it's just priming a JIT cache to reduce warmup time). The only idea I disagree with is that it's necessary to *wait* for this feature for Pythonistais to start countering the "Python is slow" meme - the 3.5 release is still almost a year away, and the migration of the overall ecosystem from Python 2 to 3 is definitely still a work in progress. As a result, I believe that in the near term, it's better to look at the barriers to adoption for existing tools like PyPy, Numba and Cython and start figuring out whether those barriers can be reduced in some way. In the case of PyPy, deployment of web services using it can be awkward relative to CPython. A prebuilt image on DockerHub that includes uWSGI + the PyPy plugin + PyPy could be a good way to reduce barriers to entry for Linux based deployments. A separate prebuilt image on DockerHub could also be a good way to make it easy for folks to experiment with building web services based on the PyPy STM branch. Another barrier to PyPy adoption is the level of use of cffi vs hand crafted C extensions. While the inclusion of pip with CPython makes cffi more readily available, there's also in principle agreement to incorporate it (and its dependencies) into the standard library. Folks interested in increasing PyPy's popularity may want to get in touch with the cffi developers and see what remaining barriers (other than available time) there are to moving forward with that proposal as a PEP for Python 3.5. (Standard library backports have more credibility as dependencies in many situations, since the standard library inclusion acts as a clear endorsement of the original project) For Numba and Cython, challenges are more likely to be centred around the process of getting started with the tools, and providing a suitable build environment. Again, Docker images can help, but in this case, by providing a preconfigured build environment that can either run in a local container on Linux, or under a Linux VM on Mac OS X or Windows. None of those Docker related ideas need to be filtered through the CPython core development team, though, nor even through the development teams of the individual projects. All it takes is someone sufficiently motivated to figure out how to do it, publish their work, and then approach the relevant project to say "Hey, I did this, does anyone else want to help promote and maintain it, and perhaps give it an official project blessing?". Unlike the Docker image ideas, putting together the cffi PEP *does* need involvement from both the developers of the cffi project and the rest of the CPython core development team, but if the cffi folks are amenable, it's conceivable someone else could do the actual PEP wrangling. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Dec 22 08:02:00 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 21 Dec 2014 23:02:00 -0800 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: <5495F193.9000402@gmail.com> Message-ID: On Sun, Dec 21, 2014 at 9:35 PM, Nick Coghlan wrote: > > > [...] I realised in trying to write this email that I don't currently > understand the consequences of not having annotation data available for a > module in terms of the ripple effects that may have on what scanners can > check - from the end user perspective, I believe that's something I'd need > to know, even though I wouldn't necessarily need to know *why* those were > the default assumptions for unannotated operations. (I'm curious about the > latter from a language *design* perspective, but I think I'd be able to use > the feature effectively just by knowing the practical consequences without > necessarily understanding the theory) > That's what "Any" is for. If an object's type is "Any", then every operation on it (including getattr) also return "Any", from the checker's POV. You stop the ripples with an explicitly declared type -- e.g. if you pass it to a function as an arg with a type annotation, inside that function the argument is assumed to have the declared type (but the "Any" prevents diagnostics about the call site). In mypy there is also an explicit cast operator (http://mypy.readthedocs.org/en/latest/casts.html), we may add this to the PEP (it's one of the many details left out from the Quip doc that need to be filled in for the actual PEP). The importance of "Any" cannot be overstated -- without it, you either have to declare types everywhere, or you end up with a "everything inherits from everything" situation. The invention of "Any" prevents this (and this is why is-consistent-with cannot be transitive). Read Jeremy Siek's Gradual Typing blog post for more. Also, consider the important difference between Any and object. They are both at the top of the class tree -- but object has *no* operations (well, almost none -- it has repr() and a few others), while Any supports *all* operations (in the sense of "is allowed by the type system/checker"). This places Any also at the *bottom* of the class tree, if you can call it that. (And hence it is more a graph than a tree -- but if you remove Any, what's left is a tree again.) Any's "contagiousness" is somewhat similar to NaN for floats, but it has its limits -- e.g. repr() of Any is still a string, and Any==Any is still a bool. Another similar concept exists IIUC in Objective-C, which has a kind of null object that can be called (sent a message), always returning another null result. (This has occasionally been proposed for Python's None, but I don't think it is appropriate.) But of course, Any only exists in the mind of the static checker -- at runtime, the object has a concrete type. "Any" is just the type checker's way to say "I don't know the type, and I don't care". I bet compilers with good error recovery have some similar concept internally, used to avoid a cascade of errors due to a single typo. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Mon Dec 22 12:34:23 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 22 Dec 2014 12:34:23 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: Message-ID: On 21/12/14 02:15, Nick Coghlan wrote: > There's also the fact that with both Numba and PyPy now supporting > selective JIT acceleration of decorated functions within the context of > a larger CPython application, as well as Cython's existing support for > precompilation as a C extension, the pattern of profiling to find > performance critical areas, and finding ways to optimise those, now > seems well established. Numba is not very mature, but it can already JIT accelerate most of Python bytecode to speeds comparable to -O2 in C. And nit just that, it does not need type-hints, it has also an "autojit" that can infere types at runtime. What the Unladen Swallow failed and Numba succeded? I think for two reasons: First, Numba was designed for a particular purpose: numerical computing. No offence to Google, but the numerics folks are the ones who really understand how to beat gigaflops out of the CPU. Numba was created by Travis Oliphant, who was also the creator of the modern NumPy package (as opposed to Jim Hugunin's original Numerics package and NASA's Numarray packages). The Unladen Swallow did the mistake of trying to accelerate "everything". Second, LLVM has improved. At the time the swallow was hatched, LLVM was not really useful as a JIT compiler framework. It was in comparison to GCC even a lousy static compiler. Now it has matured and it is excellent as as a JIT compiler as well as a static compiler. It is still a little behind GCC and Intel compilers on static optimization, but more than good enough to be the default compiler on MacOSX and FreeBSD. It should also be mentioned that the Numba (or just NumbaPro?) integrates with CUDA, and it can actually run Python bytecode on the GPU. For certain data-parallel tasks it can make your Python program yield a teraflop with a modern Nvidia GPU. I also want to mention Cython. It has also shown that compiled Python can often run at the speed of C++. But its real merit is the ease with which we can write C extensions for Python. I think in the long run, CPython could benefit from Numba as well as Cython. Numba can be used to boost performance critical code without having to resort to C. Cython can e.g. be used to implement the standard library without hand-written C modules. It would lower the threshold for contribution as well as improve readability. Sturla From encukou at gmail.com Mon Dec 22 12:54:04 2014 From: encukou at gmail.com (Petr Viktorin) Date: Mon, 22 Dec 2014 12:54:04 +0100 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: <5495F193.9000402@gmail.com> Message-ID: On Mon, Dec 22, 2014 at 8:02 AM, Guido van Rossum wrote: [...] > limits -- e.g. repr() of Any is still a string, and Any==Any is still a > bool. Why is Any==Any a bool? Comparison operators can return anything, and libraries like Numpy or SQLAlchemy take advantage of this (comparing Numpy arrays results in an array of bools, and comparing a SQLAlchemy Column to something results in a comparison expression, e.g. `query.filter(table.id == 2)`). Would type checkers be expected to reject these uses? >From Pragmatics section in the article: > If a default of None is specified, the type is implicitly optional, e.g. > def get(key: KT, default: VT = None) -> VT: ... Why is this limited to None? Can this be extended to "if there's a default argument, then that exact object is also allowed"? That would allow things like: _MISSING = object() def getattr(key: str, default: VT=_MISSING) -> VT: ... From sturla.molden at gmail.com Mon Dec 22 13:05:48 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 22 Dec 2014 13:05:48 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: Message-ID: On 22/12/14 00:20, Chris Angelico wrote: > There may be something to that. Most of the people I've heard moaning > that "Python is slow" aren't backing that up with any actual facts. One fact is that the fastest Random Forests classifier known to man is written in Python (with addition of some Cython in critical places). For those who don't know, Random Fortests is one of the strongest algorithms (if not the strongest) for estimating or predicting a probability p as a non-linear function of N variables. Here are some interesting slides: http://www.slideshare.net/glouppe/accelerating-random-forests-in-scikitlearn Just look at what they could make Python and Cython do, compared to e.g. the pure C++ solution in OpenCV. Even more interesting, the most commonly used implementation of Random Forests is the version for R, which is written in Fortran. There is actually a pure Python version which is faster... So who says Python is slow? I think today, mostly people who don't know what they are talking about. We see Python running at the biggest HPC systems today. We see Python algorithms beating anything we can throw at it with C++ or Fortran. It is time to realize that the flexibility of Python is not just a slowdown, it is also a speed boost because it makes it easier to write flexible and complex algorithms. But yes, Python needs help from Numba or Cython, and sometimes even Fortran (f2py), to achieve its full potential. The main take-home lesson from those slides, by the way, is the need to (1) profile Python code to identify bottlenecks -- humans are very bad at this kind of guesswork -- and (2) the need to use code annotation in Cython (compile with -a) to limit the Python overhead. Sturla From ncoghlan at gmail.com Mon Dec 22 13:28:55 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 22 Dec 2014 22:28:55 +1000 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: <5495F193.9000402@gmail.com> Message-ID: On 22 Dec 2014 17:02, "Guido van Rossum" wrote: > > Also, consider the important difference between Any and object. They are both at the top of the class tree -- but object has *no* operations (well, almost none -- it has repr() and a few others), while Any supports *all* operations (in the sense of "is allowed by the type system/checker"). Ah, this was the key piece I had missed, both from your article and Jeremy's: since Any conceptually allows all operations, with a result of Any, there's no need to separately represent "callable that permits arbitrary arguments, returning a result of unknown type". Very nice! Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Mon Dec 22 13:43:20 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 22 Dec 2014 13:43:20 +0100 Subject: [Python-ideas] Bug? Feature? setattr(foo, '3', 4) works! In-Reply-To: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> References: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> Message-ID: On 19/12/14 05:27, Cem Karan wrote: > I'd like to suggest that getattr(), setattr(), and hasattr() all be modified so that syntactically invalid statements raise SyntaxErrors. I'd like to object! This would break a lot of code that depends on HD5 and NetCDF databases, e.g. PyTables. Often we can do table.x but sometimes we need to do getattr(table,'some long field name') A lot of scientific software depends on this, and it would break at the moment your proposal is introduced. It would be a major disaster. It would even set back NASA's space program (no kidding!) Sturla From liam.marsh.home at gmail.com Mon Dec 22 14:06:52 2014 From: liam.marsh.home at gmail.com (Liam Marsh) Date: Mon, 22 Dec 2014 14:06:52 +0100 Subject: [Python-ideas] Fwd: improve compatibility In-Reply-To: References: <00DC45BE-A51D-4EB7-BF90-D4BA2F4B3189@yahoo.com> <54976326.4090707@trueblade.com> Message-ID: """That's suggesting mixing and matching between versions within the same process space, which is highly unlikely to work correctly (except when modules are specifically designed for that, as indicated by the availability of backported versions on PyPI).""" In fact, i was thinking about a small program, which just analize the file for these commentaries, and asks the OS to launch the appropriate "*real" *python environnement, and finishes. This small program will be also be able to chose the appropriate version by scanning the import statements, and seeking for these imported modules in the different environnements installed. 2014-12-22 1:27 GMT+01:00 Nick Coghlan : > On 22 December 2014 at 10:17, Eric V. Smith wrote: > >> I thought he was looking for something like the py launcher, implemented >> for Windows as PEP 397, and discussed for Linux here: >> https://mail.python.org/pipermail/python-ideas/2014-April/027633.html >> > > I interpreted the second suggestion in light of the first one, which > included the point "-any imported py file will be able to choose which > version it wants". That's suggesting mixing and matching between versions > within the same process space, which is highly unlikely to work correctly > (except when modules are specifically designed for that, as indicated by > the availability of backported versions on PyPI). > > Either way, there's a lot of prior art in this space that actually works > pretty well once you're aware of its existence, which means future > improvements are likely to come in the form of availability, usability, and > discoverability improvements for already popular approaches, rather than > introducing radical new concepts. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Dec 22 14:16:49 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 23 Dec 2014 00:16:49 +1100 Subject: [Python-ideas] Fwd: improve compatibility In-Reply-To: References: <00DC45BE-A51D-4EB7-BF90-D4BA2F4B3189@yahoo.com> <54976326.4090707@trueblade.com> Message-ID: On Tue, Dec 23, 2014 at 12:06 AM, Liam Marsh wrote: > """That's suggesting mixing and matching between versions within the same > process space, which is highly unlikely to work correctly (except when > modules are specifically designed for that, as indicated by the availability > of backported versions on PyPI).""" > > In fact, i was thinking about a small program, which just analize the file > for these commentaries, and asks the OS to launch the appropriate "real" > python environnement, and finishes. If you just want to have the initial script control its version, then that can be done. Put an appropriate shebang at the top of the file, use the py.exe launcher on Windows, and that'll all work. > This small program will be also be able to chose the appropriate version by > scanning the import statements, and seeking for these imported modules in > the different environnements installed. Not sure what you mean here, but if you use multiple different interpreters all installed concurrently, they'll each have their own separate module trees. Or do you mean different versions of imported libraries (so you could have, for instance, numpy 1.8.2 for this project, and numpy 1.9.1 for another)? In that case, you'd do well to use virtual environments, which cope just fine with this kind of thing. The one thing you *cannot* do easily is have an application request one Python version, and one of its imported modules request a different Python version. Similarly if you have two modules that each import the same third module; the nature of Python's import machinery is such that they have to get the same version (in fact, the same module). Within one execution space, you have to be consistent, or else go through some serious hoop-jumping to keep things separate. ChrisA From cfkaran2 at gmail.com Mon Dec 22 14:44:50 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Mon, 22 Dec 2014 08:44:50 -0500 Subject: [Python-ideas] Bug? Feature? setattr(foo, '3', 4) works! In-Reply-To: References: <55DDD42E-41AA-4C45-88A2-EBD35B927818@gmail.com> Message-ID: On Dec 22, 2014, at 7:43 AM, Sturla Molden wrote: > On 19/12/14 05:27, Cem Karan wrote: > >> I'd like to suggest that getattr(), setattr(), and hasattr() all be modified so that syntactically invalid statements raise SyntaxErrors. > > > I'd like to object! This would break a lot of code that depends on HD5 and NetCDF databases, e.g. PyTables. > > Often we can do > > table.x > > but sometimes we need to do > > getattr(table,'some long field name') > > A lot of scientific software depends on this, and it would break at the moment your proposal is introduced. It would be a major disaster. It would even set back NASA's space program (no kidding!) Yup, someone else pointed out that ctypes also relies on the current behavior, so I dropped the suggestion. (To the person that pointed about ctypes, I apologize about not giving you credit, but I deleted the email). Thanks, Cem Karan From abarnert at yahoo.com Mon Dec 22 17:19:31 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 22 Dec 2014 08:19:31 -0800 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: <5495F193.9000402@gmail.com> Message-ID: On Dec 21, 2014, at 23:02, Guido van Rossum wrote: > Any's "contagiousness" is somewhat similar to NaN for floats, but it has its limits -- e.g. repr() of Any is still a string, and Any==Any is still a bool. Another similar concept exists IIUC in Objective-C, which has a kind of null object that can be called (sent a message), always returning another null result. (This has occasionally been proposed for Python's None, but I don't think it is appropriate.) I think you're mixing up two things here. ObjC does have a concept pretty close to Any, but it's not nil, it's id. This is a type that's never seen at runtime, but at compile time, in practice,* it's both a top and bottom type** (any method can be called on an object declared as id; any argument can be passed to a covariant or contravariant parameter of type id; a value declared id can be stored or passed anywhere), and subtyping is transitive except as far as id is concerned. But nil is a whole different thing. Dynamically, it's similar to Python's None, except that it responds to every method with None. Statically, every type is effectively an Optional[T], and can hold either a T (or subclass) instance or nil.*** The static type of nil might be id, but it doesn't really matter, because that never comes up.**** The dynamic type of nil is nil.***** * In reality, this relies on the fact that ObjC objects are always referenced by C pointers, and id is effectively just a typedef for void*, and C allows any pointer to be implicitly cast to and from void*. ** Except when native C types get involved. Traditional ObjC waved that away by assuming that every C type except double and long long was the same size as a pointer and you don't use those two very often; the 64-bit transition screwed that up, and the compiler no longer lets you mix up ObjC and native C types. *** Again, this is because you always reference ObjC objects as C pointers, and C pointers can always accept null values. **** Unless you explicitly use the compile-time typeof or sizeof operators on nil, which there's no good reason to do. If you do, then you'll see it's defined as (id)0. ***** Because the equivalent of the type function is a method. From ncoghlan at gmail.com Mon Dec 22 17:42:51 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 23 Dec 2014 02:42:51 +1000 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: Message-ID: On 22 December 2014 at 22:05, Sturla Molden wrote: > So who says Python is slow? I think today, mostly people who don't know > what they are talking about. > > We see Python running at the biggest HPC systems today. We see Python > algorithms beating anything we can throw at it with C++ or Fortran. It is > time to realize that the flexibility of Python is not just a slowdown, it > is also a speed boost because it makes it easier to write flexible and > complex algorithms. But yes, Python needs help from Numba or Cython, and > sometimes even Fortran (f2py), to achieve its full potential. > Yep - the challenge is to: a) communicate the potential of these tools effectively to new Python users b) ensure these tools are readily available to them (without turning into a build nightmare) A profiling HOWTO (especially one that highlights visual tools like SnakeViz) may be helpful on the first point, but the second one is a fair bit harder to address in the general case. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Mon Dec 22 18:06:39 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 22 Dec 2014 18:06:39 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: Message-ID: On 22/12/14 17:42, Nick Coghlan wrote: > a) communicate the potential of these tools effectively to new Python users > b) ensure these tools are readily available to them (without turning > into a build nightmare) For b), Anaconda by Continuum Analytics and Canopy by Enthought are doing a great job. But today it is also possible to create a minimalistic environment with only a few 'pip install' commands. It is not a build nightmare anymore. Sturla From ethan at stoneleaf.us Mon Dec 22 21:32:27 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 22 Dec 2014 12:32:27 -0800 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: <5495F193.9000402@gmail.com> Message-ID: <54987FDB.2020602@stoneleaf.us> On 12/21/2014 08:05 PM, Guido van Rossum wrote: > On Sun, Dec 21, 2014 at 3:47 PM, Nick Coghlan > wrote: > > > On 22 December 2014 at 06:32, Andrew Svetlov > > wrote: > > Sorry, I want to ask again. > The proposal is for static checks only? > My expectations for processing annotations in runtime as-is > (just a > mark without any restrictions) will not changed? > > > Correct, there are no changes being proposed to the runtime > semantics of annotations. The type hinting proposal describes a > conventional use for them that will be of benefit to static type > checking systems and integrated development environments, but it > will be exactly that: a convention, not an enforced behaviour. > > > Well... The elephant in the room is that *eventually* other uses of > annotations *may* be frowned upon, or may need to be marked by some > decorator. But I promise that in Python 3.5 your code will not break > -- it just might not be very useful to run a static checker like mypy > on it. (IIRC mypy used to only typecheck code that imports the > typing.py module, but this seems to have changed.) > > When we discussed this earlier this year, a few other uses of > annotations were brought up, and some have proposed that static type > annotations would need to be marked by a decorator. There is even a > proposed syntax that allows multiple annotations to coexist for the > same argument (a dict with fixed keys -- to me it looks pretty ugly > though). What I ended up doing for my scription program was to move the annotations outside the def, and store them in a different attribute, applied with a decorator: @Command( file=('source file', ), dir=('target directory', ), options=('extra options', MULTI, ), ) def copy(file, dir, options): pass copy.__scription__ --> {'file':..., 'dir':..., 'options':...} -- ~Ethan~ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmludo at gmail.com Tue Dec 23 00:56:28 2014 From: gmludo at gmail.com (Ludovic Gasc) Date: Tue, 23 Dec 2014 00:56:28 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: Message-ID: Thank you everybody for all pieces of information, very interesting. I reply to everybody in the same e-mail: 1. For Python 3 usage: Where I work, we switched all new projects to Python 3 since almost one year. To be honest with you, it wasn't to be up-to-date, but it was for a new feature not natively present in Python 2: AsyncIO. We made a lot of Twisted (Telephony+WebSockets) and Django/Flask daemons (WebServices), but I wanted to: A. Create daemons all-in-one (Telephony+WebSockets+WebServices) to share easier our business logic for a same project. B. Simplify architecture: Twisted and Django are very complicated for our simple needs. The side effects are: A. Our productivity is better, we finish quicker our projects for our clients, because we share more source code and the architecture is simpler to handle. B. The performances we have are better compare to the past. Why I tell you that ? To give you a concrete example: if you want to motivate Python developers from the battlefield to migrate to Python 3, you need to add features/performances/... in Python 3. Not add all PyPI libraries in CPython, but add features you can't add easily via a library, like yield from, AsyncIO or Type Hinting. On production systems, who cares is Python 2/3, Go, Erlang... ? Certainly not clients and even management people. If you want that we use Python 3, please give us arguments to "sell" the migration to Python 3 in the company. 2. For Python 3 deployment: I use Pythonz: https://github.com/saghul/pythonz to quickly deploy a Python version on a new server with an Ansible recipe. Certainly, some sys admins could be shocked by this behaviour because it's forbidden in packaging religion, but who cares ? We lose less time to deploy, it's more reproducible between dev environment and production, and upgrades are very easy via Ansible. 3. About PyPy usage: I've made some benchmarks with the same WebService between a Flask daemon on PyPy and an AsyncIO daemon on CPython, it was very interesting: Compare to our needs, asynchronous pattern give us more performance that PyPy. Yes, I know, I've compared apples with pears, but at the end, I only want how many customers I can stack on the same server. More I stack, less it costs for my company. I'm waiting Python 3.3 support in PyPy to push that on production with a real daemon. No AsyncIO, No PyPy. 4. About Python performance: Two things changed my mind about the bias "Python is slow": A. Python High Performance: http://shop.oreilly.com/product/0636920028963.do B. Web Framework Benchmarks: http://www.techempower.com/benchmarks/ Especially with B.: this benchmark isn't the "truth", but at least, you can compare a lot of languages/frameworks based on examples more closer than my use cases, compare to others benchmarks. But, for example, if you compare Python frameworks with Erlang frameworks on "multiple queries", you can see that, in fact, Python is very good. In my mind, Erlang is very complicated to code compare to Python, but you'll have better performances in all cases, I had a wrong opinion. Finally, everybody has bias about programming languages performance. 5. Type Hinting - Performance booster (the first goal of my e-mail): Thank you Guido, your example is clear. I understand that it will be step-by-step, it's a good thing. I took the liberty to send an e-mail, because I didn't sure to understand correctly the global roadmap. I'm the first to understand that it's very difficult, and maybe that CPython will never use Type Hinting to improve performances, because it isn't really simple to implement. I'm pretty sure I can't help you to implement that, but at least, I can promote to others. Thank you everybody again for your attention. Met vriendelijke groeten, -- Ludovic Gasc On Mon, Dec 22, 2014 at 6:06 PM, Sturla Molden wrote: > On 22/12/14 17:42, Nick Coghlan wrote: > > a) communicate the potential of these tools effectively to new Python >> users >> b) ensure these tools are readily available to them (without turning >> into a build nightmare) >> > > For b), Anaconda by Continuum Analytics and Canopy by Enthought are doing > a great job. > > But today it is also possible to create a minimalistic environment with > only a few 'pip install' commands. > > It is not a build nightmare anymore. > > > Sturla > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Dec 23 01:59:33 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 22 Dec 2014 16:59:33 -0800 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: <5495F193.9000402@gmail.com> Message-ID: On Mon, Dec 22, 2014 at 3:54 AM, Petr Viktorin wrote: > > On Mon, Dec 22, 2014 at 8:02 AM, Guido van Rossum > wrote: > [...] > > limits -- e.g. repr() of Any is still a string, and Any==Any is still a > > bool. > > Why is Any==Any a bool? > Comparison operators can return anything, and libraries like Numpy or > SQLAlchemy take advantage of this (comparing Numpy arrays results in > an array of bools, and comparing a SQLAlchemy Column to something > results in a comparison expression, e.g. `query.filter(table.id == > 2)`). Would type checkers be expected to reject these uses? > Never mind, I forgot about that. :-) > > From Pragmatics section in the article: > > If a default of None is specified, the type is implicitly optional, e.g. > > def get(key: KT, default: VT = None) -> VT: ... > > Why is this limited to None? Can this be extended to "if there's a > default argument, then that exact object is also allowed"? That would > allow things like: > > _MISSING = object() > def getattr(key: str, default: VT=_MISSING) -> VT: ... > Well, that would require the type checker to be smarter. I don't want to go down the slippery path of defining compile-time expressions. Allowing None is easy (it's a keyword), useful and covers the common cases. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From liam.marsh.home at gmail.com Tue Dec 23 10:52:33 2014 From: liam.marsh.home at gmail.com (Liam Marsh) Date: Tue, 23 Dec 2014 10:52:33 +0100 Subject: [Python-ideas] Fwd: improve compatibility In-Reply-To: References: <00DC45BE-A51D-4EB7-BF90-D4BA2F4B3189@yahoo.com> <54976326.4090707@trueblade.com> Message-ID: sorry, I did'nt know it allready existed... thank you! 2014-12-22 14:16 GMT+01:00 Chris Angelico : > On Tue, Dec 23, 2014 at 12:06 AM, Liam Marsh > wrote: > > """That's suggesting mixing and matching between versions within the same > > process space, which is highly unlikely to work correctly (except when > > modules are specifically designed for that, as indicated by the > availability > > of backported versions on PyPI).""" > > > > In fact, i was thinking about a small program, which just analize the > file > > for these commentaries, and asks the OS to launch the appropriate "real" > > python environnement, and finishes. > > If you just want to have the initial script control its version, then > that can be done. Put an appropriate shebang at the top of the file, > use the py.exe launcher on Windows, and that'll all work. > > > This small program will be also be able to chose the appropriate version > by > > scanning the import statements, and seeking for these imported modules in > > the different environnements installed. > > Not sure what you mean here, but if you use multiple different > interpreters all installed concurrently, they'll each have their own > separate module trees. Or do you mean different versions of imported > libraries (so you could have, for instance, numpy 1.8.2 for this > project, and numpy 1.9.1 for another)? In that case, you'd do well to > use virtual environments, which cope just fine with this kind of > thing. > > The one thing you *cannot* do easily is have an application request > one Python version, and one of its imported modules request a > different Python version. Similarly if you have two modules that each > import the same third module; the nature of Python's import machinery > is such that they have to get the same version (in fact, the same > module). Within one execution space, you have to be consistent, or > else go through some serious hoop-jumping to keep things separate. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmueller at python-academy.de Tue Dec 23 12:14:05 2014 From: mmueller at python-academy.de (=?windows-1252?Q?Mike_M=FCller?=) Date: Tue, 23 Dec 2014 12:14:05 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: Message-ID: <54994E7D.9010308@python-academy.de> Am 23.12.14 um 00:56 schrieb Ludovic Gasc: > I'm waiting Python 3.3 support in PyPy to push that on production with a real > daemon. No AsyncIO, No PyPy. Trollius should work on PyPy already: https://pypi.python.org/pypi/trollius Trollius is a portage of the Tulip project (asyncio module, PEP 3156) on Python 2. Trollius works on Python 2.6-3.5. It has been tested on Windows, Linux, Mac OS X, FreeBSD and OpenIndiana. http://trollius.readthedocs.org/en/latest/changelog.html 2014-07-30: Version 1.0.1 This release supports PyPy and has a better support of asyncio coroutines, especially in debug mode. Mike From stefan_ml at behnel.de Wed Dec 24 12:45:11 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 24 Dec 2014 12:45:11 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: Message-ID: Ludovic Gasc schrieb am 21.12.2014 um 00:55: > With the future official support of Type Hinting in Python, is it means > that CPython could use this pieces of information to store variables with > more efficient data structures, not only check types ? > > It could possible to have better performance one day with Type Hinting like > you have with Cython (explicit types declaration) or PyPy (guessing types) ? > > Is it realistic to except that one day, or I've missed some mountains ;-) ? > > If this is correct, better performances will be a great consequence for > Type Hinting Regarding Cython, it's actually unlikely that it will bring anything. The proposed feature is about specifying Python object types for function arguments and (to a certain extent) local variables, and Cython is already pretty good in guessing those or doing optimistic optimisations that just say "if it looks like you're using a dict, let's generate special code that speeds up dict access and leaves other stuff to a fallback code path". The type semantics are even different for builtin types. When you type a variable as "dict" in Cython, it will reject subtypes as there's no use in typing a variable as "dict" if it can't be optimised as a dict due to potential subtype overrides. The normal optimistic "fast for dicts, works for all objects" code that Cython generates here is exactly as fast as code that would allow only dict or subtypes. Typing a variable as arbitrary size Python "int" is also useless as there is no advantage Cython can draw from it, but even typing a variable as Python "float", which is identical to a C double, isn't helpful due to the semantic difference for subtypes. Cython could be slightly improved to also generate special casing low level code paths for Python integer and floating point operations, but then, why waste time doing that when simply typing a variable as plain C int or double gives you full native speed without any of the hassle of runtime type checks, bloated special casing code paths, etc.? And typing variables with user implemented Python classes is equally useless as there are no special optimisations that Cython could apply to them. So, from the POV of Cython, I do see the advantage of a syntax that integrates with Python's own syntax (Cython's "pure Python mode" [1] has provided similar typing support for years now), but please don't expect any performance advantage from this. Stefan [1] http://docs.cython.org/src/tutorial/pure.html From sturla.molden at gmail.com Wed Dec 24 13:40:15 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Wed, 24 Dec 2014 13:40:15 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: Message-ID: On 24/12/14 12:45, Stefan Behnel wrote: > Regarding Cython, it's actually unlikely that it will bring anything. The > proposed feature is about specifying Python object types for function > arguments and (to a certain extent) local variables, and Cython is already > pretty good in guessing those or doing optimistic optimisations that just > say "if it looks like you're using a dict, let's generate special code that > speeds up dict access and leaves other stuff to a fallback code path". Numba is also very similar to Cython here. There is a set of types and expressions for which it can produce an efficient code path, and otherwise it falls back to calling the CPython interpreter. Psyco also used a similar strategy. This precisely where the Uladen Swallow went astray because they tried to accelerate any code path. Cython's advantage is that we can mix "down to the iron" C or C++ with Python, without having to use or know the Python C API. And of course Cython is not only for speed. It is also for making it easier to write C or C++ extensions for any thinkable reason. Sturla From ron3200 at gmail.com Wed Dec 24 20:04:01 2014 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 24 Dec 2014 13:04:01 -0600 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: Message-ID: On 12/24/2014 06:40 AM, Sturla Molden wrote: > On 24/12/14 12:45, Stefan Behnel wrote: > >> Regarding Cython, it's actually unlikely that it will bring anything. The >> proposed feature is about specifying Python object types for function >> arguments and (to a certain extent) local variables, and Cython is already >> pretty good in guessing those or doing optimistic optimisations that just >> say "if it looks like you're using a dict, let's generate special code that >> speeds up dict access and leaves other stuff to a fallback code path". > > Numba is also very similar to Cython here. There is a set of types and > expressions for which it can produce an efficient code path, and otherwise > it falls back to calling the CPython interpreter. Psyco also used a similar > strategy. This precisely where the Uladen Swallow went astray because they > tried to accelerate any code path. > > Cython's advantage is that we can mix "down to the iron" C or C++ with > Python, without having to use or know the Python C API. And of course > Cython is not only for speed. It is also for making it easier to write C or > C++ extensions for any thinkable reason. My thoughts is that making python easier to multi-process on multi-core CPUs will be where the biggest performance gains will be. Think of 100 core chips in as soon as 5 or 6 years. (Doubling about every two years, you can get 16 core CPU's now.) It's not uncommon to have multi-term expressions where each term can be processed independently of the others. result = foo(e1, e2, ... eN) Where each expression, "eN" may be call's to other functions, which may also have multiple terms. If it can be determined at parse time that they can't possibly effect each other, then these terms can be evaluated in parallel, and byte code, or compiled code, that does that can be generated. (I'm not sure it's doable, but it seems like it should be possible.) And... the same type of data dependence graphs needed to do that, can aid in checking correctness of code. Which is the near term benefit. Cheers, Ron From eltoder at gmail.com Thu Dec 25 01:50:09 2014 From: eltoder at gmail.com (Eugene Toder) Date: Thu, 25 Dec 2014 00:50:09 +0000 (UTC) Subject: [Python-ideas] Type Hinting Kick-off References: Message-ID: Guido van Rossum writes: > A few months ago we had a long discussion about type hinting. I've thought a > lot more about this. I've written up what I think is a decent "theory" > document -- writing it down like this certainly helped *me* get a lot of > clarity about some of the important issues.https://quip.com/r69HA9GhGa7J (I apologize in advance if some of this was covered in previous discussions). 1. Since there's the Union type, it's also natural to have the Intersection type. A class is a subclass of Intersection[t1, t2, ...] if it's a subclass of all t1, t2 etc. The are 2 main uses of the Intersection type: a) Require that an argument implements multiple interfaces: class Foo: @abstractmethod def foo(self): ... class Bar: @abstractmethod def bar(self): ... def fooItWithABar(obj: Intersection[Foo, Bar]): ... b) Write the type of an overloaded function: @overload def foo(x: str) -> str: ... @overload def foo(x: bytes) -> bytes: ... foo # type: Intersection[Callable[[str], str], Callable[[bytes], bytes]] 2. Constrained type variables (Var('Y', t1, t2, ...)) have a very unusual behavior. a) "subclasses of t1 etc. are replaced by the most-derived base class among t1 etc." This defeats the very common use of constrained type variables: have a type preserving function limited to classes inherited from a common base. E.g. say we have a function: def relocate(e: Employee) -> Employee: ... The function actually always returns an object of the same type as the argument, so we want to write a more precise type. We usually do it like this: XEmployee = Var('XEmployee', Employee) def relocate(e: XEmployee) -> XEmployee: ... This won't work with the definition from the proposal. b) Multiple constraints introduce an implicit Union. I'd argue that type variables are more commonly constrained by an Intersection rather than a Union. So it will be more useful if given this definition Y has to be compatible with all of t1, t2 etc, rather than just one of them. Alternatively, this can be always spelled out explicitly: Y1 = Var('Y1', Union[t1, t2, ...]) Y2 = Var('Y2', Intersection[t1, t2, ...]) Pragmatics: 3. The names Union and Intersection are standard terminology in type checking, but may not be familiar to many Python users. Names like AnyOf[] and AllOf[] can be more intuitive. 4. Similar to allowing None to mean type(None) it's nice to have shortcuts like: (t1, t2, ...) == Tuple[t1, t2, ...] [t1] == List[t1] {t1: t2} == Dict[t1, t2] {t1} == Set[t1] The last 3 can be Sequence[t1], Mapping[t1, t2] and collections.Set[t1] if we want to encourage the use of abstract types. 5. Using strings for forward references can be messy in case of generics: parsing of brackets etc in the string will be needed. I propose explicit forward declarations: C = Declare('C') class C(Generic[X]): def foo(self, other: C[X]): ... def bar(self, other: C[Y]): ... 6. On the other hand, using strings for unconstrained type variables is quite handy, and doesn't share the same problem: def head(xs: List['T']) -> 'T': ... Regards, Eugene From eltoder at gmail.com Thu Dec 25 00:22:59 2014 From: eltoder at gmail.com (Eugene Toder) Date: Wed, 24 Dec 2014 15:22:59 -0800 (PST) Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: <5495F193.9000402@gmail.com> Message-ID: <8f2e8f18-08d3-4384-87c3-e53acfa4bdcf@googlegroups.com> On Monday, December 22, 2014 2:03:28 AM UTC-5, Guido van Rossum wrote: > > This places Any also at the *bottom* of the class tree, if you can call it > that. (And hence it is more a graph than a tree -- but if you remove Any, > what's left is a tree again.) > I may be misunderstanding the context, but since Python has multiple inheritance, the "class tree" was already a graph even without Any. Eugene -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Dec 25 05:16:52 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 24 Dec 2014 20:16:52 -0800 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: Message-ID: On Wed, Dec 24, 2014 at 4:50 PM, Eugene Toder wrote: > Guido van Rossum writes: > > [...] .https://quip.com/r69HA9GhGa7J > > (I apologize in advance if some of this was covered in previous > discussions). > No problem. :-) I apologize for reformatting the text I am quoting from you, it looked as if it was sent through two different line clipping functions. > 1. Since there's the Union type, it's also natural to have the Intersection > type. A class is a subclass of Intersection[t1, t2, ...] if it's a subclass > of all t1, t2 etc. The are 2 main uses of the Intersection type: > > a) Require that an argument implements multiple interfaces: > > class Foo: > @abstractmethod > def foo(self): ... > > class Bar: > @abstractmethod > def bar(self): ... > > def fooItWithABar(obj: Intersection[Foo, Bar]): ... > Yes, we even have an issue to track this proposal. I don't recall who suggested it first. I don't know if it poses any problems to the static checked (though I doubt it). https://github.com/ambv/typehinting/issues/18 > b) Write the type of an overloaded function: > > @overload > def foo(x: str) -> str: ... > @overload > def foo(x: bytes) -> bytes: ... > > foo # type: Intersection[Callable[[str], str], Callable[[bytes], bytes]] > The static checker can figure that out for itself, but that doesn't mean we necessarily need a way to spell it. > 2. Constrained type variables (Var('Y', t1, t2, ...)) have a very unusual > behavior. > > a) "subclasses of t1 etc. are replaced by the most-derived base class > among t1etc." > This defeats the very common use of constrained type variables: have a type > preserving function limited to classes inherited from a common base. E.g. > say > we have a function: > > def relocate(e: Employee) -> Employee: ... > > The function actually always returns an object of the same type as the > argument, so we want to write a more precise type. We usually do it like > this: > > XEmployee = Var('XEmployee', Employee) > > def relocate(e: XEmployee) -> XEmployee: ... > > This won't work with the definition from the proposal. > I just copied this from mypy (where it is called typevar()). I guess in that example one would use an *unconstrained* type variable. The use case for the behavior I described is AnyStr -- if I have a function like this I don't want the type checker to assume the more precise type: def space_for(s: AnyStr) -> AnyStr: if isinstance(s, str): return ' ' else: return b' ' If someone defined a class MyStr(str), we don't want the type checker to think that space_for(MyStr(...)) returns a MyStr instance, and it would be impossible for the function to even create an instance of the proper subclass of str (it can get the class object, but it can't know the constructor signature). For strings, functions like this (which return some new string of the same type as the argument, constrained to either str or bytes) are certainly common. And for your Employee example it would also seem problematic for the function to know how to construct an instance of the proper (dynamically known) subclass. b) Multiple constraints introduce an implicit Union. I'd argue that type > variables are more commonly constrained by an Intersection rather than a > Union. So it will be more useful if given this definition Y has to be > compatible with all of t1, t2 etc, rather than just one of them. > Alternatively, this can be always spelled out explicitly: > Y1 = Var('Y1', Union[t1, t2, ...]) > Y2 = Var('Y2', Intersection[t1, t2, ...]) > Well, maybe. At this point you'd have to point us to a large body of evidence -- mypy has done well so far with its current definition of typevar(). OTOH one of the ideas on the table is to add keyword options to Var(), which might make it possible to have type variables with different semantics. There are other use cases, some of which are discussed in the tracker: https://github.com/ambv/typehinting/issues/18 > Pragmatics: > > 3. The names Union and Intersection are standard terminology in type > checking, > but may not be familiar to many Python users. Names like AnyOf[] and > AllOf[] > can be more intuitive. > I strongly disagree with this. Python's predecessor, ABC, used a number of non-standard terms for common programming language concepts, for similar reasons. But the net effect was just that it looked weird to anyone familiar with other languages, and for the users who were a completely blank slate, well, "HOW-TO" was just as much jargon that they had to learn as "procedure". Also, the Python users who will most likely need to learn about this stuff are most likely library developers. > 4. Similar to allowing None to mean type(None) it's nice to have shortcuts > like: > (t1, t2, ...) == Tuple[t1, t2, ...] > [t1] == List[t1] > {t1: t2} == Dict[t1, t2] > {t1} == Set[t1] > > The last 3 can be Sequence[t1], Mapping[t1, t2] and collections.Set[t1] if > we > want to encourage the use of abstract types. > This was proposed as the primary notation during the previous round of discussions here. You are right that if we propose to "fix up" type annotations that appear together with a default value we should also be able in principle to change these shortcuts into the proper generic type objects. Yet I am hesitant to adopt the suggestion -- people may already be using e.g. dictionaries as annotations for some other purpose, and there is the question you bring up whether we should promote these to concrete or abstract collection types. Also, I should note that, while I mentioned it as a possibility, I am hesitant to endorse the shortcut of "arg: t1 = None" as a shorthand for "arg: Union[t1, None] = None" because it's unclear whether runtime introspection of the __annotations__ object should return t1 or the inferred Union object. (The unspoken requirement here is that there will be no changes to CPython's handling of annotations -- the typing.py module will be all that is needed, and it can be backported to older Python versions.) > 5. Using strings for forward references can be messy in case of generics: > parsing of brackets etc in the string will be needed. I propose explicit > forward declarations: > > C = Declare('C') > class C(Generic[X]): > def foo(self, other: C[X]): ... > def bar(self, other: C[Y]): ... > Agreed this is an area that needs more thought. In mypy you can actually write the entire annotation in string quotes -- mypy has to be able to parse type expressions anyway (in fact it has to be able to parse all of Python :-). I do think that the example you present feels rather obscure. > 6. On the other hand, using strings for unconstrained type variables is > quite > handy, and doesn't share the same problem: > > def head(xs: List['T']) -> 'T': ... > Yeah, it does look quite handy, if the ambiguity with forward references can be resolved. Also it's no big deal to have to declare a type variable -- you can reuse them for all subsequent function definitions, and you usually don't need more than two or three. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Dec 25 05:19:23 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 24 Dec 2014 20:19:23 -0800 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: <8f2e8f18-08d3-4384-87c3-e53acfa4bdcf@googlegroups.com> References: <5495F193.9000402@gmail.com> <8f2e8f18-08d3-4384-87c3-e53acfa4bdcf@googlegroups.com> Message-ID: On Wed, Dec 24, 2014 at 3:22 PM, Eugene Toder wrote: > > On Monday, December 22, 2014 2:03:28 AM UTC-5, Guido van Rossum wrote: >> >> This places Any also at the *bottom* of the class tree, if you can call >> it that. (And hence it is more a graph than a tree -- but if you remove >> Any, what's left is a tree again.) >> > I may be misunderstanding the context, but since Python has multiple > inheritance, the "class tree" was already a graph even without Any. > Thanks, that was a simple "thinko". I meant that without Any (and perhaps barring some other pathological cases due to Python's extreme malleability) it's a DAG, but with Any it has a cycle. I've also updated the Quip doc to speak of "class graph". -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Dec 25 15:43:35 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 26 Dec 2014 01:43:35 +1100 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: Message-ID: <20141225144335.GL6580@ando.pearwood.info> On Wed, Dec 24, 2014 at 08:16:52PM -0800, Guido van Rossum wrote: > On Wed, Dec 24, 2014 at 4:50 PM, Eugene Toder wrote: [...] > > Pragmatics: > > > > 3. The names Union and Intersection are standard terminology in type > > checking, but may not be familiar to many Python users. Names like > > AnyOf[] and AllOf[] can be more intuitive. > > I strongly disagree with this. Python's predecessor, ABC, used a number of > non-standard terms for common programming language concepts, for similar > reasons. But the net effect was just that it looked weird to anyone > familiar with other languages, and for the users who were a completely > blank slate, well, "HOW-TO" was just as much jargon that they had to learn > as "procedure". Also, the Python users who will most likely need to learn > about this stuff are most likely library developers. I presume that runtime name binding will be allowed, e.g. from typing import Union as AnyOf def spam(x: AnyOf[int, float])->str: ... but not encouraged. (It's not that hard to learn a few standard names like Union.) So the above would work at runtime, but at compile time, it will depend on the specific linter or type checker: it will be a "quality of implementation" issue, with simple tools possibly not being able to recognise AnyOf as being the same as Union. Is this what you have in mind? -- Steven From guido at python.org Thu Dec 25 20:42:34 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 25 Dec 2014 11:42:34 -0800 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: <20141225144335.GL6580@ando.pearwood.info> References: <20141225144335.GL6580@ando.pearwood.info> Message-ID: On Thu, Dec 25, 2014 at 6:43 AM, Steven D'Aprano wrote: > On Wed, Dec 24, 2014 at 08:16:52PM -0800, Guido van Rossum wrote: > > On Wed, Dec 24, 2014 at 4:50 PM, Eugene Toder wrote: > [...] > > > Pragmatics: > > > > > > 3. The names Union and Intersection are standard terminology in type > > > checking, but may not be familiar to many Python users. Names like > > > AnyOf[] and AllOf[] can be more intuitive. > > > > I strongly disagree with this. Python's predecessor, ABC, used a number > of > > non-standard terms for common programming language concepts, for similar > > reasons. But the net effect was just that it looked weird to anyone > > familiar with other languages, and for the users who were a completely > > blank slate, well, "HOW-TO" was just as much jargon that they had to > learn > > as "procedure". Also, the Python users who will most likely need to learn > > about this stuff are most likely library developers. > > I presume that runtime name binding will be allowed, e.g. > > from typing import Union as AnyOf > > def spam(x: AnyOf[int, float])->str: ... > > but not encouraged. (It's not that hard to learn a few standard names > like Union.) So the above would work at runtime, but at compile time, it > will depend on the specific linter or type checker: it will be a > "quality of implementation" issue, with simple tools possibly not > being able to recognise AnyOf as being the same as Union. > > Is this what you have in mind? > I would not recommend that to anyone -- I find that use of "import ... as" is often an anti-pattern or a code smell, and in this case it would seem outright silly to fight the standard library's terminology (assuming typing.py defines Union). I don't know if mypy supports this (it's easy to try it for yourself though) but I do know it follows simple global assignments, known as type aliases, e.g. "foo = Iterator[int]". -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From eltoder at gmail.com Thu Dec 25 22:49:38 2014 From: eltoder at gmail.com (Eugene Toder) Date: Thu, 25 Dec 2014 16:49:38 -0500 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: Message-ID: On Wed, Dec 24, 2014 at 11:16 PM, Guido van Rossum wrote: > No problem. :-) I apologize for reformatting the text I am quoting from you, > it looked as if it was sent through two different line clipping functions. Oh yes, sorry for that. >> b) Write the type of an overloaded function: >> >> @overload >> def foo(x: str) -> str: ... >> @overload >> def foo(x: bytes) -> bytes: ... >> >> foo # type: Intersection[Callable[[str], str], Callable[[bytes], bytes]] > > The static checker can figure that out for itself, but that doesn't mean we > necessarily need a way to spell it. It can be useful to be able to write the type, even if type checker can infer it in some cases. For example, to annotate the type of a C function that has overloading. Also, the type checker may need to print it when reporting an error. Also, to declare the type of an argument as an overloaded function. The last one is admittedly rare. If the Intersection type is implemented for the first reason this should "just work" as well. > I just copied this from mypy (where it is called typevar()). I guess in that > example one would use an *unconstrained* type variable. The use case for the > behavior I described is AnyStr -- if I have a function like this I don't > want the type checker to assume the more precise type: > > def space_for(s: AnyStr) -> AnyStr: > if isinstance(s, str): return ' ' > else: return b' ' > > If someone defined a class MyStr(str), we don't want the type checker to > think that space_for(MyStr(...)) returns a MyStr instance, and it would be > impossible for the function to even create an instance of the proper > subclass of str (it can get the class object, but it can't know the > constructor signature). > > [snip] > > OTOH one of the ideas on the table is to add keyword options to Var(), which > might make it possible to have type variables with different semantics. > There are other use cases, some of which are discussed in the tracker: > https://github.com/ambv/typehinting/issues/18 Rather https://github.com/ambv/typehinting/issues/2? Yes, keyword arguments will make the syntax easier to understand. I thought more about this, and I think I understand what you are after. The syntax confused me somewhat. I also recalled that type variables may need lower bounds in addition to upper bounds. Is AnyStr the main use case of this feature? If that's the case, there are other ways to achieve the same effect with more general features. First, some motivational examples. Say we have a class for an immutable list: class ImmutableList(Generic[X]): def prepend(self, item: X) -> ImmutableList[X]: ... The type of prepend is actually too restrictive: we should be able to add items that are superclass of X and get a list of that more general type: Y = Var('Y') >= X # must be a superclass of X class ImmutableList(Generic[X]): def prepend(self, item: Y) -> ImmutableList[Y]: ... Alternative syntax for Y, based on the Java keyword: Y = Var('Y').super(X) This will be handy to give better types to some methods of tuple and frozenset. Next, let's try to write a type for copy.copy. There are many details that can be changed, but here's a sketch. Naturally, it should be def copy(obj: X) -> X: ... But copy doesn't work for all types, so there must be some constraints on X: X = Var('X') X <= Copyable[X] # must be a subclass of Copyable[X] (Alternative syntax: X.extends(Copyable[X]); this also shows why constraints are not listed in the constructor.) Copyable is a protocol: @protocol class Copyable(Generic[X]): def __copy__(self: X) -> X: ... And for some built-in types: Copyable.register(int) Copyable.register(str) ... This approach can be used to type functions that special-case built-in types, and rely on some known methods for everything else. In my example with XEmployee the function could either return its argument, or make a copy -- the Employee class can require all its subclasses to implement some copying protocol (e.g. a copy() method). In fact, since the longest() function from your document always returns one of its arguments, its type can be written as: X = Var('X') <= Union[str, bytes] def longest(a: X, b: X) -> X: ... that is, it doesn't need to restrict the return type to str or bytes :-) Finally, the feature from your document: AnyStr = Var('AnyStr').restrictTo(str, bytes) # the only possible values However, this can be achieved by adding more useful features to protocols: # explicit_protocol is a non-structural protocol: only explicitly registered # types are considered conforming. This is very close to type classes. # Alternatively, protocols with no methods can always be explicit. @explicit_protocol class StringLike(Generic[X]): # This type can be referenced like a class-level attribute. # The name "type" is not special in any way. type = X StringLike.register(str) StringLike.register(bytes) AnyStr = Var('AnyStr') AnyStr <= StringLike[AnyStr] AnyStrRet = StringLike[AnyStr].type def space_for(x: AnyStr) -> AnyStrRet: ... There are many details that can be tweaked, but it is quite powerful, and solves the simpler problem as well. > I strongly disagree with this. Python's predecessor, ABC, used a number of > non-standard terms for common programming language concepts, for similar > reasons. But the net effect was just that it looked weird to anyone familiar > with other languages, and for the users who were a completely blank slate, > well, "HOW-TO" was just as much jargon that they had to learn as > "procedure". Also, the Python users who will most likely need to learn about > this stuff are most likely library developers. Very good point. I should clarify that I don't suggest to change the terminology. I'm only talking about the syntax. The majority of the languages that use union type seem to use t1|t2 syntax for it. AFAIU this syntax was rejected to avoid changes in CPython. This is a shame, because it is widespread and reads really well: def foo(x: Some|Another): ... Also, Type|None is so short and clear that there's no need for the special Optional[] shorthand. Given we won't use |, I think def foo(x: AnyOf[Some, Another]): ... reads better than def foo(x: Union[Some, Another]): ... but this may be getting into the bikeshedding territory :-) > This was proposed as the primary notation during the previous round of > discussions here. You are right that if we propose to "fix up" type > annotations that appear together with a default value we should also be able > in principle to change these shortcuts into the proper generic type objects. > Yet I am hesitant to adopt the suggestion -- people may already be using > e.g. dictionaries as annotations for some other purpose, and there is the > question you bring up whether we should promote these to concrete or > abstract collection types. I have some experience using this notation internally, and it worked quite well. To be specific, I do not suggest for Python to automatically convert these annotations to proper generic types. This should be done internally in the type checker. If we want other tools to understand this syntax, we can expose functions typing.isTypeAnnotation(obj) and typing.canonicalTypeAnnotation(obj). With this approach, I don't believe this use of lists and dicts adds any more problems for the existing uses of annotations. The decision of whether to use concrete or abstract types is likely not a hard one. Given my experience, I'd use concrete types, because they are so common. But this does depend on the bigger context of how annotations are expected to be used. > Also, I should note that, while I mentioned it as a possibility, I am > hesitant to endorse the shortcut of "arg: t1 = None" as a shorthand for > "arg: Union[t1, None] = None" because it's unclear whether runtime > introspection of the __annotations__ object should return t1 or the inferred > Union object. FWIW, I'm -0.5 on this, but if this is implemented, I think __annotations__ should return t1, and typing.canonicalTypeAnnotation accept an optional second argument for the default value. > Agreed this is an area that needs more thought. In mypy you can actually > write the entire annotation in string quotes -- mypy has to be able to parse > type expressions anyway (in fact it has to be able to parse all of Python > :-). I do think that the example you present feels rather obscure. Is the intention to keep this -- i.e. require the type checker to potentially parse strings as Python code? Complex expressions in strings feel a bit like a hack :-) The code of this kind comes up regularly in containers, like the ImmutableList above. Some standard types have methods likes this as well: class Set(Generic[X]): def union(self, other: Set[X]) -> Set[X]: ... > Yeah, it does look quite handy, if the ambiguity with forward references can > be resolved. Also it's no big deal to have to declare a type variable -- you > can reuse them for all subsequent function definitions, and you usually > don't need more than two or three. Agreed. Having to declare a forward reference or a type variable both seem to add very little burden, and are possibly rare enough. I'm not sure which one should get a shorter syntax. While on the subject: what are the scoping rules for type variables? I hope they are lexically scoped: the names used in the enclosing class or function are considered bound to those values, rather than fresh variables that shadow them. I used this fact in the examples above. E.g. union() above accepts only the sets with the same elements, not with any elements, and in def foo(x: X) -> X: def bar(y: X) -> X: return y return bar(x) X in bar() must be the same type as in foo(). Eugene From guido at python.org Fri Dec 26 04:41:49 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 25 Dec 2014 19:41:49 -0800 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: Message-ID: On Thu, Dec 25, 2014 at 1:49 PM, Eugene Toder wrote: > On Wed, Dec 24, 2014 at 11:16 PM, Guido van Rossum > wrote: > [Eugene] > >> b) Write the type of an overloaded function: > >> > >> @overload > >> def foo(x: str) -> str: ... > >> @overload > >> def foo(x: bytes) -> bytes: ... > >> > >> foo # type: Intersection[Callable[[str], str], Callable[[bytes], bytes]] > > > > The static checker can figure that out for itself, but that doesn't mean > we > > necessarily need a way to spell it. > It can be useful to be able to write the type, even if type checker can > infer > it in some cases. For example, to annotate the type of a C function that > has > overloading. > mypy solves that using @overload in a stub file. That's often more precise. > Also, the type checker may need to print it when reporting an > error. Also, to declare the type of an argument as an overloaded function. > The last one is admittedly rare. > If the Intersection type is implemented for the first reason this should > "just > work" as well. > Hm, looks like the case for Intersection is still pretty weak. Anyway, we can always add stuff later. But whatever we add in 3.5 we cannot easily take back. > > I just copied this from mypy (where it is called typevar()). I guess in > that > > example one would use an *unconstrained* type variable. The use case for > the > > behavior I described is AnyStr -- if I have a function like this I don't > > want the type checker to assume the more precise type: > > > > def space_for(s: AnyStr) -> AnyStr: > > if isinstance(s, str): return ' ' > > else: return b' ' > > > > If someone defined a class MyStr(str), we don't want the type checker to > > think that space_for(MyStr(...)) returns a MyStr instance, and it would > be > > impossible for the function to even create an instance of the proper > > subclass of str (it can get the class object, but it can't know the > > constructor signature). > > > > [snip] > > > > OTOH one of the ideas on the table is to add keyword options to Var(), > which > > might make it possible to have type variables with different semantics. > > There are other use cases, some of which are discussed in the tracker: > > https://github.com/ambv/typehinting/issues/18 > Rather https://github.com/ambv/typehinting/issues/2? Yes, keyword > arguments > will make the syntax easier to understand. > Yes, that's the issue I meant. > I thought more about this, and I think I understand what you are after. The > syntax confused me somewhat. I also recalled that type variables may need > lower bounds in addition to upper bounds. Is AnyStr the main use case of > this > feature? If that's the case, there are other ways to achieve the same > effect > with more general features. > I don't know if this is the main use case (we should ask Jukka when he's back from vacation). I'm hesitant to propose more general features without at least one implementation. Perhaps you could try to see how easy those more general features would be implementable in mypy? > First, some motivational examples. Say we have a class for an immutable > list: > > class ImmutableList(Generic[X]): > def prepend(self, item: X) -> ImmutableList[X]: ... > > The type of prepend is actually too restrictive: we should be able to add > items that are superclass of X and get a list of that more general type: > > Y = Var('Y') >= X # must be a superclass of X > > class ImmutableList(Generic[X]): > def prepend(self, item: Y) -> ImmutableList[Y]: ... > > Alternative syntax for Y, based on the Java keyword: > > Y = Var('Y').super(X) > Neither syntax is acceptable to me, but let's assume we can do this with some other syntax. Your example still feels like it was carefully constructed to prove your point -- it would make sense in a language where everything is type-checked and types are the basis for everything, and users are eager to push the type system to its limits. But I'm carefully trying to avoid moving Python in that direction. > This will be handy to give better types to some methods of tuple and > frozenset. > I assume you're talking about the case where e.g. I have a frozenset of Managers and I use '+' to add an Employee; we then know that the result is a frozenset of Employees. But if we assume covariance, that frozenset of Managers is also a frozenset of Employees, so (assuming we have a way to indicate covariance) the type-checker should be able to figure this out. Or are you perhaps trying to come up with a way to spell covariance? (The issue #2 above has tons of discussion about that, although I don't think it comes to a clear conclusion.) > Next, let's try to write a type for copy.copy. > Eek. That sounds like a bad idea -- copy.copy() uses introspection and I don't think there's much hope to be able to spell its type. (Also I usually consider the use of copy.copy() a code smell. Perhaps there's a connection. :-) > There are many details that can > be changed, but here's a sketch. Naturally, it should be > > def copy(obj: X) -> X: ... > > But copy doesn't work for all types, so there must be some constraints on > X: > > X = Var('X') > X <= Copyable[X] # must be a subclass of Copyable[X] > > (Alternative syntax: X.extends(Copyable[X]); this also shows why > constraints > are not listed in the constructor.) Copyable is a protocol: > > @protocol > class Copyable(Generic[X]): > def __copy__(self: X) -> X: ... > > And for some built-in types: > > Copyable.register(int) > Copyable.register(str) > ... > > This approach can be used to type functions that special-case built-in > types, > and rely on some known methods for everything else. > Sorry, I'm not sold on this. I also worry that the register() calls are hard to track for a type checker -- but that's minor (I actually don't know if this would be a problem for mypy). I just don't see the point in trying to create a type system powerful enough to describe copy.copy(). > In my example with XEmployee the function could either return its argument, > or make a copy -- the Employee class can require all its subclasses to > implement some copying protocol (e.g. a copy() method). > That sounds like an artificial requirement on the implementation designed to help the type checker. I'm inclined to draw the line well before that point. (Otherwise Raymond Hettinger would throw a fit. :-) > In fact, since the > longest() function from your document always returns one of its arguments, > But that was just the shortest way to write such an example. The realistic examples (e.g. URL parsing or construction) aren't that simple. > its type can be written as: > > X = Var('X') <= Union[str, bytes] > > def longest(a: X, b: X) -> X: ... > > that is, it doesn't need to restrict the return type to str or bytes :-) > Now you're just wasting my time. :-) > Finally, the feature from your document: > > AnyStr = Var('AnyStr').restrictTo(str, bytes) # the only possible values > > However, this can be achieved by adding more useful features to protocols: > > # explicit_protocol is a non-structural protocol: only explicitly > registered > # types are considered conforming. This is very close to type classes. > # Alternatively, protocols with no methods can always be explicit. > @explicit_protocol > class StringLike(Generic[X]): > # This type can be referenced like a class-level attribute. > # The name "type" is not special in any way. > type = X > > StringLike.register(str) > StringLike.register(bytes) > > AnyStr = Var('AnyStr') > AnyStr <= StringLike[AnyStr] > AnyStrRet = StringLike[AnyStr].type > > def space_for(x: AnyStr) -> AnyStrRet: ... > > There are many details that can be tweaked, but it is quite powerful, and > solves the simpler problem as well. > I'm afraid you've lost me. But (as you may have noticed) I'm not really the one you should be convincing -- if you can convince Jukka to (let you) add something like this to mypy you may have a better case. Even so, I want to limit the complexity of what we add to Python 3.5 -- TBH basic generic types are already pushing the limits. I would much rather be asked to add more stuff to 3.6 than to find out that we've added so much to 3.5 that people can't follow along. Peter Norvig mentioned that the subtleties of co/contra-variance of generic types in Java were too complex for his daughter, and also reminded me that Josh Bloch has said somewhere that he believed they made it too complex. > > I strongly disagree with this. Python's predecessor, ABC, used a number > of > > non-standard terms for common programming language concepts, for similar > > reasons. But the net effect was just that it looked weird to anyone > familiar > > with other languages, and for the users who were a completely blank > slate, > > well, "HOW-TO" was just as much jargon that they had to learn as > > "procedure". Also, the Python users who will most likely need to learn > about > > this stuff are most likely library developers. > Very good point. I should clarify that I don't suggest to change the > terminology. I'm only talking about the syntax. The majority of the > languages > that use union type seem to use t1|t2 syntax for it. AFAIU this syntax was > rejected to avoid changes in CPython. This is a shame, because it is > widespread and reads really well: > > def foo(x: Some|Another): ... > Yes, but we're not going to change it, and it will be fine. > Also, Type|None is so short and clear that there's no need for the special > Optional[] shorthand. Given we won't use |, I think > > def foo(x: AnyOf[Some, Another]): ... > > reads better than > > def foo(x: Union[Some, Another]): ... > > but this may be getting into the bikeshedding territory :-) > Right. :-) > > This was proposed as the primary notation during the previous round of > > discussions here. You are right that if we propose to "fix up" type > > annotations that appear together with a default value we should also be > able > > in principle to change these shortcuts into the proper generic type > objects. > > Yet I am hesitant to adopt the suggestion -- people may already be using > > e.g. dictionaries as annotations for some other purpose, and there is the > > question you bring up whether we should promote these to concrete or > > abstract collection types. > I have some experience using this notation internally, and it worked quite > well. To be specific, I do not suggest for Python to automatically convert > these annotations to proper generic types. This should be done internally > in > the type checker. If we want other tools to understand this syntax, we can > expose functions typing.isTypeAnnotation(obj) and > typing.canonicalTypeAnnotation(obj). With this approach, I don't believe > this > use of lists and dicts adds any more problems for the existing uses of > annotations. > But I can see a serious downside as well. There will likely be multiple tools that have to be able to read the type hinting annotations, e.g. IDEs may want to use the type hints (possibly from stub files) for code completion purposes. Also someone might want to write a decorator that extracts the annotations and asserts that arguments match at run time. The more handy shorthands we invent, the more complex all such tools will have to be. > The decision of whether to use concrete or abstract types is likely not a > hard one. Given my experience, I'd use concrete types, because they are so > common. But this does depend on the bigger context of how annotations are > expected to be used. > That's how I'm leaning as well. > > Also, I should note that, while I mentioned it as a possibility, I am > > hesitant to endorse the shortcut of "arg: t1 = None" as a shorthand for > > "arg: Union[t1, None] = None" because it's unclear whether runtime > > introspection of the __annotations__ object should return t1 or the > inferred > > Union object. > FWIW, I'm -0.5 on this, but if this is implemented, I think __annotations__ > should return t1, and typing.canonicalTypeAnnotation accept an optional > second > argument for the default value. > You may just have killed the idea. Let's keep it simpler. > Agreed this is an area that needs more thought. In mypy you can actually > > write the entire annotation in string quotes -- mypy has to be able to > parse > > type expressions anyway (in fact it has to be able to parse all of Python > > :-). I do think that the example you present feels rather obscure. > Is the intention to keep this -- i.e. require the type checker to > potentially > parse strings as Python code? Complex expressions in strings feel a bit > like > a hack :-) > I know. :-) > The code of this kind comes up regularly in containers, like the > ImmutableList > above. Some standard types have methods likes this as well: > > class Set(Generic[X]): > def union(self, other: Set[X]) -> Set[X]: ... > How complex does it really have to be? Perhaps Name[Name, Name, ...] is the only form (besides a plain Name) that we really need? Anything more complex can probably be reduced using type aliases. Then again my earlier argument is clearly for keeping things simple, and perhaps an explicit forward declaration is simpler. The run-time representation would still be somewhat problematic. I'll try to remember to report back once I have tried to implement this. > > Yeah, it does look quite handy, if the ambiguity with forward references > can > > be resolved. Also it's no big deal to have to declare a type variable -- > you > > can reuse them for all subsequent function definitions, and you usually > > don't need more than two or three. > Agreed. Having to declare a forward reference or a type variable both seem > to > add very little burden, and are possibly rare enough. I'm not sure which > one > should get a shorter syntax. > I don't think it's quite a toss-up. A type variable is a special feature. But a forward reference is not much different from a backward reference -- you could easily imagine a language (e.g. C++ :-) where forward references don't require special syntax. The rule that 'X' means the same as X but is evaluated later is pretty simple, whereas the rule the 'X' introduces a type variable is pretty complex. So even if we *didn't* use string quotes for forward references I still wouldn't want to use that syntax for type variables. > While on the subject: what are the scoping rules for type variables? I hope > they are lexically scoped: the names used in the enclosing class or > function > are considered bound to those values, rather than fresh variables that > shadow > them. I used this fact in the examples above. E.g. union() above accepts > only > the sets with the same elements, not with any elements, and in > > def foo(x: X) -> X: > def bar(y: X) -> X: return y > return bar(x) > > X in bar() must be the same type as in foo(). > Why don't you install mypy and check for yourself? (I expect it's as you desire, but while I have mypy installed, I'm on vacation and my family is asking for my attention.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From fetchinson at googlemail.com Fri Dec 26 11:34:35 2014 From: fetchinson at googlemail.com (Fetchinson .) Date: Fri, 26 Dec 2014 11:34:35 +0100 Subject: [Python-ideas] python on mobile Message-ID: Hi all, maybe the best list would be python-dev but I don't dare waking the sleeping lions there :) So my question is this: is there a coherent mobile strategy among core dev people? I mean you guys release python for linux/macos/windows and the question is if there are any plans to do the same for a mobile platform. It doesn't have to be android or ios just anything that the core dev team chooses and sticks with. I've been developing python apps for smartphones (mostly hobby projects though) using sl4a but that seems like is dead. Now people suggest me using kivy which seems to be alive but who knows how long. There are some other projects qpython, etc, which are small and equally not so reliable at least looking from the outside. Is kivy now an officially blessed distribution? Since google was so integral to both python (through employing Guido) and android I'd think it would make sense for google to have an official python environment for android in cooperation with the python dev team. Does the PSF has an opinion on this? It would be great if there would be something for mobile phones that we could rely on not going away just as with linux/macos/windows. Or there are some issues which preclude this from the start? Cheers, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From sturla.molden at gmail.com Fri Dec 26 15:31:47 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 26 Dec 2014 15:31:47 +0100 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: Some thoughs on this: Kivy is based on PyGame, which is based on SDL (libsdl.org), PyOpenGL and NumPy. SDL is not going away and supports iOS and Android as well as most deskop OSes. PyOpenGL and NumPy are not going away either. Personally I would consider to use SDL and OpenGL directly from Cython, unless you really need Kivy. A mobile has limited resources so you should be careful about stacking up several layers of software. You can use Objective C APIs directly in Python with PyObjC, and thereby create "native" iOS apps. PyQt now supports Qt5 and has official iOS and Android support. You might also try your luck with PySide. Beware of the licensing issues. There is something called HTML5. I don't know much about it, but presumably you can use WebKit Framework in yout iOS Python app to make an HTML5 based UI. Mozilla is now coming to iOS and Android. Using Python and XUL might also be possible. There are apps in Apple Appstore based on Python co clearly it can be done. As I understand the rules now the app bundle must be self-contained. I.e. you must include the Python interpreter and all libraries in the bundle. Also your app cannot download executable code (e.g. python modules) from the internet. The used to be a ban on interpreted code on iOS, but I think that was mostly to fight Adobe Flash and Java mobile edition. The requirement now is a self-contained app bundle. Sturla On 26/12/14 11:34, Fetchinson . wrote: > Hi all, maybe the best list would be python-dev but I don't dare > waking the sleeping lions there :) > > So my question is this: is there a coherent mobile strategy among core > dev people? I mean you guys release python for linux/macos/windows and > the question is if there are any plans to do the same for a mobile > platform. It doesn't have to be android or ios just anything that the > core dev team chooses and sticks with. > > I've been developing python apps for smartphones (mostly hobby > projects though) using sl4a but that seems like is dead. Now people > suggest me using kivy which seems to be alive but who knows how long. > There are some other projects qpython, etc, which are small and > equally not so reliable at least looking from the outside. Is kivy now > an officially blessed distribution? Since google was so integral to > both python (through employing Guido) and android I'd think it would > make sense for google to have an official python environment for > android in cooperation with the python dev team. > > Does the PSF has an opinion on this? It would be great if there would > be something for mobile phones that we could rely on not going away > just as with linux/macos/windows. > > Or there are some issues which preclude this from the start? > > Cheers, > Daniel > > From stephen at xemacs.org Fri Dec 26 15:49:49 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 26 Dec 2014 23:49:49 +0900 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: <878uhu4gaa.fsf@uwakimon.sk.tsukuba.ac.jp> Fetchinson . writes: > Hi all, maybe the best list would be python-dev No, this is the right place if you eventually want to get Python-Dev involved. It's not obvious that's the right thing to do, though, since that list's primary development environment (for the interpreter) is C. Android might work better with a Jython derivative, and I don't know what would be the right environment to develop for iOS. So perhaps mobile platforms would develop as independent implementations of Python-the-Language like Jython or IronPython, rather than as a branch of CPython. Also, as Sturla points out, it might be better to use something like Cython. > So my question is this: is there a coherent mobile strategy among > core dev people? Not yet. Until recently (say a couple years ago), Apple officially forbid use of non-blessed languages on iDevices. Android was a little more friendly in principle but there are stringent restrictions for security reasons that require rethinking concepts that involve interactions with the OS. These mobile ports hardly get mentioned on python-dev or on this list. > Or there are some issues which preclude this from the start? I think there are a lot of people who would like to see something solid. But AFAICS the problem is more the lack of interest (ie, no promise that something that works today will work tomorrow) from the OS vendors than the lack of interest from core, plus the resources constraints Sturla mentions, which I suppose would make programming Python on a mobile device less fun. That would also help explain the proliferation of different implementations, as each would have its own approach to working around the resource restrictions. So I would say there's no "preclusion", but it's an uphill road for now. From abarnert at yahoo.com Fri Dec 26 16:26:49 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 26 Dec 2014 16:26:49 +0100 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: Message-ID: On Dec 25, 2014, at 22:49, Eugene Toder wrote: > The code of this kind comes up regularly in containers, like the ImmutableList > above. Some standard types have methods likes this as well: > > class Set(Generic[X]): > def union(self, other: Set[X]) -> Set[X]: ... I'm not sure this problem exists. The builtin set (and therefore the undocumented MyPy/typing TypeAlias Set) had a union method, but its signature is not that restrictive. It takes 1 or more arbitrary iterables of any element type, and of course it returns a set whose element type is the union of the element types of self and those. And the same is true in general for all of the builtin abstract and concrete types. So, the fact that Guido's/Jukka's proposal doesn't make it easy to define types that are more restrictive than you'd normally want to use in Python doesn't seem to be a problem. Sure, if you wanted to define more restricted C++/Swift/Haskell style collections for Python your want to be able to type them as easily as in those languages... But why do you want to define those collections? The _opposite_ problem--that it's hard to define the _actual_ type of set.union or similarly highly parameterized types--may be more serious, but Guido acknowledged that one long ago, and I think he's right that it seems like the kind of thing that could be added later. (In the initial MyPy for 3.5 you can always define a one-argument version set[X].union(Iterable[Y])->set[Union[X, Y] and a generic multi-argument overload that, say, treats all the Iterable[Any] and returns Set[Any]. If we turn out to need parameter schemas for varargs in real programs, that can surely be added in 3.6 as easily as it could now. And hopefully it won't be needed. You need some kind of complete language to write such schemas in, and C++11 is a nice warning of what it looks like to try to do that declaratively in a non-declarative language.) From abarnert at yahoo.com Fri Dec 26 17:01:17 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 26 Dec 2014 17:01:17 +0100 Subject: [Python-ideas] python on mobile In-Reply-To: <878uhu4gaa.fsf@uwakimon.sk.tsukuba.ac.jp> References: <878uhu4gaa.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <68B9C47C-1CA8-417D-A576-D7AE7C83AA3A@yahoo.com> On Dec 26, 2014, at 15:49, "Stephen J. Turnbull" wrote: > Fetchinson . writes: > >> Hi all, maybe the best list would be python-dev > > No, this is the right place if you eventually want to get Python-Dev > involved. It's not obvious that's the right thing to do, though, > since that list's primary development environment (for the > interpreter) is C. Android might work better with a Jython > derivative, and I don't know what would be the right environment to > develop for iOS. For iOS, it's probably CPython or a CPython derivative. (The whole point of ObjC is that yet includes C and can run C libraries like libpython natively, and the platform is a Unix+Cocoa platform very similar to OS X.) > So perhaps mobile platforms would develop as > independent implementations of Python-the-Language like Jython or > IronPython, rather than as a branch of CPython. > > Also, as Sturla points out, it might be better to use something like > Cython. > >> So my question is this: is there a coherent mobile strategy among >> core dev people? > > Not yet. Until recently (say a couple years ago), Apple officially > forbid use of non-blessed languages on iDevices. Android was a little > more friendly in principle but there are stringent restrictions for > security reasons that require rethinking concepts that involve > interactions with the OS. These mobile ports hardly get mentioned on > python-dev or on this list. > >> Or there are some issues which preclude this from the start? > > I think there are a lot of people who would like to see something > solid. But AFAICS the problem is more the lack of interest (ie, no > promise that something that works today will work tomorrow) from the > OS vendors than the lack of interest from core, plus the resources > constraints Sturla mentions, which I suppose would make programming > Python on a mobile device less fun. That would also help explain the > proliferation of different implementations, as each would have its own > approach to working around the resource restrictions. > > So I would say there's no "preclusion", but it's an uphill road for > now. Cross-compiling CPython itself, and most of the stdlib, is dead easy, at least for iOS. (I'll ignore Android, and other mobile platforms, but at least some things are similar.) And building an ObjC app that embeds CPython is also pretty easy. So for that part, there's very little for the core devs to do beyond listing it as a standard target and adding tests for it. CPython could I suppose also have an official (Mac) installer for a cross-compiled Python for iOS, which would also be pretty easy if anyone wanted it. But I think the problem is that nobody wants this, except for a small community of people 98% of whom have no problem building Python themselves. A standard iOS GUI might be nice, but CPython includes only Tkinter, which doesn't work on iOS and almost certainly never will, and I don't think you'd want to use it anyway. I think incorporating a different GUI framework is way out of scope for the core distribution--or, if not, it should be done for desktop first. (It would be silly to have PyQt for iOS but not for Linux.) Building an app bundle for iOS is painful on iOS, but it's just as painful on Windows and OS X, and even more so on Linux, which is why people use third-party tools like py2app and pyInstaller. Again, if those tools aren't in scope for the core distribution, building similar tools for mobile is even less so. Finally, cross-compiling C extensions had a few frustrations. Some of this is just bugs, which people will continue to find, file, and fix. But the big problem is that there isn't some standard set of steps that works for every package to use your native Python to build extensions for a cross-compiled Python and install them into its stdlib. When you compare how simple that is for native Python (pip install spam), there's clearly a lot of room for improvement. But I suspect the packaging SIG rather than -ideas is the place to get started there. Or maybe even come up with a design for a setuptools-cross library and pip-cross tool and put up a prototype first, and the try to get it standardized? Still, I don't think that'll relieve the need for things like Kivy. If you're looking to build an app for iOS, even with a standard installation, a pip-cross, a py2app-iOS, and cross wheels for, say, a statically-linked PyQt, anyone who isn't already Qt-savvy or looking to port a desktop (or Qtopia) app is likely to be happier with something like Kivy in multiple ways. Which is fine. But that implies that maybe the most important thing for the core devs to do is to help Kivy and similar teams with whatever blocking frustrations they have due to core CPython limitations, rather than trying to build something from the ground up. From solipsis at pitrou.net Fri Dec 26 18:13:59 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 26 Dec 2014 18:13:59 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? References: Message-ID: <20141226181359.3b8ac096@fsol> On Wed, 24 Dec 2014 13:04:01 -0600 Ron Adam wrote: > > My thoughts is that making python easier to multi-process on multi-core > CPUs will be where the biggest performance gains will be. Think of 100 > core chips in as soon as 5 or 6 years. Won't happen on mainstream computers (laptop/desktop/tablet/smartphone), as it's a totally silly thing to do there. Regards Antoine. From eltoder at gmail.com Fri Dec 26 18:59:28 2014 From: eltoder at gmail.com (Eugene Toder) Date: Fri, 26 Dec 2014 12:59:28 -0500 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: Message-ID: On Fri, Dec 26, 2014 at 10:26 AM, Andrew Barnert wrote: > The builtin set (and therefore the undocumented MyPy/typing TypeAlias Set) > had a union method, but its signature is not that restrictive. It takes 1 > or more arbitrary iterables of any element type, and of course it returns > a set whose element type is the union of the element types of self and > those. And the same is true in general for all of the builtin abstract and > concrete types. You are right. The union type solves the problem, and is a more precise type than with a lower bound. So we don't need lower bounds on type variables for collection methods, and maybe at all. > The _opposite_ problem--that it's hard to define the _actual_ type of > set.union or similarly highly parameterized types--may be more serious, Why is it hard? Isn't the actual type just: def union(self, *others: Iterable[Y]) -> Set[Union[X, Y]] where typing of vararg is similar to Java -- all elements must conform to the single type annotation. Also note that I posted set.union method as an example that needs a forward reference to a generic class. I was arguing that if we use strings for forward references, we'll eventually have complicated expressions in those strings, not just class names: class Set(Generic[X]): # Note that the name "Set" is not yet available, so we have to use # a forward reference. This puts the whole return type inside a string. def union(self, *others: Iterable[Y]) -> "Set[Union[X, Y]]": ... Your type for set.union seems to prove the point even better than what I used. Eugene From eltoder at gmail.com Fri Dec 26 19:22:41 2014 From: eltoder at gmail.com (Eugene Toder) Date: Fri, 26 Dec 2014 13:22:41 -0500 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: Message-ID: On Fri, Dec 26, 2014 at 12:59 PM, Eugene Toder wrote: > Why is it hard? Isn't the actual type just: > > def union(self, *others: Iterable[Y]) -> Set[Union[X, Y]] > > where typing of vararg is similar to Java -- all elements must conform to the > single type annotation. To be even more precise, both X and Y should conform to a Hashable protocol, rather than being unconstrained. So this is also a simple example of why constraints on type variables are useful. Eugene From abarnert at yahoo.com Fri Dec 26 20:15:50 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 26 Dec 2014 20:15:50 +0100 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: Message-ID: On Dec 26, 2014, at 18:59, Eugene Toder wrote: > On Fri, Dec 26, 2014 at 10:26 AM, Andrew Barnert wrote: >> The builtin set (and therefore the undocumented MyPy/typing TypeAlias Set) >> had a union method, but its signature is not that restrictive. It takes 1 >> or more arbitrary iterables of any element type, and of course it returns >> a set whose element type is the union of the element types of self and >> those. And the same is true in general for all of the builtin abstract and >> concrete types. > You are right. The union type solves the problem, and is a more precise type > than with a lower bound. So we don't need lower bounds on type variables for > collection methods, and maybe at all. > >> The _opposite_ problem--that it's hard to define the _actual_ type of >> set.union or similarly highly parameterized types--may be more serious, > > Why is it hard? Isn't the actual type just: > > def union(self, *others: Iterable[Y]) -> Set[Union[X, Y]] > > where typing of vararg is similar to Java -- all elements must conform to the > single type annotation. I'm not sure you can just skip over that last point without addressing it. In this case, given iterables of element types Y1, Y2, ..., Yn, you can say that they're all type Iterable[Union[Y1, Y2, ..., Yn]]. I _think_ an inference engine can find that Union type pretty easily, and I _think_ that at least for collection methods there won't be any harder problems--but I wouldn't just assume either of those without looking carefully. And it certainly isn't true when we go past collection methods--clearly map and zip can't be handled this way. > Also note that I posted set.union method as an example that needs a forward > reference to a generic class. I was arguing that if we use strings for forward > references, we'll eventually have complicated expressions in those strings, > not just class names: > > class Set(Generic[X]): > # Note that the name "Set" is not yet available, so we have to use > # a forward reference. This puts the whole return type inside a string. > def union(self, *others: Iterable[Y]) -> "Set[Union[X, Y]]": ... > > Your type for set.union seems to prove the point even better than what I used. Another way to write this, assuming that Set[X][Y] means Set[Y] or that there's some syntax to get from Set[X] to Set, would be to use a typeof(self) operator. Or a special magic __class_being_defined__ constant instead of an operator, or the normal type function with a slightly different meaning at compile time than runtime, or probably other ways to bikeshed it. The point is, at least this example only really needs the type of self, not an arbitrary forward declaration or an expression that has to be crammed into a string. Are there any good examples where that isn't true? Also, should Set.union be contravariant in the generic type Set, or is it always going to return a Set[Something]? The two options there could both easily be handled by type expressions, or maybe with explicit forward declarations, but with implicit forward declaration via string? I know Guido doesn't want to start allowing arbitrary expressions, but a compile-time typeof operator is a pretty simple special case; even pre-ISO C++ had that. From eltoder at gmail.com Fri Dec 26 21:00:05 2014 From: eltoder at gmail.com (Eugene Toder) Date: Fri, 26 Dec 2014 15:00:05 -0500 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: Message-ID: On Thu, Dec 25, 2014 at 10:41 PM, Guido van Rossum wrote: > I don't know if this is the main use case (we should ask Jukka when he's > back from vacation). I'm hesitant to propose more general features without > at least one implementation. Perhaps you could try to see how easy those > more general features would be implementable in mypy? Sounds like a good idea. I'll probably start with a simpler feature, though. > Neither syntax is acceptable to me, but let's assume we can do this with > some other syntax. Your example still feels like it was carefully > constructed to prove your point -- it would make sense in a language where > everything is type-checked and types are the basis for everything, and users > are eager to push the type system to its limits. Andrew pointed below that this is better solved with a union type. > I assume you're talking about the case where e.g. I have a frozenset of > Managers and I use '+' to add an Employee; we then know that the result is a > frozenset of Employees. But if we assume covariance, that frozenset of > Managers is also a frozenset of Employees, so (assuming we have a way to > indicate covariance) the type-checker should be able to figure this out. Or > are you perhaps trying to come up with a way to spell covariance? Just a quick note: typically, covariance will not help here, because of how type variables work. Once you have Set[X] with X fixed to Manager, X stays as Manager. A fresh variable is not inserted into every use of that X. So it will not get adjusted to Employee in the call to +. OTOH, covariance helps in the opposite direction -- if Set[Employee] is needed, Set[Manager] will work too. The real Set cannot be covariant, though, because it supports mutation. > Eek. That sounds like a bad idea -- copy.copy() uses introspection and I > don't think there's much hope to be able to spell its type. Does it really use more introspection than your space_for() function? Naively, copy is implemented like: def copy(obj): if isinstance(obj, int): return obj if isinstance(obj, list): return list(obj) ... return obj.__copy__() This does not seem very hard to type. There are much simpler examples, though: a) Keys of Dict and elements of Set must be Hashable, b) To use list.index() list elements must be Comparable, c) Arguments to min() and max() must be Ordered, d) Arguments to sum() must be Addable. So it's not uncommon to have generic functions that need restrictions on type variables. > That sounds like an artificial requirement on the implementation designed to > help the type checker. Perhaps I'm not explaining this right. The requirement is not for the type checker, it is to be able to implement the function. As you pointed out earlier, without any requirements on the type the function will not be able to produce the value. > Peter Norvig mentioned that the subtleties of co/contra-variance of generic > types in Java were too complex for his daughter, and also reminded me that > Josh Bloch has said somewhere that he believed they made it too complex. IIUC the context for both statements is specifically the call site annotations of variance, i.e. and . Though you can also quote Gilad Bracha, who declared that variance is too complicated notion for programmers to understand, and made all generics in Dart covariant. This was also the case in Beta, whose authors denounced invariance and contravariance, as coming from people "with type-checking background" :-) > But I can see a serious downside as well. There will likely be multiple > tools that have to be able to read the type hinting annotations, e.g. IDEs > may want to use the type hints (possibly from stub files) for code > completion purposes. Also someone might want to write a decorator that > extracts the annotations and asserts that arguments match at run time. The > more handy shorthands we invent, the more complex all such tools will have > to be. I think if these tools cannot use functions from the typing module they will have bigger problems than shorthands. And the number of shorthands is pretty limited -- there are only as many literals in Python. > How complex does it really have to be? Perhaps Name[Name, Name, ...] is the > only form (besides a plain Name) that we really need? Andrew's type of set.union is Set[Union[X, Y]], so nested names are possible. Yes, one can always declare an alias for Union[X, Y], but at that point using a forward declaration seems cleaner. > Why don't you install mypy and check for yourself? (I expect it's as you > desire, but while I have mypy installed, I'm on vacation and my family is > asking for my attention.) Will do. Thank you for your replies! Eugene From ron3200 at gmail.com Fri Dec 26 21:03:42 2014 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 26 Dec 2014 14:03:42 -0600 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: <20141226181359.3b8ac096@fsol> References: <20141226181359.3b8ac096@fsol> Message-ID: On 12/26/2014 11:13 AM, Antoine Pitrou wrote: > On Wed, 24 Dec 2014 13:04:01 -0600 > Ron Adam wrote: >> > >> >My thoughts is that making python easier to multi-process on multi-core >> >CPUs will be where the biggest performance gains will be. Think of 100 >> >core chips in as soon as 5 or 6 years. > Won't happen on mainstream computers > (laptop/desktop/tablet/smartphone), as it's a totally silly thing to > do there. Which is silly?, 100 cores, or making python easier to multi-process? The 5 or 6 years figure is my optimistic expectation for high end workstations and servers. Double that time for typical desktop, and maybe triple that for wearable devices. Currently you can get 8 core high end desktop systems, and up to 18 core work stations with windows 8. They probably run python too. I think the unknown is how much time it will take, not weather or not it will happen. Cheers, Ron From mertz at gnosis.cx Fri Dec 26 21:11:19 2014 From: mertz at gnosis.cx (David Mertz) Date: Fri, 26 Dec 2014 13:11:19 -0700 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: <20141226181359.3b8ac096@fsol> Message-ID: I think the 5-6 year estimate is pessimistic. Take a look at http://en.wikipedia.org/wiki/Xeon_Phi for some background. I have it on a sort of nudge-and-wink authority that Intel already has in-house chips with 128 cores, and has distributed prototypes to a limited set of customers/partners. More good reasons to look at PyPy-STM, which has reached the stage of "useful" I think. On Fri, Dec 26, 2014 at 1:03 PM, Ron Adam wrote: > > > On 12/26/2014 11:13 AM, Antoine Pitrou wrote: > >> On Wed, 24 Dec 2014 13:04:01 -0600 >> Ron Adam wrote: >> >>> > >>> >My thoughts is that making python easier to multi-process on multi-core >>> >CPUs will be where the biggest performance gains will be. Think of 100 >>> >core chips in as soon as 5 or 6 years. >>> >> > Won't happen on mainstream computers >> (laptop/desktop/tablet/smartphone), as it's a totally silly thing to >> do there. >> > > Which is silly?, 100 cores, or making python easier to multi-process? > > The 5 or 6 years figure is my optimistic expectation for high end > workstations and servers. Double that time for typical desktop, and maybe > triple that for wearable devices. > > Currently you can get 8 core high end desktop systems, and up to 18 core > work stations with windows 8. They probably run python too. > > I think the unknown is how much time it will take, not weather or not it > will happen. > > Cheers, > Ron > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eltoder at gmail.com Fri Dec 26 21:30:30 2014 From: eltoder at gmail.com (Eugene Toder) Date: Fri, 26 Dec 2014 15:30:30 -0500 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: Message-ID: On Fri, Dec 26, 2014 at 2:15 PM, Andrew Barnert wrote: > I'm not sure you can just skip over that last point without addressing it. > In this case, given iterables of element types Y1, Y2, ..., Yn, you can say > that they're all type Iterable[Union[Y1, Y2, ..., Yn]]. I _think_ an > inference engine can find that Union type pretty easily, and I _think_ that > at least for collection methods there won't be any harder problems--but I > wouldn't just assume either of those without looking carefully. With what I suggest the vararg is typed the same as a literal list: I assume the type of the expression [Y1, Y2, ...] is List[Union[Y1, Y2, ...]]. Alternatively, the function call is typed as if the argument was replicated the number of times equal to the number of actual arguments. Both ways should give the same result, and are already supported in the type checker. This seems intuitive, matches your analysis above, and implemented in at least C#, Java and Scala, so there's a good evidence that this is quite usable. > And it certainly isn't true when we go past collection methods--clearly map > and zip can't be handled this way. You're right, these cannot be typed without specifying some rewrite rules on the arguments types. You can go further, and say that str.format() type needs to parse the format string to determine the number and the types of what goes into varargs. I think the simple "all of the same type" rule is good enough to type the majority of uses of varargs, except for argument forwarding into a call. At least it's better than nothing. > The point is, at least this example only really needs the type of self, not > an arbitrary forward declaration or an expression that has to be crammed > into a string. Are there any good examples where that isn't true? Depends on what you consider good :-) What is in your opinion a good use of a forward reference not to the current class? Say you have a helper class to implement your collection (e.g. List[X] and ListNode[X]), and for some reason you need to reference the their types from each other. I think this is not too hard to imagine, but there are may be ways to avoid it. > Also, should Set.union be contravariant in the generic type Set, or is it > always going to return a Set[Something]? The two options there could both > easily be handled by type expressions, or maybe with explicit forward > declarations, but with implicit forward declaration via string? I assume you mean covariant? As in, subclasses of Set will return their own types from union. This can be solved without type expressions by having a special SelfType. A few languages use that -- it's useful for copy()-like methods. Eugene From solipsis at pitrou.net Fri Dec 26 21:37:28 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 26 Dec 2014 21:37:28 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? References: <20141226181359.3b8ac096@fsol> Message-ID: <20141226213728.778d5fbb@fsol> On Fri, 26 Dec 2014 14:03:42 -0600 Ron Adam wrote: > > On 12/26/2014 11:13 AM, Antoine Pitrou wrote: > > On Wed, 24 Dec 2014 13:04:01 -0600 > > Ron Adam wrote: > >> > > >> >My thoughts is that making python easier to multi-process on multi-core > >> >CPUs will be where the biggest performance gains will be. Think of 100 > >> >core chips in as soon as 5 or 6 years. > > > Won't happen on mainstream computers > > (laptop/desktop/tablet/smartphone), as it's a totally silly thing to > > do there. > > Which is silly?, 100 cores, This :-) > The 5 or 6 years figure is my optimistic expectation for high end > workstations and servers. I don't see how that's optimistic. Most workloads are intrinsically serial, not parallel. Expecting to get a 100-core general purpose CPU is expecting to get something that's unfit for most daily tasks, which is rather pessimistic. If the industry had followed the enthusiastic predictions from 5 years ago, the average desktop CPU would probably have 16+ HW threads right now - which it doesn't: the average core count stagnates between 2 and 4. Sure, some specific workloads in scientific computing may benefit - but if I understand correctly you can already release the GIL using Cython, and perhaps soon using Numba. Besides the serial nature of most workloads, there are other limits to multicore scalability, such as DRAM access latency and bandwidth. There's little point in having 100 CPU cores if they all compete for memory access as executing multiple threads simultaneously reduces the locality of accesses and therefore the efficiency of on-chip caches. Regards Antoine. From solipsis at pitrou.net Fri Dec 26 21:39:11 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 26 Dec 2014 21:39:11 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? References: <20141226181359.3b8ac096@fsol> Message-ID: <20141226213911.6c5e6818@fsol> On Fri, 26 Dec 2014 13:11:19 -0700 David Mertz wrote: > I think the 5-6 year estimate is pessimistic. Take a look at > http://en.wikipedia.org/wiki/Xeon_Phi for some background. """Intel Many Integrated Core Architecture or Intel MIC (pronounced Mick or Mike[1]) is a *coprocessor* computer architecture""" Enough said. It's not a general-purpose chip. It's meant as a competitor against the computational use of GPU, not against traditional general-purpose CPUs. Regards Antoine. From stefan_ml at behnel.de Fri Dec 26 22:55:58 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 26 Dec 2014 22:55:58 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: <20141226213728.778d5fbb@fsol> References: <20141226181359.3b8ac096@fsol> <20141226213728.778d5fbb@fsol> Message-ID: Antoine Pitrou schrieb am 26.12.2014 um 21:37: > Most workloads are intrinsically > serial, not parallel. Expecting to get a 100-core general purpose CPU > is expecting to get something that's unfit for most daily tasks, which > is rather pessimistic. If the industry had followed the enthusiastic > predictions from 5 years ago, the average desktop CPU would probably > have 16+ HW threads right now - which it doesn't: the average core count > stagnates between 2 and 4. I second this. Parallelisation continues to be difficult (and most often impossible or close enough), except for the trivial cases. As long as that holds, large multi-core chips will remain special purpose. Also, don't forget Knuth's famous quote on premature optimisation. In ~97% of the cases, (non-trivial) parallelisation is simply not needed, and thus is better not be done. Stefan From mertz at gnosis.cx Fri Dec 26 23:05:07 2014 From: mertz at gnosis.cx (David Mertz) Date: Fri, 26 Dec 2014 15:05:07 -0700 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: <20141226213911.6c5e6818@fsol> References: <20141226181359.3b8ac096@fsol> <20141226213911.6c5e6818@fsol> Message-ID: On Fri, Dec 26, 2014 at 1:39 PM, Antoine Pitrou wrote: > On Fri, 26 Dec 2014 13:11:19 -0700 > David Mertz wrote: > > I think the 5-6 year estimate is pessimistic. Take a look at > > http://en.wikipedia.org/wiki/Xeon_Phi for some background. > > """Intel Many Integrated Core Architecture or Intel MIC (pronounced > Mick or Mike[1]) is a *coprocessor* computer architecture""" > > Enough said. It's not a general-purpose chip. It's meant as a > competitor against the computational use of GPU, not against > traditional general-purpose CPUs. > Yes and no: The cores of Intel MIC are based on a modified version of P54C design, used in the original Pentium. The basis of the Intel MIC architecture is to leverage x86 legacy by creating a x86-compatible multiprocessor architecture that can utilize existing parallelization software tools. Programming tools include OpenMP, OpenCL, Cilk/Cilk Plus and specialised versions of Intel's Fortran, C++ and math libraries. x86 is pretty general purpose, but also yes it's meant to compete with GPUs too. But also, there are many projects--including Numba--that utilize GPUs for "general computation" (or at least to offload much of the computation). The distinctions seem to be blurring in my mind. But indeed, as many people have observed, parallelization is usually non-trivial, and the presence of many cores is a far different thing from their efficient utilization. > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Dec 27 01:28:14 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 27 Dec 2014 01:28:14 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: References: <20141226181359.3b8ac096@fsol> <20141226213911.6c5e6818@fsol> Message-ID: <58166E9E-867F-4F17-808C-651A18E2B3D2@yahoo.com> On Dec 26, 2014, at 23:05, David Mertz wrote: > On Fri, Dec 26, 2014 at 1:39 PM, Antoine Pitrou wrote: >> On Fri, 26 Dec 2014 13:11:19 -0700 >> David Mertz wrote: >> > I think the 5-6 year estimate is pessimistic. Take a look at >> > http://en.wikipedia.org/wiki/Xeon_Phi for some background. >> >> """Intel Many Integrated Core Architecture or Intel MIC (pronounced >> Mick or Mike[1]) is a *coprocessor* computer architecture""" >> >> Enough said. It's not a general-purpose chip. It's meant as a >> competitor against the computational use of GPU, not against >> traditional general-purpose CPUs. > > Yes and no: > > The cores of Intel MIC are based on a modified version of P54C design, used in the original Pentium. The basis of the Intel MIC architecture is to leverage x86 legacy by creating a x86-compatible multiprocessor architecture that can utilize existing parallelization software tools. Programming tools include OpenMP, OpenCL, Cilk/Cilk Plus and specialised versions of Intel's Fortran, C++ and math libraries. > > x86 is pretty general purpose, but also yes it's meant to compete with GPUs too. But also, there are many projects--including Numba--that utilize GPUs for "general computation" (or at least to offload much of the computation). The distinctions seem to be blurring in my mind. > > But indeed, as many people have observed, parallelization is usually non-trivial, and the presence of many cores is a far different thing from their efficient utilization. I think what we're eventually going to see is that optimized, explicit parallelism is very hard, but general-purpose implicit parallelism is pretty easy if you're willing to accept a lot of overhead. When people start writing a lot of code that takes 4x as much CPU but can run on 64 cores instead of 2 and work with a dumb ring cache instead of full coherence, that's when people will start selling 128-core laptops. And it's not going to be new application programming techniques that make that happen, it's going to be things like language-level STM, implicit parallelism libraries, kernel schedulers that can migrate low-utilization processes into low-power auxiliary cores, etc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Sat Dec 27 02:40:25 2014 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 26 Dec 2014 19:40:25 -0600 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: <20141226213728.778d5fbb@fsol> References: <20141226181359.3b8ac096@fsol> <20141226213728.778d5fbb@fsol> Message-ID: On 12/26/2014 02:37 PM, Antoine Pitrou wrote: > On Fri, 26 Dec 2014 14:03:42 -0600 > Ron Adam wrote: >> > >> >On 12/26/2014 11:13 AM, Antoine Pitrou wrote: >>> > >On Wed, 24 Dec 2014 13:04:01 -0600 >>> > >Ron Adam wrote: >>>>> > >> > >>>>> > >> >My thoughts is that making python easier to multi-process on multi-core >>>>> > >> >CPUs will be where the biggest performance gains will be. Think of 100 >>>>> > >> >core chips in as soon as 5 or 6 years. >> > >>> > >Won't happen on mainstream computers >>> > >(laptop/desktop/tablet/smartphone), as it's a totally silly thing to >>> > >do there. >> > >> >Which is silly?, 100 cores, > This:-) Depends on what and how it's used. >> >The 5 or 6 years figure is my optimistic expectation for high end >> >workstations and servers. > I don't see how that's optimistic. Most workloads are intrinsically > serial, not parallel. This is changing. It's a feedback loop, as new hardware becomes available, software engineers learn to take advantage of it, and as they do, it drives the market... and then more hardware improvements are added by hardware engineers. Hey, they need a pay check, and investors need dividends, and there are enough people interested to make everyone think it's worth doing. It's a huge economic machine we are talking about, and it's moving forward. I doubt we could stop it if we wanted to. So it's a process that is changing across the board. If you think of one part of that staying the same, either the hardware or the software, the other parts may seem "silly", but if you think of it all changing at once, it starts to make more sense. I wanted to find a good example, and the first thing that came to mind is the ever present web browser. That is probably the one piece of software that is run by more users directly than any other software program. A quick search brought up an interesting talk by Jack Moffitt about the Mozilla SERVO project and the Rust programming language. It's a bit long but interesting. At one point, (18:30 to 19:45 minute marks), he mentions a tree design where you spawn threads for each child, and they spawn threads for each of their children. http://www.infoq.com/presentations/servo-parallel-browser That of course is the software engineering side of things, and then you also have the hardware engineering side, but both sides are actively being developed at the same time. > Expecting to get a 100-core general purpose CPU > is expecting to get something that's unfit for most daily tasks, which > is rather pessimistic. If the industry had followed the enthusiastic > predictions from 5 years ago, the average desktop CPU would probably > have 16+ HW threads right now - which it doesn't: the average core count > stagnates between 2 and 4. Yes, I think it got off to a slower start than many expected. And with only a few cores, it makes more sense to distribute process's rather than threads. But I think this will change as the number of cores increase, and the techniques to use them also develop. Until we have the kind of fine grained threading Jack Moffitt was describing. > Sure, some specific workloads in scientific computing may benefit - > but if I understand correctly you can already release the GIL using > Cython, and perhaps soon using Numba. A web browser is about as mainstream as you can get. And it's presence is big enough to drive the computer market where ever it goes. ;-) > Besides the serial nature of most workloads, there are other limits to > multicore scalability, such as DRAM access latency and bandwidth. > There's little point in having 100 CPU cores if they all compete for > memory access as executing multiple threads simultaneously reduces the > locality of accesses and therefore the efficiency of on-chip caches. It will be interesting to see how this changes. ;-) Cheers, Ron From sturla.molden at gmail.com Sat Dec 27 02:54:02 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sat, 27 Dec 2014 01:54:02 +0000 (UTC) Subject: [Python-ideas] Type Hinting - Performance booster ? References: <20141226181359.3b8ac096@fsol> <20141226213728.778d5fbb@fsol> Message-ID: <1266584657441337410.391593sturla.molden-gmail.com@news.gmane.org> Antoine Pitrou wrote: > I don't see how that's optimistic. Most workloads are intrinsically > serial, not parallel. Computer graphics is intrinsically data-parallel, hence the GPU. A computer with a 128 core CPU would have no use for a GPU. Taking more market shared from Nvidia and AMD would be one reason why Intel might want to produce such a chip. It would also remove the need for dedicated video RAM and special vertex buffer objects, and thus simplify the coding of 3D graphics. > Sure, some specific workloads in scientific computing may benefit - > but if I understand correctly you can already release the GIL using > Cython, and perhaps soon using Numba. You can do this under Numba too. Sturla From sturla.molden at gmail.com Sat Dec 27 03:41:00 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sat, 27 Dec 2014 02:41:00 +0000 (UTC) Subject: [Python-ideas] Type Hinting - Performance booster ? References: <20141226181359.3b8ac096@fsol> <20141226213728.778d5fbb@fsol> Message-ID: <1049444120441338476.563399sturla.molden-gmail.com@news.gmane.org> Antoine Pitrou wrote: > I don't see how that's optimistic. Most workloads are intrinsically > serial, not parallel. Henry Ford invented a solution to parallelization of many repetitive serial tasks 100 years ago. His solution is known as a conveyor belt or pipeline. If you can split up a serial task into a series of smaller subtasks, you can chain them as a pipeline of worker threads. It often shows up in signal processing and multimedia. Take e.g. a look at the design of VTK. You also have it in e.g. asynchronous i/o if you use threads and queues instead of coroutines to set up a pipeline. Then there is big class of data-parallel tasks, such as e.g. in computer graphics. You e.g. have more than a million pixels on a screen, and each pixel must be processed independently. MapReduce is also a buzzword that describes a certain data parallel task. You also find it scientific computing, e.g. in linear algebra where we use libraries like BLAS and LAPACK. Then there is the ForkJoin tasks, a modern buzzword for a certain type of divide and conquer. A classical example is the FFT. Mergesort would be another example. Take a look at a statement like a = [for foobar(y) in sorted(x)] Here we have a data parallel iteration over sorted(x) and the evaluation of sorted(x) is fork-join parallel. Is it unthinkable that a future compiler could figure this out on its own? Sturla From steve at pearwood.info Sat Dec 27 03:43:03 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 27 Dec 2014 13:43:03 +1100 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: <1266584657441337410.391593sturla.molden-gmail.com@news.gmane.org> References: <20141226181359.3b8ac096@fsol> <20141226213728.778d5fbb@fsol> <1266584657441337410.391593sturla.molden-gmail.com@news.gmane.org> Message-ID: <20141227024303.GW6580@ando.pearwood.info> On Sat, Dec 27, 2014 at 01:54:02AM +0000, Sturla Molden wrote: > Antoine Pitrou wrote: > > > I don't see how that's optimistic. Most workloads are intrinsically > > serial, not parallel. > > Computer graphics is intrinsically data-parallel, hence the GPU. A computer > with a 128 core CPU would have no use for a GPU. Or, to put it another way, a computer with a GPU has no need for a 128 core CPU. I think it is more likely that general purpose desktop computers will start using GPUs than that they will start using 100+ core CPUs. Apart from graphics, and some "embarrassingly parallel" tasks, most tasks are inherently serial. E.g. it takes 9 months to make a baby, you can't do it in 1 month by using 9 women or 100 chickens. Even those which aren't inherently serial usually have some serial components, and Amadahl's Law puts an upper bound on how much of a performance increase you can get by parallising it: http://en.wikipedia.org/wiki/Amdahl's_law I expect that the number of cores used by general purpose desktops will increase very slowly. It makes sense for servers to use as many cores as possible, since they typically run many CPU-bound tasks in parallel, but that doesn't apply so much to desktops and it certainly doesn't apply to wearable computers. The idea of a wearable computer using a general-purpose CPU with 100 cores strikes me as sheer fantasy: most of the cores will be idling all of the time, the effect on battery life will be terrible, and the heat generated prohibitive. TL;DR I expect that two or four cores will remain standard for desktop computers for a long time, and one core machines will still be around for a while. Massively parallel computing will remain niche, the GIL is not really the bottleneck some people think it is, and when it is a bottleneck, existing ways of working around it are still very effective. -- Steven From stephen at xemacs.org Sat Dec 27 04:10:49 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 27 Dec 2014 12:10:49 +0900 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: <1266584657441337410.391593sturla.molden-gmail.com@news.gmane.org> References: <20141226181359.3b8ac096@fsol> <20141226213728.778d5fbb@fsol> <1266584657441337410.391593sturla.molden-gmail.com@news.gmane.org> Message-ID: <874msh4wjq.fsf@uwakimon.sk.tsukuba.ac.jp> Sturla Molden writes: > Taking more market shared from Nvidia and AMD would be one reason > why Intel might want to produce such a chip. You've put your finger on a very important class of reasons. And if history is any guide, they will become cheaper quickly. We should not underestimate the ability of consumers to be impressed by easily grasped numbers like "cores" and "GHz" once price comes within reach. From guido at python.org Sat Dec 27 05:09:36 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 26 Dec 2014 20:09:36 -0800 Subject: [Python-ideas] Type Hinting Kick-off In-Reply-To: References: Message-ID: On Fri, Dec 26, 2014 at 12:00 PM, Eugene Toder wrote: > On Thu, Dec 25, 2014 at 10:41 PM, Guido van Rossum > wrote: > [...] > > Eek. That sounds like a bad idea -- copy.copy() uses introspection and I > > don't think there's much hope to be able to spell its type. > Does it really use more introspection than your space_for() function? > Naively, > copy is implemented like: > > def copy(obj): > if isinstance(obj, int): return obj > if isinstance(obj, list): return list(obj) > ... > return obj.__copy__() > > This does not seem very hard to type. Well, it copies most class instances by just copying the __dict__. And it recognizes a bunch of other protocols (__copy__ and most pickling interfaces). > There are much simpler examples, though: > > a) Keys of Dict and elements of Set must be Hashable, > b) To use list.index() list elements must be Comparable, > c) Arguments to min() and max() must be Ordered, > d) Arguments to sum() must be Addable. > > So it's not uncommon to have generic functions that need restrictions on > type > variables. > I think you are still trying to design a type system that can express all constraints exactly. In practice I doubt if any of the examples you mention here will help catch many bugs in actual Python code; a type checker that is blissfully unaware of these requirements will still be tremendously useful. (I guess this is the point of gradual typing.) > > That sounds like an artificial requirement on the implementation > designed to > > help the type checker. > Perhaps I'm not explaining this right. The requirement is not for the type > checker, it is to be able to implement the function. As you pointed out > earlier, without any requirements on the type the function will not be > able to > produce the value. > Yeah, but my counter is that Python users today don't write classes like that, and I don't want them to have to change their habits. > > Peter Norvig mentioned that the subtleties of co/contra-variance of > generic > > types in Java were too complex for his daughter, and also reminded me > that > > Josh Bloch has said somewhere that he believed they made it too complex. > IIUC the context for both statements is specifically the call site > annotations > of variance, i.e. and . Though you can also > quote > Gilad Bracha, who declared that variance is too complicated notion for > programmers to understand, and made all generics in Dart covariant. This > was also the case in Beta, whose authors denounced invariance and > contravariance, as coming from people "with type-checking background" :-) > I'm not sure I understand why you think that is funny. I think they all have a point. > > But I can see a serious downside as well. There will likely be multiple > > tools that have to be able to read the type hinting annotations, e.g. > IDEs > > may want to use the type hints (possibly from stub files) for code > > completion purposes. Also someone might want to write a decorator that > > extracts the annotations and asserts that arguments match at run time. > The > > more handy shorthands we invent, the more complex all such tools will > have > > to be. > I think if these tools cannot use functions from the typing module they > will > have bigger problems than shorthands. And the number of shorthands is > pretty > limited -- there are only as many literals in Python. > I think there are at least three separate use cases (note that none are part of the proposal -- the proposal just enables a single notation to be used for all three): (1) Full type checkers like mypy. These have to parse everything without ever running it, so they cannot use the typing module's primitives. They may also have to parse stuff in comments (there are several places where mypy needs a little help and the best place to put it is often in a #type: comment), which rules out Python's ast module. (2) Things that use runtime introspection, e.g. decorators that try to enforce run time correctness. These can use the typing module's primitives. I wish we could just always have (generic) type objects in the annotations, so they could just look up the annotation and then use isinstance(), but I fear that forward refs will spoil that simplicity anyway. (3) IDEs. These typically need to be able to parse code that contains errors. So they end up having their own, more forgiving parser. That's enough distinct cases to make me want to compromise towards a slightly more verbose syntax that requires less special handling. (I am also compromising because I don't want to change CPython's parser and I want to be able to backport typing.py to Python 3.4 and perhaps even 3.3.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From russell at keith-magee.com Sat Dec 27 07:04:24 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Sat, 27 Dec 2014 14:04:24 +0800 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: Hi Daniel, I've been working on this problem recently; I posted about the idea to this very list a few months ago: https://mail.python.org/pipermail/python-ideas/2014-October/029856.html At the time I posted, I had posted Python builds that work on both Android and iOS: https://github.com/pybee/Python-iOS-support https://github.com/pybee/Python-Android-support and template projects that get you to "hello world" on a mobile platform without much effort. https://github.com/pybee/Python-iOS-template https://github.com/pybee/Python-Android-template I've also tackled the cross-language bridging issue: https://github.com/pybee/rubicon-objc https://github.com/pybee/rubicon-java and wrapped native widgets up into a cross-platform widget library: http://pybee.org/toga/ Now - I'm not claiming that these are anywhere near production ready - they're proof of concept at best. However, I completely agree with your original premise that Python needs a story on mobile platforms, and I'm interested in working on that story. If you go back and read the Python-ideas thread from a few months ago, it looks like there was broad support for the idea, but it would all hinge on the specifics. So, that's what I've been working on. Since I posted to Python-ideas in October, I've been focussing on getting the patches necessary for Python's own build system. However, I've been distracted by work life etc, so I haven't made as much progress as I would have liked. I'm hoping the Christmas/New year break will give me a chance to make some headway. As far as Kivy goes - I have tinkered with Kivy; I've even used Kivy's tools as a starting point for some of my own efforts. However, to me, Kivy is focussing too far up the stack - their toolchain appears to be based on getting *Kivy* working on mobile, not *Python*. This is an entirely reasonable approach if Kivy is what you want at the end of the day - but it's not what *I* want. I've been taking a much more atomic approach. The Kivy toolchain is also anchored in old versions of Python 2; something that definitely needs to be addressed. Ultimately, I want to get these patches in to Python's trunk; but I'm still a way off that being a reality. If you're interested in collaborating and helping me get to that point, please get in touch - there's plenty of work to do, and if you're keen to help, I'm happy to share what I've got so far. Yours, Russ Magee %-) On Fri, Dec 26, 2014 at 6:34 PM, Fetchinson . wrote: > > Hi all, maybe the best list would be python-dev but I don't dare > waking the sleeping lions there :) > > So my question is this: is there a coherent mobile strategy among core > dev people? I mean you guys release python for linux/macos/windows and > the question is if there are any plans to do the same for a mobile > platform. It doesn't have to be android or ios just anything that the > core dev team chooses and sticks with. > > I've been developing python apps for smartphones (mostly hobby > projects though) using sl4a but that seems like is dead. Now people > suggest me using kivy which seems to be alive but who knows how long. > There are some other projects qpython, etc, which are small and > equally not so reliable at least looking from the outside. Is kivy now > an officially blessed distribution? Since google was so integral to > both python (through employing Guido) and android I'd think it would > make sense for google to have an official python environment for > android in cooperation with the python dev team. > > Does the PSF has an opinion on this? It would be great if there would > be something for mobile phones that we could rely on not going away > just as with linux/macos/windows. > > Or there are some issues which preclude this from the start? > > Cheers, > Daniel > > > -- > Psss, psss, put it down! - http://www.cafepress.com/putitdown > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From trent at snakebite.org Sat Dec 27 08:34:33 2014 From: trent at snakebite.org (Trent Nelson) Date: Sat, 27 Dec 2014 02:34:33 -0500 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: <58166E9E-867F-4F17-808C-651A18E2B3D2@yahoo.com> References: <20141226181359.3b8ac096@fsol> <20141226213911.6c5e6818@fsol> <58166E9E-867F-4F17-808C-651A18E2B3D2@yahoo.com> Message-ID: <20141227073430.GA40003@snakebite.org> On Sat, Dec 27, 2014 at 01:28:14AM +0100, Andrew Barnert wrote: > On Dec 26, 2014, at 23:05, David Mertz wrote: > > > On Fri, Dec 26, 2014 at 1:39 PM, Antoine Pitrou > > wrote: > >> On Fri, 26 Dec 2014 13:11:19 -0700 David Mertz > >> wrote: > >> > I think the 5-6 year estimate is pessimistic. Take a look at > >> > http://en.wikipedia.org/wiki/Xeon_Phi for some background. > >> > >> """Intel Many Integrated Core Architecture or Intel MIC (pronounced > >> Mick or Mike[1]) is a *coprocessor* computer architecture""" > >> > >> Enough said. It's not a general-purpose chip. It's meant as a > >> competitor against the computational use of GPU, not against > >> traditional general-purpose CPUs. > > > > Yes and no: > > > > The cores of Intel MIC are based on a modified version of P54C > > design, used in the original Pentium. The basis of the Intel MIC > > architecture is to leverage x86 legacy by creating a x86-compatible > > multiprocessor architecture that can utilize existing > > parallelization software tools. Programming tools include OpenMP, > > OpenCL, Cilk/Cilk Plus and specialised versions of Intel's Fortran, > > C++ and math libraries. > > > > x86 is pretty general purpose, but also yes it's meant to compete > > with GPUs too. But also, there are many projects--including > > Numba--that utilize GPUs for "general computation" (or at least to > > offload much of the computation). The distinctions seem to be > > blurring in my mind. > > > > But indeed, as many people have observed, parallelization is usually > > non-trivial, and the presence of many cores is a far different thing > > from their efficient utilization. > > I think what we're eventually going to see is that optimized, explicit > parallelism is very hard, but general-purpose implicit parallelism is > pretty easy if you're willing to accept a lot of overhead. When people > start writing a lot of code that takes 4x as much CPU but can run on > 64 cores instead of 2 and work with a dumb ring cache instead of full > coherence, that's when people will start selling 128-core laptops. And > it's not going to be new application programming techniques that make > that happen, it's going to be things like language-level STM, implicit > parallelism libraries, kernel schedulers that can migrate > low-utilization processes into low-power auxiliary cores, etc. I disagree. PyParallel works fine with existing programming techniques: Just took a screen share of a load test between normal Python 3.3 release build, and the debugged-up-the-wazzo flaky PyParallel 0.1-ish, and it undeniably crushes the competition. (Then crashes, 'cause you can't have it all.) https://www.youtube.com/watch?v=JHaIaOyfldo Keep in mind that's a full debug build, but not only that, I've butchered every PyObject and added like, 6 more 8-byte pointers to it; coupled with excessive memory guard tests at every opportunity that result in a few thousand hash tables being probed to check for ptr address membership. The thing is slooooooww. And even with all that in place, check out the results: Python33: Running 10s test @ http://192.168.1.15:8000/index.html 8 threads and 64 connections Thread Stats Avg Stdev Max +/- Stdev Latency 13.69ms 11.59ms 27.93ms 52.76% Req/Sec 222.14 234.53 1.60k 86.91% Latency Distribution 50% 5.67ms 75% 26.75ms 90% 27.36ms 99% 27.93ms 16448 requests in 10.00s, 141.13MB read Socket errors: connect 0, read 7, write 0, timeout 0 Requests/sec: 1644.66 Transfer/sec: 14.11MB PyParallel v0.1, exploiting all cores: Running 10s test @ http://192.168.1.15:8080/index.html 8 threads and 8 connections Thread Stats Avg Stdev Max +/- Stdev Latency 2.32ms 2.29ms 27.57ms 92.89% Req/Sec 540.82 154.01 0.89k 75.34% Latency Distribution 50% 1.68ms 75% 2.00ms 90% 3.57ms 99% 11.26ms 40828 requests in 10.00s, 350.47MB read Requests/sec: 4082.66 Transfer/sec: 35.05MB ~2.5 times improvement even with all its warts. And it's still not even close to being loaded enough -- 35% of a gigabit link being used and about half core use. No reason it couldn't do 100,000 requests/s. Recent thread on python-ideas with a bit more information: https://mail.python.org/pipermail/python-ideas/2014-November/030196.html Core concepts: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores Trent. From solipsis at pitrou.net Sat Dec 27 11:00:15 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 27 Dec 2014 11:00:15 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? References: <20141226181359.3b8ac096@fsol> <20141226213728.778d5fbb@fsol> Message-ID: <20141227110015.04980267@fsol> On Fri, 26 Dec 2014 19:40:25 -0600 Ron Adam wrote: > > It's a bit long but interesting. At one point, (18:30 to 19:45 minute > marks), he mentions a tree design where you spawn threads for each child, > and they spawn threads for each of their children. Yeah, so what? Spawning threads isn't hard, exploiting parallelism is. > Yes, I think it got off to a slower start than many expected. It didn't "get off to a slower start". It's actually perfectly in line with predictions of people like me or Linus Torvalds :-) > A web browser is about as mainstream as you can get. Then let's talk about it again when a web browser manages to fully exploit 100 threads without thrashing CPU caches. Regards Antoine. From abarnert at yahoo.com Sat Dec 27 11:36:11 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 27 Dec 2014 11:36:11 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: <20141227073430.GA40003@snakebite.org> References: <20141226181359.3b8ac096@fsol> <20141226213911.6c5e6818@fsol> <58166E9E-867F-4F17-808C-651A18E2B3D2@yahoo.com> <20141227073430.GA40003@snakebite.org> Message-ID: <5B89E0F8-8039-48A3-AC54-465FE23398A4@yahoo.com> On Dec 27, 2014, at 8:34, Trent Nelson wrote: > On Sat, Dec 27, 2014 at 01:28:14AM +0100, Andrew Barnert wrote: >> On Dec 26, 2014, at 23:05, David Mertz wrote: >> >>> On Fri, Dec 26, 2014 at 1:39 PM, Antoine Pitrou >>> wrote: >>>> On Fri, 26 Dec 2014 13:11:19 -0700 David Mertz >>>> wrote: >>>>> I think the 5-6 year estimate is pessimistic. Take a look at >>>>> http://en.wikipedia.org/wiki/Xeon_Phi for some background. >>>> >>>> """Intel Many Integrated Core Architecture or Intel MIC (pronounced >>>> Mick or Mike[1]) is a *coprocessor* computer architecture""" >>>> >>>> Enough said. It's not a general-purpose chip. It's meant as a >>>> competitor against the computational use of GPU, not against >>>> traditional general-purpose CPUs. >>> >>> Yes and no: >>> >>> The cores of Intel MIC are based on a modified version of P54C >>> design, used in the original Pentium. The basis of the Intel MIC >>> architecture is to leverage x86 legacy by creating a x86-compatible >>> multiprocessor architecture that can utilize existing >>> parallelization software tools. Programming tools include OpenMP, >>> OpenCL, Cilk/Cilk Plus and specialised versions of Intel's Fortran, >>> C++ and math libraries. >>> >>> x86 is pretty general purpose, but also yes it's meant to compete >>> with GPUs too. But also, there are many projects--including >>> Numba--that utilize GPUs for "general computation" (or at least to >>> offload much of the computation). The distinctions seem to be >>> blurring in my mind. >>> >>> But indeed, as many people have observed, parallelization is usually >>> non-trivial, and the presence of many cores is a far different thing >>> from their efficient utilization. >> >> I think what we're eventually going to see is that optimized, explicit >> parallelism is very hard, but general-purpose implicit parallelism is >> pretty easy if you're willing to accept a lot of overhead. When people >> start writing a lot of code that takes 4x as much CPU but can run on >> 64 cores instead of 2 and work with a dumb ring cache instead of full >> coherence, that's when people will start selling 128-core laptops. And >> it's not going to be new application programming techniques that make >> that happen, it's going to be things like language-level STM, implicit >> parallelism libraries, kernel schedulers that can migrate >> low-utilization processes into low-power auxiliary cores, etc. > > I disagree. PyParallel works fine with existing programming techniques: Then what are you disagreeing with? My whole point is that it's not going to be new application programming techniques that make parallelism accessible. > Just took a screen share of a load test between normal Python 3.3 > release build, and the debugged-up-the-wazzo flaky PyParallel 0.1-ish, > and it undeniably crushes the competition. (Then crashes, 'cause you > can't have it all.) > > https://www.youtube.com/watch?v=JHaIaOyfldo > > Keep in mind that's a full debug build, but not only that, I've > butchered every PyObject and added like, 6 more 8-byte pointers to it; > coupled with excessive memory guard tests at every opportunity that > result in a few thousand hash tables being probed to check for ptr > address membership. > > The thing is slooooooww. And even with all that in place, check out the > results: Sure, sloooooww code that's 8x as parallel runs 2.5x as fast. What's held things back for so long is that people insist on code that's almost as fast on 1- or 2-core machines and also scales to 8-core machines. That silly constraint is what's held us back. And now that mainstream machines are 2 to 8 cores instead of 1 to 2, and the code you have to be almost as fast as is still sequential, things are starting to change. Even when things like PyParallel or PyPy's STM aren't optimized at all, they're already winning. From ron3200 at gmail.com Sat Dec 27 21:09:59 2014 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 27 Dec 2014 14:09:59 -0600 Subject: [Python-ideas] Type Hinting - Performance booster ? In-Reply-To: <20141227110015.04980267@fsol> References: <20141226181359.3b8ac096@fsol> <20141226213728.778d5fbb@fsol> <20141227110015.04980267@fsol> Message-ID: On 12/27/2014 04:00 AM, Antoine Pitrou wrote: > On Fri, 26 Dec 2014 19:40:25 -0600 Ron > Adam wrote: >>> It's a bit long but interesting. At one point, (18:30 to 19:45 >>> minute marks), he mentions a tree design where you spawn threads for >>> each child, and they spawn threads for each of their children. > Yeah, so what? Spawning threads isn't hard, exploiting parallelism is. I think you are using a stricter definition of parallelism here. If many programs spawn many threads, and those threads are distributed across many cores, then you have a computer that gains from having more cores. Yes, There are a lot of technical difficulties with doing that efficiently. The point I was trying to make at the start of this is type-hints can be used to identify parts of programs that don't depend on other parts, and that some of those parts can be run in parallel. Basically it involves converting a sequential program into a dependency tree, and allowing non-inter dependent child nodes to run in parallel. The type-hints may help with this. As for the technical details of building a chip with 100 cores and how to manage the cache's for them. I'm happy to let the chip engineers work on that. I do believe we will see 100 core chips in a few years regardless of weather or not it makes complete sense to do so. Keep in mind that other languages may be able to take advantage of them much easier than python can, including newly emerging languages built specifically to use parallelism more effectively. >>> Yes, I think it got off to a slower start than many expected. > It didn't "get off to a slower start". It's actually perfectly in line > with predictions of people like me or Linus Torvalds:-) "... than many expected." It wasn't surprising to me either. >>> A web browser is about as mainstream as you can get. > Then let's talk about it again when a web browser manages to fully > exploit 100 threads without thrashing CPU caches. How about 80% utilisation with only 20% thrashing? ;-) Cheers, Ron From adam.j.carruthers at hotmail.co.uk Sat Dec 27 21:15:43 2014 From: adam.j.carruthers at hotmail.co.uk (Adam Carruthers) Date: Sat, 27 Dec 2014 20:15:43 +0000 Subject: [Python-ideas] Change Behind The Scenes Attribute Access from Class to Object Message-ID: At the moment, when a python function, like print, calls an object attribute it does this (if it were written in python): type(obj).__str__(obj)This can be restrictive in lots of situations and I think it would be better to just get the attribute from the object itself. obj.__str__()It would make you free to override it on a per object basis. In some cases, this could lead to huge optimization.Obviously, it is easy to fix yourself, but it is a layer of uncharacteristic unintuitiveness: def __str__(self): self._str() -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Dec 27 21:25:32 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 27 Dec 2014 21:25:32 +0100 Subject: [Python-ideas] Type Hinting - Performance booster ? References: <20141226181359.3b8ac096@fsol> <20141226213728.778d5fbb@fsol> <20141227110015.04980267@fsol> Message-ID: <20141227212532.4ba4f10a@fsol> On Sat, 27 Dec 2014 14:09:59 -0600 Ron Adam wrote: > > How about 80% utilisation with only 20% thrashing? ;-) Well, how about you give actual examples instead of throwing numbers around. Otherwise let's wait 5 other years to see how the prediction of 100-core general purpose CPUs pans out. Regards Antoine. From rosuav at gmail.com Sun Dec 28 17:42:16 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 29 Dec 2014 03:42:16 +1100 Subject: [Python-ideas] Python hook just before NameError Message-ID: I mainly did up this patch to see how hard it would be, and now it's turned out to be fairly simple, I'm curious as to whether it would actually be useful to people. At the point where a global/builtin name lookup is about to raise NameError, first try calling a Python function, along the same lines as __getattr__. If that function hasn't been defined, raise NameError as normal; but if it has, let it either raise NameError or return some object, which is then used as if the name had been bound to it. Patch is here: http://bugs.python.org/issue23126 The idea is to allow convenient interactive use; auto-importing modules is easy, and importing names from modules ("exp" --> math.exp) can be done easily enough too, given a list of modules to try. It's probably not a good idea to use this in application code, and I definitely wouldn't encourage it in library code, but it has its uses interactively. Thoughts? ChrisA From fetchinson at googlemail.com Sun Dec 28 22:30:41 2014 From: fetchinson at googlemail.com (Fetchinson .) Date: Sun, 28 Dec 2014 22:30:41 +0100 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: On 12/27/14, Russell Keith-Magee wrote: > Hi Daniel, > > I've been working on this problem recently; I posted about the idea to this > very list a few months ago: > > https://mail.python.org/pipermail/python-ideas/2014-October/029856.html Thanks! This was a great read, I haven't noticed it before. You addressed all the issues (and much more) I thought about. And basically I got all the answers from the follow up posts. > At the time I posted, I had posted Python builds that work on both Android > and iOS: > > https://github.com/pybee/Python-iOS-support > https://github.com/pybee/Python-Android-support > > and template projects that get you to "hello world" on a mobile platform > without much effort. > > https://github.com/pybee/Python-iOS-template > https://github.com/pybee/Python-Android-template Thanks again, I'll take a look, it all seems pretty encouraging. > I've also tackled the cross-language bridging issue: > > https://github.com/pybee/rubicon-objc > https://github.com/pybee/rubicon-java > > and wrapped native widgets up into a cross-platform widget library: > > http://pybee.org/toga/ > > Now - I'm not claiming that these are anywhere near production ready - > they're proof of concept at best. However, I completely agree with your > original premise that Python needs a story on mobile platforms, and I'm > interested in working on that story. If you go back and read the > Python-ideas thread from a few months ago, it looks like there was broad > support for the idea, but it would all hinge on the specifics. > > So, that's what I've been working on. Since I posted to Python-ideas in > October, I've been focussing on getting the patches necessary for Python's > own build system. However, I've been distracted by work life etc, so I > haven't made as much progress as I would have liked. I'm hoping the > Christmas/New year break will give me a chance to make some headway. > > As far as Kivy goes - I have tinkered with Kivy; I've even used Kivy's > tools as a starting point for some of my own efforts. However, to me, Kivy > is focussing too far up the stack - their toolchain appears to be based on > getting *Kivy* working on mobile, not *Python*. What do you mean by this exactly? I thought kivy = python + some libraries. So if kivy works on android that necessarily means that python itself works as well, doesn't it? I thought I'd use kivy even if I don't need any of their fancy libraries only a basic python installation, but you'd say this is not a good approach? > This is an entirely > reasonable approach if Kivy is what you want at the end of the day - but > it's not what *I* want. I've been taking a much more atomic approach. The > Kivy toolchain is also anchored in old versions of Python 2; something that > definitely needs to be addressed. As far as I'm concerned python 2 is perfectly okay. If I don't need to worry about python 3, what drawback does kivy have, if any, according to you? > Ultimately, I want to get these patches in to Python's trunk; but I'm still > a way off that being a reality. If you're interested in collaborating and > helping me get to that point, please get in touch - there's plenty of work > to do, and if you're keen to help, I'm happy to share what I've got so far. I'm afraid I don't have the necessary expertise unfortunately. And I was under the impression kivy will bring me a working python environment on android, but please feel free to set me straight on this one :) Cheers, Daniel > Yours, > Russ Magee %-) > > On Fri, Dec 26, 2014 at 6:34 PM, Fetchinson . > wrote: >> >> Hi all, maybe the best list would be python-dev but I don't dare >> waking the sleeping lions there :) >> >> So my question is this: is there a coherent mobile strategy among core >> dev people? I mean you guys release python for linux/macos/windows and >> the question is if there are any plans to do the same for a mobile >> platform. It doesn't have to be android or ios just anything that the >> core dev team chooses and sticks with. >> >> I've been developing python apps for smartphones (mostly hobby >> projects though) using sl4a but that seems like is dead. Now people >> suggest me using kivy which seems to be alive but who knows how long. >> There are some other projects qpython, etc, which are small and >> equally not so reliable at least looking from the outside. Is kivy now >> an officially blessed distribution? Since google was so integral to >> both python (through employing Guido) and android I'd think it would >> make sense for google to have an official python environment for >> android in cooperation with the python dev team. >> >> Does the PSF has an opinion on this? It would be great if there would >> be something for mobile phones that we could rely on not going away >> just as with linux/macos/windows. >> >> Or there are some issues which preclude this from the start? >> >> Cheers, >> Daniel >> >> >> -- >> Psss, psss, put it down! - http://www.cafepress.com/putitdown >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From njs at pobox.com Sun Dec 28 23:25:09 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 28 Dec 2014 22:25:09 +0000 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: References: Message-ID: On Sun, Dec 28, 2014 at 4:42 PM, Chris Angelico wrote: > I mainly did up this patch to see how hard it would be, and now it's > turned out to be fairly simple, I'm curious as to whether it would > actually be useful to people. > > At the point where a global/builtin name lookup is about to raise > NameError, first try calling a Python function, along the same lines > as __getattr__. If that function hasn't been defined, raise NameError > as normal; but if it has, let it either raise NameError or return some > object, which is then used as if the name had been bound to it. > > Patch is here: > http://bugs.python.org/issue23126 > > The idea is to allow convenient interactive use; auto-importing > modules is easy, and importing names from modules ("exp" --> math.exp) > can be done easily enough too, given a list of modules to try. > > It's probably not a good idea to use this in application code, and I > definitely wouldn't encourage it in library code, but it has its uses > interactively. I find it weird that it's *really truly* global, instead of being scoped to a single global namespace. Random thoughts: http://bugs.python.org/issue22986 enables a similar feature for module attribute lookups, which is one substantial source of "global" lookups. (Also it's been waiting for a review for some time, *coughcough* ;-)). If evaluating a specific piece of code, you can already pass an arbitrary dict-like object as the globals argument to exec, with whatever __getitem__ you want. Of course the built-in REPL doesn't provide any way to set the interactive namespace's globals object, but it could, or it'd be easy to implement in a REPL like IPython. You could achieve a similar effect by assigning a custom object to the interactive namespace's __builtins__ variable. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From Steve.Dower at microsoft.com Sun Dec 28 18:03:47 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sun, 28 Dec 2014 17:03:47 +0000 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: References: Message-ID: +1 from me. The interactive uses would be great, and aren't really nice to implement without a hook like this. Top-posted from my Windows Phone ________________________________ From: Chris Angelico Sent: ?12/?28/?2014 10:42 To: python-ideas Subject: [Python-ideas] Python hook just before NameError I mainly did up this patch to see how hard it would be, and now it's turned out to be fairly simple, I'm curious as to whether it would actually be useful to people. At the point where a global/builtin name lookup is about to raise NameError, first try calling a Python function, along the same lines as __getattr__. If that function hasn't been defined, raise NameError as normal; but if it has, let it either raise NameError or return some object, which is then used as if the name had been bound to it. Patch is here: http://bugs.python.org/issue23126 The idea is to allow convenient interactive use; auto-importing modules is easy, and importing names from modules ("exp" --> math.exp) can be done easily enough too, given a list of modules to try. It's probably not a good idea to use this in application code, and I definitely wouldn't encourage it in library code, but it has its uses interactively. Thoughts? ChrisA _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Dec 29 01:09:24 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 29 Dec 2014 11:09:24 +1100 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: <20141229000923.GZ6580@ando.pearwood.info> On Sun, Dec 28, 2014 at 10:30:41PM +0100, Fetchinson . wrote: > On 12/27/14, Russell Keith-Magee wrote: > > This is an entirely > > reasonable approach if Kivy is what you want at the end of the day - but > > it's not what *I* want. I've been taking a much more atomic approach. The > > Kivy toolchain is also anchored in old versions of Python 2; something that > > definitely needs to be addressed. I'm not entirely sure why Russell says this. Kivy claims to support Python 3: http://kivy.org/docs/faq.html#does-kivy-support-python-3-x > As far as I'm concerned python 2 is perfectly okay. If I don't need to > worry about python 3, what drawback does kivy have, if any, according > to you? Well, you've just lost me as a potential user. I have zero interest in legacy Python versions for Android. -- Steven From steve at pearwood.info Mon Dec 29 01:37:42 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 29 Dec 2014 11:37:42 +1100 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: References: Message-ID: <20141229003742.GA6580@ando.pearwood.info> On Mon, Dec 29, 2014 at 03:42:16AM +1100, Chris Angelico wrote: > I mainly did up this patch to see how hard it would be, and now it's > turned out to be fairly simple, I'm curious as to whether it would > actually be useful to people. > > At the point where a global/builtin name lookup is about to raise > NameError, first try calling a Python function, along the same lines > as __getattr__. If that function hasn't been defined, raise NameError > as normal; but if it has, let it either raise NameError or return some > object, which is then used as if the name had been bound to it. An interesting idea, but I don't actually think much of it for interactive use. Having modules magically import themselves without an import is a bad habit for beginners to learn, and less useful for experienced users who know to import things. If I've understood it correctly, it's also process-wide global, rather than limited to a single module. That makes it much less useful, as it risks disguising bugs in library code. -- Steven From stephen at xemacs.org Mon Dec 29 01:45:06 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 29 Dec 2014 09:45:06 +0900 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: <87r3vj2sj1.fsf@uwakimon.sk.tsukuba.ac.jp> Fetchinson . writes: > As far as I'm concerned python 2 is perfectly okay. Please reconsider this position. For the 70% of the world whose native language does not fit well into ISO 8859-1, apps developed in Python 3 with no consideration for their needs are far more likely to be easily adapted to their environments than those developed in Python 2. If you don't care about the rest of the world, fine, but the rest of the world isn't going to care about you, and that matters to building a Python project given that Indians, Chinese, and a smattering of others account for a large fraction of the "pre-professional" labor we get through GSoC. They tend to be far more focused on mobile apps than their mentors in my experience. From russell at keith-magee.com Mon Dec 29 01:50:08 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Mon, 29 Dec 2014 08:50:08 +0800 Subject: [Python-ideas] Fwd: python on mobile In-Reply-To: References: <20141229000923.GZ6580@ando.pearwood.info> Message-ID: On Mon, Dec 29, 2014 at 8:09 AM, Steven D'Aprano wrote: > > On Sun, Dec 28, 2014 at 10:30:41PM +0100, Fetchinson . wrote: > > On 12/27/14, Russell Keith-Magee wrote: > > > > This is an entirely > > > reasonable approach if Kivy is what you want at the end of the day - > but > > > it's not what *I* want. I've been taking a much more atomic approach. > The > > > Kivy toolchain is also anchored in old versions of Python 2; something > that > > > definitely needs to be addressed. > > I'm not entirely sure why Russell says this. Kivy claims to support > Python 3: > > http://kivy.org/docs/faq.html#does-kivy-support-python-3-x > > Read the whole of that FAQ entry: """ However, be aware that while Kivy will run in Python 3.3+, packaging support is not yet complete. If you plan to create mobile apps for Android or iOS, you should use Python 2.7 for now. """ That is - *Kivy* works on Python 3.3, but none of the tools that let you run Kivy on Android or iOS exist for 3.X. This is the hard part - the patches against Python and related libraries are non-trivial. > As far as I'm concerned python 2 is perfectly okay. If I don't need to > > worry about python 3, what drawback does kivy have, if any, according > > to you? > > Well, you've just lost me as a potential user. I have zero interest in > legacy Python versions for Android. For what it's worth, I agree. I'll probably end up supporting Python 2, but only because my currently working code (which is derived from Kivy's toolchain) work on 2.7; but my intention is to start focussing on Python 3 ASAP. IMHO there's no point starting a new project on a platform that has a known EOL on the horizon. Yours Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From russell at keith-magee.com Mon Dec 29 02:05:31 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Mon, 29 Dec 2014 09:05:31 +0800 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: On Mon, Dec 29, 2014 at 5:30 AM, Fetchinson . wrote: > > On 12/27/14, Russell Keith-Magee wrote: > > > So, that's what I've been working on. Since I posted to Python-ideas in > > October, I've been focussing on getting the patches necessary for > Python's > > own build system. However, I've been distracted by work life etc, so I > > haven't made as much progress as I would have liked. I'm hoping the > > Christmas/New year break will give me a chance to make some headway. > > > > As far as Kivy goes - I have tinkered with Kivy; I've even used Kivy's > > tools as a starting point for some of my own efforts. However, to me, > Kivy > > is focussing too far up the stack - their toolchain appears to be based > on > > getting *Kivy* working on mobile, not *Python*. > > What do you mean by this exactly? I thought kivy = python + some > libraries. So if kivy works on android that necessarily means that > python itself works as well, doesn't it? Yes and no. Yes, Kivy is just a set of Python libraries, and a Kivy program is just a Python program. However, "Hello World" in Kivy isn't 'print("Hello world")'. It's setting up a whole Kivy stack, and starting a Kivy runloop - and the Kivy toolchain is focussed on getting you to the point where you can start that runloop. Even if you work around that, and just use Python "out of the box", you're going to be carrying at least some of the overhead of Kivy. > I thought I'd use kivy even > if I don't need any of their fancy libraries only a basic python > installation, but you'd say this is not a good approach? > It would probably work - I haven't tried, so I can't comment from experience. However, if you go down this path, you won't have a "Basic python installation" - you'll have "Python + kivy", because their tools don't give you the option to *not* install the Kivy parts (at least, not as far as I could work out). >From my experience, getting Python compiled using Kivy's tools wasn't that hard - but getting Kivy compiled was a PITA, because Kivy has a bunch of other dependencies, like sound support libraries, OpenGL libraries, Cython, and so on. These are all essential for *Kivy*, but not for Python, and not for my purposes either. Ultimately, I'm working on something that is, at one level, a competitor to Kivy. I don't want my install instructions to be "install Kivy, now throw all that stuff away and use Toga"; and I don't want my users to have to go through the same drama of getting Kivy's whole dependency chain compiled. I *just* want Python, and then I'll decide what other pieces I want. That's why I'm trying to get the build fixes into Python's core - I want *Python* to be compilable for mobile platforms, and then leave the decision of which libraries you want to use up to the end developer. Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Mon Dec 29 02:06:59 2014 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Sun, 28 Dec 2014 19:06:59 -0600 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: <20141229003742.GA6580@ando.pearwood.info> References: <20141229003742.GA6580@ando.pearwood.info> Message-ID: On Sun, Dec 28, 2014 at 6:37 PM, Steven D'Aprano wrote: > If I've understood it correctly, it's also process-wide global, rather > than limited to a single module. That makes it much less useful, as it > risks disguising bugs in library code. Though if you are using interactive mode to test library code, you might have bigger problems on your hands than implicit imports... This topic came up in a completely unrelated thread in c.l.py wherein I apologized for an automatic import of the csv module in some interactive copy-n-paste. Chris asked about it, and I posted the module I use (which I don't think was original with me) as a convenience in interactive mode. Then he looked into things and came up with this patch. S From breamoreboy at yahoo.co.uk Mon Dec 29 08:46:02 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 29 Dec 2014 07:46:02 +0000 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: References: Message-ID: On 28/12/2014 16:42, Chris Angelico wrote: > I mainly did up this patch to see how hard it would be, and now it's > turned out to be fairly simple, I'm curious as to whether it would > actually be useful to people. > > At the point where a global/builtin name lookup is about to raise > NameError, first try calling a Python function, along the same lines > as __getattr__. If that function hasn't been defined, raise NameError > as normal; but if it has, let it either raise NameError or return some > object, which is then used as if the name had been bound to it. > > Patch is here: > http://bugs.python.org/issue23126 > > The idea is to allow convenient interactive use; auto-importing > modules is easy, and importing names from modules ("exp" --> math.exp) > can be done easily enough too, given a list of modules to try. > > It's probably not a good idea to use this in application code, and I > definitely wouldn't encourage it in library code, but it has its uses > interactively. > > Thoughts? > > ChrisA +1 from me as I'm always forgetting the "obvious" imports such as sys and os when trying things interactively. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From rosuav at gmail.com Mon Dec 29 09:00:31 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 29 Dec 2014 19:00:31 +1100 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: References: Message-ID: On 28/12/2014 16:42, Chris Angelico wrote: > > At the point where a global/builtin name lookup is about to raise > NameError, first try calling a Python function, along the same lines > as __getattr__. If that function hasn't been defined, raise NameError > as normal; but if it has, let it either raise NameError or return some > object, which is then used as if the name had been bound to it. On Mon, Dec 29, 2014 at 6:46 PM, Mark Lawrence wrote: > +1 from me as I'm always forgetting the "obvious" imports such as sys and os > when trying things interactively. On Mon, Dec 29, 2014 at 11:37 AM, Steven D'Aprano wrote: > An interesting idea, but I don't actually think much of it for > interactive use. Having modules magically import themselves without an > import is a bad habit for beginners to learn, and less useful for > experienced users who know to import things. This is why the feature is not "auto-import anything", but "give Python code the chance to deal with NameError". Open to bikeshedding about whether it's better per-module or global or what, but the point is that you, as the programmer, get the flexibility. Maybe you want to white-list potential imports, which would make this like pre-importing those names but lazily. Maybe you want to allow a half-dozen common "from" imports, but not just anything. Maybe you'd like to have some completely different magic - like REXX mode: >>> import sys >>> sys.__getglobal__ = lambda name: name.upper() >>> foo = "Ham " >>> foo + spam 'Ham SPAM' Or something more suitable for floating point work: >>> sys.__getglobal__ = lambda name: float("nan") >>> x = 4 >>> x + y nan The power is in your hands. And by default, nothing is any different: $ python3 Python 3.5.0a0 (default:1c51f1650c42+, Dec 29 2014, 02:29:06) [GCC 4.7.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> foo Traceback (most recent call last): File "", line 1, in NameError: name 'foo' is not defined Don't like auto-importing? Don't use it. That simple. :) Incidentally, I fully expect that the dunder name __getglobal__ will be rejected. But the proposal works equally well with any name. ChrisA From ncoghlan at gmail.com Mon Dec 29 11:17:21 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 29 Dec 2014 20:17:21 +1000 Subject: [Python-ideas] Change Behind The Scenes Attribute Access from Class to Object In-Reply-To: References: Message-ID: On 28 Dec 2014 06:17, "Adam Carruthers" wrote: > > At the moment, when a python function, like print, calls an object attribute it does this (if it were written in python): > type(obj).__str__(obj) > This can be restrictive in lots of situations and I think it would be better to just get the attribute from the object itself. > obj.__str__() This is the way classic classes generally work in Python 2 (all classic classes are handled by a single underlying type). Removing that behaviour was one of the key changes in the new-style class model introduced in PEPs 252 and 253 (which is now the only class model in Python 3). Delegating to a lookup on the instance at the type level is the appropriate way to handle this when desired. Cheers, Nick. > It would make you free to override it on a per object basis. In some cases, this could lead to huge optimization. > Obviously, it is easy to fix yourself, but it is a layer of uncharacteristic unintuitiveness: > def __str__(self): > self._str() > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Dec 29 11:32:30 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 29 Dec 2014 20:32:30 +1000 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: On 27 Dec 2014 16:05, "Russell Keith-Magee" wrote: > Now - I'm not claiming that these are anywhere near production ready - they're proof of concept at best. However, I completely agree with your original premise that Python needs a story on mobile platforms, and I'm interested in working on that story. If you go back and read the Python-ideas thread from a few months ago, it looks like there was broad support for the idea, but it would all hinge on the specifics. Something that doesn't hinge on the specifics: perhaps it would make sense to set up a "mobile-sig at python.org" mailing list? It would presumably be Mailman 2 to start off with, but would then migrate to Mailman 3 & HyperKitty along with the rest of the python.org lists (hopefully some time in 2015, although we don't have a concrete plan for that as yet). Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Dec 29 12:43:43 2014 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 29 Dec 2014 11:43:43 +0000 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: References: Message-ID: On 29 Dec 2014 07:46, "Mark Lawrence" wrote: > > On 28/12/2014 16:42, Chris Angelico wrote: >> >> I mainly did up this patch to see how hard it would be, and now it's >> turned out to be fairly simple, I'm curious as to whether it would >> actually be useful to people. >> >> At the point where a global/builtin name lookup is about to raise >> NameError, first try calling a Python function, along the same lines >> as __getattr__. If that function hasn't been defined, raise NameError >> as normal; but if it has, let it either raise NameError or return some >> object, which is then used as if the name had been bound to it. >> >> Patch is here: >> http://bugs.python.org/issue23126 >> >> The idea is to allow convenient interactive use; auto-importing >> modules is easy, and importing names from modules ("exp" --> math.exp) >> can be done easily enough too, given a list of modules to try. >> >> It's probably not a good idea to use this in application code, and I >> definitely wouldn't encourage it in library code, but it has its uses >> interactively. >> >> Thoughts? >> >> ChrisA > > > +1 from me as I'm always forgetting the "obvious" imports such as sys and os when trying things interactively. I just set up a PYTHONSTARTUP script to pre-import the obvious things and that fixed this problem for me. You'd still have to make a startup script to set up the hook, and it's not like 'import os, sys' will appreciably affect startup time. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Dec 29 12:50:17 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 29 Dec 2014 22:50:17 +1100 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: References: Message-ID: On Mon, Dec 29, 2014 at 10:43 PM, Nathaniel Smith wrote: > I just set up a PYTHONSTARTUP script to pre-import the obvious things and > that fixed this problem for me. You'd still have to make a startup script to > set up the hook, and it's not like 'import os, sys' will appreciably affect > startup time. Maybe those two won't, but I work with a number of Python students who are using heavier libraries like psycopg2 and numpy, so I'd really rather not auto-import those into everything. On the other hand, the course is currently written for Python 2.7, so I can't actually make use of this there... yet. Another reason to nudge the course writers about updating to 3.x! ChrisA From ncoghlan at gmail.com Mon Dec 29 13:46:00 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 29 Dec 2014 22:46:00 +1000 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: References: Message-ID: On 29 December 2014 at 21:50, Chris Angelico wrote: > On Mon, Dec 29, 2014 at 10:43 PM, Nathaniel Smith wrote: > > I just set up a PYTHONSTARTUP script to pre-import the obvious things and > > that fixed this problem for me. You'd still have to make a startup > script to > > set up the hook, and it's not like 'import os, sys' will appreciably > affect > > startup time. > > Maybe those two won't, but I work with a number of Python students who > are using heavier libraries like psycopg2 and numpy, so I'd really > rather not auto-import those into everything. On the other hand, the > course is currently written for Python 2.7, so I can't actually make > use of this there... yet. Another reason to nudge the course writers > about updating to 3.x! > FWIW, making it easier to create CPython variants that pre-populate __main__ differently (ala pyp or the --pylab option to ipython) is one of the reasons I'd like to eventually implement the interpreter initialisation changes proposed in PEP 432 (or see it implemented). However, even today, it's not particularly difficult to create custom launch scripts that initialise __main__ with a few different modules based on what you plan to work on and then set os.environ["PYTHONINSPECT"] = "1" to drop into the interactive interpreter: $ python3 -c "import sys, os; os.environ['PYTHONINSPECT'] = '1'" >>> sys, os (, ) >>> Making existing pre-initialisation mechanisms easier to discover and/or use seems like a better path forward than adding yet more tools that are even more cryptic and hard to follow. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Dec 29 14:02:32 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Dec 2014 00:02:32 +1100 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: References: Message-ID: On Mon, Dec 29, 2014 at 11:46 PM, Nick Coghlan wrote: > However, even today, it's not particularly difficult to create custom launch > scripts that initialise __main__ with a few different modules based on what > you plan to work on and then set os.environ["PYTHONINSPECT"] = "1" to drop > into the interactive interpreter: > > $ python3 -c "import sys, os; os.environ['PYTHONINSPECT'] = '1'" >>>> sys, os > (, ) >>>> Once again, that's fine for sys and os, which won't take long to import. For this to work with _every_ module on the system, you'd need to couple it with a lazy import mechanism. We had some proposals along those lines recently... what happened to them? ChrisA From steve at pearwood.info Mon Dec 29 14:31:17 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 30 Dec 2014 00:31:17 +1100 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: References: Message-ID: <20141229133107.GD6580@ando.pearwood.info> On Mon, Dec 29, 2014 at 07:46:02AM +0000, Mark Lawrence wrote: > +1 from me as I'm always forgetting the "obvious" imports such as sys > and os when trying things interactively. This is why I have a startup.py file that contains (among other things): import sys, os I'm not sure how you set environment variables under Windows, but under Linux I put this in my .bashrc file: export PYTHONSTARTUP=/home/steve/python/utilities/startup.py and whenever I start an interactive session, I get the common imports that I want, a custom prompt (I use 'py>' rather than '>>>') and assorted other goodies. -- Steven From ncoghlan at gmail.com Mon Dec 29 15:11:20 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 30 Dec 2014 00:11:20 +1000 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: References: Message-ID: On 29 December 2014 at 23:02, Chris Angelico wrote: > On Mon, Dec 29, 2014 at 11:46 PM, Nick Coghlan wrote: > > However, even today, it's not particularly difficult to create custom > launch > > scripts that initialise __main__ with a few different modules based on > what > > you plan to work on and then set os.environ["PYTHONINSPECT"] = "1" to > drop > > into the interactive interpreter: > > > > $ python3 -c "import sys, os; os.environ['PYTHONINSPECT'] = '1'" > >>>> sys, os > > (, '/usr/lib64/python3.4/os.py'>) > >>>> > > Once again, that's fine for sys and os, which won't take long to > import. For this to work with _every_ module on the system, you'd need > to couple it with a lazy import mechanism. We had some proposals along > those lines recently... what happened to them? > The infrastructure landed for 3.5 already, so the custom startup script can also register a lazy loader if it wants to do so: https://docs.python.org/dev/library/importlib.html#importlib.util.LazyLoader (Lazy loading is already possible, since it's just a particular way of using the existing import customisation mechanisms that have been around for over a decade, 3.5 just makes it easier by providing more of the infrastructure to do it directly in the standard library) You can also fairly easily have different startup scripts for different scenarios, and then use explicit imports to bring in any extra pieces you decide you want for a given session. One trick I'll sometimes use myself is to do my playing around in IPython Notebooks, where any common setup code can just go in the first cell. (I'm not sure if IPython Notebook has a "clone existing notebook" feature yet, but that would be a very nice way to have template environments with different setup cells) New language features, or even default interpreter features, are generally an absolute last resort for solving problems - the vast majority of problems can be better solved externally. Incurring the long term maintenance costs of standardisation only makes sense when the perceived pay-off is deemed likely to justify that investment. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Dec 29 15:12:08 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 30 Dec 2014 01:12:08 +1100 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: References: Message-ID: <20141229141208.GE6580@ando.pearwood.info> On Tue, Dec 30, 2014 at 12:02:32AM +1100, Chris Angelico wrote: > On Mon, Dec 29, 2014 at 11:46 PM, Nick Coghlan wrote: > > However, even today, it's not particularly difficult to create custom launch > > scripts that initialise __main__ with a few different modules based on what > > you plan to work on and then set os.environ["PYTHONINSPECT"] = "1" to drop > > into the interactive interpreter: > > > > $ python3 -c "import sys, os; os.environ['PYTHONINSPECT'] = '1'" > >>>> sys, os > > (, ) > >>>> > > Once again, that's fine for sys and os, which won't take long to > import. For this to work with _every_ module on the system, you'd need > to couple it with a lazy import mechanism. We had some proposals along > those lines recently... what happened to them? I just threw this lazy import proxy object together. I haven't tested it extensively, but it seems to work: from types import ModuleType class LazyImporter(ModuleType): def __init__(self, name): self._module = None self.__name__ = name self.__package__ = None def __getattr__(self, name): if self._module is None: module = self._module = __import__(self.__name__) self.__package__ = module.__package__ else: module = self._module return getattr(module, name) def lazy_import(name): from sys import modules if name in modules: return modules[name] else: return LazyImporter(name) Usage: decimal = lazy_import('decimal') decimal.Decimal If the module has already been cached, you get the module itself, otherwise you get a proxy. This was so obvious and simple, I cannot believe I am the first to have thought of it. Perhaps there is something similar that has been extensively used and tested in the real-world that the standard library could steal? -- Steven From fetchinson at googlemail.com Mon Dec 29 15:19:29 2014 From: fetchinson at googlemail.com (Fetchinson .) Date: Mon, 29 Dec 2014 15:19:29 +0100 Subject: [Python-ideas] python on mobile In-Reply-To: <87r3vj2sj1.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87r3vj2sj1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 12/29/14, Stephen J. Turnbull wrote: > Fetchinson . writes: > > > As far as I'm concerned python 2 is perfectly okay. > > Please reconsider this position. For the 70% of the world whose > native language does not fit well into ISO 8859-1, apps developed in > Python 3 with no consideration for their needs are far more likely to > be easily adapted to their environments than those developed in Python > 2. > > If you don't care about the rest of the world, fine, but the rest of > the world isn't going to care about you, and that matters to building > a Python project given that Indians, Chinese, and a smattering of > others account for a large fraction of the "pre-professional" labor we > get through GSoC. They tend to be far more focused on mobile apps > than their mentors in my experience. I completely understand your point but at this stage I'm only thinking of hobby projects that will probably be used by myself, my wife and our family. Nothing mainstream, google playable, etc. For larger scale projects I'd completely agree with you, python 3 is the way to go. Cheers, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From fetchinson at googlemail.com Mon Dec 29 15:17:30 2014 From: fetchinson at googlemail.com (Fetchinson .) Date: Mon, 29 Dec 2014 15:17:30 +0100 Subject: [Python-ideas] python on mobile In-Reply-To: <20141229000923.GZ6580@ando.pearwood.info> References: <20141229000923.GZ6580@ando.pearwood.info> Message-ID: On 12/29/14, Steven D'Aprano wrote: > On Sun, Dec 28, 2014 at 10:30:41PM +0100, Fetchinson . wrote: >> On 12/27/14, Russell Keith-Magee wrote: > >> > This is an entirely >> > reasonable approach if Kivy is what you want at the end of the day - >> > but >> > it's not what *I* want. I've been taking a much more atomic approach. >> > The >> > Kivy toolchain is also anchored in old versions of Python 2; something >> > that >> > definitely needs to be addressed. > > I'm not entirely sure why Russell says this. Kivy claims to support > Python 3: > > http://kivy.org/docs/faq.html#does-kivy-support-python-3-x > > >> As far as I'm concerned python 2 is perfectly okay. If I don't need to >> worry about python 3, what drawback does kivy have, if any, according >> to you? > > Well, you've just lost me as a potential user. I have zero interest in > legacy Python versions for Android. I sympathize with your approach and I'd probably take the same attitude but so far I'm only using python 2 projects and developing for python 2 only (unfortunately). Cheers, Daniel > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From fetchinson at googlemail.com Mon Dec 29 15:25:03 2014 From: fetchinson at googlemail.com (Fetchinson .) Date: Mon, 29 Dec 2014 15:25:03 +0100 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: On 12/29/14, Nick Coghlan wrote: > On 27 Dec 2014 16:05, "Russell Keith-Magee" > wrote: >> Now - I'm not claiming that these are anywhere near production ready - > they're proof of concept at best. However, I completely agree with your > original premise that Python needs a story on mobile platforms, and I'm > interested in working on that story. If you go back and read the > Python-ideas thread from a few months ago, it looks like there was broad > support for the idea, but it would all hinge on the specifics. > > Something that doesn't hinge on the specifics: perhaps it would make sense > to set up a "mobile-sig at python.org" mailing list? That would be a great idea. It would be a place to point to if other people ask the same question and it would send a signal that the developers are serious about this platform (at least to the extent of setting up a mailing list which is a good start already). Cheers, Daniel > It would presumably be Mailman 2 to start off with, but would then migrate > to Mailman 3 & HyperKitty along with the rest of the python.org lists > (hopefully some time in 2015, although we don't have a concrete plan for > that as yet). > > Regards, > Nick. > -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From rosuav at gmail.com Mon Dec 29 15:37:29 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Dec 2014 01:37:29 +1100 Subject: [Python-ideas] python on mobile In-Reply-To: References: <87r3vj2sj1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Tue, Dec 30, 2014 at 1:19 AM, Fetchinson . wrote: > I completely understand your point but at this stage I'm only thinking > of hobby projects that will probably be used by myself, my wife and > our family. Nothing mainstream, google playable, etc. > > For larger scale projects I'd completely agree with you, python 3 is > the way to go. Python 3 isn't just better for large projects. It's better for _every_ project. Just because you think you won't ever need non-ASCII characters doesn't mean you won't ever get any; why not take the easy way out and use Py3? (Case in point: I just sent through a pull request on a project that had a ? (U+2264) in a comment. Py2 complained because it used a non-ASCII character and didn't have a coding cookie. Py3 would have accepted it, though sadly it wasn't a Py3-compatible script for other reasons. ASCII simply isn't enough.) ChrisA From fetchinson at googlemail.com Mon Dec 29 15:23:29 2014 From: fetchinson at googlemail.com (Fetchinson .) Date: Mon, 29 Dec 2014 15:23:29 +0100 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: On 12/29/14, Russell Keith-Magee wrote: > On Mon, Dec 29, 2014 at 5:30 AM, Fetchinson . > wrote: >> >> On 12/27/14, Russell Keith-Magee wrote: >> >> > So, that's what I've been working on. Since I posted to Python-ideas in >> > October, I've been focussing on getting the patches necessary for >> Python's >> > own build system. However, I've been distracted by work life etc, so I >> > haven't made as much progress as I would have liked. I'm hoping the >> > Christmas/New year break will give me a chance to make some headway. >> > >> > As far as Kivy goes - I have tinkered with Kivy; I've even used Kivy's >> > tools as a starting point for some of my own efforts. However, to me, >> Kivy >> > is focussing too far up the stack - their toolchain appears to be based >> on >> > getting *Kivy* working on mobile, not *Python*. >> >> What do you mean by this exactly? I thought kivy = python + some >> libraries. So if kivy works on android that necessarily means that >> python itself works as well, doesn't it? > > > Yes and no. > > Yes, Kivy is just a set of Python libraries, and a Kivy program is just a > Python program. > > However, "Hello World" in Kivy isn't 'print("Hello world")'. It's setting > up a whole Kivy stack, and starting a Kivy runloop - and the Kivy toolchain > is focussed on getting you to the point where you can start that runloop. > > Even if you work around that, and just use Python "out of the box", you're > going to be carrying at least some of the overhead of Kivy. Thanks, got it. >> I thought I'd use kivy even >> if I don't need any of their fancy libraries only a basic python >> installation, but you'd say this is not a good approach? >> > > It would probably work - I haven't tried, so I can't comment from > experience. However, if you go down this path, you won't have a "Basic > python installation" - you'll have "Python + kivy", because their tools > don't give you the option to *not* install the Kivy parts (at least, not as > far as I could work out). > > From my experience, getting Python compiled using Kivy's tools wasn't that > hard - but getting Kivy compiled was a PITA, because Kivy has a bunch of > other dependencies, like sound support libraries, OpenGL libraries, Cython, > and so on. These are all essential for *Kivy*, but not for Python, and not > for my purposes either. > > Ultimately, I'm working on something that is, at one level, a competitor to > Kivy. I don't want my install instructions to be "install Kivy, now throw > all that stuff away and use Toga"; and I don't want my users to have to go > through the same drama of getting Kivy's whole dependency chain compiled. I > *just* want Python, and then I'll decide what other pieces I want. That's > why I'm trying to get the build fixes into Python's core - I want *Python* > to be compilable for mobile platforms, and then leave the decision of which > libraries you want to use up to the end developer. Okay, now I understand your project better. And I'd say what I'd like to have is exactly what you are aiming for: having the more or less same python installation as on linux, windows, macos, but this time on android (I don't really care for ios). There is one additional feature I'd like to see though: hardware integration: accessing incoming/outgoing calls, camera, sensors, etc. But I guess that would be a secondary goal, I completely agree with you that first an ordinary python stack would be great. Cheers, Daniel > Yours, > Russ Magee %-) > -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From ram at rachum.com Mon Dec 29 17:54:46 2014 From: ram at rachum.com (Ram Rachum) Date: Mon, 29 Dec 2014 18:54:46 +0200 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin Message-ID: Hi guys, I want to have a method of `pathlib.Path` that would send a file to the recycle bin. (i.e. soft delete.) What do you think about adding this? I see there's a PyPI package `Send2Trash` that does this, but it would be nice if this was in the standard library. Thanks, Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Dec 29 18:53:26 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 29 Dec 2014 18:53:26 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin References: Message-ID: <20141229185326.19663a01@fsol> On Mon, 29 Dec 2014 18:54:46 +0200 Ram Rachum wrote: > Hi guys, > > I want to have a method of `pathlib.Path` that would send a file to the > recycle bin. (i.e. soft delete.) > > What do you think about adding this? > > I see there's a PyPI package `Send2Trash` that does this, but it would be > nice if this was in the standard library. I think the corresponding low-level function to the os module would be a good start (the hard part is probably to write the cross-platform code, anyway). Regards Antoine. From abarnert at yahoo.com Mon Dec 29 19:58:31 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 29 Dec 2014 19:58:31 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <20141229185326.19663a01@fsol> References: <20141229185326.19663a01@fsol> Message-ID: <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> On Dec 29, 2014, at 18:53, Antoine Pitrou wrote: > On Mon, 29 Dec 2014 18:54:46 +0200 > Ram Rachum wrote: >> Hi guys, >> >> I want to have a method of `pathlib.Path` that would send a file to the >> recycle bin. (i.e. soft delete.) >> >> What do you think about adding this? >> >> I see there's a PyPI package `Send2Trash` that does this, but it would be >> nice if this was in the standard library. > > I think the corresponding low-level function to the os module would be > a good start (the hard part is probably to write the cross-platform > code, anyway). There isn't really a low-level function for it on any platform but maybe Windows, except rename, which is already there. On OS X, unless you want to use higher-level CoreFoundation APIs that require a runloop, you just look up the name of the trash directory (which I believe is theoretically localized but actually always ~/.Trash on the drive with your user directory, $mnt/.Trash otherwise) and call rename. On Linux, and some other *nix platforms, you assume XDG directory layout and call rename. On other platforms, I don't know if there even is a way to do it. And I don't know what the right thing to do on OS X would be for a file on a different filesystem than the user's home that doesn't have its own .Trash or is mounted read-only or no-trash. Finder gives you a popup in each of those cases asking if you want to delete it immediately because it can't be trashed. And similarly for Linux, but the details are different (especially since you might not even have a home trash). I'd guess Send2Trash is a good place to at least start, but if this is going in the stdlib someone needs to make sure it meets Apple HIG rules and XDG standards. > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From phd at phdru.name Mon Dec 29 20:35:43 2014 From: phd at phdru.name (Oleg Broytman) Date: Mon, 29 Dec 2014 20:35:43 +0100 Subject: [Python-ideas] python on mobile In-Reply-To: References: <87r3vj2sj1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20141229193543.GA11678@phdru.name> Hi! On Tue, Dec 30, 2014 at 01:37:29AM +1100, Chris Angelico wrote: > On Tue, Dec 30, 2014 at 1:19 AM, Fetchinson . wrote: > > I completely understand your point but at this stage I'm only thinking > > of hobby projects that will probably be used by myself, my wife and > > our family. Nothing mainstream, google playable, etc. > > > > For larger scale projects I'd completely agree with you, python 3 is > > the way to go. > > Python 3 isn't just better for large projects. It's better for _every_ > project. Just because you think you won't ever need non-ASCII > characters doesn't mean you won't ever get any; why not take the easy > way out and use Py3? (Case in point: I just sent through a pull > request on a project that had a ? (U+2264) in a comment. Py2 > complained because it used a non-ASCII character and didn't have a > coding cookie. Py3 would have accepted it, though sadly it wasn't a > Py3-compatible script for other reasons. ASCII simply isn't enough.) > > ChrisA I regularly read and produce programs and texts (and text fields in binary files) in 4 non-ascii encodings -- koi8-r, utf-8, cp1251 and cp866, less frequently in utf-16. Python 2 doesn't stand on my way. Python 3 could be better -- I simple don't know -- but Py2 is still good enough. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From fetchinson at googlemail.com Mon Dec 29 21:48:07 2014 From: fetchinson at googlemail.com (Fetchinson .) Date: Mon, 29 Dec 2014 21:48:07 +0100 Subject: [Python-ideas] python on mobile In-Reply-To: References: <87r3vj2sj1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: >> I completely understand your point but at this stage I'm only thinking >> of hobby projects that will probably be used by myself, my wife and >> our family. Nothing mainstream, google playable, etc. >> >> For larger scale projects I'd completely agree with you, python 3 is >> the way to go. > > Python 3 isn't just better for large projects. It's better for _every_ > project. Just because you think you won't ever need non-ASCII > characters doesn't mean you won't ever get any; why not take the easy > way out and use Py3? (Case in point: I just sent through a pull > request on a project that had a ? (U+2264) in a comment. Py2 > complained because it used a non-ASCII character and didn't have a > coding cookie. Py3 would have accepted it, though sadly it wasn't a > Py3-compatible script for other reasons. ASCII simply isn't enough.) I'm not sure I understand you. Are you saying unicode support is not there in python 2? That's certainly not the case. I believe that python 3 is more *convenient* but certainly is not necessary, it's perfectly possible to handle any character set in python 2 as well. Or is there something I'm missing? Cheers, Daniel > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From random832 at fastmail.us Mon Dec 29 22:15:27 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Mon, 29 Dec 2014 16:15:27 -0500 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> Message-ID: <1419887727.1378173.207731021.68801678@webmail.messagingengine.com> On Mon, Dec 29, 2014, at 13:58, Andrew Barnert wrote: > There isn't really a low-level function for it on any platform but maybe > Windows, except rename, which is already there. The Windows function for this is SHFileOperation, which isn't really low-level (but doesn't require a message loop). > On OS X, unless you want to use higher-level CoreFoundation APIs that > require a runloop, you just look up the name of the trash directory > (which I believe is theoretically localized but actually always ~/.Trash > on the drive with your user directory, $mnt/.Trash otherwise) Is it not uid-specific? Windows and XDG trash directories are uid-specific. A google search shows that it's $mnt/.Trashes/$uid. > And I don't know what the right thing to do on OS X would be for a file > on a different filesystem than the user's home that doesn't have its own > .Trash or is mounted read-only or no-trash. If it's mounted read-only, you have a bigger problem. Another possible issue could be: bad permission on trash directory, bad permission or nonexistent parent directory that trash directory must be created in (home or .Trashes). These are also all issues that must be faced for XDG. XDG has a database of original location and deleted date for files in the trash - does OSX? (according to the tool I found, it does, at least for original location, but only works if you use Finder to trash the files) > Finder gives you a popup in > each of those cases asking if you want to delete it immediately because > it can't be trashed. And similarly for Linux, but the details are > different (especially since you might not even have a home trash). Another option would be to just script Finder with osascript. The tool http://hasseg.org/trash/ has objc code to "directly" script Finder without actually going through osascript. When it does not use Finder, it uses FSMoveObjectToTrashSync, which seems to be depreciated. From ethan at stoneleaf.us Mon Dec 29 23:53:49 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 29 Dec 2014 14:53:49 -0800 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: <54A1DB7D.5090807@stoneleaf.us> On 12/29/2014 02:32 AM, Nick Coghlan wrote: > > Something that doesn't hinge on the specifics: perhaps it would make sense to set up a "mobile-sig at python.org > " mailing list? +1 -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From abarnert at yahoo.com Tue Dec 30 00:46:28 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 30 Dec 2014 00:46:28 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <1419887727.1378173.207731021.68801678@webmail.messagingengine.com> References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <1419887727.1378173.207731021.68801678@webmail.messagingengine.com> Message-ID: On Dec 29, 2014, at 22:15, random832 at fastmail.us wrote: > On Mon, Dec 29, 2014, at 13:58, Andrew Barnert wrote: > >> And I don't know what the right thing to do on OS X would be for a file >> on a different filesystem than the user's home that doesn't have its own >> .Trash or is mounted read-only or no-trash. > > If it's mounted read-only, you have a bigger problem. Well, yes, but the point is that Finder gives you a special message for this rather than trying to move it and giving you a permissions error. And more importantly, if it's mounted no-trash or doesn't have a trash can, you don't have a bigger problem, but you still have this problem, so this problem needs to be solved. > Another possible issue could be: bad permission on trash directory, bad > permission or nonexistent parent directory that trash directory must be > created in (home or .Trashes). These are also all issues that must be > faced for XDG. I believe all of those issues could just raise the normal exception, because Finder treats them as normal errors--that is, rather than giving you the special "can't trash; delete?" message, it tries, fails, asks you to authenticate as an administrator, and tries again. > XDG has a database of original location and deleted date for files in > the trash - does OSX? (according to the tool I found, it does, at least > for original location, but only works if you use Finder to trash the > files) That's a whole other issue. I believe that if you use the higher-level APIs instead of POSIX functions they take care of the desktop database and alias/bookmark info and all that stuff. If not... It may happen automatically (at least if you're on an HFS+ drive) but it may not. >> Finder gives you a popup in >> each of those cases asking if you want to delete it immediately because >> it can't be trashed. And similarly for Linux, but the details are >> different (especially since you might not even have a home trash). > > Another option would be to just script Finder with osascript. The tool > http://hasseg.org/trash/ has objc code to "directly" script Finder > without actually going through osascript. If you want to script Finder, the easy way to do it (well, as easy as it gets without Appscript) is ScriptingBridge and/or the lower-level Cocoa AppleScript APIs (via PyObjC, but none of that is going to go into the stdlib, which means shutil needs to be written partly in ObjC), but that will raise the minimum OS X version, because 10.6 didn't have NSAppleScript and friends yet. (And the Carbon AEM calls are deprecated, of course.) This could also make it more difficult for Python apps to be sandboxed and AppStore'd. But the biggest problem is that you're asking Finder to do it. If you don't have permission to trash a file, instead if getting an error, you'll get an auth dialog from Finder. If you trash a bunch of files you'll get a progress bar in Finder (and your script will get a timeout from the synchronous AppleEvent and not know if the trashing succeeded). Worse, if you don't have a Finder to talk to (because you're running in an ssh shell, or in a LaunchDaemon or a detached script, or because you're using a third-party Finder replacement like PathFinder) you won't be able to trash anything. There are plenty of applications that's reasonable for, but I don't think it's reasonable stdlib behavior. > When it does not use Finder, > it uses FSMoveObjectToTrashSync, which seems to be depreciated. Yeah, that's Carbon; you're supposed to use the Cocoa/CF APIs instead. NSWorkspace had one method that's guaranteed to do exactly the same thing as trashing in Finder--which it does by asking Finder--and a more general "perform operations on files" method that can be used for trashing without the Finder, but there's no way to control whether it's synchronous or asynchronous, and requires a runloop in case it's async. Anyway, I don't think these details change the point: There is no low-level function to be added to os. There may be a high-level function to be added to shutil, but it's not trivial. From greg.ewing at canterbury.ac.nz Tue Dec 30 00:19:03 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 30 Dec 2014 12:19:03 +1300 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> Message-ID: <54A1E167.1050605@canterbury.ac.nz> Andrew Barnert wrote: > you just look up the name of the trash directory (which I believe > is theoretically localized but actually always ~/.Trash on the drive with > your user directory, $mnt/.Trash otherwise) and call rename. You may also need to invent a unique name if there's already a file in the trash with the same name. > And I don't know what the right thing to do on OS X would be for a file on a > different filesystem than the user's home that doesn't have its own .Trash or > is mounted read-only or no-trash. Raising an exception would seem to be a reasonable thing to do in that case. -- Greg From rosuav at gmail.com Tue Dec 30 04:51:13 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Dec 2014 14:51:13 +1100 Subject: [Python-ideas] python on mobile In-Reply-To: References: <87r3vj2sj1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Tue, Dec 30, 2014 at 7:48 AM, Fetchinson . wrote: >> Python 3 isn't just better for large projects. It's better for _every_ >> project. Just because you think you won't ever need non-ASCII >> characters doesn't mean you won't ever get any; why not take the easy >> way out and use Py3? (Case in point: I just sent through a pull >> request on a project that had a ? (U+2264) in a comment. Py2 >> complained because it used a non-ASCII character and didn't have a >> coding cookie. Py3 would have accepted it, though sadly it wasn't a >> Py3-compatible script for other reasons. ASCII simply isn't enough.) > > I'm not sure I understand you. Are you saying unicode support is not > there in python 2? That's certainly not the case. I believe that > python 3 is more *convenient* but certainly is not necessary, it's > perfectly possible to handle any character set in python 2 as well. You can handle data in any character set using Py2. Explicitly convert to Unicode, work with that, off you go. No problem. You can do that in any language that has encode and decode libraries. The difference is that in Py2, you can forget that you have to encode and decode, and your program will work on ASCII text - and then you'll blame "funny characters" for breaking your program. In Py3, you'll get it right straight off, and it'll work for all of Unicode equally. Plus, Py3 makes the default source code encoding UTF-8 instead of ASCII, so you can have non-ASCII comments without having to slap a coding cookie at the top of the file; and since source code is Unicode text rather than bytes, you can reliably use non-ASCII variable names, module names (I think), and even, with a few tweaks, alternate language keywords. Basically, where Py2 gives you the tools to make a Unicode-aware program, Py3 just makes your program Unicode-aware for you. We're definitely off-topic for python-ideas now. ChrisA From steve at pearwood.info Tue Dec 30 05:58:17 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 30 Dec 2014 15:58:17 +1100 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: <1dbf3f09-ac7d-48f8-90e2-abcc73679527@googlegroups.com> References: <20141229141208.GE6580@ando.pearwood.info> <1dbf3f09-ac7d-48f8-90e2-abcc73679527@googlegroups.com> Message-ID: <20141230045817.GB24472@ando.pearwood.info> On Mon, Dec 29, 2014 at 08:39:41AM -0800, Rick Johnson wrote: > > > On Monday, December 29, 2014 8:12:49 AM UTC-6, Steven D'Aprano wrote: > > I just threw this lazy import proxy object together. I haven't tested it > > extensively, but it seems to work: [...] > All you've done is to replace "import decimal" with "decimal = > lazy_import('decimal')" -- what's the advantage? It delays the actual import of the module until you try to use it. For decimal, that's not much of an advantage, but some modules are quite expensive to import the first time, and you might not want to pay that cost at application start-up. Personally, I have no use for this, but people have been talking about lazy importing recently, and I wanted to see how hard it would be to do. -- Steven From rosuav at gmail.com Tue Dec 30 06:36:32 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Dec 2014 16:36:32 +1100 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: <20141230045817.GB24472@ando.pearwood.info> References: <20141229141208.GE6580@ando.pearwood.info> <1dbf3f09-ac7d-48f8-90e2-abcc73679527@googlegroups.com> <20141230045817.GB24472@ando.pearwood.info> Message-ID: On Tue, Dec 30, 2014 at 3:58 PM, Steven D'Aprano wrote: > For decimal, that's not much of an advantage, but some modules are quite > expensive to import the first time, and you might not want to pay that > cost at application start-up. Even more so if this is done for _every module in the system_. I'm thinking of testing this out; creating a lazy import object for every single findable module, although before I start that, I'll see if I can hunt down a tab-completion routine rather than manually searching sys.path. In theory, enumerating modules shouldn't take too long (it's not like I have network mounts in PYTHONPATH), so if that's sufficiently fast, I might toss that into a permanent startup script, rather than having my current "import on NameError" trap. Small downside: This *does* require that the modules all exist at process start and never get renamed or deleted. I'm sure that'll be a problem in some obscure case somewhere, but probably not a practical issue :) ChrisA From russell at keith-magee.com Tue Dec 30 06:50:23 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Tue, 30 Dec 2014 13:50:23 +0800 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: On Mon, Dec 29, 2014 at 6:32 PM, Nick Coghlan wrote: > > > On 27 Dec 2014 16:05, "Russell Keith-Magee" > wrote: > > Now - I'm not claiming that these are anywhere near production ready - > they're proof of concept at best. However, I completely agree with your > original premise that Python needs a story on mobile platforms, and I'm > interested in working on that story. If you go back and read the > Python-ideas thread from a few months ago, it looks like there was broad > support for the idea, but it would all hinge on the specifics. > > Something that doesn't hinge on the specifics: perhaps it would make sense > to set up a "mobile-sig at python.org" mailing list? > > It would presumably be Mailman 2 to start off with, but would then migrate > to Mailman 3 & HyperKitty along with the rest of the python.org lists > (hopefully some time in 2015, although we don't have a concrete plan for > that as yet). > Sure - if it's not too much effort to establish. I'm not sure exactly what would be discussed there, especially in the short term when the big ticket item is getting the patches into the main Python tree. However, mobile is a significant platform these days, and it can't hurt to have a place where the various players can co-ordinate and collaborate where possible. Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Tue Dec 30 07:41:04 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Tue, 30 Dec 2014 06:41:04 +0000 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: References: Message-ID: On 29/12/2014 11:43, Nathaniel Smith wrote: > On 29 Dec 2014 07:46, "Mark Lawrence" > > wrote: > > > > On 28/12/2014 16:42, Chris Angelico wrote: > >> > >> I mainly did up this patch to see how hard it would be, and now it's > >> turned out to be fairly simple, I'm curious as to whether it would > >> actually be useful to people. > >> > >> At the point where a global/builtin name lookup is about to raise > >> NameError, first try calling a Python function, along the same lines > >> as __getattr__. If that function hasn't been defined, raise NameError > >> as normal; but if it has, let it either raise NameError or return some > >> object, which is then used as if the name had been bound to it. > >> > >> Patch is here: > >> http://bugs.python.org/issue23126 > >> > >> The idea is to allow convenient interactive use; auto-importing > >> modules is easy, and importing names from modules ("exp" --> math.exp) > >> can be done easily enough too, given a list of modules to try. > >> > >> It's probably not a good idea to use this in application code, and I > >> definitely wouldn't encourage it in library code, but it has its uses > >> interactively. > >> > >> Thoughts? > >> > >> ChrisA > > > > > > +1 from me as I'm always forgetting the "obvious" imports such as sys > and os when trying things interactively. > > I just set up a PYTHONSTARTUP script to pre-import the obvious things > and that fixed this problem for me. You'd still have to make a startup > script to set up the hook, and it's not like 'import os, sys' will > appreciably affect startup time. > > -n > I should really get around to that, it's been at the back of my mind to put it on the TODO list for about 12 years :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From wes.turner at gmail.com Tue Dec 30 07:39:33 2014 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 30 Dec 2014 00:39:33 -0600 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: Lots of great links in this thread: It could be useful to update the Android wiki page * https://wiki.python.org/moin/Android # I had started an awesome-python-android/README.rst on GitHub, # but I think I might've pushed over it and now cannot find it. On Sat, Dec 27, 2014 at 12:04 AM, Russell Keith-Magee < russell at keith-magee.com> wrote: > Hi Daniel, > > I've been working on this problem recently; I posted about the idea to > this very list a few months ago: > > https://mail.python.org/pipermail/python-ideas/2014-October/029856.html > > At the time I posted, I had posted Python builds that work on both Android > and iOS: > > https://github.com/pybee/Python-iOS-support > https://github.com/pybee/Python-Android-support > > and template projects that get you to "hello world" on a mobile platform > without much effort. > > https://github.com/pybee/Python-iOS-template > https://github.com/pybee/Python-Android-template > > I've also tackled the cross-language bridging issue: > > https://github.com/pybee/rubicon-objc > https://github.com/pybee/rubicon-java > > and wrapped native widgets up into a cross-platform widget library: > > http://pybee.org/toga/ > > Now - I'm not claiming that these are anywhere near production ready - > they're proof of concept at best. However, I completely agree with your > original premise that Python needs a story on mobile platforms, and I'm > interested in working on that story. If you go back and read the > Python-ideas thread from a few months ago, it looks like there was broad > support for the idea, but it would all hinge on the specifics. > > So, that's what I've been working on. Since I posted to Python-ideas in > October, I've been focussing on getting the patches necessary for Python's > own build system. However, I've been distracted by work life etc, so I > haven't made as much progress as I would have liked. I'm hoping the > Christmas/New year break will give me a chance to make some headway. > > As far as Kivy goes - I have tinkered with Kivy; I've even used Kivy's > tools as a starting point for some of my own efforts. However, to me, Kivy > is focussing too far up the stack - their toolchain appears to be based on > getting *Kivy* working on mobile, not *Python*. This is an entirely > reasonable approach if Kivy is what you want at the end of the day - but > it's not what *I* want. I've been taking a much more atomic approach. The > Kivy toolchain is also anchored in old versions of Python 2; something that > definitely needs to be addressed. > > Ultimately, I want to get these patches in to Python's trunk; but I'm > still a way off that being a reality. If you're interested in collaborating > and helping me get to that point, please get in touch - there's plenty of > work to do, and if you're keen to help, I'm happy to share what I've got so > far. > > Yours, > Russ Magee %-) > > On Fri, Dec 26, 2014 at 6:34 PM, Fetchinson . > wrote: >> >> Hi all, maybe the best list would be python-dev but I don't dare >> waking the sleeping lions there :) >> >> So my question is this: is there a coherent mobile strategy among core >> dev people? I mean you guys release python for linux/macos/windows and >> the question is if there are any plans to do the same for a mobile >> platform. It doesn't have to be android or ios just anything that the >> core dev team chooses and sticks with. >> >> I've been developing python apps for smartphones (mostly hobby >> projects though) using sl4a but that seems like is dead. Now people >> suggest me using kivy which seems to be alive but who knows how long. >> There are some other projects qpython, etc, which are small and >> equally not so reliable at least looking from the outside. Is kivy now >> an officially blessed distribution? Since google was so integral to >> both python (through employing Guido) and android I'd think it would >> make sense for google to have an official python environment for >> android in cooperation with the python dev team. >> >> Does the PSF has an opinion on this? It would be great if there would >> be something for mobile phones that we could rely on not going away >> just as with linux/macos/windows. >> >> Or there are some issues which preclude this from the start? >> >> Cheers, >> Daniel >> >> >> -- >> Psss, psss, put it down! - http://www.cafepress.com/putitdown >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anthony at xtfx.me Tue Dec 30 16:51:32 2014 From: anthony at xtfx.me (C Anthony Risinger) Date: Tue, 30 Dec 2014 09:51:32 -0600 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: <20141230045817.GB24472@ando.pearwood.info> References: <20141229141208.GE6580@ando.pearwood.info> <1dbf3f09-ac7d-48f8-90e2-abcc73679527@googlegroups.com> <20141230045817.GB24472@ando.pearwood.info> Message-ID: On Mon, Dec 29, 2014 at 10:58 PM, Steven D'Aprano wrote: > On Mon, Dec 29, 2014 at 08:39:41AM -0800, Rick Johnson wrote: > > > > > > On Monday, December 29, 2014 8:12:49 AM UTC-6, Steven D'Aprano wrote: > > > > I just threw this lazy import proxy object together. I haven't tested it > > > extensively, but it seems to work: > [...] > > All you've done is to replace "import decimal" with "decimal = > > lazy_import('decimal')" -- what's the advantage? > > It delays the actual import of the module until you try to use it. > > For decimal, that's not much of an advantage, but some modules are quite > expensive to import the first time, and you might not want to pay that > cost at application start-up. > > Personally, I have no use for this, but people have been talking about > lazy importing recently, and I wanted to see how hard it would be to do. the startup savings are significant, eg. xacto (CLI generation tool) uses lazy importing to rapidly import all "tools" without triggering cascading imports: https://github.com/xtfxme/xacto/blob/master/xacto/__init__.py#L240 ...for the purpose of scanning their signatures and generating argparse subcommands. without aggressive lazy imports, a simple `./mytool --help` can take a very long time. the method xacto uses is a bit different, and imo at least, superior to using proxy/lazy objects; instead of loading a proxy object at import-time, xacto executes code in a namespace that implements `__missing__`, therefore allowing NameErrors to become `namespace.__missing__(key)`: * install meta_path hook * code tries to import something *not* already imported: from foo import bar as baz * meta importer returns a special str representing the import instead: 'foo:bar' * interpreter tries to update namespace: namespace.__setitem__('baz', 'foo:bar') * namespace detects/remembers the details, but DISCARDS THE KEY! * next time code accesses global 'baz': namespace.__missing__('baz') * namespace performs import and updates itself ...this pattern allows the module *itself* to be lazy imported, rather than a proxy, so long at the module is only imported and not interacted with (or more precisely, referenced in any way) and is also zero-overhead after the first reference. sadly this only half works in python2 (but to my surprise, works fine in python 3.4!). in python2, module-level code will trigger __missing__, but functions (with their __globals__ bound to a custom dict) WILL NOT: python 2.7: >>> from collections import defaultdict >>> ns = defaultdict(list) >>> ne = eval('lambda: name_error', ns) >>> assert ne.__globals__ is ns >>> ne() Traceback (most recent call last): File "", line 1, in File "", line 1, in NameError: global name 'name_error' is not defined python 3.4: >>> from collections import defaultdict >>> ns = defaultdict(list) >>> ne = eval('lambda: name_error', ns) >>> assert ne.__globals__ is ns >>> ne() [] ...just some ancillary info some folks may find useful/interesting :) -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Dec 30 17:58:09 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 30 Dec 2014 08:58:09 -0800 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: On Mon, Dec 29, 2014 at 10:39 PM, Wes Turner wrote: > >> As far as Kivy goes - I have tinkered with Kivy; I've even used Kivy's >> tools as a starting point for some of my own efforts. However, to me, Kivy >> is focussing too far up the stack - their toolchain appears to be based on >> getting *Kivy* working on mobile, not *Python*. This is an entirely >> reasonable approach if Kivy is what you want at the end of the day - but >> it's not what *I* want. I've been taking a much more atomic approach. The >> Kivy toolchain is also anchored in old versions of Python 2; something that >> definitely needs to be addressed. >> > IIUC, how Kivy works is not to put a python interpreter on the platform, but rather to run all your code through Cython, creating a C program that used the python runtime as a library, essentially. So there is no python code involved at run time. So the Kivy lib aside, is this a good way to go on Mobile? I can't image it's too hard to remove the kivy UI stuff if you don't want to use it -- even if the current tools all include all that by default. But is it better to target getting Python running on Mobile just like it does on the desktop: you have a python interpreter, a bunch of python modules, and you bundle it all up with a executable launcher of some sort, al la py2exe, py2app, etc.? Something to keep in mind is that that way most Python code is run may not make sense (or even be an option with Apple's rules, anyway): you have a python environment installed, and install a bunch of application code on top of that, and run that. Makes sense to system scripts, and web apps and that like, but really doesn't make sense for "Apps". What I'm getting at is that the final goal for what exactly we want people to be be able to distribute may effect decisions about what to build and how to build it. BTW, one thing I'd like to be able to build is what I call a "BILS" (Browser Interface, Local Server) app: you run a web server bundled up in an app that points itself to the browser component for all the UI. We've been very successful doing this on Windows and OS-X, as a way to provide a self contained desktop app, and a hosted Web App with almost entirely one code base, and an identical user interface. (http://cameochemicals.noaa.gov/ and http://response.restoration.noaa.gov/cameochemicals) (Python / Pyramid on the back end, desktop is the local browser hosted by a wxPython wrapper) We'd like to be able to port that to iOS and Android, but there doesn't appear to be a solid python option that we could run a Pyramid web server on Mobile at this point -- maybe I'm wrong) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From fetchinson at googlemail.com Tue Dec 30 20:53:12 2014 From: fetchinson at googlemail.com (Fetchinson .) Date: Tue, 30 Dec 2014 20:53:12 +0100 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: >> Something that doesn't hinge on the specifics: perhaps it would make >> sense to set up a "mobile-sig at python.org" mailing list? >> >> It would presumably be Mailman 2 to start off with, but would then >> migrate >> to Mailman 3 & HyperKitty along with the rest of the python.org lists >> (hopefully some time in 2015, although we don't have a concrete plan for >> that as yet). >> > Sure - if it's not too much effort to establish. > > I'm not sure exactly what would be discussed there, especially in the short > term when the big ticket item is getting the patches into the main Python > tree. I'd think the core devs would be far more likely to include a patch if it comes with a big red note on it saying "it was discussed on mobile-sig and was blessed by all participants" as opposed to a random patch by a random person. Cheers, Daniel > However, mobile is a significant platform these days, and it can't > hurt to have a place where the various players can co-ordinate and > collaborate where possible. > > Yours, > Russ Magee %-) > -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From matthew.i.frank at intel.com Tue Dec 30 21:10:27 2014 From: matthew.i.frank at intel.com (Frank, Matthew I) Date: Tue, 30 Dec 2014 20:10:27 +0000 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: <8FEF00DDD6C58843BF4ED3C1DCF80E9F6E116C92@FMSMSX106.amr.corp.intel.com> On Tue, Dec 30, 2014 at 1058, Chris Barker chris.barker at noaa.gov wrote: > BTW, one thing I'd like to be able to build is what I call a "BILS" (Browser Interface, Local Server) app: > you run a web server bundled up in an app that points itself to the browser component for all the UI. > We've been very successful doing this on Windows and OS-X, as a way to provide a self contained > desktop app, and a hosted Web App with almost entirely one code base, and an identical user interface. > > (http://cameochemicals.noaa.gov/ and http://response.restoration.noaa.gov/cameochemicals) > > (Python / Pyramid on the back end, desktop is the local browser hosted by a wxPython wrapper) > > We'd like to be able to port that to iOS and Android, but there doesn't appear to be a solid python > option that we could run a Pyramid web server on Mobile at this point -- maybe I'm wrong) This is what I've been trying to do as well, for Android. I'm particularly interested in running an iPython server connected to the local Android browser. For iPad there is an example of someone having achieved exactly this: http://computableapp.com/. But I don't know if they've actually made any of their porting knowledge public. For Android I've been focusing on submitting patches to CPython 3.4/3.5 for things that can be argued to be "just cross-compilation bugs" in the Cpython source. For example: http://bugs.python.org/issue21668 just fixes a missing -lm flag during compilation (that doesn't matter on Linux/MacOS/Windows, but is technically required, and causes problems with the Android dynamic loader if it is missing). http://bugs.python.org/issue20306 just takes care of the fact that pw_gecos isn't actually required by Posix. -Matt From liam.marsh.home at gmail.com Tue Dec 30 21:31:51 2014 From: liam.marsh.home at gmail.com (Liam Marsh) Date: Tue, 30 Dec 2014 21:31:51 +0100 Subject: [Python-ideas] pep 397 Message-ID: hello, *the pep 397 says that any python script is able to choose the language version which will run it, between all the versions installed on the computer, using on windows a launcher in the "C:\windows" folder.* can the idle version be chosen like this too, or can the idle "run" command do it? (except it is already like this and I have a problem) thank you, and have a nice day/evening! (and sorry if this sounds irritating) -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Tue Dec 30 21:33:44 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Tue, 30 Dec 2014 15:33:44 -0500 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <1419887727.1378173.207731021.68801678@webmail.messagingengine.com> Message-ID: <1419971624.1772262.208078313.2458ACDC@webmail.messagingengine.com> On Mon, Dec 29, 2014, at 18:46, Andrew Barnert wrote: > I believe all of those issues could just raise the normal exception, > because Finder treats them as normal errors--that is, rather than giving > you the special "can't trash; delete?" message, it tries, fails, asks you > to authenticate as an administrator, and tries again. That doesn't make sense - the problem is with the state of the trash directory (it may be in a state that you can access, but the "wrong" one, e.g. one that would allow other people to delete your files - I don't know if this is a problem for OSX but it is for XDG), not your access to the file being deleted. From ronaldoussoren at mac.com Tue Dec 30 22:36:53 2014 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 30 Dec 2014 22:36:53 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <54A1E167.1050605@canterbury.ac.nz> References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> Message-ID: > On 30 Dec 2014, at 00:19, Greg Ewing wrote: > > Andrew Barnert wrote: >> you just look up the name of the trash directory (which I believe >> is theoretically localized but actually always ~/.Trash on the drive with >> your user directory, $mnt/.Trash otherwise) and call rename. > > You may also need to invent a unique name if there's already > a file in the trash with the same name. And whatever else the api does for you? IMHO its better to use the API for this on OSX, such as FSMoveObjectToTrashSync. Ronald From ronaldoussoren at mac.com Tue Dec 30 22:43:06 2014 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 30 Dec 2014 22:43:06 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <1419971624.1772262.208078313.2458ACDC@webmail.messagingengine.com> References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <1419887727.1378173.207731021.68801678@webmail.messagingengine.com> <1419971624.1772262.208078313.2458ACDC@webmail.messagingengine.com> Message-ID: <4A0EB1E7-BDC4-4922-B3C6-B2FBBBA5A8B5@mac.com> > On 30 Dec 2014, at 21:33, random832 at fastmail.us wrote: > > On Mon, Dec 29, 2014, at 18:46, Andrew Barnert wrote: >> I believe all of those issues could just raise the normal exception, >> because Finder treats them as normal errors--that is, rather than giving >> you the special "can't trash; delete?" message, it tries, fails, asks you >> to authenticate as an administrator, and tries again. > > That doesn't make sense - the problem is with the state of the trash > directory (it may be in a state that you can access, but the "wrong" > one, e.g. one that would allow other people to delete your files - I > don't know if this is a problem for OSX but it is for XDG), not your > access to the file being deleted. On OSX there are at least two reasons why the Finder can prompt you when you try to move a file to the trash. The first is a source location that doesn?t support the trash (such as a fileserver), the Finder will ask if should just permanently remove the file. The second is when you don?t have permissions to remove the file, the Finder will then ask if it should retry with Administrator privileges (and will prompt you for a username and password if you confirm). Ronald From rosuav at gmail.com Tue Dec 30 23:38:39 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 31 Dec 2014 09:38:39 +1100 Subject: [Python-ideas] pep 397 In-Reply-To: References: Message-ID: On Wed, Dec 31, 2014 at 7:31 AM, Liam Marsh wrote: > the pep 397 says that any python script is able to choose the language > version which will run it, between all the versions installed on the > computer, using on windows a launcher in the "C:\windows" folder. > > can the idle version be chosen like this too, or can the idle "run" command > do it? I don't know of a way off-hand. However, if the information is already there in the shebang (as per PEP 397), it ought to be possible to rig up a launcher/chooser script. This might be something to discuss on python-list at python.org, rather than here; there are a lot more people on python-list than there are here on -ideas, and I suspect you don't need any changes to the language to make this work. It's also entirely possible that someone there already knows exactly what you need to do! ChrisA From abarnert at yahoo.com Wed Dec 31 00:12:01 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 31 Dec 2014 00:12:01 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> Message-ID: On Dec 30, 2014, at 22:36, Ronald Oussoren wrote: > >> On 30 Dec 2014, at 00:19, Greg Ewing wrote: >> >> Andrew Barnert wrote: >>> you just look up the name of the trash directory (which I believe >>> is theoretically localized but actually always ~/.Trash on the drive with >>> your user directory, $mnt/.Trash otherwise) and call rename. >> >> You may also need to invent a unique name if there's already >> a file in the trash with the same name. At this point too much context has been stripped from the message for the replies to make sense. The original message was pointing out that there is no separate "low-level API" for trashing that could go in os along with a "high-level API" in shutil--unless you consider rename a low-level API for trashing, in which case it's already there. > And whatever else the api does for you? IMHO its better to use the API for this on OSX, such as FSMoveObjectToTrashSync. Except that API is deprecated (meaning, among other things, that if shutil linked to it, any py2app-ed or embedding app that imports shutil can't be sold in the Mac App Store), and there are no suitable replacements. By "suitable" I mean "suitable for widespread-enough use that it should be in the stdlib". As I said earlier in the thread, I think it's perfectly reasonable to have a third-party library that uses a deprecated API, requires 10.8, requires a runloop, directly or indirectly asks the Finder to do the work for you, or doesn't quite meet Apple's HIG rules, because all of those could be acceptable in _many_ applications; I just don't think any of those should be in the stdlib, because none of those are acceptable in _most_ applications. If I were designing an app right now, here's what I'd do: If you don't need 10.7 or earlier, use NSFileManager (via PyObjC if possible, via a tiny custom (Obj)C extension module if for some reason it's not, and same goes for all the following). If you need 10.5-10.7 but don't care about the App Store, check NSFileManager dynamically, and fall back to the FS function if the method is missing. If you need 10.6-10.7 and also need the App Store, you've probably got a runloop, so use the async function in NSWorkspace (recycleURLs:completionHandler:). If you need 10.6-10.7 in a helper app in the App Store, use the other function in NSWorkspace (performFileOperation:source:destination:files:tag:) and deal with all of its problems because you have no choice in the matter. From steve at pearwood.info Wed Dec 31 00:47:03 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 31 Dec 2014 10:47:03 +1100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> Message-ID: <20141230234702.GE24472@ando.pearwood.info> On Wed, Dec 31, 2014 at 12:12:01AM +0100, Andrew Barnert wrote: > By "suitable" I mean "suitable for widespread-enough use that it > should be in the stdlib". As I said earlier in the thread, I think > it's perfectly reasonable to have a third-party library that uses a > deprecated API, requires 10.8, requires a runloop, directly or > indirectly asks the Finder to do the work for you, or doesn't quite > meet Apple's HIG rules, because all of those could be acceptable in > _many_ applications; I just don't think any of those should be in the > stdlib, because none of those are acceptable in _most_ applications. What's wrong with asking the Finder to move the file? Under what circumstances would that be unacceptable? Are there scenarios on OS X where the Finder isn't available but the trash is? http://www.anthonysmith.me.uk/2008/01/08/moving-files-to-trash-from-the-mac-command-line/ Having to call out to an external program via Applescript feels rather icky, but given the difficulty of simultaneously supporting multiple OS X versions, satisfying the rules of Apple Store, and using Apple's APIs, I don't see any alternative. -- Steve From nykevin.norris at gmail.com Wed Dec 31 01:22:36 2014 From: nykevin.norris at gmail.com (Kevin Norris) Date: Tue, 30 Dec 2014 19:22:36 -0500 Subject: [Python-ideas] weakref.WeakKeyDictionary is (basically) useless Message-ID: Let's talk about weakref.WeakKeyDictionary. First, what is it? It's a dictionary whose keys are referenced weakly. That is, the dictionary takes weak references to its keys. If a key is garbage collected, it magically vanishes from the dictionary. This saves programmers much of the trouble of manually culling dead weak references from data structures. Why would you use it? The official documentation mentions implementing a [descriptor][1]. See [this paste][2] for a simple example of this sort of thing. Turning the WeakKeyDictionary into a regular dictionary would be troublesome. Since the descriptor object will (probably) never die, if it used a normal dictionary, it would keep all/most/some/more-than-one instances of Bar() alive forever. While this simple example would be better implemented as a property, or even a bare attribute, it is nice if you have many separate Bar classes and complicated per-property processing. Why is it broken? To go back to our simple example, imagine someone subclasses Bar (let's call the subclass Baz) and implements __eq__() and/or __hash__(). Instances of Baz may not work at all (if __hash__ is None) or they may "work" but behave nonsensically (all values-equal instances of Baz will have the same .foo attribute - imagine trying to track that bug down!). While we could forbid such a subclass in our API docs, this runs afoul of the open/closed principle. What can we do about it? It's possible someone is relying on this behavior, so we probably should not change the existing class. Instead, we can offer a new class called, say, WeakKeyIDDictionary. Said class would use object identity instead of object equality/hashing to uniquify keys. It would also reference keys weakly, just like WeakKeyDictionary does now. Implementing such a class is somewhat tricky, since weak references are difficult to reason about correctly. I've written up a [simplistic prototype][3], but I'm concerned about both performance and correctness. Specifically, I know that I need to call _remove_dead_refs() periodically, but I don't see where I can do that without compromising performance (e.g. making key lookup O(n) is a Bad Idea). I also would note it looks nothing like the stdlib implementations of WeakKey/ValueDictionary, which finagle raw weakrefs directly. These problems are related, of course, since the stdlib uses finalizer callbacks to cull dead references as they die. Frankly, the stdlib's code looks incredibly hairy to me and I doubt I could correctly re-implement it. Finally, I've no idea if it's subject to race conditions or other issues (it is a completely untested example I threw together in 5 minutes). Because this class is simultaneously useful and hard to write correctly, it would be a nice addition to the standard library (probably not in the form I've written it!), or failing that, some other library. I've Googled for such a library and come up empty. Should I submit a bug, a PEP, or is this whole idea just stupid? [1]: https://docs.python.org/3/reference/datamodel.html#implementing-descriptors [2]: http://pastebin.com/90j2ZAqC [3]: http://pastebin.com/cpZh2bTa -- Kevin Norris From benjamin at python.org Wed Dec 31 02:03:18 2014 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 31 Dec 2014 01:03:18 +0000 (UTC) Subject: [Python-ideas] weakref.WeakKeyDictionary is (basically) useless References: Message-ID: Kevin Norris writes: > Why is it broken? To go back to our simple example, imagine someone > subclasses Bar (let's call the subclass Baz) and implements __eq__() > and/or __hash__(). Instances of Baz may not work at all (if __hash__ > is None) or they may "work" but behave nonsensically (all values-equal > instances of Baz will have the same .foo attribute - imagine trying to > track that bug down!). While we could forbid such a subclass in our > API docs, this runs afoul of the open/closed principle. Why not just use a wrapper like this for keys? class IdentityWrapper(object): def __init__(self, obj): self.obj = obj def __hash__(self): return id(self.obj) def __eq__(self, other): return self.obj is other From russell at keith-magee.com Wed Dec 31 02:34:29 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Wed, 31 Dec 2014 09:34:29 +0800 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: On Wed, Dec 31, 2014 at 12:58 AM, Chris Barker wrote: > > On Mon, Dec 29, 2014 at 10:39 PM, Wes Turner wrote: > >> >>> As far as Kivy goes - I have tinkered with Kivy; I've even used Kivy's >>> tools as a starting point for some of my own efforts. However, to me, Kivy >>> is focussing too far up the stack - their toolchain appears to be based on >>> getting *Kivy* working on mobile, not *Python*. This is an entirely >>> reasonable approach if Kivy is what you want at the end of the day - but >>> it's not what *I* want. I've been taking a much more atomic approach. The >>> Kivy toolchain is also anchored in old versions of Python 2; something that >>> definitely needs to be addressed. >>> >> > IIUC, how Kivy works is not to put a python interpreter on the platform, > but rather to run all your code through Cython, creating a C program that > used the python runtime as a library, essentially. So there is no python > code involved at run time. > Incorrect. On both iOS and Android, Kivy uses Cython as the mechanism to get a C-native platform binding (PyJNIus/PyObjCus) that allows your Python code to invoke native libraries directly. The deployed app is a Python interpreter that is packaged with all the Python code and libraries, and it's executed as byte code at runtime. Toga/Rubicon takes a similar approach, but using ctypes directly, instead of using Cython. > So the Kivy lib aside, is this a good way to go on Mobile? I can't image > it's too hard to remove the kivy UI stuff if you don't want to use it -- > even if the current tools all include all that by default. > Sure, it could be done. And in a sense, that *is* what I'm doing - taking all the Kivy-specific parts out of Kivy's tools. And what I've been left with is things that, IMHO, should be in Python's own source tree. Along the way, I've discovered all the ways that Kivy's tools are deficient (for example, as a result of the way they've done their build, sys.platform reports the compilation platform, not the runtime platform). But is it better to target getting Python running on Mobile just like it > does on the desktop: you have a python interpreter, a bunch of python > modules, and you bundle it all up with a executable launcher of some sort, > al la py2exe, py2app, etc.? > That approach might work. However, py2app etc are just optimisations - you still need to have all the language bindings etc. On top of that, on Android, the end product must essentially be a JAR file. Jython doesn't currently work on the Dalvik VM, so there's a whole other level of tooling required. This might be changing with Android 5/Lollipop - I've only had a brief look at the VM changed in Android 5. > Something to keep in mind is that that way most Python code is run may not > make sense (or even be an option with Apple's rules, anyway): you have a > python environment installed, and install a bunch of application code on > top of that, and run that. Makes sense to system scripts, and web apps and > that like, but really doesn't make sense for "Apps". > > What I'm getting at is that the final goal for what exactly we want people > to be be able to distribute may effect decisions about what to build and > how to build it. > However, at the core, an "app" is just iOS/Android are running a process. Just as on desktop or web, that process can, if you so choose, start a Python interpreter and run a script. It's just a set of instructions. Yes, those instructions need to play nice with your local platform conditions, but that's no different to the distinction between desktop apps and web apps - I've spent plenty of time on the Django-users mailing list teaching newcomers to the web that you can't just take a long lived process and treat a web request as a button click starting that process - you need to restructure your code to take into account the short lived, stateless nature of HTTP requests. As for the "Apple" factor - that's not an issue (at least, not at the moment). Apple doesn't place restrictions on "interpreted" apps in the app store, as long as the scripts are all bundled as part of the app (i.e., you can't deliver updates/enhancements by sending an installed app a new script to run). There are Kivy apps in the App Store right now. Of course, this is subject to the whims and fancies of Apple, but this has been a stable policy for a couple of years, and I suspect there would be a riot if this policy were to change substantially, because it's central to the operation of many games engines. BTW, one thing I'd like to be able to build is what I call a "BILS" > (Browser Interface, Local Server) app: you run a web server bundled up in > an app that points itself to the browser component for all the UI. We've > been very successful doing this on Windows and OS-X, as a way to provide a > self contained desktop app, and a hosted Web App with almost entirely one > code base, and an identical user interface. > > (http://cameochemicals.noaa.gov/ and > http://response.restoration.noaa.gov/cameochemicals) > > (Python / Pyramid on the back end, desktop is the local browser hosted by > a wxPython wrapper) > > We'd like to be able to port that to iOS and Android, but there doesn't > appear to be a solid python option that we could run a Pyramid web server > on Mobile at this point -- maybe I'm wrong) > I'm aware of the approach - I've even used it myself in the past (to get a UI on an old iPaq handheld). However, once you've got a working Python interpreter on mobile with good bindings to the underlying platform, it shouldn't be too hard to write - getting a WebKit view that consumes the entire screen is easy to write, no matter what platform you're on, so then you just need to set up the server stuff, and for that all you need is a Python interpreter with a working socket and threads module. I'm not sure it would be great for battery life, but that's a separate problem. I can't say that particular approach something I'm particularly interested in myself, but my point is that this shouldn't matter - once you can do Python on mobile, it's up to you to decide what works or doesn't. But the first step is getting Python supported on mobile. Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From nykevin.norris at gmail.com Wed Dec 31 03:41:33 2014 From: nykevin.norris at gmail.com (Kevin Norris) Date: Tue, 30 Dec 2014 21:41:33 -0500 Subject: [Python-ideas] weakref.WeakKeyDictionary is (basically) useless Message-ID: On Tue, Dec 30, 2014 at 8:34 PM, Benjamin Peterson wrote: > Why not just use a wrapper like this for keys? > > class IdentityWrapper(object): > def __init__(self, obj): > self.obj = obj > def __hash__(self): > return id(self.obj) > def __eq__(self, other): > return self.obj is other I considered an approach like this one, but couldn't quite make it work. How do we make the IdentityWrapper object vanish at the right time? If we place it directly into a vanilla WeakKeyDictionary and forget it, it will immediately die and our WeakKeyDictionary will remove it. On the other hand, if we modify the IdentityWrapper to hold a weakref to obj, the IdentityWrapper will not vanish out of the dictionary at all, and its hash() will become invalid to boot (though we can deal with the latter issue relatively easily). If we could somehow make the target incorporate a reference to the wrapper, that would probably solve these issues, but I can't see a general way to do that without potentially breaking the target. I could do something like target._Foo__backref = self (or equivalently, invoke Python's name mangling), but that's a sloppy solution at best and a recipe for name collisions at worst (what if the target's class has multiple instances of Foo() attached to it?). It also falls apart in the face of __slots__. -- Kevin Norris From nykevin.norris at gmail.com Wed Dec 31 03:50:56 2014 From: nykevin.norris at gmail.com (Kevin Norris) Date: Tue, 30 Dec 2014 21:50:56 -0500 Subject: [Python-ideas] weakref.WeakKeyDictionary is (basically) useless In-Reply-To: <5aff1bfc-5911-410d-82a3-48a5f70d7d18@googlegroups.com> References: <5aff1bfc-5911-410d-82a3-48a5f70d7d18@googlegroups.com> Message-ID: On Tue, Dec 30, 2014 at 9:33 PM, Nils Bruin wrote: > We ran into this for sage (www.sagemath.org) and we implemented > "MonoDict" with basically the semantics you describe. It is > implemented as a cython extension class. See: > > http://git.sagemath.org/sage.git/tree/src/sage/structure/coerce_dict.pyx#n241 > > I can confirm your observation that it's hard to get right, but I think our > implementation by now is both correct and fast. The idiosyncratic > name reveals its origin in "TripleDict" defined in the same file, which > was developed first. > > Incidentally, we also have a WeakValueDict implementation that is faster > and a little safer than at least the Python2 version: > > http://git.sagemath.org/sage.git/tree/src/sage/misc/weak_dict.pyx > > These implementations receive quite a bit of exercise in sage, > so I would recommend that you look at them before developing > something from scratch yourself. This is quite interesting. Is there any interest in maintaining that code in a separate library? Sage is a rather large dependency which, for better or for worse, has little to do with my own work. I'm reluctant to add something that large for a single utility class. If there is no such interest, I might consider forking. In my code, I've worked around it by having the owner class and the descriptor collude (basically, the owner's instances have a private attribute which the descriptor uses), but it's far from the cleanest solution. -- Kevin Norris From stephen at xemacs.org Wed Dec 31 04:16:35 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 31 Dec 2014 12:16:35 +0900 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: <87k31833vw.fsf@uwakimon.sk.tsukuba.ac.jp> Fetchinson . writes: > > I'm not sure exactly what would be discussed [on mobile-sig], > > especially in the short term when the big ticket item is getting > > the patches into the main Python tree. It probably matters more that there *is* substantive discussion than *what* is discussed. There are probably a lot of people who are far more interested in "how to do 'it' *now*" than "get 'it' into the main Python tree 'someday'", for example. > I'd think the core devs would be far more likely to include a patch > if it comes with a big red note on it saying "it was discussed on > mobile-sig and was blessed by all participants" as opposed to a > random patch by a random person. Nope. It helps somewhat to have consensus backing for a particular patch by the (random) interested parties, but not that much. What helps a lot is to get someone trusted to have a good sense for "Pythonicity" and known to have experience in getting patches committed as spokesman. IIUC correctly it is Russ's opinion that his set of patches is not very invasive and wouldn't fall afoul of disturbing some other applications or platforms. If so, the real problem is to get core attention. Although it's a couple of months off, I would suggest a talk (if it's still possible to schedule), a BOF, and a sprint at Pycon North America in March. That should get things moving, and several of the "important" folks who have already expressed interest should be there, too. Regards, From russell at keith-magee.com Wed Dec 31 04:14:28 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Wed, 31 Dec 2014 11:14:28 +0800 Subject: [Python-ideas] python on mobile In-Reply-To: <8FEF00DDD6C58843BF4ED3C1DCF80E9F6E116C92@FMSMSX106.amr.corp.intel.com> References: <8FEF00DDD6C58843BF4ED3C1DCF80E9F6E116C92@FMSMSX106.amr.corp.intel.com> Message-ID: On Wed, Dec 31, 2014 at 4:10 AM, Frank, Matthew I wrote: > > > For Android I've been focusing on submitting patches to CPython 3.4/3.5 > for things that can be > argued to be "just cross-compilation bugs" in the Cpython source. > > For example: > http://bugs.python.org/issue21668 just fixes a missing -lm flag during > compilation (that doesn't > matter on Linux/MacOS/Windows, but is technically required, and causes > problems with the > Android dynamic loader if it is missing). > > http://bugs.python.org/issue20306 just takes care of the fact that > pw_gecos isn't actually required > by Posix. > Thanks for the heads up - I'll keep an eye on those tickets. And, if you're interested in collaborating on getting the rest of the compilation steps working on Android, let me know. Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From russell at keith-magee.com Wed Dec 31 04:42:59 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Wed, 31 Dec 2014 11:42:59 +0800 Subject: [Python-ideas] python on mobile In-Reply-To: <87k31833vw.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87k31833vw.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Wed, Dec 31, 2014 at 11:16 AM, Stephen J. Turnbull wrote: > > Fetchinson . writes: > > > > I'm not sure exactly what would be discussed [on mobile-sig], > > > especially in the short term when the big ticket item is getting > > > the patches into the main Python tree. > > It probably matters more that there *is* substantive discussion than > *what* is discussed. There are probably a lot of people who are far > more interested in "how to do 'it' *now*" than "get 'it' into the main > Python tree 'someday'", for example. > I'm not sure if a mailing list would help with this as much as a good, simple set of docs. It's not like there's unknown territory here; once you've got Python installed on a mobile device, the "how" is reasonably easy. The catch is that someone with Python experience won't necessarily have experience managing a cross-platform C/Automake build, which is what you need to get Python installed in the first place. However, once someone provides a binary for Python, others can just use that binary. That's what I've done with my platform support libraries and templates; they're a set of step-by-step instructions for going from a clean sheet to a running Python install. Again, I'm not claiming my code is complete and production ready - just that this is an achievable goal IMHO, and my intention is to document "getting Python working" independent of getting any particular platform working. To my mind, while this *could* be maintained as an external set of patches etc, it makes sense to get the patches into the main source tree. > I'd think the core devs would be far more likely to include a patch > > if it comes with a big red note on it saying "it was discussed on > > mobile-sig and was blessed by all participants" as opposed to a > > random patch by a random person. > > Nope. It helps somewhat to have consensus backing for a particular > patch by the (random) interested parties, but not that much. What > helps a lot is to get someone trusted to have a good sense for > "Pythonicity" and known to have experience in getting patches > committed as spokesman. > > IIUC correctly it is Russ's opinion that his set of patches is not > very invasive and wouldn't fall afoul of disturbing some other > applications or platforms. If so, the real problem is to get core > attention. > That is correct. There are a couple of tweaks needed to improve cross platform build support, a couple of tweaks introducing sys.platform support for iOS and Android, and some other miscellaneous patches to correct support for individual features. There's a bunch of little changes required, but it's not a big patch by any measure. Although it's a couple of months off, I would suggest a talk (if it's > still possible to schedule), a BOF, and a sprint at Pycon North > America in March. That should get things moving, and several of the > "important" folks who have already expressed interest should be there, > too. > Unfortunately, due to the tyranny of distance (I'm based on the west coast of Australia), plus work commitments, it's unlikely that I'll be attending PyCon in Montreal. Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Dec 31 09:41:55 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 31 Dec 2014 09:41:55 +0100 Subject: [Python-ideas] weakref.WeakKeyDictionary is (basically) useless References: Message-ID: <20141231094155.122ef8d4@fsol> On Tue, 30 Dec 2014 19:22:36 -0500 Kevin Norris wrote: > Let's talk about weakref.WeakKeyDictionary. > > First, what is it? It's a dictionary whose keys are referenced > weakly. That is, the dictionary takes weak references to its keys. > If a key is garbage collected, it magically vanishes from the > dictionary. This saves programmers much of the trouble of manually > culling dead weak references from data structures. > > Why would you use it? The official documentation mentions > implementing a [descriptor][1]. See [this paste][2] for a simple > example of this sort of thing. How about something like this (untested): import weakref class Foo: def __init__(self): self._data = {} def __get__(self, obj, owner): _, val = self._data[id(obj)] return val def __set__(self, obj, value): key = id(obj) try: ref, _ = self._data[key] except KeyError: def on_destroy(_): del self._data[key] ref = weakref.ref(obj, on_destroy) self._data[key] = ref, value def __delete__(self, obj): del self._data[id(obj)] class Bar: foo = Foo() Regards Antoine. From abarnert at yahoo.com Wed Dec 31 11:50:04 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 31 Dec 2014 11:50:04 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <20141230234702.GE24472@ando.pearwood.info> References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <20141230234702.GE24472@ando.pearwood.info> Message-ID: On Dec 31, 2014, at 0:47, Steven D'Aprano wrote: > On Wed, Dec 31, 2014 at 12:12:01AM +0100, Andrew Barnert wrote: > >> By "suitable" I mean "suitable for widespread-enough use that it >> should be in the stdlib". As I said earlier in the thread, I think >> it's perfectly reasonable to have a third-party library that uses a >> deprecated API, requires 10.8, requires a runloop, directly or >> indirectly asks the Finder to do the work for you, or doesn't quite >> meet Apple's HIG rules, because all of those could be acceptable in >> _many_ applications; I just don't think any of those should be in the >> stdlib, because none of those are acceptable in _most_ applications. > > What's wrong with asking the Finder to move the file? Under what > circumstances would that be unacceptable? Are there scenarios on OS X > where the Finder isn't available but the trash is? I already answered this briefly earlier in the thread, but in more detail: 1. If you don't have a GUI login session (e.g., because you've ssh'd into the box, or are running as a LaunchDaemon or a traditional detached script), you have no Finder in your session to script. 2. If you're running in a sandbox that doesn't have scripting rights for Finder added in--as will be the case in most Mac App Store apps--you can't script Finder. Of course you can include a sandbox exception to allow for this, but the only way to do this that's allowed in the App Store requires 10.8.0 to run and, IIRC, 10.8.2 to run without serious bugs. And the Python docs would have to explain that any app that may call shutil.trash must use a custom sandbox. 3. If you're running in a sandbox, you can't execute osascript; if you need pre-10.8 compat, you can't use the modern APIs; this means the only thing you can do is include a compiled AppleScript as a resource and use the 10.7 APIs to execute it--which still requires 10.7, and it seems to have problems with the new 10.9.5/10.10.1 code signing rules (although it may be easy to work around that; I haven't tried). 4. If you're using a third-party Finder replacement like PathFinder, it won't respond to messages sent to Finder. 5. If you've killed the Finder to save resources (pretty rare nowadays, but still legal) you'll relaunch it. 6. The Finder may pop up a progress bar--or, worse, an auth dialog, or a "delete immediately?" dialog, on behalf of your app. This could be surprising from a console program. 7. Depending on how you do the scripting, your request may time out after 5 or 30 seconds, leaving you no way to know whether the trashing succeeded. 8. If Finder is hidden, it may unhide. If another app is fullscreen in front of Finder, it may slide offscreen, or Finder's dialogs may just be invisible. (You can guarantee a fix for the latter by forcing the former.) 9. If your session or system is screwed up and you're trying to shut down cleanly or repair things, Finder could be unresponsive or even hung. For example, on 10.6, if you have for separate moves blocked on an NFS mount, a trash will block forever, but (assuming you aren't trying to trash something on the same mount) there's nothing stopping you from trashing on your own. In some cases, this may all be fine. For example, if you're writing a trash script that you intend to use on your specific computer in local Terminal sessions only, you may _want_ the Finder to take care of auth, no-trash mounts, and other issues for you, and show you progress on long tasks. Fortunately, you can already write that script with about 5 lines of ScriptingBridge code that will run on the Python 2.7 with PyObjC and other extras that Apple has pre-installed as /usr/bin/python on all 10.6-10.10 versions, so you're already covered. > http://www.anthonysmith.me.uk/2008/01/08/moving-files-to-trash-from-the-mac-command-line/ Even if you want to use osascript, you don't want to do it that way. Build a single AppleScript command that trashes all of your files in a single call. One of the other blog posts someone linked earlier in this thread shows how to do that. Also, for the usual reasons, os.system is not the best way to call osascript (you'll end up with a success retcode even on failure, quoting for both the shell and AppleScript is double the fun, etc.). > Having to call out to an external program via Applescript feels rather > icky, but given the difficulty of simultaneously supporting > multiple OS X versions, satisfying the rules of Apple Store, and using > Apple's APIs, I don't see any alternative. Since this doesn't satisfy the rules of the App Store, it doesn't solve the problem it's intended to. But see my previous email. For almost any app, there is a much better alternative. The problem is that there's no _single_ alternative that's good for most apps; there are different ones for different use cases. In particular, for just writing a little trash script for use on the command line on your own 10.8+ computer, the NSFileManager version is even simpler than the scripting version (again using Apple's pre-installed 2.7 with PyObjC), and without any of its problems. For older computers, it's a _bit_ more work with the FS function, but still very simple, and it's a known problem that multiple people have solved years ago (and linked to earlier in this thread), so Python 3.5 doesn't need to retroactively solve it. From j.wielicki at sotecware.net Wed Dec 31 12:32:20 2014 From: j.wielicki at sotecware.net (Jonas Wielicki) Date: Wed, 31 Dec 2014 12:32:20 +0100 Subject: [Python-ideas] Python hook just before NameError In-Reply-To: References: Message-ID: <54A3DEC4.6070009@sotecware.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 29.12.2014 09:00, Chris Angelico wrote: > [?] Maybe you'd like to have some completely different magic - > like REXX mode: [?] Or something more suitable for floating point > work: > >>>> sys.__getglobal__ = lambda name: float("nan") x = 4 x + y > nan > > The power is in your hands. [?] Or with sympy, have undefined names be a sympy.core.symbol.Symbol automatically. This is something which cannot be accomplished with the PYTHONSTARTUP-script I already have for calculator work in python (importing sympy and defining a few more constants and such). regards, jwi p.s.: Sorry for the messed up quote, didn?t manage to figure out how to trick my client into not re-flowing the interactive prompt. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCgAGBQJUo968AAoJEMBiAyWXYliK2tAQAJfqwGn5vBhck1bdzbPVFMED Vu65jOAssvCk7reCTmtKtpY4PYDHde+XfDawRShagTPnMFpVvADAAxfOcLb11blr 6vrcCTitAPoy7NbOU44TjHDeAsrDkibF6HDtDRMy3Z6Vu+srRe/+teHQTjSZaR1W zFhBh0lCVYnL/I67ADEJbU7AzDK4na/3CrwgSrcTVCXzgdaKeVixtKypdli5AHis IiXURVy1Zy52xDpzbfhUgEMWMuMANOnX9wXjKaS2JlaHIT+YJ5MCCbqE7tlVElJb HAKz1nLwaN6QE8tXtb1EWNV/3Lnsr8dcd0BBNU0v6YU5JvZGK69rcpGwUxJJzjNK 5HT64Z/LQB64yRwlf6butIPppzsvMWCAzGmlZoarvwnSh+gnbuj8M9syVnbqhcTS 8tHyuOqvRU+VPGY5TdXOR4Qq56nSPyEsDHB3hPeXsRh0+1sn4AQEMUhhKrLfRdPP nyTUYvnMWMcZr4b3CR2nNujz3gq5ON/IbINPrrQwU4fSGxqTHkofi4fOfjB56nNw Fzbuv6wwXzLoFPmknTU9EiZD1EsIzgvW3bKzN5qzgKBsN6nN23UBroTU3vz2/zM9 jkgQhQ7lgp9+x/myRwxc7LAefJ1DqFiXubcS95Y9tzGh6HFeSc4t1cBLEOVRQ3xy PKzIobEoSFL5M3QakFDU =6GR9 -----END PGP SIGNATURE----- From ronaldoussoren at mac.com Wed Dec 31 09:20:20 2014 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 31 Dec 2014 09:20:20 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> Message-ID: > On 31 Dec 2014, at 00:12, Andrew Barnert wrote: > > On Dec 30, 2014, at 22:36, Ronald Oussoren wrote: > >> >>> On 30 Dec 2014, at 00:19, Greg Ewing wrote: >>> >>> Andrew Barnert wrote: >>>> you just look up the name of the trash directory (which I believe >>>> is theoretically localized but actually always ~/.Trash on the drive with >>>> your user directory, $mnt/.Trash otherwise) and call rename. >>> >>> You may also need to invent a unique name if there's already >>> a file in the trash with the same name. > > At this point too much context has been stripped from the message for the replies to make sense. The original message was pointing out that there is no separate "low-level API" for trashing that could go in os along with a "high-level API" in shutil--unless you consider rename a low-level API for trashing, in which case it's already there. > >> And whatever else the api does for you? IMHO its better to use the API for this on OSX, such as FSMoveObjectToTrashSync. > > Except that API is deprecated (meaning, among other things, that if shutil linked to it, any py2app-ed or embedding app that imports shutil can't be sold in the Mac App Store), and there are no suitable replacements. There?s always the Cocoa APIs for moving files to the Trash, if its fine to depend on OSX 10.6 or later (as you mentioned below). Whether or not you?d want to do that in the stdlib is a different question. My point was that just using os.rename move a file to the trash is a bad idea. BTW. I?m not really in favour of using either FSMoveObjectToTrash or the Cocoa APIs in the stdlib, given the problems we?ve had with proxy support in the _scproxy extension (basically resulting in ?random? crashes when using urllib in a child proces started with os.fork because Apple?s system frameworks don?t support that). Ronald From abarnert at yahoo.com Wed Dec 31 17:17:24 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 31 Dec 2014 17:17:24 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> Message-ID: <735A853C-FAA1-4D50-A9A5-913B58FD4643@yahoo.com> On Dec 31, 2014, at 9:20, Ronald Oussoren wrote: > >> On 31 Dec 2014, at 00:12, Andrew Barnert wrote: >> >> On Dec 30, 2014, at 22:36, Ronald Oussoren wrote: >> >>> >>>> On 30 Dec 2014, at 00:19, Greg Ewing wrote: >>>> >>>> Andrew Barnert wrote: >>>>> you just look up the name of the trash directory (which I believe >>>>> is theoretically localized but actually always ~/.Trash on the drive with >>>>> your user directory, $mnt/.Trash otherwise) and call rename. >>>> >>>> You may also need to invent a unique name if there's already >>>> a file in the trash with the same name. >> >> At this point too much context has been stripped from the message for the replies to make sense. The original message was pointing out that there is no separate "low-level API" for trashing that could go in os along with a "high-level API" in shutil--unless you consider rename a low-level API for trashing, in which case it's already there. >> >>> And whatever else the api does for you? IMHO its better to use the API for this on OSX, such as FSMoveObjectToTrashSync. >> >> Except that API is deprecated (meaning, among other things, that if shutil linked to it, any py2app-ed or embedding app that imports shutil can't be sold in the Mac App Store), and there are no suitable replacements. > > There?s always the Cocoa APIs for moving files to the Trash, if its fine to depend on OSX 10.6 or later (as you mentioned below). Yes, but the 10.6 Cocoa APIs both have problems; the useful one requires 10.8. So as long as Python standard builds are 10.6? (I suppose you could have it in the stdlib but only available at runtime on 10.8+, but I think that would be weird.) > Whether or not you?d want to do that in the stdlib is a different question. My point was that just using os.rename move a file to the trash is a bad idea. Yes, and therefore there just _is_ no low-level, suitable-for-os, function that would help for a higher-level shutil/pathlib function. > BTW. I?m not really in favour of using either FSMoveObjectToTrash or the Cocoa APIs in the stdlib, given the problems we?ve had with proxy support in the _scproxy extension (basically resulting in ?random? crashes when using urllib in a child proces started with os.fork because Apple?s system frameworks don?t support that). I forgot about that, but you're right, that's another potential reason not to put this in the stdlib. Anyway, maybe what we need is something in the docs explaining the various ways to do it (with links to PyPI projects as appropriate) with the pros and cons of each. What I wrote before isn't right, because it doesn't take into account the forking issue. It could be easily extended, although that might get a bit long for stdlib documentation. From chris.barker at noaa.gov Wed Dec 31 18:01:58 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 31 Dec 2014 09:01:58 -0800 Subject: [Python-ideas] python on mobile In-Reply-To: References: Message-ID: On Tue, Dec 30, 2014 at 5:34 PM, Russell Keith-Magee < russell at keith-magee.com> wrote: > Incorrect. On both iOS and Android, Kivy uses Cython as the mechanism to > get a C-native platform binding (PyJNIus/PyObjCus) that allows your Python > code to invoke native libraries directly. The deployed app is a Python > interpreter that is packaged with all the Python code and libraries, and > it's executed as byte code at runtime. > Thanks for the clarification -- it does sound like they are headed in the right direction, then. Sure, it could be done. And in a sense, that *is* what I'm doing - taking > all the Kivy-specific parts out of Kivy's tools. And what I've been left > with is things that, IMHO, should be in Python's own source tree. Along the > way, I've discovered all the ways that Kivy's tools are deficient (for > example, as a result of the way they've done their build, sys.platform > reports the compilation platform, not the runtime platform). > sounds good -- and you are right, the core building stuff should be in the main source. > But is it better to target getting Python running on Mobile just like it >> does on the desktop: you have a python interpreter, a bunch of python >> modules, and you bundle it all up with a executable launcher of some sort, >> al la py2exe, py2app, etc.? >> > > That approach might work. However, py2app etc are just optimizations - you > still need to have all the language bindings etc. > well, they are not optimizations, exactly, but they are extra stuff -- the final bundling up as an app. The thing is, that that last stage is optional, and as far as I can tell, a small fraction of the python use cases, for current python use. But for mobile, it is essential: no one is going to be delivering an app that first requires you to install a python environment and associated packages for your iPhone. Of course, it still requires the basic python environment and standard library to be built in a way that makes sense, but some of that way that makes sense may be influenced by the target delivery method. I"d love to see the bundling be standardized as well, but that probably isn't a project for the core development team. Though maybe it is -- the truth is that Python does not currently "have a story" on Mobile. And a real "story" would be a soup to nuts -- this is how you build an app with python for iOS/Android/WindowsMobile. If we can only say: cPython compiles fine for mobile platforms, but you still need to figure out how to call system libs, and bundle it up as an app, and here are a dozen scattered projects that all do that differently, each designed for a different use case, We really aren't there. All that being said, it seems you are very much on the right track -- start at the bottom, if you can get the core building stuff in the standard distribution, then at least the dozen different systems could b built on the same python.... > As for the "Apple" factor - that's not an issue (at least, not at the > moment). Apple doesn't place restrictions on "interpreted" apps in the app > store, as long as the scripts are all bundled as part of the app > However, I can't see that one could build a python "run time" that individual apps could all share, rather than each app bundling up python and all with it. Maybe flash space and RAM is cheap enough these days that we don't care but it is still a bit constrained on mobile. I'm aware of the approach - I've even used it myself in the past (to get a >> UI on an old iPaq handheld). However, once you've got a working Python >> interpreter on mobile with good bindings to the underlying platform, it >> shouldn't be too hard to write - getting a WebKit view that consumes the >> entire screen is easy to write, no matter what platform you're on, so then >> you just need to set up the server stuff, and for that all you need is a >> Python interpreter with a working socket and threads module. I'm not sure >> it would be great for battery life, but that's a separate problem. >> > > I can't say that particular approach something I'm particularly interested > in myself, but my point is that this shouldn't matter - once you can do > Python on mobile, it's up to you to decide what works or doesn't. But the > first step is getting Python supported on mobile. > yup -- I'm really looking forward to the results of your work! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From nykevin.norris at gmail.com Wed Dec 31 19:26:44 2014 From: nykevin.norris at gmail.com (Kevin Norris) Date: Wed, 31 Dec 2014 13:26:44 -0500 Subject: [Python-ideas] weakref.WeakKeyDictionary is (basically) useless Message-ID: On Wed, Dec 31, 2014 at 6:00 AM, Antoine Pitrou wrote: >> Why would you use it? The official documentation mentions >> implementing a [descriptor][1]. See [this paste][2] for a simple >> example of this sort of thing. > > How about something like this (untested): > A cursory glance suggests that might work for my use-case, but I would want to examine it more thoroughly for race conditions and other hazards. In particular, I'd be concerned about the potential performance cost of removing things immediately on finalization (the stdlib implementation uses a semi-lazy approach here, which I still don't entirely understand). I'm also a little nervous about doing anything remotely interesting from a finalizer. With a little fine-tuning, however, this could be pulled into a separate WeakKeyIDDictionary implementation, which would be more modular and easier to work with. -- Kevin Norris