From marko.ristin at gmail.com Mon Oct 1 00:32:42 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Mon, 1 Oct 2018 06:32:42 +0200 Subject: [Python-ideas] "old" values in postconditions In-Reply-To: References: <13186373-6FB6-4C8B-A8D8-2C3E028CDC3D@gmail.com> <3C33B6FF-FC19-47D6-AD2A-FC0B17C50A8D@gmail.com> <0061278F-4243-42BD-945D-A93B4A0FC21D@gmail.com> <769938C2-9DE6-42B2-99E7-FED5325F8510@gmail.com> Message-ID: Hi James, Sure, let's do that! I'm fine with emails, too. I needed only a bit of structure since we lumped so many issues together, but maybe it's just my OCD :) Le lun. 1 oct. 2018 ? 04:05, James Lu a ?crit : > Hi Marko, > > Regarding switching over to GitHub issues: > * I copy-pasted the MockP original code to GitHub issues. > * There's a clunky way to view the discussion at > https://mail.python.org/pipermail/python-ideas/2018-September/subject.html#start > . > * The less clunky way to view the discussion is to subscribe to the > mailing list and use Gmail to move all the messages from python-ideas to > python ideas list and all the messages from the discussions we have to a > "contracts" label and view the discussion with your email client. > * A week earlier I didn't think I'd be saying this, but I like email for > discussion better. It works on mobile and I can send messages offline, and > I send 90% of my messages on my phone and when I'm offline. Unless you know > an alternative (WhatsApp, maybe?) that fits my use cases off the top of > your head, I think we should stick to email. > * My proposal: We split the discussion into a new email thread, we keep > the latest agreed upon proposal on GitHub issues. > > On Sun, Sep 30, 2018 at 4:32 PM Marko Ristin-Kaufmann < > marko.ristin at gmail.com> wrote: > >> Hi James, >> (I'm just about to go to sleep, so I'll answer the other messages >> tomorrow.) >> >> Should we keep some kind of document to keep track of all the different >>> proposals? I?m thinking an editable document like HackMD where we can label >>> all the different ideas to keep them straight in our head. >>> >> >> I thought github issues would be a suitable place for that: >> https://github.com/Parquery/icontract/issues >> >> It reads a bit easier as a discussion rather than a single document -- if >> anybody else needs to follow. What do you think? >> >> On Sun, 30 Sep 2018 at 22:07, James Lu wrote: >> >>> Hi Marko, >>> >>> Going back to your proposal on repeating lambda P as a convention. >>> >>> I do find >>> >>> @snapshot(some_identifier=P -> P.self(P.arg1), >>> some_identifier2=P -> P.arg1 + P.arg2) >>> >>> acceptable. >>> >>> Should we keep some kind of document to keep track of all the different >>> proposals? I?m thinking an editable document like HackMD where we can label >>> all the different ideas to keep them straight in our head. >>> >>> James Lu >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko.ristin at gmail.com Mon Oct 1 00:41:41 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Mon, 1 Oct 2018 06:41:41 +0200 Subject: [Python-ideas] "old" values in postconditions In-Reply-To: <559132CB-DD2B-4422-853C-B97EA7FCF2D2@gmail.com> References: <13186373-6FB6-4C8B-A8D8-2C3E028CDC3D@gmail.com> <3C33B6FF-FC19-47D6-AD2A-FC0B17C50A8D@gmail.com> <0061278F-4243-42BD-945D-A93B4A0FC21D@gmail.com> <559132CB-DD2B-4422-853C-B97EA7FCF2D2@gmail.com> Message-ID: Hi James, I agree. Having a snapshot decorator per variable also has the advantage that we can add parameters to the decorator (e.g., snapshots also need "enabled" and "description" argument). How should we distinguish the two approaches? (I suppose you would also apply them to requests, ensures and invariant decorator, right?) Is there a terminology? Lambda-based conditions might be the name for the conventional approach with @requests(lambda P: ...). What should we term the approach you proposed? Re naming: @requests(lambda P: P.arg1 < P.arg2) and @requests_that(P.arg1 < P.arg2)? Le dim. 30 sept. 2018 ? 22:07, James Lu a ?crit : > Hi Marko, > > I just found the time to reply to these. > > I reread the proposal with MockP. I still don't get the details, but if > I think I understand the basic idea. You put a placeholder and whenever one > of its methods is called (including dunders), you record it and finally > assemble an AST and compile a lambda function to be executed at actual call > later. > Precisely. > > > > But that would still fail if you want to have: > > @snapshot(var1=some_func(MockP.arg1, MockP.arg2)) > > , right? Or there is a way to record that? > > This would still fail. You would record it like this: > @snapshot(var1=thunk(some_func)(MockP.arg1, MockP.arg2)) > > thunk stores the function for later and produces another MockP object that > listens for __call__. > > By the way, MockP is the class while P is a virgin instance of MockP. > MockP instances are immutable, so any operation on a MockP instance creates > a new object or MockP instance. > > I?m also beginning to lean towards > @snapshot(var1=...) > @snapshot(var2=...) > > I suspect this would deal better with VCS. > > This syntax does have a a nice visual alignment. I?m not entirely sure > what kind of indentation PEP 8 recommends and editors give, so the point > may be moot if the natural indentation also gives the same visual alignment. > > Though both should be supported so the best syntax may win. > > James Lu > > > > On Sep 29, 2018, at 3:22 PM, Marko Ristin-Kaufmann < > marko.ristin at gmail.com> wrote: > > > > I reread the proposal with MockP. I still don't get the details, but if > I think I understand the basic idea. You put a placeholder and whenever one > of its methods is called (including dunders), you record it and finally > assemble an AST and compile a lambda function to be executed at actual call > later. > > > > But that would still fail if you want to have: > > @snapshot(var1=some_func(MockP.arg1, MockP.arg2)) > > , right? Or there is a way to record that? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko.ristin at gmail.com Mon Oct 1 01:01:34 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Mon, 1 Oct 2018 07:01:34 +0200 Subject: [Python-ideas] Transpiling contracts Message-ID: Hi James, Regarding the ?transpile into Python? syntax with with statements: Can I > see an example of this syntax when used in pathlib? I?m a bit worried this > syntax is too long and ?in the way?, unlike decorators which are before the > function body. Or do you mean that both MockP and your syntax should be > supported? > > Would > > with requiring: assert arg1 < arg2, ?message? > > Be the code you type or the code that?s actually run? > That's the code you would type. Let me make a full example (I'm omitting "with contracts" since we actually don't need it). You would read/write this: def some_func(arg1: List[int])->int: with requiring: assert len(arg1) > 3, "some description" with oldie as O, resultie as result, ensuring: if SLOW: O.var1 = sum(arg1) assert result > sum(arg1) > O.var1 assert len(result) > 5 This would run: @requires(lambda P: len(P.arg1) > 3, "some description") @snapshot(lambda P, var1: sum(P.arg1), enabled=SLOW) @ensures(lambda O, P, result: result > sum(arg1) > O.var1, enabled=SLOW) @ensures(lambda result: len(result) > 5) I omitted in the example how to specify a custom exception to avoid confusion. If we decide that transpiler is the way to go, I'd say it's easier to just stick with lambdas and allow no mock-based approach in transpilation. Cheers, Marko > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko.ristin at gmail.com Mon Oct 1 01:25:23 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Mon, 1 Oct 2018 07:25:23 +0200 Subject: [Python-ideas] Renaming icontract to pcontract Message-ID: Hi, I'd like to rename icontract into pcontract to avoid name conflict with java's icontract library. Do you have any better suggestion? Thanks! Marko -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Mon Oct 1 01:57:51 2018 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 1 Oct 2018 01:57:51 -0400 Subject: [Python-ideas] Upgrade to Mailman 3 In-Reply-To: References: Message-ID: re: message URL in mm3 footers (after sig dashes: '\n--\n') "This list will soon be migrating to Mailman 3" https://mail.python.org/mm3/archives/list/distutils-sig at python.org/thread/IHWUGVND3FU2UH3GEC2GYOSOJJMKLO5H/ On Sunday, September 30, 2018, James Lu wrote: > It has a nice GUI for people who spectate a discussion to read emails > without having to subscribe to the list. > > http://docs.mailman3.org/en/latest/migration.html > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at cskk.id.au Mon Oct 1 03:15:04 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 1 Oct 2018 17:15:04 +1000 Subject: [Python-ideas] Renaming icontract to pcontract In-Reply-To: References: Message-ID: <20181001071504.GA54129@cskk.homeip.net> On 01Oct2018 07:25, Marko Ristin-Kaufmann wrote: >I'd like to rename icontract into pcontract to avoid name conflict with >java's icontract library. > >Do you have any better suggestion? No, sounds ok to me. What was the "i" for in the old name? Cheers, Cameron Simpson From marko.ristin at gmail.com Mon Oct 1 03:28:06 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Mon, 1 Oct 2018 09:28:06 +0200 Subject: [Python-ideas] Renaming icontract to pcontract In-Reply-To: <20181001071504.GA54129@cskk.homeip.net> References: <20181001071504.GA54129@cskk.homeip.net> Message-ID: Hi Cameron, A nerdy way to make it sound like a sentence: "I contract that ...". Pcontract would stand for python contract. Pycontract is already taken. Cheers, Marko Le lun. 1 oct. 2018 ? 09:15, Cameron Simpson a ?crit : > On 01Oct2018 07:25, Marko Ristin-Kaufmann wrote: > >I'd like to rename icontract into pcontract to avoid name conflict with > >java's icontract library. > > > >Do you have any better suggestion? > > No, sounds ok to me. What was the "i" for in the old name? > > Cheers, > Cameron Simpson > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Mon Oct 1 06:38:34 2018 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 1 Oct 2018 11:38:34 +0100 Subject: [Python-ideas] Suggestion: Extend integers to include iNaN In-Reply-To: References: <20180930010053.GF19437@ando.pearwood.info> Message-ID: On Mon, 1 Oct 2018 at 00:00, Chris Angelico wrote: > > On Mon, Oct 1, 2018 at 8:53 AM Oscar Benjamin > wrote: > > > > On Sun, 30 Sep 2018 at 02:01, Steven D'Aprano wrote: > > > > > > On Sat, Sep 29, 2018 at 09:43:42PM +0100, Oscar Benjamin wrote: > > > > On Sat, 29 Sep 2018 at 19:38, Steve Barnes wrote: > > > > > > > > > I converted to int because I needed a whole number, this was intended to > > > > > represent some more complex process where a value is converted to a > > > > > whole number down in the depths of the processing. > > > > > > > > Your requirement to have a whole number cannot meaningfully be > > > > satisfied if your input is nan so an exception is the most useful > > > > result. > > > > > > Not to Steve it isn't. > > > > > > Be careful about making value judgements like that: Steve is asking for > > > an integer NAN because for *him* an integer NAN is more useful than an > > > exception. You shouldn't tell him that he is wrong, unless you know his > > > use-case and his code, which you don't. > > > > Then he can catch the exception and do something else. If I called > > int(x) because my subsequent code "needed a whole number" then I would > > definitely not want to end up with a nan. The proposal requested is > > that int(x) could return something other than a well defined integer. > > That would break a lot of code! > > At no point was the behaviour of int(x) ever proposed to be changed. > Don't overreact here. The context got trimmed a bit too much. You can see here the messages preceding what is quoted above: https://mail.python.org/pipermail/python-ideas/2018-September/053840.html Quoting here: > One simplistic example would be print(int(float('nan'))) (gives a ValueError) while print(int(iNaN)) should give 'nan' or maybe 'inan' That would mean that the result of int(x) is no longer guaranteed to be a well-defined integer. > The recommended use-case was for a library to > return iNaN instead of None when it is unable to return an actual > value. You can already use None or float('nan') for this. You can also create your own singleton nan object if you want. When I said I haven't seen a use-case what I mean is that no one has presented a situation where the existing language facilities are considered insufficient (apart from the suggestion about int(iNaN) that I refer to above). -- Oscar From jfine2358 at gmail.com Mon Oct 1 08:02:58 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Mon, 1 Oct 2018 13:02:58 +0100 Subject: [Python-ideas] Suggestion: Extend integers to include iNaN In-Reply-To: References: Message-ID: Hi Steve You've suggested that we add to Python an integer NaN object, similar to the already existing float NaN object. >>> x = float('nan') >>> x, type(x) (nan, ) I've learnt that decimal also has a NaN object, but not fractions. https://stackoverflow.com/questions/19374254/ >>> from decimal import Decimal >>> y = Decimal('nan') >>> y, type(y) (Decimal('NaN'), ) Numpy has its own fixed size int classes >>> from numpy import int16 >>> x = int16(2358); x, type(x) (2358, ) >>> x = x * x; x, type(x) __main__:1: RuntimeWarning: overflow encountered in short_scalars (-10396, ) I'm confident that one could create classes similar to numpy.int16, except that >>> z = int16('nan') Traceback (most recent call last): ValueError: invalid literal for int() with base 10: 'nan' would not raise an exception (and would have the semantics you wish for). I wonder, would this be sufficient for the use cases you have in mind? -- Jonathan From klahnakoski at mozilla.com Mon Oct 1 08:10:26 2018 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Mon, 1 Oct 2018 08:10:26 -0400 Subject: [Python-ideas] Suggestion: Extend integers to include iNaN In-Reply-To: References: <20180930010726.GG19437@ando.pearwood.info> <20180930125542.GL19437@ando.pearwood.info> <91cbff66-68bc-8453-559c-03ec27387ebf@mozilla.com> Message-ID: <9c7fc345-a2cf-d372-caf5-53302e960383@mozilla.com> On 2018-09-30 10:45, Anders Hovm?ller wrote: > >>> I am roughing out such a class and some test cases which will hopefully >>> include some cases where the hoped for advantages can be realised. >>> >>> My thinking on bitwise operations is to do the same as arithmetic >>> operations, i.e. (anything op iNaN) = iNaN and likewise for shift >>> operations. >> Steve, >> >> While you are extending a number system, can every int be truthy, while >> only iNan be falsey? I found that behaviour more useful because >> checking if there is a value is more common than checking if it is a >> zero value. > I?m not saying you?re wrong in principle but such a change to Python seems extremely disruptive. And if we?re talking about robustness of code then truthiness would be better like in Java (!) imo, where only true is true and false is false and everything else is an error. If we?re actually talking about changing the truth table of Python for basic types then this is the logical next step. > > But making any change to the basic types truth table is a big -1 from me. This seems like a Python 2-3 transition to me. Sorry, I can see I was unclear:? I was only asking that the new number system (and the class that implements it) have truthy defined differently.? My imagination never considered extending ints with iNaN. I would imagine the iNaN checks on every int operation to be noticeably slower, so out of the question. From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Oct 1 09:09:23 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 1 Oct 2018 22:09:23 +0900 Subject: [Python-ideas] Simplicity of C (was why is design-by-contracts not widely) In-Reply-To: References: <20180930015413.GH19437@ando.pearwood.info> <20180930042635.GI19437@ando.pearwood.info> <20180930080249.GJ19437@ando.pearwood.info> <23472.48495.587491.729759@turnbull.sk.tsukuba.ac.jp> Message-ID: <23474.7299.429212.519412@turnbull.sk.tsukuba.ac.jp> Elazar writes: > On Sun, Sep 30, 2018, 15:12 Stephen J. Turnbull < > turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > > What does "inherited" mean? Just that methods that are not overridden > > retain their contracts? > > Contracts are attached to interfaces, not to specifications. So > when you have abstract base class, it defines contracts, and > implementing classes must adhere to these contracts - the can only > strengthen it, not weaken it. Stated in words, it sounds very reasonable. Thinking about the implications for writing Python code, it sounds very constraining in the context of duck-typing. Or, to give a contrapositive example, you can write few, if any, contracts for dunders beyond their signatures. For example, len(x) + len(y) == len(x + y) is not an invariant over the classes that define __len__ (eg, set and dict, taking the "x + y" loosely for them). Of course if you restrict to Sequence, you can impose that invariant. The question is the balance. It sounds to me like the proof of the pudding for Python will be annotating the ABC module with contracts. If that turns out to be a mostly fruitless exercise, it's hard to see how "real" contracts (vs. using asserts to "program in contracting style") are an important feature for Python in general. That said, given the passion of the proponents, if we can come up with an attractive style for implementing contracts without a language change, I'd be +0 at least for adding a module to the stdlib. But this is a Python 3.9 project, given the state of the art. > This is precisely like with types, since types are contracts (and > vice versa, in a way). Not Python types (== classes), though. As a constraint on behavior, subclassing is purely nominal, although Python deliberately makes it tedious to turn a subclass of str into a clone of int. Steve From marko.ristin at gmail.com Mon Oct 1 09:26:42 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Mon, 1 Oct 2018 15:26:42 +0200 Subject: [Python-ideas] Simplicity of C (was why is design-by-contracts not widely) In-Reply-To: <23474.7299.429212.519412@turnbull.sk.tsukuba.ac.jp> References: <20180930015413.GH19437@ando.pearwood.info> <20180930042635.GI19437@ando.pearwood.info> <20180930080249.GJ19437@ando.pearwood.info> <23472.48495.587491.729759@turnbull.sk.tsukuba.ac.jp> <23474.7299.429212.519412@turnbull.sk.tsukuba.ac.jp> Message-ID: Hi Steve (Turnbull), It sounds to me like the proof of the pudding for Python will be > annotating the ABC module with contracts. If that turns out to be a > mostly fruitless exercise, it's hard to see how "real" contracts > (vs. using asserts to "program in contracting style") are an important > feature for Python in general. > Do you mean annotating an abstract class with contracts or annotating the actual functions in abc module? If you meant abstract classes in general, we do that for quite some time in our code base and had no problems so far. icontract already supports inheritance of contracts (as well as weakening/strengthening of the contracts). Inheriting of contracts also proved useful when you define an interface as a group of functions together with the contracts. You define the functions as an ABC with purely static methods (no state) and specify the contracts in that abstract class. The concrete classes implement the static functions, but don't need to repeat the contracts since they are inherited from the abstract class. I can copy/paste from our code base if you are interested. Cheers, Marko On Mon, 1 Oct 2018 at 15:09, Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > Elazar writes: > > On Sun, Sep 30, 2018, 15:12 Stephen J. Turnbull < > > turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > > > > What does "inherited" mean? Just that methods that are not overridden > > > retain their contracts? > > > > Contracts are attached to interfaces, not to specifications. So > > when you have abstract base class, it defines contracts, and > > implementing classes must adhere to these contracts - the can only > > strengthen it, not weaken it. > > Stated in words, it sounds very reasonable. > > Thinking about the implications for writing Python code, it sounds > very constraining in the context of duck-typing. Or, to give a > contrapositive example, you can write few, if any, contracts for > dunders beyond their signatures. For example, len(x) + len(y) == > len(x + y) is not an invariant over the classes that define __len__ > (eg, set and dict, taking the "x + y" loosely for them). Of course if > you restrict to Sequence, you can impose that invariant. The question > is the balance. > > It sounds to me like the proof of the pudding for Python will be > annotating the ABC module with contracts. If that turns out to be a > mostly fruitless exercise, it's hard to see how "real" contracts > (vs. using asserts to "program in contracting style") are an important > feature for Python in general. > > That said, given the passion of the proponents, if we can come up with > an attractive style for implementing contracts without a language > change, I'd be +0 at least for adding a module to the stdlib. But > this is a Python 3.9 project, given the state of the art. > > > This is precisely like with types, since types are contracts (and > > vice versa, in a way). > > Not Python types (== classes), though. As a constraint on behavior, > subclassing is purely nominal, although Python deliberately makes it > tedious to turn a subclass of str into a clone of int. > > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Oct 1 10:18:21 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 2 Oct 2018 00:18:21 +1000 Subject: [Python-ideas] Simplicity of C (was why is design-by-contracts not widely) In-Reply-To: <99d136ef-4492-968a-e7e2-10576ac9ffa0@trueblade.com> References: <20180930015413.GH19437@ando.pearwood.info> <20180930042635.GI19437@ando.pearwood.info> <20180930080249.GJ19437@ando.pearwood.info> <23472.48495.587491.729759@turnbull.sk.tsukuba.ac.jp> <99d136ef-4492-968a-e7e2-10576ac9ffa0@trueblade.com> Message-ID: <20181001141821.GS19437@ando.pearwood.info> On Sun, Sep 30, 2018 at 06:19:11PM -0400, Eric V. Smith wrote: > On 9/30/2018 8:11 AM, Stephen J. Turnbull wrote: > >Steven D'Aprano writes: > > > (7) You can't unit test loop invariants > > > >I don't see how a loop invariant can be elegantly specified without > >mixing it in to the implementation. Can you show an example of code > >written in a language with support for loop invariants *not* mixed > >into the implementation? > > I'd be very interested in this, too. Any pointers? I'm not sure how Stephen went from my comment that you can't unit test loop invariants to the idea that loop invariants aren't in the implementation. Obviously the loop invariant has to be attached to the loop. You can't attach a loop invariant to a method or class, because they might contain more than one loop. https://en.wikipedia.org/wiki/Loop_invariant#Eiffel from x := 0 invariant x <= 10 until x > 10 loop x := x + 1 end -- Steve From jamtlu at gmail.com Mon Oct 1 11:18:09 2018 From: jamtlu at gmail.com (James Lu) Date: Mon, 1 Oct 2018 11:18:09 -0400 Subject: [Python-ideas] Transpiling contracts In-Reply-To: References: Message-ID: <67F07F63-C3B0-4CB2-A818-C8EAFB3C24AC@gmail.com> Hi Marko, I?m going to refer to the transpiler syntax as ?block syntax.? 1. How does Eiffel do formal contracts? 2. What?s the use case for preferring block syntax over lambda syntax? Is the block syntax only better when multiple statements are needed to test a single contract condition? 2. If the last proposition is true, how often do we need multiple statements to test or specify a single contract condition? 3. When we do need multiple statements to formally verify/specify, is it usually a better idea to factor out major functionality of the contract testing to an external function? (So other contracts can use the same external function and the intent of the contract is clear trough the function name) 4. If this is true, is it a good idea to include a contracts/ folder at the same folder level or a .contracts.py file to list ?helper? or verification functions for a contract? I think we should answer questions two and three by looking at real code. ? If MockP is supported, it could be used for all the contract decorators: @snapshot(var=P.self.name) @post(var=R.attr == P.old.var) The decorator can also check the type of the returned object, and if it?s MockP it will use MockP protocol and otherwise it will check the object is callable and treat it as a lambda. Your approach has better type hinting, but I?m not sure if type hinting would be that useful if you?re adding a contract whilst writing the function. James Lu > On Oct 1, 2018, at 1:01 AM, Marko Ristin-Kaufmann wrote: > > Hi James, > >> Regarding the ?transpile into Python? syntax with with statements: Can I see an example of this syntax when used in pathlib? I?m a bit worried this syntax is too long and ?in the way?, unlike decorators which are before the function body. Or do you mean that both MockP and your syntax should be supported? >> >> Would >> >> with requiring: assert arg1 < arg2, ?message? >> >> Be the code you type or the code that?s actually run? > > > That's the code you would type. Let me make a full example (I'm omitting "with contracts" since we actually don't need it). > > You would read/write this: > def some_func(arg1: List[int])->int: > with requiring: > assert len(arg1) > 3, "some description" > > with oldie as O, resultie as result, ensuring: > if SLOW: > O.var1 = sum(arg1) > assert result > sum(arg1) > O.var1 > > assert len(result) > 5 > > This would run: > @requires(lambda P: len(P.arg1) > 3, "some description") > @snapshot(lambda P, var1: sum(P.arg1), enabled=SLOW) > @ensures(lambda O, P, result: result > sum(arg1) > O.var1, enabled=SLOW) > @ensures(lambda result: len(result) > 5) > > I omitted in the example how to specify a custom exception to avoid confusion. > > If we decide that transpiler is the way to go, I'd say it's easier to just stick with lambdas and allow no mock-based approach in transpilation. > > Cheers, > Marko > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Oct 1 11:28:55 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 2 Oct 2018 01:28:55 +1000 Subject: [Python-ideas] Simplicity of C (was why is design-by-contracts not widely) In-Reply-To: References: <20180930015413.GH19437@ando.pearwood.info> <20180930042635.GI19437@ando.pearwood.info> <20180930080249.GJ19437@ando.pearwood.info> Message-ID: <20181001152855.GT19437@ando.pearwood.info> On Sun, Sep 30, 2018 at 06:33:08PM +1000, Chris Angelico wrote: > On Sun, Sep 30, 2018 at 6:03 PM Steven D'Aprano wrote: > > > > On Sun, Sep 30, 2018 at 02:50:28PM +1000, Chris Angelico wrote: > > > > > And yet all the examples I've seen have just been poor substitutes for > > > unit tests. Can we get some examples that actually do a better job of > > > selling contracts? > > > > In no particular order... > > > > (1) Distance > > Don't doctests deal with this too? With the exact same downsides? I'm not sure that I understand the question. Yes, doctests are usually (but not always) close to the function too. That's a plus in favour of doctests. The downside is that if you want to show a metric tonne of examples, your docstring becomes huge. (Or move it out into a separate text file, which doctest can still run.) Sometimes its a real struggle to write a good docstring that isn't huge. But on the other hand, a good editor should allow you to collapse docstrings to a single line. > > (2) Self-documenting code > > Ditto Sorry, are you asking if doctests are self-documenting code? Since they aren't part of the implementation of the function, no they aren't. > > (3) The "Have you performed the *right* tests?" problem > > Great if your contracts can actually be perfectly defined. Why do you think the contracts have to be perfectly defined? I keep reading people accusing pro-Contracts people of being zealots who believe that Contracts are a magic bullet that solve every problem 100% perfectly. I don't know where people get this idea. Contracts are a tool, like unit tests. Just as a single unit test is better than no unit tests, and two unit tests is better than one, same goes for contracts. If you have an imperfect (incomplete, not buggy!) contract, that's better than *no* contract at all. If you would like to check five things, but can only specify three of them in a contract, that's better than not specifying any of them. > Every time > a weird case is mentioned, those advocating contracts (mainly Marko) > give examples showing "hey, contracts can do that too", and they're > just testing specifics. Can you give an example? Of course, contracts *can* be specific (at least in Eiffel), the syntax allows it. In pseudo-code: def spam(x, y): require: if x is None: y > 0 # implementation goes here... The precondition checks that if x is None, y must be greater than zero. On its own, that's not a very good contract since it tells us nothing about the case when x is not None, but (as per my comments above) its better than nothing. How would you do this as a unit test? For starters, you need to move the precondition into the function implementation block, giving it an explicit raise: def spam(x, y): if x is None and y <= 0: raise SomeError # implementation continues... Now you've given up any hope of disabling that specific check, short of editing the source code. Then you have to make a unit test to check that it works the way you think it does: def test_if_x_is_none_y_cannot_be_negative_or_zero(self): # Naming things is hard... self.AssertRaise(SomeError, spam, None, -1) self.AssertRaise(SomeError, spam, None, 0) Great! Now we know that if spam is passed None and -1, or None and 0, it will correctly fail. But the unit test doesn't give us any confidence that None and -9137 will fail, since we haven't tested that case. Work-around: I often write unit tests like this: def test_if_x_is_none_y_cannot_be_negative_or_zero(self): for i in range(20): # I never know how many or how few values to check... n = random.randint(1, 1000) # or which values... self.AssertRaise(SomeError, spam, None, -n) self.AssertRaise(SomeError, spam, None, 0) As David points out in another post, it is true that Contracts also can only check the values that you actually pass to the application during the testing phase, but that is likely to be a larger set of values than those we put in the unit tests, which lets us gain confidence in the code more quickly and with less effort. Rather than write lots of explicit tests, we can just exercise the routine with realistic data and see if the contract checks fail. [...] > > (6) Separation of concerns: function algorithm versus error checking > > Agreed, so long as you can define the contract in a way that isn't > just duplicating the function's own body. Sure. That's a case where contracts aren't really suitable and a few unit tests would be appropriate, but that's only likely to apply to post-conditions, not pre-conditions. > Contracts are great for some situations, but I'm seeing a lot of cases > where they're just plain not, yet advocates still say "use contracts, > use contracts". Why? For the same reason advocates say "Use tests, use tests" despite there being some cases where formal automated tests are too hard to use. Use contracts for the 90% of the cases they help, don't dismiss them because of the 10% of cases they don't. But honestly, if you prefer unit testing, that's great too. Nobody is trying to say that unit tests suck or that you shouldn't use them. This is about giving people the choice between contracts or unit tests or both. When I say "Use contracts", what I mean is "Use as many or as few contracts as makes sense for your application, according to your taste". Right now, that choice is very restrictive, and I personally don't like the look and feel of existing contract libraries. I would give my right eye for Cobra-like syntax for contracts in Python: http://cobra-language.com/trac/cobra/wiki/Contracts -- Steve From mertz at gnosis.cx Mon Oct 1 12:36:18 2018 From: mertz at gnosis.cx (David Mertz) Date: Mon, 1 Oct 2018 12:36:18 -0400 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: <20180930153132.GO19437@ando.pearwood.info> References: <37e7961b-528d-309f-114f-7194b9051892@kynesim.co.uk> <20180929111925.GU19437@ando.pearwood.info> <20180930153132.GO19437@ando.pearwood.info> Message-ID: On Sun, Sep 30, 2018 at 11:34 AM Steven D'Aprano wrote: > On Sun, Sep 30, 2018 at 10:29:50AM -0400, David Mertz wrote: > But given that in general unit tests tend to only exercise a handful of > values (have a look at the tests in the Python stdlib) I think it is > fair to say that in practice unit tests typically do not have anywhere > near the coverage of live data used during alpha and beta testing. > I still think it's a mixture. I write tests to also address "this really shouldn't happen" cases as well (often in a loop, or using a Nose class for scaffolding). There's some saying/joke about software testing along the lines of: For an argument that should be in range 1-100: try 50; try 1; try 100 try 101; try 0; try -1, try 3.14 try 1+1j; try the string "fish", try a null pointer; etc. Many of those oddball cases can easily be in a list of values to test in unit tests, but may be impossible or unlikely to make it to the function call in the normal/possible flow of the program. > I'm curious. When you write a function or method, do you include input > checks? Here's an example from the Python stdlib (docstring removed for > brevity): > > # bisect.py > def insort_right(a, x, lo=0, hi=None): > if lo < 0: > raise ValueError('lo must be non-negative') > if hi is None: > hi = len(a) > while lo < hi: > mid = (lo+hi)//2 > if x < a[mid]: hi = mid > else: lo = mid+1 > a.insert(lo, x) > > Do you consider that check for lo < 0 to be disruptive? How would you > put that in a unit test? > I definitely put in checks like that. However, I might well write a test like: assert lo >= 0, "lo must be non-negative" That would allow disabling the check for production. However, I presume the stdlib does this for a reason; it presumably wants to allow callers to catch a specific exception. I haven't seen anything in the contract discussion that allows raising a particular exception when a contract is violated, only a general one for all contracts. The case of 'if hi is None' is different. It's remediation of a missing value where it's perfectly fine to impute an unspecified value. So that would be a poor contract. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Oct 1 13:43:57 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 2 Oct 2018 02:43:57 +0900 Subject: [Python-ideas] Simplicity of C (was why is design-by-contracts not widely) In-Reply-To: <20181001141821.GS19437@ando.pearwood.info> References: <20180930015413.GH19437@ando.pearwood.info> <20180930042635.GI19437@ando.pearwood.info> <20180930080249.GJ19437@ando.pearwood.info> <23472.48495.587491.729759@turnbull.sk.tsukuba.ac.jp> <99d136ef-4492-968a-e7e2-10576ac9ffa0@trueblade.com> <20181001141821.GS19437@ando.pearwood.info> Message-ID: <23474.23773.790046.329945@turnbull.sk.tsukuba.ac.jp> Steven D'Aprano writes: > I'm not sure how Stephen went from my comment that you can't unit test > loop invariants to the idea that loop invariants aren't in the > implementation. Obviously I didn't. I went from your statement that (1) contracts are separated from the implementation and the inference from (7) that contracts can specify loop invariants to the question of whether both can be true at the same time. > Obviously the loop invariant has to be attached to the loop. You > can't attach a loop invariant to a method or class, because they > might contain more than one loop. That's not so obvious. At least in theory you could provide a label for the loop (Common Lisp for example has named blocks), and refer to that in an "invariant" section outside the function. I couldn't imagine a pleasing syntax without help from someone who's used contracts, though. That's why I asked. It's not a big deal that they aren't both simultaneously true in practice, it's just that there are a huge number of abstract claims being made for contracts, and for my money they don't look anywhere near as much of an improvement over asserts in practice as they sound in the abstract, in decorator form or in Eiffel syntax. As I said before, since a number of people are asking for them, I'm at least +0 on including a contracts module with attractive syntax for contracts in the stdlib on the general principle that we should provide batteries for as many common patterns or styles as possible. But I'm not ready to go that high for language changes for this purpose, at least not yet. Since I don't expect to be using contracts any time soon, I'll leave the definition of "attractive" up to those who expect to have to read them frequently. Steve From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Oct 1 13:50:34 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 2 Oct 2018 02:50:34 +0900 Subject: [Python-ideas] Simplicity of C (was why is design-by-contracts not widely) In-Reply-To: References: <20180930015413.GH19437@ando.pearwood.info> <20180930042635.GI19437@ando.pearwood.info> <20180930080249.GJ19437@ando.pearwood.info> <23472.48495.587491.729759@turnbull.sk.tsukuba.ac.jp> <23474.7299.429212.519412@turnbull.sk.tsukuba.ac.jp> Message-ID: <23474.24170.876365.723271@turnbull.sk.tsukuba.ac.jp> Marko Ristin-Kaufmann writes: > Do you mean annotating an abstract class with contracts or > annotating the actual functions in abc module? I mean the functions in the abc module. > Inheriting of contracts also proved useful when you define an > interface as a group of functions together with the contracts. You > define the functions as an ABC with purely static methods (no > state) and specify the contracts in that abstract class. I can envision how that would work in a situation with application- specific classes, even in a deep hierarchy, with some discipline. However, Python does not necessarily conform to that kind of discipline. I suspect that Pythonic programming styles are not terribly amenable to inheritance of contracts. I think the ABC module is most likely to provide a strong evidence that I'm wrong: if contract inheritance, and in general the contracting style of programming, works well for that module, it will work for anything written in Python. By the way, I'm not asking you to do this; you've already provided both the icontract module and a rather complete annotation of a sizeable stdlib module. Your work is appreciated! Of course I'd be happy to see a similar treatment of abc, but it's also a reasonable candidate for thought experiments, or a sample of a few cases. Steve From jamtlu at gmail.com Mon Oct 1 14:43:40 2018 From: jamtlu at gmail.com (James Lu) Date: Mon, 1 Oct 2018 14:43:40 -0400 Subject: [Python-ideas] Renaming icontract to pcontract In-Reply-To: References: <20181001071504.GA54129@cskk.homeip.net> Message-ID: <1CD22DA7-E63F-49AC-AA83-B8BACEE73909@gmail.com> I think the confusion would be minimal and not worth the hassle of renaming to pcontract. People would just say ?Python icontract? when it?s ambiguous. Sent from my iPhone > On Oct 1, 2018, at 3:28 AM, Marko Ristin-Kaufmann wrote: > > Hi Cameron, > A nerdy way to make it sound like a sentence: "I contract that ...". > > Pcontract would stand for python contract. Pycontract is already taken. > > Cheers, > Marko > >> Le lun. 1 oct. 2018 ? 09:15, Cameron Simpson a ?crit : >> On 01Oct2018 07:25, Marko Ristin-Kaufmann wrote: >> >I'd like to rename icontract into pcontract to avoid name conflict with >> >java's icontract library. >> > >> >Do you have any better suggestion? >> >> No, sounds ok to me. What was the "i" for in the old name? >> >> Cheers, >> Cameron Simpson >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Oct 1 21:12:07 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 2 Oct 2018 11:12:07 +1000 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: References: <37e7961b-528d-309f-114f-7194b9051892@kynesim.co.uk> <20180929111925.GU19437@ando.pearwood.info> <20180930153132.GO19437@ando.pearwood.info> Message-ID: <20181002011207.GV19437@ando.pearwood.info> On Mon, Oct 01, 2018 at 12:36:18PM -0400, David Mertz wrote: > There's some saying/joke about software testing along the lines of: > > For an argument that should be in range 1-100: > try 50; try 1; try 100 > try 101; try 0; > try -1, > try 3.14 > try 1+1j; > try the string "fish", > try a null pointer; > etc. > > Many of those oddball cases can easily be in a list of values to test in > unit tests, but may be impossible or unlikely to make it to the function > call in the normal/possible flow of the program. Right. And for a *library*, you must handle out of range errors, because the library cannot trust the input passed to it by third parties. If a third party passes "fish" as an argument, that's not a bug in the library and the library cannot fix it, it should handle the error gracefully with a sensible error message. But for an *application*, the internal routines are completely under control of the one developer team. If an internal routine passes "fish", that's a bug that the dev team can and ought to fix. Once that bug is fixed, there's no need to check for "fish" in production code. Its a waste of time. (In principle at least -- it depends on how confident you or your QA team are that all the bugs are ironed out. Do you ship your apps with debugging turned off? Then you might ship with contract checking turned off, or at least dialled back to a lower level.) For an application, it doesn't matter if my function sets the computer on fire when passed the string "fish", if there is no way for the application to pass that string to the function. If it can't happen, it can't happen and there's no need to defend against it beyond a regression test. (Modulo comments about how confident you are about the software quality.) For a library, anyone can pass "fish" to anything, so you better handle it gracefully rather than setting the computer on fire. For application developers, the point of contracts is to make error conditions *impossible* rather than to just defend against them, so you can turn off the defensive error checking in production because they aren't needed. > > I'm curious. When you write a function or method, do you include input > > checks? Here's an example from the Python stdlib (docstring removed for > > brevity): > > > > # bisect.py > > def insort_right(a, x, lo=0, hi=None): > > if lo < 0: > > raise ValueError('lo must be non-negative') > > if hi is None: > > hi = len(a) > > while lo < hi: > > mid = (lo+hi)//2 > > if x < a[mid]: hi = mid > > else: lo = mid+1 > > a.insert(lo, x) > > > > Do you consider that check for lo < 0 to be disruptive? How would you > > put that in a unit test? > > > > I definitely put in checks like that. However, I might well write a test > like: > > assert lo >= 0, "lo must be non-negative" > > That would allow disabling the check for production. For a library function, I would consider that an abuse of assert, since: (1) It raises the wrong exception (AssertionError); (2) It can be disabled; (3) And it sends the wrong message, telling the reader that lo is an internal implementation detail rather than part of the function's external API. > However, I presume > the stdlib does this for a reason; it presumably wants to allow callers to > catch a specific exception. I haven't seen anything in the contract > discussion that allows raising a particular exception when a contract is > violated, only a general one for all contracts. I think that the Eiffel philosophy is that all contract violations are assertion errors, but I'm not really confident in my understanding of Eiffel's exception mechanism. It doesn't have a generalised try...except statement like Python. If Python had contracts, I'd want to see: - by default, contract violations raise specific subclasses of AssertionError, e.g. RequireError, EnsureError; - but there ought to be a mechanism to allow specific errors to raise more specific exceptions. I haven't given any thought to that mechanism. It might simply be "the status quo" -- there's no reason why we *must* move defensive input checking into the (hypothetical) require block just because it is available. You can still write your test the old fashioned way in the body of the function and raise the exception of your choice. > The case of 'if hi is None' is different. It's remediation of a missing > value where it's perfectly fine to impute an unspecified value. So that > would be a poor contract. Since the check for None is not error checking at all, I didn't talk about it or make it a precondition test. -- Steve From steve at pearwood.info Mon Oct 1 21:23:31 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 2 Oct 2018 11:23:31 +1000 Subject: [Python-ideas] Simplicity of C (was why is design-by-contracts not widely) In-Reply-To: <23474.23773.790046.329945@turnbull.sk.tsukuba.ac.jp> References: <20180930015413.GH19437@ando.pearwood.info> <20180930042635.GI19437@ando.pearwood.info> <20180930080249.GJ19437@ando.pearwood.info> <23472.48495.587491.729759@turnbull.sk.tsukuba.ac.jp> <99d136ef-4492-968a-e7e2-10576ac9ffa0@trueblade.com> <20181001141821.GS19437@ando.pearwood.info> <23474.23773.790046.329945@turnbull.sk.tsukuba.ac.jp> Message-ID: <20181002012331.GW19437@ando.pearwood.info> On Tue, Oct 02, 2018 at 02:43:57AM +0900, Stephen J. Turnbull wrote: > Steven D'Aprano writes: > > > I'm not sure how Stephen went from my comment that you can't unit test > > loop invariants to the idea that loop invariants aren't in the > > implementation. > > Obviously I didn't. I went from your statement that (1) contracts are > separated from the implementation and the inference from (7) that > contracts can specify loop invariants to the question of whether both > can be true at the same time. Okay, fair enough. The loop invariant is separate in the sense that it is in a distinctly named block. But it connected in the sense that the loop invariant is attached to the loop, which is inside the function implementation, rather than in a separate block outside of the implementation. In Python terms, we might say something like: for x in sequence: BLOCK else: BLOCK invariant: BLOCK Static tools could parse the source and extract the invariant blocks, editors could allow you to collapse them out of sight. But unlike the (hypothetical) require and ensure blocks, its right there inside the function implementation. We presumably wouldn't want the invariant to be far away from the loop: ... code ... more code for x in sequence: BLOCK ... more code ... more code ... # and far, far away invariant: BLOCK Even if we had some mechanism to connect the two (some sort of label, as you suggest) it would suck from a readability perspective to have to jump backwards and forwards from the loop to the loop invariant. Things which are closely connected ought to be close together. -- Steve From mertz at gnosis.cx Mon Oct 1 21:48:18 2018 From: mertz at gnosis.cx (David Mertz) Date: Mon, 1 Oct 2018 21:48:18 -0400 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: <20181002011207.GV19437@ando.pearwood.info> References: <37e7961b-528d-309f-114f-7194b9051892@kynesim.co.uk> <20180929111925.GU19437@ando.pearwood.info> <20180930153132.GO19437@ando.pearwood.info> <20181002011207.GV19437@ando.pearwood.info> Message-ID: On Mon, Oct 1, 2018, 9:13 PM Steven D'Aprano wrote: > For an application, it doesn't matter if my function sets the computer on > fire when passed the string "fish", if there is no way for the application > to pass that string to the function. If it can't happen, it can't happen > and there's no need to defend against it beyond a regression test. > How many times have you written or seen a comment in code similar to "This can't possibly happen!!" ... Usually in response to a failed debugging attempt. It's really hard to understands all possible execution paths that might result from all possible inputs and program states. "Fail early and fail hard" is a good principle... And indeed one generally in a spirit compatible with DbC. Beware of bugs in the above code; I have only proved it correct, not tried it. - Donald Knuth -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko.ristin at gmail.com Tue Oct 2 00:47:58 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Tue, 2 Oct 2018 06:47:58 +0200 Subject: [Python-ideas] Renaming icontract to pcontract In-Reply-To: <1CD22DA7-E63F-49AC-AA83-B8BACEE73909@gmail.com> References: <20181001071504.GA54129@cskk.homeip.net> <1CD22DA7-E63F-49AC-AA83-B8BACEE73909@gmail.com> Message-ID: Ok, let's keep icontract as it was already referenced a couple of times at least on this list. I'll rename the pypi package to icontract2 and the module as well as soon as I change the interface. On Mon, 1 Oct 2018 at 20:43, James Lu wrote: > I think the confusion would be minimal and not worth the hassle of > renaming to pcontract. People would just say ?Python icontract? when it?s > ambiguous. > > Sent from my iPhone > > On Oct 1, 2018, at 3:28 AM, Marko Ristin-Kaufmann > wrote: > > Hi Cameron, > A nerdy way to make it sound like a sentence: "I contract that ...". > > Pcontract would stand for python contract. Pycontract is already taken. > > Cheers, > Marko > > Le lun. 1 oct. 2018 ? 09:15, Cameron Simpson a ?crit : > >> On 01Oct2018 07:25, Marko Ristin-Kaufmann wrote: >> >I'd like to rename icontract into pcontract to avoid name conflict with >> >java's icontract library. >> > >> >Do you have any better suggestion? >> >> No, sounds ok to me. What was the "i" for in the old name? >> >> Cheers, >> Cameron Simpson >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko.ristin at gmail.com Tue Oct 2 01:14:39 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Tue, 2 Oct 2018 07:14:39 +0200 Subject: [Python-ideas] Transpiling contracts In-Reply-To: <67F07F63-C3B0-4CB2-A818-C8EAFB3C24AC@gmail.com> References: <67F07F63-C3B0-4CB2-A818-C8EAFB3C24AC@gmail.com> Message-ID: Hi James, I had another take at it. I wrote it down in the github issue ( https://github.com/Parquery/icontract/issues/48#issuecomment-426147468): SLOW=os.environ.get("SOME_ENVIRONMENT_VARIABLE", "") != "" class SomeClass: def __init__(self)->None: self.some_prop = 1984 def some_func(arg1: List[int], arg2: List[int])->SomeClass: with requiring: len(arg1) > 3, "some description" len(arg2) > 5, ValueError("some format {} {}".format(arg1, arg2)) if SLOW: all(elt in arg2 for elt in arg1) len(arg2) == len(set(arg2)) result = None # type: SomeClass with ensuring, oldie as O: len(arg1) == len(O.arg1) result.some_prop < len(arg2) This transpiles to/from: SLOW=os.environ.get("SOME_ENVIRONMENT_VARIABLE", "") != "" class SomeClass: def __init__(self)->None: self.some_prop = 1984 @requires(lambda P: len(P.arg1) > 3, "some description")@requires(lambda P: len(P.arg2), error=lambda P: ValueError("some format {} {}".format(P.arg1, P.arg2)))@requires(lambda P: all(elt in P.arg2 for elt in P.arg1), enabled=SLOW)@requires(lambda P: len(arg2) == len(set(arg2)))@ensures(lambda O, P: len(P.arg1) == len(O.arg1))@ensures(lambda P, result: result.some_prop < len(P.arg2))def some_func(arg1: List[int], arg2: List[int])->SomeClass: ... 2. What?s the use case for preferring block syntax over lambda syntax? Is > the block syntax only better when multiple statements are needed to test a > single contract condition? > Actually no. The condition must be a single statement (specified as a boolean expression) regardless of the form (readable or executable). My idea was that contracts would be a more readable + easier to type and group by the respective enabled flags. It should be a 1-to-1 back and forth transformation. I didn't aim for any other advantages other than readability/easier modification. The use case I have in mind is not to separate each file into two (executable and readable). I'm more thinking of a tool that I can attach to key shortcuts in the IDE that let me quickly transform the contracts into readable form when I need to read them / make my modifications and then convert them back into executable form. 3. When we do need multiple statements to formally verify/specify, is it > usually a better idea to factor out major functionality of the contract > testing to an external function? (So other contracts can use the same > external function and the intent of the contract is clear trough the > function name) > I would say so (i.e. no multi-statement conditions) . Making multi-statement conditions would be rather confusing and also harder to put into documentation. And it would be hard to implement :) 4. If this is true, is it a good idea to include a contracts/ folder at the > same folder level or a .contracts.py file to list ?helper? or verification > functions for a contract? > So far, we never had to make a function external to a contract -- it was often the signal that something had to be refactored out of the function if the condition became complex, but the resulting functions usually either stayed in the same file or were more general and had to move to a separate module anyhow. Our Python code base has about 95K LOC, I don't know how contracts behave on larger code bases or how they would need to be structured. Cheers, Marko On Mon, 1 Oct 2018 at 17:18, James Lu wrote: > Hi Marko, > > I?m going to refer to the transpiler syntax as ?block syntax.? > > 1. How does Eiffel do formal contracts? > 2. What?s the use case for preferring block syntax over lambda syntax? Is > the block syntax only better when multiple statements are needed to test a > single contract condition? > 2. If the last proposition is true, how often do we need multiple > statements to test or specify a single contract condition? > 3. When we do need multiple statements to formally verify/specify, is it > usually a better idea to factor out major functionality of the contract > testing to an external function? (So other contracts can use the same > external function and the intent of the contract is clear trough the > function name) > 4. If this is true, is it a good idea to include a contracts/ folder at > the same folder level or a .contracts.py file to list ?helper? or > verification functions for a contract? > > I think we should answer questions two and three by looking at real code. > ? > > If MockP is supported, it could be used for all the contract decorators: > @snapshot(var=P.self.name) > @post(var=R.attr == P.old.var) > > The decorator can also check the type of the returned object, and if it?s > MockP it will use MockP protocol and otherwise it will check the object is > callable and treat it as a lambda. > > Your approach has better type hinting, but I?m not sure if type hinting > would be that useful if you?re adding a contract whilst writing the > function. > > James Lu > > On Oct 1, 2018, at 1:01 AM, Marko Ristin-Kaufmann > wrote: > > Hi James, > > Regarding the ?transpile into Python? syntax with with statements: Can I >> see an example of this syntax when used in pathlib? I?m a bit worried this >> syntax is too long and ?in the way?, unlike decorators which are before the >> function body. Or do you mean that both MockP and your syntax should be >> supported? >> >> Would >> >> with requiring: assert arg1 < arg2, ?message? >> >> Be the code you type or the code that?s actually run? >> > > That's the code you would type. Let me make a full example (I'm omitting > "with contracts" since we actually don't need it). > > You would read/write this: > def some_func(arg1: List[int])->int: > with requiring: > assert len(arg1) > 3, "some description" > > with oldie as O, resultie as result, ensuring: > if SLOW: > O.var1 = sum(arg1) > assert result > sum(arg1) > O.var1 > > assert len(result) > 5 > > This would run: > @requires(lambda P: len(P.arg1) > 3, "some description") > @snapshot(lambda P, var1: sum(P.arg1), enabled=SLOW) > @ensures(lambda O, P, result: result > sum(arg1) > O.var1, enabled=SLOW) > @ensures(lambda result: len(result) > 5) > > I omitted in the example how to specify a custom exception to avoid > confusion. > > If we decide that transpiler is the way to go, I'd say it's easier to just > stick with lambdas and allow no mock-based approach in transpilation. > > Cheers, > Marko > > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko.ristin at gmail.com Tue Oct 2 01:25:46 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Tue, 2 Oct 2018 07:25:46 +0200 Subject: [Python-ideas] Simplicity of C (was why is design-by-contracts not widely) In-Reply-To: <20181001152855.GT19437@ando.pearwood.info> References: <20180930015413.GH19437@ando.pearwood.info> <20180930042635.GI19437@ando.pearwood.info> <20180930080249.GJ19437@ando.pearwood.info> <20181001152855.GT19437@ando.pearwood.info> Message-ID: Hi Steven (D'Aprano), Right now, that choice is very restrictive, and I personally don't like > the look and feel of existing contract libraries. I would give my right > eye for Cobra-like syntax for contracts in Python: > > http://cobra-language.com/trac/cobra/wiki/Contracts > You might be interested in the discussion on transpiling contracts back and forth. See the other thread "Transpile contracts" and the issue in github: https://github.com/Parquery/icontract/issues/48 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfine2358 at gmail.com Tue Oct 2 07:31:35 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Tue, 2 Oct 2018 12:31:35 +0100 Subject: [Python-ideas] "while:" for the loop In-Reply-To: References: <23468.29167.517011.19676@turnbull.sk.tsukuba.ac.jp> Message-ID: Mikhail V has suggested "while:" as a shorthand for "while True:". He's also provided a helpful list of URLs for related discussion. I'd like to suggest another approach. My suggestion is to improved documentation and help. For me a search for python "while True" produces as the top result https://wiki.python.org/moin/WhileLoop. Please take a look at this page. I don't think it's suitable for novices. In particular, about 25% of the page compares "while 1:" to "while True:" and changes between Python2 and Python3. Searching for site:docs.python.org "while True" produces as top result https://docs.python.org/3/tutorial/controlflow.html. Please also look at this page. It has two examples of "while True:", but does not say that "while True:" means loop forever (unless there is a break or an exception). Today I saw https://blog.codeclub.org.uk/2018/06/13/how-to-move-from-scratch-to-python-in-code-club/ On this page, a Code Club volunteer gives as debugging advice I use comparisons of Scratch to Python at the beginning, for example to explain that ?while True? is the same as the ?Forever? block. I also have some Python cheat sheets which give examples of common Python code. Some of the learners find it easier to debug Python than Scratch, because Python gives you error messages to indicate what went wrong. It?s usually down to spelling errors! My final URL is https://www.youtube.com/watch?v=QEJ8tE9lAeE. This is a 3 minute video: Python For Beginners - How to use a While True Loop in Python. Something that can be done now, which can soon bring benefit to many, is to find or write material that will help novices master Python idioms such as "while True:". I'm sure the Code Club would like to put such material on its web site. -- Jonathan On Sat, Sep 29, 2018 at 4:58 AM Mikhail V wrote: > > I put the list of related discussion here, just in case. > > Same suggestion: > https://mail.python.org/pipermail/python-dev/2005-July/054914.html > > Idea for the "loop" keyword: > https://mail.python.org/pipermail/python-ideas/2014-June/028202.html > (followed by the same suggestion from @Random832: > https://mail.python.org/pipermail/python-ideas/2014-June/028220.html) > > Somewhat related PEP (rejected) + discussion links inside: > https://legacy.python.org/dev/peps/pep-0315/ > (I've meditated a bit on this - but could not get what was > actually the point of that idea.) > > Plus some other related discussions, maybe not directly related, > but something along the lines might be interesting. > > An old and lively one: > https://mail.python.org/pipermail/python-list/1999-April/002557.html > https://mail.python.org/pipermail/python-list/1999-May/008535.html > Another one: > https://mail.python.org/pipermail/python-list/2000-December/029972.html > ( Particularly this: > https://mail.python.org/pipermail/python-list/2000-December/052694.html > ) > Yet another "1" vs "True" debate: > https://mail.python.org/pipermail/python-list/2012-January/618649.html > New era: > https://mail.python.org/pipermail/python-list/2017-April/721182.html > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rhodri at kynesim.co.uk Tue Oct 2 10:26:34 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Tue, 2 Oct 2018 15:26:34 +0100 Subject: [Python-ideas] "while:" for the loop In-Reply-To: References: <23468.29167.517011.19676@turnbull.sk.tsukuba.ac.jp> Message-ID: <6642b2fd-d7d5-74b1-b82f-e659f30480c2@kynesim.co.uk> On 02/10/18 12:31, Jonathan Fine wrote: > Mikhail V has suggested "while:" as a shorthand for "while True:". > He's also provided a helpful list of URLs for related discussion. I'd > like to suggest another approach. Before we hare off into the middle distance, could you show that there is a problem that needs to be addressed here? I killfiled Mikhail a long time ago, so I haven't actually seen any justification here beyond "I don't want to have to type five more characters." > My suggestion is to improved documentation and help. For me a search for > python "while True" > produces as the top result https://wiki.python.org/moin/WhileLoop. > > Please take a look at this page. I don't think it's suitable for > novices. In particular, about 25% of the page compares "while 1:" to > "while True:" and changes between Python2 and Python3. Is it intended solely for novices? > > Searching for > site:docs.python.org "while True" > produces as top result https://docs.python.org/3/tutorial/controlflow.html. > > Please also look at this page. It has two examples of "while True:", > but does not say that "while True:" means loop forever (unless there > is a break or an exception). This is an invidious example. "while" is actually introduced on the previous page, as the first paragraph says. Expecting a statement of the blindingly obvious here is inappropriate, surely. (And yes, my experience with early teens is that it *is* blindingly obvious.) > > Today I saw https://blog.codeclub.org.uk/2018/06/13/how-to-move-from-scratch-to-python-in-code-club/ > > On this page, a Code Club volunteer gives as debugging advice > > I use comparisons of Scratch to Python at the beginning, for example > to explain that ?while True? is the same as the ?Forever? block. I > also have some Python cheat sheets which give examples of common > Python code. Some of the learners find it easier to debug Python than > Scratch, because Python gives you error messages to indicate what went > wrong. It?s usually down to spelling errors! > Um, yay? I'm not sure what your point here is. So Scratch has a Forever block, so what? > My final URL is https://www.youtube.com/watch?v=QEJ8tE9lAeE. > This is a 3 minute video: Python For Beginners - How to use a While > True Loop in Python. > > Something that can be done now, which can soon bring benefit to many, > is to find or write material that will help novices master Python > idioms such as "while True:". I'm sure the Code Club would like to put > such material on its web site. I'm sorry, but I find the idea that idioms like "while True:" need to be *mastered* a tad patronising. I have genuinely not had to take more than five minutes to get it through to an uninterested and not particularly technical 12-year old. -- Rhodri James *-* Kynesim Ltd From ned at nedbatchelder.com Tue Oct 2 10:54:51 2018 From: ned at nedbatchelder.com (Ned Batchelder) Date: Tue, 2 Oct 2018 10:54:51 -0400 Subject: [Python-ideas] Transpiling contracts In-Reply-To: References: <67F07F63-C3B0-4CB2-A818-C8EAFB3C24AC@gmail.com> Message-ID: I'm getting confused: is this still about an idea for Python, or development of a third-party library? --Ned. On 10/2/18 1:14 AM, Marko Ristin-Kaufmann wrote: > Hi James, > > I had another take at it. I wrote it down in the github issue > (https://github.com/Parquery/icontract/issues/48#issuecomment-426147468): > > SLOW=os.environ.get("SOME_ENVIRONMENT_VARIABLE","")!= "" > > class SomeClass: > def __init__(self)->None: > self.some_prop= 1984 > > def some_func(arg1: List[int],arg2: List[int])->SomeClass: > with requiring: > len(arg1)> 3,"some description" > len(arg2)> 5,ValueError("some format {} {}".format(arg1, arg2)) > > if SLOW: > all(eltin arg2for eltin arg1) > len(arg2)== len(set(arg2)) > > result= None # type: SomeClass > with ensuring, oldieas O: > len(arg1)== len(O.arg1) > result.some_prop< len(arg2) > > This transpiles to/from: > > SLOW=os.environ.get("SOME_ENVIRONMENT_VARIABLE","")!= "" > > class SomeClass: > def __init__(self)->None: > self.some_prop= 1984 > > @requires(lambda P:len(P.arg1)> 3,"some description") > @requires(lambda P:len(P.arg2), > error=lambda P:ValueError("some format {} {}".format(P.arg1, P.arg2))) > @requires(lambda P:all(eltin P.arg2for eltin P.arg1),enabled=SLOW) > @requires(lambda P:len(arg2)== len(set(arg2))) > @ensures(lambda O,P:len(P.arg1)== len(O.arg1)) > @ensures(lambda P,result: result.some_prop< len(P.arg2)) > def some_func(arg1: List[int],arg2: List[int])->SomeClass: > ... > > 2. What?s the use case for preferring block syntax over lambda > syntax? Is the block syntax only better when multiple statements > are needed to test a single contract condition? > > Actually no. The condition must be a single statement (specified as a > boolean expression) regardless of the form (readable or executable). > > My idea was that contracts would be a more readable + easier to type > and group by the respective enabled flags. It should be a 1-to-1 back > and forth transformation. I didn't aim for any other advantages other > than readability/easier modification. > > The use case I have in mind is not to separate each file into two > (executable and readable). I'm more thinking of a tool that I can > attach to key shortcuts in the IDE that let me quickly transform the > contracts into readable form when I need to read them / make my > modifications and then convert them back into executable form. > > 3. When we do need multiple statements to formally verify/specify, > is it usually a better idea to factor out major functionality of > the contract testing to an external function? (So other contracts > can use the same external function and the intent of the contract > is clear trough the function name) > > > I would say so (i.e. no multi-statement conditions) . Making > multi-statement conditions would be rather confusing and also harder > to put into documentation. And it would be hard to implement :) > > 4. If this is true, is it a good idea to include a contracts/ > folder at the same folder level or a .contracts.py file to list > ?helper? or verification functions for a contract? > > > So far, we never had to make a function external to a contract -- it > was often the signal that something had to be refactored out of the > function if the condition became complex, but the resulting functions > usually either stayed in the same file or were more general and had to > move to a separate module anyhow. Our Python code base has about 95K > LOC, I don't know how contracts behave on larger code bases or how > they would need to be structured. > > Cheers, > Marko > > > > On Mon, 1 Oct 2018 at 17:18, James Lu > wrote: > > Hi Marko, > > I?m going to refer to the transpiler syntax as ?block syntax.? > > 1. How does Eiffel do formal contracts? > 2. What?s the use case for preferring block syntax over lambda > syntax? Is the block syntax only better when multiple statements > are needed to test a single contract condition? > 2. If the last proposition is true, how often do we need multiple > statements to test or specify a single contract condition? > 3. When we do need multiple statements to formally verify/specify, > is it usually a better idea to factor out major functionality of > the contract testing to an external function? (So other contracts > can use the same external function and the intent of the contract > is clear trough the function name) > 4. If this is true, is it a good idea to include a contracts/ > folder at the same folder level or a .contracts.py file to list > ?helper? or verification functions for a contract? > > I think we should answer questions two and three by looking at > real code. > ? > > If MockP is supported, it could be used for all the contract > decorators: > @snapshot(var=P.self.name ) > @post(var=R.attr == P.old.var) > > The decorator can also check the type of the returned object, and > if it?s MockP it will use MockP protocol and otherwise it will > check the object is callable and treat it as a lambda. > > Your approach has better type hinting, but I?m not sure if type > hinting would be that useful if you?re adding a contract whilst > writing the function. > > James Lu > > On Oct 1, 2018, at 1:01 AM, Marko Ristin-Kaufmann > > wrote: > >> Hi James, >> >> Regarding the ?transpile into Python? syntax with with >> statements: Can I see an example of this syntax when used in >> pathlib? I?m a bit worried this syntax is too long and ?in >> the way?, unlike decorators which are before the function >> body. Or do you mean that both MockP and your syntax should >> be supported? >> >> Would >> >> with requiring: assert arg1 < arg2, ?message? >> >> Be the code you type or the code that?s actually run? >> >> >> That's the code you would type. Let me make a full example (I'm >> omitting "with contracts" since we actually don't need it). >> >> You would read/write this: >> def some_func(arg1: List[int])->int: >> ? with requiring: >> ? ? assert len(arg1) > 3, "some description" >> >> ? with oldie as O, resultie as result, ensuring: >> ? ? if SLOW: >> ? ? ? O.var1 = sum(arg1) >> ? ? ? assert result > sum(arg1) > O.var1 >> >> ? ? assert len(result) > 5 >> >> This would run: >> @requires(lambda P: len(P.arg1) > 3, "some description") >> @snapshot(lambda P, var1: sum(P.arg1), enabled=SLOW) >> @ensures(lambda O, P, result: result > sum(arg1) > O.var1, >> enabled=SLOW) >> @ensures(lambda result: len(result) > 5) >> >> I omitted in the example how to specify a custom exception to >> avoid confusion. >> >> If we decide that transpiler is the way to go, I'd say it's >> easier to just stick with lambdas and allow no mock-based >> approach in transpilation. >> >> Cheers, >> Marko >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From van.lindberg at gmail.com Tue Oct 2 11:01:00 2018 From: van.lindberg at gmail.com (VanL) Date: Tue, 2 Oct 2018 10:01:00 -0500 Subject: [Python-ideas] Maintaining civility - the core of the Python community Message-ID: Hello everyone, You all have probably noted that there have been some contentious threads recently, ultimately ending in a few people being given a time-out from posting on these lists. I don't normally get into things on this list, but it has been generally discouraging to see a bunch of generally nice and reasonable people get sidetracked into unproductive, sarcastic, and unhelpful remarks. There are some efforts underway to formalize what is in and out of bounds - but I would suggest that we are losing our way when we need to get to explicit lists of things not to do. Unfortunately, we are getting there. I would like to reemphasize that we are bound together by the things that we share. We love working on Python. We love being in the community and seeing it grow. We can make our community stronger and more pleasant by choosing to emphasize the things that we have in common and by ignoring or avoiding topics that are more likely to generate unproductive discussion. We can and should also try to remember that not everyone is coming from the same place, and so we should actively assume the best of others and interpret what they say in the most charitable way possible. Think of it as Postel's law [1] as applied to people. I'd also suggest that generally, 1) use of profanity, 2) use of sexual terms and imagery, and 3) use of specifically denigrating terms to refer to a person [2][3][4] are completely out of bounds for the professional environment we want to maintain. It is ok to attack arguments, and ideas, but ad hominem arguments - those attack a person, rather than the person's argument - are also inappropriate. Use of sarcasm should be strongly moderated, as it is not correctly interpreted by many people. No reply is needed to this email. Instead, I'd prefer to see a continuation of solid technical discussion, rather than meta-discussion. Thanks, Van [1] https://en.wikipedia.org/wiki/Robustness_principle [2] https://en.wikipedia.org/wiki/List_of_ethnic_slurs [3] https://en.wikipedia.org/wiki/List_of_religious_slurs [4] https://en.wikipedia.org/wiki/List_of_disability-related_terms_with_negative_connotations -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko.ristin at gmail.com Tue Oct 2 11:05:25 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Tue, 2 Oct 2018 17:05:25 +0200 Subject: [Python-ideas] Transpiling contracts In-Reply-To: References: <67F07F63-C3B0-4CB2-A818-C8EAFB3C24AC@gmail.com> Message-ID: Hi Ned, The idea is to polish a proof-of-concept library and then try to introduce it into the standard libs eventually. On Tue, 2 Oct 2018 at 16:57, Ned Batchelder wrote: > I'm getting confused: is this still about an idea for Python, or > development of a third-party library? > > --Ned. > > On 10/2/18 1:14 AM, Marko Ristin-Kaufmann wrote: > > Hi James, > > I had another take at it. I wrote it down in the github issue ( > https://github.com/Parquery/icontract/issues/48#issuecomment-426147468): > > SLOW=os.environ.get("SOME_ENVIRONMENT_VARIABLE", "") != "" > class SomeClass: > def __init__(self)->None: > self.some_prop = 1984 > def some_func(arg1: List[int], arg2: List[int])->SomeClass: > with requiring: > len(arg1) > 3, "some description" > len(arg2) > 5, ValueError("some format {} {}".format(arg1, arg2)) > > if SLOW: > all(elt in arg2 for elt in arg1) > len(arg2) == len(set(arg2)) > > result = None # type: SomeClass > with ensuring, oldie as O: > len(arg1) == len(O.arg1) > result.some_prop < len(arg2) > > > This transpiles to/from: > > SLOW=os.environ.get("SOME_ENVIRONMENT_VARIABLE", "") != "" > class SomeClass: > def __init__(self)->None: > self.some_prop = 1984 > @requires(lambda P: len(P.arg1) > 3, "some description")@requires(lambda P: len(P.arg2), > error=lambda P: ValueError("some format {} {}".format(P.arg1, P.arg2)))@requires(lambda P: all(elt in P.arg2 for elt in P.arg1), enabled=SLOW)@requires(lambda P: len(arg2) == len(set(arg2)))@ensures(lambda O, P: len(P.arg1) == len(O.arg1))@ensures(lambda P, result: result.some_prop < len(P.arg2))def some_func(arg1: List[int], arg2: List[int])->SomeClass: > ... > > > 2. What?s the use case for preferring block syntax over lambda syntax? Is >> the block syntax only better when multiple statements are needed to test a >> single contract condition? >> > Actually no. The condition must be a single statement (specified as a > boolean expression) regardless of the form (readable or executable). > > My idea was that contracts would be a more readable + easier to type and > group by the respective enabled flags. It should be a 1-to-1 back and forth > transformation. I didn't aim for any other advantages other than > readability/easier modification. > > The use case I have in mind is not to separate each file into two > (executable and readable). I'm more thinking of a tool that I can attach to > key shortcuts in the IDE that let me quickly transform the contracts into > readable form when I need to read them / make my modifications and then > convert them back into executable form. > > 3. When we do need multiple statements to formally verify/specify, is it >> usually a better idea to factor out major functionality of the contract >> testing to an external function? (So other contracts can use the same >> external function and the intent of the contract is clear trough the >> function name) >> > > I would say so (i.e. no multi-statement conditions) . Making > multi-statement conditions would be rather confusing and also harder to put > into documentation. And it would be hard to implement :) > > 4. If this is true, is it a good idea to include a contracts/ folder at >> the same folder level or a .contracts.py file to list ?helper? or >> verification functions for a contract? >> > > So far, we never had to make a function external to a contract -- it was > often the signal that something had to be refactored out of the function if > the condition became complex, but the resulting functions usually either > stayed in the same file or were more general and had to move to a separate > module anyhow. Our Python code base has about 95K LOC, I don't know how > contracts behave on larger code bases or how they would need to be > structured. > > Cheers, > Marko > > > > On Mon, 1 Oct 2018 at 17:18, James Lu wrote: > >> Hi Marko, >> >> I?m going to refer to the transpiler syntax as ?block syntax.? >> >> 1. How does Eiffel do formal contracts? >> 2. What?s the use case for preferring block syntax over lambda syntax? Is >> the block syntax only better when multiple statements are needed to test a >> single contract condition? >> 2. If the last proposition is true, how often do we need multiple >> statements to test or specify a single contract condition? >> 3. When we do need multiple statements to formally verify/specify, is it >> usually a better idea to factor out major functionality of the contract >> testing to an external function? (So other contracts can use the same >> external function and the intent of the contract is clear trough the >> function name) >> 4. If this is true, is it a good idea to include a contracts/ folder at >> the same folder level or a .contracts.py file to list ?helper? or >> verification functions for a contract? >> >> I think we should answer questions two and three by looking at real code. >> ? >> >> If MockP is supported, it could be used for all the contract decorators: >> @snapshot(var=P.self.name) >> @post(var=R.attr == P.old.var) >> >> The decorator can also check the type of the returned object, and if it?s >> MockP it will use MockP protocol and otherwise it will check the object is >> callable and treat it as a lambda. >> >> Your approach has better type hinting, but I?m not sure if type hinting >> would be that useful if you?re adding a contract whilst writing the >> function. >> >> James Lu >> >> On Oct 1, 2018, at 1:01 AM, Marko Ristin-Kaufmann >> wrote: >> >> Hi James, >> >> Regarding the ?transpile into Python? syntax with with statements: Can I >>> see an example of this syntax when used in pathlib? I?m a bit worried this >>> syntax is too long and ?in the way?, unlike decorators which are before the >>> function body. Or do you mean that both MockP and your syntax should be >>> supported? >>> >>> Would >>> >>> with requiring: assert arg1 < arg2, ?message? >>> >>> Be the code you type or the code that?s actually run? >>> >> >> That's the code you would type. Let me make a full example (I'm omitting >> "with contracts" since we actually don't need it). >> >> You would read/write this: >> def some_func(arg1: List[int])->int: >> with requiring: >> assert len(arg1) > 3, "some description" >> >> with oldie as O, resultie as result, ensuring: >> if SLOW: >> O.var1 = sum(arg1) >> assert result > sum(arg1) > O.var1 >> >> assert len(result) > 5 >> >> This would run: >> @requires(lambda P: len(P.arg1) > 3, "some description") >> @snapshot(lambda P, var1: sum(P.arg1), enabled=SLOW) >> @ensures(lambda O, P, result: result > sum(arg1) > O.var1, enabled=SLOW) >> @ensures(lambda result: len(result) > 5) >> >> I omitted in the example how to specify a custom exception to avoid >> confusion. >> >> If we decide that transpiler is the way to go, I'd say it's easier to >> just stick with lambdas and allow no mock-based approach in transpilation. >> >> Cheers, >> Marko >> >> >> > > _______________________________________________ > Python-ideas mailing listPython-ideas at python.orghttps://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Tue Oct 2 11:20:59 2018 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 2 Oct 2018 11:20:59 -0400 Subject: [Python-ideas] Transpiling contracts In-Reply-To: References: <67F07F63-C3B0-4CB2-A818-C8EAFB3C24AC@gmail.com> Message-ID: <22262f4d-3b1a-b3c4-6417-482eb0fe04dd@trueblade.com> On 10/2/2018 11:05 AM, Marko Ristin-Kaufmann wrote: > Hi Ned, > > The idea is to polish a proof-of-concept library and then try to > introduce it into the standard libs eventually. I'd suggest taking this off-list until such a library is developed, then. But, if the library needs some hook provided by Python to make it work (or work better), then I think it's appropriate to discuss that on this list. Eric From alan.cristh at gmail.com Tue Oct 2 13:47:57 2018 From: alan.cristh at gmail.com (Alan Cristhian) Date: Tue, 2 Oct 2018 14:47:57 -0300 Subject: [Python-ideas] Simplicity of C (was why is design-by-contracts not widely) In-Reply-To: References: <20180930015413.GH19437@ando.pearwood.info> <20180930042635.GI19437@ando.pearwood.info> <20180930080249.GJ19437@ando.pearwood.info> <20181001152855.GT19437@ando.pearwood.info> Message-ID: Hi, I implement my own design-by-contract module. You can see it here: https://github.com/AlanCristhian/eiffel I did this as an experiment, I have no real life experience with the design by contract approach. El mar., 2 oct. 2018 a las 2:28, Marko Ristin-Kaufmann (< marko.ristin at gmail.com>) escribi?: > Hi Steven (D'Aprano), > > Right now, that choice is very restrictive, and I personally don't like >> the look and feel of existing contract libraries. I would give my right >> eye for Cobra-like syntax for contracts in Python: >> >> http://cobra-language.com/trac/cobra/wiki/Contracts >> > > You might be interested in the discussion on transpiling contracts back > and forth. See the other thread "Transpile contracts" and the issue in > github: https://github.com/Parquery/icontract/issues/48 > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Tue Oct 2 20:27:03 2018 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 2 Oct 2018 20:27:03 -0400 Subject: [Python-ideas] f-string "debug" conversion Message-ID: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> This idea was proposed to me at the core sprints last month by Larry Hastings. I've discussed it with a few people, who seem generally positive about it, and we've tweaked it a little bit. I've spent some time implementing it, and I think it's doable. I thought I'd post it here for any additional feedback. Here?s the idea: for f-strings, we add a !d conversion operator, which is superficially similar to !s, !r, and !a. The meaning of !d is: produce the text of the expression (not its value!), followed by an equal sign, followed by the repr of the value of the expression. So: value = 10 s = 'a string!' print(f'{value!d}') print(f'next: {value+1!d}') print(f'{s!d}') produces: value=10 next: value+1=11 s='a string!' I?m not proposing this for str.format(). It would only really make sense for named arguments, and I don?t think print('{value!d}'.format(value=value) is much of a win. The result is a string, so if you really wanted to, you could use a string formatting spec. So: print(f'*{value!d:^20}*' would produce: * value=10 * Although I don?t think that would be very useful in general. The mnemonic is !d for ?debugging?. I?d wanted to use !=, because there?s an equal sign involved in the result, but = is the one character that can?t be used after ! (it?s ?not equal? in expressions, and f-strings look specifically for that case). I also mentioned !!, but I think I prefer !d as being less confusing. This would be used in debugging print statements, that currently end up looking like: print(f'value={value!r}') and would now be: print(f'{value!d}') There have been discussions about ways to specify str() vs. repr(), using characters other than '=', adding spaces, etc. But they all end up over-complicating what should be a simple tool, not a Swiss Army knife. Thoughts? Eric From dkteresi at gmail.com Tue Oct 2 23:44:11 2018 From: dkteresi at gmail.com (David Teresi) Date: Tue, 2 Oct 2018 23:44:11 -0400 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> Message-ID: print(f'{value!d}') is a lot of symbols and boilerplate to type out just for a debugging statement that will be deleted later. Especially now that breakpoint() exists, I can't really see myself using this. I also don't see the use case of it being within an f-string, because I've never had to interpolate a debug string within some other string or format it in a fancy way. You said it yourself, taking advantage of other f-string features isn't very useful in this case. If other people can find a use for it, I'd suggest making it ita own function -- debug(value) or something similar. David On Tue, Oct 2, 2018, 8:27 PM Eric V. Smith wrote: > This idea was proposed to me at the core sprints last month by Larry > Hastings. I've discussed it with a few people, who seem generally > positive about it, and we've tweaked it a little bit. I've spent some > time implementing it, and I think it's doable. I thought I'd post it > here for any additional feedback. > > Here?s the idea: for f-strings, we add a !d conversion operator, which > is superficially similar to !s, !r, and !a. The meaning of !d is: > produce the text of the expression (not its value!), followed by an > equal sign, followed by the repr of the value of the expression. So: > > value = 10 > s = 'a string!' > print(f'{value!d}') > print(f'next: {value+1!d}') > print(f'{s!d}') > > produces: > > value=10 > next: value+1=11 > s='a string!' > > I?m not proposing this for str.format(). It would only really make sense > for named arguments, and I don?t think > print('{value!d}'.format(value=value) is much of a win. > > The result is a string, so if you really wanted to, you could use a > string formatting spec. So: > > print(f'*{value!d:^20}*' > > would produce: > > * value=10 * > > Although I don?t think that would be very useful in general. > > The mnemonic is !d for ?debugging?. I?d wanted to use !=, because > there?s an equal sign involved in the result, but = is the one character > that can?t be used after ! (it?s ?not equal? in expressions, and > f-strings look specifically for that case). I also mentioned !!, but I > think I prefer !d as being less confusing. > > This would be used in debugging print statements, that currently end up > looking like: > > print(f'value={value!r}') > > and would now be: > > print(f'{value!d}') > > There have been discussions about ways to specify str() vs. repr(), > using characters other than '=', adding spaces, etc. But they all end up > over-complicating what should be a simple tool, not a Swiss Army knife. > > Thoughts? > > Eric > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Oct 2 23:49:37 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Oct 2018 13:49:37 +1000 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> Message-ID: On Wed, Oct 3, 2018 at 1:45 PM David Teresi wrote: > > print(f'{value!d}') is a lot of symbols and boilerplate to type out just for a debugging statement that will be deleted later. Especially now that breakpoint() exists, I can't really see myself using this. > What about when you want to log something without stopping the program? TBH, I almost never use breakpoint. > I also don't see the use case of it being within an f-string, because I've never had to interpolate a debug string within some other string or format it in a fancy way. You said it yourself, taking advantage of other f-string features isn't very useful in this case. > It's an f-string because f-strings are already compiler magic, so the ability to display the expression as well as its result is more logical and plausible. > If other people can find a use for it, I'd suggest making it ita own function -- debug(value) or something similar. > As a normal function, that wouldn't be able to print out the text of the expression, only the resulting value. If debug(value) could print out "value: " and the value, then sure, but otherwise, it's not what this is proposing. ChrisA From boxed at killingar.net Wed Oct 3 00:11:04 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Wed, 3 Oct 2018 06:11:04 +0200 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> Message-ID: > This would be used in debugging print statements, that currently end up looking like: > > print(f'value={value!r}') > > and would now be: > > print(f'{value!d}') It seems to me that a short form for keyword arguments would improve this situation too. So instead of your suggestion one could do: print(dict(=value)) And of course this feature wouldn?t be a minor feature on f-strings but a feature that is generally useful and composable so the above could be improved: def debug(**kwargs): for k, v in kwargs.items(): print(f?{k}={v}?) debug(=value, =another) / Anders From rosuav at gmail.com Wed Oct 3 00:17:44 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Oct 2018 14:17:44 +1000 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> Message-ID: On Wed, Oct 3, 2018 at 2:11 PM Anders Hovm?ller wrote: > > > > This would be used in debugging print statements, that currently end up looking like: > > > > print(f'value={value!r}') > > > > and would now be: > > > > print(f'{value!d}') > > It seems to me that a short form for keyword arguments would improve this situation too. So instead of your suggestion one could do: > > print(dict(=value)) > > And of course this feature wouldn?t be a minor feature on f-strings but a feature that is generally useful and composable so the above could be improved: > > def debug(**kwargs): > for k, v in kwargs.items(): > print(f?{k}={v}?) > > debug(=value, =another) What if it's not a simple name, though? The OP gave this (somewhat simplistic, but indicative) example: print(f'next: {value+1!d}') AIUI, keyword arguments are all supposed to be legal names/atoms, so you aren't supposed to do something like this: debug(**{"value+1":value+1}) even though it does work in current versions of CPython. So even if your "=value" syntax did permit it, I wouldn't want to guarantee that in the language. (Side point: Please watch your mailer. The debug() function above has smart quotes in it, which means it can't be copied and pasted into the interpreter. Not a big problem with trivially-simple functions, but if it's something larger, it's annoying to have to track down "SyntaxError: invalid character in identifier" to figure out why it's not doing what you think it is.) ChrisA From tim.peters at gmail.com Wed Oct 3 00:23:15 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 2 Oct 2018 23:23:15 -0500 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> Message-ID: [Eric V. Smith ] > Here?s the idea: for f-strings, we add a !d conversion operator, which > is superficially similar to !s, !r, and !a. The meaning of !d is: > produce the text of the expression (not its value!), followed by an > equal sign, followed by the repr of the value of the expression. ... > The result is a string, so if you really wanted to, you could use a > string formatting spec. So: > > print(f'*{value!d:^20}*' > > would produce: > > * value=10 * > > Although I don?t think that would be very useful in general. Me neither ;-) But what if {EXPR!d:FMT} acted like the current EXPR={EXPR:FMT} ? I'd find _that_ useful often. For example, when displaying floats, where the repe is almost never what I want to see. >>> f"math.pi={math.pi:.2f}" 'math.pi=3.14' I have plenty of code already embedding stuff of the form EXPR {EXPR:FMT} and don't really care whether there's a space or an "=" between the chunks. "!d" could act like a macro-expansion operator automating a mechanical transformation inside the f-string. Then I could read "!d" as "duplicate" instead of as "debug" ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From boxed at killingar.net Wed Oct 3 00:57:19 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Wed, 3 Oct 2018 06:57:19 +0200 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> Message-ID: >> debug(=value, =another) > > What if it's not a simple name, though? The OP gave this (somewhat > simplistic, but indicative) example: > > print(f'next: {value+1!d}') debug(next=value+1) Still shorter than the proposed syntax and much more readable. If you do this a lot you?d probably call the function just ?d? too. > AIUI, keyword arguments are all supposed to be legal names/atoms, so > you aren't supposed to do something like this: > > debug(**{"value+1":value+1}) Really? That seems pretty weird to me. I?ve used that type of thing in production code from time to time. > even though it does work in current versions of CPython. So even if > your "=value" syntax did permit it, I wouldn't want to guarantee that > in the language. Well I haven?t suggested my syntax would support that. But yea, I realize I was pretty vague in my last email! > (Side point: Please watch your mailer. The debug() function above has > smart quotes in it, which means it can't be copied and pasted into the > interpreter. Not a big problem with trivially-simple functions, but if > it's something larger, it's annoying to have to track down > "SyntaxError: invalid character in identifier" to figure out why it's > not doing what you think it is.) Huh. I?ll look into it. Thanks for the heads up. / Anders From njs at pobox.com Wed Oct 3 01:40:39 2018 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 2 Oct 2018 22:40:39 -0700 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> Message-ID: On Tue, Oct 2, 2018 at 8:44 PM, David Teresi wrote: > print(f'{value!d}') is a lot of symbols and boilerplate to type out just for > a debugging statement that will be deleted later. Especially now that > breakpoint() exists, I can't really see myself using this. > > I also don't see the use case of it being within an f-string, because I've > never had to interpolate a debug string within some other string or format > it in a fancy way. You said it yourself, taking advantage of other f-string > features isn't very useful in this case. > > If other people can find a use for it, I'd suggest making it ita own > function -- debug(value) or something similar. There was some discussion of this back in April: https://mail.python.org/pipermail/python-ideas/2018-April/050113.html I think the way I'd do it would be: Step 1: Take the current "lnotab" that lets us map bytecode offsets -> line numbers, and extend it with more detailed information, so that we can map e.g. a CALL operation to the exact start and end positions of that call expression in the source. This is free at runtime, and would allow more detailed tracebacks (see [1] for an example), and more detailed coverage information. It would definitely take some work to thread the necessary information through the compiler infrastructure, but I think this would be a worthwhile feature even without the debug() use case. Step 2: Add a 'debug' helper function that exploits the detailed information to reconstruct its call, by peeking into the calling frame and finding the source for the call. Of course this would be a strange and ugly thing to do for a regular function, but for a debugging helper it's reasonable. So e.g. if you had the code: total = debug(x) + debug(y / 10) The output might be: debug:myfile.py:10: 'x' is 3 debug:myfile.py:10: 'y / 10' is 7 Or if you have a clever UI, like in an IDE or ipython, maybe it overrides the debug() operator to print something like: total = debug(*x*) + debug(y / 10) ^ *3* total = debug(x) + debug(*y / 10*) ^^^^^^^ *7* (for anyone for whom the rendering is borked: on my screen the "x" on the first line and the "y / 10" on the second line are highlighted in a different font, and the carets draw an underline beneath them.) -n [1] https://mail.python.org/pipermail/python-ideas/2018-April/050137.html -- Nathaniel J. Smith -- https://vorpus.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Oct 3 03:54:10 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 3 Oct 2018 17:54:10 +1000 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> Message-ID: <20181003075409.GF21220@ando.pearwood.info> On Tue, Oct 02, 2018 at 08:27:03PM -0400, Eric V. Smith wrote: > Here?s the idea: for f-strings, we add a !d conversion operator, which > is superficially similar to !s, !r, and !a. The meaning of !d is: > produce the text of the expression (not its value!), I SO WANT THIS AS A GENERAL FEATURE, not just for f-strings, it hurts. Actually what I want is an executable object (a function?) which has the AST and text of the expression attached. If putting this into f-strings is a first step towards getting this thunk-like thing, then I don't need to read any further, I'm +10000 :-) But considering the proposal as you give it: > followed by an > equal sign, followed by the repr of the value of the expression. So: > > value = 10 > s = 'a string!' > print(f'{value!d}') > print(f'next: {value+1!d}') > print(f'{s!d}') > > produces: > > value=10 > next: value+1=11 > s='a string!' I can see lots of arguments about whether the equals sign should have spaces around it. Maybe !d for no spaces and !D for spaces? print(f'next: {value+1!d}') print(f'next: {value+1!D}') would print next: value+1=11 next: value+1 = 11 > I?m not proposing this for str.format(). It would only really make sense > for named arguments, and I don?t think > print('{value!d}'.format(value=value) is much of a win. I'm not so sure that it only makes sense for named arguments. I think it works for arbitrary expressions too: f'{len("NOBODY expects the Spanish Inquisition!")!d}' ought to return 'len("NOBODY expects the Spanish Inquisition!")=39' which I can see being very useful in debugging expressions. This is perhaps the first time I've been excited and enthusiastic about f-strings. A definite +1 on this. -- Steve From boxed at killingar.net Wed Oct 3 04:02:30 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Wed, 3 Oct 2018 10:02:30 +0200 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: <20181003075409.GF21220@ando.pearwood.info> References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <20181003075409.GF21220@ando.pearwood.info> Message-ID: <0AB8D3D0-2213-41D6-9804-02A1FC278D66@killingar.net> >> Here?s the idea: for f-strings, we add a !d conversion operator, which >> is superficially similar to !s, !r, and !a. The meaning of !d is: >> produce the text of the expression (not its value!), > > I SO WANT THIS AS A GENERAL FEATURE, not just for f-strings, it hurts. > > Actually what I want is an executable object (a function?) which has the > AST and text of the expression attached. If putting this into f-strings > is a first step towards getting this thunk-like thing, then I don't > need to read any further, I'm +10000 :-) Well... this is a trivial expansion of the keyword argument short form proposal you were very strongly opposed ^_- So, I suggested `foo(=a)` as short form for `foo(a=a)`, but if we interpret it instead as `foo(=a)` -> `foo(**{'a': a}`, which is really the same thing for most purposes, then we could trivially get `foo(=1+3*bar)` -> 'foo(**{'1+3*bar': 1+3*bar})' I'm fine with this proposal. And thanks Chris for the idea! / Anders From steve at pearwood.info Wed Oct 3 04:05:39 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 3 Oct 2018 18:05:39 +1000 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> Message-ID: <20181003080539.GG21220@ando.pearwood.info> On Wed, Oct 03, 2018 at 06:57:19AM +0200, Anders Hovm?ller wrote: > debug(next=value+1) > > Still shorter than the proposed syntax Are we trying to emulate Perl now? *wink* > and much more readable. So you say. To me that looks like a regular function call, which calls an ordinary function "debug" and takes a simple keyword argument next with value "value+1". Things which contain compiler magic should look special, not like ordinary function calls. > > AIUI, keyword arguments are all supposed to be legal names/atoms, so > > you aren't supposed to do something like this: > > > > debug(**{"value+1":value+1}) > > Really? That seems pretty weird to me. I?ve used that type of thing in > production code from time to time. The fact that this works is, I think, an accident of implementation: py> def spam(**kw): ... print(kw) ... py> spam(**{"value+1": 42}) {'value+1': 42} rather than a guaranteed language feature. I can't find any relevent documentation on it, but I'd be very wary about relying on it. (To be honest, I expected it to fail until I tried it.) You certainly can't do this: py> spam(value+1=42) File "", line 1 SyntaxError: keyword can't be an expression -- Steve From rosuav at gmail.com Wed Oct 3 04:29:32 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Oct 2018 18:29:32 +1000 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: <20181003080539.GG21220@ando.pearwood.info> References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <20181003080539.GG21220@ando.pearwood.info> Message-ID: On Wed, Oct 3, 2018 at 6:06 PM Steven D'Aprano wrote: > > On Wed, Oct 03, 2018 at 06:57:19AM +0200, Anders Hovm?ller wrote: > > > debug(next=value+1) > > > > Still shorter than the proposed syntax > > Are we trying to emulate Perl now? *wink* > > > > and much more readable. > > So you say. > > To me that looks like a regular function call, which calls an ordinary > function "debug" and takes a simple keyword argument next with value > "value+1". > > Things which contain compiler magic should look special, not like > ordinary function calls. > > > > > AIUI, keyword arguments are all supposed to be legal names/atoms, so > > > you aren't supposed to do something like this: > > > > > > debug(**{"value+1":value+1}) > > > > Really? That seems pretty weird to me. I?ve used that type of thing in > > production code from time to time. > > The fact that this works is, I think, an accident of implementation: > > py> def spam(**kw): > ... print(kw) > ... > py> spam(**{"value+1": 42}) > {'value+1': 42} > > rather than a guaranteed language feature. I can't find any relevent > documentation on it, but I'd be very wary about relying on it. > > (To be honest, I expected it to fail until I tried it.) I can't find any documentation either, but ISTR it's been stated as a CPython implementation detail, not a language feature. Other Pythons are entirely free to reject this. ChrisA From rosuav at gmail.com Wed Oct 3 04:34:04 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Oct 2018 18:34:04 +1000 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: <20181003075409.GF21220@ando.pearwood.info> References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <20181003075409.GF21220@ando.pearwood.info> Message-ID: On Wed, Oct 3, 2018 at 5:54 PM Steven D'Aprano wrote: > I'm not so sure that it only makes sense for named arguments. I think it > works for arbitrary expressions too: > > f'{len("NOBODY expects the Spanish Inquisition!")!d}' > > ought to return > > 'len("NOBODY expects the Spanish Inquisition!")=39' > > which I can see being very useful in debugging expressions. Absolutely! I can't imagine doing *hugely* complex expressions, but consider: print(f"{request.args!d}") Technically that's an "arbitrary expression" and is most definitely not a named argument, but it's the sort of thing where you really do want the tag to say "request.args" and not just "args" or anything like that. Ideally, I'd like to be able to write this as: dump(request.args) and have it pick up both the string "request.args" and the value of request.args, but the f-string variant is absolutely okay with me as a partial solution. ChrisA From boxed at killingar.net Wed Oct 3 04:48:31 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Wed, 3 Oct 2018 10:48:31 +0200 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: <20181003080539.GG21220@ando.pearwood.info> References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <20181003080539.GG21220@ando.pearwood.info> Message-ID: >> and much more readable. > > So you say. > > To me that looks like a regular function call, which calls an ordinary > function "debug" and takes a simple keyword argument next with value > "value+1". That was what it was. That was not intended to be about magic, that was the normal case where you didn't want the magic. I think you might be taking a habit of interpreting my mails in the least flattering way due to our previous disagreements. I hope we can put this behind us going forward. > Things which contain compiler magic should look special, not like > ordinary function calls. Good. I agree :P >>> AIUI, keyword arguments are all supposed to be legal names/atoms, so >>> you aren't supposed to do something like this: >>> >>> debug(**{"value+1":value+1}) >> >> Really? That seems pretty weird to me. I?ve used that type of thing in >> production code from time to time. > > The fact that this works is, I think, an accident of implementation: > > py> def spam(**kw): > ... print(kw) > ... > py> spam(**{"value+1": 42}) > {'value+1': 42} > > rather than a guaranteed language feature. I can't find any relevent > documentation on it, but I'd be very wary about relying on it. > > (To be honest, I expected it to fail until I tried it.) I don't really think accidents of implementation are different from hard requirements in Python, as it applies to alternative implementations. In practice if it deviates from CPython then it's broken. There is no language spec, there is only CPython. This has been the experience and approach of PyPy as far as I've understood it after having followed their blog over the years. / Anders From rosuav at gmail.com Wed Oct 3 04:55:27 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Oct 2018 18:55:27 +1000 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <20181003080539.GG21220@ando.pearwood.info> Message-ID: On Wed, Oct 3, 2018 at 6:49 PM Anders Hovm?ller wrote: > > I don't really think accidents of implementation are different from hard requirements in Python, as it applies to alternative implementations. In practice if it deviates from CPython then it's broken. There is no language spec, there is only CPython. This has been the experience and approach of PyPy as far as I've understood it after having followed their blog over the years. > Definitely not true. There have been times when other implementors have come to python-dev and said, hey, is this part of the spec or is it an implementation detail? And the answer determines whether they care about that or not. For just a few examples: 1) Reference counting vs nondeterministic garbage collection 2) O(1) indexing/slicing of Unicode strings 3) "\N{...}" string escapes (okay, that's standardized, but optional) 4) Reuse of id() values 5) The "slots" mechanism for dunder method lookup The language spec determines, in some cases, that a CPython implementation detail has been promoted to standard. More often, it determines that other Pythons are permitted to behave differently. ChrisA From boxed at killingar.net Wed Oct 3 05:09:27 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Wed, 3 Oct 2018 11:09:27 +0200 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <20181003080539.GG21220@ando.pearwood.info> Message-ID: <6A1C9BA8-5773-4F8B-A269-81B6856FFB0E@killingar.net> >> I don't really think accidents of implementation are different from hard requirements in Python, as it applies to alternative implementations. In practice if it deviates from CPython then it's broken. There is no language spec, there is only CPython. This has been the experience and approach of PyPy as far as I've understood it after having followed their blog over the years. >> > > Definitely not true. There have been times when other implementors > have come to python-dev and said, hey, is this part of the spec or is > it an implementation detail? And the answer determines whether they > care about that or not. For just a few examples: > > 1) Reference counting vs nondeterministic garbage collection > 2) O(1) indexing/slicing of Unicode strings > 3) "\N{...}" string escapes (okay, that's standardized, but optional) > 4) Reuse of id() values > 5) The "slots" mechanism for dunder method lookup > > The language spec determines, in some cases, that a CPython > implementation detail has been promoted to standard. More often, it > determines that other Pythons are permitted to behave differently. Sometimes they will come away from this list thinking they don't care but then their users will report bugs over and over again and they'll just have to do it anyway. You probably won't hear about most of those. Trees that fall in the forest when no one is there do in fact make a sound. / Anders From rosuav at gmail.com Wed Oct 3 05:17:08 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Oct 2018 19:17:08 +1000 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: <6A1C9BA8-5773-4F8B-A269-81B6856FFB0E@killingar.net> References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <20181003080539.GG21220@ando.pearwood.info> <6A1C9BA8-5773-4F8B-A269-81B6856FFB0E@killingar.net> Message-ID: On Wed, Oct 3, 2018 at 7:09 PM Anders Hovm?ller wrote: > > > >> I don't really think accidents of implementation are different from hard requirements in Python, as it applies to alternative implementations. In practice if it deviates from CPython then it's broken. There is no language spec, there is only CPython. This has been the experience and approach of PyPy as far as I've understood it after having followed their blog over the years. > >> > > > > Definitely not true. There have been times when other implementors > > have come to python-dev and said, hey, is this part of the spec or is > > it an implementation detail? And the answer determines whether they > > care about that or not. For just a few examples: > > > > 1) Reference counting vs nondeterministic garbage collection > > 2) O(1) indexing/slicing of Unicode strings > > 3) "\N{...}" string escapes (okay, that's standardized, but optional) > > 4) Reuse of id() values > > 5) The "slots" mechanism for dunder method lookup > > > > The language spec determines, in some cases, that a CPython > > implementation detail has been promoted to standard. More often, it > > determines that other Pythons are permitted to behave differently. > > Sometimes they will come away from this list thinking they don't care but then their users will report bugs over and over again and they'll just have to do it anyway. You probably won't hear about most of those. Trees that fall in the forest when no one is there do in fact make a sound. > And sometimes, people all over the world learn to write "with open(...) as f:" instead of just letting f expire. There IS a language spec, and pretending there isn't one doesn't change that. ChrisA From nicolas.rolin at tiime.fr Wed Oct 3 05:20:55 2018 From: nicolas.rolin at tiime.fr (Nicolas Rolin) Date: Wed, 3 Oct 2018 11:20:55 +0200 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <20181003080539.GG21220@ando.pearwood.info> <6A1C9BA8-5773-4F8B-A269-81B6856FFB0E@killingar.net> Message-ID: As a user I would be really pleased with the change proposed, as I use extensively use f-string in my logging (the fact that I have to evaluate the string whatever the logger level is is not a performance hit for my application), and usually the most readable logging format is something akin to f"interesting_variable_name={interesting_variable_name}, big_list[:10]={big_list[:10]}". 2018-10-03 11:17 GMT+02:00 Chris Angelico : > On Wed, Oct 3, 2018 at 7:09 PM Anders Hovm?ller > wrote: > > > > > > >> I don't really think accidents of implementation are different from > hard requirements in Python, as it applies to alternative implementations. > In practice if it deviates from CPython then it's broken. There is no > language spec, there is only CPython. This has been the experience and > approach of PyPy as far as I've understood it after having followed their > blog over the years. > > >> > > > > > > Definitely not true. There have been times when other implementors > > > have come to python-dev and said, hey, is this part of the spec or is > > > it an implementation detail? And the answer determines whether they > > > care about that or not. For just a few examples: > > > > > > 1) Reference counting vs nondeterministic garbage collection > > > 2) O(1) indexing/slicing of Unicode strings > > > 3) "\N{...}" string escapes (okay, that's standardized, but optional) > > > 4) Reuse of id() values > > > 5) The "slots" mechanism for dunder method lookup > > > > > > The language spec determines, in some cases, that a CPython > > > implementation detail has been promoted to standard. More often, it > > > determines that other Pythons are permitted to behave differently. > > > > Sometimes they will come away from this list thinking they don't care > but then their users will report bugs over and over again and they'll just > have to do it anyway. You probably won't hear about most of those. Trees > that fall in the forest when no one is there do in fact make a sound. > > > > And sometimes, people all over the world learn to write "with > open(...) as f:" instead of just letting f expire. There IS a language > spec, and pretending there isn't one doesn't change that. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- -- *Nicolas Rolin* | Data Scientist + 33 631992617 - nicolas.rolin at tiime.fr *15 rue Auber, **75009 Paris* *www.tiime.fr * -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfine2358 at gmail.com Wed Oct 3 05:29:29 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Wed, 3 Oct 2018 10:29:29 +0100 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <20181003080539.GG21220@ando.pearwood.info> <6A1C9BA8-5773-4F8B-A269-81B6856FFB0E@killingar.net> Message-ID: Hi Eric Summary: This email is mainly about process. One discussion thread or several. I think the decision is yours. You wrote suggesting an enhancement for debugging: > Here?s the idea: for f-strings, we add a !d conversion operator, which > is superficially similar to !s, !r, and !a. The meaning of !d is: > produce the text of the expression (not its value!), followed by an > equal sign, followed by the repr of the value of the expression. You gave as one of several examples that given value = 10 the expression f'{value!d}' evaluates to 'value=10' which can then be printed or logged. I thank you and Larry Hastings for coming up with and exploring this idea. Here's my personal opinion, which does not count for much. I like the output and ease of use. I have doubts about the implementation, and how it might work with other things. And also, I'd like the change to go through the PEP process. The present discussion thread is spreading into other areas, such as what constitutes the Python language specification. I'd like to contribute to a broader discussion, of how we can improve the debugging experience. But I feel uncomfortable doing so in a discussion thread that started with your admirably clear focus. Briefly, Eric, would you like the general discussion of improving debugging to take place in this thread, or in another. Finally, if I may I thank you and Larry for your work on this important topic, and for starting this useful discussion. -- Jonathan From boxed at killingar.net Wed Oct 3 05:33:19 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Wed, 3 Oct 2018 11:33:19 +0200 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <20181003080539.GG21220@ando.pearwood.info> <6A1C9BA8-5773-4F8B-A269-81B6856FFB0E@killingar.net> Message-ID: > The present discussion thread is spreading into other areas, such as > what constitutes the Python language specification. I let myself get side tracked. This will be my last mail discussing that off topic thread. Sorry about that. > I'd like to > contribute to a broader discussion, of how we can improve the > debugging experience. But I feel uncomfortable doing so in a > discussion thread that started with your admirably clear focus. Please reconsider! We need more view points and good ideas. / Anders From rosuav at gmail.com Wed Oct 3 05:55:25 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Oct 2018 19:55:25 +1000 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <20181003080539.GG21220@ando.pearwood.info> <6A1C9BA8-5773-4F8B-A269-81B6856FFB0E@killingar.net> Message-ID: On Wed, Oct 3, 2018 at 7:30 PM Jonathan Fine wrote: > Here's my personal opinion, which does not count for much. I like the > output and ease of use. I have doubts about the implementation, and > how it might work with other things. And also, I'd like the change to > go through the PEP process. At this point, I don't think this requires a PEP. If it does in the future, or if competing proposals emerge, then a PEP could be written, but it'd be overkill at the moment. Chris Angelico PEP Editor From steve at pearwood.info Wed Oct 3 05:52:53 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 3 Oct 2018 19:52:53 +1000 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <20181003080539.GG21220@ando.pearwood.info> Message-ID: <20181003095251.GJ21220@ando.pearwood.info> On Wed, Oct 03, 2018 at 10:48:31AM +0200, Anders Hovm?ller wrote: > > >> and much more readable. > > > > So you say. > > > > To me that looks like a regular function call, which calls an ordinary > > function "debug" and takes a simple keyword argument next with value > > "value+1". > > That was what it was. That was not intended to be about magic, that > was the normal case where you didn't want the magic. Oh I'm sorry, I admit I read your earlier post wrongly. I missed that you were focused on the "=args" syntactic sugar and imagined you were proposing a magic debug() function as an alternative to this proposed f-string feature. The perils of skim reading. Mea culpa. -- Steve From eric at trueblade.com Wed Oct 3 06:44:37 2018 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 3 Oct 2018 06:44:37 -0400 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: <20181003075409.GF21220@ando.pearwood.info> References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <20181003075409.GF21220@ando.pearwood.info> Message-ID: <8f57431f-541c-76eb-a6df-2dff8f809663@trueblade.com> On 10/3/2018 3:54 AM, Steven D'Aprano wrote: > On Tue, Oct 02, 2018 at 08:27:03PM -0400, Eric V. Smith wrote: > >> Here?s the idea: for f-strings, we add a !d conversion operator, which >> is superficially similar to !s, !r, and !a. The meaning of !d is: >> produce the text of the expression (not its value!), > > I SO WANT THIS AS A GENERAL FEATURE, not just for f-strings, it hurts. > > Actually what I want is an executable object (a function?) which has the > AST and text of the expression attached. If putting this into f-strings > is a first step towards getting this thunk-like thing, then I don't > need to read any further, I'm +10000 :-) I feel your pain, but we're a long way from that. >> produces: >> >> value=10 >> next: value+1=11 >> s='a string!' > > I can see lots of arguments about whether the equals sign should have > spaces around it. > > Maybe !d for no spaces and !D for spaces? > > print(f'next: {value+1!d}') > print(f'next: {value+1!D}') > > would print > > next: value+1=11 > next: value+1 = 11 I'm not sure it's worth the hassle. I considered a whole format spec language for this, but it sort of got out of hand. >> I?m not proposing this for str.format(). It would only really make sense >> for named arguments, and I don?t think >> print('{value!d}'.format(value=value) is much of a win. > > I'm not so sure that it only makes sense for named arguments. I think it > works for arbitrary expressions too: > > f'{len("NOBODY expects the Spanish Inquisition!")!d}' > > ought to return > > 'len("NOBODY expects the Spanish Inquisition!")=39' > > which I can see being very useful in debugging expressions. Yes, that's part of the proposal, but I wasn't very clear. That's what the "value+1" example was supposed to convey. My comment about named parameters applied only to str.format(), and why it won't be supported there. Eric From eric at trueblade.com Wed Oct 3 06:49:37 2018 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 3 Oct 2018 06:49:37 -0400 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <20181003080539.GG21220@ando.pearwood.info> <6A1C9BA8-5773-4F8B-A269-81B6856FFB0E@killingar.net> Message-ID: On 10/3/2018 5:29 AM, Jonathan Fine wrote: > Hi Eric > > Summary: This email is mainly about process. One discussion thread or > several. I think the decision is yours. > > You wrote suggesting an enhancement for debugging: > >> Here?s the idea: for f-strings, we add a !d conversion operator, which >> is superficially similar to !s, !r, and !a. The meaning of !d is: >> produce the text of the expression (not its value!), followed by an >> equal sign, followed by the repr of the value of the expression. > > You gave as one of several examples that given > value = 10 > the expression > f'{value!d}' > evaluates to > 'value=10' > which can then be printed or logged. > > I thank you and Larry Hastings for coming up with and exploring this idea. > > Here's my personal opinion, which does not count for much. I like the > output and ease of use. I have doubts about the implementation, and > how it might work with other things. And also, I'd like the change to > go through the PEP process. I don't think a simple f-string feature needs a PEP. The implementation is pretty simple, since the f-string parser already has access to the string. > The present discussion thread is spreading into other areas, such as > what constitutes the Python language specification. I'd like to > contribute to a broader discussion, of how we can improve the > debugging experience. But I feel uncomfortable doing so in a > discussion thread that started with your admirably clear focus. > > Briefly, Eric, would you like the general discussion of improving > debugging to take place in this thread, or in another. I think it would be more productive as a separate thread. > Finally, if I may I thank you and Larry for your work on this > important topic, and for starting this useful discussion. Thanks. It's what we do! Eric From eric at trueblade.com Wed Oct 3 06:55:44 2018 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 3 Oct 2018 06:55:44 -0400 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> Message-ID: On 10/3/2018 1:40 AM, Nathaniel Smith wrote: > On Tue, Oct 2, 2018 at 8:44 PM, David Teresi > wrote: > > print(f'{value!d}') is a lot of symbols and boilerplate to type out > just for > > a debugging statement that will be deleted later. Especially now that > > breakpoint() exists, I can't really see myself using this. > > > > I also don't see the use case of it being within an f-string, because > I've > > never had to interpolate a debug string within some other string or > format > > it in a fancy way. You said it yourself, taking advantage of other > f-string > > features isn't very useful in this case. > > > > If other people can find a use for it, I'd suggest making it ita own > > function -- debug(value) or something similar. > > There was some discussion of this back in April: > > https://mail.python.org/pipermail/python-ideas/2018-April/050113.html Worth pointing out from that discussion is the "q" library: https://pypi.org/project/q/. I got excited about it back then, but never installed it. > I think the way I'd do it would be: > > Step 1: Take the current "lnotab" that lets us map bytecode offsets -> > line numbers, and extend it with more detailed information, so that we > can map e.g. a CALL operation to the exact start and end positions of > that call expression in the source. This is free at runtime, and would > allow more detailed tracebacks (see [1] for an example), and more > detailed coverage information. It would definitely take some work to > thread the necessary information through the compiler infrastructure, > but I think this would be a worthwhile feature even without the debug() > use case. > > Step 2: Add a 'debug' helper function that exploits the detailed > information to reconstruct its call, by peeking into the calling frame > and finding the source for the call. Of course this would be a strange > and ugly thing to do for a regular function, but for a debugging helper > it's reasonable. So e.g. if you had the code: > > ? total = debug(x) + debug(y / 10) > > The output might be: > > ? debug:myfile.py:10: 'x' is 3 > ? debug:myfile.py:10: 'y / 10' is 7 I'm not positive, but isn't this what q does? Eric From eric at trueblade.com Wed Oct 3 07:07:44 2018 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 3 Oct 2018 07:07:44 -0400 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> Message-ID: <8b914470-a7c3-764a-7c30-a500ef1acf8a@trueblade.com> On 10/3/2018 12:23 AM, Tim Peters wrote: > > [Eric V. Smith >] > > Here?s the idea: for f-strings, we add a !d conversion operator, which > is superficially similar to !s, !r, and !a. The meaning of !d is: > produce the text of the expression (not its value!), followed by an > equal sign, followed by the repr of the value of the expression. > > ... > > The result is a string, so if you really wanted to, you could use a > string formatting spec. So: > > print(f'*{value!d:^20}*' > > would produce: > > *? ? ? value=10? ? ? * > > Although I don?t think that would be very useful in general. > > > Me neither ;-)? But what if > > ? ? {EXPR!d:FMT} > > acted like the current > > ? ? EXPR={EXPR:FMT} > > ?? I'd find _that_ useful often.? For example, when displaying floats, > where the repe is almost never what I want to see. > > >>> f"math.pi={math.pi:.2f}" > 'math.pi=3.14' > > I have plenty of code already embedding stuff of the form > > ? ? EXPR {EXPR:FMT} > > and don't really care whether there's a space or an "=" between the > chunks.? "!d" could act like a macro-expansion operator automating a > mechanical transformation inside the f-string.? Then I could read "!d" > as "duplicate" instead of as "debug" ;-) That's a very interesting idea. It breaks the expectation (which might be mine alone!) that the format spec is used on the result of the conversion specifier (!s, !r). But maybe that's okay. It does seem generally more useful than the format spec being used on the repr of the evaluated expression. Eric From levkivskyi at gmail.com Wed Oct 3 07:10:11 2018 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Wed, 3 Oct 2018 12:10:11 +0100 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: <8f57431f-541c-76eb-a6df-2dff8f809663@trueblade.com> References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <20181003075409.GF21220@ando.pearwood.info> <8f57431f-541c-76eb-a6df-2dff8f809663@trueblade.com> Message-ID: On Wed, 3 Oct 2018 at 11:45, Eric V. Smith wrote: > On 10/3/2018 3:54 AM, Steven D'Aprano wrote: > > On Tue, Oct 02, 2018 at 08:27:03PM -0400, Eric V. Smith wrote: > > > >> Here?s the idea: for f-strings, we add a !d conversion operator, which > >> is superficially similar to !s, !r, and !a. The meaning of !d is: > >> produce the text of the expression (not its value!), > > > > I SO WANT THIS AS A GENERAL FEATURE, not just for f-strings, it hurts. > > > > Actually what I want is an executable object (a function?) which has the > > AST and text of the expression attached. If putting this into f-strings > > is a first step towards getting this thunk-like thing, then I don't > > need to read any further, I'm +10000 :-) > > I feel your pain, but we're a long way from that. > Maybe we are actually not so far? PEP 563 added functionality for transforming expressions to strings, maybe it can be reused for this purpose? I would love to have this since in my experience I mostly print some (relatively complex) expressions. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfine2358 at gmail.com Wed Oct 3 10:51:43 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Wed, 3 Oct 2018 15:51:43 +0100 Subject: [Python-ideas] Debugging: some problems and possible solutions Message-ID: This thread is about debugging. I suggest we start by collecting problems and possible solutions. And that after about a week of that, we start discussing what we've gathered. We already have a problem and possible solution, provided by Eric Smith and Larry Hastings. TITLE: f-string "debug" conversion URL: https://mail.python.org/pipermail/python-ideas/2018-October/053956.html PROBLEM Writing print('value = ', value) is tedious, particularly for more complicated expressions, such as print('big_array[5:20] =', big_array[5:20]) POSSIBLE SOLUTION For f-strings, we add a !d conversion operator, which produces the text of an expression followed by its value. Thus, the two previous examples can be written more concisely as print(f'{value !d}') print(f'{big_array[5:20] !d}') I suggest for the moment that we just gather problem-solution pairs, much as above. I think they'll be between 5 and 15 such pairs. I'll post one to the current discussion thread in an hour or so. And that after about a week, we move to discussing what we have. Finally, many thanks to Eric and Larry for their useful contribution to the important problem of debugging. -- Jonathan From jfine2358 at gmail.com Wed Oct 3 11:15:22 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Wed, 3 Oct 2018 16:15:22 +0100 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: Message-ID: TITLE: Saving and sharing debug code PROBLEM: Sometimes debug-only code should be saved and shared. For example, one person's code written to solve a bug might be needed a second time, by someone else. Or perhaps the same person, again. This is particularly likely when the bug is difficult or stubborn. POSSIBLE SOLUTION At present, __debug__ is a keyword identifier, which can take only the values True and False. It gets its value on Python startup, and then its value can't be changed. The possible solution is to allow Python startup to give __debug__ an additional value, namely 2. Once this is done, code blocks such as if __debug__ >= 2: # my debug-only code goes here will effectively be ignored, except when requested. Python already does this for blocks such as if __debug__: # ignored if optimised code is being generated See also: https://docs.python.org/3/library/constants.html#__debug__ __debug__ This constant is true if Python was not started with an -O option. See also the assert statement. From rosuav at gmail.com Wed Oct 3 11:23:26 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Oct 2018 01:23:26 +1000 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: Message-ID: On Thu, Oct 4, 2018 at 12:52 AM Jonathan Fine wrote: > > This thread is about debugging. I suggest we start by collecting > problems and possible solutions. And that after about a week of that, > we start discussing what we've gathered. > Why not just discuss each proposal on its own merits, independently of any other proposals? Do they interact with each other? ChrisA From wes.turner at gmail.com Wed Oct 3 11:25:24 2018 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 3 Oct 2018 11:25:24 -0400 Subject: [Python-ideas] PEPs: Theory of operation [was: Moving to another forum system ...] In-Reply-To: References: <859A4CFF-23ED-4329-9FAA-DA59AA65FC80@gmail.com> <23459.25799.174825.591496@turnbull.sk.tsukuba.ac.jp> <93AF02F1-CE05-4D91-B25C-C0D6DC1FC73D@killingar.net> <23461.63786.966966.519896@turnbull.sk.tsukuba.ac.jp> Message-ID: On Saturday, September 22, 2018, Wes Turner wrote: > Here are links to the Apache governance docs: > > https://www.apache.org/foundation/governance/#technical > > https://www.apache.org/foundation/governance/pmcs.html > > Which are the PSF docs for these exact same processes for open source > governance? (In re: to transitioning from BDFL is not dead, but) > > https://devguide.python.org/#contributing > > https://devguide.python.org/experts/ > - is there a different BDFL-delegate org chart, or would this be the page > to add to and refer to? > re: Documenting the BDFL-Delegate process (in PEP 1?) I'm just not good with a MUA; Markdown in a linear GH issue is far easier to unsubscribe from; so I just added triple quotes: >From "[Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580": ""' Jeroen Demeyer Hello, I would like to propose Petr Viktorin as BDFL-Delegate for PEP 580, titled "The C call protocol". He has co-authored several PEPs [...] """ """ INADA Naoki 2018?10?3?(?) 21:24 Jeroen Demeyer : > > Hello, Really? I don't know process to assign BDFL-delegate without BDFL. This PEP is ... ?ukasz Langa > My understand is that accepting *any* PEP by anyone is out of the question until the governance situation gets resolved. That's the only reason why... """ """ On Wednesday, October 3, 2018, INADA Naoki wrote: 2018?10?3?(?) 21:24 Jeroen Demeyer : Hello, >> I am well aware of the current governance issues, but several people have mentioned that the BDFL-Delegate process can still continue for now. > Really? > I don't know process to assign BDFL-delegate without BDFL. """ """ AFAIU, there is not yet a documented process for BDFL-delegate assignment. There's this in the devguide; which links to PEP1: "20.2. PEP Process?" https://devguide.python.org/langchanges/#pep-process https://github.com/python/devguide/blob/master/langchanges.rst And PEP 1: "PEP 1 -- PEP Purpose and Guidelines" "PEP Workflow" https://www.python.org/dev/peps/pep-0001/#pep-workflow "PEP Editors" https://www.python.org/dev/peps/pep-0001/#pep-editors "PEP Editor Responsibilities & Workflow" https://www.python.org/dev/peps/pep-0001/#pep-editor-responsibilities-workflow https://github.com/python/peps/blob/master/pep-0001.txt And the devguide has a list of experts: https://devguide.python.org/experts/ Maybe PEP1 is the place to list current BDFL-Delegates (in addition to in the PEP metadata as in the OT PR: python/peps#797 "PEP 580: Petr Viktorin as BDFL-Delegate" )? Not to bikeshed, but is BDFL-Delegate still the current term because that's what's in all the other PEPs' metadata? """ Maybe a "delegation" GH Issue label or similar (and a commit prefix) on the python/peps repo would be helpful? > > On Saturday, September 22, 2018, Wes Turner wrote: > >> >> >> On Saturday, September 22, 2018, Wes Turner wrote: >> >>> >>> It seems like everything's fine, but I would have no idea, BTW >>> >> >> Would project boards be helpful for coordinating proposal status >> information, or extra process for something that's already working just >> fine? >> >> https://github.com/python/peps/projects >> >> https://github.com/pypa/interoperability-peps/projects >> >> TBH, I like Waffle.io boards, but core team may be more comfortable with >> GH projects with swimlanes? >> >> >>> [] https://en.wikipedia.org/wiki/Quorum_call >>> >>> On Saturday, September 22, 2018, Stephen J. Turnbull < >>> turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: >>> >>>> Executive summary: Writing a PEP is an inherently uncertain process. >>>> Achieving "community consensus" is the goal of the process, not a >>>> precondition. >>>> >>>> Anders Hovm?ller writes: >>>> >>>> > In general pep1 is frustratingly vague. Terms like ?community >>>> > consensus? without defining community or what numbers would >>>> > constitute a consensus are not fun to read as someone who doesn?t >>>> > personally know anyone of the core devs. Further references to >>>> > Guido are even more frustrating now that he?s bowed out. >>>> >>>> These terms have little to do with what a new PEP's proponent needs to >>>> think about, though. A PEP-able proposal by definition involves >>>> uncertainty. Nobody, not even Guido, can tell you in advance whether >>>> a PEP will be accepted (for implementation). The PEP process is >>>> rigorous enough that by the time you get close to needing consensus to >>>> proceed, you'll know what it means. >>>> >>>> "Community consensus" is not a condition for *anything* in the PEP >>>> process, except final acceptance. It is the *goal* of the process. >>>> PEPs are approved (for publication) by default; the only requirement >>>> is editorial completeness. PEPs are needed for two reasons: (1) to >>>> get the input of the community, both highly competent engineers for >>>> implementation and a variety of users for requirements, to refine a >>>> complex proposal or one with far-reaching implications for the >>>> language, and/or (2) to build a consensus for implementation. Either >>>> way, by definition the outcome is unclear at the beginning. >>>> >>>> If your concern about "consensus" is that you want to know whether >>>> you're likely to get to consensus, and an accepted PEP, ask somebody >>>> who seems sympathetic and experienced enough to know about what it >>>> looks like on the list when a PEP is going to succeed. Anything >>>> PEP-able is sufficiently unclear that rules can't be given in PEP 1. >>>> It is possible only to say that Python is now very mature, and there's >>>> a strong conservative bias against change. That doesn't mean there >>>> aren't changes: Python attracts a lot of feature proposals, so the >>>> rate of change isn't slowing although the acceptance rate is declining >>>> gradually. >>>> >>>> "Consensus" is never defined by numbers in the English language, and >>>> it does not mean "unanimity". In PEP 1, it means that some people >>>> agree, most people don't disagree, and even if a senior person >>>> disagrees, they're willing to go along with the "sense of the >>>> community". As that adjective "senior" implies, some people count >>>> more to the consensus than others. Usually when I write "senior" I'm >>>> referring to core developers (committers), but here there >>>> people who are "senior" enough despite not having commit bits.[1] >>>> >>>> "The community" is not well defined, and it can't be, short of a >>>> doctoral dissertation in anthropology. The relevant channels are >>>> open-participation, some people speak for themselves, some people are >>>> "official" representatives of important constituencies such as the >>>> leaders of large independent projects or alternative implementations, >>>> and some people have acquired sufficient reputation to be considered >>>> representative of a group of people (especially when other members of >>>> the group rarely participate in the dev lists but for some reason are >>>> considered important to the community -- I'm thinking in particular of >>>> sysadmins and devops, and the problems we can cause them by messing >>>> with packaging and installation). >>>> >>>> References to the BDFL are, of course, in limbo. AFAIK we don't have >>>> one at the moment. Until we do, any PEPs will presumably be accepted >>>> either by a self-nominated BDFL-Delegate acceptable to the core devs, >>>> or by an ad hoc committee of interested core devs, and that part of >>>> PEP 1 can't be usefully updated yet. This is not a weakness of the >>>> Python project, IMO. Rather, the fact that, despite a sort of >>>> constitutional crisis, the whole process is continuing pretty much as >>>> usual shows its strength. >>>> >>>> This is possible because the BDFL is not, and has not been for many >>>> years, a "hands-on" manager. It's true that where a proposal affects >>>> his own "development *in* Python", he's likely to work closely with a >>>> proponent, off- and on-list, or even *be* the proponent. Of course >>>> such proposals are more likely to be approved, and a few community >>>> members have pushed back on that because it appears undemocratic. But >>>> the general reaction is "maybe 'Although that way may not be obvious >>>> at first unless you're Dutch' applies to me in such cases!" For most >>>> proposals, he's "just" a very senior developer whose comments are >>>> important because he's a great developer, but he is easily swayed by >>>> the sense of the community. Bottom line: except in the rare case >>>> where your proposal directly affects the BDFL's own coding, the BDFL's >>>> now-traditional role is to declare that consensus has been achieved, >>>> postpone the PEP because it's clear that consensus is not forming, or >>>> in rare cases, make a choice despite the lack of consensus. >>>> >>>> But none of this is really of importance to a PEP proponent >>>> ("champion" in the terminology of PEP 1). PEP 1 is quite specific >>>> about the required components of the document, and many points of >>>> formatting and style. Accept the uncertainty, and do what you need to >>>> do to meet those requirements, that's all there is to it. If the >>>> community wants more, or wants changes, it will tell you, either as a >>>> demand about style or missing content from an editor or as a technical >>>> comment on the list. Whether you accept those technical comments is >>>> up to you, but your star will rise far more rapidly if you are very >>>> sensitive to claims that "this change to the PEP will a big >>>> improvement for some significant consituency in the community". If >>>> you want advice on whether the chance of acceptance is high enough to >>>> be worth putting in more work, ask the BDFL-Delegate (or the BDFL if >>>> she/he has "claimed" the PEP) where the proposal has an official >>>> adjudicator, and if not, a senior core developer. >>>> >>>> If one doesn't know who the senior developers are yet, she should think >>>> twice about whether she's ready to PEP anything. That's not a litmus >>>> test; some PEPs have eventually succeeded though the proponent was new >>>> to the project development process.[2] But it's a lot less painful if >>>> you can tell who's likely to be able to sway the whole project one way >>>> or the other. And as a matter of improving your proposal, who surely >>>> does know more about what your proposal implies for the implementation >>>> than you do, so you should strongly consider whether *you* are the one >>>> who's missing something when you disagree with them. >>>> >>>> >>>> Footnotes: >>>> [1] They are familiar to some of the core developers as drivers of >>>> important projects developing *in* Python. >>>> >>>> [2] The ones I can think of involve the same kind of person as >>>> footnote 1, and a co-proponent who was a core developer. >>>> >>>> >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfine2358 at gmail.com Wed Oct 3 11:30:10 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Wed, 3 Oct 2018 16:30:10 +0100 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: Message-ID: Chris Angelico wrote: > Why not just discuss each proposal on its own merits, independently of > any other proposals? Do they interact with each other? Good question. I think they will interact. Not in terms of implementation, but in terms of benefit. To quote the Zen of Python: > There should be one-- and preferably only one --obvious way to do it. > Although that way may not be obvious at first unless you're Dutch. However, we won't know for sure until the proposals are in. -- Jonathan From rosuav at gmail.com Wed Oct 3 11:33:28 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Oct 2018 01:33:28 +1000 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: Message-ID: On Thu, Oct 4, 2018 at 1:30 AM Jonathan Fine wrote: > > Chris Angelico wrote: > > > Why not just discuss each proposal on its own merits, independently of > > any other proposals? Do they interact with each other? > > Good question. I think they will interact. Not in terms of > implementation, but in terms of benefit. To quote the Zen of Python: > > > There should be one-- and preferably only one --obvious way to do it. > > Although that way may not be obvious at first unless you're Dutch. > > However, we won't know for sure until the proposals are in. Debugging is a massively broad area, and we commonly have and use multiple completely different tools. Let's not start assuming that proposals will interact or conflict until we actually have some that do. Your proposal for __debug__ to be able to have integer values seems like something that should be discussed in more detail, independently of exactly what the debugging code governed by it ends up doing. If you do "if __debug__>2: print(f'{blah!d}')", it'd use two different proposals, but they're completely orthogonal. ChrisA From boxed at killingar.net Wed Oct 3 11:42:30 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Wed, 3 Oct 2018 17:42:30 +0200 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: Message-ID: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> TITLE: output debug information easily and quickly POSSIBLE SOLUTION Short form of keyword arguments where foo(=a, =1+bar) Is expanded at compile time to foo(**{'a': a, '1+bar': 1+bar}) Then we can just create a normal debug function: def debug_print(**vars): for k, v in vars.items(): print(f'{k}={v}') this is of course useful for many other things as Steven has pointed out. / Anders From wes.turner at gmail.com Wed Oct 3 11:56:36 2018 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 3 Oct 2018 11:56:36 -0400 Subject: [Python-ideas] PEPs: Theory of operation [was: Moving to another forum system ...] In-Reply-To: References: <859A4CFF-23ED-4329-9FAA-DA59AA65FC80@gmail.com> <23459.25799.174825.591496@turnbull.sk.tsukuba.ac.jp> <93AF02F1-CE05-4D91-B25C-C0D6DC1FC73D@killingar.net> <23461.63786.966966.519896@turnbull.sk.tsukuba.ac.jp> Message-ID: re: Updating the BDFL-Delegate policy (in PEP 1) On Wednesday, October 3, 2018, Wes Turner wrote: > > > On Saturday, September 22, 2018, Wes Turner wrote: > >> [...] >> >> https://devguide.python.org/#contributing >> >> https://devguide.python.org/experts/ >> - is there a different BDFL-delegate org chart, or would this be the page >> to add to and refer to? >> > > re: Documenting the BDFL-Delegate process (in PEP 1?) > > I'm just not good with a MUA; Markdown in a linear GH issue is far easier > to unsubscribe from; so I just added triple quotes: > > From "[Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580": > > ""' > Jeroen Demeyer > Hello, I would like to propose Petr Viktorin as BDFL-Delegate for PEP 580, > titled "The C call protocol". He has co-authored several PEPs [...] > """ > > """ > INADA Naoki > 2018?10?3?(?) 21:24 Jeroen Demeyer : > > > Hello, Really? I don't know process to assign BDFL-delegate without > BDFL. This PEP is ... > > ?ukasz Langa > > My understand is that accepting *any* PEP by anyone is out of the > question until the governance situation gets resolved. That's the only > reason why... > """ > > """ > On Wednesday, October 3, 2018, INADA Naoki wrote: > > 2018?10?3?(?) 21:24 Jeroen Demeyer : > Hello, > > >> I am well aware of the current governance issues, but several people > have mentioned that the BDFL-Delegate process can still continue for > now. > > > Really? > > I don't know process to assign BDFL-delegate without BDFL. > """ > > """ > AFAIU, there is not yet a documented process for BDFL-delegate assignment. > > There's this in the devguide; which links to PEP1: > > "20.2. PEP Process?" > https://devguide.python.org/langchanges/#pep-process > https://github.com/python/devguide/blob/master/langchanges.rst > > > And PEP 1: > > "PEP 1 -- PEP Purpose and Guidelines" > "PEP Workflow" > https://www.python.org/dev/peps/pep-0001/#pep-workflow > "PEP Editors" > https://www.python.org/dev/peps/pep-0001/#pep-editors > "PEP Editor Responsibilities & Workflow" > https://www.python.org/dev/peps/pep-0001/#pep-editor- > responsibilities-workflow > > https://github.com/python/peps/blob/master/pep-0001.txt > > And the devguide has a list of experts: > https://devguide.python.org/experts/ > > > Maybe PEP1 is the place to list current BDFL-Delegates > (in addition to in the PEP metadata as in the OT PR: > python/peps#797 > "PEP 580: Petr Viktorin as BDFL-Delegate" > )? > > > Not to bikeshed, but is BDFL-Delegate still the current term because > that's what's in all the other PEPs' metadata? > """ > > Maybe a "delegation" GH Issue label or similar (and a commit prefix) on > the python/peps repo would be helpful? > ''' On 2018-10-03 17:12, Wes Turner wrote: > AFAIU, there is not yet a documented process for BDFL-delegate assignment. PEP 1 says: """ However, whenever a new PEP is put forward, any core developer that believes they are suitably experienced to make the final decision on that PEP may offer to serve as the BDFL's delegate (or "PEP czar") for that PEP. If their self-nomination is accepted by the other core developers and the BDFL, then they will have the authority to approve (or reject) that PEP. """ I know that it says "core developers and the BDFL". However, if the core developers agree that Petr can become BDFL-Delegate, I don't see why that wouldn't be possible. ''' The phrase "core developers and the BDFL" is where some sort of alternate quorum/majority policy would need to be specified if this is a contentious issue in practice? (TBC, I'm +1 on the particular delegation that's not the question here) > > >> >> On Saturday, September 22, 2018, Wes Turner wrote: >> >>> >>> >>> On Saturday, September 22, 2018, Wes Turner >>> wrote: >>> >>>> >>>> It seems like everything's fine, but I would have no idea, BTW >>>> >>> >>> Would project boards be helpful for coordinating proposal status >>> information, or extra process for something that's already working just >>> fine? >>> >>> https://github.com/python/peps/projects >>> >>> https://github.com/pypa/interoperability-peps/projects >>> >>> TBH, I like Waffle.io boards, but core team may be more comfortable with >>> GH projects with swimlanes? >>> >>> >>>> [] https://en.wikipedia.org/wiki/Quorum_call >>>> >>>> On Saturday, September 22, 2018, Stephen J. Turnbull < >>>> turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: >>>> >>>>> Executive summary: Writing a PEP is an inherently uncertain process. >>>>> Achieving "community consensus" is the goal of the process, not a >>>>> precondition. >>>>> >>>>> Anders Hovm?ller writes: >>>>> >>>>> > In general pep1 is frustratingly vague. Terms like ?community >>>>> > consensus? without defining community or what numbers would >>>>> > constitute a consensus are not fun to read as someone who doesn?t >>>>> > personally know anyone of the core devs. Further references to >>>>> > Guido are even more frustrating now that he?s bowed out. >>>>> >>>>> These terms have little to do with what a new PEP's proponent needs to >>>>> think about, though. A PEP-able proposal by definition involves >>>>> uncertainty. Nobody, not even Guido, can tell you in advance whether >>>>> a PEP will be accepted (for implementation). The PEP process is >>>>> rigorous enough that by the time you get close to needing consensus to >>>>> proceed, you'll know what it means. >>>>> >>>>> "Community consensus" is not a condition for *anything* in the PEP >>>>> process, except final acceptance. It is the *goal* of the process. >>>>> PEPs are approved (for publication) by default; the only requirement >>>>> is editorial completeness. PEPs are needed for two reasons: (1) to >>>>> get the input of the community, both highly competent engineers for >>>>> implementation and a variety of users for requirements, to refine a >>>>> complex proposal or one with far-reaching implications for the >>>>> language, and/or (2) to build a consensus for implementation. Either >>>>> way, by definition the outcome is unclear at the beginning. >>>>> >>>>> If your concern about "consensus" is that you want to know whether >>>>> you're likely to get to consensus, and an accepted PEP, ask somebody >>>>> who seems sympathetic and experienced enough to know about what it >>>>> looks like on the list when a PEP is going to succeed. Anything >>>>> PEP-able is sufficiently unclear that rules can't be given in PEP 1. >>>>> It is possible only to say that Python is now very mature, and there's >>>>> a strong conservative bias against change. That doesn't mean there >>>>> aren't changes: Python attracts a lot of feature proposals, so the >>>>> rate of change isn't slowing although the acceptance rate is declining >>>>> gradually. >>>>> >>>>> "Consensus" is never defined by numbers in the English language, and >>>>> it does not mean "unanimity". In PEP 1, it means that some people >>>>> agree, most people don't disagree, and even if a senior person >>>>> disagrees, they're willing to go along with the "sense of the >>>>> community". As that adjective "senior" implies, some people count >>>>> more to the consensus than others. Usually when I write "senior" I'm >>>>> referring to core developers (committers), but here there >>>>> people who are "senior" enough despite not having commit bits.[1] >>>>> >>>>> "The community" is not well defined, and it can't be, short of a >>>>> doctoral dissertation in anthropology. The relevant channels are >>>>> open-participation, some people speak for themselves, some people are >>>>> "official" representatives of important constituencies such as the >>>>> leaders of large independent projects or alternative implementations, >>>>> and some people have acquired sufficient reputation to be considered >>>>> representative of a group of people (especially when other members of >>>>> the group rarely participate in the dev lists but for some reason are >>>>> considered important to the community -- I'm thinking in particular of >>>>> sysadmins and devops, and the problems we can cause them by messing >>>>> with packaging and installation). >>>>> >>>>> References to the BDFL are, of course, in limbo. AFAIK we don't have >>>>> one at the moment. Until we do, any PEPs will presumably be accepted >>>>> either by a self-nominated BDFL-Delegate acceptable to the core devs, >>>>> or by an ad hoc committee of interested core devs, and that part of >>>>> PEP 1 can't be usefully updated yet. This is not a weakness of the >>>>> Python project, IMO. Rather, the fact that, despite a sort of >>>>> constitutional crisis, the whole process is continuing pretty much as >>>>> usual shows its strength. >>>>> >>>>> This is possible because the BDFL is not, and has not been for many >>>>> years, a "hands-on" manager. It's true that where a proposal affects >>>>> his own "development *in* Python", he's likely to work closely with a >>>>> proponent, off- and on-list, or even *be* the proponent. Of course >>>>> such proposals are more likely to be approved, and a few community >>>>> members have pushed back on that because it appears undemocratic. But >>>>> the general reaction is "maybe 'Although that way may not be obvious >>>>> at first unless you're Dutch' applies to me in such cases!" For most >>>>> proposals, he's "just" a very senior developer whose comments are >>>>> important because he's a great developer, but he is easily swayed by >>>>> the sense of the community. Bottom line: except in the rare case >>>>> where your proposal directly affects the BDFL's own coding, the BDFL's >>>>> now-traditional role is to declare that consensus has been achieved, >>>>> postpone the PEP because it's clear that consensus is not forming, or >>>>> in rare cases, make a choice despite the lack of consensus. >>>>> >>>>> But none of this is really of importance to a PEP proponent >>>>> ("champion" in the terminology of PEP 1). PEP 1 is quite specific >>>>> about the required components of the document, and many points of >>>>> formatting and style. Accept the uncertainty, and do what you need to >>>>> do to meet those requirements, that's all there is to it. If the >>>>> community wants more, or wants changes, it will tell you, either as a >>>>> demand about style or missing content from an editor or as a technical >>>>> comment on the list. Whether you accept those technical comments is >>>>> up to you, but your star will rise far more rapidly if you are very >>>>> sensitive to claims that "this change to the PEP will a big >>>>> improvement for some significant consituency in the community". If >>>>> you want advice on whether the chance of acceptance is high enough to >>>>> be worth putting in more work, ask the BDFL-Delegate (or the BDFL if >>>>> she/he has "claimed" the PEP) where the proposal has an official >>>>> adjudicator, and if not, a senior core developer. >>>>> >>>>> If one doesn't know who the senior developers are yet, she should think >>>>> twice about whether she's ready to PEP anything. That's not a litmus >>>>> test; some PEPs have eventually succeeded though the proponent was new >>>>> to the project development process.[2] But it's a lot less painful if >>>>> you can tell who's likely to be able to sway the whole project one way >>>>> or the other. And as a matter of improving your proposal, who surely >>>>> does know more about what your proposal implies for the implementation >>>>> than you do, so you should strongly consider whether *you* are the one >>>>> who's missing something when you disagree with them. >>>>> >>>>> >>>>> Footnotes: >>>>> [1] They are familiar to some of the core developers as drivers of >>>>> important projects developing *in* Python. >>>>> >>>>> [2] The ones I can think of involve the same kind of person as >>>>> footnote 1, and a co-proponent who was a core developer. >>>>> >>>>> >>>>> _______________________________________________ >>>>> Python-ideas mailing list >>>>> Python-ideas at python.org >>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>>> >>>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Wed Oct 3 12:02:14 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Wed, 3 Oct 2018 18:02:14 +0200 Subject: [Python-ideas] PEPs: Theory of operation [was: Moving to another forum system ...] In-Reply-To: <4b8e985c07a4464c8a39055b95b38d0e@xmail103.UGent.be> References: <859A4CFF-23ED-4329-9FAA-DA59AA65FC80@gmail.com> <23459.25799.174825.591496@turnbull.sk.tsukuba.ac.jp> <93AF02F1-CE05-4D91-B25C-C0D6DC1FC73D@killingar.net> <23461.63786.966966.519896@turnbull.sk.tsukuba.ac.jp> <4b8e985c07a4464c8a39055b95b38d0e@xmail103.UGent.be> Message-ID: <5BB4E806.6010805@UGent.be> On 2018-10-03 17:56, Wes Turner wrote: > The phrase "core developers and the BDFL" > is where some sort of alternate quorum/majority policy would need to be > specified if this is a contentious issue in practice? The whole point of the process described in PEP 8000 is to make this more precise. The only question for now (in particular for PEP 580) is whether the old BDFL-Delegate system can still work until the new governance model has been decided. From mike at selik.org Wed Oct 3 12:12:53 2018 From: mike at selik.org (Michael Selik) Date: Wed, 3 Oct 2018 09:12:53 -0700 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> Message-ID: On Wed, Oct 3, 2018 at 8:42 AM Anders Hovm?ller wrote: > foo(=a, =1+bar) Unfortunately, that won't help with Jonathan's inital example expression "big_array[5:20]" as it's not a valid keyword. From boxed at killingar.net Wed Oct 3 12:29:42 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Wed, 3 Oct 2018 18:29:42 +0200 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> Message-ID: <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> >> foo(=a, =1+bar) > > Unfortunately, that won't help with Jonathan's inital example > expression "big_array[5:20]" as it's not a valid keyword. I didn't understand that. The example you are referring to is print('big_array[5:20] =', big_array[5:20]) Right? Nothing is a keyword in that example or in my example. My suggestion is that we could do: my_func(=big_array[5:20]) And it would be compile time transformed into my_func(**{'big_array[5:20]': big_array[5:20]}) and then my_func is just a normal function: def my_func(**kwargs): Whatever It's a very simple textual transformation. / Anders -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Wed Oct 3 12:48:58 2018 From: mike at selik.org (Michael Selik) Date: Wed, 3 Oct 2018 09:48:58 -0700 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> Message-ID: On Wed, Oct 3, 2018 at 9:29 AM Anders Hovm?ller wrote: >>> foo(=a, =1+bar) > >> Unfortunately, that won't help with Jonathan's inital example >> expression "big_array[5:20]" as it's not a valid keyword. > > I didn't understand that. The example you are referring to is > print('big_array[5:20] =', big_array[5:20]) > > Nothing is a keyword in that example or in my example. My suggestion is that we could do: > my_func(=big_array[5:20]) > > And it would be compile time transformed into > my_func(**{'big_array[5:20]': big_array[5:20]}) > > and then my_func is just a normal function: > > def my_func(**kwargs): > Whatever You're right, that case will work. I was thinking of In [1]: foo(a+b=1) File "", line 1 foo(a+b=1) ^ SyntaxError: keyword can't be an expression From boxed at killingar.net Wed Oct 3 13:36:48 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Wed, 3 Oct 2018 19:36:48 +0200 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> Message-ID: <1AF3BDEE-9628-4021-8F0B-432E7D491537@killingar.net> >>>> foo(=a, =1+bar) >> >>> Unfortunately, that won't help with Jonathan's inital example >>> expression "big_array[5:20]" as it's not a valid keyword. >> >> I didn't understand that. The example you are referring to is >> print('big_array[5:20] =', big_array[5:20]) >> >> Nothing is a keyword in that example or in my example. My suggestion is that we could do: >> my_func(=big_array[5:20]) >> >> And it would be compile time transformed into >> my_func(**{'big_array[5:20]': big_array[5:20]}) >> >> and then my_func is just a normal function: >> >> def my_func(**kwargs): >> Whatever > > You're right, that case will work. I was thinking of > > In [1]: foo(a+b=1) > File "", line 1 > foo(a+b=1) > ^ > SyntaxError: keyword can't be an expression Sure. But no one suggested it either so I don't see how it's relevant. / Anders From rosuav at gmail.com Wed Oct 3 13:48:41 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Oct 2018 03:48:41 +1000 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> Message-ID: On Thu, Oct 4, 2018 at 2:30 AM Anders Hovm?ller wrote: > > Nothing is a keyword in that example or in my example. My suggestion is that we could do: > > my_func(=big_array[5:20]) > > And it would be compile time transformed into > > my_func(**{'big_array[5:20]': big_array[5:20]}) > > and then my_func is just a normal function: > > def my_func(**kwargs): > Whatever > > It's a very simple textual transformation. > That is not guaranteed to work. In another thread it was pointed out that this is merely a CPython implementation detail, NOT a language feature. ChrisA From jfine2358 at gmail.com Wed Oct 3 14:12:28 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Wed, 3 Oct 2018 19:12:28 +0100 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> Message-ID: Anders Hovm?ller suggested > Short form of keyword arguments where > foo(=a, =1+bar) > Is expanded at compile time to > foo(**{'a': a, '1+bar': 1+bar}) Chris Angelico wrote: > That is not guaranteed to work. In another thread it was pointed out > that this is merely a CPython implementation detail, NOT a language > feature. Here's a variant of Anders' suggestion. First, here's a dict literal {'a':1, 'b': 2, 'c':3} and here's another way to write an equivalent dict dict(a=1, b=2, c=3) So how about extending Python so that, for example, {=(1 + bar), } is equivalent to {'1 + bar': 1 + bar, } The basic idea is Anders's, recast to avoid Chris's problem. Anders: Are you willing to accept this change, if need be? Chris: Please speak up, if you think this may depend on CPython. Off topic: https://data.grammarbook.com/blog/apostrophes/apostrophes-with-names-ending-in-s-ch-or-z/ To show singular possession of a name ending in s or z, some writers add just an apostrophe. Others also add another s. -- Jonathan From rosuav at gmail.com Wed Oct 3 14:17:41 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Oct 2018 04:17:41 +1000 Subject: [Python-ideas] Dictionary initialization to capture syntax and value (was Re: Debugging: some problems and possible solutions) In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> Message-ID: On Thu, Oct 4, 2018 at 4:12 AM Jonathan Fine wrote: > > Anders Hovm?ller suggested > > Short form of keyword arguments where > > foo(=a, =1+bar) > > Is expanded at compile time to > > foo(**{'a': a, '1+bar': 1+bar}) > > Chris Angelico wrote: > > That is not guaranteed to work. In another thread it was pointed out > > that this is merely a CPython implementation detail, NOT a language > > feature. > > Here's a variant of Anders' suggestion. First, here's a dict literal > {'a':1, 'b': 2, 'c':3} > and here's another way to write an equivalent dict > dict(a=1, b=2, c=3) > > So how about extending Python so that, for example, > {=(1 + bar), } > is equivalent to > {'1 + bar': 1 + bar, } > > The basic idea is Anders's, recast to avoid Chris's problem. > > Anders: Are you willing to accept this change, if need be? > Chris: Please speak up, if you think this may depend on CPython. And then you just pass the dictionary as-is? That would be plausible, but I'm not a big fan of the syntax. Feels very clunky and forced. Spun off as a new thread because this isn't really specific to debugging. ChrisA From jfine2358 at gmail.com Wed Oct 3 14:31:46 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Wed, 3 Oct 2018 19:31:46 +0100 Subject: [Python-ideas] Dictionary initialization to capture syntax and value (was Re: Debugging: some problems and possible solutions) In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> Message-ID: > Spun off as a new thread because this isn't really specific to debugging. Thank you for starting a new thread, to discuss this idea. -- Jonathan From boxed at killingar.net Wed Oct 3 14:36:34 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Wed, 3 Oct 2018 20:36:34 +0200 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> Message-ID: >> It's a very simple textual transformation. >> > > That is not guaranteed to work. In another thread it was pointed out > that this is merely a CPython implementation detail, NOT a language > feature. Pedantry. Ok. Let's be pedantic: it's a simple textual transformation AND a promotion of an implementation detail to a spec requirement. Happy? / Anders From rosuav at gmail.com Wed Oct 3 14:38:04 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Oct 2018 04:38:04 +1000 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> Message-ID: On Thu, Oct 4, 2018 at 4:36 AM Anders Hovm?ller wrote: > > > >> It's a very simple textual transformation. > >> > > > > That is not guaranteed to work. In another thread it was pointed out > > that this is merely a CPython implementation detail, NOT a language > > feature. > > Pedantry. Ok. Let's be pedantic: it's a simple textual transformation AND a promotion of an implementation detail to a spec requirement. Happy? Then I'm strongly -1 on it. Happy? :) ChrisA From rosuav at gmail.com Wed Oct 3 14:41:22 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Oct 2018 04:41:22 +1000 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> Message-ID: On Thu, Oct 4, 2018 at 4:38 AM Chris Angelico wrote: > > On Thu, Oct 4, 2018 at 4:36 AM Anders Hovm?ller wrote: > > > > > > >> It's a very simple textual transformation. > > >> > > > > > > That is not guaranteed to work. In another thread it was pointed out > > > that this is merely a CPython implementation detail, NOT a language > > > feature. > > > > Pedantry. Ok. Let's be pedantic: it's a simple textual transformation AND a promotion of an implementation detail to a spec requirement. Happy? > > Then I'm strongly -1 on it. Happy? :) And In case it's not clear why I said that, btw: It's not mere pedantry. By restating your proposal in those terms, you make it far broader than a simple textual transformation. Imagine if you'd said it like this: "Okay, let's be pedantic. As well as my other proposals, I'm also requiring that you be able to use 'a+b' as a variable name." That is most definitely not a simple proposal. And that means it should be discussed as a much much broader change: disconnecting keyword arguments from variable names. That should NOT just slide through as part of a separate change. ChrisA From boxed at killingar.net Wed Oct 3 14:46:01 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Wed, 3 Oct 2018 20:46:01 +0200 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> Message-ID: > So how about extending Python so that, for example, > {=(1 + bar), } > is equivalent to > {'1 + bar': 1 + bar, } > > The basic idea is Anders's, recast to avoid Chris's problem. Chris' problem isn't an actual problem though. Its just a few sentences in a PEP. It might be a problem for other python implementations but I'm gonna put say 100 dollars on that it's not actually so. Pypy, jython, ironpython, who else? Without looking I'm betting they have the same implementation detail. Any takers on such a bet? One is not allowed to look before taking the bet but I can't check that so scout honor applies. > Anders: Are you willing to accept this change, if need be? I would but I think it's worse in every possible way :P It looks like a set, not a dict for example. / Anders From boxed at killingar.net Wed Oct 3 14:54:14 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Wed, 3 Oct 2018 20:54:14 +0200 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> Message-ID: <07391D84-0065-47DD-A4B5-DA6F0803E1C1@killingar.net> >> Then I'm strongly -1 on it. Happy? :) > > And In case it's not clear why I said that, btw: It's not mere > pedantry. Good to see you understood yourself why that mail wasn't so good. > By restating your proposal in those terms, you make it far > broader than a simple textual transformation. Imagine if you'd said it > like this: > > "Okay, let's be pedantic. As well as my other proposals, I'm also > requiring that you be able to use 'a+b' as a variable name." > > That is most definitely not a simple proposal. And that means it > should be discussed as a much much broader change: disconnecting > keyword arguments from variable names. That should NOT just slide > through as part of a separate change. Imagine if I said something other totally irrelevant and that is bigger change indeed. But I didn't. I suggested not a change of CPython or PyPy or IronPython but a few sentences in a PEP. I also didn't suggest that it be snuck into the same PEP as my proposed syntax changes. I agree that would be bad. It should obviously be a separate PEP. You could try first discussing the idea before requiring that I first state the legalese minutiae in exactly the right way before you even discuss the idea itself. We're pretty far away from a PEP at this stage anyway so hold your horses. This is python-ideas@ after all, not pep-lawyering at . I noticed you wouldn't take the bet too. / Anders From rosuav at gmail.com Wed Oct 3 14:59:18 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Oct 2018 04:59:18 +1000 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: <07391D84-0065-47DD-A4B5-DA6F0803E1C1@killingar.net> References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <07391D84-0065-47DD-A4B5-DA6F0803E1C1@killingar.net> Message-ID: On Thu, Oct 4, 2018 at 4:54 AM Anders Hovm?ller wrote: > > > > >> Then I'm strongly -1 on it. Happy? :) > > > > And In case it's not clear why I said that, btw: It's not mere > > pedantry. > > Good to see you understood yourself why that mail wasn't so good. > > > By restating your proposal in those terms, you make it far > > broader than a simple textual transformation. Imagine if you'd said it > > like this: > > > > "Okay, let's be pedantic. As well as my other proposals, I'm also > > requiring that you be able to use 'a+b' as a variable name." > > > > That is most definitely not a simple proposal. And that means it > > should be discussed as a much much broader change: disconnecting > > keyword arguments from variable names. That should NOT just slide > > through as part of a separate change. > > Imagine if I said something other totally irrelevant and that is bigger change indeed. But I didn't. I suggested not a change of CPython or PyPy or IronPython but a few sentences in a PEP. I also didn't suggest that it be snuck into the same PEP as my proposed syntax changes. I agree that would be bad. It should obviously be a separate PEP. > I'm not sure what you're calling irrelevant here. But sure. If you want to propose that, propose it. Start a new thread in which you propose that, as a language feature, kwargs are allowed to be invalid variable names. > I noticed you wouldn't take the bet too. No, because the bet isn't the point. You're asking if any existing Python implementation assumes that kwargs are valid identifiers. I could easily create one and win the bet, and that still wouldn't be the point. You're proposing a change to the language specification, and that's not something to just gloss over. When PEP 572 started proposing changes to other semantics than just assignment expressions, there was a lot of pushback because that was seen as an independent proposal (even though it was fairly tightly bound to the assignment expressions themselves, in that it'd be extremely hard to observe the difference else). What you're proposing here is, similarly, a completely independent proposal, and not all that tightly bound. ChrisA From wolfram.hinderer at googlemail.com Wed Oct 3 15:38:46 2018 From: wolfram.hinderer at googlemail.com (Wolfram Hinderer) Date: Wed, 3 Oct 2018 21:38:46 +0200 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> Message-ID: <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> An HTML attachment was scrubbed... URL: From jfine2358 at gmail.com Wed Oct 3 15:52:18 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Wed, 3 Oct 2018 20:52:18 +0100 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> Message-ID: Hi Wolfram You tried > def f(a): > .. print(a) > f(**{"a":2}) > 2 > f(**{"a+1":2}) > Traceback (most recent call last): > File "python", line 1, in > TypeError: f() got an unexpected keyword argument 'a+1' This is exactly what I would have expected. Please consider the following: >>> def f(a): pass >>> f(**dict(b=1)) TypeError: f() got an unexpected keyword argument 'b' > Does CPython count as "other python implementation"? Yes and No. Both are half correct. CPython is the reference implementation. Please also consider >>> def f(a, **kwargs): pass >>> f(a=1, **{'a+1': 2}) >>> f(a=1, **{(0, 1): 2}) TypeError: f() keywords must be strings So far as I know, that a keyword be a string is the only constraint, at least in CPython. For example >>> def f(a, **kwargs): pass >>> f(a=1, **{'': 2}) >>> f(a=1, **{'def': 2}) So I think Anders proposal works in CPython. I think you forgot the **kwargs in the parameters to f. best regards Jonathan From mh-python at online.de Wed Oct 3 16:29:45 2018 From: mh-python at online.de (Marcus Harnisch) Date: Wed, 3 Oct 2018 22:29:45 +0200 Subject: [Python-ideas] Better error messages for missing optional stdlib packages Message-ID: <23943232-e980-11bd-04c4-157210c732d3@gmx.net> Hi all When trying to import lzma on one of my machines, I was suprised to get a normal import error like for any other module. According to the docs lzma has been part of stdlib since 3.3. Further digging revealed that the error is due to the fact that xz wasn't compiled in when building Python. Since I suspect that there are other optional stdlib modules, this made me think whether the message in those cases should look a little more polished. Perhaps installing a stub module that prints some informative text before raising the relevant exception or similar. Also, maybe add a little note in the docs, stating that despite being part of stdlib this module might not be available on all systems. Thanks, Marcus From wolfram.hinderer at googlemail.com Wed Oct 3 17:12:58 2018 From: wolfram.hinderer at googlemail.com (Wolfram Hinderer) Date: Wed, 3 Oct 2018 23:12:58 +0200 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> Message-ID: Am 03.10.2018 um 21:52 schrieb Jonathan Fine: > >>>> def f(a, **kwargs): pass >>>> f(a=1, **{'': 2}) >>>> f(a=1, **{'def': 2}) > So I think Anders proposal works in CPython. I think you forgot the > **kwargs in the parameters to f. > > Ah, yes. Thank you. So it works in CPython 2.7. But I'm curious, does it work in very old versions? I'm not saying that this is important, because language changes always are for new versions. However, Anders' claim that this not a language change seemed too broad to me. It may be that this change has very little cost, but it should not be dismissed. Wolfram From boxed at killingar.net Wed Oct 3 17:44:53 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Wed, 3 Oct 2018 23:44:53 +0200 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <07391D84-0065-47DD-A4B5-DA6F0803E1C1@killingar.net> Message-ID: <4FA022ED-4B38-4FD8-8BA1-868187F4B5F8@killingar.net> >> Imagine if I said something other totally irrelevant and that is bigger change indeed. But I didn't. I suggested not a change of CPython or PyPy or IronPython but a few sentences in a PEP. I also didn't suggest that it be snuck into the same PEP as my proposed syntax changes. I agree that would be bad. It should obviously be a separate PEP. >> > > I'm not sure what you're calling irrelevant here. But sure. If you > want to propose that, propose it. Start a new thread in which you > propose that, as a language feature, kwargs are allowed to be invalid > variable names. But wouldn't it make sense to have a motivating example? Like the one we're discussing? Not just suggest it out of the blue? > You're proposing a change to the language specification, > and that's not something to just gloss over. Many people are suggesting language spec changes in this thread and quite a few others. This is the forum for it. > When PEP 572 started proposing changes to other semantics than just > assignment expressions, there was a lot of pushback because that was > seen as an independent proposal (even though it was fairly tightly > bound to the assignment expressions themselves, in that it'd be > extremely hard to observe the difference else). What you're proposing > here is, similarly, a completely independent proposal, and not all > that tightly bound. Sure. But I'm only proposing it in the "what if?" way. It's a discussion to see what other solutions exists for the problem that the thread started discussing. A You and me keep derailing. It's quite depressing. I don't want to be in a constant shouting game with you. I want to discuss ideas. / Anders From boxed at killingar.net Wed Oct 3 18:14:32 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Thu, 4 Oct 2018 00:14:32 +0200 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> Message-ID: <04C45FA6-7DB3-4483-981B-577CC47CDA18@killingar.net> > Ah, yes. Thank you. So it works in CPython 2.7. But I'm curious, does it work in very old versions? My bet is still on. I take paypay. I will not accept python 1 let's say. It's just easier that way. If **kwargs exists they take strings. > I'm not saying that this is important, because language changes always are for new versions. However, Anders' claim that this not a language change seemed too broad to me. Not what I meant. I meant it's not a change to any python implementation. It might be a change to promote an implementation detail to the spec as pointed out by Chris. > It may be that this change has very little cost, but it should not be dismissed. I wasn't dismissing it. I just didn't think it was yet time to bring it up. We were still discussing larger themes. But if we do have to bring it up: It has zero technical cost in actuality right now but might not in the future have zero cost. It's the same argument against the guaranteed order of dicts in 3.7: it might have a cost in the future. / Anders From mistersheik at gmail.com Wed Oct 3 18:39:56 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 3 Oct 2018 15:39:56 -0700 (PDT) Subject: [Python-ideas] Better error messages for missing optional stdlib packages In-Reply-To: <23943232-e980-11bd-04c4-157210c732d3@gmx.net> References: <23943232-e980-11bd-04c4-157210c732d3@gmx.net> Message-ID: I agree with this. These small changes can save hundreds of people an hour each. In the long run, it pays off. On Wednesday, October 3, 2018 at 4:31:08 PM UTC-4, Marcus Harnisch wrote: > > Hi all > > When trying to import lzma on one of my machines, I was suprised to get > a normal import error like for any other module. According to the docs > lzma has been part of stdlib since 3.3. Further digging revealed that > the error is due to the fact that xz wasn't compiled in when building > Python. Since I suspect that there are other optional stdlib modules, > this made me think whether the message in those cases should look a > little more polished. Perhaps installing a stub module that prints some > informative text before raising the relevant exception or similar. > Also, maybe add a little note in the docs, stating that despite being > part of stdlib this module might not be available on all systems. > > Thanks, > Marcus > > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Oct 3 18:32:55 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 4 Oct 2018 08:32:55 +1000 Subject: [Python-ideas] Better error messages for missing optional stdlib packages In-Reply-To: <23943232-e980-11bd-04c4-157210c732d3@gmx.net> References: <23943232-e980-11bd-04c4-157210c732d3@gmx.net> Message-ID: <20181003223255.GN21220@ando.pearwood.info> On Wed, Oct 03, 2018 at 10:29:45PM +0200, Marcus Harnisch wrote: > Hi all > > When trying to import lzma on one of my machines, I was suprised to get > a normal import error like for any other module. According to the docs > lzma has been part of stdlib since 3.3. Further digging revealed that > the error is due to the fact that xz wasn't compiled in when building > Python. Since I suspect that there are other optional stdlib modules, > this made me think whether the message in those cases should look a > little more polished. Perhaps installing a stub module that prints some > informative text before raising the relevant exception or similar. This sounds to me like something that the various Python distributors could do, e.g. Activestate, the Linux distros, etc. Especially since they're the ones compiling Python, they can control whether or not XY is supplied or not. > Also, maybe add a little note in the docs, stating that despite being > part of stdlib this module might not be available on all systems. That should be uncontroversial. Raise an issue on the bug tracker for that, or a patch on Github. -- Steve From steve at pearwood.info Wed Oct 3 19:13:52 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 4 Oct 2018 09:13:52 +1000 Subject: [Python-ideas] Dictionary initialization to capture syntax and value (was Re: Debugging: some problems and possible solutions) In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> Message-ID: <20181003231351.GO21220@ando.pearwood.info> On Thu, Oct 04, 2018 at 04:17:41AM +1000, Chris Angelico wrote: > > {=(1 + bar), } > > is equivalent to > > {'1 + bar': 1 + bar, } What is so special about dicts that this only works there? If we're going to have syntax to capture the source (and AST) of an expression, it ought to work everywhere. And it ought to be callable, without having to call eval() on the source, which is slow. -- Steve From boxed at killingar.net Wed Oct 3 19:15:53 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Thu, 4 Oct 2018 01:15:53 +0200 Subject: [Python-ideas] Better error messages for missing optional stdlib packages In-Reply-To: <20181003223255.GN21220@ando.pearwood.info> References: <23943232-e980-11bd-04c4-157210c732d3@gmx.net> <20181003223255.GN21220@ando.pearwood.info> Message-ID: <7FA88FBE-6F8E-4F63-85A7-77799E80CDF6@killingar.net> >> Also, maybe add a little note in the docs, stating that despite being >> part of stdlib this module might not be available on all systems. > > That should be uncontroversial. Raise an issue on the bug tracker for > that, or a patch on Github. I believe a PR that is more complex than a spelling fix needs a bug track number. At least that's what the PR submit form lead me to believe when I submitted a minor fix last week. / Anders From boxed at killingar.net Wed Oct 3 19:24:17 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Thu, 4 Oct 2018 01:24:17 +0200 Subject: [Python-ideas] Dictionary initialization to capture syntax and value (was Re: Debugging: some problems and possible solutions) In-Reply-To: <20181003231351.GO21220@ando.pearwood.info> References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <20181003231351.GO21220@ando.pearwood.info> Message-ID: >>> {=(1 + bar), } >>> is equivalent to >>> {'1 + bar': 1 + bar, } > > What is so special about dicts that this only works there? > > If we're going to have syntax to capture the source (and AST) of an > expression, it ought to work everywhere. And it ought to be callable, > without having to call eval() on the source, which is slow. So... what are you thinking? We store the AST in a way that can be inspected by the inspect module? Seems a bit vague how the API would look. Would you look at the stack and traverse it and the frames contain detailed information on where a function call originated in the AST? That's how I interpret the "works everywhere" idea. It seems pretty complex! I hope you have a simpler idea. On a related note: any access of the AST will be annoying to use if the standard library doesn't have an unparse feature, and if it does, it'll be slightly annoying if all formatting and comments are thrown away like the current AST does. / Anders From eric at trueblade.com Wed Oct 3 19:38:52 2018 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 3 Oct 2018 19:38:52 -0400 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: <8b914470-a7c3-764a-7c30-a500ef1acf8a@trueblade.com> References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <8b914470-a7c3-764a-7c30-a500ef1acf8a@trueblade.com> Message-ID: <6a0bf16d-9959-babf-bbcc-33222e570f1c@trueblade.com> On 10/3/2018 7:07 AM, Eric V. Smith wrote: > On 10/3/2018 12:23 AM, Tim Peters wrote: >> >> [Eric V. Smith >] >> >> ??? Here?s the idea: for f-strings, we add a !d conversion operator, >> which >> ??? is superficially similar to !s, !r, and !a. The meaning of !d is: >> ??? produce the text of the expression (not its value!), followed by an >> ??? equal sign, followed by the repr of the value of the expression. >> ... >> >> ??? The result is a string, so if you really wanted to, you could use a >> ??? string formatting spec. So: >> >> ??? print(f'*{value!d:^20}*' >> >> ??? would produce: >> >> ??? *? ? ? value=10? ? ? * >> >> ??? Although I don?t think that would be very useful in general. >> >> >> Me neither ;-)? But what if >> >> ?? ? {EXPR!d:FMT} >> >> acted like the current >> >> ?? ? EXPR={EXPR:FMT} >> >> ?? I'd find _that_ useful often.? For example, when displaying floats, >> where the repe is almost never what I want to see. >> >> ?>>> f"math.pi={math.pi:.2f}" >> 'math.pi=3.14' >> >> I have plenty of code already embedding stuff of the form >> >> ?? ? EXPR {EXPR:FMT} >> >> and don't really care whether there's a space or an "=" between the >> chunks.? "!d" could act like a macro-expansion operator automating a >> mechanical transformation inside the f-string.? Then I could read "!d" >> as "duplicate" instead of as "debug" ;-) > > That's a very interesting idea. It breaks the expectation (which might > be mine alone!) that the format spec is used on the result of the > conversion specifier (!s, !r). But maybe that's okay. It does seem > generally more useful than the format spec being used on the repr of the > evaluated expression. After giving this some more thought, the problem with this approach is that there's no way to get the repr of the object, which for debugging can be pretty useful (think strings). Remember, by default object.__format__ calls object.__str__. I guess we could get around this by combining !d and !r and assigning some meaning to that, which I'd rather not do. Or, this occurred to me right before hitting "send": if there's no format spec, use repr(expr). If there is one (even if it's zero-length), use format(expr, spec). I'll have to think on that for a while. Maybe there's too much voodoo going on there. Eric From tim.peters at gmail.com Wed Oct 3 20:01:39 2018 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 3 Oct 2018 19:01:39 -0500 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: <6a0bf16d-9959-babf-bbcc-33222e570f1c@trueblade.com> References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <8b914470-a7c3-764a-7c30-a500ef1acf8a@trueblade.com> <6a0bf16d-9959-babf-bbcc-33222e570f1c@trueblade.com> Message-ID: [Tim] > >> .... But what if >> > >> {EXPR!d:FMT} > >> > >> acted like the current > >> > >> EXPR={EXPR:FMT} > >> > >> ? I'd find _that_ useful often. For example, when displaying floats, > >> where the repr is almost never what I want to see. > >> ... [Eric V. Smith ] > After giving this some more thought, the problem with this approach is > that there's no way to get the repr of the object, which for debugging > can be pretty useful (think strings). Sure there is: f"{repr(var)!d"} would expand to, as a matter of course (nothing special about it): f"{repr(var)={repr(var)}" which would yield, e.g., repr(var)=`a\n' Special shortcuts for calling `repr()` went out of style when Guido got rid of that meaning for the backtick symbol ;-) Remember, by default > object.__format__ calls object.__str__. > Which - since there's already a related default - makes it a Dubious Idea to make some other spelling use a different default. > I guess we could get around this by combining !d and !r and assigning > some meaning to that, which I'd rather not do. > > Or, this occurred to me right before hitting "send": if there's no > format spec, use repr(expr). If there is one (even if it's zero-length), > use format(expr, spec). I'll have to think on that for a while. Maybe > there's too much voodoo going on there. > The alternative: if I want repr,(which I usually don't), make me call "repr()" (which I don't mind at all). If there must be a shortcut, "!dr" or "!rd" are at least cryptically explicit, and {EXPR!dr} would expand to EXPR={repr(EXPR)} -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Oct 3 20:06:26 2018 From: guido at python.org (Guido van Rossum) Date: Wed, 3 Oct 2018 17:06:26 -0700 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <8b914470-a7c3-764a-7c30-a500ef1acf8a@trueblade.com> <6a0bf16d-9959-babf-bbcc-33222e570f1c@trueblade.com> Message-ID: Gah! You are overthinking it. This idea is only worth it if it's dead simple, and the version that Eric posted to start this thread, where !d uses the repr() of the expression, is the only one simple enough to bother implementing. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Wed Oct 3 20:47:45 2018 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 3 Oct 2018 19:47:45 -0500 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <8b914470-a7c3-764a-7c30-a500ef1acf8a@trueblade.com> <6a0bf16d-9959-babf-bbcc-33222e570f1c@trueblade.com> Message-ID: [Guido] > Gah! You are overthinking it. This idea is only worth it if it's dead > simple, and the version that Eric posted to start this thread, where !d > uses the repr() of the expression, is the only one simple enough to bother > implementing. > In that case, I have no real use for it, for reasons already explained (I rarely want the repr, and already have lots of code that tediously repeats the EXPR {EXPR:FMT} pattern inside f-strings). Even for displaying integers, where you might _assume_ I'd be happy with the repr, I'm often not: print(f"nballs {nballs:,} into nbins {nbins:,}") is one I just happened to write today. Without comma formatting, the output is much harder to read. Note that transforming {EXPR!d:FMT} into EXPR={repr(EXPR):FMT} is actually slightly more involved than transforming it into EXPR={EXPR:FMT} so I don't buy the argument that the original idea is simpler. More magical and less useful, yes ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Oct 3 21:24:06 2018 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 3 Oct 2018 18:24:06 -0700 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> Message-ID: On Wed, Oct 3, 2018, 03:55 Eric V. Smith wrote: > > On 10/3/2018 1:40 AM, Nathaniel Smith wrote: > > I think the way I'd do it would be: > > > > Step 1: Take the current "lnotab" that lets us map bytecode offsets -> > > line numbers, and extend it with more detailed information, so that we > > can map e.g. a CALL operation to the exact start and end positions of > > that call expression in the source. This is free at runtime, and would > > allow more detailed tracebacks (see [1] for an example), and more > > detailed coverage information. It would definitely take some work to > > thread the necessary information through the compiler infrastructure, > > but I think this would be a worthwhile feature even without the debug() > > use case. > > > > Step 2: Add a 'debug' helper function that exploits the detailed > > information to reconstruct its call, by peeking into the calling frame > > and finding the source for the call. Of course this would be a strange > > and ugly thing to do for a regular function, but for a debugging helper > > it's reasonable. So e.g. if you had the code: > > > > total = debug(x) + debug(y / 10) > > > > The output might be: > > > > debug:myfile.py:10: 'x' is 3 > > debug:myfile.py:10: 'y / 10' is 7 > > I'm not positive, but isn't this what q does? The difference is that without "step 1", there's no reliable way to figure out the value's source text. q does it by grabbing the source line and making some guesses based on heuristics, but e.g. in the example here it gets confused and prints: 0.0s : x) + q(y / 10=3 0.0s : x) + q(y / 10=7 So you can think of this idea as (1) make it possible to implement a reliable version of q, (2) add an built-in implementation. -n From njs at pobox.com Wed Oct 3 22:04:25 2018 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 3 Oct 2018 19:04:25 -0700 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> Message-ID: On Wed, Oct 3, 2018 at 10:48 AM, Chris Angelico wrote: > On Thu, Oct 4, 2018 at 2:30 AM Anders Hovm?ller wrote: >> >> Nothing is a keyword in that example or in my example. My suggestion is that we could do: >> >> my_func(=big_array[5:20]) >> >> And it would be compile time transformed into >> >> my_func(**{'big_array[5:20]': big_array[5:20]}) >> >> and then my_func is just a normal function: >> >> def my_func(**kwargs): >> Whatever >> >> It's a very simple textual transformation. >> > > That is not guaranteed to work. In another thread it was pointed out > that this is merely a CPython implementation detail, NOT a language > feature. I'm curious where this is written down. Can you point to the relevant part of the language spec or pronouncement or whatever it was? -n -- Nathaniel J. Smith -- https://vorpus.org From boxed at killingar.net Thu Oct 4 01:07:49 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Thu, 4 Oct 2018 07:07:49 +0200 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> Message-ID: <742BDCDB-13A6-4F8F-9EFC-1CD7695CB9F7@killingar.net> >> That is not guaranteed to work. In another thread it was pointed out >> that this is merely a CPython implementation detail, NOT a language >> feature. > > I'm curious where this is written down. Can you point to the relevant > part of the language spec or pronouncement or whatever it was? I'm going to be charitable here and suggest that Chris is referring to something implicit. The spec clearly says that identifiers must be valid keys because otherwise normal functions wouldn't work. But that's it. So in Chris' view you're probably asking him to prove a negative. The burden of proof would lie on those who say otherwise to show how the spec allows arbitrary strings. But no one has made such a claim so it's a bit moot. / Anders From hpolak at polak.es Thu Oct 4 03:41:48 2018 From: hpolak at polak.es (Hans Polak) Date: Thu, 4 Oct 2018 09:41:48 +0200 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: Message-ID: <933bb2fc-cad2-3ec4-7253-13fa12cd38d4@polak.es> Good morning, I read about a "time machine" debugger a long time ago. The debugger would collect all the information of all the calls and the programmer can just execute the code without breakpoints. Later, the programmer can follow the evolution of a variable until it reaches an erroneous value. I've never worked with that and it sounds really memory intensive, but the examples were quite interesting. When you are in a loop, you want to break at a certain iteration. When you are in a recursive function, you want to stop at the right point. etc. Cheers, Hans On 03/10/18 16:51, Jonathan Fine wrote: > This thread is about debugging. I suggest we start by collecting > problems and possible solutions. And that after about a week of that, > we start discussing what we've gathered. > > We already have a problem and possible solution, provided by Eric > Smith and Larry Hastings. > > > TITLE: f-string "debug" conversion > URL: https://mail.python.org/pipermail/python-ideas/2018-October/053956.html > PROBLEM > Writing > print('value = ', value) > is tedious, particularly for more complicated expressions, such as > print('big_array[5:20] =', big_array[5:20]) > > POSSIBLE SOLUTION > > For f-strings, we add a !d conversion operator, which produces the > text of an expression followed by its value. Thus, the two previous > examples can be written more concisely as > print(f'{value !d}') > print(f'{big_array[5:20] !d}') > > > I suggest for the moment that we just gather problem-solution pairs, > much as above. I think they'll be between 5 and 15 such pairs. I'll > post one to the current discussion thread in an hour or so. > > And that after about a week, we move to discussing what we have. > Finally, many thanks to Eric and Larry for their useful contribution > to the important problem of debugging. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From boxed at killingar.net Thu Oct 4 04:32:53 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Thu, 4 Oct 2018 10:32:53 +0200 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> Message-ID: > Ah, yes. Thank you. So it works in CPython 2.7. But I'm curious, does it work in very old versions? > I'm not saying that this is important, because language changes always are for new versions. However, Anders' claim that this not a language change seemed too broad to me. > It may be that this change has very little cost, but it should not be dismissed. It works in: Python 1 Python 2 Python 3 PyPy 6 IronPython Jython micropython Are there more I should try? I highly recommend https://tio.run/ for trying this out. It's pretty cool! It doesn't have micropython but it has the others. / Anders -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Thu Oct 4 07:41:44 2018 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 4 Oct 2018 07:41:44 -0400 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <8b914470-a7c3-764a-7c30-a500ef1acf8a@trueblade.com> <6a0bf16d-9959-babf-bbcc-33222e570f1c@trueblade.com> Message-ID: <44ec8378-2a3c-e615-2a19-d02d3d10443f@trueblade.com> On 10/3/2018 8:47 PM, Tim Peters wrote: > Note that transforming > > ? ?{EXPR!d:FMT} > > into > > ? ?EXPR={repr(EXPR):FMT} > > is actually slightly more involved than transforming it into > > ? ?EXPR={EXPR:FMT} > > so I don't buy the argument that the original idea is simpler.? More > magical and less useful, yes ;-) Actually, my proposal is to apply FMT to the entire result of EXPR={repr(EXPR)}, not just the repr(EXPR) part. I'm not sure either is particularly useful. Eric From rosuav at gmail.com Thu Oct 4 08:10:21 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Oct 2018 22:10:21 +1000 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> Message-ID: On Thu, Oct 4, 2018 at 6:33 PM Anders Hovm?ller wrote: > > > Ah, yes. Thank you. So it works in CPython 2.7. But I'm curious, does it work in very old versions? > I'm not saying that this is important, because language changes always are for new versions. However, Anders' claim that this not a language change seemed too broad to me. > It may be that this change has very little cost, but it should not be dismissed. > > > It works in: > > Python 1 > Python 2 > Python 3 > PyPy 6 > IronPython > Jython > micropython > > Are there more I should try? I've no idea what you actually tried or what actually worked, since you haven't shown your code. However, it doesn't matter. This IS a language change, and it makes no difference how many implementations are lax enough to permit it currently. Citation: https://mail.python.org/pipermail/python-dev/2018-October/155437.html It's just like a program assuming that there will always be a user with UID 0, or assuming that every human being has a name that consists of a given name and a family name, or assuming that data sent across the internet will arrive unchanged. You can show millions, billions, trillions of examples that support your assumption, but that doesn't make any difference - the assumption is false as soon as there is a single counter-example, or as soon as the specification is shown to *permit* a counter-example. ChrisA From boxed at killingar.net Thu Oct 4 08:27:55 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Thu, 4 Oct 2018 14:27:55 +0200 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> Message-ID: <5FE3F758-2D89-4682-88FC-D266AFEF91E1@killingar.net> >> Ah, yes. Thank you. So it works in CPython 2.7. But I'm curious, does it work in very old versions? >> I'm not saying that this is important, because language changes always are for new versions. However, Anders' claim that this not a language change seemed too broad to me. >> It may be that this change has very little cost, but it should not be dismissed. >> >> >> It works in: >> >> Python 1 >> Python 2 >> Python 3 >> PyPy 6 >> IronPython >> Jython >> micropython >> >> Are there more I should try? > > I've no idea what you actually tried or what actually worked, since > you haven't shown your code. Please take a deep breath. The question was not directed at you and I just answered a clear and simply stated question. All the examples cited, including the one you link to below work. No need to get angry about this. If you are upset if I discuss implementation details don't reply. This is all I'm doing at this point. > However, it doesn't matter. Of course it matters. It's the difference between changing the spec and changing the spec AND some implementation. There is a difference between those two things. You might not care but that's another topic. > This IS a > language change Yes I agree. I have said so many times. > , and it makes no difference how many implementations > are lax enough to permit it currently. Sure it does. See above. > You can show millions, > billions, trillions of examples that support your assumption, but that > doesn't make any difference - the assumption is false as soon as there > is a single counter-example, ...which there isn't. But that's irrelevant. We both agree it's irrelevant. > or as soon as the specification is shown > to *permit* a counter-example. Maybe. But I haven't argued that this implementation detail is already in the spec have I? I have just argued that it's easy to IMPLEMENT because it is in fact already implemented in all existing pythons. I don't see why this is such a hard concept for you to grasp. Yes I know it would be a change to the spec. I have conceded this point MANY TIMES. You don't need to argue that point, you've already won it. By walk over even because I never argued against it. Can we drop this now? Your point has been made. / Anders From rhodri at kynesim.co.uk Thu Oct 4 10:35:19 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Thu, 4 Oct 2018 15:35:19 +0100 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: <5FE3F758-2D89-4682-88FC-D266AFEF91E1@killingar.net> References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> <5FE3F758-2D89-4682-88FC-D266AFEF91E1@killingar.net> Message-ID: <9cdae6fe-06b7-4d22-7141-f0bc081f63d9@kynesim.co.uk> On 04/10/18 13:27, Anders Hovm?ller wrote: [I think >> = ChrisA] >> However, it doesn't matter. > Of course it matters. It's the difference between changing the spec and changing the spec AND some implementation. There is a difference between those two things. You might not care but that's another topic. In terms of defining specs, not there isn't a difference between those things. Changing the language spec is not a thing to do lightly regardless of how many implementations happen to do what you already want. While gather evidence that the chaos your change will cause is minimal is a good thing, it doesn't actually lower the bar for getting a language spec change through. Think of it this way: you are proposing to break a promise. Even if it's only a little thing and you're fairly sure everyone will forgive you, that's still not something you should ever want to get comfortable doing. -- Rhodri James *-* Kynesim Ltd From mikhailwas at gmail.com Thu Oct 4 11:31:24 2018 From: mikhailwas at gmail.com (Mikhail V) Date: Thu, 4 Oct 2018 18:31:24 +0300 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> Message-ID: On Wed, Oct 3, 2018 at 3:28 AM Eric V. Smith wrote: > > Here?s the idea: for f-strings, we add a !d conversion operator, which > is superficially similar to !s, !r, and !a. The meaning of !d is: > produce the text of the expression (not its value!), followed by an > equal sign, followed by the repr of the value of the expression. So: > [...] > The mnemonic is !d for ?debugging?. I?d wanted to use !=, because > there?s an equal sign involved in the result, but = is the one character > that can?t be used after ! (it?s ?not equal? in expressions, and > f-strings look specifically for that case). I also mentioned !!, but I > think I prefer !d as being less confusing. > > This would be used in debugging print statements, that currently end up > looking like: > > print(f'value={value!r}') > > and would now be: > > print(f'{value!d}') > > There have been discussions about ways to specify str() vs. repr(), > using characters other than '=', adding spaces, etc. But they all end up > over-complicating what should be a simple tool, not a Swiss Army knife. > > Thoughts? > > Eric I think the feature is useful to avoid re-typing the expressions. But still, the syntax is a bit obscure, and the feature is very limited (can't construct anything but "x=x"). That makes me personally less enthusiastic about using it instead of the common: f" x = {x}". (yes I use spaces around = most of the time and probably only spaces for table-like output) Did you think about more general syntax for repetitions? If there would be a way to repeat entries, one could do similar output without much extra typing. E.g. an asterisk for "same expression" ? price = 10 s = f"{price} {*}" # 10 10 And then using say {expr !} would print _only_ the expression string. So with e.g. some variable: variable = 13.18654937 f"{variable !} = {*:.4f} {*:.2f} {*:.0f}" prints: variable = 13.1865 13.19 13 Something like that. It is more complex implementation, etc, but has better, verbose appearance Imo and not so limited. And custom formatting is important to achieve a readable output. Mikhail From boxed at killingar.net Thu Oct 4 11:57:45 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Thu, 4 Oct 2018 17:57:45 +0200 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: <9cdae6fe-06b7-4d22-7141-f0bc081f63d9@kynesim.co.uk> References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> <5FE3F758-2D89-4682-88FC-D266AFEF91E1@killingar.net> <9cdae6fe-06b7-4d22-7141-f0bc081f63d9@kynesim.co.uk> Message-ID: <22F1CC9D-CC3E-4602-8D14-99A9A852570E@killingar.net> > [I think >> = ChrisA] >>> However, it doesn't matter. >> Of course it matters. It's the difference between changing the spec and changing the spec AND some implementation. There is a difference between those two things. You might not care but that's another topic. > > In terms of defining specs, not there isn't a difference between those things. So in terms of X there is no difference in Y. No kidding. Which is why I said so. Repeatedly. > Changing the language spec is not a thing to do lightly regardless of how many implementations happen to do what you already want. Where do you think I'm taking it lightly? I'm taking changing all python implementations seriously. That has non-zero weight to me. Do you and Chris think that changing the implementations of all python implementations and the complexity of that change is totally irrelevant to all discussions? I sure hope not! Changing the spec AND all implementations is pretty much by definition a bigger change and should be taken more seriously than only the spec. I don't take changing the spec lightly because I think changing the spec AND all implementation is MORE. I can't believe I'm arguing that 1 + 1 > 1 but here we are. I would hope you could drop this idea that I'm taking things "lightly" whatever that means. If I were taking it lightly I would have already submitted a PEP. > While gather evidence that the chaos your change will cause is minimal is a good thing, it doesn't actually lower the bar for getting a language spec change through. Why do you repeat that? No one is saying the opposite. I'm saying the facts lowers the bar for IMPLEMENTATION. Because it's literally zero. That is totally orthogonal to the bar for getting a spec change through of course. Which I have already conceded like ten times now. How many times do I have to concede this point before you stop saying I need to concede it? Do I need to write it with my own blood on parchment or something? > Think of it this way: you are proposing to break a promise. Even if it's only a little thing and you're fairly sure everyone will forgive you, that's still not something you should ever want to get comfortable doing. Every new thread on this mailing list does this. This is the entire point of this list. And I've conceded this point so you're just beating a dead horse. / Anders From jfine2358 at gmail.com Thu Oct 4 12:15:42 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Thu, 4 Oct 2018 17:15:42 +0100 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: <22F1CC9D-CC3E-4602-8D14-99A9A852570E@killingar.net> References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> <5FE3F758-2D89-4682-88FC-D266AFEF91E1@killingar.net> <9cdae6fe-06b7-4d22-7141-f0bc081f63d9@kynesim.co.uk> <22F1CC9D-CC3E-4602-8D14-99A9A852570E@killingar.net> Message-ID: Hello all I think we've established enough basic facts for now, about using a non-identifier string as a keyword in a kwargs dictionary. I'd be most grateful if discussion of this particular topic could be suspended for now, to be resumed when it again becomes relevant. If you are keen to discuss this particular topic right now, please start a new discussion thread for that purpose. Later today, I'll be posting another problem and proposed solution to this thread, and would like your help on that. Many thanks Jonathan From mh-python at online.de Thu Oct 4 12:46:52 2018 From: mh-python at online.de (Marcus Harnisch) Date: Thu, 4 Oct 2018 18:46:52 +0200 Subject: [Python-ideas] Better error messages for missing optional stdlib packages In-Reply-To: <20181003223255.GN21220@ando.pearwood.info> References: <23943232-e980-11bd-04c4-157210c732d3@gmx.net> <20181003223255.GN21220@ando.pearwood.info> Message-ID: <09afccaf-7959-4e85-270a-f31c851c7e04@gmx.net> Hi Steven On 10/04/2018 12:32 AM, Steven D'Aprano wrote: > On Wed, Oct 03, 2018 at 10:29:45PM +0200, Marcus Harnisch wrote: >> When trying to import lzma on one of my machines, I was suprised to get >> a normal import error like for any other module. According to the docs >> lzma has been part of stdlib since 3.3. Further digging revealed that >> the error is due to the fact that xz wasn't compiled in when building >> Python. Since I suspect that there are other optional stdlib modules, >> this made me think whether the message in those cases should look a >> little more polished. Perhaps installing a stub module that prints some >> informative text before raising the relevant exception or similar. > This sounds to me like something that the various Python distributors > could do, e.g. Activestate, the Linux distros, etc. Especially since > they're the ones compiling Python, they can control whether or not XY is > supplied or not. I'd argue that it is Python that creates the dependency and is therefore responsible to handle this gracefully. But, I guess having this documented first is the most important step before discussing further measures. >> Also, maybe add a little note in the docs, stating that despite being >> part of stdlib this module might not be available on all systems. > That should be uncontroversial. Raise an issue on the bug tracker for > that, or a patch on Github. Done. https://bugs.python.org/issue34895 Thanks, Marcus From solipsis at pitrou.net Thu Oct 4 12:51:20 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 4 Oct 2018 18:51:20 +0200 Subject: [Python-ideas] Better error messages for missing optional stdlib packages References: <23943232-e980-11bd-04c4-157210c732d3@gmx.net> <20181003223255.GN21220@ando.pearwood.info> Message-ID: <20181004185120.29080416@fsol> On Thu, 4 Oct 2018 08:32:55 +1000 Steven D'Aprano wrote: > > > Also, maybe add a little note in the docs, stating that despite being > > part of stdlib this module might not be available on all systems. > > That should be uncontroversial. Raise an issue on the bug tracker for > that, or a patch on Github. I don't know. Realistically, any decent distributor of Python should include those optional modules, and I don't remember encountering a Python that doesn't has them. Adding a note saying otherwise in the docs may pointlessly scare users away. Regards Antoine. From jfine2358 at gmail.com Thu Oct 4 13:41:22 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Thu, 4 Oct 2018 18:41:22 +0100 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <7FE0D881-481B-4FE3-A0B8-6C36D68ABDBB@killingar.net> <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> <5FE3F758-2D89-4682-88FC-D266AFEF91E1@killingar.net> <9cdae6fe-06b7-4d22-7141-f0bc081f63d9@kynesim.co.uk> <22F1CC9D-CC3E-4602-8D14-99A9A852570E@killingar.net> Message-ID: TITLE: PROBLEM: Debug print() statements cause doctests to fail Adding debug print(...) statements to code can cause doctests to fail. This is because both use sys.stdout as the output stream. POSSIBLE SOLUTION: Provide and use a special stream for debug output. In other words, something like >>> import sys >>> sys.stddebug = sys.stderr >>> debug = lambda *argv, **kwargs: print(*argv, file=sys.stddebug, flush=True, **kwargs) Note: Need to use current value of sys.stddebug, so can't use functools.partial. From rhodri at kynesim.co.uk Thu Oct 4 13:54:08 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Thu, 4 Oct 2018 18:54:08 +0100 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> <5FE3F758-2D89-4682-88FC-D266AFEF91E1@killingar.net> <9cdae6fe-06b7-4d22-7141-f0bc081f63d9@kynesim.co.uk> <22F1CC9D-CC3E-4602-8D14-99A9A852570E@killingar.net> Message-ID: On 04/10/18 18:41, Jonathan Fine wrote: > TITLE: > PROBLEM: Debug print() statements cause doctests to fail > Adding debug print(...) statements to code can cause doctests to fail. > This is because both use sys.stdout as the output stream. > > POSSIBLE SOLUTION: > Provide and use a special stream for debug output. In other words, > something like > >>>> import sys >>>> sys.stddebug = sys.stderr >>>> debug = lambda *argv, **kwargs: print(*argv, file=sys.stddebug, flush=True, **kwargs) > > Note: Need to use current value of sys.stddebug, so can't use functools.partial. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > Or write your debug output to stderr? -- Rhodri James *-* Kynesim Ltd From jfine2358 at gmail.com Thu Oct 4 14:10:46 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Thu, 4 Oct 2018 19:10:46 +0100 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> <5FE3F758-2D89-4682-88FC-D266AFEF91E1@killingar.net> <9cdae6fe-06b7-4d22-7141-f0bc081f63d9@kynesim.co.uk> <22F1CC9D-CC3E-4602-8D14-99A9A852570E@killingar.net> Message-ID: In response to my problem-solution pair (fixing a typo) > TITLE: Debug print() statements cause doctests to fail Rhodri James wrote: > Or write your debug output to stderr? Perhaps I've been too concise. If so, I apologise. My proposal is that the system be set up so that debug(a, b, c) sends output to the correct stream, whatever it should be. Rhodri: Thank you for your contribution. Are you saying that because the developer can write print(a, b, c, file=sys.stderr) there's not a problem to solve here? -- Jonathan From mh-python at online.de Thu Oct 4 16:18:21 2018 From: mh-python at online.de (Marcus Harnisch) Date: Thu, 4 Oct 2018 22:18:21 +0200 Subject: [Python-ideas] Better error messages for missing optional stdlib packages In-Reply-To: <20181004185120.29080416@fsol> References: <23943232-e980-11bd-04c4-157210c732d3@gmx.net> <20181003223255.GN21220@ando.pearwood.info> <20181004185120.29080416@fsol> Message-ID: <7ddd798e-200a-4d8f-46ab-1b2c21ddf2e5@gmx.net> On 10/04/2018 06:51 PM, Antoine Pitrou wrote: > I don't know. Realistically, any decent distributor of Python > should include those optional modules, and I don't remember > encountering a Python that doesn't has them. What is the point, then, of making these packages optional in the first place? Shouldn't by that logic a missing dependency rather trigger a build failure /unless/ someone enables a certain switch (e.g. ?--without-xz?), e.g. for special-purpose distribution. > Adding a note saying otherwise in the docs may pointlessly scare users away. Possibly. But not saying anything isn't the best alternative under these circumstances. Thanks, Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Thu Oct 4 18:24:10 2018 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 4 Oct 2018 17:24:10 -0500 Subject: [Python-ideas] f-string "debug" conversion In-Reply-To: <44ec8378-2a3c-e615-2a19-d02d3d10443f@trueblade.com> References: <3473f538-7683-28c0-c45e-3e0776bfd7ed@trueblade.com> <8b914470-a7c3-764a-7c30-a500ef1acf8a@trueblade.com> <6a0bf16d-9959-babf-bbcc-33222e570f1c@trueblade.com> <44ec8378-2a3c-e615-2a19-d02d3d10443f@trueblade.com> Message-ID: [Tim] > > > Note that transforming > > > > {EXPR!d:FMT} > > > > into > > > > EXPR={repr(EXPR):FMT} > > > > is actually slightly more involved than transforming it into > > > > EXPR={EXPR:FMT} > > > > so I don't buy the argument that the original idea is simpler. More > > magical and less useful, yes ;-) > [Eric V. Smith] > Actually, my proposal is to apply FMT to the entire result of EXPR={repr(EXPR)}, not just the repr(EXPR) part. I'm not sure either is > particularly useful. > So the actual transformation is from {EXPR!d:FMT} to {"EXPR=" + repr(EXPR):FMT} I expect I'll barely use "!d" at all if the format doesn't apply to the result of EXPR, so I have no non-contrived ;-) opinion about that. BTW, I checked, and I've never used !r, !s, or !a. So the idea that the format could apply to a string - when EXPR itself doesn't evaluate to a string - is simply foreign to me. I suppose it's natural to people who do use ![rsa] all the time - if such people exist ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfine2358 at gmail.com Fri Oct 5 11:48:49 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Fri, 5 Oct 2018 16:48:49 +0100 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> <5FE3F758-2D89-4682-88FC-D266AFEF91E1@killingar.net> <9cdae6fe-06b7-4d22-7141-f0bc081f63d9@kynesim.co.uk> <22F1CC9D-CC3E-4602-8D14-99A9A852570E@killingar.net> Message-ID: I wrote: > Perhaps I've been too concise. If so, I apologise. My proposal is that > the system be set up so that > debug(a, b, c) > sends output to the correct stream, whatever it should be. See also: https://docs.python.org/3/whatsnew/3.7.html#pep-553-built-in-breakpoint > Python 3.7 includes the new built-in breakpoint() function as > an easy and consistent way to enter the Python debugger. So I'm suggesting a new built-in debug() function as an easy and consistent way to write out debugging information. -- Jonathan From S at muelcolvin.com Fri Oct 5 11:57:52 2018 From: S at muelcolvin.com (Samuel Colvin) Date: Fri, 5 Oct 2018 16:57:52 +0100 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> <5FE3F758-2D89-4682-88FC-D266AFEF91E1@killingar.net> <9cdae6fe-06b7-4d22-7141-f0bc081f63d9@kynesim.co.uk> <22F1CC9D-CC3E-4602-8D14-99A9A852570E@killingar.net> Message-ID: > So I'm suggesting a new built-in debug() function as an easy and > consistent way to write out debugging information. Python definitely needs a dedicated debug print command. Since "debug" is take in 3.7, perhaps "debug.print" would work? I've built python devtools with has such a command: https://github.com/samuelcolvin/python-devtools (I'm not trying to promote devtools, just demonstrate what is possible.) The packge needs a little work but it's already extremely useful, I use it all day every day. A few things with devtools' debug does: * pretty printing of objects (like pprint but IMHO clearer) * coloured output of objects * extra info about objects, eg. length, type etc. * line numbers and function names where debug() was called * names of the varible (or expression) which is being printed It would be difficult to fully integrate a package like this into the standard lib since it relies on pygments for colouring output, but not impossible. Is this the kind of thing you were thinking about? Samuel Colvin -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at rachum.com Sat Oct 6 03:22:20 2018 From: ram at rachum.com (Ram Rachum) Date: Sat, 6 Oct 2018 10:22:20 +0300 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: Hi, I'd like to use the re module to parse a long text file, 1GB in size. I wish that the re module could parse a stream, so I wouldn't have to load the whole thing into memory. I'd like to iterate over matches from the stream without keeping the old matches and input in RAM. What do you think? Thanks, Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sat Oct 6 04:21:48 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 6 Oct 2018 11:21:48 +0300 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: 06.10.18 10:22, Ram Rachum ????: > I'd like to use the re module to parse a long text file, 1GB in size. I > wish that the re module could parse a stream, so I wouldn't have to load > the whole thing into memory. I'd like to iterate over matches from the > stream without keeping the old matches and input in RAM. How would you match 'a.*b' without loading the whole thing into memory? From ram at rachum.com Sat Oct 6 04:32:48 2018 From: ram at rachum.com (Ram Rachum) Date: Sat, 6 Oct 2018 11:32:48 +0300 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: It'll load as much as it needs to in order to match or rule out a match on a pattern. If you'd try to match `a.*b` it'll load the whole thing. The use cases that are relevant to a stream wouldn't have these kinds of problems. On Sat, Oct 6, 2018 at 11:22 AM Serhiy Storchaka wrote: > 06.10.18 10:22, Ram Rachum ????: > > I'd like to use the re module to parse a long text file, 1GB in size. I > > wish that the re module could parse a stream, so I wouldn't have to load > > the whole thing into memory. I'd like to iterate over matches from the > > stream without keeping the old matches and input in RAM. > > How would you match 'a.*b' without loading the whole thing into memory? > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfine2358 at gmail.com Sat Oct 6 06:53:15 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Sat, 6 Oct 2018 11:53:15 +0100 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: Hi Ram You wrote: > I'd like to use the re module to parse a long text file, 1GB in size. I > wish that the re module could parse a stream, so I wouldn't have to load > the whole thing into memory. I'd like to iterate over matches from the > stream without keeping the old matches and input in RAM. This is a regular expression problem, rather than a Python problem. A search for regular expression large file brings up some URLs that might help you, starting with https://stackoverflow.com/questions/23773669/grep-pattern-match-between-very-large-files-is-way-too-slow This might also be helpful https://svn.boost.org/trac10/ticket/11776 What will work for your problem depends on the nature of the problem you have. The simplest thing that might work is to iterate of the file line-by-line, and use a regular expression to extract matches from each line. In other words, something like (not tested) def helper(lines): for line in lines: yield from re.finditer(pattern, line) lines = open('my-big-file.txt') for match in helper(lines): # Do your stuff here Parsing is not the same as lexing, see https://en.wikipedia.org/wiki/Lexical_analysis I suggest you use regular expressions ONLY for the lexing phase. If you'd like further help, perhaps first ask yourself this. Can the lexing be done on a line-by-line basis? And if not, why not? If line-by-line not possible, then you'll have to modify the helper. At the end of each line, they'll be a residue / remainder, which you'll have to bring into the next line. In other words, the helper will have to record (and change) the state that exists at the end of each line. A bit like the 'carry' that is used when doing long addition. I hope this helps. -- Jonathan From jfine2358 at gmail.com Sat Oct 6 07:20:16 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Sat, 6 Oct 2018 12:20:16 +0100 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> <5FE3F758-2D89-4682-88FC-D266AFEF91E1@killingar.net> <9cdae6fe-06b7-4d22-7141-f0bc081f63d9@kynesim.co.uk> <22F1CC9D-CC3E-4602-8D14-99A9A852570E@killingar.net> Message-ID: Samuel Colvin wrote: > Python definitely needs a dedicated debug print command. > I've built python devtools with has such a command: > https://github.com/samuelcolvin/python-devtools > Is this the kind of thing you were thinking about? Thank you for this comment, Samuel. And also very much for your work on devtools. To answer your question: Yes, and No. I'm thinking of providing a builtin debug() command that 'does the right thing' according to the context. And the context would include the user's preferences. Which might be to use the print command in your devtools package. But the user might be in an IDE, which provides a separate window for the debug output. I suggest the builtin debug() would determine the API, and provide a reference implementation. And that many users (perhaps through their IDE) would choose a different implementation. -- Jonathan From ram at rachum.com Sat Oct 6 07:25:55 2018 From: ram at rachum.com (Ram Rachum) Date: Sat, 6 Oct 2018 14:25:55 +0300 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: "This is a regular expression problem, rather than a Python problem." Do you have evidence for this assertion, except that other regex implementations have this limitation? Is there a regex specification somewhere that specifies that streams aren't supported? Is there a fundamental reason that streams aren't supported? "Can the lexing be done on a line-by-line basis?" For my use case, it unfortunately can't. On Sat, Oct 6, 2018 at 1:53 PM Jonathan Fine wrote: > Hi Ram > > You wrote: > > > I'd like to use the re module to parse a long text file, 1GB in size. I > > wish that the re module could parse a stream, so I wouldn't have to load > > the whole thing into memory. I'd like to iterate over matches from the > > stream without keeping the old matches and input in RAM. > > This is a regular expression problem, rather than a Python problem. A > search for > regular expression large file > brings up some URLs that might help you, starting with > > https://stackoverflow.com/questions/23773669/grep-pattern-match-between-very-large-files-is-way-too-slow > > This might also be helpful > https://svn.boost.org/trac10/ticket/11776 > > What will work for your problem depends on the nature of the problem > you have. The simplest thing that might work is to iterate of the file > line-by-line, and use a regular expression to extract matches from > each line. > > In other words, something like (not tested) > > def helper(lines): > for line in lines: > yield from re.finditer(pattern, line) > > lines = open('my-big-file.txt') > for match in helper(lines): > # Do your stuff here > > Parsing is not the same as lexing, see > https://en.wikipedia.org/wiki/Lexical_analysis > > I suggest you use regular expressions ONLY for the lexing phase. If > you'd like further help, perhaps first ask yourself this. Can the > lexing be done on a line-by-line basis? And if not, why not? > > If line-by-line not possible, then you'll have to modify the helper. > At the end of each line, they'll be a residue / remainder, which > you'll have to bring into the next line. In other words, the helper > will have to record (and change) the state that exists at the end of > each line. A bit like the 'carry' that is used when doing long > addition. > > I hope this helps. > > -- > Jonathan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfine2358 at gmail.com Sat Oct 6 07:37:00 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Sat, 6 Oct 2018 12:37:00 +0100 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: I wrote: > This is a regular expression problem, rather than a Python problem. Ram wrote: > Do you have evidence for this assertion, except that > other regex implementations have this limitation? Yes. 1. I've already supplied: https://svn.boost.org/trac10/ticket/11776 2. https://en.wikipedia.org/wiki/Regular_expression does not contain the word 'stream'. Ram wrote: > For my use case, it unfortunately can't. For that case, Ram, I've already given my best advice. If you get a better idea, please do follow it. And share it with us, so we can all learn. -- Jonathan From ned at nedbatchelder.com Sat Oct 6 11:57:35 2018 From: ned at nedbatchelder.com (Ned Batchelder) Date: Sat, 6 Oct 2018 11:57:35 -0400 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: <74106724-00c6-baa1-ac69-464ceeaf0bc4@nedbatchelder.com> On 10/6/18 7:25 AM, Ram Rachum wrote: > "This is a regular expression problem, rather than a Python problem." > > Do you have evidence for this assertion, except that other regex > implementations have this limitation? Is there a regex specification > somewhere that specifies that streams aren't supported? Is there a > fundamental reason that streams aren't supported? > > > "Can the lexing be done on a line-by-line basis?" > > For my use case, it unfortunately can't. You mentioned earlier that your use case doesn't have to worry about the "a.*b" problem.? Can you tell us more about your scenario?? How would the stream know it had read enough to match or not match? Perhaps that same logic can be used to feed the data in chunks? --Ned. > > On Sat, Oct 6, 2018 at 1:53 PM Jonathan Fine > wrote: > > Hi Ram > > You wrote: > > > I'd like to use the re module to parse a long text file, 1GB in > size. I > > wish that the re module could parse a stream, so I wouldn't have > to load > > the whole thing into memory. I'd like to iterate over matches > from the > > stream without keeping the old matches and input in RAM. > > This is a regular expression problem, rather than a Python > problem. A search for > ? ? regular expression large file > brings up some URLs that might help you, starting with > https://stackoverflow.com/questions/23773669/grep-pattern-match-between-very-large-files-is-way-too-slow > > This might also be helpful > https://svn.boost.org/trac10/ticket/11776 > > What will work for your problem depends on the nature of the problem > you have. The simplest thing that might work is to iterate of the file > line-by-line, and use a regular expression to extract matches from > each line. > > In other words, something like (not tested) > > ? ?def helper(lines): > ? ? ? ?for line in lines: > ? ? ? ? ? ?yield from re.finditer(pattern, line) > > ? ? lines = open('my-big-file.txt') > ? ? for match in helper(lines): > ? ? ? ? # Do your stuff here > > Parsing is not the same as lexing, see > https://en.wikipedia.org/wiki/Lexical_analysis > > I suggest you use regular expressions ONLY for the lexing phase. If > you'd like further help, perhaps first ask yourself this. Can the > lexing be done on a line-by-line basis? And if not, why not? > > If line-by-line not possible, then you'll have to modify the helper. > At the end of each line, they'll be a residue / remainder, which > you'll have to bring into the next line. In other words, the helper > will have to record (and change) the state that exists at the end of > each line. A bit like the 'carry' that is used when doing long > addition. > > I hope this helps. > > -- > Jonathan > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Oct 6 17:00:27 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 6 Oct 2018 14:00:27 -0700 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: On Sat, Oct 6, 2018 at 12:22 AM, Ram Rachum wrote: > I'd like to use the re module to parse a long text file, 1GB in size. I wish > that the re module could parse a stream, so I wouldn't have to load the > whole thing into memory. I'd like to iterate over matches from the stream > without keeping the old matches and input in RAM. > > What do you think? This has frustrated me too. The case where I've encountered this is parsing HTTP/1.1. We have data coming in incrementally over the network, and we want to find the end of the headers. To do this, we're looking for the first occurrence of b"\r\n\r\n" OR b"\n\n". So our requirements are: 1. Search a bytearray for the regex b"\r\n\r\n|\n\n" 2. If there's no match yet, wait for more data to arrive and try again 3. When more data arrives, start searching again *where the last search left off* The last requirement is subtle, but important. The naive approach would be to rescan your whole receive buffer after each new packet arrives: end_of_headers = re.compile(b"\r\n\r\n|\n\n") while True: m = end_of_headers.search(receive_buffer) if m is None: receive_buffer += await get_more_data_from_network() # loop around and try again else: break But the code above is quadratic! If the headers are N bytes long, then on each pass through the loop we perform an O(N) regex search, and we do O(N) passes through the loop, so the whole thing is O(N**2). That means your HTTP client-or-server can be trivially DoSed by a peer who sends their headers broken into lots of small fragments. Fortunately, there's an elegant and natural solution: Just save the regex engine's internal state when it hits the end of the string, and then when more data arrives, use the saved state to pick up the search where we left off. Theoretically, any regex engine *could* support this ? it's especially obvious for DFA-based matchers, but even backtrackers like Python's re could support it, basically by making the matching engine a coroutine that can suspend itself when it hits the end of the input, then resume it when new input arrives. Like, if you asked Knuth for the theoretically optimal design for this parser, I'm pretty sure this is what he'd tell you to use, and it's what people do when writing high-performance HTTP parsers in C. But unfortunately, in reality, re *doesn't* support this kind of pause/resume functionality, and you can't write efficient character-by-character algorithms in Python, so you have to use really awkward hacks instead. For the HTTP header case, the best I've been able to come up with is to manually analyze the regex to figure out the maximum size string it could match (in this case, 4 bytes), and then write a loop that tracks how long the string was before the last time we appended new data, and on each iteration searches the substring receive_buffer[old_length - 4 + 1:]. This is super finicky, and especially annoying if you want to offer this as a generic API for using regexes to deconstruct network streams. (There are a lot of Python network libraries that have accidentally-quadratic parsers in them.) In practice I suspect retrofitting this functionality into 're' would be a lot of work. But it's definitely frustrating that we have 90% of the machinery we'd need to do things the natural/efficient way, but then are thwarted by this arbitrary API limitation. -n -- Nathaniel J. Smith -- https://vorpus.org From rosuav at gmail.com Sat Oct 6 17:04:20 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 7 Oct 2018 08:04:20 +1100 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: On Sun, Oct 7, 2018 at 8:01 AM Nathaniel Smith wrote: > > On Sat, Oct 6, 2018 at 12:22 AM, Ram Rachum wrote: > > I'd like to use the re module to parse a long text file, 1GB in size. I wish > > that the re module could parse a stream, so I wouldn't have to load the > > whole thing into memory. I'd like to iterate over matches from the stream > > without keeping the old matches and input in RAM. > > > > What do you think? > > This has frustrated me too. > > The case where I've encountered this is parsing HTTP/1.1. We have data > coming in incrementally over the network, and we want to find the end > of the headers. To do this, we're looking for the first occurrence of > b"\r\n\r\n" OR b"\n\n". > > So our requirements are: > > 1. Search a bytearray for the regex b"\r\n\r\n|\n\n" > 2. If there's no match yet, wait for more data to arrive and try again > 3. When more data arrives, start searching again *where the last > search left off* > > The last requirement is subtle, but important. The naive approach > would be to rescan your whole receive buffer after each new packet > arrives: > > end_of_headers = re.compile(b"\r\n\r\n|\n\n") > while True: > m = end_of_headers.search(receive_buffer) > if m is None: > receive_buffer += await get_more_data_from_network() > # loop around and try again > else: > break > > But the code above is quadratic! If the headers are N bytes long, then > on each pass through the loop we perform an O(N) regex search, and we > do O(N) passes through the loop, so the whole thing is O(N**2). That > means your HTTP client-or-server can be trivially DoSed by a peer who > sends their headers broken into lots of small fragments. Quadratic in the size of the headers only, so you could just cap it - if the receive buffer is too large, just reject it. Sometimes, the simplest approach is the best. ChrisA From rosuav at gmail.com Sat Oct 6 18:56:59 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 7 Oct 2018 09:56:59 +1100 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: On Sun, Oct 7, 2018 at 9:54 AM Nathaniel Smith wrote: > > On Sat, Oct 6, 2018 at 2:04 PM, Chris Angelico wrote: > > On Sun, Oct 7, 2018 at 8:01 AM Nathaniel Smith wrote: > >> > >> On Sat, Oct 6, 2018 at 12:22 AM, Ram Rachum wrote: > >> > I'd like to use the re module to parse a long text file, 1GB in size. I wish > >> > that the re module could parse a stream, so I wouldn't have to load the > >> > whole thing into memory. I'd like to iterate over matches from the stream > >> > without keeping the old matches and input in RAM. > >> > > >> > What do you think? > >> > >> This has frustrated me too. > >> > >> The case where I've encountered this is parsing HTTP/1.1. We have data > >> coming in incrementally over the network, and we want to find the end > >> of the headers. To do this, we're looking for the first occurrence of > >> b"\r\n\r\n" OR b"\n\n". > >> > >> So our requirements are: > >> > >> 1. Search a bytearray for the regex b"\r\n\r\n|\n\n" > >> 2. If there's no match yet, wait for more data to arrive and try again > >> 3. When more data arrives, start searching again *where the last > >> search left off* > >> > >> The last requirement is subtle, but important. The naive approach > >> would be to rescan your whole receive buffer after each new packet > >> arrives: > >> > >> end_of_headers = re.compile(b"\r\n\r\n|\n\n") > >> while True: > >> m = end_of_headers.search(receive_buffer) > >> if m is None: > >> receive_buffer += await get_more_data_from_network() > >> # loop around and try again > >> else: > >> break > >> > >> But the code above is quadratic! If the headers are N bytes long, then > >> on each pass through the loop we perform an O(N) regex search, and we > >> do O(N) passes through the loop, so the whole thing is O(N**2). That > >> means your HTTP client-or-server can be trivially DoSed by a peer who > >> sends their headers broken into lots of small fragments. > > > > Quadratic in the size of the headers only, so you could just cap it - > > if the receive buffer is too large, just reject it. Sometimes, the > > simplest approach is the best. > > But OTOH, every problem has a solution that's simple, obvious, and wrong :-). > > Of course you set a cap on the header size, to prevent other kinds of > DoS (e.g. memory exhaustion). But it turns out people stuff a lot of > data into HTTP headers [1], so if the cap is large enough to support > non-malicious usage, then it's also large enough to let people DoS the > naive O(N**2) algorithm. Production-quality HTTP/1.1 parsers really do > have to use an O(N) algorithm here. Fair enough! You could instead use a simple linear parser rather than searching for end of headers first. That way, all you're looking for is a \n, no regex needed. Sometimes the VERY simplest solution doesn't work, but I still do like to keep things as simple as possible :) ChrisA From njs at pobox.com Sat Oct 6 18:54:30 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 6 Oct 2018 15:54:30 -0700 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: On Sat, Oct 6, 2018 at 2:04 PM, Chris Angelico wrote: > On Sun, Oct 7, 2018 at 8:01 AM Nathaniel Smith wrote: >> >> On Sat, Oct 6, 2018 at 12:22 AM, Ram Rachum wrote: >> > I'd like to use the re module to parse a long text file, 1GB in size. I wish >> > that the re module could parse a stream, so I wouldn't have to load the >> > whole thing into memory. I'd like to iterate over matches from the stream >> > without keeping the old matches and input in RAM. >> > >> > What do you think? >> >> This has frustrated me too. >> >> The case where I've encountered this is parsing HTTP/1.1. We have data >> coming in incrementally over the network, and we want to find the end >> of the headers. To do this, we're looking for the first occurrence of >> b"\r\n\r\n" OR b"\n\n". >> >> So our requirements are: >> >> 1. Search a bytearray for the regex b"\r\n\r\n|\n\n" >> 2. If there's no match yet, wait for more data to arrive and try again >> 3. When more data arrives, start searching again *where the last >> search left off* >> >> The last requirement is subtle, but important. The naive approach >> would be to rescan your whole receive buffer after each new packet >> arrives: >> >> end_of_headers = re.compile(b"\r\n\r\n|\n\n") >> while True: >> m = end_of_headers.search(receive_buffer) >> if m is None: >> receive_buffer += await get_more_data_from_network() >> # loop around and try again >> else: >> break >> >> But the code above is quadratic! If the headers are N bytes long, then >> on each pass through the loop we perform an O(N) regex search, and we >> do O(N) passes through the loop, so the whole thing is O(N**2). That >> means your HTTP client-or-server can be trivially DoSed by a peer who >> sends their headers broken into lots of small fragments. > > Quadratic in the size of the headers only, so you could just cap it - > if the receive buffer is too large, just reject it. Sometimes, the > simplest approach is the best. But OTOH, every problem has a solution that's simple, obvious, and wrong :-). Of course you set a cap on the header size, to prevent other kinds of DoS (e.g. memory exhaustion). But it turns out people stuff a lot of data into HTTP headers [1], so if the cap is large enough to support non-malicious usage, then it's also large enough to let people DoS the naive O(N**2) algorithm. Production-quality HTTP/1.1 parsers really do have to use an O(N) algorithm here. And similarly, if you're building a generic helper library for people implementing arbitrary unknown protocols, then you can't assume their protocols were designed to use small frames only, to avoid hitting arbitrary limitations in Python's re module. -n [1] E.g., 16 KiB total header size is already enough that on my laptop, the naive O(N**2) algorithm takes ~750 ms CPU time, versus ~16 ms for the O(N) algorithm. HTTP/2 tried hard to simplify their header encoding scheme by putting limits on header size, but got so much push-back that they were eventually forced to add special hacks to allow for arbitrarily large headers ? in particular, it turns out that people use *individual cookies* that are larger than 16 KiB: https://http2.github.io/faq/#why-the-rules-around-continuation-on-headers-frames -- Nathaniel J. Smith -- https://vorpus.org From steve at pearwood.info Sat Oct 6 21:39:48 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 7 Oct 2018 12:39:48 +1100 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: <20181007013948.GA3817@ando.pearwood.info> On Sat, Oct 06, 2018 at 02:00:27PM -0700, Nathaniel Smith wrote: > Fortunately, there's an elegant and natural solution: Just save the > regex engine's internal state when it hits the end of the string, and > then when more data arrives, use the saved state to pick up the search > where we left off. Theoretically, any regex engine *could* support > this ? it's especially obvious for DFA-based matchers, but even > backtrackers like Python's re could support it, basically by making > the matching engine a coroutine that can suspend itself when it hits > the end of the input, then resume it when new input arrives. Like, if > you asked Knuth for the theoretically optimal design for this parser, > I'm pretty sure this is what he'd tell you to use, and it's what > people do when writing high-performance HTTP parsers in C. The message I take from this is: - regex engines certainly can be written to support streaming data; - but few of them are; - and it is exceedingly unlikely to be able to easily (or at all) retro-fit that support to Python's existing re module. Perhaps the solution is a lightweight streaming DFA regex parser? Does anyone know whether MRAB's regex library supports this? https://pypi.org/project/regex/ > you can't write efficient > character-by-character algorithms in Python I'm sure that Python will never be as efficient as C in that regard (although PyPy might argue the point) but is there something we can do to ameliorate this? If we could make char-by-char processing only 10 times less efficient than C instead of 100 times (let's say...) perhaps that would help Ram (and you?) with your use-cases? -- Steve From ram at rachum.com Sun Oct 7 00:30:32 2018 From: ram at rachum.com (Ram Rachum) Date: Sun, 7 Oct 2018 07:30:32 +0300 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <74106724-00c6-baa1-ac69-464ceeaf0bc4@nedbatchelder.com> References: <74106724-00c6-baa1-ac69-464ceeaf0bc4@nedbatchelder.com> Message-ID: Hi Ned! I'm happy to see you here. I'm doing multi-color 3d-printing. The slicing software generates a GCode file, which is a text file of instructions for the printer, each command meaning something like "move the head to coordinates x,y,z while extruding plastic at a rate of w" and lots of other administrative commands. (Turn the print fan on, heat the extruder to a certain temperature, calibrate the printer, etc.) Here's an example of a simple GCode from a print I did last week: https://www.dropbox.com/s/kzmm6v8ilcn0aik/JPL%20Go%20hook.gcode?dl=0 It's 1.8MB in size. They could get to 1GB for complex prints. Multi-color prints means that at some points in the print, usually in a layer change, I'm changing the color. This means I need to insert an M600 command, which tells the printer to pause the print, move the head around, and give me a prompt to change the filament before continuing printing. I'm sick of injecting the M600 manually after every print. I've been doing that for the last year. I'm working on a script so I could say "Insert an M600 command after layers 5, 88 and 234, and also before process Foo." The slicer inserts comments saying "; layer 234" Or "; process Foo". I want to identify the entire layer as one match. That's because I want to find the layer and possibly process at the start, I want to find the last retraction command, the first extrusion command in the new layer, etc. So it's a regex that spans potentially thousands of lines. Then I'll know just where to put my M600 and how much retraction to do afterwards. Thanks, Ram. On Sat, Oct 6, 2018 at 6:58 PM Ned Batchelder wrote: > On 10/6/18 7:25 AM, Ram Rachum wrote: > > "This is a regular expression problem, rather than a Python problem." > > Do you have evidence for this assertion, except that other regex > implementations have this limitation? Is there a regex specification > somewhere that specifies that streams aren't supported? Is there a > fundamental reason that streams aren't supported? > > > "Can the lexing be done on a line-by-line basis?" > > For my use case, it unfortunately can't. > > > You mentioned earlier that your use case doesn't have to worry about the > "a.*b" problem. Can you tell us more about your scenario? How would the > stream know it had read enough to match or not match? Perhaps that same > logic can be used to feed the data in chunks? > > --Ned. > > > On Sat, Oct 6, 2018 at 1:53 PM Jonathan Fine wrote: > >> Hi Ram >> >> You wrote: >> >> > I'd like to use the re module to parse a long text file, 1GB in size. I >> > wish that the re module could parse a stream, so I wouldn't have to load >> > the whole thing into memory. I'd like to iterate over matches from the >> > stream without keeping the old matches and input in RAM. >> >> This is a regular expression problem, rather than a Python problem. A >> search for >> regular expression large file >> brings up some URLs that might help you, starting with >> >> https://stackoverflow.com/questions/23773669/grep-pattern-match-between-very-large-files-is-way-too-slow >> >> This might also be helpful >> https://svn.boost.org/trac10/ticket/11776 >> >> What will work for your problem depends on the nature of the problem >> you have. The simplest thing that might work is to iterate of the file >> line-by-line, and use a regular expression to extract matches from >> each line. >> >> In other words, something like (not tested) >> >> def helper(lines): >> for line in lines: >> yield from re.finditer(pattern, line) >> >> lines = open('my-big-file.txt') >> for match in helper(lines): >> # Do your stuff here >> >> Parsing is not the same as lexing, see >> https://en.wikipedia.org/wiki/Lexical_analysis >> >> I suggest you use regular expressions ONLY for the lexing phase. If >> you'd like further help, perhaps first ask yourself this. Can the >> lexing be done on a line-by-line basis? And if not, why not? >> >> If line-by-line not possible, then you'll have to modify the helper. >> At the end of each line, they'll be a residue / remainder, which >> you'll have to bring into the next line. In other words, the helper >> will have to record (and change) the state that exists at the end of >> each line. A bit like the 'carry' that is used when doing long >> addition. >> >> I hope this helps. >> >> -- >> Jonathan >> >> > > _______________________________________________ > Python-ideas mailing listPython-ideas at python.orghttps://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at rachum.com Sun Oct 7 00:32:01 2018 From: ram at rachum.com (Ram Rachum) Date: Sun, 7 Oct 2018 07:32:01 +0300 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <20181007013948.GA3817@ando.pearwood.info> References: <20181007013948.GA3817@ando.pearwood.info> Message-ID: On Sun, Oct 7, 2018 at 4:40 AM Steven D'Aprano wrote: > I'm sure that Python will never be as efficient as C in that regard > (although PyPy might argue the point) but is there something we can do > to ameliorate this? If we could make char-by-char processing only 10 > times less efficient than C instead of 100 times (let's say...) perhaps > that would help Ram (and you?) with your use-cases? > > Does that mean I'll have to write that character-by-character algorithm? I could already do that now I guess, the speed doesn't matter for my use case, but I'm trying to avoid writing an algorithm. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at cskk.id.au Sun Oct 7 02:58:46 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 7 Oct 2018 17:58:46 +1100 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: <20181007065846.GA57216@cskk.homeip.net> On 07Oct2018 07:30, Ram Rachum wrote: >I'm doing multi-color 3d-printing. The slicing software generates a GCode >file, which is a text file of instructions for the printer, each command >meaning something like "move the head to coordinates x,y,z while extruding >plastic at a rate of w" and lots of other administrative commands. (Turn >the print fan on, heat the extruder to a certain temperature, calibrate the >printer, etc.) > >Here's an example of a simple GCode from a print I did last week: >https://www.dropbox.com/s/kzmm6v8ilcn0aik/JPL%20Go%20hook.gcode?dl=0 > >It's 1.8MB in size. They could get to 1GB for complex prints. > >Multi-color prints means that at some points in the print, usually in a >layer change, I'm changing the color. This means I need to insert an M600 >command, which tells the printer to pause the print, move the head around, >and give me a prompt to change the filament before continuing printing. > >I'm sick of injecting the M600 manually after every print. I've been doing >that for the last year. I'm working on a script so I could say "Insert an >M600 command after layers 5, 88 and 234, and also before process Foo." > >The slicer inserts comments saying "; layer 234" Or "; process Foo". I want >to identify the entire layer as one match. That's because I want to find >the layer and possibly process at the start, I want to find the last >retraction command, the first extrusion command in the new layer, etc. So >it's a regex that spans potentially thousands of lines. > >Then I'll know just where to put my M600 and how much retraction to do >afterwards. Aha. Yeah, don't use a regexp for "the whole layer". I've fetched your file, and it is one instruction or comment per line. This is _easy_ to parse. Consider this totally untested sketch: layer_re = re.compile('^; layer (\d+), Z = (.*)') with open("JPL.gcode") as gcode: current_layer = None for lineno, line in enumerate(gcode, 1): m = layer_re.match(line) if m: # new layer new_layer = int(m.group(1)) new_z = float(m.group(2)) if current_layer is not None: # process the saved previous layer .......... current_layer = new_layer accrued_layer = [] if current_layer is not None: # we're saving lines for later work accrued_layer.append(line) continue # otherwise, copy the line straight out sys.stdout.write(line) The idea is that you scan the data on a per-line basis, adjusting some state variables as you see important lines. If you're "saving" a chunk of lines such as the instructions in a layer (in the above code: "current_layer is not None") you can stuff just those lines into a list for use when complete. On changes of state you deal with what you may have saved, etc. But just looking at your examples, you may not need to save anything; just insert or append lines during the copy. Example: with open("JPL.gcode") as gcode: for lineno, line in enumerate(gcode, 1): # pre line actions if line.startswith('; process '): print("M600 instruction...") # copy out the line sys.stdout.write(line) # post line actions if ... So you don't need to apply a regexp to a huge chunk of file. Process the file on an instruction basis and insert/append your extra instructions as you see the boundaries of the code you're after. A minor note. This incantation: for lineno, line in enumerate(gcode, 1): is to make it easy to print error message which recite the file line number to aid debugging. If you don't need that you'd just run with: for line in gcode: Cheers, Cameron Simpson From cs at cskk.id.au Sun Oct 7 02:59:43 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 7 Oct 2018 17:59:43 +1100 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: <20181007065943.GA8777@cskk.homeip.net> On 07Oct2018 07:32, Ram Rachum wrote: >On Sun, Oct 7, 2018 at 4:40 AM Steven D'Aprano wrote: >> I'm sure that Python will never be as efficient as C in that regard >> (although PyPy might argue the point) but is there something we can do >> to ameliorate this? If we could make char-by-char processing only 10 >> times less efficient than C instead of 100 times (let's say...) perhaps >> that would help Ram (and you?) with your use-cases? >> >Does that mean I'll have to write that character-by-character algorithm? I >could already do that now I guess, the speed doesn't matter for my use >case, but I'm trying to avoid writing an algorithm. No. You can write a line by line loop. See my other post. Cheers, Cameron Simpson From marko.ristin at gmail.com Sun Oct 7 03:25:13 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Sun, 7 Oct 2018 09:25:13 +0200 Subject: [Python-ideas] Dynamic getting of __doc__ of a function Message-ID: Hi, I'm working on decorators that have a dynamic __doc__ property computed at runtime and not at decoration time. The decorator must return the wrapper as a function and can not return a callable object with __call__ since the "self" argument would not be properly passed through the decorator. (Making __call__ a static method drops implicitly the first argument). If __call__ could be a static method and get the arguments as-are, I could make __doc__ a property. But this is not possible, right? Assigning a class satisfying the property protocol to the function wrapper.__doc__ did not work either since the property object is returned as-is instead of invoking __get__ on it. Could someone explain a bit -- is this the desired behavior or a bug or an underspecification? Is there any way for a callable object to be used as a decorator? Thanks! Cheers Marko -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Oct 7 04:31:01 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 7 Oct 2018 19:31:01 +1100 Subject: [Python-ideas] Dynamic getting of __doc__ of a function In-Reply-To: References: Message-ID: <20181007083101.GD3817@ando.pearwood.info> On Sun, Oct 07, 2018 at 09:25:13AM +0200, Marko Ristin-Kaufmann wrote: > Hi, > I'm working on decorators that have a dynamic __doc__ property computed at > runtime and not at decoration time. You mean like this? py> class X: ... @property ... def __doc__(self): ... return "NOBODY expects the %s Inquisition!" % self.nationality ... def __init__(self, nationality): ... self.nationality = nationality ... py> a = X("Spanish") py> b = X("Belgian") py> a.__doc__ 'NOBODY expects the Spanish Inquisition!' py> b.__doc__ 'NOBODY expects the Belgian Inquisition!' The only downside is that help(a) doesn't do the right thing. I believe that counts as a bug in help(). > The decorator must return the wrapper as a function and can not return a > callable object with __call__ since the "self" argument would not be > properly passed through the decorator. Sorry, I don't get this. Can you demonstrate? Code speaks much louder than words. -- Steve From amjadhedhili at outlook.com Sun Oct 7 06:16:08 2018 From: amjadhedhili at outlook.com (Amjad Ben Hedhili) Date: Sun, 7 Oct 2018 10:16:08 +0000 Subject: [Python-ideas] add a time decorator to timeit.py Message-ID: I think that a time decorator will be a useful addition to the sandard library, as i find the current way of measuring execution time a bit tedious: timeit.timeit("fun_to_time(a, b)", setup="from __main__ import a, b", number=1) compared to: @timef def fun_to_time(a, b): ... or timef(print)("Hello world!"). I already made a basic implementation of it, and it's working well. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Oct 7 07:15:27 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 7 Oct 2018 22:15:27 +1100 Subject: [Python-ideas] add a time decorator to timeit.py In-Reply-To: References: Message-ID: <20181007111527.GE3817@ando.pearwood.info> On Sun, Oct 07, 2018 at 10:16:08AM +0000, Amjad Ben Hedhili wrote: > I think that a time decorator will be a useful addition to the sandard > library, as i find the current way of measuring execution time a bit > tedious: > > timeit.timeit("fun_to_time(a, b)", setup="from __main__ import a, b", number=1) There are lots of ways to call timeit, but generally if you are calling it with number=1 then your results may not be trustworthy. (That depends on the function, of course.) There's a reason why timeit defaults to number=1000000 rather than 1. > compared to: > > @timef > def fun_to_time(a, b): > ... The problem with that as an API is that once you have decorated the function to call timeit, how do you use it *without* calling timeit? If you use functools.wraps, there will be a func_to_time.__wrapped__ but I don't think that calling that directly is a good interface. > or > > timef(print)("Hello world!"). > > I already made a basic implementation of it, and it's working well. Can you explain what the timef decorator does? What arguments does it take, how do we use it? -- Steve From amjadhedhili at outlook.com Sun Oct 7 07:43:40 2018 From: amjadhedhili at outlook.com (Amjad Ben Hedhili) Date: Sun, 7 Oct 2018 11:43:40 +0000 Subject: [Python-ideas] add a time decorator to timeit.py In-Reply-To: <20181007111527.GE3817@ando.pearwood.info> References: , <20181007111527.GE3817@ando.pearwood.info> Message-ID: this is my implementation: ``` units = {"auto": -1, "s": 1, "ms": 1e3, "?s": 1e6, "ns": 1e9} def timef(unit="auto", number=1, garcol=True, get_time=False): """A decorator to measure the execution time of a function Returns the return of the tested function if get_time is False or the execution time in unit if not unit: one of "ns" (nanoseconds), "?s" (microseconds), "ms" (milliseconds), "s" (seconds) and "auto": (automatic) (default: "auto") garcol: if True enables garbage collection (default: True) get_time: if true return the execution time of the function (default: False) """ fun = False if callable(unit): fun = True func = unit unit = "auto" elif unit not in units: raise ValueError( "valid options: " "s(seconds), " "ms(milliseconds), " "?s(microseconds), " "ns(nanoseconds) " "and auto(automatic: default)") def decorating_function(func): return _wrapper(func, number, units, unit, units[unit], garcol, get_time) return decorating_function(func) if fun else decorating_function def _wrapper(func, number, units, unit, factor, garcol, get_time): def timed(*args, **kwargs): nonlocal unit, factor gcold = gc.isenabled() # disable garbage collection on demand if not garcol: gc.disable() try: if number > 0: start = default_timer() for _ in range(number): func(*args, **kwargs) time = default_timer() - start result = func(*args, **kwargs) finally: if gcold: gc.enable() if unit == "auto": for u, f in units.items(): if 1 <= int(time * f) < 1000: unit, factor = u, f break else: unit, factor = "s", 1 time *= factor print(f"{func.__qualname__}: {time:.4f} {unit}") return time if get_time else result return timed ``` it's not perfect but it works; you can control number of repetitions, garbage collection, unit ... and it produces a formatted output such as: `fun_to_time: 24.1056 ms`. ________________________________ De : Python-ideas de la part de Steven D'Aprano Envoy? : dimanche 7 octobre 2018 04:15 ? : python-ideas at python.org Objet : Re: [Python-ideas] add a time decorator to timeit.py On Sun, Oct 07, 2018 at 10:16:08AM +0000, Amjad Ben Hedhili wrote: > I think that a time decorator will be a useful addition to the sandard > library, as i find the current way of measuring execution time a bit > tedious: > > timeit.timeit("fun_to_time(a, b)", setup="from __main__ import a, b", number=1) There are lots of ways to call timeit, but generally if you are calling it with number=1 then your results may not be trustworthy. (That depends on the function, of course.) There's a reason why timeit defaults to number=1000000 rather than 1. > compared to: > > @timef > def fun_to_time(a, b): > ... The problem with that as an API is that once you have decorated the function to call timeit, how do you use it *without* calling timeit? If you use functools.wraps, there will be a func_to_time.__wrapped__ but I don't think that calling that directly is a good interface. > or > > timef(print)("Hello world!"). > > I already made a basic implementation of it, and it's working well. Can you explain what the timef decorator does? What arguments does it take, how do we use it? -- Steve _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Oct 7 08:15:29 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 7 Oct 2018 05:15:29 -0700 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <20181007013948.GA3817@ando.pearwood.info> References: <20181007013948.GA3817@ando.pearwood.info> Message-ID: On Sat, Oct 6, 2018, 18:40 Steven D'Aprano wrote: > The message I take from this is: > > - regex engines certainly can be written to support streaming data; > - but few of them are; > - and it is exceedingly unlikely to be able to easily (or at all) > retro-fit that support to Python's existing re module. > I don't know enough about the re module internals to make an informed guess about the difficulty. On a quick glance, it does seem to store most intermediate match state in explicit structs rather than on the call stack... -n > -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Sun Oct 7 08:31:21 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sun, 7 Oct 2018 21:31:21 +0900 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> <5FE3F758-2D89-4682-88FC-D266AFEF91E1@killingar.net> <9cdae6fe-06b7-4d22-7141-f0bc081f63d9@kynesim.co.uk> <22F1CC9D-CC3E-4602-8D14-99A9A852570E@killingar.net> Message-ID: <23481.64665.177894.364111@turnbull.sk.tsukuba.ac.jp> Jonathan Fine writes: > I'm thinking of providing a builtin debug() command "Command" doesn't make much sense in this context. I presume you mean "function". > that 'does the right thing' according to the context. And the > context would include the user's preferences. Which might be to use > the print command in your devtools package. "Do what I mean" is inappropriate for a built-in because you have no idea who might be calling it. I can see an API (say, that of class Pdb ;-) and various implementations (starting with Pdb and finally evolving as Jonathan Fine's Magic Mind-Reading Debugging Environment). Wait, wut???!!! Look here: > I suggest the builtin debug() would determine the API, and provide a > reference implementation. And that many users (perhaps through their > IDE) would choose a different implementation. The necessary builtin for this is already available. Its name is "breakpoint". From turnbull.stephen.fw at u.tsukuba.ac.jp Sun Oct 7 08:32:14 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sun, 7 Oct 2018 21:32:14 +0900 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <2EBB19B1-2E39-48A0-AA3D-30203D8A5BDB@killingar.net> <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> <5FE3F758-2D89-4682-88FC-D266AFEF91E1@killingar.net> <9cdae6fe-06b7-4d22-7141-f0bc081f63d9@kynesim.co.uk> <22F1CC9D-CC3E-4602-8D14-99A9A852570E@killingar.net> Message-ID: <23481.64718.880329.910243@turnbull.sk.tsukuba.ac.jp> Samuel Colvin writes: > Python definitely needs a dedicated debug print command. For *commands* (ie, a REPL), you have Pdb already. It's unlikely a *statement* would be accepted, given that print was demoted from a statement to a function. And it's not obvious why a "debug.print()" function can't just as well be imported from a 3rd party library you get from PyPI. (There's already pdb.Pdb.prettyprint in the stdlib, though it may not be terribly easy to use outside of the Pdb REPL.) > (I'm not trying to promote devtools, just demonstrate what is > possible.) It seems to me that promoting devtools (unless you know a solution that's better for some reason) is exactly what you should do. Not that you should pretend it's already optimal. But it would be a concrete proposal that would serve to focus suggestions, and evaluate whether there is a single module that is sufficiently useful to enough people to deserve inclusion in the stdlib, or whether this is yet another case where a number of modules with varied feature sets available from PyPI is a better solution. > It would be difficult to fully integrate a package like this into > the standard lib since it relies on pygments for colouring output, > but not impossible. I don't see why that's a problem, as long as the coloring feature is optional: have_color = None try: import pygments have_color = 'pygments' except ImportError: pass # or try other color libraries here and you don't even try to colorize if have_color is None. See the thread on xz and other optional external libraries that don't exist on some platforms and don't give nice errors when the import fails for the wrapper Python library. A bigger problem is if devtools does stuff on which there is no consensus that it's wanted in the stdlib. That "stuff" would need to be split out, which would possibly be a pain for you and any other existing users. From steve at pearwood.info Sun Oct 7 08:34:10 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 7 Oct 2018 23:34:10 +1100 Subject: [Python-ideas] add a time decorator to timeit.py In-Reply-To: References: <20181007111527.GE3817@ando.pearwood.info> Message-ID: <20181007123410.GF3817@ando.pearwood.info> On Sun, Oct 07, 2018 at 11:43:40AM +0000, Amjad Ben Hedhili wrote: > this is my implementation: I didn't ask about the current implementation, as that might change. I asked about the interface: what it does, not how it does it. For the purposes of Python-Ideas, we want to discuss the feature, not the details of how it works, why it is better than other existing solutions, and whether or not it belongs in the std lib. You're selling the *idea*, not the *implementation* (although of course you can use the implementation to demonstrate the idea). Unfortunately, not every good idea can or will make it into the std lib. Even if the idea is good, we might decide that it belongs as a third- party library, or that the benefit is not enough to make up for the cost in effort or complexity. Or the idea is simply too controversial. -- Steve From rosuav at gmail.com Sun Oct 7 08:44:20 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 7 Oct 2018 23:44:20 +1100 Subject: [Python-ideas] add a time decorator to timeit.py In-Reply-To: <20181007123410.GF3817@ando.pearwood.info> References: <20181007111527.GE3817@ando.pearwood.info> <20181007123410.GF3817@ando.pearwood.info> Message-ID: On Sun, Oct 7, 2018 at 11:34 PM Steven D'Aprano wrote: > Unfortunately, not every good idea can or will make it into the std lib. > Even if the idea is good, we might decide that it belongs as a third- > party library, or that the benefit is not enough to make up for the cost > in effort or complexity. Or the idea is simply too controversial. Or just too detaily. The given function has four keyword arguments that vary its behaviour slightly; if it gets into the stdlib, someone will ask for a way to control the output format, someone else will request that it be able to send to stderr instead of stdout, etc, etc, and suddenly the proposed function has two dozen kwargs. One of the powerful tools available to a programmer is having *your own* debug library, without having to worry about what other people would do that might be similar. -0.5 on adding to the stdlib, -0 on putting on PyPI (I wouldn't bother looking on PyPI for something like this - I'd just write my own, specific to the need). ChrisA From ram at rachum.com Sun Oct 7 09:11:57 2018 From: ram at rachum.com (Ram Rachum) Date: Sun, 7 Oct 2018 16:11:57 +0300 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <20181007065846.GA57216@cskk.homeip.net> References: <20181007065846.GA57216@cskk.homeip.net> Message-ID: Hi Cameron, Thanks for putting in the time to study my problem and sketch a solution. Unfortunately, it's not helpful. I was developing a solution similar to yours before I came to the conclusion that a multilne regex would be more elegant. I find this algorithm to be quite complicated. It's basically a poor man's regex engine. I'm more likely to use a shim to make the re package work on streams (like regexy or reading chunks until I get a match) than to use an algorithm like that. Thanks, Ram. On Sun, Oct 7, 2018 at 9:58 AM Cameron Simpson wrote: > On 07Oct2018 07:30, Ram Rachum wrote: > >I'm doing multi-color 3d-printing. The slicing software generates a GCode > >file, which is a text file of instructions for the printer, each command > >meaning something like "move the head to coordinates x,y,z while extruding > >plastic at a rate of w" and lots of other administrative commands. (Turn > >the print fan on, heat the extruder to a certain temperature, calibrate > the > >printer, etc.) > > > >Here's an example of a simple GCode from a print I did last week: > >https://www.dropbox.com/s/kzmm6v8ilcn0aik/JPL%20Go%20hook.gcode?dl=0 > > > >It's 1.8MB in size. They could get to 1GB for complex prints. > > > >Multi-color prints means that at some points in the print, usually in a > >layer change, I'm changing the color. This means I need to insert an M600 > >command, which tells the printer to pause the print, move the head around, > >and give me a prompt to change the filament before continuing printing. > > > >I'm sick of injecting the M600 manually after every print. I've been doing > >that for the last year. I'm working on a script so I could say "Insert an > >M600 command after layers 5, 88 and 234, and also before process Foo." > > > >The slicer inserts comments saying "; layer 234" Or "; process Foo". I > want > >to identify the entire layer as one match. That's because I want to find > >the layer and possibly process at the start, I want to find the last > >retraction command, the first extrusion command in the new layer, etc. So > >it's a regex that spans potentially thousands of lines. > > > >Then I'll know just where to put my M600 and how much retraction to do > >afterwards. > > Aha. > > Yeah, don't use a regexp for "the whole layer". I've fetched your file, > and it > is one instruction or comment per line. This is _easy_ to parse. Consider > this > totally untested sketch: > > layer_re = re.compile('^; layer (\d+), Z = (.*)') > with open("JPL.gcode") as gcode: > current_layer = None > for lineno, line in enumerate(gcode, 1): > m = layer_re.match(line) > if m: > # new layer > new_layer = int(m.group(1)) > new_z = float(m.group(2)) > if current_layer is not None: > # process the saved previous layer > .......... > current_layer = new_layer > accrued_layer = [] > if current_layer is not None: > # we're saving lines for later work > accrued_layer.append(line) > continue > # otherwise, copy the line straight out > sys.stdout.write(line) > > The idea is that you scan the data on a per-line basis, adjusting some > state > variables as you see important lines. If you're "saving" a chunk of lines > such > as the instructions in a layer (in the above code: "current_layer is not > None") > you can stuff just those lines into a list for use when complete. > > On changes of state you deal with what you may have saved, etc. > > But just looking at your examples, you may not need to save anything; just > insert or append lines during the copy. Example: > > with open("JPL.gcode") as gcode: > for lineno, line in enumerate(gcode, 1): > # pre line actions > if line.startswith('; process '): > print("M600 instruction...") > # copy out the line > sys.stdout.write(line) > # post line actions > if ... > > So you don't need to apply a regexp to a huge chunk of file. Process the > file > on an instruction basis and insert/append your extra instructions as you > see > the boundaries of the code you're after. > > A minor note. This incantation: > > for lineno, line in enumerate(gcode, 1): > > is to make it easy to print error message which recite the file line > number to > aid debugging. If you don't need that you'd just run with: > > for line in gcode: > > Cheers, > Cameron Simpson > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From 2015 at jmunch.dk Sun Oct 7 09:24:13 2018 From: 2015 at jmunch.dk (2015 at jmunch.dk) Date: Sun, 7 Oct 2018 15:24:13 +0200 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <20181007065846.GA57216@cskk.homeip.net> References: <20181007065846.GA57216@cskk.homeip.net> Message-ID: <855d8842-35e3-4032-6b70-eb601a0b2fc9@stofanet.dk> On 18-10-07 15.11, Ram Rachum wrote: > Unfortunately, it's not helpful. I was developing a solution similar to yours before I came to the conclusion that a multilne regex would be more elegant. How about memory mapping your 1GB file? bytes patterns work on memoryviews. regards, Anders From ram at rachum.com Sun Oct 7 10:15:48 2018 From: ram at rachum.com (Ram Rachum) Date: Sun, 7 Oct 2018 17:15:48 +0300 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <855d8842-35e3-4032-6b70-eb601a0b2fc9@stofanet.dk> References: <20181007065846.GA57216@cskk.homeip.net> <855d8842-35e3-4032-6b70-eb601a0b2fc9@stofanet.dk> Message-ID: I tested it now and indeed bytes patterns work on memoryview objects. But how do I use this to scan for patterns through a stream without loading it to memory? On Sun, Oct 7, 2018 at 4:24 PM <2015 at jmunch.dk> wrote: > On 18-10-07 15.11, Ram Rachum wrote: > > > Unfortunately, it's not helpful. I was developing a solution similar > to yours before I came to the conclusion that a multilne regex would be > more elegant. > > > How about memory mapping your 1GB file? > > bytes patterns work on memoryviews. > > regards, Anders > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko.ristin at gmail.com Sun Oct 7 10:40:17 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Sun, 7 Oct 2018 16:40:17 +0200 Subject: [Python-ideas] Dynamic getting of __doc__ of a function In-Reply-To: <20181007083101.GD3817@ando.pearwood.info> References: <20181007083101.GD3817@ando.pearwood.info> Message-ID: Hi Steve, Here are a couple of code snippets. I hope it's not too verbose :) Setting property on the __doc__ works on the instance, but not on the class: class A: @property def __doc__(self): return "Works only on the instance" a = A() print('A.__doc__ is {}'.format(A.__doc__)) # A.__doc__ is print('a.__doc__ is {}'.format(a.__doc__)) # a.__doc__ is Works only on the instance help(A) # Help on class A in module __main__: # # class A(builtins.object) # | Data descriptors defined here: # | # | __dict__ # | dictionary for instance variables (if defined) # | # | __weakref__ # | list of weak references to the object (if defined) However, you can fix this with a separate class that follows the property protocol: some_var = "oi" class DocProperty: def __get__(self, instance, owner): return "Dynamic class doc: {}".format(some_var) class A: """Static doc""" __doc__ = DocProperty() print(A.__doc__) # Dynamic class doc: oi some_var = "noi" print(A.__doc__) # Dynamic class doc: noi help(A) # Help on class A in module __main__: # # class A(builtins.object) # | Dynamic class doc: noi # | # | Data descriptors defined here: # | # | __dict__ # | dictionary for instance variables (if defined) # | # | __weakref__ # | list of weak references to the object (if defined) This worked well when I decorate a *class.* Consider now the case where we have to decorate a *function*. I could decorate with a callable class, but __doc__ now does not work and help() becomes misleading (the function signature is lost and the help says "Help on Wrapper"): import functools class DocProperty: def __init__(self, func): self.func = func def __get__(self, instance, owner): return "Func doc is: {!r} and now my part...".format(self.func.__doc__) def decorate(func): class Wrapper: __doc__ = DocProperty(func) def __call__(self, *args, **kwargs): print("Before the call") return func(*args, **kwargs) wrapper = Wrapper() functools.update_wrapper(wrapper=wrapper, wrapped=func) wrapper.__doc__ = DocProperty(func) return wrapper @decorate def some_func(x): """Original doc.""" print("Hi from some_func") print('some_func.__doc__ is {}'.format(some_func.__doc__)) # some_func.__doc__ is <__main__.DocProperty object at 0x7fd134a1d668> help(some_func) # Help on Wrapper in module __main__ object: # # some_func = class Wrapper(builtins.object) # | Func doc is: 'Original doc.' and now my part... # | # | Methods defined here: # | # | __call__(self, *args, **kwargs) # | # | ---------------------------------------------------------------------- # | Data descriptors defined here: # | # | __dict__ # | dictionary for instance variables (if defined) # | # | __weakref__ # | list of weak references to the object (if defined) I tried to decorate the function with a DocProperty, but then hit the wall. Both __doc__ is not "getted" nor does help() works: import functools class DocProperty: def __init__(self, func): self.func = func def __get__(self, instance, owner): return "Func doc is: {!r} and now my part...".format(self.func.__doc__) def decorate(func): def wrapper(*args, **kwargs): print("Before the call") return func(*args, **kwargs) functools.update_wrapper(wrapper=wrapper, wrapped=func) wrapper.__doc__ = DocProperty(func) return wrapper @decorate def some_func(x): """Original doc.""" print("Hi from some_func") some_func(x=3) # Before the call # Hi from some_func print('some_func.__doc__ is {}'.format(some_func.__doc__)) # some_func.__doc__ is <__main__.DocProperty object at 0x7f0551eea438> help(some_func) # Help on function some_func in module __main__: # # some_func(x) I just couldn't figure out how to make the __doc__ attribute of a function a getter. Is this a bug or am I making a mistake? If it's my mistake, is there any other way how to dynamically __doc__ a function or am I forced to set __doc__ of a function at the decoration time? (If you wonder about the use case: I'd like to dynamically generate the docstrings when functions are decorated with contracts from icontract library. Condition functions need to be parsed and re-formatted, so this is something that should be done on-demand, when the user either wants to see the help() or when the sphinx documentation is automatically generated. The docs should not inflict computational overhead during the decoration since normal course of program operation does not need pretty-printed contracts.) Thanks for any pointers! Please let me know if you'd like me to compress one or the other example or explain something in more detail. Cheers, Marko On Sun, 7 Oct 2018 at 11:07, Steven D'Aprano wrote: > On Sun, Oct 07, 2018 at 09:25:13AM +0200, Marko Ristin-Kaufmann wrote: > > Hi, > > I'm working on decorators that have a dynamic __doc__ property computed > at > > runtime and not at decoration time. > > You mean like this? > > py> class X: > ... @property > ... def __doc__(self): > ... return "NOBODY expects the %s Inquisition!" % self.nationality > ... def __init__(self, nationality): > ... self.nationality = nationality > ... > py> a = X("Spanish") > py> b = X("Belgian") > py> a.__doc__ > 'NOBODY expects the Spanish Inquisition!' > py> b.__doc__ > 'NOBODY expects the Belgian Inquisition!' > > > The only downside is that help(a) doesn't do the right thing. I believe > that counts as a bug in help(). > > > > > > The decorator must return the wrapper as a function and can not return a > > callable object with __call__ since the "self" argument would not be > > properly passed through the decorator. > > Sorry, I don't get this. Can you demonstrate? Code speaks much louder > than words. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Oct 7 10:45:01 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 8 Oct 2018 01:45:01 +1100 Subject: [Python-ideas] Dynamic getting of __doc__ of a function In-Reply-To: References: <20181007083101.GD3817@ando.pearwood.info> Message-ID: On Mon, Oct 8, 2018 at 1:41 AM Marko Ristin-Kaufmann wrote: > (If you wonder about the use case: I'd like to dynamically generate the docstrings when functions are decorated with contracts from icontract library. Condition functions need to be parsed and re-formatted, so this is something that should be done on-demand, when the user either wants to see the help() or when the sphinx documentation is automatically generated. The docs should not inflict computational overhead during the decoration since normal course of program operation does not need pretty-printed contracts.) > Have you tried just generating the docstrings at decoration time and applying them? By the sound of it, they won't change after that, and the only reason to run it later is performance... but have you actually measured a performance hit resulting from that? ChrisA From marko.ristin at gmail.com Sun Oct 7 10:56:14 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Sun, 7 Oct 2018 16:56:14 +0200 Subject: [Python-ideas] Dynamic getting of __doc__ of a function In-Reply-To: References: <20181007083101.GD3817@ando.pearwood.info> Message-ID: Hi Crhis, Have you tried just generating the docstrings at decoration time and > applying them? By the sound of it, they won't change after that, and > the only reason to run it later is performance... but have you > actually measured a performance hit resulting from that? I did. On my laptop (Intel i7-4700MQ), it takes about 4-12 milliseconds to parse and reformat a single condition lambda function of a contract. This quickly adds up to a couple of seconds if you have thousands of condition functions to parse. This might not be a problem for a limited code base, but I imagine that it could get inefficient very quickly as soon as the contract usage would spread to the dependencies as well. Not dead-slow inefficient, but inefficient enough to consider whether the automatic doc generation is worth it. Cheers, Marko On Sun, 7 Oct 2018 at 16:46, Chris Angelico wrote: > On Mon, Oct 8, 2018 at 1:41 AM Marko Ristin-Kaufmann > wrote: > > (If you wonder about the use case: I'd like to dynamically generate the > docstrings when functions are decorated with contracts from icontract > library. Condition functions need to be parsed and re-formatted, so this is > something that should be done on-demand, when the user either wants to see > the help() or when the sphinx documentation is automatically generated. The > docs should not inflict computational overhead during the decoration since > normal course of program operation does not need pretty-printed contracts.) > > > > Have you tried just generating the docstrings at decoration time and > applying them? By the sound of it, they won't change after that, and > the only reason to run it later is performance... but have you > actually measured a performance hit resulting from that? > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Oct 7 11:11:20 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 8 Oct 2018 02:11:20 +1100 Subject: [Python-ideas] Dynamic getting of __doc__ of a function In-Reply-To: References: <20181007083101.GD3817@ando.pearwood.info> Message-ID: On Mon, Oct 8, 2018 at 1:56 AM Marko Ristin-Kaufmann wrote: > > Hi Crhis, > >> Have you tried just generating the docstrings at decoration time and >> applying them? By the sound of it, they won't change after that, and >> the only reason to run it later is performance... but have you >> actually measured a performance hit resulting from that? > > > I did. On my laptop (Intel i7-4700MQ), it takes about 4-12 milliseconds to parse and reformat a single condition lambda function of a contract. This quickly adds up to a couple of seconds if you have thousands of condition functions to parse. This might not be a problem for a limited code base, but I imagine that it could get inefficient very quickly as soon as the contract usage would spread to the dependencies as well. Not dead-slow inefficient, but inefficient enough to consider whether the automatic doc generation is worth it. > Thanks. Having actual numbers is helpful. It may be worth looking into this as part of a broader lazy-generation feature, as has recently been rolled out to annotations, and is periodically discussed elsewhere. ChrisA From steve at pearwood.info Sun Oct 7 11:13:31 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 8 Oct 2018 02:13:31 +1100 Subject: [Python-ideas] Dynamic getting of __doc__ of a function In-Reply-To: References: <20181007083101.GD3817@ando.pearwood.info> Message-ID: <20181007151330.GG3817@ando.pearwood.info> On Mon, Oct 08, 2018 at 01:45:01AM +1100, Chris Angelico wrote: > On Mon, Oct 8, 2018 at 1:41 AM Marko Ristin-Kaufmann > wrote: > > (If you wonder about the use case: I'd like to dynamically generate the docstrings when functions are decorated with contracts from icontract library. Condition functions need to be parsed and re-formatted, so this is something that should be done on-demand, when the user either wants to see the help() or when the sphinx documentation is automatically generated. The docs should not inflict computational overhead during the decoration since normal course of program operation does not need pretty-printed contracts.) > > > > Have you tried just generating the docstrings at decoration time and > applying them? By the sound of it, they won't change after that, and > the only reason to run it later is performance... but have you > actually measured a performance hit resulting from that? That might work for Marko's use-case, but the problem is more general and I'd like to consider the broader picture. Currently instance __doc__ attributes are sometimes ignored by help(). Given these: py> class X: ... pass ... py> a = X() py> b = X() py> a.__doc__ = "Hello" py> b.__doc__ = "Goodbye" I would expect to see the per-instance docstrings, but in 3.6 at least, help(a) and help(b) both give the unhelpful: Help on X in module __main__ object: class X(builtins.object) | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) (END) As per my earlier post, making the docstrings a property fails from help(). So I think there's some improvement here: - fix help() so it picks up per-instance docstrings, not just the class docstring; - including the case where the docstring is a property. -- Steve From rosuav at gmail.com Sun Oct 7 11:26:06 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 8 Oct 2018 02:26:06 +1100 Subject: [Python-ideas] Dynamic getting of __doc__ of a function In-Reply-To: <20181007151330.GG3817@ando.pearwood.info> References: <20181007083101.GD3817@ando.pearwood.info> <20181007151330.GG3817@ando.pearwood.info> Message-ID: On Mon, Oct 8, 2018 at 2:14 AM Steven D'Aprano wrote: > That might work for Marko's use-case, but the problem is more general > and I'd like to consider the broader picture. Currently instance __doc__ > attributes are sometimes ignored by help(). Given these: > > py> class X: > ... pass > ... > py> a = X() > py> b = X() > py> a.__doc__ = "Hello" > py> b.__doc__ = "Goodbye" > > I would expect to see the per-instance docstrings, but in 3.6 at least, > help(a) and help(b) both give the unhelpful: > > > Help on X in module __main__ object: Each of them is actually giving help(X). So having that respect *any* per-instance information first means changing it so help(inst) is different from help(cls). > As per my earlier post, making the docstrings a property fails from > help(). So I think there's some improvement here: > > - fix help() so it picks up per-instance docstrings, not just the > class docstring; > > - including the case where the docstring is a property. Which are really the same thing - having help() care about the actual instance, not the type. Currently, it works if you put a property on the metaclass: >>> class mX(type): ... @property ... def __doc__(self): ... return "Docstring" ... >>> class X(metaclass=mX): pass ... >>> help(X()) class X(builtins.object) | Docstring | ... Maybe what we really need here is a convenient way to create class properties. class X: @classproperty def __doc__(cls): return "Docstring" I know this has been discussed before, though I can't find a discussion thread right now. ChrisA From jfine2358 at gmail.com Sun Oct 7 11:36:52 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Sun, 7 Oct 2018 16:36:52 +0100 Subject: [Python-ideas] Dynamic getting of __doc__ of a function In-Reply-To: References: <20181007083101.GD3817@ando.pearwood.info> Message-ID: Hi Marko You wrote: > I just couldn't figure out how to make the __doc__ attribute of a function a getter. Is this a bug or am I making a mistake? If it's my mistake, is there any other way how to dynamically __doc__ a function or am I forced to set __doc__ of a function at the decoration time? Thank you for your clear examples. I'm not sure, but I may have found the problem. As I recall, setting obj.attr to a property won't work. Instead, you have to set type(obj).attr to the property. And now you come up against a problem. You can't set __doc__ on the type 'function', nor can you subclass the type 'function'. >>> def fn(): pass >>> type(fn) >>> type(fn).__doc__ 'function(code, globals[, name[, argdefs[, closure]]]) [snip]' >>> type(fn).__doc__ = None TypeError: can't set attributes of built-in/extension type 'function' >>> class FN(type(fn)): pass TypeError: type 'function' is not an acceptable base type As I (more vaguely) recall, something was done recently to add a get attribute property that works directly on objects, rather than on instances of a class. But I've not been able to find it. Perhaps I misunderstood PEP 562. Some perhaps relevant URLs: https://www.python.org/dev/peps/pep-0224/ Attribute Docstrings https://www.python.org/dev/peps/pep-0549/ Instance Descriptors https://www.python.org/dev/peps/pep-0562/ Module __getattr__ and __dir__ https://stackoverflow.com/questions/2447353/getattr-on-a-module -- Jonathan From amjadhedhili at outlook.com Sun Oct 7 11:51:49 2018 From: amjadhedhili at outlook.com (Amjad Ben Hedhili) Date: Sun, 7 Oct 2018 15:51:49 +0000 Subject: [Python-ideas] add a time decorator to timeit.py In-Reply-To: References: <20181007111527.GE3817@ando.pearwood.info> <20181007123410.GF3817@ando.pearwood.info>, Message-ID: @Steven D'Aprano , I know you didn't ask for the implementation, but it does supports the idea and it contains comments describing how it's used. @Chris Angelico, I don't know if it needs to be further customizable (apart from a repeat keyword maybe) as it's meant to be similar to the timeit function, and it's not meant to be complicated + you can use this argument for basically any function in the standard library as it might not be customizable enough for someone's needs, I don't know if that's a big problem in the first place as there are functions in the standard library with lots of keywords (e.g the Popen constructor with 18 kw). I see that it deserves to be in the std library as the current way might not be suitable for many, i mean: `timeit.timeit("func_to_time(a, b)", setup="from __main__ import a, b") or `timeit.timeit("func_to_time(a, b)", globals=globals())` is not as: ``` timef func_to_time(a, b): ... ``` when you don't want it for now you simply comment the decorator. Also consider the case where you have a dozen of functions and methods to measure, it'll be tedious to go with timeit version. in addition other languages that compete with python have easy ways to measure times in their standard library that are more pythonic than python itself (e.g Julia with @timev, @timed, @elapsed macros, Kotlin with the measureTimeMillis and measureNanoTime functions ...) and the implementation is open for change, it can be made better. ________________________________ De : Python-ideas de la part de Chris Angelico Envoy? : dimanche 7 octobre 2018 05:44 ? : python-ideas Objet : Re: [Python-ideas] add a time decorator to timeit.py On Sun, Oct 7, 2018 at 11:34 PM Steven D'Aprano wrote: > Unfortunately, not every good idea can or will make it into the std lib. > Even if the idea is good, we might decide that it belongs as a third- > party library, or that the benefit is not enough to make up for the cost > in effort or complexity. Or the idea is simply too controversial. Or just too detaily. The given function has four keyword arguments that vary its behaviour slightly; if it gets into the stdlib, someone will ask for a way to control the output format, someone else will request that it be able to send to stderr instead of stdout, etc, etc, and suddenly the proposed function has two dozen kwargs. One of the powerful tools available to a programmer is having *your own* debug library, without having to worry about what other people would do that might be similar. -0.5 on adding to the stdlib, -0 on putting on PyPI (I wouldn't bother looking on PyPI for something like this - I'd just write my own, specific to the need). ChrisA _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfine2358 at gmail.com Sun Oct 7 12:15:03 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Sun, 7 Oct 2018 17:15:03 +0100 Subject: [Python-ideas] add a time decorator to timeit.py In-Reply-To: References: <20181007111527.GE3817@ando.pearwood.info> <20181007123410.GF3817@ando.pearwood.info> Message-ID: Summary: Python's timeit.timeit() has an undocumented feature / implementation detail that gives much of what the original poster has asked for. Perhaps revising the docs will solve the problem. This thread has prompted me to look at timeit again. Usually, I look at the command line help first. >>> import timeit >>> help(timeit) Classes: Timer Functions: timeit(string, string) -> float repeat(string, string) -> list default_timer() -> float This time, to my surprise, I found the following works: >>> def fn(): return 2 + 2 >>> timeit.timeit(fn) 0.10153918000287376 Until today, as I recall, I didn't know this. Now for: https://docs.python.org/3/library/timeit.html I don't see any examples there, that show that timeit.timeit can take a callable as its first argument. So my ignorance can, I hope be forgiven. Now for: https://github.com/python/cpython/blob/3.7/Lib/timeit.py#L100 This contains, for both the stmt and setup parameters, explicit tests such as if isinstance(stmt, str): # string case elif callable(stmt): # callable case So I think it's an undocumented feature, rather than an implementation detail. And if you're a software historian, now perhaps look at https://github.com/python/cpython/commits/3.7/Lib/timeit.py And also, if you wish, for the tests for timeit.py. -- Jonathan From marko.ristin at gmail.com Sun Oct 7 12:17:47 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Sun, 7 Oct 2018 18:17:47 +0200 Subject: [Python-ideas] Dynamic getting of __doc__ of a function In-Reply-To: References: <20181007083101.GD3817@ando.pearwood.info> Message-ID: Hi, @Jonathan: thanks! I'll have a look at your links. So there are actually two issues if I understand correctly: * help() ignoring the property getter when invoked on an instance * built-in class function ignoring the property getter (some_func.__doc__ set to a property returns the property instead of invoking the getter) Is the second issue also problematic or the built-ins are meant to ignore properties and there should be no fix? I think I'm misunderstanding something here. Cheers, Marko Le dim. 7 oct. 2018 ? 17:37, Jonathan Fine a ?crit : > Hi Marko > > You wrote: > > > I just couldn't figure out how to make the __doc__ attribute of a > function a getter. Is this a bug or am I making a mistake? If it's my > mistake, is there any other way how to dynamically __doc__ a function or am > I forced to set __doc__ of a function at the decoration time? > > Thank you for your clear examples. I'm not sure, but I may have found > the problem. > > As I recall, setting obj.attr to a property won't work. Instead, you > have to set type(obj).attr to the property. And now you come up > against a problem. You can't set __doc__ on the type 'function', nor > can you subclass the type 'function'. > > >>> def fn(): pass > >>> type(fn) > > > >>> type(fn).__doc__ > 'function(code, globals[, name[, argdefs[, closure]]]) [snip]' > > >>> type(fn).__doc__ = None > TypeError: can't set attributes of built-in/extension type 'function' > > >>> class FN(type(fn)): pass > TypeError: type 'function' is not an acceptable base type > > As I (more vaguely) recall, something was done recently to add a get > attribute property that works directly on objects, rather than on > instances of a class. But I've not been able to find it. Perhaps I > misunderstood PEP 562. > > Some perhaps relevant URLs: > https://www.python.org/dev/peps/pep-0224/ Attribute Docstrings > https://www.python.org/dev/peps/pep-0549/ Instance Descriptors > https://www.python.org/dev/peps/pep-0562/ Module __getattr__ and __dir__ > https://stackoverflow.com/questions/2447353/getattr-on-a-module > > -- > Jonathan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From 2015 at jmunch.dk Sun Oct 7 12:45:28 2018 From: 2015 at jmunch.dk (2015 at jmunch.dk) Date: Sun, 7 Oct 2018 18:45:28 +0200 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <855d8842-35e3-4032-6b70-eb601a0b2fc9@stofanet.dk> References: <20181007065846.GA57216@cskk.homeip.net> <855d8842-35e3-4032-6b70-eb601a0b2fc9@stofanet.dk> Message-ID: <8e0057b7-b3c5-d10f-876a-bcdc70b9e3fc@stofanet.dk> On 18-10-07 16.15, Ram Rachum wrote: > I tested it now and indeed bytes patterns work on memoryview objects. > But how do I use this to scan for patterns through a stream without > loading it to memory? An mmap object is one of the things you can make a memoryview of, although looking again, it seems you don't even need to, you can just re.search the mmap object directly. re.search'ing the mmap object means the operating system takes care of the streaming for you, reading in parts of the file only as necessary. regards, Anders From jfine2358 at gmail.com Sun Oct 7 13:13:22 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Sun, 7 Oct 2018 18:13:22 +0100 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <8e0057b7-b3c5-d10f-876a-bcdc70b9e3fc@stofanet.dk> References: <20181007065846.GA57216@cskk.homeip.net> <855d8842-35e3-4032-6b70-eb601a0b2fc9@stofanet.dk> <8e0057b7-b3c5-d10f-876a-bcdc70b9e3fc@stofanet.dk> Message-ID: Anders wrote > An mmap object is one of the things you can make a memoryview of, > although looking again, it seems you don't even need to, you can > just re.search the mmap object directly. > > re.search'ing the mmap object means the operating system takes care of > the streaming for you, reading in parts of the file only as necessary. Provided mmap releases memory when possible, this is a good solution, and probably in many cases better than the one I suggested. And closer to what the original poster asked for. So I've learnt something. Thank you all. -- Jonathan From jfine2358 at gmail.com Sun Oct 7 14:40:17 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Sun, 7 Oct 2018 19:40:17 +0100 Subject: [Python-ideas] add a time decorator to timeit.py In-Reply-To: References: <20181007111527.GE3817@ando.pearwood.info> <20181007123410.GF3817@ando.pearwood.info> Message-ID: Summary: It's an undocumented feature, not an implementation detail. I wrote: > And if you're a software historian, now perhaps look at > https://github.com/python/cpython/commits/3.7/Lib/timeit.py Well, I've bitten my own bait, and have found it. See https://github.com/python/cpython/commit/d8faa3654 Merged revisions 53952-54987 via svnmerge from svn+ssh://pythondev at svn.python.org/python/trunk This is a compendium commit, which contains r54348 | georg.brandl | 2007-03-13 12:32:21 -0700 (Tue, 13 Mar 2007) | 4 lines Patch #1533909: the timeit module now accepts callables in addition to strings for the code to time and the setup code. Also added two convenience functions for instantiating a Timer and calling its methods. For the diff see https://github.com/python/cpython/commit/d8faa3654#diff-f21e8a2e7addb312903b3058066b6994R138 So to me that's conclusive. An undocumented feature. -- Jonathan From jfine2358 at gmail.com Sun Oct 7 15:49:21 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Sun, 7 Oct 2018 20:49:21 +0100 Subject: [Python-ideas] add a time decorator to timeit.py In-Reply-To: References: <20181007111527.GE3817@ando.pearwood.info> <20181007123410.GF3817@ando.pearwood.info> Message-ID: On 7 Oct 2018 20:39, "Michael Selik" wrote: Isn't there some rule about noticing an undocumented feature, like, whoever pours the last cup of coffee needs to brew a fresh pot? :-) Most of the credit belongs, I think, to the original poster. His statement, that Python wasn't very Pythonic here, really drew attention to the empty coffee pot. -- Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Oct 7 16:24:58 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 7 Oct 2018 16:24:58 -0400 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: References: Message-ID: On 9/24/2018 3:46 AM, Marko Ristin-Kaufmann wrote: I am responding to your request "Please do point me to what is not obvious to you". I think some of your claims are not only not obvious, but are wrong. I have read some (probably less than half) of the responses and avoid saying what I know others have covered adequately. A mathematical function is defined or specified by a input domain, output range, and a mapping from inputs to outputs. The mapping can be defined either by an explicit listing of input-output pairs or by a rule specifying either a) the process, what is done to inputs to produce outputs or, b) the result, how the output relates to the input. > https://www.win.tue.nl/~wstomv/edu/2ip30/references/design-by-contract/index.html defines contracts as "precise (legally unambiguous) specifications" (5.2 Business Contracting/Sub-contracting Metaphor) It is not obvious to me that the metaphor of contracts adds anything worthwhile to the idea of 'function'. 1. Only a small sliver of human interactions are governed by formal legal contracts read, understood, and agreed to by both (all) parties. 2. The idealized definition is naive in practice. Most legal contracts, unlike the example in the link article, are written in language that most people cannot read. Many contracts are imprecise and legally ambiguous, which is why we have contract dispute courts. And even then, the expense means that most people who feel violated in a transaction do not use courts. Post-conditions specify a function by result. I claim that this is not always sensible. I said above that functions may be specified by process rather than result. Ironically, the contract metaphor reinforces my claim. Many contracts, such as in teaching and medicine, only specify process and explicitly disclaim any particular result of concern to the client. > b)//If you write contracts in text, they will become stale over time Not true for good docstrings. We very seldom change the essential meaning of public functions. How has "Return the sine of x (measured in radians).", for math.sin, become stale? Why would it ever? What formal executable post condition would help someone who does not understand 'sine', or 'sine of x'? The function is only useful for someone who either understands 'sine' or is copying a formula written elsewhere without needing to understand it. Or consider "Stable sort *IN PLACE*." for list.sort. A formal executable post-condition for this *can* be written. But for someone who understands the words 'stable', 'sort', and 'in place', this compact English post condition is much more readable than any formal version could be. More over, that addition of 'stable', when we were ready to do so, did not make "Sort *IN PLACE*" stale in any sense of being wrong. I said above that functions definitions and contracts can specify the process rather than the result. For functions, the code is the process, and the code may then be the specification and the contract. For instance, def append_first(seq): "Append seq[0] to seq." So seq must have a first element accessed as seq[0] and an append method that accepts the first element as an argument. One might guess the minimal body. return seq.append(seq[0]) The pre-condition here is that the return value can be calculated as specified. Whenever this is true, python is already testing pre-conditions, as efficiently as possible. We could elaborate the body as try: return seq.append(seq[0]) except Exception as e: raise ValueError('raised ' + repr(e)[0]) and add 'Raise ValueError with explanation on bad input' to the doc string. If the domain were restricted to lists, we could write a post-condition, and use that in a list-based unittest. However, it likely be less readable than the docstring, and would execute in O(n) time (for an O(1) function). The latter is acceptable for unittests, which will use short tests, but not for production. But with duck-typing, no post condition is possible. Neither is a generic unittest. The code *is* the contract. This is generally true of duck-typed Python code. -- Terry Jan Reedy From guido at python.org Sun Oct 7 16:44:09 2018 From: guido at python.org (Guido van Rossum) Date: Sun, 7 Oct 2018 13:44:09 -0700 Subject: [Python-ideas] add a time decorator to timeit.py In-Reply-To: References: <20181007111527.GE3817@ando.pearwood.info> <20181007123410.GF3817@ando.pearwood.info> Message-ID: So someone ought to submit a PR that adds (brief) documentation for this, with reference to this thread. On Sun, Oct 7, 2018 at 12:50 PM Jonathan Fine wrote: > On 7 Oct 2018 20:39, "Michael Selik" wrote: > > Isn't there some rule about noticing an undocumented feature, like, > whoever pours the last cup of coffee needs to brew a fresh pot? :-) > > > Most of the credit belongs, I think, to the original poster. His > statement, that Python wasn't very Pythonic here, really drew attention to > the empty coffee pot. > > -- > Jonathan > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Oct 7 17:18:21 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 7 Oct 2018 17:18:21 -0400 Subject: [Python-ideas] Simplicity of C (was why is design-by-contracts not widely) In-Reply-To: References: <20180930044422.GA57629@cskk.homeip.net> Message-ID: On 9/30/2018 2:19 AM, Marko Ristin-Kaufmann wrote: > The library name will also need to change. When I started developing it, > I was not aware of Java icontract library. It will be probably renamed > to "pcontract" or any other suggested better name :) 'icontract' immediatly looks to me like 'internet contract', which is wrong (the Apple effect). I am guessing 'interface contract'. I would prefer 'pycontract'. -- Terry Jan Reedy From tjreedy at udel.edu Sun Oct 7 18:29:19 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 7 Oct 2018 18:29:19 -0400 Subject: [Python-ideas] Better error messages for missing optional stdlib packages In-Reply-To: <23943232-e980-11bd-04c4-157210c732d3@gmx.net> References: <23943232-e980-11bd-04c4-157210c732d3@gmx.net> Message-ID: On 10/3/2018 4:29 PM, Marcus Harnisch wrote: > When trying to import lzma on one of my machines, I was suprised to get > a normal import error like for any other module. What was the traceback and message? Did you get an import error for one of the three imports in lzma.py. I don't know why you would expect anything else. Any import in any stdlib module can potential fail if the file is buggy, corrupted, or missing. > According to the docs > lzma has been part of stdlib since 3.3. Further digging revealed that > the error is due to the fact that xz wasn't compiled in when building > Python. Perhaps this is a buggy build. Where did you get it from? Have you complained to the distributor? lzma is documented as wrapping liblzma and supporting .xz files. If 'xz' is a separate library, then perhaps lzma should tolerate it missing. > Since I suspect that there are other optional stdlib modules, I agree that modules that are necessarily optional should be documented as such, and as I mentioned on https://bugs.python.org/issue34895, many are so documented. In the absence of such documentation, I would considered it to be not optional except as some distributor decides to omit it. But then it is the responsibility of the distributor to document the omission. > this made me think whether the message in those cases should look a > little more polished. Perhaps installing a stub module that prints some > informative text before raising the relevant exception or similar. > Also, maybe add a little note in the docs, stating that despite being > part of stdlib this module might not be available on all systems. ImportError messages and the documented status versus actual status of a module are two different issues. Let's keep them separate. -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Sun Oct 7 17:32:55 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 08 Oct 2018 10:32:55 +1300 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: <20181007065846.GA57216@cskk.homeip.net> <855d8842-35e3-4032-6b70-eb601a0b2fc9@stofanet.dk> <8e0057b7-b3c5-d10f-876a-bcdc70b9e3fc@stofanet.dk> Message-ID: <5BBA7B87.7010202@canterbury.ac.nz> Jonathan Fine wrote: > Provided mmap releases memory when possible, It will. The virtual memory system will read pages from the file into RAM when needed, and re-use those RAM pages for other purposes when needed. It should be pretty much the most efficient solution possible. -- Greg From steve at pearwood.info Sun Oct 7 18:52:31 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 8 Oct 2018 09:52:31 +1100 Subject: [Python-ideas] Dynamic getting of __doc__ of a function In-Reply-To: References: <20181007083101.GD3817@ando.pearwood.info> Message-ID: <20181007225231.GI3817@ando.pearwood.info> On Sun, Oct 07, 2018 at 06:17:47PM +0200, Marko Ristin-Kaufmann wrote: > Hi, > @Jonathan: thanks! I'll have a look at your links. > > So there are actually two issues if I understand correctly: > * help() ignoring the property getter when invoked on an instance Not just the property getter, but if you set instance.__doc__ to a string, help ignores that string too. > * built-in class function ignoring the property getter (some_func.__doc__ > set to a property returns the property instead of invoking the getter) Not just functions, it is all instances. Properties are only invoked if they are on the class object, not the instance. py> class X: ... pass ... py> x = X() py> x.spam = property(lambda self: "computed result") py> x.spam py> x.spam.__get__(x) 'computed result' -- Steve From tjreedy at udel.edu Sun Oct 7 20:09:02 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 7 Oct 2018 20:09:02 -0400 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: On 10/6/2018 5:00 PM, Nathaniel Smith wrote: > On Sat, Oct 6, 2018 at 12:22 AM, Ram Rachum wrote: >> I'd like to use the re module to parse a long text file, 1GB in size. I wish >> that the re module could parse a stream, so I wouldn't have to load the >> whole thing into memory. I'd like to iterate over matches from the stream >> without keeping the old matches and input in RAM. >> >> What do you think? > > This has frustrated me too. > > The case where I've encountered this is parsing HTTP/1.1. We have data > coming in incrementally over the network, and we want to find the end > of the headers. To do this, we're looking for the first occurrence of > b"\r\n\r\n" OR b"\n\n". > > So our requirements are: > > 1. Search a bytearray for the regex b"\r\n\r\n|\n\n" I believe that re is both overkill and slow for this particular problem. For O(n), search forward for \n with str.index('\n') (or .find) [I assume that this searches forward faster than for i, c in enumerate(s): if c == '\n': break and leave you to test this.] If not found, continue with next chunk of data. If found, look back for \r to determine whether to look forward for \n or \r\n *whenever there is enough data to do so. > 2. If there's no match yet, wait for more data to arrive and try again > 3. When more data arrives, start searching again *where the last > search left off* s.index has an optional start parameter. And keep chunks in a list until you have a match and can join all at once. -- Terry Jan Reedy From tjreedy at udel.edu Sun Oct 7 20:14:00 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 7 Oct 2018 20:14:00 -0400 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: <20181007013948.GA3817@ando.pearwood.info> Message-ID: On 10/7/2018 12:32 AM, Ram Rachum wrote: > Does that mean I'll have to write that character-by-character algorithm? I would not be surprised if you could make use of str.index, which scans at C speed. See my answer to Nathaniel. -- Terry Jan Reedy From njs at pobox.com Sun Oct 7 20:54:16 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 7 Oct 2018 17:54:16 -0700 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: On Sun, Oct 7, 2018 at 5:09 PM, Terry Reedy wrote: > On 10/6/2018 5:00 PM, Nathaniel Smith wrote: >> >> On Sat, Oct 6, 2018 at 12:22 AM, Ram Rachum wrote: >>> >>> I'd like to use the re module to parse a long text file, 1GB in size. I >>> wish >>> that the re module could parse a stream, so I wouldn't have to load the >>> whole thing into memory. I'd like to iterate over matches from the stream >>> without keeping the old matches and input in RAM. >>> >>> What do you think? >> >> >> This has frustrated me too. >> >> The case where I've encountered this is parsing HTTP/1.1. We have data >> coming in incrementally over the network, and we want to find the end >> of the headers. To do this, we're looking for the first occurrence of >> b"\r\n\r\n" OR b"\n\n". >> >> So our requirements are: >> >> 1. Search a bytearray for the regex b"\r\n\r\n|\n\n" > > > I believe that re is both overkill and slow for this particular problem. > For O(n), search forward for \n with str.index('\n') (or .find) > [I assume that this searches forward faster than > for i, c in enumerate(s): > if c == '\n': break > and leave you to test this.] > > If not found, continue with next chunk of data. > If found, look back for \r to determine whether to look forward for \n or > \r\n *whenever there is enough data to do so. Are you imagining something roughly like this? (Ignoring chunk boundary handling for the moment.) def find_double_line_end(buf): start = 0 while True: next_idx = buf.index(b"\n", start) if buf[next_idx - 1:next_idx + 1] == b"\n" or buf[next_idx - 3:next_idx] == b"\r\n\r": return next_idx start = next_idx + 1 That's much more complicated than using re.search, and on some random HTTP headers I have lying around it benchmarks ~70% slower too. Which makes sense, since we're basically trying to replicate re engine's work by hand in a slower language. BTW, if we only want to find a fixed string like b"\r\n\r\n", then re.search and bytearray.index are almost identical in speed. If you have a problem that can be expressed as a regular expression, then regular expression engines are actually pretty good at solving those :-) -n -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Sun Oct 7 21:40:47 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 7 Oct 2018 18:40:47 -0700 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: On Sun, Oct 7, 2018 at 5:54 PM, Nathaniel Smith wrote: > Are you imagining something roughly like this? (Ignoring chunk > boundary handling for the moment.) > > def find_double_line_end(buf): > start = 0 > while True: > next_idx = buf.index(b"\n", start) > if buf[next_idx - 1:next_idx + 1] == b"\n" or buf[next_idx - > 3:next_idx] == b"\r\n\r": > return next_idx > start = next_idx + 1 > > That's much more complicated than using re.search, and on some random > HTTP headers I have lying around it benchmarks ~70% slower too. Which > makes sense, since we're basically trying to replicate re engine's > work by hand in a slower language. > > BTW, if we only want to find a fixed string like b"\r\n\r\n", then > re.search and bytearray.index are almost identical in speed. If you > have a problem that can be expressed as a regular expression, then > regular expression engines are actually pretty good at solving those > :-) Though... here's something strange. Here's another way to search for the first appearance of either \r\n\r\n or \n\n in a bytearray: def find_double_line_end_2(buf): idx1 = buf.find(b"\r\n\r\n") idx2 = buf.find(b"\n\n", 0, idx1) if idx1 == -1: return idx2 elif idx2 == -1: return idx1 else: return min(idx1, idx2) So this is essentially equivalent to our regex (notice they both pick out position 505 as the end of the headers): In [52]: find_double_line_end_2(sample_headers) Out[52]: 505 In [53]: double_line_end_re = re.compile(b"\r\n\r\n|\n\n") In [54]: double_line_end_re.search(sample_headers) Out[54]: <_sre.SRE_Match object; span=(505, 509), match=b'\r\n\r\n'> But, the Python function that calls bytearray.find twice is about ~3x faster than the re module: In [55]: %timeit find_double_line_end_2(sample_headers) 1.18 ?s ? 40 ns per loop (mean ? std. dev. of 7 runs, 1000000 loops each) In [56]: %timeit double_line_end_re.search(sample_headers) 3.3 ?s ? 23.9 ns per loop (mean ? std. dev. of 7 runs, 100000 loops each) The regex module is even slower: In [57]: double_line_end_regex = regex.compile(b"\r\n\r\n|\n\n") In [58]: %timeit double_line_end_regex.search(sample_headers) 4.95 ?s ? 76.4 ns per loop (mean ? std. dev. of 7 runs, 100000 loops each) -n -- Nathaniel J. Smith -- https://vorpus.org From marko.ristin at gmail.com Mon Oct 8 00:51:54 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Mon, 8 Oct 2018 06:51:54 +0200 Subject: [Python-ideas] Dynamic getting of __doc__ of a function In-Reply-To: <20181007225231.GI3817@ando.pearwood.info> References: <20181007083101.GD3817@ando.pearwood.info> <20181007225231.GI3817@ando.pearwood.info> Message-ID: Hi Steve, > > * built-in class function ignoring the property getter (some_func.__doc__ > > set to a property returns the property instead of invoking the getter) > > Not just functions, it is all instances. Properties are only invoked if > they are on the class object, not the instance. > I see. The concrete function is simply handled like an instance of the class "function" and not in any special way. Is that something I could fix? Or is it still not clear how this should be fixed? I am totally unaware of the bigger picture and wider repercussions here. Cheers Marko > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko.ristin at gmail.com Mon Oct 8 01:25:22 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Mon, 8 Oct 2018 07:25:22 +0200 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: References: Message-ID: Hi Terry, I would encourage you to read all the messages on the thread. The points you raised were actually already discussed: * Using formal contracts in the code does not imply that you must specify *all* the contracts; you specify what you consider meaningful. * Distinction of result/process (the example that was already discussed was related to GUIs and pathlib standard library) > b)//If you write contracts in text, they will become stale over time > > Not true for good docstrings. We very seldom change the essential > meaning of public functions. In my team, we have a stale docstring once every two weeks or even more often. If it weren't for doctests and contracts, I could imagine we would have them even more often :) I suppose good docstrings require many reviewers and slow refactoring processes. Our public interfaces are changing daily and there is one reviewer per pull request. If github/bitbucket allowed for better highlighting of docstrings in code reviews (*e.g., *if it highlighted the docstring of every function that changed), the miss rate would be probably lower. I'm always happy to hear other experiences how people dealt with requirements changing at a fast pace and how they dealt with code rot in a team of limited size and time resources. I agree with you that a good docstring is not stale by definition. I have just never experienced a team that had good docstrings in my career. Cheers, Marko On Sun, 7 Oct 2018 at 22:25, Terry Reedy wrote: > On 9/24/2018 3:46 AM, Marko Ristin-Kaufmann wrote: > > I am responding to your request "Please do point me to what is not > obvious to you". I think some of your claims are not only not obvious, > but are wrong. I have read some (probably less than half) of the > responses and avoid saying what I know others have covered adequately. > > A mathematical function is defined or specified by a input domain, > output range, and a mapping from inputs to outputs. The mapping can be > defined either by an explicit listing of input-output pairs or by a rule > specifying either a) the process, what is done to inputs to produce > outputs or, b) the result, how the output relates to the input. > > > > > https://www.win.tue.nl/~wstomv/edu/2ip30/references/design-by-contract/index.html > > defines contracts as "precise (legally unambiguous) specifications" (5.2 > Business Contracting/Sub-contracting Metaphor) It is not obvious to me > that the metaphor of contracts adds anything worthwhile to the idea of > 'function'. > > 1. Only a small sliver of human interactions are governed by formal > legal contracts read, understood, and agreed to by both (all) parties. > > 2. The idealized definition is naive in practice. Most legal contracts, > unlike the example in the link article, are written in language that > most people cannot read. Many contracts are imprecise and legally > ambiguous, which is why we have contract dispute courts. And even then, > the expense means that most people who feel violated in a transaction do > not use courts. > > Post-conditions specify a function by result. I claim that this is not > always sensible. I said above that functions may be specified by > process rather than result. Ironically, the contract metaphor > reinforces my claim. Many contracts, such as in teaching and medicine, > only specify process and explicitly disclaim any particular result of > concern to the client. > > > b)//If you write contracts in text, they will become stale over time > > Not true for good docstrings. We very seldom change the essential > meaning of public functions. > > How has "Return the sine of x (measured in radians).", for math.sin, > become stale? Why would it ever? What formal executable post condition > would help someone who does not understand 'sine', or 'sine of x'? The > function is only useful for someone who either understands 'sine' or is > copying a formula written elsewhere without needing to understand it. > > Or consider "Stable sort *IN PLACE*." for list.sort. A formal > executable post-condition for this *can* be written. But for someone > who understands the words 'stable', 'sort', and 'in place', this compact > English post condition is much more readable than any formal version > could be. More over, that addition of 'stable', when we were ready to > do so, did not make "Sort *IN PLACE*" stale in any sense of being wrong. > > I said above that functions definitions and contracts can specify the > process rather than the result. For functions, the code is the process, > and the code may then be the specification and the contract. For instance, > > def append_first(seq): > "Append seq[0] to seq." > > So seq must have a first element accessed as seq[0] and an append method > that accepts the first element as an argument. One might guess the > minimal body. > > return seq.append(seq[0]) > > The pre-condition here is that the return value can be calculated as > specified. Whenever this is true, python is already testing > pre-conditions, as efficiently as possible. We could elaborate the body as > > try: > return seq.append(seq[0]) > except Exception as e: > raise ValueError('raised ' + repr(e)[0]) > > and add 'Raise ValueError with explanation on bad input' to the doc string. > > If the domain were restricted to lists, we could write a post-condition, > and use that in a list-based unittest. However, it likely be less > readable than the docstring, and would execute in O(n) time (for an O(1) > function). The latter is acceptable for unittests, which will use short > tests, but not for production. But with duck-typing, no post condition > is possible. Neither is a generic unittest. The code *is* the > contract. This is generally true of duck-typed Python code. > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Oct 8 01:29:34 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 8 Oct 2018 16:29:34 +1100 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: References: Message-ID: On Mon, Oct 8, 2018 at 4:26 PM Marko Ristin-Kaufmann wrote: >> Not true for good docstrings. We very seldom change the essential >> meaning of public functions. > > In my team, we have a stale docstring once every two weeks or even more often. If it weren't for doctests and contracts, I could imagine we would have them even more often :) > In other words, you change the *public interface* of your functions all the time? How do you not have massive breakage all the time? ChrisA From marko.ristin at gmail.com Mon Oct 8 02:10:07 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Mon, 8 Oct 2018 08:10:07 +0200 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: References: <37e7961b-528d-309f-114f-7194b9051892@kynesim.co.uk> <20180929111925.GU19437@ando.pearwood.info> <23471.39350.891499.861435@turnbull.sk.tsukuba.ac.jp> Message-ID: Hi, I'd like to share a use pattern for contracts that might have got lost in the discussion and which I personally grow to like more and more. I'm not making any claims; this use pattern work for our team and I can't judge how much of a benefit it would be to others. Imagine there are two developers, Alice working on a package A, and Betty working on a package B. Package A depends on package B. Betty tested her package B with some test data D_B. Alice tests her package A with some test data D_A. Now assume Betty did not write any contracts for her package B. When Alice tests her package, she is actually making an integration test. While she controls the inputs to B from A, she can only observe the results from B, but not whether they are correct by coincidence or B did its job correctly. Let's denote D'_B the data that is given to B from her original test data D_A during Alice's integration testing. How can she test that package B gives the correct results on D'_B ? She needs to manually record the data somehow (by dynamically mocking package B and intercepting what gets passed from A to B?). She might fetch the tests from the package B, copy/paste the test cases and append D'_B. Or she could make a pull request and provide the extra test data directly to package B. She needs to understand how Betty's unit tests work and see how D'_B fits in there and what needs to be mocked. All in all, not a trivial task if Alice is not familiar with the package B and even less so if Alice and Betty don't work in the same organization. Most of the time, Alice would not bother to test the dependencies on her testing data D_A. She would assume that her dependencies work, and just tests what comes out of them. If the results make sense, she would call it a tick on her to-do list and move on with the next task. Let's assume now that Betty wrote some contracts in her code. When Alice runs the integration test of her package A, the contracts of B are automatically verified on D'_B. While the contracts might not cover all the cases that were covered in Betty's unit tests, they still cover some of them. Alice can be a bit more confident that at least *something* was checked on D'_B. Without the contracts, she would have checked *nothing* on D'_B in most of her everyday programming. You can consider writing contracts as a matter of economy in this story. Betty might not need contracts for maintaining her package B -- she can read her code, she can extend her test cases. However, you can see contracts as a service to the package users, Alice in this case. Betty helps Alice have some integration tests free-of-charge (free for Alice; Betty of course pays the overhead of writing and maintaining the contracts). Alice does not need to understand how B can be tested nor needs to manually record data that needs to be passed to B. She merely runs her test code and the checker library will do the testing of B on D'_B automatically. The utility of this service tends to grow exponentially in cases where dependency trees grow exponentially as well. Imagine if we had Carol with the package C, with the dependencies A -> B -> C. When Carol writes contracts, she does a service not only to her direct users (Betty) but also to the users of B (Alice). I don't see how Alice could practically cover the case with dependencies A -> B -> C and test C with D'_C (*i.e. *test C with the data coming from D_A) without the contracts unless she really takes her time and gets familiar with dependencies of all here immediate dependencies. We found this pattern helpful in the team, especially during refactorings where contracts provide an additional security net. We don't have time to record and add tests of B for D'_B, and even less so of C for D'_C. The contracts work thus as a good compromise for us (marginal overhead, but better documentation and "free" integration tests rather than none). Cheers, Marko On Sun, 30 Sep 2018 at 08:17, Marko Ristin-Kaufmann wrote: > Hi, > > I compiled a couple of issues on github to provide a more structured > ground for discussions on icontract features: > https://github.com/Parquery/icontract/issues (@David Maertz: I also > included the issue with automatically generated __doc__ in case you are > still interested in it). > > Cheers, > Marko > > On Sat, 29 Sep 2018 at 17:27, Stephen J. Turnbull < > turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > >> Steven D'Aprano writes: >> >> > put (x: ELEMENT; key: STRING) is >> > -- Insert x so that it will be retrievable through key. >> > require >> > count <= capacity >> > not key.empty >> > do >> > ... Some insertion algorithm ... >> > ensure >> > has (x) >> > item (key) = x >> > count = old count + 1 >> > end >> > >> > Two pre-conditions, and three post-conditions. That's hardly >> > complex. >> >> You can already do this: >> >> def put(self, x: Element, key: str) -> None: >> """Insert x so that it will be retrievable through key.""" >> >> # CHECKING PRECONDITIONS >> _old_count = self.count >> assert self.count <= self.capacity, >> assert key >> >> # IMPLEMENTATION >> ... some assertion algorithm ... >> >> # CHECKING POSTCONDITIONS >> assert x in self >> assert self[key] == x >> assert self.count == _old_count >> >> return >> >> I don't see a big advantage to having syntax, unless the syntax allows >> you to do things like turn off "expensive" contracts only. Granted, >> you save a little bit of typing and eye movement (you can omit >> "assert" and have syntax instead of an assignment for checking >> postconditions dependent on initial state). >> >> A document generator can look for the special comments (as with >> encoding cookies), and suck in all the asserts following until a >> non-assert line of code (or the next special comment). The >> assignments will need special handling, an additional special comment >> or something. With PEP 572, I think you could even do this: >> >> assert ((_old_count := self.count),) >> >> to get the benefit of python -O here. >> >> > If I were writing this in Python, I'd write something like this: >> > >> > def put(self, x, key): >> > """Insert x so that it will be retrievable through key.""" >> > # Input checks are pre-conditions! >> > if self.count > capacity: >> > raise DatabaseFullError >> > if not key: >> > raise ValueError >> > # .. Some insertion algorithm ... >> >> But this is quite different, as I understand it. Nothing I've seen in >> the discussion so far suggests that a contract violation allows >> raising differentiated exceptions, and it seems very unlikely from the >> syntax in your example above. I could easily see both of these errors >> being retryable: >> >> for _ in range(3): >> try: >> db.put(x, key) >> except DatabaseFullError: >> db.resize(expansion_factor=1.5) >> db.put(x, key) >> except ValueError: >> db.put(x, alternative_key) >> >> > and then stick the post-conditions in a unit test, usually in a >> > completely different file: >> >> If you like the contract-writing style, why would you do either of >> these instead of something like the code I wrote above? >> >> > So what's wrong with the status quo? >> > >> > - The pre-condition checks are embedded right there in the >> > method implementation, mixing up the core algorithm with the >> > associated error checking. >> >> You don't need syntax to separate them, you can use a convention, as I >> did above. >> >> > - Which in turn makes it hard to distinguish the checks from >> > the implementation, and impossible to do so automatically. >> >> sed can do it, why can't we? >> >> > - Half of the checks are very far away, in a separate file, >> > assuming I even remembered or bothered to write the test. >> >> That was your choice. There's nothing about the assert statement that >> says you're not allowed to use it at the end of a definition. >> >> > - The post-conditions aren't checked unless I run my test suite, and >> > then they only check the canned input in the test suite. >> >> Ditto. >> >> > - The pre-conditions can't be easily disabled in production. >> >> What's so hard about python -O? >> >> > - No class invariants. >> >> Examples? >> >> > - Inheritance is not handled correctly. >> >> Examples? Mixins and classes with additional functionality should >> work fine AFAICS. I guess you'd have to write the contracts in each >> subclass of an abstract class, which is definitely a minus for some of >> the contracts. But I don't see offhand why you would expect that the >> full contract of a method of a parent class would typically make sense >> without change for an overriding implementation, and might not make >> sense for a class with restricted functionality. >> >> > The status quo is all so very ad-hoc and messy. Design By Contract >> > syntax would allow (not force, allow!) us to add some structure to the >> > code: >> > >> > - requirements of the function >> > - the implementation of the function >> > - the promise made by the function >> >> Possible already as far as I can see. OK, you could have the compiler >> enforce the structure to some extent, but the real problem IMO is >> going to be like documentation and testing: programmers just won't do >> it regardless of syntax to make it nice and compiler checkable. >> >> > Most of us already think about these as three separate things, and >> > document them as such. Our code should reflect the structure of how we >> > think about the code. >> >> But what's the need for syntax? How about the common (in this thread) >> complaint that even as decorators, the contract is annoying, verbose, >> and distracts the reader from understanding the code? Note: I think >> that, as with static typing, this could be mitigated by allowing >> contracts to be optionally specified in a stub file. As somebody >> pointed out, it shouldn't be hard to write contract strippers and >> contract folding in many editors. (As always, we have to admit it's >> very difficult to get people to change their editor!) >> >> > > In my experience this is very rarely true. Most functions I >> > > write are fairly short and easily grokked, even if they do >> complicated >> > > things. That's part of the skill of breaking a problem down, IMHO; >> if >> > > the function is long and horrible-looking, I've already got it wrong >> and >> > > no amount of protective scaffolding like DbC is going to help. >> > >> > That's like saying that if a function is horrible-looking, then >> there's >> > no point in writing tests for it. >> > >> > I'm not saying that contracts are only for horrible functions, but >> > horrible functions are the ones which probably benefit the most from >> > specifying exactly what they promise to do, and checking on every >> > invocation that they live up to that promise. >> >> I think you're missing the point then: ISTM that the implicit claim >> here is that the time spent writing contracts for a horrible function >> would be better spent refactoring it. As you mention in connection >> with the Eiffel example, it's not easy to get all the relevant >> contracts, and for a horrible function it's going to be hard to get >> some of the ones you do write correct. >> >> > Python (the interpreter) does type checking. Any time you get a >> > TypeError, that's a failed type check. And with type annotations, we >> can >> > run a static type checker on our code too, which will catch many of >> > these failures before we run the code. >> >> But an important strength of contracts is that they are *always* run, >> on any input you actually give the function. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko.ristin at gmail.com Mon Oct 8 02:22:05 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Mon, 8 Oct 2018 08:22:05 +0200 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: References: Message-ID: Hi Chris, In other words, you change the *public interface* of your functions > all the time? How do you not have massive breakage all the time? I suppose that Pycharm helps the most with its refactoring tools. We use type annotations, contracts, static checks (mypy, pylint, pydocstyle) and unit, integration and end-to-end tests, so actually unexpected breakages in production are not that frequent. What does happen often, though, is that documentation gets stale. Cheers, Marko On Mon, 8 Oct 2018 at 07:29, Chris Angelico wrote: > On Mon, Oct 8, 2018 at 4:26 PM Marko Ristin-Kaufmann > wrote: > >> Not true for good docstrings. We very seldom change the essential > >> meaning of public functions. > > > > In my team, we have a stale docstring once every two weeks or even more > often. If it weren't for doctests and contracts, I could imagine we would > have them even more often :) > > > > In other words, you change the *public interface* of your functions > all the time? How do you not have massive breakage all the time? > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ijkl at netc.fr Mon Oct 8 03:10:40 2018 From: ijkl at netc.fr (Jimmy Girardet) Date: Mon, 8 Oct 2018 09:10:40 +0200 Subject: [Python-ideas] support toml for pyproject support Message-ID: <7fd0e3a3-34e2-321a-86d6-48af3504b539@netc.fr> Hi, I don't know if this was already debated? but I don't know how to search in the whole archive of the list. For? now the? adoption of pyproject.toml file is more difficult because toml is not in the standard library. Each tool which wants to use pyproject.toml has to add a toml lib? as a conditional or hard dependency. Since toml is now the standard configuration file format, it's strange the python does not support it in the stdlib lije it would have been strange to not have the configparser module. I know it's complicated to add more and more thing to the stdlib but I really think it is necessary for python packaging being more consistent. Maybe we could thought to a readonly lib to limit the added code. If it's conceivable, I'd be happy to help in it. Nice Day guys and girls. Jimmy From ram at rachum.com Mon Oct 8 03:56:15 2018 From: ram at rachum.com (Ram Rachum) Date: Mon, 8 Oct 2018 10:56:15 +0300 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <8e0057b7-b3c5-d10f-876a-bcdc70b9e3fc@stofanet.dk> References: <20181007065846.GA57216@cskk.homeip.net> <855d8842-35e3-4032-6b70-eb601a0b2fc9@stofanet.dk> <8e0057b7-b3c5-d10f-876a-bcdc70b9e3fc@stofanet.dk> Message-ID: That's incredibly interesting. I've never used mmap before. However, there's a problem. I did a few experiments with mmap now, this is the latest: path = pathlib.Path(r'P:\huge_file') with path.open('r') as file: mmap = mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) for match in re.finditer(b'.', mmap): pass The file is 338GB in size, and it seems that Python is trying to load it into memory. The process is now taking 4GB RAM and it's growing. I saw the same behavior when searching for a non-existing match. Should I open a Python bug for this? On Sun, Oct 7, 2018 at 7:49 PM <2015 at jmunch.dk> wrote: > On 18-10-07 16.15, Ram Rachum wrote: > > I tested it now and indeed bytes patterns work on memoryview objects. > > But how do I use this to scan for patterns through a stream without > > loading it to memory? > > An mmap object is one of the things you can make a memoryview of, > although looking again, it seems you don't even need to, you can > just re.search the mmap object directly. > > re.search'ing the mmap object means the operating system takes care of > the streaming for you, reading in parts of the file only as necessary. > > regards, Anders > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Oct 8 05:55:52 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 8 Oct 2018 20:55:52 +1100 Subject: [Python-ideas] support toml for pyproject support In-Reply-To: <7fd0e3a3-34e2-321a-86d6-48af3504b539@netc.fr> References: <7fd0e3a3-34e2-321a-86d6-48af3504b539@netc.fr> Message-ID: <20181008095551.GJ3817@ando.pearwood.info> Hi Jimmy, and welcome, On Mon, Oct 08, 2018 at 09:10:40AM +0200, Jimmy Girardet wrote: > Hi, > > I don't know if this was already debated? but I don't know how to search > in the whole archive of the list. > > > For? now the? adoption of pyproject.toml file is more difficult because > toml is not in the standard library. It is true that using third-party libraries is more difficult than using the std lib. That alone is not a reason to add a library to the std lib. > Each tool which wants to use pyproject.toml has to add a toml lib? as a > conditional or hard dependency. > > Since toml is now the standard configuration file format, It is? Did I miss the memo? Because I've never even heard of TOML before this very moment. Google Trends doesn't really support your assertion that TOML has become "the standard" for config files: # compare TOML, JSON and YAML https://trends.google.com/trends/explore?q=%2Fg%2F11c5zwr35t,%2Fm%2F05cntt,%2Fm%2F01w6k2 although it is trending upwards: https://trends.google.com/trends/explore?q=%2Fg%2F11c5zwr35t > it's strange > the python does not support it in the stdlib lije it would have been > strange to not have the configparser module. We don't even ship a YAML library, and that seems to be far more popular than TOML. On the other hand, we do ship a plist library. > I know it's complicated to add more and more thing to the stdlib but I > really think it is necessary for python packaging being more consistent. > > > Maybe we could thought to a readonly lib to limit the added code. What is a readonly lib? -- Steve From cs at cskk.id.au Mon Oct 8 06:19:57 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 8 Oct 2018 21:19:57 +1100 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: <20181008101957.GA85076@cskk.homeip.net> On 08Oct2018 10:56, Ram Rachum wrote: >That's incredibly interesting. I've never used mmap before. >However, there's a problem. >I did a few experiments with mmap now, this is the latest: > >path = pathlib.Path(r'P:\huge_file') > >with path.open('r') as file: > mmap = mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) Just a remark: don't tromp on the "mmap" name. Maybe "mapped"? > for match in re.finditer(b'.', mmap): > pass > >The file is 338GB in size, and it seems that Python is trying to load it >into memory. The process is now taking 4GB RAM and it's growing. I saw the >same behavior when searching for a non-existing match. > >Should I open a Python bug for this? Probably not. First figure out what is going on. BTW, how much RAM have you got? As you access the mapped file the OS will try to keep it in memory in case you need that again. In the absense of competition, most stuff will get paged out to accomodate it. That's normal. All the data are "clean" (unmodified) so the OS can simply release the older pages instantly if something else needs the RAM. However, another possibility is the the regexp is consuming lots of memory. The regexp seems simple enough (b'.'), so I doubt it is leaking memory like mad; I'm guessing you're just seeing the OS page in as much of the file as it can. Also, does the loop iterate? i.e. does it find multiple matches as the memory gets consumed, or is the first iateration blocking and consuming gobs of memory before the first match comes back? A print() call will tell you that. Cheers, Cameron Simpson From njs at pobox.com Mon Oct 8 06:22:59 2018 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 8 Oct 2018 03:22:59 -0700 Subject: [Python-ideas] support toml for pyproject support In-Reply-To: <20181008095551.GJ3817@ando.pearwood.info> References: <7fd0e3a3-34e2-321a-86d6-48af3504b539@netc.fr> <20181008095551.GJ3817@ando.pearwood.info> Message-ID: On Mon, Oct 8, 2018 at 2:55 AM, Steven D'Aprano wrote: > > On Mon, Oct 08, 2018 at 09:10:40AM +0200, Jimmy Girardet wrote: >> Each tool which wants to use pyproject.toml has to add a toml lib as a >> conditional or hard dependency. >> >> Since toml is now the standard configuration file format, > > It is? Did I miss the memo? Because I've never even heard of TOML before > this very moment. He's referring to PEPs 518 and 517 [1], which indeed standardize on TOML as a file format for Python package build metadata. I think moving anything into the stdlib would be premature though ? TOML libraries are under active development, and the general trend in the packaging space has been to move things *out* of the stdlib (e.g. there's repeated rumblings about moving distutils out), because the stdlib release cycle doesn't work well for packaging infrastructure. -n [1] https://www.python.org/dev/peps/pep-0518/ https://www.python.org/dev/peps/pep-0517 -- Nathaniel J. Smith -- https://vorpus.org From steve at pearwood.info Mon Oct 8 06:26:04 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 8 Oct 2018 21:26:04 +1100 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: References: Message-ID: <20181008102603.GK3817@ando.pearwood.info> On Mon, Oct 08, 2018 at 04:29:34PM +1100, Chris Angelico wrote: > On Mon, Oct 8, 2018 at 4:26 PM Marko Ristin-Kaufmann > wrote: > >> Not true for good docstrings. We very seldom change the essential > >> meaning of public functions. > > > > In my team, we have a stale docstring once every two weeks or even more often. "At Resolver we've found it useful to short-circuit any doubt and just refer to comments in code as 'lies'. " --Michael Foord paraphrases Christian Muirhead on python-dev, 2009-03-22 > If it weren't for doctests and contracts, I could imagine we would > have them even more often :) > > > > In other words, you change the *public interface* of your functions > all the time? How do you not have massive breakage all the time? I can't comment about Marko's actual use-case, but *in general* contracts are aimed at application *internal* interfaces, not so much library *public* interfaces. That's not to say that contracts can't be used for libraries at all, but they're not so useful for public interfaces that could be called by arbitrary third-parties. They are more useful for internal interfaces, where you don't break anyone's code but your own if you change the API. Think about it this way: you probably wouldn't hesitate much to change the interface of a _private method or function, aside from discussing it with your dev team. Sure it will break some code, but you have tests to identify the breakage, and maybe refactoring tools to help. And of course the contracts themselves are de facto tests. Such changes are manageable. And since its a private function, nobody outside of your team need care. Same with contracts. (At least in the ideal case.) -- Steve From rosuav at gmail.com Mon Oct 8 06:32:23 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 8 Oct 2018 21:32:23 +1100 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: <20181008102603.GK3817@ando.pearwood.info> References: <20181008102603.GK3817@ando.pearwood.info> Message-ID: On Mon, Oct 8, 2018 at 9:26 PM Steven D'Aprano wrote: > > In other words, you change the *public interface* of your functions > > all the time? How do you not have massive breakage all the time? > > I can't comment about Marko's actual use-case, but *in general* > contracts are aimed at application *internal* interfaces, not so much > library *public* interfaces. Yet we keep having use-cases shown to us involving one person with one module, and another person with another module, and the interaction between the two. Which way is it? Do the contracts change frequently or not? Are they public or not? How are we supposed to understand the point of contracts if the use-cases being shown all involve bad code and/or bad coding practices? Contracts, apparently, allow people to violate versioning expectations and feel good about it. (Am I really exaggerating all that much here?) ChrisA From ram at rachum.com Mon Oct 8 06:36:04 2018 From: ram at rachum.com (Ram Rachum) Date: Mon, 8 Oct 2018 13:36:04 +0300 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <20181008101957.GA85076@cskk.homeip.net> References: <20181008101957.GA85076@cskk.homeip.net> Message-ID: I'm not an expert on memory. I used Process Explorer to look at the Process. The Working Set of the current run is 11GB. The Private Bytes is 708MB. Actually, see all the info here: https://www.dropbox.com/s/tzoud028pzdkfi7/screenshot_TURING_2018-10-08_133355.jpg?dl=0 I've got 16GB of RAM on this computer, and Process Explorer says it's almost full, just ~150MB left. This is physical memory. To your question: The loop does iterate, i.e. finding multiple matches. On Mon, Oct 8, 2018 at 1:20 PM Cameron Simpson wrote: > On 08Oct2018 10:56, Ram Rachum wrote: > >That's incredibly interesting. I've never used mmap before. > >However, there's a problem. > >I did a few experiments with mmap now, this is the latest: > > > >path = pathlib.Path(r'P:\huge_file') > > > >with path.open('r') as file: > > mmap = mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) > > Just a remark: don't tromp on the "mmap" name. Maybe "mapped"? > > > for match in re.finditer(b'.', mmap): > > pass > > > >The file is 338GB in size, and it seems that Python is trying to load it > >into memory. The process is now taking 4GB RAM and it's growing. I saw the > >same behavior when searching for a non-existing match. > > > >Should I open a Python bug for this? > > Probably not. First figure out what is going on. BTW, how much RAM have > you > got? > > As you access the mapped file the OS will try to keep it in memory in case > you > need that again. In the absense of competition, most stuff will get paged > out > to accomodate it. That's normal. All the data are "clean" (unmodified) so > the > OS can simply release the older pages instantly if something else needs > the > RAM. > > However, another possibility is the the regexp is consuming lots of memory. > > The regexp seems simple enough (b'.'), so I doubt it is leaking memory > like > mad; I'm guessing you're just seeing the OS page in as much of the file as > it > can. > > Also, does the loop iterate? i.e. does it find multiple matches as the > memory > gets consumed, or is the first iateration blocking and consuming gobs of > memory > before the first match comes back? A print() call will tell you that. > > Cheers, > Cameron Simpson > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Mon Oct 8 07:52:33 2018 From: erik.m.bray at gmail.com (Erik Bray) Date: Mon, 8 Oct 2018 13:52:33 +0200 Subject: [Python-ideas] support toml for pyproject support In-Reply-To: References: <7fd0e3a3-34e2-321a-86d6-48af3504b539@netc.fr> <20181008095551.GJ3817@ando.pearwood.info> Message-ID: On Mon, Oct 8, 2018 at 12:23 PM Nathaniel Smith wrote: > > On Mon, Oct 8, 2018 at 2:55 AM, Steven D'Aprano wrote: > > > > On Mon, Oct 08, 2018 at 09:10:40AM +0200, Jimmy Girardet wrote: > >> Each tool which wants to use pyproject.toml has to add a toml lib as a > >> conditional or hard dependency. > >> > >> Since toml is now the standard configuration file format, > > > > It is? Did I miss the memo? Because I've never even heard of TOML before > > this very moment. > > He's referring to PEPs 518 and 517 [1], which indeed standardize on > TOML as a file format for Python package build metadata. > > I think moving anything into the stdlib would be premature though ? > TOML libraries are under active development, and the general trend in > the packaging space has been to move things *out* of the stdlib (e.g. > there's repeated rumblings about moving distutils out), because the > stdlib release cycle doesn't work well for packaging infrastructure. If I had the energy to argue it I would also argue against using TOML in those PEPs. I personally don't especially care for TOML and what's "obvious" to Tom is not at all obvious to me. I'd rather just stick with YAML or perhaps something even simpler than either one. From marko.ristin at gmail.com Mon Oct 8 07:56:41 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Mon, 8 Oct 2018 13:56:41 +0200 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: References: <20181008102603.GK3817@ando.pearwood.info> Message-ID: Hi Crhis, > In other words, you change the *public interface* of your functions > > all the time? How do you not have massive breakage all the time? > > I can't comment about Marko's actual use-case, but *in general* > contracts are aimed at application *internal* interfaces, not so much > library *public* interfaces. > Sorry, I might have misunderstood the question -- I was referring to modules used within the company, not outside. Of course, public libraries put on pypi don't change their interfaces weekly. Just to clear the confusion, both Steve and I would claim that the contracts do count as part of the interface. For everything internal, we make changes frequently (including the interface) and more often than not, the docstring is not updated when the implementation of the function is. Contracts help our team catch breaking changes more easily. When we change the behavior of the function, we use "Find usage" in Pycharm, fix manually what we can obviously see that was affected by the changed implementation, then statically check with mypy that the changed return type did not affect the callers, and contracts (of other functions!) catch some of the bugs during testing that we missed when we changed the implementation. End-to-end test with testing contracts turned off catch some more bugs on the real data, and then it goes into production where hopefully we see no errors. Cheers, Marko On Mon, 8 Oct 2018 at 12:32, Chris Angelico wrote: > On Mon, Oct 8, 2018 at 9:26 PM Steven D'Aprano > wrote: > > > In other words, you change the *public interface* of your functions > > > all the time? How do you not have massive breakage all the time? > > > > I can't comment about Marko's actual use-case, but *in general* > > contracts are aimed at application *internal* interfaces, not so much > > library *public* interfaces. > > Yet we keep having use-cases shown to us involving one person with one > module, and another person with another module, and the interaction > between the two. Which way is it? Do the contracts change frequently > or not? Are they public or not? How are we supposed to understand the > point of contracts if the use-cases being shown all involve bad code > and/or bad coding practices? > > Contracts, apparently, allow people to violate versioning expectations > and feel good about it. > > (Am I really exaggerating all that much here?) > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From boxed at killingar.net Mon Oct 8 08:02:11 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Mon, 8 Oct 2018 14:02:11 +0200 Subject: [Python-ideas] support toml for pyproject support In-Reply-To: References: <7fd0e3a3-34e2-321a-86d6-48af3504b539@netc.fr> <20181008095551.GJ3817@ando.pearwood.info> Message-ID: <0A82538D-0BC0-4A6E-90B8-247252301807@killingar.net> >> He's referring to PEPs 518 and 517 [1], which indeed standardize on >> TOML as a file format for Python package build metadata. >> >> I think moving anything into the stdlib would be premature though ? >> TOML libraries are under active development, and the general trend in >> the packaging space has been to move things *out* of the stdlib (e.g. >> there's repeated rumblings about moving distutils out), because the >> stdlib release cycle doesn't work well for packaging infrastructure. > > If I had the energy to argue it I would also argue against using TOML > in those PEPs. I personally don't especially care for TOML and what's > "obvious" to Tom is not at all obvious to me. I'd rather just stick > with YAML or perhaps something even simpler than either one. This thread isn't about regretting past decisions but what makes sense given current realities though. / Anders From erik.m.bray at gmail.com Mon Oct 8 08:02:05 2018 From: erik.m.bray at gmail.com (Erik Bray) Date: Mon, 8 Oct 2018 14:02:05 +0200 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <20181008101957.GA85076@cskk.homeip.net> References: <20181008101957.GA85076@cskk.homeip.net> Message-ID: On Mon, Oct 8, 2018 at 12:20 PM Cameron Simpson wrote: > > On 08Oct2018 10:56, Ram Rachum wrote: > >That's incredibly interesting. I've never used mmap before. > >However, there's a problem. > >I did a few experiments with mmap now, this is the latest: > > > >path = pathlib.Path(r'P:\huge_file') > > > >with path.open('r') as file: > > mmap = mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) > > Just a remark: don't tromp on the "mmap" name. Maybe "mapped"? > > > for match in re.finditer(b'.', mmap): > > pass > > > >The file is 338GB in size, and it seems that Python is trying to load it > >into memory. The process is now taking 4GB RAM and it's growing. I saw the > >same behavior when searching for a non-existing match. > > > >Should I open a Python bug for this? > > Probably not. First figure out what is going on. BTW, how much RAM have you > got? > > As you access the mapped file the OS will try to keep it in memory in case you > need that again. In the absense of competition, most stuff will get paged out > to accomodate it. That's normal. All the data are "clean" (unmodified) so the > OS can simply release the older pages instantly if something else needs the > RAM. > > However, another possibility is the the regexp is consuming lots of memory. > > The regexp seems simple enough (b'.'), so I doubt it is leaking memory like > mad; I'm guessing you're just seeing the OS page in as much of the file as it > can. Yup. Windows will aggressively fill up your RAM in cases like this because after all why not? There's no use to having memory just sitting around unused. For read-only, non-anonymous mappings it's not much problem for the OS to drop pages that haven't been recently accessed and use them for something else. So I wouldn't be too worried about the process chewing up RAM. I feel like this is veering more into python-list territory for further discussion though. From steve at pearwood.info Mon Oct 8 08:11:08 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 8 Oct 2018 23:11:08 +1100 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: References: <20181008102603.GK3817@ando.pearwood.info> Message-ID: <20181008121108.GL3817@ando.pearwood.info> On Mon, Oct 08, 2018 at 09:32:23PM +1100, Chris Angelico wrote: > On Mon, Oct 8, 2018 at 9:26 PM Steven D'Aprano wrote: > > > In other words, you change the *public interface* of your functions > > > all the time? How do you not have massive breakage all the time? > > > > I can't comment about Marko's actual use-case, but *in general* > > contracts are aimed at application *internal* interfaces, not so much > > library *public* interfaces. > > Yet we keep having use-cases shown to us involving one person with one > module, and another person with another module, and the interaction > between the two. Do we? I haven't noticed anything that matches that description, although I admit I haven't read every single post in these threads religiously. But "application" != "one module" or "one developer". I fail to see the contradiction. An application can be split over dozens of modules, written by teams of developers. Whether one or a dozen modules, it still has no public interface that third-party code can call. It is *all* internal. Obviously if you are using contracts in public library code, the way you will manage them is different from the way you would manage them if you are using them for private or internal code. That's no different from (say) docstrings and doctests: there are implied stability promises for those in *some* functions (the public ones) but not *other* functions (the private ones). Of course some devs don't believe in stability promises, and treat all APIs as unstable. So what? That has nothing to do with contracts. People can "move fast and break everything" in any programming style they like. > Which way is it? Do the contracts change frequently or not? "Mu." https://en.wikipedia.org/wiki/Mu_(negative) They change as frequently as you, the developer writing them, chooses to change them. Just like your tests, your type annotations, your doc strings, and every other part of your code. > Are they public or not? That's up to you. Contracts were originally designed for application development, where the concept of "public" versus "private" is meaningless. The philosophy of DbC is always going to be biased towards that mind-set. Nevertheless, people can choose to use them for library code where there is a meaningful distinction. If they do so, then how they choose to manage the contracts is up to them. If you want to make a contract a public part of the interface, then you can (but that would rule out disabling that specific contract, at least for pre-conditions). If you only want to use it for internal interfaces, you can do that too. If you want to mix and match and make some contracts internal and some public, there is no DbC Police to tell you that you can't. > How are we supposed to understand the point of contracts You could start by reading the explanations given on the Eiffel page, which I've linked to about a bazillion times. Then you could read about another bazillion blog posts and discussions that describe it (some pro, some con, some mixed). And you can read the Wikipedia page that shows how DbC is supported natively by at least 17 languages (not just Eiffel) and via libraries in at least 15 others. Not just new experimental languages, but old, established and conservative languages like Java, C and Ada. There are heaps of discussions on DbC on Stackoverflow: https://stackoverflow.com/search?q=design%20by%20contract and a good page on wiki.c2: http://wiki.c2.com/?DesignByContract TIL: Pre- and postconditions were first supported natively Barbara Liskov's CLU in the 1970s. This is not some "weird bizarre Eiffel thing", as people seem to believe. If it hasn't quite gone mainstream, it is surely at least as common as functional programming style. It has been around for over forty years in one way or another, not four weeks, and is a standard, well-established if minority programming style and development process. Of course it is always valid to debate the pros and cons of DbC versus other development paradigms, but questioning the very basis of DbC as people here keep doing is as ludicrous and annoying as questioning the basis of OOP or FP or TDD would be. Just as functional programming is a paradigm that says (among other things) "no side effects", "no global variables holding state" etc, and we can choose to apply that paradigm even in non-FP languages, so DbC is in part a paradigm that tells you how to design the internals of your application. We can apply the same design concepts to any code we want, even if we're not buying into the whole Contract metaphor: - pre-conditions can be considered argument validation; - post-conditions can be considered a kind of test; - class invariants can be considered a kind of defensive assertion. > if the use-cases being shown all involve bad code > and/or bad coding practices? How do you draw that conclusion? > Contracts, apparently, allow people to violate versioning expectations > and feel good about it. > > (Am I really exaggerating all that much here?) "Exaggerating" is not the word I would choose to use. "Talking utter bollocks" is probably more accurate *grin* -- Steve From ram at rachum.com Mon Oct 8 08:11:14 2018 From: ram at rachum.com (Ram Rachum) Date: Mon, 8 Oct 2018 15:11:14 +0300 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: <20181008101957.GA85076@cskk.homeip.net> Message-ID: " Windows will aggressively fill up your RAM in cases like this because after all why not? There's no use to having memory just sitting around unused." Two questions: 1. Is the "why not" sarcastic, as in you're agreeing it's a waste? 2. Will this be different on Linux? Which command do I run on Linux to verify that the process isn't taking too much RAM? Thanks, Ram. On Mon, Oct 8, 2018 at 3:02 PM Erik Bray wrote: > On Mon, Oct 8, 2018 at 12:20 PM Cameron Simpson wrote: > > > > On 08Oct2018 10:56, Ram Rachum wrote: > > >That's incredibly interesting. I've never used mmap before. > > >However, there's a problem. > > >I did a few experiments with mmap now, this is the latest: > > > > > >path = pathlib.Path(r'P:\huge_file') > > > > > >with path.open('r') as file: > > > mmap = mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) > > > > Just a remark: don't tromp on the "mmap" name. Maybe "mapped"? > > > > > for match in re.finditer(b'.', mmap): > > > pass > > > > > >The file is 338GB in size, and it seems that Python is trying to load it > > >into memory. The process is now taking 4GB RAM and it's growing. I saw > the > > >same behavior when searching for a non-existing match. > > > > > >Should I open a Python bug for this? > > > > Probably not. First figure out what is going on. BTW, how much RAM have > you > > got? > > > > As you access the mapped file the OS will try to keep it in memory in > case you > > need that again. In the absense of competition, most stuff will get > paged out > > to accomodate it. That's normal. All the data are "clean" (unmodified) > so the > > OS can simply release the older pages instantly if something else needs > the > > RAM. > > > > However, another possibility is the the regexp is consuming lots of > memory. > > > > The regexp seems simple enough (b'.'), so I doubt it is leaking memory > like > > mad; I'm guessing you're just seeing the OS page in as much of the file > as it > > can. > > Yup. Windows will aggressively fill up your RAM in cases like this > because after all why not? There's no use to having memory just > sitting around unused. For read-only, non-anonymous mappings it's not > much problem for the OS to drop pages that haven't been recently > accessed and use them for something else. So I wouldn't be too > worried about the process chewing up RAM. > > I feel like this is veering more into python-list territory for > further discussion though. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From boxed at killingar.net Mon Oct 8 08:15:12 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Mon, 8 Oct 2018 14:15:12 +0200 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: <20181008101957.GA85076@cskk.homeip.net> Message-ID: <17E6FE52-5CFD-41AB-8807-4B08F9CFCCA6@killingar.net> >> However, another possibility is the the regexp is consuming lots of memory. >> >> The regexp seems simple enough (b'.'), so I doubt it is leaking memory like >> mad; I'm guessing you're just seeing the OS page in as much of the file as it >> can. > > Yup. Windows will aggressively fill up your RAM in cases like this > because after all why not? There's no use to having memory just > sitting around unused. For read-only, non-anonymous mappings it's not > much problem for the OS to drop pages that haven't been recently > accessed and use them for something else. So I wouldn't be too > worried about the process chewing up RAM. > > I feel like this is veering more into python-list territory for > further discussion though. Last time I worked on windows, which admittedly was a long time, the file cache was not attributed to a process, so this doesn't seem to be relevant to this situation. / Anders -------------- next part -------------- An HTML attachment was scrubbed... URL: From Richard at Damon-Family.org Mon Oct 8 08:25:53 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Mon, 8 Oct 2018 08:25:53 -0400 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: <20181008101957.GA85076@cskk.homeip.net> Message-ID: <7969bb36-8834-7178-e56d-bc353b3995f5@Damon-Family.org> On 10/8/18 8:11 AM, Ram Rachum wrote: > " Windows will aggressively fill up your RAM in cases like this > because after all why not?? There's no use to having memory just > sitting around unused." > > Two questions: > > 1. Is the "why not" sarcastic, as in you're agreeing it's a waste? > 2. Will this be different on Linux? Which command do I run on Linux to > verify that the process isn't taking too much RAM? > > > Thanks, > Ram. I would say the 'why not' isn't being sarcastic but pragmatic. (And I would expect Linux to work similarly). After all if you have a system with X amount of memory, and total memory demand for the other processes is 10% of X, what is the issue with letting one process use 80% of X with memory usages that is easy to clear out if something else wants it. A read only page that is already backed on the disk is trivial to make available for another usage. Memory just sitting idle is the real waste. -- Richard Damon From ram at rachum.com Mon Oct 8 08:29:37 2018 From: ram at rachum.com (Ram Rachum) Date: Mon, 8 Oct 2018 15:29:37 +0300 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <7969bb36-8834-7178-e56d-bc353b3995f5@Damon-Family.org> References: <20181008101957.GA85076@cskk.homeip.net> <7969bb36-8834-7178-e56d-bc353b3995f5@Damon-Family.org> Message-ID: Thanks for your help everybody! I'm very happy to have learned about mmap. On Mon, Oct 8, 2018 at 3:27 PM Richard Damon wrote: > On 10/8/18 8:11 AM, Ram Rachum wrote: > > " Windows will aggressively fill up your RAM in cases like this > > because after all why not? There's no use to having memory just > > sitting around unused." > > > > Two questions: > > > > 1. Is the "why not" sarcastic, as in you're agreeing it's a waste? > > 2. Will this be different on Linux? Which command do I run on Linux to > > verify that the process isn't taking too much RAM? > > > > > > Thanks, > > Ram. > I would say the 'why not' isn't being sarcastic but pragmatic. (And I > would expect Linux to work similarly). After all if you have a system > with X amount of memory, and total memory demand for the other processes > is 10% of X, what is the issue with letting one process use 80% of X > with memory usages that is easy to clear out if something else wants it. > A read only page that is already backed on the disk is trivial to make > available for another usage. > > Memory just sitting idle is the real waste. > > -- > Richard Damon > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhodri at kynesim.co.uk Mon Oct 8 09:15:35 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Mon, 8 Oct 2018 14:15:35 +0100 Subject: [Python-ideas] Debugging: some problems and possible solutions In-Reply-To: References: <2f3f92a8-fe84-9984-8238-283e44c456d6@googlemail.com> <5FE3F758-2D89-4682-88FC-D266AFEF91E1@killingar.net> <9cdae6fe-06b7-4d22-7141-f0bc081f63d9@kynesim.co.uk> <22F1CC9D-CC3E-4602-8D14-99A9A852570E@killingar.net> Message-ID: <26f2cf6b-0014-3eb6-a66e-69b54b00b243@kynesim.co.uk> On 04/10/18 19:10, Jonathan Fine wrote: > In response to my problem-solution pair (fixing a typo) >> TITLE: Debug print() statements cause doctests to fail > > Rhodri James wrote: >> Or write your debug output to stderr? > > Perhaps I've been too concise. If so, I apologise. My proposal is that > the system be set up so that > debug(a, b, c) > sends output to the correct stream, whatever it should be. > > Rhodri: Thank you for your contribution. Are you saying that because > the developer can write > print(a, b, c, file=sys.stderr) > there's not a problem to solve here? Exactly so. If you want a quick drop of debug information, print() will do that just fine. If you want detailed or tunable information, that's what the logging module is for. I'm not sure where on the line between the two your debug() sits and what it's supposed to offer that is better than either of the alternatives. -- Rhodri James *-* Kynesim Ltd From mertz at gnosis.cx Mon Oct 8 09:26:12 2018 From: mertz at gnosis.cx (David Mertz) Date: Mon, 8 Oct 2018 09:26:12 -0400 Subject: [Python-ideas] support toml for pyproject support In-Reply-To: <0A82538D-0BC0-4A6E-90B8-247252301807@killingar.net> References: <7fd0e3a3-34e2-321a-86d6-48af3504b539@netc.fr> <20181008095551.GJ3817@ando.pearwood.info> <0A82538D-0BC0-4A6E-90B8-247252301807@killingar.net> Message-ID: I agree here. I briefly urged against using the less used TOML format, but I have no real skin in the game around packaging. I like YAML, but that's also not in the standard library, even if more widely used. But given that packaging is committed to TOML, I think that's a strong case for including a library in stdlib. The PEP 517/518 authors had their reasons that were accepted. Now there is broad ecosystem that is built on that choice. Let's support it. On Mon, Oct 8, 2018, 8:03 AM Anders Hovm?ller wrote: > > >> He's referring to PEPs 518 and 517 [1], which indeed standardize on > >> TOML as a file format for Python package build metadata. > >> > >> I think moving anything into the stdlib would be premature though ? > >> TOML libraries are under active development, and the general trend in > >> the packaging space has been to move things *out* of the stdlib (e.g. > >> there's repeated rumblings about moving distutils out), because the > >> stdlib release cycle doesn't work well for packaging infrastructure. > > > > If I had the energy to argue it I would also argue against using TOML > > in those PEPs. I personally don't especially care for TOML and what's > > "obvious" to Tom is not at all obvious to me. I'd rather just stick > > with YAML or perhaps something even simpler than either one. > > This thread isn't about regretting past decisions but what makes sense > given current realities though. > > / Anders > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Oct 8 10:21:57 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Oct 2018 01:21:57 +1100 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: <20181008121108.GL3817@ando.pearwood.info> References: <20181008102603.GK3817@ando.pearwood.info> <20181008121108.GL3817@ando.pearwood.info> Message-ID: On Mon, Oct 8, 2018 at 11:11 PM Steven D'Aprano wrote: > > On Mon, Oct 08, 2018 at 09:32:23PM +1100, Chris Angelico wrote: > > On Mon, Oct 8, 2018 at 9:26 PM Steven D'Aprano wrote: > > > > In other words, you change the *public interface* of your functions > > > > all the time? How do you not have massive breakage all the time? > > > > > > I can't comment about Marko's actual use-case, but *in general* > > > contracts are aimed at application *internal* interfaces, not so much > > > library *public* interfaces. > > > > Yet we keep having use-cases shown to us involving one person with one > > module, and another person with another module, and the interaction > > between the two. > > Do we? I haven't noticed anything that matches that description, > although I admit I haven't read every single post in these threads > religiously. Try this: On Mon, Oct 8, 2018 at 5:11 PM Marko Ristin-Kaufmann wrote: > Alice tests her package A with some test data D_A. Now assume Betty did not write any contracts for her package B. When Alice tests her package, she is actually making an integration test. While she controls the inputs to B from A, she can only observe the results from B, but not whether they are correct by coincidence or B did its job correctly. Let's denote D'_B the data that is given to B from her original test data D_A during Alice's integration testing. > If you're regularly changing your function contracts, such that you need to continually test in case something in the other package changed, then yes, that's exactly what I'm talking about. I'm tired of debating this. Have fun. If you love contracts so much, marry them. I'm not interested in using them, because nothing in any of these threads has shown me any good use-cases that aren't just highlighting bad coding practices. ChrisA From steve at pearwood.info Mon Oct 8 10:26:32 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 9 Oct 2018 01:26:32 +1100 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: References: Message-ID: <20181008142632.GM3817@ando.pearwood.info> On Sun, Oct 07, 2018 at 04:24:58PM -0400, Terry Reedy wrote: > A mathematical function is defined or specified by a input domain, > output range, and a mapping from inputs to outputs. The mapping can be > defined either by an explicit listing of input-output pairs or by a rule > specifying either a) the process, what is done to inputs to produce > outputs or, b) the result, how the output relates to the input. Most code does not define pure mathematical functions, unless you're writing in Haskall :-) > https://www.win.tue.nl/~wstomv/edu/2ip30/references/design-by-contract/index.html > > defines contracts as "precise (legally unambiguous) specifications" (5.2 > Business Contracting/Sub-contracting Metaphor) You are quoting that out of context. The full context says (emphasis added): IN THE BUSINESS WORLD, contracts are precise (legally unambiguous) specifications that define the obligations and benefits of the (usually two) parties involved. and later goes on to say: How does this apply to software correctness? Consider the execution of a routine. The called routine provides a service - it is a supplier. The caller is the client that is requesting the service. We can impose a contract that spells out precisely the obligations and benefits of both the caller (client) and the callee (supplier). This contract SERVES AS THE INTERFACE SPECIFICATION FOR THE ROUTINE. (I would add *executable* interface specification.) > It is not obvious to me > that the metaphor of contracts adds anything worthwhile to the idea of > 'function'. It doesn't. That's not what the metaphor is for. Design By Contract is not a redefinition of "function", it is a software methodology, a paradigm for helping programmers reason better about functions and specify the interface so that bugs are discovered earlier. > 1. Only a small sliver of human interactions are governed by formal > legal contracts read, understood, and agreed to by both (all) parties. Irrelevant. > 2. The idealized definition is naive in practice. Most legal contracts, > unlike the example in the link article, are written in language that > most people cannot read. Irrelevant. Dicts aren't actual paper books filled with definitions of words, floats don't actually float, neural nets are not made of neurons nor can you catch fish in them, and software contracts are code, not legally binding contracts. It is a *metaphor*. > Many contracts are imprecise and legally > ambiguous, which is why we have contract dispute courts. And even then, > the expense means that most people who feel violated in a transaction do > not use courts. Is this a critique of the legal system? What relevance does it have to Design By Contract? Honestly Terry, you seem to be arguing: "Hiring a lawyer is too expensive, and that's why Design By Contract doesn't work as a software methodology." > Post-conditions specify a function by result. I claim that this is not > always sensible. In this context, "result" can mean either "the value returned by the function" OR "the action performed by the function (its side-effect)". Post-conditions can check both. > I said above that functions may be specified by > process rather than result. Fine. What of it? Can you describe what the function does? "It sorts the list in place." "It deletes the given record from the database." "It deducts the given amount from Account A and transfers it to Account B, guaranteeing that either both transactions occur or neither of them, but never one and not the other." These are all post-conditions. Write them as code, and they are contracts. If you can't write them as code, okay, move on to the next function. (By the way, since you started off talking about mathematical functions, functions which perform a process rather than return a result aren't mathematical functions.) > Ironically, the contract metaphor > reinforces my claim. Many contracts, such as in teaching and medicine, > only specify process and explicitly disclaim any particular result of > concern to the client. Irrelevant. > >b)//If you write contracts in text, they will become stale over time > > Not true for good docstrings. We very seldom change the essential > meaning of public functions. What about public functions while they are still under active development with an unstable interface? > How has "Return the sine of x (measured in radians).", for math.sin, > become stale? Why would it ever? Of course a stable function with a fixed API is unlikely to change. What's your point? The sin() function implementation on many platforms probably hasn't changed in 10 or even 20 years. (It probably just calls the hardware routines.) Should we conclude that unit testing is therefore bunk and nobody needs to write unit tests? > What formal executable post condition > would help someone who does not understand 'sine', or 'sine of x'? Does x have to be a float? A number? (Decimal, Fraction, int, complex?) What if x is an INF or NAN? Can sin(x) return -0.0 rather than 0.0? Let's consider some related functions... Will tan(x) ever return NAN or INF? What does sqrt(x) do if x is negative? Will it raise? What are the possible valid arguments for the three-value form of pow(a, b, c)? [...] > Or consider "Stable sort *IN PLACE*." for list.sort. A formal > executable post-condition for this *can* be written. But for someone > who understands the words 'stable', 'sort', and 'in place', this compact > English post condition is much more readable than any formal version > could be. Not everybody knows what those terms mean. https://stackoverflow.com/questions/15125552/what-is-the-meaning-of-stable-and-unstable-for-various-sorting-algorithms?noredirect=1 https://stackoverflow.com/questions/1517793/what-is-stability-in-sorting-algorithms-and-why-is-it-important And even if they can read the comment, it is a comment, and therefore a lie until proven different. Post-conditions can be checked. Comments can't. > More over, that addition of 'stable', when we were ready to > do so, did not make "Sort *IN PLACE*" stale in any sense of being wrong. I don't understand what point you think you are making here. It is still stale -- the documentation is incomplete. Whether that is "wrong" or merely "stale" is a matter of how fine a line we wish to draw. (Is a *partially* true but incomplete statement right, or wrong?) > I said above that functions definitions and contracts can specify the > process rather than the result. For functions, the code is the process, > and the code may then be the specification and the contract. For instance, > > def append_first(seq): > "Append seq[0] to seq." [...] > But with duck-typing, no post condition is possible. That's incorrect. def append_first(seq): require: len(seq) > 0 hasattr(seq, "append") ensure: len(seq) == len(OLD.seq) + 1 seq[0] == seq[-1] Nothing in the above requires seq to be a specific type. It just needs to support the len, append and getitem interfaces. If one wanted to be *really* pedantic, we could add a O(N) check as well, requiring only slicing and equality: seq[0:-1] == OLD.seq but just because we CAN write a condition, doesn't mean we MUST write the condition. You still get to choose whether to check that as a contract, or a test, or both, or neither. -- Steve From rosuav at gmail.com Mon Oct 8 10:28:01 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Oct 2018 01:28:01 +1100 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <17E6FE52-5CFD-41AB-8807-4B08F9CFCCA6@killingar.net> References: <20181008101957.GA85076@cskk.homeip.net> <17E6FE52-5CFD-41AB-8807-4B08F9CFCCA6@killingar.net> Message-ID: On Mon, Oct 8, 2018 at 11:15 PM Anders Hovm?ller wrote: > > > However, another possibility is the the regexp is consuming lots of memory. > > The regexp seems simple enough (b'.'), so I doubt it is leaking memory like > mad; I'm guessing you're just seeing the OS page in as much of the file as it > can. > > > Yup. Windows will aggressively fill up your RAM in cases like this > because after all why not? There's no use to having memory just > sitting around unused. For read-only, non-anonymous mappings it's not > much problem for the OS to drop pages that haven't been recently > accessed and use them for something else. So I wouldn't be too > worried about the process chewing up RAM. > > I feel like this is veering more into python-list territory for > further discussion though. > > > Last time I worked on windows, which admittedly was a long time, the file cache was not attributed to a process, so this doesn't seem to be relevant to this situation. Depends whether it's a file cache or a memory-mapped file, though. On Linux, if I open a file, read it, then close it, I'm not using that file any more, but it might remain in cache (which will mean that re-reading it will be fast, regardless of whether that's from the same or a different process). That usage shows up as either "buffers" or "cache", and doesn't belong to any process. In contrast, a mmap'd file is memory that you do indeed own. If the system runs short of physical memory, it can simply discard those pages (rather than saving them to the swap file), but they're still owned by one specific process, and should count in that process's virtual memory. (That's based on my knowledge of Linux today and OS/2 back in the 90s. It may or may not be accurate to Windows, but I suspect it won't be very far wrong.) ChrisA From marko.ristin at gmail.com Mon Oct 8 11:04:27 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Mon, 8 Oct 2018 17:04:27 +0200 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: References: <20181008102603.GK3817@ando.pearwood.info> <20181008121108.GL3817@ando.pearwood.info> Message-ID: Hi Chris, I hope you don't mind me responding though you would like to stop participating. This message is meant for other readers in case they are interested. > Alice tests her package A with some test data D_A. Now assume Betty did > not write any contracts for her package B. When Alice tests her package, > she is actually making an integration test. While she controls the inputs > to B from A, she can only observe the results from B, but not whether they > are correct by coincidence or B did its job correctly. Let's denote D'_B > the data that is given to B from her original test data D_A during Alice's > integration testing. > > > > If you're regularly changing your function contracts, such that you > need to continually test in case something in the other package > changed, then yes, that's exactly what I'm talking about. > The user story I put above had nothing to do with change. I was telling how manually performing integration tests between A and B is tedious for us (since it involves some form or the other of manual recording of input/outputs to the module B and adapting unit tests of B) while contracts are much better (*for us*) since they incur little overhead (write them once for B, anybody runs them automatically). I did not want to highlight the *change* in my user story, but the ease of integration tests with contracts. If it were not for contracts, we would have never performed them. Cheers, Marko On Mon, 8 Oct 2018 at 16:22, Chris Angelico wrote: > On Mon, Oct 8, 2018 at 11:11 PM Steven D'Aprano > wrote: > > > > On Mon, Oct 08, 2018 at 09:32:23PM +1100, Chris Angelico wrote: > > > On Mon, Oct 8, 2018 at 9:26 PM Steven D'Aprano > wrote: > > > > > In other words, you change the *public interface* of your functions > > > > > all the time? How do you not have massive breakage all the time? > > > > > > > > I can't comment about Marko's actual use-case, but *in general* > > > > contracts are aimed at application *internal* interfaces, not so much > > > > library *public* interfaces. > > > > > > Yet we keep having use-cases shown to us involving one person with one > > > module, and another person with another module, and the interaction > > > between the two. > > > > Do we? I haven't noticed anything that matches that description, > > although I admit I haven't read every single post in these threads > > religiously. > > Try this: > > On Mon, Oct 8, 2018 at 5:11 PM Marko Ristin-Kaufmann > wrote: > > Alice tests her package A with some test data D_A. Now assume Betty did > not write any contracts for her package B. When Alice tests her package, > she is actually making an integration test. While she controls the inputs > to B from A, she can only observe the results from B, but not whether they > are correct by coincidence or B did its job correctly. Let's denote D'_B > the data that is given to B from her original test data D_A during Alice's > integration testing. > > > > If you're regularly changing your function contracts, such that you > need to continually test in case something in the other package > changed, then yes, that's exactly what I'm talking about. > > I'm tired of debating this. Have fun. If you love contracts so much, > marry them. I'm not interested in using them, because nothing in any > of these threads has shown me any good use-cases that aren't just > highlighting bad coding practices. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Oct 8 10:46:17 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 9 Oct 2018 01:46:17 +1100 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: References: <20181008102603.GK3817@ando.pearwood.info> <20181008121108.GL3817@ando.pearwood.info> Message-ID: <20181008144616.GN3817@ando.pearwood.info> On Tue, Oct 09, 2018 at 01:21:57AM +1100, Chris Angelico wrote: > > > Yet we keep having use-cases shown to us involving one person with one > > > module, and another person with another module, and the interaction > > > between the two. > > > > Do we? I haven't noticed anything that matches that description, > > although I admit I haven't read every single post in these threads > > religiously. > > Try this: Thanks for the example, that's from one of the posts I haven't read. > If you're regularly changing your function contracts, such that you > need to continually test in case something in the other package > changed, then yes, that's exactly what I'm talking about. Presumably you're opposed to continuous integration testing too. > I'm tired of debating this. Is that what you were doing? I had wondered. http://www.montypython.net/scripts/argument.php *wink* -- Steve From pradyunsg at gmail.com Mon Oct 8 11:51:17 2018 From: pradyunsg at gmail.com (Pradyun Gedam) Date: Mon, 8 Oct 2018 21:21:17 +0530 Subject: [Python-ideas] support toml for pyproject support In-Reply-To: <7fd0e3a3-34e2-321a-86d6-48af3504b539@netc.fr> References: <7fd0e3a3-34e2-321a-86d6-48af3504b539@netc.fr> Message-ID: On Mon, Oct 8, 2018 at 12:49 PM Jimmy Girardet wrote: > > Hi, Hi Jimmy and welcome! :) > > I don't know if this was already debated but I don't know how to search > in the whole archive of the list. > > > For now the adoption of pyproject.toml file is more difficult because > toml is not in the standard library. > > Each tool which wants to use pyproject.toml has to add a toml lib as a > conditional or hard dependency. > > Since toml is now the standard configuration file format, it's strange > the python does not support it in the stdlib lije it would have been > strange to not have the configparser module. > Let's wait till TOML hits 1.0 before adding it to the standard library. It's still at 0.5 right now. I am personally in favor of adding a standard library module for TOML, after it hits 1.0 and there's some stability after the release. > I know it's complicated to add more and more thing to the stdlib but I > really think it is necessary for python packaging being more consistent. > TOML has a fairly unambiguous specification so I don't think the choice of library should really affect what data gets loaded. If there are differences across implementations, due to the TOML specification unintentionally being ambiguous, please do file an issue on GitHub. :) > > Maybe we could thought to a readonly lib to limit the added code. I don't think that would be as helpful as possibly a round-tripping parser-writer combination but I'll refrain from pushing for that *right now*. > > > If it's conceivable, I'd be happy to help in it. > > > Nice Day guys and girls. > > Jimmy Cheers, Pradyun (pip maintainer, TOML Core member) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From guido at python.org Mon Oct 8 12:01:22 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 8 Oct 2018 09:01:22 -0700 Subject: [Python-ideas] support toml for pyproject support In-Reply-To: References: <7fd0e3a3-34e2-321a-86d6-48af3504b539@netc.fr> <20181008095551.GJ3817@ando.pearwood.info> Message-ID: On Mon, Oct 8, 2018 at 4:53 AM Erik Bray wrote: > If I had the energy to argue it I would also argue against using TOML > in those PEPs. I personally don't especially care for TOML and what's > "obvious" to Tom is not at all obvious to me. I'd rather just stick > with YAML or perhaps something even simpler than either one. > I feel the same way. (Somebody was requesting extensive TOML support for mypy and was also waving those PEPs in front of us.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mh-python at online.de Mon Oct 8 12:49:30 2018 From: mh-python at online.de (Marcus Harnisch) Date: Mon, 8 Oct 2018 18:49:30 +0200 Subject: [Python-ideas] Better error messages for missing optional stdlib packages In-Reply-To: References: <23943232-e980-11bd-04c4-157210c732d3@gmx.net> Message-ID: On 10/08/2018 12:29 AM, Terry Reedy wrote: > On 10/3/2018 4:29 PM, Marcus Harnisch wrote: > >> When trying to import lzma on one of my machines, I was suprised to >> get a normal import error like for any other module. > What was the traceback and message?? Did you get an import error for > one of the three imports in lzma.py.? I don't know why you would > expect anything else.? Any import in any stdlib module can potential > fail if the file is buggy, corrupted, or missing. $ /usr/bin/python3 Python 3.7.0 (default, Oct? 4 2018, 03:21:59) [GCC 8.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import lzma Traceback (most recent call last): ? File "", line 1, in ? File "/usr/lib/python3.7/lzma.py", line 27, in ModuleNotFoundError: No module named '_lzma' >>> > > According to the docs >> lzma has been part of stdlib since 3.3. Further digging revealed that >> the error is due to the fact that xz wasn't compiled in when building >> Python. > > Perhaps this is a buggy build. This, I reckon, depends on the perspective and the definition of ?buggy?. If the build process finishes without error, can we assume that the build is not buggy? If we make claims along the lines of ?nobody in their right mind would build Python without lzma? it would only be fair to break the build if liblzma can't be detected. Unless I missed anything it doesn't happen until after the build has finished successfully, that a message is printed which lists the modules which couldn't be detected by setup.py. Here is a list of modules, which I believe are affected: $ grep -F missing.append setup.py ??????????? missing.append('spwd') ??????????? missing.append('readline') ??????????? missing.append('_ssl') ??????????? missing.append('_hashlib') ??????????? missing.append('_sqlite3') ??????????????? missing.append('_dbm') ??????????? missing.append('_gdbm') ??????????? missing.append('nis') ??????????? missing.append('_curses') ??????????? missing.append('_curses_panel') ??????????????????? missing.append('zlib') ??????????????? missing.append('zlib') ??????????? missing.append('zlib') ??????????? missing.append('_bz2') ??????????? missing.append('_lzma') ??????????? missing.append('_elementtree') ??????????? missing.append('ossaudiodev') ??????????? missing.append('_tkinter') ??????????? missing.append('_uuid') > ? Have you complained to the distributor? After finding the root cause of the missing import I did file a request for including lzma in future releases of the distribution. All I am asking is that unsuspecting users not be left in the dark when it comes to debugging unexpected import errors. I believe a missing stdlib module qualifies for ?unexpected?. This could happen in form of documentation or by means of an import error handler that prints some helpful message in case that a stdlib module couldn't be found. Regards, Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Oct 8 15:34:51 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 8 Oct 2018 15:34:51 -0400 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: <20181008142632.GM3817@ando.pearwood.info> References: <20181008142632.GM3817@ando.pearwood.info> Message-ID: On 10/8/2018 10:26 AM, Steven D'Aprano wrote: > On Sun, Oct 07, 2018 at 04:24:58PM -0400, Terry Reedy wrote: >> https://www.win.tue.nl/~wstomv/edu/2ip30/references/design-by-contract/index.html >> >> defines contracts as "precise (legally unambiguous) specifications" (5.2 >> Business Contracting/Sub-contracting Metaphor) > > You are quoting that out of context. The full context says (emphasis > added): > > IN THE BUSINESS WORLD, contracts are precise (legally unambiguous) > specifications that define the obligations and benefits of the > (usually two) parties involved. This is silly. Every quote that is not complete is literally 'out of context'. However, 'quoting out of context', in the colloquial sense, means selectively quoting so as to distort the original meaning, whereas I attempted to focus on the core meaning I was about to discuss. Marko asked an honest question about why things obvious to him are not obvious to others. I attempted to give an honest answer. If my answer suggested that I have not undertstood Marko properly, as is likely, he can use it as a hint as to how communicate his position better. >> I said above that functions may be specified by >> process rather than result. > > Fine. What of it? Can you describe what the function does? > > "It sorts the list in place." > > "It deletes the given record from the database." > These are all post-conditions. No they are not. They are descriptions of the process. Additional mental work is required to turn them into formal descriptions of the result that can be coded. Marko appears to claim that such coded formal descriptions are easier to read and understand than the short English description. I disagree. It is therefore not obvious to me that the extra work is worthwhile. >> def append_first(seq): >> "Append seq[0] to seq." > [...] The snipped body (revised to omit irrelevant 'return') seq.append(seq[0]) >> But with duck-typing, no post condition is possible. > > That's incorrect. > > def append_first(seq): > require: > len(seq) > 0 seq does not neccessarily have a __len__ method > hasattr(seq, "append") The actual precondition is that seq[0] be in the domain of seq.append. The only absolutely sure way to test this is to run the code body. Or one could run seq[0] and check it against the preconditions, if formally specified, of seq.append. > ensure: > len(seq) == len(OLD.seq) + 1 > seq[0] == seq[-1] Not even all sequences implement negative indexing. This is true for lists, as I said, but not for every object the meets the preconditions. As others have said, duck typing means that we don't know what unexpected things methods of user-defined classes might do. class Unexpected(): def __init__(self, first): self.first = first def __getitem__(self, key): if key == 0: return self.first else: raise ValueError(f'key {key} does not equal 0') def append(self, item): if isinstance(item, int): self.last = item else: raise TypeError(f'item {item} is not an int') def append_first(seq): seq.append(seq[0]) x = Unexpected(42) append_first(x) print(x.first, x.last) # 42 42 A more realistic example: def add(a, b): return a + b The simplified precondition is that a.__add__ exists and applies to b or that b.__radd__ exists and applies to a. I see no point in formally specifying this as part of 'def add' as it is part of the language definition. It is not just laziness that makes me averse to such redundancy. Even ignoring user classes, a *useful* post-condition that applies to both numbers and sequences is hard to write. I believe + is distributive for both, so that a + (b + b) = (a + b) + b, but -- Terry Jan Reedy From amjadhedhili at outlook.com Mon Oct 8 15:46:53 2018 From: amjadhedhili at outlook.com (Amjad Ben Hedhili) Date: Mon, 8 Oct 2018 19:46:53 +0000 Subject: [Python-ideas] add a time decorator to timeit.py In-Reply-To: References: <20181007111527.GE3817@ando.pearwood.info> <20181007123410.GF3817@ando.pearwood.info> , Message-ID: > Summary: Python's timeit.timeit() has an undocumented feature / > implementation detail that gives much of what the original poster has > asked for. Perhaps revising the docs will solve the problem. although timeit can be used with a callable, you need to create a lambda expression if the function has args: ``` def func_to_time(a, b): ... timeit.timeit(lambda: func_to_time(a, b), globals=globals()) ``` and you can't use it as a decorator. ________________________________ De : Python-ideas de la part de Jonathan Fine Envoy? : dimanche 7 octobre 2018 09:15 ? : python-ideas Objet : Re: [Python-ideas] add a time decorator to timeit.py Summary: Python's timeit.timeit() has an undocumented feature / implementation detail that gives much of what the original poster has asked for. Perhaps revising the docs will solve the problem. This thread has prompted me to look at timeit again. Usually, I look at the command line help first. >>> import timeit >>> help(timeit) Classes: Timer Functions: timeit(string, string) -> float repeat(string, string) -> list default_timer() -> float This time, to my surprise, I found the following works: >>> def fn(): return 2 + 2 >>> timeit.timeit(fn) 0.10153918000287376 Until today, as I recall, I didn't know this. Now for: https://docs.python.org/3/library/timeit.html I don't see any examples there, that show that timeit.timeit can take a callable as its first argument. So my ignorance can, I hope be forgiven. Now for: https://github.com/python/cpython/blob/3.7/Lib/timeit.py#L100 This contains, for both the stmt and setup parameters, explicit tests such as if isinstance(stmt, str): # string case elif callable(stmt): # callable case So I think it's an undocumented feature, rather than an implementation detail. And if you're a software historian, now perhaps look at https://github.com/python/cpython/commits/3.7/Lib/timeit.py And also, if you wish, for the tests for timeit.py. -- Jonathan _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at cskk.id.au Mon Oct 8 16:30:39 2018 From: cs at cskk.id.au (cs at cskk.id.au) Date: Tue, 9 Oct 2018 07:30:39 +1100 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: Message-ID: <20181008203039.GA91809@cskk.homeip.net> On 08Oct2018 13:36, Ram Rachum wrote: >I'm not an expert on memory. I used Process Explorer to look at the >Process. The Working Set of the current run is 11GB. The Private Bytes is >708MB. Actually, see all the info here: >https://www.dropbox.com/s/tzoud028pzdkfi7/screenshot_TURING_2018-10-08_133355.jpg?dl=0 And the process' virtual size is about 353GB, which matches having your file mmaped (its contents is now part of your process virtual memory space). >I've got 16GB of RAM on this computer, and Process Explorer says it's >almost full, just ~150MB left. This is physical memory. I'd say this is expected behaviour. As you access the memory it is paged into physical memory, and since it may be wanted again (the OS can't tell) it isn't paged out until that becomes necessary to make room for other virtual pages. I suspect (but would need to test to find out) that sequentially reading the file instead of memory mapping it might not be so aggressive because your process would be reusing that same small pool of memory to hold data as you scan the file. Cheers, Cameron Simpson From robert.hoelzl at posteo.de Mon Oct 8 18:02:10 2018 From: robert.hoelzl at posteo.de (robert.hoelzl at posteo.de) Date: Tue, 09 Oct 2018 00:02:10 +0200 Subject: [Python-ideas] Introduce typing.SupportsFsPath Message-ID: Hello, Since __fspath__ was introduced in PEP 519 it is possible to create object classes that are representing file system paths. But there is no corresponding type object in the "typing" module. Thus I cannot specify functions, that accept any kind of object which supports the __fspath__ protocol. Please note that "Path" is not a replacement for "SupportsFsPath", since the concept of PEP 519 is, that I could implement new objects (without dependency to "Path") that are implementing the __fspath__ protocol. robert From guido at python.org Mon Oct 8 21:12:18 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 8 Oct 2018 18:12:18 -0700 Subject: [Python-ideas] Introduce typing.SupportsFsPath In-Reply-To: References: Message-ID: In typeshed there is os.PathLike which is close. You should be able to use Union[str, os.PathLike[str]] for what you want (or define an alias). We generally don't want to add more things to typing that aren't closely related to the type system. (Adding the io and re classes was already less than ideal, and we don't want to do more of those.) On Mon, Oct 8, 2018 at 3:10 PM wrote: > Hello, > > Since __fspath__ was introduced in PEP 519 it is possible to create > object classes that are representing file system paths. > But there is no corresponding type object in the "typing" module. Thus I > cannot specify functions, that accept any kind of object which supports > the __fspath__ protocol. > > Please note that "Path" is not a replacement for "SupportsFsPath", since > the concept of PEP 519 is, that I could implement new objects (without > dependency to "Path") > that are implementing the __fspath__ protocol. > > robert > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Oct 9 03:23:16 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 9 Oct 2018 18:23:16 +1100 Subject: [Python-ideas] Why is design-by-contracts not widely adopted? In-Reply-To: References: <20181008142632.GM3817@ando.pearwood.info> Message-ID: <20181009072314.GT3817@ando.pearwood.info> On Mon, Oct 08, 2018 at 03:34:51PM -0400, Terry Reedy wrote: > >> I said above that functions may be specified by > >> process rather than result. > > > > Fine. What of it? Can you describe what the function does? > > > > "It sorts the list in place." > > > > "It deletes the given record from the database." > > > These are all post-conditions. > > No they are not. Yes they are; they are textual descriptions of the post-condition, not executable code, but since software contracts are both specification and code, they certainly count as post-conditions. By the way, you seem to have deleted one of my examples, leaving only the two relatively simple ones. I don't understand why you deleted the example, but here it is again: "It deducts the given amount from Account A and transfers it to Account B, guaranteeing that either both transactions occur or neither of them, but never one and not the other." > They are descriptions of the process. Additional > mental work is required to turn them into formal descriptions of the > result that can be coded. Yes. That's called "programming". You might have done it from time to time :-) > Marko appears to claim that such coded formal > descriptions are easier to read and understand than the short English > description. I disagree. What does "sort in place"? *Precisely*? How do you know if the software *actually does* sort in place? They are not rhetorical questions. You can document that the function sorts in place, but if it actually reverses a copy of the list, how would you know? How do we know the software meets the specified requirements if you haven't nailed down the specification, and if the specification exists only in documentation and is never checked in the code? Design By Contract is a methodology that helps ensure that the specification is (1) nailed down and (2) actually checked. > It is therefore not obvious to me that the > extra work is worthwhile. Tell me how you, personally, would find out whether or not the function actually sorts the list in place. What process would YOU do? As I see it, you either have to take it on trust, or you have to write tests. Would you argue that the "extra work" to write tests are not worthwhile? > >>def append_first(seq): > >> "Append seq[0] to seq." > >[...] > > The snipped body (revised to omit irrelevant 'return') > seq.append(seq[0]) > > >>But with duck-typing, no post condition is possible. > > > >That's incorrect. > > > >def append_first(seq): > > require: > > len(seq) > 0 > > seq does not neccessarily have a __len__ method Yes it must, because I wrote the specification and I demand that it must support len. You lost your opportunity to write the spec yourself when you wrongly said no post-condition is possible. I just showed that post-conditions ARE possible, by writing some. We could debate what those conditions are: can we just test for the Sequence interface, using isinstance() and the Sequence ABC? Do I need to bother checking that seq supports __getitem__? But I'm not really interested in getting bogged down in the minutia of what specific post-conditions need to go with this toy function. Its a made-up example, so who cares? In practice, you specify in as much detail as you need. You can always come back and add more detail later. > > hasattr(seq, "append") > > The actual precondition is that seq[0] be in the domain of seq.append. That is not part of the contract of the append_first function. It doesn't care about any domain limits of seq. If an object could be appended into seq[0] in the first place, surely it can append again later. We don't have to try to detect every imaginable bizarre weird piece of code intentionally written to perplex us. > The only absolutely sure way to test this is to run the code body. Sure. So what? The code body does actually get run you know. If it raises an exception because seq can't append seq[0] to itself, that's a bug that needs to be fixed. > Or > one could run seq[0] and check it against the preconditions, if formally > specified, of seq.append. That is not the responsibility of the append_first function. > > ensure: > > len(seq) == len(OLD.seq) + 1 > > seq[0] == seq[-1] > > Not even all sequences implement negative indexing. I don't care. Since I wrote the spec, I demand that only those which meet my requirements are used. But okay, fine, I'll re-write it: seq[0] == seq[len(seq)-1] and then you can tell me off for writing un-Pythonic code, and then I'll change it back. > This is true for lists, as I said, but not for every object the meets > the preconditions. As others have said, duck typing means that we don't > know what unexpected things methods of user-defined classes might do. That's not how we use duck-typing. When I call len(obj), I don't expect that it will format my hard drive, even though obj.__len__ might do that, and if it did, I would treat it as a bug. Why am I passing such an object as input? That makes no sense. I *don't want* that behaviour, so why would I pass an object that does that? I wouldn't say "Oh well, that means that formatting my head drive must be the correct behaviour for my application, good thing I have backups." I would say "That's a bug. Fix it." Here is the scenario: I am writing an application that has some specific purpose. I don't know, let's say it is a payroll application. It talks to a database and generates payroll reports or something. Whatever. The point is it has some *concrete* purpose, it's not just Mickey Mouse toy functions. As part of that application, I need (for some unspecified reason!) to take a sequence, extract the first item, and append it to the end of the sequence. Hence you append_first function. Why? Who knows. Let's say it makes sense in context. I trust you. In context, the sequence I pass to that function is not going to be "some arbitrary object that could do absolutely anything". It is going to be from a tiny subset of *useful* sequences, specifically the ones which actually do append the first item to the end of sequence. Why would I write, and then use, a class that does something unhelpful? Am I a malicious coder trying to get myself fired? The purpose of the contract is to detect ACCIDENTAL bugs early, not to defend against intentional self-sabotage. -- Steve From levkivskyi at gmail.com Tue Oct 9 06:16:00 2018 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Tue, 9 Oct 2018 12:16:00 +0200 Subject: [Python-ideas] Introduce typing.SupportsFsPath In-Reply-To: References: Message-ID: On Tue, 9 Oct 2018 at 03:13, Guido van Rossum wrote: > In typeshed there is os.PathLike which is close. You should be able to use > Union[str, os.PathLike[str]] for what you want (or define an alias). > > We generally don't want to add more things to typing that aren't closely > related to the type system. (Adding the io and re classes was already less > than ideal, and we don't want to do more of those.) > The problem however is that `PathLike` is not a protocol in typeshed. This should be updated when protocols will be official. Until that, you can just easily define your own protocol: from typing import AnyStr from typing_extensions import Protocol class PathLike(Protocol[AnyStr]): def __fspath__(self) -> AnyStr: ... -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Oct 9 07:04:28 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 10 Oct 2018 00:04:28 +1300 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: <20181008101957.GA85076@cskk.homeip.net> <17E6FE52-5CFD-41AB-8807-4B08F9CFCCA6@killingar.net> Message-ID: <5BBC8B3C.70308@canterbury.ac.nz> Chris Angelico wrote: > In contrast, a mmap'd file is memory that you do indeed own. Although it's not really accurate to say that it's owned by a particular process. If two processes mmap the same file, the physical memory pages holding it appear in the address spaces of both processes. -- Greg From rosuav at gmail.com Tue Oct 9 07:10:45 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Oct 2018 22:10:45 +1100 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <5BBC8B3C.70308@canterbury.ac.nz> References: <20181008101957.GA85076@cskk.homeip.net> <17E6FE52-5CFD-41AB-8807-4B08F9CFCCA6@killingar.net> <5BBC8B3C.70308@canterbury.ac.nz> Message-ID: On Tue, Oct 9, 2018 at 10:05 PM Greg Ewing wrote: > > Chris Angelico wrote: > > In contrast, a mmap'd file is memory that you do indeed own. > > Although it's not really accurate to say that it's owned by > a particular process. If two processes mmap the same file, > the physical memory pages holding it appear in the address > spaces of both processes. > Yeah, which is a constant problem when you ask "why am I out of memory". Same thing happens if you have any other (non-mmap'd) pages and then you fork, or if two processes are using the same executable (though I think that's actually mmap again), etc, etc, etc. Tell me, which process is responsible for libc being in memory? Other than, like, all of them? ChrisA From ericfahlgren at gmail.com Tue Oct 9 10:16:34 2018 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Tue, 9 Oct 2018 07:16:34 -0700 Subject: [Python-ideas] Introduce typing.SupportsFsPath In-Reply-To: References: Message-ID: On Tue, Oct 9, 2018 at 3:16 AM Ivan Levkivskyi wrote: > class PathLike(Protocol[AnyStr]): > I had been working on this same problem intermittently for several months, so thanks, but... error: Invariant type variable 'AnyStr' used in protocol where covariant one is expected is called out on the class by mypy 0.630 (Python 3.6.6). Do I just need to wait for 0.640? Or should I define a new TypeVar for AnyStr_co and use that? -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Oct 9 11:42:04 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 10 Oct 2018 00:42:04 +0900 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: <20181008101957.GA85076@cskk.homeip.net> <17E6FE52-5CFD-41AB-8807-4B08F9CFCCA6@killingar.net> <5BBC8B3C.70308@canterbury.ac.nz> Message-ID: <23484.52300.242521.93443@turnbull.sk.tsukuba.ac.jp> Chris Angelico writes: > On Tue, Oct 9, 2018 at 10:05 PM Greg Ewing wrote: > > > > Chris Angelico wrote: > > > In contrast, a mmap'd file is memory that you do indeed own. > > > > Although it's not really accurate to say that it's owned by > > a particular process. If two processes mmap the same file, > > the physical memory pages holding it appear in the address > > spaces of both processes. Subject to COW, I presume. Probably in units smaller than the whole file (per page?) > Tell me, which process is responsible for libc being in memory? > Other than, like, all of them? Yes. Why would you want a different answer? From rosuav at gmail.com Tue Oct 9 11:46:31 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 10 Oct 2018 02:46:31 +1100 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <23484.52300.242521.93443@turnbull.sk.tsukuba.ac.jp> References: <20181008101957.GA85076@cskk.homeip.net> <17E6FE52-5CFD-41AB-8807-4B08F9CFCCA6@killingar.net> <5BBC8B3C.70308@canterbury.ac.nz> <23484.52300.242521.93443@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, Oct 10, 2018 at 2:42 AM Stephen J. Turnbull wrote: > > Chris Angelico writes: > > On Tue, Oct 9, 2018 at 10:05 PM Greg Ewing wrote: > > > > > > Chris Angelico wrote: > > > > In contrast, a mmap'd file is memory that you do indeed own. > > > > > > Although it's not really accurate to say that it's owned by > > > a particular process. If two processes mmap the same file, > > > the physical memory pages holding it appear in the address > > > spaces of both processes. > > Subject to COW, I presume. Probably in units smaller than the whole > file (per page?) Both processes are using the virtual memory. Either or both could be using physical memory. Assuming they haven't written to the pages (which is the case with executables - the system mmaps the binary into your memory space as read-only), and assuming that those pages are backed by physical memory, which process is using that memory? > > Tell me, which process is responsible for libc being in memory? > > Other than, like, all of them? > > Yes. Why would you want a different answer? Because that would mean that I have way more *physical* memory in use than I actually have chips on the motherboard for. Yes, virtual memory can be over-allocated, but physical memory can't. How do you measure where your physical memory is being used, if adding up the usage of all processes exceeds the available resources? It's like trying to figure out which countries the people of this planet live in, and saying that there are X billion in China, Y million in Australia, etc, etc, etc, and then discovering that you're counting people multiple times if they ever move... which means that there are ten or twenty billion people on the planet. ChrisA From chris.barker at noaa.gov Tue Oct 9 11:55:55 2018 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 9 Oct 2018 08:55:55 -0700 Subject: [Python-ideas] support toml for pyproject support In-Reply-To: References: <7fd0e3a3-34e2-321a-86d6-48af3504b539@netc.fr> <20181008095551.GJ3817@ando.pearwood.info> Message-ID: If I had the energy to argue it I would also argue against using TOML > in those PEPs. I partook in that discussion, and I still have no idea why TOML was chosen, over, say, a defined subset of YAML, or a slightly extended JSON. But the folks that were highly invested and putting the work in made a choice, so here we are. But if it?s in the PEPs, it seems time to define a version used ( 1.0 would be good, but often that?s actually pretty arbitrary) and get an implementation into the stdlib. If the TOML folks don?t think it?s stable enough for that, I?ve got to wonder if it was a good choice! We?re have enough trouble with really slow adoption of the new ways of doing packaging, not even providing the tools seems to really defeat the purpose. -CHB -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Oct 9 12:05:36 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 9 Oct 2018 18:05:36 +0200 Subject: [Python-ideas] support toml for pyproject support References: <7fd0e3a3-34e2-321a-86d6-48af3504b539@netc.fr> <20181008095551.GJ3817@ando.pearwood.info> <0A82538D-0BC0-4A6E-90B8-247252301807@killingar.net> Message-ID: <20181009180536.0fbba705@fsol> On Mon, 8 Oct 2018 09:26:12 -0400 David Mertz wrote: > I agree here. I briefly urged against using the less used TOML format, but > I have no real skin in the game around packaging. I like YAML, but that's > also not in the standard library, even if more widely used. Agreed with David. Also, please note that one argument against YAML is its complexity, which logically also entails implementation complexity (and therefore makes it less likely that a YAML implementation would enter the stdlib). Regards Antoine. From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Oct 9 14:09:56 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 10 Oct 2018 03:09:56 +0900 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: <20181008101957.GA85076@cskk.homeip.net> <17E6FE52-5CFD-41AB-8807-4B08F9CFCCA6@killingar.net> <5BBC8B3C.70308@canterbury.ac.nz> <23484.52300.242521.93443@turnbull.sk.tsukuba.ac.jp> Message-ID: <23484.61172.389958.589@turnbull.sk.tsukuba.ac.jp> Chris Angelico writes: > Both processes are using the virtual memory. Either or both could be > using physical memory. Assuming they haven't written to the pages > (which is the case with executables - the system mmaps the binary into > your memory space as read-only), and assuming that those pages are > backed by physical memory, which process is using that memory? One doesn't know. Clever side-channel attacks aside, I don't care, and I don't see how it could matter. > > > Tell me, which process is responsible for libc being in memory? > > > Other than, like, all of them? > > > > Yes. Why would you want a different answer? > > Because that would mean that I have way more *physical* memory in use > than I actually have chips on the motherboard for. No, that's like saying that because you have multiple links to a file on disk you're using more physical disk than you have. > Yes, virtual memory can be over-allocated, but physical memory > can't. How do you measure where your physical memory is being used, > if adding up the usage of all processes exceeds the available > resources? lsof can do it, I guess. The kernel knows which pages of which processes' virtual memory are backed by physical memory. But just as running over the array of inodes tells you how much physical disk is allocated, without looking at directories to find out what it's allocated to, the kernel probably has a bitmap or similar so it knows which pages of physical memory are in use. All problems are easy once you have enough indirection. :^) From rosuav at gmail.com Tue Oct 9 14:12:42 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 10 Oct 2018 05:12:42 +1100 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <23484.61172.389958.589@turnbull.sk.tsukuba.ac.jp> References: <20181008101957.GA85076@cskk.homeip.net> <17E6FE52-5CFD-41AB-8807-4B08F9CFCCA6@killingar.net> <5BBC8B3C.70308@canterbury.ac.nz> <23484.52300.242521.93443@turnbull.sk.tsukuba.ac.jp> <23484.61172.389958.589@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, Oct 10, 2018 at 5:09 AM Stephen J. Turnbull wrote: > > Chris Angelico writes: > > > Both processes are using the virtual memory. Either or both could be > > using physical memory. Assuming they haven't written to the pages > > (which is the case with executables - the system mmaps the binary into > > your memory space as read-only), and assuming that those pages are > > backed by physical memory, which process is using that memory? > > One doesn't know. Clever side-channel attacks aside, I don't care, > and I don't see how it could matter. It matters a lot when you're trying to figure out what your system is doing. > > > > Tell me, which process is responsible for libc being in memory? > > > > Other than, like, all of them? > > > > > > Yes. Why would you want a different answer? > > > > Because that would mean that I have way more *physical* memory in use > > than I actually have chips on the motherboard for. > > No, that's like saying that because you have multiple links to a file > on disk you're using more physical disk than you have. Actually, that's exactly the same problem, with exactly the same consequences. How do you figure out why your disk is full? How do you enforce disk quotas? How can you get any sort of reasonable metrics about anything when the sum of everything vastly exceeds the actual usage? ChrisA From greg.ewing at canterbury.ac.nz Tue Oct 9 16:58:56 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 10 Oct 2018 09:58:56 +1300 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <23484.52300.242521.93443@turnbull.sk.tsukuba.ac.jp> References: <20181008101957.GA85076@cskk.homeip.net> <17E6FE52-5CFD-41AB-8807-4B08F9CFCCA6@killingar.net> <5BBC8B3C.70308@canterbury.ac.nz> <23484.52300.242521.93443@turnbull.sk.tsukuba.ac.jp> Message-ID: <5BBD1690.4020803@canterbury.ac.nz> Stephen J. Turnbull wrote: > Subject to COW, I presume. Probably in units smaller than the whole > file (per page?) It can be COW or not, depending on the options passed to mmap. And yes, it's mapped in units of pages. -- Greg From cs at cskk.id.au Tue Oct 9 17:25:46 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 10 Oct 2018 08:25:46 +1100 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <23484.52300.242521.93443@turnbull.sk.tsukuba.ac.jp> References: <23484.52300.242521.93443@turnbull.sk.tsukuba.ac.jp> Message-ID: <20181009212546.GA37654@cskk.homeip.net> On 10Oct2018 00:42, Stephen J. Turnbull wrote: >Chris Angelico writes: > > On Tue, Oct 9, 2018 at 10:05 PM Greg Ewing wrote: > > > Chris Angelico wrote: > > > > In contrast, a mmap'd file is memory that you do indeed own. > > > > > > Although it's not really accurate to say that it's owned by > > > a particular process. If two processes mmap the same file, > > > the physical memory pages holding it appear in the address > > > spaces of both processes. > >Subject to COW, I presume. Probably in units smaller than the whole >file (per page?) Yes, pages (whatever using the memory paging service works in). But not COW. Changes to the memory affect the file contents and are visible to the other process. It is mapped, not pseudocopied. Cheers, Cameron Simpson From turnbull.stephen.fw at u.tsukuba.ac.jp Wed Oct 10 01:27:56 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 10 Oct 2018 14:27:56 +0900 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: References: <20181008101957.GA85076@cskk.homeip.net> <17E6FE52-5CFD-41AB-8807-4B08F9CFCCA6@killingar.net> <5BBC8B3C.70308@canterbury.ac.nz> <23484.52300.242521.93443@turnbull.sk.tsukuba.ac.jp> <23484.61172.389958.589@turnbull.sk.tsukuba.ac.jp> Message-ID: <23485.36316.625236.863737@turnbull.sk.tsukuba.ac.jp> Chris Angelico writes: > On Wed, Oct 10, 2018 at 5:09 AM Stephen J. Turnbull > wrote: > > > > Chris Angelico writes: > > > > > Both processes are using the virtual memory. Either or both could be > > > using physical memory. Assuming they haven't written to the pages > > > (which is the case with executables - the system mmaps the binary into > > > your memory space as read-only), and assuming that those pages are > > > backed by physical memory, which process is using that memory? > > > > One doesn't know. Clever side-channel attacks aside, I don't care, > > and I don't see how it could matter. > > It matters a lot when you're trying to figure out what your system > is doing. Sure, but knowing how your system works is far more important. Eg, create a 1TB file on a POSIX system, delete it while a process still has it opened, and it doesn't matter how you process the output of du or ls, you still have 1TB of used file space not accounted for. The same applies to swapfiles. But "df" knows and will tell you. In fact, "ps" will tell you how much shared memory a process is using. I just don't see a problem here, on the "I'm not getting the data I need" side. You do have access to the data you need. > > > > > Tell me, which process is responsible for libc being in memory? > > > > > Other than, like, all of them? > > > > > > > > Yes. Why would you want a different answer? > > > > > > Because that would mean that I have way more *physical* memory in use > > > than I actually have chips on the motherboard for. > > > > No, that's like saying that because you have multiple links to a file > > on disk you're using more physical disk than you have. > > Actually, that's exactly the same problem, with exactly the same > consequences. How do you figure out why your disk is full? How do you > enforce disk quotas? How can you get any sort of reasonable metrics > about anything when the sum of everything vastly exceeds the actual > usage? You add up the right things, of course, and avoid paradoxes. The disk quota enforcement problem is indeed hard. This sounds to me like a special problem studied in cost accounting, a problem which was solved (a computation that satisfies certain axioms was shown to exist and be unique) in a sense by Aumann and Shapley in the 1970s. The A-S prices have been used by telephone carriers to allocate costs of fixed assets with capacity constraints to individual calls, though I don't know if the method is still in use. I'm not sure if the disk quota problem fits the A-S theorem (which imposes certain monotonicity conditions), but the general paradigm does. However, the quota problem (and in general, the problem of allocation of overhead) is "hard" even if you have complete information about the system, because it's a values problem: what events are bad? what events are worse? what events are unacceptable (result in bankruptcy and abandonment of the system)? Getting very complete, accurate information about the physical consequences of individual events in the system (linking to a file on disk, allocating a large quantity of viritual memory) is not difficult, in the sense that you throw money and engineers at it, and you get "df". Getting very complete, accurate information about the values you're trying to satisfy is possible only for an omniscient god, even if, as in business, they can be measured in currency units. Steve From boxed at killingar.net Wed Oct 10 02:56:13 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Wed, 10 Oct 2018 08:56:13 +0200 Subject: [Python-ideas] add a time decorator to timeit.py In-Reply-To: References: <20181007111527.GE3817@ando.pearwood.info> <20181007123410.GF3817@ando.pearwood.info> Message-ID: > On 7 Oct 2018, at 22:44, Guido van Rossum wrote: > > So someone ought to submit a PR that adds (brief) documentation for this, with reference to this thread. I was trying to write this documentation when I noticed that the docs already mention this! "The stmt and setup parameters can also take objects that are callable without arguments." Maybe we should reword this? Obviously many people here have looked at the docs and not spotted it. I'll post a PR with lambda examples under "Basic Examples", I think this should make people take notice of this feature. / Anders From daveshawley at gmail.com Wed Oct 10 07:10:55 2018 From: daveshawley at gmail.com (David Shawley) Date: Wed, 10 Oct 2018 07:10:55 -0400 Subject: [Python-ideas] async unittest.TestCase Message-ID: Hi everyone and good morning to some of you, Since asyncio and the async/await syntax are both part of Python, I think that we should extend TestCase to support it. The simplest solution that I can think of is to create unittest.AsyncTestCase sub-class with the following extensions: - create a new event loop and install it in AsyncTestCase.run - make it possible for setUp, tearDown, and test methods to be async by calling asyncio.iscoroutinefunction I wrote my own in a local test before noticing that Martin Richard had already written and published asynctest [1]. Since then I found the following projects as well: - https://github.com/kwarunek/aiounittest - https://github.com/pytest-dev/pytest-asyncio I think that the community as a whole would benefit from basic support in unittest for async/await test "user methods". I am personally fond of the approach of extending unittest.TestCase in asynctest [2] over some of the other approaches. Is this something that we want in our Standard Library? - dave -- [1]: https://github.com/Martiusweb/asynctest [2]: https://github.com/Martiusweb/asynctest/blob/master/asynctest/case.py -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfine2358 at gmail.com Wed Oct 10 11:24:34 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Wed, 10 Oct 2018 16:24:34 +0100 Subject: [Python-ideas] Paul Romer, 2018 Economics Nobel Laureate, uses Python and Jupyter Message-ID: Terry Reedy wrote (to comp.lang.python) > https://paulromer.net/jupyter-mathematica-and-the-future-of-the-research-paper/ > Jupyter, Mathematica, and the Future of the Research Paper > Paul Romer, new Nobel prize winner in economics, for research on how > ideas interact with economic growth, explained last April why he has > switched from Mathematica to Jupyter. Well done, Terry, for spotting this. I hope you don't mind, I've changed the subject to give Paul Romer star billing. I think he deserves it. Here's some URLs on Romer and Python. https://qz.com/1417145/economics-nobel-laureate-paul-romer-is-a-python-programming-convert/ https://developers.slashdot.org/story/18/10/09/0042240/economics-nobel-laureate-paul-romer-is-a-python-programming-convert https://www.reddit.com/r/Python/comments/9mhxq2/this_years_nobel_prize_in_economics_was_awarded/ https://news.ycombinator.com/item?id=18173812&ref=hvper.com&utm_source=hvper.com&utm_medium=website https://www.wsj.com/articles/nobel-in-economics-goes-to-american-pair-1538992672 And some related URLs https://www.nature.com/articles/d41586-018-01322-9 # Future of online publishing https://pypi.org/project/nobel/ # Python interface to Nobel Prize API! https://jfine2358.github.io/slides/2018-nature-jupyter-altair-vega-binder.html And some Python code: >>> import nobel >>> api = nobel.Api() >>> api.prizes.filter(year=2018, category='economics')[0].laureates[1].surname u'Romer' -- Jonathan From geoffspear at gmail.com Wed Oct 10 11:56:47 2018 From: geoffspear at gmail.com (Geoffrey Spear) Date: Wed, 10 Oct 2018 11:56:47 -0400 Subject: [Python-ideas] Paul Romer, 2018 Economics Nobel Laureate, uses Python and Jupyter In-Reply-To: References: Message-ID: On Wed, Oct 10, 2018 at 11:29 AM Jonathan Fine wrote: > Terry Reedy wrote (to comp.lang.python) > > > > https://paulromer.net/jupyter-mathematica-and-the-future-of-the-research-paper/ > > Jupyter, Mathematica, and the Future of the Research Paper > > Paul Romer, new Nobel prize winner in economics, for research on how > > ideas interact with economic growth, explained last April why he has > > switched from Mathematica to Jupyter. > > Well done, Terry, for spotting this. I hope you don't mind, I've > changed the subject to give Paul Romer star billing. I think he > deserves it. > > Here's some URLs on Romer and Python. > > > https://qz.com/1417145/economics-nobel-laureate-paul-romer-is-a-python-programming-convert/ > > https://developers.slashdot.org/story/18/10/09/0042240/economics-nobel-laureate-paul-romer-is-a-python-programming-convert > > https://www.reddit.com/r/Python/comments/9mhxq2/this_years_nobel_prize_in_economics_was_awarded/ > > https://news.ycombinator.com/item?id=18173812&ref=hvper.com&utm_source=hvper.com&utm_medium=website > > https://www.wsj.com/articles/nobel-in-economics-goes-to-american-pair-1538992672 > > And some related URLs > > https://www.nature.com/articles/d41586-018-01322-9 # Future of online > publishing > https://pypi.org/project/nobel/ # Python interface to Nobel Prize API! > > https://jfine2358.github.io/slides/2018-nature-jupyter-altair-vega-binder.html > > And some Python code: > > >>> import nobel > >>> api = nobel.Api() > >>> api.prizes.filter(year=2018, > category='economics')[0].laureates[1].surname > u'Romer' > > > Is there an idea for Python hidden somewhere in this message? -------------- next part -------------- An HTML attachment was scrubbed... URL: From philip.martin2007 at gmail.com Wed Oct 10 13:04:48 2018 From: philip.martin2007 at gmail.com (Philip Martin) Date: Wed, 10 Oct 2018 12:04:48 -0500 Subject: [Python-ideas] Revisiting Immutable Mappings Message-ID: Hi, I first want to thank everyone in the community for the contributions over the years. I know the idea of a frozendict has been proposed before and rejected. I have a use case for a frozendict implementation that to my knowledge was not discussed during previous debates. My reasoning for a frozendict class stems from patterns I typically see arise when performing ETL or data integrations and conversions. I generally have used MappingProxyType as a way to set default mapping to a function or to set an empty mapping to a function. I've created a gist with an example use case: https://gist.github.com/pmart123/493edf84d9aa61691ca7321325ebb6ab I've included an example of what code typically looks like when using MappingProxyType and what it could look like with a frozendict implementation. I believe this use case may also be under-reported in open source code as it often crops up when integrating third-party data sources, which at times can't be open sourced due to licensing issues. I would love to hear if anyone has used MappingProxyType in a similar manner, or if this use case could help warrant a frozendict in the standard library. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfine2358 at gmail.com Wed Oct 10 13:15:07 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Wed, 10 Oct 2018 18:15:07 +0100 Subject: [Python-ideas] Paul Romer, 2018 Economics Nobel Laureate, uses Python and Jupyter In-Reply-To: References: Message-ID: Geoffrey Spear wrote: > Is there an idea for Python hidden somewhere in this message? Thank you, Geoffrey, for pointing this out. I'd have done better to prefixed the title with OFF-TOPIC. That would have been more polite. To answer your question: Django created a new community of Python users. As did, for example, NumPy, Raspberry Pi, SciPy and other worthy examples, such as PyData. Python, in part because of the shared experience we have on this list, has made many good decisions that work well across the board. I think we're pragmatic, and concerned about our users. Paul Romer's Nobel Prize draws attention to an emerging new community. I don't mean economics. I mean scholarly publishing. See: https://paulromer.net/jupyter-mathematica-and-the-future-of-the-research-paper/ https://www.nature.com/articles/d41586-018-01322-9 # Future of online publishing By the way, back in the 1980s TeX and LaTeX were the future of STEM publishing. I think Jupyter is well placed to be 'the Django of STEM publishing', or if you prefer 'the LaTeX of the 2020s'. Things change. The relevance to this list? Off-topic, but I hope adding to our shared relevant background knowledge. I will try to remember the OFF-TOPIC in future. -- Jonathan From yselivanov.ml at gmail.com Wed Oct 10 13:19:11 2018 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 10 Oct 2018 13:19:11 -0400 Subject: [Python-ideas] async unittest.TestCase In-Reply-To: References: Message-ID: Thanks for proposing this. Yes, it makes sense to have unittest.AsyncTestCase in 3.8. AFAIK Lisa Roach (copied) was working on that (as well as on async Mock object), but I'm not sure what's the status of her work. I suggest to search for an open issue for this on bugs.python.org; if there's none, please create one. And let's make this happen. Yury On Wed, Oct 10, 2018 at 7:11 AM David Shawley wrote: > > Hi everyone and good morning to some of you, > > Since asyncio and the async/await syntax are both part of Python, I think > that we should extend TestCase to support it. The simplest solution that > I can think of is to create unittest.AsyncTestCase sub-class with the > following extensions: > > - create a new event loop and install it in AsyncTestCase.run > - make it possible for setUp, tearDown, and test methods to be async > by calling asyncio.iscoroutinefunction > > I wrote my own in a local test before noticing that Martin Richard had > already written and published asynctest [1]. Since then I found the > following projects as well: > > - https://github.com/kwarunek/aiounittest > - https://github.com/pytest-dev/pytest-asyncio > > I think that the community as a whole would benefit from basic support in > unittest for async/await test "user methods". I am personally fond of the > approach of extending unittest.TestCase in asynctest [2] over some of the > other approaches. > > Is this something that we want in our Standard Library? > > - dave > -- > [1]: https://github.com/Martiusweb/asynctest > [2]: https://github.com/Martiusweb/asynctest/blob/master/asynctest/case.py > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Yury From storchaka at gmail.com Wed Oct 10 13:34:57 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 10 Oct 2018 20:34:57 +0300 Subject: [Python-ideas] async unittest.TestCase In-Reply-To: References: Message-ID: 10.10.18 20:19, Yury Selivanov ????: > Thanks for proposing this. Yes, it makes sense to have > unittest.AsyncTestCase in 3.8. AFAIK Lisa Roach (copied) was working > on that (as well as on async Mock object), but I'm not sure what's the > status of her work. I suggest to search for an open issue for this on > bugs.python.org; if there's none, please create one. And let's make > this happen. https://bugs.python.org/issue32972 https://bugs.python.org/issue26467 From mike at selik.org Wed Oct 10 15:36:33 2018 From: mike at selik.org (Michael Selik) Date: Wed, 10 Oct 2018 12:36:33 -0700 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: Message-ID: How does a frozendict help in that example? It's not obvious to me. Despite not understanding that example, I'm +1 for having a frozendict. I don't think it'll increase cognitive load much, as it'll sit right next to frozenset when someone reads the builtins in alphabetical order. In my own experience, I want to use a dict as a dict key about once or twice a year. It'd be nice to have a quick way to convert to a frozendict. On Wed, Oct 10, 2018 at 10:05 AM Philip Martin wrote: > Hi, I first want to thank everyone in the community for the contributions > over the years. I know the idea of a frozendict has been proposed before > and rejected. I have a use case for a frozendict implementation that to my > knowledge was not discussed during previous debates. My reasoning for a > frozendict class stems from patterns I typically see arise when performing > ETL or data integrations and conversions. I generally have used > MappingProxyType as a way to set default mapping to a function or to set an > empty mapping to a function. I've created a gist with an example use case: > > https://gist.github.com/pmart123/493edf84d9aa61691ca7321325ebb6ab > > I've included an example of what code typically looks like when using > MappingProxyType and what it could look like with a > frozendict implementation. I believe this use case may also be > under-reported in open source code as it often crops up when integrating > third-party data sources, which at times can't be open sourced due to > licensing issues. I would love to hear if anyone has used MappingProxyType > in a similar manner, or if this use case could help warrant a frozendict in > the standard library. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Oct 10 19:23:48 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 11 Oct 2018 10:23:48 +1100 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: Message-ID: <20181010232348.GZ3817@ando.pearwood.info> Hi Philiip, and welcome, On Wed, Oct 10, 2018 at 12:04:48PM -0500, Philip Martin wrote: > I generally have used MappingProxyType as a way to set default mapping > to a function or to set an empty mapping to a function. > I've created a gist with an example use case: > > https://gist.github.com/pmart123/493edf84d9aa61691ca7321325ebb6ab Please try to keep the discussion in one place (i.e. here), for the benefit of the archives and for those who have email access but not unrestricted web access. Can you explain (in English) your use-case, and why MappingProxyType isn't suitable? If it *is* suitable, how does your proposal differ? If the only proposal is to rename types.MappingProxyType to a builtin "frozendict", that's one thing; if the proposal is something else, you should explain what. > I've included an example of what code typically looks like when using > MappingProxyType and what it could look like with a > frozendict implementation. Wouldn't that be just: from types import MappingProxyType as frozendict d = frozendict({'spam': 1, 'eggs': 2}) versus: d = frozendict({'spam': 1, 'eggs': 2}) Apart from the initial import, how would they be different? You want a frozendict; the existing MappingProxyType provides a frozendict (with a surprising name, but never mind...). Wouldn't you use them exactly the same way? They both (I presume) offer precisely the same read-only access to the mapping interface. -- Steve From philip.martin2007 at gmail.com Wed Oct 10 21:25:38 2018 From: philip.martin2007 at gmail.com (Philip Martin) Date: Wed, 10 Oct 2018 20:25:38 -0500 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: <20181010232348.GZ3817@ando.pearwood.info> References: <20181010232348.GZ3817@ando.pearwood.info> Message-ID: Steven, that's a great idea, and I would be 100% up for your suggestion to have types.MappingProxyType renamed to frozendict. However, the differences in the behavior of MappingProxyType's constructor versus dict's would make the API's behavior confusing IMO. For example, MappingProxyType(x=5, y=10) throws a TypeError. I don't think most people would expect this. MappingProxyType to me though does seem to be a non-obvious name compared to say frozenset as you have mentioned. Plus, it's included in a module that I would say is very low level alongside functions like prepare_class, new_class, etc. On Wed, Oct 10, 2018 at 6:24 PM Steven D'Aprano wrote: > Hi Philiip, and welcome, > > On Wed, Oct 10, 2018 at 12:04:48PM -0500, Philip Martin wrote: > > > I generally have used MappingProxyType as a way to set default mapping > > to a function or to set an empty mapping to a function. > > > I've created a gist with an example use case: > > > > https://gist.github.com/pmart123/493edf84d9aa61691ca7321325ebb6ab > > Please try to keep the discussion in one place (i.e. here), for the > benefit of the archives and for those who have email access but not > unrestricted web access. > > Can you explain (in English) your use-case, and why MappingProxyType > isn't suitable? If it *is* suitable, how does your proposal differ? > > If the only proposal is to rename types.MappingProxyType to a builtin > "frozendict", that's one thing; if the proposal is something else, you > should explain what. > > > > I've included an example of what code typically looks like when using > > MappingProxyType and what it could look like with a > > frozendict implementation. > > Wouldn't that be just: > > from types import MappingProxyType as frozendict > d = frozendict({'spam': 1, 'eggs': 2}) > > versus: > > d = frozendict({'spam': 1, 'eggs': 2}) > > Apart from the initial import, how would they be different? You want a > frozendict; the existing MappingProxyType provides a frozendict (with a > surprising name, but never mind...). Wouldn't you use them exactly the > same way? They both (I presume) offer precisely the same read-only > access to the mapping interface. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From philip.martin2007 at gmail.com Wed Oct 10 21:26:37 2018 From: philip.martin2007 at gmail.com (Philip Martin) Date: Wed, 10 Oct 2018 20:26:37 -0500 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: <20181010232348.GZ3817@ando.pearwood.info> References: <20181010232348.GZ3817@ando.pearwood.info> Message-ID: It would help over using a regular dict as a default argument to a function by preventing accidental mutation of the default or constant mapping. This is a quickly contrived example of the convert_price function now having a side effect by changing the translation_map. from unicodedata import normalize prices = [{'croissant': 1}, {'coffee': 3}] translation_map = {'apple': 'pomme', 'coffee': 'caf?'} def normalize_text(s): return normalize('NFD', s).encode('ascii', 'ignore').decode("utf-8") def usd_to_eur(v): return v / 1.2 def passthrough(v): return v def convert_price(record, convert_map=translation_map): # remove accents for price mapping. Oops! for key, value in convert_map.items(): convert_map[key] = normalize_text(value) record = { convert_map[k]: usd_to_eur(v) for k, v in record.items() } return record On Wed, Oct 10, 2018 at 6:24 PM Steven D'Aprano wrote: > Hi Philiip, and welcome, > > On Wed, Oct 10, 2018 at 12:04:48PM -0500, Philip Martin wrote: > > > I generally have used MappingProxyType as a way to set default mapping > > to a function or to set an empty mapping to a function. > > > I've created a gist with an example use case: > > > > https://gist.github.com/pmart123/493edf84d9aa61691ca7321325ebb6ab > > Please try to keep the discussion in one place (i.e. here), for the > benefit of the archives and for those who have email access but not > unrestricted web access. > > Can you explain (in English) your use-case, and why MappingProxyType > isn't suitable? If it *is* suitable, how does your proposal differ? > > If the only proposal is to rename types.MappingProxyType to a builtin > "frozendict", that's one thing; if the proposal is something else, you > should explain what. > > > > I've included an example of what code typically looks like when using > > MappingProxyType and what it could look like with a > > frozendict implementation. > > Wouldn't that be just: > > from types import MappingProxyType as frozendict > d = frozendict({'spam': 1, 'eggs': 2}) > > versus: > > d = frozendict({'spam': 1, 'eggs': 2}) > > Apart from the initial import, how would they be different? You want a > frozendict; the existing MappingProxyType provides a frozendict (with a > surprising name, but never mind...). Wouldn't you use them exactly the > same way? They both (I presume) offer precisely the same read-only > access to the mapping interface. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at cskk.id.au Wed Oct 10 22:01:29 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Thu, 11 Oct 2018 13:01:29 +1100 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: Message-ID: <20181011020129.GA11038@cskk.homeip.net> On 10Oct2018 20:25, Philip Martin wrote: >Steven, that's a great idea, and I would be 100% up for your suggestion to >have types.MappingProxyType renamed to frozendict. I'm not for the rename, myself. Though I'd not be against a frozendict factory in builtins, a tiny shim for MappingProxyType. >However, the differences >in the behavior of MappingProxyType's constructor versus dict's would make >the API's behavior confusing IMO. For example, MappingProxyType(x=5, y=10) >throws a TypeError. I don't think most people would expect this. Well, if it were called frozendict, indeed not. It should act like dict. So: def frozendict(**kw): return MappingProxyType(kw) You could make an argument for that (or a slightly heftier version accepting the various things dict accepts). Or... you could just keep such a thing in your personal kit as a trivial way to spell "frozendict". One could argue for the above as a nice example to live in the docs perhaps. But not everything needs a special name. Cheers, Cameron Simpson From rosuav at gmail.com Wed Oct 10 22:27:50 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Oct 2018 13:27:50 +1100 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: <20181011020129.GA11038@cskk.homeip.net> References: <20181011020129.GA11038@cskk.homeip.net> Message-ID: On Thu, Oct 11, 2018 at 1:02 PM Cameron Simpson wrote: > > On 10Oct2018 20:25, Philip Martin wrote: > >Steven, that's a great idea, and I would be 100% up for your suggestion to > >have types.MappingProxyType renamed to frozendict. > > I'm not for the rename, myself. Though I'd not be against a frozendict > factory in builtins, a tiny shim for MappingProxyType. > > >However, the differences > >in the behavior of MappingProxyType's constructor versus dict's would make > >the API's behavior confusing IMO. For example, MappingProxyType(x=5, y=10) > >throws a TypeError. I don't think most people would expect this. > > Well, if it were called frozendict, indeed not. It should act like dict. > > So: > > def frozendict(**kw): > return MappingProxyType(kw) > > You could make an argument for that (or a slightly heftier version > accepting the various things dict accepts). Or... you could just keep > such a thing in your personal kit as a trivial way to spell > "frozendict". One could argue for the above as a nice example to live in > the docs perhaps. > > But not everything needs a special name. > Point of note: A mapping proxy is NOT immutable; it is merely read-only. >>> from types import MappingProxyType >>> d = {'a':1, 'b':2} >>> p = MappingProxyType(d) >>> p mappingproxy({'a': 1, 'b': 2}) >>> d['a'] = 3 >>> p mappingproxy({'a': 3, 'b': 2}) A frozendict type, if it's meant to parallel frozenset, ought to be hashable (subject to the hashability of its members, of course), but you can't just take MPT and toss in a __hash__ method. No idea how important that use-case is, but Michael Selik mentioned "want[ing] to use a dict as a dict key about once or twice a year", which MPT is not able to do. Interestingly, frozenset isn't a subclass of set. I was going to say that a frozendict ought to be a dict, but maybe that isn't so important. Which might make a simple pure-Python frozendict actually pretty easy. ChrisA From philip.martin2007 at gmail.com Wed Oct 10 22:28:32 2018 From: philip.martin2007 at gmail.com (Philip Martin) Date: Wed, 10 Oct 2018 21:28:32 -0500 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: <20181011020129.GA11038@cskk.homeip.net> References: <20181011020129.GA11038@cskk.homeip.net> Message-ID: Cameron, That's a good suggestion. Ultimately, if there are not enough various use cases for a frozendict class, I think we could add something like this as an example recipe similar to the recipe section in itertools. I would be hesitant to add a quick shim to the standard library as I can't think of another instance where a developer calls a function expecting a specific class, and receives a different class. I'm happy to draft up some documentation if we decide to take this route because there aren't enough use cases. It would be great though to hear what other use cases developers have for a frozendict to ultimately decide whether this is the case. On Wed, Oct 10, 2018 at 9:01 PM Cameron Simpson wrote: > On 10Oct2018 20:25, Philip Martin wrote: > >Steven, that's a great idea, and I would be 100% up for your suggestion to > >have types.MappingProxyType renamed to frozendict. > > I'm not for the rename, myself. Though I'd not be against a frozendict > factory in builtins, a tiny shim for MappingProxyType. > > >However, the differences > >in the behavior of MappingProxyType's constructor versus dict's would make > >the API's behavior confusing IMO. For example, MappingProxyType(x=5, y=10) > >throws a TypeError. I don't think most people would expect this. > > Well, if it were called frozendict, indeed not. It should act like dict. > > So: > > def frozendict(**kw): > return MappingProxyType(kw) > > You could make an argument for that (or a slightly heftier version > accepting the various things dict accepts). Or... you could just keep > such a thing in your personal kit as a trivial way to spell > "frozendict". One could argue for the above as a nice example to live in > the docs perhaps. > > But not everything needs a special name. > > Cheers, > Cameron Simpson > -------------- next part -------------- An HTML attachment was scrubbed... URL: From philip.martin2007 at gmail.com Wed Oct 10 22:49:16 2018 From: philip.martin2007 at gmail.com (Philip Martin) Date: Wed, 10 Oct 2018 21:49:16 -0500 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: <20181011020129.GA11038@cskk.homeip.net> Message-ID: That is interesting. From my recollection, when OrderedDict was reimplemented in C, there was advice on the thread to not implement it as a subclass of dict. https://bugs.python.org/issue16991 I'm far from the right person to comment on the exact reasons, but perhaps frozenset being decoupled from set gives precedence for a frozendict not subclassing dict? The methods clear, pop, popitem, update and setdefault would not be necessary. On Wed, Oct 10, 2018 at 9:28 PM Chris Angelico wrote: > On Thu, Oct 11, 2018 at 1:02 PM Cameron Simpson wrote: > > > > On 10Oct2018 20:25, Philip Martin wrote: > > >Steven, that's a great idea, and I would be 100% up for your suggestion > to > > >have types.MappingProxyType renamed to frozendict. > > > > I'm not for the rename, myself. Though I'd not be against a frozendict > > factory in builtins, a tiny shim for MappingProxyType. > > > > >However, the differences > > >in the behavior of MappingProxyType's constructor versus dict's would > make > > >the API's behavior confusing IMO. For example, MappingProxyType(x=5, > y=10) > > >throws a TypeError. I don't think most people would expect this. > > > > Well, if it were called frozendict, indeed not. It should act like dict. > > > > So: > > > > def frozendict(**kw): > > return MappingProxyType(kw) > > > > You could make an argument for that (or a slightly heftier version > > accepting the various things dict accepts). Or... you could just keep > > such a thing in your personal kit as a trivial way to spell > > "frozendict". One could argue for the above as a nice example to live in > > the docs perhaps. > > > > But not everything needs a special name. > > > > Point of note: A mapping proxy is NOT immutable; it is merely read-only. > > >>> from types import MappingProxyType > >>> d = {'a':1, 'b':2} > >>> p = MappingProxyType(d) > >>> p > mappingproxy({'a': 1, 'b': 2}) > >>> d['a'] = 3 > >>> p > mappingproxy({'a': 3, 'b': 2}) > > A frozendict type, if it's meant to parallel frozenset, ought to be > hashable (subject to the hashability of its members, of course), but > you can't just take MPT and toss in a __hash__ method. No idea how > important that use-case is, but Michael Selik mentioned "want[ing] to > use a dict as a dict key about once or twice a year", which MPT is not > able to do. > > Interestingly, frozenset isn't a subclass of set. I was going to say > that a frozendict ought to be a dict, but maybe that isn't so > important. Which might make a simple pure-Python frozendict actually > pretty easy. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmcs at jsantos.eu Thu Oct 11 00:20:30 2018 From: jmcs at jsantos.eu (=?UTF-8?B?Sm/Do28gU2FudG9z?=) Date: Thu, 11 Oct 2018 06:20:30 +0200 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: <20181010232348.GZ3817@ando.pearwood.info> References: <20181010232348.GZ3817@ando.pearwood.info> Message-ID: One important difference between MappingProxyType and a "proper" frozendict, as analog to frozenset, is that MappingProxyType doesn't have any method to return mutated versions of itself. On Thu, 11 Oct 2018 at 01:24, Steven D'Aprano wrote: > Hi Philiip, and welcome, > > On Wed, Oct 10, 2018 at 12:04:48PM -0500, Philip Martin wrote: > > > I generally have used MappingProxyType as a way to set default mapping > > to a function or to set an empty mapping to a function. > > > I've created a gist with an example use case: > > > > https://gist.github.com/pmart123/493edf84d9aa61691ca7321325ebb6ab > > Please try to keep the discussion in one place (i.e. here), for the > benefit of the archives and for those who have email access but not > unrestricted web access. > > Can you explain (in English) your use-case, and why MappingProxyType > isn't suitable? If it *is* suitable, how does your proposal differ? > > If the only proposal is to rename types.MappingProxyType to a builtin > "frozendict", that's one thing; if the proposal is something else, you > should explain what. > > > > I've included an example of what code typically looks like when using > > MappingProxyType and what it could look like with a > > frozendict implementation. > > Wouldn't that be just: > > from types import MappingProxyType as frozendict > d = frozendict({'spam': 1, 'eggs': 2}) > > versus: > > d = frozendict({'spam': 1, 'eggs': 2}) > > Apart from the initial import, how would they be different? You want a > frozendict; the existing MappingProxyType provides a frozendict (with a > surprising name, but never mind...). Wouldn't you use them exactly the > same way? They both (I presume) offer precisely the same read-only > access to the mapping interface. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From boxed at killingar.net Thu Oct 11 01:02:40 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Thu, 11 Oct 2018 07:02:40 +0200 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: Message-ID: <765EA045-EB78-4FDC-B1BA-0D52BCA266AD@killingar.net> In tri.struct we have a class Frozen https://github.com/TriOptima/tri.struct/blob/master/lib/tri/struct/__init__.py that can be used to freeze stuff. I think something like this would be even better in the standard library, especially now with data classes! If we had this frozendict would just be: class frozendict(dict, Frozen): pass / Anders > On 10 Oct 2018, at 19:04, Philip Martin wrote: > > Hi, I first want to thank everyone in the community for the contributions over the years. I know the idea of a frozendict has been proposed before and rejected. I have a use case for a frozendict implementation that to my knowledge was not discussed during previous debates. My reasoning for a frozendict class stems from patterns I typically see arise when performing ETL or data integrations and conversions. I generally have used MappingProxyType as a way to set default mapping to a function or to set an empty mapping to a function. I've created a gist with an example use case: > > https://gist.github.com/pmart123/493edf84d9aa61691ca7321325ebb6ab > > I've included an example of what code typically looks like when using MappingProxyType and what it could look like with a frozendict implementation. I believe this use case may also be under-reported in open source code as it often crops up when integrating third-party data sources, which at times can't be open sourced due to licensing issues. I would love to hear if anyone has used MappingProxyType in a similar manner, or if this use case could help warrant a frozendict in the standard library. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Thu Oct 11 01:35:40 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 11 Oct 2018 08:35:40 +0300 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: <20181010232348.GZ3817@ando.pearwood.info> Message-ID: 11.10.18 07:20, Jo?o Santos ????: > One important difference between MappingProxyType and a "proper" > frozendict, as analog to frozenset, is that MappingProxyType doesn't > have any method to return mutated versions of itself. MappingProxyType.copy()? From steve at pearwood.info Thu Oct 11 02:39:57 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 11 Oct 2018 17:39:57 +1100 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: <20181011020129.GA11038@cskk.homeip.net> Message-ID: <20181011063957.GA3817@ando.pearwood.info> On Thu, Oct 11, 2018 at 01:27:50PM +1100, Chris Angelico wrote: [...] [Cameron Simpson] > > Well, if it were called frozendict, indeed not. It should act like dict. > > > > So: > > > > def frozendict(**kw): > > return MappingProxyType(kw) > > > > You could make an argument for that (or a slightly heftier version > > accepting the various things dict accepts). How would we make the opposite argument? That [frozen]dict *shouldn't* accept the same constructions that dict actually does? Seems to me that since we've proven the utility of the various constructor signatures that regular dicts already support, the addition of "read-only" doesn't change that. Now for the limited use-cases that MappingProxyType was originally designed for, as a wrapper around an existing dict, it made historical sense for it to only support a single dict argument. After all, it is a proxy, so there needs to be a dict for it to proxy. But if we promoted it to a real mapping, not just a proxy, or introduced a new read-only mapping, it seems to me that the full dict constructor interface ought to be a no-brainer. > > But not everything needs a special name. > > > > Point of note: A mapping proxy is NOT immutable; it is merely read-only. [...] > A frozendict type, if it's meant to parallel frozenset, ought to be > hashable (subject to the hashability of its members, of course) Good point! That's exactly the sort of difference between MappingProxyType and a true frozendict that I was looking for. So there's still a use-case for a true frozendict. > Interestingly, frozenset isn't a subclass of set. I was going to say > that a frozendict ought to be a dict, but maybe that isn't so > important. If there were a subclass relationship between frozendict and dict, it ought to be equivalent to this: py> issubclass(collections.MutableMapping, collections.Mapping) True but having two independent types works for me too. > Which might make a simple pure-Python frozendict actually > pretty easy. There's historical precedent: sets (and frozensets?) were initially introduced as a std library module, and only later elevated to builtins. I would be +1 for a frozendict module in 3.8, and if it proves sufficient useful to elevate to a builtin, that can happen in 3.9 or above. -- Steve From szport at gmail.com Thu Oct 11 05:18:52 2018 From: szport at gmail.com (Zaur Shibzukhov) Date: Thu, 11 Oct 2018 02:18:52 -0700 (PDT) Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: Message-ID: May be the following simple prototype of frozendict could be useful? def frozen_error(): return RuntimeError("frozendict is not mutable") class frozendict(dict): # def __setitem__(self, key, val): raise frozen_error() # def setdefault(self, key, val=None): raise frozen_error() # def update(self, ob): raise frozen_error() # def pop(self): raise frozen_error() # def popitem(self, ob): raise frozen_error() # def clear(self, ob): raise frozen_error() # def __delitem__(self, key): raise frozen_error() # def __repr__(self): return "frozendict(" + dict.__repr__(self)[1:-1] + ")" # def __str__(self): return "frozendict(" + dict.__str__(self)[1:-1] + ")" # def fromkeys(self, keys, val=None): return frozendict([(key,val) for key in keys]) # def copy(self): return frozendict(self.items()) ?????, 10 ??????? 2018 ?., 20:06:03 UTC+3 ???????????? Philip Martin ???????: > > Hi, I first want to thank everyone in the community for the contributions > over the years. I know the idea of a frozendict has been proposed before > and rejected. I have a use case for a frozendict implementation that to my > knowledge was not discussed during previous debates. My reasoning for a > frozendict class stems from patterns I typically see arise when performing > ETL or data integrations and conversions. I generally have used > MappingProxyType as a way to set default mapping to a function or to set an > empty mapping to a function. I've created a gist with an example use case: > > https://gist.github.com/pmart123/493edf84d9aa61691ca7321325ebb6ab > > I've included an example of what code typically looks like when using > MappingProxyType and what it could look like with a > frozendict implementation. I believe this use case may also be > under-reported in open source code as it often crops up when integrating > third-party data sources, which at times can't be open sourced due to > licensing issues. I would love to hear if anyone has used MappingProxyType > in a similar manner, or if this use case could help warrant a frozendict in > the standard library. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Oct 11 07:41:40 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 11 Oct 2018 22:41:40 +1100 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: Message-ID: <20181011114138.GB3817@ando.pearwood.info> On Thu, Oct 11, 2018 at 02:18:52AM -0700, Zaur Shibzukhov wrote: > > May be the following simple prototype of frozendict could be useful? > > def frozen_error(): > return RuntimeError("frozendict is not mutable") > > class frozendict(dict): This violates the Liskov Substitution Principle. https://en.wikipedia.org/wiki/Liskov_substitution_principle Why is that bad? Let's say you have a function that requires a dict, and following the principle of "Fail Fast", you check the input up front: def spam(adict): """Input: adict: an instance of dict """ if isinstance(adict, dict): # lots of code here # and deep inside the function, we end up calling: adict[key] = "something" I hear people objecting that isinstance checks are "unPythonic". Perhaps; however, the basic principle applies regardless of whether we are doing LBYL isinstance checks, test for the presence of a __setitem__ method, or do no up-front tests at all and rely on duck-typing and EAFP. Our spam() function now accepts dicts, and rejects non-dicts, and works well, until somebody passes a frozendict: spam(frozendict()) This passes the isinstance test (or the check for __setitem__). It passes the documented interface: frozendicts certainly are dicts, since they are a subclass of dict. But nevertheless, the call to setitem fails, and the function breaks. We can get a good idea of how this ought to work by looking at set and frozenset. frozenset is *not* a subclass of set; it is a distinct class, so that frozensets are not sets. Rather than having mutator methods which raise an exception, they simply don't provide those methods at all: py> f = frozenset() py> isinstance(f, set) False py> hasattr(f, 'add') False That's how we should design frozendict. -- Steve From jfine2358 at gmail.com Thu Oct 11 08:51:46 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Thu, 11 Oct 2018 13:51:46 +0100 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: <20181011114138.GB3817@ando.pearwood.info> References: <20181011114138.GB3817@ando.pearwood.info> Message-ID: Steve D'Aprano wrote > Zaur Shibzukhov wrote > > class frozendict(dict): > This violates the Liskov Substitution Principle. > https://en.wikipedia.org/wiki/Liskov_substitution_principle and so, Steve said, we should not make frozendict a subclass of dict. Perhaps Zaur was thinking of Don't Repeat Yourself (DRY) https://en.wikipedia.org/wiki/Don%27t_repeat_yourself Certainly, by writing class frozendict(dict): # methods Zaur was able to use unchanged superclass methods such as __init__ and __getitem__, and also to delegate most of __str__ to the superclass. I see much merit in both https://en.wikipedia.org/wiki/Liskov_substitution_principle https://en.wikipedia.org/wiki/Don%27t_repeat_yourself I wonder whether, in pure Python, we can nicely have them both, when we implement frozenset. The best I can come up with is (not tested) is class frozendict: __slots__ = ('_self',) def __init__(self, *argv, **kwargs): self._self = dict(*argv, **kwargs) def __getitem__(self, key): return self._self.__getitem__(key) # And so on. -- Jonathan From jfine2358 at gmail.com Thu Oct 11 09:15:16 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Thu, 11 Oct 2018 14:15:16 +0100 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: <20181011114138.GB3817@ando.pearwood.info> Message-ID: > https://en.wikipedia.org/wiki/Liskov_substitution_principle > https://en.wikipedia.org/wiki/Don%27t_repeat_yourself I did an internet search for: python liskov (over the past year). The first result was a Ruby page (but principle probably the same) https://www.netguru.co/codestories/solid-principles-3-lsp The second result was "Incompatibile signature with supertype" https://github.com/python/mypy/issues/4250 And the code example was class FrozenDict(MutableMapping): # code So there's prior Python art for FrozenDict and Liskov inheritance. -- Jonathan From chris.barker at noaa.gov Thu Oct 11 11:40:07 2018 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 11 Oct 2018 11:40:07 -0400 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: <20181011114138.GB3817@ando.pearwood.info> References: <20181011114138.GB3817@ando.pearwood.info> Message-ID: > This violates the Liskov Substitution Principle. If we REALLY had a time machine, then dict would subclass frozendict, and we?d be all set. But to what extent do we need to support ALL the ways to check for an interface? Personally, I think EAFTP is the most ?Pythonic?, but if folks want to check up front, then isn?t that what ABCs are for? In which case , we already have Mapping and MutableMapping. So if you have code that checks for Mapping when you need it mutable, then that?s arguably a bug. And if you code checks for dict directly, then it?s arguably unpythonic. That being said, it probably is best not to break working code. Would there be unpleasant consequences to making dict a subclass of FrozenDict ? Or maybe come up with a new name: after all, lists and tuples are independent, even though you *could* think of a tuple as a FrozenList ... -CHB From rosuav at gmail.com Thu Oct 11 11:45:30 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 12 Oct 2018 02:45:30 +1100 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: <20181011114138.GB3817@ando.pearwood.info> Message-ID: On Fri, Oct 12, 2018 at 2:41 AM Chris Barker - NOAA Federal via Python-ideas wrote: > > > This violates the Liskov Substitution Principle. > > If we REALLY had a time machine, then dict would subclass frozendict, > and we?d be all set. Thanks to virtual subclassing, we can still do this. The question is, should we? Intuition tells me that a frozen dictionary is a form of dictionary that adds restrictions, not that a dictionary is a frozen dictionary that you left out to thaw. But as we see from [frozen]set, it's probably best to treat them as completely independent classes, both implementing the basic Mapping interface. ChrisA From jfine2358 at gmail.com Thu Oct 11 12:54:05 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Thu, 11 Oct 2018 17:54:05 +0100 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: <20181011114138.GB3817@ando.pearwood.info> Message-ID: Summary: Long post. Because of LSP, neither dict nor frozendict are a subclass of the other. Chris Barker wrote: > If we REALLY had a time machine, then dict would subclass frozendict, > and we?d be all set. Prediction is difficult, particularly when it involves the future. - Neils Bohr. And be careful about what you wish for. Perhaps in Python 4 we can fix this problem. Chris Angelico wrote: > Thanks to virtual subclassing, we can still do this. The question is, should we? > Intuition tells me that a frozen dictionary is a form of dictionary > that adds restrictions, not that a dictionary is a frozen dictionary > that you left out to thaw. It's not quite so simple. Sometimes, less is more. https://www.phrases.org.uk/meanings/226400.html Suppose every instance of class A has useful property P, and B is a subclass of A. If the Liskov substitution principle (LSP) holds, then every instance of B also has useful property P. Notice that I said USEFUL property P. The negation Q of property P is also a property. Often, Q is NOT USEFUL. But what if both P and its negation Q are BOTH USEFUL? Well, if there should be some strange useful property P, whose negation is also useful then (drum roll) Liskov substitution says that neither class is a subclass of the other. (I guess that's one of the applications of LSP.) Here's an example of such a property. A tuple of integers is immutable, and so has a hash and can be used as a dictionary key. Being immutable is a useful property. A list of integers is mutable, and so can be used to record changing state. Being mutable is a useful property. Aside: Last month I prepared the show part of the show-and-tell for this. https://github.com/jfine2358/python-show-and-tell/issues/1 How do we explain this to a novice >>> dct = dict() >>> point = [0, 1] >>> dct[point] = 'red' Traceback (most recent call last): TypeError: unhashable type: 'list' Conclusion: If we have LSP, then dict is not a subclass of frozendict, and frozendict is not a subclass of dict. Neither is a subclass of the other. One way to fix this in Python 4 would be to create a common ancestor, which has only the shared methods of dict and frozendict. Or do something similar with abstract base classes. Remark: The issues here might be related to mypy typing, which I've not used. From what I've read, it seems to rely on LSP. I've nothing to say at present about the hardest problem here, which is naming things. We have list and tuple. But dict and what name for frozen, and what name for common ancestor. https://martinfowler.com/bliki/TwoHardThings.html -- Jonathan From jfine2358 at gmail.com Thu Oct 11 13:43:06 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Thu, 11 Oct 2018 18:43:06 +0100 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: <20181011114138.GB3817@ando.pearwood.info> Message-ID: A link on https://en.wikipedia.org/wiki/Liskov_substitution_principle goes to http://www.engr.mun.ca/~theo/Courses/sd/5895-downloads/sd-principles-3.ppt.pdf which has a very nice example (slides 14 and 15). Slide 14: Is immutable Square a behavioural subtype of immutable Rectangle? And vice versa? Slide 15: Is mutable Square a behavioural subtype of mutable Rectangle? And vice versa? And relationship between mutable and immutable Square (resp Rectangle). -- Jonathan From chris.barker at noaa.gov Thu Oct 11 15:34:13 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 11 Oct 2018 12:34:13 -0700 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: <20181011114138.GB3817@ando.pearwood.info> Message-ID: On Thu, Oct 11, 2018 at 9:54 AM, Jonathan Fine wrote: > Summary: Long post. Because of LSP, neither dict nor frozendict are a > subclass of the other. given Python's dynamic typing, these issues are kinda academic :-) > Intuition tells me that a frozen dictionary is a form of dictionary > > that adds restrictions, not that a dictionary is a frozen dictionary > > that you left out to thaw. > well, IIUC the Liskov principle correctly then a subclass is never a version of a class that does less, but rather always one that does more. Think you intuition may be driven by the choice of names and history: yes, a frozen dict sounds like a regular dict that has been altered (specifically frozen) -- but if we called them "hash_table" and "mutable_has_table", then your intuition may be different :-) As for subclassing or not, for most Python code is makes no difference -- polymorphism is not achieved through subclassing. and the "Pythonic" way to test for type is through ABCs, and we already have Mapping and MutableMapping, which kind of surprised me, as there is no builtin Mapping that isn't also a MutableMapping. Notice that I said USEFUL property P. The negation Q of property P is > also a property. well, yeah, but I think the concept of "useful" is pretty vague. in this case, is "imutablilty" the "useful" property -- or is "hashability"?, which would want us to use abc.Hashable. So: I don't care what is or isn't a subclass of what -- I don't think that's a Pythonic question. But I do think : issubclass(frozendict, abc.Mapping) and issubclass(frozendict, abc.Hashable) would be useful. BTW, I just noticed that: A tuple, as in ``issubclass(x, (A, B, ...))``, may be given as the target to check against. This is equivalent to ``issubclass(x, A) or issubclass(x, B) or ...`` etc. which seems less useful than "and" -- at least for ABCs I suppose that API pre-dates ABCs.... One way to fix this in Python 4 would be to create a common ancestor, > which has only the shared methods of dict and frozendict. which would be an immutable, but not hashable, mapping ?!? Or do > something similar with abstract base classes. > already done -- see above. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Oct 11 15:44:52 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 11 Oct 2018 12:44:52 -0700 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: <20181011114138.GB3817@ando.pearwood.info> Message-ID: BTW: In [7]: issubclass(set, frozenset) Out[7]: False In [8]: issubclass(frozenset, set) Out[8]: False no reason to do anything different here. and: In [13]: issubclass(MappingProxyType, abc.Hashable) Out[13]: False so yes, there is a need for something different -CHB On Thu, Oct 11, 2018 at 12:34 PM, Chris Barker wrote: > On Thu, Oct 11, 2018 at 9:54 AM, Jonathan Fine > wrote: > >> Summary: Long post. Because of LSP, neither dict nor frozendict are a >> subclass of the other. > > > given Python's dynamic typing, these issues are kinda academic :-) > > > Intuition tells me that a frozen dictionary is a form of dictionary >> > that adds restrictions, not that a dictionary is a frozen dictionary >> > that you left out to thaw. >> > > well, IIUC the Liskov principle correctly then a subclass is never a > version of a class that does less, but rather always one that does more. > > Think you intuition may be driven by the choice of names and history: yes, > a frozen dict sounds like a regular dict that has been altered > (specifically frozen) -- but if we called them "hash_table" and > "mutable_has_table", then your intuition may be different :-) > > As for subclassing or not, for most Python code is makes no difference -- > polymorphism is not achieved through subclassing. and the "Pythonic" way to > test for type is through ABCs, and we already have Mapping and > MutableMapping, which kind of surprised me, as there is no builtin Mapping > that isn't also a MutableMapping. > > Notice that I said USEFUL property P. The negation Q of property P is >> also a property. > > > well, yeah, but I think the concept of "useful" is pretty vague. > > in this case, is "imutablilty" the "useful" property -- or is > "hashability"?, which would want us to use abc.Hashable. > > So: > > I don't care what is or isn't a subclass of what -- I don't think that's a > Pythonic question. But I do think : > > issubclass(frozendict, abc.Mapping) and issubclass(frozendict, > abc.Hashable) > > would be useful. > > BTW, I just noticed that: > > A tuple, as in ``issubclass(x, (A, B, ...))``, may be given as the target > to check against. This is equivalent to ``issubclass(x, A) or issubclass(x, > B) or ...`` etc. > > which seems less useful than "and" -- at least for ABCs > > I suppose that API pre-dates ABCs.... > > One way to fix this in Python 4 would be to create a common ancestor, >> which has only the shared methods of dict and frozendict. > > > which would be an immutable, but not hashable, mapping ?!? > > Or do >> something similar with abstract base classes. >> > > already done -- see above. > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Thu Oct 11 16:47:46 2018 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Thu, 11 Oct 2018 21:47:46 +0100 Subject: [Python-ideas] Introduce typing.SupportsFsPath In-Reply-To: References: Message-ID: On Tue, 9 Oct 2018 at 15:17, Eric Fahlgren wrote: > On Tue, Oct 9, 2018 at 3:16 AM Ivan Levkivskyi > wrote: > >> class PathLike(Protocol[AnyStr]): >> > > I had been working on this same problem intermittently for several months, > so thanks, but... > > error: Invariant type variable 'AnyStr' used in protocol where > covariant one is expected > > is called out on the class by mypy 0.630 (Python 3.6.6). Do I just need > to wait for 0.640? Or should I define a new TypeVar for AnyStr_co and use > that? > Hm, it looks like mypy overreacts here. I think it should be safe to use a constrained type variable if there are no constraints that are subtypes of other constraints (which is the case for AnyStr on Python 3, where bytes is not a subtype of str). Could you please open an issue about this on mypy tracker? In the meantime, you can just silence the error with a `# type: ignore`. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Oct 11 18:16:00 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 12 Oct 2018 09:16:00 +1100 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: <20181011114138.GB3817@ando.pearwood.info> Message-ID: <20181011221600.GC3817@ando.pearwood.info> On Fri, Oct 12, 2018 at 02:45:30AM +1100, Chris Angelico wrote: > On Fri, Oct 12, 2018 at 2:41 AM Chris Barker - NOAA Federal via > Python-ideas wrote: > > > > > This violates the Liskov Substitution Principle. > > > > If we REALLY had a time machine, then dict would subclass frozendict, > > and we?d be all set. > > Thanks to virtual subclassing, we can still do this. The question is, should we? > > Intuition tells me that a frozen dictionary is a form of dictionary > that adds restrictions, not that a dictionary is a frozen dictionary > that you left out to thaw. No offence Chris, but that's why we shouldn't program by intuition :-) We don't build Smart cars by starting with a Hummer and slicing off the bits that aren't needed. We already have prior art demonstrating best practice in the form of the Mapping and MutableMapping ABCs, and frozenset versus set for concrete classes. Best practice is not to subclass a type and restrict existing functionality, but to start with a restricted class and extend functionality. Or not to subclass at all. Subclassing is not the only way to DRY -- there are other ways for frozendict and dict to share code other than subclassing, although they're not necessarily efficient or easy if frozendict is written in pure Python. One way is delegation to a hidden dict (like the MappingProxyType, except we ensure the dict is private and not shared). # Just a sketch, not a full implementation. class frozendict(object): def __new__(cls, *args, **kwargs): instance = super().__new__(cls) instance.__d = dict(*args, **kwargs) return instance # Define only the methods we want, delegating to the # hidden dict. def __getitem__(self, key): return self.__d[key] def keys(self): return self.__d.keys() def items(self): return self.__d.items() The trickiest part is probably getting hashing right. I think this would work: def __hash__(self): return hash(tuple(self.items())) > But as we see from [frozen]set, it's > probably best to treat them as completely independent classes, both > implementing the basic Mapping interface. Indeed. -- Steve From ericfahlgren at gmail.com Thu Oct 11 18:34:15 2018 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Thu, 11 Oct 2018 15:34:15 -0700 Subject: [Python-ideas] Introduce typing.SupportsFsPath In-Reply-To: References: Message-ID: Done https://github.com/python/mypy/issues/5775 On Thu, Oct 11, 2018 at 1:47 PM Ivan Levkivskyi wrote: > On Tue, 9 Oct 2018 at 15:17, Eric Fahlgren wrote: > >> On Tue, Oct 9, 2018 at 3:16 AM Ivan Levkivskyi >> wrote: >> >>> class PathLike(Protocol[AnyStr]): >>> >> >> I had been working on this same problem intermittently for several >> months, so thanks, but... >> >> error: Invariant type variable 'AnyStr' used in protocol where >> covariant one is expected >> >> is called out on the class by mypy 0.630 (Python 3.6.6). Do I just need >> to wait for 0.640? Or should I define a new TypeVar for AnyStr_co and use >> that? >> > > Hm, it looks like mypy overreacts here. I think it should be safe to use a > constrained type variable if there are no constraints that are subtypes of > other constraints (which is the case for AnyStr on Python 3, where bytes is > not a subtype of str). Could you please open an issue about this on mypy > tracker? In the meantime, you can just silence the error with a `# type: > ignore`. > > -- > Ivan > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Oct 11 18:35:28 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 12 Oct 2018 09:35:28 +1100 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: <20181011114138.GB3817@ando.pearwood.info> Message-ID: <20181011223528.GD3817@ando.pearwood.info> On Thu, Oct 11, 2018 at 12:34:13PM -0700, Chris Barker via Python-ideas wrote: > I don't care what is or isn't a subclass of what -- I don't think that's a > Pythonic question. But I do think : > > issubclass(frozendict, abc.Mapping) and issubclass(frozendict, abc.Hashable) > > would be useful. So you do care about what is and isn't a subclass :-) It is 2018 and we've had isinstance and issubclass in the language since Python 2.2 in 2002, I really wish that we could get away from the idea that checking types is unPythonic and duck-typing is the One True Way to do things. The old, Python 1.5 form of type-checking: type(spam) is dict was not so great, since it tested only for a specific type by identity. But with ABCs and virtual subclasses and inheritance, isinstance is a great way to check that your duck both quacks and swims, rather than watching it sink and drown in the middle of your function :-) -- Steve From rosuav at gmail.com Thu Oct 11 18:38:22 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 12 Oct 2018 09:38:22 +1100 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: <20181011221600.GC3817@ando.pearwood.info> References: <20181011114138.GB3817@ando.pearwood.info> <20181011221600.GC3817@ando.pearwood.info> Message-ID: On Fri, Oct 12, 2018 at 9:16 AM Steven D'Aprano wrote: > > On Fri, Oct 12, 2018 at 02:45:30AM +1100, Chris Angelico wrote: > > On Fri, Oct 12, 2018 at 2:41 AM Chris Barker - NOAA Federal via > > Python-ideas wrote: > > > > > > > This violates the Liskov Substitution Principle. > > > > > > If we REALLY had a time machine, then dict would subclass frozendict, > > > and we?d be all set. > > > > Thanks to virtual subclassing, we can still do this. The question is, should we? > > > > Intuition tells me that a frozen dictionary is a form of dictionary > > that adds restrictions, not that a dictionary is a frozen dictionary > > that you left out to thaw. > > No offence Chris, but that's why we shouldn't program by intuition :-) > > > But as we see from [frozen]set, it's > > probably best to treat them as completely independent classes, both > > implementing the basic Mapping interface. > > Indeed. > Which was my point. They probably should NOT be in a direct hierarchy, partly because people's intuition WILL lead them to a dangerous expectation. ChrisA From chris.barker at noaa.gov Thu Oct 11 19:30:47 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 11 Oct 2018 16:30:47 -0700 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: <20181011223528.GD3817@ando.pearwood.info> References: <20181011114138.GB3817@ando.pearwood.info> <20181011223528.GD3817@ando.pearwood.info> Message-ID: On Thu, Oct 11, 2018 at 3:35 PM, Steven D'Aprano wrote: > On Thu, Oct 11, 2018 at 12:34:13PM -0700, Chris Barker via Python-ideas > wrote: > > > I don't care what is or isn't a subclass of what -- I don't think that's > a > > Pythonic question. But I do think : > > > > issubclass(frozendict, abc.Mapping) and issubclass(frozendict, > abc.Hashable) > > > > would be useful. > > So you do care about what is and isn't a subclass :-) > well, kinda -- I don't care whether dict and frozen dict have a subclassing relationship, and I don't care what class a given object is that gets passed to my code. What I *might* care about is what interface it presents, which is what ABCs are for. If Python had a way to check ABCs without issubclass, then I wouldn't care about subclasses at all. I'd actually kind of like to have: hasinterface(an_object, (ABC1, ABC2, ABC3)) Even though, it would be the same as issubclass() (though I'd like an AND relationship with the tuple of ABCs..) Maybe I'm making a distinction that isn't really there, but I see a subclass of an ABC is quite different than a subclass of another concrete class. > It is 2018 and we've had isinstance and issubclass in the language since > Python 2.2 in 2002, I really wish that we could get away from the idea > that checking types is unPythonic and duck-typing is the One True Way to > do things. The old, Python 1.5 form of type-checking: > > type(spam) is dict > > was not so great, since it tested only for a specific type by identity. > But with ABCs and virtual subclasses and inheritance, isinstance is a > great way to check that your duck both quacks and swims, rather than > watching it sink and drown in the middle of your function :-) > yeah, I'm not so sure -- I"ve been watching Python go more and more in that direction -- and I don't think I'm happy about it. Personally, the only time I check a type is the darn list-of-strings vs single-string issue -- ever since we got true_division, that's the only type error I commonly see (that isn't trivial to debug). And I still wish Python had a Character type, and then I wouldn't need to do that either :-) And actually, if you're talking "modern" Python, isn't static type checking the way to do it now ? Now that I think about it - how does MyPy/Typeshed handle the iterable_of_strings problem? Since a single string IS and iterable of strings? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamtlu at gmail.com Fri Oct 12 10:17:36 2018 From: jamtlu at gmail.com (James Lu) Date: Fri, 12 Oct 2018 10:17:36 -0400 Subject: [Python-ideas] Support parsing stream with `re` In-Reply-To: <23485.36316.625236.863737@turnbull.sk.tsukuba.ac.jp> References: <20181008101957.GA85076@cskk.homeip.net> <17E6FE52-5CFD-41AB-8807-4B08F9CFCCA6@killingar.net> <5BBC8B3C.70308@canterbury.ac.nz> <23484.52300.242521.93443@turnbull.sk.tsukuba.ac.jp> <23484.61172.389958.589@turnbull.sk.tsukuba.ac.jp> <23485.36316.625236.863737@turnbull.sk.tsukuba.ac.jp> Message-ID: <39ED4C2C-1D23-4BE4-87AB-7086C512E2EF@gmail.com> An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Fri Oct 12 10:05:54 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 13 Oct 2018 03:05:54 +1300 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: <20181011114138.GB3817@ando.pearwood.info> Message-ID: <5BC0AA42.7010900@canterbury.ac.nz> Chris Barker - NOAA Federal via Python-ideas wrote: > Or maybe come up with a new name We should call it a birdseyedict, because of this: http://www.vulture.com/2016/12/unearthing-a-rare-1971-monty-python-film-all-about-peas.html -- Greg From hasan.diwan at gmail.com Mon Oct 15 14:15:47 2018 From: hasan.diwan at gmail.com (Hasan Diwan) Date: Mon, 15 Oct 2018 11:15:47 -0700 Subject: [Python-ideas] Powerset Message-ID: [if this isn't the correct spot, let me know and I'll gladly take it elsewhere] I have found myself needing powerset functionality several times recently to the point where I wondered if there's interest in making it part of the standard library. I have a working implementation and tests. Would take but 3 minutes to add it in and issue a pull request, but I'd like to know if there's interest from esteemed group members. Thanks for the quick turnaround. -- H -- OpenPGP: https://sks-keyservers.net/pks/lookup?op=get&search=0xFEBAD7FFD041BBA1 If you wish to request my time, please do so using bit.ly/hd1AppointmentRequest. Si vous voudrais faire connnaisance, allez a bit.ly/hd1AppointmentRequest. Sent from my mobile device Envoye de mon portable From skreft at gmail.com Mon Oct 15 14:42:45 2018 From: skreft at gmail.com (Sebastian Kreft) Date: Mon, 15 Oct 2018 20:42:45 +0200 Subject: [Python-ideas] Powerset In-Reply-To: References: Message-ID: Hi Hassan, I think this is unlikely to get added to the standard library as it is listed as a recipe in the itertools module ( https://docs.python.org/3/library/itertools.html#itertools-recipes). You may want to check out the more-itertools package ( https://more-itertools.readthedocs.io/en/latest/) which includes all the recipes listed, as well as some other functions. On Mon, Oct 15, 2018 at 8:16 PM Hasan Diwan wrote: > [if this isn't the correct spot, let me know and I'll gladly take it > elsewhere] > I have found myself needing powerset functionality several times > recently to the point where I wondered if there's interest in making > it part of the standard library. I have a working implementation and > tests. Would take but 3 minutes to add it in and issue a pull request, > but I'd like to know if there's interest from esteemed group members. > Thanks for the quick turnaround. -- H > > -- > OpenPGP: > https://sks-keyservers.net/pks/lookup?op=get&search=0xFEBAD7FFD041BBA1 > If you wish to request my time, please do so using > bit.ly/hd1AppointmentRequest. > Si vous voudrais faire connnaisance, allez a bit.ly/hd1AppointmentRequest. > > Sent from my mobile device > Envoye de mon portable > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Sebastian Kreft -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Oct 15 19:02:52 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 16 Oct 2018 10:02:52 +1100 Subject: [Python-ideas] Powerset In-Reply-To: References: Message-ID: <20181015230251.GM3817@ando.pearwood.info> Hi Hasan, and welcome, On Mon, Oct 15, 2018 at 11:15:47AM -0700, Hasan Diwan wrote: > [if this isn't the correct spot, let me know and I'll gladly take it elsewhere] > I have found myself needing powerset functionality several times > recently to the point where I wondered if there's interest in making > it part of the standard library. This is certainly the right place to discuss this, but you shouldn't assume that everyone reading will know what powerset functionality you are referring to. Is it the same as the recipe in the itertools documentation? https://docs.python.org/3/library/itertools.html#itertools-recipes Can you make a case for why it is important and useful enough to be put into the std lib? Every new function increases the burden on both the Python developers maintaining the stdlib, and new users learning to use the language. So you need to make a case for why the benefit outweighs the costs. -- Steve From hasan.diwan at gmail.com Mon Oct 15 19:45:39 2018 From: hasan.diwan at gmail.com (Hasan Diwan) Date: Mon, 15 Oct 2018 16:45:39 -0700 Subject: [Python-ideas] Powerset In-Reply-To: <20181015230251.GM3817@ando.pearwood.info> References: <20181015230251.GM3817@ando.pearwood.info> Message-ID: > This is certainly the right place to discuss this, but you shouldn't > assume that everyone reading will know what powerset functionality you > are referring to. > Is it the same as the recipe in the itertools documentation? Yes pretty much. > > https://docs.python.org/3/library/itertools.html#itertools-recipes > > Can you make a case for why it is important and useful enough to be put > into the std lib? Every new function increases the burden on both the > Python developers maintaining the stdlib, and new users learning to use > the language. So you need to make a case for why the benefit outweighs > the costs. The best case I've come up with is one of obviousness. When looking for the powerset implementation in the cpython source, grepping for itertools.chain is not an obvious, at least to me, vector for a solution. -- H -- OpenPGP: https://sks-keyservers.net/pks/lookup?op=get&search=0xFEBAD7FFD041BBA1 If you wish to request my time, please do so using bit.ly/hd1AppointmentRequest. Si vous voudrais faire connnaisance, allez a bit.ly/hd1AppointmentRequest. Sent from my mobile device Envoye de mon portable From wes.turner at gmail.com Tue Oct 16 00:31:20 2018 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 16 Oct 2018 00:31:20 -0400 Subject: [Python-ideas] Powerset In-Reply-To: References: <20181015230251.GM3817@ando.pearwood.info> Message-ID: Is there a name for an iteration of the powerset which is more useful for binary search? I.e. instead of starting with null set, start with the "middle" ( r/2 ). Maybe a bit OT, but is there a name for such a combinatorial search? On Monday, October 15, 2018, Hasan Diwan wrote: > > This is certainly the right place to discuss this, but you shouldn't > > assume that everyone reading will know what powerset functionality you > > are referring to. > > Is it the same as the recipe in the itertools documentation? > > Yes pretty much. > > > > https://docs.python.org/3/library/itertools.html#itertools-recipes > > > > Can you make a case for why it is important and useful enough to be put > > into the std lib? Every new function increases the burden on both the > > Python developers maintaining the stdlib, and new users learning to use > > the language. So you need to make a case for why the benefit outweighs > > the costs. > > The best case I've come up with is one of obviousness. When looking > for the powerset implementation in the cpython source, grepping for > itertools.chain is not an obvious, at least to me, vector for a > solution. -- H > -- > OpenPGP: https://sks-keyservers.net/pks/lookup?op=get&search= > 0xFEBAD7FFD041BBA1 > If you wish to request my time, please do so using > bit.ly/hd1AppointmentRequest. > Si vous voudrais faire connnaisance, allez a bit.ly/hd1AppointmentRequest. > > Sent from my mobile device > Envoye de mon portable > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Oct 16 01:40:47 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 16 Oct 2018 18:40:47 +1300 Subject: [Python-ideas] Powerset In-Reply-To: References: <20181015230251.GM3817@ando.pearwood.info> Message-ID: <5BC579DF.6020603@canterbury.ac.nz> Wes Turner wrote: > Is there a name for an iteration of the powerset which is more useful > for binary search? I.e. instead of starting with null set, start with > the "middle" ( r/2 ). You'll have to provide more detail about what you want to search and how you intend to search it. There isn't a single "middle" to the set of powersets, since in general there are many subsets with about half the elements of the original set. Also there is no obvious ordering to use for bisection. -- Greg From paal.drange at gmail.com Tue Oct 16 02:24:25 2018 From: paal.drange at gmail.com (=?UTF-8?B?UMOlbCBHcsO4bsOlcyBEcmFuZ2U=?=) Date: Tue, 16 Oct 2018 08:24:25 +0200 Subject: [Python-ideas] Powerset In-Reply-To: References: <20181015230251.GM3817@ando.pearwood.info> Message-ID: > Is there a name for an iteration of the powerset which is more useful for binary search? I.e. instead of starting with null set, start with the "middle" ( r/2 ). > > Maybe a bit OT, but is there a name for such a combinatorial search? Not that I know of, especially since this has the unfortunate drawback that you're going through the largest part of the search space first. Usually, I find that I want to do the opposite from this; go from both ends simultaneously. Hasan, if you recall that the powerset is just `yield from S choose k for k from 0 to |S|+1`, you see that that is exactly the implementation in the examples page. Best regards, P?l GD -------------- next part -------------- An HTML attachment was scrubbed... URL: From hasan.diwan at gmail.com Tue Oct 16 02:27:26 2018 From: hasan.diwan at gmail.com (Hasan Diwan) Date: Mon, 15 Oct 2018 23:27:26 -0700 Subject: [Python-ideas] Powerset In-Reply-To: References: <20181015230251.GM3817@ando.pearwood.info> Message-ID: On Mon, 15 Oct 2018 at 23:25, P?l Gr?n?s Drange wrote: > Hasan, if you recall that the powerset is just > `yield from S choose k for k from 0 to |S|+1`, > you see that that is exactly the implementation in the examples page. I know that, but when one searches for a powerset function, the logical place to look isn't itertools, it's the set class. -- H -- OpenPGP: https://sks-keyservers.net/pks/lookup?op=get&search=0xFEBAD7FFD041BBA1 If you wish to request my time, please do so using bit.ly/hd1AppointmentRequest. Si vous voudrais faire connnaisance, allez a bit.ly/hd1AppointmentRequest. Sent from my mobile device Envoye de mon portable From waksman at gmail.com Tue Oct 16 04:02:03 2018 From: waksman at gmail.com (George Leslie-Waksman) Date: Tue, 16 Oct 2018 01:02:03 -0700 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: <5BC0AA42.7010900@canterbury.ac.nz> References: <20181011114138.GB3817@ando.pearwood.info> <5BC0AA42.7010900@canterbury.ac.nz> Message-ID: Would a frozendict require that keys and values be hashable? It seems to me that we would need this restriction to make a reasonably universal frozendict that is, itself, hashable. With this restriction for the general case, is there still sufficient value for everyone that is asking for a frozendict? Without this restriction and without frozendict being hashable, is there still sufficient value for everyone that is asking for a frozendict? On Fri, Oct 12, 2018 at 7:31 AM Greg Ewing wrote: > Chris Barker - NOAA Federal via Python-ideas wrote: > > > Or maybe come up with a new name > > We should call it a birdseyedict, because of this: > > > http://www.vulture.com/2016/12/unearthing-a-rare-1971-monty-python-film-all-about-peas.html > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paal.drange at gmail.com Tue Oct 16 04:36:53 2018 From: paal.drange at gmail.com (=?UTF-8?B?UMOlbCBHcsO4bsOlcyBEcmFuZ2U=?=) Date: Tue, 16 Oct 2018 10:36:53 +0200 Subject: [Python-ideas] Powerset In-Reply-To: References: <20181015230251.GM3817@ando.pearwood.info> Message-ID: > [...] when one searches for a powerset function, the > logical place to look isn't itertools, it's the set class. -- H That's a rather object-oriented view, I think. So you look for the permutation function in the list class? I prefer these functions gathered in one place, and I find that itertools does the job. I do however agree that there could be a powerset function there for convenience, but only +0. - P?l GD -------------- next part -------------- An HTML attachment was scrubbed... URL: From hasan.diwan at gmail.com Tue Oct 16 04:41:53 2018 From: hasan.diwan at gmail.com (Hasan Diwan) Date: Tue, 16 Oct 2018 01:41:53 -0700 Subject: [Python-ideas] Powerset In-Reply-To: References: <20181015230251.GM3817@ando.pearwood.info> Message-ID: Pal, On Tue, 16 Oct 2018 at 01:36, P?l Gr?n?s Drange wrote: > I do however agree that there could be a powerset function there for convenience, but only +0. That is the best argument I could come up with to justify a set#powerset method. -- H -- OpenPGP: https://sks-keyservers.net/pks/lookup?op=get&search=0xFEBAD7FFD041BBA1 If you wish to request my time, please do so using bit.ly/hd1AppointmentRequest. Si vous voudrais faire connnaisance, allez a bit.ly/hd1AppointmentRequest. Sent from my mobile device Envoye de mon portable From nicolas.rolin at tiime.fr Tue Oct 16 05:01:50 2018 From: nicolas.rolin at tiime.fr (Nicolas Rolin) Date: Tue, 16 Oct 2018 11:01:50 +0200 Subject: [Python-ideas] Powerset In-Reply-To: References: <20181015230251.GM3817@ando.pearwood.info> Message-ID: Can we get an utilisation context ? I don't think it belongs in the stdlib alone on the basis that its output is not linear in the size of its input (but exponential), so it explode even for a mid-sized list, which by nature limits greatly its usage. The question would be wether or not it is used enough to push it to itertools, which is pretty hard to decide with no examples. 2018-10-16 10:41 GMT+02:00 Hasan Diwan : > Pal, > On Tue, 16 Oct 2018 at 01:36, P?l Gr?n?s Drange > wrote: > > I do however agree that there could be a powerset function there for > convenience, but only +0. > > That is the best argument I could come up with to justify a > set#powerset method. -- H > -- > OpenPGP: https://sks-keyservers.net/pks/lookup?op=get&search= > 0xFEBAD7FFD041BBA1 > If you wish to request my time, please do so using > bit.ly/hd1AppointmentRequest. > Si vous voudrais faire connnaisance, allez a bit.ly/hd1AppointmentRequest. > > Sent from my mobile device > Envoye de mon portable > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- -- *Nicolas Rolin* | Data Scientist + 33 631992617 - nicolas.rolin at tiime.fr *15 rue Auber, **75009 Paris* *www.tiime.fr * -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Oct 16 05:26:52 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 16 Oct 2018 20:26:52 +1100 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: <20181011114138.GB3817@ando.pearwood.info> <5BC0AA42.7010900@canterbury.ac.nz> Message-ID: On Tue, Oct 16, 2018 at 7:02 PM George Leslie-Waksman wrote: > > Would a frozendict require that keys and values be hashable? Keys would already have to be hashable - regular dicts demand this, and there's no reason not to for frozendict. Values? Not so sure. Personally I would say that no, they don't HAVE to be hashable - but that the frozendict itself would then not be hashable. > It seems to me that we would need this restriction to make a reasonably universal frozendict that is, itself, hashable. > The tuple provides a good precedent here: >>> hash((1,2,3)) 2528502973977326415 >>> hash((1,2,[])) Traceback (most recent call last): File "", line 1, in TypeError: unhashable type: 'list' >>> ChrisA From steve at pearwood.info Tue Oct 16 05:23:39 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 16 Oct 2018 20:23:39 +1100 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: <20181011114138.GB3817@ando.pearwood.info> <5BC0AA42.7010900@canterbury.ac.nz> Message-ID: <20181016092338.GN3817@ando.pearwood.info> On Tue, Oct 16, 2018 at 01:02:03AM -0700, George Leslie-Waksman wrote: > Would a frozendict require that keys and values be hashable? Keys, yes. Values, no. If the values were hashable, the frozendict itself would also be hashable. If not, then it would be like trying to hash a tuple with unhashable items: py> hash((1, 2, {}, 3)) Traceback (most recent call last): File "", line 1, in TypeError: unhashable type: 'dict' > It seems to me that we would need this restriction to make a reasonably > universal frozendict that is, itself, hashable. When people talk about frozendicts being hashable, they mean it in the same sense that tuples are hashable. For what it is worth, here's an incomplete, quick and dirty proof of concept frozendict, using automatic delegation: # Not good enough for production, not tested, buyer beware, etc. class frozendict: def __new__(cls, *args, **kwargs): d = dict(*args, **kwargs) proxy = types.MappingProxyType(d) instance = super().__new__(cls) instance.__proxy = proxy return instance def __hash__(self): return hash(tuple(self.items())) def __getattr__(self, name): return getattr(self.__proxy, name) def __getitem__(self, key): return self.__proxy[key] def __repr__(self): return "%s(%r)" % (type(self).__name__, dict(self)) -- Steve From wes.turner at gmail.com Tue Oct 16 06:45:44 2018 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 16 Oct 2018 06:45:44 -0400 Subject: [Python-ideas] Powerset In-Reply-To: <5BC579DF.6020603@canterbury.ac.nz> References: <20181015230251.GM3817@ando.pearwood.info> <5BC579DF.6020603@canterbury.ac.nz> Message-ID: On Tuesday, October 16, 2018, Greg Ewing wrote: > Wes Turner wrote: > >> Is there a name for an iteration of the powerset which is more useful for >> binary search? I.e. instead of starting with null set, start with the >> "middle" ( r/2 ). >> > > You'll have to provide more detail about what you want to search > and how you intend to search it. There isn't a single "middle" to > the set of powersets, since in general there are many subsets with > about half the elements of the original set. Also there is no > obvious ordering to use for bisection. When searching for combinations of factors which most correlate to the dependent variable, it doesn't always make sense to start with single factors; especially when other factors 'cancel out'. For example, in clinical medicine, differential diagnosis is a matter of determining what the most likely diagnosis/es is/are; given lots of noise and one or more differentiating factors. Testing individual factors first may not be the most efficient because combinations/permutations are more likely to be highly correlated with specific diagnoses. Random search of the powerset and mutation (or a neuralnet) may be faster anyways. Just wondering whether there's a name for differently ordered powerset (and Cartesian product) traversals? Obviously, this is combinatorics and set theory (category theory (HOTT)); here in the itertools library for iterables. > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfine2358 at gmail.com Tue Oct 16 07:28:38 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Tue, 16 Oct 2018 12:28:38 +0100 Subject: [Python-ideas] Powerset In-Reply-To: References: <20181015230251.GM3817@ando.pearwood.info> <5BC579DF.6020603@canterbury.ac.nz> Message-ID: Wes Turner wrote: > Obviously, this is combinatorics and set theory (category theory (HOTT)); here in the itertools library for iterables. Powerset is a small problem, and HoTT is a big solution. I know this because I've got the standard book sitting, largely unread, on my bookshelf for over a year. https://en.wikipedia.org/wiki/Homotopy_type_theory https://homotopytypetheory.org/book/ HoTT is based on the Coq Proof Assistant: https://coq.inria.fr/. One of the applications of CoQ is certification of properties of programming languages. And now we're a long way from the original post and topic. -- Jonathan From szport at gmail.com Tue Oct 16 10:42:55 2018 From: szport at gmail.com (Zaur Shibzukhov) Date: Tue, 16 Oct 2018 07:42:55 -0700 (PDT) Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: <20181016092338.GN3817@ando.pearwood.info> References: <20181011114138.GB3817@ando.pearwood.info> <5BC0AA42.7010900@canterbury.ac.nz> <20181016092338.GN3817@ando.pearwood.info> Message-ID: <4c640043-9138-491e-8929-1b5d67d539f2@googlegroups.com> ???????, 16 ??????? 2018 ?., 12:29:55 UTC+3 ???????????? Steven D'Aprano ???????: > > > > It seems to me that we would need this restriction to make a reasonably > > universal frozendict that is, itself, hashable. > > When people talk about frozendicts being hashable, they mean it in the > same sense that tuples are hashable. > > > For what it is worth, here's an incomplete, quick and dirty proof of > concept frozendict, using automatic delegation: > > # Not good enough for production, not tested, buyer beware, etc. > class frozendict: > def __new__(cls, *args, **kwargs): > d = dict(*args, **kwargs) > proxy = types.MappingProxyType(d) > instance = super().__new__(cls) > instance.__proxy = proxy > return instance > def __hash__(self): > return hash(tuple(self.items())) > def __getattr__(self, name): > return getattr(self.__proxy, name) > def __getitem__(self, key): > return self.__proxy[key] > def __repr__(self): > return "%s(%r)" % (type(self).__name__, dict(self)) > > For those who need more performant variant (Cython compiled) of frozendict frozenmap project can be offer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From prometheus235 at gmail.com Tue Oct 16 13:09:10 2018 From: prometheus235 at gmail.com (Nick Timkovich) Date: Tue, 16 Oct 2018 12:09:10 -0500 Subject: [Python-ideas] Powerset In-Reply-To: References: <20181015230251.GM3817@ando.pearwood.info> Message-ID: "more-itertools" seems to be a modestly popular library (top 100-500?) that includes the stdlib itertools recipes and more. https://more-itertools.readthedocs.io/en/stable/api.html#more_itertools.powerset On Tue, Oct 16, 2018 at 3:42 AM Hasan Diwan wrote: > Pal, > On Tue, 16 Oct 2018 at 01:36, P?l Gr?n?s Drange > wrote: > > I do however agree that there could be a powerset function there for > convenience, but only +0. > > That is the best argument I could come up with to justify a > set#powerset method. -- H > -- > OpenPGP: > https://sks-keyservers.net/pks/lookup?op=get&search=0xFEBAD7FFD041BBA1 > If you wish to request my time, please do so using > bit.ly/hd1AppointmentRequest. > Si vous voudrais faire connnaisance, allez a bit.ly/hd1AppointmentRequest. > > Sent from my mobile device > Envoye de mon portable > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From philip.martin2007 at gmail.com Tue Oct 16 23:57:46 2018 From: philip.martin2007 at gmail.com (Philip Martin) Date: Tue, 16 Oct 2018 22:57:46 -0500 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: <20181016092338.GN3817@ando.pearwood.info> References: <20181011114138.GB3817@ando.pearwood.info> <5BC0AA42.7010900@canterbury.ac.nz> <20181016092338.GN3817@ando.pearwood.info> Message-ID: I think it could inherit from the Mapping abc? class frozendict(Mapping): def __new__(cls, *args, **kwargs): d = dict(*args, **kwargs) proxy = MappingProxyType(d) instance = super().__new__(cls) instance.__proxy = proxy return instance def __hash__(self): return hash(tuple(self.items())) def __getattr__(self, name): return getattr(self.__proxy, name) def __getitem__(self, key): return self.__proxy[key] def __iter__(self): return self.__proxy.__iter__() def __len__(self): return len(self.__proxy) def __repr__(self): return "%s(%r)" % (type(self).__name__, dict(self)) On Tue, Oct 16, 2018 at 4:29 AM Steven D'Aprano wrote: > On Tue, Oct 16, 2018 at 01:02:03AM -0700, George Leslie-Waksman wrote: > > Would a frozendict require that keys and values be hashable? > > Keys, yes. Values, no. > > If the values were hashable, the frozendict itself would also be > hashable. If not, then it would be like trying to hash a tuple with > unhashable items: > > py> hash((1, 2, {}, 3)) > Traceback (most recent call last): > File "", line 1, in > TypeError: unhashable type: 'dict' > > > > It seems to me that we would need this restriction to make a reasonably > > universal frozendict that is, itself, hashable. > > When people talk about frozendicts being hashable, they mean it in the > same sense that tuples are hashable. > > > For what it is worth, here's an incomplete, quick and dirty proof of > concept frozendict, using automatic delegation: > > # Not good enough for production, not tested, buyer beware, etc. > class frozendict: > def __new__(cls, *args, **kwargs): > d = dict(*args, **kwargs) > proxy = types.MappingProxyType(d) > instance = super().__new__(cls) > instance.__proxy = proxy > return instance > def __hash__(self): > return hash(tuple(self.items())) > def __getattr__(self, name): > return getattr(self.__proxy, name) > def __getitem__(self, key): > return self.__proxy[key] > def __repr__(self): > return "%s(%r)" % (type(self).__name__, dict(self)) > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpic at yourlabs.org Wed Oct 17 10:19:16 2018 From: jpic at yourlabs.org (Jamesie Pic) Date: Wed, 17 Oct 2018 16:19:16 +0200 Subject: [Python-ideas] [Distutils] Pypi private repo's In-Reply-To: References: <3075001d3cc57$370df900$a529eb00$@sdamon.com> <2A893F9F-C4CE-4470-BC1D-F556A7DA5253@me.com> Message-ID: Hi Nick, Well, when I created my company I had no intention to work on closed source projects, so "private repositories" is definitely not interesting for us as a feature. However, we're all for helping PyPA to make sustainable revenue, and also having more infra, and why not one day integrate gpg signature checking on packages we've been uploading with python setup.py sdist upload --sign so far .... Please contact me if interested. Have a great day. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpic at yourlabs.org Wed Oct 17 10:20:44 2018 From: jpic at yourlabs.org (Jamesie Pic) Date: Wed, 17 Oct 2018 16:20:44 +0200 Subject: [Python-ideas] [Distutils] Pypi private repo's In-Reply-To: References: <3075001d3cc57$370df900$a529eb00$@sdamon.com> <2A893F9F-C4CE-4470-BC1D-F556A7DA5253@me.com> Message-ID: PS: forgot to say, the name of the company i'm putting at your disposal for this project is YourLabs Business Service, we have hackers and funds at your disposal for this project. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Oct 17 10:44:25 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 18 Oct 2018 01:44:25 +1100 Subject: [Python-ideas] [Distutils] Pypi private repo's In-Reply-To: References: <3075001d3cc57$370df900$a529eb00$@sdamon.com> <2A893F9F-C4CE-4470-BC1D-F556A7DA5253@me.com> Message-ID: <20181017144424.GQ3817@ando.pearwood.info> Hi Jamesie, and welcome. You are responding to a six-month old email from Nick. (And I only know that much by the merest chance.) A bit of context would be nice, otherwise your post is rather mysterious and not very interesting. -- Steve From srkunze at mail.de Thu Oct 18 12:20:38 2018 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 18 Oct 2018 18:20:38 +0200 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: <20181011114138.GB3817@ando.pearwood.info> <20181011223528.GD3817@ando.pearwood.info> Message-ID: On 12.10.18 01:30, Chris Barker via Python-ideas wrote: > If Python had a way to check ABCs without issubclass, then I wouldn't > care about subclasses at all. I'd actually kind of like to have: > > hasinterface(an_object, (ABC1, ABC2, ABC3)) I actually like your idea and could imagine using "hasinterface" as some asserts. Great idea. +1 > Even though, it would be the same as issubclass() (though I'd like an > AND relationship with the tuple of ABCs..) When "hasinterface" ANDs the tuple, it's already different, isn't it? Btw., issubclass would require a class, not an instance. hasinterface, though, would work on all type of objects. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From boxed at killingar.net Thu Oct 18 12:49:19 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Thu, 18 Oct 2018 18:49:19 +0200 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: References: <20181011114138.GB3817@ando.pearwood.info> <20181011223528.GD3817@ando.pearwood.info> Message-ID: <34E2C837-7748-47C5-A2E3-5E2D5253927D@killingar.net> >> Even though, it would be the same as issubclass() (though I'd like an AND relationship with the tuple of ABCs..) > > When "hasinterface" ANDs the tuple, it's already different, isn't it? > If it's AND, shouldn't it be "hasinterfaces" (notice the s!)? One could also imagine that isinstance and issubclass taking a keyword argument for the logical operator. Maybe just something as simple as "isinstance(foo, (a, b), all=True)" / Anders -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Thu Oct 18 13:12:33 2018 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 18 Oct 2018 19:12:33 +0200 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: <34E2C837-7748-47C5-A2E3-5E2D5253927D@killingar.net> References: <20181011114138.GB3817@ando.pearwood.info> <20181011223528.GD3817@ando.pearwood.info> <34E2C837-7748-47C5-A2E3-5E2D5253927D@killingar.net> Message-ID: <5be9d8d1-074b-e44f-62a8-08c566c81801@mail.de> On 18.10.18 18:49, Anders Hovm?ller wrote: > If it's AND, shouldn't it be "hasinterfaces" (notice the s!)? Yeah, could be. To be sure, we are on the same page here: "interface" refers to a set of attributes of the object in question, does it? E.g. like the __iter__ iterface. I usually don't care about the actual inheritance hierarchy but care about functionality. > One could also imagine that isinstance and issubclass taking a keyword > argument for the logical operator. Maybe just something as simple as > "isinstance(foo, (a, b), all=True)" Does AND even make sense for isinstance/issubclass? Cheers, Sven From boxed at killingar.net Thu Oct 18 13:26:57 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Thu, 18 Oct 2018 19:26:57 +0200 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: <5be9d8d1-074b-e44f-62a8-08c566c81801@mail.de> References: <20181011114138.GB3817@ando.pearwood.info> <20181011223528.GD3817@ando.pearwood.info> <34E2C837-7748-47C5-A2E3-5E2D5253927D@killingar.net> <5be9d8d1-074b-e44f-62a8-08c566c81801@mail.de> Message-ID: > Does AND even make sense for isinstance/issubclass? Why wouldn't it? Python supports multiple inheritance. / Anders From chris.barker at noaa.gov Thu Oct 18 15:35:36 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 18 Oct 2018 12:35:36 -0700 Subject: [Python-ideas] Revisiting Immutable Mappings In-Reply-To: <5be9d8d1-074b-e44f-62a8-08c566c81801@mail.de> References: <20181011114138.GB3817@ando.pearwood.info> <20181011223528.GD3817@ando.pearwood.info> <34E2C837-7748-47C5-A2E3-5E2D5253927D@killingar.net> <5be9d8d1-074b-e44f-62a8-08c566c81801@mail.de> Message-ID: On Thu, Oct 18, 2018 at 10:12 AM, Sven R. Kunze wrote: > On 18.10.18 18:49, Anders Hovm?ller wrote: > >> If it's AND, shouldn't it be "hasinterfaces" (notice the s!)? > > yeah, that would make sense. Is someone proposing something here? The point I was making, is in the case of ABCs: issubclass(a_class, an_abc) isn't "Really" testing a subclass, but really an interface. That is - the ABC exists solely to define an interface, not provide any functionality (though there is some in there), and you can, in fact, use ABCs to provide a way to check a duck-typed class that isn't an actual subclass. So do we need a new function that makes that clear? would it functionally be an different than issubclass with an ABC? I honestly don't know. Though an easy way to get the "and" behaviour -- i.e. interfaceS would be nice. -CHB To be sure, we are on the same page here: "interface" refers to a set of > attributes of the object in question, does it? > > E.g. like the __iter__ iterface. I usually don't care about the actual > inheritance hierarchy but care about functionality. > > One could also imagine that isinstance and issubclass taking a keyword >> argument for the logical operator. Maybe just something as simple as >> "isinstance(foo, (a, b), all=True)" >> > > Does AND even make sense for isinstance/issubclass? > > Cheers, > Sven > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Fri Oct 19 03:18:02 2018 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Fri, 19 Oct 2018 09:18:02 +0200 Subject: [Python-ideas] TypeHinting: From variable name to type Message-ID: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> Imaging you are developing in the django context. Everytime you use a variable named "request" or "response" your human brains knows that this is a subclass of django.http.HttpRequest and django.http.HttpResponse. How to give the IDE this knowledge? One solution is the do typehinting everywhere the veriable gets used. But why does the human brain not need this? Because it is intelligent? I would not call this intelligence. There is a simple dictionary in the brain of the developer, which maps: variable-name --> Type And this mapping dict exists once per library. If you are developing in the requests http lib, then there is a different mapping. Then "response" means type requests.Response. Now my idea: Per module and/or per file type hinting from variable name. Maybe a magic docstring in the __init__.py file: """ variable-name-mapping: { request: django.http.HttpRequest, ... } """ This makes the type mapping available for all files in this directory (or sub-directories). What do you think? Disclaimer: For reasons I don't want to explain in detail, I am not allowed to do implement things like this. This is just an idea. I would feel proud and thankfull if someone likes it that much, that he implements it. -- Thomas Guettler http://www.thomas-guettler.de/ I am looking for feedback: https://github.com/guettli/programming-guidelines From boxed at killingar.net Fri Oct 19 03:29:38 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Fri, 19 Oct 2018 09:29:38 +0200 Subject: [Python-ideas] TypeHinting: From variable name to type In-Reply-To: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> Message-ID: <3A295990-52A3-4347-865E-096E9126C6C7@killingar.net> > On 19 Oct 2018, at 09:18, Thomas G?ttler wrote: > > Imaging you are developing in the django context. > > Everytime you use a variable named "request" or "response" your human brains > knows that this is a subclass of django.http.HttpRequest and django.http.HttpResponse. > > How to give the IDE this knowledge? > > One solution is the do typehinting everywhere the veriable gets used. > > But why does the human brain not need this? > > Because it is intelligent? > > I would not call this intelligence. There is a simple > dictionary in the brain of the developer, which maps: > > variable-name --> Type > > And this mapping dict exists once per library. > > If you are developing in the requests http lib, then > there is a different mapping. Then "response" means > type requests.Response. > > > Now my idea: Per module and/or per file type hinting from variable name. > > Maybe a magic docstring in the __init__.py file: > > """ > variable-name-mapping: > { > request: django.http.HttpRequest, > ... > } > """ > > This makes the type mapping available for all files in this directory (or sub-directories). > > What do you think? I tried to implement this in mypy quite recently actually, but gave up for various reasons. I am very much +1 on this. This would be a huge boon to preexisting code bases. / Anders From robertve92 at gmail.com Fri Oct 19 04:22:08 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Fri, 19 Oct 2018 10:22:08 +0200 Subject: [Python-ideas] TypeHinting: From variable name to type In-Reply-To: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> Message-ID: That would be a feature in my "python dialect" future library. Taking one valid python program and producing another python program : def f(request): return HttpResponse('ok') ? def f(request: HttpRequest): return HttpResponse('ok') The dialect options would include "all top functions in this module" for example, and parameters about the naming convention (explicit list of conversion or rules to create the mapping). Le ven. 19 oct. 2018 ? 09:18, Thomas G?ttler a ?crit : > Imaging you are developing in the django context. > > Everytime you use a variable named "request" or "response" your human > brains > knows that this is a subclass of django.http.HttpRequest and > django.http.HttpResponse. > > How to give the IDE this knowledge? > > One solution is the do typehinting everywhere the veriable gets used. > > But why does the human brain not need this? > > Because it is intelligent? > > I would not call this intelligence. There is a simple > dictionary in the brain of the developer, which maps: > > variable-name --> Type > > And this mapping dict exists once per library. > > If you are developing in the requests http lib, then > there is a different mapping. Then "response" means > type requests.Response. > > > Now my idea: Per module and/or per file type hinting from variable name. > > Maybe a magic docstring in the __init__.py file: > > """ > variable-name-mapping: > { > request: django.http.HttpRequest, > ... > } > """ > > This makes the type mapping available for all files in this directory (or > sub-directories). > > What do you think? > > > > > Disclaimer: For reasons I don't want to explain in detail, I am not > allowed to > do implement things like this. This is just an idea. I would feel proud > and thankfull > if someone likes it that much, that he implements it. > > > > > -- > Thomas Guettler http://www.thomas-guettler.de/ > I am looking for feedback: > https://github.com/guettli/programming-guidelines > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Fri Oct 19 05:30:53 2018 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Fri, 19 Oct 2018 11:30:53 +0200 Subject: [Python-ideas] TypeHinting: From variable name to type In-Reply-To: References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> Message-ID: <0c2edac3-36de-723f-2571-a782fa118965@thomas-guettler.de> Am 19.10.18 um 10:22 schrieb Robert Vanden Eynde: > That would be a feature in my "python dialect" future library. Taking one valid python program and producing another > python program : > > def f(request): > ? ? return HttpResponse('ok') > > ? > > def f(request: HttpRequest): > return HttpResponse('ok') > > The dialect options would include "all top functions in this module" for example, and parameters about the naming > convention (explicit list of conversion or rules to create the mapping). I think creating source code from source code is most of the time not the best solution. Except you call the result not "source code" but "binary executable code". [1] I think it would be great if this could work for local variables (which are not in the arg/kwargs of the method), too: Example: def foo(): response = get_response_from_paradise() If `get_response_from_paradise()` has not spec for the return type, then it would be great to guess the return type from the general "wisdom". Regards, Thomas [1]: https://github.com/guettli/programming-guidelines/blob/master/README.rst#source-code-generation-is-a-stupid-idea -- Thomas Guettler http://www.thomas-guettler.de/ I am looking for feedback: https://github.com/guettli/programming-guidelines From levkivskyi at gmail.com Fri Oct 19 05:42:04 2018 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Fri, 19 Oct 2018 10:42:04 +0100 Subject: [Python-ideas] TypeHinting: From variable name to type In-Reply-To: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> Message-ID: On Fri, 19 Oct 2018 at 08:19, Thomas G?ttler wrote: > [...] > This makes the type mapping available for all files in this directory (or > sub-directories). > What do you think? > I don't think this should be a language feature. _Maybe_ it can be some kind of an agreed format between IDEs. But in any case I would first discuss this on PyCharm and VScode (IntelliSense?) trackers. Also I don't think it is a good fit for mypy (and probably other type checkers, but better ask pytype and pyre devs). I would rather expect that IDEs auto-add annotations using some guided heuristics, and then type checkers can verify them. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bmbartnicki at gmail.com Fri Oct 19 05:55:42 2018 From: bmbartnicki at gmail.com (=?UTF-8?Q?Bart=C5=82omiej_Bartnicki?=) Date: Fri, 19 Oct 2018 11:55:42 +0200 Subject: [Python-ideas] Python-ideas Digest, Vol 143, Issue 73 In-Reply-To: References: Message-ID: I agree with Ivan. This feater is stands in contradiction with second Python Zen principle: "Explicit is better than implicit.". Isn't it? On Fri, Oct 19, 2018 at 11:41 AM wrote: > Send Python-ideas mailing list submissions to > python-ideas at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/python-ideas > or, via email, send a message with subject or body 'help' to > python-ideas-request at python.org > > You can reach the person managing the list at > python-ideas-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-ideas digest..." > > > Today's Topics: > > 1. TypeHinting: From variable name to type (Thomas G?ttler) > 2. Re: TypeHinting: From variable name to type (Anders Hovm?ller) > 3. Re: TypeHinting: From variable name to type (Robert Vanden Eynde) > 4. Re: TypeHinting: From variable name to type (Thomas G?ttler) > 5. Re: TypeHinting: From variable name to type (Ivan Levkivskyi) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 19 Oct 2018 09:18:02 +0200 > From: Thomas G?ttler > To: python-ideas at python.org > Subject: [Python-ideas] TypeHinting: From variable name to type > Message-ID: <2d9d1825-2619-e4a0-1b19-6a614666816a at thomas-guettler.de> > Content-Type: text/plain; charset=utf-8; format=flowed > > Imaging you are developing in the django context. > > Everytime you use a variable named "request" or "response" your human > brains > knows that this is a subclass of django.http.HttpRequest and > django.http.HttpResponse. > > How to give the IDE this knowledge? > > One solution is the do typehinting everywhere the veriable gets used. > > But why does the human brain not need this? > > Because it is intelligent? > > I would not call this intelligence. There is a simple > dictionary in the brain of the developer, which maps: > > variable-name --> Type > > And this mapping dict exists once per library. > > If you are developing in the requests http lib, then > there is a different mapping. Then "response" means > type requests.Response. > > > Now my idea: Per module and/or per file type hinting from variable name. > > Maybe a magic docstring in the __init__.py file: > > """ > variable-name-mapping: > { > request: django.http.HttpRequest, > ... > } > """ > > This makes the type mapping available for all files in this directory (or > sub-directories). > > What do you think? > > > > > Disclaimer: For reasons I don't want to explain in detail, I am not > allowed to > do implement things like this. This is just an idea. I would feel proud > and thankfull > if someone likes it that much, that he implements it. > > > > > -- > Thomas Guettler http://www.thomas-guettler.de/ > I am looking for feedback: > https://github.com/guettli/programming-guidelines > > > ------------------------------ > > Message: 2 > Date: Fri, 19 Oct 2018 09:29:38 +0200 > From: Anders Hovm?ller > To: Thomas G?ttler > Cc: python-ideas at python.org > Subject: Re: [Python-ideas] TypeHinting: From variable name to type > Message-ID: <3A295990-52A3-4347-865E-096E9126C6C7 at killingar.net> > Content-Type: text/plain; charset=utf-8 > > > > > On 19 Oct 2018, at 09:18, Thomas G?ttler > wrote: > > > > Imaging you are developing in the django context. > > > > Everytime you use a variable named "request" or "response" your human > brains > > knows that this is a subclass of django.http.HttpRequest and > django.http.HttpResponse. > > > > How to give the IDE this knowledge? > > > > One solution is the do typehinting everywhere the veriable gets used. > > > > But why does the human brain not need this? > > > > Because it is intelligent? > > > > I would not call this intelligence. There is a simple > > dictionary in the brain of the developer, which maps: > > > > variable-name --> Type > > > > And this mapping dict exists once per library. > > > > If you are developing in the requests http lib, then > > there is a different mapping. Then "response" means > > type requests.Response. > > > > > > Now my idea: Per module and/or per file type hinting from variable name. > > > > Maybe a magic docstring in the __init__.py file: > > > > """ > > variable-name-mapping: > > { > > request: django.http.HttpRequest, > > ... > > } > > """ > > > > This makes the type mapping available for all files in this directory > (or sub-directories). > > > > What do you think? > > > I tried to implement this in mypy quite recently actually, but gave up for > various reasons. > > I am very much +1 on this. This would be a huge boon to preexisting code > bases. > > / Anders > > > > > ------------------------------ > > Message: 3 > Date: Fri, 19 Oct 2018 10:22:08 +0200 > From: Robert Vanden Eynde > To: Thomas G?ttler > Cc: python-ideas > Subject: Re: [Python-ideas] TypeHinting: From variable name to type > Message-ID: > < > CA+msPNkhxXxCWkB4f_YbeR3oJQLNBWLGzvsOfouW+BQ7nonFyA at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > That would be a feature in my "python dialect" future library. Taking one > valid python program and producing another python program : > > def f(request): > return HttpResponse('ok') > > ? > > def f(request: HttpRequest): > return HttpResponse('ok') > > The dialect options would include "all top functions in this module" for > example, and parameters about the naming convention (explicit list of > conversion or rules to create the mapping). > > Le ven. 19 oct. 2018 ? 09:18, Thomas G?ttler > > a ?crit : > > > Imaging you are developing in the django context. > > > > Everytime you use a variable named "request" or "response" your human > > brains > > knows that this is a subclass of django.http.HttpRequest and > > django.http.HttpResponse. > > > > How to give the IDE this knowledge? > > > > One solution is the do typehinting everywhere the veriable gets used. > > > > But why does the human brain not need this? > > > > Because it is intelligent? > > > > I would not call this intelligence. There is a simple > > dictionary in the brain of the developer, which maps: > > > > variable-name --> Type > > > > And this mapping dict exists once per library. > > > > If you are developing in the requests http lib, then > > there is a different mapping. Then "response" means > > type requests.Response. > > > > > > Now my idea: Per module and/or per file type hinting from variable name. > > > > Maybe a magic docstring in the __init__.py file: > > > > """ > > variable-name-mapping: > > { > > request: django.http.HttpRequest, > > ... > > } > > """ > > > > This makes the type mapping available for all files in this directory (or > > sub-directories). > > > > What do you think? > > > > > > > > > > Disclaimer: For reasons I don't want to explain in detail, I am not > > allowed to > > do implement things like this. This is just an idea. I would feel proud > > and thankfull > > if someone likes it that much, that he implements it. > > > > > > > > > > -- > > Thomas Guettler http://www.thomas-guettler.de/ > > I am looking for feedback: > > https://github.com/guettli/programming-guidelines > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/python-ideas/attachments/20181019/cdfa7215/attachment-0001.html > > > > ------------------------------ > > Message: 4 > Date: Fri, 19 Oct 2018 11:30:53 +0200 > From: Thomas G?ttler > To: Robert Vanden Eynde > Cc: python-ideas > Subject: Re: [Python-ideas] TypeHinting: From variable name to type > Message-ID: <0c2edac3-36de-723f-2571-a782fa118965 at thomas-guettler.de> > Content-Type: text/plain; charset=utf-8; format=flowed > > > > Am 19.10.18 um 10:22 schrieb Robert Vanden Eynde: > > That would be a feature in my "python dialect" future library. Taking > one valid python program and producing another > > python program : > > > > def f(request): > > ? ? return HttpResponse('ok') > > > > ? > > > > def f(request: HttpRequest): > > return HttpResponse('ok') > > > > The dialect options would include "all top functions in this module" for > example, and parameters about the naming > > convention (explicit list of conversion or rules to create the mapping). > > > I think creating source code from source code is most of the time not the > best solution. > Except you call the result not "source code" but "binary executable code". > [1] > > I think it would be great if this could work for local variables (which > are not in the arg/kwargs of the method), too: > > Example: > > def foo(): > response = get_response_from_paradise() > > If `get_response_from_paradise()` has not spec for the return type, then > it would be great to guess > the return type from the general "wisdom". > > Regards, > Thomas > > > > > > [1]: > https://github.com/guettli/programming-guidelines/blob/master/README.rst#source-code-generation-is-a-stupid-idea > > > > > > -- > Thomas Guettler http://www.thomas-guettler.de/ > I am looking for feedback: > https://github.com/guettli/programming-guidelines > > > ------------------------------ > > Message: 5 > Date: Fri, 19 Oct 2018 10:42:04 +0100 > From: Ivan Levkivskyi > To: guettliml at thomas-guettler.de > Cc: python-ideas > Subject: Re: [Python-ideas] TypeHinting: From variable name to type > Message-ID: > LNM7vJCNLQq-j4t7h8Oz0T6Q at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On Fri, 19 Oct 2018 at 08:19, Thomas G?ttler > > wrote: > > > [...] > > This makes the type mapping available for all files in this directory (or > > sub-directories). > > What do you think? > > > > I don't think this should be a language feature. _Maybe_ it can be some > kind of an agreed format between IDEs. > But in any case I would first discuss this on PyCharm and VScode > (IntelliSense?) trackers. > > Also I don't think it is a good fit for mypy (and probably other type > checkers, but better ask pytype and pyre devs). > I would rather expect that IDEs auto-add annotations using some guided > heuristics, and then type checkers can verify them. > > -- > Ivan > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/python-ideas/attachments/20181019/5cdb785c/attachment.html > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > > ------------------------------ > > End of Python-ideas Digest, Vol 143, Issue 73 > ********************************************* > -- Pozdrawiam, Bart?omiej Bartnicki -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Fri Oct 19 06:08:29 2018 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Fri, 19 Oct 2018 12:08:29 +0200 Subject: [Python-ideas] TypeHinting: From variable name to type (IDE Support only) In-Reply-To: References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> Message-ID: Am 19.10.18 um 11:42 schrieb Ivan Levkivskyi: > On Fri, 19 Oct 2018 at 08:19, Thomas G?ttler > wrote: > > [...] > This makes the type mapping available for all files in this directory (or sub-directories). > What do you think? > > > I don't think this should be a language feature. _Maybe_ it can be some kind of an agreed format between IDEs. > But in any case I would first discuss this on PyCharm and VScode (IntelliSense?) trackers. > > Also I don't think it is a good fit for mypy (and probably other type checkers, but better ask pytype and pyre devs). > I would rather expect that IDEs auto-add annotations using some guided heuristics, and then type checkers can verify them. I agree. For me IDE support is all I need. Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ I am looking for feedback: https://github.com/guettli/programming-guidelines From steve at pearwood.info Fri Oct 19 06:15:49 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 19 Oct 2018 21:15:49 +1100 Subject: [Python-ideas] TypeHinting: From variable name to type In-Reply-To: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> Message-ID: <20181019101549.GR3817@ando.pearwood.info> On Fri, Oct 19, 2018 at 09:18:02AM +0200, Thomas G?ttler wrote: > Imaging you are developing in the django context. > > Everytime you use a variable named "request" or "response" your human brains > knows that this is a subclass of django.http.HttpRequest and > django.http.HttpResponse. Not *my* human brain. I've seen far too many variables called (let's say) "mylist" which actually hold a dict or a tuple (or in one memorable case, a string!) to unconditionally believe the name. But I'll accept that given the context, there's a very strong likelihood that a variable called "request" might be a HttpRequest object, and one called "response" might be a HttpResponse object. > How to give the IDE this knowledge? > > One solution is the do typehinting everywhere the veriable gets used. You shouldn't need to. Your IDE or type-checker should be able to do type-inference and infer the type. You only need to add a hint if it cannot infer the type. If your IDE doesn't do type inference, get a better IDE *wink* > But why does the human brain not need this? > > Because it is intelligent? Yes. > I would not call this intelligence. There is a simple > dictionary in the brain of the developer, which maps: > > variable-name --> Type Is that a fact? How do you know? How does the information in this dict get filled in? I think it is FAR more likely that it is intelligence: the human programmer understands the *meaning of the code* and would equally recognise that myresponse response2 reply answer rspns are all HttpResponse objects too, from the context in which they appear. We do that because we know the meaning of the words, and start from the default assumption that the coder isn't lying to us by using a name like "mylist" to represent a floating point number or "page_size" to represent the name of a file. In other words, we *read* and *understand* the code, not just mechanically map names to types. That's why we have no problem with dynamically typed languages like Python were the one name can refer to objects of many different types. When the context is different, we interpret the name differently: response = input("Continue? Y/n ") response = chatbot.lookup(question) response = mapping[challenge] would all be interpreted differently, even if the module used Django. I doubt any reasonable programmer would imagine that they were HttpResponse objects. > And this mapping dict exists once per library. Or more likely, doesn't exist at all. > If you are developing in the requests http lib, then > there is a different mapping. Then "response" means > type requests.Response. What if you are using *both* django and requests in the same module? You could have both of these: response = something_returning_requests_Response() response = something_returning_django_HttpResponse() in the one module. -- Steve From boxed at killingar.net Fri Oct 19 07:14:39 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Fri, 19 Oct 2018 13:14:39 +0200 Subject: [Python-ideas] TypeHinting: From variable name to type In-Reply-To: <20181019101549.GR3817@ando.pearwood.info> References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> <20181019101549.GR3817@ando.pearwood.info> Message-ID: > On 19 Oct 2018, at 12:15, Steven D'Aprano wrote: > > On Fri, Oct 19, 2018 at 09:18:02AM +0200, Thomas G?ttler wrote: > >> Imaging you are developing in the django context. >> >> Everytime you use a variable named "request" or "response" your human brains >> knows that this is a subclass of django.http.HttpRequest and >> django.http.HttpResponse. > > Not *my* human brain. > > I've seen far too many variables called (let's say) "mylist" which > actually hold a dict or a tuple (or in one memorable case, a string!) to > unconditionally believe the name. This is an even stronger argument for the proposal I think. If IDEs and static analysis tools looked at stuff like this and assumed a type _and then found that this assumption is violated_, it would be a huge win! >> How to give the IDE this knowledge? >> >> One solution is the do typehinting everywhere the veriable gets used. > > You shouldn't need to. Your IDE or type-checker should be able to do > type-inference and infer the type. You only need to add a hint if it > cannot infer the type. > > If your IDE doesn't do type inference, get a better IDE *wink* Which IDE would this be? PyCharm doesn't do this in the general case. Not even close in the code base I work on. >> And this mapping dict exists once per library. > > Or more likely, doesn't exist at all. You seem to argue here, and generally, that the normal case for code bases is that you have no consistent naming. This seems strange to me. Can I ask what type of code and how big code bases you work on? To me it seems we have radically different perspectives, and we'll continue talking past each other if we don't explain those perspectives to each other. I'll go first: I work on a code base that: - is ~240k dry lines - is ~15 years old - it a web app with just one production install - has very good consistent naming, for example: - the "request" example above exists over 6000 times in the code base and all of them are HttpRequestBase-derived (or in a few cases a mock that pretends to be that). - the word "party" exists over 11 thousand times, all of them are an instance of the Party django model Being able to tell PyCharm to assume the type of these and flag obvious violations would be great. >> If you are developing in the requests http lib, then >> there is a different mapping. Then "response" means >> type requests.Response. > > What if you are using *both* django and requests in the same module? You > could have both of these: > > response = something_returning_requests_Response() > > response = something_returning_django_HttpResponse() > > in the one module. Then you'd use the normal typing annotations to override the default. / Anders From elazarg at gmail.com Fri Oct 19 09:32:34 2018 From: elazarg at gmail.com (Elazar) Date: Fri, 19 Oct 2018 16:32:34 +0300 Subject: [Python-ideas] TypeHinting: From variable name to type In-Reply-To: References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> <20181019101549.GR3817@ando.pearwood.info> Message-ID: Related: https://github.com/python/mypy/issues/2103 - Elazar On Fri, Oct 19, 2018 at 2:15 PM Anders Hovm?ller wrote: > > > > On 19 Oct 2018, at 12:15, Steven D'Aprano wrote: > > > > On Fri, Oct 19, 2018 at 09:18:02AM +0200, Thomas G?ttler wrote: > > > >> Imaging you are developing in the django context. > >> > >> Everytime you use a variable named "request" or "response" your human > brains > >> knows that this is a subclass of django.http.HttpRequest and > >> django.http.HttpResponse. > > > > Not *my* human brain. > > > > I've seen far too many variables called (let's say) "mylist" which > > actually hold a dict or a tuple (or in one memorable case, a string!) to > > unconditionally believe the name. > > This is an even stronger argument for the proposal I think. If IDEs and > static analysis tools looked at stuff like this and assumed a type _and > then found that this assumption is violated_, it would be a huge win! > > >> How to give the IDE this knowledge? > >> > >> One solution is the do typehinting everywhere the veriable gets used. > > > > You shouldn't need to. Your IDE or type-checker should be able to do > > type-inference and infer the type. You only need to add a hint if it > > cannot infer the type. > > > > If your IDE doesn't do type inference, get a better IDE *wink* > > > Which IDE would this be? PyCharm doesn't do this in the general case. Not > even close in the code base I work on. > > > >> And this mapping dict exists once per library. > > > > Or more likely, doesn't exist at all. > > You seem to argue here, and generally, that the normal case for code bases > is that you have no consistent naming. This seems strange to me. Can I ask > what type of code and how big code bases you work on? To me it seems we > have radically different perspectives, and we'll continue talking past each > other if we don't explain those perspectives to each other. > > I'll go first: I work on a code base that: > - is ~240k dry lines > - is ~15 years old > - it a web app with just one production install > - has very good consistent naming, for example: > - the "request" example above exists over 6000 times in the code base > and all of them are HttpRequestBase-derived (or in a few cases a mock that > pretends to be that). > - the word "party" exists over 11 thousand times, all of them are an > instance of the Party django model > > Being able to tell PyCharm to assume the type of these and flag obvious > violations would be great. > > > >> If you are developing in the requests http lib, then > >> there is a different mapping. Then "response" means > >> type requests.Response. > > > > What if you are using *both* django and requests in the same module? You > > could have both of these: > > > > response = something_returning_requests_Response() > > > > response = something_returning_django_HttpResponse() > > > > in the one module. > > Then you'd use the normal typing annotations to override the default. > > > / Anders > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Sat Oct 20 13:53:18 2018 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler_Lists?=) Date: Sat, 20 Oct 2018 19:53:18 +0200 Subject: [Python-ideas] TypeHinting: From variable name to type In-Reply-To: <20181019101549.GR3817@ando.pearwood.info> References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> <20181019101549.GR3817@ando.pearwood.info> Message-ID: <3cad7e64-6781-f3ad-1d14-c3a4e93cd037@thomas-guettler.de> Am 19.10.18 um 12:15 schrieb Steven D'Aprano: > On Fri, Oct 19, 2018 at 09:18:02AM +0200, Thomas G?ttler wrote: > >> Imaging you are developing in the django context. >> >> Everytime you use a variable named "request" or "response" your human brains >> knows that this is a subclass of django.http.HttpRequest and >> django.http.HttpResponse. > Not *my* human brain. I know that I know nothing, or at least not much. My laptop does compute much faster than I do > I've seen far too many variables called (let's say) "mylist" which > actually hold a dict or a tuple (or in one memorable case, a string!) to > unconditionally believe the name. > > But I'll accept that given the context, there's a very strong likelihood > that a variable called "request" might be a HttpRequest object, and one > called "response" might be a HttpResponse object. Great, we agree here. > >> How to give the IDE this knowledge? >> >> One solution is the do typehinting everywhere the veriable gets used. > You shouldn't need to. Your IDE or type-checker should be able to do > type-inference and infer the type. You only need to add a hint if it > cannot infer the type. > > If your IDE doesn't do type inference, get a better IDE *wink* I use the free community version of pyCharm, and it often does not know which type a variable has. A simple name2type mapping would improve the situation a lot. > >> But why does the human brain not need this? >> >> Because it is intelligent? > Yes. > > >> I would not call this intelligence. There is a simple >> dictionary in the brain of the developer, which maps: >> >> variable-name --> Type > Is that a fact? How do you know? > > How does the information in this dict get filled in? as I already said before a docstring in the __init__.py file would be a pragmatic solution. > I think it is FAR more likely that it is intelligence: the human > programmer understands the *meaning of the code* and would equally > recognise that > > myresponse > response2 > reply > answer > rspns I try to use the same variable name again and again. I guess in my code base in 95% an instance of HttpRequest is stored in a variable called "request". You say:? the human programmer understands the *meaning of the code* Great if that is the case for you. I often look at the code and don't get it at all. I wrote the lines two years ago, and today I don't understand why they exist. This does not happen daily, but at least once a month. > In other words, we *read* and *understand* the code, not just > mechanically map names to types. That's why we have no problem with > dynamically typed languages like Python were the one name can refer to > objects of many different types. > > When the context is different, we interpret the name differently: > > response = input("Continue? Y/n ") > > response = chatbot.lookup(question) > > response = mapping[challenge] > > > would all be interpreted differently, even if the module used Django. I > doubt any reasonable programmer would imagine that they were > HttpResponse objects. relax. I mean this like I said. Please relax. Imagine you have this: reposense = obj.get_response() First: if the variable is the return value of a method, and the method has type-annotated the return value, then this type information gets used. End of type detect. If the type information can't be found from the method, then use the name2type mapping which I suggest in this mail. > >> And this mapping dict exists once per library. > Or more likely, doesn't exist at all. > > >> If you are developing in the requests http lib, then >> there is a different mapping. Then "response" means >> type requests.Response. > What if you are using *both* django and requests in the same module? You > could have both of these: > > response = something_returning_requests_Response() > > response = something_returning_django_HttpResponse() > > in the one module. > Do you really care for this edge-case? I don't. The way to get the type info is the same: Check the method, if it has it use this. Do not use name2type mapping. End of type detection. Then check the current file or the above __init__.py of the code. Take name2type mapping from there. Regards, ? Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ I am looking for feedback: https://github.com/guettli/programming-guidelines From initalize.damla at gmail.com Sun Oct 21 06:58:19 2018 From: initalize.damla at gmail.com (Damla Altun) Date: Sun, 21 Oct 2018 13:58:19 +0300 Subject: [Python-ideas] IPLike objects Message-ID: Hi, Python's ipaddress module is really good for working with network / IP related thing. I would like to use ipaddress.IPv4/v6Address in creating network interfaces. Like a socket but i can't, the internal things and some standard libraries doesn't support them (even socket). >>> server_address = (ipaddress.IPv4Address('127.0.0.1'), 10000) >>> sock.bind(server_address) Traceback (most recent call last): File "", line 1, in TypeError: str, bytes or bytearray expected, not IPv4Address Proposal ======== The best solution we've got[0] is implementing something like PathLike[1] for IP Address objects. For PoC[2] we called it IPLike. It is a protocol (some kind of interface) created with Abstract Base Classes and it forces class to implement __ipaddr__ method (like __fspath__ method). The __ipaddr__ method should return string or bytes formed notation of IP address. It is dotted notation in IPv4, long formed notation in IPv6. The __ipaddr__ method is also an identifier for IPLike classes. IPLike interface's subclasshook checks the __ipaddr__ method of given class when issubclass called. >>> class SomethingRepresentIP: ... def __ipaddr__(self) -> str: ... return '127.0.0.1' ... >>> issubclass(SomethingRepresentIP, IPLike) True >>> isinstance(SomethingRepresentIP(), IPLike) True For PoC[2] we implented __ipaddr__ method to ipaddress._BaseAddress class. It is parent class of IPv4/v6Adress. >>> issubclass(IPv4Address, IPLike) True >>> IPv4Address('127.0.0.1').__ipaddr__() '127.0.0.1' Like os.fspath we need a general-purpose ip-getter. So we implemented ipaddress.get_ipaddr(address) function. When you give it an valid IP like object it returns the value of __ipaddr__ method. >>> get_ipaddr(IPv4Address('127.0.0.1')) '127.0.0.1' When you give it string or bytes or integer object it calls ipaddr.ip_address(address) and re-calls itself with the value it got from ip_address. It helps get_ipaddr to convert integer-formed / byte-formed IPs to string formed IPs with a valid notation. >>> get_ipaddr('127.0.0.1') '127.0.0.1' >>> get_ipaddr(3232235521) '192.168.0.1' >>> get_ipaddr(b'\xC0\xA8\x00\x01') '192.168.0.1' If you give any type else it raises TypeError. Also if you give it a IP like object (an object that has __ipaddr__ method), but the object's __ipaddr__ method returns something that is not string/bytes it raises TypeError. For PoC[2] i added socket to support IP like objects. >>> sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) >>> server_address = (SomethingRepresentIP(), 10000) >>> sock.bind(server_address) >>> sock.listen(1) >>> conn, client = sock.accept() >>> conn I didn't implement a C API for ipaddress.get_ipaddr. The function is too small for creating a new module, new header and the ipaddress library currently has no C-side module. If needed i can write a function inside the socketmodule.c . Proof-of-Concept (Implementation) ================================= The implementation can be found on github (DamlaAltun/cpython[2]) We have a Abstract Base Class for IP Address protocol called IPLike. (ipaddress.py[3]) class IPLike(abc.ABC): """ Abstract base class for IP (address) like objects. """ @abc.abstractmethod def __ipaddr__(self): """ If it is a IPv4 return in the dotted form, If it is a IPv6 return in the long form. """ raise NotImplementedError @classmethod def __subclasshook__(cls, subclass): return hasattr(subclass, '__ipaddr__') It has a abstract method __ipaddr__ and a subclasshook. The __ipaddr__ method returns ip representation of object. The subclasshook checks for __ipaddr__ method. Then there is a ip-getter called get_ipaddr (ipaddress.py[4]) def get_ipaddr(address): """Return the representation of a ip address like object. If bytes or string or int passed in, get_ipaddr converts it to a IPv4/v6 Address and calls it self again for getting, IP address of IPv4/v6 object. If a IPLike passed it uses IPLike interface to get the value. """ if isinstance(address, (str, bytes, int)): return get_ipaddr(ip_address(address)) addr_type = type(address) try: ip_addr = address.__ipaddr__() except AttributeError: if hasattr(address, '__ipaddr__'): raise else: raise TypeError("expected str, bytes or ipaddress.IPLike object, " "not {}".format(addr_type.__name__)) if isinstance(ip_addr, (str, bytes)): return ip_addr else: raise TypeError("expected {}.__ipaddr__() to return str or bytes, " "not {}".format(addr_type.__name__, type(addr_type).__name__)) It takes one parameter, address. The address might be string, integer, bytes or ipaddress.IPLike If it is string or bytes or integer the get_ipaddr calls ipaddress.ip_address for getting IPv4/v6Address object. The purpose of that is converting bytes/int IP addresses to valid formed string IP addresses. And calls it self again with result of ipaddress.ip_address It tries to get address' __ipaddr__ method, if it can successfully perform it checks the return value of __ipaddr__ method. If return value is string or bytes it returns the ip address. If it is not it raises a TypeError. In the C-Side we can check via _PyObject_LookupSpecial. (socketmodule.c[5]) idna_converter(PyObject *obj, struct maybe_idna *data) { ... _Py_IDENTIFIER(__ipaddr__); ... func = _PyObject_LookupSpecial(obj, &PyId___ipaddr__); if (func != NULL) { obj = PyObject_CallFunctionObjArgs(func, NULL); Py_DECREF(func); } } Thanks for your attention, Damla [0] https://github.com/python/cpython/pull/9956#pullrequestreview-166756568 [1] https://www.python.org/dev/peps/pep-0519 [2] https://github.com/DamlaAltun/cpython/tree/iplike-object [3] https://github.com/DamlaAltun/cpython/blob/iplike-object/Lib/ipaddress.py#L413 [4] https://github.com/DamlaAltun/cpython/blob/iplike-object/Lib/ipaddress.py#L384 [5] https://github.com/DamlaAltun/cpython/blob/iplike-object/Modules/socketmodule.c#L1537 -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Sun Oct 21 08:16:12 2018 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sun, 21 Oct 2018 15:16:12 +0300 Subject: [Python-ideas] IPLike objects In-Reply-To: References: Message-ID: Should we support bytes? Why? For path name, the bytes are sometimes the only way to encode the file name, Posix operates with bytes internally. But IP addresses are always ASCII strings. I doubt if we shall support bytes for it, complicating the implementation. Support for `int` in `get_addr()` is event more questionable and error-prone. The main consumer of IPLIke API is socket module. Moving IPLike from ipaddress.py into socket.py makes sense for me. Calling `ip_address` from `get_ipaddr()` is another question. I doubt if `get_ipaddr()` is required at all. In my mind `socket.ipaddr(arg)` should be very similar to `os,fspath` and implemented in the same way: 1. Return argument as is if it is a string, 2. Otherwise try to return `type(arg).__ipaddr__(arg)` `socket.ipaddr` should be as fast as possible, don't process extra conversions and have a C Accelerated version used by default. Basically it is a mirror of `os.fspath()`, `socket.IPLike` is a mirror of `os.PathLike`. Your thoughts? On Sun, Oct 21, 2018 at 1:59 PM Damla Altun wrote: > Hi, > > Python's ipaddress module is really good for working with network / IP > related thing. > I would like to use ipaddress.IPv4/v6Address in creating network > interfaces. Like a socket but i can't, > the internal things and some standard libraries doesn't support them (even > socket). > > >>> server_address = (ipaddress.IPv4Address('127.0.0.1'), 10000) > >>> sock.bind(server_address) > Traceback (most recent call last): > File "", line 1, in > TypeError: str, bytes or bytearray expected, not IPv4Address > > > Proposal > ======== > > The best solution we've got[0] is implementing something like PathLike[1] > for IP Address objects. For PoC[2] > we called it IPLike. It is a protocol (some kind of interface) created > with Abstract Base Classes and > it forces class to implement __ipaddr__ method (like __fspath__ method). > The __ipaddr__ method should return > string or bytes formed notation of IP address. It is dotted notation in > IPv4, long formed notation in IPv6. > > The __ipaddr__ method is also an identifier for IPLike classes. IPLike > interface's subclasshook > checks the __ipaddr__ method of given class when issubclass called. > > >>> class SomethingRepresentIP: > ... def __ipaddr__(self) -> str: > ... return '127.0.0.1' > ... > >>> issubclass(SomethingRepresentIP, IPLike) > True > >>> isinstance(SomethingRepresentIP(), IPLike) > True > > For PoC[2] we implented __ipaddr__ method to ipaddress._BaseAddress class. > It is parent class of IPv4/v6Adress. > > >>> issubclass(IPv4Address, IPLike) > True > >>> IPv4Address('127.0.0.1').__ipaddr__() > '127.0.0.1' > > Like os.fspath we need a general-purpose ip-getter. So we implemented > ipaddress.get_ipaddr(address) function. > When you give it an valid IP like object it returns the value of > __ipaddr__ method. > > >>> get_ipaddr(IPv4Address('127.0.0.1')) > '127.0.0.1' > > When you give it string or bytes or integer object it calls > ipaddr.ip_address(address) and re-calls itself with > the value it got from ip_address. It helps get_ipaddr to convert > integer-formed / byte-formed IPs to string formed IPs > with a valid notation. > > >>> get_ipaddr('127.0.0.1') > '127.0.0.1' > >>> get_ipaddr(3232235521) > '192.168.0.1' > >>> get_ipaddr(b'\xC0\xA8\x00\x01') > '192.168.0.1' > > If you give any type else it raises TypeError. Also if you give it a IP > like object (an object that has __ipaddr__ method), > but the object's __ipaddr__ method returns something that is not > string/bytes it raises TypeError. > > For PoC[2] i added socket to support IP like objects. > > >>> sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > >>> server_address = (SomethingRepresentIP(), 10000) > >>> sock.bind(server_address) > >>> sock.listen(1) > >>> conn, client = sock.accept() > >>> conn > type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 10000), > raddr=('127.0.0.1', 49452)> > > I didn't implement a C API for ipaddress.get_ipaddr. The function is too > small for creating a new module, new header and the ipaddress library > currently has no > C-side module. If needed i can write a function inside the socketmodule.c . > > Proof-of-Concept (Implementation) > ================================= > > The implementation can be found on github (DamlaAltun/cpython[2]) We have > a Abstract Base Class for IP Address protocol called IPLike. > > (ipaddress.py[3]) > > class IPLike(abc.ABC): > """ > Abstract base class for IP (address) like objects. > """ > > @abc.abstractmethod > def __ipaddr__(self): > """ > If it is a IPv4 return in the dotted form, > If it is a IPv6 return in the long form. > """ > raise NotImplementedError > > @classmethod > def __subclasshook__(cls, subclass): > return hasattr(subclass, '__ipaddr__') > > It has a abstract method __ipaddr__ and a subclasshook. The __ipaddr__ > method returns ip representation of object. > The subclasshook checks for __ipaddr__ method. > > Then there is a ip-getter called get_ipaddr > > (ipaddress.py[4]) > > def get_ipaddr(address): > """Return the representation of a ip address like object. > > If bytes or string or int passed in, get_ipaddr converts it to > a IPv4/v6 Address and calls it self again for getting, > IP address of IPv4/v6 object. If a IPLike passed it uses IPLike > interface to get the value. > """ > > if isinstance(address, (str, bytes, int)): > return get_ipaddr(ip_address(address)) > > addr_type = type(address) > try: > ip_addr = address.__ipaddr__() > except AttributeError: > if hasattr(address, '__ipaddr__'): > raise > else: > raise TypeError("expected str, bytes or ipaddress.IPLike > object, " > "not {}".format(addr_type.__name__)) > > if isinstance(ip_addr, (str, bytes)): > return ip_addr > else: > raise TypeError("expected {}.__ipaddr__() to return str or > bytes, " > "not {}".format(addr_type.__name__, > type(addr_type).__name__)) > > > It takes one parameter, address. The address might be string, integer, > bytes or ipaddress.IPLike > If it is string or bytes or integer the get_ipaddr calls > ipaddress.ip_address for getting IPv4/v6Address > object. The purpose of that is converting bytes/int IP addresses to valid > formed string IP addresses. And calls > it self again with result of ipaddress.ip_address > > It tries to get address' __ipaddr__ method, if it can successfully perform > it checks the return value of __ipaddr__ method. > If return value is string or bytes it returns the ip address. If it is not > it raises a TypeError. > > In the C-Side we can check via _PyObject_LookupSpecial. > > (socketmodule.c[5]) > > idna_converter(PyObject *obj, struct maybe_idna *data) > { > ... > _Py_IDENTIFIER(__ipaddr__); > ... > func = _PyObject_LookupSpecial(obj, &PyId___ipaddr__); > if (func != NULL) { > obj = PyObject_CallFunctionObjArgs(func, NULL); > Py_DECREF(func); > } > } > > > Thanks for your attention, > > Damla > > [0] > https://github.com/python/cpython/pull/9956#pullrequestreview-166756568 > [1] https://www.python.org/dev/peps/pep-0519 > [2] https://github.com/DamlaAltun/cpython/tree/iplike-object > [3] > https://github.com/DamlaAltun/cpython/blob/iplike-object/Lib/ipaddress.py#L413 > [4] > https://github.com/DamlaAltun/cpython/blob/iplike-object/Lib/ipaddress.py#L384 > [5] > https://github.com/DamlaAltun/cpython/blob/iplike-object/Modules/socketmodule.c#L1537 > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: From sukurcf at gmail.com Sun Oct 21 09:00:44 2018 From: sukurcf at gmail.com (Siva Sukumar Reddy) Date: Sun, 21 Oct 2018 18:30:44 +0530 Subject: [Python-ideas] Python Enhancement Proposal for List methods Message-ID: Hey everyone, I am really new to Python contribution community want to propose below methods for List object. Forgive me if this is not the format to send an email. 1. *list.replace( item_to_be_replaced, new_item )*: which replaces all the occurrences of an element in the list instead of writing a new list comprehension in place. 2. *list.replace( item_to_be_replaced, new_item, number_of_occurrences )*: which replaces the occurrences of an element in the list till specific number of occurrences of that element. The number_of_occurrences can defaulted to 0 which will replace all the occurrences in place. 3. *list.removeall( item_to_be_removed )*: which removes all the occurrences of an element in a list in place. What do you think about these features? Are they PEP-able? Did anyone tried to implement these features before? Please let me know. Thank you, Sukumar -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sun Oct 21 10:18:28 2018 From: mertz at gnosis.cx (David Mertz) Date: Sun, 21 Oct 2018 10:18:28 -0400 Subject: [Python-ideas] Python Enhancement Proposal for List methods In-Reply-To: References: Message-ID: The list comprehensions are not very hard, and are more general. EXCEPT with the limited number of occurrences. We have this for str.replace(..., max=n), and it is useful fairly often. I'm +0.5 on .replace() with that capability. But -1 on .removeall() that adds nothing to an easy listcomp. On Sun, Oct 21, 2018, 9:01 AM Siva Sukumar Reddy Hey everyone, > > I am really new to Python contribution community want to propose below > methods for List object. Forgive me if this is not the format to send an > email. > > 1. *list.replace( item_to_be_replaced, new_item )*: which replaces all > the occurrences of an element in the list instead of writing a new list > comprehension in place. > 2. *list.replace( item_to_be_replaced, new_item, number_of_occurrences )*: > which replaces the occurrences of an element in the list till specific > number of occurrences of that element. The number_of_occurrences can > defaulted to 0 which will replace all the occurrences in place. > 3. *list.removeall( item_to_be_removed )*: which removes all the > occurrences of an element in a list in place. > > What do you think about these features? > Are they PEP-able? Did anyone tried to implement these features before? > Please let me know. > > Thank you, > Sukumar > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sun Oct 21 10:44:05 2018 From: mertz at gnosis.cx (David Mertz) Date: Sun, 21 Oct 2018 10:44:05 -0400 Subject: [Python-ideas] TypeHinting: From variable name to type In-Reply-To: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> Message-ID: On Fri, Oct 19, 2018 at 3:18 AM Thomas G?ttler wrote: > Now my idea: Per module and/or per file type hinting from variable name. > Maybe a magic docstring in the __init__.py file: > variable-name-mapping: > { > request: django.http.HttpRequest, > } > In general, I like this idea; but really only on a per-file basis. A short header at the top would be easy enough for an IDE or linter to scan. But imposing the conventions project-wide feels too broad. There might, of course, be cases where the same name is used for different purposes in the same file. But the tool can alert the developer of that... and in that one file, she could either remove the header of refactor the names used, as made sense for that particular code. This is certainly not something that requires language support. It can easily be purely a convention, as long as different IDEs, linters, type checkers, etc. agree on what the convention is. Maybe at some point in the future, if the convention becomes adopted, there might be some help in having a standard library module, or even minimal language recognition, of the convention. But let's work on adopting a convention first. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sun Oct 21 10:51:18 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 21 Oct 2018 17:51:18 +0300 Subject: [Python-ideas] Python Enhancement Proposal for List methods In-Reply-To: References: Message-ID: 21.10.18 16:00, Siva Sukumar Reddy ????: > I am really new to Python contribution community want to propose below > methods for List object. Forgive me if this is not the format to send an > email. > > 1. *list.replace( item_to_be_replaced, new_item )*: which replaces all > the occurrences of an element in the list instead of writing a new list > comprehension in place. > 2. *list.replace( item_to_be_replaced, new_item, number_of_occurrences > )*: which replaces the occurrences of an element in the list till > specific number of occurrences of that element. The > number_of_occurrences can defaulted to 0 which will replace all the > occurrences in place. > 3. *list.removeall( item_to_be_removed )*: which removes all the > occurrences of an element in a list in place. > > What do you think about these features? > Are they PEP-able? Did anyone tried to implement these features before? > Please let me know. See the previous discussion about this three years ago: https://mail.python.org/pipermail/python-ideas/2015-October/036770.html The conclusion is that this feature is too niche. It works only with lists, and supports only one test predicate (equality to a specified value). filter() and comprehensions are more general. From andrej.wins at gmail.com Sun Oct 21 12:06:21 2018 From: andrej.wins at gmail.com (Andreas Winschu) Date: Sun, 21 Oct 2018 18:06:21 +0200 Subject: [Python-ideas] Multi Statement Lambdas Message-ID: There was an extensive discussion about this in the past. https://www.artima.com/weblogs/viewpost.jsp?thread=147358 In the end Guido felt that his effort seems to him like building a Rube Goldberg machine. By looking at the amount of people still stumbling on this issue we definitely feel, that it is not the. Such a machine is a useful one. "Naming things" was from the beginning one the two hardest things in programming. Good naming is important for understandable code. The balance of extracted, named functions, as most of things in programming, is subjective though. The python style guide itself encourages people to use simple data transformations like `map`, `filter`, `reduce` for expressiveness. Those transformations can still be simple with 2 or 3 statements and will not require naming. Naming the transformation becomes harder, than reading the actual code. Coffeescript, which is another indent based language, handles multi statement lambdas without curly braces. *arr = []* *arr.map((x)-> * * y = x+1* * y * 2* *)* There is also an argument in this discussion, that just using curly braces for lambdas (which is easier to implement) would be pretty unpythonic. I really agree that python curly braces free block style is really helpful for readability in contrast to javascript for example. However using curly braces for lambdas in ruby is not hurting the eyes that much. Normally blocks in ruby use `do .. end` notation. Thus lambdas are clearly visible and also people try not to nest multiple levels of lamda transfromations. We strive to keep such a chain flat. *def transform arr* * .map { ... }* * .filter { ... } .reduce { ... }* In the end curly branes for lambdas is also a mathematical notation. Guido had a hard opinion on this, but i hope the python community does not. A powerful general purpose language should not limit itself to one statement in a closure. Lets add mutli-statement lambdas to python either with just curly braces or with an indent based machine. -------------- next part -------------- An HTML attachment was scrubbed... URL: From klahnakoski at mozilla.com Sun Oct 21 12:08:02 2018 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Sun, 21 Oct 2018 12:08:02 -0400 Subject: [Python-ideas] TypeHinting: From variable name to type In-Reply-To: References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> Message-ID: On 2018-10-21 10:44, David Mertz wrote: > On Fri, Oct 19, 2018 at 3:18 AM Thomas G?ttler > > > wrote: > > Now my idea: Per module and/or per file type hinting from variable > name. > Maybe a magic docstring in the __init__.py file: > variable-name-mapping: > ? { > ? ? request: django.http.HttpRequest, > } > > > In general, I like this idea; but really only on a per-file basis.? A > short header at the top would be easy enough for an IDE or linter to > scan.? But imposing the conventions project-wide feels too broad. > Maybe in a .editorconfig file?? This would allow type hinting at a number of directory levels. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Oct 21 12:14:50 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 22 Oct 2018 03:14:50 +1100 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: Message-ID: On Mon, Oct 22, 2018 at 3:07 AM Andreas Winschu wrote: > Guido had a hard opinion on this, but i hope the python community does not. A powerful general purpose language should not limit itself to one statement in a closure. > Lets add mutli-statement lambdas to python either with just curly braces or with an indent based machine. > Just to be clear here, a *closure* is fully capable of having multiple statements in it. You can use 'def' inside a function, and it will create a closure. Proposals like this stand or fall on the syntax to be used. Can you be completely specific about exactly what the syntax is that you're proposing, and give some examples? Don't forget that this is currently valid code: func(lambda x: {}) so you'll need to clearly distinguish your block delimiters from set/dict display. ChrisA From boxed at killingar.net Sun Oct 21 12:17:48 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Sun, 21 Oct 2018 18:17:48 +0200 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: Message-ID: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> > A powerful general purpose language should not limit itself to one statement in a closure. Nitpick on language: It doesn't. Lambdas are not the only way to do a closure. It's important to be precise when discussing these things. > Lets add mutli-statement lambdas to python either with just curly braces or with an indent based machine. Mostly it's just better with a function. Do you have some more compelling examples where you can't use a function smoothly? / Anders From andrej.wins at gmail.com Sun Oct 21 12:28:40 2018 From: andrej.wins at gmail.com (Andreas Winschu) Date: Sun, 21 Oct 2018 18:28:40 +0200 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: A def function has to be named. Wheres a lambda expression can be passed anonymously to any other function as an argument. *map(lambda x: x**2, array)* vs *def powers2(x)* * x**2* *map(powers2, array)* Coming up with a name here is not needed, as the operation is expressive enough. Am So., 21. Okt. 2018 um 18:17 Uhr schrieb Anders Hovm?ller < boxed at killingar.net>: > > > A powerful general purpose language should not limit itself to one > statement in a closure. > > Nitpick on language: It doesn't. Lambdas are not the only way to do a > closure. > > It's important to be precise when discussing these things. > > > Lets add mutli-statement lambdas to python either with just curly braces > or with an indent based machine. > > Mostly it's just better with a function. Do you have some more compelling > examples where you can't use a function smoothly? > > / Anders -- Gru? Andreas Winschu -------------- next part -------------- An HTML attachment was scrubbed... URL: From boxed at killingar.net Sun Oct 21 12:33:39 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Sun, 21 Oct 2018 18:33:39 +0200 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: > Wheres a lambda expression can be passed anonymously to any other function as an argument. > > map(lambda x: x**2, array) > > vs > > def powers2(x) > x**2 > map(powers2, array) > > Coming up with a name here is not needed, as the operation is expressive enough. Sure, but there is a well known convention for such non-names already: meta syntactical variables. In this case the name could be "foo" and all readers will interpret this as "no name". I admit to have wanted lambdas that are just as powerful as normal functions, but often when I want it the code is nicer when a normal function is used. / Anders -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Oct 21 12:38:17 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 22 Oct 2018 03:38:17 +1100 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: On Mon, Oct 22, 2018 at 3:29 AM Andreas Winschu wrote: > > A def function has to be named. > > Wheres a lambda expression can be passed anonymously to any other function as an argument. > > map(lambda x: x**2, array) > > vs > > def powers2(x) > x**2 > map(powers2, array) > > Coming up with a name here is not needed, as the operation is expressive enough. > Sure, but can you give examples where you can't use lambda, but still want it to have no name? Then show what your proposed syntax is, and how it would improve the code. Side point: I hardly ever use map with lambda functions in Python. I'll use map with an existing function (eg "map(int, list_of_strings)" to intify everything), but otherwise, I'll use a comprehension. Comprehensions are also restricted to expressions only, but your proposal won't help them. ChrisA From ron.reiter at gmail.com Sun Oct 21 12:36:49 2018 From: ron.reiter at gmail.com (Ron Reiter) Date: Sun, 21 Oct 2018 19:36:49 +0300 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: Multi-line lambdas may be nice for functional languages, but Python is very imperative in nature and prefers clean syntax over conciseness, which means it will make it REALLY hard to fit with Python. I personally find multi-line lambdas an unreadable abomination. As for some more concrete reasons on why this should not be a part of Python, see here for an example which shows why it would even be hard to come up with a plausible syntax for multi-line lambdas: https://stackoverflow.com/questions/1233448/no-multiline-lambda-in-python-why-not Guido's words: "But the complexity of any proposed solution for this puzzle is immense, to me: it requires the parser (or more precisely, the lexer) to be able to switch back and forth between indent-sensitive and indent-insensitive modes, keeping a stack of previous modes and indentation level. Technically that can all be solved (there's already a stack of indentation levels that could be generalized). But none of that takes away my gut feeling that it is all an elaborate Rube Goldberg contraption." If one would find a syntax which is Pythonic and clean then I am in favor of adding it, but I argue it simply does not exist. - Ron [image: Facebook] [image: Twitter] [image: LinkedIn] On Sun, Oct 21, 2018 at 7:34 PM Anders Hovm?ller wrote: > > Wheres a lambda expression can be passed anonymously to any other function > as an argument. > > *map(lambda x: x**2, array)* > > vs > > *def powers2(x)* > * x**2* > *map(powers2, array)* > > Coming up with a name here is not needed, as the operation is expressive > enough. > > > Sure, but there is a well known convention for such non-names already: > meta syntactical variables. In this case the name could be "foo" and all > readers will interpret this as "no name". > > I admit to have wanted lambdas that are just as powerful as normal > functions, but often when I want it the code is nicer when a normal > function is used. > > / Anders > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From initalize.damla at gmail.com Sun Oct 21 12:47:44 2018 From: initalize.damla at gmail.com (Damla Altun) Date: Sun, 21 Oct 2018 19:47:44 +0300 Subject: [Python-ideas] IPLike objects In-Reply-To: References: Message-ID: > I followed your instructions and appended them to the last commit of > DamlaAltun/cpython (branch: _iplike-socket)[0] . > >> > - Bytes and Integer support removed, only allowed string and IPLike. If > string passed in it returns without any func call / modification. [1] > - IPLike moved to socket [2] > - `get_ipaddr()` renamed as `_ipaddr` and moved to socket. [3] > - IPLike & ipaddr() tests moved to test_socket [4] > - Created a header (ipaddrmodule.h) file and imported into Python.h and > added header files list in makefile.pre.in [5] > - In socketmodule.c; > - Created `PySocket_IPAddr(PyObject *address)[6], its behavior exactly > same with socket._ipaddr's behavior. In Lib/socket.py it checks builtins > for ipaddr method if it can not find it it renames _ipaddr as ipaddr [7]. > - Created `socket_ipaddr_impl(PyObject *module, PyObject *address) > [8], it used for exporting c method to python [9]. It calls PySocket_IPAddr. > - Added IPLike objects support to socket's idna_converter [10] > > [0] https://github.com/DamlaAltun/cpython/tree/_iplike-socket > [1] > https://github.com/DamlaAltun/cpython/commit/d39a261edb91d6e423d319b1dd20f94998f61539#diff-1f037cd6b68ccd47326d99aeb7ae2c6aR763 > [2] > https://github.com/DamlaAltun/cpython/commit/d39a261edb91d6e423d319b1dd20f94998f61539#diff-1f037cd6b68ccd47326d99aeb7ae2c6aR789 > [3] > https://github.com/DamlaAltun/cpython/commit/d39a261edb91d6e423d319b1dd20f94998f61539#diff-1f037cd6b68ccd47326d99aeb7ae2c6aR756 > [4] > https://github.com/DamlaAltun/cpython/commit/d39a261edb91d6e423d319b1dd20f94998f61539#diff-30065f853d7944a19b3e17b55db67733R147 > [5] > https://github.com/DamlaAltun/cpython/commit/d39a261edb91d6e423d319b1dd20f94998f61539#diff-45e8b91057f0c5b60efcb5944125b585R970 > [6] > https://github.com/DamlaAltun/cpython/commit/d39a261edb91d6e423d319b1dd20f94998f61539#diff-a47fd74731aeb547ad780900bb8e6953R4698 > [7] > https://github.com/DamlaAltun/cpython/commit/d39a261edb91d6e423d319b1dd20f94998f61539#diff-1f037cd6b68ccd47326d99aeb7ae2c6aR785 > [8] > https://github.com/DamlaAltun/cpython/commit/d39a261edb91d6e423d319b1dd20f94998f61539#diff-a47fd74731aeb547ad780900bb8e6953R4723 > [9] > https://github.com/DamlaAltun/cpython/commit/d39a261edb91d6e423d319b1dd20f94998f61539#diff-a47fd74731aeb547ad780900bb8e6953R6749 > [10] > https://github.com/DamlaAltun/cpython/commit/d39a261edb91d6e423d319b1dd20f94998f61539#diff-a47fd74731aeb547ad780900bb8e6953R1550 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hemflit at gmail.com Sun Oct 21 13:58:05 2018 From: hemflit at gmail.com (=?UTF-8?Q?Vladimir_Filipovi=C4=87?=) Date: Sun, 21 Oct 2018 19:58:05 +0200 Subject: [Python-ideas] Add closing and iteration to threading.Queue Message-ID: Hi! I originally submitted this as a pull request. Raymond Hettinger suggested it should be given a shakeout in python-ideas first. https://github.com/python/cpython/pull/10018 https://bugs.python.org/issue35034 ------ Briefly: Add a close() method to Queue, which should simplify many common uses of the class and reduce the space for some easy-to-make errors. Also add an __iter__() method which in conjunction with close() would further simplify some common use patterns. ------ At eye-watering length: Apologies in advance for the length of this message. This isn't a PEP in disguise, it's a proposal for a very small, simple and I dare imagine uncontroversial feature. I'm new to contributing to Python and after the BPO/github submission I didn't manage to come up with a better way to present it than this. The issue Code using threading.Queue often needs to coordinate a "work is finished as far as I care" state between the producing and consuming side. Not "work" in the task_done() sense of completion of processing of queue items, "work" in the simpler sense of just passing data through the queue. For example, a producer can be driving the communication by enqueuing e.g. names of files that need to be processed, and once it's enqueued the last filename, it can be useful to inform the consumers that no further names will be coming, so after they've retrieved what's in-flight currently, they don't need to bother waiting for any more. Alternatively, a consumer can be driving the communication, and may need to let the producers know "I'm not interested in any more, so you can stop wasting resources on producing and enqueuing them". Also, a third, coordinating component may need to let both sides know that "Our collective work here is done. Start wrapping it up y'all, but don't drop any items that are still in-flight." In practice it's probably the exception, not the rule, when any piece of code interacting with a Queue _doesn't_ have to either inform another component that its interest in transferring the data has ended, or watch for such information. In the most common case of producer letting consumers know that it's done, this is usually implemented (over and over again) with sentinel objects, which is at best needlessly verbose and at worst error-prone. A recipe for multiple consumers making sure nobody misses the sentinel is not complicated, but neither is it obvious the first time one needs to do it. When a generic sentinel (None or similar) isn't adequate, some component needs to create the sentinel object and communicate it to the others, which complicates code, and especially complicates interfaces between components that are not being developed together (e.g. if one of them is part of a library and expects the library-user code to talk to it through a Queue). In the less common cases where the producers are the ones being notified, there isn't even a typical solution - everything needs to be cooked up from scratch using synchronization primitives. ------ A solution Adding a close() method to the Queue that simply prohibits all further put()'s (with other methods acting appropriately when the queue is closed) would simplify a lot of this in a clean and safe way - for the most obvious example, multi-consumer code would not have to juggle sentinel objects. Adding a further __iter__() method (that would block as necessary, and stop its iteration once the queue is closed and exhausted) would especially simplify many unsophisticated consumers. This is a current fairly ordinary pattern: # Producer: while some_condition: q.put(generate_item()) q.put(sentinel) # Consumer: while True: item = q.get() if item == sentinel: q.put(sentinel) break process(item) (This consumer could be simplified a little with an assignment expression or an iter(q.get, sentinel), but one of those is super new and the other seems little-known in spite of being nearly old enough to vote.) With the close() and __iter__(), this would become: # Producer: with closing(q): while some_condition: q.put(generate_item()) # Consumer: for item in q: process(item) Apart from it being shorter and less error-prone (e.g. if generate_item raises), the implied interface for initializing the two components is also simplified, because there's no sentinel to pass around. More complex consumers that couldn't take advantage of the __iter__() would still benefit from being able to explicitly and readably find out (via exception or querying) that the queue has been closed and exhausted, without juggling the sentinel. I honestly think this small addition would be an unqualified win. And it would not change anything for code that doesn't want to use it. ------ I've got a sample implementation ready for Queue and its children. (Link is at the start of this message. It includes documentation updates too, in case those clarify anything further at this stage.) If this is going in the right direction, I'm happy to do the same for SimpleQueue, but I haven't done it yet. I'm still getting my bearings around the C code base. ------ To immediately answer some of Raymond's initial comments at BPO: This is completely independent from Queue's existing task-tracking protocol. One is about controlling transport, and the other about tracking the completion of processing after transport. I hesitate to call them "orthogonal", but there is no functional overlap between them at all. I keep talking about "common cases" and such weaselly concepts above, and really it's just conjecture based on my own limited experience and informal conversations. BUT, I've also done a survey of the standard library itself. There are four packages that use threading.Queue: concurrent, idlelib, logging, multiprocessing. All except idlelib have at least one piece that would have benefited from a Queue.close() if it had been available when they were being written. I've now had a look at a handful of other languages too: - Java and C# have nothing like this. They basically start from a deque as a collection, and add synchronization features to it; C++ STL doesn't even go that far. None of them do the Python thing where a Queue is a dedicated communication tool that just uses a collection as part of its implementation. - Ruby (to my mild surprise) also does nothing like this. - Clojure does just about the same thing as this proposal, yay. - Go does approximately the same thing, just more finicky about practical usage. The producer can say `close(q)` and the consumers can say `for item := range q { process(item) }` which do exactly the same thing as the proposed Python equivalents, but it has some harsh limitations that are not applicable to Python. (Can't re-close a closed channel, can't query whether a channel is closed outside of retrieving an item). ------ To anticipate a couple more possible questions: - What would this proposal do about multiple producers/consumers needing to jointly decide _when_ to close the queue? Explicitly nothing. The queue's state is either closed or not, and it doesn't care who closed it. It needs to interact correctly with multiple consumers and multiple producers, but once any one piece of code closes it, the correct interaction is acting like a closed queue for everybody. When e.g. multiple producers need to arrange among themselves that the queue be closed only after all of them are done producing, then it's not the queue's job to coordinate _that_. They probably need a Semaphore or a more complex coordinator in charge of the closing. Yes, this too is a non-trivial-to-get-right situation, but trying to solve this one inside the Queue class seems like bloat. - Why not make the Queue a context manager while you're at it? Most closeable classes do it. I don't see that as a clear win in this case. Happy to add it if there's consensus in its favour, of course. I think `with closing(q):` is much more expressive than `with q:` while still brief. The Queue is more versatile in use than files, database cursors and many other resource-type classes, so the meaning of a `with q:` would not be as immediately suggestive as with them. It also often needs to be passed around between creation and initial use (the whole point is that more than one piece of code has access to it) so the common idiom `with Queue() as q:` would be a slightly less natural fit than with resource-like classes. - Should SimpleQueue do the same thing as Queue? Yes, I would propose so and volunteer to implement it. Some details would be adapted to SimpleQueue's existing promises (put() can't fail) but I think this feature is very much in the spirit of SimpleQueue. - Should multiprocessing.Queue do the same thing? I think so, though I'm not proposing it. It already has a close() method, whose meaning is very similar but not identical to (a subset of) the proposed threading.Queue.close's meaning (with resource-management effects not relevant to threading.Queue either way). I'd hope that this "not identical" part would not cause any problems in practice (lots of things aren't identical between those two Queues), but all hoping aside, maybe people more experienced than me can evaluate if that's really the case. I also have no clear idea if the feature would be as valuable, and if the implementation would be as easy and clean as they are with threading.Queue. - Should asyncio.Queue do the same thing? Probably? I'm too unfamiliar with asyncio for my opinion on this to be of value. So I'm not proposing it. ------ From python at mrabarnett.plus.com Sun Oct 21 14:42:14 2018 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 21 Oct 2018 19:42:14 +0100 Subject: [Python-ideas] Add closing and iteration to threading.Queue In-Reply-To: References: Message-ID: <2866bc8a-c9ec-bc24-ba37-38d2752d9d2d@mrabarnett.plus.com> On 2018-10-21 18:58, Vladimir Filipovi? wrote: > Hi! > > I originally submitted this as a pull request. Raymond Hettinger > suggested it should be given a shakeout in python-ideas first. > > https://github.com/python/cpython/pull/10018 > https://bugs.python.org/issue35034 > [snip] FTR, this has been discussed before: [Python-ideas] `__iter__` for queues? https://mail.python.org/pipermail/python-ideas/2010-January/006711.html From hemflit at gmail.com Sun Oct 21 15:19:21 2018 From: hemflit at gmail.com (=?UTF-8?Q?Vladimir_Filipovi=C4=87?=) Date: Sun, 21 Oct 2018 21:19:21 +0200 Subject: [Python-ideas] Add closing and iteration to threading.Queue In-Reply-To: <2866bc8a-c9ec-bc24-ba37-38d2752d9d2d@mrabarnett.plus.com> References: <2866bc8a-c9ec-bc24-ba37-38d2752d9d2d@mrabarnett.plus.com> Message-ID: On Sun, Oct 21, 2018 at 8:45 PM MRAB wrote: > FTR, this has been discussed before: > > [Python-ideas] `__iter__` for queues? > https://mail.python.org/pipermail/python-ideas/2010-January/006711.html Thank you! For the sake of clarity, I want to outline a few differences between that discussion and my proposal: 1. Much of the discussion there seemed to implicitly limit itself to consideration of FIFO queues. This proposal works cleanly for child classes too, including any (API-compliant) user-written children. 2. Throughout that discussion, iteration is the A feature, and closing is occasionally mentioned as a possible prerequisite. In this proposal, the A feature is closing, which enables sensible iteration (as a B feature) but is useful even if iteration isn't used. 3. There's naturally a lot of quick spitballing of various mutually-incompatible ideas there, whereas this is one rounded self-consistent proposal. Most of what I've come up with has already been anticipated there but it's all mixed up textually. 4. This proposal sidesteps a lot of the doubts and difficulties by just not using sentinels at all. Being closed is part of the queue's state that can be queried at any time, and will affect put() calls immediately, without waiting for a sentinel to float up to the front. (With recognition that your (MRAB's) message towards that thread's end already proposed the same approach.) From marko.ristin at gmail.com Sun Oct 21 15:19:48 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Sun, 21 Oct 2018 21:19:48 +0200 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: Hi, What about writing longer map&reduces in pyspark? When I worked with Spark and Scala, it was easy to script a longer chain of transformations in Scala. Does anybody know how that works in Python without multiline lambdas? Cheers Marko Le dim. 21 oct. 2018 ? 18:40, Ron Reiter a ?crit : > Multi-line lambdas may be nice for functional languages, but Python is > very imperative in nature and prefers clean syntax over conciseness, which > means it will make it REALLY hard to fit with Python. I personally find > multi-line lambdas an unreadable abomination. > > As for some more concrete reasons on why this should not be a part of > Python, see here for an example which shows why it would even be hard to > come up with a plausible syntax for multi-line lambdas: > > https://stackoverflow.com/questions/1233448/no-multiline-lambda-in-python-why-not > > Guido's words: > > "But the complexity of any proposed solution for this puzzle is immense, > to me: it requires the parser (or more precisely, the lexer) to be able to > switch back and forth between indent-sensitive and indent-insensitive > modes, keeping a stack of previous modes and indentation level. Technically > that can all be solved (there's already a stack of indentation levels that > could be generalized). But none of that takes away my gut feeling that it > is all an elaborate Rube Goldberg contraption." > > If one would find a syntax which is Pythonic and clean then I am in favor > of adding it, but I argue it simply does not exist. > > - Ron > > > [image: Facebook] [image: Twitter] > [image: LinkedIn] > > > > On Sun, Oct 21, 2018 at 7:34 PM Anders Hovm?ller > wrote: > >> >> Wheres a lambda expression can be passed anonymously to any other >> function as an argument. >> >> *map(lambda x: x**2, array)* >> >> vs >> >> *def powers2(x)* >> * x**2* >> *map(powers2, array)* >> >> Coming up with a name here is not needed, as the operation is expressive >> enough. >> >> >> Sure, but there is a well known convention for such non-names already: >> meta syntactical variables. In this case the name could be "foo" and all >> readers will interpret this as "no name". >> >> I admit to have wanted lambdas that are just as powerful as normal >> functions, but often when I want it the code is nicer when a normal >> function is used. >> >> / Anders >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.reiter at gmail.com Sun Oct 21 15:34:47 2018 From: ron.reiter at gmail.com (Ron Reiter) Date: Sun, 21 Oct 2018 22:34:47 +0300 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: Just write all your transformations at the top of the file and then write the mapreduce chain at the end. If you need to use closures that depend on each other then it may be a bit more ugly but still feasible to implement. - Ron [image: Facebook] [image: Twitter] [image: LinkedIn] On Sun, Oct 21, 2018 at 10:20 PM Marko Ristin-Kaufmann < marko.ristin at gmail.com> wrote: > Hi, > What about writing longer map&reduces in pyspark? When I worked with Spark > and Scala, it was easy to script a longer chain of transformations in > Scala. Does anybody know how that works in Python without multiline lambdas? > > Cheers Marko > > > Le dim. 21 oct. 2018 ? 18:40, Ron Reiter a ?crit : > >> Multi-line lambdas may be nice for functional languages, but Python is >> very imperative in nature and prefers clean syntax over conciseness, which >> means it will make it REALLY hard to fit with Python. I personally find >> multi-line lambdas an unreadable abomination. >> >> As for some more concrete reasons on why this should not be a part of >> Python, see here for an example which shows why it would even be hard to >> come up with a plausible syntax for multi-line lambdas: >> >> https://stackoverflow.com/questions/1233448/no-multiline-lambda-in-python-why-not >> >> Guido's words: >> >> "But the complexity of any proposed solution for this puzzle is immense, >> to me: it requires the parser (or more precisely, the lexer) to be able to >> switch back and forth between indent-sensitive and indent-insensitive >> modes, keeping a stack of previous modes and indentation level. Technically >> that can all be solved (there's already a stack of indentation levels that >> could be generalized). But none of that takes away my gut feeling that it >> is all an elaborate Rube Goldberg contraption." >> >> If one would find a syntax which is Pythonic and clean then I am in favor >> of adding it, but I argue it simply does not exist. >> >> - Ron >> >> >> [image: Facebook] [image: Twitter] >> [image: LinkedIn] >> >> >> >> On Sun, Oct 21, 2018 at 7:34 PM Anders Hovm?ller >> wrote: >> >>> >>> Wheres a lambda expression can be passed anonymously to any other >>> function as an argument. >>> >>> *map(lambda x: x**2, array)* >>> >>> vs >>> >>> *def powers2(x)* >>> * x**2* >>> *map(powers2, array)* >>> >>> Coming up with a name here is not needed, as the operation is expressive >>> enough. >>> >>> >>> Sure, but there is a well known convention for such non-names already: >>> meta syntactical variables. In this case the name could be "foo" and all >>> readers will interpret this as "no name". >>> >>> I admit to have wanted lambdas that are just as powerful as normal >>> functions, but often when I want it the code is nicer when a normal >>> function is used. >>> >>> / Anders >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kohnt at tobiaskohn.ch Sun Oct 21 15:40:00 2018 From: kohnt at tobiaskohn.ch (Tobias Kohn) Date: Sun, 21 Oct 2018 21:40:00 +0200 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: <20181021214000.Horde.KOuVMROobsatG25ZGmehLAj@webmail.tobiaskohn.ch> it might just be a detail, but Python does not even have single-statement lambdas.? The body of a lambda is an expression yielding a value, not a statement. Function languages (from which the idea of the Lambda in Python probably came from) do not have statements at all.? Something like the example with "y = x+1; y*2" given earlier is usually expressed as a let-expression, which is syntactic sugar for a lambda itself.? Hence, the example could actually be written in Python like so (I am not saying it is beautiful, but just that it is possible ^_^): ? arr.map( lambda x: (lambda y:y*2)(x+1) ) Or, if you prefer (this seems to me to be syntactically very close to the original): ? arr.map( lambda x: (lambda y=x+1: y*2)() ) Moreover, in Python 3.8, we will have assignments in expressions, and (even though I obviously can't test this) I wonder if you could then write the same thing as: ? arr.map( lambda x: (y := x+1, y*2)[1] ) I guess, the original request is therefore not really about having multi-statement lambdas, but more about extending lambdas to anonymous functions with the possibility to have full statements inside the body. Finally, I absolutely agree with that good naming is paramount for understandable code.? But IMHO, good naming means that I write a small /named/ function (describing its intent) if its body contains more than just a simple expression, anyway. Cheers, Tobias Quoting Anders Hovm?ller : >> A powerful general purpose language should not limit itself to one >> statement in a closure. > > Nitpick on language: It doesn't. Lambdas are not the only way to do > a closure. > > It's important to be precise when discussing these things. > >> Lets add mutli-statement lambdas to python either with just curly >> braces or with an indent based machine. > > Mostly it's just better with a function. Do you have some more > compelling examples where you can't use a function smoothly? > > / Anders > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideasCode of > Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Oct 21 16:34:12 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 21 Oct 2018 13:34:12 -0700 Subject: [Python-ideas] Add closing and iteration to threading.Queue In-Reply-To: References: Message-ID: Hi Vladimir, It's great to see people revisiting these old stdlib tools. Closure tracking is definitely a big point of awkwardness for Queues. In Trio we started with a straight copy of threading.Queue, and this turned out to be a major friction point for users. We just deprecated our version of Queue and replaced it with a new design. Our new thing is probably more radical than you want to get in the stdlib (we ended up splitting the object into two pieces, a sender object and a receiver object), but you might find the discussions interesting: Manual: https://trio.readthedocs.io/en/latest/reference-core.html#using-channels-to-pass-values-between-tasks A more minimal proposal to add closure tracking to trio.Queue: https://github.com/python-trio/trio/pull/573 Follow-up issue with design questions we're still thinking about (also links to earlier design discussions): https://github.com/python-trio/trio/issues/719 We only started shipping this last week, so we're still getting experience with it. -n On Sun, Oct 21, 2018, 10:59 Vladimir Filipovi? wrote: > Hi! > > I originally submitted this as a pull request. Raymond Hettinger > suggested it should be given a shakeout in python-ideas first. > > https://github.com/python/cpython/pull/10018 > https://bugs.python.org/issue35034 > > ------ > > Briefly: > > Add a close() method to Queue, which should simplify many common uses > of the class and reduce the space for some easy-to-make errors. > > Also add an __iter__() method which in conjunction with close() would > further simplify some common use patterns. > > ------ > > At eye-watering length: > > Apologies in advance for the length of this message. This isn't a PEP > in disguise, it's a proposal for a very small, simple and I dare > imagine uncontroversial feature. I'm new to contributing to Python and > after the BPO/github submission I didn't manage to come up with a > better way to present it than this. > > The issue > > Code using threading.Queue often needs to coordinate a "work is > finished as far as I care" state between the producing and consuming > side. Not "work" in the task_done() sense of completion of processing > of queue items, "work" in the simpler sense of just passing data > through the queue. > > For example, a producer can be driving the communication by enqueuing > e.g. names of files that need to be processed, and once it's enqueued > the last filename, it can be useful to inform the consumers that no > further names will be coming, so after they've retrieved what's > in-flight currently, they don't need to bother waiting for any more. > Alternatively, a consumer can be driving the communication, and may > need to let the producers know "I'm not interested in any more, so you > can stop wasting resources on producing and enqueuing them". > Also, a third, coordinating component may need to let both sides know > that "Our collective work here is done. Start wrapping it up y'all, > but don't drop any items that are still in-flight." > > In practice it's probably the exception, not the rule, when any piece > of code interacting with a Queue _doesn't_ have to either inform > another component that its interest in transferring the data has > ended, or watch for such information. > > In the most common case of producer letting consumers know that it's > done, this is usually implemented (over and over again) with sentinel > objects, which is at best needlessly verbose and at worst error-prone. > A recipe for multiple consumers making sure nobody misses the sentinel > is not complicated, but neither is it obvious the first time one needs > to do it. > When a generic sentinel (None or similar) isn't adequate, some > component needs to create the sentinel object and communicate it to > the others, which complicates code, and especially complicates > interfaces between components that are not being developed together > (e.g. if one of them is part of a library and expects the library-user > code to talk to it through a Queue). > > In the less common cases where the producers are the ones being > notified, there isn't even a typical solution - everything needs to be > cooked up from scratch using synchronization primitives. > > ------ > > A solution > > Adding a close() method to the Queue that simply prohibits all further > put()'s (with other methods acting appropriately when the queue is > closed) would simplify a lot of this in a clean and safe way - for the > most obvious example, multi-consumer code would not have to juggle > sentinel objects. > > Adding a further __iter__() method (that would block as necessary, and > stop its iteration once the queue is closed and exhausted) would > especially simplify many unsophisticated consumers. > > This is a current fairly ordinary pattern: > > # Producer: > while some_condition: > q.put(generate_item()) > q.put(sentinel) > > # Consumer: > while True: > item = q.get() > if item == sentinel: > q.put(sentinel) > break > process(item) > > (This consumer could be simplified a little with an assignment > expression or an iter(q.get, sentinel), but one of those is super new > and the other seems little-known in spite of being nearly old enough > to vote.) > > With the close() and __iter__(), this would become: > > # Producer: > with closing(q): > while some_condition: > q.put(generate_item()) > > # Consumer: > for item in q: > process(item) > > Apart from it being shorter and less error-prone (e.g. if > generate_item raises), the implied interface for initializing the two > components is also simplified, because there's no sentinel to pass > around. > > More complex consumers that couldn't take advantage of the __iter__() > would still benefit from being able to explicitly and readably find > out (via exception or querying) that the queue has been closed and > exhausted, without juggling the sentinel. > > I honestly think this small addition would be an unqualified win. And > it would not change anything for code that doesn't want to use it. > > ------ > > I've got a sample implementation ready for Queue and its children. > (Link is at the start of this message. It includes documentation > updates too, in case those clarify anything further at this stage.) > > If this is going in the right direction, I'm happy to do the same for > SimpleQueue, but I haven't done it yet. I'm still getting my bearings > around the C code base. > > ------ > > To immediately answer some of Raymond's initial comments at BPO: > > This is completely independent from Queue's existing task-tracking > protocol. One is about controlling transport, and the other about > tracking the completion of processing after transport. I hesitate to > call them "orthogonal", but there is no functional overlap between > them at all. > > I keep talking about "common cases" and such weaselly concepts above, > and really it's just conjecture based on my own limited experience and > informal conversations. > BUT, I've also done a survey of the standard library itself. There are > four packages that use threading.Queue: concurrent, idlelib, logging, > multiprocessing. All except idlelib have at least one piece that would > have benefited from a Queue.close() if it had been available when they > were being written. > > I've now had a look at a handful of other languages too: > - Java and C# have nothing like this. They basically start from a > deque as a collection, and add synchronization features to it; C++ STL > doesn't even go that far. None of them do the Python thing where a > Queue is a dedicated communication tool that just uses a collection as > part of its implementation. > - Ruby (to my mild surprise) also does nothing like this. > - Clojure does just about the same thing as this proposal, yay. > - Go does approximately the same thing, just more finicky about > practical usage. The producer can say `close(q)` and the consumers can > say `for item := range q { process(item) }` which do exactly the same > thing as the proposed Python equivalents, but it has some harsh > limitations that are not applicable to Python. (Can't re-close a > closed channel, can't query whether a channel is closed outside of > retrieving an item). > > ------ > > To anticipate a couple more possible questions: > > - What would this proposal do about multiple producers/consumers > needing to jointly decide _when_ to close the queue? > > Explicitly nothing. > > The queue's state is either closed or not, and it doesn't care who > closed it. It needs to interact correctly with multiple consumers and > multiple producers, but once any one piece of code closes it, the > correct interaction is acting like a closed queue for everybody. > > When e.g. multiple producers need to arrange among themselves that the > queue be closed only after all of them are done producing, then it's > not the queue's job to coordinate _that_. They probably need a > Semaphore or a more complex coordinator in charge of the closing. Yes, > this too is a non-trivial-to-get-right situation, but trying to solve > this one inside the Queue class seems like bloat. > > - Why not make the Queue a context manager while you're at it? Most > closeable classes do it. > > I don't see that as a clear win in this case. Happy to add it if > there's consensus in its favour, of course. > > I think `with closing(q):` is much more expressive than `with q:` > while still brief. The Queue is more versatile in use than files, > database cursors and many other resource-type classes, so the meaning > of a `with q:` would not be as immediately suggestive as with them. It > also often needs to be passed around between creation and initial use > (the whole point is that more than one piece of code has access to it) > so the common idiom `with Queue() as q:` would be a slightly less > natural fit than with resource-like classes. > > - Should SimpleQueue do the same thing as Queue? > > Yes, I would propose so and volunteer to implement it. > Some details would be adapted to SimpleQueue's existing promises > (put() can't fail) but I think this feature is very much in the spirit > of SimpleQueue. > > - Should multiprocessing.Queue do the same thing? > > I think so, though I'm not proposing it. > > It already has a close() method, whose meaning is very similar but not > identical to (a subset of) the proposed threading.Queue.close's > meaning (with resource-management effects not relevant to > threading.Queue either way). I'd hope that this "not identical" part > would not cause any problems in practice (lots of things aren't > identical between those two Queues), but all hoping aside, maybe > people more experienced than me can evaluate if that's really the > case. > > I also have no clear idea if the feature would be as valuable, and if > the implementation would be as easy and clean as they are with > threading.Queue. > > - Should asyncio.Queue do the same thing? > > Probably? I'm too unfamiliar with asyncio for my opinion on this to be > of value. So I'm not proposing it. > > ------ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Oct 21 16:58:00 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 21 Oct 2018 16:58:00 -0400 Subject: [Python-ideas] Python Enhancement Proposal for List methods In-Reply-To: References: Message-ID: On 10/21/2018 9:00 AM, Siva Sukumar Reddy wrote: > I am really new to Python contribution community want to propose below > methods for List object. For most classes, there is an infinite number of possible functions that could be made into methods. The Python philosophy is to include as methods the basic functions needed to write other functions. So number classes include the basic operations of +, -, *, /, and ** as methods. The math and cmath modules contain number functions that use these basic operations. The list class also has a fairly small set of methods. Nearly all are either original or were added before list.pop, about 20 years ago. list.clear was added more recently, in analogy with dict.clear, and there was some opposition because there are other ways to clear a list. > 1. *list.replace( item_to_be_replaced, new_item )*: which replaces all > the occurrences of an element in the list instead of writing a new list > comprehension in place. > 2. *list.replace( item_to_be_replaced, new_item, number_of_occurrences > )*: which replaces the occurrences of an element in the list till > specific number of occurrences of that element. The > number_of_occurrences can defaulted to 0 which will replace all the > occurrences in place. I presume these are somewhat in analogy with str.replace(old, new, count=-1), except for being limited to one sequence member and being done in place. Note that only 1 method is needed and the default count should be -1, not 0. However, the need for a built-in method is much, much less. Nearly all cases that need to be done inline code can be done like this. for i, item in enumerate(mylist): if item == old: mylist[i] = new As Serhiy noted from previous discussion, other conditions are possible, such as 'item in {old1, old2, old3}' or, given a list of numbers, if item > max: mylist[i] = max > 3. *list.removeall( item_to_be_removed )*: which removes all the > occurrences of an element in a list in place. str.replace(old, '') does this (not in place) for strings. There are also str.translate and re.sub and re.subn. An indefinite number of in-place removes is a generally a bad idea unless done carefully. alist.remove is inherently O(n). alist.removeall implemented naively would be O(n*n)/ alist = [x for x in alist if x == old] or alist = list(x for x in alist if x == old) or alist = list(filter(lambda x: x == old)) is O(n). Again, as Serhiy noted, 'x == old' is only one possible condition. The last example illustrates another point. In Python 3, one can usually and often should avoid turning streams of objects into a concrete list until one needs one of the particular list operations. I noted above that we have only added 1 list method (that I know of) since .pop. On the other hand, we added generators and then the itertools module with multiple basic functions and multiple recipes based on those functions. Since lists are no longer the basic representation of sequences of items, adding list methods is somewhat obsolete. If possible, it is best to filter out multiple items when creating the list instead of later. Or to put it another way, it is best to avoid turning a stream into a list for as long as possible. If space and aliasing are not issues, replacing a list is easier and possibly faster that editing it in place. Note that editors for adding, replacing, and deleting items tend to use tree structures rather than a single linear sequence. An alternative to removing items is to replace them with a null value, such as None, '', 0, 1 (for multiplication), 'pass' (for code execution), or sentinal=object(). The choice depends on what is to be done later. In fact, temporary replacement is the key to efficient multiple removals. -- Terry Jan Reedy From hemflit at gmail.com Sun Oct 21 17:04:02 2018 From: hemflit at gmail.com (=?UTF-8?Q?Vladimir_Filipovi=C4=87?=) Date: Sun, 21 Oct 2018 23:04:02 +0200 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: On Sun, Oct 21, 2018 at 6:39 PM Chris Angelico wrote: > Sure, but can you give examples where you can't use lambda, but still > want it to have no name? I can try kinda answering this: It's less with everyday built-ins and the standard library (though see first example below), and more with libraries and frameworks that want callbacks (or callables in general) passed in. Sometimes I need that callable to be something very simple, but it technically can't be a lambda because it has a statement in it. Defining it as a function forces me to give it not only a name, but a place too. Defining it just before the point of use is a natural location in one sense, but it makes the visual flow of the code worse, especially if I need to make several such steps (with different almost-lambdas) in a row. >From the standard library, I can actually think of Thread: t1 = Thread(target=(lambda: counter += 1)) t2 = Thread(target=(lambda: sum += current_quux())) Because they include statements, it needs to be: def inc_counter(): counter += 1 t1 = Thread(target=inc_counter) def update_sum(): sum += current_quux() t2 = Thread(target=update_sum) >From one popular library: ws = websocket.WebSocketApp( url, on_open = (lambda ws: ws.send('subscribe'); conn_counter += 1), on_message = (lambda ws, msg: print(msg); msg_counter += 1)) Because they include statements, I again need to create those functions separately. Naming isn't so much a problem here, but it would turn kludgey if I needed to make several such calls in the same scope. In any case, that's a common little frustration I often put up with. I think most other languages around Python's power level don't make that a problem. To expand that argument a bit, lambdas have access to names in their immediate context that can be more of a pain with less-local functions. For a toy example: [(lambda a: print(x + a)) for x in l] When print was a statement, defining this lambda as a named function would have had to use more arguments. The richer the context, the more noise it would have added. BTW, don't treat the above `(lambda: stmt)` syntax as a _proposal_ per se, BUT: No matter what we allow inside a lambda, from the outside it's an expression. How do we make a Python expression span multiple lines? In simplified terms, we put it in parentheses. How do we separate an expression whose boundaries would be ambiguous in a given context (e.g. a tuple literal inside a parameter list)? We put it inside a pair of parentheses. Hmm. I mean, these already work: (lambda: 2 + 3) lambda: 2 + ( 3) They're multi-LINE lambdas, just not multi-statement (or even uni-statement). If Python _was_ to somehow allow statements inside lambdas at all at some point in the future (which I don't expect), I don't see that that would then need any additional special syntax for multi-line-ness or for separating them from the surrounding context. From solipsis at pitrou.net Sun Oct 21 17:30:29 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 21 Oct 2018 23:30:29 +0200 Subject: [Python-ideas] Add closing and iteration to threading.Queue References: Message-ID: <20181021233029.6d897061@fsol> On Sun, 21 Oct 2018 19:58:05 +0200 Vladimir Filipovi? wrote: > > To anticipate a couple more possible questions: > > - What would this proposal do about multiple producers/consumers > needing to jointly decide _when_ to close the queue? > > Explicitly nothing. > > The queue's state is either closed or not, and it doesn't care who > closed it. It needs to interact correctly with multiple consumers and > multiple producers, but once any one piece of code closes it, the > correct interaction is acting like a closed queue for everybody. Ah. This is the one statement that makes me favorable to this idea. When there is a single consumer, it's easy enough to send a sentinel. But when there are multiple consumers, suddenly you must send exactly the right number of sentinels (which means you also have to careful keep track of their number, which isn't always easy). There's some delicate code doing exactly that in concurrent.futures. > - Should multiprocessing.Queue do the same thing? > > I think so, though I'm not proposing it. > > It already has a close() method, whose meaning is very similar but not > identical to (a subset of) the proposed threading.Queue.close's > meaning (with resource-management effects not relevant to > threading.Queue either way). Not really similar, unfortunately. mp.Queue.close() isn't a logical thing, but releases the queue's internal resources. It doesn't signal consumers that the producers has finished with the queue. Perhaps if you renamed close() to something else ("put_eof" or "put_end" perhaps?) that would allow porting it to mp.Queue? Regards Antoine. From tjreedy at udel.edu Sun Oct 21 17:58:45 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 21 Oct 2018 17:58:45 -0400 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: Message-ID: On 10/21/2018 12:06 PM, Andreas Winschu wrote: > > There was an extensive discussion about this in the past. > > https://www.artima.com/weblogs/viewpost.jsp?thread=147358 And perhaps 20 other threads, now including this one. > In the end Guido felt that his effort seems to him like building a Rube > Goldberg machine. > > By looking at the amount of people still stumbling on this issue What I have seen for 20 years on python-list and perhaps 5 on Stackoverflow is beginners repeatedly stumbling on properly writing and using lambda expressions as they are. > we?definitely feel, that it is not the. ?? The royal 'we'?? > "Naming things" was from the beginning one the two hardest things in > programming. I completely disagree. I would put naming maybe 20th. Perhaps this explains why you would want anonymous multiline functions while I think they are a foolish idea. > Guido had a hard opinion on this, but i hope the python community does > not. Hah. If I were the new BDFL, I would be tempted to ban further discussion. However, your revelation about how you have a difficulty with naming that I do not helps explain our different viewpoints. -- Terry Jan Reedy From steve at pearwood.info Sun Oct 21 18:46:22 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 22 Oct 2018 09:46:22 +1100 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: <20181021224622.GW3817@ando.pearwood.info> On Sun, Oct 21, 2018 at 11:04:02PM +0200, Vladimir Filipovi? wrote: > Sometimes I need that callable to be something very simple, but it > technically can't be a lambda because it has a statement in it. > Defining it as a function forces me to give it not only a name, but a > place too. Defining it just before the point of use is a natural > location in one sense, but it makes the visual flow of the code worse, > especially if I need to make several such steps (with different > almost-lambdas) in a row. > > > From the standard library, I can actually think of Thread: > > t1 = Thread(target=(lambda: counter += 1)) > t2 = Thread(target=(lambda: sum += current_quux())) > > Because they include statements, it needs to be: > > def inc_counter(): > counter += 1 > t1 = Thread(target=inc_counter) I don't think that's a real working example. py> counter = 0 py> def f(): ... counter += 1 ... py> f() Traceback (most recent call last): File "", line 1, in File "", line 2, in f UnboundLocalError: local variable 'counter' referenced before assignment You need to declare counter and sum as global variables. > def update_sum(): > sum += current_quux() > t2 = Thread(target=update_sum) > > > From one popular library: > > ws = websocket.WebSocketApp( > url, > on_open = (lambda ws: ws.send('subscribe'); conn_counter += 1), > on_message = (lambda ws, msg: print(msg); msg_counter += 1)) > > Because they include statements, I again need to create those > functions separately. Naming isn't so much a problem here, but it > would turn kludgey if I needed to make several such calls in the same > scope. Its not the naming that would make it turn kludgy, but the use of global variables. If you had three websockets, would you want them all to share the same counter? > In any case, that's a common little frustration I often put up with. I > think most other languages around Python's power level don't make that > a problem. > > > To expand that argument a bit, lambdas have access to names in their > immediate context that can be more of a pain with less-local > functions. For a toy example: > > [(lambda a: print(x + a)) for x in l] You've fallen straight into the classic eager versus late binding of closures Gotcha. py> l = "spam eggs cheese aardvark".split() py> funcs = [(lambda a: print(x + a)) for x in l] py> for f in funcs: ... f("") ... aardvark aardvark aardvark aardvark If your intention was to demonstrate that multi-statement lambda would be a bug-magnet, you have done an excellent job :-) -- Steve From steve at pearwood.info Sun Oct 21 18:52:31 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 22 Oct 2018 09:52:31 +1100 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: <20181021225230.GX3817@ando.pearwood.info> On Sun, Oct 21, 2018 at 06:28:40PM +0200, Andreas Winschu wrote: > A def function has to be named. That's a feature, not a lack. As a general rule of thumb, if the body of a function is complex enough to require multiple statements, it is probably complex enough to require documentation, testing and a nice name to display in tracebacks. I wish that somebody would do an objective scientific study on the bug density of code using named functions versus anonymous code blocks. My intuition (which is, of course, completely subjective) tells me that code written in languages that allow and encourage anonymous code blocks will have more bugs, simply because coders will be less likely to name and test their functions. -- Steve From tjreedy at udel.edu Sun Oct 21 19:16:02 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 21 Oct 2018 19:16:02 -0400 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: On 10/21/2018 12:28 PM, Andreas Winschu wrote > A def function has to be named. In general, this is a good thing. It often improves tracebacks. Perhaps more importantly, name facilitate testing and mocking. > Wheres a lambda expression can be passed anonymously to any other > function as an argument. Except for relatively trivial expressions, this is a bad thing. All functions created from lambda expressions get the same pseudo-name ''. This can make tracebacks worse. Perhaps more importantly, proper testing may become harder. > /map(lambda x: x**2, array)/ It is obvious what this trivial expression does, but without a name or comment or context indicating intent, how do I know that it is correct? Trivial examples evade the issues that arise even with multiline expressions, let alone multiple statements. >>> for i in map(lambda x: x ** 2, 'abc'): print(i) Traceback (most recent call last): File "", line 2, in 2, 'abc'): File "", line 2, in 2, 'abc'): TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int' > vs > /def powers2(x)/ > /? ?x**2/ > /map(powers2, array)/ >>> def pow2(x): return (x ** 2) >>> for i in map(pow2, 'abc'): print(i) Traceback (most recent call last): File "", line 1, in for i in map(pow2, 'abc'): print(i) File "", line 3, in pow2 2) TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int' Given that there might be a hundred functions named '', I think the specific name is a bit helpful. > Coming up with a name here is not needed, as the operation is expressive > enough. It is clear as to the meaning. And testing that pow2(3) equals 9 is pretty useless. But you are proposing something else: adding anonymous multiple statement functions, where a name *and testing* is usually needed. Justification for what we have today is not justification for a major change. tkinter uses callbacks with widget commands and scheduling. idlelib has 77 matches for the re '[( ,]command='. 47 are followed by 'self.somemethod'; 26 by other function names, and 4 by lambda expressions. All four of the latter are trivial calls of a named method: self.somemethod(value). So when all named non-test functions are tested, all non-test callbacks will be tested. In spite of newbie problems with lambda, I am happy to not have to define trivial test functions like this. def undoNone(): return d.undo_event(None) But doubt that I would rather write any of the independent production-code functions and methods as anonymous lambda constructs. -- Terry Jan Reedy From hemflit at gmail.com Sun Oct 21 19:16:38 2018 From: hemflit at gmail.com (=?UTF-8?Q?Vladimir_Filipovi=C4=87?=) Date: Mon, 22 Oct 2018 01:16:38 +0200 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: <20181021224622.GW3817@ando.pearwood.info> References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> <20181021224622.GW3817@ando.pearwood.info> Message-ID: On Mon, Oct 22, 2018 at 12:52 AM Steven D'Aprano wrote: > > def inc_counter(): > > counter += 1 > > I don't think that's a real working example. > ... > You need to declare counter and sum as global variables. Good catch! None of the examples were real, in the sense of being copied directly from code in actual use. They were synthesized to illustrate a point while being as brief as possible. I'm not even really advocating for multi-statement (or statement) lambdas BTW, I was just answering the question of when one might even want an anonymous function that can't be expressed using the current lambda syntax. > Its not the naming that would make it turn kludgy, but the use of global > variables. If you had three websockets, would you want them all to share > the same counter? No, the naming would turn it kludgey precisely because I'd want different functions. If there's only one "on_open", it suggests a natural name for itself. If there are multiple, they can either be called "on_open1", "on_open2"... or I can keep reusing the same name in the same scope but that would make the code less clear, not more. > You've fallen straight into the classic eager versus late binding of > closures Gotcha. I did! /facepalm Thanks. I constructed a very poor example, and it's beginning to look like the particular point I was trying to illustrate with that doesn't actually stand. > If your intention was to demonstrate that multi-statement lambda would > be a bug-magnet, you have done an excellent job :-) I don't know why you'd say that. That was a zero-statement lambda :) From rosuav at gmail.com Sun Oct 21 19:20:49 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 22 Oct 2018 10:20:49 +1100 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: On Mon, Oct 22, 2018 at 8:05 AM Vladimir Filipovi? wrote: > From one popular library: > > ws = websocket.WebSocketApp( > url, > on_open = (lambda ws: ws.send('subscribe'); conn_counter += 1), > on_message = (lambda ws, msg: print(msg); msg_counter += 1)) > > Because they include statements, I again need to create those > functions separately. Naming isn't so much a problem here, but it > would turn kludgey if I needed to make several such calls in the same > scope. Cool! We have an example. Next step: Offer a variety of alternate syntaxes that *do* currently work, and then the proposed multi-statement lambda, and show how the current syntaxes are all ugly. One syntax that currently can be used to great effect is decorators. I don't know the specific websocket library that you're using, but it's generally not difficult to design something so that a function call can do the assignment. It'd end up looking something like: ws = websocket.WebSocketApp(url) @ws.event def on_open(sock): sock.send('subscribe') nonlocal conn_counter # or global, not sure conn_counter += 1 @ws.event def on_message(sock, msg): print(msg) nonlocal msg_counter msg_counter += 1 Yes, it's a lot more vertical than your version. However, adding in the nonlocal declarations definitely makes it look a LOT less lambda-friendly. This also takes a bit of extra support. But it can be third-party support if you need it to. Worst case, you can make a little helper: events = {} def event(func): events[func.__name__] = func return func @event def on_open... @event def on_message... ws = websocket.WebSocketApp(url, **events) which will work with the same API. Great for larger and more complex setups. ChrisA From rosuav at gmail.com Sun Oct 21 19:25:19 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 22 Oct 2018 10:25:19 +1100 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: On Mon, Oct 22, 2018 at 10:16 AM Terry Reedy wrote: > > On 10/21/2018 12:28 PM, Andreas Winschu wrote > > > A def function has to be named. > > In general, this is a good thing. It often improves tracebacks. > Perhaps more importantly, name facilitate testing and mocking. > > > Wheres a lambda expression can be passed anonymously to any other > > function as an argument. > > Except for relatively trivial expressions, this is a bad thing. All > functions created from lambda expressions get the same pseudo-name > ''. This can make tracebacks worse. Perhaps more importantly, > proper testing may become harder. The same considerations bite comprehensions, too, but we don't discourage their use. So I don't think this should be a killer - not on its own, anyhow. I do not currently support any proposed syntax for multi-statement lambda functions, mainly because they've all been ugly. But maybe there'll be one, somewhere, some day, that makes sense. ChrisA From rosuav at gmail.com Sun Oct 21 19:31:42 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 22 Oct 2018 10:31:42 +1100 Subject: [Python-ideas] Python Enhancement Proposal for List methods In-Reply-To: References: Message-ID: On Mon, Oct 22, 2018 at 7:58 AM Terry Reedy wrote: > > An indefinite number of in-place removes is a generally a bad idea > unless done carefully. > alist.remove is inherently O(n). > alist.removeall implemented naively would be O(n*n)/ > alist = [x for x in alist if x == old] or > alist = list(x for x in alist if x == old) or > alist = list(filter(lambda x: x == old)) is O(n). > That's actually a good reason to make it part of the language or stdlib - it's very easy to get it wrong. All of the above are not in-place. If I were to make a stdlib function to do removeall, it would be: seq[:] = [x for x in seq if x != removeme] (And that one probably missed some subtlety somewhere too.) So it'd be less buggy for people to reach for a standard solution than to try to craft their own. That said, though, I think this probably belongs as a recipe, not as a stdlib function or method. ChrisA From python at mrabarnett.plus.com Sun Oct 21 19:48:14 2018 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 22 Oct 2018 00:48:14 +0100 Subject: [Python-ideas] Add closing and iteration to threading.Queue In-Reply-To: <20181021233029.6d897061@fsol> References: <20181021233029.6d897061@fsol> Message-ID: On 2018-10-21 22:30, Antoine Pitrou wrote: > On Sun, 21 Oct 2018 19:58:05 +0200 > Vladimir Filipovi? > wrote: >> >> To anticipate a couple more possible questions: >> >> - What would this proposal do about multiple producers/consumers >> needing to jointly decide _when_ to close the queue? >> >> Explicitly nothing. >> >> The queue's state is either closed or not, and it doesn't care who >> closed it. It needs to interact correctly with multiple consumers and >> multiple producers, but once any one piece of code closes it, the >> correct interaction is acting like a closed queue for everybody. > > Ah. This is the one statement that makes me favorable to this idea. > When there is a single consumer, it's easy enough to send a sentinel. > But when there are multiple consumers, suddenly you must send exactly > the right number of sentinels (which means you also have to careful > keep track of their number, which isn't always easy). There's some > delicate code doing exactly that in concurrent.futures. > You don't need more than one sentinel. When a consumer sees the sentinel, it just needs to put it back for the other consumers. [snip] From tjreedy at udel.edu Sun Oct 21 20:02:38 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 21 Oct 2018 20:02:38 -0400 Subject: [Python-ideas] Add closing and iteration to threading.Queue In-Reply-To: <2866bc8a-c9ec-bc24-ba37-38d2752d9d2d@mrabarnett.plus.com> References: <2866bc8a-c9ec-bc24-ba37-38d2752d9d2d@mrabarnett.plus.com> Message-ID: On 10/21/2018 2:42 PM, MRAB wrote: > On 2018-10-21 18:58, Vladimir Filipovi? wrote: >> Hi! >> >> I originally submitted this as a pull request. Raymond Hettinger >> suggested it should be given a shakeout in python-ideas first. >> >> https://github.com/python/cpython/pull/10018 >> https://bugs.python.org/issue35034 The proposed close method would only half-close the queue: closed to puts, open to gets (but perhaps close completely when the last item is gotten. > FTR, this has been discussed before: > [Python-ideas] `__iter__` for queues? https://mail.python.org/pipermail/python-ideas/2010-January/006711.html Worth reading in relation to the new proposal. One person reported having an IterableQueue with close that is probably similar to the current proposal. -- Terry Jan Reedy From steve at pearwood.info Sun Oct 21 20:13:32 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 22 Oct 2018 11:13:32 +1100 Subject: [Python-ideas] TypeHinting: From variable name to type In-Reply-To: <3cad7e64-6781-f3ad-1d14-c3a4e93cd037@thomas-guettler.de> References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> <20181019101549.GR3817@ando.pearwood.info> <3cad7e64-6781-f3ad-1d14-c3a4e93cd037@thomas-guettler.de> Message-ID: <20181022001332.GY3817@ando.pearwood.info> On Sat, Oct 20, 2018 at 07:53:18PM +0200, Thomas G?ttler Lists wrote: > > Am 19.10.18 um 12:15 schrieb Steven D'Aprano: > >>One solution is the do typehinting everywhere the veriable gets used. > > > >You shouldn't need to. Your IDE or type-checker should be able to do > >type-inference and infer the type. You only need to add a hint if it > >cannot infer the type. > > > >If your IDE doesn't do type inference, get a better IDE *wink* > > > I use the free community version of pyCharm, and it often does not know > which type a variable has. A simple name2type mapping would improve > the situation a lot. If the free community version of PyCharm doesn't support type-inference, what makes you think it would support this "name2type" mapping? I don't use PyCharm, and I don't know if this feature applies to the free version, but according to their blog, PyCharm not only supports type-inference but also uses dynamic runtime information to enhance static type checking: https://blog.jetbrains.com/pycharm/2013/02/dynamic-runtime-type-inference-in-pycharm-2-7/ If you find yourself needing to annotate lots of variables, I suspect that you're fighting the IDE rather than using it correctly. Without seeing your code, and knowing a lot more about PyCharm, I cannot say further. But the critical point here is that we should not add a language feature to make up for the limitations of a single IDE. If the free version of PyCharm is underpowered, perhaps you ought to try the paid version, or another IDE, or submit a feature request to PyCharm, *before* turning to the Python language. > >>But why does the human brain not need this? > >> > >>Because it is intelligent? > >Yes. > > > > > >>I would not call this intelligence. There is a simple > >>dictionary in the brain of the developer, which maps: > >> > >>variable-name --> Type > >Is that a fact? How do you know? > > > >How does the information in this dict get filled in? > > > as I already said before a docstring in the __init__.py file would > be a pragmatic solution. You misunderstand me. You claim that there is a dictionary in the brain of the developer, right now. How does that dictionary *in the brain* get there, if it isn't through the use of human intelligence? I believe you are too focused on a single specific implementation: "use a dict to map variable name to type -- I know how to implement this, therefore that is what the brain must do" whereas I think that the reality is more complex: what you see as a mapping, I expect is just a special case of more general, and far more powerful, general reasoning intelligence. We can infer types from variable names, because we know what the name means, not because "there is a simple dictionary in the brain of the developer" mapping name to type. exchange_rate # probably a float num_rows # number of rows, therefore an int name # a string phone_number # not a float! should be a string response # a response from something (depends on context) If we read Greek, we'd probably even recognise that a variable called "?????" was likely to be a string. [...] > >When the context is different, we interpret the name differently: > > > > response = input("Continue? Y/n ") > > response = chatbot.lookup(question) > > response = mapping[challenge] > > > > > >would all be interpreted differently, even if the module used Django. I > >doubt any reasonable programmer would imagine that they were > >HttpResponse objects. > > relax. I mean this like I said. Please relax. Please don't patronise me. Just because I disagree with your suggestion doesn't mean I am freaking out and need to be told to relax. We should not implement new features after ONLY considering when the feature would be useful. We should also consider when the feature is not needed, and when it would be actively harmful, and weigh it on the balance. Your suggestion is useful when: - you use the same variable name over and over again; - always (or almost always) to mean the same type; - and type inference won't work; - and you don't want to annotate the variable each time; - and the IDE supports this new feature; - and there is little or no chance that you might want to use the same name for a different type in the same module. It is not useful, maybe even harmful, if any of those conditions are not met. I'm especially concerned about the last one: type-checking false alarms due to a clash between the name2type mapping and the programmer's intention. But on the other hand, if this is a convention purely implemented by IDEs, then I don't give a damn about this one bit. (Except to say, take the discussion to the "code-quality" mailing list, not here.) It doesn't sound like something that needs language support. > >What if you are using *both* django and requests in the same module? You > >could have both of these: > > > > response = something_returning_requests_Response() > > > > response = something_returning_django_HttpResponse() > > > >in the one module. > > > > Do you really care for this edge-case? I don't. Of course I do. It isn't an edge-case, it is representative of the vast majority of variable names: - "A single variable name is always the same type" is the edge-case. - "A single variable name can represent different types in different contexts" is the general case in dynamically-typed languages like Python. So we should care about the case where variables are reused in different contexts. > The way to get the type info is the same: Check the method, > if it has it use this. Do not use name2type mapping. End of type detection. So why doesn't that already work for you? Why do you need name2type mapping *at all* if this problem is so easy to solve? The bottom line here is that I don't think this needs language support, I would be against adding language support, but if IDEs want to support this, so long as they offer a switch to turn it off I don't care what they do :-) -- Steve From steve at pearwood.info Sun Oct 21 20:34:27 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 22 Oct 2018 11:34:27 +1100 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: <20181022003426.GA3817@ando.pearwood.info> On Mon, Oct 22, 2018 at 10:25:19AM +1100, Chris Angelico wrote: > On Mon, Oct 22, 2018 at 10:16 AM Terry Reedy wrote: > > Except for relatively trivial expressions, this is a bad thing. All > > functions created from lambda expressions get the same pseudo-name > > ''. This can make tracebacks worse. Perhaps more importantly, > > proper testing may become harder. > > The same considerations bite comprehensions, too, but we don't > discourage their use. So I don't think this should be a killer - not > on its own, anyhow. But we *do* discourage large, complex comprehensions, especially nested ones. And more importantly, comprehensions are also limited to a single expression, like lambda, and if you need a multi-statement comprehension we say "turn it into a for-loop". > I do not currently support any proposed syntax for multi-statement > lambda functions, mainly because they've all been ugly. But maybe > there'll be one, somewhere, some day, that makes sense. After 25 years, I think the odds of that are pretty slim. -- Steve From rosuav at gmail.com Sun Oct 21 20:44:47 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 22 Oct 2018 11:44:47 +1100 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: <20181022003426.GA3817@ando.pearwood.info> References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> <20181022003426.GA3817@ando.pearwood.info> Message-ID: On Mon, Oct 22, 2018 at 11:40 AM Steven D'Aprano wrote: > > On Mon, Oct 22, 2018 at 10:25:19AM +1100, Chris Angelico wrote: > > On Mon, Oct 22, 2018 at 10:16 AM Terry Reedy wrote: > > > > Except for relatively trivial expressions, this is a bad thing. All > > > functions created from lambda expressions get the same pseudo-name > > > ''. This can make tracebacks worse. Perhaps more importantly, > > > proper testing may become harder. > > > > The same considerations bite comprehensions, too, but we don't > > discourage their use. So I don't think this should be a killer - not > > on its own, anyhow. > > But we *do* discourage large, complex comprehensions, especially nested > ones. And more importantly, comprehensions are also limited to a single > expression, like lambda, and if you need a multi-statement comprehension > we say "turn it into a for-loop". This is true. But nobody ever says "oh but it's impossible to unit-test that for-loop". It's just a block of code within a function, and doesn't NEED to be separately tested. A lambda function should also behave as a subsection of its surrounding code, and thus not need to be unit-tested. I don't know why this is a problem for lambda functions and not for comprehensions or statements. *shrug* > > I do not currently support any proposed syntax for multi-statement > > lambda functions, mainly because they've all been ugly. But maybe > > there'll be one, somewhere, some day, that makes sense. > > After 25 years, I think the odds of that are pretty slim. Also very true... but I'm trying to be welcoming rather than just rudely say "it hasn't been solved for 25 years so there's no way you could possibly have solved this". After all, we've had plenty of other proposals that took many years to finally happen. Slim is nonzero and I'm open to the possibility that there is now a new syntax that would work. ChrisA From mertz at gnosis.cx Sun Oct 21 20:48:41 2018 From: mertz at gnosis.cx (David Mertz) Date: Sun, 21 Oct 2018 20:48:41 -0400 Subject: [Python-ideas] Python Enhancement Proposal for List methods In-Reply-To: References: Message-ID: I have changed my opinion to -1 on list.replace(). While max=n is a useful argument, a plain function could work equally well on every sequence and not be specific to lists. On Sun, Oct 21, 2018, 10:18 AM David Mertz The list comprehensions are not very hard, and are more general. EXCEPT > with the limited number of occurrences. We have this for str.replace(..., > max=n), and it is useful fairly often. > > I'm +0.5 on .replace() with that capability. But -1 on .removeall() that > adds nothing to an easy listcomp. > > On Sun, Oct 21, 2018, 9:01 AM Siva Sukumar Reddy >> Hey everyone, >> >> I am really new to Python contribution community want to propose below >> methods for List object. Forgive me if this is not the format to send an >> email. >> >> 1. *list.replace( item_to_be_replaced, new_item )*: which replaces all >> the occurrences of an element in the list instead of writing a new list >> comprehension in place. >> 2. *list.replace( item_to_be_replaced, new_item, number_of_occurrences )*: >> which replaces the occurrences of an element in the list till specific >> number of occurrences of that element. The number_of_occurrences can >> defaulted to 0 which will replace all the occurrences in place. >> 3. *list.removeall( item_to_be_removed )*: which removes all the >> occurrences of an element in a list in place. >> >> What do you think about these features? >> Are they PEP-able? Did anyone tried to implement these features before? >> Please let me know. >> >> Thank you, >> Sukumar >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Oct 21 21:06:43 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 21 Oct 2018 18:06:43 -0700 Subject: [Python-ideas] Add closing and iteration to threading.Queue In-Reply-To: References: <20181021233029.6d897061@fsol> Message-ID: On Sun, Oct 21, 2018, 16:48 MRAB wrote: > On 2018-10-21 22:30, Antoine Pitrou wrote: > > On Sun, 21 Oct 2018 19:58:05 +0200 > > Vladimir Filipovi? > > wrote: > >> > >> To anticipate a couple more possible questions: > >> > >> - What would this proposal do about multiple producers/consumers > >> needing to jointly decide _when_ to close the queue? > >> > >> Explicitly nothing. > >> > >> The queue's state is either closed or not, and it doesn't care who > >> closed it. It needs to interact correctly with multiple consumers and > >> multiple producers, but once any one piece of code closes it, the > >> correct interaction is acting like a closed queue for everybody. > > > > Ah. This is the one statement that makes me favorable to this idea. > > When there is a single consumer, it's easy enough to send a sentinel. > > But when there are multiple consumers, suddenly you must send exactly > > the right number of sentinels (which means you also have to careful > > keep track of their number, which isn't always easy). There's some > > delicate code doing exactly that in concurrent.futures. > > > You don't need more than one sentinel. When a consumer sees the > sentinel, it just needs to put it back for the other consumers. > I'm not sure if this is an issue the way Queue is used in practice, but in general you have to be careful with this kind of circular flow because if your queue communicates backpressure (which it should) then circular flows can deadlock. -n > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at cskk.id.au Sun Oct 21 23:21:46 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 22 Oct 2018 14:21:46 +1100 Subject: [Python-ideas] Add closing and iteration to threading.Queue In-Reply-To: References: Message-ID: <20181022032146.GA60729@cskk.homeip.net> On 21Oct2018 21:19, Vladimir Filipovi? wrote: >On Sun, Oct 21, 2018 at 8:45 PM MRAB wrote: >> FTR, this has been discussed before: >> [Python-ideas] `__iter__` for queues? >> https://mail.python.org/pipermail/python-ideas/2010-January/006711.html > >Thank you! Hmm, yes. My post there is this one: https://mail.python.org/pipermail/python-ideas/2010-January/006716.html I want to point out that in my code the single consumer of a Queue is incredibly common, so common that I can't off the top of my head think of _any_ uses of Queue directly: I _always_ make an IterableQueue and simply have the consumer iterate over the iterable queue. This is _exactly_ like Vladimir's proposal to my mind: my IterableQueue is iterable, and has a .close method just like his (prevent further puts and indicates end of stream) mediated with a sentinel internally. Arbitrary number of putters, _usually_ only one getter but of course there's no need for that. So to my mind his proposal is very simple and sensible, and matches almost universally my own use of Queues. Cheers, Cameron Simpson From cs at cskk.id.au Sun Oct 21 23:24:41 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 22 Oct 2018 14:24:41 +1100 Subject: [Python-ideas] Add closing and iteration to threading.Queue In-Reply-To: References: Message-ID: <20181022032441.GA74772@cskk.homeip.net> On 21Oct2018 18:06, Nathaniel Smith wrote: >On Sun, Oct 21, 2018, 16:48 MRAB wrote: >> On 2018-10-21 22:30, Antoine Pitrou wrote: >> > Ah. This is the one statement that makes me favorable to this >> > idea. >> > When there is a single consumer, it's easy enough to send a sentinel. >> > But when there are multiple consumers, suddenly you must send exactly >> > the right number of sentinels (which means you also have to careful >> > keep track of their number, which isn't always easy). There's some >> > delicate code doing exactly that in concurrent.futures. >> > >> You don't need more than one sentinel. When a consumer sees the >> sentinel, it just needs to put it back for the other consumers. Yes, this is exactly what my own IterableQUeue does. >I'm not sure if this is an issue the way Queue is used in practice, but in >general you have to be careful with this kind of circular flow because if >your queue communicates backpressure (which it should) then circular flows >can deadlock. Haven't come across this myself. A closeable queue doesn't seem circular to me. The handling of the sentinel is internal to the IterableQueue, so external users never see it. Cheers, Cameron Simpson From guido at python.org Sun Oct 21 23:31:45 2018 From: guido at python.org (Guido van Rossum) Date: Sun, 21 Oct 2018 20:31:45 -0700 Subject: [Python-ideas] Add closing and iteration to threading.Queue In-Reply-To: References: <20181021233029.6d897061@fsol> Message-ID: On Sun, Oct 21, 2018 at 6:08 PM Nathaniel Smith wrote: I'm not sure if this is an issue the way Queue is used in practice, but in general you have to be careful with this kind of circular flow because if your queue communicates backpressure (which it should) then circular flows can deadlock. Nathaniel, would you be able to elaborate more on the issue of backpressure? I think a lot of people here are not really familiar with the concepts and its importance, and it changes how you have to think about queues and the like. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From sukurcf at gmail.com Mon Oct 22 02:23:20 2018 From: sukurcf at gmail.com (Siva Sukumar Reddy) Date: Mon, 22 Oct 2018 11:53:20 +0530 Subject: [Python-ideas] Python Enhancement Proposal for List methods In-Reply-To: References: Message-ID: Sounds good to me. Thanks. On Mon 22 Oct, 2018, 06:18 David Mertz, wrote: > I have changed my opinion to -1 on list.replace(). While max=n is a useful > argument, a plain function could work equally well on every sequence and > not be specific to lists. > > On Sun, Oct 21, 2018, 10:18 AM David Mertz >> The list comprehensions are not very hard, and are more general. EXCEPT >> with the limited number of occurrences. We have this for str.replace(..., >> max=n), and it is useful fairly often. >> >> I'm +0.5 on .replace() with that capability. But -1 on .removeall() that >> adds nothing to an easy listcomp. >> >> On Sun, Oct 21, 2018, 9:01 AM Siva Sukumar Reddy > wrote: >> >>> Hey everyone, >>> >>> I am really new to Python contribution community want to propose below >>> methods for List object. Forgive me if this is not the format to send an >>> email. >>> >>> 1. *list.replace( item_to_be_replaced, new_item )*: which replaces all >>> the occurrences of an element in the list instead of writing a new list >>> comprehension in place. >>> 2. *list.replace( item_to_be_replaced, new_item, number_of_occurrences >>> )*: which replaces the occurrences of an element in the list till >>> specific number of occurrences of that element. The number_of_occurrences >>> can defaulted to 0 which will replace all the occurrences in place. >>> 3. *list.removeall( item_to_be_removed )*: which removes all the >>> occurrences of an element in a list in place. >>> >>> What do you think about these features? >>> Are they PEP-able? Did anyone tried to implement these features before? >>> Please let me know. >>> >>> Thank you, >>> Sukumar >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From hemflit at gmail.com Mon Oct 22 03:21:34 2018 From: hemflit at gmail.com (=?UTF-8?Q?Vladimir_Filipovi=C4=87?=) Date: Mon, 22 Oct 2018 09:21:34 +0200 Subject: [Python-ideas] Add closing and iteration to threading.Queue In-Reply-To: References: <20181021233029.6d897061@fsol> Message-ID: Nathaniel, thank you for the pointer to Trio. Its approach seems very robust. I'm relieved to see that a solution so fundamentally rebuilt has also settled on very similar semantics for its `.close_put()`. I think your `.clone()` idiom is very clear when the communication objects are treated as distinct endpoints. Something with similar effects on closing (not necessarily similar in idiom) would probably be a neat enhancement to the standard Queue, though if I was making that, I'd do it in an external package. ------ Antoine, regarding multiprocessing.Queue: The similarity of meaning behind closing that I was getting at is that mp.Q.close() means "I am done writing to this queue, and I don't care about the rest of you", whereas the proposed meaning of q.Q.close() is "Listen up, we are all done writing to this queue". I don't know yet that this difference necessarily creates a true incompatibility. That the effects (in terms of eager OS-resource cleanup) are different shouldn't be a problem in itself - every implementation does the right thing for itself. ------ On Mon, Oct 22, 2018 at 2:03 AM Terry Reedy wrote: > The proposed close method would only half-close the queue: closed to > puts, open to gets (but perhaps close completely when the last item is > gotten. In other words: in this proposal, there is no such thing as "closed for retrieval". A closed queue means exactly that it's closed for insertion. Retrieval becomes permanently impossible once the queue is closed and exhausted, and that's a condition that get() must treat correctly and usefully, but calling that condition "closed / completely closed / closed for retrieval" would muddle up the terminology. In the proposed implementation I've called it "exhausted", a name I've picked up god-knows-when and from god-knows-where, but it seemed reasonable. ------ Regarding sentinels in general: They are a limited-purpose solution, and this proposal should make them unnecessary in 99% of the cases. Firstly, they only naturally apply to FIFO queues. You could hack your use of LIFO and priority queues to also rely on sentinels, but it's very kludgey in the general cases, not a natural fit, and not generalizable to user-created children of Queue (which Queue otherwise explicitly aspires to support). Secondly, they only work when the producer is the one driving the flow and notifying the consumer that "no more is forthcoming". They don't work when the producer is the one who needs to be notified. Thirdly, they're a potential cause of deadlocks when the same threads act as both producers and consumers. (E.g. in a parallelized breadth-first-search.) I'm sure this is the circular flow that Nathaniel was referring to, but I'll let him detail it more or correct me. Fourthly, they don't make it easy to query the Queue about whether it's closed. This probably isn't a big deal admittedly. Sure, when sentinels are adequate, they're adequate. This proposal aims to be more general-purpose than that. From guettliml at thomas-guettler.de Mon Oct 22 04:53:27 2018 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Mon, 22 Oct 2018 10:53:27 +0200 Subject: [Python-ideas] TypeHinting: From variable name to type. Yes: no change to language, just convetion In-Reply-To: References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> Message-ID: Am 21.10.18 um 16:44 schrieb David Mertz: > On Fri, Oct 19, 2018 at 3:18 AM Thomas G?ttler > wrote: > > Now my idea: Per module and/or per file type hinting from variable name. > Maybe a magic docstring in the __init__.py file: > variable-name-mapping: > ? { > ? ? request: django.http.HttpRequest, > } > > > In general, I like this idea; but really only on a per-file basis.? A short header at the top would be easy enough for > an IDE or linter to scan.? But imposing the conventions project-wide feels too broad. > > There might, of course, be cases where the same name is used for different purposes in the same file.? But the tool can > alert the developer of that... and in that one file, she could either remove the header of refactor the names used, as > made sense for that particular code. > > This is certainly not something that requires language support.? It can easily be purely a convention, as long as > different IDEs, linters, type checkers, etc. agree on what the convention is.? Maybe at some?point in the future, if the > convention becomes adopted, there might be some help in having a standard library module, or even minimal language > recognition, of the convention.? But let's work on adopting a convention first. Yes, this sounds good. There is no need to change the python language, it is should be a convetion. Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ I am looking for feedback: https://github.com/guettli/programming-guidelines From storchaka at gmail.com Mon Oct 22 06:10:51 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 22 Oct 2018 13:10:51 +0300 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: 22.10.18 02:16, Terry Reedy ????: > All > functions created from lambda expressions get the same pseudo-name > ''.? This can make tracebacks worse.? Perhaps more importantly, > proper testing may become harder. See https://bugs.python.org/issue34856. But this can work only while lambda's body is a simple expression. > >>> for i in map(lambda x: x ** > ????????? 2, 'abc'): > ????print(i) > > Traceback (most recent call last): > ? File "", line 2, in > ??? 2, 'abc'): > ? File "", line 2, in > ??? 2, 'abc'): > TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int' In 3.8 the traceback is different (an can be even more informative with resolved issue34856). Traceback (most recent call last): File "1.py", line 1, in for i in map(lambda x: x ** File "1.py", line 1, in for i in map(lambda x: x ** TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int' > Given that there might be a hundred functions named '', I think > the specific name is a bit helpful. I think the main problem is not with tracebacks, but with reprs. If you have a pack of callbacks, it is not easy to figure out what they do if they are anonymous functions. From boxed at killingar.net Mon Oct 22 08:35:40 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Mon, 22 Oct 2018 14:35:40 +0200 Subject: [Python-ideas] TypeHinting: From variable name to type In-Reply-To: <20181022001332.GY3817@ando.pearwood.info> References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> <20181019101549.GR3817@ando.pearwood.info> <3cad7e64-6781-f3ad-1d14-c3a4e93cd037@thomas-guettler.de> <20181022001332.GY3817@ando.pearwood.info> Message-ID: <8EB3F115-0B1D-40C1-9ADA-EDACB6FEB0F4@killingar.net> > But the critical point here is that we should not add a language feature > to make up for the limitations of a single IDE. If the free version of > PyCharm is underpowered, perhaps you ought to try the paid version, or > another IDE, or submit a feature request to PyCharm, *before* turning to > the Python language. Which IDE do you use that is so much greater than PyCharm? I would love to try it! > Of course I do. It isn't an edge-case, it is representative of the vast > majority of variable names: > > - "A single variable name is always the same type" is the edge-case. I strongly disagree. I also wrote a mail with my motivation and context where I asked you to supply your basis for believing this. You have not replied. I think we need to try to understand each others perspective here, but it's really hard to understand yours if you won't give us any information on it. / Anders From boxed at killingar.net Mon Oct 22 08:40:01 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Mon, 22 Oct 2018 14:40:01 +0200 Subject: [Python-ideas] TypeHinting: From variable name to type. Yes: no change to language, just convetion In-Reply-To: References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> Message-ID: <30AA8719-DE26-4121-9155-51ADEC7BD948@killingar.net> >> This is certainly not something that requires language support. It can easily be purely a convention, as long as different IDEs, linters, type checkers, etc. agree on what the convention is. Maybe at some point in the future, if the convention becomes adopted, there might be some help in having a standard library module, or even minimal language recognition, of the convention. But let's work on adopting a convention first. > > Yes, this sounds good. There is no need to change the python language, it is should be a convetion. Might not even be a convention. I have now tried using https://github.com/edreamleo/make-stub-files on the code base at work. I had to fix some fatal bugs, and it's still pretty buggy even after those fixes, AND PyCharm has some nasty bugs with stub files that are blockers, but it's clear that generating stub files to do this is feasible. / Anders -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Oct 22 09:04:35 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 23 Oct 2018 00:04:35 +1100 Subject: [Python-ideas] TypeHinting: From variable name to type In-Reply-To: <8EB3F115-0B1D-40C1-9ADA-EDACB6FEB0F4@killingar.net> References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> <20181019101549.GR3817@ando.pearwood.info> <3cad7e64-6781-f3ad-1d14-c3a4e93cd037@thomas-guettler.de> <20181022001332.GY3817@ando.pearwood.info> <8EB3F115-0B1D-40C1-9ADA-EDACB6FEB0F4@killingar.net> Message-ID: <20181022130434.GE3817@ando.pearwood.info> On Mon, Oct 22, 2018 at 02:35:40PM +0200, Anders Hovm?ller wrote: > > > But the critical point here is that we should not add a language feature > > to make up for the limitations of a single IDE. If the free version of > > PyCharm is underpowered, perhaps you ought to try the paid version, or > > another IDE, or submit a feature request to PyCharm, *before* turning to > > the Python language. > > Which IDE do you use that is so much greater than PyCharm? I would love to try it! My IDE is Unix. (Technically, Linux.) https://sanctum.geek.nz/arabesque/series/unix-as-ide/ Or just google https://duckduckgo.com/?q=unix+as+an+ide Although I'm not as hard-core as some (I use an actual GUI editor, not Vim or Emacs). I'm not opposed to the concept of IDEs as such, my first two development environments (THINK Pascal, and Hypercard) were IDEs and they were great, especially considing the limited resources they had available back in 1988 or so. But... modern IDEs... I dunno... I don't begrudge you if you like them, but I don't think they're for me. > > Of course I do. It isn't an edge-case, it is representative of the vast > > majority of variable names: > > > > - "A single variable name is always the same type" is the edge-case. > > I strongly disagree. Okay. -- Steve From 2QdxY4RzWzUUiLuE at potatochowder.com Mon Oct 22 15:12:13 2018 From: 2QdxY4RzWzUUiLuE at potatochowder.com (Dan Sommers) Date: Mon, 22 Oct 2018 15:12:13 -0400 Subject: [Python-ideas] TypeHinting: From variable name to type In-Reply-To: <20181022130434.GE3817@ando.pearwood.info> References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> <20181019101549.GR3817@ando.pearwood.info> <3cad7e64-6781-f3ad-1d14-c3a4e93cd037@thomas-guettler.de> <20181022001332.GY3817@ando.pearwood.info> <8EB3F115-0B1D-40C1-9ADA-EDACB6FEB0F4@killingar.net> <20181022130434.GE3817@ando.pearwood.info> Message-ID: <63da050b-acd1-5fa7-a93a-ea79121093f6@potatochowder.com> On 10/22/18 9:04 AM, Steven D'Aprano wrote: > My IDE is Unix. (Technically, Linux.) +1 > Or just google https://duckduckgo.com/?q=unix+as+an+ide Thank you for not verbing DuckDuckGo! :-) > ... (I use an actual GUI editor, not > Vim or Emacs) ... [ ... must ... resist ... holy war ... ] Dan From mike at selik.org Mon Oct 22 15:46:37 2018 From: mike at selik.org (Michael Selik) Date: Mon, 22 Oct 2018 12:46:37 -0700 Subject: [Python-ideas] [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals In-Reply-To: References: <20181012144847.7280967b@fsol> Message-ID: I switched this thread to the python-ideas list, since this is proposing a new feature. On Mon, Oct 22, 2018 at 12:13 PM Sean Harrington wrote: > I contend that multiprocessing.Pool is used most frequently with a single > task. I am proposing a feature that enforces this invariant, optimizes task > memory-footprints & thus serialization time, and preserves the > well-established interface to Pool through subclassing. > We've got a disagreement over interface design. That's more easily discussed with concrete examples. In my own work, I often create a procmap or threadmap function to make that pattern more pleasant. Since you're making the proposal, why not share some examples of using Pool in the manner you'd like? As real a chunk of code as you're able to share publicly (and concisely). -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Oct 22 19:02:05 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 23 Oct 2018 10:02:05 +1100 Subject: [Python-ideas] TypeHinting: From variable name to type In-Reply-To: References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> <20181019101549.GR3817@ando.pearwood.info> Message-ID: <20181022230203.GF3817@ando.pearwood.info> On Fri, Oct 19, 2018 at 01:14:39PM +0200, Anders Hovm?ller wrote: [I wrote this] > > I've seen far too many variables called (let's say) "mylist" which > > actually hold a dict or a tuple (or in one memorable case, a string!) to > > unconditionally believe the name. > > This is an even stronger argument for the proposal I think. If IDEs > and static analysis tools looked at stuff like this and assumed a type > _and then found that this assumption is violated_, it would be a huge > win! That's an argument for *static type checking*, not necessarily an argument for *this* specific proposal. Any method of checking types, be it static typing, mandatory type declarations, type inference, stub files, annotations, or anything else, would help in that situation. This specific proposal assumes that when you have a single name, it always (or at least, nearly always) means the same type, across an entire package. Therefore it is safe to map the name *alone* to the type, rather than annotate the variable (name in a specific context) with the type. Of course I get it that you only choose *selected* names to label with this name2type mapping. I just don't think this happens often enough to make it a language feature, or would be valuable enough when it does happen, to make up for the disadvantages. You're using a *project-wide global* declaration, distant from the variable's context, possibly even in a completely different file from where the variable is used. That's just about the worst possible way to annotate a variable with a type. But if some, or all, IDEs want to support this, I have no objections. Just don't use it in any code I have to read :-) [...] > > If your IDE doesn't do type inference, get a better IDE *wink* > > > Which IDE would this be? PyCharm doesn't do this in the general case. > Not even close in the code base I work on. I don't know whether PyCharm does type-inference, but it *claims* to, and it does have a plugin which runs MyPy, and MyPy certainly does. If PyCharm lacks this feature, have you checked out the competition (e.g. Wing IDE, PyDev, Spyder)? Have you submitted a feature request? It could integrate better with MyPy, or possibly even the new type-checking tool from Facebook: https://pypi.org/project/pyre-check/ or Google's: https://github.com/google/pytype Jedi is another project which claims to implement type-inference and only require hints as a fallback: https://jedi.readthedocs.io/en/latest/docs/features.html#type-hinting > >> And this mapping dict exists once per library. > > > > Or more likely, doesn't exist at all. > > You seem to argue here, and generally, that the normal case for code > bases is that you have no consistent naming. This seems strange to me. The point of modules, and functions, is *encapsulation*. If I name a variable "spam" in function 1, why *must* a similar variable use the same name in function 2 let alone function 202, distant in another module? That implies a level of coupling that makes me uncomfortable. And probably an excess of generic names like "response" and too few *specific* names like "client_response" and "server_response". I am impressed by you and your team's attention to detail at requiring consistent names across such a large code base. I don't know if it is a good thing or a bad thing or just a thing, but I can admire the discipline it requires. I am certainly not that meticulous. But I'm also faintly horrified at how much pointless pretend- productivity this book-keeping may (or not!) have involved. You know the sort of time-wasting busy-work coders can get up to when they want to look busy without actually thinking too hard: - prettifying code layout and data structures; - pointless PEP-8-ifying code, whether it needs it or not; - obsessing about consistent names everywhere; etc. I know this because I've wasted many hours doing all of these. (I shouldn't *need* to say this, but I will anyway: I am making no comment on *you and your team* specifically, as I don't know you. I'm making a general observation about the tendency of many programmers, myself included, to waste time in unproductive "refactoring" which doesn't actually help code quality.) I don't see "inconsistent" names in different functions or modules to be a problem that needs to be avoided. Hence, I don't avoid it. I don't go out of my way to use unique names, but nor do I try to impose a single-name policy. In any case, we're still up against the fact that in 2018, the state of the art in type checkers is that they ought to be able to do what a human reader does and infer the type of your variables. It shouldn't matter whether you call it "response" or "spam" or "zpaqxerag", if your type-checker can't work out that foo = HttpResponse(*args) is a HttpResponse object, something has gone wrong somewhere. Adding yet another way to annotate variables instead of improving type-inference seems like a sub-optimal choice to me. https://mypy.readthedocs.io/en/latest/type_inference_and_annotations.html -- Steve From boxed at killingar.net Tue Oct 23 01:06:45 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Tue, 23 Oct 2018 07:06:45 +0200 Subject: [Python-ideas] TypeHinting: From variable name to type In-Reply-To: <20181022230203.GF3817@ando.pearwood.info> References: <2d9d1825-2619-e4a0-1b19-6a614666816a@thomas-guettler.de> <20181019101549.GR3817@ando.pearwood.info> <20181022230203.GF3817@ando.pearwood.info> Message-ID: I'll just simply reply here that Mypy on our entire code base just silently outputs nothing. Nothing at all. I tried to look at the code to find out why and maybe implement the name checking feature discussed here and was terrified by the Byzantine complexity of the code base. I gave up. From what I later understood of Mypy it is supposed to be used to check all types or none on a file level, not just check what can be checked so you can gradually add typing in reasonable places. This seems the totally wrong approach to me. I think we agree on this point. As for why we have consistent names, it was before my time but keeping this consistency isn't much work, we certainly don't go back and change code to enforce conventions. > On 23 Oct 2018, at 01:02, Steven D'Aprano wrote: > > On Fri, Oct 19, 2018 at 01:14:39PM +0200, Anders Hovm?ller wrote: > > [I wrote this] >>> I've seen far too many variables called (let's say) "mylist" which >>> actually hold a dict or a tuple (or in one memorable case, a string!) to >>> unconditionally believe the name. >> >> This is an even stronger argument for the proposal I think. If IDEs >> and static analysis tools looked at stuff like this and assumed a type >> _and then found that this assumption is violated_, it would be a huge >> win! > > That's an argument for *static type checking*, not necessarily an > argument for *this* specific proposal. Any method of checking types, be > it static typing, mandatory type declarations, type inference, stub > files, annotations, or anything else, would help in that situation. > > This specific proposal assumes that when you have a single name, it > always (or at least, nearly always) means the same type, across an > entire package. Therefore it is safe to map the name *alone* to the > type, rather than annotate the variable (name in a specific context) > with the type. > > Of course I get it that you only choose *selected* names to label with > this name2type mapping. I just don't think this happens often enough to > make it a language feature, or would be valuable enough when it does > happen, to make up for the disadvantages. > > You're using a *project-wide global* declaration, distant from the > variable's context, possibly even in a completely different file from > where the variable is used. That's just about the worst possible way to > annotate a variable with a type. > > But if some, or all, IDEs want to support this, I have no objections. > Just don't use it in any code I have to read :-) > > > [...] >>> If your IDE doesn't do type inference, get a better IDE *wink* >> >> >> Which IDE would this be? PyCharm doesn't do this in the general case. >> Not even close in the code base I work on. > > I don't know whether PyCharm does type-inference, but it *claims* to, > and it does have a plugin which runs MyPy, and MyPy certainly does. > > If PyCharm lacks this feature, have you checked out the competition > (e.g. Wing IDE, PyDev, Spyder)? Have you submitted a feature request? It > could integrate better with MyPy, or possibly even the new type-checking > tool from Facebook: > > https://pypi.org/project/pyre-check/ > > or Google's: > > https://github.com/google/pytype > > Jedi is another project which claims to implement type-inference and > only require hints as a fallback: > > https://jedi.readthedocs.io/en/latest/docs/features.html#type-hinting > > >>>> And this mapping dict exists once per library. >>> >>> Or more likely, doesn't exist at all. >> >> You seem to argue here, and generally, that the normal case for code >> bases is that you have no consistent naming. This seems strange to me. > > The point of modules, and functions, is *encapsulation*. If I name a > variable "spam" in function 1, why *must* a similar variable use the > same name in function 2 let alone function 202, distant in another > module? That implies a level of coupling that makes me uncomfortable. > > And probably an excess of generic names like "response" and too few > *specific* names like "client_response" and "server_response". > > I am impressed by you and your team's attention to detail at requiring > consistent names across such a large code base. I don't know if it is a > good thing or a bad thing or just a thing, but I can admire the > discipline it requires. I am certainly not that meticulous. > > But I'm also faintly horrified at how much pointless pretend- > productivity this book-keeping may (or not!) have involved. You know the > sort of time-wasting busy-work coders can get up to when they want to > look busy without actually thinking too hard: > > - prettifying code layout and data structures; > - pointless PEP-8-ifying code, whether it needs it or not; > - obsessing about consistent names everywhere; > > etc. I know this because I've wasted many hours doing all of these. > > (I shouldn't *need* to say this, but I will anyway: I am making no > comment on *you and your team* specifically, as I don't know you. I'm > making a general observation about the tendency of many programmers, > myself included, to waste time in unproductive "refactoring" which > doesn't actually help code quality.) > > I don't see "inconsistent" names in different functions or modules to be > a problem that needs to be avoided. Hence, I don't avoid it. I don't go > out of my way to use unique names, but nor do I try to impose a > single-name policy. > > In any case, we're still up against the fact that in 2018, the state of > the art in type checkers is that they ought to be able to do what a > human reader does and infer the type of your variables. It shouldn't > matter whether you call it "response" or "spam" or "zpaqxerag", if your > type-checker can't work out that > > foo = HttpResponse(*args) > > is a HttpResponse object, something has gone wrong somewhere. Adding yet > another way to annotate variables instead of improving type-inference > seems like a sub-optimal choice to me. > > https://mypy.readthedocs.io/en/latest/type_inference_and_annotations.html > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From njs at pobox.com Tue Oct 23 01:13:38 2018 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 22 Oct 2018 22:13:38 -0700 Subject: [Python-ideas] Add closing and iteration to threading.Queue In-Reply-To: References: <20181021233029.6d897061@fsol> Message-ID: On Sun, Oct 21, 2018 at 8:31 PM, Guido van Rossum wrote: > On Sun, Oct 21, 2018 at 6:08 PM Nathaniel Smith wrote: >> I'm not sure if this is an issue the way Queue is used in practice, but in >> general you have to be careful with this kind of circular flow because if >> your queue communicates backpressure (which it should) then circular flows >> can deadlock. > > Nathaniel, would you be able to elaborate more on the issue of backpressure? > I think a lot of people here are not really familiar with the concepts and > its importance, and it changes how you have to think about queues and the > like. Sure. Suppose you have some kind of producer connected to some kind of consumer. If the producer consistently runs faster than the consumer, what should happen? By default with queue.Queue, there's no limit on its internal buffer, so if the producer puts, say, 10 items per second, and the consumer only gets, say, 1 item per second, then the internal buffer grows by 9 items per second. Basically you have a memory leak, which will eventually crash your program. And well before that, your latency will become terrible. How can we avoid this? I guess we could avoid this by carefully engineering our systems to make sure that producers always run slower than consumers, but that's difficult and fragile. Instead, what we usually want to do is to dynamically detect when a producer is outrunning a consumer, and apply *backpressure*. (It's called that b/c it involves the consumer "pushing back" against the producer.) The simplest way is to put a limit on how large our Queue's buffer can grow, and make put() block if it would exceed this limit. That way producers are automatically slowed down, because they have to wait for the consumer to drain the buffer before they can continue executing. This simple approach also works well when you have several tasks arranged in a pipeline like A -> B -> C, where B gets objects from A, does some processing, and then puts new items on to C. If C is running slow, this will eventually apply backpressure to B, which will block in put(), and then since B is blocked and not calling get(), then A will eventually get backpressure too. In fact, this works fine for any acyclic network topology. If you have a cycle though, like A -> B -> C -> A, then you at least potentially have the risk of deadlock, where every task is blocked in put(), and can't continue until the downstream task calls get(), but it never will because it's blocked in put() too. Sometimes it's OK and won't deadlock, but you need to think carefully about the details to figure that out. If a task gets and puts to the same queue, like someone suggested doing for the sentinel value upthread, then that's a cycle and you need to do some more analysis. (I guess if you have a single sentinel value, then queue.Queue is probably OK, since the minimal buffer size it supports is 1? So when the last thread get()s the sentinel, it knows that there's at least 1 free space in the buffer, and can put() it back without blocking. But if there's a risk of somehow getting multiple sentinel values, or if Queues ever gain support for zero-sized buffers, then this pattern could deadlock.) There's a runnable example here: https://trio.readthedocs.io/en/latest/reference-core.html#buffering-in-channels And I also wrote about backpressure and asyncio here: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#bug-1-backpressure -n -- Nathaniel J. Smith -- https://vorpus.org From hemflit at gmail.com Tue Oct 23 05:03:21 2018 From: hemflit at gmail.com (=?UTF-8?Q?Vladimir_Filipovi=C4=87?=) Date: Tue, 23 Oct 2018 11:03:21 +0200 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: Chris, I'm happy to work with you to hammer out comparisons of various solutions. But I can't take on the role of an advocate for "multi-statement lambdas". I don't even understand what precisely that covers, since we don't have uni-statement lambdas. _If that role would be needed for this discussion, the rest of what I'm about to write may be a waste of your time!_ I got involved in this thread because examples of something were asked for, and I had a couple. I could defend a claim like "some _limited set_ of statements inside lambdas would be useful and pythonic", if that's a useful conversation to have. (My view BTW is that _indented compound statements_ in lambdas would be _irreducibly_ unpythonic, and the only way that can change is if the concept of pythonicity happens to drift over time.) ------ > Next step: Offer a variety of alternate > syntaxes that *do* currently work, and then the proposed > multi-statement lambda, and show how the current syntaxes are all > ugly. Good, but we may have a disconnect about which "ugliness"/problem is being solved. The central point of Python's lambdas, for me, is that when one needs to refer to a _very simple_, _once-off_ function, lambdas _make the code better._ The ways they make it better go beyond saving vertical space and can be subtle: they improve the locality of code by allowing the function's definition to be exactly in the place where it's used; a lot like the difference between (in pseudocode) `output("Hello, world")` and having to say `String hw = new String("Hello, world"); output(hw)`. So, what does the existing lambda feature offer in those situations? 1. Virtually no pomp and ceremony (one keyword, sometimes needs parentheses, no compulsory newlines). 2. Relieves me from having to decide on the place for the definition (right before use? at the beginning of the using function/scope? right before the using function/scope?). This could be resolved by just having a convention, but I note there isn't an existing convention ("Schelling point") for the just-define-a-named-function approach. 3. That code-locality thing. It relieves me as reader from having to mentally connect its definition to its usage, and from having to deduce that it's not also used elsewhere. 4. Relieves me from having to come up with a one-use name. No, `foo` isn't as good. 5. (I shouldn't have to include this but) Expressions don't lose clarity by being moved into a lambda. Past the keyword itself, the contents of the lambda look exactly like (= as simple as) what I would write in non-lambda Python code. I don't need to contort them into special lambda-specific constructs. None of those things are _universally_ desirable! But the existing lambda lets me _choose_ that in some particular case (very simple, once-off function) they do make the code better, so I can get them. Except that sometimes I need a _very simple_, _once-off_ function that would do something that's a statement. It seems like a very similar situation, the concept doesn't seem antithetical to the philosophy of Python's existing lambdas, so it's a bit frustrating that I can't choose to use the same feature and reap those same benefits. Other times, I want the lambda to do a couple of things in sequence. No ifs or try-finallys, just one thing with its side-effects, followed by the other. This is already doable as evaluation of a tuple (if the things are expressions), and that's not _too_ ugly, but it would become more direct (as in quality #5 above) if it was expressible as a pair of expression-statements. ------ Okay now, what does the decorator solution do in that WebSocket situation that just defining named functions doesn't already? I suppose it helps somewhat with property #2 above (finding a place for the definition). Outside of that list, it lets us shorten the constructor call. Am I missing something? Or am I moving the goalposts - were you explicitly trying to solve something else? ------ Let me try to anticipate one more existing solution (the best I came up with): Where the statement is about updating counters or similar state variables, we can make them items in a dict (or collections.Counter) instead, and update them in a lambda via dict.update(). It's not terrible, but it means all other references to the state variable must change in a way that makes them less clear. ("Why are they using this dict at all? Ohh, because lambda.") And having to replace `c += 1` with `d.update(c = d['c'] + 1)` or `counter.update(c=1)` is still ugly. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Oct 23 07:15:11 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 23 Oct 2018 22:15:11 +1100 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: On Tue, Oct 23, 2018 at 8:04 PM Vladimir Filipovi? wrote: > > Chris, I'm happy to work with you to hammer out comparisons of various solutions. > > But I can't take on the role of an advocate for "multi-statement lambdas". I don't even understand what precisely that covers, since we don't have uni-statement lambdas. > _If that role would be needed for this discussion, the rest of what I'm about to write may be a waste of your time!_ > If you were confused by my use of the generic "you", then I apologize. This wasn't aimed personally at you, but more generally at "this is what we need". And you're right that we don't have uni-statement lambdas, but I suspect people would be more confused by "suite lambdas" as an alternative to "expression lambdas", unless they happen to be familiar with the internal details of CPython's grammar :) I suppose in theory you could design a syntax for lambda functions that permits a simple_stmt, or maybe a compound_stmt, but both would be as messy as supporting a full suite (or multiple statements, in vernacular) and would be unnecessarily restrictive. > The central point of Python's lambdas, for me, is that when one needs to refer to a _very simple_, _once-off_ function, lambdas _make the code better._ > The ways they make it better go beyond saving vertical space and can be subtle: they improve the locality of code by allowing the function's definition to be exactly in the place where it's used; a lot like the difference between (in pseudocode) `output("Hello, world")` and having to say `String hw = new String("Hello, world"); output(hw)`. > My view, which I suspect is the same as yours but written in different words, is that a lambda function should be treated as a block of code inside another function, and not as its own entity. You don't give a name to the block of code between "if" and "else", because it's just code as part of the implementation of its containing function. You don't give a name to the implicit function created by a list comprehension, because it's just ... you get the idea. > Okay now, what does the decorator solution do in that WebSocket situation that just defining named functions doesn't already? > > I suppose it helps somewhat with property #2 above (finding a place for the definition). > Outside of that list, it lets us shorten the constructor call. > > Am I missing something? Or am I moving the goalposts - were you explicitly trying to solve something else? Part of getting a proposal accepted is showing that there is no adequate solution using existing tools. The best way to show that is to come up with the best possible solution that we currently have, and demonstrate (if it isn't self-evident) that it's sub-par. The decorator solution is one of two that came to my mind as existing options (the other being an abuse of class syntax). Criticizing it is exactly correct, and is a way to support the assertion "we need better lambda functions". > Let me try to anticipate one more existing solution (the best I came up with): > > Where the statement is about updating counters or similar state variables, we can make them items in a dict (or collections.Counter) instead, and update them in a lambda via dict.update(). > > It's not terrible, but it means all other references to the state variable must change in a way that makes them less clear. ("Why are they using this dict at all? Ohh, because lambda.") > > And having to replace `c += 1` with `d.update(c = d['c'] + 1)` or `counter.update(c=1)` is still ugly. Also a point worth raising. Currently, lambda functions are permitted to mutate objects in surrounding scopes, but not to rebind them. Well, actually, PEP 572 might well be the solution there, but then you reopen all that lovely controversy... I mentioned abusing class syntax as an alternative solution. This would require a new API, but it wouldn't be hard to write a wrapper. It'd end up something like this: @args(websocket.WebSocketApp, url) class ws: def on_open(...): def on_message(...): The decorator could look something like this (untested): def args(func, *a, **kw): def wrapper(cls): for name, val in cls.__dict__.items(): if not name.startswith("_"): assert name not in kw kw[name] = val return func(*a, **kw) return wrapper In some contexts, that would be every bit as good as a lambda-based solution. To compete, the proposed lambda syntax would have to be better than all these - by no means impossible, but it's a target to aim at. ChrisA From python at mrabarnett.plus.com Tue Oct 23 11:46:38 2018 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 23 Oct 2018 16:46:38 +0100 Subject: [Python-ideas] Add closing and iteration to threading.Queue In-Reply-To: References: <20181021233029.6d897061@fsol> Message-ID: <9c2f1dff-d9a0-e752-88dd-221584eb8d17@mrabarnett.plus.com> On 2018-10-23 06:13, Nathaniel Smith wrote: > On Sun, Oct 21, 2018 at 8:31 PM, Guido van Rossum wrote: >> On Sun, Oct 21, 2018 at 6:08 PM Nathaniel Smith wrote: >>> I'm not sure if this is an issue the way Queue is used in practice, but in >>> general you have to be careful with this kind of circular flow because if >>> your queue communicates backpressure (which it should) then circular flows >>> can deadlock. >> >> Nathaniel, would you be able to elaborate more on the issue of backpressure? >> I think a lot of people here are not really familiar with the concepts and >> its importance, and it changes how you have to think about queues and the >> like. > > Sure. > > Suppose you have some kind of producer connected to some kind of > consumer. If the producer consistently runs faster than the consumer, > what should happen? By default with queue.Queue, there's no limit on > its internal buffer, so if the producer puts, say, 10 items per > second, and the consumer only gets, say, 1 item per second, then the > internal buffer grows by 9 items per second. Basically you have a > memory leak, which will eventually crash your program. And well before > that, your latency will become terrible. How can we avoid this? > [snip] The purpose of the sentinel is to tell the consumer(s) that there are no more items, that the producer has finished producing. The sentinel is the only item in the queue, there will be no more items after it, and backpressure is not an issue. From philip.martin2007 at gmail.com Tue Oct 23 23:13:08 2018 From: philip.martin2007 at gmail.com (Philip Martin) Date: Tue, 23 Oct 2018 22:13:08 -0500 Subject: [Python-ideas] Python 3.7 dataclasses attribute order Message-ID: Hi, I just started to use the new dataclasses module. My initial use case boils down to somewhere between a namedtuple and a class where I want a record with a few methods. Mainly, I am trying to build a specific record from various sources, and then have the class handle validation and serialization. One roadblock I have found is that I can't use the field order I define for the class to say write out a csv file if any of the fields have default value. I know this bucks Python's args, kwargs ordering, but I think having the ability define optional arguments and required arguments in any order helps improve the classes intended usability. For instance, imagine "account_description_2" is None for most sources, but must appear before "other_required_field" in a CSV exported file. I think it would be useful to be able to do the following: import csv from datetime import date from dataclasses import dataclass, fields from typing import List OBJECT_SERIALIZERS = {date: date.isoformat} @dataclass class Account: account_id: str account_description: str account_description_2: str = None # INVALID other_required_field: str def serialize(self): for field in fields(self): value = getattr(self, field.name) serializer = OBJECT_SERIALIZERS.get(field.type, None) if serializer: value = serializer(value) yield value @property def field_names(self): return [field.name for field in fields(self)] @classmethod def from_source_a(cls, record): return cls(account_id=record['account_code'], account_description=record['account_name'], other_required_field=record['other_field']) @dataclass class AccountList: accounts: List[Account] @property def field_names(self): return [ field.name for field in fields(fields(self)[0].type.__args__[0]) ] @property def record_field(self): return fields(self)[0].name def to_csv(self, path): with open(path, 'w') as file: self.write_file(file) def write_file(self, file): records_field = self.record_field writer = csv.writer(file) writer.writerow(self.field_names) writer.writerows( record.serialize() for record in getattr(self, records_field) ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From reg.python at poti.sk Wed Oct 24 02:38:29 2018 From: reg.python at poti.sk (reg.python at poti.sk) Date: Wed, 24 Oct 2018 08:38:29 +0200 Subject: [Python-ideas] asymcio.TimerHandle.done() Message-ID: I'm rewriting an existing python program using a 3rd party async library to standard asyncio. That library allows one to check if a timer created with: timer = loop.call_later(delay, callback) is active, i.e. not cancelled and the scheduled callback not executed yet. I'm missing that functionality in asyncio and would like to propose: TimerHandle.done() = True if cancelled or no longer waiting in the event loop's queue; otherwise False In that program sometimes a state is defined just by the mere existence of an active timer - no state variables, no real callbacks. I find that quite a natural mini-pattern. From marko.ristin at gmail.com Wed Oct 24 03:40:25 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Wed, 24 Oct 2018 09:40:25 +0200 Subject: [Python-ideas] Contracts in python -- a report & next steps Message-ID: Hi, I would like to give you a short report on the development of icontract library following the discussion about the introduction of contracts into Python (e.g., see [1, 2, 3, 4]). *Features* The functionality of icontract library is basically on par with Eiffel, Cobra and other languages supporting contracts except for loop invariants and less readable syntax. The features include: * "require", "ensure" and "invariant" decorators to define the contracts * "snapshot" decorator to capture the values prior to the function invocation to allow postconditions to verify the state transitions * inheritance of the contracts (with strengthening/weakening) * tracing of the argument values on contract violation * informative violation messages automatically parsed from the code of the condition function * The user can specify custom errors to be raised on a specific contract violation * individual toggling of the contracts * linter to check the contract arguments * sphinx extension to render the contracts automatically in the documentation (including a sophisticated matching of logical implications) We covered all the use cases we could find in our code base (at Parquery AG) such as: * functions * instance methods * class and static methods * property getters, setters and deleters * slot wrappers *Roadblocks* During the development, the following roadblocks were encountered: * We wanted to include the contracts in the output of help(). Unfortunately, help() renders the __doc__ of the class and not of the instance. For functions, this is the class "function" which you can not inherit from. See [5] for more details. * We need to inspect the source code of the condition and error lambdas to generate the violation message and infer the error type in the documentation, respectively. inspect.getsource(.) is broken on lambdas defined in decorators in Python 3.5.2+ (see [6]). We circumvented this bug by using inspect.findsource(.), inspect.getsourcefile(.) and examining the local source code of the lambda by searching for other decorators above and other decorators and a function or class definition below. The decorator code is parsed and then we match the condition and error arguments in the AST of the decorator. This is brittle as it prevents us from having partial definitions of contract functions or from sharing the contracts among functions. Here is a short code snippet to demonstrate where the current implementation fails: import icontract require_x_positive = icontract.require( lambda x: x > 0, error=lambda: ValueError("x must be positive")) @require_x_positive def some_func(x: int) -> None: pass However, we haven't faced a situation in the code base where we would do something like the above, so I am unsure whether this is a big issue. As long as decorators are directly applied to functions and classes, everything worked fine on our code base. *Our Experience* We have been using icontract in our team for our production code base (~100K LOC) as well as for a couple of open source projects (each <10K LOC) since August 1st 2018 (~ 3 months). In contrast to points raised during the discussions in [1, 2, 3, 4], we did not have issues with readability and modifying contracts. Thus far, we have not encountered problems with variable renames and major code refactorings. We use Pycharm, so it remains open whether this also applies to development environments such as emacs and vim. We confirm that contracts help us improve the documentation, keep the documentation up-to-date, better reason about the function's input and output and catch bugs early and more easily during the unit and integration tests. *Status* The library interface is currently frozen after the version bump to 2.0.0 and is not expected to change in the next six months. All the reported bugs have been fixed and no bug fixes are pending. *Next Steps?* I personally doubt that we are enough people to form a party to push for a change in the language. A standardized library seems to me like a realizable compromise given the state of the discussion on this mail list. Before we organize a collective to write a proposal to standardize the library, I would suggest that a couple of other interested teams adopt icontract, apply it to their code bases and report their experiences on this mail list. I hope that 2-3 reports would be insightful enough to either convince other people that contracts in python are worth standardizing (and highlight missing features yet to be implemented) or provide solid material to discard the endeavor at the current moment. In the meanwhile, it would be of great help if somebody could vet the documentation and the code of icontract. *Authors* The library was mainly written by me (with some help of my colleague Adam Radomski). We discussed the features within the dev team at Parquery (Zurich, Switzerland) as well as in email correspondence with James Lu. [1] https://groups.google.com/forum/#!topic/python-ideas/mvzLsdwGuww [2] https://groups.google.com/forum/#!topic/python-ideas/JtMgpSyODTU [3] https://groups.google.com/forum/#!topic/python-ideas/mvzLsdwGuww [4] https://groups.google.com/forum/#!topic/python-ideas/dmXz_7LH4GI%5B1-25%5D [5] https://groups.google.com/forum/#!topic/python-ideas/c9ntrVuh6WE [6] https://bugs.python.org/issue21217 -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko.ristin at gmail.com Wed Oct 24 03:41:34 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Wed, 24 Oct 2018 09:41:34 +0200 Subject: [Python-ideas] Contracts in python -- a report & next steps In-Reply-To: References: Message-ID: P.S. Here is the link to the github repo: https://github.com/Parquery/icontract On Wed, 24 Oct 2018 at 09:40, Marko Ristin-Kaufmann wrote: > Hi, > I would like to give you a short report on the development of icontract > library following the discussion about the introduction of contracts into > Python (e.g., see [1, 2, 3, 4]). > > *Features* > The functionality of icontract library is basically on par with Eiffel, > Cobra and other languages supporting contracts except for loop invariants > and less readable syntax. > > The features include: > * "require", "ensure" and "invariant" decorators to define the contracts > * "snapshot" decorator to capture the values prior to the function > invocation to allow postconditions to verify the state transitions > * inheritance of the contracts (with strengthening/weakening) > * tracing of the argument values on contract violation > * informative violation messages automatically parsed from the code of the > condition function > * The user can specify custom errors to be raised on a specific contract > violation > * individual toggling of the contracts > * linter to check the contract arguments > * sphinx extension to render the contracts automatically in the > documentation (including a sophisticated matching of logical implications) > > We covered all the use cases we could find in our code base (at Parquery > AG) such as: > * functions > * instance methods > * class and static methods > * property getters, setters and deleters > * slot wrappers > > *Roadblocks* > During the development, the following roadblocks were encountered: > > * We wanted to include the contracts in the output of help(). > Unfortunately, help() renders the __doc__ of the class and not of the > instance. For functions, this is the class "function" which you can not > inherit from. See [5] for more details. > > * We need to inspect the source code of the condition and error lambdas to > generate the violation message and infer the error type in the > documentation, respectively. inspect.getsource(.) is broken on lambdas > defined in decorators in Python 3.5.2+ (see [6]). We circumvented this bug > by using inspect.findsource(.), inspect.getsourcefile(.) and examining the > local source code of the lambda by searching for other decorators above and > other decorators and a function or class definition below. The decorator > code is parsed and then we match the condition and error arguments in the > AST of the decorator. This is brittle as it prevents us from having partial > definitions of contract functions or from sharing the contracts among > functions. > > Here is a short code snippet to demonstrate where the current > implementation fails: > import icontract > > require_x_positive = icontract.require( > lambda x: x > 0, error=lambda: ValueError("x must be positive")) > > @require_x_positive > def some_func(x: int) -> None: > pass > > However, we haven't faced a situation in the code base where we would do > something like the above, so I am unsure whether this is a big issue. As > long as decorators are directly applied to functions and classes, > everything worked fine on our code base. > > > *Our Experience* > We have been using icontract in our team for our production code base > (~100K LOC) as well as for a couple of open source projects (each <10K LOC) > since August 1st 2018 (~ 3 months). > > In contrast to points raised during the discussions in [1, 2, 3, 4], we > did not have issues with readability and modifying contracts. Thus far, we > have not encountered problems with variable renames and major code > refactorings. We use Pycharm, so it remains open whether this also applies > to development environments such as emacs and vim. > > We confirm that contracts help us improve the documentation, keep the > documentation up-to-date, better reason about the function's input and > output and catch bugs early and more easily during the unit and integration > tests. > > > *Status* > The library interface is currently frozen after the version bump to 2.0.0 > and is not expected to change in the next six months. All the reported bugs > have been fixed and no bug fixes are pending. > > *Next Steps?* > I personally doubt that we are enough people to form a party to push for a > change in the language. A standardized library seems to me like a > realizable compromise given the state of the discussion on this mail list. > > Before we organize a collective to write a proposal to standardize the > library, I would suggest that a couple of other interested teams adopt > icontract, apply it to their code bases and report their experiences on > this mail list. I hope that 2-3 reports would be insightful enough to > either convince other people that contracts in python are worth > standardizing (and highlight missing features yet to be implemented) or > provide solid material to discard the endeavor at the current moment. > > In the meanwhile, it would be of great help if somebody could vet the > documentation and the code of icontract. > > *Authors* > The library was mainly written by me (with some help of my colleague Adam > Radomski). We discussed the features within the dev team at Parquery > (Zurich, Switzerland) as well as in email correspondence with James Lu. > > [1] https://groups.google.com/forum/#!topic/python-ideas/mvzLsdwGuww > [2] https://groups.google.com/forum/#!topic/python-ideas/JtMgpSyODTU > [3] https://groups.google.com/forum/#!topic/python-ideas/mvzLsdwGuww > [4] > https://groups.google.com/forum/#!topic/python-ideas/dmXz_7LH4GI%5B1-25%5D > [5] https://groups.google.com/forum/#!topic/python-ideas/c9ntrVuh6WE > [6] https://bugs.python.org/issue21217 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Oct 24 03:57:48 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 24 Oct 2018 18:57:48 +1100 Subject: [Python-ideas] Contracts in python -- a report & next steps In-Reply-To: References: Message-ID: On Wed, Oct 24, 2018 at 6:41 PM Marko Ristin-Kaufmann wrote: > Next Steps? > I personally doubt that we are enough people to form a party to push for a change in the language. A standardized library seems to me like a realizable compromise given the state of the discussion on this mail list. > > Before we organize a collective to write a proposal to standardize the library, I would suggest that a couple of other interested teams adopt icontract, apply it to their code bases and report their experiences on this mail list. I hope that 2-3 reports would be insightful enough to either convince other people that contracts in python are worth standardizing (and highlight missing features yet to be implemented) or provide solid material to discard the endeavor at the current moment. > For the sake of those of us who REALLY don't feel like diving back into the extensive threads on this subject, can you please summarize the benefits of having this in the stdlib rather than as a third-party library? Also - have you benchmarked the performance cost of adding contracts? Particularly: if you're planning to contractify the stdlib, what is the impact on startup performance? ChrisA From boxed at killingar.net Wed Oct 24 05:30:53 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Wed, 24 Oct 2018 11:30:53 +0200 Subject: [Python-ideas] Python 3.7 dataclasses attribute order In-Reply-To: References: Message-ID: <238D40B8-2D52-48C2-8F92-89A05ECD224C@killingar.net> Well that seems super unfortunate. You can opt out of the auto generate constructor and do it yourself: @dataclass(init=False) class Foo: foo: str bar: str = None baz: str def __init__(self, *, foo, bar = None, baz): self.foo = foo self.bar = bar self.baz = baz Foo(foo='a', bar='b', baz='c') but this seems to take away from the utility of dataclasses. One could imagine there being a new argument to @dataclass that would make this work. Something like: @dataclass(init_kwargs_only=True) class Foo: foo: str bar: str = None baz: str where you would then get an auto generated constructor like with keyword only arguments. Personally I think this should have been the default, but it's at least a nice addition now. / Anders > On 24 Oct 2018, at 05:13, Philip Martin wrote: > > Hi, I just started to use the new dataclasses module. My initial use case boils down to somewhere between a namedtuple and a class where I want a record with a few methods. > > Mainly, I am trying to build a specific record from various sources, and then have the class handle validation and serialization. One roadblock I have found is that I can't use the field order I define for the class to say write out a csv file if any of the fields have default value. I know this bucks Python's args, kwargs ordering, but I think having the ability define optional arguments and required arguments in any order helps improve the classes intended usability. For instance, imagine "account_description_2" is None for most sources, but must appear before "other_required_field" in a CSV exported file. I think it would be useful to be able to do the following: > > import csv > from datetime import date > from dataclasses import dataclass, fields > from typing import List > > OBJECT_SERIALIZERS = {date: date.isoformat} > > @dataclass > class Account: > account_id: str > account_description: str > account_description_2: str = None > > # INVALID > other_required_field: str > > def serialize(self): > for field in fields(self): > value = getattr(self, field.name ) > serializer = OBJECT_SERIALIZERS.get(field.type, None) > > if serializer: > value = serializer(value) > yield value > > @property > def field_names(self): > return [field.name for field in fields(self)] > > @classmethod > def from_source_a(cls, record): > return cls(account_id=record['account_code'], > account_description=record['account_name'], > other_required_field=record['other_field']) > > @dataclass > class AccountList: > accounts: List[Account] > > @property > def field_names(self): > return [ > field.name for field in fields(fields(self)[0].type.__args__[0]) > ] > > @property > def record_field(self): > return fields(self)[0].name > > def to_csv(self, path): > with open(path, 'w') as file: > self.write_file(file) > > def write_file(self, file): > records_field = self.record_field > > writer = csv.writer(file) > writer.writerow(self.field_names) > writer.writerows( > record.serialize() for record in getattr(self, records_field) > ) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From boxed at killingar.net Wed Oct 24 05:34:51 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Wed, 24 Oct 2018 11:34:51 +0200 Subject: [Python-ideas] Contracts in python -- a report & next steps In-Reply-To: References: Message-ID: <3852DAE8-F137-4BB1-AA30-92AE7CF919C6@killingar.net> > Roadblocks > During the development, the following roadblocks were encountered: > > * We wanted to include the contracts in the output of help(). Unfortunately, help() renders the __doc__ of the class and not of the instance. For functions, this is the class "function" which you can not inherit from. See [5] for more details. > > * We need to inspect the source code of the condition and error lambdas to generate the violation message and infer the error type in the documentation, respectively. inspect.getsource(.) is broken on lambdas defined in decorators in Python 3.5.2+ (see [6]). Both these things seem like we should look at separately! I know I'm always annoyed that repr(lambda x: x*2) != "lambda x: x*2" for some reason. Fixing these two things in python seems like obvious wins to me, especially the latter. / Anders -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko.ristin at gmail.com Wed Oct 24 06:08:41 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Wed, 24 Oct 2018 12:08:41 +0200 Subject: [Python-ideas] Contracts in python -- a report & next steps In-Reply-To: References: Message-ID: Hi Chris, For the sake of those of us who REALLY don't feel like diving back > into the extensive threads on this subject, can you please summarize > the benefits of having this in the stdlib rather than as a third-party > library? > Certainly. We need a standard approach to contracts as opposed to third-party libraries for the following technical reasons: * There is no dependency hell if two packages use different versions of the same contract library. * Two packages need to inherit each other's contracts (*e.g.*, one package defines a class which inherits a base class from a different package and hence needs to inherit its contracts as well). * Third-party libraries for testing and static verification need a standard approach to contracts in order to be usable. Otherwise, the authors of these libraries (*e.g. *Hypothesis) need to support multiple contrat libraries (if they could ever bother to support multiple of them). There are many more, but these are the reasons that I find critical. > Also - have you benchmarked the performance cost of adding contracts? > Particularly: if you're planning to contractify the stdlib, what is > the impact on startup performance? > I made some preliminary benchmarks. The source code is available at: https://github.com/Parquery/icontract/tree/master/benchmarks/startup Average over ten runs in milliseconds on my machine (Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz, 8 cores, 4 GB RAM): Duration to import the module functions_100_with_no_contract : 795.59 ? 10.47 Duration to import the module functions_100_with_1_contract : 919.53 ? 61.22 Duration to import the module functions_100_with_5_contracts : 1075.81 ? 59.87 Duration to import the module functions_100_with_10_contracts : 1290.22 ? 90.04 Duration to import the module functions_100_with_1_disabled_contract : 833.60 ? 32.07 Duration to import the module functions_100_with_5_disabled_contracts : 851.31 ? 66.93 Duration to import the module functions_100_with_10_disabled_contracts : 897.90 ? 143.02 Duration to import the module classes_100_with_no_invariant : 843.61 ? 28.21 Duration to import the module classes_100_with_1_invariant : 3409.71 ? 95.78 Duration to import the module classes_100_with_5_invariants : 4005.93 ? 131.97 Duration to import the module classes_100_with_10_invariants : 4801.82 ? 157.56 Duration to import the module classes_100_with_1_disabled_invariant : 885.88 ? 44.24 Duration to import the module classes_100_with_5_disabled_invariants : 912.53 ? 101.91 Duration to import the module classes_100_with_10_disabled_invariants : 963.77 ? 161.76 Please let me know if these are the benchmarks you had in mind or you would like to see something else. I have neither optimized the code for speed nor investigated why the performance of class invariants differs so much from the functions. The important bit for me was that disabling contracts basically had no impact on import time. Cheers, Marko -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Wed Oct 24 05:43:44 2018 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 24 Oct 2018 05:43:44 -0400 Subject: [Python-ideas] Python 3.7 dataclasses attribute order In-Reply-To: <238D40B8-2D52-48C2-8F92-89A05ECD224C@killingar.net> References: <238D40B8-2D52-48C2-8F92-89A05ECD224C@killingar.net> Message-ID: On 10/24/2018 5:30 AM, Anders Hovm?ller wrote: > Well that seems super unfortunate. You can opt out of the auto generate > constructor and do it yourself: > > ??@dataclass(init=False) > ??class?Foo: > ? ? ? foo:?str > ? ? ? bar:?str?=?None > ? ? ? baz:?str > > ? ? ??def?__init__(self, *, foo, bar =?None, baz): > ? ? ? ? ??self.foo = foo > ? ? ? ? ??self.bar = bar > ? ? ? ? ??self.baz = baz > > > ? Foo(foo='a', bar='b', baz='c') > > but this seems to take away from the utility of dataclasses. One could > imagine there being a new argument to @dataclass that would make this > work. Something like: > > > @dataclass(init_kwargs_only=True) > ? class Foo: > ? ? ? foo: str > ? ? ? bar: str = None > ? ? ? baz: str > > where you would then get an auto generated constructor like with keyword > only arguments. Personally I think this should have been the default, > but it's at least a nice addition now. https://bugs.python.org/issue33129 I definitely wouldn't want this to be the default. And as you say, it's too late anyway. I haven't decided how the interaction of per-field and whole class versions of keyword-only should operate. Eric From rosuav at gmail.com Wed Oct 24 06:24:24 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 24 Oct 2018 21:24:24 +1100 Subject: [Python-ideas] Contracts in python -- a report & next steps In-Reply-To: References: Message-ID: On Wed, Oct 24, 2018 at 9:08 PM Marko Ristin-Kaufmann wrote: > > Hi Chris, > >> For the sake of those of us who REALLY don't feel like diving back >> into the extensive threads on this subject, can you please summarize >> the benefits of having this in the stdlib rather than as a third-party >> library? > > > Certainly. We need a standard approach to contracts as opposed to third-party libraries for the following technical reasons: > * There is no dependency hell if two packages use different versions of the same contract library. Wouldn't version differences happen just as much whether it's in the stdlib or a third-party library? The contracts API has to be built to cope with this, regardless. But having it in the stdlib would at least mean that you don't have to worry about incompatible DIFFERENT contracts libraries. > * Two packages need to inherit each other's contracts (e.g., one package defines a class which inherits a base class from a different package and hence needs to inherit its contracts as well). Ditto. So, small benefit. Enough to be of value, but not (on its own) compelling. > * Third-party libraries for testing and static verification need a standard approach to contracts in order to be usable. Otherwise, the authors of these libraries (e.g. Hypothesis) need to support multiple contrat libraries (if they could ever bother to support multiple of them). > And ditto again. So I would disagree that these benefits are critical, but they are definitely benefits. >> Also - have you benchmarked the performance cost of adding contracts? >> Particularly: if you're planning to contractify the stdlib, what is >> the impact on startup performance? > The main question is: are you intending for the standard library to be annotated with contracts? If not, all your proposed benefits can be achieved at the level of a single project, by just saying "we're going to use THIS contracts library", unless I'm misunderstanding something here. If so, you suddenly have to figure out how much longer it takes *across all of Python's startup*. How much longer will it take to run "hg help" with annotations set up? That's the kind of benchmark that's going to make or break this proposal. Adding a notable delay to command-line tools like that would seriously hurt. ChrisA From cspealma at redhat.com Wed Oct 24 09:18:14 2018 From: cspealma at redhat.com (Calvin Spealman) Date: Wed, 24 Oct 2018 09:18:14 -0400 Subject: [Python-ideas] Return for assignment blocks Message-ID: I'd like to suggest what I think would be a simple addition to `def` and `class` blocks. I don't know if calling those "Assignment Blocks" is accurate, but I just mean to refer to block syntaxes that assign to a name. Anyway, I propose a combined return-def structure, and optionally also allowing a return-class version. Omitting the name would be allowable, as well. This would only apply to a `def` or `class` statement made as the last part of the function body, of course. def ignore_exc(exc_type): return def (func): @wraps(func) return def (*args, **kwargs): try: return func(*args, **kwargs) except exc_type: pass Thanks for considering and for any comments, thoughts, or feedback on the idea! -------------- next part -------------- An HTML attachment was scrubbed... URL: From 1benediktwerner at gmail.com Wed Oct 24 09:51:09 2018 From: 1benediktwerner at gmail.com (Benedikt Werner) Date: Wed, 24 Oct 2018 15:51:09 +0200 Subject: [Python-ideas] Return for assignment blocks In-Reply-To: References: Message-ID: <1580a88e-e654-50e6-ea51-378c30f6b245@gmail.com> Would you mind providing a bit more details about your proposal? What exactly are those "Assignment Blocks" supposed to do? If I understand your proposal correctly you want this: def my_func(): ??? return def(): ??????? print("abc") to be the same as this: def my_func(): ??? def inner_func(): ??????? print("abc") ??? return inner_func But this is only my assumption as your proposal doesn't give very much details. Maybe you should provide a few simple examples and explain what they are supposed to do or what they should be equivalent to. Your example is quite complex and to me it's not clear at all what it is supposed to mean. Also and probably most importantly what is the reason you want this? Are there any good usecases where this would help? If my assumption above is correct this just looks like a bit of syntactic sugar that IMO isn't really neccessary. It doesn't really improve readability or save many characters. The existing way to do this is totally fine. Benedikt Am 24.10.2018 um 15:18 schrieb Calvin Spealman: > I'd like to suggest what I think would be a simple addition to `def` > and `class` blocks. I don't know if calling those "Assignment Blocks" > is accurate, but I just mean to refer to block syntaxes that assign to > a name. Anyway, I propose a combined return-def structure, and > optionally also allowing a return-class version. Omitting the name > would be allowable, as well. > > This would only apply to a `def` or `class` statement made as the last > part of the function body, of course. > > def ignore_exc(exc_type): > ??? return def (func): > ??????? @wraps(func) > ??????? return def (*args, **kwargs): > ??????????? try: > ??????????????? return func(*args, **kwargs) > ??????????? except exc_type: > ??????????????? pass > > Thanks for considering and for any comments, thoughts, or feedback on > the idea! > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cspealma at redhat.com Wed Oct 24 10:04:30 2018 From: cspealma at redhat.com (Calvin Spealman) Date: Wed, 24 Oct 2018 10:04:30 -0400 Subject: [Python-ideas] Return for assignment blocks In-Reply-To: <1580a88e-e654-50e6-ea51-378c30f6b245@gmail.com> References: <1580a88e-e654-50e6-ea51-378c30f6b245@gmail.com> Message-ID: My idea is not "assignment blocks" those already exist. `def` and `class` blocks are both syntaxes that assign to some name. I'm just using the term to refer to them as a group. The proposal is just being able to return them. These two examples become equivalent: def ignore_exc(exc_type): return def (func): @wraps(func) return def (*args, **kwargs): try: return func(*args, **kwargs) except exc_type: pass def ignore_exc(exc_type): def decorator(func): @wraps(func) def wrapped_func(*args, **kwargs): try: return func(*args, **kwargs) except exc_type: pass return wrapped_func return decorator On Wed, Oct 24, 2018 at 9:51 AM Benedikt Werner <1benediktwerner at gmail.com> wrote: > Would you mind providing a bit more details about your proposal? > > What exactly are those "Assignment Blocks" supposed to do? > > If I understand your proposal correctly you want this: > > def my_func(): > return def(): > print("abc") > > to be the same as this: > > def my_func(): > def inner_func(): > print("abc") > return inner_func > > But this is only my assumption as your proposal doesn't give very much > details. > Maybe you should provide a few simple examples and explain what they are > supposed to do or what they should be equivalent to. > > Your example is quite complex and to me it's not clear at all what it is > supposed to mean. > > Also and probably most importantly what is the reason you want this? Are > there any good usecases where this would help? > > If my assumption above is correct this just looks like a bit of syntactic > sugar that IMO isn't really neccessary. It doesn't really improve > readability or save many characters. The existing way to do this is totally > fine. > > Benedikt > > Am 24.10.2018 um 15:18 schrieb Calvin Spealman: > > I'd like to suggest what I think would be a simple addition to `def` and > `class` blocks. I don't know if calling those "Assignment Blocks" is > accurate, but I just mean to refer to block syntaxes that assign to a name. > Anyway, I propose a combined return-def structure, and optionally also > allowing a return-class version. Omitting the name would be allowable, as > well. > > This would only apply to a `def` or `class` statement made as the last > part of the function body, of course. > > def ignore_exc(exc_type): > return def (func): > @wraps(func) > return def (*args, **kwargs): > try: > return func(*args, **kwargs) > except exc_type: > pass > > Thanks for considering and for any comments, thoughts, or feedback on the > idea! > > > _______________________________________________ > Python-ideas mailing listPython-ideas at python.orghttps://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhodri at kynesim.co.uk Wed Oct 24 10:50:54 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Wed, 24 Oct 2018 15:50:54 +0100 Subject: [Python-ideas] Return for assignment blocks In-Reply-To: References: <1580a88e-e654-50e6-ea51-378c30f6b245@gmail.com> Message-ID: <347d3913-c854-a0a3-584b-9214aaa80a54@kynesim.co.uk> On 24/10/2018 15:04, Calvin Spealman wrote: > My idea is not "assignment blocks" those already exist. `def` and `class` > blocks are both syntaxes that assign to some name. I'm just using the term > to refer to them as a group. > > The proposal is just being able to return them. These two examples become > equivalent: > > def ignore_exc(exc_type): > return def (func): > @wraps(func) > return def (*args, **kwargs): > try: > return func(*args, **kwargs) > except exc_type: > pass > > def ignore_exc(exc_type): > def decorator(func): > @wraps(func) > def wrapped_func(*args, **kwargs): > try: > return func(*args, **kwargs) > except exc_type: > pass > return wrapped_func > return decorator Essentially this is a limited multi-line lambda. Either people are going to be surprised that you can only use it in a return statement or you have to open the whole can of worms about multi-line lambdas. Good luck on the latter. -- Rhodri James *-* Kynesim Ltd From guido at python.org Wed Oct 24 12:40:58 2018 From: guido at python.org (Guido van Rossum) Date: Wed, 24 Oct 2018 09:40:58 -0700 Subject: [Python-ideas] Return for assignment blocks In-Reply-To: <347d3913-c854-a0a3-584b-9214aaa80a54@kynesim.co.uk> References: <1580a88e-e654-50e6-ea51-378c30f6b245@gmail.com> <347d3913-c854-a0a3-584b-9214aaa80a54@kynesim.co.uk> Message-ID: On Wed, Oct 24, 2018 at 7:55 AM Rhodri James wrote: > On 24/10/2018 15:04, Calvin Spealman wrote: > > My idea is not "assignment blocks" those already exist. `def` and `class` > > blocks are both syntaxes that assign to some name. I'm just using the > term > > to refer to them as a group. > > > > The proposal is just being able to return them. These two examples become > > equivalent: > > > > def ignore_exc(exc_type): > > return def (func): > > @wraps(func) > > return def (*args, **kwargs): > > try: > > return func(*args, **kwargs) > > except exc_type: > > pass > > > > def ignore_exc(exc_type): > > def decorator(func): > > @wraps(func) > > def wrapped_func(*args, **kwargs): > > try: > > return func(*args, **kwargs) > > except exc_type: > > pass > > return wrapped_func > > return decorator > > Essentially this is a limited multi-line lambda. Either people are > going to be surprised that you can only use it in a return statement or > you have to open the whole can of worms about multi-line lambdas. Good > luck on the latter. > Let's close that can quickly. Syntactically this is much simpler because because there's no trouble with switching between expression-mode and statement-mode. Also note that syntactically it is clearly a special form of `def` statement -- it can even be decorated! So let's review the proposal as a shorthand for defining a function and immediately returning it. It saves one line plus picking a name. I personally don't think that's enough of a benefit to warrant the extra syntactic complexity (even if modest). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko.ristin at gmail.com Wed Oct 24 14:23:12 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Wed, 24 Oct 2018 20:23:12 +0200 Subject: [Python-ideas] Contracts in python -- a report & next steps In-Reply-To: References: Message-ID: Hi Chris, If not, all your proposed benefits can be achieved at the level of a > single project, by just saying "we're going to use THIS contracts > library", unless I'm misunderstanding something here. > I think we are having a big disconnect in the discussion. Please apologize for my vagueness and point me where you would like me to elaborate more. Let me try to have another iteration where I'll try to paraphrase myself and illustrate what I think that you are thinking. I imagine that you conceive contracts purely as an approach to a testing to be applied to a single project. I'm not talking about that. I'm talking about two packages on pypi, both specifying contracts, each developed by a separate team from different organizations. Let's call these packages package_a and package_b, respectively (and team A and B, analogously). Imagine now that the contracts are not standardized. For example, team A uses dpcontracts whereas team B uses icontract. Enter team C working on the package_c. There is a class package_c.SomeClass that needs to inherit both from package_a.some_module.BaseClass and from package_b.another_module.AnotherBaseClass. Imagine the mess that developers in team C have to deal with -- how are contracts supposed to be strengthened and weakened in package_c.SomeClass? A nightmare with different contract libraries. Even if they don't have to work with multiple inheritance, they would need to use different syntaxes for different concrete classes inheriting from the two packages, respectively. And now think of contract groups that can be specified and applied to functions: @require(...) @require(...) @ensure(...) def some_func(...): ... @enforce_contracts_of(some_func) @require(...) def some_other_func(...): ... some_group = [ require(...), ensure(...), ensure(...) ] @enforce_contracts(some_group) @ensure(...) def yet_another_function(...): ... How are these contract groups supposed to play together between package_a and package_b when applied in package_c? (A side remark: the name and implementation of "enforce_contracts_of" and "enforce_contracts" are still in discussion for icontract and have not been implemented yet.) Furthermore, what about toggling contracts? Team C needs to know both how to toggle the contracts of package_a.some_module and also how to toggle the contracts of package_b.another_module. They need to be aware of and probably set two different environment variables with different toggling granularities *etc*. What if the toggling conflicts? Even if they both used, say, icontract. Package_a might require icontract 1.x whereas package_b requires 2.x. This results in a dependency hell. A standardized lib would force both packages to use the same version of contract lib (or not be supported in that given python version at all) -- the compatibility would be dictated by the python version, not by the version of a non-standard contract lib. Let's focus now on the team working on mypy. For example, consider the code: @invariant(lambda self: self.something is not None or self.another is not None) class SomeClass: ... # later in code somewhere def some_func(): s = SomeClass(...) if s.something is None: # mypy knows here that s.another is not None. ... But if there existed multiple non-standard contract libraries, can we really expect that mypy deals with all of them? I couldn't possibly imagine that. If these points are still not clear -- imagine having multiple non-standard, but widely used libraries and approaches for abstract base classes or object-oriented programming. A horrifying thought :) I'd probably even prefer the python world without contracts than with a widespread use of multiple contract libraries. If pythonistas are to use contracts more widely without stomping on each other's feet, there needs to be some form of standardization so that different packages can inter-operate. There needs to be a convention how to toggle the contracts at different levels of granularity (individual contracts, module-level, pypi package-level, only in testing *etc.*). Environment variables? Setting global variables before importing a module? Some mechanism I'm not aware of? Are the contracts on or off by default? There are so many other details that needs to be fleshed out and this needs to be a collective effort if it is to succeed. I don't have the knowledge to figure out all the conceptual details alone and need help from more experienced people. --- Now a completely different point. Whether we write contracts *for* the modules of the standard library is another discussion. It is not directly connected with the standardization of contracts (and also not part of the current thread, IMO). To write contracts or not to write contracts for a standard library is a per-module decision and concerns the authors and users of each module in separation. Some modules might include contracts that are run only in test mode (*e.g., *in case of icontract, that would be the situation when the environment variable ICONTRACT_SLOW is set); other modules might verify only some of them in production; yet other modules might always verify all the contracts. Or standard lib might take the approach that each module has a corresponding environment variable to turn *on *the contracts, while its contracts are disabled by default. In current implementation of icontract, there is a negligible overhead at the moment during the import of a module with disabled contracts (numbers from the previous message; average over 10 runs, in milliseconds): Duration to import the module functions_100_with_no_contract : 795.59 ? 10.47 Duration to import the module functions_100_with_1_disabled_contract : 833.60 ? 32.07 Duration to import the module functions_100_with_5_disabled_contracts : 851.31 ? 66.93 Duration to import the module functions_100_with_10_disabled_contracts : 897.90 ? 143.02 The whole import overhead is attributed to the interpreter having to parse the decorators and the call to the decorator at the import time (when the decorator immediately returns the wrapped function). If this overhead is unacceptable -- then we need to figure out how to circumvent the parsing of contracts in the interpreter directly and ignore the contract decorators which are disabled. Or have a code base which is shipped for production and another one which is shipped for testing (the former would have the contracts automatically stripped out). This overhead is a limitation of *any *decorator -- *e.g.,* if abstract base classes are OK, I don't see why contracts wouldn't be. I could think of many other optimizations in CPython (*e.g., *it could in-line the code to verify the condition in the function body to avoid making two calls, one to the wrapper and another to the wrapped function), but this is really not the topic of the current thread (please feel free to fork to another thread if you'd like to discuss that particular topic). --- My conclusion is at this point is that the best course of action would be that other teams pick up a contract library (now that there is at least icontract library with a full feature set for design-by-contract akin to Eiffel and other languages), and present us how they use contracts and what problems they were facing (*e.g., *were there any problems with the toggling granularity? readability? maintainability?). Then we decide how to proceed. What are the opinions of the other folks that would like to have a standardized contract library? Cheers, Marko -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Oct 24 15:29:54 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 24 Oct 2018 12:29:54 -0700 Subject: [Python-ideas] Contracts in python -- a report & next steps In-Reply-To: References: Message-ID: On Wed, Oct 24, 2018 at 12:57 AM, Chris Angelico wrote: > For the sake of those of us who REALLY don't feel like diving back > into the extensive threads on this subject, can you please summarize > the benefits of having this in the stdlib rather than as a third-party > library? > I'm not (currently) a fan of design by contract, and don't expect I'll end up using it, whether or not we get a nifty standard library for it.... That being said -- this is analogous to why PEP 484 -- Type Hints was adopted, even though the syntax changes in Python had long been adopted to enable it. That is: the IF folks are going to use DbC in Python, the community is MUCH better off if everyone does it in the same way. And the way to make that happen is to put it in the stdlib. However, I'm not sure it's anywhere near time to actually do that -- before we get there, there needs to be a pretty good community of folks using icontract (or maybe something else?) and ideally some interest from a core developer or two. After all, PEP 484 didn't get rollling until Guido got interested in MyPy. But by all means -- start building that community -- maybe some of us skeptics will be drawn in .... > Also - have you benchmarked the performance cost of adding contracts? > Particularly: if you're planning to contractify the stdlib, what is > the impact on startup performance? > Contractifying the stdlib is an entirely independent question -- yes, if we wanted to add contract to the stdlib, we'd need a contract library in the stdlib, but there's no need for the other way around to be true. Is the stdlib being fully typhinted since PEP 484 was added? a quick scan of the PEP didn't seem to mention it at all. Performance (particularly start up time) would absolutely be a consideration if and when we get there, but I don't know that it has any impact to anything anyone might do now. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Oct 24 16:35:12 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 25 Oct 2018 07:35:12 +1100 Subject: [Python-ideas] Return for assignment blocks In-Reply-To: References: Message-ID: <20181024203512.GI3817@ando.pearwood.info> On Wed, Oct 24, 2018 at 09:18:14AM -0400, Calvin Spealman wrote: > I'd like to suggest what I think would be a simple addition to `def` and > `class` blocks. I don't know if calling those "Assignment Blocks" is > accurate, but I just mean to refer to block syntaxes that assign to a name. > Anyway, I propose a combined return-def structure, and optionally also > allowing a return-class version. Omitting the name would be allowable, as > well. > > This would only apply to a `def` or `class` statement made as the last part > of the function body, of course. > > def ignore_exc(exc_type): > return def (func): > @wraps(func) > return def (*args, **kwargs): > try: > return func(*args, **kwargs) > except exc_type: > pass Your example is too complex for me this early in the morning -- I can't tell what it actually *does*, as it is obscured by what looks like a bunch of irrelevent code. I *think* you are proposing the following syntax. Am I right? return def (func): # body of func which is equivalent to: def func: # body of func return func And similar for classes: return class (K): # body of K being equivalent to: class K: # body of K return K Is it intentional that your example function takes no arguments? If the function did, where would the parameter list go in your syntax? Aside from saving one line, what is the purpose of this? -- Steve From greg.ewing at canterbury.ac.nz Wed Oct 24 17:23:16 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 25 Oct 2018 10:23:16 +1300 Subject: [Python-ideas] Return for assignment blocks In-Reply-To: References: <1580a88e-e654-50e6-ea51-378c30f6b245@gmail.com> Message-ID: <5BD0E2C4.7010804@canterbury.ac.nz> Calvin Spealman wrote: > def ignore_exc(exc_type): > return def (func): > ... Something very similar has been suggested before, you might like to review the previous discussion. -- Greg From tripsvt at gmail.com Wed Oct 24 21:15:09 2018 From: tripsvt at gmail.com (Virendra Tripathi) Date: Wed, 24 Oct 2018 18:15:09 -0700 Subject: [Python-ideas] Return for assignment blocks In-Reply-To: <5BD0E2C4.7010804@canterbury.ac.nz> References: <1580a88e-e654-50e6-ea51-378c30f6b245@gmail.com> <5BD0E2C4.7010804@canterbury.ac.nz> Message-ID: In addition to what GVR wrote, a generator function would have to have its own syntax - backward compatibility would be another issue. Virendra Tripathi Santa Clara, CA 415-910-4955 tripsvt at gmail.com On Wed, Oct 24, 2018 at 2:31 PM Greg Ewing wrote: > Calvin Spealman wrote: > > > def ignore_exc(exc_type): > > return def (func): > > ... > > Something very similar has been suggested before, you might > like to review the previous discussion. > > -- > Greg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Wed Oct 24 22:40:35 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 25 Oct 2018 11:40:35 +0900 Subject: [Python-ideas] Contracts in python -- a report & next steps In-Reply-To: References: Message-ID: <23505.11555.41561.739883@turnbull.sk.tsukuba.ac.jp> Chris Barker via Python-ideas writes: > On Wed, Oct 24, 2018 at 12:57 AM, Chris Angelico wrote: > > > For the sake of those of us who REALLY don't feel like diving back > > into the extensive threads on this subject, can you please summarize > > the benefits of having this in the stdlib rather than as a third-party > > library? > That being said -- this is analogous to why PEP 484 -- Type Hints was > adopted, even though the syntax changes in Python had long been adopted to > enable it. This is historically imprecise, though: function annotations were adopted with an eye to some form of optional typing. > After all, PEP 484 didn't get rollling until Guido got interested > in MyPy. I don't know how much effort Guido put into type hinting himself until the run-up to PEP 484. But he has said that type hinting was the intended use case all along, and he wasn't very sympathetic to arguments that people had found other uses that conflicted. That said, stub files could serve that purpose (with the stub overriding inline annotations, and a flag to mypy to not check the inlines). > That is: the IF folks are going to use DbC in Python, the community > is MUCH better off if everyone does it in the same way. I don't think that's true for icontracts (note, I'm being specific to the implementation here!) in the same way as for type hints, though. Type hints are part of the syntax (even though analyzed by a separate program). You would really need pragmas (function by function, maybe even argument by argument) flags to allow use of multiple type-checkers. Python never liked pragmas at all (this could change with Guido's retirement as BDFL, but of course we're all here because to a great degree we like Guido's taste in such things). With decorator-based contracts, however, you have a function wrapped by code generated by a decorator. It's transparent to the compiler. I just don't see Marko's "dependency hell" being that big a problem. Consider receiving a binary module "contraction" using icontracts. Your project uses bfcontracts (b for banana, f for flavor), also implemented as a set of decorators. As far as I can see, to use bf contracts at top level, all you need to do is import icontracts icontracts.verify_contracts = True # Note: no further mention of icontracts below. import bfcontracts import contraction class inherited(iparent): @bfcontracts.require(additional_contract) def recontract(old_arg_1, ...): return contraction.iparent.recontract(old_arg_1, ...) This would be annoying, but not impossible. You can't weaken the contracts for iparent.recontract as far as I can see in a decorator- based contract library. I haven't thought about it, but with a little bit of trickery it might even be possible in the simple case above to do @bfcontracts.require(additional_contract) @bfcontracts.call_with_inherited_contracts(recontract) def recontract(old_arg_1, ...): pass It's true that you can't replace the implementation of recontract() (ie, without calling contractions.iparent.recontract) and preserve the contracts automatically. This means that contract implementations will have to know something about how other implementations (that do inherit contracts on interfaces rather than function/method implementation) store such contracts. This certainly would be worth standardizing with a dunder attribute or something like that. > And the way to make that happen is to put it in the stdlib. True, but > However, I'm not sure it's anywhere near time to actually do that -- before > we get there, there needs to be a pretty good community of folks using > icontract (or maybe something else?) and ideally some interest from a core > developer or two. +1 > Is the stdlib being fully typhinted since PEP 484 was added? a quick scan > of the PEP didn't seem to mention it at all. There was discussion about stub hint files for the stdlib, but I believe it's being implemented outside of the stdlib. Sorry, don't have time to check. -- Associate Professor Division of Policy and Planning Science http://turnbull.sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull at sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN From ronmarti18 at gmail.com Wed Oct 24 23:47:18 2018 From: ronmarti18 at gmail.com (Ronie Martinez) Date: Thu, 25 Oct 2018 11:47:18 +0800 Subject: [Python-ideas] Set starting point for itertools.product() Message-ID: Hi, My idea is to set the starting point for itertools.product() since it becomes very slow if the point of interest is in the middle. For example when working with datetime tuples with seconds resolution (worst case, milli/microseconds), you need to skip a lot of items. There are several methods I have tried as an alternative but itertools.product() is the fastest: - datetime and timedelta - naive looping This is where I applied it but the issue at seconds resolution is still a bottleneck: https://github.com/Code-ReaQtor/DocCron/blob/1.0.0/doccron/job.py#L171 Setting the starting point might come in handy if we want to skip several data. What do you think? Best Regards, Ronie -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Oct 25 00:25:27 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 25 Oct 2018 17:25:27 +1300 Subject: [Python-ideas] Return for assignment blocks In-Reply-To: <20181024203512.GI3817@ando.pearwood.info> References: <20181024203512.GI3817@ando.pearwood.info> Message-ID: <5BD145B7.5060402@canterbury.ac.nz> The idea is okay, but I'm not keen about the syntax, because it doesn't read well as English. Currently "def" can be read as the verb "define", but "return define" doesn't make sense. Maybe it would be better as a pseudo-decorator: @return def f(args): ... But I'm inclined to agree with Guido that this isn't a frequent enough thing to warrant special syntax. -- Greg From njs at pobox.com Thu Oct 25 01:12:48 2018 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 24 Oct 2018 22:12:48 -0700 Subject: [Python-ideas] Contracts in python -- a report & next steps In-Reply-To: References: Message-ID: On Wed, Oct 24, 2018 at 11:23 AM, Marko Ristin-Kaufmann wrote: > I imagine that you conceive contracts purely as an approach to a testing to > be applied to a single project. I'm not talking about that. I'm talking > about two packages on pypi, both specifying contracts, each developed by a > separate team from different organizations. Let's call these packages > package_a and package_b, respectively (and team A and B, analogously). > > Imagine now that the contracts are not standardized. For example, team A > uses dpcontracts whereas team B uses icontract. Enter team C working on the > package_c. I appreciate your enthusiasm making Python better, and concern for the ecosystem as a whole! But, right now, this particular problem is mostly a theoretical issue, right? Currently 0% of the packages on PyPI use any contracts library (within rounding error), so it'd be very unlikely to have conflicts like this. So in the future, a few different things could happen: Maybe you fail to convince users that contracts are useful, so the usage remains rare. In that case, the problem never happens. Maybe icontracts is super successful, everyone adopts it, and it becomes a de facto standard. Great! Now your problem is solved, because everyone is using icontract. Maybe two different contracts libraries become successful, and people start hitting this problem for real. Something has to be done! But... what? The Python core devs aren't going to know which library is better, or where the pain points are, or how to best mitigate them. The people best placed to figure that out are the developers of the libraries, and their users. And even if we move one of them into the stdlib, that won't magically force people to adopt it. The stdlib has urllib, but most people use requests; the stdlib has ssl, but lots of projects use pyopenssl... In your position, I wouldn't be talking to the core devs; I'd be writing blog posts to proselytize the advantages of contracts, working with popular projects that are interested in adopting them, writing case studies, going to meetups, submitting talks to pycons, that sort of thing. If you want contracts to become a widely-used thing, you have to convince people to use them. The core team doesn't have a magic wand that can short-circuit that process. > But if there existed multiple non-standard contract libraries, can we really > expect that mypy deals with all of them? I couldn't possibly imagine that. mypy currently ships with special support for both dataclasses and attrs, which are basically two implementations of the same idea. If there were multiple popular contract libraries, and there was some benefit to mypy supporting popular contract libraries, then I assume it would support all of them too. Why not? -n -- Nathaniel J. Smith -- https://vorpus.org From steve at pearwood.info Thu Oct 25 01:49:30 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 25 Oct 2018 16:49:30 +1100 Subject: [Python-ideas] Set starting point for itertools.product() In-Reply-To: References: Message-ID: <20181025054929.GJ3817@ando.pearwood.info> On Thu, Oct 25, 2018 at 11:47:18AM +0800, Ronie Martinez wrote: > Hi, > > My idea is to set the starting point for itertools.product() > > since it becomes very slow if the point of interest is in the middle. For > example when working with datetime tuples with seconds resolution (worst > case, milli/microseconds), you need to skip a lot of items. I don't understand what you mean by "skip a lot of items" or why this applies to datetime tuples. Can you give a SHORT and SIMPLE example, showing both the existing solution and your proposed solution? -- Steve From ronmarti18 at gmail.com Thu Oct 25 02:31:05 2018 From: ronmarti18 at gmail.com (Ronie Martinez) Date: Thu, 25 Oct 2018 14:31:05 +0800 Subject: [Python-ideas] Set starting point for itertools.product() In-Reply-To: <20181025054929.GJ3817@ando.pearwood.info> References: <20181025054929.GJ3817@ando.pearwood.info> Message-ID: Hi Steve, Here is an example: import itertools import time def main(): datetime_odometer = itertools.product( range(2018, 10_000), # year range(1, 13), # month range(1, 31), # days range(0, 24), # hours range(0, 60), # minutes range(0, 60) # seconds ) datetime_of_interest = (2050, 6, 15, 10, 5, 0) for i in datetime_odometer: if i == datetime_of_interest: # target start time break if __name__ == '__main__': start = time.time() main() duration = time.time() - start print(duration, 'seconds') # 91.9426908493042 seconds It took 92 seconds to get to the target start time. It does not only apply to datetimes but for other purposes that uses "odometer-like" patterns. I don't have any propose solution for now, but I guess adding this feature within itertools will come in handy. Regards, Ronie On Thu, Oct 25, 2018 at 1:49 PM Steven D'Aprano wrote: > On Thu, Oct 25, 2018 at 11:47:18AM +0800, Ronie Martinez wrote: > > Hi, > > > > My idea is to set the starting point for itertools.product() > > > > since it becomes very slow if the point of interest is in the middle. For > > example when working with datetime tuples with seconds resolution (worst > > case, milli/microseconds), you need to skip a lot of items. > > I don't understand what you mean by "skip a lot of items" or why this > applies to datetime tuples. > > Can you give a SHORT and SIMPLE example, showing both the existing > solution and your proposed solution? > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Thu Oct 25 02:59:21 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 25 Oct 2018 15:59:21 +0900 Subject: [Python-ideas] Set starting point for itertools.product() In-Reply-To: References: <20181025054929.GJ3817@ando.pearwood.info> Message-ID: <23505.27081.682975.230067@turnbull.sk.tsukuba.ac.jp> Ronie Martinez writes: > def main(): > datetime_odometer = itertools.product( > range(2018, 10_000), # year > range(1, 13), # month > range(1, 31), # days > range(0, 24), # hours > range(0, 60), # minutes > range(0, 60) # seconds > ) > > datetime_of_interest = (2050, 6, 15, 10, 5, 0) > > for i in datetime_odometer: > if i == datetime_of_interest: # target start time > break I don't understand the issue. Doesn't def make_odometer(year, month, day, hour, minute, second): return itertools.product( range(year, 10_000), range(month, 13), range(day, 31), range(hour, 24), range(minute, 60), range(second, 61) # leap seconds! ) def main(): datetime_of_interest = (2050, 6, 15, 10, 5, 0) datetime_odometer = make_odometer(*datetime_of_interest) do what you want? If you have a task where that *doesn't* do what's needed, eg, where the "odometer wheels" are an iterated function system, I don't see any way to avoid the problem. If you have some range-like object that doesn't support starting values, you need to fix that or there's nothing that could be done about it in itertools. (Yet Another) Steve -- Associate Professor Division of Policy and Planning Science http://turnbull.sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull at sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN From ronmarti18 at gmail.com Thu Oct 25 03:07:15 2018 From: ronmarti18 at gmail.com (Ronie Martinez) Date: Thu, 25 Oct 2018 15:07:15 +0800 Subject: [Python-ideas] Set starting point for itertools.product() In-Reply-To: <23505.27081.682975.230067@turnbull.sk.tsukuba.ac.jp> References: <20181025054929.GJ3817@ando.pearwood.info> <23505.27081.682975.230067@turnbull.sk.tsukuba.ac.jp> Message-ID: Hi another Steve, Your code will not work! You are skipping the other values below it. When, let's say, the minute value reaches 59, the next iterator should return 0. In your code, it will not. It will start with 5. That is not how an odometer wheel works. Best Regards, Ronie On Thu, Oct 25, 2018 at 2:59 PM Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > Ronie Martinez writes: > > > def main(): > > datetime_odometer = itertools.product( > > range(2018, 10_000), # year > > range(1, 13), # month > > range(1, 31), # days > > range(0, 24), # hours > > range(0, 60), # minutes > > range(0, 60) # seconds > > ) > > > > datetime_of_interest = (2050, 6, 15, 10, 5, 0) > > > > for i in datetime_odometer: > > if i == datetime_of_interest: # target start time > > break > > I don't understand the issue. Doesn't > > def make_odometer(year, month, day, hour, minute, second): > return itertools.product( > range(year, 10_000), > range(month, 13), > range(day, 31), > range(hour, 24), > range(minute, 60), > range(second, 61) # leap seconds! > ) > > def main(): > datetime_of_interest = (2050, 6, 15, 10, 5, 0) > > datetime_odometer = make_odometer(*datetime_of_interest) > > do what you want? If you have a task where that *doesn't* do what's > needed, eg, where the "odometer wheels" are an iterated function > system, I don't see any way to avoid the problem. If you have some > range-like object that doesn't support starting values, you need to > fix that or there's nothing that could be done about it in itertools. > > (Yet Another) Steve > > -- > Associate Professor Division of Policy and Planning Science > http://turnbull.sk.tsukuba.ac.jp/ Faculty of Systems and Information > Email: turnbull at sk.tsukuba.ac.jp University of Tsukuba > Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sf at fermigier.com Thu Oct 25 03:30:38 2018 From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=) Date: Thu, 25 Oct 2018 09:30:38 +0200 Subject: [Python-ideas] Contracts in python -- a report & next steps In-Reply-To: References: Message-ID: A few thoughts: 1) Just as with with PEP 484, we could distinguish between contract specification and checking. To achieve some of the goals that you state, we need to standardise the way people would state contracts in their code (and provide a small implementation in the stdlib, similar to the typing module), but how these contracts are verified (at runtime or statically) or leveraged in the documentation are left to third party projects (similar to mypy & al). 2) Building momentum is paramount. First, since there are many contracts libraries out there, some dating more than 15 years (ex: PyDBC), we'd need to build consensus between the people who wrote and use these libraries. And of course, get feedback from people who use this approach on significant projects. I'm willing to try your library on one of my projects. 3) Having contracts for most of the stdlib would be a worthy goal (for contracts users) but, as noted by some, would come with some speed issues. And of course, would represent a multi-year effort. Hopefully your project (i.e. standardising a library, but not "contracting" the stdlib) would still be valuable if this never happen. 4) Interaction between contracts and typing is an open question. S. On Wed, Oct 24, 2018 at 9:40 AM Marko Ristin-Kaufmann < marko.ristin at gmail.com> wrote: > Hi, > I would like to give you a short report on the development of icontract > library following the discussion about the introduction of contracts into > Python (e.g., see [1, 2, 3, 4]). > > *Features* > The functionality of icontract library is basically on par with Eiffel, > Cobra and other languages supporting contracts except for loop invariants > and less readable syntax. > > The features include: > * "require", "ensure" and "invariant" decorators to define the contracts > * "snapshot" decorator to capture the values prior to the function > invocation to allow postconditions to verify the state transitions > * inheritance of the contracts (with strengthening/weakening) > * tracing of the argument values on contract violation > * informative violation messages automatically parsed from the code of the > condition function > * The user can specify custom errors to be raised on a specific contract > violation > * individual toggling of the contracts > * linter to check the contract arguments > * sphinx extension to render the contracts automatically in the > documentation (including a sophisticated matching of logical implications) > > We covered all the use cases we could find in our code base (at Parquery > AG) such as: > * functions > * instance methods > * class and static methods > * property getters, setters and deleters > * slot wrappers > > *Roadblocks* > During the development, the following roadblocks were encountered: > > * We wanted to include the contracts in the output of help(). > Unfortunately, help() renders the __doc__ of the class and not of the > instance. For functions, this is the class "function" which you can not > inherit from. See [5] for more details. > > * We need to inspect the source code of the condition and error lambdas to > generate the violation message and infer the error type in the > documentation, respectively. inspect.getsource(.) is broken on lambdas > defined in decorators in Python 3.5.2+ (see [6]). We circumvented this bug > by using inspect.findsource(.), inspect.getsourcefile(.) and examining the > local source code of the lambda by searching for other decorators above and > other decorators and a function or class definition below. The decorator > code is parsed and then we match the condition and error arguments in the > AST of the decorator. This is brittle as it prevents us from having partial > definitions of contract functions or from sharing the contracts among > functions. > > Here is a short code snippet to demonstrate where the current > implementation fails: > import icontract > > require_x_positive = icontract.require( > lambda x: x > 0, error=lambda: ValueError("x must be positive")) > > @require_x_positive > def some_func(x: int) -> None: > pass > > However, we haven't faced a situation in the code base where we would do > something like the above, so I am unsure whether this is a big issue. As > long as decorators are directly applied to functions and classes, > everything worked fine on our code base. > > > *Our Experience* > We have been using icontract in our team for our production code base > (~100K LOC) as well as for a couple of open source projects (each <10K LOC) > since August 1st 2018 (~ 3 months). > > In contrast to points raised during the discussions in [1, 2, 3, 4], we > did not have issues with readability and modifying contracts. Thus far, we > have not encountered problems with variable renames and major code > refactorings. We use Pycharm, so it remains open whether this also applies > to development environments such as emacs and vim. > > We confirm that contracts help us improve the documentation, keep the > documentation up-to-date, better reason about the function's input and > output and catch bugs early and more easily during the unit and integration > tests. > > > *Status* > The library interface is currently frozen after the version bump to 2.0.0 > and is not expected to change in the next six months. All the reported bugs > have been fixed and no bug fixes are pending. > > *Next Steps?* > I personally doubt that we are enough people to form a party to push for a > change in the language. A standardized library seems to me like a > realizable compromise given the state of the discussion on this mail list. > > Before we organize a collective to write a proposal to standardize the > library, I would suggest that a couple of other interested teams adopt > icontract, apply it to their code bases and report their experiences on > this mail list. I hope that 2-3 reports would be insightful enough to > either convince other people that contracts in python are worth > standardizing (and highlight missing features yet to be implemented) or > provide solid material to discard the endeavor at the current moment. > > In the meanwhile, it would be of great help if somebody could vet the > documentation and the code of icontract. > > *Authors* > The library was mainly written by me (with some help of my colleague Adam > Radomski). We discussed the features within the dev team at Parquery > (Zurich, Switzerland) as well as in email correspondence with James Lu. > > [1] https://groups.google.com/forum/#!topic/python-ideas/mvzLsdwGuww > [2] https://groups.google.com/forum/#!topic/python-ideas/JtMgpSyODTU > [3] https://groups.google.com/forum/#!topic/python-ideas/mvzLsdwGuww > [4] > https://groups.google.com/forum/#!topic/python-ideas/dmXz_7LH4GI%5B1-25%5D > [5] https://groups.google.com/forum/#!topic/python-ideas/c9ntrVuh6WE > [6] https://bugs.python.org/issue21217 > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group @ Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfine2358 at gmail.com Thu Oct 25 04:03:42 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Thu, 25 Oct 2018 09:03:42 +0100 Subject: [Python-ideas] Set starting point for itertools.product() In-Reply-To: References: <20181025054929.GJ3817@ando.pearwood.info> <23505.27081.682975.230067@turnbull.sk.tsukuba.ac.jp> Message-ID: I see three issues emerging from Ronie's post. The first issue. Each combinatorial iterator, such as itertools.product, has an internal state. This internal state, roughly, is 'where am I'? For the product of three tuples, the internal state is the value of (i, j, k), the indexes into the tuples. The first issue is: how to set the internal state of a combinatorial iterator? And should this be added to the standard library? The second issue. Again, it's about combinatorial iterators. Here's an example. By using enumerate(combinations('ABCD', 2)) we set up a bijection between range(0, 4 * 3 // 2) and the length two subsets of set('ABCD'). When and how should this bijection be made available. The third issue is solving the specific date and time problem he provides. Because of the complexities of date and time, this is I think best done using the datetime module. For me, something like the below (not tested) is the obvious thing to try. def my_moments(start, step): i = 0 while True: value = start + i * step yield value i = i + 1 Here, start is to be a datetime, and step a timedelta. And instead of yielding the value as is, convert it if you wish to a tuple of (year, month, day, hour, minute, second). Ronie. Does this solve your problem? And if not, why not? -- Jonathan From storchaka at gmail.com Thu Oct 25 06:22:00 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 25 Oct 2018 13:22:00 +0300 Subject: [Python-ideas] Set starting point for itertools.product() In-Reply-To: References: <20181025054929.GJ3817@ando.pearwood.info> Message-ID: 25.10.18 09:31, Ronie Martinez ????: > Here is an example: > > import itertools > import time > > > def main(): > datetime_odometer = itertools.product( > range(2018,10_000),# year > range(1,13),# month > range(1,31),# days > range(0,24),# hours > range(0,60),# minutes > range(0,60)# seconds > ) > > datetime_of_interest = (2050,6,15,10,5,0) > > for iin datetime_odometer: > if i == datetime_of_interest:# target start time > break > > > if __name__ =='__main__': > start = time.time() > main() > duration = time.time() - start > print(duration,'seconds')# 91.9426908493042 seconds > > > It took 92 seconds to get to the target start time. It does not only > apply to datetimes but for other purposes that uses "odometer-like" > patterns. > > I don't have any propose solution for now, but I guess adding this > feature within itertools will come in handy. Thank you for clarification. Now I understand your idea. For datetimes it is better to use the datetime classes: def iterdatetimes(): delta = timedelta(microseconds=1) dt = datetime(2050,6,15,10,5,0) while True: yield dt dt += delta Note that in your example you missed 31th days, but iterate 29th and 30th February. See also the calendar module which provides date range iterators (although not with microsecond precision). Currently for general "odometer-like" patterns you can use the undocumented __setstate__ method of itertools.product. But this is on your own risk. From steve at pearwood.info Thu Oct 25 07:31:00 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 25 Oct 2018 22:31:00 +1100 Subject: [Python-ideas] Set starting point for itertools.product() In-Reply-To: References: <20181025054929.GJ3817@ando.pearwood.info> Message-ID: <20181025113058.GL3817@ando.pearwood.info> On Thu, Oct 25, 2018 at 02:31:05PM +0800, Ronie Martinez wrote: > def main(): > datetime_odometer = itertools.product( > range(2018, 10_000), # year > range(1, 13), # month > range(1, 31), # days > range(0, 24), # hours > range(0, 60), # minutes > range(0, 60) # seconds > ) When you talked about datetime, I thought you meant actual datetime objects. The above is buggy: it ignores the 31st day of January, etc, but includes February 29 and 30 every year. > datetime_of_interest = (2050, 6, 15, 10, 5, 0) > > for i in datetime_odometer: > if i == datetime_of_interest: # target start time > break In the most general case, there is no way to jump into the middle of an arbitrary iterator, except to start at the beginning and compute the values until you see the one that you want. Arbitrary iterators compute their values on request, and there is no way to jump ahead except by inspecting each value in turn, skipping the ones you don't want. So unless I have missed something, I think what you are asking for is impossible except for special cases like lists. But in *this* case, modelling an odometer, that special case works: def rotate(iterable, position): L = list(iterable) return L[position:] + L[:position] Which gives us this: py> months = rotate(range(1, 13), 5) py> months [6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5] Now pass that to itertools.product. In other words, I think that the right solution here is to construct your iterables to start at the position you want, rather than expect product() to jump into the middle of the sequence. (Which may not be possible.) -- Steve From rhodri at kynesim.co.uk Thu Oct 25 08:31:26 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Thu, 25 Oct 2018 13:31:26 +0100 Subject: [Python-ideas] Return for assignment blocks In-Reply-To: References: <1580a88e-e654-50e6-ea51-378c30f6b245@gmail.com> <5BD0E2C4.7010804@canterbury.ac.nz> Message-ID: On 25/10/2018 02:15, Virendra Tripathi wrote: > In addition to what GVR wrote, a generator function would have to have its > own syntax - backward compatibility would be another issue. No it wouldn't. This is about returning the function/generator/class itself, not its results. I still don't think it's worth it, mind you. > On Wed, Oct 24, 2018 at 2:31 PM Greg Ewing > wrote: > >> Calvin Spealman wrote: >> >>> def ignore_exc(exc_type): >>> return def (func): >> > ... >> >> Something very similar has been suggested before, you might >> like to review the previous discussion. -- Rhodri James *-* Kynesim Ltd From cspealma at redhat.com Thu Oct 25 08:44:44 2018 From: cspealma at redhat.com (Calvin Spealman) Date: Thu, 25 Oct 2018 08:44:44 -0400 Subject: [Python-ideas] Return for assignment blocks In-Reply-To: <20181024203512.GI3817@ando.pearwood.info> References: <20181024203512.GI3817@ando.pearwood.info> Message-ID: On Wed, Oct 24, 2018 at 4:41 PM Steven D'Aprano wrote: > On Wed, Oct 24, 2018 at 09:18:14AM -0400, Calvin Spealman wrote: > > I'd like to suggest what I think would be a simple addition to `def` and > > `class` blocks. I don't know if calling those "Assignment Blocks" is > > accurate, but I just mean to refer to block syntaxes that assign to a > name. > > Anyway, I propose a combined return-def structure, and optionally also > > allowing a return-class version. Omitting the name would be allowable, as > > well. > > > > This would only apply to a `def` or `class` statement made as the last > part > > of the function body, of course. > > > > def ignore_exc(exc_type): > > return def (func): > > @wraps(func) > > return def (*args, **kwargs): > > try: > > return func(*args, **kwargs) > > except exc_type: > > pass > > Your example is too complex for me this early in the morning -- I can't > tell what it actually *does*, as it is obscured by what looks like a > bunch of irrelevent code. > > I *think* you are proposing the following syntax. Am I right? > > > return def (func): > # body of func > > which is equivalent to: > > def func: > # body of func > return func > > And similar for classes: > > return class (K): > # body of K > > being equivalent to: > > class K: > # body of K > return K > > > Is it intentional that your example function takes no arguments? If the > function did, where would the parameter list go in your syntax? > > Aside from saving one line, what is the purpose of this? > The point is not saving a line or typing, but saving a thought. Expressing the intent of the factory function more clearly. Decorators don't do more than "saving one line", either. But the biggest reason I'd like something like this is that it solves a *specific* version of the multi-line anonymous function that comes up over and over and over again, and maybe by chipping away at those use-cases we can stop seeing *that* debate over and over and over again. :-) So, no, it doesn't save a lot of typing, but I'm never interested in that. I don't spend a lot of time typing code, I spend it reading code, and something like this would certainly help there, imho. > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From boxed at killingar.net Thu Oct 25 09:28:05 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Thu, 25 Oct 2018 15:28:05 +0200 Subject: [Python-ideas] Return for assignment blocks In-Reply-To: References: <20181024203512.GI3817@ando.pearwood.info> Message-ID: > The point is not saving a line or typing, but saving a thought. Expressing the intent of the factory function more clearly. Could you give some other usage examples? To me it seems like a nicer and less confusing way to create decorators is warranted but could then be more specialized and this much nicer. We wouldn't necessarily get to that future point by adding super tiny improvements to what we have, but instead take a step back and rethink what the syntax could be. Personally I often just copy paste an existing decorator and then tweak it instead of writing from scratch because it's too confusing and error prone to do that. / Anders From cspealma at redhat.com Thu Oct 25 10:39:49 2018 From: cspealma at redhat.com (Calvin Spealman) Date: Thu, 25 Oct 2018 10:39:49 -0400 Subject: [Python-ideas] Return for assignment blocks In-Reply-To: References: <20181024203512.GI3817@ando.pearwood.info> Message-ID: On Thu, Oct 25, 2018 at 9:28 AM Anders Hovm?ller wrote: > > > The point is not saving a line or typing, but saving a thought. > Expressing the intent of the factory function more clearly. > > Could you give some other usage examples? > > To me it seems like a nicer and less confusing way to create decorators is > warranted but could then be more specialized and this much nicer. We > wouldn't necessarily get to that future point by adding super tiny > improvements to what we have, but instead take a step back and rethink what > the syntax could be. Personally I often just copy paste an existing > decorator and then tweak it instead of writing from scratch because it's > too confusing and error prone to do that. > You know what, I *can't* think of other good examples than slightly improved decorators. So, maybe its not that great an idea if it can't be applied to at least a few more use cases. / Anders -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Oct 25 11:25:04 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 25 Oct 2018 08:25:04 -0700 Subject: [Python-ideas] Return for assignment blocks In-Reply-To: References: <20181024203512.GI3817@ando.pearwood.info> Message-ID: On Thu, Oct 25, 2018 at 7:41 AM Calvin Spealman wrote: > > On Thu, Oct 25, 2018 at 9:28 AM Anders Hovm?ller > wrote: > >> [Calvin] >> > The point is not saving a line or typing, but saving a thought. >> Expressing the intent of the factory function more clearly. >> > [...] > You know what, I *can't* think of other good examples than slightly > improved decorators. So, maybe its not that great an idea if it can't be > applied to at least a few more use cases. > I appreciate that you're saving a thought, and I think it isn't a bad idea. It is indeed a very common pattern in decorators. But it would be a bit of a one-trick pony, since there aren't many *other* situations where it's useful. Compared to decorators themselves (which also, as was pointed out, do so much more than "saving a line") the use case is just much narrower. So unless we find more use cases, or until we can convince ourselves that we can use `def (args): block` in all expression contexts, I guess it'll have to remain an idea. Thank you though! It was a fascinating one. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfine2358 at gmail.com Thu Oct 25 12:28:24 2018 From: jfine2358 at gmail.com (Jonathan Fine) Date: Thu, 25 Oct 2018 17:28:24 +0100 Subject: [Python-ideas] Problems (and solutions?) in writing decorators In-Reply-To: References: <20181024203512.GI3817@ando.pearwood.info> Message-ID: Was: Return for assignment blocks Anders Hovm?ller wrote: > Personally I often just copy paste an existing decorator and then tweak it instead of writing from scratch because it's too confusing and error prone to do that. Thank you, Anders, for your honesty. I also find writing decorators a bit hard. It seems to be something I have to learn anew each time I do it. Particularly for the pattern @deco(arg1, arg2) def fn(arg3, arg4): # function body Perhaps doing something with partial might help here. Anyone here interested in exploring this? -- Jonathan From guettliml at thomas-guettler.de Thu Oct 25 16:04:05 2018 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler_Lists?=) Date: Thu, 25 Oct 2018 22:04:05 +0200 Subject: [Python-ideas] name2type mapping Message-ID: <29c1c7e4-16b0-064c-88cc-1532eeab30df@thomas-guettler.de> I created a first draft of the name2type mapping which was discussed here some days ago: https://github.com/guettli/python-name2type-mapping/blob/master/README.rst Feedback is welcome. Regards, ? Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ I am looking for feedback: https://github.com/guettli/programming-guidelines From steve at pearwood.info Thu Oct 25 16:29:55 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 26 Oct 2018 07:29:55 +1100 Subject: [Python-ideas] Return for assignment blocks In-Reply-To: References: <20181024203512.GI3817@ando.pearwood.info> Message-ID: <20181025202955.GM3817@ando.pearwood.info> On Thu, Oct 25, 2018 at 08:44:44AM -0400, Calvin Spealman wrote: > On Wed, Oct 24, 2018 at 4:41 PM Steven D'Aprano wrote: [...] > > I *think* you are proposing the following syntax. Am I right? > > > > > > return def (func): > > # body of func > > > > which is equivalent to: > > > > def func: > > # body of func > > return func [...] > > Aside from saving one line, what is the purpose of this? > > > > The point is not saving a line or typing, but saving a thought. Expressing > the intent of the factory function more clearly. I don't find that it expresses the intent clearly. Perhaps because I have read, and written, many factory functions that post-process the inner function in some fashion, to me creating the inner function is just one step out of possibly many steps, no more special than the others and not deserving of any special syntax. To give a toy example, adding an extra attribute to the function: def factory(argument): def inner(): ... inner.spam = "spam" return inner Another common pattern for me is changing the name of the inner function. Most of the time wraps() does this, but if the factory doesn't wrap another function, I frequently change the inner function name manually: inner.__name__ = some_name() or "default" Admittedly even for me, the pattern of returning the function immediately after it is created (usually after being decorated by functools.wraps) is very *common*, but I don't find it *special* enough to justify dedicated syntax. > Decorators don't do more than "saving one line", either. I didn't say they did. > But the biggest reason I'd like something like this is that it solves a > *specific* version of the multi-line anonymous function that comes up over > and over and over again, and maybe by chipping away at those use-cases we > can stop seeing *that* debate over and over and over again. :-) I don't see the connection. "def" will still be a statement, not an expression, so you can't put it in expressions with or without a leading "return" statement: x = [spam, eggs, return def func(): block, aardvark] nor can you put it in a lambda. So I don't see why this would help with the anonymous code block requests. > So, no, it doesn't save a lot of typing, but I'm never interested in that. > I don't spend a lot of time typing code, I spend it reading code, and > something like this would certainly help there, imho. I don't think it will "certainly" help. I think you're projecting your own opinion onto others. I don't find it very readable, and neither have some others who commented. When I read a function def, I read it as short for "define", an imperative command. "return def(ine)" doesn't read like English to me. I also see return, expect an expression, and instead find a statement, so I find it jarring. If *everything was an expression* in Python, perhaps I would agree with you, but then we wouldn't need special syntax because we could already write "return def ...". But given that we do have the statement vs expression dichotomy in Python, making def a special case but only in return statements causes an exception in my brain when I read it :-) -- Steve From steve at pearwood.info Thu Oct 25 16:34:10 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 26 Oct 2018 07:34:10 +1100 Subject: [Python-ideas] name2type mapping In-Reply-To: <29c1c7e4-16b0-064c-88cc-1532eeab30df@thomas-guettler.de> References: <29c1c7e4-16b0-064c-88cc-1532eeab30df@thomas-guettler.de> Message-ID: <20181025203409.GO3817@ando.pearwood.info> On Thu, Oct 25, 2018 at 10:04:05PM +0200, Thomas G?ttler Lists wrote: > I created a first draft of the name2type mapping which was discussed > here some days ago: Since this is not a language feature, please take the discussion to a more appropriate forum such as this: https://mail.python.org/mailman/listinfo/code-quality -- Steve From greg.ewing at canterbury.ac.nz Thu Oct 25 17:31:19 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 26 Oct 2018 10:31:19 +1300 Subject: [Python-ideas] Return for assignment blocks In-Reply-To: References: <20181024203512.GI3817@ando.pearwood.info> Message-ID: <5BD23627.8060109@canterbury.ac.nz> Calvin Spealman wrote: > You know what, I /can't/ think of other good examples than slightly > improved decorators. So, maybe its not that great an idea if it can't be > applied to at least a few more use cases. The class version might be useful for things like namedtuple that dynamically create classes. But that's even rarer than decorators. -- Greg From marko.ristin at gmail.com Thu Oct 25 17:44:13 2018 From: marko.ristin at gmail.com (Marko Ristin-Kaufmann) Date: Thu, 25 Oct 2018 23:44:13 +0200 Subject: [Python-ideas] Contracts in python -- a report & next steps In-Reply-To: References: Message-ID: Hi, Stephen J. Turnbull wrote: > You can't weaken the > contracts for iparent.recontract as far as I can see in a decorator- > based contract library Of course you can weaken the contracts for iparent.recontract with a decorator-based contract library. From my first message: > The features include: > ... > * inheritance of the contracts (with strengthening/weakening) > ... > Please also have a look at the readme on https://github.com/Parquery/icontract/, Section "Inheritance". Chris Barker wrote: > However, I'm not sure it's anywhere near time to actually do that -- > before we get there, there needs to be a pretty good community of folks > using icontract (or maybe something else?) and ideally some interest from a > core developer or two. I absolutely agree. In my first message I wrote: > Before we organize a collective to write a proposal to standardize the > library, I would suggest that a couple of other interested teams adopt > icontract, apply it to their code bases and report their experiences on > this mail list. I hope that 2-3 reports would be insightful enough to > either convince other people that contracts in python are worth > standardizing (and highlight missing features yet to be implemented) or > provide solid material to discard the endeavor at the current moment. > and in my second message (before yours), I wrote: > My conclusion is at this point is that the best course of action would be > that other teams pick up a contract library (now that there is at least > icontract library with a full feature set for design-by-contract akin to > Eiffel and other languages), and present us how they use contracts and what > problems they were facing (*e.g., *were there any problems with the > toggling granularity? readability? maintainability?). Then we decide how to > proceed. > Stefane Fermigier wrote: > 1) Just as with with PEP 484, we could distinguish between contract > specification and checking. To achieve some of the goals that you state, we > need to standardise the way people would state contracts in their code (and > provide a small implementation in the stdlib, similar to the typing > module), but how these contracts are verified (at runtime or statically) or > leveraged in the documentation are left to third party projects (similar to > mypy & al). > This is a great idea -- though I'm not sure how it could be implemented other than already specifying the implementation (i.e. how the contracts are verified). If you have some time, could you elaborate a bit how this might work? I'm suffering from tunnel vision at this point after implementing the library. > 2) Building momentum is paramount. First, since there are many contracts > libraries out there, some dating more than 15 years (ex: PyDBC), we'd need > to build consensus between the people who wrote and use these libraries. > And of course, get feedback from people who use this approach on > significant projects. I'm willing to try your library on one of my projects. > Thanks, I'm looking forward to hearing your experience! I contacted the authors of dpcontracts, but there is still no clear plan of action how to merge icontract and dpcontracts (see https://github.com/deadpixi/contracts/issues/15). Other libraries seem either unmaintained or not applicable except for single-argument checks and involve custom syntax (like https://pypi.org/project/PyContracts/; I only played a bit with PyContracts, so it would be great if anybody could tell their experience with it in a larger code base). We found that all the present libraries were far from usable for a larger code base (so we rolled out our own). To the best of our knowledge, none could deal with inheritance properly (weakening/strengthening) and you always had to write out the condition along the lambda function (unless the code was parsed from the documentation which increases the import time significantly even if you want to disable some of the contracts). As others already mentioned in the previous discussion, part of the reason for not adopting DbC was the lack of tools. I'm not saying that icontract should be that standard tool -- not at all! We need to first find out what such standard tool would need to fulfill*. *Assuming that we don't make language changes, what would we like the standard contract library to look like? What features should it have? What is too much of a niche to be left out? > 3) Having contracts for most of the stdlib would be a worthy goal (for > contracts users) but, as noted by some, would come with some speed issues. > And of course, would represent a multi-year effort. Hopefully your project > (i.e. standardising a library, but not "contracting" the stdlib) would > still be valuable if this never happen. > I agree. Nathaniel Smith wrote: > In your position, I wouldn't be talking to the core devs; I'd be > writing blog posts to proselytize the advantages of contracts, working > with popular projects that are interested in adopting them, writing > case studies, going to meetups, submitting talks to pycons, that sort > of thing. If you want contracts to become a widely-used thing, you > have to convince people to use them. The core team doesn't have a > magic wand that can short-circuit that process. > I thought python-ideas is an open list to generally discuss ideas, not the core dev list (isn't that python-dev)? That's why I wrote here (after being forwarded from python-dev list). I agree that these efforts you mention would be worthwhile. Implementation of a (prototype) library and participating in the discussions on this mail list are the maximum effort I can put up at the moment. Maybe in 2019 I could manage to go to the local Python summit. Cheers, Marko -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Oct 25 18:04:17 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 26 Oct 2018 11:04:17 +1300 Subject: [Python-ideas] Problems (and solutions?) in writing decorators In-Reply-To: References: <20181024203512.GI3817@ando.pearwood.info> Message-ID: <5BD23DE1.1000909@canterbury.ac.nz> Jonathan Fine wrote: > I also find writing decorators a bit > hard. It seems to be something I have to learn anew each time I do it. > Particularly for the pattern > > @deco(arg1, arg2) def fn(arg3, arg4): > # function body > > Perhaps doing something with partial might help here. Anyone here interested > in exploring this? > I can't think of a way that partial would help. But would you find it easier if you could do something like this? class deco(Decorator): def __init__(self, arg1, arg2): self.arg1 = arg1 self.arg2 = arg2 def invoke(self, func, arg3, arg4): # function body Implementation: class Decorator: def __call__(self, func): self._wrapped_function = func return self._wrapper def _wrapper(self, *args, **kwds): return self.invoke(self._wrapped_function, *args, **kwds) -- Greg From guido at python.org Thu Oct 25 19:19:43 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 25 Oct 2018 16:19:43 -0700 Subject: [Python-ideas] name2type mapping In-Reply-To: <20181025203409.GO3817@ando.pearwood.info> References: <29c1c7e4-16b0-064c-88cc-1532eeab30df@thomas-guettler.de> <20181025203409.GO3817@ando.pearwood.info> Message-ID: On Thu, Oct 25, 2018 at 1:35 PM Steven D'Aprano wrote: > On Thu, Oct 25, 2018 at 10:04:05PM +0200, Thomas G?ttler Lists wrote: > > I created a first draft of the name2type mapping which was discussed > > here some days ago: > > Since this is not a language feature, please take the discussion to a > more appropriate forum such as this: > > https://mail.python.org/mailman/listinfo/code-quality > Honestly I think you're a little quick to request this discussion be taken away from python-ideas. Since the feature is intended to be closely integrated with static type checking (PEPs 484, 526, 544, 561) it may be PEP-worthy. As a compromise, perhaps some discussion can take place in the issue tracker of the repo (https://github.com/guettli/python-name2type-mapping) Thomas created? If someone with PEP experience is interested they can guide Thomas into drafting a PEP. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Oct 25 19:54:06 2018 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 25 Oct 2018 16:54:06 -0700 Subject: [Python-ideas] Contracts in python -- a report & next steps In-Reply-To: References: Message-ID: On Thu, Oct 25, 2018, 14:44 Marko Ristin-Kaufmann wrote: > > Nathaniel Smith wrote: > >> In your position, I wouldn't be talking to the core devs; I'd be >> writing blog posts to proselytize the advantages of contracts, working >> with popular projects that are interested in adopting them, writing >> case studies, going to meetups, submitting talks to pycons, that sort >> of thing. If you want contracts to become a widely-used thing, you >> have to convince people to use them. The core team doesn't have a >> magic wand that can short-circuit that process. >> > > I thought python-ideas is an open list to generally discuss ideas, not the > core dev list (isn't that python-dev)? That's why I wrote here (after being > forwarded from python-dev list). > > I agree that these efforts you mention would be worthwhile. Implementation > of a (prototype) library and participating in the discussions on this mail > list are the maximum effort I can put up at the moment. Maybe in 2019 I > could manage to go to the local Python summit. > Python-ideas is for ideas for changing the python language or standard library. And the subthread I was replying to was about adding contracts support to the stdlib, so you're in the right place for that discussion. I was trying to explain why trying to add contract support to the stdlib doesn't make much sense right now. And if you switch your focus to trying to recruit users and collaborators, like I recommend, then python-ideas isn't the best place to do that. Most of your potential users aren't here! -n > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronmarti18 at gmail.com Thu Oct 25 22:20:35 2018 From: ronmarti18 at gmail.com (Ronie Martinez) Date: Fri, 26 Oct 2018 10:20:35 +0800 Subject: [Python-ideas] Set starting point for itertools.product() In-Reply-To: <20181025113058.GL3817@ando.pearwood.info> References: <20181025054929.GJ3817@ando.pearwood.info> <20181025113058.GL3817@ando.pearwood.info> Message-ID: Hi Steve, I tried this when writing DocCron but it will not work correctly. Suppose we have minutes with range 0-59 and seconds with range 0-59 but starting at, say, 5: When the seconds value reaches 59, on the next iteration the minute value should increase by 1 (or back to zero) but it will not work that way since the next rotation will be at 4 transitioning to 5. So the correct solution is not modifying any of the arrangements. Best Regards, Ronie On Thu, Oct 25, 2018 at 7:31 PM Steven D'Aprano wrote: > On Thu, Oct 25, 2018 at 02:31:05PM +0800, Ronie Martinez wrote: > > > def main(): > > datetime_odometer = itertools.product( > > range(2018, 10_000), # year > > range(1, 13), # month > > range(1, 31), # days > > range(0, 24), # hours > > range(0, 60), # minutes > > range(0, 60) # seconds > > ) > > When you talked about datetime, I thought you meant actual datetime > objects. The above is buggy: it ignores the 31st day of January, etc, > but includes February 29 and 30 every year. > > > > datetime_of_interest = (2050, 6, 15, 10, 5, 0) > > > > for i in datetime_odometer: > > if i == datetime_of_interest: # target start time > > break > > In the most general case, there is no way to jump into the middle of an > arbitrary iterator, except to start at the beginning and compute the > values until you see the one that you want. Arbitrary iterators compute > their values on request, and there is no way to jump ahead except by > inspecting each value in turn, skipping the ones you don't want. > > So unless I have missed something, I think what you are asking for is > impossible except for special cases like lists. > > But in *this* case, modelling an odometer, that special case works: > > def rotate(iterable, position): > L = list(iterable) > return L[position:] + L[:position] > > Which gives us this: > > py> months = rotate(range(1, 13), 5) > py> months > [6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5] > > Now pass that to itertools.product. > > In other words, I think that the right solution here is to construct > your iterables to start at the position you want, rather than expect > product() to jump into the middle of the sequence. (Which may not be > possible.) > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronmarti18 at gmail.com Thu Oct 25 22:31:54 2018 From: ronmarti18 at gmail.com (Ronie Martinez) Date: Fri, 26 Oct 2018 10:31:54 +0800 Subject: [Python-ideas] Set starting point for itertools.product() In-Reply-To: References: <20181025054929.GJ3817@ando.pearwood.info> Message-ID: Hi Serhiy, I missed the 31st on days range. Thanks for spotting it. I do verify if the datetime tuple is correct by passing them to datetime() and raising exception later on on the code (see https://github.com/Code-ReaQtor/DocCron/blob/1.0.0/doccron/job.py#L180) Datetime and timedeltas can solve the problem if there is a "pattern" but this is not an efficient way. For example, if there are steps on all the categories or the items have no pattern: datetime_odometer = itertools.product( range(2018, 10_000, 5), # year with steps range(1, 13, 3), # month with steps range(1, 32, 4), # days with steps [0, 5, 6, 10, 13, 24], # hours without steps range(0, 60, 6), # minutes with steps range(0, 60, 2) # seconds with steps ) Datetime and timedelta will create a lot of overhead and is not the best solution. I still believe itertools.product() is the fastest and best solution. Let me read the code for __setstate__ first. Thanks for spotting this! Best Regards, Ronie Martinez On Thu, Oct 25, 2018 at 6:22 PM Serhiy Storchaka wrote: > 25.10.18 09:31, Ronie Martinez ????: > > Here is an example: > > > > import itertools > > import time > > > > > > def main(): > > datetime_odometer = itertools.product( > > range(2018,10_000),# year > > range(1,13),# month > > range(1,31),# days > > range(0,24),# hours > > range(0,60),# minutes > > range(0,60)# seconds > > ) > > > > datetime_of_interest = (2050,6,15,10,5,0) > > > > for iin datetime_odometer: > > if i == datetime_of_interest:# target start time > > break > > > > > > if __name__ =='__main__': > > start = time.time() > > main() > > duration = time.time() - start > > print(duration,'seconds')# 91.9426908493042 seconds > > > > > > It took 92 seconds to get to the target start time. It does not only > > apply to datetimes but for other purposes that uses "odometer-like" > > patterns. > > > > I don't have any propose solution for now, but I guess adding this > > feature within itertools will come in handy. > > Thank you for clarification. Now I understand your idea. > > For datetimes it is better to use the datetime classes: > > def iterdatetimes(): > delta = timedelta(microseconds=1) > dt = datetime(2050,6,15,10,5,0) > while True: > yield dt > dt += delta > > Note that in your example you missed 31th days, but iterate 29th and > 30th February. > > See also the calendar module which provides date range iterators > (although not with microsecond precision). > > Currently for general "odometer-like" patterns you can use the > undocumented __setstate__ method of itertools.product. But this is on > your own risk. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronmarti18 at gmail.com Fri Oct 26 00:16:55 2018 From: ronmarti18 at gmail.com (Ronie Martinez) Date: Fri, 26 Oct 2018 12:16:55 +0800 Subject: [Python-ideas] Set starting point for itertools.product() In-Reply-To: References: <20181025054929.GJ3817@ando.pearwood.info> Message-ID: Hi Serhiy, __setstate__() made it possible to specify the starting point. Using this method and combining it with binning (if the items does not have any pattern) worked well! Thanks for the suggestion. Cheers! Best Regards, Ronie On Fri, Oct 26, 2018 at 10:31 AM Ronie Martinez wrote: > Hi Serhiy, > > I missed the 31st on days range. Thanks for spotting it. I do verify if > the datetime tuple is correct by passing them to datetime() and raising > exception later on on the code (see > https://github.com/Code-ReaQtor/DocCron/blob/1.0.0/doccron/job.py#L180) > > Datetime and timedeltas can solve the problem if there is a "pattern" but > this is not an efficient way. > > For example, if there are steps on all the categories or the items have no > pattern: > > datetime_odometer = itertools.product( > range(2018, 10_000, 5), # year with steps > range(1, 13, 3), # month with steps > range(1, 32, 4), # days with steps > [0, 5, 6, 10, 13, 24], # hours without steps > range(0, 60, 6), # minutes with steps > range(0, 60, 2) # seconds with steps > ) > > > Datetime and timedelta will create a lot of overhead and is not the best > solution. I still believe itertools.product() is the fastest and best > solution. > > Let me read the code for __setstate__ first. Thanks for spotting this! > > Best Regards, > Ronie Martinez > > > On Thu, Oct 25, 2018 at 6:22 PM Serhiy Storchaka > wrote: > >> 25.10.18 09:31, Ronie Martinez ????: >> > Here is an example: >> > >> > import itertools >> > import time >> > >> > >> > def main(): >> > datetime_odometer = itertools.product( >> > range(2018,10_000),# year >> > range(1,13),# month >> > range(1,31),# days >> > range(0,24),# hours >> > range(0,60),# minutes >> > range(0,60)# seconds >> > ) >> > >> > datetime_of_interest = (2050,6,15,10,5,0) >> > >> > for iin datetime_odometer: >> > if i == datetime_of_interest:# target start time >> > break >> > >> > >> > if __name__ =='__main__': >> > start = time.time() >> > main() >> > duration = time.time() - start >> > print(duration,'seconds')# 91.9426908493042 seconds >> > >> > >> > It took 92 seconds to get to the target start time. It does not only >> > apply to datetimes but for other purposes that uses "odometer-like" >> > patterns. >> > >> > I don't have any propose solution for now, but I guess adding this >> > feature within itertools will come in handy. >> >> Thank you for clarification. Now I understand your idea. >> >> For datetimes it is better to use the datetime classes: >> >> def iterdatetimes(): >> delta = timedelta(microseconds=1) >> dt = datetime(2050,6,15,10,5,0) >> while True: >> yield dt >> dt += delta >> >> Note that in your example you missed 31th days, but iterate 29th and >> 30th February. >> >> See also the calendar module which provides date range iterators >> (although not with microsecond precision). >> >> Currently for general "odometer-like" patterns you can use the >> undocumented __setstate__ method of itertools.product. But this is on >> your own risk. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Fri Oct 26 03:26:00 2018 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Fri, 26 Oct 2018 09:26:00 +0200 Subject: [Python-ideas] name2type mapping In-Reply-To: <20181025203409.GO3817@ando.pearwood.info> References: <29c1c7e4-16b0-064c-88cc-1532eeab30df@thomas-guettler.de> <20181025203409.GO3817@ando.pearwood.info> Message-ID: <9bdb93dd-34c8-a6eb-eced-2f80851824dc@thomas-guettler.de> Am 25.10.18 um 22:34 schrieb Steven D'Aprano: > On Thu, Oct 25, 2018 at 10:04:05PM +0200, Thomas G?ttler Lists wrote: >> I created a first draft of the name2type mapping which was discussed >> here some days ago: > > Since this is not a language feature, please take the discussion to a > more appropriate forum such as this: > > https://mail.python.org/mailman/listinfo/code-quality The discussion was already taken to a an other forum: https://github.com/guettli/python-name2type-mapping I feel where proud and happy since Guido created an issue there. Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ I am looking for feedback: https://github.com/guettli/programming-guidelines From daveshawley at gmail.com Fri Oct 26 08:48:57 2018 From: daveshawley at gmail.com (David Shawley) Date: Fri, 26 Oct 2018 08:48:57 -0400 Subject: [Python-ideas] async unittest.TestCase In-Reply-To: References: Message-ID: On Oct 10, 2018, at 1:34 PM, Serhiy Storchaka wrote: > 10.10.18 20:19, Yury Selivanov ????: > > Thanks for proposing this. Yes, it makes sense to have > > unittest.AsyncTestCase in 3.8. AFAIK Lisa Roach (copied) was working > > on that (as well as on async Mock object), but I'm not sure what's the > > status of her work. I suggest to search for an open issue for this on > > bugs.python.org; if there's none, please create one. And let's make > > this happen. > > https://bugs.python.org/issue32972 > https://bugs.python.org/issue26467 I have a working prototype of asynchronous testing in a branch on github [1] but I took a different approach than what was discussed in bpo-32972. Should I open a new bpo or continue the discussion on the existing bpo? I'm leaning in the direction of opening a new bpo so as to not muddy the waters with the discussion around the other branch. Here's a summary of the approach that I took: - create a new class unittest.AsyncioTestCase that implements a few new methods and attributes: - asyncSetUp / asyncTearDown: are awaitable and call the synchronous setUp and tearDown methods. These only exist on the new sub-class. - _runTest: a new method on unittest.TestCase that encapsulates running the test case. This is simply the portion of unittest.TestCase.run() that runs setUp()/testMethod()/tearDown(). - _terminateTest: a new method on unittest.TestCase that is *always* called from within run() immediately *before* calling stopTest() on the result. The AsyncioTestCase specialization of this method is responsible to cleaning up the event loop in the same manner that asyncio.run() does. In particular, it shuts down asynchronous generators. [a] - event_loop: an overridable property on the new sub-class that creates and installs a new event loop. This is referred to from within the AsyncioTestCase specialization of _runTest to create the event loop - event loops are created and destroyed for each test method [b] - the same event loop persists from before setUp until after the cleanups are called - test methods that asyncio.iscoroutinefunction thinks are co-routines are run on the event loop; other test methods are called directly [c] With that said, should I open a new bpo or start discussion on the existing one anew? Cheers, dave. -- [1]: https://github.com/dave-shawley/cpython/pull/1 [2]: https://github.com/python/cpython/pull/9296 [a]: I explicitly chose to not shutdown outstanding tasks in _terminateTest so that the event loop complains loudly about them. This is an area that I would like to discuss further. [b]: I chose to create and destroy an event loop with each test method to avoid having tasks and other event loop settings from leaking between test calls. [c]: This is problematic since mock.patch decorated co-routines won't be called correctly. I'm going to discuss this with @lisroach on her bpo-26467 patch [2]. I have a feeling that the "AwaitableMock" approach may work out and change this code. See the most recent comments on [2] for details. -- People think right angles produce harmony, but they don't. They produce sleep. -- James Krenov -------------- next part -------------- An HTML attachment was scrubbed... URL: From mathias.laurin at gmail.com Fri Oct 26 11:41:26 2018 From: mathias.laurin at gmail.com (Mathias Laurin) Date: Fri, 26 Oct 2018 17:41:26 +0200 Subject: [Python-ideas] PEP 543-conform TLS library Message-ID: <70EB8AEB-AC3D-455F-BBB3-6153D9CB6D77@gmail.com> Hello Python ideas, I have implemented an (I believe) PEP 543-conform TLS library and released TLS support in the latest version yesterday: https://github.com/Synss/python-mbedtls/tree/0.13.0 https://pypi.org/project/python-mbedtls/0.13.0/ As far as I know, I am the first one to follow PEP 543. So one point is that the API works. However, I have a couple of questions regarding the PEP: - I do not know what to do in `TLSWrappedBuffer.do_handshake()`. The full TLS handshake requires writing to the server, reading back, etc., (ClientHello, ServerHello, KeyExchange, etc.), which cannot be accomplished in a single buffer. For now, I am doing the handshake in `TLSWrappedSocket.do_handshake()`: I set the BIO to using the socket directly, then perform the handshake on the socket thus entirely bypassing the TLSWrappedBuffer. Once this is done, I swap the BIO to using the buffer and go on encrypting and decrypting from the buffer. That is, the encrypted communication is buffered. - The PEP sometimes mentions an "input buffer" and an "output buffer", and some other times just "the buffer". I believe that both implementations are possible. That is, with two different buffers for input and output, or a single one. I have implemented it with a single circular buffer (that is a stream after all). What the PEP is expecting is nonetheless not clear to me. So, can anybody clarify these two points from the PEP? Or should I just address Cory Benfield (who does not seem very active anymore lately) and Christian Heimes directly? Cheers, Mathias From solipsis at pitrou.net Fri Oct 26 12:08:27 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 26 Oct 2018 18:08:27 +0200 Subject: [Python-ideas] PEP 543-conform TLS library References: <70EB8AEB-AC3D-455F-BBB3-6153D9CB6D77@gmail.com> Message-ID: <20181026180827.51e08845@fsol> On Fri, 26 Oct 2018 17:41:26 +0200 Mathias Laurin wrote: > > So, can anybody clarify these two points from the PEP? > > Or should I just address Cory Benfield (who does not seem very > active anymore lately) and Christian Heimes directly? Either that, or post on python-dev where you'll be more likely to be read by the relevant people :-) Regards Antoine. From jpic at yourlabs.org Sat Oct 27 09:03:13 2018 From: jpic at yourlabs.org (Jamesie Pic) Date: Sat, 27 Oct 2018 15:03:13 +0200 Subject: [Python-ideas] [Distutils] Pypi private repo's In-Reply-To: <20181017144424.GQ3817@ando.pearwood.info> References: <3075001d3cc57$370df900$a529eb00$@sdamon.com> <2A893F9F-C4CE-4470-BC1D-F556A7DA5253@me.com> <20181017144424.GQ3817@ando.pearwood.info> Message-ID: Hi Steven, I thought they were looking for a company to help them with this project. I didn't want to take the lead on this but if I am then I'm sorry to say that the user story list for the beta is extremely boring: - user can subscribe for free during beta with an email - user can create a private repository - user can create a token - user can upload packages to the repository with their token - user can delete a package - user can delete a repository - user can delete their account But then if we want to pay for that it has to be much more interesting, you could go completely crazy on ideas: - user can create groups, assign permissions on packages etc - user has a command to upload that supports encryption for their packages to host on the public server, - user has a command to spawn a server for a given repository given a token, then packages could be uploaded/downloaded directly from their own network - that could also run on their private network in which case the public server can serve as proxy, - fun & profit Best regards -------------- next part -------------- An HTML attachment was scrubbed... URL: From python.gem at gmail.com Sat Oct 27 14:02:53 2018 From: python.gem at gmail.com (Joy Diamond) Date: Sat, 27 Oct 2018 14:02:53 -0400 Subject: [Python-ideas] Fix documentation for __instancecheck__ Message-ID: Greetings, This is a request to fix the documentation for __instancecheck__. Please add the following (please rewrite better than I can -- I am not good at explaining concepts in short sentences): NOTE: As an optimization, isinstance(*object*, *classinfo*) does NOT call classinfo.__instancecheck__(instance) when type(object) == classinfo. Consider the following program: class M(type): def __instancecheck__(m, t): print('instancecheck(%s, %s)' % (m, t)) return False # LIE! Test = M('Test', ((object,)), {}) something = Test() print('Does *NOT* call __instancecheck__:') print('isinstance(something, Test): %s' % isinstance(something, Test)) print() print('Does call __instancecheck__:') print('isinstance(0, Test): %s' % isinstance(0, Test)) Under python 2, python 3, and pypy, in all cases, the first examples does *NOT* call __instancecheck__. You can see the optimization here: https://github.com/python/cpython/blob/master/Objects/abstract.c#L2397-L2405 Which reads: int PyObject_IsInstance(PyObject *inst, PyObject *cls) { _Py_IDENTIFIER(__instancecheck__); PyObject *checker; /* Quick test for an exact match */ if (Py_TYPE(inst) == (PyTypeObject *)cls) return 1; I'm fine with the optimization -- I am not suggesting to get rid of it. I just want the documentation to match the actual implementation. The following documentation needs to be fixed: https://docs.python.org/2/reference/datamodel.html https://docs.python.org/3/reference/datamodel.html https://www.python.org/dev/peps/pep-3119/ It took me half an hour to figure out why my version of __instancecheck__ was not working, as I tried to test it using the super simple code above ... One of the best things about python is how accurate and consistent the documentation is. This request is to keep these high standards. Thanks, Joy Diamond. NOTE: I'm not sure where to post this, so posting to python-ideas, in case people want to discuss getting rid of the optimization in PyObject_IsInstance ... which I am not suggesting. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Oct 27 14:24:43 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 28 Oct 2018 05:24:43 +1100 Subject: [Python-ideas] Fix documentation for __instancecheck__ In-Reply-To: References: Message-ID: On Sun, Oct 28, 2018 at 5:03 AM Joy Diamond wrote: > > Greetings, > > This is a request to fix the documentation for __instancecheck__. > > Please add the following (please rewrite better than I can -- I am not good at explaining concepts in short sentences): > > NOTE: As an optimization, isinstance(object, classinfo) does NOT call classinfo.__instancecheck__(instance) when type(object) == classinfo. > > Consider the following program: > > class M(type): > def __instancecheck__(m, t): > print('instancecheck(%s, %s)' % (m, t)) > return False # LIE! > > Test = M('Test', ((object,)), {}) > > something = Test() > > print('Does *NOT* call __instancecheck__:') > print('isinstance(something, Test): %s' % isinstance(something, Test)) Here's the passage in question, for reference: """ The following methods are used to override the default behavior of the isinstance() and issubclass() built-in functions. In particular, the metaclass abc.ABCMeta implements these methods in order to allow the addition of Abstract Base Classes (ABCs) as ?virtual base classes? to any class or type (including built-in types), including other ABCs. """ https://docs.python.org/3/reference/datamodel.html#customizing-instance-and-subclass-checks Since it uses the word "override", I agree that it's not entirely correct. The implication of "override" is that you can completely replace the normal behaviour. In this case, you can change the behaviour of subclass testing (for instance, you can "disown" a subclass by denying that instances of it are instances of yourself), and of course, you can claim an object as an instance of a class it didn't directly inherit from (the way ABCs work), but you cannot fib about direct instances. I think the behaviour is close enough to accurate that it doesn't need major rewording; how about adding this parenthesis: """ (Note that any object `x` is always considered to be an instance of `x.__class__`, and this cannot be overridden.) """ Would that work? ChrisA From python.gem at gmail.com Sat Oct 27 14:53:56 2018 From: python.gem at gmail.com (Joy Diamond) Date: Sat, 27 Oct 2018 14:53:56 -0400 Subject: [Python-ideas] Fix documentation for __instancecheck__ In-Reply-To: References: Message-ID: Chris, Yes, the following works: """ (Note that any object `x` is always considered to be an instance of `type(x)`, and this cannot be overridden.) """ NOTE: I fixed your sentence of `x.__class__` to `type(x)` since it is not always true that `x.__class__ == type(x)`. For example the actual code reference above: https://github.com/python/cpython/blob/master/Objects/abstract.c#L2397-L2405 Says "if (Py_TYPE(inst) == (PyTypeObject *)cls)" in the actual C Python implementation: So it using `type(x)` not `x.__class__` Thanks, Joy Diamond. ==== ADDENDUM: Here is an example where `type(x) is not x.__class__` (there are other not as perverse examples where you want `.__class__` to differ from `type`) NOTE: The final assert will fail, showing that `type(x) is not x.__class__` # # This really perverse code demonstrates that `t.__class__` is *NOT* # always the same as type(t). # # The code shows an metaclass that when you derive objects from its classes, creates a `7` # instead of a derived class. # def not_really_a_metaclass__make_a_7(name, bases, member): return 7 @property def not_really_a_class__make_a_7(t): return not_really_a_metaclass__make_a_7 class Metaclass__Make_A_7(type): __class__ = not_really_a_class__make_a_7 Make_A_7 = Metaclass__Make_A_7('Make_A_7', ((object,)), {}) # # Now then: # # `Make_A_7` is a class that when inherited from creates a '7' instead # of a class ... :( # # Works for python 2 & pypy (not yet fixed to work for python 3) # class Seven(Make_A_7): # Calls `Make_A_7.__class__('Seven, (('Make_A_7,)), {})` which returns `7` pass print('Seven is: %s' % Seven) assert isinstance(Make_A_7, Metaclass__Make_A_7) assert Make_A_7.__class__ == Metaclass__Make_A_7 # This will *FAIL* due to the perverse definition of `.__class__` On Sat, Oct 27, 2018 at 2:25 PM Chris Angelico wrote: > On Sun, Oct 28, 2018 at 5:03 AM Joy Diamond wrote: > > > > Greetings, > > > > This is a request to fix the documentation for __instancecheck__. > > > > Please add the following (please rewrite better than I can -- I am not > good at explaining concepts in short sentences): > > > > NOTE: As an optimization, isinstance(object, classinfo) does NOT call > classinfo.__instancecheck__(instance) when type(object) == classinfo. > > > > Consider the following program: > > > > class M(type): > > def __instancecheck__(m, t): > > print('instancecheck(%s, %s)' % (m, t)) > > return False # LIE! > > > > Test = M('Test', ((object,)), {}) > > > > something = Test() > > > > print('Does *NOT* call __instancecheck__:') > > print('isinstance(something, Test): %s' % isinstance(something, Test)) > > Here's the passage in question, for reference: > > """ > The following methods are used to override the default behavior of the > isinstance() and issubclass() built-in functions. > > In particular, the metaclass abc.ABCMeta implements these methods in > order to allow the addition of Abstract Base Classes (ABCs) as > ?virtual base classes? to any class or type (including built-in > types), including other ABCs. > """ > > https://docs.python.org/3/reference/datamodel.html#customizing-instance-and-subclass-checks > > Since it uses the word "override", I agree that it's not entirely > correct. The implication of "override" is that you can completely > replace the normal behaviour. In this case, you can change the > behaviour of subclass testing (for instance, you can "disown" a > subclass by denying that instances of it are instances of yourself), > and of course, you can claim an object as an instance of a class it > didn't directly inherit from (the way ABCs work), but you cannot fib > about direct instances. I think the behaviour is close enough to > accurate that it doesn't need major rewording; how about adding this > parenthesis: > > """ > (Note that any object `x` is always considered to be an instance of > `x.__class__`, and this cannot be overridden.) > """ > > Would that work? > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Oct 27 16:09:03 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 27 Oct 2018 16:09:03 -0400 Subject: [Python-ideas] Fix documentation for __instancecheck__ In-Reply-To: References: Message-ID: On 10/27/2018 2:53 PM, Joy Diamond wrote: > Chris, > > Yes, the following works: > """ > (Note that any object `x` is always considered to be an instance of > `type(x)`, and this cannot be overridden.) > """ Open a doc issue on bugs.python.org with the problem, motivation, and proposed solution and it should get considered. > NOTE:? I fixed your sentence of `x.__class__` to `type(x)` since it is > not always true that `x.__class__ == type(x)`. > > For example the actual code reference above: > > https://github.com/python/cpython/blob/master/Objects/abstract.c#L2397-L2405 > > Says "if (Py_TYPE(inst) == (PyTypeObject *)cls)" in the actual C Python > implementation: > > So it using `type(x)` not `x.__class__` > > Thanks, > > Joy Diamond. > > ==== > > ADDENDUM: > > Here is an example where `type(x) is not x.__class__`?? (there are other > not as perverse examples where you want `.__class__` to differ from `type`) > > NOTE:? The final assert will fail, showing that `type(x) is not x.__class__` > > # > #?? This really perverse code demonstrates that `t.__class__` is *NOT* > #?? always the same as type(t). > # > #? The code shows an metaclass that when you derive objects from its > classes, creates a `7` > #? instead of a derived class. > # > def not_really_a_metaclass__make_a_7(name, bases, member): > ??? return 7 > > @property > def not_really_a_class__make_a_7(t): > ??? return not_really_a_metaclass__make_a_7 > > class Metaclass__Make_A_7(type): > ??? __class__ = not_really_a_class__make_a_7 > > Make_A_7 = Metaclass__Make_A_7('Make_A_7', ((object,)), {}) > > > # > #?? Now then: > # > #?????? `Make_A_7` is a class that when inherited from creates a '7' instead > #?????? of a class ... :( > # > #?? Works for python 2 & pypy (not yet fixed to work for python 3) > # > class Seven(Make_A_7):? # Calls `Make_A_7.__class__('Seven, > (('Make_A_7,)), {})` which returns `7` > ??? pass > > print('Seven is: %s' % Seven) > > assert isinstance(Make_A_7, Metaclass__Make_A_7) > assert Make_A_7.__class__ == Metaclass__Make_A_7?? #? This will *FAIL* > due to the perverse definition of `.__class__` > > On Sat, Oct 27, 2018 at 2:25 PM Chris Angelico > > wrote: > > On Sun, Oct 28, 2018 at 5:03 AM Joy Diamond > > wrote: > > > > Greetings, > > > > This is a request to fix the documentation for __instancecheck__. > > > > Please add the following (please rewrite better than I can -- I > am not good at explaining concepts in short sentences): > > > > NOTE:? As an optimization, isinstance(object, classinfo) does NOT > call classinfo.__instancecheck__(instance) when type(object) == > classinfo. > > > > Consider the following program: > > > > class M(type): > >? ? ?def __instancecheck__(m, t): > >? ? ? ? ?print('instancecheck(%s, %s)' % (m, t)) > >? ? ? ? ?return False? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? #? ?LIE! > > > > Test = M('Test', ((object,)), {}) > > > > something = Test() > > > > print('Does *NOT* call __instancecheck__:') > > print('isinstance(something, Test): %s' % isinstance(something, > Test)) > > Here's the passage in question, for reference: > > """ > The following methods are used to override the default behavior of the > isinstance() and issubclass() built-in functions. > > In particular, the metaclass abc.ABCMeta implements these methods in > order to allow the addition of Abstract Base Classes (ABCs) as > ?virtual base classes? to any class or type (including built-in > types), including other ABCs. > """ > https://docs.python.org/3/reference/datamodel.html#customizing-instance-and-subclass-checks > > Since it uses the word "override", I agree that it's not entirely > correct. The implication of "override" is that you can completely > replace the normal behaviour. In this case, you can change the > behaviour of subclass testing (for instance, you can "disown" a > subclass by denying that instances of it are instances of yourself), > and of course, you can claim an object as an instance of a class it > didn't directly inherit from (the way ABCs work), but you cannot fib > about direct instances. I think the behaviour is close enough to > accurate that it doesn't need major rewording; how about adding this > parenthesis: > > """ > (Note that any object `x` is always considered to be an instance of > `x.__class__`, and this cannot be overridden.) > """ > > Would that work? > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Terry Jan Reedy From python.gem at gmail.com Sat Oct 27 16:15:46 2018 From: python.gem at gmail.com (Joy Diamond) Date: Sat, 27 Oct 2018 16:15:46 -0400 Subject: [Python-ideas] Fix documentation for __instancecheck__ In-Reply-To: References: Message-ID: Thanks, Terry. As per your suggestion, I have submitted it as a bug report to: https://bugs.python.org/issue35083 On Sat, Oct 27, 2018 at 4:09 PM Terry Reedy wrote: > On 10/27/2018 2:53 PM, Joy Diamond wrote: > > Chris, > > > > Yes, the following works: > > """ > > (Note that any object `x` is always considered to be an instance of > > `type(x)`, and this cannot be overridden.) > > """ > > Open a doc issue on bugs.python.org with the problem, motivation, and > proposed solution and it should get considered. > > > NOTE: I fixed your sentence of `x.__class__` to `type(x)` since it is > > not always true that `x.__class__ == type(x)`. > > > > For example the actual code reference above: > > > > > https://github.com/python/cpython/blob/master/Objects/abstract.c#L2397-L2405 > > > > Says "if (Py_TYPE(inst) == (PyTypeObject *)cls)" in the actual C Python > > implementation: > > > > So it using `type(x)` not `x.__class__` > > > > Thanks, > > > > Joy Diamond. > > > > ==== > > > > ADDENDUM: > > > > Here is an example where `type(x) is not x.__class__` (there are other > > not as perverse examples where you want `.__class__` to differ from > `type`) > > > > NOTE: The final assert will fail, showing that `type(x) is not > x.__class__` > > > > # > > # This really perverse code demonstrates that `t.__class__` is *NOT* > > # always the same as type(t). > > # > > # The code shows an metaclass that when you derive objects from its > > classes, creates a `7` > > # instead of a derived class. > > # > > def not_really_a_metaclass__make_a_7(name, bases, member): > > return 7 > > > > @property > > def not_really_a_class__make_a_7(t): > > return not_really_a_metaclass__make_a_7 > > > > class Metaclass__Make_A_7(type): > > __class__ = not_really_a_class__make_a_7 > > > > Make_A_7 = Metaclass__Make_A_7('Make_A_7', ((object,)), {}) > > > > > > # > > # Now then: > > # > > # `Make_A_7` is a class that when inherited from creates a '7' > instead > > # of a class ... :( > > # > > # Works for python 2 & pypy (not yet fixed to work for python 3) > > # > > class Seven(Make_A_7): # Calls `Make_A_7.__class__('Seven, > > (('Make_A_7,)), {})` which returns `7` > > pass > > > > print('Seven is: %s' % Seven) > > > > assert isinstance(Make_A_7, Metaclass__Make_A_7) > > assert Make_A_7.__class__ == Metaclass__Make_A_7 # This will *FAIL* > > due to the perverse definition of `.__class__` > > > > On Sat, Oct 27, 2018 at 2:25 PM Chris Angelico > > > > wrote: > > > > On Sun, Oct 28, 2018 at 5:03 AM Joy Diamond > > > > wrote: > > > > > > Greetings, > > > > > > This is a request to fix the documentation for __instancecheck__. > > > > > > Please add the following (please rewrite better than I can -- I > > am not good at explaining concepts in short sentences): > > > > > > NOTE: As an optimization, isinstance(object, classinfo) does NOT > > call classinfo.__instancecheck__(instance) when type(object) == > > classinfo. > > > > > > Consider the following program: > > > > > > class M(type): > > > def __instancecheck__(m, t): > > > print('instancecheck(%s, %s)' % (m, t)) > > > return False # LIE! > > > > > > Test = M('Test', ((object,)), {}) > > > > > > something = Test() > > > > > > print('Does *NOT* call __instancecheck__:') > > > print('isinstance(something, Test): %s' % isinstance(something, > > Test)) > > > > Here's the passage in question, for reference: > > > > """ > > The following methods are used to override the default behavior of > the > > isinstance() and issubclass() built-in functions. > > > > In particular, the metaclass abc.ABCMeta implements these methods in > > order to allow the addition of Abstract Base Classes (ABCs) as > > ?virtual base classes? to any class or type (including built-in > > types), including other ABCs. > > """ > > > https://docs.python.org/3/reference/datamodel.html#customizing-instance-and-subclass-checks > > > > Since it uses the word "override", I agree that it's not entirely > > correct. The implication of "override" is that you can completely > > replace the normal behaviour. In this case, you can change the > > behaviour of subclass testing (for instance, you can "disown" a > > subclass by denying that instances of it are instances of yourself), > > and of course, you can claim an object as an instance of a class it > > didn't directly inherit from (the way ABCs work), but you cannot fib > > about direct instances. I think the behaviour is close enough to > > accurate that it doesn't need major rewording; how about adding this > > parenthesis: > > > > """ > > (Note that any object `x` is always considered to be an instance of > > `x.__class__`, and this cannot be overridden.) > > """ > > > > Would that work? > > > > ChrisA > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Sat Oct 27 19:37:32 2018 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Sat, 27 Oct 2018 19:37:32 -0400 Subject: [Python-ideas] loop: statement In-Reply-To: References: Message-ID: A `loop:` statement to replace `while True:` has probably been discussed at length more than once. Does anyone keep links to the discussions? TIA! -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew at aeracode.org Sat Oct 27 19:42:52 2018 From: andrew at aeracode.org (Andrew Godwin) Date: Sat, 27 Oct 2018 16:42:52 -0700 Subject: [Python-ideas] Standardising ASGI as a PEP Message-ID: Hi everyone, I'd like to breach the topic of standardising an asynchronous successor to WSGI - specifically, the ASGI specification I and a group of other Python web developers have been refining over the past couple of years (you can read more at https://asgi.readthedocs.io/en/latest/). I'm unsure of the best approach to take for this, given a couple of factors: - Web SIG has been basically dead for at least two years and several maintainers I know unsubscribed from it last time as things got toxic. It appears to not be a good place to start this discussion, but maybe it can be revived? - The specification as I would propose it is two or three parts - an overall interface for asynchronous servers to talk to applications and then a separate specification(s) of how to transport HTTP and WebSockets over that. Would this be multiple PEPs? I'd appreciate advice from you all on these questions as well as what you think the best way to even approach something like "let's add a WSGI successor" is. My initial approach was to go away and prove something in real-world use and across a variety of frameworks, and we seem to have got to that point, and so now I would like to start making it more official. I'm more than ready to take the specification we have and start prepping it to land into the PEP repo for further discussion, but I wanted to check in here first before jumping the gun (and besides, there's already plenty of specs, write ups, and reference code to discuss the merits of this). Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Oct 27 20:01:46 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 28 Oct 2018 11:01:46 +1100 Subject: [Python-ideas] Fix documentation for __instancecheck__ In-Reply-To: References: Message-ID: <20181028000145.GT3817@ando.pearwood.info> On Sun, Oct 28, 2018 at 05:24:43AM +1100, Chris Angelico wrote: > On Sun, Oct 28, 2018 at 5:03 AM Joy Diamond wrote: > > NOTE: As an optimization, isinstance(object, classinfo) does NOT > > call classinfo.__instancecheck__(instance) when type(object) == > > classinfo. I'd like to discuss this optimization. It seems very strange to me that a method designed specifically to override the isinstance check isn't actually called to allow it to override the isinstance check. [Chris] > Here's the passage in question, for reference: > > """ > The following methods are used to override the default behavior of the > isinstance() and issubclass() built-in functions. > > In particular, the metaclass abc.ABCMeta implements these methods in > order to allow the addition of Abstract Base Classes (ABCs) as > ?virtual base classes? to any class or type (including built-in > types), including other ABCs. > """ > https://docs.python.org/3/reference/datamodel.html#customizing-instance-and-subclass-checks > > Since it uses the word "override", I agree that it's not entirely > correct. Is that a polite way of saying "wrong"? The question we should be asking, is the optimization implementing the desired behaviour: * classes can disown instances of subclasses * but they cannot disown their own instances or is the optimization over-zealous and does too much? I don't think it is obvious that the behaviour is correct. Presumably Joy had a use-case for overriding isinstance(), and this optimization prevented it. Joy, can you comment on your use-case, and did you come up with a work-around? > The implication of "override" is that you can completely > replace the normal behaviour. Indeed. > In this case, you can change the > behaviour of subclass testing (for instance, you can "disown" a > subclass by denying that instances of it are instances of yourself), > and of course, you can claim an object as an instance of a class it > didn't directly inherit from (the way ABCs work), but you cannot fib > about direct instances. But we can change the class of direct instances, by changing their __class__ attribute, and that is intentional, supported behaviour. So "fibbing" is allowed. Perhaps changing the __class__ is enough to work-around this optimization, and there's nothing to do here except to document it better. But I think that if it is useful to own a non-instance, we shouldn't be so blas? about prohibiting disowning an instance. > I think the behaviour is close enough to > accurate that it doesn't need major rewording; how about adding this > parenthesis: > > """ > (Note that any object `x` is always considered to be an instance of > `x.__class__`, and this cannot be overridden.) > """ I would rather be precise about what is going on, and state that X.__instancecheck__(x) is not called if type(x) is X, rather than merely imply it. It is not just that the method is called and ignored, but that it isn't called at all. I find the process of checking types rather opaque and mysterious. Do we have a documented (other than the source) algorithm for deciding what is an instance of what? - type(x) and x.__class__ don't necessarily agree; under what circumstances are each used? (I've asked this before, and either never got a good answer, or I can't keep it straight in my head.) - what precisely does type(x) do? - when is __instancecheck__ called? A flowchart would be good :-) -- Steve From rosuav at gmail.com Sat Oct 27 20:08:48 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 28 Oct 2018 11:08:48 +1100 Subject: [Python-ideas] Fix documentation for __instancecheck__ In-Reply-To: <20181028000145.GT3817@ando.pearwood.info> References: <20181028000145.GT3817@ando.pearwood.info> Message-ID: On Sun, Oct 28, 2018 at 11:02 AM Steven D'Aprano wrote: > > """ > > (Note that any object `x` is always considered to be an instance of > > `x.__class__`, and this cannot be overridden.) > > """ > > I would rather be precise about what is going on, and state that > X.__instancecheck__(x) is not called if type(x) is X, rather than merely > imply it. It is not just that the method is called and ignored, but > that it isn't called at all. Are there any situations where something is called and ignored, such that the distinction needs to be drawn? Not a rhetorical question. I have no idea what dark corners there are (like the difference between __class__ and type(), which you also are unsure of). ChrisA From steve at pearwood.info Sat Oct 27 20:33:12 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 28 Oct 2018 11:33:12 +1100 Subject: [Python-ideas] loop: statement In-Reply-To: References: Message-ID: <20181028003312.GC18897@ando.pearwood.info> On Sat, Oct 27, 2018 at 07:37:32PM -0400, Juancarlo A?ez wrote: > A `loop:` statement to replace `while True:` has probably been discussed at > length more than once. > > Does anyone keep links to the discussions? Google :-) This might get you started: https://duckduckgo.com/?q=Mikhail+loop+site%3Ahttps%3A%2F%2Fmail.python.org%2Fpipermail%2Fpython-ideas I think Mikhail has raised the same question on the Python-List mailing list, but the level of noise there is probably higher. This thread might also be relevant: https://mail.python.org/pipermail/python-ideas/2017-March/045344.html -- Steve From python.gem at gmail.com Sat Oct 27 21:52:55 2018 From: python.gem at gmail.com (Joy Diamond) Date: Sat, 27 Oct 2018 21:52:55 -0400 Subject: [Python-ideas] Fix documentation for __instancecheck__ In-Reply-To: <20181028000145.GT3817@ando.pearwood.info> References: <20181028000145.GT3817@ando.pearwood.info> Message-ID: On Sat, Oct 27, 2018 at 8:02 PM Steven D'Aprano wrote: > I don't think it is obvious that the behaviour is correct. Presumably > Joy had a use-case for overriding isinstance(), and this optimization > prevented it. > > Joy, can you comment on your use-case, and did you come up with a > work-around? > I'm implementing traits (basically interfaces + members). I am thus replacing the whole object/class/inheritance mechanism [Also replacing __slots__ with actual members] Thus, what I am doing is replacing all of: __cmp__, __eq__, __ge__ ,__gt__, __le__, __lt__, __ne__ (you cannot compare objects, unless they inherit from interface 'Comparable', or 'Hashable'). __delattr__ (thus my classes properly implement 'immutable', to make immutable members). __format__ (you cannot, currently, format objects) __init__ (you cannot write a constructor; instead, it automatically creates a hidden constructor, like C++. You cannot write a constructor, as assignment to members often fails, since they are immutable & __delattr__ & __setattr_ have been fixed to enforce that -- in debug mode) __hash__ (You cannot hash an object, unless it inherits from interface 'Hashable', in which case you can). __new__ (you cannot write a construtor -- see above under __init__) __reduce__, and __reduce__ex__ (you, currently, cannot pickle or unpickle objects -- this has to do with immutable members; and will be fixed if pickling is required). __setattr__ (thus my classes properly implement 'immutable', to make immutable members) __subclasshook__ (classes do not use inheritance, instead they are all trait based) And also, for metaclasses: __call__ (calls an automatically created constructor to construct the object, and work around the fact that some of its members are immutale). __instancecheck__ (classes do not use inheritance, instead they are all trait based) __subclasscheck__ (classes do not use inheritance, instead they are all trait based) __subclasses__ (classes do not use inheritance, instead they are all trait based). My particular use case was: I simply wanted to disable __instancecheck__, which I did, but my unit test code, failed, when the call to __instancecheck__ was bypassed. > > - type(x) and x.__class__ don't necessarily agree; under what > circumstances are each used? > > (I've asked this before, and either never got a good answer, or I can't > keep it straight in my head.) > > - what precisely does type(x) do? > 1. `type(x)` gets the true actual type of `x`. 2. `x.__class__` gets the `.__class__` attribute for `x`, which by default gets the actual true type of `x`, but may be replace by the user to do other stuff. In pythonic terms, `type(x)` does the following: 1. It looks like a constructor to `type` -- Hence it calls type's metaclass `__call__` function. 2. To be precise it calls: `type.__class__.__call__(type, x)` 3. This code can be seen here: https://github.com/python/cpython/blob/master/Objects/typeobject.c#L914-L964 4. Which basically calls `type.__new__` and if `type.__new__` returns a type, calls `type.__init__` (i.e.: the usual way an object is constructed). 5. `type.__new__` is special, it actually returns "the true actual type of `x`". 6. The code for `type.__new__` is here: https://github.com/python/cpython/blob/master/Objects/typeobject.c#L2354-L2392 7. As the code reads: /* Special case: type(x) should return x->ob_type */ /* We only want type itself to accept the one-argument form (#27157) Note: We don't call PyType_CheckExact as that also allows subclasses */ if (metatype == &PyType_Type) { const Py_ssize_t nargs = PyTuple_GET_SIZE(args); const Py_ssize_t nkwds = kwds == NULL ? 0 : PyDict_GET_SIZE(kwds); if (nargs == 1 && nkwds == 0) { PyObject *x = PyTuple_GET_ITEM(args, 0); Py_INCREF(Py_TYPE(x)); return (PyObject *) Py_TYPE(x); } 8. So after going through `type.__call__`, `type.__new__` returns the true actual type of `x` (why this is not optimized in `type.__call__` I've never understood). 9. Thus `type(x)` actually looks like: A CONSTRUCTION of a type object; which is short-circuited to return the previous actual type of `x`. Thus, in many ways, I find `type(x)` to be usage, as I am sure many others do, since it looks like a CONSTRUCtON of a type object, even though it is not. In pythonic terms, `x.__class__` does the following: 1. Call `x.__getattribute__('__class__')`. In general this will find `Object.__dict__["__class__]"` (unless the user is playing games). 2. The default implementation (if not replaced by the user) is here: https://github.com/python/cpython/blob/master/Objects/typeobject.c#L4017-L4021 3. Which calls `object_get_class` as defined here: https://github.com/python/cpython/blob/master/Objects/typeobject.c#L3831-L3836 static PyObject * object_get_class(PyObject *self, void *closure) { Py_INCREF(Py_TYPE(self)); return (PyObject *)(Py_TYPE(self)); } 4. Thus the default implementation of `.__class__` is to return the true type of `x`. In terms of actual usage: 1. It is cleaner, to use `.__class__` 2. It is better to use `.__class__`, and the user can replace it, in rare circumstances. 3. *SOME* internal python code attempts to use `.__class__` (and if this fails; then does a fall back to the true type of `x`) 4. *SOME* internal ptyhon code, bypasses `.__class__` and uses the true type of `x` directly [as for example the implementation of `isinstance` does] 5. Use of `type(x)` does not looks as good [since it looks like a construtor]; but is always accurate, and returns the true type of `x`, and cannot be replaced by the user. Hopefully this helps :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Oct 27 21:59:13 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 28 Oct 2018 12:59:13 +1100 Subject: [Python-ideas] Fix documentation for __instancecheck__ In-Reply-To: References: <20181028000145.GT3817@ando.pearwood.info> Message-ID: On Sun, Oct 28, 2018 at 12:53 PM Joy Diamond wrote: >> - type(x) and x.__class__ don't necessarily agree; under what >> circumstances are each used? >> >> (I've asked this before, and either never got a good answer, or I can't >> keep it straight in my head.) >> >> - what precisely does type(x) do? > > > 1. `type(x)` gets the true actual type of `x`. > 2. `x.__class__` gets the `.__class__` attribute for `x`, which by default gets the actual true type of `x`, but may be replace by the user to do other stuff. > Not that simple. >>> class X: ... cls = "X" ... >>> class Y: ... cls = "Y" ... >>> x = X() >>> x.__class__ = Y >>> x.cls 'Y' >>> type(x) I don't know what the "true actual type" is, since I just changed it. In this case, type() and __class__ do exactly the same thing. The only way I know of (in Py3) to have them show different values is to make __class__ a property, and if you do that, I have no idea what uses the property and what uses type(). Maybe this needs to be actually documented somewhere. It keeps on being a point of confusion. ChrisA From njs at pobox.com Sat Oct 27 22:16:09 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 27 Oct 2018 19:16:09 -0700 Subject: [Python-ideas] Standardising ASGI as a PEP In-Reply-To: References: Message-ID: The WSGI PEP is a bit of a funny thing, since it's a PEP that doesn't really involve the language or stdlib. (I guess there's wsgiref, but I don't think it being in the stdlib actually affects much these days.) What are you hoping to accomplish by making ASGI a PEP? -n On Sat, Oct 27, 2018 at 4:42 PM, Andrew Godwin wrote: > Hi everyone, > > I'd like to breach the topic of standardising an asynchronous successor to > WSGI - specifically, the ASGI specification I and a group of other Python > web developers have been refining over the past couple of years (you can > read more at https://asgi.readthedocs.io/en/latest/). > > I'm unsure of the best approach to take for this, given a couple of factors: > > - Web SIG has been basically dead for at least two years and several > maintainers I know unsubscribed from it last time as things got toxic. It > appears to not be a good place to start this discussion, but maybe it can be > revived? > > - The specification as I would propose it is two or three parts - an > overall interface for asynchronous servers to talk to applications and then > a separate specification(s) of how to transport HTTP and WebSockets over > that. Would this be multiple PEPs? > > I'd appreciate advice from you all on these questions as well as what you > think the best way to even approach something like "let's add a WSGI > successor" is. > > My initial approach was to go away and prove something in real-world use and > across a variety of frameworks, and we seem to have got to that point, and > so now I would like to start making it more official. > > I'm more than ready to take the specification we have and start prepping it > to land into the PEP repo for further discussion, but I wanted to check in > here first before jumping the gun (and besides, there's already plenty of > specs, write ups, and reference code to discuss the merits of this). > > Andrew > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Nathaniel J. Smith -- https://vorpus.org From python.gem at gmail.com Sat Oct 27 22:21:04 2018 From: python.gem at gmail.com (Joy Diamond) Date: Sat, 27 Oct 2018 22:21:04 -0400 Subject: [Python-ideas] Fix documentation for __instancecheck__ In-Reply-To: References: <20181028000145.GT3817@ando.pearwood.info> Message-ID: On Sat, Oct 27, 2018 at 10:00 PM Chris Angelico wrote: > On Sun, Oct 28, 2018 at 12:53 PM Joy Diamond wrote: > >> - type(x) and x.__class__ don't necessarily agree; under what > >> circumstances are each used? > >> > >> (I've asked this before, and either never got a good answer, or I can't > >> keep it straight in my head.) > >> > >> - what precisely does type(x) do? > > > > > > 1. `type(x)` gets the true actual type of `x`. > > 2. `x.__class__` gets the `.__class__` attribute for `x`, which by > default gets the actual true type of `x`, but may be replace by the user to > do other stuff. > > > > Not that simple. > > >>> class X: > ... cls = "X" > ... > >>> class Y: > ... cls = "Y" > ... > >>> x = X() > >>> x.__class__ = Y > >>> x.cls > 'Y' > >>> type(x) > > > I don't know what the "true actual type" is, since I just changed it. > In this case, type() and __class__ do exactly the same thing. The only > way I know of (in Py3) to have them show different values is to make > __class__ a property, and if you do that, I have no idea what uses the > property and what uses type(). > > Maybe this needs to be actually documented somewhere. It keeps on > being a point of confusion. > > ChrisA > Yes, in your example, you actually changed the true actual type of `x` from an `X` to a `Y`. This is permitted since `X` and `Y` are "compatible". For example, if instead you did [making `X` and `Y` no longer compatible]: class X(object): __slots__ = (('x',)) class Y(object): __slots__ = (('y', 'z')) x = X() x.__class__ = Y You get the following: TypeError: __class__ assignment: 'X' object layout differs from 'Y' Thus assigning to `.__class__` (when it has not been replaced by the user), actually transforms an instance to a new true type. [This is actually really useful in some rare edge cases, which is why Python supports it]. Correct, you can make `__class__` a property to replace it (or play some really difficult games with metaclasses to support a different `__class__`). And, yes it is a point of confusion: 1. As per my earlier email, its best to use `.__class__`, as this is the new way of doing things. 2. `type(x)` was the old [Python 1] way of doing things, looks incorrectly like a constructor. 3. It would take weeks to fully document it; and there are many subtle bugs probably in the python source code as to when it uses `.__class__` and when it uses the true type -- i.e.: bypasses it all by using PY_Type(x). CLARIFICATION: By "True actual type" -- I mean it's actual type as implemented in the C code, and returns by the C [macro] `Py_Type(x)`. That is, as defined at https://github.com/python/cpython/blob/master/Include/object.h#L121 #define Py_TYPE(ob) (((PyObject*)(ob))->ob_type) There are hundreds (704 in Python 2.7.12 for example) references to `Py_TYPE` in the C code; thus it would take weeks to document it all. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew at aeracode.org Sun Oct 28 00:15:27 2018 From: andrew at aeracode.org (Andrew Godwin) Date: Sat, 27 Oct 2018 21:15:27 -0700 Subject: [Python-ideas] Standardising ASGI as a PEP In-Reply-To: References: Message-ID: On Sat, Oct 27, 2018 at 7:16 PM Nathaniel Smith wrote: > The WSGI PEP is a bit of a funny thing, since it's a PEP that doesn't > really involve the language or stdlib. (I guess there's wsgiref, but I > don't think it being in the stdlib actually affects much these days.) > Right. This is why I think I'm unsure quite how to approach replacing it. > > What are you hoping to accomplish by making ASGI a PEP? > Essentially to put it on the same platform as things like WSGI and DBAPI2 - to have a directly accepted standard that forms part of the language core. Obviously this is not required for things to function and people to develop against it, but it wasn't required for WSGI either, so in some ways the reason I think it should be a PEP is pretty much purely because WSGI is. Andrew > On Sat, Oct 27, 2018 at 4:42 PM, Andrew Godwin > wrote: > > Hi everyone, > > > > I'd like to breach the topic of standardising an asynchronous successor > to > > WSGI - specifically, the ASGI specification I and a group of other Python > > web developers have been refining over the past couple of years (you can > > read more at https://asgi.readthedocs.io/en/latest/). > > > > I'm unsure of the best approach to take for this, given a couple of > factors: > > > > - Web SIG has been basically dead for at least two years and several > > maintainers I know unsubscribed from it last time as things got toxic. It > > appears to not be a good place to start this discussion, but maybe it > can be > > revived? > > > > - The specification as I would propose it is two or three parts - an > > overall interface for asynchronous servers to talk to applications and > then > > a separate specification(s) of how to transport HTTP and WebSockets over > > that. Would this be multiple PEPs? > > > > I'd appreciate advice from you all on these questions as well as what you > > think the best way to even approach something like "let's add a WSGI > > successor" is. > > > > My initial approach was to go away and prove something in real-world use > and > > across a variety of frameworks, and we seem to have got to that point, > and > > so now I would like to start making it more official. > > > > I'm more than ready to take the specification we have and start prepping > it > > to land into the PEP repo for further discussion, but I wanted to check > in > > here first before jumping the gun (and besides, there's already plenty of > > specs, write ups, and reference code to discuss the merits of this). > > > > Andrew > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > -- > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Sun Oct 28 10:35:13 2018 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 28 Oct 2018 15:35:13 +0100 Subject: [Python-ideas] Standardising ASGI as a PEP In-Reply-To: References: Message-ID: <39090561-5d86-5ace-32b5-a0090f02bf84@egenix.com> On 28.10.2018 05:15, Andrew Godwin wrote: > On Sat, Oct 27, 2018 at 7:16 PM Nathaniel Smith > wrote: > > The WSGI PEP is a bit of a funny thing, since it's a PEP that doesn't > really involve the language or stdlib. (I guess there's wsgiref, but I > don't think it being in the stdlib actually affects much these days.) > > > Right. This is why I think I'm unsure quite how to approach replacing it. ? Why would you want to replace it, if what you have in mind is a different standard for a different audience and use case ? WSGI is not going to go away as a standard since it is useful and works well in the context of non-async web applications. You'll win more fans for ASGI by not going confrontational, since that just sidetracks discussions into non-productive terrains. > What are you hoping to accomplish by making ASGI a PEP? > > > Essentially to put it on the same platform as things like WSGI and > DBAPI2 - to have a directly accepted standard that forms part of the > language core. Obviously this is not required for things to function and > people to develop against it, but it wasn't required for WSGI either, so > in some ways the reason I think it should be a PEP is pretty much purely > because WSGI is. It's a good idea to turn this into an informational PEP similar to the DB API. These standards are developed outside the usual Python development process, but provide good guidance for the Python community at large. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 28 2018) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From mikhailwas at gmail.com Sun Oct 28 11:44:22 2018 From: mikhailwas at gmail.com (Mikhail V) Date: Sun, 28 Oct 2018 18:44:22 +0300 Subject: [Python-ideas] Off-topic: can't access mail.python.org Message-ID: Sorry for posting it here on the list, but I don't know the right contact for technical questions. The problem is, for 3 weeks or so the mail.python.org site is not accessible for me. What can be causing this? This means I can't manage the mail delivery, subscribe or unsubscribe or open any mail archive page. I have no idea what is it. I've sent this question to Ethan Furman few days ago because I thought he is the moderator, but there was no answer from him. Mikhail -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Sun Oct 28 12:07:46 2018 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 28 Oct 2018 16:07:46 +0000 Subject: [Python-ideas] Off-topic: can't access mail.python.org In-Reply-To: References: Message-ID: On 2018-10-28 15:44, Mikhail V wrote: > Sorry for posting it here on the list, but I don't know the right > contact for technical questions. > The problem is, for 3 weeks or so the mail.python.org > > site is not accessible for me. What can be causing this? > This means I can't manage the mail delivery, subscribe or > unsubscribe or open any mail archive page. > I have no idea what is it. > I've sent this question to Ethan Furman few days ago because > I thought he is the moderator, but there was no answer from him. > That link took me to https://mail.python.org/mailman/listinfo. The text says: """ Below is a listing of all the public Mailman 2 mailing lists on mail.python.org. Click on a list name to get more information about the list, or to subscribe, unsubscribe, and change the preferences on your subscription. To visit the general information page for an unadvertised list, open a URL similar to this one, but with a '/' and the list name appended. There are also Mailman 3 lists on mail.python.org. If you don't find a list you're looking for here, it may be a Mailman 3 list. Please go to https://mail.python.org/mm3/mailman3/lists/ for information on Mailman 3 lists. List administrators, you can visit the list admin overview page to find the management interface for your list. If you are having trouble using the lists, please contact mailman at python.org.""" From jpic at yourlabs.org Sun Oct 28 13:43:30 2018 From: jpic at yourlabs.org (Jamesie Pic) Date: Sun, 28 Oct 2018 18:43:30 +0100 Subject: [Python-ideas] [Distutils] Pypi private repo's In-Reply-To: References: <3075001d3cc57$370df900$a529eb00$@sdamon.com> <2A893F9F-C4CE-4470-BC1D-F556A7DA5253@me.com> <20181017144424.GQ3817@ando.pearwood.info> Message-ID: This actively developed tool to manage private package indexes I just found by chance, leaving here for the record: https://github.com/helpshift/pypiprivate -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew at aeracode.org Sun Oct 28 18:34:43 2018 From: andrew at aeracode.org (Andrew Godwin) Date: Sun, 28 Oct 2018 15:34:43 -0700 Subject: [Python-ideas] Standardising ASGI as a PEP In-Reply-To: <39090561-5d86-5ace-32b5-a0090f02bf84@egenix.com> References: <39090561-5d86-5ace-32b5-a0090f02bf84@egenix.com> Message-ID: On Sun, Oct 28, 2018 at 7:35 AM M.-A. Lemburg wrote: > On 28.10.2018 05:15, Andrew Godwin wrote: > > Right. This is why I think I'm unsure quite how to approach replacing it. > > Why would you want to replace it, if what you have in mind is a > different standard for a different audience and use case ? > > WSGI is not going to go away as a standard since it is useful > and works well in the context of non-async web applications. > The word "replace" there was maybe not quite right - I think we need an asynchronous equivalent of WSGI for the same reasons we have WSGI (inter-operability between servers, frameworks, and applications). The use case is quite similar, to be honest. ASGI is designed to be able to encapsulate WSGI applications if desired for backwards compatibility, but I wouldn't expect it to "replace" it per se for a long time, if ever. > > You'll win more fans for ASGI by not going confrontational, > since that just sidetracks discussions into non-productive > terrains. > Oh totally. I'm not here to argue, just to work out how to best get to a Python-wide standard we can all benefit from in the same way as WSGI. > > Essentially to put it on the same platform as things like WSGI and > > DBAPI2 - to have a directly accepted standard that forms part of the > > language core. Obviously this is not required for things to function and > > people to develop against it, but it wasn't required for WSGI either, so > > in some ways the reason I think it should be a PEP is pretty much purely > > because WSGI is. > > It's a good idea to turn this into an informational PEP similar > to the DB API. These standards are developed outside the usual > Python development process, but provide good guidance for the > Python community at large. > I agree - I don't think it makes sense as either of the other options, and both PEP 3333 and PEP 249 are Informational. They're the two most similar things I can think of in the PEP ecosystem. Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: From hemflit at gmail.com Sun Oct 28 21:21:35 2018 From: hemflit at gmail.com (=?UTF-8?Q?Vladimir_Filipovi=C4=87?=) Date: Mon, 29 Oct 2018 02:21:35 +0100 Subject: [Python-ideas] Multi Statement Lambdas In-Reply-To: References: <825275BC-3AD8-4508-9232-5772BDC9CD84@killingar.net> Message-ID: On Tue, Oct 23, 2018 at 1:16 PM Chris Angelico wrote: > a lambda function should be treated as a block of code > inside another function, and not as its own entity. You don't give a > name to the block of code between "if" and "else" ... Very well put! Thanks. > Currently, lambda functions are permitted > to mutate objects in surrounding scopes, but not to rebind them. Well, > actually, PEP 572 might well be the solution there, but then you > reopen all that lovely controversy... I don't think I see how PEP 572 could help solve even a part of it. I mean, I get it enables expressions in general to rebind names in the surrounding scope, but it seems (I'm re-reading the scoping section right now) that lambdas are in effect excluded from that; that any `x := 1` inside a lambda necessarily refers to a local `x`. > I mentioned abusing class syntax as an alternative solution. This > would require a new API, but it wouldn't be hard to write a wrapper. > ... > In some contexts, that would be every bit as good as a lambda-based > solution. To compete, the proposed lambda syntax would have to be > better than all these - by no means impossible, but it's a target to > aim at. Okay, that's... indeed looking abusive :) In situations where it's applicable, it's arguably no worse than a statement lambda for points 2-5 (placement, code locality, naming, natural code inside it). But in terms of "pomp and ceremony" (point 1) it's a bit of a step backwards compared to even writing plain named functions: it calls for multiple keywords and compulsory newlines for something very simple that shouldn't visually command that much attention. Minor point, but also understanding this construct (when reading code) needs a bit of a learning curve, and statement lambdas wouldn't. And it's only applicable with keyword arguments. I understand you expect real examples. `Thread` and `websocket.WebSocketApp` with their keyword arguments were genuine examples from my experience, whereas I couldn't say I've ever felt a need to pass a statement-lambda to `map` or `reduce`. So maybe that disqualifies the positional-arguments objection against this solution, but the verbosity complaint stands. From mistersheik at gmail.com Mon Oct 29 03:07:19 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 29 Oct 2018 00:07:19 -0700 (PDT) Subject: [Python-ideas] async unittest.TestCase In-Reply-To: References: Message-ID: <51dccf8d-8d6c-43cb-b433-782d4b8f5056@googlegroups.com> Why not just use pytest? On Wednesday, October 10, 2018 at 7:12:02 AM UTC-4, David Shawley wrote: > > Hi everyone and good morning to some of you, > > Since asyncio and the async/await syntax are both part of Python, I think > that we should extend TestCase to support it. The simplest solution that > I can think of is to create unittest.AsyncTestCase sub-class with the > following extensions: > > - create a new event loop and install it in AsyncTestCase.run > - make it possible for setUp, tearDown, and test methods to be async > by calling asyncio.iscoroutinefunction > > I wrote my own in a local test before noticing that Martin Richard had > already written and published asynctest [1]. Since then I found the > following projects as well: > > - https://github.com/kwarunek/aiounittest > - https://github.com/pytest-dev/pytest-asyncio > > I think that the community as a whole would benefit from basic support in > unittest for async/await test "user methods". I am personally fond of the > approach of extending unittest.TestCase in asynctest [2] over some of the > other approaches. > > Is this something that we want in our Standard Library? > > - dave > -- > [1]: https://github.com/Martiusweb/asynctest > [2]: https://github.com/Martiusweb/asynctest/blob/master/asynctest/case.py > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daveshawley at gmail.com Mon Oct 29 07:23:14 2018 From: daveshawley at gmail.com (David Shawley) Date: Mon, 29 Oct 2018 07:23:14 -0400 Subject: [Python-ideas] async unittest.TestCase In-Reply-To: <51dccf8d-8d6c-43cb-b433-782d4b8f5056@googlegroups.com> References: <51dccf8d-8d6c-43cb-b433-782d4b8f5056@googlegroups.com> Message-ID: <9BD5345B-EEDF-4C96-82CE-90983482FC2D@gmail.com> Sorry if this double posted but I got a bounce from python-ideas at googlegroups.com the first time that I sent it. I resent to python-ideas at python.org. On Oct 29, 2018, at 3:07 AM, Neil Girdhar wrote: > > Why not just use pytest? > I could use pytest or Martin Richard's asynctest[1]. I want the ability to test asynchronous code without pulling in a 3rd party library. I noticed that a number of the libraries in https://github.com/aio-libs/ have their own custom-rolled test support in addition to those that use pytest. Now that the async/await syntax has landed, the Standard Library should provide the ability to test event loop driven code. This will remove at least on barrier to entry for async/await based code -- you can simply test it using unittest. - cheers, dave. [1]: https://github.com/Martiusweb/asynctest -- "There are only two hard things in Computer Science: cache invalidation and naming things." -- Phil Karlton -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Mon Oct 29 08:26:46 2018 From: mikhailwas at gmail.com (Mikhail V) Date: Mon, 29 Oct 2018 15:26:46 +0300 Subject: [Python-ideas] loop: statement In-Reply-To: References: Message-ID: See my post a month ago (29 September) in the archive with links to some related discussions. Subject "while: for the loop". (I don't have access to the archive now so I can't link to the post) One proposal was exactly about the "loop" keyword: https://mail.python.org/pipermail/python-ideas/2014-June/028202.html I was proposing "while" to replace "while True". My opinion is that adding a new keyword just to replace "while True:" would be too much, also it can cause problems with syntax highlighting, some code analysis and replacements if a variable "loop" is used. Since it was not a keyword, the probability that someone had used such variable can be significant. Mikhail On Sun, Oct 28, 2018 at 2:38 AM Juancarlo A?ez wrote: > > A `loop:` statement to replace `while True:` has probably been discussed > at length more than once. > > Does anyone keep links to the discussions? > > TIA! > > -- > Juancarlo *A?ez* > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Oct 29 09:56:20 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 30 Oct 2018 00:56:20 +1100 Subject: [Python-ideas] Off-topic: can't access mail.python.org In-Reply-To: References: Message-ID: <20181029135619.GY3817@ando.pearwood.info> On Sun, Oct 28, 2018 at 06:44:22PM +0300, Mikhail V wrote: > Sorry for posting it here on the list, but I don't know the right > contact for technical questions. > The problem is, for 3 weeks or so the mail.python.org > site is not accessible for me. What can be causing this? Your DNS is broken? Your ISP has been given a DRM takedown notice, and responded by blocking the entire site? You are running nannyware software which has blocked access to it? Your government has decided that Python is a subversive American plot and secretly banned access? Some virus or malware on your PC is blocking access, perhaps by accident? (I actually think that the last is the most likely. Malware is frequently buggy and has unintended side-effects.) https://downforeveryoneorjustme.com/mail.python.org currently says that the site is up. When you say the site is not accessible, what error do you get? > This means I can't manage the mail delivery, subscribe or > unsubscribe or open any mail archive page. You can't access the archives, but you should be able to manage the rest by email. Try emailing with the word "help" (and nothing else) in the body of the email. See also: https://www.gnu.org/software/mailman/mailman-member/node10.html -- Steve From python at mrabarnett.plus.com Mon Oct 29 13:20:46 2018 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 29 Oct 2018 17:20:46 +0000 Subject: [Python-ideas] Off-topic: can't access mail.python.org In-Reply-To: <20181029135619.GY3817@ando.pearwood.info> References: <20181029135619.GY3817@ando.pearwood.info> Message-ID: <84071c10-6a9b-4575-9e80-f7f552273b1f@mrabarnett.plus.com> On 2018-10-29 13:56, Steven D'Aprano wrote: > On Sun, Oct 28, 2018 at 06:44:22PM +0300, Mikhail V wrote: >> Sorry for posting it here on the list, but I don't know the right >> contact for technical questions. >> The problem is, for 3 weeks or so the mail.python.org >> site is not accessible for me. What can be causing this? > > Your DNS is broken? > > Your ISP has been given a DRM takedown notice, and responded by blocking > the entire site? > > You are running nannyware software which has blocked access to it? > > Your government has decided that Python is a subversive American plot > and secretly banned access? > Or a subversive /Dutch/ plot? [snip] From python.gem at gmail.com Mon Oct 29 14:11:03 2018 From: python.gem at gmail.com (Joy Diamond) Date: Mon, 29 Oct 2018 14:11:03 -0400 Subject: [Python-ideas] Decide if `type.__subclasses__` is part of future Python or not Message-ID: Greetings, Currently `type.__subclasses__` is not documented at docs.python.org (and works in both python 2 & python 3). I would like to keep `.__subclasses__` and have it documented at docs.python.org. Is there a reason it is not documented? I realize the whole type system is going an overhaul, and the concept of what is "subclass" of another class is a complicated issue. See for example: https://github.com/python/typing/issues/135 which talks about the complexity of deciding subclassing. I think though, despite the overall, `types.__subclasses__` should be kept & documented. Thanks, Joy Diamond. 1. The google search: "site:docs.python.org __subclasses__" finds no instances of the word "__subclasses__" 2. As for what the function does, it returns the subclasses of a class: Consider the following program: class CryptographicActors(object): pass class Alice(CryptographicActors): pass class Bob(CryptographicActors): pass print("CryptographicActors subclases: %s" % CryptographicActors.__subclasses__()) help(CryptographicActors.__subclasses__) 3. It properly prints out: CryptographicActors subclases: [, ] Showing the two subclasses (`Alice` and `Bob`) 4. The internal documentation explains "__subclasses__() -> list of immediate subclasses" To fully figure out what it did, I had to read the source code to Python -- which really is not the best way to figure out what a function does; hence the request to document it (and indicate it's future existence in python) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Oct 29 14:15:32 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 29 Oct 2018 11:15:32 -0700 Subject: [Python-ideas] Decide if `type.__subclasses__` is part of future Python or not In-Reply-To: References: Message-ID: __subclasses__ is a runtime thing. As such it should probably be documented -- someone should submit a doc PR. I don't think that "overhaul" is the right word to describe what's going to happen with types in Python -- yes, we're adding optional static type checking, and yes, this will occasionally result in changed recommendations for how the language should be used or how APIs should be designed, but no, this does not mean that we're planning to deprecate or remove existing runtime features. On Mon, Oct 29, 2018 at 11:12 AM Joy Diamond wrote: > Greetings, > > Currently `type.__subclasses__` is not documented at docs.python.org (and > works in both python 2 & python 3). > > I would like to keep `.__subclasses__` and have it documented at > docs.python.org. > > Is there a reason it is not documented? > > I realize the whole type system is going an overhaul, and the concept of > what is "subclass" of another class is a complicated issue. > > See for example: https://github.com/python/typing/issues/135 which talks > about the complexity of deciding subclassing. > > I think though, despite the overall, `types.__subclasses__` should be kept > & documented. > > Thanks, > > Joy Diamond. > > 1. The google search: "site:docs.python.org __subclasses__" finds no > instances of the word "__subclasses__" > > 2. As for what the function does, it returns the subclasses of a class: > > Consider the following program: > > class CryptographicActors(object): pass > class Alice(CryptographicActors): pass > class Bob(CryptographicActors): pass > > print("CryptographicActors subclases: %s" % > CryptographicActors.__subclasses__()) > > help(CryptographicActors.__subclasses__) > > 3. It properly prints out: > > CryptographicActors subclases: [, '__main__.Bob'>] > > Showing the two subclasses (`Alice` and `Bob`) > > 4. The internal documentation explains "__subclasses__() -> list of > immediate subclasses" > > To fully figure out what it did, I had to read the source code to Python > -- which really is not the best way to figure out what a function does; > hence the request to document it (and indicate it's future existence in > python) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Mon Oct 29 14:22:34 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 29 Oct 2018 20:22:34 +0200 Subject: [Python-ideas] Decide if `type.__subclasses__` is part of future Python or not In-Reply-To: References: Message-ID: 29.10.18 20:11, Joy Diamond ????: > Currently `type.__subclasses__` is not documented at docs.python.org > (and works in both python 2 & python 3). > > I would like to keep `.__subclasses__` and have it documented at > docs.python.org . > > Is there a reason it is not documented? It is documented. https://docs.python.org/3/library/stdtypes.html#class.__subclasses__ From python.gem at gmail.com Mon Oct 29 14:22:59 2018 From: python.gem at gmail.com (Joy Diamond) Date: Mon, 29 Oct 2018 14:22:59 -0400 Subject: [Python-ideas] Decide if `type.__subclasses__` is part of future Python or not In-Reply-To: References: Message-ID: Thanks for clarification that `type.__subclasses__` is remaining. I submitted a bug request to document `type.__subclasses__` URL: https://bugs.python.org/issue35106 (Hopefully I did it correctly, as I am just learning how to submit python bugs). On Mon, Oct 29, 2018 at 2:15 PM Guido van Rossum wrote: > __subclasses__ is a runtime thing. As such it should probably be > documented -- someone should submit a doc PR. I don't think that "overhaul" > is the right word to describe what's going to happen with types in Python > -- yes, we're adding optional static type checking, and yes, this will > occasionally result in changed recommendations for how the language should > be used or how APIs should be designed, but no, this does not mean that > we're planning to deprecate or remove existing runtime features. > > On Mon, Oct 29, 2018 at 11:12 AM Joy Diamond wrote: > >> Greetings, >> >> Currently `type.__subclasses__` is not documented at docs.python.org >> (and works in both python 2 & python 3). >> >> I would like to keep `.__subclasses__` and have it documented at >> docs.python.org. >> >> Is there a reason it is not documented? >> >> I realize the whole type system is going an overhaul, and the concept of >> what is "subclass" of another class is a complicated issue. >> >> See for example: https://github.com/python/typing/issues/135 which talks >> about the complexity of deciding subclassing. >> >> I think though, despite the overall, `types.__subclasses__` should be >> kept & documented. >> >> Thanks, >> >> Joy Diamond. >> >> 1. The google search: "site:docs.python.org __subclasses__" finds no >> instances of the word "__subclasses__" >> >> 2. As for what the function does, it returns the subclasses of a class: >> >> Consider the following program: >> >> class CryptographicActors(object): pass >> class Alice(CryptographicActors): pass >> class Bob(CryptographicActors): pass >> >> print("CryptographicActors subclases: %s" % >> CryptographicActors.__subclasses__()) >> >> help(CryptographicActors.__subclasses__) >> >> 3. It properly prints out: >> >> CryptographicActors subclases: [, > '__main__.Bob'>] >> >> Showing the two subclasses (`Alice` and `Bob`) >> >> 4. The internal documentation explains "__subclasses__() -> list of >> immediate subclasses" >> >> To fully figure out what it did, I had to read the source code to Python >> -- which really is not the best way to figure out what a function does; >> hence the request to document it (and indicate it's future existence in >> python) >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python.gem at gmail.com Mon Oct 29 14:25:36 2018 From: python.gem at gmail.com (Joy Diamond) Date: Mon, 29 Oct 2018 14:25:36 -0400 Subject: [Python-ideas] Decide if `type.__subclasses__` is part of future Python or not In-Reply-To: References: Message-ID: Thanks, Storchaka for finding the documentation. Though it is really annoying I could not find it with a google search... I looked really hard before taking the time to write my message. Sorry to bother everyone with this. On Mon, Oct 29, 2018 at 2:22 PM Serhiy Storchaka wrote: > 29.10.18 20:11, Joy Diamond ????: > > Currently `type.__subclasses__` is not documented at docs.python.org > > (and works in both python 2 & python 3). > > > > I would like to keep `.__subclasses__` and have it documented at > > docs.python.org . > > > > Is there a reason it is not documented? > > It is documented. > > https://docs.python.org/3/library/stdtypes.html#class.__subclasses__ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Mon Oct 29 14:38:38 2018 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Mon, 29 Oct 2018 14:38:38 -0400 Subject: [Python-ideas] loop: statement In-Reply-To: <20181028003312.GC18897@ando.pearwood.info> References: <20181028003312.GC18897@ando.pearwood.info> Message-ID: Thanks, Steven. Those links led to the complete discussions, including this one with a "loop:" keyword. https://mail.python.org/pipermail/python-ideas/2014-June/028202.html In short "while:" won't likely ever make it, and adding new keywords to Python only happens on a _absolute-need_ basis. On Sat, Oct 27, 2018 at 8:33 PM Steven D'Aprano wrote: > On Sat, Oct 27, 2018 at 07:37:32PM -0400, Juancarlo A?ez wrote: > > > A `loop:` statement to replace `while True:` has probably been discussed > at > > length more than once. > > > > Does anyone keep links to the discussions? > > Google :-) > > This might get you started: > > > https://duckduckgo.com/?q=Mikhail+loop+site%3Ahttps%3A%2F%2Fmail.python.org%2Fpipermail%2Fpython-ideas > > I think Mikhail has raised the same question on the Python-List mailing > list, but the level of noise there is probably higher. > > This thread might also be relevant: > > https://mail.python.org/pipermail/python-ideas/2017-March/045344.html > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Oct 29 18:06:56 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 29 Oct 2018 18:06:56 -0400 Subject: [Python-ideas] Decide if `type.__subclasses__` is part of future Python or not In-Reply-To: References: Message-ID: On 10/29/2018 2:25 PM, Joy Diamond wrote: > Thanks, Storchaka for finding the documentation. > > Though it is really annoying I could not find it with a google search... Next time, start with our excellent index, which includes a page for words starting with '_'. There is another page for symbols, which are impossible to lookup on Google. -- Terry Jan Reedy From ron.reiter at gmail.com Tue Oct 30 02:58:07 2018 From: ron.reiter at gmail.com (Ron Reiter) Date: Tue, 30 Oct 2018 08:58:07 +0200 Subject: [Python-ideas] gevent-like Coroutines in Python Message-ID: One of Golang's advantages is that goroutines act like gevent's coroutines instead of relying on an async/await syntax. In my opinion, it makes Golang code much more readable. I feel like having the await syntax trigger by default on any awaitable invocation in a coroutine context makes much more sense. I consider async/await syntax to be too complicated for the average developer since it opens up too many abilities whereas most developers would always mean they prefer to do this: result = [await fun(x) for fun in funcs] versus: result = [fun(x) for fun in funcs] await asyncio.gather(*result) Moreso, having it become the default makes statements like this: result = [await fun(x) for fun in funcs if await smth] Look like this: result = [fun(x) for fun in funcs if smth] Therefore, my suggestion is to create a new "async" definition which basically turns every function invocation into an "await" if a generator is returned. Instead of "async def" I propose the alternative "coroutine def" syntax. However, a better solution may be to imply the word "async" in every function definition given some sort of trigger (however I assume that this won't be the preferable approach as it is not something that can be implemented at the parsing level). For me, I believe that Python should aspire to be both concise and straightforward, which means no boilerplate code just because it's more "correct" but rather assume there's some logical default behavior going on in the back - and to me this logical behavior is that every invocation can trigger an await and go back to the main loop. Our assumption that a function invocation has to tie back to instant execution unless an "await" statement has been placed should be challenged in favor of readability, conciseness, and to always aim to appeal to novice developers (which I believe is the reason Python is on such a rise these days). I do have to admit I have not thought through what is the best syntax or the implications of this because I would like to see what is the general opinion on this idea first. - Ron -------------- next part -------------- An HTML attachment was scrubbed... URL: From python.gem at gmail.com Tue Oct 30 04:40:40 2018 From: python.gem at gmail.com (Joy Diamond) Date: Tue, 30 Oct 2018 04:40:40 -0400 Subject: [Python-ideas] Add a way to test for a descriptor in Python Code Message-ID: Greetings, I am trying to emulate attribute lookup, and want to test if symbol found in __dict__ is an attribute or not in python code (i.e.: does its have a `tp_descr_get` slot?) Reading the documentation carefully, I am supposed to test if the attribute has a `.__get__` method; however, I see no way to test for this properly (easily). Currently the only way I know to test for this is (See code at end of this message): any('__get__' in m.__dict__ for m in type(v).__mro__) Which seems terribly inefficient. The documentation at: https://docs.python.org/2/howto/descriptor.html https://docs.python.org/3/howto/descriptor.html Both says: """ For classes, the machinery is in type.__getattribute__() which transforms B.x into B.__dict__['x'].__get__(None, B). In pure Python, it looks like: def __getattribute__(self, key): "Emulate type_getattro() in Objects/typeobject.c" v = object.__getattribute__(self, key) if hasattr(v, '__get__'): return v.__get__(None, self) return v """ However, the call to `hasattr(v, '__get__')` appears to me to be incorrect. The question is *NOT* whether 'v' has an attribute '__get__'; *BUT* whether `v` has a symbol `__get__` in any of the classes in it's method resolution order. Looking at `type_getattro` in "Objects/typeobject.c" here: https://github.com/python/cpython/blob/master/Objects/typeobject.c#L3177 Reads: meta_get = Py_TYPE(meta_attribute)->tp_descr_get So I really want to know if the `tp_descr_get` slot is set or not. (Which is a different question than whether `v` has a `__get__` attribute). The code below shows that: 1. The valid value of `Point.y` is 2. The valid value is returned by `Point.y`, `type.__getattribute__(Point, y)`, and `fixed__Type__getattribute` 3. The invalid value of `Point.y` is `2` as returned by the [emulated] `__getattribute__` documented https://docs.python.org/3/howto/descriptor.html So I am requesting: 1. An efficient python way to test for `tp_descr_get` slot (and other descriptor) slots. 2. Fix the documentation referenced above. Thanks, Joy Diamond. NOTE #1: This email describes the very subtle difference between whether an instance has a `__get__` symbol or not, which I believe is *NOT* the same question as whether `hasattr(instance, "__get__")` returns true or not. The first question is does it have the symbol `__get_` [Which python put in the `tp_descr_slot`] while `hasattr` answers the question does it have the `__get__` attribute. NOTE #2: Also using `hasattr(type(v), "__get__")` would not answer the question I want, because then it might find a `__get__` method in the meta-class of `type(v)` which again would return an incorrect answer. Example program that shows that using `hasattr(v, '__get__')` is not a valid way to test if something is a descriptor (works in python 2, python 3, and pypy): def get_1(self, a, b): return 1 def get_2(a, b): return 2 class Descriptor(object): __get__ = get_1 class Not_A_Descriptor(object): def __init__(self): self.__get__ = get_2 def __repr__(self): return '' class Point(object): __slots__ = (()) x = Descriptor() y = Not_A_Descriptor() # # Copied from https://docs.python.org/3/howto/descriptor.html # def __getattribute__(self, key): "Emulate type_getattro() in Objects/typeobject.c" v = object.__getattribute__(self, key) if hasattr(v, '__get__'): return v.__get__(None, self) return v # # My fixed version # def fixed__Type__getattribute(self, key): "FIXED: Emulate type_getattro() in Objects/typeobject.c" v = object.__getattribute__(self, key) if any('__get__' in m.__dict__ for m in type(v).__mro__): return v.__get__(None, self) return v print('Point.x: %s' % Point.x) print('Point.y: %s' % Point.y) print("type.__getattribute__(Point, 'x'): %s" % type.__getattribute__(Point, 'x')) print("type.__getattribute__(Point, 'y'): %s" % type.__getattribute__(Point, 'y')) print("__getattribute__(Point, 'x'): %s" % __getattribute__(Point, 'x')) print("__getattribute__(Point, 'y'): %s ***WRONG***" % __getattribute__(Point, 'y')) print("fixed__Type__getattribute(Point, 'x'): %s" % fixed__Type__getattribute(Point, 'x')) print("fixed__Type__getattribute(Point, 'y'): %s ***CORRECT***" % fixed__Type__getattribute(Point, 'y')) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Oct 30 05:34:10 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 30 Oct 2018 22:34:10 +1300 Subject: [Python-ideas] gevent-like Coroutines in Python In-Reply-To: References: Message-ID: <5BD82592.7010400@canterbury.ac.nz> Ron Reiter wrote: > I feel like having the await syntax trigger by default on any awaitable > invocation in a coroutine context makes much more sense. Guido seems to regard the requirement to use 'await' as a feature, not a bug. He says he likes to be able to see where all the potential suspension points are. > Therefore, my suggestion is to create a new "async" definition which > basically turns every function invocation into an "await" if a generator > is returned. Instead of "async def" I propose the alternative "coroutine > def" syntax. However, a better solution may be to imply the word "async" > in every function definition given some sort of trigger (however I > assume that this won't be the preferable approach as it is not something > that can be implemented at the parsing level). I don't think it's feasible to automatically infer both 'async' *and* 'await', even at run time. An async function needs to be treated differently from the moment it starts running, so there must be something that statically identifies it as such. Anyway, if you want to pursue these ideas further, you should take a look at PEP 3152, which was my attempt at a nicer syntax for generator-based coroutines, before async/await came along. I think it would have been better in some ways, but it was rejected. -- Greg From steve at pearwood.info Tue Oct 30 06:30:31 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 30 Oct 2018 21:30:31 +1100 Subject: [Python-ideas] Add a way to test for a descriptor in Python Code In-Reply-To: References: Message-ID: <20181030103031.GB3817@ando.pearwood.info> On Tue, Oct 30, 2018 at 04:40:40AM -0400, Joy Diamond wrote: > """ > For classes, the machinery is in type.__getattribute__() which transforms > B.x into B.__dict__['x'].__get__(None, B). In pure Python, it looks like: > > def __getattribute__(self, key): > "Emulate type_getattro() in Objects/typeobject.c" > v = object.__getattribute__(self, key) > if hasattr(v, '__get__'): > return v.__get__(None, self) > return v > > """ [...] > However, the call to `hasattr(v, '__get__')` appears to me to be incorrect. I agree, but only because it fails to take into account that dunder methods like __get__ are only looked up on the class, not the instance. I believe a more accurate eumulation would be: if hasattr(type(v), '__get__'): return type(v).__get__(None, self) Actually, on further investigation, I think it ought to be: if inspect.hasattr_static(type(v), '__get__') except that there is no hasattr_static, there's only a getattr_static. So perhaps there ought to be a hasattr_static as well. > The question is *NOT* whether 'v' has an attribute '__get__'; *BUT* whether > `v` has a symbol `__get__` in any of the classes in it's method resolution > order. What's the difference as you see it? -- Steve From rosuav at gmail.com Tue Oct 30 06:40:41 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Oct 2018 21:40:41 +1100 Subject: [Python-ideas] gevent-like Coroutines in Python In-Reply-To: References: Message-ID: On Tue, Oct 30, 2018 at 6:01 PM Ron Reiter wrote: > > ... most developers would always mean they prefer to do this: > > result = [await fun(x) for fun in funcs] > > versus: > > result = [fun(x) for fun in funcs] > await asyncio.gather(*result) > > Moreso, having it become the default makes statements like this: > > result = [await fun(x) for fun in funcs if await smth] > > Look like this: > > result = [fun(x) for fun in funcs if smth] > > Therefore, my suggestion is to create a new "async" definition which basically turns every function invocation into an "await" if a generator is returned. > I'm not sure what you're driving at here. From your first example, I gather that (pun intended) you're expecting the 'result' list to contain all the results from the different function calls, running them all in parallel; but your second example and described suggestion seem to imply that the waitings would continue to be sequential. Unless you're asking for straight-up magic ("do everything in parallel unless they need to be serialized"), there still needs to be a clear way to differentiate between "wait for this right now and give me a result before this function continues" and "gather all these jobs together, get me the results, and then move on once you have them all". It might perhaps be nice to have an easier/more obvious syntax for gather(), but it definitely needs to have some form of spelling. If you're not asking for them to be run in parallel, you're asking for an implicit way for a function call to block its caller, and for the calling function to act sequentially. Python already has that - it's called threading :) ChrisA From adelfino at gmail.com Tue Oct 30 06:43:27 2018 From: adelfino at gmail.com (Andre Delfino) Date: Tue, 30 Oct 2018 07:43:27 -0300 Subject: [Python-ideas] Make fnmatch.filter accept a tuple of patterns Message-ID: Frequently, while globbing, one needs to work with multiple extensions. I?d like to propose for fnmatch.filter to handle a tuple of patterns (while preserving the single str argument functionality, alas str.endswith), as a first step for glob.i?glob to accept multiple patterns as well. Here is the implementation I came up with: https://github.com/python/cpython/compare/master...andresdelfino:fnmatch-multiple-patterns?expand=1 If this is deemed reasonable, I?ll write tests and documentation updates. Any opinion? -------------- next part -------------- An HTML attachment was scrubbed... URL: From python.gem at gmail.com Tue Oct 30 07:31:26 2018 From: python.gem at gmail.com (Joy Diamond) Date: Tue, 30 Oct 2018 07:31:26 -0400 Subject: [Python-ideas] Add a way to test for a descriptor in Python Code In-Reply-To: <20181030103031.GB3817@ando.pearwood.info> References: <20181030103031.GB3817@ando.pearwood.info> Message-ID: Clarifications: 1. I miswrote part of my first post where I wrote "I want to test if symbol found in __dict__ is an attribute or not in python code". I meant to write "is a DESCRIPTOR" or not. 2. The example in https://docs.python.org/3/howto/descriptor.html for reproducing `type.__getattribute__` has a second bug, in that it does not look at the class inheritance properly. (See fixed example below named `type_getattro` and based on the same function in the C code). In the example below if you call ` __getattribute__(Child, 'x')` it will incorrectly fail with "Catch `AttributeError: 'type' object has no attribute 'x'`" Responding to Steve: On Tue, Oct 30, 2018 at 6:31 AM Steven D'Aprano wrote: > Actually, on further investigation, I think it ought to be: > > if inspect.hasattr_static(type(v), '__get__') > > except that there is no hasattr_static, there's only a getattr_static. > So perhaps there ought to be a hasattr_static as well. > > `inspect.hasattr_static` gets me half way there (but it still looks in two chains of inheritance, where I only want to look in one). In particular if the metaclass of `type(v)` has a `.__get__` method it will incorrectly find that. So it will still misidentify if an instance is a descriptor or not. > > The question is *NOT* whether 'v' has an attribute '__get__'; *BUT* > whether > > `v` has a symbol `__get__` in any of the classes in it's method > resolution > > order. > > What's the difference as you see it? > I want to be able to look in the method resolution order (one inheritance chain). `getattr` and `inspect.getattr_static` both look in two inheritance chains (the instance & it's type; or the case of a class, the class and it's metaclass). I need to look in only one chain (and disable descriptors like `inspect.getattr_static` does). To put it succinctly: I am trying to reproduce the behavior of `_PyType_Lookup` from "Objects/typeobject.c" (see example below). Below is a full reproduction of `object.__getattribute__` and `type.__getattribute__` based on reading the Python source code. Note this reproduction of `type.__getattribute__` is much more accurate than what is at: https://docs.python.org/2/howto/descriptor.html https://docs.python.org/3/howto/descriptor.html Both of which need to be updated. (This is not yet filed as a bug report; as first I am requesting a call to something like `_PyType_Lookup` that is efficent; and once we agree on that, we can created an updated reproduction of `type.__getattribute__`). Thanks, Joy Diamond. # # The following reproduces (and tests) `object.__getattribute__` and `type.__getattribute__` based on reading the C source code. # absent = object() def _PyType_Lookup(model, name): '''Based on `_PyType_Lookup` in "Objects/typeobject.c"''' mro = model.__mro__ if mro is None: return absent for m in mro: symbol_table = m.__dict__ if name in symbol_table: return symbol_table[name] return absent def lookup__tp_descr_get(model): tp_descr_get = _PyType_Lookup(model, '__get__') return tp_descr_get def has__tp_descr_set(model): tp_descr_set = _PyType_Lookup(model, '__set__') return tp_descr_set is not absent # # Reproduction of `object.__getattribute__` # def PyObject_GenericGetAttr(instance, name): '''Based on `PyObject_GenericGetAttr` in "Objects/object.c"''' instance_type = type(instance) descriptor = _PyType_Lookup(instance_type, name) if descriptor is absent: get = absent else: descriptor_type = type(descriptor) get = lookup__tp_descr_get(descriptor_type) if (get is not absent) and (has__tp_descr_set(descriptor_type)): # # "Data Descriptor" (a `__set__` method exists) has precedence. # return get(descriptor, instance, instance_type) if instance_type.__dictoffset__: instance__mapping = instance.__dict__ if name in instance__mapping: return instance__mapping[name] if get is not absent: return get(descriptor, instance, instance_type) raise AttributeError("cannot find attribute `{}` in instance of `{}`".format(name, instance_type.__name__)) # # Reproduction of `type.__getattribute__` # def type_getattro(model, name): '''Based on `type_getattro` in "Objects/type_object.c"''' metatype = type(model) descriptor = _PyType_Lookup(metatype, name) if descriptor is absent: get = absent else: descriptor_type = type(descriptor) get = lookup__tp_descr_get(descriptor_type) if (get is not absent) and (has__tp_descr_set(descriptor_type)): # # "Data Descriptor" (a `__set__` method exists) has precedence. # return get(descriptor, instance, instance_type) symbol = _PyType_Lookup(model, name) if symbol is not absent: # # Implement descriptor functionality, if any # symbol_type = type(symbol) symbol_get = lookup__tp_descr_get(symbol_type) if symbol_get is not absent: # # None 2nd argument indicates the descriptor was # found on the target object itself (or a base) # return symbol_get(symbol, None, model) return symbol if get is not absent: return get(descriptor, instance, instance_type) raise AttributeError("cannot find attribute `{}` in class `{}`".format(name, instance_type.__name__)) # # object Example # print('=== object example ===') class Readonly_Descriptor(object): __slots__ = (( 'value', )) def __init__(self, value): self.value = value def __get__(self, object, object_type): return self.value class Data_Descriptor(object): __slots__ = (( 'value', )) def __init__(self, value): self.value = value def __get__(self, object, object_type): return self.value def __set__(self, object, value): self.value = value class Point(object): __slots__ = (( '__dict__', '_y', )) def __init__(self, x, y): self.x = x self.y = y self._y = 1 p23 = Point(2, 3) Point.x = Readonly_Descriptor(4) Point.y = Data_Descriptor(5) Point.z = Readonly_Descriptor(6) p23.y = 7 # Uses the Data_Descriptor print("p23.x: %d # p23.__dict__['x']; *IGNORES* ReadOnly_Descriptor" % p23.x) print("p23.y: %d # type(p23).__dict__['y'].__get__(p23, Point)" % p23.y) print("p23.z: %d # type(p23).__dict__['z'].__get__(p23, Point)" % p23.z) print('') print("PyObject_GenericGetAttr(p23, 'x'): %d" % PyObject_GenericGetAttr(p23, 'x')) print("PyObject_GenericGetAttr(p23, 'y'): %d" % PyObject_GenericGetAttr(p23, 'y')) print("PyObject_GenericGetAttr(p23, 'z'): %d" % PyObject_GenericGetAttr(p23, 'z')) # # Type Example # print('') print('=== Type example ===') def get_1(self, a, b): return 1 def get_2(a, b): return 2 class Descriptor(object): __get__ = get_1 class Not_A_Descriptor(object): def __init__(self): self.__get__ = get_2 def __repr__(self): return '' class Parent(object): __slots__ = (()) x = Descriptor() y = Not_A_Descriptor() class Child(Parent): __slots__ = (()) # # Copied from https://docs.python.org/3/howto/descriptor.html # def __getattribute__(self, key): "Emulate type_getattro() in Objects/typeobject.c" v = object.__getattribute__(self, key) if hasattr(v, '__get__'): return v.__get__(None, self) return v print('Child.x: %s' % Child.x) print('Child.y: %s' % Child.y) print("type.__getattribute__(Child, 'x'): %s" % type.__getattribute__(Child, 'x')) print("type.__getattribute__(Child, 'y'): %s" % type.__getattribute__(Child, 'y')) try: print("__getattribute__(Child, 'x'): %s" % __getattribute__(Child, 'x')) print("__getattribute__(Child, 'y'): %s ***WRONG***" % __getattribute__(Child, 'y')) except AttributeError as e: # # Catch `AttributeError: 'type' object has no attribute 'x'` # pass print("type_getattro(Child, 'x'): %s" % type_getattro(Child, 'x')) print("type_getattro(Child, 'y'): %s ***CORRECT***" % type_getattro(Child, 'y')) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.reiter at gmail.com Tue Oct 30 08:35:53 2018 From: ron.reiter at gmail.com (Ron Reiter) Date: Tue, 30 Oct 2018 14:35:53 +0200 Subject: [Python-ideas] gevent-like Coroutines in Python In-Reply-To: References: Message-ID: You are right that they are different, I was actually assuming that developers by default don't try to parallelize and would rather go ahead and write code to yield one function at a time, which is fine. The need to separate "await" from the invocation is something which is rarely used. Not sure what you mean about "threading" as it is still more efficient and lightweight to parallelize workers on an event loop rather than using blocking threads. As I said - I don't think we should downgrade Python's current ability to do so, my suggestion is to create something like the "codef" proposal, which will also await on every function invocation - for readability. - Ron [image: Facebook] [image: Twitter] [image: LinkedIn] On Tue, Oct 30, 2018 at 12:41 PM Chris Angelico wrote: > On Tue, Oct 30, 2018 at 6:01 PM Ron Reiter wrote: > > > > ... most developers would always mean they prefer to do this: > > > > result = [await fun(x) for fun in funcs] > > > > versus: > > > > result = [fun(x) for fun in funcs] > > await asyncio.gather(*result) > > > > Moreso, having it become the default makes statements like this: > > > > result = [await fun(x) for fun in funcs if await smth] > > > > Look like this: > > > > result = [fun(x) for fun in funcs if smth] > > > > Therefore, my suggestion is to create a new "async" definition which > basically turns every function invocation into an "await" if a generator is > returned. > > > > I'm not sure what you're driving at here. From your first example, I > gather that (pun intended) you're expecting the 'result' list to > contain all the results from the different function calls, running > them all in parallel; but your second example and described suggestion > seem to imply that the waitings would continue to be sequential. > > Unless you're asking for straight-up magic ("do everything in parallel > unless they need to be serialized"), there still needs to be a clear > way to differentiate between "wait for this right now and give me a > result before this function continues" and "gather all these jobs > together, get me the results, and then move on once you have them > all". It might perhaps be nice to have an easier/more obvious syntax > for gather(), but it definitely needs to have some form of spelling. > > If you're not asking for them to be run in parallel, you're asking for > an implicit way for a function call to block its caller, and for the > calling function to act sequentially. Python already has that - it's > called threading :) > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Oct 30 08:41:04 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Oct 2018 23:41:04 +1100 Subject: [Python-ideas] gevent-like Coroutines in Python In-Reply-To: References: Message-ID: On Tue, Oct 30, 2018 at 11:36 PM Ron Reiter wrote: > > You are right that they are different, I was actually assuming that developers by default don't try to parallelize and would rather go ahead and write code to yield one function at a time, which is fine. The need to separate "await" from the invocation is something which is rarely used. Not sure what you mean about "threading" as it is still more efficient and lightweight to parallelize workers on an event loop rather than using blocking threads. > Okay, so it's actually nothing to do with asyncio.gather(). Sure. So what you're looking for is JUST the removal of the "await" keywords. As Greg already said, Guido considers the explicit await markers as a feature, not a bug; these are the exact points where an intrathread context switch can occur. As to the efficiency of parallelizing on an event loop rather than using threads, that's a tradeoff; threads aren't going anywhere just because asyncio is here. When you want the extreme simplicity of "just do this stuff, okay?", the easiest way to get it is to just use threads, and pay a bit of overhead. You'll often find that the overhead isn't actually all that significant until you get to extremes of throughput - most Python apps are not trying to run tens of thousands of concurrent TCP sockets, for instance. ChrisA From mehaase at gmail.com Tue Oct 30 09:56:48 2018 From: mehaase at gmail.com (Mark E. Haase) Date: Tue, 30 Oct 2018 09:56:48 -0400 Subject: [Python-ideas] gevent-like Coroutines in Python In-Reply-To: References: Message-ID: Python's coroutines are designed to make suspension points visible, which enhances "local reasoning" about code. This concept has been written up very well over here: https://glyph.twistedmatrix.com/2014/02/unyielding.html On Tue, Oct 30, 2018 at 8:37 AM Ron Reiter wrote: > You are right that they are different, I was actually assuming that > developers by default don't try to parallelize and would rather go ahead > and write code to yield one function at a time, which is fine. The need to > separate "await" from the invocation is something which is rarely used. Not > sure what you mean about "threading" as it is still more efficient and > lightweight to parallelize workers on an event loop rather than using > blocking threads. > > As I said - I don't think we should downgrade Python's current ability to > do so, my suggestion is to create something like the "codef" proposal, > which will also await on every function invocation - for readability. > > - Ron > > > [image: Facebook] [image: Twitter] > [image: LinkedIn] > > > > On Tue, Oct 30, 2018 at 12:41 PM Chris Angelico wrote: > >> On Tue, Oct 30, 2018 at 6:01 PM Ron Reiter wrote: >> > >> > ... most developers would always mean they prefer to do this: >> > >> > result = [await fun(x) for fun in funcs] >> > >> > versus: >> > >> > result = [fun(x) for fun in funcs] >> > await asyncio.gather(*result) >> > >> > Moreso, having it become the default makes statements like this: >> > >> > result = [await fun(x) for fun in funcs if await smth] >> > >> > Look like this: >> > >> > result = [fun(x) for fun in funcs if smth] >> > >> > Therefore, my suggestion is to create a new "async" definition which >> basically turns every function invocation into an "await" if a generator is >> returned. >> > >> >> I'm not sure what you're driving at here. From your first example, I >> gather that (pun intended) you're expecting the 'result' list to >> contain all the results from the different function calls, running >> them all in parallel; but your second example and described suggestion >> seem to imply that the waitings would continue to be sequential. >> >> Unless you're asking for straight-up magic ("do everything in parallel >> unless they need to be serialized"), there still needs to be a clear >> way to differentiate between "wait for this right now and give me a >> result before this function continues" and "gather all these jobs >> together, get me the results, and then move on once you have them >> all". It might perhaps be nice to have an easier/more obvious syntax >> for gather(), but it definitely needs to have some form of spelling. >> >> If you're not asking for them to be run in parallel, you're asking for >> an implicit way for a function call to block its caller, and for the >> calling function to act sequentially. Python already has that - it's >> called threading :) >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From julien at tayon.net Tue Oct 30 11:31:12 2018 From: julien at tayon.net (julien tayon) Date: Tue, 30 Oct 2018 16:31:12 +0100 Subject: [Python-ideas] Implementing a set of operation (+, /, - *) on dict consistent with linearAlgebrae Message-ID: Hello :) the idea is described here:http://jul.github.io/cv/pres.html#printable Summary of the idea : Take a linear algebrae book, and implements all the rules as a TDD.https://github.com/jul/archery/blob/master/consistent_algebrae.py make it works based on abstract base class and sets of Mixins.https://archery.readthedocs.io/en/latest/ And see if we can make cos/__abs__/dot and if it gives naively the intended results ? (spoiler: yes) Making it work with dict, and "other" dictionary like counter by using ineritancehttps://archery.readthedocs.io/en/latest/#advanced-usage My idea is : wouldn't it be nice if we introduced geometries as sets of mixins for objects ? (Hilbertian algebrae could be nice too, and we could make MutableMapping behave like bra/kets). So I was proposing a soft discussion on : could we agree that it would be nice to consider operation overloading as a whole set of behaviours that could profit from being consistent in a categorized way ? (like the + of [] could be the + of "RecordAlgebrae") Meaning we could define sets of "expected behaviour consistent interaction between operators" as we defined the abc and call them algebrae? I offer the LinearAlgebrae Mixins as a POC, and was thinking of creating a unittest to qualify if an object is following the rules of linear algebrae. What are your opinions ? I don't actually see a lot of use case except it was funny to build. But maybe it can be of use. Cordialement -- Julien -------------- next part -------------- An HTML attachment was scrubbed... URL: From boxed at killingar.net Tue Oct 30 11:43:31 2018 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Tue, 30 Oct 2018 16:43:31 +0100 Subject: [Python-ideas] Implementing a set of operation (+, /, - *) on dict consistent with linearAlgebrae In-Reply-To: References: Message-ID: <12111354-2B7E-472A-97E5-A193CD9A0F2A@killingar.net> > What are your opinions ? > I don't actually see a lot of use case except it was funny to build. But > maybe it can be of use. This list is for suggesting additions and changes to python. Broad usefulness is a prerequisite. So please build your lib but this seems off topic on this list. / Anders From robertve92 at gmail.com Tue Oct 30 14:10:55 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Tue, 30 Oct 2018 19:10:55 +0100 Subject: [Python-ideas] Implementing a set of operation (+, /, - *) on dict consistent with linearAlgebrae In-Reply-To: <12111354-2B7E-472A-97E5-A193CD9A0F2A@killingar.net> References: <12111354-2B7E-472A-97E5-A193CD9A0F2A@killingar.net> Message-ID: Julien, your article is very pleasant to read (and funny) but as other say the mailing list is not there to share some articles, but for proposition to the standard python library, do our own lib on github and pypi first if you want to Share some code to the world ! And if project becomes super useful to everyone one day, it may come one day to the standard library so that everybody will have it. Cheers, Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From julien at tayon.net Tue Oct 30 15:06:51 2018 From: julien at tayon.net (julien tayon) Date: Tue, 30 Oct 2018 20:06:51 +0100 Subject: [Python-ideas] Implementing a set of operation (+, /, - *) on dict consistent with linearAlgebrae In-Reply-To: References: <12111354-2B7E-472A-97E5-A193CD9A0F2A@killingar.net> Message-ID: Thanks robert for the praise. It feels nice. I may be bold, but I really hate to come empty handed to a discussion. So this lib is nothing more than doing my homework when I don't have a PhD. Actually, science (in my opinion) is about measuring. What I propose is nothing more than (if you add Vector traits) giving native metrics to objects (having coded in Perl for too long I still see objects as a hierarchy of blessed MutableMappings, I am sorry). And I think that measurements are a corner stone of science, thus of data science. (my opinion you may not share). Thus it could be kind of extending some of the concepts of datasets : https://www.python.org/dev/peps/pep-0557/ to additionnal default behaviour (that could be subscribed optionnally). As an everyday coder, this behaviour does solve problems I can illustrate with code (like aggregating data, or measuring if I might have doubon in a set of dataset, transforming objects into objects). I do not want to force feed the community with my "brilliant" ideas, I much more would like to plead my case on how adopting "consistent geometric behaviours" at the language level would ease our lives as coders, if this is not inappropriate. Please don't look at the lib. Look at the idea of making operators behave in a consistent way that gives the property of well known mathematic constructions to the core of the language. It also enables parallelisation without side effects (aka the map reduce of the poors), which are a first order consequence of the linear algebrae. I may not be gifted with writing long dissertations, however, I have a pragmatic mind. So I don't mind being challenged a tad, as long as we talk about stuffs like : how does it profit python coders to be standard, can you show me real life example ? However, if a "no (answer)" is a "no", I do understand. I like python the way it is, and I don't want to introduce friction in the process of improving python by being off topic. Thus if no one is interested, I still have a last word : keep up the good work! And thank you all for what you bring us. Cheers On Tue, 30 Oct 2018 at 19:11, Robert Vanden Eynde wrote: > Julien, your article is very pleasant to read (and funny) but as other say > the mailing list is not there to share some articles, but for proposition > to the standard python library, > > do our own lib on github and pypi first if you want to Share some code to > the world ! > > And if project becomes super useful to everyone one day, it may come one > day to the standard library so that everybody will have it. > > Cheers, > > Robert > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Oct 30 17:30:41 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 31 Oct 2018 10:30:41 +1300 Subject: [Python-ideas] Implementing a set of operation (+, /, - *) on dict consistent with linearAlgebrae In-Reply-To: References: Message-ID: <5BD8CD81.4020709@canterbury.ac.nz> julien tayon wrote: > like the + of [] could be the + of "RecordAlgebrae" If you're proposing to change the behaviour of '+' on the built-in list type, that's not going to happen. -- Greg From julien at tayon.net Tue Oct 30 18:28:54 2018 From: julien at tayon.net (julien tayon) Date: Tue, 30 Oct 2018 23:28:54 +0100 Subject: [Python-ideas] Implementing a set of operation (+, /, - *) on dict consistent with linearAlgebrae In-Reply-To: <5BD8CD81.4020709@canterbury.ac.nz> References: <5BD8CD81.4020709@canterbury.ac.nz> Message-ID: On Tue, 30 Oct 2018 at 22:33, Greg Ewing wrote: > julien tayon wrote: > > like the + of [] could be the + of "RecordAlgebrae" > > If you're proposing to change the behaviour of '+' on the > built-in list type, that's not going to happen. > > I dont suggest to change something that already exists and works (I am pretty conservative too, and expect stuff to not be broken by any changes) And all behaviours can coexists quite peacefully. "RecordAlgebra" In [5]: [2] + [2] Out[5]: [2, 2] In [6]: [2] * 2 Out[6]: [2, 2] In [7]: "a" + "a" Out[7]: 'aa' In [8]: "a" * 2 Out[8]: 'aa' (adding n times the same value is equal to multiplying by n // that is totally consistent to me) Mixed scenario : In [12]: a= mdict(a=[2], b='a') In [13]: a+a Out[14]: {'a': [2, 2], b='aa'} In [17]: a * 4 Out[17]: {'a': [2, 2, 2, 2], b='aaaa'} I propose the operators to be propagated, and any value to still follow its logic. LibearAlgebraic MutableMapping would be as algebraic as their values. No more. -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Tue Oct 30 18:53:20 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 30 Oct 2018 18:53:20 -0400 Subject: [Python-ideas] Implementing a set of operation (+, /, - *) on dict consistent with linearAlgebrae In-Reply-To: References: <5BD8CD81.4020709@canterbury.ac.nz> Message-ID: > In [12]: a= mdict(a=[2], b='a') > In [13]: a+a Aren't you reinventing the Counter type? >>> from collections import Counter >>> c = Counter(a=1,b=2) >>> c + c Counter({'b': 4, 'a': 2}) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Tue Oct 30 19:19:59 2018 From: mertz at gnosis.cx (David Mertz) Date: Tue, 30 Oct 2018 19:19:59 -0400 Subject: [Python-ideas] Implementing a set of operation (+, /, - *) on dict consistent with linearAlgebrae In-Reply-To: References: <5BD8CD81.4020709@canterbury.ac.nz> Message-ID: Counter doesn't QUITE do the same thing as this `mdict`. But it's pretty close. I think if .__add__() became a synonym for .update() that wouldn't break anything that currently works. But I'm probably wrong, and missing a case in my quick thought: >>> from collections import Counter >>> c = Counter(a=[2], b='a') >>> c.update(c) >>> c Counter({'a': [2, 2], 'b': 'aa'}) >>> c2 = Counter(a=1, b=2) >>> c2 + c2 Counter({'b': 4, 'a': 2}) >>> c2.update(c2) >>> c2 Counter({'b': 4, 'a': 2}) >>> c + c Traceback (most recent call last): File "", line 1, in c + c File "/anaconda3/lib/python3.6/collections/__init__.py", line 705, in __add__ if newcount > 0: TypeError: '>' not supported between instances of 'list' and 'int' On Tue, Oct 30, 2018 at 6:54 PM Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > In [12]: a= mdict(a=[2], b='a') > > In [13]: a+a > > Aren't you reinventing the Counter type? > > >>> from collections import Counter > >>> c = Counter(a=1,b=2) > >>> c + c > Counter({'b': 4, 'a': 2}) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Tue Oct 30 19:22:45 2018 From: mertz at gnosis.cx (David Mertz) Date: Tue, 30 Oct 2018 19:22:45 -0400 Subject: [Python-ideas] Implementing a set of operation (+, /, - *) on dict consistent with linearAlgebrae In-Reply-To: References: <5BD8CD81.4020709@canterbury.ac.nz> Message-ID: Actually, they are definitely different as in-place mutation versus returning a new Counter. But in some arithmetic way they look mostly the same. On Tue, Oct 30, 2018, 7:19 PM David Mertz Counter doesn't QUITE do the same thing as this `mdict`. But it's pretty > close. > > I think if .__add__() became a synonym for .update() that wouldn't break > anything that currently works. But I'm probably wrong, and missing a case > in my quick thought: > > >>> from collections import Counter > >>> c = Counter(a=[2], b='a') > >>> c.update(c) > >>> c > Counter({'a': [2, 2], 'b': 'aa'}) > >>> c2 = Counter(a=1, b=2) > >>> c2 + c2 > Counter({'b': 4, 'a': 2}) > >>> c2.update(c2) > >>> c2 > Counter({'b': 4, 'a': 2}) > >>> c + c > Traceback (most recent call last): > File "", line 1, in > c + c > File "/anaconda3/lib/python3.6/collections/__init__.py", line 705, in > __add__ > if newcount > 0: > TypeError: '>' not supported between instances of 'list' and 'int' > > On Tue, Oct 30, 2018 at 6:54 PM Alexander Belopolsky < > alexander.belopolsky at gmail.com> wrote: > >> > In [12]: a= mdict(a=[2], b='a') >> > In [13]: a+a >> >> Aren't you reinventing the Counter type? >> >> >>> from collections import Counter >> >>> c = Counter(a=1,b=2) >> >>> c + c >> Counter({'b': 4, 'a': 2}) >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Tue Oct 30 19:44:55 2018 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Wed, 31 Oct 2018 00:44:55 +0100 Subject: [Python-ideas] Add "default" kwarg to list.pop() Message-ID: Sorry in advance if this has been proposed in the past but I couldn't find anything on python-ideas: >>> l = [] >>> l.pop(default=1) 1 FWIW my use case consists in reading entries from /proc/diskstats where lines can have a variable number of fields depending on the kernel version: https://github.com/giampaolo/psutil/issues/1354#issuecomment-434495870 https://github.com/giampaolo/psutil/blob/d8b05151e65f9348aff9b58da977abd8cacb2127/psutil/_pslinux.py#L1068 As such it would be convenient to set missing fields to 0 as "reads = fields.pop(default=0)" instead of catching IndexError every time. Extra: for consistency set.pop() should probably have the same. Thoughts? -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From julien at tayon.net Tue Oct 30 20:22:28 2018 From: julien at tayon.net (julien tayon) Date: Wed, 31 Oct 2018 01:22:28 +0100 Subject: [Python-ideas] Implementing a set of operation (+, /, - *) on dict consistent with linearAlgebrae In-Reply-To: References: <5BD8CD81.4020709@canterbury.ac.nz> Message-ID: On Wed, 31 Oct 2018 at 00:20, David Mertz wrote: > Counter doesn't QUITE do the same thing as this `mdict`. But it's pretty > close. > > I think if .__add__() became a synonym for .update() that wouldn't break > anything that currently works. But I'm probably wrong, and missing a case > in my quick thought: > > My quick thoughts too is that it achieve coincidently Counter features as a subset of its features. I never noticed it, and both approaches seem consistent in their results (pfiou, close one since I did not thought of checking it) And .... if you add the trait to Counter .... you have the following results : >>> from collections import Counter > > >>> from archery.quiver import LinearAlgebrae >>> class ACounter(LinearAlgebrae, Counter): pass > >>> c = ACounter(a=[2], b='a') > >>> c.update(c) > >>> c > ACounter({'a': [2, 2], 'b': 'aa'}) (same) >>> c2 = ACounter(a=1, b=2) > >>> c2 + c2 > ACounter({'b': 4, 'a': 2}) > >>> c2.update(c2) > >>> c2 > ACounter({'b': 4, 'a': 2}) > >>> c2 + c2 > ACounter({'a': 4, 'b': 8}) >>> c2 + .5 * c2 ACounter({'a': 1.5, 'b': 3.0}) On Tue, Oct 30, 2018 at 6:54 PM Alexander Belopolsky < > alexander.belopolsky at gmail.com> wrote: > >> > In [12]: a= mdict(a=[2], b='a') >> > In [13]: a+a >> >> Aren't you reinventing the Counter type? >> >> nop. It is an unintended subset of the possibilities. I do have though >>> c2 / 2 Out[17]: ACounter({'a': 0.5, 'b': 1.0}) >>> c / 2 TypeError: can't multiply sequence by non-int of type 'float' And talking about Counter, by inheriting from the mixins of Vector (dot, abs, cos) we give it out of the box the cosine simlarities. Which given its wide use in textual indexation is pretty reassuring. It would also enable to normalize Counter (with value that supports truediv) easily by writing >>> class VCounter(LinearAlgebrae,Vector, Counter): pass >>> c2 = VCounter(a=1, b=2) >>> c2/abs(c2) Out[20]: VCounter({'a': 0.4472135954999579, 'b': 0.8944271909999159}) And since it is mixins it touches nothing of the MutableMapping class it relies on. It just gives behaviours associated with operators. (ofc c2.cos(c) willl normally raise a TypeError since it would have no sense) It really is a proof of concept of adding linear/vectorial algebrae to ANY kind of mutable mapping be it : dict, Counter, OrderedDict, defaultDict ... It only relies on what mutableMapping (from abc) offers and does its life with it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Oct 30 20:25:25 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 31 Oct 2018 02:25:25 +0200 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: References: Message-ID: 31.10.18 01:44, Giampaolo Rodola' ????: > Sorry in advance if this has been proposed in the past but I couldn't > find anything on python-ideas: > > >>> l = [] > >>> l.pop(default=1) > 1 > > FWIW my use case consists in reading entries from /proc/diskstats where > lines can have a variable number of fields depending on the kernel version: > https://github.com/giampaolo/psutil/issues/1354#issuecomment-434495870 > https://github.com/giampaolo/psutil/blob/d8b05151e65f9348aff9b58da977abd8cacb2127/psutil/_pslinux.py#L1068 > As such it would be convenient to set missing fields to 0 as "reads = > fields.pop(default=0)" instead of catching IndexError every time. Extra: > for consistency set.pop() should probably have the same. > > Thoughts? It is just l.pop() if l else default or (l or [default]).pop() From steve at pearwood.info Tue Oct 30 21:08:51 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 31 Oct 2018 12:08:51 +1100 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: References: Message-ID: <20181031010851.GC3817@ando.pearwood.info> On Wed, Oct 31, 2018 at 02:25:25AM +0200, Serhiy Storchaka wrote: > 31.10.18 01:44, Giampaolo Rodola' ????: > >Sorry in advance if this has been proposed in the past but I couldn't > >find anything on python-ideas: > > > > >>> l = [] > > >>> l.pop(default=1) > >1 [...] > It is just > > l.pop() if l else default It might *do* the same thing, but it doesn't communicate the programmer's intention as well. {}.pop('key', default) could be written using LBYL too, but the intention is much clearer given an explicit default argument. The only advantage of the "if l" version is that if the default is expensive to calculate, we can short-circuit it. > or > > (l or [default]).pop() That's clever, but it is also wasteful, building a single-item list only to immediately pop the item out of it and throw the list away. [steve at ando ~]$ python3.5 -m timeit -s "l = []" "l.pop() if l else None" 10000000 loops, best of 3: 0.0739 usec per loop [steve at ando ~]$ python3.5 -m timeit -s "l = []" "(l or [None]).pop()" 1000000 loops, best of 3: 0.421 usec per loop -- Steve From nicolas.rolin at tiime.fr Wed Oct 31 05:23:48 2018 From: nicolas.rolin at tiime.fr (Nicolas Rolin) Date: Wed, 31 Oct 2018 10:23:48 +0100 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: <20181031010851.GC3817@ando.pearwood.info> References: <20181031010851.GC3817@ando.pearwood.info> Message-ID: As a user I always found a bit disurbing that dict pop method have a default while list and set doesn't. While it is way more computationally easy to check wether a list or a set is empty that to check if a key is in a dict, it still create a signature difference for no real reason (having a default to a built-in in python is pretty standard). It would be nice if every built-in/method of built-in type that returns a value and raise in some case have access to a default instead of raise, and not having to check the doc to see if it supports a default. We could for exemple ask ourselves wether or not list.index should have a default, as it is a method that we explecitely excpect to return a value and might just raise instead. 2018-10-31 2:08 GMT+01:00 Steven D'Aprano : > On Wed, Oct 31, 2018 at 02:25:25AM +0200, Serhiy Storchaka wrote: > > 31.10.18 01:44, Giampaolo Rodola' ????: > > >Sorry in advance if this has been proposed in the past but I couldn't > > >find anything on python-ideas: > > > > > > >>> l = [] > > > >>> l.pop(default=1) > > >1 > [...] > > > It is just > > > > l.pop() if l else default > > It might *do* the same thing, but it doesn't communicate the > programmer's intention as well. > > {}.pop('key', default) could be written using LBYL too, but the > intention is much clearer given an explicit default argument. > > The only advantage of the "if l" version is that if the default is > expensive to calculate, we can short-circuit it. > > > > or > > > > (l or [default]).pop() > > That's clever, but it is also wasteful, building a single-item list only > to immediately pop the item out of it and throw the list away. > > [steve at ando ~]$ python3.5 -m timeit -s "l = []" "l.pop() if l else None" > 10000000 loops, best of 3: 0.0739 usec per loop > > [steve at ando ~]$ python3.5 -m timeit -s "l = []" "(l or [None]).pop()" > 1000000 loops, best of 3: 0.421 usec per loop > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- -- *Nicolas Rolin* | Data Scientist + 33 631992617 - nicolas.rolin at tiime.fr *15 rue Auber, **75009 Paris* *www.tiime.fr * -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.reiter at gmail.com Wed Oct 31 05:36:04 2018 From: ron.reiter at gmail.com (Ron Reiter) Date: Wed, 31 Oct 2018 11:36:04 +0200 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: References: <20181031010851.GC3817@ando.pearwood.info> Message-ID: I think l.pop() if l else None is good enough. I think it's pretty obvious that a developer means "Pop the list if the list is not empty, and return None if the list is empty ". - Ron [image: Facebook] [image: Twitter] [image: LinkedIn] On Wed, Oct 31, 2018 at 11:24 AM Nicolas Rolin wrote: > > As a user I always found a bit disurbing that dict pop method have a > default while list and set doesn't. > While it is way more computationally easy to check wether a list or a set > is empty that to check if a key is in a dict, it still create a signature > difference for no real reason (having a default to a built-in in python is > pretty standard). > It would be nice if every built-in/method of built-in type that returns a > value and raise in some case have access to a default instead of raise, and > not having to check the doc to see if it supports a default. > > We could for exemple ask ourselves wether or not list.index should have a > default, as it is a method that we explecitely excpect to return a value > and might just raise instead. > > 2018-10-31 2:08 GMT+01:00 Steven D'Aprano : > >> On Wed, Oct 31, 2018 at 02:25:25AM +0200, Serhiy Storchaka wrote: >> > 31.10.18 01:44, Giampaolo Rodola' ????: >> > >Sorry in advance if this has been proposed in the past but I couldn't >> > >find anything on python-ideas: >> > > >> > > >>> l = [] >> > > >>> l.pop(default=1) >> > >1 >> [...] >> >> > It is just >> > >> > l.pop() if l else default >> >> It might *do* the same thing, but it doesn't communicate the >> programmer's intention as well. >> >> {}.pop('key', default) could be written using LBYL too, but the >> intention is much clearer given an explicit default argument. >> >> The only advantage of the "if l" version is that if the default is >> expensive to calculate, we can short-circuit it. >> >> >> > or >> > >> > (l or [default]).pop() >> >> That's clever, but it is also wasteful, building a single-item list only >> to immediately pop the item out of it and throw the list away. >> >> [steve at ando ~]$ python3.5 -m timeit -s "l = []" "l.pop() if l else None" >> 10000000 loops, best of 3: 0.0739 usec per loop >> >> [steve at ando ~]$ python3.5 -m timeit -s "l = []" "(l or [None]).pop()" >> 1000000 loops, best of 3: 0.421 usec per loop >> >> >> >> -- >> Steve >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > > -- > *Nicolas Rolin* | Data Scientist > + 33 631992617 - nicolas.rolin at tiime.fr > > > *15 rue Auber, **75009 Paris* > *www.tiime.fr * > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertve92 at gmail.com Wed Oct 31 05:37:35 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Wed, 31 Oct 2018 10:37:35 +0100 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: References: <20181031010851.GC3817@ando.pearwood.info> Message-ID: I think the same way about set.pop, list.pop. About .index I agree adding default= would make sense but that's not exactly the same thing as the others. Do we have somewhere else a place where a linear search already accepts a default= kwarg ? def index(self, x): for i,y in enumerate(self): if x == y: return i raise ValueError We do have "next", the default version of .index is currently implentable as : next((i for i, y in enumerate(self) if x == y), -1) If I want -1 for default. Le mer. 31 oct. 2018 ? 10:23, Nicolas Rolin a ?crit : > > As a user I always found a bit disurbing that dict pop method have a > default while list and set doesn't. > While it is way more computationally easy to check wether a list or a set > is empty that to check if a key is in a dict, it still create a signature > difference for no real reason (having a default to a built-in in python is > pretty standard). > It would be nice if every built-in/method of built-in type that returns a > value and raise in some case have access to a default instead of raise, and > not having to check the doc to see if it supports a default. > > We could for exemple ask ourselves wether or not list.index should have a > default, as it is a method that we explecitely excpect to return a value > and might just raise instead. > > 2018-10-31 2:08 GMT+01:00 Steven D'Aprano : > >> On Wed, Oct 31, 2018 at 02:25:25AM +0200, Serhiy Storchaka wrote: >> > 31.10.18 01:44, Giampaolo Rodola' ????: >> > >Sorry in advance if this has been proposed in the past but I couldn't >> > >find anything on python-ideas: >> > > >> > > >>> l = [] >> > > >>> l.pop(default=1) >> > >1 >> [...] >> >> > It is just >> > >> > l.pop() if l else default >> >> It might *do* the same thing, but it doesn't communicate the >> programmer's intention as well. >> >> {}.pop('key', default) could be written using LBYL too, but the >> intention is much clearer given an explicit default argument. >> >> The only advantage of the "if l" version is that if the default is >> expensive to calculate, we can short-circuit it. >> >> >> > or >> > >> > (l or [default]).pop() >> >> That's clever, but it is also wasteful, building a single-item list only >> to immediately pop the item out of it and throw the list away. >> >> [steve at ando ~]$ python3.5 -m timeit -s "l = []" "l.pop() if l else None" >> 10000000 loops, best of 3: 0.0739 usec per loop >> >> [steve at ando ~]$ python3.5 -m timeit -s "l = []" "(l or [None]).pop()" >> 1000000 loops, best of 3: 0.421 usec per loop >> >> >> >> -- >> Steve >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > > -- > *Nicolas Rolin* | Data Scientist > + 33 631992617 - nicolas.rolin at tiime.fr > > > *15 rue Auber, **75009 Paris* > *www.tiime.fr * > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Oct 31 05:41:28 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 31 Oct 2018 20:41:28 +1100 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: References: <20181031010851.GC3817@ando.pearwood.info> Message-ID: On Wed, Oct 31, 2018 at 8:24 PM Nicolas Rolin wrote: > > > As a user I always found a bit disurbing that dict pop method have a default while list and set doesn't. > While it is way more computationally easy to check wether a list or a set is empty that to check if a key is in a dict, it still create a signature difference for no real reason (having a default to a built-in in python is pretty standard). > It would be nice if every built-in/method of built-in type that returns a value and raise in some case have access to a default instead of raise, and not having to check the doc to see if it supports a default. > https://www.python.org/dev/peps/pep-0463/ wants to say hi. ChrisA From steve at pearwood.info Wed Oct 31 06:14:07 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 31 Oct 2018 21:14:07 +1100 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: References: <20181031010851.GC3817@ando.pearwood.info> Message-ID: <20181031101407.GE3817@ando.pearwood.info> On Wed, Oct 31, 2018 at 08:41:28PM +1100, Chris Angelico wrote: > On Wed, Oct 31, 2018 at 8:24 PM Nicolas Rolin wrote: > > > > > > As a user I always found a bit disurbing that dict pop method have a default while list and set doesn't. > > While it is way more computationally easy to check wether a list or a set is empty that to check if a key is in a dict, it still create a signature difference for no real reason (having a default to a built-in in python is pretty standard). > > It would be nice if every built-in/method of built-in type that returns a value and raise in some case have access to a default instead of raise, and not having to check the doc to see if it supports a default. > > > > https://www.python.org/dev/peps/pep-0463/ wants to say hi. PEP 463 is too busy crying bitter tears in the corner after being rejected to say "Hi". I don't think this is the same thing: PEP 464 is about being able to catch arbitrary exceptions in arbitrary expressions in an ad-hoc manner. Nicholas' suggestion is about making a consistent strategy of avoiding the need to catch exceptions. I don't think I would agree with a broad rule "anything that raises can return a default value" -- I don't think it makes sense to have, let's say, len(obj, default=2). But on a case-by-case basis, it works for me. If this were Ruby, and we could monkey-patch the built-in types, my reasoning would go something like this: - if I have to write "L.pop() if L else default", I should write it in place; - the third time I write it in any one module, I ought to factor it out into a helper function; - the third time I write the helper function, I ought to just monkey-patch the built-in and be done with it. I haven't been counting, but I'm pretty sure I've written "get(L, index, default)" and "pop(L, index, default)" helpers at least three times now. Possibly even three times each :-) In Python, of course, we can't add methods to built-ins. But I think this functionality makes good sense, especially once we remember that pop takes an optional index. Spot the bug in this: L.pop(idx) if len(L) > idx else default -- Steve From rosuav at gmail.com Wed Oct 31 06:31:37 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 31 Oct 2018 21:31:37 +1100 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: <20181031101407.GE3817@ando.pearwood.info> References: <20181031010851.GC3817@ando.pearwood.info> <20181031101407.GE3817@ando.pearwood.info> Message-ID: On Wed, Oct 31, 2018 at 9:14 PM Steven D'Aprano wrote: > > On Wed, Oct 31, 2018 at 08:41:28PM +1100, Chris Angelico wrote: > > On Wed, Oct 31, 2018 at 8:24 PM Nicolas Rolin wrote: > > > > > > > > > As a user I always found a bit disurbing that dict pop method have a default while list and set doesn't. > > > While it is way more computationally easy to check wether a list or a set is empty that to check if a key is in a dict, it still create a signature difference for no real reason (having a default to a built-in in python is pretty standard). > > > It would be nice if every built-in/method of built-in type that returns a value and raise in some case have access to a default instead of raise, and not having to check the doc to see if it supports a default. > > > > > > > https://www.python.org/dev/peps/pep-0463/ wants to say hi. > > > PEP 463 is too busy crying bitter tears in the corner after being > rejected to say "Hi". > > I don't think this is the same thing: PEP 464 is about being able to > catch arbitrary exceptions in arbitrary expressions in an ad-hoc manner. > Nicholas' suggestion is about making a consistent strategy of avoiding > the need to catch exceptions. > > I don't think I would agree with a broad rule "anything that raises can > return a default value" -- I don't think it makes sense to have, let's > say, len(obj, default=2). But on a case-by-case basis, it works for me. And that's exactly why a broad rule of "anything that raises can be wrapped in a catcher" does make sense. Hence it may not be the same thing, but it is an alternative solution that doesn't require specifically angling for consistency. ChrisA From solipsis at pitrou.net Wed Oct 31 07:07:10 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 31 Oct 2018 12:07:10 +0100 Subject: [Python-ideas] Add "default" kwarg to list.pop() References: Message-ID: <20181031120710.0824d82d@fsol> On Wed, 31 Oct 2018 02:25:25 +0200 Serhiy Storchaka wrote: > 31.10.18 01:44, Giampaolo Rodola' ????: > > Sorry in advance if this has been proposed in the past but I couldn't > > find anything on python-ideas: > > > > >>> l = [] > > >>> l.pop(default=1) > > 1 > > > > FWIW my use case consists in reading entries from /proc/diskstats where > > lines can have a variable number of fields depending on the kernel version: > > https://github.com/giampaolo/psutil/issues/1354#issuecomment-434495870 > > https://github.com/giampaolo/psutil/blob/d8b05151e65f9348aff9b58da977abd8cacb2127/psutil/_pslinux.py#L1068 > > As such it would be convenient to set missing fields to 0 as "reads = > > fields.pop(default=0)" instead of catching IndexError every time. Extra: > > for consistency set.pop() should probably have the same. > > > > Thoughts? > > It is just > > l.pop() if l else default > > or > > (l or [default]).pop() l.pop(default=...) has the potential to be multi-thread-safe, while your alternatives haven't. Regards Antoine. From desmoulinmichel at gmail.com Wed Oct 31 07:11:23 2018 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Wed, 31 Oct 2018 12:11:23 +0100 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: References: Message-ID: +1 one this. And a list.get method has well, with a default value. Le 31/10/2018 ? 00:44, Giampaolo Rodola' a ?crit?: > Sorry in advance if this has been proposed in the past but I couldn't > find anything on python-ideas: > > ?>>> l = [] > ?>>> l.pop(default=1) > 1 > > FWIW my use case consists in reading entries from /proc/diskstats where > lines can have a variable number of fields depending on the kernel version: > https://github.com/giampaolo/psutil/issues/1354#issuecomment-434495870 > https://github.com/giampaolo/psutil/blob/d8b05151e65f9348aff9b58da977abd8cacb2127/psutil/_pslinux.py#L1068 > As such it would be convenient to set missing fields to 0 as "reads = > fields.pop(default=0)" instead of catching IndexError every time. Extra: > for consistency set.pop() should probably have the same. > > Thoughts? > > -- > Giampaolo - http://grodola.blogspot.com > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From solipsis at pitrou.net Wed Oct 31 07:08:13 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 31 Oct 2018 12:08:13 +0100 Subject: [Python-ideas] Add "default" kwarg to list.pop() References: Message-ID: <20181031120813.69b9a23b@fsol> On Wed, 31 Oct 2018 00:44:55 +0100 "Giampaolo Rodola'" wrote: > Sorry in advance if this has been proposed in the past but I couldn't find > anything on python-ideas: > > >>> l = [] > >>> l.pop(default=1) > 1 > > FWIW my use case consists in reading entries from /proc/diskstats where > lines can have a variable number of fields depending on the kernel version: > https://github.com/giampaolo/psutil/issues/1354#issuecomment-434495870 > https://github.com/giampaolo/psutil/blob/d8b05151e65f9348aff9b58da977abd8cacb2127/psutil/_pslinux.py#L1068 > As such it would be convenient to set missing fields to 0 as "reads = > fields.pop(default=0)" instead of catching IndexError every time. Extra: > for consistency set.pop() should probably have the same. > > Thoughts? +1 from me. dict.pop() already has an optional default. This is a straight-forward improvement to the API and no Python programmer will be surprised. Regards Antoine. From steve at pearwood.info Wed Oct 31 07:21:21 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 31 Oct 2018 22:21:21 +1100 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: References: <20181031010851.GC3817@ando.pearwood.info> <20181031101407.GE3817@ando.pearwood.info> Message-ID: <20181031112121.GF3817@ando.pearwood.info> On Wed, Oct 31, 2018 at 09:31:37PM +1100, Chris Angelico wrote: > > I don't think I would agree with a broad rule "anything that raises can > > return a default value" -- I don't think it makes sense to have, let's > > say, len(obj, default=2). But on a case-by-case basis, it works for me. > > And that's exactly why a broad rule of "anything that raises can be > wrapped in a catcher" does make sense. Hence it may not be the same > thing, but it is an alternative solution that doesn't require > specifically angling for consistency. True. I'm not arguing against the earlier PEP, I was in favour of it too. But even if we had exception-catching expressions, it would still make sense to add a default value to list.pop and perhaps a list.get method too. -- Steve From ericfahlgren at gmail.com Wed Oct 31 13:15:46 2018 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Wed, 31 Oct 2018 10:15:46 -0700 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: References: <20181031010851.GC3817@ando.pearwood.info> Message-ID: On Wed, Oct 31, 2018 at 2:42 AM Chris Angelico wrote: > > https://www.python.org/dev/peps/pep-0463/ wants to say hi. > > That was exactly my reaction, too, and usually is whenever one of these "add a default" or similar ideas pops up. 463 should be re-examined, I was very hopeful when it went through the wringer the first time, but alas it was not to be. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hemflit at gmail.com Wed Oct 31 13:35:09 2018 From: hemflit at gmail.com (=?UTF-8?Q?Vladimir_Filipovi=C4=87?=) Date: Wed, 31 Oct 2018 18:35:09 +0100 Subject: [Python-ideas] Implementing a set of operation (+, /, - *) on dict consistent with linearAlgebrae In-Reply-To: References: <5BD8CD81.4020709@canterbury.ac.nz> Message-ID: Julien, would I be correct if I summarized the changes you have in mind like this: for dictionaries d1 and d2, non-Mapping ("scalar") sc, binary operation ?, and unary operation ? (such as negation or abs()): d1 ? sc == {k: (v ? sc) for k, v in d1.items()} sc ? d1 == {k: (sc ? v) for k, v in d1.items()} ?(d1) == {k: ?(v) for k, v in d1.items()} d1 ? d2 == {k: (d1[k] ? d2[k]) for k in d1.keys() & d2.keys()} From mike at selik.org Wed Oct 31 13:35:58 2018 From: mike at selik.org (Michael Selik) Date: Wed, 31 Oct 2018 10:35:58 -0700 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: References: <20181031010851.GC3817@ando.pearwood.info> Message-ID: On Wed, Oct 31, 2018 at 10:17 AM Eric Fahlgren wrote: > On Wed, Oct 31, 2018 at 2:42 AM Chris Angelico wrote: > >> https://www.python.org/dev/peps/pep-0463/ wants to say hi. >> > > That was exactly my reaction, too, and usually is whenever one of these > "add a default" or similar ideas pops up. 463 should be re-examined, I was > very hopeful when it went through the wringer the first time, but alas it > was not to be. > I'll add my vote to re-examining PEP 463. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertve92 at gmail.com Wed Oct 31 15:23:49 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Wed, 31 Oct 2018 20:23:49 +0100 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: References: <20181031010851.GC3817@ando.pearwood.info> Message-ID: Oooh, PEP463, you're reason with I switch to LBYL or write studpid try except functions so much times. Oh and also the local assignement "let/where/statement" :D x = (y+1 where y = 3.14) because x = [y+1 for y in [3.14]][0] is an overkill and ugly. Should I write a PEP even though I know it's going to be rejected because the mailing list was not really into it ? Le mer. 31 oct. 2018 ? 18:37, Michael Selik a ?crit : > On Wed, Oct 31, 2018 at 10:17 AM Eric Fahlgren > wrote: > >> On Wed, Oct 31, 2018 at 2:42 AM Chris Angelico wrote: >> >>> https://www.python.org/dev/peps/pep-0463/ wants to say hi. >>> >> >> That was exactly my reaction, too, and usually is whenever one of these >> "add a default" or similar ideas pops up. 463 should be re-examined, I was >> very hopeful when it went through the wringer the first time, but alas it >> was not to be. >> > > I'll add my vote to re-examining PEP 463. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertve92 at gmail.com Wed Oct 31 15:29:09 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Wed, 31 Oct 2018 20:29:09 +0100 Subject: [Python-ideas] Implementing a set of operation (+, /, - *) on dict consistent with linearAlgebrae In-Reply-To: References: <5BD8CD81.4020709@canterbury.ac.nz> Message-ID: And with libraries like pip install funcoperators or pip install infix, you can even write it infix :D from funcoperators import infix @infix def superop(d1, sc): return {k: (v *superopp* sc) for k, v in d1.items()} print({'a': 8} *superop* 5) Le mer. 31 oct. 2018 ? 18:35, Vladimir Filipovi? a ?crit : > Julien, would I be correct if I summarized the changes you have in > mind like this: > > for dictionaries d1 and d2, > non-Mapping ("scalar") sc, > binary operation ?, > and unary operation ? (such as negation or abs()): > > d1 ? sc == {k: (v ? sc) for k, v in d1.items()} > sc ? d1 == {k: (sc ? v) for k, v in d1.items()} > ?(d1) == {k: ?(v) for k, v in d1.items()} > d1 ? d2 == {k: (d1[k] ? d2[k]) for k in d1.keys() & d2.keys()} > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Oct 31 15:29:43 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 1 Nov 2018 06:29:43 +1100 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: References: <20181031010851.GC3817@ando.pearwood.info> Message-ID: On Thu, Nov 1, 2018 at 6:25 AM Robert Vanden Eynde wrote: > > Oooh, PEP463, you're reason with I switch to LBYL or write studpid try except functions so much times. > > Oh and also the local assignement "let/where/statement" :D > > x = (y+1 where y = 3.14) because x = [y+1 for y in [3.14]][0] is an overkill and ugly. > > Should I write a PEP even though I know it's going to be rejected because the mailing list was not really into it ? > There's nothing wrong with writing PEPs that have a low likelihood of being accepted. In fact, some PEPs have specifically been written with the purpose of rejecting them - creating a sort of FAQ document ("this has been proposed many times, here's the arguments in favour of it, and it's rejected because X and Y"). That said, though, you may well not need to go to that effort. What is being asked for here (if I'm not misreading) is a relatively simple enhancement to a method on a built-in type (or a small handful of types). If that garners reasonable support, the next step wouldn't be a PEP, it'd just go straight to a tracker issue and a pull request. For myself, I'm +0 on adding default=. It'd be a minor convenience on a very small number of cases. But it wouldn't bother me at all so it wouldn't be a problem if it were to land. ChrisA From mike at selik.org Wed Oct 31 16:29:35 2018 From: mike at selik.org (Michael Selik) Date: Wed, 31 Oct 2018 13:29:35 -0700 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: References: <20181031010851.GC3817@ando.pearwood.info> Message-ID: On Wed, Oct 31, 2018 at 12:31 PM Chris Angelico wrote: > What is being asked for here (if I'm not misreading) is a relatively simple > enhancement to a method on a built-in type (or a small handful of > types). If that garners reasonable support, the next step wouldn't be > a PEP, it'd just go straight to a tracker issue and a pull request. > > For myself, I'm +0 on adding default=. It'd be a minor convenience on > a very small number of cases. But it wouldn't bother me at all so it > wouldn't be a problem if it were to land. > If the consensus is to add a default keyword parameter for the rest of the get/pop methods on built-in types, it'd be reasonable to write an addendum to PEP 463 that mentions what is being established as the Pythonic interface: When a method can raise IndexError or KeyError, the method should provide an optional default which suppresses the error. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertve92 at gmail.com Wed Oct 31 16:46:25 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Wed, 31 Oct 2018 21:46:25 +0100 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: References: <20181031010851.GC3817@ando.pearwood.info> Message-ID: > > > That said, though, you may well not need to go to that effort. What is > being asked for here (if I'm not misreading) is a relatively simple > enhancement to a method on a built-in type (or a small handful of > types). If that garners reasonable support, the next step wouldn't be > a PEP, it'd just go straight to a tracker issue and a pull request. > So, not all modification come from a PEP ? Where are the functionalities discussed for future python version ? Only on the list ? Is there a place where people or big projects can vote for functionalities ? Recently I saw on the 3.8 changelog "added math.dist(a,b)" and saw the discussion on the bug tracker. The last comment was "I don't understand why this was added, as the discussion was more in favor of supporting *args for math.hypot". Who selects which PR are merged ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Oct 31 17:29:19 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 1 Nov 2018 08:29:19 +1100 Subject: [Python-ideas] Add "default" kwarg to list.pop() In-Reply-To: References: <20181031010851.GC3817@ando.pearwood.info> Message-ID: On Thu, Nov 1, 2018 at 7:46 AM Robert Vanden Eynde wrote: >> >> >> That said, though, you may well not need to go to that effort. What is >> being asked for here (if I'm not misreading) is a relatively simple >> enhancement to a method on a built-in type (or a small handful of >> types). If that garners reasonable support, the next step wouldn't be >> a PEP, it'd just go straight to a tracker issue and a pull request. > > > So, not all modification come from a PEP ? Correct. Some more info here: https://www.python.org/dev/peps/pep-0001/#what-is-a-pep > Where are the functionalities discussed for future python version ? Only on the list ? A number of places. This list is for all manner of discussions, but doesn't have any authority to make actual decisions. If a proposal has decent support here, it will usually be taken then to python-dev and then the tracker, or sometimes straight to the tracker (if it's considered a trivial change). > Is there a place where people or big projects can vote for functionalities ? No. Voting is not a thing in Python. That might change now that Guido has stepped down, but even if it does, it isn't likely to be where big projects get to vote - it'll be a matter of the governing body voting on the change. What you can do is *express support* for a proposal, giving your reasons for such support, as a means of convincing the core devs to accept the plan. (Offering to write the code can remove a barrier to acceptance, but that's not the only barrier.) > Who selects which PR are merged ? The core devs. Exactly how a team of core devs can make unified decisions is a little up in the air at the moment, but broadly speaking, it's up to the people who have the power to push code to the main repository. Which, in turn, means that it's the core devs that you ultimately need to convince if you have something you want changed. Convincing people here on this list has no intrinsic meaning, but it's a great way to hash out the proposal, the objections to the proposal, and the counter-arguments, etc, etc. Which is pretty important :) ChrisA From allemang.d at gmail.com Wed Oct 31 22:52:44 2018 From: allemang.d at gmail.com (David Allemang) Date: Wed, 31 Oct 2018 22:52:44 -0400 Subject: [Python-ideas] Allow Context Managers to Support Suspended Execution Message-ID: I do not think there is currently a good way for Context Managers to support suspended execution, as in await or yield. Both of these instructions cause the interpreter to leave the with block, yet no indication of this (temporary) exit or subsequent re-entrance is given to the context manager. If the intent of a Context Manager is to say "no matter how this block is entered or exited, the context will be correctly maintained", then this needs to be possible. I would propose magic methods __suspend__ and __resume__ as companions to the existing __enter__ and __exit__ methods (and their async variants). __suspend__, if present, would be called upon suspending execution on an await or yield statement, and __resume__, if present, would be called when execution is resumed. If __suspend__ or __resume__ are not present then nothing should be done, so that the behavior of existing context managers is preserved. Here is an example demonstrating the issue with await: https://gist.github.com/allemangD/bba8dc2d059310623f752ebf65bb6cdc and one with yield: https://gist.github.com/allemangD/f2534f16d3a0c642c2cdc02c544e854f The context manager used is clearly not thread-safe, and I'm not actually sure how to approach a thread-safe implementation with the proposed __suspend__ and __resume__ - but I don't believe that introducing these new methods would create any issues that aren't already present with __enter__ and __exit__. It's worth noting that the context manager used in those examples is, essentially, identical contextlib's redirect_stdout and decimal's localcontext managers. Any context manager such as these which modify global state or the behavior of global functions would benefit from this. It may also make sense to, for example, have the __suspend__ method on file objects flush buffers without closing the file, similar to their current __exit__ behavior, but I'm unsure what impact this would have on performance. It is important, though, that yield and await not use __enter__ or __exit__, as not all context-managers are reusable. I'm unsure what the best term would be to describe this type of context, as the documentation for contextlib already gives a different definition for "reentrant" - I would then call them "suspendable" contexts. It would make sense to have an @suspendable decorator, probably in contextlib, to indicate that a context manager can use __enter__ and __exit__ methods rather than __suspend__ and __resume__. All it would need to do is define __suspend__ to call __enter__() and __resume__ to call __exit__(None, None, None). It is also important, since __suspend__ and __resume__ would be called after a context is entered but before it is exited, that __suspend__ not accept any parameters and that __resume__ not use its return value. __suspend__ could not be triggered by an exception, only by a yield or await, and __resume__ could not have its return value named with as. Thanks, David From klahnakoski at mozilla.com Wed Oct 31 22:54:28 2018 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Wed, 31 Oct 2018 22:54:28 -0400 Subject: [Python-ideas] Implementing a set of operation (+, /, - *) on dict consistent with linearAlgebrae In-Reply-To: References: Message-ID: Julien, Personally, I would be able to use the module you are proposing to accumulate arbitrarily-named measures. I can not think of a use case for division, but it would be nice for completion.? I have made my own library that implements a small part of what you propose [1]. I was looking through the pstats.py [2] source code; and I thought it could benefit from vector operations.? I have seen other code that collect measures have the same redundant pattern.? Maybe some fancy regex can identify other += code sequences that would benefit.? If you make a module, and show how it can simplify pstats.py, maybe you have a winner?? [1] "vector" addition? - https://github.com/klahnakoski/mo-dots/blob/dev/tests/test_dot.py#L610 [2] pstats.py source code - https://github.com/python/cpython/blob/3.7/Lib/pstats.py#L156 On 2018-10-30 11:31, julien tayon wrote: > Hello :) > > the idea is described here: > http://jul.github.io/cv/pres.html#printable > > Summary of the idea : > > Take a linear algebrae book, and implements all the rules as a TDD. > https://github.com/jul/archery/blob/master/consistent_algebrae.py > > make it works based on abstract base class and sets of Mixins. > https://archery.readthedocs.io/en/latest/ > > And see if we can make cos/__abs__/dot and if it gives naively the intended > results ? (spoiler: yes) > > Making it work with dict, and "other" dictionary like counter by using > ineritance > https://archery.readthedocs.io/en/latest/#advanced-usage > > My idea is : wouldn't it be nice if we introduced geometries as sets of > mixins for objects ? > (Hilbertian algebrae could be nice too, and we could make MutableMapping > behave like bra/kets). > > So I was proposing a soft discussion on : could we agree that it would be > nice to consider operation overloading as a whole set of behaviours that > could profit from being consistent in a categorized way ? (like the + of [] > could be the + of "RecordAlgebrae") > Meaning we could define sets of "expected behaviour consistent interaction > between operators" as we defined the abc and call them algebrae? > > I offer the LinearAlgebrae Mixins as a POC, and was thinking of creating a > unittest to qualify if an object is following the rules of linear algebrae. > > What are your opinions ? > I don't actually see a lot of use case except it was funny to build. But > maybe it can be of use. > > > Cordialement > > -- > Julien > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: