From steve at pearwood.info Sun May 1 00:28:25 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 01 May 2016 14:28:25 +1000 Subject: What should Python apps do when asked to show help? References: <5724253e$0$22142$c3e8da3$5496439d@news.astraweb.com> <20160430235132.GA83556@cskk.homeip.net> Message-ID: <572585eb$0$1605$c3e8da3$5496439d@news.astraweb.com> On Sun, 1 May 2016 09:51 am, cs at zip.com.au wrote: > Let me recite one of my favourite rules of thumb: > > If it can't be turned off, it's not a feature. - Karl Heuer My microwave oven has a safety lock which prevents the mechanism from operating (generating microwaves) while the door is open. Since I can't turn it off, it is not a feature! What if I want to stick my head in the microwave and cook it? *wink* Often-wanting-to-microwave-somebody-else's-head-ly y'rs, -- Steven From cs at zip.com.au Sun May 1 01:24:34 2016 From: cs at zip.com.au (cs at zip.com.au) Date: Sun, 1 May 2016 15:24:34 +1000 Subject: What should Python apps do when asked to show help? In-Reply-To: <1462074388.273747.594588777.5E705539@webmail.messagingengine.com> References: <1462074388.273747.594588777.5E705539@webmail.messagingengine.com> Message-ID: <20160501052434.GA98660@cskk.homeip.net> On 30Apr2016 23:46, Random832 wrote: >On Sat, Apr 30, 2016, at 22:30, Grant Edwards wrote: >> We don't want to use a PAGER variable to specify when we want a pager >> and when we don't want a pager. If we want a pager we append "| less" >> to the command. If we don't want a pager, we don't append that to the >> command. > >Setting PAGER=cat - permanently, in your profile, and never ever >changing it - gives this result, yet it's described by some people here >as "So I have to cripple my shell [...]?" What is crippled? Evidently >you're not "we", since this "crippled" state of affairs seems to be a >perfectly acceptable one to you. This is an interesting argument. Yes, PAGER=cat would make "man" also not page, and likely almost everything. And yet I am unwilling to do so. Why? On reflection, my personal problems with this approach are twofold: - I want $PAGER to specify my preferred pager when I do want a pager, so setting it to "cat" does not inform apps about my wishes - in an interactive shell, yes I can type "| less"; it seems painful (oh for the happy days of yore when this was spelt "|pg":-) I am going to try PAGER=cat as an experiment to seem how it feels. Now, I _do_ wish "man" to page for a number of reasons (a) my brain expects it (b) I like the nice colour highlighting of headings etc (c) manual pages and other _lengthy_ documents _are_ better paged than not. This last is in contrast to "help" messages, even lengthy usage messages: these are short enough that I feel they should not be paged, and if I come across a glaring exception i can always page them myself. Fortunately for me, I am already in the position of mucking with "man"; my despite for GNU info with its weird non-paging browser and the accompanying GNU manual pages which all too frequently say "man is defunct, go look in info - we don't bother with man" drove me to rewrite "man" to try to suck in info files into flat pagable text. So telling my shell this: man(){ PAGER=less command man ${1+"$@"}; } is no hardship. I'll give this a go and see how it feels. Cheers, Cameron Simpson From rosuav at gmail.com Sun May 1 02:44:11 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 1 May 2016 16:44:11 +1000 Subject: What should Python apps do when asked to show help? In-Reply-To: <20160501052434.GA98660@cskk.homeip.net> References: <1462074388.273747.594588777.5E705539@webmail.messagingengine.com> <20160501052434.GA98660@cskk.homeip.net> Message-ID: On Sun, May 1, 2016 at 3:24 PM, wrote: > Yes, PAGER=cat would make "man" also not page, and likely almost everything. > And yet I am unwilling to do so. Why? > > On reflection, my personal problems with this approach are twofold: > > - I want $PAGER to specify my preferred pager when I do want a pager, so > setting it to "cat" does not inform apps about my wishes So you expect the environment variable to say which of multiple pagers you might want, but only when you already want a pager. Okay. How is an app supposed to know whether or not to use a pager? How do you expect them to mindread? ChrisA From cs at zip.com.au Sun May 1 03:28:53 2016 From: cs at zip.com.au (cs at zip.com.au) Date: Sun, 1 May 2016 17:28:53 +1000 Subject: What should Python apps do when asked to show help? In-Reply-To: References: Message-ID: <20160501072853.GA37568@cskk.homeip.net> On 01May2016 16:44, Chris Angelico wrote: >On Sun, May 1, 2016 at 3:24 PM, wrote: >> Yes, PAGER=cat would make "man" also not page, and likely almost everything. >> And yet I am unwilling to do so. Why? >> >> On reflection, my personal problems with this approach are twofold: >> >> - I want $PAGER to specify my preferred pager when I do want a pager, so >> setting it to "cat" does not inform apps about my wishes > >So you expect the environment variable to say which of multiple pagers >you might want, but only when you already want a pager. Okay. How is >an app supposed to know whether or not to use a pager? How do you >expect them to mindread? I think for several of us, we do not expect the app to mindread. Don't page for short output! As the rest of my article remarks, I at least think "man" should page on the premise than manual pages will be long enough to benefit, as they should be. Aside: especially if one uses "less" and includes the -d and -F options in the $LESS envvar, which suppresses the warning about "dumb" terminals and autoquits if the file fits on the screen - these two provide most of the painfree behaviour for short outputs and embedded ttys at least. We could fork a separate discussion on making pagers more seamless, and terminal emulators with nice modes to reduce the need for pagers. Cheers, Cameron Simpson From alister.ware at ntlworld.com Sun May 1 05:28:53 2016 From: alister.ware at ntlworld.com (alister) Date: Sun, 01 May 2016 09:28:53 GMT Subject: What should Python apps do when asked to show help? Message-ID: On Sun, 01 May 2016 17:28:53 +1000, cs wrote: > On 01May2016 16:44, Chris Angelico wrote: >>On Sun, May 1, 2016 at 3:24 PM, wrote: >>> Yes, PAGER=cat would make "man" also not page, and likely almost >>> everything. >>> And yet I am unwilling to do so. Why? >>> >>> On reflection, my personal problems with this approach are twofold: >>> >>> - I want $PAGER to specify my preferred pager when I do want a pager, >>> so setting it to "cat" does not inform apps about my wishes >> >>So you expect the environment variable to say which of multiple pagers >>you might want, but only when you already want a pager. Okay. How is an >>app supposed to know whether or not to use a pager? How do you expect >>them to mindread? > > I think for several of us, we do not expect the app to mindread. Don't > page for short output! > > As the rest of my article remarks, I at least think "man" should page on > the premise than manual pages will be long enough to benefit, as they > should be. > > Aside: especially if one uses "less" and includes the -d and -F options > in the $LESS envvar, which suppresses the warning about "dumb" terminals > and autoquits if the file fits on the screen - these two provide most of > the painfree behaviour for short outputs and embedded ttys at least. > > We could fork a separate discussion on making pagers more seamless, and > terminal emulators with nice modes to reduce the need for pagers. > > Cheers, > Cameron Simpson all the discussion on the pager variable is interesting but it overlooks what I consider to be a very important lesson on program output. You have no way of knowing what the users output device is & have no right to dictate what that should be. alternative outputs for the command line could be. a teletype printer a text to speech reader a Braille terminal or a computer to (dead) parrot) interface which is why mot of us here all agree, just output the data, let the end users environment decide how to present it, that is the users choice not the programmers. -- Get in touch with your feelings of hostility against the dying light. -- Dylan Thomas [paraphrased periphrastically] From steve at pearwood.info Sun May 1 06:47:38 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 01 May 2016 20:47:38 +1000 Subject: What should Python apps do when asked to show help? References: <1462074388.273747.594588777.5E705539@webmail.messagingengine.com> <20160501052434.GA98660@cskk.homeip.net> Message-ID: <5725decb$0$1596$c3e8da3$5496439d@news.astraweb.com> On Sun, 1 May 2016 04:44 pm, Chris Angelico wrote: > On Sun, May 1, 2016 at 3:24 PM, wrote: >> Yes, PAGER=cat would make "man" also not page, and likely almost >> everything. And yet I am unwilling to do so. Why? >> >> On reflection, my personal problems with this approach are twofold: >> >> - I want $PAGER to specify my preferred pager when I do want a pager, so >> setting it to "cat" does not inform apps about my wishes > > So you expect the environment variable to say which of multiple pagers > you might want, but only when you already want a pager. Okay. How is > an app supposed to know whether or not to use a pager? How do you > expect them to mindread? Easy: if the READPAGERENVIRONVAR is set, then the application should read the PAGER environment variable, otherwise it should ignore it. -- Steven From steve at pearwood.info Sun May 1 06:55:04 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 01 May 2016 20:55:04 +1000 Subject: What should Python apps do when asked to show help? References: <20160501072853.GA37568@cskk.homeip.net> Message-ID: <5725e08a$0$1620$c3e8da3$5496439d@news.astraweb.com> On Sun, 1 May 2016 05:28 pm, cs at zip.com.au wrote: > On 01May2016 16:44, Chris Angelico wrote: >>So you expect the environment variable to say which of multiple pagers >>you might want, but only when you already want a pager. Okay. How is >>an app supposed to know whether or not to use a pager? How do you >>expect them to mindread? > > I think for several of us, we do not expect the app to mindread. Don't > page for short output! Is there an environment variable to tell the application what you consider "short", or should it read your mind? Personally, I'd rather use a pager for 3 lines than print 30 lines of help text directly to the console, but others may feel differently. -- Steven From rosuav at gmail.com Sun May 1 07:23:27 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 1 May 2016 21:23:27 +1000 Subject: What should Python apps do when asked to show help? In-Reply-To: <5725e08a$0$1620$c3e8da3$5496439d@news.astraweb.com> References: <20160501072853.GA37568@cskk.homeip.net> <5725e08a$0$1620$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sun, May 1, 2016 at 8:55 PM, Steven D'Aprano wrote: > On Sun, 1 May 2016 05:28 pm, cs at zip.com.au wrote: > >> On 01May2016 16:44, Chris Angelico wrote: > >>>So you expect the environment variable to say which of multiple pagers >>>you might want, but only when you already want a pager. Okay. How is >>>an app supposed to know whether or not to use a pager? How do you >>>expect them to mindread? >> >> I think for several of us, we do not expect the app to mindread. Don't >> page for short output! > > Is there an environment variable to tell the application what you > consider "short", or should it read your mind? How about $LINES? If it's less than that, it'll fit on one screen. Of course, that still won't be perfect, but it's a definite improvement over guessing. ChrisA From grant.b.edwards at gmail.com Sun May 1 09:53:39 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Sun, 1 May 2016 13:53:39 +0000 (UTC) Subject: What should Python apps do when asked to show help? References: <1462074388.273747.594588777.5E705539@webmail.messagingengine.com> <20160501052434.GA98660@cskk.homeip.net> Message-ID: On 2016-05-01, Chris Angelico wrote: > On Sun, May 1, 2016 at 3:24 PM, wrote: >> Yes, PAGER=cat would make "man" also not page, and likely almost everything. >> And yet I am unwilling to do so. Why? >> >> On reflection, my personal problems with this approach are twofold: >> >> - I want $PAGER to specify my preferred pager when I do want a pager, so >> setting it to "cat" does not inform apps about my wishes > > So you expect the environment variable to say which of multiple pagers > you might want, but only when you already want a pager. Yes! Just like EDITOR specifies which editor to use _when_ _you_ _want_ _to_ _use_ _an_ _editor_. It doesn't tell programs to invoke an editor all the time. > Okay. How is an app supposed to know whether or not to use a pager? Command line option. > How do you expect them to mindread? Nope, just recognize '-p' or somesuch. -- Grant From zljubisic at gmail.com Sun May 1 10:19:26 2016 From: zljubisic at gmail.com (zljubisic at gmail.com) Date: Sun, 1 May 2016 07:19:26 -0700 (PDT) Subject: Python3 html scraper that supports javascript Message-ID: <2a0c92ed-352d-455c-832d-c9a9438f318b@googlegroups.com> Hi, can you please recommend to me a python3 library that I can use for scrapping JS that works on windows as well as linux? Regards. From marko at pacujo.net Sun May 1 11:08:53 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Sun, 01 May 2016 18:08:53 +0300 Subject: What should Python apps do when asked to show help? References: <1462074388.273747.594588777.5E705539@webmail.messagingengine.com> <20160501052434.GA98660@cskk.homeip.net> Message-ID: <87r3dlolx6.fsf@elektro.pacujo.net> Grant Edwards : > On 2016-05-01, Chris Angelico wrote: >> Okay. How is an app supposed to know whether or not to use a pager? > Command line option. > >> How do you expect them to mindread? > Nope, just recognize '-p' or somesuch. In discussions like these, it would be important to draw from precedents. Are there commands that have such an option? I could only find: mysql --pager CMD which seems sensible but nothing like an industry standard. Personally, I wouldn't bother with builtin paging. Marko From grant.b.edwards at gmail.com Sun May 1 12:30:03 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Sun, 1 May 2016 16:30:03 +0000 (UTC) Subject: What should Python apps do when asked to show help? References: <1462074388.273747.594588777.5E705539@webmail.messagingengine.com> <20160501052434.GA98660@cskk.homeip.net> <87r3dlolx6.fsf@elektro.pacujo.net> Message-ID: On 2016-05-01, Marko Rauhamaa wrote: > Grant Edwards : > >> On 2016-05-01, Chris Angelico wrote: >>> Okay. How is an app supposed to know whether or not to use a pager? >> Command line option. >> >>> How do you expect them to mindread? >> Nope, just recognize '-p' or somesuch. > > In discussions like these, it would be important to draw from > precedents. Are there commands that have such an option? It's pretty rare. It is assumed that Unix uses can type " | less" if they want to view the output of a program with a pager. That's simpler and faster than spending time to try to figure out if and how you tell some particular application to invoke a pager for you. > I could only find: > > mysql --pager CMD > > which seems sensible but nothing like an industry standard. > > Personally, I wouldn't bother with builtin paging. I agree completely. Builtin paging is pretty much pointless -- but if you _are_ going to do, make it something that you invoke with a command line option. -- Grant From steve at pearwood.info Sun May 1 12:36:48 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 02 May 2016 02:36:48 +1000 Subject: What should Python apps do when asked to show help? References: <1462074388.273747.594588777.5E705539@webmail.messagingengine.com> <20160501052434.GA98660@cskk.homeip.net> <87r3dlolx6.fsf@elektro.pacujo.net> Message-ID: <572630a1$0$1595$c3e8da3$5496439d@news.astraweb.com> On Mon, 2 May 2016 02:30 am, Grant Edwards wrote: >> In discussions like these, it would be important to draw from >> precedents. Are there commands that have such an option? > > It's pretty rare. ?It is assumed that Unix uses can type " | less" Is nobody except me questioning the assumption that we're only talking about Unix users? -- Steven From bgailer at gmail.com Sun May 1 13:01:08 2016 From: bgailer at gmail.com (Bob Gailer) Date: Sun, 1 May 2016 13:01:08 -0400 Subject: Python3 html scraper that supports javascript In-Reply-To: <2a0c92ed-352d-455c-832d-c9a9438f318b@googlegroups.com> References: <2a0c92ed-352d-455c-832d-c9a9438f318b@googlegroups.com> Message-ID: On May 1, 2016 10:20 AM, wrote: > > Hi, > > can you please recommend to me a python3 library that I can use for scrapping JS I'm not sure what you mean by that. The tool I use is Splinter. Install it using pip. that works on windows as well as linux? From grant.b.edwards at gmail.com Sun May 1 13:04:06 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Sun, 1 May 2016 17:04:06 +0000 (UTC) Subject: What should Python apps do when asked to show help? References: <1462074388.273747.594588777.5E705539@webmail.messagingengine.com> <20160501052434.GA98660@cskk.homeip.net> <87r3dlolx6.fsf@elektro.pacujo.net> <572630a1$0$1595$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-01, Steven D'Aprano wrote: > On Mon, 2 May 2016 02:30 am, Grant Edwards wrote: > >>> In discussions like these, it would be important to draw from >>> precedents. Are there commands that have such an option? >> >> It's pretty rare. ?It is assumed that Unix uses can type " | less" > > Is nobody except me questioning the assumption that we're only > talking about Unix users? Didn't the OP specify that he was writing a command-line utility for Linux/Unix? Discussing command line operation for Windows or OS-X seems rather pointless. -- Grant From gheskett at wdtv.com Sun May 1 13:09:42 2016 From: gheskett at wdtv.com (Gene Heskett) Date: Sun, 1 May 2016 13:09:42 -0400 Subject: What should Python apps do when asked to show help? In-Reply-To: <572630a1$0$1595$c3e8da3$5496439d@news.astraweb.com> References: <1462074388.273747.594588777.5E705539@webmail.messagingengine.com> <572630a1$0$1595$c3e8da3$5496439d@news.astraweb.com> Message-ID: <201605011309.42971.gheskett@wdtv.com> On Sunday 01 May 2016 12:36:48 Steven D'Aprano wrote: > On Mon, 2 May 2016 02:30 am, Grant Edwards wrote: > >> In discussions like these, it would be important to draw from > >> precedents. Are there commands that have such an option? > > > > It's pretty rare. ?It is assumed that Unix uses can type " | less" > > Is nobody except me questioning the assumption that we're only talking > about Unix users? > linux, unix, mauche nichs. Are there others?> > > -- > Steven Cheers, Gene Heskett -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) Genes Web page From random832 at fastmail.com Sun May 1 13:49:22 2016 From: random832 at fastmail.com (Random832) Date: Sun, 01 May 2016 13:49:22 -0400 Subject: What should Python apps do when asked to show help? In-Reply-To: References: <1462074388.273747.594588777.5E705539@webmail.messagingengine.com> <20160501052434.GA98660@cskk.homeip.net> <87r3dlolx6.fsf@elektro.pacujo.net> <572630a1$0$1595$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1462124962.1982396.594925177.787078AD@webmail.messagingengine.com> On Sun, May 1, 2016, at 13:04, Grant Edwards wrote: > On 2016-05-01, Steven D'Aprano wrote: > > Is nobody except me questioning the assumption that we're only > > talking about Unix users? > > Didn't the OP specify that he was writing a command-line utility for > Linux/Unix? We've been talking about pydoc instead of the OP's program for a while now. From ethan at stoneleaf.us Sun May 1 13:54:26 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 01 May 2016 10:54:26 -0700 Subject: What should Python apps do when asked to show help? In-Reply-To: <572630a1$0$1595$c3e8da3$5496439d@news.astraweb.com> References: <1462074388.273747.594588777.5E705539@webmail.messagingengine.com> <20160501052434.GA98660@cskk.homeip.net> <87r3dlolx6.fsf@elektro.pacujo.net> <572630a1$0$1595$c3e8da3$5496439d@news.astraweb.com> Message-ID: <572642D2.7040405@stoneleaf.us> On 05/01/2016 09:36 AM, Steven D'Aprano wrote: > On Mon, 2 May 2016 02:30 am, Grant Edwards wrote: >> It's pretty rare. It is assumed that Unix uses can type " | less" > > Is nobody except me questioning the assumption that we're only talking about > Unix users? Even Windows has "more". -- ~Ethan~ From davidgshi at yahoo.co.uk Sun May 1 14:27:39 2016 From: davidgshi at yahoo.co.uk (David Shi) Date: Sun, 1 May 2016 18:27:39 +0000 (UTC) Subject: How to fill in abbreviation in one column based on state name in another column? References: <919174787.8635501.1462127259949.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <919174787.8635501.1462127259949.JavaMail.yahoo@mail.yahoo.com> Hello, I am back. ?Thank you very much for your positive response. I am trying to use Pandas apply to execute a lookup function, so that we can put abbreviation in a new column, in accordance to a state name in another column. Does anyone knows how to make this to work? Regards.DavidLook up functionstate_to_code = {"VERMONT": "VT", "GEORGIA": "GA", "IOWA": "IA"}#table['moa_state_name'] = map(lambda x: x.upper(), table['moa_state_name'])def convert_state(row): ? ?abbrev1 = ?state_to_code(table['moa_state_name']) #'aatest' ? ?if abbrev1: ? ? ? ? return abbrev1 ##state_to_code[abbrev[0]] ? ?return np.nan#print convert_state(table['moa_state_name']) table.insert(0, "abbrev", np.nan) table['abbrev'] = table.apply(convert_state, axis=1)print state_to_code['ARKANSAS'] From pierre.quentel at gmail.com Sun May 1 15:16:18 2016 From: pierre.quentel at gmail.com (Pierre Quentel) Date: Sun, 1 May 2016 12:16:18 -0700 (PDT) Subject: do_POST not working on http.server with python In-Reply-To: References: Message-ID: Le jeudi 28 avril 2016 10:36:27 UTC+2, Rahul Raghunath a ?crit?: > 0 > down vote > favorite > > > I'm trying to create a simple http server with basic GET and POST functionality. The program is supposed to GET requests by printing out a simple webpage that greets a user and askes how he would rather be greeted. When the user enters a greeting of his choice, the webpage should now greet him as he had chosen. > > While GET seems to be working fine, POST is not. I tried debugging by printing at every code execution and it seems to be getting stuck here: > > ctype, pdict = cgi.parse_header(self.headers.getheader('content-type')) > > I'll paste the code full code below, along with my terminal output. > > Code: > > from http.server import BaseHTTPRequestHandler, HTTPServer > import cgi > > > class webServerHandler(BaseHTTPRequestHandler): > > def do_GET(self): > try: > if self.path.endswith("/hello"): > self.send_response(200) > self.send_header('Content-type', 'text/html') > self.end_headers() > output = "" > output += "" > output += "

Hello!

" > output += '''

What would you like me to say?

''' > output += "" > self.wfile.write(output.encode(encoding = 'utf_8')) > print (output) > return > > if self.path.endswith("/hola"): > self.send_response(200) > self.send_header('Content-type', 'text/html') > self.end_headers() > output = "" > output += "" > output += "

¡ Hola !

" > output += '''

What would you like me to say?

''' > output += "" > self.wfile.write(output.encode(encoding = 'utf_8')) > print (output) > return > > except IOError: > self.send_error(404, 'File Not Found: %s' % self.path) > > def do_POST(self): > try: > self.send_response(201) > print("Sent response") > self.send_header('Content-type', 'text/html') > print("Sent headers") > self.end_headers() > print("Ended header") > ctype, pdict = cgi.parse_header(self.headers.getheader('content-type')) > print("Parsed headers") > if ctype == 'multipart/form-data': > fields = cgi.parse_multipart(self.rfile, pdict) > messagecontent = fields.get('message') > print("Receiver message content") > output = "" > output += "" > output += "

Okay, how about this:

" > output += "

%s

" % messagecontent[0] > output += '''

What would you like me to say?

''' > output += "" > print(output) > self.wfile.write(output.encode(encoding = 'utf_8')) > print ("Wrote through CGI") > except: > pass > > > def main(): > try: > port = 8080 > server = HTTPServer(('', port), webServerHandler) > print ("Web Server running on port", port) > server.serve_forever() > except KeyboardInterrupt: > print (" ^C entered, stopping web server....") > server.socket.close() > > if __name__ == '__main__': > main() > > Terminal Output: > > Web Server running on port 8080 > 127.0.0.1 - - [28/Apr/2016 13:28:59] "GET /hello HTTP/1.1" 200 - >

Hello!

What would you like me to say?

> 127.0.0.1 - - [28/Apr/2016 13:29:09] "POST /hello HTTP/1.1" 201 - > Sent response > Sent headers > Ended header > > As you can see, the POST function does not seem to go beyong the parse_header command. I cannot figure this out, and any help would be usefu! Hi, It's generally not considered good practise to silently ignore exceptions in a try/except where the except clause is just : except: pass If you remove the try/except in do_POST you will see this interesting error message : AttributeError: 'HTTPMessage' object has no attribute 'getheader' From cs at zip.com.au Sun May 1 18:23:39 2016 From: cs at zip.com.au (cs at zip.com.au) Date: Mon, 2 May 2016 08:23:39 +1000 Subject: What should Python apps do when asked to show help? In-Reply-To: <5725e08a$0$1620$c3e8da3$5496439d@news.astraweb.com> References: <5725e08a$0$1620$c3e8da3$5496439d@news.astraweb.com> Message-ID: <20160501222339.GA37734@cskk.homeip.net> On 01May2016 20:55, Steven D'Aprano wrote: >On Sun, 1 May 2016 05:28 pm, cs at zip.com.au wrote: >> On 01May2016 16:44, Chris Angelico wrote: >>>So you expect the environment variable to say which of multiple pagers >>>you might want, but only when you already want a pager. Okay. How is >>>an app supposed to know whether or not to use a pager? How do you >>>expect them to mindread? >> >> I think for several of us, we do not expect the app to mindread. Don't >> page for short output! > >Is there an environment variable to tell the application what you >consider "short", or should it read your mind? We're getting into matters of taste here. It shouldn't read my mind, but of course when it differs it shows bad taste! I am taking the line that usage and help messages should fall into the "short" category, both simply by their nature and also as a design/style criterion for program authors. Manuals, be they man pages or info or whatever, should be "long", with specification and ideally explainations for rationale and some examples. >Personally, I'd rather use a pager for 3 lines than print 30 lines of help >text directly to the console, but others may feel differently. And I am very much the opposite. ["foo --help"; "types next command; huh? I'm in a pager, not back at my prompt?"] However, with "less" configured to quit if the text fits on the screen (which is can usually determine by querying the terminal directly, no magic required), I get the best of both wolds, possibly to the point that I have rarely noticed that Python's help() pages. And I've got mixed feelings about git. It seems that "git help" and "git --help" produces sensible unpaged short help (42 lines of it, but that is ok to me). It is "git help " which runs "man git-subcommand", and that is somewhat defensible because most of the git subcommands have an outrageous number of options. (And it really does just invoke "man" (by default - that is also tunable I see); I can tell because it invokes my personal "man" command instead of the system one or some internally sourced text.) My constrast, Mercurial (hg) always produces unpaged output for "help" and "help ", and the "help " is succinct and fitting for a help text. There is a single large "man hg" for when you want the detailed manual. This approach is more to my liking. Cheers, Cameron Simpson From cs at zip.com.au Sun May 1 18:27:12 2016 From: cs at zip.com.au (cs at zip.com.au) Date: Mon, 2 May 2016 08:27:12 +1000 Subject: What should Python apps do when asked to show help? In-Reply-To: References: Message-ID: <20160501222712.GA18558@cskk.homeip.net> On 01May2016 21:23, Chris Angelico wrote: >On Sun, May 1, 2016 at 8:55 PM, Steven D'Aprano wrote: >> Is there an environment variable to tell the application what you >> consider "short", or should it read your mind? > >How about $LINES? If it's less than that, it'll fit on one screen. Of >course, that still won't be perfect, but it's a definite improvement >over guessing. On terminal emulators you can normally query the terminal directly for its current size (look at the output of "stty -a" for example), and pagers do. This is better than $LINES, which is really a convenience thing presented by some shells like bash and which won't magicly change if you resize your terminal until bash gets another look. If your pager can be told to autoquit if the output fits then this pain point (whatever your preference) can be largely obviated. Cheers, Cameron Simpson From cs at zip.com.au Sun May 1 18:30:45 2016 From: cs at zip.com.au (cs at zip.com.au) Date: Mon, 2 May 2016 08:30:45 +1000 Subject: What should Python apps do when asked to show help? In-Reply-To: References: Message-ID: <20160501223045.GA38957@cskk.homeip.net> On 01May2016 17:04, Grant Edwards wrote: >On 2016-05-01, Steven D'Aprano wrote: >> On Mon, 2 May 2016 02:30 am, Grant Edwards wrote: >> >>>> In discussions like these, it would be important to draw from >>>> precedents. Are there commands that have such an option? >>> >>> It's pretty rare. ?It is assumed that Unix uses can type " | less" >> >> Is nobody except me questioning the assumption that we're only >> talking about Unix users? > >Didn't the OP specify that he was writing a command-line utility for >Linux/Unix? > >Discussing command line operation for Windows or OS-X seems rather >pointless. OS-X _is_ UNIX. I spent almost all my time on this Mac in terminals. It is a very nice to use UNIX in many regards. Cheers, Cameron Simpson Mac OS X. Because making Unix user-friendly is easier than debugging Windows. - Mike Dawson, Macintosh Systems Administrator and Consultation. mdawson at mac.com http://herowars.onestop.net From steve at pearwood.info Sun May 1 21:48:35 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 02 May 2016 11:48:35 +1000 Subject: What should Python apps do when asked to show help? References: <1462074388.273747.594588777.5E705539@webmail.messagingengine.com> <20160501052434.GA98660@cskk.homeip.net> <87r3dlolx6.fsf@elektro.pacujo.net> <572630a1$0$1595$c3e8da3$5496439d@news.astraweb.com> Message-ID: <5726b1f3$0$1603$c3e8da3$5496439d@news.astraweb.com> On Mon, 2 May 2016 03:04 am, Grant Edwards wrote: > On 2016-05-01, Steven D'Aprano wrote: >> On Mon, 2 May 2016 02:30 am, Grant Edwards wrote: >> >>>> In discussions like these, it would be important to draw from >>>> precedents. Are there commands that have such an option? >>> >>> It's pretty rare. ?It is assumed that Unix uses can type " | less" >> >> Is nobody except me questioning the assumption that we're only >> talking about Unix users? > > Didn't the OP specify that he was writing a command-line utility for > Linux/Unix? *cough* I'm the OP, and no I didn't. Obviously I'm a Linux user myself, but I'm presumptuous enough to hope that when I release the utility publicly[1], others may find it of some small use. Including Windows users. [1] Real Soon Now. -- Steven From flebber.crue at gmail.com Sun May 1 23:17:52 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Sun, 1 May 2016 20:17:52 -0700 (PDT) Subject: Code Opinion - Enumerate Message-ID: <0a2a0652-85c2-4283-a3c4-3ca8bc4e5481@googlegroups.com> Looking at various Python implementations of Conway's game of life. I came across one on rosetta using defaultdict. http://rosettacode.org/wiki/Conway%27s_Game_of_Life#Python Just looking for your opinion on style would you write it like this continually calling range or would you use enumerate instead, or neither (something far better) ? import random from collections import defaultdict printdead, printlive = '-#' maxgenerations = 3 cellcount = 3,3 celltable = defaultdict(int, { (1, 2): 1, (1, 3): 1, (0, 3): 1, } ) # Only need to populate with the keys leading to life ## ## Start States ## # blinker u = universe = defaultdict(int) u[(1,0)], u[(1,1)], u[(1,2)] = 1,1,1 for i in range(maxgenerations): print "\nGeneration %3i:" % ( i, ) for row in range(cellcount[1]): print " ", ''.join(str(universe[(row,col)]) for col in range(cellcount[0])).replace( '0', printdead).replace('1', printlive) nextgeneration = defaultdict(int) for row in range(cellcount[1]): for col in range(cellcount[0]): nextgeneration[(row,col)] = celltable[ ( universe[(row,col)], -universe[(row,col)] + sum(universe[(r,c)] for r in range(row-1,row+2) for c in range(col-1, col+2) ) ) ] universe = nextgeneration Just finished watching ned batchelders talk and wondering how far I should take his advice. http://nedbatchelder.com/text/iter.html Thanks Sayth From rustompmody at gmail.com Sun May 1 23:31:57 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Sun, 1 May 2016 20:31:57 -0700 (PDT) Subject: How to fill in abbreviation in one column based on state name in another column? In-Reply-To: References: <919174787.8635501.1462127259949.JavaMail.yahoo.ref@mail.yahoo.com> <919174787.8635501.1462127259949.JavaMail.yahoo@mail.yahoo.com> Message-ID: Your code (below) is too garbled to be able to read On Monday, May 2, 2016 at 12:00:59 AM UTC+5:30, David Shi wrote: > Hello, I am back. ?Thank you very much for your positive response. > I am trying to use Pandas apply to execute a lookup function, so that we can put abbreviation in a new column, in accordance to a state name in another column. > Does anyone knows how to make this to work? > Regards.DavidLook up functionstate_to_code = {"VERMONT": "VT", "GEORGIA": "GA", "IOWA": "IA"}#table['moa_state_name'] = map(lambda x: x.upper(), table['moa_state_name'])def convert_state(row): ? ?abbrev1 = ?state_to_code(table['moa_state_name']) #'aatest' ? ?if abbrev1: ? ? ? ? return abbrev1 ##state_to_code[abbrev[0]] ? ?return np.nan#print convert_state(table['moa_state_name']) > table.insert(0, "abbrev", np.nan) > table['abbrev'] = table.apply(convert_state, axis=1)print state_to_code['ARKANSAS'] From flebber.crue at gmail.com Sun May 1 23:37:57 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Sun, 1 May 2016 20:37:57 -0700 (PDT) Subject: Code Opinion - Enumerate In-Reply-To: <0a2a0652-85c2-4283-a3c4-3ca8bc4e5481@googlegroups.com> References: <0a2a0652-85c2-4283-a3c4-3ca8bc4e5481@googlegroups.com> Message-ID: <257326e1-57b7-4c96-9b42-2b72d039704a@googlegroups.com> Also not using enumerate but no ugly for i range implementation this one from code review uses a generator on live cells only. http://codereview.stackexchange.com/a/108121/104381 def neighbors(cell): x, y = cell yield x - 1, y - 1 yield x , y - 1 yield x + 1, y - 1 yield x - 1, y yield x + 1, y yield x - 1, y + 1 yield x , y + 1 yield x + 1, y + 1 def apply_iteration(board): new_board = set([]) candidates = board.union(set(n for cell in board for n in neighbors(cell))) for cell in candidates: count = sum((n in board) for n in neighbors(cell)) if count == 3 or (count == 2 and cell in board): new_board.add(cell) return new_board if __name__ == "__main__": board = {(0,1), (1,2), (2,0), (2,1), (2,2)} number_of_iterations = 10 for _ in xrange(number_of_iterations): board = apply_iteration(board) print board Sayth From nospam at dfs.com Sun May 1 23:39:22 2016 From: nospam at dfs.com (DFS) Date: Sun, 1 May 2016 23:39:22 -0400 Subject: You gotta love a 2-line python solution Message-ID: To save a webpage to a file: ------------------------------------- 1. import urllib 2. urllib.urlretrieve("http://econpy.pythonanywhere.com /ex/001.html","D:\file.html") ------------------------------------- That's it! Coming from VB/A background, some of the stuff you can do with python - with ease - is amazing. VBScript version ------------------------------------------------------ 1. Option Explicit 2. Dim xmlHTTP, fso, fOut 3. Set xmlHTTP = CreateObject("MSXML2.serverXMLHTTP") 4. xmlHTTP.Open "GET", "http://econpy.pythonanywhere.com/ex/001.html" 5. xmlHTTP.Send 6. Set fso = CreateObject("Scripting.FileSystemObject") 7. Set fOut = fso.CreateTextFile("D:\file.html", True) 8. fOut.WriteLine xmlHTTP.ResponseText 9. fOut.Close 10. Set fOut = Nothing 11. Set fso = Nothing 12. Set xmlHTTP = Nothing ------------------------------------------------------ Technically, that VBS will run with just lines 3-9, but that's still 6 lines of code vs 2 for python. From nospam at dfs.com Mon May 2 00:06:54 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 00:06:54 -0400 Subject: Fastest way to retrieve and write html contents to file Message-ID: I posted a little while ago about how short the python code was: ------------------------------------- 1. import urllib 2. urllib.urlretrieve(webpage, filename) ------------------------------------- Which is very sweet compared to the VBScript version: ------------------------------------------------------ 1. Option Explicit 2. Dim xmlHTTP, fso, fOut 3. Set xmlHTTP = CreateObject("MSXML2.serverXMLHTTP") 4. xmlHTTP.Open "GET", webpage 5. xmlHTTP.Send 6. Set fso = CreateObject("Scripting.FileSystemObject") 7. Set fOut = fso.CreateTextFile(filename, True) 8. fOut.WriteLine xmlHTTP.ResponseText 9. fOut.Close 10. Set fOut = Nothing 11. Set fso = Nothing 12. Set xmlHTTP = Nothing ------------------------------------------------------ Then I tested them in loops - the VBScript is MUCH faster: 0.44 for 10 iterations, vs 0.88 for python. webpage = 'http://econpy.pythonanywhere.com/ex/001.html' So I tried: --------------------------- import urllib2 r = urllib2.urlopen(webpage) f = open(filename,"w") f.write(r.read()) f.close --------------------------- and --------------------------- import requests r = requests.get(webpage) f = open(filename,"w") f.write(r.text) f.close --------------------------- and --------------------------------- import pycurl with open(filename, 'wb') as f: c = pycurl.Curl() c.setopt(c.URL, webpage) c.setopt(c.WRITEDATA, f) c.perform() c.close() --------------------------------- urllib2 and requests were about the same speed as urllib.urlretrieve, while pycurl was significantly slower (1.2 seconds). I'm running Win 8.1. python 2.7.11 32-bit. I know it's asking a lot, but is there a really fast AND really short python solution for this simple thing? Thanks! From me+python at ixokai.io Mon May 2 00:31:41 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sun, 01 May 2016 21:31:41 -0700 Subject: You gotta love a 2-line python solution In-Reply-To: References: Message-ID: <1462163501.1158885.595255233.34023BE4@webmail.messagingengine.com> On Sun, May 1, 2016, at 08:39 PM, DFS wrote: > To save a webpage to a file: > ------------------------------------- > 1. import urllib > 2. urllib.urlretrieve("http://econpy.pythonanywhere.com > /ex/001.html","D:\file.html") > ------------------------------------- Note, for paths on windows you really want to use a rawstring. Ie, r"D:\file.html". -- Stephen Hansen m e @ i x o k a i . i o From me+python at ixokai.io Mon May 2 00:34:31 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sun, 01 May 2016 21:34:31 -0700 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: Message-ID: <1462163671.1159335.595256017.204863FE@webmail.messagingengine.com> On Sun, May 1, 2016, at 09:06 PM, DFS wrote: > Then I tested them in loops - the VBScript is MUCH faster: 0.44 for 10 > iterations, vs 0.88 for python. ... > I know it's asking a lot, but is there a really fast AND really short > python solution for this simple thing? 0.88 is not fast enough for you? That's less then a second. -- Stephen Hansen m e @ i x o k a i . i o From rosuav at gmail.com Mon May 2 00:40:31 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 May 2016 14:40:31 +1000 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: <1462163671.1159335.595256017.204863FE@webmail.messagingengine.com> References: <1462163671.1159335.595256017.204863FE@webmail.messagingengine.com> Message-ID: On Mon, May 2, 2016 at 2:34 PM, Stephen Hansen wrote: > On Sun, May 1, 2016, at 09:06 PM, DFS wrote: >> Then I tested them in loops - the VBScript is MUCH faster: 0.44 for 10 >> iterations, vs 0.88 for python. > ... >> I know it's asking a lot, but is there a really fast AND really short >> python solution for this simple thing? > > 0.88 is not fast enough for you? That's less then a second. Also, this is timings of network and disk operations. Unless something pathological is happening, the language used won't make any difference. ChrisA From ben+python at benfinney.id.au Mon May 2 00:49:57 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 02 May 2016 14:49:57 +1000 Subject: Fastest way to retrieve and write html contents to file References: Message-ID: <85vb2xgj2i.fsf@benfinney.id.au> DFS writes: > Then I tested them in loops - the VBScript is MUCH faster: 0.44 for 10 > iterations, vs 0.88 for python. > > [?] > > urllib2 and requests were about the same speed as urllib.urlretrieve, > while pycurl was significantly slower (1.2 seconds). Network access is notoriously erratic in its timing. The program, and the machine on which it runs, is subject to a great many external effects once the request is sent ? effects which will significantly alter the delay before a response is completed. How have you controlled for the wide variability in the duration, for even a given request by the *same code on the same machine*, at different points in time? One simple way to do that: Run the exact same test many times (say, 10?000 or so) on the same machine, and then compute the average of all the durations. Do the same for each different program, and then you may have more meaningfully comparable measurements. -- \ ?We are no more free to believe whatever we want about God than | `\ we are free to adopt unjustified beliefs about science or | _o__) history [?].? ?Sam Harris, _The End of Faith_, 2004 | Ben Finney From nospam at dfs.com Mon May 2 00:50:51 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 00:50:51 -0400 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <1462163671.1159335.595256017.204863FE@webmail.messagingengine.com> Message-ID: On 5/2/2016 12:40 AM, Chris Angelico wrote: > On Mon, May 2, 2016 at 2:34 PM, Stephen Hansen wrote: >> On Sun, May 1, 2016, at 09:06 PM, DFS wrote: >>> Then I tested them in loops - the VBScript is MUCH faster: 0.44 for 10 >>> iterations, vs 0.88 for python. >> ... >>> I know it's asking a lot, but is there a really fast AND really short >>> python solution for this simple thing? >> >> 0.88 is not fast enough for you? That's less then a second. > > Also, this is timings of network and disk operations. Unless something > pathological is happening, the language used won't make any > difference. > > ChrisA Unfortunately, the VBScript is twice as fast as any python method. From nospam at dfs.com Mon May 2 00:51:27 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 00:51:27 -0400 Subject: You gotta love a 2-line python solution In-Reply-To: References: <1462163501.1158885.595255233.34023BE4@webmail.messagingengine.com> Message-ID: On 5/2/2016 12:31 AM, Stephen Hansen wrote: > On Sun, May 1, 2016, at 08:39 PM, DFS wrote: >> To save a webpage to a file: >> ------------------------------------- >> 1. import urllib >> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com >> /ex/001.html","D:\file.html") >> ------------------------------------- > > Note, for paths on windows you really want to use a rawstring. Ie, > r"D:\file.html". Thanks. I actually use "D:\\file.html" in my code. From rosuav at gmail.com Mon May 2 00:53:59 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 May 2016 14:53:59 +1000 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: <85vb2xgj2i.fsf@benfinney.id.au> References: <85vb2xgj2i.fsf@benfinney.id.au> Message-ID: On Mon, May 2, 2016 at 2:49 PM, Ben Finney wrote: > One simple way to do that: Run the exact same test many times (say, > 10?000 or so) on the same machine, and then compute the average of all > the durations. > > Do the same for each different program, and then you may have more > meaningfully comparable measurements. And also find the minimum and maximum durations, too. Averages don't always tell the whole story. ChrisA From me+python at ixokai.io Mon May 2 01:00:32 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sun, 01 May 2016 22:00:32 -0700 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <1462163671.1159335.595256017.204863FE@webmail.messagingengine.com> Message-ID: <1462165232.1164853.595266801.5DB17C45@webmail.messagingengine.com> On Sun, May 1, 2016, at 09:50 PM, DFS wrote: > On 5/2/2016 12:40 AM, Chris Angelico wrote: > > On Mon, May 2, 2016 at 2:34 PM, Stephen Hansen wrote: > >> On Sun, May 1, 2016, at 09:06 PM, DFS wrote: > >>> Then I tested them in loops - the VBScript is MUCH faster: 0.44 for 10 > >>> iterations, vs 0.88 for python. > >> ... > >>> I know it's asking a lot, but is there a really fast AND really short > >>> python solution for this simple thing? > >> > >> 0.88 is not fast enough for you? That's less then a second. > > > > Also, this is timings of network and disk operations. Unless something > > pathological is happening, the language used won't make any > > difference. > > > > ChrisA > > > Unfortunately, the VBScript is twice as fast as any python method. And 0.2 is twice as fast as 0.1. When you have two small numbers, 'twice as fast' isn't particularly meaningful as a metric. -- Stephen Hansen m e @ i x o k a i . i o From nospam at dfs.com Mon May 2 01:00:38 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 01:00:38 -0400 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> Message-ID: On 5/2/2016 12:49 AM, Ben Finney wrote: > DFS writes: > >> Then I tested them in loops - the VBScript is MUCH faster: 0.44 for 10 >> iterations, vs 0.88 for python. >> >> [?] >> >> urllib2 and requests were about the same speed as urllib.urlretrieve, >> while pycurl was significantly slower (1.2 seconds). > > Network access is notoriously erratic in its timing. The program, and > the machine on which it runs, is subject to a great many external > effects once the request is sent ? effects which will significantly > alter the delay before a response is completed. > > How have you controlled for the wide variability in the duration, for > even a given request by the *same code on the same machine*, at > different points in time? > > One simple way to do that: Run the exact same test many times (say, > 10?000 or so) on the same machine, and then compute the average of all > the durations. > > Do the same for each different program, and then you may have more > meaningfully comparable measurements. I tried the 10-loop test several times with all versions. The results were 100% consistent: VBSCript xmlHTTP was always 2x faster than any python method. From me+python at ixokai.io Mon May 2 01:02:01 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sun, 01 May 2016 22:02:01 -0700 Subject: You gotta love a 2-line python solution In-Reply-To: References: <1462163501.1158885.595255233.34023BE4@webmail.messagingengine.com> Message-ID: <1462165321.1165390.595267481.3403F202@webmail.messagingengine.com> On Sun, May 1, 2016, at 09:51 PM, DFS wrote: > On 5/2/2016 12:31 AM, Stephen Hansen wrote: > > On Sun, May 1, 2016, at 08:39 PM, DFS wrote: > >> To save a webpage to a file: > >> ------------------------------------- > >> 1. import urllib > >> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com > >> /ex/001.html","D:\file.html") > >> ------------------------------------- > > > > Note, for paths on windows you really want to use a rawstring. Ie, > > r"D:\file.html". > > > Thanks. > > I actually use "D:\\file.html" in my code. Or you can do that. But the whole point of raw strings is not having to escape slashes :) -- Stephen Hansen m e @ i x o k a i . i o From nospam at dfs.com Mon May 2 01:04:23 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 01:04:23 -0400 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <1462163671.1159335.595256017.204863FE@webmail.messagingengine.com> <1462165232.1164853.595266801.5DB17C45@webmail.messagingengine.com> Message-ID: On 5/2/2016 1:00 AM, Stephen Hansen wrote: > On Sun, May 1, 2016, at 09:50 PM, DFS wrote: >> On 5/2/2016 12:40 AM, Chris Angelico wrote: >>> On Mon, May 2, 2016 at 2:34 PM, Stephen Hansen wrote: >>>> On Sun, May 1, 2016, at 09:06 PM, DFS wrote: >>>>> Then I tested them in loops - the VBScript is MUCH faster: 0.44 for 10 >>>>> iterations, vs 0.88 for python. >>>> ... >>>>> I know it's asking a lot, but is there a really fast AND really short >>>>> python solution for this simple thing? >>>> >>>> 0.88 is not fast enough for you? That's less then a second. >>> >>> Also, this is timings of network and disk operations. Unless something >>> pathological is happening, the language used won't make any >>> difference. >>> >>> ChrisA >> >> >> Unfortunately, the VBScript is twice as fast as any python method. > > And 0.2 is twice as fast as 0.1. When you have two small numbers, 'twice > as fast' isn't particularly meaningful as a metric. 0.2 is half as fast as 0.1, here. And two small numbers turn into bigger numbers when the webpage is big, and soon the download time differences are measured in minutes, not half a second. So, any ideas? From nospam at dfs.com Mon May 2 01:08:24 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 01:08:24 -0400 Subject: You gotta love a 2-line python solution In-Reply-To: References: <1462163501.1158885.595255233.34023BE4@webmail.messagingengine.com> <1462165321.1165390.595267481.3403F202@webmail.messagingengine.com> Message-ID: On 5/2/2016 1:02 AM, Stephen Hansen wrote: > On Sun, May 1, 2016, at 09:51 PM, DFS wrote: >> On 5/2/2016 12:31 AM, Stephen Hansen wrote: >>> On Sun, May 1, 2016, at 08:39 PM, DFS wrote: >>>> To save a webpage to a file: >>>> ------------------------------------- >>>> 1. import urllib >>>> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com >>>> /ex/001.html","D:\file.html") >>>> ------------------------------------- >>> >>> Note, for paths on windows you really want to use a rawstring. Ie, >>> r"D:\file.html". >>> >> Thanks. >> >> I actually use "D:\\file.html" in my code. > > Or you can do that. But the whole point of raw strings is not having to > escape slashes :) Nice. Where/how else is 'r' used? I'm new to python, but I learned that one the hard way. I was using "D\testfile.txt" for something, and my code kept failing. Took me a while to figure it out. I tried various letters after the slash. I finally stumbled across the escape slashes in the docs somewhere. From rosuav at gmail.com Mon May 2 01:12:22 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 May 2016 15:12:22 +1000 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <1462163671.1159335.595256017.204863FE@webmail.messagingengine.com> <1462165232.1164853.595266801.5DB17C45@webmail.messagingengine.com> Message-ID: On Mon, May 2, 2016 at 3:04 PM, DFS wrote: > And two small numbers turn into bigger numbers when the webpage is big, and > soon the download time differences are measured in minutes, not half a > second. > > So, any ideas? So, measure with bigger web pages, and find out whether it's really a 2:1 ratio or a half-second difference. When download times are measured in minutes, a half second difference is insignificant. Extrapolating is dangerous. https://xkcd.com/605/ ChrisA From me+python at ixokai.io Mon May 2 01:15:36 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sun, 01 May 2016 22:15:36 -0700 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> Message-ID: <1462166136.1167243.595273897.291B0865@webmail.messagingengine.com> On Sun, May 1, 2016, at 10:00 PM, DFS wrote: > I tried the 10-loop test several times with all versions. Also how, _exactly_, are you testing this? C:\Python27>python -m timeit "filename='C:\\test.txt'; webpage='http://econpy.pythonanywhere.com/ex/001.html'; import urllib2; r = urllib2.urlopen(webpage); f = open(filename, 'w'); f.write(r.read()); f.close();" 10 loops, best of 3: 175 msec per loop That's a whole lot less the 0.88secs. -- Stephen Hansen m e @ i x o k a i . i o From me+python at ixokai.io Mon May 2 01:17:41 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sun, 01 May 2016 22:17:41 -0700 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <1462163671.1159335.595256017.204863FE@webmail.messagingengine.com> <1462165232.1164853.595266801.5DB17C45@webmail.messagingengine.com> Message-ID: <1462166261.1167799.595274273.1190901B@webmail.messagingengine.com> On Sun, May 1, 2016, at 10:04 PM, DFS wrote: > And two small numbers turn into bigger numbers when the webpage is big, > and soon the download time differences are measured in minutes, not half > a second. Are you sure of that? Have you determined that the time is not a constant overhead verses that the time is directly relational to the size of the page? If so, how have you determined that? You aren't showing how you're testing. 0.4s difference is meaningless to me, if its a constant overhead. If its twice as slow for a 1 meg file, then you might have an issue. Maybe. You haven't shown that. -- Stephen Hansen m e @ i x o k a i . i o From me+python at ixokai.io Mon May 2 01:21:18 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sun, 01 May 2016 22:21:18 -0700 Subject: You gotta love a 2-line python solution In-Reply-To: References: <1462163501.1158885.595255233.34023BE4@webmail.messagingengine.com> <1462165321.1165390.595267481.3403F202@webmail.messagingengine.com> Message-ID: <1462166478.1168179.595275257.0A4C1AD8@webmail.messagingengine.com> On Sun, May 1, 2016, at 10:08 PM, DFS wrote: > On 5/2/2016 1:02 AM, Stephen Hansen wrote: > >> I actually use "D:\\file.html" in my code. > > > > Or you can do that. But the whole point of raw strings is not having to > > escape slashes :) > > > Nice. Where/how else is 'r' used? Raw strings are primarily used A) for windows paths, and more universally, B) for regular expressions. But in theory they're useful anywhere you have static/literal data that might include backslashes where you don't actually intend to use any escape characters. -- Stephen Hansen m e @ i x o k a i . i o From nospam at dfs.com Mon May 2 01:23:48 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 01:23:48 -0400 Subject: You gotta love a 2-line python solution In-Reply-To: References: <1462163501.1158885.595255233.34023BE4@webmail.messagingengine.com> <1462165321.1165390.595267481.3403F202@webmail.messagingengine.com> Message-ID: On 5/2/2016 1:02 AM, Stephen Hansen wrote: > On Sun, May 1, 2016, at 09:51 PM, DFS wrote: >> On 5/2/2016 12:31 AM, Stephen Hansen wrote: >>> On Sun, May 1, 2016, at 08:39 PM, DFS wrote: >>>> To save a webpage to a file: >>>> ------------------------------------- >>>> 1. import urllib >>>> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com >>>> /ex/001.html","D:\file.html") >>>> ------------------------------------- >>> >>> Note, for paths on windows you really want to use a rawstring. Ie, >>> r"D:\file.html". >>> >> Thanks. >> >> I actually use "D:\\file.html" in my code. > > Or you can do that. But the whole point of raw strings is not having to > escape slashes :) Trying the rawstring thing (say it fast 3x): webpage = "http://econpy.pythonanywhere.com/ex/001.html" ---------------------------------------------------- webfile = "D:\\econpy001.html" urllib.urlretrieve(webpage,webfile) WORKS ---------------------------------------------------- webfile = "rD:\econpy001.html" urllib.urlretrieve(webpage,webfile) FAILS ---------------------------------------------------- webfile = "D:\econpy001.html" urllib.urlretrieve(webpage,"r" + webfile) FAILS ---------------------------------------------------- webfile = "D:\econpy001.html" urllib.urlretrieve(webpage,"r" + "" + webfile + "") FAILS ---------------------------------------------------- The FAILs throw: Traceback (most recent call last): File "webscraper.py", line 54, in urllib.urlretrieve(webpage,webfile) File "D:\development\python\python_2.7.11\lib\urllib.py", line 98, in urlretrieve return opener.retrieve(url, filename, reporthook, data) File "D:\development\python\python_2.7.11\lib\urllib.py", line 249, in retrieve tfp = open(filename, 'wb') IOError: [Errno 22] invalid mode ('wb') or filename: 'rD:\\econpy001.html' ---------------------------------------------------- What am I doing wrong? From me+python at ixokai.io Mon May 2 01:24:58 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sun, 01 May 2016 22:24:58 -0700 Subject: Code Opinion - Enumerate In-Reply-To: <0a2a0652-85c2-4283-a3c4-3ca8bc4e5481@googlegroups.com> References: <0a2a0652-85c2-4283-a3c4-3ca8bc4e5481@googlegroups.com> Message-ID: <1462166698.1168771.595278561.2A184D4B@webmail.messagingengine.com> On Sun, May 1, 2016, at 08:17 PM, Sayth Renshaw wrote: > Just looking for your opinion on style would you write it like this > continually calling range or would you use enumerate instead, or neither > (something far better) ? I can't comment on your specific code because there's too much noise to it, but in general: Using enumerate increases readability, and I use it whenever the idiom: for index, item in enumerate(thing): ... is used. Enumerate is your friend. Hug it. -- Stephen Hansen m e @ i x o k a i . i o From me+python at ixokai.io Mon May 2 01:37:04 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sun, 01 May 2016 22:37:04 -0700 Subject: You gotta love a 2-line python solution In-Reply-To: References: <1462163501.1158885.595255233.34023BE4@webmail.messagingengine.com> <1462165321.1165390.595267481.3403F202@webmail.messagingengine.com> Message-ID: <1462167424.1170963.595282785.2F78C0C6@webmail.messagingengine.com> On Sun, May 1, 2016, at 10:23 PM, DFS wrote: > Trying the rawstring thing (say it fast 3x): > > webpage = "http://econpy.pythonanywhere.com/ex/001.html" > > ---------------------------------------------------- > webfile = "D:\\econpy001.html" > urllib.urlretrieve(webpage,webfile) WORKS > ---------------------------------------------------- > webfile = "rD:\econpy001.html" The r is *outside* the string. Its: r"D:\econpy001.html" -- Stephen Hansen m e @ i x o k a i . i o From steve+comp.lang.python at pearwood.info Mon May 2 01:51:53 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 02 May 2016 15:51:53 +1000 Subject: You gotta love a 2-line python solution References: <1462163501.1158885.595255233.34023BE4@webmail.messagingengine.com> <1462165321.1165390.595267481.3403F202@webmail.messagingengine.com> <1462166478.1168179.595275257.0A4C1AD8@webmail.messagingengine.com> Message-ID: <5726eafb$0$2905$c3e8da3$76491128@news.astraweb.com> On Monday 02 May 2016 15:21, Stephen Hansen wrote: > On Sun, May 1, 2016, at 10:08 PM, DFS wrote: >> On 5/2/2016 1:02 AM, Stephen Hansen wrote: >> >> I actually use "D:\\file.html" in my code. >> > >> > Or you can do that. But the whole point of raw strings is not having to >> > escape slashes :) >> >> >> Nice. Where/how else is 'r' used? > > Raw strings are primarily used A) for windows paths, and more > universally, B) for regular expressions. Raw strings are designed for regular expressions. They can be used for Windows paths, except for one minor gotcha: you can't end a raw string with an odd number of backspaces. So this doesn't work: directory = r'D:\some\path\dir\' So it's more of a half-cooked string than a raw string. -- Steve From steve+comp.lang.python at pearwood.info Mon May 2 01:57:02 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 02 May 2016 15:57:02 +1000 Subject: Fastest way to retrieve and write html contents to file References: <1462163671.1159335.595256017.204863FE@webmail.messagingengine.com> <1462165232.1164853.595266801.5DB17C45@webmail.messagingengine.com> Message-ID: <5726ec2f$0$2905$c3e8da3$76491128@news.astraweb.com> On Monday 02 May 2016 15:04, DFS wrote: > 0.2 is half as fast as 0.1, here. > > And two small numbers turn into bigger numbers when the webpage is big, > and soon the download time differences are measured in minutes, not half > a second. It takes twice as long to screw a screw into timber than to hammer a nail into the same timber. Therefore if builders change from nails to screws, they can finish building the house in half the time. -- Steve From nospam at dfs.com Mon May 2 01:59:51 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 01:59:51 -0400 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> <1462166136.1167243.595273897.291B0865@webmail.messagingengine.com> Message-ID: On 5/2/2016 1:15 AM, Stephen Hansen wrote: > On Sun, May 1, 2016, at 10:00 PM, DFS wrote: >> I tried the 10-loop test several times with all versions. > > Also how, _exactly_, are you testing this? > > C:\Python27>python -m timeit "filename='C:\\test.txt'; > webpage='http://econpy.pythonanywhere.com/ex/001.html'; import urllib2; > r = urllib2.urlopen(webpage); f = open(filename, 'w'); > f.write(r.read()); f.close();" > 10 loops, best of 3: 175 msec per loop > > That's a whole lot less the 0.88secs. Indeed. --------------------------------------------------------------------- import requests, urllib, urllib2, pycurl import time webpage = "http://econpy.pythonanywhere.com/ex/001.html" webfile = "D:\\econpy001.html" loops = 10 startTime = time.clock() for i in range(loops): urllib.urlretrieve(webpage,webfile) endTime = time.clock() print "Finished urllib in %.2g seconds" %(endTime-startTime) startTime = time.clock() for i in range(loops): r = urllib2.urlopen(webpage) f = open(webfile,"w") f.write(r.read()) f.close endTime = time.clock() print "Finished urllib2 in %.2g seconds" %(endTime-startTime) startTime = time.clock() for i in range(loops): r = requests.get(webpage) f = open(webfile,"w") f.write(r.text) f.close endTime = time.clock() print "Finished requests in %.2g seconds" %(endTime-startTime) startTime = time.clock() for i in range(loops): with open(webfile + str(i) + ".txt", 'wb') as f: c = pycurl.Curl() c.setopt(c.URL, webpage) c.setopt(c.WRITEDATA, f) c.perform() c.close() endTime = time.clock() print "Finished pycurl in %.2g seconds" %(endTime-startTime) --------------------------------------------------------------------- $ python getHTML.py Finished urllib in 0.88 seconds Finished urllib2 in 0.83 seconds Finished requests in 0.89 seconds Finished pycurl in 1.1 seconds Those results are consistent. They go up or down a little, but never below 0.82 seconds (for urllib2), or above 1.2 seconds (for pycurl) VBScript is consistently 0.44 to 0.48 From steve+comp.lang.python at pearwood.info Mon May 2 02:05:35 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 02 May 2016 16:05:35 +1000 Subject: Fastest way to retrieve and write html contents to file References: <85vb2xgj2i.fsf@benfinney.id.au> Message-ID: <5726ee33$0$1617$c3e8da3$5496439d@news.astraweb.com> On Monday 02 May 2016 15:00, DFS wrote: > I tried the 10-loop test several times with all versions. > > The results were 100% consistent: VBSCript xmlHTTP was always 2x faster > than any python method. Are you absolutely sure you're comparing the same job in two languages? Is VB using a local web cache, and Python not? Are you saving files with both tests? To the same local drive? (To ensure you aren't measuring the difference between "write this file to a slow IDE hard disk, write that file to a fast SSD".) Once you are sure that you are comparing the same task in two languages, then make sure the measurement is meaningful. If you change from a (let's say) 1 KB file to a 100 KB file, do you see the same 2 x difference? What if you increase it to a 10000 KB file? -- Steve From nospam at dfs.com Mon May 2 02:13:12 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 02:13:12 -0400 Subject: You gotta love a 2-line python solution In-Reply-To: References: <1462163501.1158885.595255233.34023BE4@webmail.messagingengine.com> <1462165321.1165390.595267481.3403F202@webmail.messagingengine.com> <1462167424.1170963.595282785.2F78C0C6@webmail.messagingengine.com> Message-ID: On 5/2/2016 1:37 AM, Stephen Hansen wrote: > On Sun, May 1, 2016, at 10:23 PM, DFS wrote: >> Trying the rawstring thing (say it fast 3x): >> >> webpage = "http://econpy.pythonanywhere.com/ex/001.html" >> >> ---------------------------------------------------- >> webfile = "D:\\econpy001.html" >> urllib.urlretrieve(webpage,webfile) WORKS >> ---------------------------------------------------- >> webfile = "rD:\econpy001.html" > > The r is *outside* the string. > > Its: r"D:\econpy001.html" Got it. Thanks. From me+python at ixokai.io Mon May 2 02:27:32 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sun, 01 May 2016 23:27:32 -0700 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> <1462166136.1167243.595273897.291B0865@webmail.messagingengine.com> Message-ID: <1462170452.1180117.595306673.68B64F02@webmail.messagingengine.com> On Sun, May 1, 2016, at 10:59 PM, DFS wrote: > startTime = time.clock() > for i in range(loops): > r = urllib2.urlopen(webpage) > f = open(webfile,"w") > f.write(r.read()) > f.close > endTime = time.clock() > print "Finished urllib2 in %.2g seconds" %(endTime-startTime) Yeah on my system I get 1.8 out of this, amounting to 0.18s. I'm again going back to the point of: its fast enough. When comparing two small numbers, "twice as slow" is meaningless. You have an assumption you haven't answered, that downloading a 10 meg file will be twice as slow as downloading this tiny file. You haven't proven that at all. I suspect you have a constant overhead of X, and in this toy example, that makes it seem twice as slow. But when downloading a file of size, you'll have the same constant factor, at which point the difference is irrelevant. If you believe otherwise, demonstrate it. -- Stephen Hansen m e @ i x o k a i . i o From flebber.crue at gmail.com Mon May 2 02:28:55 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Sun, 1 May 2016 23:28:55 -0700 (PDT) Subject: Code Opinion - Enumerate In-Reply-To: References: <0a2a0652-85c2-4283-a3c4-3ca8bc4e5481@googlegroups.com> <1462166698.1168771.595278561.2A184D4B@webmail.messagingengine.com> Message-ID: <1767aac2-ae13-485d-9184-84827c44bd5c@googlegroups.com> Thanks for the opinion. I should add that is not my code in first post it's the code from Rosetta on how to do Conway's GOL. I thought it looked ugly. Sayth From tjreedy at udel.edu Mon May 2 02:34:13 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 2 May 2016 02:34:13 -0400 Subject: What should Python apps do when asked to show help? In-Reply-To: <5726b1f3$0$1603$c3e8da3$5496439d@news.astraweb.com> References: <1462074388.273747.594588777.5E705539@webmail.messagingengine.com> <20160501052434.GA98660@cskk.homeip.net> <87r3dlolx6.fsf@elektro.pacujo.net> <572630a1$0$1595$c3e8da3$5496439d@news.astraweb.com> <5726b1f3$0$1603$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5/1/2016 9:48 PM, Steven D'Aprano wrote: > On Mon, 2 May 2016 03:04 am, Grant Edwards wrote: > >> On 2016-05-01, Steven D'Aprano wrote: >>> On Mon, 2 May 2016 02:30 am, Grant Edwards wrote: >>> >>>>> In discussions like these, it would be important to draw from >>>>> precedents. Are there commands that have such an option? >>>> >>>> It's pretty rare. It is assumed that Unix uses can type " | less" >>> >>> Is nobody except me questioning the assumption that we're only >>> talking about Unix users? >> >> Didn't the OP specify that he was writing a command-line utility for >> Linux/Unix? > > *cough* I'm the OP, and no I didn't. > > Obviously I'm a Linux user myself, but I'm presumptuous enough to hope that > when I release the utility publicly[1], others may find it of some small > use. Including Windows users. > > [1] Real Soon Now. As a Windows user in recent years, I expect -h to give me a list of options, hopefully with some annotation beyond the bare bones, that give the signature of the command (regarding it as a function call). 'python -h' is pretty bare bones. 'python -m test -h' is much better. I expect both to tell me how to properly pass a file argument. I don't expect either to tell me how write a python or unittest file. I use the manual for this. -- Terry Jan Reedy From tjreedy at udel.edu Mon May 2 02:46:27 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 2 May 2016 02:46:27 -0400 Subject: You gotta love a 2-line python solution In-Reply-To: <1462163501.1158885.595255233.34023BE4@webmail.messagingengine.com> References: <1462163501.1158885.595255233.34023BE4@webmail.messagingengine.com> Message-ID: On 5/2/2016 12:31 AM, Stephen Hansen wrote: > On Sun, May 1, 2016, at 08:39 PM, DFS wrote: >> To save a webpage to a file: >> ------------------------------------- >> 1. import urllib >> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com >> /ex/001.html","D:\file.html") >> ------------------------------------- > > Note, for paths on windows you really want to use a rawstring. Ie, > r"D:\file.html". Or use forward slashes "D:/file.html" and avoid the issue. I don't know of anywhere this does not work for file names sent from python directly to Windows. -- Terry Jan Reedy From nospam at dfs.com Mon May 2 02:47:01 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 02:47:01 -0400 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: <5726ee33$0$1617$c3e8da3$5496439d@news.astraweb.com> References: <85vb2xgj2i.fsf@benfinney.id.au> <5726ee33$0$1617$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5/2/2016 2:05 AM, Steven D'Aprano wrote: > On Monday 02 May 2016 15:00, DFS wrote: > >> I tried the 10-loop test several times with all versions. >> >> The results were 100% consistent: VBSCript xmlHTTP was always 2x faster >> than any python method. > > > Are you absolutely sure you're comparing the same job in two languages? As near as I can tell. In VBScript I'm actually dereferencing various objects (that adds to the time), but I don't do that in python. I don't know enough to even know if it's necessary, or good practice, or what. > Is VB using a local web cache, and Python not? I'm not specifying a local web cache with either (wouldn't know how or where to look). If you have Windows, you can try it. ------------------------------------------------------------------- Option Explicit Dim xmlHTTP, fso, fOut, startTime, endTime, webpage, webfile,i webpage = "http://econpy.pythonanywhere.com/ex/001.html" webfile = "D:\econpy001.html" startTime = Timer For i = 1 to 10 Set xmlHTTP = CreateObject("MSXML2.serverXMLHTTP") xmlHTTP.Open "GET", webpage xmlHTTP.Send Set fso = CreateObject("Scripting.FileSystemObject") Set fOut = fso.CreateTextFile(webfile, True) fOut.WriteLine xmlHTTP.ResponseText fOut.Close Set fOut = Nothing Set fso = Nothing Set xmlHTTP = Nothing Next endTime = Timer wscript.echo "Finished VBScript in " & FormatNumber(endTime - startTime,3) & " seconds" ------------------------------------------------------------------- save it to a .vbs file and run it like this: $cscript /nologo filename.vbs > Are you saving files with both > tests? To the same local drive? (To ensure you aren't measuring the > difference between "write this file to a slow IDE hard disk, write that file > to a fast SSD".) Identical functionality (retrieve webpage, write html to file). Same webpage, written to the same folder on the same hard drive (not SSD). The 10 file writes (open/write/close) don't make a meaningful difference at all: VBScript 0.0156 seconds urllib2 0.0034 seconds This file is 3.55K. > Once you are sure that you are comparing the same task in two languages, > then make sure the measurement is meaningful. If you change from a (let's > say) 1 KB file to a 100 KB file, do you see the same 2 x difference? What if > you increase it to a 10000 KB file? Do you know a webpage I can hit 10x repeatedly to download a good size file? I'm always paranoid they'll block me thinking I'm a "professional" web scraper or something. Thanks From rosuav at gmail.com Mon May 2 03:19:01 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 May 2016 17:19:01 +1000 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> <5726ee33$0$1617$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Mon, May 2, 2016 at 4:47 PM, DFS wrote: > I'm not specifying a local web cache with either (wouldn't know how or where > to look). If you have Windows, you can try it. > ------------------------------------------------------------------- > Option Explicit > Dim xmlHTTP, fso, fOut, startTime, endTime, webpage, webfile,i > webpage = "http://econpy.pythonanywhere.com/ex/001.html" > webfile = "D:\econpy001.html" > startTime = Timer > For i = 1 to 10 > Set xmlHTTP = CreateObject("MSXML2.serverXMLHTTP") > xmlHTTP.Open "GET", webpage > xmlHTTP.Send > Set fso = CreateObject("Scripting.FileSystemObject") > Set fOut = fso.CreateTextFile(webfile, True) > fOut.WriteLine xmlHTTP.ResponseText > fOut.Close > Set fOut = Nothing > Set fso = Nothing > Set xmlHTTP = Nothing > Next > endTime = Timer > wscript.echo "Finished VBScript in " & FormatNumber(endTime - startTime,3) & > " seconds" > ------------------------------------------------------------------- There's an easier way to test if there's caching happening. Just crank the iterations up from 10 to 100 and see what happens to the times. If your numbers are perfectly fair, they should be perfectly linear in the iteration count; eg a 1.8 second ten-iteration loop should become an 18 second hundred-iteration loop. Obviously they won't be exactly that, but I would expect them to be reasonably close (eg 17-19 seconds, but not 2 seconds). Then the next thing to test would be to create a deliberately-slow web server, and connect to that. Put a two-second delay into it, to simulate a distant or overloaded server, and see if your logs show the correct result. Something like this: -------- import time try: import http.server as BaseHTTPServer # Python 3 except ImportError: import BaseHTTPServer # Python 2 class SlowHTTP(BaseHTTPServer.BaseHTTPRequestHandler): def do_GET(self): self.send_response(200) self.send_header("Content-type","text/html") self.end_headers() self.wfile.write(b"Hello, ") time.sleep(2) self.wfile.write(b"world!") server = BaseHTTPServer.HTTPServer(("", 1234), SlowHTTP) server.serve_forever() ------- Test that with a web browser or command-line downloader (go to http://127.0.0.1:1234/), and make sure that (a) it produces "Hello, world!", and (b) it takes two seconds. Then set your test scripts to downloading that URL. (Be sure to set them back to low iteration counts first!) If the times are true and fair, they should all come out pretty much the same - ten iterations, twenty seconds. And since all that's changed is the server, this will be an accurate demonstration of what happens in the real world: network requests aren't always fast. Incidentally, you can also watch the server's log to see if it's getting the appropriate number of requests. It may turn out that changing the web server actually materially changes your numbers. Comment out the sleep call and try it again - you might find that your numbers come closer together, because this naive server doesn't send back 204 NOT MODIFIED responses or anything. Again, though, this would prove that you're not actually measuring language performance, because the tests are more dependent on the server than the client. Even if the files themselves aren't being cached, you might find that DNS is. So if you truly want to eliminate variables, replace the name in your URL with an IP address. It's another thing that might mess with your timings, without actually being a language feature. Networking has about four billion variables in it. You're messing with one of the least significant: the programming language :) ChrisA From nospam at dfs.com Mon May 2 03:37:09 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 03:37:09 -0400 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> <1462166136.1167243.595273897.291B0865@webmail.messagingengine.com> <1462170452.1180117.595306673.68B64F02@webmail.messagingengine.com> Message-ID: On 5/2/2016 2:27 AM, Stephen Hansen wrote: > On Sun, May 1, 2016, at 10:59 PM, DFS wrote: >> startTime = time.clock() >> for i in range(loops): >> r = urllib2.urlopen(webpage) >> f = open(webfile,"w") >> f.write(r.read()) >> f.close >> endTime = time.clock() >> print "Finished urllib2 in %.2g seconds" %(endTime-startTime) > > Yeah on my system I get 1.8 out of this, amounting to 0.18s. You get 1.8 seconds total for the 10 loops? That's less than half as fast as my results. Surprising. > I'm again going back to the point of: its fast enough. When comparing > two small numbers, "twice as slow" is meaningless. Speed is always meaningful. I know python is relatively slow, but it's a cool, concise, powerful language. I'm extremely impressed by how tight the code can get. > You have an assumption you haven't answered, that downloading a 10 meg > file will be twice as slow as downloading this tiny file. You haven't > proven that at all. True. And it has been my assumption - tho not with 10MB file. > I suspect you have a constant overhead of X, and in this toy example, > that makes it seem twice as slow. But when downloading a file of size, > you'll have the same constant factor, at which point the difference is > irrelevant. Good point. Test below. > If you believe otherwise, demonstrate it. http://www.usdirectory.com/ypr.aspx?fromform=qsearch&qs=ga&wqhqn=2&qc=Atlanta&rg=30&qhqn=restaurant&sb=zipdisc&ap=2 It's a 58854 byte file when saved to disk (smaller file was 3546 bytes), so this is 16.6x larger. So I would expect python to linearly run in 16.6 * 0.88 = 14.6 seconds. 10 loops per run 1st run $ python timeGetHTML.py Finished urllib in 8.5 seconds Finished urllib2 in 5.6 seconds Finished requests in 7.8 seconds Finished pycurl in 6.5 seconds wait a couple minutes, then 2nd run $ python timeGetHTML.py Finished urllib in 5.6 seconds Finished urllib2 in 5.7 seconds Finished requests in 5.2 seconds Finished pycurl in 6.4 seconds It's a little more than 1/3 of my estimate - so good news. (when I was doing these tests, some of the python results were 0.75 seconds - way too fast, so I checked and no data was written to file, and I couldn't even open the webpage with a browser. Looks like I had been temporarily blocked from the site. After a couple minutes, I was able to access it again). I noticed urllib and curl returned the html as is, but urllib2 and requests added enhancements that should make the data easier to parse. Based on speed and functionality and documentation, I believe I'll be using the requests HTTP library (I will actually be doing a small amount of web scraping). VBScript 1st run: 7.70 seconds 2nd run: 5.38 3rd run: 7.71 So python matches or beats VBScript at this much larger file. Kewl. From flebber.crue at gmail.com Mon May 2 03:52:43 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Mon, 2 May 2016 00:52:43 -0700 (PDT) Subject: Code Opinion - Enumerate In-Reply-To: <0a2a0652-85c2-4283-a3c4-3ca8bc4e5481@googlegroups.com> References: <0a2a0652-85c2-4283-a3c4-3ca8bc4e5481@googlegroups.com> Message-ID: As a reference here is a functional implementation of conways GOL. http://programmablelife.blogspot.com.au/2012/08/conways-game-of-life-in-clojure.html The author first does it in clojure and then transliterates it to python. Just good for a different view. Sayth From me+python at ixokai.io Mon May 2 03:58:02 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Mon, 02 May 2016 00:58:02 -0700 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> <1462166136.1167243.595273897.291B0865@webmail.messagingengine.com> <1462170452.1180117.595306673.68B64F02@webmail.messagingengine.com> Message-ID: <1462175882.1196948.595349401.3B7C84C9@webmail.messagingengine.com> On Mon, May 2, 2016, at 12:37 AM, DFS wrote: > On 5/2/2016 2:27 AM, Stephen Hansen wrote: > > I'm again going back to the point of: its fast enough. When comparing > > two small numbers, "twice as slow" is meaningless. > > Speed is always meaningful. > > I know python is relatively slow, but it's a cool, concise, powerful > language. I'm extremely impressed by how tight the code can get. I'm sorry, but no. Speed is not always meaningful. It's not even usually meaningful, because you can't quantify what "speed is". In context, you're claiming this is twice as slow (even though my tests show dramatically better performance), but what details are different? You're ignoring the fact that Python might have a constant overhead -- meaning, for a 1k download, it might have X speed cost. For a 1meg download, it might still have the exact same X cost. Looking narrowly, that overhead looks like "twice as slow", but that's not meaningful at all. Looking larger, that overhead is a pittance. You aren't measuring that. > > You have an assumption you haven't answered, that downloading a 10 meg > > file will be twice as slow as downloading this tiny file. You haven't > > proven that at all. > > True. And it has been my assumption - tho not with 10MB file. And that assumption is completely invalid. > I noticed urllib and curl returned the html as is, but urllib2 and > requests added enhancements that should make the data easier to parse. > Based on speed and functionality and documentation, I believe I'll be > using the requests HTTP library (I will actually be doing a small amount > of web scraping). The requests library's added-value is ease-of-use, and its overhead is likely tiny: so using it means you spend less effort making a thing happen. I recommend you embrace this. > VBScript > 1st run: 7.70 seconds > 2nd run: 5.38 > 3rd run: 7.71 > > So python matches or beats VBScript at this much larger file. Kewl. This is what I'm talking about: Python might have a constant overhead, but looking at larger operations, its probably comparable. Not fast, mind you. Python isn't the fastest language out there. But in real world work, its usually fast enough. -- Stephen Hansen m e @ i x o k a i . i o From __peter__ at web.de Mon May 2 04:42:36 2016 From: __peter__ at web.de (Peter Otten) Date: Mon, 02 May 2016 10:42:36 +0200 Subject: Fastest way to retrieve and write html contents to file References: <85vb2xgj2i.fsf@benfinney.id.au> <5726ee33$0$1617$c3e8da3$5496439d@news.astraweb.com> Message-ID: DFS wrote: >> Is VB using a local web cache, and Python not? > > I'm not specifying a local web cache with either (wouldn't know how or > where to look). If you have Windows, you can try it. I don't have Windows, but if I'm to believe http://stackoverflow.com/questions/5235464/how-to-make-microsoft-xmlhttprequest-honor-cache-control-directive the page is indeed cached and you can disable caching with > Option Explicit > Dim xmlHTTP, fso, fOut, startTime, endTime, webpage, webfile,i > webpage = "http://econpy.pythonanywhere.com/ex/001.html" > webfile = "D:\econpy001.html" > startTime = Timer > For i = 1 to 10 > Set xmlHTTP = CreateObject("MSXML2.serverXMLHTTP") > xmlHTTP.Open "GET", webpage xmlHTTP.setRequestHeader "Cache-Control", "max-age=0" > xmlHTTP.Send > Set fso = CreateObject("Scripting.FileSystemObject") > Set fOut = fso.CreateTextFile(webfile, True) > fOut.WriteLine xmlHTTP.ResponseText > fOut.Close > Set fOut = Nothing > Set fso = Nothing > Set xmlHTTP = Nothing > Next > endTime = Timer > wscript.echo "Finished VBScript in " & FormatNumber(endTime - > startTime,3) & " seconds" > ------------------------------------------------------------------- > save it to a .vbs file and run it like this: > $cscript /nologo filename.vbs > From bc at freeuk.com Mon May 2 05:26:17 2016 From: bc at freeuk.com (BartC) Date: Mon, 2 May 2016 10:26:17 +0100 Subject: You gotta love a 2-line python solution In-Reply-To: References: Message-ID: On 02/05/2016 04:39, DFS wrote: > To save a webpage to a file: > ------------------------------------- > 1. import urllib > 2. urllib.urlretrieve("http://econpy.pythonanywhere.com > /ex/001.html","D:\file.html") > ------------------------------------- > > That's it! > > Coming from VB/A background, some of the stuff you can do with python - > with ease - is amazing. > > > VBScript version > ------------------------------------------------------ > 1. Option Explicit > 2. Dim xmlHTTP, fso, fOut > 3. Set xmlHTTP = CreateObject("MSXML2.serverXMLHTTP") > 4. xmlHTTP.Open "GET", "http://econpy.pythonanywhere.com/ex/001.html" > 5. xmlHTTP.Send > 6. Set fso = CreateObject("Scripting.FileSystemObject") > 7. Set fOut = fso.CreateTextFile("D:\file.html", True) > 8. fOut.WriteLine xmlHTTP.ResponseText > 9. fOut.Close > 10. Set fOut = Nothing > 11. Set fso = Nothing > 12. Set xmlHTTP = Nothing > ------------------------------------------------------ > > Technically, that VBS will run with just lines 3-9, but that's still 6 > lines of code vs 2 for python. It seems Python provides a higher level solution compared with VBS. Python presumably also has to do those Opens and Sends, but they are hidden away inside urllib.urlretrieve. You can do the same with VB just by wrapping up these lines in a subroutine. As you would if this had to be executed in a dozen different places for example. Then you could just write: getfile("http://econpy.pythonanywhere.com/ex/001.html", "D:/file.html") in VBS too. (The forward slash in the file name ought to work.) (I don't know VBS; I assume it does /have/ subroutines? What I haven't factored in here is error handling which might yet require more coding in VBS compared with Python) -- Bartc From ulf.worsoe at mosek.com Mon May 2 05:28:00 2016 From: ulf.worsoe at mosek.com (ulf.worsoe at mosek.com) Date: Mon, 2 May 2016 02:28:00 -0700 (PDT) Subject: loading multiple module with same name using importlib.machinery.SourceFileLoader Message-ID: I have observed this behaviour, for some reason only on OS X (and Python 3.5.1): I use importlib.machinery.SourceFileLoader to load a long list of modules. The modules are not located in the loader path, and many of them have the same name, i.e. I would have: m1 = importlib.machinery.SourceFileLoader("Module","path/to/m1/Module.py") m2 = importlib.machinery.SourceFileLoader("Module","path/to/m2/Module.py") Sometimes the modules will contain members from other modules with the same name, e.g. m1/module.py would define a function "m1func" that does not exist in m2/module.py, but the function would appear in m2, and examining m2.m1func.__code__.co_filename shows that it comes from m1. Members that are defined in both m1 and m2 are not overwritten, though. Is this a bug in importlib.machinery.SourceFileLoader or are we in the Land of Undefined Behaviour here? From marko at pacujo.net Mon May 2 06:12:08 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Mon, 02 May 2016 13:12:08 +0300 Subject: You gotta love a 2-line python solution References: Message-ID: <8760uwojk7.fsf@elektro.pacujo.net> BartC : > On 02/05/2016 04:39, DFS wrote: >> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com >> /ex/001.html","D:\file.html") > [...] > > It seems Python provides a higher level solution compared with VBS. > Python presumably also has to do those Opens and Sends, but they are > hidden away inside urllib.urlretrieve. Relevant questions include: * Is a solution available? * Is the solution well thought out? Python does have a lot of great stuff available, which is nice. Unfortunately, many of the handy facilities are lacking in the well-thought-out department. For example, the urlretrieve() function above blocks. You can't use it with the asyncio or select modules. You are left with: Database facilities are notorious offenders. Also, json.load and json.loads don't allow you to decode JSON in chunks. If asyncio breaks through, I expect all blocking stdlib function calls to be adapted for it over the coming years. I'm not overly fond of the asyncio programming model, but it does sport two new killer features: * any blocking operation can be interrupted * events can be multiplexed Marko From steve at pearwood.info Mon May 2 08:05:39 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 02 May 2016 22:05:39 +1000 Subject: You gotta love a 2-line python solution References: <8760uwojk7.fsf@elektro.pacujo.net> Message-ID: <57274296$0$1599$c3e8da3$5496439d@news.astraweb.com> On Mon, 2 May 2016 08:12 pm, Marko Rauhamaa wrote: > For example, the urlretrieve() function above blocks. You can't use it > with the asyncio or select modules. The urlretrieve function is one of the oldest functions in the std library. It literally only exists because Guido was working on a computer somewhere, found that he did have wget, and decided it would be faster to write his own in Python than download and install wget. And because this was very early in Python's history, the barrier to getting into the std lib was much less, especially for stuff Guido wrote himself, so there it is. These days, I doubt it would be included. It would probably be a recipe in the docs. Compared to a full-featured tool like wget or curl, urlretrieve is missing a lot of stuff which is considered essential, like limiting/configuring the rate, support for cookies and authentication, retrying on error, etc. -- Steven From python.list at tim.thechases.com Mon May 2 08:38:21 2016 From: python.list at tim.thechases.com (Tim Chase) Date: Mon, 2 May 2016 07:38:21 -0500 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: Message-ID: <20160502073821.4fe1a305@bigbox.christie.dr> On 2016-05-02 00:06, DFS wrote: > Then I tested them in loops - the VBScript is MUCH faster: 0.44 for > 10 iterations, vs 0.88 for python. In addition to the other debugging recommendations in sibling threads, a couple other things to try: 1) use a local debugging proxy so that you can compare the headers to see if anything stands out 2) in light of #1, can you confirm/deny whether one is using gzip compression and the other isn't? -tkc From ian.g.kelly at gmail.com Mon May 2 09:47:02 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 2 May 2016 07:47:02 -0600 Subject: Private message regarding: Howw to prevent the duplication of any value in a column within a CSV file (python) In-Reply-To: <5ce80235-417b-4deb-8101-58b4d9507be6@googlegroups.com> References: <75e5d8dc-e15b-4350-95e8-490abd4aa14f@googlegroups.com> <531e2dda-c522-4b11-a030-f3f37078f344@googlegroups.com> <49f4ffd8-d140-4f1b-95a9-75cb122e8b2f@googlegroups.com> <5ce80235-417b-4deb-8101-58b4d9507be6@googlegroups.com> Message-ID: On Mon, May 2, 2016 at 3:52 AM, Adam Davis wrote: > Hi Ian, > > I'm really struggling to implement a set into my code as I'm a beginner, > it's taking me a while to grasp the idea of it. If I was to show you my code > so you get an idea of my aim/function of the code, would you be able to help > me at all? Sure, although I'd recommend posting it to the list so that others might also be able to help. From larry.martell at gmail.com Mon May 2 09:52:23 2016 From: larry.martell at gmail.com (Larry Martell) Date: Mon, 2 May 2016 09:52:23 -0400 Subject: starting docker container messes up terminal settings Message-ID: I am starting a docker container from a subprocess.Popen and it works, but when the script returns, the terminal settings of my shell are messed up. Nothing is echoed and return doesn't cause a newline. I can fix this with 'tset' in the terminal, but I don't want to require that. Has anyone here worked with docker and had seen and solved this issue? From grant.b.edwards at gmail.com Mon May 2 10:07:02 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 2 May 2016 14:07:02 +0000 (UTC) Subject: What should Python apps do when asked to show help? References: <20160501223045.GA38957@cskk.homeip.net> Message-ID: On 2016-05-01, cs at zip.com.au wrote: >>Didn't the OP specify that he was writing a command-line utility for >>Linux/Unix? >> >>Discussing command line operation for Windows or OS-X seems rather >>pointless. > > OS-X _is_ UNIX. I spent almost all my time on this Mac in terminals. It is a > very nice to use UNIX in many regards. I include what you're doing under the category "Unix". When I talk about "OS X", I mean what my 84 year old mother is using. I assumed everybody thought that way. ;) -- Grant Edwards grant.b.edwards Yow! If I am elected no one at will ever have to do their gmail.com laundry again! From Joaquin.Alzola at lebara.com Mon May 2 10:08:04 2016 From: Joaquin.Alzola at lebara.com (Joaquin Alzola) Date: Mon, 2 May 2016 14:08:04 +0000 Subject: starting docker container messes up terminal settings In-Reply-To: References: Message-ID: >I am starting a docker container from a subprocess.Popen and it works, but when the script returns, the terminal settings of my shell are messed up. Nothing is echoed and return doesn't cause a >newline. I can fix this with 'tset' in the terminal, but I don't want to require that. Has anyone here worked with docker and had seen and solved this issue? It is good to put part of the code you think is causing the error (Popen subprocess) This email is confidential and may be subject to privilege. If you are not the intended recipient, please do not copy or disclose its content but contact the sender immediately upon receipt. From larry.martell at gmail.com Mon May 2 10:28:02 2016 From: larry.martell at gmail.com (Larry Martell) Date: Mon, 2 May 2016 10:28:02 -0400 Subject: starting docker container messes up terminal settings In-Reply-To: References: Message-ID: On Mon, May 2, 2016 at 10:08 AM, Joaquin Alzola wrote: >>I am starting a docker container from a subprocess.Popen and it works, but when the script returns, the terminal settings of my shell are messed up. Nothing is echoed and return doesn't cause a >newline. I can fix this with 'tset' in the terminal, but I don't want to require that. Has anyone here worked with docker and had seen and solved this issue? > > It is good to put part of the code you think is causing the error (Popen subprocess) cmd = ['sudo', 'docker', 'run', '-t', '-i', 'elucidbio/capdata:v2', 'bash' ] p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) From nospam at dfs.com Mon May 2 11:15:08 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 11:15:08 -0400 Subject: You gotta love a 2-line python solution In-Reply-To: References: Message-ID: On 5/2/2016 5:26 AM, BartC wrote: > On 02/05/2016 04:39, DFS wrote: >> To save a webpage to a file: >> ------------------------------------- >> 1. import urllib >> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com >> /ex/001.html","D:\file.html") >> ------------------------------------- >> >> That's it! >> >> Coming from VB/A background, some of the stuff you can do with python - >> with ease - is amazing. >> >> >> VBScript version >> ------------------------------------------------------ >> 1. Option Explicit >> 2. Dim xmlHTTP, fso, fOut >> 3. Set xmlHTTP = CreateObject("MSXML2.serverXMLHTTP") >> 4. xmlHTTP.Open "GET", "http://econpy.pythonanywhere.com/ex/001.html" >> 5. xmlHTTP.Send >> 6. Set fso = CreateObject("Scripting.FileSystemObject") >> 7. Set fOut = fso.CreateTextFile("D:\file.html", True) >> 8. fOut.WriteLine xmlHTTP.ResponseText >> 9. fOut.Close >> 10. Set fOut = Nothing >> 11. Set fso = Nothing >> 12. Set xmlHTTP = Nothing >> ------------------------------------------------------ >> >> Technically, that VBS will run with just lines 3-9, but that's still 6 >> lines of code vs 2 for python. > > It seems Python provides a higher level solution compared with VBS. > Python presumably also has to do those Opens and Sends, but they are > hidden away inside urllib.urlretrieve. > > You can do the same with VB just by wrapping up these lines in a > subroutine. As you would if this had to be executed in a dozen different > places for example. Then you could just write: > > getfile("http://econpy.pythonanywhere.com/ex/001.html", "D:/file.html") > > in VBS too. (The forward slash in the file name ought to work.) Of course. Taken to its extreme, I could eventually replace you with one line of code :) But python does it for me. That would save me 8 lines... > (I don't know VBS; I assume it does /have/ subroutines? What I haven't > factored in here is error handling which might yet require more coding > in VBS compared with Python) Yeah, VBS has subs and functions. And strange, limited error handling. And a single data type, called Variant. But it's installed with Windows so it's easy to get going with. From larry.martell at gmail.com Mon May 2 11:24:40 2016 From: larry.martell at gmail.com (Larry Martell) Date: Mon, 2 May 2016 11:24:40 -0400 Subject: You gotta love a 2-line python solution In-Reply-To: References: Message-ID: On Mon, May 2, 2016 at 11:15 AM, DFS wrote: > Of course. Taken to its extreme, I could eventually replace you with one > line of code :) That reminds me of something I heard many years ago. Every non-trivial program can be simplified by at least one line of code. Every non trivial program has at least one bug. Therefore every non-trivial program can be reduced to one line of code with a bug. From manolo at austrohungaro.com Mon May 2 11:32:46 2016 From: manolo at austrohungaro.com (Manolo =?iso-8859-1?Q?Mart=EDnez?=) Date: Mon, 2 May 2016 17:32:46 +0200 Subject: You gotta love a 2-line python solution In-Reply-To: References: Message-ID: <20160502153246.GA16381@beagle.localdomain> On 05/02/16 at 11:24am, Larry Martell wrote: > That reminds me of something I heard many years ago. > > Every non-trivial program can be simplified by at least one line of code. > Every non trivial program has at least one bug. > > Therefore every non-trivial program can be reduced to one line of code > with a bug. Well, not really. Every non-trivial program can be reduced to one line of code, but then the resulting program is not non-trivial (as it cannot be further reduced), and therefore there are no guarantees that it will have a bug. M From zljubisic at gmail.com Mon May 2 11:33:51 2016 From: zljubisic at gmail.com (zljubisic at gmail.com) Date: Mon, 2 May 2016 08:33:51 -0700 (PDT) Subject: Python3 html scraper that supports javascript In-Reply-To: References: <2a0c92ed-352d-455c-832d-c9a9438f318b@googlegroups.com> Message-ID: I tried to use the following code: from bs4 import BeautifulSoup from selenium import webdriver PHANTOMJS_PATH = 'C:\\Users\\Zoran\\Downloads\\Obrisi\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe' url = 'https://hrti.hrt.hr/#/video/show/2203605/trebizat-prica-o-jednoj-vodi-i-jednom-narodu-dokumentarni-film' browser = webdriver.PhantomJS(PHANTOMJS_PATH) browser.get(url) soup = BeautifulSoup(browser.page_source, "html.parser") x = soup.prettify() print(x) When I print x variable, I would expect to see something like this: but I can't come to that point. Regards. From nospam at dfs.com Mon May 2 12:33:01 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 12:33:01 -0400 Subject: Best way to clean up list items? Message-ID: Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n '] Want: list1 = ['Item 1','Item 2'] I wrote this, which works fine, but maybe it can be tidier? 1. list2 = [t.replace("\r\n", "") for t in list1] #remove \r\n 2. list3 = [t.strip(' ') for t in list2] #trim whitespace 3. list1 = filter(None, list3) #remove empty items After each step: 1. list2 = [' Item 1 ',' Item 2 ',' '] #remove \r\n 2. list3 = ['Item 1','Item 2',''] #trim whitespace 3. list1 = ['Item 1','Item 2'] #remove empty items Thanks! From nospam at dfs.com Mon May 2 12:39:35 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 12:39:35 -0400 Subject: Python3 html scraper that supports javascript In-Reply-To: References: <2a0c92ed-352d-455c-832d-c9a9438f318b@googlegroups.com> Message-ID: On 5/2/2016 11:33 AM, zljubisic at gmail.com wrote: > > > I tried to use the following code: > > from bs4 import BeautifulSoup > from selenium import webdriver > > PHANTOMJS_PATH = 'C:\\Users\\Zoran\\Downloads\\Obrisi\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe' > > url = 'https://hrti.hrt.hr/#/video/show/2203605/trebizat-prica-o-jednoj-vodi-i-jednom-narodu-dokumentarni-film' > > browser = webdriver.PhantomJS(PHANTOMJS_PATH) > browser.get(url) > > soup = BeautifulSoup(browser.page_source, "html.parser") > > x = soup.prettify() > > print(x) > > > When I print x variable, I would expect to see something like this: > > > but I can't come to that point. > > Regards. I was doing something similar recently. Try this: f = open(somefilename) soup = BeautifulSoup.BeautifulSoup(f) f.close() print soup.prettify() From jussi.piitulainen at helsinki.fi Mon May 2 12:57:07 2016 From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen) Date: Mon, 02 May 2016 19:57:07 +0300 Subject: Best way to clean up list items? References: Message-ID: DFS writes: > Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n '] > Want: list1 = ['Item 1','Item 2'] > > > I wrote this, which works fine, but maybe it can be tidier? > > 1. list2 = [t.replace("\r\n", "") for t in list1] #remove \r\n > 2. list3 = [t.strip(' ') for t in list2] #trim whitespace > 3. list1 = filter(None, list3) #remove empty items > > After each step: > > 1. list2 = [' Item 1 ',' Item 2 ',' '] #remove \r\n > 2. list3 = ['Item 1','Item 2',''] #trim whitespace > 3. list1 = ['Item 1','Item 2'] #remove empty items Try filter(None, (t.strip() for t in list1)). The default. Funny-looking data you have. From walters.justin01 at gmail.com Mon May 2 13:10:02 2016 From: walters.justin01 at gmail.com (justin walters) Date: Mon, 2 May 2016 10:10:02 -0700 Subject: Best way to clean up list items? In-Reply-To: References: Message-ID: On May 2, 2016 10:03 AM, "Jussi Piitulainen" wrote: > > DFS writes: > > > Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n '] > > Want: list1 = ['Item 1','Item 2'] > > > > > > I wrote this, which works fine, but maybe it can be tidier? > > > > 1. list2 = [t.replace("\r\n", "") for t in list1] #remove \r\n > > 2. list3 = [t.strip(' ') for t in list2] #trim whitespace > > 3. list1 = filter(None, list3) #remove empty items > > > > After each step: > > > > 1. list2 = [' Item 1 ',' Item 2 ',' '] #remove \r\n > > 2. list3 = ['Item 1','Item 2',''] #trim whitespace > > 3. list1 = ['Item 1','Item 2'] #remove empty items You could also try compiled regex to remove unwanted characters. Then loop through the list and do a replace for each item. From me+python at ixokai.io Mon May 2 13:25:30 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Mon, 02 May 2016 10:25:30 -0700 Subject: Best way to clean up list items? In-Reply-To: References: Message-ID: <1462209930.1313497.595817481.73635112@webmail.messagingengine.com> On Mon, May 2, 2016, at 09:33 AM, DFS wrote: > Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n '] I'm curious how you got to this point, it seems like you can solve the problem in how this is generated. > Want: list1 = ['Item 1','Item 2'] That said: list1 = [t.strip() for t in list1 if t and not t.isspace()] -- Stephen Hansen m e @ i x o k a i . i o From __peter__ at web.de Mon May 2 13:30:41 2016 From: __peter__ at web.de (Peter Otten) Date: Mon, 02 May 2016 19:30:41 +0200 Subject: Best way to clean up list items? References: Message-ID: DFS wrote: > Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n '] > Want: list1 = ['Item 1','Item 2'] > > > I wrote this, which works fine, but maybe it can be tidier? > > 1. list2 = [t.replace("\r\n", "") for t in list1] #remove \r\n > 2. list3 = [t.strip(' ') for t in list2] #trim whitespace > 3. list1 = filter(None, list3) #remove empty items > > > After each step: > > 1. list2 = [' Item 1 ',' Item 2 ',' '] #remove \r\n > 2. list3 = ['Item 1','Item 2',''] #trim whitespace > 3. list1 = ['Item 1','Item 2'] #remove empty items > > > Thanks! s.strip() strips all whitespace, so you can combine steps 1 and 2: >>> items = ['\r\n Item 1 ',' Item 2 ','\r\n '] >>> stripped = (s.strip() for s in items) The (...) instead of [...] denote a generator expression, so the iteration has not started yet. The final step uses a list comprehension instead of filter(): >>> [s for s in stripped if s] ['Item 1', 'Item 2'] That way the same code works with both Python 2 and Python 3. Note that you can iterate over the generator expression only once; if you try it again you'll end empty-handed: >>> [s for s in stripped if s] [] If you want to do it in one step here are two options that both involve some duplicate work: >>> [s.strip() for s in items if s and not s.isspace()] ['Item 1', 'Item 2'] >>> [s.strip() for s in items if s.strip()] ['Item 1', 'Item 2'] From me+python at ixokai.io Mon May 2 14:00:24 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Mon, 02 May 2016 11:00:24 -0700 Subject: Python3 html scraper that supports javascript In-Reply-To: References: <2a0c92ed-352d-455c-832d-c9a9438f318b@googlegroups.com> Message-ID: <1462212024.1321698.595856209.333A021A@webmail.messagingengine.com> On Mon, May 2, 2016, at 08:33 AM, zljubisic at gmail.com wrote: > I tried to use the following code: > > from bs4 import BeautifulSoup > from selenium import webdriver > > PHANTOMJS_PATH = > 'C:\\Users\\Zoran\\Downloads\\Obrisi\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe' > > url = > 'https://hrti.hrt.hr/#/video/show/2203605/trebizat-prica-o-jednoj-vodi-i-jednom-narodu-dokumentarni-film' > > browser = webdriver.PhantomJS(PHANTOMJS_PATH) > browser.get(url) > > soup = BeautifulSoup(browser.page_source, "html.parser") > > x = soup.prettify() > > print(x) > > When I print x variable, I would expect to see something like this: > > > but I can't come to that point. Why? As important as it is to show code, you need to show what actually happens and what error message is produced. -- Stephen Hansen m e @ i x o k a i . i o From nospam at dfs.com Mon May 2 14:06:34 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 14:06:34 -0400 Subject: Best way to clean up list items? In-Reply-To: References: Message-ID: On 5/2/2016 12:57 PM, Jussi Piitulainen wrote: > DFS writes: > >> Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n '] >> Want: list1 = ['Item 1','Item 2'] >> >> >> I wrote this, which works fine, but maybe it can be tidier? >> >> 1. list2 = [t.replace("\r\n", "") for t in list1] #remove \r\n >> 2. list3 = [t.strip(' ') for t in list2] #trim whitespace >> 3. list1 = filter(None, list3) #remove empty items >> >> After each step: >> >> 1. list2 = [' Item 1 ',' Item 2 ',' '] #remove \r\n >> 2. list3 = ['Item 1','Item 2',''] #trim whitespace >> 3. list1 = ['Item 1','Item 2'] #remove empty items > > Try filter(None, (t.strip() for t in list1)). The default. Works and drops a line of code. Thx. > Funny-looking data you have. I know - sadly, it's actual data: -------------------------------------------------------------------- from lxml import html import requests webpage = "http://www.usdirectory.com/ypr.aspx?fromform=qsearch&qs=TN&wqhqn=2&qc=Nashville&rg=30&qhqn=restaurant&sb=zipdisc&ap=2" page = requests.get(webpage) tree = html.fromstring(page.content) addr1 = tree.xpath('//span[@class="text3"]/text()') print 'Addresses: ', addr1 -------------------------------------------------------------------- I couldn't figure out a better way to extract it from the HTML (maybe XML and DOM?) From nospam at dfs.com Mon May 2 14:09:39 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 14:09:39 -0400 Subject: Best way to clean up list items? In-Reply-To: References: <1462209930.1313497.595817481.73635112@webmail.messagingengine.com> Message-ID: On 5/2/2016 1:25 PM, Stephen Hansen wrote: > On Mon, May 2, 2016, at 09:33 AM, DFS wrote: >> Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n '] > > I'm curious how you got to this point, it seems like you can solve the > problem in how this is generated. -------------------------------------------------------------------- from lxml import html import requests webpage = "http://www.usdirectory.com/ypr.aspx?fromform=qsearch&qs=TN&wqhqn=2&qc=Nashville&rg=30&qhqn=restaurant&sb=zipdisc&ap=2" page = requests.get(webpage) tree = html.fromstring(page.content) addr1 = tree.xpath('//span[@class="text3"]/text()') print 'Addresses: ', addr1 -------------------------------------------------------------------- I'd prefer to get clean data in the first place, but I don't know a better way to extract it from the HTML. From me+python at ixokai.io Mon May 2 14:23:51 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Mon, 02 May 2016 11:23:51 -0700 Subject: Best way to clean up list items? In-Reply-To: References: <1462209930.1313497.595817481.73635112@webmail.messagingengine.com> Message-ID: <1462213431.1326811.595881033.1348150C@webmail.messagingengine.com> On Mon, May 2, 2016, at 11:09 AM, DFS wrote: > I'd prefer to get clean data in the first place, but I don't know a > better way to extract it from the HTML. Ah, right. I didn't know you were scraping HTML. Scraping HTML is rarely clean so you have to do a lot of cleanup. -- Stephen Hansen m e @ i x o k a i . i o From jussi.piitulainen at helsinki.fi Mon May 2 14:27:17 2016 From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen) Date: Mon, 02 May 2016 21:27:17 +0300 Subject: Best way to clean up list items? References: Message-ID: DFS writes: > On 5/2/2016 12:57 PM, Jussi Piitulainen wrote: >> DFS writes: >> >>> Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n '] >>> Want: list1 = ['Item 1','Item 2'] . . >> Funny-looking data you have. > > I know - sadly, it's actual data: > > -------------------------------------------------------------------- > from lxml import html > import requests > > webpage = > "http://www.usdirectory.com/ypr.aspx?fromform=qsearch&qs=TN&wqhqn=2&qc=Nashville&rg=30&qhqn=restaurant&sb=zipdisc&ap=2" > > page = requests.get(webpage) > tree = html.fromstring(page.content) > addr1 = tree.xpath('//span[@class="text3"]/text()') > print 'Addresses: ', addr1 > -------------------------------------------------------------------- > > I couldn't figure out a better way to extract it from the HTML (maybe > XML and DOM?) I should have guessed :) But now I'm a bit worried about those spaces inside your items. Can it happen that item text is split into strings in the middle? Then the above sanitation does the wrong thing. If someone has the right solution, I'm watching, too. From nospam at dfs.com Mon May 2 15:04:10 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 15:04:10 -0400 Subject: Best way to clean up list items? In-Reply-To: References: Message-ID: On 5/2/2016 2:27 PM, Jussi Piitulainen wrote: > DFS writes: > >> On 5/2/2016 12:57 PM, Jussi Piitulainen wrote: >>> DFS writes: >>> >>>> Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n '] >>>> Want: list1 = ['Item 1','Item 2'] > > . . > >>> Funny-looking data you have. >> >> I know - sadly, it's actual data: >> >> -------------------------------------------------------------------- >> from lxml import html >> import requests >> >> webpage = >> "http://www.usdirectory.com/ypr.aspx?fromform=qsearch&qs=TN&wqhqn=2&qc=Nashville&rg=30&qhqn=restaurant&sb=zipdisc&ap=2" >> >> page = requests.get(webpage) >> tree = html.fromstring(page.content) >> addr1 = tree.xpath('//span[@class="text3"]/text()') >> print 'Addresses: ', addr1 >> -------------------------------------------------------------------- >> >> I couldn't figure out a better way to extract it from the HTML (maybe >> XML and DOM?) > > I should have guessed :) But now I'm a bit worried about those spaces > inside your items. Can it happen that item text is split into strings in > the middle? Meaning split by me, or comes 'malformed' from the data source? > Then the above sanitation does the wrong thing. > > If someone has the right solution, I'm watching, too. Here's the raw data as stored in the tree: --------------------------------------------------------------------------- 1st page ['\r\n ', '\r\n 1918 W End Ave, Nashville, TN 37203', '\r\n ', '\r\n 1806 Hayes St, Nashville, TN 37203', '\r\n ', '\r\n 1701 Broadway, Nashville, TN 37203', '\r\n ', '\r\n 209 10th Ave S, Nashville, TN 37203', '\r\n ', '\r\n 907 20th Ave S, Nashville, TN 37212', '\r\n ', '\r\n 911 20th Ave S, Nashville, TN 37212', '\r\n ', '\r\n 1722 W End Ave, Nashville, TN 37203', '\r\n ', '\r\n 1905 Hayes St, Nashville, TN 37203', '\r\n ', '\r\n 2000 W End Ave, Nashville, TN 37203'] --------------------------------------------------------------------------- Next page ['\r\n ', '\r\n 120 19th Ave N, Nashville, TN 37203', '\r\n ', '\r\n 1719 W End Ave Ste 101, Nashville, TN 37203', '\r\n ', '\r\n 1922 W End Ave, Nashville, TN 37203', '\r\n ', '\r\n 909 20th Ave S, Nashville, TN 37212', '\r\n ', '\r\n 1807 Church St, Nashville, TN 37203', '\r\n ', '\r\n 1721 Church St, Nashville, TN 37203', '\r\n ', '\r\n 718 Division St, Nashville, TN 37203', '\r\n ', '\r\n 907 12th Ave S, Nashville, TN 37203', '\r\n ', '\r\n 204 21st Ave S, Nashville, TN 37203', '\r\n ', '\r\n 1811 Division St, Nashville, TN 37203', '\r\n ', '\r\n 903 Gleaves St, Nashville, TN 37203', '\r\n ', '\r\n 1720 W End Ave Ste 530, Nashville, TN 37203', '\r\n ', '\r\n 1200 Division St Ste 100-A, Nashville, TN 37203', '\r\n ', '\r\n 422 7th Ave S, Nashville, TN 37203', '\r\n ', '\r\n 605 8th Ave S, Nashville, TN 37203'] and so on --------------------------------------------------------------------------- I've checked a couple hundred addresses visually, and so far I've only seen 2 formats: 1. '\r\n ' 2. '\r\n address ' From zljubisic at gmail.com Mon May 2 16:11:33 2016 From: zljubisic at gmail.com (zljubisic at gmail.com) Date: Mon, 2 May 2016 13:11:33 -0700 (PDT) Subject: Python3 html scraper that supports javascript In-Reply-To: References: <2a0c92ed-352d-455c-832d-c9a9438f318b@googlegroups.com> <1462212024.1321698.595856209.333A021A@webmail.messagingengine.com> Message-ID: <3d4a30a6-44bc-42bb-a3ef-4c993240558a@googlegroups.com> > Why? As important as it is to show code, you need to show what actually > happens and what error message is produced. If you run the code you will see that html that I got doesn't have link to the flash video. I should somehow do something (press play video button maybe) in order to get html with reference to the video file on this page. Regards From moa47401 at gmail.com Mon May 2 17:30:48 2016 From: moa47401 at gmail.com (moa47401 at gmail.com) Date: Mon, 2 May 2016 14:30:48 -0700 (PDT) Subject: Need help understanding list structure Message-ID: I've been using an old text parsing library and have been able to accomplish most of what I wanted to do. But I don't understand the list structure it uses well enough to build additional methods. If I print the list, it has thousands of elements within its brackets separated by commas as I would expect. But the elements appear to be memory pointers not the actual text. Here's an example: If I iterate over the list, I do get the actual text of each element and am able to use it. Also, if I iterate over the list and place each element in a new list using append, then each element in the new list is the text I expect not memory pointers. But... if I copy the old list to a new list using new = old[:] or new = list(old) the new list is exactly like the original with memory pointers. Can someone help me understand why or under what circumstances a list shows pointers instead of the text data? From python at lucidity.plus.com Mon May 2 17:48:33 2016 From: python at lucidity.plus.com (Erik) Date: Mon, 2 May 2016 22:48:33 +0100 Subject: Need help understanding list structure In-Reply-To: References: Message-ID: <5727CB31.5060309@lucidity.plus.com> On 02/05/16 22:30, moa47401 at gmail.com wrote: > Can someone help me understand why or under what circumstances a list > shows pointers instead of the text data? When Python's "print" statement/function is invoked, it will print the textual representation of the object according to its class's __str__ or __repr__ method. That is, the print function prints out whatever text the class says it should. For classes which don't implement a __str__ or __repr__ method, then the text "" is used - where CLASS is the class name and ADDRESS is the "memory pointer". > If I iterate over the list, I do get the actual text of each element > and am able to use it. > > Also, if I iterate over the list and place each element in a new list > using append, then each element in the new list is the text I expect > not memory pointers. Look at the __iter__ method of the class of the object you are iterating over. I suspect that it returns string objects, not the objects that are in the list itself. String objects have a __str__ or __repr__ method that represents them as the text, so that is what 'print' will output. Hope that helps, E. From moa47401 at gmail.com Mon May 2 18:33:25 2016 From: moa47401 at gmail.com (moa47401 at gmail.com) Date: Mon, 2 May 2016 15:33:25 -0700 (PDT) Subject: Need help understanding list structure In-Reply-To: References: <5727CB31.5060309@lucidity.plus.com> Message-ID: > When Python's "print" statement/function is invoked, it will print the > textual representation of the object according to its class's __str__ or > __repr__ method. That is, the print function prints out whatever text > the class says it should. > > For classes which don't implement a __str__ or __repr__ method, then > the text "" is used - where CLASS is the class > name and ADDRESS is the "memory pointer". > > > If I iterate over the list, I do get the actual text of each element > > and am able to use it. > > > > Also, if I iterate over the list and place each element in a new list > > using append, then each element in the new list is the text I expect > > not memory pointers. > > Look at the __iter__ method of the class of the object you are iterating > over. I suspect that it returns string objects, not the objects that are > in the list itself. > > String objects have a __str__ or __repr__ method that represents them as > the text, so that is what 'print' will output. > > Hope that helps, E. Yes, that does help. You're right. The author of the library I'm using didn't implement either a __str__ or __repr__ method. Am I correct in assuming that parsing a large text file would be quicker returning pointers instead of strings? I've never run into this before. From cs at zip.com.au Mon May 2 18:52:30 2016 From: cs at zip.com.au (cs at zip.com.au) Date: Tue, 3 May 2016 08:52:30 +1000 Subject: What should Python apps do when asked to show help? In-Reply-To: References: Message-ID: <20160502225230.GA2167@cskk.homeip.net> On 02May2016 14:07, Grant Edwards wrote: >On 2016-05-01, cs at zip.com.au wrote: > >>>Didn't the OP specify that he was writing a command-line utility for >>>Linux/Unix? >>> >>>Discussing command line operation for Windows or OS-X seems rather >>>pointless. >> >> OS-X _is_ UNIX. I spent almost all my time on this Mac in terminals. It is a >> very nice to use UNIX in many regards. > >I include what you're doing under the category "Unix". When I talk >about "OS X", I mean what my 84 year old mother is using. I assumed >everybody thought that way. ;) Weird. My 79 year old mother uses "Apple". I can only presume there's no "OS X" for her. Cheers, Cameron Simpson From torriem at gmail.com Mon May 2 19:25:24 2016 From: torriem at gmail.com (Michael Torrie) Date: Mon, 2 May 2016 17:25:24 -0600 Subject: Need help understanding list structure In-Reply-To: References: <5727CB31.5060309@lucidity.plus.com> Message-ID: <5727E1E4.7080403@gmail.com> On 05/02/2016 04:33 PM, moa47401 at gmail.com wrote: > Yes, that does help. You're right. The author of the library I'm > using didn't implement either a __str__ or __repr__ method. Am I > correct in assuming that parsing a large text file would be quicker > returning pointers instead of strings? I've never run into this > before. I'm not sure what you mean by "returning pointers." The list isn't returning pointers. It's a list of *objects*. To be specific, a list of gedcom.Element objects, though they could be anything, including numbers or strings. If you refer to the source code where the Element class is defined you can see what these objects contain. I suspect they contain a lot more information than simply text. Lists of objects is a common idiom in Python. As you've discovered, if you shallow copy a list, the new list will contain the exact same objects. In many cases, this does not matter. For example a list of numbers, which are immutable or unchangeable objects. It doesn't matter that the instances are shared, since the instances themselves will never change. If the objects are mutable, as they are in your case, a shallow copy may not always be what you want. As to your question. A list never shows "pointers" as you say. A list always contains objects, and if you simply "print" the list, it will try to show a representation of the list, using the objects' repr dunder methods. Some classes I have used have their repr methods print out what the constructor would look like, if you were to construct the object yourself. This is very useful. If I recall, this is what BeautifulSoup objects do, which is incredibly useful. In your case, as Erik said, the objects you are dealing with don't provide repr dunder methods, so Python just lets you know they are objects of a certain class, and what their ids are, which is helpful if you're trying to determine if two objects are the same object. These are not "pointers" in the sense you're talking. You'll get text if the object prints text for you. This is true of any object you might store in the list. I hope this helps a bit. Exploring from the interactive prompt as you are doing is very useful, once you understand what it's saying to you. From ben+python at benfinney.id.au Mon May 2 19:43:48 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 03 May 2016 09:43:48 +1000 Subject: Need help understanding list structure References: <5727CB31.5060309@lucidity.plus.com> Message-ID: <85mvo8gh57.fsf@benfinney.id.au> moa47401 at gmail.com writes: > Am I correct in assuming that parsing a large text file would be > quicker returning pointers instead of strings? What do you mean by ?return a pointer?? Python doesn't have pointers. In the Python language, a container type (such as ?set?, ?list?, ?dict?, etc.) contains the objects directly. There are no ?pointers? there; by accessing the items of a container, you access the items directly. What do you mean by ?would be quicker?? I am concerned you are seeking speed of the program at the expense of understandability and clarity of the code. Instead, you should be writing clear, maintainable code. *Only if* the clear, maintainable code you write then actually ends up being too slow, should you then worry about what parts are quick or slow by *measuring* the specific parts of code to discover what is actually occupying the time. -- \ ?All television is educational television. The question is: | `\ what is it teaching?? ?Nicholas Johnson | _o__) | Ben Finney From jfong at ms4.hinet.net Mon May 2 20:45:13 2016 From: jfong at ms4.hinet.net (jfong at ms4.hinet.net) Date: Mon, 2 May 2016 17:45:13 -0700 (PDT) Subject: You gotta love a 2-line python solution In-Reply-To: References: Message-ID: DFS at 2016/5/2 UTC+8 11:39:33AM wrote: > To save a webpage to a file: > ------------------------------------- > 1. import urllib > 2. urllib.urlretrieve("http://econpy.pythonanywhere.com > /ex/001.html","D:\file.html") > ------------------------------------- > > That's it! Why my system can't do it? Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (In tel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from urllib import urlretrieve Traceback (most recent call last): File "", line 1, in ImportError: cannot import name 'urlretrieve' From nospam at dfs.com Mon May 2 21:12:13 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 21:12:13 -0400 Subject: You gotta love a 2-line python solution In-Reply-To: References: Message-ID: On 5/2/2016 8:45 PM, jfong at ms4.hinet.net wrote: > DFS at 2016/5/2 UTC+8 11:39:33AM wrote: >> To save a webpage to a file: >> ------------------------------------- >> 1. import urllib >> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com >> /ex/001.html","D:\file.html") >> ------------------------------------- >> >> That's it! > > Why my system can't do it? > > Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (In > tel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. >>>> from urllib import urlretrieve > Traceback (most recent call last): > File "", line 1, in > ImportError: cannot import name 'urlretrieve' try from urllib.request import urlretrieve http://stackoverflow.com/questions/21171718/urllib-urlretrieve-file-python-3-3 I'm running python 2.7.11 (32-bit) From nospam at dfs.com Mon May 2 21:51:31 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 21:51:31 -0400 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> <5726ee33$0$1617$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5/2/2016 3:19 AM, Chris Angelico wrote: > There's an easier way to test if there's caching happening. Just crank > the iterations up from 10 to 100 and see what happens to the times. If > your numbers are perfectly fair, they should be perfectly linear in > the iteration count; eg a 1.8 second ten-iteration loop should become > an 18 second hundred-iteration loop. Obviously they won't be exactly > that, but I would expect them to be reasonably close (eg 17-19 > seconds, but not 2 seconds). 100 loops Finished VBScript in 3.953 seconds Finished VBScript in 3.608 seconds Finished VBScript in 3.610 seconds Bit of a per-loop speedup going from 10 to 100. > Then the next thing to test would be to create a deliberately-slow web > server, and connect to that. Put a two-second delay into it, to > simulate a distant or overloaded server, and see if your logs show the > correct result. Something like this: > > -------- > > import time > try: > import http.server as BaseHTTPServer # Python 3 > except ImportError: > import BaseHTTPServer # Python 2 > > class SlowHTTP(BaseHTTPServer.BaseHTTPRequestHandler): > def do_GET(self): > self.send_response(200) > self.send_header("Content-type","text/html") > self.end_headers() > self.wfile.write(b"Hello, ") > time.sleep(2) > self.wfile.write(b"world!") > > server = BaseHTTPServer.HTTPServer(("", 1234), SlowHTTP) > server.serve_forever() > > ------- > > Test that with a web browser or command-line downloader (go to > http://127.0.0.1:1234/), and make sure that (a) it produces "Hello, > world!", and (b) it takes two seconds. Then set your test scripts to > downloading that URL. (Be sure to set them back to low iteration > counts first!) If the times are true and fair, they should all come > out pretty much the same - ten iterations, twenty seconds. And since > all that's changed is the server, this will be an accurate > demonstration of what happens in the real world: network requests > aren't always fast. Incidentally, you can also watch the server's log > to see if it's getting the appropriate number of requests. > > It may turn out that changing the web server actually materially > changes your numbers. Comment out the sleep call and try it again - > you might find that your numbers come closer together, because this > naive server doesn't send back 204 NOT MODIFIED responses or anything. > Again, though, this would prove that you're not actually measuring > language performance, because the tests are more dependent on the > server than the client. > > Even if the files themselves aren't being cached, you might find that > DNS is. So if you truly want to eliminate variables, replace the name > in your URL with an IP address. It's another thing that might mess > with your timings, without actually being a language feature. > > Networking has about four billion variables in it. You're messing with > one of the least significant: the programming language :) > > ChrisA Thanks for the good feedback. From nospam at dfs.com Mon May 2 21:52:04 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 21:52:04 -0400 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> <5726ee33$0$1617$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5/2/2016 4:42 AM, Peter Otten wrote: > DFS wrote: > >>> Is VB using a local web cache, and Python not? >> >> I'm not specifying a local web cache with either (wouldn't know how or >> where to look). If you have Windows, you can try it. > > I don't have Windows, but if I'm to believe > > http://stackoverflow.com/questions/5235464/how-to-make-microsoft-xmlhttprequest-honor-cache-control-directive > > the page is indeed cached and you can disable caching with > >> Option Explicit >> Dim xmlHTTP, fso, fOut, startTime, endTime, webpage, webfile,i >> webpage = "http://econpy.pythonanywhere.com/ex/001.html" >> webfile = "D:\econpy001.html" >> startTime = Timer >> For i = 1 to 10 >> Set xmlHTTP = CreateObject("MSXML2.serverXMLHTTP") >> xmlHTTP.Open "GET", webpage > > xmlHTTP.setRequestHeader "Cache-Control", "max-age=0" Tried that, and from later on that stackoverflow page: xmlHTTP.setRequestHeader "Cache-Control", "private" Neither made a difference. In fact, I saw faster times than ever - as low as 0.41 for 10 loops. From rosuav at gmail.com Mon May 2 22:00:23 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 3 May 2016 12:00:23 +1000 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> <5726ee33$0$1617$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, May 3, 2016 at 11:51 AM, DFS wrote: > On 5/2/2016 3:19 AM, Chris Angelico wrote: > >> There's an easier way to test if there's caching happening. Just crank >> the iterations up from 10 to 100 and see what happens to the times. If >> your numbers are perfectly fair, they should be perfectly linear in >> the iteration count; eg a 1.8 second ten-iteration loop should become >> an 18 second hundred-iteration loop. Obviously they won't be exactly >> that, but I would expect them to be reasonably close (eg 17-19 >> seconds, but not 2 seconds). > > > 100 loops > Finished VBScript in 3.953 seconds > Finished VBScript in 3.608 seconds > Finished VBScript in 3.610 seconds > > Bit of a per-loop speedup going from 10 to 100. How many seconds was it for 10 loops? ChrisA From nospam at dfs.com Mon May 2 22:01:44 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 22:01:44 -0400 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> <5726ee33$0$1617$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5/2/2016 10:00 PM, Chris Angelico wrote: > On Tue, May 3, 2016 at 11:51 AM, DFS wrote: >> On 5/2/2016 3:19 AM, Chris Angelico wrote: >> >>> There's an easier way to test if there's caching happening. Just crank >>> the iterations up from 10 to 100 and see what happens to the times. If >>> your numbers are perfectly fair, they should be perfectly linear in >>> the iteration count; eg a 1.8 second ten-iteration loop should become >>> an 18 second hundred-iteration loop. Obviously they won't be exactly >>> that, but I would expect them to be reasonably close (eg 17-19 >>> seconds, but not 2 seconds). >> >> >> 100 loops >> Finished VBScript in 3.953 seconds >> Finished VBScript in 3.608 seconds >> Finished VBScript in 3.610 seconds >> >> Bit of a per-loop speedup going from 10 to 100. > > How many seconds was it for 10 loops? > > ChrisA ~0.44 From jfong at ms4.hinet.net Mon May 2 23:27:45 2016 From: jfong at ms4.hinet.net (jfong at ms4.hinet.net) Date: Mon, 2 May 2016 20:27:45 -0700 (PDT) Subject: You gotta love a 2-line python solution In-Reply-To: References: Message-ID: DFS at 2016/5/3 9:12:24AM wrote: > try > > from urllib.request import urlretrieve > > http://stackoverflow.com/questions/21171718/urllib-urlretrieve-file-python-3-3 > > > I'm running python 2.7.11 (32-bit) Alright, it works...someway. I try to get a zip file. It works, the file can be unzipped correctly. >>> from urllib.request import urlretrieve >>> urlretrieve("http://www.caprilion.com.tw/fed.zip", "d:\\temp\\temp.zip") ('d:\\temp\\temp.zip', ) >>> But when I try to get this forum page, it does get a html file but can't be viewed normally. >>> urlretrieve("https://groups.google.com/forum/#!topic/comp.lang.python/jFl3GJ bmR7A", "d:\\temp\\temp.html") ('d:\\temp\\temp.html', ) >>> I suppose the html is a much complex situation where more processes need to be done before it can be opened by a web browser:-) From me+python at ixokai.io Mon May 2 23:49:00 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Mon, 02 May 2016 20:49:00 -0700 Subject: You gotta love a 2-line python solution In-Reply-To: References: Message-ID: <1462247340.1450624.596312857.04969060@webmail.messagingengine.com> On Mon, May 2, 2016, at 08:27 PM, jfong at ms4.hinet.net wrote: > But when I try to get this forum page, it does get a html file but can't > be viewed normally. What does that mean? -- Stephen Hansen m e @ i x o k a i . i o From nospam at dfs.com Mon May 2 23:56:55 2016 From: nospam at dfs.com (DFS) Date: Mon, 2 May 2016 23:56:55 -0400 Subject: You gotta love a 2-line python solution In-Reply-To: References: Message-ID: On 5/2/2016 11:27 PM, jfong at ms4.hinet.net wrote: > DFS at 2016/5/3 9:12:24AM wrote: >> try >> >> from urllib.request import urlretrieve >> >> http://stackoverflow.com/questions/21171718/urllib-urlretrieve-file-python-3-3 >> >> >> I'm running python 2.7.11 (32-bit) > > Alright, it works...someway. > > I try to get a zip file. It works, the file can be unzipped correctly. > >>>> from urllib.request import urlretrieve >>>> urlretrieve("http://www.caprilion.com.tw/fed.zip", "d:\\temp\\temp.zip") > ('d:\\temp\\temp.zip', ) >>>> > > But when I try to get this forum page, it does get a html file but can't be viewed normally. > >>>> urlretrieve("https://groups.google.com/forum/#!topic/comp.lang.python/jFl3GJ > bmR7A", "d:\\temp\\temp.html") > ('d:\\temp\\temp.html', ) >>>> > > I suppose the html is a much complex situation where more processes need to be done before it can be opened by a web browser:-) Who knows what Google has done... it won't open in Opera. The tab title shows up, but after 20-30 seconds the screen just stays blank and the cursor quits loading. It's a mess - try running it thru BeautifulSoup.prettify() and it looks better. ------------------------------------------------------------ import BeautifulSoup from urllib.request import urlretrieve webfile = "D:\\afile.html" urllib.urlretrieve("https://groups.google.com/forum/#!topic/comp.lang.python/jFl3GJbmR7A",webfile) f = open(webfile) soup = BeautifulSoup.BeautifulSoup(f) f.close() print soup.prettify() ------------------------------------------------------------ From jfong at ms4.hinet.net Mon May 2 23:57:21 2016 From: jfong at ms4.hinet.net (jfong at ms4.hinet.net) Date: Mon, 2 May 2016 20:57:21 -0700 (PDT) Subject: You gotta love a 2-line python solution In-Reply-To: References: <1462247340.1450624.596312857.04969060@webmail.messagingengine.com> Message-ID: <21c6d3df-3346-49b5-ac66-76b977c5aef5@googlegroups.com> Stephen Hansen at 2016/5/3 11:49:22AM wrote: > On Mon, May 2, 2016, at 08:27 PM, jfong at ms4.hinet.net wrote: > > But when I try to get this forum page, it does get a html file but can't > > be viewed normally. > > What does that mean? > > -- > Stephen Hansen > m e @ i x o k a i . i o The page we are looking at:-) https://groups.google.com/forum/#!topic/comp.lang.python/jFl3GJbmR7A From torriem at gmail.com Tue May 3 00:06:48 2016 From: torriem at gmail.com (Michael Torrie) Date: Mon, 2 May 2016 22:06:48 -0600 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> <1462166136.1167243.595273897.291B0865@webmail.messagingengine.com> <1462170452.1180117.595306673.68B64F02@webmail.messagingengine.com> Message-ID: <572823D8.6050204@gmail.com> On 05/02/2016 01:37 AM, DFS wrote: > So python matches or beats VBScript at this much larger file. Kewl. If you download something large enough to be meaningful, you'll find the runtime speeds should all converge to something showing your internet connection speed. Try downloading a 4 GB file, for example. You're trying to benchmark an io-bound operation. After you move past the very small and meaningless examples that simply benchmark the overhead of the connection building, you'll find that all languages, even compiled languages like C, should run at the same speed on average. Neither VBS nor Python will be faster than each other. Now if you want to talk about processing the data once you have it, there we can talk about speeds and optimization. From nospam at dfs.com Tue May 3 00:24:06 2016 From: nospam at dfs.com (DFS) Date: Tue, 3 May 2016 00:24:06 -0400 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> <1462166136.1167243.595273897.291B0865@webmail.messagingengine.com> <1462170452.1180117.595306673.68B64F02@webmail.messagingengine.com> <572823D8.6050204@gmail.com> Message-ID: On 5/3/2016 12:06 AM, Michael Torrie wrote: > Now if you want to talk about processing the data once you have it, > there we can talk about speeds and optimization. Be glad to. Helps me learn python, so bring whatever challenge you want and I'll try to keep up. One small comparison I was able to make was VBA vs python/pyodbc to summarize an Access database. Not quite a fair test, but interesting nonetheless. --------------------------------------------------- Access 2003 file Access 2003 VBA code 2,099,101 rows 114 tables (max row = 600288) 971 columns text: 503 boolean: 4 numeric: 351 date-time: 108 binary: 5 309 indexes (25 foreign keys) 333,549,568 bytes on disk Time: 0.18 seconds --------------------------------------------------- same Access 2003 file 32-bit python 2.7.11 + 32-bit pyodbc 3.0.6 2,099,101 rows 114 tables (max row = 600288) 971 columns text: 503 numeric: 351 date-time: 108 binary: 5 boolean: 4 309 indexes (foreign keys na via ODBC*) 333,549,568 bytes on disk Time: 0.49 seconds * the Access ODBC driver doesn't support the SQLForeignKeys function --------------------------------------------------- From kewiloo at gmail.com Tue May 3 04:02:30 2016 From: kewiloo at gmail.com (musoke wilson) Date: Tue, 3 May 2016 01:02:30 -0700 (PDT) Subject: Client support automation and self service Message-ID: Hi Guys Currently working with a team to automate business operations and client support for a small enterprise. Key requirements: Clients to register, log queries and initiate service request through The Web and/or Mobile APP Clear tracking by the CRM team (SR alert through email/mobile APP) Real time support (IM/email/mobile phone APP) Analysis & Visualization of enrollment/Usage stats We are considering using possibility of using the Python-twisted framework Kindly advise if python & twisted framework are appropriate tools and any other possible open source solutions to avoid re-inventing the wheel regards Wilson From jeanmichel at sequans.com Tue May 3 05:36:52 2016 From: jeanmichel at sequans.com (jmp) Date: Tue, 03 May 2016 11:36:52 +0200 Subject: Client support automation and self service In-Reply-To: References: Message-ID: On 05/03/2016 10:02 AM, musoke wilson wrote: > Hi Guys > > Currently working with a team to automate business operations and client support for a small enterprise. > > Key requirements: > Clients to register, log queries and initiate service request through The Web and/or Mobile APP > Clear tracking by the CRM team (SR alert through email/mobile APP) > Real time support (IM/email/mobile phone APP) > Analysis & Visualization of enrollment/Usage stats > > We are considering using possibility of using the Python-twisted framework > > Kindly advise if python & twisted framework are appropriate tools and any other possible open source solutions to avoid re-inventing the wheel > > regards > > > Wilson If no one in your team is able to answer that question I'd rather seek for professional support than asking this mailing list. You need something better than "yes twisted can do what you want". (it surely can) However, regarding open source alternative, you could implement a client support using bugzilla, probably with a litle bit of tuning. https://www.bugzilla.org/ Regarding "business operations" which I'm not sure what it means exactly, look at http://www.tryton.org/ jm From pavlovevidence at gmail.com Tue May 3 06:00:22 2016 From: pavlovevidence at gmail.com (pavlovevidence at gmail.com) Date: Tue, 3 May 2016 03:00:22 -0700 (PDT) Subject: Not x.islower() has different output than x.isupper() in list output... In-Reply-To: References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: On Friday, April 29, 2016 at 6:55:56 PM UTC-7, Christopher Reimer wrote: > On 4/29/2016 6:29 PM, Stephen Hansen wrote: > > If isupper/islower were perfect opposites of each-other, there'd be no > > need for both. But since characters can be upper, lower, or *neither*, > > you run into this situation. > > Based upon the official documentation, I was expecting perfect opposites. > > str.islower(): "Return true if all cased characters [4] in the string > are lowercase and there is at least one cased character, false otherwise." > > https://docs.python.org/3/library/stdtypes.html?highlight=islower#str.islower > > str.isupper(): "Return true if all cased characters [4] in the string > are uppercase and there is at least one cased character, false otherwise." > > https://docs.python.org/3/library/stdtypes.html?highlight=isupper#str.isupper Just to take this discussion in a more pure logic direction. What you call perfect opposites (that is, the functions being negations of each other) is not what the similar wording in the documentation actually implies: you shouldn't have been expecting that. What you should have been expecting is a symmetry. Say you have a string G. islower(G) will return a certain result. Now take every letter in G and swap the case, and call that string g. isupper(g) will always return the same result is islower(G). More succinctly, for any string x, the following is always ture: islower(x) == isupper(swapcase(x)) But that is not the same thing, and does not imply, as the following identity (which it turns out is not always true, as we've seen): islower(x) == not isupper(x) Another example of functions that behave like this are ispositive and isnegative. The identity "ispositive(x) == isnegative(-x)" is always true. However, "ispositive(x) == not isnegative(x)" is false if x == 0. However, I can understand your confusion, because there are some pairs of functions where both identities are true, and if you've seen a few of them it's fairly easy for your intuition to overgeneralize a bit. An example I can think of offhand is iseven(x) and isodd(x), for any integer x. The identities "iseven(x) == isodd(x^1)" and "iseven(x) == not isodd(x)" are both always true. Carl Banks From myth.services.eg at gmail.com Tue May 3 06:04:19 2016 From: myth.services.eg at gmail.com (Tarek Abulnaga) Date: Tue, 3 May 2016 03:04:19 -0700 (PDT) Subject: Python with Oracle Database Course. Message-ID: <5fe33743-a1dd-4d31-b2c8-4aab568bf358@googlegroups.com> I have created Python with Oracle Database Course, and offer 50% discount till 10-5-2016. Any one interested in course just click on below link. https://www.udemy.com/using-python-with-oracle-db/? Then write coupon Code=PYTHON_ORACLE_50%25 From rosuav at gmail.com Tue May 3 06:25:17 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 3 May 2016 20:25:17 +1000 Subject: Not x.islower() has different output than x.isupper() in list output... In-Reply-To: References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: On Tue, May 3, 2016 at 8:00 PM, wrote: > > What you should have been expecting is a symmetry. Say you have a string G. islower(G) will return a certain result. Now take every letter in G and swap the case, and call that string g. isupper(g) will always return the same result is islower(G). > > More succinctly, for any string x, the following is always ture: > > islower(x) == isupper(swapcase(x)) > > But that is not the same thing, and does not imply, as the following identity (which it turns out is not always true, as we've seen): > > islower(x) == not isupper(x) > > > Another example of functions that behave like this are ispositive and isnegative. The identity "ispositive(x) == isnegative(-x)" is always true. However, "ispositive(x) == not isnegative(x)" is false if x == 0. > This assumes, of course, that there is a function swapcase which can return a string with case inverted. I'm not sure such a function exists. ChrisA From gengyangcai at gmail.com Tue May 3 07:12:55 2016 From: gengyangcai at gmail.com (Cai Gengyang) Date: Tue, 3 May 2016 04:12:55 -0700 (PDT) Subject: Python Madlibs.py code and error message In-Reply-To: <5721baee$0$22141$c3e8da3$5496439d@news.astraweb.com> References: <3cff626c-28a2-499c-9877-de2df4d459d0@googlegroups.com> <6c0d96c5-c7f5-4659-a2e9-20f8d202d701@googlegroups.com> <1461820617.3238425.591942249.1484DAC3@webmail.messagingengine.com> <85fuu6jo1o.fsf@benfinney.id.au> <1461826246.3257796.591992793.54C5CC58@webmail.messagingengine.com> <85bn4ujk7p.fsf@benfinney.id.au> <1461827330.3261212.591998673.4D215991@webmail.messagingengine.com> <5721baee$0$22141$c3e8da3$5496439d@news.astraweb.com> Message-ID: <944d06f6-4a96-4cf7-96f9-1df082880bdc@googlegroups.com> Ok, I got it to work with no error message finally ... Enter a name: cai gengyang Enter an adjective: beautiful Enter a second adjective: honest Enter a third adjective: pretty Enter a verb: hit Enter a second verb: run Enter a third verb: jump Enter a noun: honesty Enter a noun: patience Enter a noun: happiness Enter a noun: danger Enter an animal: elephant Enter a food: burger Enter a fruit: watermelon Enter a number: 1985 Enter a superhero_name: batman Enter a country: america Enter a dessert: icekachang Enter a year: 1984 This morning I woke up and felt because _ was going to finally h e big _ honest. On the other sid onesty were many patiences prote eep elephant in stores. The crow _ to the rythym of the burger, all of the runs very _. happine o _ into the sewers and found wa ats. Needing help, pretty quickl ump. 1985 appeared and saved cai by flying to batman and dropping puddle of america. icekachang th leep and woke up in the year 198 rld where dangers ruled the worl $ On Thursday, April 28, 2016 at 3:25:46 PM UTC+8, Steven D'Aprano wrote: > On Thursday 28 April 2016 17:08, Stephen Hansen wrote: > > > On Wed, Apr 27, 2016, at 11:55 PM, Ben Finney wrote: > >> Stephen Hansen writes: > >> > >> > On Wed, Apr 27, 2016, at 10:32 PM, Ben Finney wrote: > >> > > Better: when you have many semantically-different values, use named > >> > > (not positional) parameters in the format string. [...] > >> > > > >> > > > >> > > >> > Except the poster is not using Python 3, so all of this is for naught. > >> > >> Everything I described above works fine in Python 2. Any still-supported > >> version has 'str.format'. > > > > This response is completely unhelpful. The OP is using Python 2, and > > using %-formatting, and so you give a series of examples of using > > str.format, to, what? Confuse matters? > > How about we assume good faith and give Ben the benefit of the doubt that he > simply made a minor and trivial misjudgement rather than accusing him of > intentionally trying to confuse matters? > > You are correct that the OP can use % formatting with named arguments. Ben > is correct that the OP can also change his code to use str.format. Some > people hate %-formatting and cannot wait to migrate to {}-formatting, and > some people don't. > > (For the record, Internet rumours that %-formatting is deprecated are simply > not correct.) > > > > -- > Steve From gengyangcai at gmail.com Tue May 3 07:20:54 2016 From: gengyangcai at gmail.com (Cai Gengyang) Date: Tue, 3 May 2016 04:20:54 -0700 (PDT) Subject: How to become more motivated to learn Python Message-ID: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> So I have completed up to CodeAcademy's Python Unit 2 , now moving on to Unit3 : Conditionals and Control Flow. But I feel my motivation wavering , at times I get stuck and frustrated when trying to learn a new programming language ? This might not be a technical question per say, but it is a Python programming related one. How do you motivate a person (either yourself or your child) to become more interested in programming and stick with it ? Is determination in learning (especially in a tough field like software) partly genetic ? Related , This is a very well written essay on determination by Paul Graham ----------------------------------------http://www.paulgraham.com/determination.html Gengyang From jussi.piitulainen at helsinki.fi Tue May 3 07:25:18 2016 From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen) Date: Tue, 03 May 2016 14:25:18 +0300 Subject: Not x.islower() has different output than x.isupper() in list output... References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: Chris Angelico writes: > This assumes, of course, that there is a function swapcase which can > return a string with case inverted. I'm not sure such a function > exists. str.swapcase("foO") 'FOo' From flebber.crue at gmail.com Tue May 3 07:34:09 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Tue, 3 May 2016 04:34:09 -0700 (PDT) Subject: Trouble porting glob bash behavior with argparse to windows shell Message-ID: <80f6672a-2b69-4749-821d-a92be107862a@googlegroups.com> Hi I had a simple argparse working on ubuntu bash. However now I am trying to run the script on windows and it cannot work because cmd doesn't handle the glob like bash does. So I am attempting to modify my script to accommodate. As i am running python 3.5 i can use glob.glob for a list of files I believe. Now I am specifying my arguments as 2 arguments path and extension python script.py /mypath/XML *xml This is my current error I have had many. TypeError was unhandled by user code Message: can only concatenate list (not "str") to list ' import argparse import glob parser = argparse.ArgumentParser(description=None) def GetArgs(parser): """Parser function using argparse""" # parser.add_argument('directory', help='directory use', # action='store', nargs='*') parser.add_argument("path", nargs="+") parser.add_argument('-e', '--extension', default='', help='File extension to filter by.') args = parser.parse_args() files = set() files |= set(glob.glob(args.path + '/*' + args.extension)) return files fileList = GetArgs(parser) for file in fileList: print(file) At this became unsure of where to troubleshoot further. Sayth From flebber.crue at gmail.com Tue May 3 07:46:57 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Tue, 3 May 2016 04:46:57 -0700 (PDT) Subject: How to become more motivated to learn Python In-Reply-To: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> References: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> Message-ID: <54ef0b25-438f-4c7e-b5c7-72b9193d27d0@googlegroups.com> On Tuesday, 3 May 2016 21:21:21 UTC+10, Cai Gengyang wrote: > So I have completed up to CodeAcademy's Python Unit 2 , now moving on to Unit3 : Conditionals and Control Flow. > > But I feel my motivation wavering , at times I get stuck and frustrated when trying to learn a new programming language ? > > This might not be a technical question per say, but it is a Python programming related one. How do you motivate a person (either yourself or your child) to become more interested in programming and stick with it ? Is determination in learning (especially in a tough field like software) partly genetic ? > > Related , This is a very well written essay on determination by Paul Graham ----------------------------------------http://www.paulgraham.com/determination.html > > Gengyang I have more determination than skill, look at my code :-). Why, and how can you benefit? Goals, not just related to Python but goals about what it and my other leanings combined will provide. I doubt whether if you are learning for learning sake that there is a possible motivation. In short I coach in corporate world for a living and it becomes quite clear and simple on how to get people to achieve goals and meet KRA's its not magic. Listen -> Identify values, beliefs and goals -> Discuss vision and outcomes of taking a new action (ambitions) -> Relate the business request to this action -> then tie the new action to their values what it brings and you have also tied them to the business strategy and goals. As Tony Robbins said it takes a least 3 legs to make a stool. Values - Ambitions - Actions Listen (in this case to yourself), identify key values, beliefs and family goals. Then ask yourself how by learning python can I will enrich one or all of these facets of my life. Tie the action of learning to a goal which defines your vision and your action learning python. With a bonus it should hopefully give you more clarity into what sectors of Python you want to learn, the great thing about Python is that it is in so many domains. Hope it helps Sayth From fabiofz at gmail.com Tue May 3 07:55:40 2016 From: fabiofz at gmail.com (Fabio Zadrozny) Date: Tue, 3 May 2016 08:55:40 -0300 Subject: installing scipy In-Reply-To: References: Message-ID: Are you sure that the Python34 you installed is 64-bits and not the 32-bit version? (you can check that by just executing 'python'... the prompt will show the proper info to you). On Tue, Apr 26, 2016 at 12:33 PM, Heli wrote: > Hi all, > > I have a python34 installed on a windows-64bit machine. I am using Eclipse > pydev editor. I need to used griddata from scipy.interpolate. > > I have installed scipy using by downloading the followng wheel file: > scipy-0.17.0-cp34-none-win_amd64 > > and isntalling using pip install. The install is successful, but > > It seems like scipy is not installed correctly and I get the following > error when trying to import and use scipy modules: > > C:\Python34\Scripts>python -c "import scipy.interpolate" > Traceback (most recent call last): > File "", line 1, in > File "C:\Python34\lib\site-packages\scipy\interpolate\__init__.py", line > 158, in > from .interpolate import * > File "C:\Python34\lib\site-packages\scipy\interpolate\interpolate.py", > line 11, in > import scipy.linalg > File "C:\Python34\lib\site-packages\scipy\linalg\__init__.py", line 174, > in > from .misc import * > File "C:\Python34\lib\site-packages\scipy\linalg\misc.py", line 5, in > > from .blas import get_blas_funcs > File "C:\Python34\lib\site-packages\scipy\linalg\blas.py", line 155, in > > from scipy.linalg import _fblas > ImportError: DLL load failed > > How can I fix this problem? > > Thanks, > -- > https://mail.python.org/mailman/listinfo/python-list > From flebber.crue at gmail.com Tue May 3 07:59:36 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Tue, 3 May 2016 04:59:36 -0700 (PDT) Subject: installing scipy In-Reply-To: References: Message-ID: <19c314cd-66fd-455f-acde-478015bf6024@googlegroups.com> On Wednesday, 27 April 2016 01:33:57 UTC+10, Heli wrote: > Hi all, > > I have a python34 installed on a windows-64bit machine. I am using Eclipse pydev editor. I need to used griddata from scipy.interpolate. > > I have installed scipy using by downloading the followng wheel file: > scipy-0.17.0-cp34-none-win_amd64 > > and isntalling using pip install. The install is successful, but > > It seems like scipy is not installed correctly and I get the following error when trying to import and use scipy modules: > > C:\Python34\Scripts>python -c "import scipy.interpolate" > Traceback (most recent call last): > File "", line 1, in > File "C:\Python34\lib\site-packages\scipy\interpolate\__init__.py", line 158, in > from .interpolate import * > File "C:\Python34\lib\site-packages\scipy\interpolate\interpolate.py", line 11, in > import scipy.linalg > File "C:\Python34\lib\site-packages\scipy\linalg\__init__.py", line 174, in > from .misc import * > File "C:\Python34\lib\site-packages\scipy\linalg\misc.py", line 5, in > from .blas import get_blas_funcs > File "C:\Python34\lib\site-packages\scipy\linalg\blas.py", line 155, in > from scipy.linalg import _fblas > ImportError: DLL load failed > > How can I fix this problem? > > Thanks, On windows use Anaconda. https://www.continuum.io/downloads Sayth From rosuav at gmail.com Tue May 3 08:00:45 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 3 May 2016 22:00:45 +1000 Subject: Not x.islower() has different output than x.isupper() in list output... In-Reply-To: References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: On Tue, May 3, 2016 at 9:25 PM, Jussi Piitulainen wrote: > Chris Angelico writes: > >> This assumes, of course, that there is a function swapcase which can >> return a string with case inverted. I'm not sure such a function >> exists. > > str.swapcase("foO") > 'FOo' I suppose for this discussion it doesn't matter if it's imperfect. >>> "\N{ANGSTROM SIGN}".swapcase().swapcase() == "\N{ANGSTROM SIGN}" False >>> "\N{LATIN SMALL LETTER SHARP S}".swapcase().swapcase() 'ss' But drawing the analogy with the negation of real numbers implies something that doesn't exist. ChrisA From drewes.mil at gmail.com Tue May 3 08:14:47 2016 From: drewes.mil at gmail.com (drewes.mil at gmail.com) Date: Tue, 3 May 2016 05:14:47 -0700 (PDT) Subject: Saving Consol outputs in a python script Message-ID: Hello, I'm new to python and have a Question. I'm running a c++ file with a python script like: import os import subprocess subprocess.call(["~/caffe/build/examples/cpp_classification/classification", "deploy.prototxt", "this.caffemodel", "mean.binaryproto", "labels.txt", "Bild2.jpg"]) and it runes fine. On the console it gives me the output: ~/Desktop/Downloader/Sym+$ python Run_C.py ---------- Prediction for Bild2.jpg ---------- 0.9753 - "Class 1" 0.0247 - "Class 2" What I need are the 2 values for the 2 classes saved in a variable in the .py script, so that I can write them into a text file. Would be super nice if someone could help me! have a nice day! Steffen From nospam at dfs.com Tue May 3 09:01:53 2016 From: nospam at dfs.com (DFS) Date: Tue, 3 May 2016 09:01:53 -0400 Subject: Not x.islower() has different output than x.isupper() in list output... In-Reply-To: References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: On 5/3/2016 8:00 AM, Chris Angelico wrote: > On Tue, May 3, 2016 at 9:25 PM, Jussi Piitulainen > wrote: >> Chris Angelico writes: >> >>> This assumes, of course, that there is a function swapcase which can >>> return a string with case inverted. I'm not sure such a function >>> exists. >> >> str.swapcase("foO") >> 'FOo' > > I suppose for this discussion it doesn't matter if it's imperfect. What was imperfect? From rosuav at gmail.com Tue May 3 09:13:21 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 3 May 2016 23:13:21 +1000 Subject: Not x.islower() has different output than x.isupper() in list output... In-Reply-To: References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: On Tue, May 3, 2016 at 11:01 PM, DFS wrote: > On 5/3/2016 8:00 AM, Chris Angelico wrote: >> >> On Tue, May 3, 2016 at 9:25 PM, Jussi Piitulainen >> wrote: >>> >>> Chris Angelico writes: >>> >>>> This assumes, of course, that there is a function swapcase which can >>>> return a string with case inverted. I'm not sure such a function >>>> exists. >>> >>> >>> str.swapcase("foO") >>> 'FOo' >> >> >> I suppose for this discussion it doesn't matter if it's imperfect. > > > > What was imperfect? It doesn't invert, the way numeric negation does. And if you try to define exactly what it does, you'll come right back to isupper()/islower(), so it's not much help in defining those. ChrisA From nospam at dfs.com Tue May 3 09:19:06 2016 From: nospam at dfs.com (DFS) Date: Tue, 3 May 2016 09:19:06 -0400 Subject: Not x.islower() has different output than x.isupper() in list output... In-Reply-To: References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: On 5/3/2016 9:13 AM, Chris Angelico wrote: > On Tue, May 3, 2016 at 11:01 PM, DFS wrote: >> On 5/3/2016 8:00 AM, Chris Angelico wrote: >>> >>> On Tue, May 3, 2016 at 9:25 PM, Jussi Piitulainen >>> wrote: >>>> >>>> Chris Angelico writes: >>>> >>>>> This assumes, of course, that there is a function swapcase which can >>>>> return a string with case inverted. I'm not sure such a function >>>>> exists. >>>> >>>> >>>> str.swapcase("foO") >>>> 'FOo' >>> >>> >>> I suppose for this discussion it doesn't matter if it's imperfect. >> >> >> >> What was imperfect? > > It doesn't invert, the way numeric negation does. What do you mean by 'case inverted'? It looks like it swaps the case correctly between upper and lower. > And if you try to > define exactly what it does, you'll come right back to > isupper()/islower(), so it's not much help in defining those. > > ChrisA From moa47401 at gmail.com Tue May 3 09:21:22 2016 From: moa47401 at gmail.com (moa47401 at gmail.com) Date: Tue, 3 May 2016 06:21:22 -0700 (PDT) Subject: Need help understanding list structure In-Reply-To: References: <5727CB31.5060309@lucidity.plus.com> <85mvo8gh57.fsf@benfinney.id.au> Message-ID: Thanks for the replies. I definitely need a better understanding of "" when using Python objects. So far no luck with web searches or my Python books. Could someone point (no pun intended) me to a good resource? Not that it matters, but the reason I got off track is there are pointers within my data that point to other pieces of the data and have nothing to do with Python. From rosuav at gmail.com Tue May 3 09:23:37 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 3 May 2016 23:23:37 +1000 Subject: Not x.islower() has different output than x.isupper() in list output... In-Reply-To: References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: On Tue, May 3, 2016 at 11:19 PM, DFS wrote: > What do you mean by 'case inverted'? > > It looks like it swaps the case correctly between upper and lower. I gave two examples in my previous post. Did you read them? You trimmed them from the quote. ChrisA From rosuav at gmail.com Tue May 3 09:47:18 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 3 May 2016 23:47:18 +1000 Subject: Need help understanding list structure In-Reply-To: References: <5727CB31.5060309@lucidity.plus.com> <85mvo8gh57.fsf@benfinney.id.au> Message-ID: On Tue, May 3, 2016 at 11:21 PM, wrote: > Thanks for the replies. I definitely need a better understanding of "" when using Python objects. So far no luck with web searches or my Python books. Could someone point (no pun intended) me to a good resource? > > Not that it matters, but the reason I got off track is there are pointers within my data that point to other pieces of the data and have nothing to do with Python. What you're seeing there is the default object representation. It isn't actually quoting an address but an "identity number". The only meaning of that number is that it's unique [1]; so if you see the same number come up twice in the list, you can be confident that the list has two references to the same object. Beyond that, it basically tells you nothing. So you know what kind of object it is, but anything else you'll have to figure out for yourself. ChrisA [1] Among concurrently-existing objects; if an object disappears, its ID can be reused. From nospam at dfs.com Tue May 3 10:32:19 2016 From: nospam at dfs.com (DFS) Date: Tue, 3 May 2016 10:32:19 -0400 Subject: Saving Consol outputs in a python script In-Reply-To: References: Message-ID: On 5/3/2016 8:14 AM, drewes.mil at gmail.com wrote: > Hello, I'm new to python and have a Question. > > I'm running a c++ file with a python script like: > > import os > import subprocess > > subprocess.call(["~/caffe/build/examples/cpp_classification/classification", "deploy.prototxt", "this.caffemodel", "mean.binaryproto", "labels.txt", "Bild2.jpg"]) > > and it runes fine. On the console it gives me the output: > > ~/Desktop/Downloader/Sym+$ python Run_C.py > ---------- Prediction for Bild2.jpg ---------- > 0.9753 - "Class 1" > 0.0247 - "Class 2" > > > What I need are the 2 values for the 2 classes saved in a variable in the .py script, so that I can write them into a text file. > > Would be super nice if someone could help me! This looks like the ticket: http://eli.thegreenplace.net/2015/redirecting-all-kinds-of-stdout-in-python/ > have a nice day! > > Steffen From jussi.piitulainen at helsinki.fi Tue May 3 10:42:28 2016 From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen) Date: Tue, 03 May 2016 17:42:28 +0300 Subject: Not x.islower() has different output than x.isupper() in list output... References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: Chris Angelico writes: > On Tue, May 3, 2016 at 9:25 PM, Jussi Piitulainen wrote: >> Chris Angelico writes: >> >>> This assumes, of course, that there is a function swapcase which can >>> return a string with case inverted. I'm not sure such a function >>> exists. >> >> str.swapcase("foO") >> 'FOo' > > I suppose for this discussion it doesn't matter if it's imperfect. Not sure. I may have misunderstood what this discussion is about - I thought you had forgotten that Python has this function :) >>>> "\N{ANGSTROM SIGN}".swapcase().swapcase() == "\N{ANGSTROM SIGN}" > False >>>> "\N{LATIN SMALL LETTER SHARP S}".swapcase().swapcase() > 'ss' > > But drawing the analogy with the negation of real numbers implies > something that doesn't exist. I missed the relevance of that analogy. Ceterum censeo, the only suggested use for .swapcase I've ever heard of is encryption. From jussi.piitulainen at helsinki.fi Tue May 3 10:49:48 2016 From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen) Date: Tue, 03 May 2016 17:49:48 +0300 Subject: Not x.islower() has different output than x.isupper() in list output... References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: DFS writes: > On 5/3/2016 9:13 AM, Chris Angelico wrote: >> It doesn't invert, the way numeric negation does. > > What do you mean by 'case inverted'? > > It looks like it swaps the case correctly between upper and lower. There's letters that do not come in exact pairs of upper and lower case, so _some_ swaps are not invertible: you swap twice and end up somewhere else than your starting point. The "\N{ANSGTROM SIGN}" looks like the Swedish upper-case a-with-ring-above but isn't the same character, yet Python swaps its case to the actual lower-case a-with-ring above. It can't go back to _both_ the Angstrom sign and the actual upper case letter. (Not sure why the sign is considered a cased letter at all.) From nospam at dfs.com Tue May 3 11:12:50 2016 From: nospam at dfs.com (DFS) Date: Tue, 3 May 2016 11:12:50 -0400 Subject: Not x.islower() has different output than x.isupper() in list output... In-Reply-To: References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: On 5/3/2016 10:49 AM, Jussi Piitulainen wrote: > DFS writes: > >> On 5/3/2016 9:13 AM, Chris Angelico wrote: > >>> It doesn't invert, the way numeric negation does. >> >> What do you mean by 'case inverted'? >> >> It looks like it swaps the case correctly between upper and lower. > > There's letters that do not come in exact pairs of upper and lower case, > so _some_ swaps are not invertible: you swap twice and end up somewhere > else than your starting point. > > The "\N{ANSGTROM SIGN}" looks like the Swedish upper-case > a-with-ring-above but isn't the same character, yet Python swaps its > case to the actual lower-case a-with-ring above. It can't go back to > _both_ the Angstrom sign and the actual upper case letter. > > (Not sure why the sign is considered a cased letter at all.) Thanks for the explanation. Does that mean: lower(?) != ? ? and upper(?) != ? ? From jussi.piitulainen at helsinki.fi Tue May 3 11:27:13 2016 From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen) Date: Tue, 03 May 2016 18:27:13 +0300 Subject: Not x.islower() has different output than x.isupper() in list output... References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: DFS writes: > On 5/3/2016 10:49 AM, Jussi Piitulainen wrote: >> DFS writes: >> >>> On 5/3/2016 9:13 AM, Chris Angelico wrote: >> >>>> It doesn't invert, the way numeric negation does. >>> >>> What do you mean by 'case inverted'? >>> >>> It looks like it swaps the case correctly between upper and lower. >> >> There's letters that do not come in exact pairs of upper and lower case, >> so _some_ swaps are not invertible: you swap twice and end up somewhere >> else than your starting point. >> >> The "\N{ANSGTROM SIGN}" looks like the Swedish upper-case >> a-with-ring-above but isn't the same character, yet Python swaps its >> case to the actual lower-case a-with-ring above. It can't go back to >> _both_ the Angstrom sign and the actual upper case letter. >> >> (Not sure why the sign is considered a cased letter at all.) > > > Thanks for the explanation. > > Does that mean: > > lower(?) != ? ? > > and > > upper(?) != ? ? It means "\N{ANGSTROM SIGN}" != "?", yet both lower to "?", which then uppers back to "?" (U+00c5). The ?ngstr?m sign (U+212b) looks like this: ?. Indistinguishable from ? in the font that I'm seeing - for all I know, it's the same glyph. From python.list at tim.thechases.com Tue May 3 11:28:50 2016 From: python.list at tim.thechases.com (Tim Chase) Date: Tue, 3 May 2016 10:28:50 -0500 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> <1462166136.1167243.595273897.291B0865@webmail.messagingengine.com> <1462170452.1180117.595306673.68B64F02@webmail.messagingengine.com> <572823D8.6050204@gmail.com> Message-ID: <20160503102850.63cec18a@bigbox.christie.dr> On 2016-05-03 00:24, DFS wrote: > One small comparison I was able to make was VBA vs python/pyodbc to > summarize an Access database. Not quite a fair test, but > interesting nonetheless. > > Access 2003 file > Access 2003 VBA code > Time: 0.18 seconds > > same Access 2003 file > 32-bit python 2.7.11 + 32-bit pyodbc 3.0.6 > Time: 0.49 seconds Curious whether you're forcing Access VBA to talk over ODBC or whether Access is using native access/file-handling (and thus bypassing the ODBC overhead)? -tkc From grant.b.edwards at gmail.com Tue May 3 11:42:32 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 3 May 2016 15:42:32 +0000 (UTC) Subject: Not x.islower() has different output than x.isupper() in list output... References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: On 2016-05-03, Jussi Piitulainen wrote: >> Does that mean: >> >> lower(?) != ? ? >> >> and >> >> upper(?) != ? ? > > It means "\N{ANGSTROM SIGN}" != "?", yet both lower to "?", which then > uppers back to "?" (U+00c5). > > The ?ngstr?m sign (U+212b) looks like this: ?. Indistinguishable from ? > in the font that I'm seeing - for all I know, it's the same glyph. Interesting. FWIW, ? and ? definitely look different with the terminal and font I'm using (urxvt with -misc-fixed-medium-r-normal-*-18-120-*-*-*-90-iso10646-*) Expecting upper/lower operations to be 100% invertible is probably a ASCII-centric mindset that will falls over as soon as you start dealing with non-ASCII encodings. -- Grant Edwards grant.b.edwards Yow! Xerox your lunch at and file it under "sex gmail.com offenders"! From me+python at ixokai.io Tue May 3 11:43:59 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Tue, 03 May 2016 08:43:59 -0700 Subject: Saving Consol outputs in a python script In-Reply-To: References: Message-ID: <1462290239.1609819.596871457.7356F4C8@webmail.messagingengine.com> On Tue, May 3, 2016, at 05:14 AM, drewes.mil at gmail.com wrote: > What I need are the 2 values for the 2 classes saved in a variable in the > .py script, so that I can write them into a text file. > > Would be super nice if someone could help me! You shouldn't use the call() convienence function, but instead create a process using the Popen constructor, passing PIPE to stdout. Then use communicate() to get the output. This should get you started: process = subprocess.Popen(["commandline"], stdout=subprocess.PIPE) output, error = process.communicate() Output will be a string, string has a splitlines method, etc. -- Stephen Hansen m e @ i x o k a i . i o From moa47401 at gmail.com Tue May 3 12:01:07 2016 From: moa47401 at gmail.com (moa47401 at gmail.com) Date: Tue, 3 May 2016 09:01:07 -0700 (PDT) Subject: Need help understanding list structure In-Reply-To: References: <5727CB31.5060309@lucidity.plus.com> <85mvo8gh57.fsf@benfinney.id.au> Message-ID: At the risk of coming across as a complete dunder-head, I think my confusion has to do with the type of data the library returns in the list. Any kind of text or integer list I manually create, doesn't do this. See my questions down below at the end. If I run the following statements on the list returned by the gedcom library: print(type(myList)) print(len(myList)) print(myList[0]) print(myList[0:29]) print(myList) for x in myList: print(x) I get this: 29 0 HEAD [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 0 HEAD 1 SOUR AncestQuest 2 NAME Ancestral Quest 2 VERS 14.00.9 2 CORP Incline Software, LC 3 ADDR PO Box 95543 4 CONT South Jordan, UT 84095 4 CONT USA 1 DATE 3 MAY 2016 2 TIME 10:44:10 1 FILE test_gedcom.ged 1 GEDC 2 VERS 5.5 2 FORM LINEAGE-LINKED 1 CHAR ANSEL 0 @I1@ INDI 1 NAME John /Allen/ 1 SEX M 1 BIRT 2 DATE 1750 2 PLAC VA 1 DEAT 2 DATE 1804 2 PLAC KY 1 _UID D6C103E6105D654B85D47DA1B36E474BC7D1 1 CHAN 2 DATE 3 MAY 2016 3 TIME 10:43:35 0 TRLR Questions: Why does printing a single item print the actual text of the object? Why does printing a range print the "representations" of the objects? Why does iterating over the list print the actual text of the objects? How can I determine what type of data is in the list? From me+python at ixokai.io Tue May 3 12:09:38 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Tue, 03 May 2016 09:09:38 -0700 Subject: You gotta love a 2-line python solution In-Reply-To: <21c6d3df-3346-49b5-ac66-76b977c5aef5@googlegroups.com> References: <1462247340.1450624.596312857.04969060@webmail.messagingengine.com> <21c6d3df-3346-49b5-ac66-76b977c5aef5@googlegroups.com> Message-ID: <1462291778.1617121.596905937.17231963@webmail.messagingengine.com> On Mon, May 2, 2016, at 08:57 PM, jfong at ms4.hinet.net wrote: > Stephen Hansen at 2016/5/3 11:49:22AM wrote: > > On Mon, May 2, 2016, at 08:27 PM, jfong at ms4.hinet.net wrote: > > > But when I try to get this forum page, it does get a html file but can't > > > be viewed normally. > > > > What does that mean? > > > > -- > > Stephen Hansen > > m e @ i x o k a i . i o > > The page we are looking at:-) > https://groups.google.com/forum/#!topic/comp.lang.python/jFl3GJbmR7A Try scraping gmane. Google Groups is one big javascript application. -- Stephen Hansen m e @ i x o k a i . i o From tjreedy at udel.edu Tue May 3 12:37:25 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 3 May 2016 12:37:25 -0400 Subject: Not x.islower() has different output than x.isupper() in list output... In-Reply-To: References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: On 5/3/2016 11:42 AM, Grant Edwards wrote: > Interesting. FWIW, ? and ? definitely look different with the terminal > and font I'm using (urxvt with -misc-fixed-medium-r-normal-*-18-120-*-*-*-90-iso10646-*) In the fixed pitch font used by Thunderbird (Courier?), Angstrom ? has the circle touching the A while letter ? has the circle spaced above. -- Terry Jan Reedy From hemla21 at gmail.com Tue May 3 12:39:35 2016 From: hemla21 at gmail.com (Heli) Date: Tue, 3 May 2016 09:39:35 -0700 (PDT) Subject: installing scipy In-Reply-To: References: Message-ID: Yes, the python I have installed is 64bit. Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32 and the scipy wheel I am trying to install from is : scipy-0.17.0-cp34-none-win_amd64.whl At Sayth: Thanks for recommending Anaconda. I already am familiar with it, but I was wondering why the scipy installed on my normal python gives the following error : mportError: DLL load failed Thanks for your comments, From tjreedy at udel.edu Tue May 3 12:50:00 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 3 May 2016 12:50:00 -0400 Subject: Saving Consol outputs in a python script In-Reply-To: References: Message-ID: On 5/3/2016 8:14 AM, drewes.mil at gmail.com wrote: > Hello, I'm new to python and have a Question. > > I'm running a c++ file with a python script like: > > import os > import subprocess > > subprocess.call(["~/caffe/build/examples/cpp_classification/classification", "deploy.prototxt", "this.caffemodel", "mean.binaryproto", "labels.txt", "Bild2.jpg"]) > > and it runes fine. On the console it gives me the output: > > ~/Desktop/Downloader/Sym+$ python Run_C.py > ---------- Prediction for Bild2.jpg ---------- > 0.9753 - "Class 1" > 0.0247 - "Class 2" > > > What I need are the 2 values for the 2 classes saved in a variable in the .py script, so that I can write them into a text file. pycaffe is a python interface to caffe. You should look into it and probably use it. Its functions will return python objects. I suspect that is has a function that will return the result of calling the caffe classification function Stackoverflow has question/answers tagged with 'caffe' and 'pycaffe'. (There is also a caffe-users group on google groups.) -- Terry Jan Reedy From rgaddi at highlandtechnology.invalid Tue May 3 12:50:36 2016 From: rgaddi at highlandtechnology.invalid (Rob Gaddi) Date: Tue, 3 May 2016 16:50:36 -0000 (UTC) Subject: How to become more motivated to learn Python References: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> Message-ID: Cai Gengyang wrote: > So I have completed up to CodeAcademy's Python Unit 2 , now moving on to Unit3 : Conditionals and Control Flow. > > But I feel my motivation wavering , at times I get stuck and frustrated when trying to learn a new programming language ? > > This might not be a technical question per say, but it is a Python programming related one. How do you motivate a person (either yourself or your child) to become more interested in programming and stick with it ? Is determination in learning (especially in a tough field like software) partly genetic ? > > Related , This is a very well written essay on determination by Paul Graham ----------------------------------------http://www.paulgraham.com/determination.html > > Gengyang You don't. Learning programming is dull and sloggy and inherently unmotiving. Now, solving a problem, on the other hand. Solving a problem is fun. A real problem, an actual task that you actually need to do, not FizzBang. The thorny, nasty, horrible problems are great fun, and when you beat them into submission and mount their heads on your wall, but even the little ones like "I've got an 8GB USB stick, I want to put a random selection of all my MP3 files onto it." are entertaining. The Python's not the point. It can never be the point. Have a thing you want to do, and not just "Get a high paying job." If the tool for doing that thing is Python, so be it. If you need a soldering iron, or a hammer and chisel, or a structural engineering degree instead, then go figure out how to use one of those and Python will still be waiting when you do need it. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. From D.Strohl at F5.com Tue May 3 12:52:53 2016 From: D.Strohl at F5.com (Dan Strohl) Date: Tue, 3 May 2016 16:52:53 +0000 Subject: Need help understanding list structure In-Reply-To: References: <5727CB31.5060309@lucidity.plus.com> <85mvo8gh57.fsf@benfinney.id.au> Message-ID: Take a look at the docs for print() https://docs.python.org/3.5/library/functions.html#print str() https://docs.python.org/3.5/library/stdtypes.html#str repr() https://docs.python.org/3.5/library/functions.html#repr When you do "print(object)", python will run everything through str() and output it. Str() will try to return a string representation of the object, what actually comes back will depend on how the objects author defined it. If object.__str__() has been defined, it will use that, if __str__() is not defined, it will use object.__repr__(). If object.__repr__() has not been defined, it will something that looks like "object_name object at xxxxxxx". So, as to your specific questions / comments: > At the risk of coming across as a complete dunder-head, I think my confusion > has to do with the type of data the library returns in the list. Any kind of text > or integer list I manually create, doesn't do this. Actually, they do, but strings and integers have well defined __str__ and __repr__ methods, and behave pretty well. So... think about what is actually being passed to the print() function in each case: > print(type(myList)) Passing the string object of the results of type(myList) > print(len(myList)) Passing the integer object returning from len(myList) > print(myList[0]) Passing the gedcom ELEMENT object found at location 0 in the list > print(myList[0:29]) Passing a list object created by copying the items from location 0 to location 29 in the original list > print(myList) Passing the entire list object > for x in myList: > print(x) Passing the individual gedcom ELEMENT objects from the list. So, in each of these cases, you are passing different types of objects, and each one (string, integer, list, gedcom) will behave differently depending on how it is coded. > Why does printing a single item print the actual text of the object? If you are printing a single item (print(myList[0]), you are printing the __str__ or __repr__ for the object stored at that location. > Why does printing a range print the "representations" of the objects? If you are printing a range of objects, you are printing the __str__ or __repr__ for the RANGE OBJECT or LIST object, not the object itself, which apparently only checks for a __repr__() method in the contained gedcom ELEMENT objects, which is not defined (see below). > Why does iterating over the list print the actual text of the objects? When iterating over the list, you are printing the specific items again (just like when you did print(myList[0]) ) > How can I determine what type of data is in the list? Try print(type(myList[0])), which will give you the type of data in the first object in the list (though, keep in mind that the object type could be different in each item in the list). If we look at the gedcom library (assuming you are talking about this one: https://github.com/madprime/python-gedcom/blob/master/gedcom/__init__.py) at the end of the file you can see they defined __str__() in the ELEMENT object, but there is no definition for __repr__(), which matches what we surmised above. If you want to fix it by editing the gedcom library, you could simply add a line at the end like: __repr__() = __str__() Hope that helps. Dan Strohl From nospam at dfs.com Tue May 3 13:00:34 2016 From: nospam at dfs.com (DFS) Date: Tue, 3 May 2016 13:00:34 -0400 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> <1462166136.1167243.595273897.291B0865@webmail.messagingengine.com> <1462170452.1180117.595306673.68B64F02@webmail.messagingengine.com> <572823D8.6050204@gmail.com> <20160503102850.63cec18a@bigbox.christie.dr> Message-ID: On 5/3/2016 11:28 AM, Tim Chase wrote: > On 2016-05-03 00:24, DFS wrote: >> One small comparison I was able to make was VBA vs python/pyodbc to >> summarize an Access database. Not quite a fair test, but >> interesting nonetheless. >> >> Access 2003 file >> Access 2003 VBA code >> Time: 0.18 seconds >> >> same Access 2003 file >> 32-bit python 2.7.11 + 32-bit pyodbc 3.0.6 >> Time: 0.49 seconds > > Curious whether you're forcing Access VBA to talk over ODBC or > whether Access is using native access/file-handling (and thus > bypassing the ODBC overhead)? The latter, which is why I said "not quite a fair test". From moa47401 at gmail.com Tue May 3 13:31:10 2016 From: moa47401 at gmail.com (moa47401 at gmail.com) Date: Tue, 3 May 2016 10:31:10 -0700 (PDT) Subject: Need help understanding list structure In-Reply-To: References: <5727CB31.5060309@lucidity.plus.com> <85mvo8gh57.fsf@benfinney.id.au> Message-ID: <221dcc70-39d8-4c9c-8827-8e0bc1ec1fda@googlegroups.com> I added a __repr__ method at the end of the gedcom library like so: def __repr__(self): """ Format this element as its original string """ result = repr(self.level()) if self.pointer() != "": result += ' ' + self.pointer() result += ' ' + self.tag() if self.value() != "": result += ' ' + self.value() return result and now I can print myList properly. Eric and Michael also mentioned repr above, but I guess I needed someone to spell it out for me. Thanks for taking the time to put it in terms an old dog could understand. From none at invalid.com Tue May 3 13:44:20 2016 From: none at invalid.com (mm0fmf) Date: Tue, 03 May 2016 18:44:20 +0100 Subject: How to become more motivated to learn Python In-Reply-To: References: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> Message-ID: On 03/05/2016 17:50, Rob Gaddi wrote: > Cai Gengyang wrote: > >> So I have completed up to CodeAcademy's Python Unit 2 , now moving on to Unit3 : Conditionals and Control Flow. >> >> But I feel my motivation wavering , at times I get stuck and frustrated when trying to learn a new programming language ? >> >> This might not be a technical question per say, but it is a Python programming related one. How do you motivate a person (either yourself or your child) to become more interested in programming and stick with it ? Is determination in learning (especially in a tough field like software) partly genetic ? >> >> Related , This is a very well written essay on determination by Paul Graham ----------------------------------------http://www.paulgraham.com/determination.html >> >> Gengyang > > You don't. Learning programming is dull and sloggy and inherently > unmotiving. > > Now, solving a problem, on the other hand. Solving a problem is fun. A > real problem, an actual task that you actually need to do, not > FizzBang. The thorny, nasty, horrible problems are great fun, and when > you beat them into submission and mount their heads on your wall, but > even the little ones like "I've got an 8GB USB stick, I want to put a > random selection of all my MP3 files onto it." are entertaining. > > The Python's not the point. It can never be the point. Have a thing > you want to do, and not just "Get a high paying job." If the tool for > doing that thing is Python, so be it. If you need a soldering iron, or a > hammer and chisel, or a structural engineering degree instead, then go > figure out how to use one of those and Python will still be waiting when > you do need it. > +1 I learned Python because... I had a problem to solve. First I had to recompile a Linux kernel for an oddball PPC based NAS I used a lowpower computer. Then I had to find the patches that fixed some USB issues. Then I had to make a USB GSM modem work as it was a nasty Windows Zero-CD system. Then I wrote a program in C/C++ to process data from the modem and play with the ASCII strings and then push them to a website. I used libcurl to do the web work. It was a nightmare to debug/enhance and handling raw C strings in C or C++ was a pain. And I thought there has to be a better way. I kept seeing mention of Python so I thought I'll rewrite this in Python. But I couldn't find Python 2.7.x for PPC so I had to get the Python source and build that from scrath and check it worked OK. It took about 7 days of deep hacking to be able to rewrite on a line by line basis the C/C++ in Python. Boy was it ugly. But it worked. Then as I enhanced the program it became more Pythonic and I starting using the language and not abusing it. And the more I used it the easier it became. That code is online on a cheap VM running 24/7 and has about 450 users worldwide growing by several a week. This group was huge source of inspiration looking at the replies people gave to questions. It also motivated me to stop faffing about and convert my Python2 code to Python3 which again was straightforward. So I learned Python to solve a real world problem and it took a huge amount of effort along the way to realise I had to get a better way to maintain a program that was now being relied on by others. I didn't set out to learn Python but I'm sure glad I did. In 33 years of code bashing, I found a language that let me think about the problem not the programming language. YMMV From D.Strohl at F5.com Tue May 3 13:54:38 2016 From: D.Strohl at F5.com (Dan Strohl) Date: Tue, 3 May 2016 17:54:38 +0000 Subject: Need help understanding list structure In-Reply-To: <221dcc70-39d8-4c9c-8827-8e0bc1ec1fda@googlegroups.com> References: <5727CB31.5060309@lucidity.plus.com> <85mvo8gh57.fsf@benfinney.id.au> <221dcc70-39d8-4c9c-8827-8e0bc1ec1fda@googlegroups.com> Message-ID: > I added a __repr__ method at the end of the gedcom library like so: > > def __repr__(self): > """ Format this element as its original string """ > result = repr(self.level()) > if self.pointer() != "": > result += ' ' + self.pointer() > result += ' ' + self.tag() > if self.value() != "": > result += ' ' + self.value() > return result > > and now I can print myList properly. > > Eric and Michael also mentioned repr above, but I guess I needed someone > to spell it out for me. Thanks for taking the time to put it in terms an old dog > could understand. > Glad to help! (being an old dog myself, I know the feeling!) One other point for you, if your "__repr__(self)" code is the same as the "__str__(self)" code (which it looks like it is, at a glance at least), you can instead reference the __str__ method and save having a duplicate code block... some examples: ========= Option 1: This is the easiest to read (IMHO) and allows for the possibility that str() is doing something here like formatting or whatever. (in this case it shouldn't be though). However, to call this actually is taking multiple steps (calling object.__repr__, whch calls str(), which calls object.__str__(). ) def __repr__(self): return str(self) ========= Option 2: this isn't hard to read, and just takes two steps (calling object.__repr__(), which calls object.__str__(). def __repr__(self): return self.__str__() ======== Option 3: it's not that this is hard to read, but since it doesn't follow the standard "def blah(self):" pattern, sometimes I overlook these in the code (even when I put them there). This however is the shortest since it really just tells the object to return object.__str__() if either object.__repr__() OR object.__str__() is called. __repr__ = __str__ This probably doesn't matter much in this case, since it probably isn't called that much in normal use (though there are always exceptions), and in the end, Python is fast enough that unless you really need to slice off a few milliseconds, you will never notice the difference, but just some food for thought. From ben+python at benfinney.id.au Tue May 3 14:14:39 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 04 May 2016 04:14:39 +1000 Subject: Use __repr__ to show the programmer's representation (was: Need help understanding list structure) References: <5727CB31.5060309@lucidity.plus.com> <85mvo8gh57.fsf@benfinney.id.au> <221dcc70-39d8-4c9c-8827-8e0bc1ec1fda@googlegroups.com> Message-ID: <85inyvgga8.fsf_-_@benfinney.id.au> Dan Strohl via Python-list writes: > One other point for you, if your "__repr__(self)" code is the same as > the "__str__(self)" code (which it looks like it is, at a glance at > least), you can instead reference the __str__ method and save having a > duplicate code block... Alternatively, consider: the ?__repr__? method is intended to return a *programmer's* representation of the object. Commonly, this is text which looks like the Python expression which would create an equal instance:: >>> foo = datetime.date.fromtimestamp(13012345678) >>> print(repr(foo)) datetime.date(2382, 5, 7) So if there is a sensible ?here is the expression that could have been used to create this instance? text, have the ?__repr__? method return that text:: >>> foo = LoremIpsum(bingle, bongle, bungle) >>> print(repr(foo)) packagename.LoremIpsum("spam", 753, frob=True) That text is very useful because it can be fed back into the interactive interpreter to make an equal-valued instance and experiment further. For some types, there isn't such an expression that would evaluate to an equal-valued instance of the type. So the conventional non-evaluating representation is used:: >>> foo = frobnicate_the_widget(widget) >>> print(repr(foo)) This gives the crucial information of what the type is, and also gives other interesting (to the programmer) attributes that characterise the specific instance. The fallback ?? is the least helpful; it gives the type and identity of the instance, but only because that's the lowest common information ?object? can guarantee. Always implement a more informative representation for your custom type, if you can. -- \ ?Intellectual property is to the 21st century what the slave | `\ trade was to the 16th.? ?David Mertz | _o__) | Ben Finney From D.Strohl at F5.com Tue May 3 14:35:48 2016 From: D.Strohl at F5.com (Dan Strohl) Date: Tue, 3 May 2016 18:35:48 +0000 Subject: Use __repr__ to show the programmer's representation (was: Need help understanding list structure) In-Reply-To: <85inyvgga8.fsf_-_@benfinney.id.au> References: <5727CB31.5060309@lucidity.plus.com> <85mvo8gh57.fsf@benfinney.id.au> <221dcc70-39d8-4c9c-8827-8e0bc1ec1fda@googlegroups.com> <85inyvgga8.fsf_-_@benfinney.id.au> Message-ID: <4c15c01e66314265b7251a16c3c3c235@seaexchmbx03.olympus.F5Net.com> > > One other point for you, if your "__repr__(self)" code is the same as > > the "__str__(self)" code (which it looks like it is, at a glance at > > least), you can instead reference the __str__ method and save having a > > duplicate code block... > > Alternatively, consider: the ?__repr__? method is intended to return a > *programmer's* representation of the object. Commonly, this is text which > looks like the Python expression which would create an equal > instance:: Definitely true per what _repr__ is supposed to do per python docs. However, in this case, that might not have solved the problem if the goal was to return the same as a __str__. (Though to be fair, I don?t really know what the actual problem was, so I might provide a different approach with a different goal ). I also have never actually used repr() to create code that could be fed back to the interpreter (not saying it isn?t done, just that I haven?t run into needing it), and there are so many of the libraries that do not return a usable repr string that I would hesitate to even try it outside of a very narrow use case. Personally, I normally use __repr__ to give me a useful troubleshooting representation of the object... but that representation might not be exactly the same as what would recreate the object since I might show information that is dynamic to the current state (like, the name of the parent instead of just a pointer or something). I know that isn?t per the rules, but in the end, it makes it easier for me to troubleshoot the code. Having said that, Ben is totally correct in terms of the "right" way to do it, my earlier suggestion was not "right". Dan From python.list at tim.thechases.com Tue May 3 14:41:21 2016 From: python.list at tim.thechases.com (Tim Chase) Date: Tue, 3 May 2016 13:41:21 -0500 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> <1462166136.1167243.595273897.291B0865@webmail.messagingengine.com> <1462170452.1180117.595306673.68B64F02@webmail.messagingengine.com> <572823D8.6050204@gmail.com> <20160503102850.63cec18a@bigbox.christie.dr> Message-ID: <20160503134121.19f71792@bigbox.christie.dr> On 2016-05-03 13:00, DFS wrote: > On 5/3/2016 11:28 AM, Tim Chase wrote: > > On 2016-05-03 00:24, DFS wrote: > >> One small comparison I was able to make was VBA vs python/pyodbc > >> to summarize an Access database. Not quite a fair test, but > >> interesting nonetheless. > >> > >> Access 2003 file > >> Access 2003 VBA code > >> Time: 0.18 seconds > >> > >> same Access 2003 file > >> 32-bit python 2.7.11 + 32-bit pyodbc 3.0.6 > >> Time: 0.49 seconds > > > > Curious whether you're forcing Access VBA to talk over ODBC or > > whether Access is using native access/file-handling (and thus > > bypassing the ODBC overhead)? > > The latter, which is why I said "not quite a fair test". Can you try the same tests, getting Access/VBA to use ODBC instead to see how much overhead ODBC entails? -tkc From moa47401 at gmail.com Tue May 3 15:24:25 2016 From: moa47401 at gmail.com (moa47401 at gmail.com) Date: Tue, 3 May 2016 12:24:25 -0700 (PDT) Subject: Use __repr__ to show the programmer's representation (was: Need help understanding list structure) In-Reply-To: References: <5727CB31.5060309@lucidity.plus.com> <85mvo8gh57.fsf@benfinney.id.au> <221dcc70-39d8-4c9c-8827-8e0bc1ec1fda@googlegroups.com> <85inyvgga8.fsf_-_@benfinney.id.au> <4c15c01e66314265b7251a16c3c3c235@seaexchmbx03.olympus.F5Net.com> Message-ID: quote - (Though to be fair, I don't really know what the actual problem was, so I might provide a different approach with a different goal ) Originally I was trying to understand the exact structure of the list being returned by the gedcom library. It worked as it was, but I wanted to add additional functionality. I also wanted to understand what character set it was returning. I was giving it a gedcom file with ansel encoding, which is normal. My genealogy program can also export its database to gedcom using UTF-8 and Unicode. But both of those character sets caused the gedcom library to generate an error msg that the file violated GEDCOM format. Keep in mind the gedcom format established by the Latter-day Saints hasn't been updated in 20+ years. From adam at no_thanks.com Tue May 3 15:33:48 2016 From: adam at no_thanks.com (Adam) Date: Tue, 3 May 2016 12:33:48 -0700 Subject: crash while using PyCharm / Python3 References: <56f03080$0$5924$e4fe514c@news.xs4all.nl> Message-ID: "Jonathan N. Little" wrote in message news:ncqc7j$na1$1 at dont-email.me... > Adam wrote: >> "Adam" wrote in message >> news:ncprqb$tl9$1 at news.albasani.net... >>> >>> "Jonathan N. Little" wrote in message >>> news:ncpjj0$7ug$1 at dont-email.me... >>>> Adam wrote: >>>>> There ought to be a way to just reinstall the graphics subsystem >>>>> rather >>>>> than >>>>> an all-or-none installation approach. >>>> >>>> Yes you can. Did it for a borked install of the nVidia driver. >>>> reference >>>> this: >>>> >>>> >>> >>> Thanks, even after doing the following... >>> >>> Problem: Need to purge -fglrx >>> >>> Typically, the following manual commands will properly uninstall -fglrx: >>> >>> $ sudo apt-get remove --purge xorg-driver-fglrx fglrx* >>> $ sudo apt-get install --reinstall libgl1-mesa-glx libgl1-mesa-dri >>> xserver-xorg-core >>> $ sudo dpkg-reconfigure xserver-xorg >>> >>> I still get that dreaded "The system is running in low-graphics mode" >>> error. >>> And, recovery mode failsafeX and Ctrl+Alt+F1 hangs with the following... >>> >>> >>> Initializing built-in extension MIT-SCREEN-SAVER >>> Initializing built-in extension DOUBLE-BUFFER >>> Initializing built-in extension RECORD >>> Initializing built-in extension DPMS >>> Initializing built-in extension Present >>> Initializing built-in extension DRI3 >>> Initializing built-in extension X-Resource >>> Initializing built-in extension XVideo >>> Initializing built-in extension XVideo-MotionCompensation >>> Initializing built-in extension SELinux >>> Initializing built-in extension XFree86-VidModeExtension >>> Initializing built-in extension XFree86-DGA >>> Initializing built-in extension XFree86-DRI >>> Initializing built-in extension DRI2 >>> Loading extension GLX >>> >>> >> >> After trying the following... >> >> http://askubuntu.com/questions/577093/how-to-install-gnome-desktop >> >> $ sudo apt-get update >> $ sudo apt-get install gnome-shell >> >> >> http://tipsonubuntu.com/2014/06/06/change-display-manager-ubuntu-14-04/ >> >> $ sudo dpkg-reconfigure lightdm > > Not sure about ATI, but when you purge the nVidia driver it takes Unity > with it so I had to reinstall unity-desktop afterwards. If you want > lightdm but are now using gdm, maybe it happened because unity-desktop was > uninstalled and you just installed gnome-shell, your can reset: > > sudo dpkg-reconfigure gdm > > and select lightdm from the list. > Today, I tried to switch from gdm back to lightdm and still encountered the "The system is running in low-graphics mode" problem until I found... NumLock Key ON... http://ubuntuforums.org/showthread.php?t=2218111 grumblebum2 provided the fix in post #9... For NumLock key ON (in Ubuntu 14.04), add... greeter-setup-script=/usr/bin/numlockx on to /usr/share/lightdm/lightdm.conf.d/50-unity-greeter.conf (not /etc/lightdm/lightdm.conf, which I deleted since there were no other settings) > > -- > Take care, > > Jonathan > ------------------- > LITTLE WORKS STUDIO > http://www.LittleWorksStudio.com From random832 at fastmail.com Tue May 3 15:37:24 2016 From: random832 at fastmail.com (Random832) Date: Tue, 03 May 2016 15:37:24 -0400 Subject: Use __repr__ to show the programmer's representation (was: Need help understanding list structure) In-Reply-To: References: <5727CB31.5060309@lucidity.plus.com> <85mvo8gh57.fsf@benfinney.id.au> <221dcc70-39d8-4c9c-8827-8e0bc1ec1fda@googlegroups.com> <85inyvgga8.fsf_-_@benfinney.id.au> <4c15c01e66314265b7251a16c3c3c235@seaexchmbx03.olympus.F5Net.com> Message-ID: <1462304244.1459718.597124625.092579A1@webmail.messagingengine.com> On Tue, May 3, 2016, at 15:24, moa47401 at gmail.com wrote: > I also wanted to understand what character set it was returning. I was > giving it a gedcom file with ansel encoding, which is normal. My > genealogy program can also export its database to gedcom using UTF-8 and > Unicode. But both of those character sets caused the gedcom library to > generate an error msg that the file violated GEDCOM format. You haven't said where this library can be found, or what the error message was. From python at mrabarnett.plus.com Tue May 3 15:57:12 2016 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 3 May 2016 20:57:12 +0100 Subject: Need help understanding list structure In-Reply-To: References: <5727CB31.5060309@lucidity.plus.com> <85mvo8gh57.fsf@benfinney.id.au> <221dcc70-39d8-4c9c-8827-8e0bc1ec1fda@googlegroups.com> Message-ID: <1ca9650f-3f4f-f4b5-529d-c7d57e8a2171@mrabarnett.plus.com> On 2016-05-03 18:54, Dan Strohl via Python-list wrote: > >> I added a __repr__ method at the end of the gedcom library like so: >> >> def __repr__(self): >> """ Format this element as its original string """ >> result = repr(self.level()) >> if self.pointer() != "": >> result += ' ' + self.pointer() >> result += ' ' + self.tag() >> if self.value() != "": >> result += ' ' + self.value() >> return result >> >> and now I can print myList properly. >> >> Eric and Michael also mentioned repr above, but I guess I needed someone >> to spell it out for me. Thanks for taking the time to put it in terms an old dog >> could understand. >> > > Glad to help! (being an old dog myself, I know the feeling!) > > One other point for you, if your "__repr__(self)" code is the same as the "__str__(self)" code (which it looks like it is, at a glance at least), you can instead reference the __str__ method and save having a duplicate code block... some examples: > > ========= > Option 1: This is the easiest to read (IMHO) and allows for the possibility that str() is doing something here like formatting or whatever. (in this case it shouldn't be though). However, to call this actually is taking multiple steps (calling object.__repr__, whch calls str(), which calls object.__str__(). ) > > def __repr__(self): > return str(self) > > ========= > Option 2: this isn't hard to read, and just takes two steps (calling object.__repr__(), which calls object.__str__(). > > def __repr__(self): > return self.__str__() > > ======== > Option 3: it's not that this is hard to read, but since it doesn't follow the standard "def blah(self):" pattern, sometimes I overlook these in the code (even when I put them there). This however is the shortest since it really just tells the object to return object.__str__() if either object.__repr__() OR object.__str__() is called. > > > __repr__ = __str__ > > > This probably doesn't matter much in this case, since it probably isn't called that much in normal use (though there are always exceptions), and in the end, Python is fast enough that unless you really need to slice off a few milliseconds, you will never notice the difference, but just some food for thought. > > Option 4: Delete the __str__ method. If there's no __str__, Python falls back to __repr__. If there's no __repr__, Python falls back to the format. From larry.martell at gmail.com Tue May 3 16:36:22 2016 From: larry.martell at gmail.com (Larry Martell) Date: Tue, 3 May 2016 16:36:22 -0400 Subject: Broken pipe from gevent/pywsgi.py Message-ID: I have a python server that has this in the main: from gevent import pywsgi try: httpd = pywsgi.WSGIServer(('0.0.0.0', 8000), app) httpd.serve_forever() except KeyboardInterrupt: pass Recently we began getting HTTPError: 504 Server Error: Gateway Time-out on requests to the server. Looking in the logs I saw this: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line 508, in handle_one_response self.run_application() File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line 495, in run_application self.process_result() File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line 486, in process_result self.write(data) File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line 380, in write self._write_with_headers(data) File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line 400, in _write_with_headers self._sendall(towrite) File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line 355, in _sendall self.socket.sendall(data) File "/usr/local/lib/python2.7/dist-packages/gevent/socket.py", line 460, in sendall data_sent += self.send(_get_memory(data, data_sent), flags) File "/usr/local/lib/python2.7/dist-packages/gevent/socket.py", line 437, in send return sock.send(data, flags) error: [Errno 32] Broken pipe {'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_ACCEPT': '*/*', 'HTTP_ACCEPT_ENCODING': 'gzip, deflate', 'HTTP_CONNECTION': 'close', 'HTTP_HOST': 'bekku.bbmsc.com', 'HTTP_USER_AGENT': 'python-requests/2.9.1', 'HTTP_X_FORWARDED_FOR': '192.168.10.5', 'HTTP_X_REAL_IP': '192.168.10.5', 'PATH_INFO': '/readings', 'QUERY_STRING': 'target=2225217193085335062&tool=HM', 'REMOTE_ADDR': '172.17.42.1', 'REMOTE_PORT': '41708', 'REQUEST_METHOD': 'GET', 'SCRIPT_NAME': '', 'SERVER_NAME': 'fdffcd119506', 'SERVER_PORT': '8000', 'SERVER_PROTOCOL': 'HTTP/1.0', 'SERVER_SOFTWARE': 'gevent/1.0 Python/2.7', 'wsgi.errors': ', mode 'w' at 0x7f4c299bc1e0>, 'wsgi.input': , 'wsgi.multiprocess': False, 'wsgi.multithread': False, 'wsgi.run_once': False, 'wsgi.url_scheme': 'http', 'wsgi.version': (1, 0)} failed with error Once this occurs it seems that no further requests are served. I googled this and found many places where people said this was happening, and many places where people explained why it was happening, but no place did I find a way to fix or workaround this. Does anyone know how to get it to continue processing after this occurs? From flebber.crue at gmail.com Tue May 3 16:55:41 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Tue, 3 May 2016 13:55:41 -0700 (PDT) Subject: Trouble porting glob bash behavior with argparse to windows shell In-Reply-To: <80f6672a-2b69-4749-821d-a92be107862a@googlegroups.com> References: <80f6672a-2b69-4749-821d-a92be107862a@googlegroups.com> Message-ID: <664703cb-ac1f-411d-a0d7-0a50a683d2fb@googlegroups.com> Is there something obvious to this I am doing wrong? Sayth From __peter__ at web.de Tue May 3 17:15:45 2016 From: __peter__ at web.de (Peter Otten) Date: Tue, 03 May 2016 23:15:45 +0200 Subject: Trouble porting glob bash behavior with argparse to windows shell References: <80f6672a-2b69-4749-821d-a92be107862a@googlegroups.com> <664703cb-ac1f-411d-a0d7-0a50a683d2fb@googlegroups.com> Message-ID: Sayth Renshaw wrote: > Is there something obvious to this I am doing wrong? > parser.add_argument("path", nargs="+") The "+" implicitly turns args.path into a list > files |= set(glob.glob(args.path + '/*' + args.extension)) so the glob() argument is evaluated as list + str + str Try name_pattern = "*" + args.extension for path in args.path: files.update(glob.glob(os.path.join(path, name_pattern))) From crk at godblessthe.us Tue May 3 17:50:41 2016 From: crk at godblessthe.us (Clayton Kirkwood) Date: Tue, 3 May 2016 14:50:41 -0700 Subject: How to become more motivated to learn Python In-Reply-To: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> References: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> Message-ID: <06d201d1a585$d6bda040$8438e0c0$@godblessthe.us> Find some thing that you really need done, but you've put off because you didn't have the programming knowledge to do. Program parts of some thing that you are really interested in doing. For instance, at some point, I need to find duplicate copies of hardcopy photos that I have. I'd love to write a program that sorts, compares, and marks as duplicates. I think I would go about doing this similar to facial recognition or eye recognition where I would look for 30 points on each photo and compare them instead of scanning and comparing every bit. crk > -----Original Message----- > From: Python-list [mailto:python-list- > bounces+crk=godblessthe.us at python.org] On Behalf Of Cai Gengyang > Sent: Tuesday, May 03, 2016 4:21 AM > To: python-list at python.org > Subject: How to become more motivated to learn Python > > So I have completed up to CodeAcademy's Python Unit 2 , now moving on to > Unit3 : Conditionals and Control Flow. > > But I feel my motivation wavering , at times I get stuck and frustrated when > trying to learn a new programming language ? > > This might not be a technical question per say, but it is a Python programming > related one. How do you motivate a person (either yourself or your child) to > become more interested in programming and stick with it ? Is determination > in learning (especially in a tough field like software) partly genetic ? > > Related , This is a very well written essay on determination by Paul Graham -- > -------------------------------------- > http://www.paulgraham.com/determination.html > > Gengyang > -- > https://mail.python.org/mailman/listinfo/python-list From tjreedy at udel.edu Tue May 3 18:00:27 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 3 May 2016 18:00:27 -0400 Subject: Trouble porting glob bash behavior with argparse to windows shell In-Reply-To: <664703cb-ac1f-411d-a0d7-0a50a683d2fb@googlegroups.com> References: <80f6672a-2b69-4749-821d-a92be107862a@googlegroups.com> <664703cb-ac1f-411d-a0d7-0a50a683d2fb@googlegroups.com> Message-ID: On 5/3/2016 4:55 PM, Sayth Renshaw wrote: > Is there something obvious to this I am doing wrong? > > Sayth Somethin happened so that I don't see what you did. Fortunately, it did show up for Peter, between the '?' and name, so he could answer. -- Terry Jan Reedy From davidgshi at yahoo.co.uk Tue May 3 18:56:25 2016 From: davidgshi at yahoo.co.uk (David Shi) Date: Tue, 3 May 2016 22:56:25 +0000 (UTC) Subject: How to call a Python Class? References: <1659145888.152847.1462316185321.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <1659145888.152847.1462316185321.JavaMail.yahoo@mail.yahoo.com> I found a Python class within an Open Source software. I would like to use it in my own Python script. I tried to import it, but I got following message. from intersection import *Traceback (most recent call last):? File "", line 1, in ? ? from intersection import *ImportError: bad magic number in 'intersection': b'\x03\xf3\r\n' Can any one help? Regards. David From rosuav at gmail.com Tue May 3 19:14:40 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 May 2016 09:14:40 +1000 Subject: How to call a Python Class? In-Reply-To: <1659145888.152847.1462316185321.JavaMail.yahoo@mail.yahoo.com> References: <1659145888.152847.1462316185321.JavaMail.yahoo.ref@mail.yahoo.com> <1659145888.152847.1462316185321.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Wed, May 4, 2016 at 8:56 AM, David Shi via Python-list wrote: > I found a Python class within an Open Source software. > I would like to use it in my own Python script. > I tried to import it, but I got following message. > from intersection import *Traceback (most recent call last): File "", line 1, in from intersection import *ImportError: bad magic number in 'intersection': b'\x03\xf3\r\n' > Can any one help? > Regards. Did you get a .py file, or only a .pyc? Try deleting all .pyc files that you downloaded, and try again. ChrisA From ben+python at benfinney.id.au Tue May 3 19:31:52 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 04 May 2016 09:31:52 +1000 Subject: How to call a Python Class? References: <1659145888.152847.1462316185321.JavaMail.yahoo.ref@mail.yahoo.com> <1659145888.152847.1462316185321.JavaMail.yahoo@mail.yahoo.com> Message-ID: <85eg9ihg5z.fsf@benfinney.id.au> David Shi via Python-list writes: > I found a Python class within an Open Source software. > I would like to use it in my own Python script. > I tried to import it, but I got following message. Your text is mangled in transit. Please post only plain text messages (avoid HTML or other ?rich? content), so the formatting survives. > from intersection import *Traceback (most recent call last):? File "", line 1, in ? ? from intersection import *ImportError: bad magic number in 'intersection': b'\x03\xf3\r\n' > Can any one help? If we can see the exact code you tried, perhaps. -- \ ?Holy hole in a donut, Batman!? ?Robin | `\ | _o__) | Ben Finney From rosuav at gmail.com Tue May 3 19:40:24 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 May 2016 09:40:24 +1000 Subject: Use __repr__ to show the programmer's representation (was: Need help understanding list structure) In-Reply-To: <4c15c01e66314265b7251a16c3c3c235@seaexchmbx03.olympus.F5Net.com> References: <5727CB31.5060309@lucidity.plus.com> <85mvo8gh57.fsf@benfinney.id.au> <221dcc70-39d8-4c9c-8827-8e0bc1ec1fda@googlegroups.com> <85inyvgga8.fsf_-_@benfinney.id.au> <4c15c01e66314265b7251a16c3c3c235@seaexchmbx03.olympus.F5Net.com> Message-ID: On Wed, May 4, 2016 at 4:35 AM, Dan Strohl via Python-list wrote: > I also have never actually used repr() to create code that could be fed back to the interpreter (not saying it isn?t done, just that I haven?t run into needing it), and there are so many of the libraries that do not return a usable repr string that I would hesitate to even try it outside of a very narrow use case. Here's a repr that I like using with SQLAlchemy: def __repr__(self): return (self.__class__.__name__ + "(" + ", ".join("%s=%r" % (col.name, getattr(self, col.name)) for col in self.__table__.columns) + ")") That results in something that *looks* like you could eval it, but you shouldn't ever actually do that (because it'd create a new object). It's still an effective way to make the repr readable; imagine a list that prints out like this: [Person(id=3, name="Fred"), Person(id=6, name="Barney"), Person(id=8, name="Joe")] You can tell exactly where one starts and another ends; you can read what's going on with these record objects. Making them "pseudo-evalable" is worth doing, even if you should never *actually* eval them. ChrisA From me at vmesel.com Tue May 3 19:48:37 2016 From: me at vmesel.com (Vinicius Mesel) Date: Tue, 03 May 2016 20:48:37 -0300 Subject: Analytical Geometry in Python with GeoMath Message-ID: <154790603d7.1040ee3bd58689.8427362743863775803@vmesel.com> Hey guys, I'm back after some time in the darkness(lol). So I'm here to announce my contribution for the mathematicians and physicians and other guys who love geometry like me! I created a library called "GeoMath" that it's intent is to solve all Analytical Geometry problems in a simple way using Python. If you want to check it out, here is the link: https://github.com/vmesel/GeoMath And if you want to install it and start solving your problems with it, just run: pip install geomath! Tnx, Vinicius Mesel From steve at pearwood.info Tue May 3 21:20:28 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 04 May 2016 11:20:28 +1000 Subject: You gotta love a 2-line python solution References: Message-ID: <57294e5d$0$1606$c3e8da3$5496439d@news.astraweb.com> On Tue, 3 May 2016 01:56 pm, DFS wrote: > On 5/2/2016 11:27 PM, jfong at ms4.hinet.net wrote: >> DFS at 2016/5/3 9:12:24AM wrote: >>> try >>> >>> from urllib.request import urlretrieve >>> >>> http://stackoverflow.com/questions/21171718/urllib-urlretrieve-file-python-3-3 >>> >>> >>> I'm running python 2.7.11 (32-bit) >> >> Alright, it works...someway. >> >> I try to get a zip file. It works, the file can be unzipped correctly. >> >>>>> from urllib.request import urlretrieve >>>>> urlretrieve("http://www.caprilion.com.tw/fed.zip", >>>>> "d:\\temp\\temp.zip") >> ('d:\\temp\\temp.zip', ) >>>>> >> >> But when I try to get this forum page, it does get a html file but can't >> be viewed normally. >> >>>>> urlretrieve("https://groups.google.com/forum/#!topic/comp.lang.python/jFl3GJ >> bmR7A", "d:\\temp\\temp.html") >> ('d:\\temp\\temp.html', ) >>>>> >> >> I suppose the html is a much complex situation where more processes need >> to be done before it can be opened by a web browser:-) > > > Who knows what Google has done... it won't open in Opera. The tab title > shows up, but after 20-30 seconds the screen just stays blank and the > cursor quits loading. Dennis has given the answer to this, but since he has X-No-Archive=Yes, his useful and well-written answer will be lost forever. So I've taken the liberty of copying his answer here: Dennis Lee Bieber says: ????????There's practically no HTML in that page -- just miles of Javascript. The one obvious item is: -=-=-=-=-=- -=-=-=-=-=- which is a RELATIVE path. If you copied the file to your machine and then load it in a browser, it will be looking for /forum/C53652DA8B67255A46256B72F0D65A40.cache.js to be on your machine in a subdirectory of where you saved the main file. ????????You'd have to recreate most of the Google environment and fetch anything that was referenced through a relative path first, to get the content to display. Of course, you may find, for example, that the Javascript at some point is doing a database lookup -- and you'd maybe have to now duplicate the database... -- Steven From steve at pearwood.info Tue May 3 21:30:00 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 04 May 2016 11:30:00 +1000 Subject: Not x.islower() has different output than x.isupper() in list output... References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: <57295099$0$1612$c3e8da3$5496439d@news.astraweb.com> On Wed, 4 May 2016 12:42 am, Jussi Piitulainen wrote: > Ceterum censeo, the only suggested use for .swapcase I've ever heard of > is encryption. iF YOU'RE PROGRAMMING AN EDITOR, sWAP cASE IS REALLY USEFUL FOR THOSE LITTLE capslock ACCIDENTS THAT PLAGUE TYPISTS. -- Steven From christopher_reimer at icloud.com Tue May 3 22:12:48 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Tue, 03 May 2016 19:12:48 -0700 Subject: How to become more motivated to learn Python In-Reply-To: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> References: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> Message-ID: <57295AA0.6080203@icloud.com> On 5/3/2016 4:20 AM, Cai Gengyang wrote: > So I have completed up to CodeAcademy's Python Unit 2 , now moving on to Unit3 : Conditionals and Control Flow. > > But I feel my motivation wavering , at times I get stuck and frustrated when trying to learn a new programming language ? > > This might not be a technical question per say, but it is a Python programming related one. How do you motivate a person (either yourself or your child) to become more interested in programming and stick with it ? Is determination in learning (especially in a tough field like software) partly genetic ? > > Related , This is a very well written essay on determination by Paul Graham ----------------------------------------http://www.paulgraham.com/determination.html > > Gengyang I started out translating old BASIC games into Python. These are the same BASIC games that I tried to program into my Commodore 64 without much success when I was much younger. Many of these BASIC games are a good introduction to classical programming problems like rolling dice and playing cards. http://www.atariarchives.org/basicgames/ When I realized that I wasn't learning enough about the Python language from translating BASIC games, I started coding a chess engine. If you ever look at the academic literature for chess programming from the last 50+ years, you can spend a lifetime solving the programming challenges from implementing the game of kings. Thank you, Chris R. From gengyangcai at gmail.com Tue May 3 22:55:35 2016 From: gengyangcai at gmail.com (Cai Gengyang) Date: Tue, 3 May 2016 19:55:35 -0700 (PDT) Subject: How to become more motivated to learn Python In-Reply-To: References: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> <57295AA0.6080203@icloud.com> Message-ID: <38df902f-27dd-45d6-81ba-01c4c2f7668b@googlegroups.com> Cool, I have finally summoned up enough activation energy to start on Unit 3, now going through the topic on Conditionals and Control Flows (stuff like this) >>> boolthree = 200 == (50 * 5) >>> boolthree False Guess it would be really cool to work on AI and games. ( I have been addicted to computer games for a long time lol --- To be able to design a blockbuster like Starcraft 2, Diablo 3 or Final Fantasy 7 would be an incredible feat !) Didn't know that chess programming goes back 50+ years --- I always thought that computer programs were a recent invention. As it turns out , Ada Lovelace wrote the world's first computer program in 1842 --------------------------------------------------------------------------------------------(http://www.todayifoundout.com/index.php/2011/02/in-1842-ada-lovelace-wrote-the-worlds-first-computer-program/) if this piece is true On Wednesday, May 4, 2016 at 10:13:13 AM UTC+8, Christopher Reimer wrote: > On 5/3/2016 4:20 AM, Cai Gengyang wrote: > > So I have completed up to CodeAcademy's Python Unit 2 , now moving on to Unit3 : Conditionals and Control Flow. > > > > But I feel my motivation wavering , at times I get stuck and frustrated when trying to learn a new programming language ? > > > > This might not be a technical question per say, but it is a Python programming related one. How do you motivate a person (either yourself or your child) to become more interested in programming and stick with it ? Is determination in learning (especially in a tough field like software) partly genetic ? > > > > Related , This is a very well written essay on determination by Paul Graham ----------------------------------------http://www.paulgraham.com/determination.html > > > > Gengyang > > I started out translating old BASIC games into Python. These are the > same BASIC games that I tried to program into my Commodore 64 without > much success when I was much younger. Many of these BASIC games are a > good introduction to classical programming problems like rolling dice > and playing cards. > > http://www.atariarchives.org/basicgames/ > > When I realized that I wasn't learning enough about the Python language > from translating BASIC games, I started coding a chess engine. If you > ever look at the academic literature for chess programming from the last > 50+ years, you can spend a lifetime solving the programming challenges > from implementing the game of kings. > > Thank you, > > Chris R. From nospam at dfs.com Tue May 3 23:00:09 2016 From: nospam at dfs.com (DFS) Date: Tue, 3 May 2016 23:00:09 -0400 Subject: How to become more motivated to learn Python In-Reply-To: References: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> <57295AA0.6080203@icloud.com> Message-ID: On 5/3/2016 10:12 PM, Christopher Reimer wrote: > When I realized that I wasn't learning enough about the Python language > from translating BASIC games, I started coding a chess engine. If you > ever look at the academic literature for chess programming from the last > 50+ years, you can spend a lifetime solving the programming challenges > from implementing the game of kings. We can have a good thread on python chess engines some time. I'm also going to write a chess engine in python - follow the UCI protocol and all. You're way ahead of me, I'm sure, but I did already look into algebraic notation, game recording, FEN and all that. pyChess is a nice little game: www.pychess.org The one thing I'm not going to do is review anyone else's code until I put out v1.0 of my own. My goal with v1.0 is for the pieces to make valid moves. That's it. Following that, I'll work in getting the game recording right. No 'strategy' at first. Maybe later I can load a library of well-known openings and try to utilize them. How far along are you in your engine development? Getting the code for en passant and castling right looks to be a bit of an obstacle. What's nice is the strongest engine (Stockfish) is totally open source. From steve at pearwood.info Tue May 3 23:28:57 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 04 May 2016 13:28:57 +1000 Subject: Not x.islower() has different output than x.isupper() in list output... References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: <57296c7a$0$1589$c3e8da3$5496439d@news.astraweb.com> On Wed, 4 May 2016 12:49 am, Jussi Piitulainen wrote: > DFS writes: > >> On 5/3/2016 9:13 AM, Chris Angelico wrote: > >>> It doesn't invert, the way numeric negation does. >> >> What do you mean by 'case inverted'? >> >> It looks like it swaps the case correctly between upper and lower. > > There's letters that do not come in exact pairs of upper and lower case, Languages with two distinct lettercases, like English, are called bicameral. The two cases are technically called majuscule and minuscule, but colloquially known as uppercase and lowercase since movable type printers traditionally used to keep the majuscule letters in a drawer above the minuscule letters. Many alphabets are unicameral, that is, they only have a single lettercase. Examples include Hebrew, Arabic, Hangul, and many others. Georgian is an interesting example, as it is the only known written alphabet that started as a bicameral script and then became unicameral. Consequently, many letters are neither upper nor lower case, and have Unicode category "Letter other": py> c = u'\N{ARABIC LETTER FEH}' py> unicodedata.category(c) 'Lo' py> c.isalpha() True py> c.isupper() False py> c.islower() False Even among bicameral alphabets, there are a few anomalies. The three most obvious ones are Greek sigma, German Eszett (or "sharp S") and Turkish I. (1) The Greek sigma is usually written as ? or ? in uppercase and lowercase respectively, but at the end of a word, lowercase sigma is written as ?. (This final sigma is sometimes called "stigma", but should not be confused with the archaic Greek letter stigma, which has two cases ? ?, at least when it is not being written as digamma ?? -- and if you're confused, so are the Greeks :-) Python 3.3 correctly handles the sigma/final sigma when upper- and lowercasing: py> '?????'.lower() '?????' py> '?????'.lower().upper() '?????' (2) The German Eszett ? traditionally existed in only lowercase forms, but despite the existence of an uppercase form since at least the 19th century, when the Germans moved away from blackletter to Roman-style letters, the uppercase form was left out. In recent years, printers in Germany have started to reintroduce an uppercase version, and the German government have standardized on its use for placenames, but not other words. (Aside: in Germany, ? is not considered a distinct letter of the alphabet, but a ligature of ss; historically it derived from a ligature of ?s, ?z or ??. The funny characters you may or may not be able to see are the long-S and round-Z.) Python follows common, but not universal, German practice for eszett: py> '?'.lower() '?' py> '?'.upper() 'SS' Note that this is lossy: given a name like "STRASSER", it is impossible to tell whether it should be title-cased to "Strasser" or "Stra?er". It also means that uppercasing a string can make it longer. For more on the uppercase eszett, see: https://typography.guru/journal/germanys-new-character/ https://typography.guru/journal/how-to-draw-a-capital-sharp-s-r18/ (3) In most Latin alphabets, the lowercase i and j have a "tittle" diacritic on them, but not the uppercase forms I and J. Turkish and a few other languages have both I-with-tittle and I-without-tittle. (As far as I know, there is no language with a dotless J.) So in Turkish, the correct uppercase to lowercase and back again should go: Dotless I: I -> ? -> I Dotted I: ? -> i -> ? Python does not quite manage to handle this correctly for Turkish applications, since it loses the dotted/dotless distinction: py> '?'.upper() 'I' py> '?'.lower() 'i' and further case conversions follow the non-Turkish rules. Note that sometimes getting this wrong can have serious consequences: http://gizmodo.com/382026/a-cellphones-missing-dot-kills-two-people-puts-three-more-in-jail -- Steven From christopher_reimer at icloud.com Tue May 3 23:39:16 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Tue, 03 May 2016 20:39:16 -0700 Subject: How to become more motivated to learn Python In-Reply-To: References: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> <57295AA0.6080203@icloud.com> Message-ID: <57296EE4.6070001@icloud.com> On 5/3/2016 8:00 PM, DFS wrote: > How far along are you in your engine development? I can display a text-based chess board on the console (looks better with a mono font). 8 BR BN BB BQ BK BB BN BR 7 BP BP BP BP BP BP BP BP 6 __ __ __ __ __ __ __ __ 5 __ __ __ __ __ __ __ __ 4 __ __ __ __ __ __ __ __ 3 __ __ __ __ __ __ __ __ 2 WP WP WP WP WP WP WP WP 1 WR WN WB WQ WK WB WN WR A B C D E F G H With feedback from this list, I had to break a lot of bad Java habits to make the code more Pythonic. Right now I'm going back and forth between writing documentation and unit tests. Once I finalized the code in its current state, I'll post it up on GitHub under the MIT license. Future updates will have a fuller console interface and moves for individual pieces implemented. Thank you, Chris R. From nospam at dfs.com Wed May 4 01:13:05 2016 From: nospam at dfs.com (DFS) Date: Wed, 4 May 2016 01:13:05 -0400 Subject: python chess engines Message-ID: > On 5/3/2016 8:00 PM, DFS wrote: >> How far along are you in your engine development? > I can display a text-based chess board on the console (looks better > with a mono font). > > 8 BR BN BB BQ BK BB BN BR > > 7 BP BP BP BP BP BP BP BP > > 6 __ __ __ __ __ __ __ __ > > 5 __ __ __ __ __ __ __ __ > > 4 __ __ __ __ __ __ __ __ > > 3 __ __ __ __ __ __ __ __ > > 2 WP WP WP WP WP WP WP WP > > 1 WR WN WB WQ WK WB WN WR > > A B C D E F G H > > > With feedback from this list, I had to break a lot of bad Java habits > to make the code more Pythonic. Right now I'm going back and forth > between writing documentation and unit tests. Once I finalized the > code in its current state, I'll post it up on GitHub under the MIT > license. Future updates will have a fuller console interface and > moves for individual pieces implemented. > Thank you, > > Chris R. Wanted to start a new thread, rather than use the 'motivated' thread. Can you play your game at the console? The way I think about a chess engine is it doesn't even display a board. It accepts a move as input, records the move, analyzes the positions after the move, and returns the next move. Here's the UCI protocol. http://download.shredderchess.com/div/uci.zip From nospam at dfs.com Wed May 4 02:10:11 2016 From: nospam at dfs.com (DFS) Date: Wed, 4 May 2016 02:10:11 -0400 Subject: Fastest way to retrieve and write html contents to file In-Reply-To: References: <85vb2xgj2i.fsf@benfinney.id.au> <1462166136.1167243.595273897.291B0865@webmail.messagingengine.com> <1462170452.1180117.595306673.68B64F02@webmail.messagingengine.com> <572823D8.6050204@gmail.com> <20160503102850.63cec18a@bigbox.christie.dr> <20160503134121.19f71792@bigbox.christie.dr> Message-ID: On 5/3/2016 2:41 PM, Tim Chase wrote: > On 2016-05-03 13:00, DFS wrote: >> On 5/3/2016 11:28 AM, Tim Chase wrote: >>> On 2016-05-03 00:24, DFS wrote: >>>> One small comparison I was able to make was VBA vs python/pyodbc >>>> to summarize an Access database. Not quite a fair test, but >>>> interesting nonetheless. >>>> >>>> Access 2003 file >>>> Access 2003 VBA code >>>> Time: 0.18 seconds >>>> >>>> same Access 2003 file >>>> 32-bit python 2.7.11 + 32-bit pyodbc 3.0.6 >>>> Time: 0.49 seconds >>> >>> Curious whether you're forcing Access VBA to talk over ODBC or >>> whether Access is using native access/file-handling (and thus >>> bypassing the ODBC overhead)? >> >> The latter, which is why I said "not quite a fair test". > > Can you try the same tests, getting Access/VBA to use ODBC instead to > see how much overhead ODBC entails? > > -tkc Done. I dropped a few extraneous tables from the database (was 114 tables): Access 2003 .mdb file 2,009,164 rows 97 tables (max row = 600288) 725 columns text: 389 boolean: 4 numeric: 261 date-time: 69 binary: 2 264 indexes (25 foreign keys)* 299,167,744 bytes on disk 1. DAO Time: 0.15 seconds 2. ADODB, Access ODBC driver, OpenSchema method** Time: 0.26 seconds 3. python, pyodbc, Access ODBC driver Time: 0.42 seconds * despite being written by Microsoft, the Access ODBC driver doesn't support the ODBC SQLForeignKeys function, so the python code doesn't show a count of foreign keys ** the Access ODBC driver doesn't support the adSchemaIndexes or adSchemaForeignKeys query types, so I used DAO code to count indexes and foreign keys. From orgnut at yahoo.com Wed May 4 02:53:31 2016 From: orgnut at yahoo.com (Larry Hudson) Date: Tue, 3 May 2016 23:53:31 -0700 Subject: How to become more motivated to learn Python In-Reply-To: <38df902f-27dd-45d6-81ba-01c4c2f7668b@googlegroups.com> References: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> <57295AA0.6080203@icloud.com> <38df902f-27dd-45d6-81ba-01c4c2f7668b@googlegroups.com> Message-ID: <1-SdnfwrZPXxAbTKnZ2dnUU7-YvNnZ2d@giganews.com> On 05/03/2016 07:55 PM, Cai Gengyang wrote: > Cool, I have finally summoned up enough activation energy to start on Unit 3, now going through the topic on Conditionals and Control Flows (stuff like this) > >>>> boolthree = 200 == (50 * 5) >>>> boolthree > False > > Guess it would be really cool to work on AI and games. ( I have been addicted to computer games for a long time lol --- To be able to design a blockbuster like Starcraft 2, Diablo 3 or Final Fantasy 7 would be an incredible feat !) > For an introduction to Python via games you might want to check out the book "Invent Your Own Computer Games With Python". (However, these are strictly text-based games -- no graphics, but a fairly good intro to Python programming.) It's available to read on-line, or for download as a .pdf at http://inventwithpython.com/ Use the "PDF and All Source Code" link to download it. From tjreedy at udel.edu Wed May 4 03:44:16 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 4 May 2016 03:44:16 -0400 Subject: How to become more motivated to learn Python In-Reply-To: <38df902f-27dd-45d6-81ba-01c4c2f7668b@googlegroups.com> References: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> <57295AA0.6080203@icloud.com> <38df902f-27dd-45d6-81ba-01c4c2f7668b@googlegroups.com> Message-ID: On 5/3/2016 10:55 PM, Cai Gengyang wrote: > Guess it would be really cool to work on AI and games. ( I have been > addicted to computer games for a long time lol --- To be able to > design a blockbuster like Starcraft 2, Diablo 3 or Final Fantasy 7 > would be an incredible feat !) In case you like minecraft, I just discovered this today https://www.nostarch.com/programwithminecraft "Learn to Program with Minecraft" It uses a socket client written in 3.5 to interface to a minecraft 1.8 socket server written in java 7. One can at least do simple things like move the player, identify and place blocks, and put text in the chat box. -- Terry Jan Reedy From flebber.crue at gmail.com Wed May 4 03:56:15 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Wed, 4 May 2016 00:56:15 -0700 (PDT) Subject: Trouble porting glob bash behavior with argparse to windows shell In-Reply-To: References: <80f6672a-2b69-4749-821d-a92be107862a@googlegroups.com> <664703cb-ac1f-411d-a0d7-0a50a683d2fb@googlegroups.com> Message-ID: Thank you Peter. I was starting to flail and thought my use of glob.glob was wrong. As an aside should I be using os.path to negate system inconsistency? Thanks Sayth From flebber.crue at gmail.com Wed May 4 03:57:16 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Wed, 4 May 2016 00:57:16 -0700 (PDT) Subject: Trouble porting glob bash behavior with argparse to windows shell In-Reply-To: References: <80f6672a-2b69-4749-821d-a92be107862a@googlegroups.com> <664703cb-ac1f-411d-a0d7-0a50a683d2fb@googlegroups.com> Message-ID: <125a6208-8752-482e-ac8f-8ef2be012dae@googlegroups.com> Oops sorry noticed you did in the glob. Sorry squinting at phone. Sayth From greg.ewing at canterbury.ac.nz Wed May 4 04:34:07 2016 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Wed, 04 May 2016 20:34:07 +1200 Subject: Not x.islower() has different output than x.isupper() in list output... In-Reply-To: References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: Jussi Piitulainen wrote: > Ceterum censeo, the only suggested use for .swapcase I've ever heard of > is encryption. Yep, all the smart terrorists these days are using a combination of swapcase and rot13. Totally bamboozles the FBI. -- Greg From mviljamaa at kapsi.fi Wed May 4 04:41:37 2016 From: mviljamaa at kapsi.fi (mviljamaa) Date: Wed, 04 May 2016 11:41:37 +0300 Subject: How to become more motivated to learn Python In-Reply-To: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> References: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> Message-ID: <6e2d9770eddbc7a71c2a63851bce14b7@kapsi.fi> I tend to not have the patience to go through programming tutorials, because I think they're boring. I sometimes use them as reference to see or recall how something is done, but I don't step through them in order to learn a language. Rather, I write programs to learn programming and languages. It's easier if you have some background in programming in some other language. I merely pick personal projects that interest me and then try to develop them to finish. I seek resources that help me do what I need to. I think it's easier to be motivated in actual projects that interest you, rather than programming assignments. -Matti Cai Gengyang kirjoitti 2016-05-03 14:20: > So I have completed up to CodeAcademy's Python Unit 2 , now moving on > to Unit3 : Conditionals and Control Flow. > > But I feel my motivation wavering , at times I get stuck and > frustrated when trying to learn a new programming language ? > > This might not be a technical question per say, but it is a Python > programming related one. How do you motivate a person (either yourself > or your child) to become more interested in programming and stick with > it ? Is determination in learning (especially in a tough field like > software) partly genetic ? > > Related , This is a very well written essay on determination by Paul > Graham > ----------------------------------------http://www.paulgraham.com/determination.html > > Gengyang From steve+comp.lang.python at pearwood.info Wed May 4 04:59:03 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Wed, 04 May 2016 18:59:03 +1000 Subject: After a year using Node.js, the prodigal son returns Message-ID: <5729b9d8$0$1612$c3e8da3$5496439d@news.astraweb.com> A year ago, Gavin Vickery decided to move away from Python and give Javascript with Node.js a try. Twelve months later, he has written about his experiences: http://geekforbrains.com/post/after-a-year-of-nodejs-in-production -- Steve From steve+comp.lang.python at pearwood.info Wed May 4 05:06:26 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Wed, 04 May 2016 19:06:26 +1000 Subject: Not x.islower() has different output than x.isupper() in list output... References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> Message-ID: <5729bb93$0$2927$c3e8da3$76491128@news.astraweb.com> On Wednesday 04 May 2016 18:34, Gregory Ewing wrote: > Jussi Piitulainen wrote: >> Ceterum censeo, the only suggested use for .swapcase I've ever heard of >> is encryption. > > Yep, all the smart terrorists these days are using a > combination of swapcase and rot13. Totally bamboozles > the FBI. > Heh, the Australian government is getting their panties in a twist over the whole encryption thing, because Apple versus FBI proves that encryption is evil or something. Which lead to this exchange in the IRC channel we use at work: (handles have been removed to anonymize the discussion) XXXX: newsflash, most criminals are too stupid to use encryption XXXX: https://en.wikipedia.org/wiki/Mujahedeen_Secrets XXXX: It's hilarious, they turn off all the "western" ciphers YYYY: the great thing about Mujahedeen Secrets (apart fromt he crypto apparently being poor) is the cyphertext is trivially detectable YYYY: like "-----BEGIN PGP MESSAGE-----" but it's "-----DEATH TO INFIDELS-----" or something YYYY: so XKeyscore can just go "this person is using Mujahedeen Secrets" in their regular traffic matching ZZZZ: Wait, Mujahedeen Secrets was released anonymously, it has easily broken encryption and is trivially detectable? Can you say "schmuck bait"? I think I know who the authors were... -- Steve From steve+comp.lang.python at pearwood.info Wed May 4 05:08:42 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Wed, 04 May 2016 19:08:42 +1000 Subject: Analytical Geometry in Python with GeoMath References: <154790603d7.1040ee3bd58689.8427362743863775803@vmesel.com> Message-ID: <5729bc1a$0$2927$c3e8da3$76491128@news.astraweb.com> On Wednesday 04 May 2016 09:48, Vinicius Mesel wrote: > I created a library called "GeoMath" that it's intent is to solve all > Analytical Geometry problems in a simple way using Python. > > If you want to check it out, here is the link: > https://github.com/vmesel/GeoMath > > And if you want to install it and start solving your problems with it, > just run: pip install geomath! Can you show us some examples? -- Steve From marko at pacujo.net Wed May 4 05:51:10 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Wed, 04 May 2016 12:51:10 +0300 Subject: How to become more motivated to learn Python References: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> <57295AA0.6080203@icloud.com> <38df902f-27dd-45d6-81ba-01c4c2f7668b@googlegroups.com> Message-ID: <87k2jakv75.fsf@elektro.pacujo.net> Terry Reedy : > In case you like minecraft, I just discovered this today > https://www.nostarch.com/programwithminecraft "Learn to Program with > Minecraft" > > It uses a socket client written in 3.5 to interface to a minecraft 1.8 > socket server written in java 7. One can at least do simple things > like move the player, identify and place blocks, and put text in the > chat box. Hm, I think it would be more exciting to construct a small Python engine using redstone. (I actually went pretty far trying to build a tiny, functional CPU using redstone. I can tell you it's tedious work. Makes you appreciate the advances in transistor density in the past half-century. Too bad redstone is glitchy -- by design, I suppose. After your contraption becomes large enough, it keeps glitching out and you spend all your time debugging and repairing it.) Marko From flebber.crue at gmail.com Wed May 4 06:51:51 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Wed, 4 May 2016 03:51:51 -0700 (PDT) Subject: Mentor Request Message-ID: Hi Wanting to ask if there was anyone on the list that would like to be a mentor? I have no expectation that you would do it for free, so would accept fair offers as well monetary or skill trade, or something regional from Australia that you would otherwise not have access to etc. I want to be clear on what I am wanting and why. Wanting to learn. * Troubleshooting, reading and locating what the errors mean. This is a key area for me, I struggle to understand what I am doing wrong. * What's idiomatic and good practise, I keep finding code examples I doubt the quality of. In practical terms the things I want to do effectively. * Move, clean and obtain data from various sources scraped, streaming json/xml, xml/csv files. * Data into and out of Excel and SQL stores, best storage and use with pandas and bokeh/matplotlib/plotly other? (D3). * Present the data as interactive format flask or django or pyramid etc. About me I have completed codeacademy Python, Pluralsight python (Austin Bingham), flask projects in Real Python course and bits from various books online courses, conference videos etc. It's really point one, errors and troubleshooting which causes me the frustration, I feel like I am shooting in the dark, well except for Flask debug=true which seems to give fairly clear errors. I have several other orielly books and what not, but the examples never seem to relate to what you actually do in the real world. About me. Live in New South Wales Australia somewhat regional, closest local python group is 2 and a half hours away in Sydney. I am a coach in a financial company, also I do trend analysis with excel and sharepoint and write coaching, training and facilitation documentation. Note: I am a coach so if you accept to help, I wont be second guessing what tools, approach or actions to do, I know the key to learning is to accept you are learning and go with it. Thanks Sayth NB: Unless referred directly by someone common and known to this list I will not accept your offer, only members I have seen and post honestly and with a level of group trust will I accept. From rosuav at gmail.com Wed May 4 07:15:48 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 May 2016 21:15:48 +1000 Subject: Mentor Request In-Reply-To: References: Message-ID: On Wed, May 4, 2016 at 8:51 PM, Sayth Renshaw wrote: > Wanting to ask if there was anyone on the list that would like to be a mentor? > > I have no expectation that you would do it for free, so would accept fair offers as well monetary or skill trade, or something regional from Australia that you would otherwise not have access to etc. > > I want to be clear on what I am wanting and why. > > Wanting to learn. > > * Troubleshooting, reading and locating what the errors mean. This is a key area for me, I struggle to understand what I am doing wrong. > > * What's idiomatic and good practise, I keep finding code examples I doubt the quality of. > > In practical terms the things I want to do effectively. > > * Move, clean and obtain data from various sources scraped, streaming json/xml, xml/csv files. > > * Data into and out of Excel and SQL stores, best storage and use with pandas and bokeh/matplotlib/plotly other? (D3). > > * Present the data as interactive format flask or django or pyramid etc. > This fits in nicely with the kinds of things I do. The usual arrangement is a regular (eg weekly) meeting via videoconference, for which I'd give you an invoice; in between times, you'd be studying/practising on your own. Code review is one of the most efficient uses of our time, so if you spend a few days or a week working on a piece of code, we can discuss the results of that. Contact me off-list if you're interested in more information, including the sordid details. ChrisA From wrightalexw at gmail.com Wed May 4 07:37:56 2016 From: wrightalexw at gmail.com (alex wright) Date: Wed, 4 May 2016 07:37:56 -0400 Subject: How to become more motivated to learn Python In-Reply-To: <6e2d9770eddbc7a71c2a63851bce14b7@kapsi.fi> References: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> <6e2d9770eddbc7a71c2a63851bce14b7@kapsi.fi> Message-ID: Beyond motivation, it's not likely just reading a flat text will be retained meaningfully without creative problem solving. Personal projects are the best route in my opinion. I like reading technical books but I figure I retain a small percentage of the specifics. On May 4, 2016 6:32 AM, "mviljamaa" wrote: > I tend to not have the patience to go through programming tutorials, > because I think they're boring. I sometimes use them as reference to see or > recall how something is done, but I don't step through them in order to > learn a language. > > Rather, I write programs to learn programming and languages. It's easier > if you have some background in programming in some other language. > > I merely pick personal projects that interest me and then try to develop > them to finish. I seek resources that help me do what I need to. > > I think it's easier to be motivated in actual projects that interest you, > rather than programming assignments. > > -Matti > > Cai Gengyang kirjoitti 2016-05-03 14:20: > >> So I have completed up to CodeAcademy's Python Unit 2 , now moving on >> to Unit3 : Conditionals and Control Flow. >> >> But I feel my motivation wavering , at times I get stuck and >> frustrated when trying to learn a new programming language ? >> >> This might not be a technical question per say, but it is a Python >> programming related one. How do you motivate a person (either yourself >> or your child) to become more interested in programming and stick with >> it ? Is determination in learning (especially in a tough field like >> software) partly genetic ? >> >> Related , This is a very well written essay on determination by Paul >> Graham >> ---------------------------------------- >> http://www.paulgraham.com/determination.html >> >> Gengyang >> > -- > https://mail.python.org/mailman/listinfo/python-list > From grant.b.edwards at gmail.com Wed May 4 09:44:41 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Wed, 4 May 2016 13:44:41 +0000 (UTC) Subject: After a year using Node.js, the prodigal son returns References: <5729b9d8$0$1612$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-04, Steven D'Aprano wrote: > A year ago, Gavin Vickery decided to move away from Python and give > Javascript with Node.js a try. Twelve months later, he has written about his > experiences: > > http://geekforbrains.com/post/after-a-year-of-nodejs-in-production [Regarding Python] Its stood the test of time, has some great standards, libraries, its easy to debug and performs very well. Sure it has its worts. It even makes Beer! Talk about "batteries included"... -- Grant Edwards grant.b.edwards Yow! We are now enjoying at total mutual interaction in gmail.com an imaginary hot tub ... From nospam at dfs.com Wed May 4 10:09:11 2016 From: nospam at dfs.com (DFS) Date: Wed, 4 May 2016 10:09:11 -0400 Subject: Not x.islower() has different output than x.isupper() in list output... In-Reply-To: <57296c7a$0$1589$c3e8da3$5496439d@news.astraweb.com> References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> <57296c7a$0$1589$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5/3/2016 11:28 PM, Steven D'Aprano wrote: > On Wed, 4 May 2016 12:49 am, Jussi Piitulainen wrote: > >> DFS writes: >> >>> On 5/3/2016 9:13 AM, Chris Angelico wrote: >> >>>> It doesn't invert, the way numeric negation does. >>> >>> What do you mean by 'case inverted'? >>> >>> It looks like it swaps the case correctly between upper and lower. >> >> There's letters that do not come in exact pairs of upper and lower case, > > Languages with two distinct lettercases, like English, are called bicameral. > The two cases are technically called majuscule and minuscule, but > colloquially known as uppercase and lowercase since movable type printers > traditionally used to keep the majuscule letters in a drawer above the > minuscule letters. > > Many alphabets are unicameral, that is, they only have a single lettercase. > Examples include Hebrew, Arabic, Hangul, and many others. Georgian is an > interesting example, as it is the only known written alphabet that started > as a bicameral script and then became unicameral. > > Consequently, many letters are neither upper nor lower case, and have > Unicode category "Letter other": > > py> c = u'\N{ARABIC LETTER FEH}' > py> unicodedata.category(c) > 'Lo' > py> c.isalpha() > True > py> c.isupper() > False > py> c.islower() > False > > > Even among bicameral alphabets, there are a few anomalies. The three most > obvious ones are Greek sigma, German Eszett (or "sharp S") and Turkish I. > > (1) The Greek sigma is usually written as ? or ? in uppercase and lowercase > respectively, but at the end of a word, lowercase sigma is written as ?. > > (This final sigma is sometimes called "stigma", but should not be confused > with the archaic Greek letter stigma, which has two cases ? ?, at least > when it is not being written as digamma ?? -- and if you're confused, so > are the Greeks :-) > > Python 3.3 correctly handles the sigma/final sigma when upper- and > lowercasing: > > py> '?????'.lower() > '?????' > > py> '?????'.lower().upper() > '?????' > > > > (2) The German Eszett ? traditionally existed in only lowercase forms, but > despite the existence of an uppercase form since at least the 19th century, > when the Germans moved away from blackletter to Roman-style letters, the > uppercase form was left out. In recent years, printers in Germany have > started to reintroduce an uppercase version, and the German government have > standardized on its use for placenames, but not other words. > > (Aside: in Germany, ? is not considered a distinct letter of the alphabet, > but a ligature of ss; historically it derived from a ligature of ?s, ?z or > ??. The funny characters you may or may not be able to see are the long-S > and round-Z.) > > Python follows common, but not universal, German practice for eszett: > > py> '?'.lower() > '?' > py> '?'.upper() > 'SS' > > Note that this is lossy: given a name like "STRASSER", it is impossible to > tell whether it should be title-cased to "Strasser" or "Stra?er". It also > means that uppercasing a string can make it longer. > > > For more on the uppercase eszett, see: > > https://typography.guru/journal/germanys-new-character/ > https://typography.guru/journal/how-to-draw-a-capital-sharp-s-r18/ > > > (3) In most Latin alphabets, the lowercase i and j have a "tittle" diacritic > on them, but not the uppercase forms I and J. Turkish and a few other > languages have both I-with-tittle and I-without-tittle. > > (As far as I know, there is no language with a dotless J.) > > So in Turkish, the correct uppercase to lowercase and back again should go: > > Dotless I: I -> ? -> I > > Dotted I: ? -> i -> ? > > Python does not quite manage to handle this correctly for Turkish > applications, since it loses the dotted/dotless distinction: > > py> '?'.upper() > 'I' > py> '?'.lower() > 'i' > > and further case conversions follow the non-Turkish rules. > > Note that sometimes getting this wrong can have serious consequences: > > http://gizmodo.com/382026/a-cellphones-missing-dot-kills-two-people-puts-three-more-in-jail Linguist much? From ericandbella at yahoo.com Wed May 4 10:28:29 2016 From: ericandbella at yahoo.com (Eric and Bella) Date: Wed, 4 May 2016 14:28:29 +0000 (UTC) Subject: windows 7 install trouble References: <1388951192.8059240.1462372109668.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <1388951192.8059240.1462372109668.JavaMail.yahoo@mail.yahoo.com> hello, could someone please help.? I had installed python 3.? on my computer.? it had issues.? so i tried to uninstall and it wouldn't.? so i deleted all the files and used microsoft fixit to uninstall.? that worked.? but i now can't re-install it.? on the off chance it was firewall related, tried all combinations of firewall. here are the links to the error logs. any help would be appreciated. -eric https://docs.google.com/document/d/1V_1BrmKdCMOwXH9Lt4Zl2xlXp8wkM-sjcrteSRzkucM/pub Python 3.5.1 (32-bit)_20160503210453_000_core_JustForMe.log | | From rosuav at gmail.com Wed May 4 10:37:17 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 May 2016 00:37:17 +1000 Subject: Not x.islower() has different output than x.isupper() in list output... In-Reply-To: References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> <57296c7a$0$1589$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, May 5, 2016 at 12:09 AM, DFS wrote: > On 5/3/2016 11:28 PM, Steven D'Aprano wrote: >> [ lengthy piece about text, Unicode, and letter case ] > > Linguist much? As an English-only speaker who writes code that needs to be used around the world, you end up accruing tidbits of language and text trivia in the form of edge cases that you need to remember to test. Among them: * Turkish dotless and dotted i * Greek medial and final sigma * German eszett * Hebrew and Arabic right-to-left text * Chinese non-BMP characters * Combining characters (eg diacriticals starting U+0300) * Non-characters eg U+FFFE And then a post like Steven's basically comes from pulling up all those from your memory, and maybe doing a spot of quick testing and/or research to get some explanatory details. You don't have to be a linguist, necessarily - just a competent debugger. ChrisA From gengyangcai at gmail.com Wed May 4 10:41:10 2016 From: gengyangcai at gmail.com (Cai Gengyang) Date: Wed, 4 May 2016 07:41:10 -0700 (PDT) Subject: Conditionals And Control Flows Message-ID: I am trying to understand the boolean operator "and" in Python. It is supposed to return "True" when the expression on both sides of "and" are true For instance, 1 < 3 and 10 < 20 is True --- (because both statements are true) 1 < 5 and 5 > 12 is False --- (because both statements are false) bool_one = False and False --- This should give False because none of the statements are False bool_two = True and False --- This should give False because only 1 statement is True bool_three = False and True --- This should give False because only 1 statement is True bool_five = True and True --- This should give True because only 1 statement is True Am I correct ? From jussi.piitulainen at helsinki.fi Wed May 4 10:44:43 2016 From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen) Date: Wed, 04 May 2016 17:44:43 +0300 Subject: After a year using Node.js, the prodigal son returns References: <5729b9d8$0$1612$c3e8da3$5496439d@news.astraweb.com> Message-ID: Grant Edwards writes: > On 2016-05-04, Steven D'Aprano wrote: >> A year ago, Gavin Vickery decided to move away from Python and give >> Javascript with Node.js a try. Twelve months later, he has written about his >> experiences: >> >> http://geekforbrains.com/post/after-a-year-of-nodejs-in-production > > [Regarding Python] > > Its stood the test of time, has some great standards, libraries, > its easy to debug and performs very well. Sure it has its worts. > > It even makes Beer! > > Talk about "batteries included"... Since _I_ didn't know this wort, let the joke not go missing: https://en.wikipedia.org/wiki/Wort Google asked if I meant "worst" :) From bgailer at gmail.com Wed May 4 10:54:03 2016 From: bgailer at gmail.com (Bob Gailer) Date: Wed, 4 May 2016 10:54:03 -0400 Subject: Conditionals And Control Flows In-Reply-To: References: Message-ID: On May 4, 2016 10:45 AM, "Cai Gengyang" wrote: > > I am trying to understand the boolean operator "and" in Python. It is supposed to return "True" when the expression on both sides of "and" are true > > For instance, > > 1 < 3 and 10 < 20 is True --- (because both statements are true) > 1 < 5 and 5 > 12 is False --- (because both statements are false) > > bool_one = False and False --- This should give False because none of the statements are False > bool_two = True and False --- This should give False because only 1 statement is True > bool_three = False and True --- This should give False because only 1 statement is True > bool_five = True and True --- This should give True because only 1 statement is True > > Am I correct ? Yes. Caveat : python Boolean operators return the value of the last argument necessary to determine the outcome. Example : 2 or 3 results in 2 2 and 3 results in 3. 0 and 3 results in 0. HTH. From michael.selik at gmail.com Wed May 4 10:56:25 2016 From: michael.selik at gmail.com (Michael Selik) Date: Wed, 04 May 2016 14:56:25 +0000 Subject: Conditionals And Control Flows In-Reply-To: References: Message-ID: On Wed, May 4, 2016 at 10:46 AM Cai Gengyang wrote: > I am trying to understand the boolean operator "and" in Python. It is > supposed to return "True" when the expression on both sides of "and" are > true > Not exactly, because they will short-circuit. Take a look at the docs. ( https://docs.python.org/3/library/stdtypes.html#boolean-operations-and-or-not ) From rosuav at gmail.com Wed May 4 10:58:25 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 May 2016 00:58:25 +1000 Subject: Conditionals And Control Flows In-Reply-To: References: Message-ID: On Thu, May 5, 2016 at 12:41 AM, Cai Gengyang wrote: > I am trying to understand the boolean operator "and" in Python. It is supposed to return "True" when the expression on both sides of "and" are true > > For instance, > > 1 < 3 and 10 < 20 is True --- (because both statements are true) > 1 < 5 and 5 > 12 is False --- (because both statements are false) > > bool_one = False and False --- This should give False because none of the statements are False > bool_two = True and False --- This should give False because only 1 statement is True > bool_three = False and True --- This should give False because only 1 statement is True > bool_five = True and True --- This should give True because only 1 statement is True > > Am I correct ? Not entirely so, but very close. So long as you stick to the exact values True and False (or simple conditional expressions, like your examples), yes, that's what you'd get. The best way to try these out is the interactive interpreter. On Windows, look in your Start menu for "IDLE"; on other platforms, open up a terminal and type "python3". Then just start messing around: >>> 1 < 3 and 10 < 20 True >>> 1 < 5 and 5 > 12 False This is far and away the easiest way to learn how Python works. You can even play with some other things, and learn how Python's 'and' operator handles other types of data: >>> 1 and 4 4 >>> 0 and 3 0 Confused? Keep messing around. Build up a theory as to what's going on, test your theory, then go check your theory against the documentation. (Or come and ask here, if you can't find it in the docs.) Python doesn't mind how much you poke around with it, and you will learn ever so much more than we can explain! ChrisA From jussi.piitulainen at helsinki.fi Wed May 4 11:10:14 2016 From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen) Date: Wed, 04 May 2016 18:10:14 +0300 Subject: Conditionals And Control Flows References: Message-ID: Cai Gengyang writes: > I am trying to understand the boolean operator "and" in Python. It is > supposed to return "True" when the expression on both sides of "and" > are true > > For instance, > > 1 < 3 and 10 < 20 is True --- (because both statements are true) Yes. > 1 < 5 and 5 > 12 is False --- (because both statements are false) No :) > bool_one = False and False --- This should give False because none of the statements are False > bool_two = True and False --- This should give False because only 1 statement is True > bool_three = False and True --- This should give False because only 1 statement is True Yes. > bool_five = True and True --- This should give True because only 1 statement is True No :) > Am I correct ? Somewhat. In a technical programming-language sense, these are "expressions", not "statements". Technically, if the first expression evaluates to a value that counts as true in Python, the compound expression "E and F" evaluates to the value of the second expression. Apart from False, "empty" values like 0, "", [] count as false in Python, and all the others count as true. But it's true that "E and F" only evaluates to a true value when both E and F evaluate to a true value. Your subject line is good: Python's "and" is indeed a conditional, control-flow operator. From ben.usenet at bsb.me.uk Wed May 4 11:21:58 2016 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Wed, 04 May 2016 16:21:58 +0100 Subject: Conditionals And Control Flows References: Message-ID: <87pot1vofd.fsf@bsb.me.uk> Jussi Piitulainen writes: > Cai Gengyang writes: > >> I am trying to understand the boolean operator "and" in Python. It is >> supposed to return "True" when the expression on both sides of "and" >> are true >> >> For instance, >> >> 1 < 3 and 10 < 20 is True --- (because both statements are true) > > Yes. > >> 1 < 5 and 5 > 12 is False --- (because both statements are false) > > No :) > >> bool_one = False and False --- This should give False because none >> of the statements are False >> bool_two = True and False --- This should give False because only 1 >> statement is True >> bool_three = False and True --- This should give False because only >> 1 statement is True > > Yes. > >> bool_five = True and True --- This should give True because only 1 >> statement is True > > No :) > >> Am I correct ? > > Somewhat. Just an observation on the language... Change "only 1 statement" to "only statement 1" and you get much closer to a correct explanation. -- Ben. From gengyangcai at gmail.com Wed May 4 11:23:00 2016 From: gengyangcai at gmail.com (Cai Gengyang) Date: Wed, 4 May 2016 08:23:00 -0700 (PDT) Subject: Conditionals And Control Flows In-Reply-To: References: Message-ID: <27d8c2c9-304e-4d25-81fd-fb350cf913c1@googlegroups.com> Sorry I mistyped , this should be correct : bool_one = False and False --- This should give False because none of the statements are True bool_two = True and False --- This should give False because only 1 statement is True bool_three = False and True --- This should give False because only 1 statement is True bool_five = True and True --- This should give True because both statements are True On Wednesday, May 4, 2016 at 11:10:28 PM UTC+8, Jussi Piitulainen wrote: > Cai Gengyang writes: > > > I am trying to understand the boolean operator "and" in Python. It is > > supposed to return "True" when the expression on both sides of "and" > > are true > > > > For instance, > > > > 1 < 3 and 10 < 20 is True --- (because both statements are true) > > Yes. > > > 1 < 5 and 5 > 12 is False --- (because both statements are false) > > No :) > > > bool_one = False and False --- This should give False because none of the statements are False > > bool_two = True and False --- This should give False because only 1 statement is True > > bool_three = False and True --- This should give False because only 1 statement is True > > Yes. > > > bool_five = True and True --- This should give True because only 1 statement is True > > No :) > > > Am I correct ? > > Somewhat. > > In a technical programming-language sense, these are "expressions", not > "statements". Technically, if the first expression evaluates to a value > that counts as true in Python, the compound expression "E and F" > evaluates to the value of the second expression. > > Apart from False, "empty" values like 0, "", [] count as false in > Python, and all the others count as true. > > But it's true that "E and F" only evaluates to a true value when both E > and F evaluate to a true value. > > Your subject line is good: Python's "and" is indeed a conditional, > control-flow operator. From steve at pearwood.info Wed May 4 11:37:53 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 05 May 2016 01:37:53 +1000 Subject: Not x.islower() has different output than x.isupper() in list output... References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> <57296c7a$0$1589$c3e8da3$5496439d@news.astraweb.com> Message-ID: <572a1752$0$1614$c3e8da3$5496439d@news.astraweb.com> On Thu, 5 May 2016 12:09 am, DFS wrote: > On 5/3/2016 11:28 PM, Steven D'Aprano wrote: >> Languages with two distinct lettercases, like English, are called >> bicameral. [...] > Linguist much? Possibly even a cunning one. Somebody-had-to-say-it-ly y'rs, -- Steven From no.email at nospam.invalid Wed May 4 12:03:26 2016 From: no.email at nospam.invalid (Paul Rubin) Date: Wed, 04 May 2016 09:03:26 -0700 Subject: Mentor Request References: Message-ID: <8737pxerox.fsf@nightsong.com> Sayth Renshaw writes: > Live in New South Wales Australia somewhat regional, closest local > python group is 2 and a half hours away in Sydney. Try here: https://wiki.hackerspaces.org/RoboDojo From jussi.piitulainen at helsinki.fi Wed May 4 12:41:58 2016 From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen) Date: Wed, 04 May 2016 19:41:58 +0300 Subject: Conditionals And Control Flows References: <27d8c2c9-304e-4d25-81fd-fb350cf913c1@googlegroups.com> Message-ID: Cai Gengyang writes: > Sorry I mistyped , this should be correct : > > bool_one = False and False --- This should give False because none of the statements are True > bool_two = True and False --- This should give False because only 1 statement is True > bool_three = False and True --- This should give False because only 1 statement is True > bool_five = True and True --- This should give True because both statements are True Yes. (That is, afterwards bool_one, bool_two, ... evaluate to the stated truth values, for the stated reasons.) From me at vmesel.com Wed May 4 14:16:36 2016 From: me at vmesel.com (Vinicius) Date: Wed, 4 May 2016 15:16:36 -0300 Subject: Analytical Geometry in Python with GeoMath In-Reply-To: <5729bc1a$0$2927$c3e8da3$76491128@news.astraweb.com> References: <154790603d7.1040ee3bd58689.8427362743863775803@vmesel.com> <5729bc1a$0$2927$c3e8da3$76491128@news.astraweb.com> Message-ID: <8F08F6A8-09F9-46BE-9D88-900490515868@vmesel.com> Yes sure, To add a point, you do: From geomath import point A = point.Point(x,y) A.distance(PointB) A.mispoint(PointB) A.quadrant() To make usage of the Figure class From geomath import figure Square = figure.Figure() Square.addpoints(pointA, PointB, PointC, PointD) To make use of the line class From geomath import line Line = line.Line(PointA, PointB ) Line.equation() These are some basic functions, the others we are still implementing. Tnx, Vinicius Mesel > Em 4 de mai de 2016, ?s 6:08 AM, Steven D'Aprano escreveu: > >> On Wednesday 04 May 2016 09:48, Vinicius Mesel wrote: >> >> I created a library called "GeoMath" that it's intent is to solve all >> Analytical Geometry problems in a simple way using Python. >> >> If you want to check it out, here is the link: >> https://github.com/vmesel/GeoMath >> >> And if you want to install it and start solving your problems with it, >> just run: pip install geomath! > > > Can you show us some examples? > > > > -- > Steve > > From sohcahtoa82 at gmail.com Wed May 4 14:35:21 2016 From: sohcahtoa82 at gmail.com (sohcahtoa82 at gmail.com) Date: Wed, 4 May 2016 11:35:21 -0700 (PDT) Subject: After a year using Node.js, the prodigal son returns In-Reply-To: <5729b9d8$0$1612$c3e8da3$5496439d@news.astraweb.com> References: <5729b9d8$0$1612$c3e8da3$5496439d@news.astraweb.com> Message-ID: <12817a3a-facb-450e-9863-bd9c644b9535@googlegroups.com> On Wednesday, May 4, 2016 at 1:59:15 AM UTC-7, Steven D'Aprano wrote: > A year ago, Gavin Vickery decided to move away from Python and give > Javascript with Node.js a try. Twelve months later, he has written about his > experiences: > > > http://geekforbrains.com/post/after-a-year-of-nodejs-in-production > > > > -- > Steve "Packages that consist of trivial code no more than 10 lines of code are downloaded in the thousands every day from NPM." *cough* left-pad *cough* From encore1 at cox.net Wed May 4 14:41:28 2016 From: encore1 at cox.net (Dick Holmes) Date: Wed, 4 May 2016 11:41:28 -0700 Subject: Interacting with Subprocesses Message-ID: I am attempting to write a Python program that will interact with a (non-Python) process. The programs will run under MinGW. The process can use stdin/stdout commands and responses and can work with pipes. The problem I'm having is that I can't find any way in Python to have a continuing dialog with the process. I have tried Popen communicate, but that protocol seems to be limited to a single message/response pair, and the response is not returned to the message originator until the process terminates. Unfortunately I don't have access to the process' source code so I can't change the communication medium. Is there some feature that will allow me to initiate the process and execute multiple message/response pairs between the Python program and the process during a single execution of the process? Thanks! Dick From fabien.maussion at gmail.com Wed May 4 14:53:49 2016 From: fabien.maussion at gmail.com (Fabien) Date: Wed, 4 May 2016 20:53:49 +0200 Subject: Analytical Geometry in Python with GeoMath References: <154790603d7.1040ee3bd58689.8427362743863775803@vmesel.com> Message-ID: On 05/04/2016 01:48 AM, Vinicius Mesel wrote: > f you want to check it out, here is the link:https://github.com/vmesel/GeoMath Thanks! What differences will your package have with shapely? https://github.com/Toblerity/Shapely Cheers Fabien From me+python at ixokai.io Wed May 4 14:54:29 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Wed, 04 May 2016 11:54:29 -0700 Subject: Conditionals And Control Flows In-Reply-To: References: Message-ID: <1462388069.1988878.598232289.669D262F@webmail.messagingengine.com> On Wed, May 4, 2016, at 07:41 AM, Cai Gengyang wrote: > I am trying to understand the boolean operator "and" in Python. It is > supposed to return "True" when the expression on both sides of "and" are > true The thing is, its kinda dubious to think of 'and' as a 'boolean operator', because once you go down that road, some people start wanting it to be a *pure* boolean operator. Something that always returns True or False. Instead, 'and' and 'or' return something that is true, or something that is false. Notice the lower case. (I know the docs call them Boolean Operations, but I still think saying 'boolean' is unhelpful) Python defines false things as False, None, 0 (of any numeric type), an empty container (lists, tuples, mappings, something else that defines __len__ and it returns 0), and instances of classes that define __nonzero__ that return 0 or False. Everything else is a true thing. If you see "x and y", the rule is: if x is a false thing, it'll return something false. As it happens, it has x handy, and since its decided x is false, it'll return that. Therefore, "x and y" is false. If x is true, though, it'll return y. In this case, "x and y" will be a true thing if y is a true thing, and a false thing if y is a false thing. As you can see, all of this logic happens without ever using True or False. -- Stephen Hansen m e @ i x o k a i . i o From 4kir4.1i at gmail.com Wed May 4 17:04:02 2016 From: 4kir4.1i at gmail.com (Akira Li) Date: Thu, 05 May 2016 00:04:02 +0300 Subject: Interacting with Subprocesses References: Message-ID: <871t5h4jst.fsf@gmail.com> Dick Holmes writes: > I am attempting to write a Python program that will interact with > a (non-Python) process. The programs will run under MinGW. The > process can use stdin/stdout commands and responses and can work > with pipes. The problem I'm having is that I can't find any > way in Python to have a continuing dialog with the process. I > have tried Popen communicate, but that protocol seems to be > limited to a single message/response pair, and the response > is not returned to the message originator until the process > terminates. Unfortunately I don't have access to the process' > source code so I can't change the communication medium. > > Is there some feature that will allow me to initiate the process > and execute multiple message/response pairs between the Python > program and the process during a single execution of the process? > Pass stdin=PIPE, stdout=PIPE and use p.stdin, p.stdout file objects to write input, read output from the child process. Beware, there could be buffering issues or the child process may change its behavior some other way when the standard input/output streams are redirected. See http://pexpect.readthedocs.io/en/stable/FAQ.html#whynotpipe btw, If pexpect module works in MingGW environment (if pty is available); you could try it to communicate with the process interactively. You might also find the list of Stackoverflow question related to the subprocess module useful http://stackoverflow.com/tags/subprocess/info Akira From nospam at dfs.com Wed May 4 17:05:07 2016 From: nospam at dfs.com (DFS) Date: Wed, 4 May 2016 17:05:07 -0400 Subject: Not x.islower() has different output than x.isupper() in list output... In-Reply-To: <572a1752$0$1614$c3e8da3$5496439d@news.astraweb.com> References: <572407AE.1070703@icloud.com> <1461979797.3824480.593944273.0B8D8DF3@webmail.messagingengine.com> <57241097.7020801@icloud.com> <57296c7a$0$1589$c3e8da3$5496439d@news.astraweb.com> <572a1752$0$1614$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5/4/2016 11:37 AM, Steven D'Aprano wrote: > On Thu, 5 May 2016 12:09 am, DFS wrote: > >> On 5/3/2016 11:28 PM, Steven D'Aprano wrote: > >>> Languages with two distinct lettercases, like English, are called >>> bicameral. > [...] > >> Linguist much? > > > Possibly even a cunning one. I see you as more of a Colonel Angus. From nospam at dfs.com Wed May 4 18:46:46 2016 From: nospam at dfs.com (DFS) Date: Wed, 4 May 2016 18:46:46 -0400 Subject: No SQLite newsgroup, so I'll ask here about SQLite, python and MS Access Message-ID: Both of the following python commands successfully create a SQLite3 datafile which crashes Access 2003 immediately upon trying to open it (via an ODBC linked table). import sqlite3 conn = sqlite3.connect("dfile.db") import pyodbc conn = pyodbc.connect('Driver={SQLite3 ODBC Driver};Database=dfile.db') The file is created, a table is added, I add rows to the table in code, etc., and it can be read by 'DB Browser for SQLite' so it's a valid SQLite3 database, but Access won't read it. I can create and store a link to the table - using that ODBC driver - but as soon as I try to open it: "Microsoft Access has stopped working" On the other hand, a SQLite3 file created in VBScript, using the same ODBC driver, /is/ readable with Access 2003: Set conn = CreateObject("ADODB.Connection") conn.Open "Driver={SQLite3 ODBC Driver};Database=dfile.db;" python 2.7.11, pyodbc 3.0.6, ODBC driver, and Access 2003: all 32-bit OS is Win8.1Pro 64-bit. I can't find anything on the web. Any ideas? Thanks From ethan at stoneleaf.us Wed May 4 19:55:36 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 04 May 2016 16:55:36 -0700 Subject: [ANN] Aenum 1.4.1 Message-ID: <572A8BF8.7000705@stoneleaf.us> aenum 1.4.1 Advanced Enumerations (compatible with Python's stdlib Enum), NamedTuples, and NamedConstants aenum includes a Python stdlib Enum-compatible data type, as well as a metaclass-based NamedTuple implementation and a NamedConstant class. An Enum is a set of symbolic names (members) bound to unique, constant values. Within an enumeration, the members can be compared by identity, and the enumeration itself can be iterated over. If using Python 3 there is built-in support for unique values, multiple values, auto-numbering, and suspension of aliasing (members with the same value are not identical), plus the ability to have values automatically bound to attributes. A NamedTuple is a class-based, fixed-length tuple with a name for each possible position accessible using attribute-access notation as well as the standard index notation. A NamedConstant is a class whose members cannot be rebound; it lacks all other Enum capabilities, however; consequently, it can have duplicate values. Utility functions include: - skip: class that prevents attributes from being converted to a constant or enum member - module: inserts NamedConstant and Enum classes into sys.modules where it will appear to be a module whose top-level names cannot be rebound - extend_enum: add new members to enumerations after creation - enum: helper class for creating members with keywords - constant: helper class for creating constant members https://pypi.python.org/pypi/aenum https://bitbucket.org/stoneleaf/aenum From tjreedy at udel.edu Wed May 4 20:33:56 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 4 May 2016 20:33:56 -0400 Subject: Interacting with Subprocesses In-Reply-To: References: Message-ID: On 5/4/2016 2:41 PM, Dick Holmes wrote: > I am attempting to write a Python program that will interact with > a (non-Python) process. The programs will run under MinGW. The > process can use stdin/stdout commands and responses and can work > with pipes. The problem I'm having is that I can't find any > way in Python to have a continuing dialog with the process. I > have tried Popen communicate, but that protocol seems to be > limited to a single message/response pair, and the response > is not returned to the message originator until the process > terminates. Unfortunately I don't have access to the process' > source code so I can't change the communication medium. > > Is there some feature that will allow me to initiate the process > and execute multiple message/response pairs between the Python > program and the process during a single execution of the process? I have been told that multiprocessing works better for this. Not sure is true. -- Terry Jan Reedy From 4kir4.1i at gmail.com Wed May 4 21:05:30 2016 From: 4kir4.1i at gmail.com (Akira Li) Date: Thu, 05 May 2016 04:05:30 +0300 Subject: Interacting with Subprocesses References: Message-ID: <87wpn92u1x.fsf@gmail.com> Terry Reedy writes: > On 5/4/2016 2:41 PM, Dick Holmes wrote: >> I am attempting to write a Python program that will interact with >> a (non-Python) process. The programs will run under MinGW. The >> process can use stdin/stdout commands and responses and can work >> with pipes. The problem I'm having is that I can't find any >> way in Python to have a continuing dialog with the process. I >> have tried Popen communicate, but that protocol seems to be >> limited to a single message/response pair, and the response >> is not returned to the message originator until the process >> terminates. Unfortunately I don't have access to the process' >> source code so I can't change the communication medium. >> >> Is there some feature that will allow me to initiate the process >> and execute multiple message/response pairs between the Python >> program and the process during a single execution of the process? > > I have been told that multiprocessing works better for this. Not sure > is true. OP wants to interact with a *non-Python* process?*multiprocessing* module won't help here. From me+python at ixokai.io Wed May 4 22:02:06 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Wed, 04 May 2016 19:02:06 -0700 Subject: No SQLite newsgroup, so I'll ask here about SQLite, python and MS Access In-Reply-To: References: Message-ID: <1462413726.2080215.598577889.63D1ACCF@webmail.messagingengine.com> On Wed, May 4, 2016, at 03:46 PM, DFS wrote: > I can't find anything on the web. Have you tried: http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users If you really must access it over a newsgroup, you can use the Gmane mirror: http://gmane.org/info.php?group=gmane.comp.db.sqlite.general > Any ideas? Sorry, I don't use Access. -- Stephen Hansen m e @ i x o k a i . i o From michael.selik at gmail.com Wed May 4 22:12:57 2016 From: michael.selik at gmail.com (Michael Selik) Date: Thu, 05 May 2016 02:12:57 +0000 Subject: No SQLite newsgroup, so I'll ask here about SQLite, python and MS Access In-Reply-To: References: Message-ID: On Wed, May 4, 2016, 6:51 PM DFS wrote: > Both of the following python commands successfully create a SQLite3 > datafile which crashes Access 2003 immediately upon trying to open it > (via an ODBC linked table). > Have you tried using Access 2013? On the other hand, a SQLite3 file created in VBScript, using the same > ODBC driver, /is/ readable with Access 2003 Microsoft is pretty intense about backwards compatibility, sometimes even staying compatibile with old bugs. Not saying that's the case here, but it wouldn't surprise me. > From nospam at dfs.com Thu May 5 00:58:12 2016 From: nospam at dfs.com (DFS) Date: Thu, 5 May 2016 00:58:12 -0400 Subject: Whittle it on down Message-ID: Want to whittle a list like this: [u'Espa\xf1ol', 'Health & Fitness Clubs (36)', 'Health Clubs & Gymnasiums (42)', 'Health Fitness Clubs', 'Name', 'Atlanta city guide', 'edit address', 'Tweet', 'PHYSICAL FITNESS CONSULTANTS & TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', 'www.custombuiltpt.com/', 'RACQUETBALL COURTS PRIVATE', 'www.lafitness.com', 'GYMNASIUMS', 'HEALTH & FITNESS CLUBS', 'www.lafitness.com', 'HEALTH & FITNESS CLUBS', 'www.lafitness.com', 'PERSONAL FITNESS TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'EXERCISE & PHYSICAL FITNESS PROGRAMS', 'FITNESS CENTERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', 'PERSONAL FITNESS TRAINERS', '5', '4', '3', '2', '1', 'Yellow Pages', 'About Us', 'Contact Us', 'Support', 'Terms of Use', 'Privacy Policy', 'Advertise With Us', 'Add/Update Listing', 'Business Profile Login', 'F.A.Q.'] down to ['PHYSICAL FITNESS CONSULTANTS & TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', 'RACQUETBALL COURTS PRIVATE', 'GYMNASIUMS', 'HEALTH & FITNESS CLUBS', 'HEALTH & FITNESS CLUBS', 'PERSONAL FITNESS TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'EXERCISE & PHYSICAL FITNESS PROGRAMS', 'FITNESS CENTERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', 'PERSONAL FITNESS TRAINERS'] Want to keep all elements containing only upper case letters or upper case letters and ampersand (where ampersand is surrounded by spaces) Is it easier to extract elements meeting those conditions, or remove elements meeting the following conditions: * elements with a lower-case letter in them * elements with a number in them * elements with a period in them ? So far all I figured out is remove items with a period: newlist = [ x for x in oldlist if "." not in x ] Thanks for help, python gurus. From nospam at dfs.com Thu May 5 00:59:58 2016 From: nospam at dfs.com (DFS) Date: Thu, 5 May 2016 00:59:58 -0400 Subject: No SQLite newsgroup, so I'll ask here about SQLite, python and MS Access In-Reply-To: References: <1462413726.2080215.598577889.63D1ACCF@webmail.messagingengine.com> Message-ID: On 5/4/2016 10:02 PM, Stephen Hansen wrote: > On Wed, May 4, 2016, at 03:46 PM, DFS wrote: >> I can't find anything on the web. > > Have you tried: > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > If you really must access it over a newsgroup, you can use the Gmane > mirror: > http://gmane.org/info.php?group=gmane.comp.db.sqlite.general Thanks >> Any ideas? > > Sorry, I don't use Access. From me+python at ixokai.io Thu May 5 01:39:15 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Wed, 04 May 2016 22:39:15 -0700 Subject: Whittle it on down In-Reply-To: References: Message-ID: <1462426755.15465.598690257.42990546@webmail.messagingengine.com> On Wed, May 4, 2016, at 09:58 PM, DFS wrote: > Want to whittle a list like this: > > [u'Espa\xf1ol', 'Health & Fitness Clubs (36)', 'Health Clubs & > Gymnasiums (42)', 'Health Fitness Clubs', 'Name', 'Atlanta city guide', > 'edit address', 'Tweet', 'PHYSICAL FITNESS CONSULTANTS & TRAINERS', > 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', > 'www.custombuiltpt.com/', 'RACQUETBALL COURTS PRIVATE', > 'www.lafitness.com', 'GYMNASIUMS', 'HEALTH & FITNESS CLUBS', > 'www.lafitness.com', 'HEALTH & FITNESS CLUBS', 'www.lafitness.com', > 'PERSONAL FITNESS TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'EXERCISE & > PHYSICAL FITNESS PROGRAMS', 'FITNESS CENTERS', 'HEALTH CLUBS & > GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', 'PERSONAL FITNESS TRAINERS', > '5', '4', '3', '2', '1', 'Yellow Pages', 'About Us', 'Contact Us', > 'Support', 'Terms of Use', 'Privacy Policy', 'Advertise With Us', > 'Add/Update Listing', 'Business Profile Login', 'F.A.Q.'] > > down to > > ['PHYSICAL FITNESS CONSULTANTS & TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', > 'HEALTH CLUBS & GYMNASIUMS', 'RACQUETBALL COURTS PRIVATE', 'GYMNASIUMS', > 'HEALTH & FITNESS CLUBS', 'HEALTH & FITNESS CLUBS', 'PERSONAL FITNESS > TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'EXERCISE & PHYSICAL FITNESS > PROGRAMS', 'FITNESS CENTERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS > & GYMNASIUMS', 'PERSONAL FITNESS TRAINERS'] Sometimes regular expressions are the tool to do the job: Given: >>> input = [u'Espa\xf1ol', 'Health & Fitness Clubs (36)', 'Health Clubs & Gymnasiums (42)', 'Health Fitness Clubs', 'Name', 'Atlanta city guide', 'edit address', 'Tweet', 'PHYSICAL FITNESS CONSULTANTS & TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', 'www.custombuiltpt.com/', 'RACQUETBALL COURTS PRIVATE', 'www.lafitness.com', 'GYMNASIUMS', 'HEALTH & FITNESS CLUBS', 'www.lafitness.com', 'HEALTH & FITNESS CLUBS', 'www.lafitness.com', 'PERSONAL FITNESS TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'EXERCISE & PHYSICAL FITNESS PROGRAMS', 'FITNESS CENTERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', 'PERSONAL FITNESS TRAINERS', '5', '4', '3', '2', '1', 'Yellow Pages', 'About Us', 'Contact Us', 'Support', 'Terms of Use', 'Privacy Policy', 'Advertise With Us', 'Add/Update Listing', 'Business Profile Login', 'F.A.Q.'] Then: >>> pattern = re.compile(r"^[A-Z\s&]+$") >>> output = [x for x in list if pattern.match(x)] >>> output ['PHYSICAL FITNESS CONSULTANTS & TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', 'RACQUETBALL COURTS PRIVATE', 'GYMNASIUMS', 'HEALTH & FITNESS CLUBS', 'HEALTH & FITNESS CLUBS', 'PERSONAL FITNESS TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'EXERCISE & PHYSICAL FITNESS PROGRAMS', 'FITNESS CENTERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', 'PERSONAL FITNESS TRAINERS'] -- Stephen Hansen m e @ i x o k a i . i o From jussi.piitulainen at helsinki.fi Thu May 5 01:53:54 2016 From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen) Date: Thu, 05 May 2016 08:53:54 +0300 Subject: Whittle it on down References: Message-ID: DFS writes: . . > Want to keep all elements containing only upper case letters or upper > case letters and ampersand (where ampersand is surrounded by spaces) > > Is it easier to extract elements meeting those conditions, or remove > elements meeting the following conditions: > > * elements with a lower-case letter in them > * elements with a number in them > * elements with a period in them > > ? > > > So far all I figured out is remove items with a period: > newlist = [ x for x in oldlist if "." not in x ] > Either way is easy to approximate with a regex: import re upper = re.compile(r'[A-Z &]+') lower = re.compile(r'[^A-Z &]') print([datum for datum in data if upper.fullmatch(datum)]) print([datum for datum in data if not lower.search(datum)]) I've skipped testing that the ampersand is between spaces, and I've skipped the period. Adjust. This considers only ASCII upper case letters. You can add individual letters that matter to you, or you can reach for the documentation to find if there is some generic notation for all upper case letters. The newer regex package on PyPI supports POSIX character classes like [:upper:], I think, and there may or may not be notation for Unicode character categories in re or regex - LU would be Letter, Uppercase. From steve+comp.lang.python at pearwood.info Thu May 5 02:04:13 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Thu, 05 May 2016 16:04:13 +1000 Subject: Whittle it on down References: Message-ID: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> On Thursday 05 May 2016 14:58, DFS wrote: > Want to whittle a list like this: [...] > Want to keep all elements containing only upper case letters or upper > case letters and ampersand (where ampersand is surrounded by spaces) Start by writing a function or a regex that will distinguish strings that match your conditions from those that don't. A regex might be faster, but here's a function version. def isupperalpha(string): return string.isalpha() and string.isupper() def check(string): if isupperalpha(string): return True parts = string.split("&") if len(parts) < 2: return False # Don't strip leading spaces from the start of the string. parts[0] = parts[0].rstrip(" ") # Or trailing spaces from the end of the string. parts[-1] = parts[-1].lstrip(" ") # But strip leading and trailing spaces from the middle parts # (if any). for i in range(1, len(parts)-1): parts[i] = parts[i].strip(" ") return all(isupperalpha(part) for part in parts) Now you have two ways of filtering this. The obvious way is to extract elements which meet the condition. Here are two ways: # List comprehension. newlist = [item for item in oldlist if check(item)] # Filter, Python 2 version newlist = filter(check, oldlist) # Filter, Python 3 version newlist = list(filter(check, oldlist)) In practice, this is the best (fastest, simplest) way. But if you fear that you will run out of memory dealing with absolutely humongous lists with hundreds of millions or billions of strings, you can remove items in place: def remove(func, alist): for i in range(len(alist)-1, -1, -1): if not func(alist[i]): del alist[i] Note the magic incantation to iterate from the end of the list towards the front. If you do it the other way, Bad Things happen. Note that this will use less memory than extracting the items, but it will be much slower. You can combine the best of both words. Here is a version that uses a temporary list to modify the original in place: # works in both Python 2 and 3 def remove(func, alist): # Modify list in place, the fast way. alist[:] = filter(check, alist) -- Steve From me+python at ixokai.io Thu May 5 02:46:06 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Wed, 04 May 2016 23:46:06 -0700 Subject: Whittle it on down In-Reply-To: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> Message-ID: <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> On Wed, May 4, 2016, at 11:04 PM, Steven D'Aprano wrote: > Start by writing a function or a regex that will distinguish strings that > match your conditions from those that don't. A regex might be faster, but > here's a function version. > ... snip ... Yikes. I'm all for the idea that one shouldn't go to regex when Python's powerful string type can answer the problem more clearly, but this seems to go out of its way to do otherwise. I don't even care about faster: Its overly complicated. Sometimes a regular expression really is the clearest way to solve a problem. -- Stephen Hansen m e @ i x o k a i . i o From steve+comp.lang.python at pearwood.info Thu May 5 03:04:59 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Thu, 05 May 2016 17:04:59 +1000 Subject: Whittle it on down References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> Message-ID: <572af09d$0$1508$c3e8da3$5496439d@news.astraweb.com> On Thursday 05 May 2016 16:46, Stephen Hansen wrote: > On Wed, May 4, 2016, at 11:04 PM, Steven D'Aprano wrote: >> Start by writing a function or a regex that will distinguish strings that >> match your conditions from those that don't. A regex might be faster, but >> here's a function version. >> ... snip ... > > Yikes. I'm all for the idea that one shouldn't go to regex when Python's > powerful string type can answer the problem more clearly, but this seems > to go out of its way to do otherwise. > > I don't even care about faster: Its overly complicated. Sometimes a > regular expression really is the clearest way to solve a problem. You're probably right, but I find it easier to reason about matching in Python rather than the overly terse, cryptic regular expression mini- language. I haven't tested my function version, but I'm 95% sure that it is correct. It trickiest part of it is the logic about splitting around ampersands. And I'll cheerfully admit that it isn't easy to extend to (say) "ampersand, or at signs". But your regex solution: r"^[A-Z\s&]+$" is much smaller and more compact, but *wrong*. For instance, your regex wrongly accepts both "&&&&&" and " " as valid strings, and wrongly rejects "????". Your Greek customers will be sad... Oh, I just realised, I should have looked more closely at the examples given. because the specification given by DFS does not match the examples. DFS says that only uppercase letters and ampersands are allowed, but their examples include strings with spaces, e.g. 'FITNESS CENTERS' despite the lack of ampersands. (I read the spec literally as spaces only allowed if they surround an ampersand.) Oops, mea culpa. That makes the check function much simpler and easier to extend: def check(string): string = string.replace("&", "").replace(" ", "") return string.isalpha() and string.isupper() and now I'm 95% confident it is correct without testing, this time for sure! ;-) -- Steve From me+python at ixokai.io Thu May 5 03:34:29 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Thu, 05 May 2016 00:34:29 -0700 Subject: Whittle it on down In-Reply-To: <572af09d$0$1508$c3e8da3$5496439d@news.astraweb.com> References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af09d$0$1508$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1462433669.32454.598751585.63AF2D12@webmail.messagingengine.com> On Thu, May 5, 2016, at 12:04 AM, Steven D'Aprano wrote: > On Thursday 05 May 2016 16:46, Stephen Hansen wrote: > > > On Wed, May 4, 2016, at 11:04 PM, Steven D'Aprano wrote: > >> Start by writing a function or a regex that will distinguish strings that > >> match your conditions from those that don't. A regex might be faster, but > >> here's a function version. > >> ... snip ... > > > > Yikes. I'm all for the idea that one shouldn't go to regex when Python's > > powerful string type can answer the problem more clearly, but this seems > > to go out of its way to do otherwise. > > > > I don't even care about faster: Its overly complicated. Sometimes a > > regular expression really is the clearest way to solve a problem. > > You're probably right, but I find it easier to reason about matching in > Python rather than the overly terse, cryptic regular expression mini- > language. > > I haven't tested my function version, but I'm 95% sure that it is > correct. > It trickiest part of it is the logic about splitting around ampersands. > And > I'll cheerfully admit that it isn't easy to extend to (say) "ampersand, > or > at signs". But your regex solution: > > r"^[A-Z\s&]+$" > > is much smaller and more compact, but *wrong*. For instance, your regex > wrongly accepts both "&&&&&" and " " as valid strings, and wrongly > rejects "????". Your Greek customers will be sad... Meh. You have a pedantic definition of wrong. Given the inputs, it produced right output. Very often that's enough. Perfect is the enemy of good, it's said. There's no situation where "&&&&&" and " " will exist in the given dataset, and recognizing that is important. You don't have to account for every bit of nonsense. If the OP needs a unicode-aware solution that redefines "A-Z" as perhaps "\w" with an isupper call. Its still far simpler then you're suggesting. -- Stephen Hansen m e @ i x o k a i . i o From steve+comp.lang.python at pearwood.info Thu May 5 03:36:47 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Thu, 05 May 2016 17:36:47 +1000 Subject: Whittle it on down References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> Message-ID: <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> Oh, a further thought... On Thursday 05 May 2016 16:46, Stephen Hansen wrote: > On Wed, May 4, 2016, at 11:04 PM, Steven D'Aprano wrote: >> Start by writing a function or a regex that will distinguish strings that >> match your conditions from those that don't. A regex might be faster, but >> here's a function version. >> ... snip ... > > Yikes. I'm all for the idea that one shouldn't go to regex when Python's > powerful string type can answer the problem more clearly, but this seems > to go out of its way to do otherwise. > > I don't even care about faster: Its overly complicated. Sometimes a > regular expression really is the clearest way to solve a problem. Putting non-ASCII letters aside for the moment, how would you match these specs as a regular expression? - All uppercase ASCII letters (A to Z only), optionally separated into words by either a bare ampersand (e.g. "AAA&AAA") or an ampersand with leading and trailing spaces (spaces only, not arbitrary whitespace): "AAA & AAA". - The number of spaces on either side of the ampersands need not be the same: "AAA& BBB & CCC" should match. - Leading or trailing spaces, or spaces not surrounding an ampersand, must not match: "AAA BBB" must be rejected. - Leading or trailing ampersands must also be rejected. This includes the case where the string is nothing but ampersands. - Consecutive ampersands "AAA&&&BBB" and the empty string must be rejected. I get something like this: r"(^[A-Z]+$)|(^([A-Z]+[ ]*\&[ ]*[A-Z]+)+$)" but it fails on strings like "AA & A & A". What am I doing wrong? For the record, here's my brief test suite: def test(pat): for s in ("", " ", "&" "A A", "A&", "&A", "A&&A", "A& &A"): assert re.match(pat, s) is None for s in ("A", "A & A", "AA&A", "AA & A & A"): assert re.match(pat, s) -- Steve From __peter__ at web.de Thu May 5 04:17:47 2016 From: __peter__ at web.de (Peter Otten) Date: Thu, 05 May 2016 10:17:47 +0200 Subject: Whittle it on down References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> Message-ID: Steven D'Aprano wrote: > Oh, a further thought... > > > On Thursday 05 May 2016 16:46, Stephen Hansen wrote: > >> On Wed, May 4, 2016, at 11:04 PM, Steven D'Aprano wrote: >>> Start by writing a function or a regex that will distinguish strings >>> that match your conditions from those that don't. A regex might be >>> faster, but here's a function version. >>> ... snip ... >> >> Yikes. I'm all for the idea that one shouldn't go to regex when Python's >> powerful string type can answer the problem more clearly, but this seems >> to go out of its way to do otherwise. >> >> I don't even care about faster: Its overly complicated. Sometimes a >> regular expression really is the clearest way to solve a problem. > > Putting non-ASCII letters aside for the moment, how would you match these > specs as a regular expression? > > - All uppercase ASCII letters (A to Z only), optionally separated into > words by either a bare ampersand (e.g. "AAA&AAA") or an ampersand with > leading and > trailing spaces (spaces only, not arbitrary whitespace): "AAA & AAA". > > - The number of spaces on either side of the ampersands need not be the > same: "AAA& BBB & CCC" should match. > > - Leading or trailing spaces, or spaces not surrounding an ampersand, must > not match: "AAA BBB" must be rejected. > > - Leading or trailing ampersands must also be rejected. This includes the > case where the string is nothing but ampersands. > > - Consecutive ampersands "AAA&&&BBB" and the empty string must be > rejected. > > > I get something like this: > > r"(^[A-Z]+$)|(^([A-Z]+[ ]*\&[ ]*[A-Z]+)+$)" > > > but it fails on strings like "AA & A & A". What am I doing wrong? > > > For the record, here's my brief test suite: > > > def test(pat): > for s in ("", " ", "&" "A A", "A&", "&A", "A&&A", "A& &A"): > assert re.match(pat, s) is None > for s in ("A", "A & A", "AA&A", "AA & A & A"): > assert re.match(pat, s) >>> def test(pat): ... for s in ("", " ", "&" "A A", "A&", "&A", "A&&A", "A& &A"): ... assert re.match(pat, s) is None ... for s in ("A", "A & A", "AA&A", "AA & A & A"): ... assert re.match(pat, s) ... >>> test("^A+( *& *A+)*$") >>> From steve+comp.lang.python at pearwood.info Thu May 5 04:41:31 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Thu, 05 May 2016 18:41:31 +1000 Subject: Whittle it on down References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af09d$0$1508$c3e8da3$5496439d@news.astraweb.com> <1462433669.32454.598751585.63AF2D12@webmail.messagingengine.com> Message-ID: <572b073e$0$1611$c3e8da3$5496439d@news.astraweb.com> On Thursday 05 May 2016 17:34, Stephen Hansen wrote: > Meh. You have a pedantic definition of wrong. Given the inputs, it > produced right output. Very often that's enough. Perfect is the enemy of > good, it's said. And this is a *perfect* example of why we have things like this: http://www.bbc.com/future/story/20160325-the-names-that-break-computer- systems "Nobody will ever be called Null." "Nobody has quotation marks in their name." "Nobody will have a + sign in their email address." "Nobody has a legal gender other than Male or Female." "Nobody will lean on the keyboard and enter gobbledygook into our form." "Nobody will try to write more data than the space they allocated for it." > There's no situation where "&&&&&" and " " will exist in the given > dataset, and recognizing that is important. You don't have to account > for every bit of nonsense. Whenever a programmer says "This case will never happen", ten thousand computers crash. http://www.kr41.net/2016/05-03-shit_driven_development.html -- Steven D'Aprano From cl at isbd.net Thu May 5 05:01:36 2016 From: cl at isbd.net (cl at isbd.net) Date: Thu, 5 May 2016 10:01:36 +0100 Subject: No SQLite newsgroup, so I'll ask here about SQLite, python and MS Access References: Message-ID: There's a gmane 'newsgroup from a mailing list' for sqlite:- gmane.comp.db.sqlite.general It's quite active and helpful too. (Also 'announce' and others) -- Chris Green ? From jldunn2000 at gmail.com Thu May 5 05:22:54 2016 From: jldunn2000 at gmail.com (loial) Date: Thu, 5 May 2016 02:22:54 -0700 (PDT) Subject: smtplib not working when python run under windows service via Local System account Message-ID: <4157043d-2b09-47e8-b4ba-d7f21323f298@googlegroups.com> I have a python 2.7.10 script which is being run under a windows service on windows 2012 server . The python script uses smtplib to send an email. It works fine when the windows service is run as a local user, but not when the windows service is configured to run as Local System account. I get no exception from smtplib, but the email fails to arrive. Any ideas? From eryksun at gmail.com Thu May 5 05:38:47 2016 From: eryksun at gmail.com (eryk sun) Date: Thu, 5 May 2016 04:38:47 -0500 Subject: Interacting with Subprocesses In-Reply-To: <871t5h4jst.fsf@gmail.com> References: <871t5h4jst.fsf@gmail.com> Message-ID: On Wed, May 4, 2016 at 4:04 PM, Akira Li <4kir4.1i at gmail.com> wrote: > > Pass stdin=PIPE, stdout=PIPE and use p.stdin, p.stdout file objects to > write input, read output from the child process. > > Beware, there could be buffering issues or the child process may change > its behavior some other way when the standard input/output streams are > redirected. See > http://pexpect.readthedocs.io/en/stable/FAQ.html#whynotpipe On Linux, you may be able to use stdbuf [1] to modify standard I/O buffering. stdbuf sets the LD_PRELOAD [2] environment variable to load libstdbuf.so [3]. For example, the following shows the environment variables created by "stdbuf -oL": $ stdbuf -oL python -c 'import os;print os.environ["LD_PRELOAD"]' /usr/lib/coreutils/libstdbuf.so $ stdbuf -oL python -c 'import os;print os.environ["_STDBUF_O"]' L [1]: http://www.gnu.org/software/coreutils/manual/html_node/stdbuf-invocation.html [2]: http://www.linuxjournal.com/article/7795 [3]: http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/libstdbuf.c?id=v8.21 On Windows, if you can modify the program, then you can check for a command-line option or an environment variable, like Python's -u and PYTHONUNBUFFERED. If you can't modify the source, I think you might be able to hack something similar to the Linux LD_PRELOAD environment variable by creating a stdbuf.exe launcher that debugs the process and injects a DLL after the loader's first-chance breakpoint. The injected stdbuff.dll would need to be able to get the standard streams for common CRTs, such as by calling __acrt_iob_func for ucrtbase.dll. Also, unlike the Linux command, stdbuf.exe would have to wait on the child, since the Windows API doesn't have fork/exec. From nospam at dfs.com Thu May 5 08:31:00 2016 From: nospam at dfs.com (DFS) Date: Thu, 5 May 2016 08:31:00 -0400 Subject: Whittle it on down In-Reply-To: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> Message-ID: On 5/5/2016 2:04 AM, Steven D'Aprano wrote: > On Thursday 05 May 2016 14:58, DFS wrote: > >> Want to whittle a list like this: > [...] >> Want to keep all elements containing only upper case letters or upper >> case letters and ampersand (where ampersand is surrounded by spaces) > > > Start by writing a function or a regex that will distinguish strings that > match your conditions from those that don't. A regex might be faster, but > here's a function version. > > def isupperalpha(string): > return string.isalpha() and string.isupper() > > def check(string): > if isupperalpha(string): > return True > parts = string.split("&") > if len(parts) < 2: > return False > # Don't strip leading spaces from the start of the string. > parts[0] = parts[0].rstrip(" ") > # Or trailing spaces from the end of the string. > parts[-1] = parts[-1].lstrip(" ") > # But strip leading and trailing spaces from the middle parts > # (if any). > for i in range(1, len(parts)-1): > parts[i] = parts[i].strip(" ") > return all(isupperalpha(part) for part in parts) > > > Now you have two ways of filtering this. The obvious way is to extract > elements which meet the condition. Here are two ways: > > # List comprehension. > newlist = [item for item in oldlist if check(item)] > > # Filter, Python 2 version > newlist = filter(check, oldlist) > > # Filter, Python 3 version > newlist = list(filter(check, oldlist)) > > > In practice, this is the best (fastest, simplest) way. But if you fear that > you will run out of memory dealing with absolutely humongous lists with > hundreds of millions or billions of strings, you can remove items in place: > > > def remove(func, alist): > for i in range(len(alist)-1, -1, -1): > if not func(alist[i]): > del alist[i] > > > Note the magic incantation to iterate from the end of the list towards the > front. If you do it the other way, Bad Things happen. Note that this will > use less memory than extracting the items, but it will be much slower. > > You can combine the best of both words. Here is a version that uses a > temporary list to modify the original in place: > > # works in both Python 2 and 3 > def remove(func, alist): > # Modify list in place, the fast way. > alist[:] = filter(check, alist) You are out of your mind. From nospam at dfs.com Thu May 5 08:44:48 2016 From: nospam at dfs.com (DFS) Date: Thu, 5 May 2016 08:44:48 -0400 Subject: Whittle it on down In-Reply-To: References: <1462426755.15465.598690257.42990546@webmail.messagingengine.com> Message-ID: On 5/5/2016 1:39 AM, Stephen Hansen wrote: > pattern = re.compile(r"^[A-Z\s&]+$") > output = [x for x in list if pattern.match(x)] Holy Shr"^[A-Z\s&]+$" One line of parsing! I was figuring a few list comprehensions would do it - this is better. (note: the reason I specified 'spaces around ampersand' is so it would remove 'Q&A' if that ever came up - but some people write 'Q & A', so I'll live with that exception, or try to tweak it myself. You're the man, man. Thank you! From nospam at dfs.com Thu May 5 08:57:31 2016 From: nospam at dfs.com (DFS) Date: Thu, 5 May 2016 08:57:31 -0400 Subject: Whittle it on down In-Reply-To: References: Message-ID: On 5/5/2016 1:53 AM, Jussi Piitulainen wrote: > Either way is easy to approximate with a regex: > > import re > upper = re.compile(r'[A-Z &]+') > lower = re.compile(r'[^A-Z &]') > print([datum for datum in data if upper.fullmatch(datum)]) > print([datum for datum in data if not lower.search(datum)]) This is similar to Hansen's solution. > I've skipped testing that the ampersand is between spaces, and I've > skipped the period. Adjust. Will do. > This considers only ASCII upper case letters. You can add individual > letters that matter to you, or you can reach for the documentation to > find if there is some generic notation for all upper case letters. > > The newer regex package on PyPI supports POSIX character classes like > [:upper:], I think, and there may or may not be notation for Unicode > character categories in re or regex - LU would be Letter, Uppercase. Thanks. From random832 at fastmail.com Thu May 5 09:13:34 2016 From: random832 at fastmail.com (Random832) Date: Thu, 05 May 2016 09:13:34 -0400 Subject: Whittle it on down In-Reply-To: <572b073e$0$1611$c3e8da3$5496439d@news.astraweb.com> References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af09d$0$1508$c3e8da3$5496439d@news.astraweb.com> <1462433669.32454.598751585.63AF2D12@webmail.messagingengine.com> <572b073e$0$1611$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1462454014.2960218.598997953.58FF04DC@webmail.messagingengine.com> On Thu, May 5, 2016, at 04:41, Steven D'Aprano wrote: > > There's no situation where "&&&&&" and " " will exist in the given > > dataset, and recognizing that is important. You don't have to account > > for every bit of nonsense. > > Whenever a programmer says "This case will never happen", ten thousand > computers crash. What crash can including such an entry in the output list cause? Should the regex also ensure that the data only includes *english words* separated by space-ampersand-space? From random832 at fastmail.com Thu May 5 09:21:39 2016 From: random832 at fastmail.com (Random832) Date: Thu, 05 May 2016 09:21:39 -0400 Subject: Whittle it on down In-Reply-To: <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1462454499.2962191.598999745.40BB8A1E@webmail.messagingengine.com> On Thu, May 5, 2016, at 03:36, Steven D'Aprano wrote: > Putting non-ASCII letters aside for the moment, how would you match these > specs as a regular expression? Well, obviously *your* language (not the OP's), given the cases you reject, is "one or more sequences of letters separated by space*-ampersand-space*", and that is actually one of the easiest kinds of regex to write: "[A-Z]+( *& *[A-Z]+)*". However, your spec is wrong: > - Leading or trailing spaces, or spaces not surrounding an ampersand, > must not match: "AAA BBB" must be rejected. The *very first* item in OP's list of good outputs is 'PHYSICAL FITNESS CONSULTANTS & TRAINERS'. If you want something that's extremely conservative (except for the *very odd in context* choice of allowing arbitrary numbers of spaces - why would you allow this but reject leading or trailing space?) and accepts all of OP's input: [A-Z]+(( *& *| +)[A-Z]+)* From me+python at ixokai.io Thu May 5 09:32:24 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Thu, 05 May 2016 06:32:24 -0700 Subject: Whittle it on down In-Reply-To: <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1462455144.93995.599007201.4350517C@webmail.messagingengine.com> On Thu, May 5, 2016, at 12:36 AM, Steven D'Aprano wrote: > Oh, a further thought... > > On Thursday 05 May 2016 16:46, Stephen Hansen wrote: > > I don't even care about faster: Its overly complicated. Sometimes a > > regular expression really is the clearest way to solve a problem. > > Putting non-ASCII letters aside for the moment, how would you match these > specs as a regular expression? I don't know, but mostly because I wouldn't even try. The requirements are over-specified. If you look at the OP's data (and based on previous conversation), he's doing web scraping and trying to pull out good data. There's no absolutely perfect way to do that because the system he's scraping isn't meant for data processing. The data isn't cleanly articulated. Instead, he wants a heuristic to pull out what look like section titles. The OP looked at the data and came up with a simple set of rules that identify these section titles: >> Want to keep all elements containing only upper case letters or upper case letters and ampersand (where ampersand is surrounded by spaces) This translates naturally into a simple regular expression: an uppercase string with spaces and &'s. Now, that expression doesn't 100% encode every detail of that rule-- it allows both Q&A and Q & A-- but on my own looking at the data, I suspect its good enough. The titles are clearly separate from the other data scraped by their being upper cased. We just need to expand our allowed character range into spaces and &'s. Nothing in the OP's request demands the kind of rigorous matching that your scenario does. Its a practical problem with a simple, practical answer. -- Stephen Hansen m e @ i x o k a i . i o From nospam at dfs.com Thu May 5 10:36:27 2016 From: nospam at dfs.com (DFS) Date: Thu, 5 May 2016 10:36:27 -0400 Subject: Whittle it on down In-Reply-To: References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> <1462455144.93995.599007201.4350517C@webmail.messagingengine.com> Message-ID: On 5/5/2016 9:32 AM, Stephen Hansen wrote: > On Thu, May 5, 2016, at 12:36 AM, Steven D'Aprano wrote: >> Oh, a further thought... >> >> On Thursday 05 May 2016 16:46, Stephen Hansen wrote: >>> I don't even care about faster: Its overly complicated. Sometimes a >>> regular expression really is the clearest way to solve a problem. >> >> Putting non-ASCII letters aside for the moment, how would you match these >> specs as a regular expression? > > I don't know, but mostly because I wouldn't even try. The requirements > are over-specified. If you look at the OP's data (and based on previous > conversation), he's doing web scraping and trying to pull out good data. > There's no absolutely perfect way to do that because the system he's > scraping isn't meant for data processing. The data isn't cleanly > articulated. > > Instead, he wants a heuristic to pull out what look like section titles. Assigned by a company named localeze, apparently. http://www.usdirectory.com/cat/g0 https://www.neustarlocaleze.biz/welcome/ > The OP looked at the data and came up with a simple set of rules that > identify these section titles: > >>> Want to keep all elements containing only upper case letters or upper > case letters and ampersand (where ampersand is surrounded by spaces) > > This translates naturally into a simple regular expression: an uppercase > string with spaces and &'s. Now, that expression doesn't 100% encode > every detail of that rule-- it allows both Q&A and Q & A-- but on my own > looking at the data, I suspect its good enough. The titles are clearly > separate from the other data scraped by their being upper cased. We just > need to expand our allowed character range into spaces and &'s. > > Nothing in the OP's request demands the kind of rigorous matching that > your scenario does. Its a practical problem with a simple, practical > answer. Yes. And simplicity + practicality = successfulality. And I do a sanity check before using the data anyway: after parse and cleanup and regex matching, I make sure all lists have the same number of elements: lenData = [len(title),len(names),len(addr),len(street),len(city),len(state),len(zip)] if len(set(lenData)) != 1: alert the media From steve at pearwood.info Thu May 5 11:39:40 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 06 May 2016 01:39:40 +1000 Subject: Whittle it on down References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> Message-ID: <572b693d$0$1599$c3e8da3$5496439d@news.astraweb.com> On Thu, 5 May 2016 06:17 pm, Peter Otten wrote: >> I get something like this: >> >> r"(^[A-Z]+$)|(^([A-Z]+[ ]*\&[ ]*[A-Z]+)+$)" >> >> >> but it fails on strings like "AA & A & A". What am I doing wrong? > test("^A+( *& *A+)*$") Thanks Peter, that's nice! -- Steven From jcasale at activenetwerx.com Thu May 5 11:42:10 2016 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Thu, 5 May 2016 15:42:10 +0000 Subject: Ctypes c_void_p overflow Message-ID: <7492a519afee44b1875f7ec51edf44a7@activenetwerx.com> I have CDLL function I use to get a pointer, several other functions happily accept this pointer which is really a long when passed to ctypes.c_void_p. However, only one with same type def in the prototype overflows. Docs suggest c_void_p takes an int but that is not what the first call returns, nor what all but one function happily accept? Anyone familiar enough with ctypes that can shed some light? Thanks, jlc From steve at pearwood.info Thu May 5 13:13:35 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 06 May 2016 03:13:35 +1000 Subject: Whittle it on down References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af09d$0$1508$c3e8da3$5496439d@news.astraweb.com> <1462433669.32454.598751585.63AF2D12@webmail.messagingengine.com> <572b073e$0$1611$c3e8da3$5496439d@news.astraweb.com> <1462454014.2960218.598997953.58FF04DC@webmail.messagingengine.com> Message-ID: <572b7f41$0$1598$c3e8da3$5496439d@news.astraweb.com> On Thu, 5 May 2016 11:13 pm, Random832 wrote: > On Thu, May 5, 2016, at 04:41, Steven D'Aprano wrote: >> > There's no situation where "&&&&&" and " " will exist in the given >> > dataset, and recognizing that is important. You don't have to account >> > for every bit of nonsense. >> >> Whenever a programmer says "This case will never happen", ten thousand >> computers crash. > > What crash can including such an entry in the output list cause? How do I know? It depends what you do with that list. But if you assume that your list contains alphabetical strings, and pass it on to code that expects alphabetical strings, why is it so hard to believe that it might choke when it receives a non-alphabetical string? > Should the regex also ensure that the data only includes *english words* > separated by space-ampersand-space? That wasn't part of the specification. But for some applications, yes, you should ensure the data includes only English words. -- Steven From steve at pearwood.info Thu May 5 13:15:42 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 06 May 2016 03:15:42 +1000 Subject: Ctypes c_void_p overflow References: <7492a519afee44b1875f7ec51edf44a7@activenetwerx.com> Message-ID: <572b7fbf$0$1598$c3e8da3$5496439d@news.astraweb.com> On Fri, 6 May 2016 01:42 am, Joseph L. Casale wrote: > I have CDLL function I use to get a pointer, several other functions > happily accept this pointer which is really a long when passed to > ctypes.c_void_p. However, only one with same type def in the prototype > overflows. Docs suggest c_void_p takes an int but that is not what the > first call returns, nor what all but one function happily accept? > > Anyone familiar enough with ctypes that can shed some light? I'm not a ctypes expert, but you might get better responses if you show the code you're using, the expected result, and the result you actually get. -- Steven From steve at pearwood.info Thu May 5 13:43:40 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 06 May 2016 03:43:40 +1000 Subject: Whittle it on down References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> <1462455144.93995.599007201.4350517C@webmail.messagingengine.com> Message-ID: <572b864e$0$1622$c3e8da3$5496439d@news.astraweb.com> On Thu, 5 May 2016 11:32 pm, Stephen Hansen wrote: > On Thu, May 5, 2016, at 12:36 AM, Steven D'Aprano wrote: >> Oh, a further thought... >> >> On Thursday 05 May 2016 16:46, Stephen Hansen wrote: >> > I don't even care about faster: Its overly complicated. Sometimes a >> > regular expression really is the clearest way to solve a problem. >> >> Putting non-ASCII letters aside for the moment, how would you match these >> specs as a regular expression? > > I don't know, but mostly because I wouldn't even try. Really? Peter Otten seems to have found a solution, and Random832 almost found it too. > The requirements > are over-specified. If you look at the OP's data (and based on previous > conversation), he's doing web scraping and trying to pull out good data. I'm not talking about the OP's data. I'm talking about *my* requirements. I thought that this was a friendly discussion about regexes, but perhaps I was mistaken. Because I sure am feeling a lot of hostility to the ideas that regexes are not necessarily the only way to solve this, and that data validation is a good thing. > There's no absolutely perfect way to do that because the system he's > scraping isn't meant for data processing. The data isn't cleanly > articulated. Right. Which makes it *more*, not less, important to be sure that your regex doesn't match too much, because your data is likely to be contaminated by junk strings that don't belong in the data and shouldn't be accepted. I've done enough web scraping to realise just how easy it is to start grabbing data from the wrong part of the file. > Instead, he wants a heuristic to pull out what look like section titles. Good for him. I asked a different question. Does my question not count? > The OP looked at the data and came up with a simple set of rules that > identify these section titles: > >>> Want to keep all elements containing only upper case letters or upper > case letters and ampersand (where ampersand is surrounded by spaces) That simple rule doesn't match his examples, as I know too well because I made the silly mistake of writing to the written spec as written without reading the examples as well. As I already admitted. That was a silly mistake because I know very well that people are really bad at writing detailed specs that neither match too much nor too little. But you know, I was more focused on the rest of his question, namely whether it was better to extract the matches strings into a new list, or delete the non-matches from the existing string, and just got carried away writing the match function. I didn't actually expect anyone to use it. It was untested, and I hinted that a regex would probably be better. I was trying to teach DFS a generic programming technique, not solve his stupid web scraping problem for him. What happens next time when he's trying to filter a list of floats, or Widgets? Should he convert them to strings so he can use a regex to match them, or should he learn about general filtering techniques? > This translates naturally into a simple regular expression: an uppercase > string with spaces and &'s. Now, that expression doesn't 100% encode > every detail of that rule-- it allows both Q&A and Q & A-- but on my own > looking at the data, I suspect its good enough. The titles are clearly > separate from the other data scraped by their being upper cased. We just > need to expand our allowed character range into spaces and &'s. > > Nothing in the OP's request demands the kind of rigorous matching that > your scenario does. Its a practical problem with a simple, practical > answer. Yes, and that practical answer needs to reject: - the empty string, because it is easy to mistakenly get empty strings when scraping data, especially if you post-process the data; - strings that are all spaces, because " " cannot possibly be a title; - strings that are all ampersands, because "&&&&&" is not a title, and it almost surely indicates that your scraping has gone wrong and you're reading junk from somewhere; - even leading and trailing spaces are suspect: " FOO " doesn't match any of the examples given, and it seems unlikely to be a title. Presumably the strings have already been filtered or post-processed to have leading and trailing spaces removed, in which case " FOO " reveals a bug. -- Steven From jussi.piitulainen at helsinki.fi Thu May 5 13:49:26 2016 From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen) Date: Thu, 05 May 2016 20:49:26 +0300 Subject: Whittle it on down References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> Message-ID: Steven D'Aprano writes: > I get something like this: > > r"(^[A-Z]+$)|(^([A-Z]+[ ]*\&[ ]*[A-Z]+)+$)" > > > but it fails on strings like "AA & A & A". What am I doing wrong? It cannot split the string as (LETTERS & LETTERS)(LETTERS & LETTERS) when the middle part is just one LETTER. That's something of a misanalysis anyway. I notice that the correct pattern has already been posted at least thrice and you have acknowledged one of them. But I think you are also trying to do too much with a single regex. A more promising start is to think of the whole string as "parts" joined with "glue", then split with a glue pattern and test the parts: import re glue = re.compile(" *& *| +") keep, drop = [], [] for datum in data: items = glue.split(datum) if all(map(str.isupper, items)): keep.append(datum) else: drop.append(datum) That will cope with Greek, by the way. It's annoying that the order of the branches of the glue pattern above matters. One _does_ have problems when one uses the usual regex engines. Capturing groups in the glue pattern would produce glue items in the split output. Either avoid them or deal with them: one could split with the underspecific "([ &]+)" and then check that each glue item contains at most one ampersand. One could also allow other punctuation, and then check afterwards. One can use _another_ regex to test individual parts. Code above used str.isupper to test a part. The improved regex package (from PyPI, to cope with Greek) can do the same: import regex part = regex.compile("[[:upper:]]+") glue = regex.compile(" *& *| *") keep, drop = [], [] for datum in data: items = glue.split(datum) if all(map(part.fullmatch, items)): keep.append(datum) else: drop.append(datum) Just "[A-Z]+" suffices for ASCII letters, and "[A-Z??]+" copes with most of Finnish; the [:upper:] class is nicer and there's much more that is nicer in the newer regex package. The point of using a regex for this is that the part pattern can then be generalized to allow some punctuation or digits in a part, for example. Anything that the glue pattern doesn't consume. (Nothing wrong with using other techniques for this, either; str.isupper worked nicely above.) It's also possible to swap the roles of the patterns. Split with a part pattern. Then check that the text between such parts is glue: keep, drop = [], [] for datum in data: items = part.split(datum) if all(map(glue.fullmatch, items)): keep.append(datum) else: drop.append(datum) The point is to keep the patterns simple by making them more local, or more relaxed, followed by a further test. This way they can be made to do more, but not more than they reasonably can. Note also the use of re.fullmatch instead of re.match (let alone re.search) when a full match is required! This gets rid of all anchors in the pattern, which may in turn allow fewer parentheses inside the pattern. The usual regex engines are not perfect, but parts of them are fantastic. From steve at pearwood.info Thu May 5 13:54:11 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 06 May 2016 03:54:11 +1000 Subject: Whittle it on down References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> Message-ID: <572b88c5$0$1601$c3e8da3$5496439d@news.astraweb.com> On Thu, 5 May 2016 10:31 pm, DFS wrote: > You are out of your mind. That's twice you've tried to put me down, first by dismissing my comments about text processing with "Linguist much", and now an outright insult. The first time I laughed it off and made a joke about it. I won't do that again. You asked whether it was better to extract the matching strings into a new list, or remove them in place in the existing list. I not only showed you how to do both, but I tried to give you the mental tools to understand when you should pick one answer over the other. And your response is to insult me and question my sanity. Well, DFS, I might be crazy, but I'm not stupid. If that's really how you feel about my answers, I won't make the mistake of wasting my time answering your questions in the future. Over to you now. -- Steven From steve at pearwood.info Thu May 5 14:03:23 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 06 May 2016 04:03:23 +1000 Subject: Whittle it on down References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> <1462454499.2962191.598999745.40BB8A1E@webmail.messagingengine.com> Message-ID: <572b8aee$0$1589$c3e8da3$5496439d@news.astraweb.com> On Thu, 5 May 2016 11:21 pm, Random832 wrote: > On Thu, May 5, 2016, at 03:36, Steven D'Aprano wrote: >> Putting non-ASCII letters aside for the moment, how would you match these >> specs as a regular expression? > > Well, obviously *your* language (not the OP's), given the cases you > reject, is "one or more sequences of letters separated by > space*-ampersand-space*", and that is actually one of the easiest kinds > of regex to write: "[A-Z]+( *& *[A-Z]+)*". One of the easiest kind of regex to write incorrectly: py> re.match("[A-Z]+( *& *[A-Z]+)*", "A----") <_sre.SRE_Match object at 0xb7bf4aa0> It doesn't even get the "all uppercase" part of the specification: py> re.match("[A-Z]+( *& *[A-Z]+)*", "Azzz") <_sre.SRE_Match object at 0xb7bf4aa0> You failed to anchor the string at the beginning and end of the string, an easy mistake to make, but that's the point. It's easy to make mistakes with regexes because the syntax is so overly terse and unforgiving. But I think I just learned something important today. I learned that's it's not actually regexes that I dislike, it's regex culture that I dislike. What I learned from this thread: - Nobody could possibly want to support non-ASCII text. (Apart from the approximately 6.5 billion people in the world that don't speak English of course, an utterly insignificant majority.) - Data validity doesn't matter, because there's no possible way that you might accidentally scrape data from the wrong part of a HTML file and end up with junk input. - Even if you do somehow end up with junk, there couldn't possibly be any real consequences to that. - It doesn't matter if you match too much, or to little, that just means the specs are too pedantic. Hence the famous quote: Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems. It's not really regexes that are the problem. > However, your spec is wrong: How can you say that? It's *my* spec, I can specify anything I want. >> - Leading or trailing spaces, or spaces not surrounding an ampersand, >> must not match: "AAA BBB" must be rejected. > > The *very first* item in OP's list of good outputs is 'PHYSICAL FITNESS > CONSULTANTS & TRAINERS'. That's very nice, but irrelevant. I'm not talking about the OP's outputs. I'm giving my own. -- Steven From steve at pearwood.info Thu May 5 14:14:05 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 06 May 2016 04:14:05 +1000 Subject: Whittle it on down References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> Message-ID: <572b8d6f$0$1602$c3e8da3$5496439d@news.astraweb.com> On Fri, 6 May 2016 03:49 am, Jussi Piitulainen wrote: > Steven D'Aprano writes: > >> I get something like this: >> >> r"(^[A-Z]+$)|(^([A-Z]+[ ]*\&[ ]*[A-Z]+)+$)" >> >> >> but it fails on strings like "AA & A & A". What am I doing wrong? > > It cannot split the string as (LETTERS & LETTERS)(LETTERS & LETTERS) > when the middle part is just one LETTER. That's something of a > misanalysis anyway. I notice that the correct pattern has already been > posted at least thrice and you have acknowledged one of them. Thrice? I've seen Peter's response (he made the trivial and obvious simplification of just using A instead of [A-Z], but that was easy to understand), and Random832 almost got it, missing only that you need to match the entire string, not just a substring. If there was a third response, I missed it. > But I think you are also trying to do too much with a single regex. A > more promising start is to think of the whole string as "parts" joined > with "glue", then split with a glue pattern and test the parts: > > import re > glue = re.compile(" *& *| +") > keep, drop = [], [] > for datum in data: > items = glue.split(datum) > if all(map(str.isupper, items)): > keep.append(datum) > else: > drop.append(datum) Ah, the penny drops! For a while I thought you were suggesting using this to assemble a regex, and it just wasn't making sense to me. Then I realised you were using this as a matcher: feed in the list of strings, and it splits it into strings to keep and strings to discard. Nicely done, that is a good technique to remember. Thanks for the analysis! -- Steven From jussi.piitulainen at helsinki.fi Thu May 5 14:27:05 2016 From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen) Date: Thu, 05 May 2016 21:27:05 +0300 Subject: Whittle it on down References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> <572b8d6f$0$1602$c3e8da3$5496439d@news.astraweb.com> Message-ID: Steven D'Aprano writes: > On Fri, 6 May 2016 03:49 am, Jussi Piitulainen wrote: > >> Steven D'Aprano writes: >> >>> I get something like this: >>> >>> r"(^[A-Z]+$)|(^([A-Z]+[ ]*\&[ ]*[A-Z]+)+$)" >>> >>> >>> but it fails on strings like "AA & A & A". What am I doing wrong? >> >> It cannot split the string as (LETTERS & LETTERS)(LETTERS & LETTERS) >> when the middle part is just one LETTER. That's something of a >> misanalysis anyway. I notice that the correct pattern has already been >> posted at least thrice and you have acknowledged one of them. > > Thrice? I've seen Peter's response (he made the trivial and obvious > simplification of just using A instead of [A-Z], but that was easy to > understand), and Random832 almost got it, missing only that you need to > match the entire string, not just a substring. If there was a third > response, I missed it. I think I saw another. I may be mistaken. Random832's pattern is fine. You need to use re.fullmatch with it. . . From fabiofz at gmail.com Thu May 5 14:46:53 2016 From: fabiofz at gmail.com (Fabio Zadrozny) Date: Thu, 5 May 2016 15:46:53 -0300 Subject: PyDev 5.0.0 Released Message-ID: PyDev 5.0.0 Released Release Highlights: ------------------------------- * **Important** PyDev now requires Java 8. * PyDev 4.5.5 is the last release supporting Java 7. * See: http://www.pydev.org/update_sites/index.html for the update site of older versions of PyDev. * See: the **PyDev does not appear after install** section on http://www.pydev.org/download.html for help on using a Java 8 vm in Eclipse. * PyUnit view now persists its state across restarts. * Fixed issue in super() code completion. * PyDev.Debugger updated to the latest version. * No longer showing un-needed shell on Linux on startup when showing donation dialog. * Fixed pyedit_wrap_expression to avoid halt of the IDE on Ctrl+1 -> Wrap expression. What is PyDev? --------------------------- PyDev is an open-source Python IDE on top of Eclipse for Python, Jython and IronPython development. It comes with goodies such as code completion, syntax highlighting, syntax analysis, code analysis, refactor, debug, interactive console, etc. Details on PyDev: http://pydev.org Details on its development: http://pydev.blogspot.com What is LiClipse? --------------------------- LiClipse is a PyDev standalone with goodies such as support for Multiple cursors, theming, TextMate bundles and a number of other languages such as Django Templates, Jinja2, Kivy Language, Mako Templates, Html, Javascript, etc. It's also a commercial counterpart which helps supporting the development of PyDev. Details on LiClipse: http://www.liclipse.com/ Cheers, -- Fabio Zadrozny ------------------------------------------------------ Software Developer LiClipse http://www.liclipse.com PyDev - Python Development Environment for Eclipse http://pydev.org http://pydev.blogspot.com PyVmMonitor - Python Profiler http://www.pyvmmonitor.com/ From random832 at fastmail.com Thu May 5 14:52:05 2016 From: random832 at fastmail.com (Random832) Date: Thu, 05 May 2016 14:52:05 -0400 Subject: Whittle it on down In-Reply-To: <572b8aee$0$1589$c3e8da3$5496439d@news.astraweb.com> References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> <1462454499.2962191.598999745.40BB8A1E@webmail.messagingengine.com> <572b8aee$0$1589$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1462474325.3044031.599340073.61F19F49@webmail.messagingengine.com> On Thu, May 5, 2016, at 14:03, Steven D'Aprano wrote: > You failed to anchor the string at the beginning and end of the string, > an easy mistake to make, but that's the point. I don't think anchoring is properly a concern of the regex itself - .match is anchored implicitly at the beginning, and one could easily imagine an API that implicitly anchors at the end - or you can simply check that the match length == the string length. > - Data validity doesn't matter, because there's no possible way that you > might accidentally scrape data from the wrong part of a HTML file and end > up with junk input. If you've scraped data from the wrong part of the file, then nothing you do to your regex can prevent the junk input from coincidentally matching the input format. From random832 at fastmail.com Thu May 5 14:54:56 2016 From: random832 at fastmail.com (Random832) Date: Thu, 05 May 2016 14:54:56 -0400 Subject: Whittle it on down In-Reply-To: References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> <572b8d6f$0$1602$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1462474496.3044412.599343033.5A3676DD@webmail.messagingengine.com> On Thu, May 5, 2016, at 14:27, Jussi Piitulainen wrote: > Random832's pattern is fine. You need to use re.fullmatch with it. Heh, in my previous post I said "and one could easily imagine an API that implicitly anchors at the end". So easy to imagine it turns out that someone already did, as it turns out. Batteries included indeed. From me+python at ixokai.io Thu May 5 14:55:11 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Thu, 05 May 2016 11:55:11 -0700 Subject: Whittle it on down In-Reply-To: <572b864e$0$1622$c3e8da3$5496439d@news.astraweb.com> References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> <1462455144.93995.599007201.4350517C@webmail.messagingengine.com> <572b864e$0$1622$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1462474511.158594.599310737.0972E5A5@webmail.messagingengine.com> On Thu, May 5, 2016, at 10:43 AM, Steven D'Aprano wrote: > On Thu, 5 May 2016 11:32 pm, Stephen Hansen wrote: > > > On Thu, May 5, 2016, at 12:36 AM, Steven D'Aprano wrote: > >> Oh, a further thought... > >> > >> On Thursday 05 May 2016 16:46, Stephen Hansen wrote: > >> > I don't even care about faster: Its overly complicated. Sometimes a > >> > regular expression really is the clearest way to solve a problem. > >> > >> Putting non-ASCII letters aside for the moment, how would you match these > >> specs as a regular expression? > > > > I don't know, but mostly because I wouldn't even try. > > Really? Peter Otten seems to have found a solution, and Random832 almost > found it too. > > > > The requirements > > are over-specified. If you look at the OP's data (and based on previous > > conversation), he's doing web scraping and trying to pull out good data. > > I'm not talking about the OP's data. I'm talking about *my* requirements. > > I thought that this was a friendly discussion about regexes, but perhaps > I > was mistaken. Because I sure am feeling a lot of hostility to the ideas > that regexes are not necessarily the only way to solve this, and that > data > validation is a good thing. Umm, what? Hostility? I have no idea where you're getting that. I didn't say that regexs are the only way to solve problems; in fact they're something I avoid using in most cases. In the OP's case, though, I did say I thought was a natural fit. Usually, I'd go for startswith/endswith, "in", slicing and such string primitives before I go for a regular expression. "Find all upper cased phrases that may have &'s in them" is something just specific enough that the built in string primitives are awkward tools. In my experience, most of the problems with regexes is people think they're the hammer and every problem is a nail: and then they get into ever more convoluted expressions that become brittle. More specific in a regular expression is not, necessarily, a virtue. In fact its exactly the opposite a lot of times. > > There's no absolutely perfect way to do that because the system he's > > scraping isn't meant for data processing. The data isn't cleanly > > articulated. > > Right. Which makes it *more*, not less, important to be sure that your > regex > doesn't match too much, because your data is likely to be contaminated by > junk strings that don't belong in the data and shouldn't be accepted. > I've > done enough web scraping to realise just how easy it is to start grabbing > data from the wrong part of the file. I have nothing against data validation: I don't think it belongs in regular expressions, though. That can be a step done afterwards. > > Instead, he wants a heuristic to pull out what look like section titles. > > Good for him. I asked a different question. Does my question not count? Sure it counts, but I don't want to engage in your theoretical exercise. That's not being hostile, that's me not wanting to think about a complex set of constraints for a regular expression for purely intellectual reasons. > I was trying to teach DFS a generic programming technique, not solve his > stupid web scraping problem for him. What happens next time when he's > trying to filter a list of floats, or Widgets? Should he convert them to > strings so he can use a regex to match them, or should he learn about > general filtering techniques? Come on. This is a bit presumptuous, don't you think? > > This translates naturally into a simple regular expression: an uppercase > > string with spaces and &'s. Now, that expression doesn't 100% encode > > every detail of that rule-- it allows both Q&A and Q & A-- but on my own > > looking at the data, I suspect its good enough. The titles are clearly > > separate from the other data scraped by their being upper cased. We just > > need to expand our allowed character range into spaces and &'s. > > > > Nothing in the OP's request demands the kind of rigorous matching that > > your scenario does. Its a practical problem with a simple, practical > > answer. > > Yes, and that practical answer needs to reject: > > - the empty string, because it is easy to mistakenly get empty strings > when > scraping data, especially if you post-process the data; > > - strings that are all spaces, because " " cannot possibly be a > title; > > - strings that are all ampersands, because "&&&&&" is not a title, and it > almost surely indicates that your scraping has gone wrong and you're > reading junk from somewhere; > > - even leading and trailing spaces are suspect: " FOO " doesn't match > any > of the examples given, and it seems unlikely to be a title. Presumably > the > strings have already been filtered or post-processed to have leading and > trailing spaces removed, in which case " FOO " reveals a bug. We're going to have to agree to disagree. I find all of that unnecessary. Any validation can be easily done before or after matching, you don't need to over-complicate the regular expression itself. The urge to find an ever more perfect regular expression that manages to encapsulate what is precisely correct and what is not leads itself to over-complicated expressions. And regular expressions are ugly. I'd rather keep them simple and straight-forward and deal with the rest in Python. -- Stephen Hansen m e @ i x o k a i . i o From me+python at ixokai.io Thu May 5 14:56:52 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Thu, 05 May 2016 11:56:52 -0700 Subject: Whittle it on down In-Reply-To: References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> Message-ID: <1462474612.159192.599345593.57C2CB88@webmail.messagingengine.com> On Thu, May 5, 2016, at 05:31 AM, DFS wrote: > You are out of your mind. Whoa, now. I might disagree with Steven D'Aprano about how to approach this problem, but there's no need to be rude. Everyone's trying to help you, after all. -- Stephen Hansen m e @ i x o k a i . i o From tjreedy at udel.edu Thu May 5 14:59:32 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 5 May 2016 14:59:32 -0400 Subject: Python is an Equal Opportunity Programming Language Message-ID: https://motherboard.vice.com/blog/python-is-an-equal-opportunity-programming-language from an 'Intel? Software Evangelist' -- Terry Jan Reedy From me+python at ixokai.io Thu May 5 15:09:42 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Thu, 05 May 2016 12:09:42 -0700 Subject: Whittle it on down In-Reply-To: <572b8aee$0$1589$c3e8da3$5496439d@news.astraweb.com> References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> <1462454499.2962191.598999745.40BB8A1E@webmail.messagingengine.com> <572b8aee$0$1589$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1462475382.161356.599349169.50845B61@webmail.messagingengine.com> On Thu, May 5, 2016, at 11:03 AM, Steven D'Aprano wrote: > - Nobody could possibly want to support non-ASCII text. (Apart from the > approximately 6.5 billion people in the world that don't speak English of > course, an utterly insignificant majority.) Oh, I'd absolutely want to support non-ASCII text. If I have unicode input, though, I unfortunately have to rely on https://pypi.python.org/pypi/regex as 're' doesn't support matching on character properties. I keep hoping it'll replace "re", then we could do: pattern = regex.compile(ru"^\p{Lu}\s&]+$") where \p{property} matches against character properties in the unicode database. > - Data validity doesn't matter, because there's no possible way that you > might accidentally scrape data from the wrong part of a HTML file and end > up with junk input. Um, no one said that. I was arguing that the *regular expression* doesn't need to be responsible for validation. > - Even if you do somehow end up with junk, there couldn't possibly be any > real consequences to that. No one said that either... > - It doesn't matter if you match too much, or to little, that just means > the > specs are too pedantic. Or that... -- Stephen Hansen m e @ i x o k a i . i o From ethan at stoneleaf.us Thu May 5 15:52:21 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 05 May 2016 12:52:21 -0700 Subject: Comparing Python enums to Java, was: How much sanity checking is required for function inputs? In-Reply-To: References: Message-ID: <572BA475.10307@stoneleaf.us> On 04/24/2016 08:20 AM, Ian Kelly wrote: > On Sun, Apr 24, 2016 at 1:20 AM, Ethan Furman wrote: >> What fun things can Java enums do? > > Everything that Python enums can do, plus: > >--> Planet.EARTH.value > (5.976e+24, 6378140.0) >--> Planet.EARTH.surface_gravity > 9.802652743337129 > > This is incredibly useful, but it has a flaw: the value of each member > of the enum is just the tuple of its arguments. Suppose we added a > value for COUNTER_EARTH describing a hypothetical planet with the same > mass and radius existing on the other side of the sun. [1] Then: > >--> Planet.EARTH is Planet.COUNTER_EARTH > True If using Python 3 and aenum 1.4.1+, you can do --> class Planet(Enum, settings=NoAlias, init='mass radius'): ... MERCURY = (3.303e+23, 2.4397e6) ... VENUS = (4.869e+24, 6.0518e6) ... EARTH = (5.976e+24, 6.37814e6) ... COUNTER_EARTH = EARTH ... @property ... def surface_gravity(self): ... # universal gravitational constant (m3 kg-1 s-2) ... G = 6.67300E-11 ... return G * self.mass / (self.radius * self.radius) ... --> Planet.EARTH.value (5.976e+24, 6378140.0) --> Planet.EARTH.surface_gravity 9.802652743337129 --> Planet.COUNTER_EARTH.value (5.976e+24, 6378140.0) --> Planet.COUNTER_EARTH.surface_gravity 9.802652743337129 >>> Planet.EARTH is Planet.COUNTER_EARTH False > * Speaking of AutoNumber, since Java enums don't have the > instance/value distinction, they effectively do this implicitly, only > without generating a bunch of ints that are entirely irrelevant to > your enum type. With Python enums you have to follow a somewhat arcane > recipe to avoid specifying values, which just generates some values > and then hides them away. And it also breaks the Enum alias feature: > >--> class Color(AutoNumber): > ... red = default = () # not an alias! > ... blue = () > ... Another thing you could do here: --> class Color(Enum, settings=AutoNumber): ... red ... default = red ... blue ... --> list(Color) [, ] --> Color.default is Color.red True -- ~Ethan~ From eryksun at gmail.com Thu May 5 15:56:28 2016 From: eryksun at gmail.com (eryk sun) Date: Thu, 5 May 2016 14:56:28 -0500 Subject: Ctypes c_void_p overflow In-Reply-To: <7492a519afee44b1875f7ec51edf44a7@activenetwerx.com> References: <7492a519afee44b1875f7ec51edf44a7@activenetwerx.com> Message-ID: On Thu, May 5, 2016 at 10:42 AM, Joseph L. Casale wrote: > I have CDLL function I use to get a pointer, several other functions happily accept this > pointer which is really a long when passed to ctypes.c_void_p. However, only one with > same type def in the prototype overflows. Docs suggest c_void_p takes an int but that > is not what the first call returns, nor what all but one function happily accept? What you're describing isn't clear to me, so I'll describe the general case of handling pointers with ctypes functions. If a function returns a pointer, you must set the function's restype to a pointer type since the default c_int restype truncates the upper half of a 64-bit pointer. Generally you also have to do the same for pointer parameters in argtypes. Otherwise integer arguments are converted to C int values. Note that when restype is set to c_void_p, the result gets converted to a Python integer (or None for a NULL result). If you pass this result back as a ctypes function argument, the function must have the parameter set to c_void_p in argtypes. If argtypes isn't set, the default integer conversion may truncate the pointer value. This problem won't occur on a 32-bit platform, so there's a lot of carelessly written ctypes code that makes this mistake. Simple types are also automatically converted when accessed as a field of a struct or union or as an index of an array or pointer. To avoid this, you can use a subclass of the type, since ctypes won't automatically convert subclasses of simple types. I generally avoid c_void_p because its lenient from_param method (called to convert arguments) doesn't provide much type safety. If a bug causes an incorrect argument to be passed, I prefer getting an immediate ctypes.ArgumentError rather than a segfault or data corruption. For example, when a C API returns a void pointer as a handle for an opaque structure or object, I prefer to handle it as a pointer to an empty Structure subclass, as follows: class _ContosoHandle(ctypes.Structure): pass ContosoHandle = ctypes.POINTER(_ContosoHandle) lib.CreateContoso.restype = ContosoHandle lib.DestroyContoso.argtypes = (ContosoHandle,) ctypes will raise an ArgumentError if DestroyContoso is called with arguments such as 123456789 or "Crash Me". From mymyxin at gmail.com Thu May 5 16:47:05 2016 From: mymyxin at gmail.com (mymyxin at gmail.com) Date: Thu, 5 May 2016 13:47:05 -0700 (PDT) Subject: python, ctypes and GetIconInfo issue Message-ID: Hello, I try to make the GetIconInfo function work, but I can't figure out what I'm doing wrong. >From the MSDN documentation the function is https://msdn.microsoft.com/en-us/library/windows/desktop/ms648070%28v=vs.85%29.aspx # BOOL WINAPI GetIconInfo( # _In_ HICON hIcon, # _Out_ PICONINFO piconinfo # ); which I defined as GetIconInfo = windll.user32.GetIconInfo GetIconInfo.argtypes = [HICON, POINTER(ICONINFO)] GetIconInfo.restype = BOOL GetIconInfo.errcheck = ErrorIfZero The structure piconinfo is described as https://msdn.microsoft.com/en-us/library/windows/desktop/ms648052%28v=vs.85%29.aspx # typedef struct _ICONINFO { # BOOL fIcon; # DWORD xHotspot; # DWORD yHotspot; # HBITMAP hbmMask; # HBITMAP hbmColor; # } ICONINFO, *PICONINFO; my implementation is class ICONINFO(Structure): __fields__ = [ ('fIcon', BOOL), ('xHotspot', DWORD), ('yHotspot', DWORD), ('hbmMask', HBITMAP), ('hbmColor', HBITMAP), ] #### not part of the problem but needed to get the icon handle hicon = ImageList_GetIcon(def_il_handle,1,ILD_NORMAL) print hicon #### As the documentation states, the function run successful if return code is none zero. Well I get 1 returned but as soon as I try to access a class member the program crashes. iconinfo = ICONINFO() lres = GetIconInfo(hicon, pointer(iconinfo)) print lres print '{0}'.format(sizeof(iconinfo)) # <- crash If I comment the print of sizeof... the program keeps running but if I call the same code a second time then it crashes at GetIconInfo(hicon, ...) So it looks like I'm doing something terribly wrong but don't see it. Can someone shed some light on it? Thank you Hubert From jcasale at activenetwerx.com Thu May 5 17:20:08 2016 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Thu, 5 May 2016 21:20:08 +0000 Subject: Ctypes c_void_p overflow In-Reply-To: References: <7492a519afee44b1875f7ec51edf44a7@activenetwerx.com> Message-ID: <9387a5b591d843dca78cfd90717932cf@activenetwerx.com> > I generally avoid c_void_p because its lenient from_param method > (called to convert arguments) doesn't provide much type safety. If a > bug causes an incorrect argument to be passed, I prefer getting an > immediate ctypes.ArgumentError rather than a segfault or data > corruption. For example, when a C API returns a void pointer as a > handle for an opaque structure or object, I prefer to handle it as a > pointer to an empty Structure subclass, as follows: > > class _ContosoHandle(ctypes.Structure): > pass > > ContosoHandle = ctypes.POINTER(_ContosoHandle) > > lib.CreateContoso.restype = ContosoHandle > lib.DestroyContoso.argtypes = (ContosoHandle,) > > ctypes will raise an ArgumentError if DestroyContoso is called with > arguments such as 123456789 or "Crash Me". After typing up a response with all the detail, your reply helped me see the error. Thank you so much for all that detail, it was very much appreciated! jlc From nospam at dfs.com Thu May 5 17:36:11 2016 From: nospam at dfs.com (DFS) Date: Thu, 5 May 2016 17:36:11 -0400 Subject: Whittle it on down In-Reply-To: <572b88c5$0$1601$c3e8da3$5496439d@news.astraweb.com> References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <572b88c5$0$1601$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5/5/2016 1:54 PM, Steven D'Aprano wrote: > On Thu, 5 May 2016 10:31 pm, DFS wrote: > >> You are out of your mind. > > That's twice you've tried to put me down, first by dismissing my comments > about text processing with "Linguist much", and now an outright insult. The > first time I laughed it off and made a joke about it. I won't do that > again. > > You asked whether it was better to extract the matching strings into a new > list, or remove them in place in the existing list. I not only showed you > how to do both, but I tried to give you the mental tools to understand when > you should pick one answer over the other. And your response is to insult > me and question my sanity. > > Well, DFS, I might be crazy, but I'm not stupid. If that's really how you > feel about my answers, I won't make the mistake of wasting my time > answering your questions in the future. > > Over to you now. heh! Relax, pal. I was just trying to be funny - no insult intended either time, of course. Look for similar responses from me in the future. Usenet brings out the smart-aleck in me. Actually, you should've accepted the 'Linguist much?' as a compliment, because I seriously thought you were. But you ARE out of your mind if you prefer that convoluted "function" method over a simple 1-line regex method (as per S. Hansen). def isupperalpha(string): return string.isalpha() and string.isupper() def check(string): if isupperalpha(string): return True parts = string.split("&") if len(parts) < 2: return False parts[0] = parts[0].rstrip(" ") parts[-1] = parts[-1].lstrip(" ") for i in range(1, len(parts)-1): parts[i] = parts[i].strip(" ") return all(isupperalpha(part) for part in parts) I'm sure it does the job well, but that style brings back [bad] memories of the VBA I used to write. I expected something very concise and 'pythonic' (which I'm learning is everyone's favorite mantra here in python-land). Anyway, I appreciate ALL replies to my queries. So thank you for taking the time. Whenever I'm able, I'll try to contribute to clp as well. From nospam at dfs.com Thu May 5 17:45:54 2016 From: nospam at dfs.com (DFS) Date: Thu, 5 May 2016 17:45:54 -0400 Subject: Whittle it on down In-Reply-To: References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462474612.159192.599345593.57C2CB88@webmail.messagingengine.com> Message-ID: On 5/5/2016 2:56 PM, Stephen Hansen wrote: > On Thu, May 5, 2016, at 05:31 AM, DFS wrote: >> You are out of your mind. > > Whoa, now. I might disagree with Steven D'Aprano about how to approach > this problem, but there's no need to be rude. Seriously not trying to be rude - more smart-alecky than anything. Hope D'Aprano doesn't stay butthurt... > Everyone's trying to help you, after all. Yes, and I do appreciate it. I've only been working with python for about a month, but I feel like I'm making good progress. clp is a great resource, and I'll be hanging around for a long time, and will contribute when possible. Thanks for your help. From nospam at dfs.com Thu May 5 19:31:33 2016 From: nospam at dfs.com (DFS) Date: Thu, 5 May 2016 19:31:33 -0400 Subject: Whittle it on down In-Reply-To: References: <1462426755.15465.598690257.42990546@webmail.messagingengine.com> Message-ID: On 5/5/2016 1:39 AM, Stephen Hansen wrote: > Given: > >>>> input = [u'Espa\xf1ol', 'Health & Fitness Clubs (36)', 'Health Clubs & Gymnasiums (42)', 'Health Fitness Clubs', 'Name', 'Atlanta city guide', 'edit address', 'Tweet', 'PHYSICAL FITNESS CONSULTANTS & TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', 'www.custombuiltpt.com/', 'RACQUETBALL COURTS PRIVATE', 'www.lafitness.com', 'GYMNASIUMS', 'HEALTH & FITNESS CLUBS', 'www.lafitness.com', 'HEALTH & FITNESS CLUBS', 'www.lafitness.com', 'PERSONAL FITNESS TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'EXERCISE & PHYSICAL FITNESS PROGRAMS', 'FITNESS CENTERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', 'PERSONAL FITNESS TRAINERS', '5', '4', '3', '2', '1', 'Yellow Pages', 'About Us', 'Contact Us', 'Support', 'Terms of Use', 'Privacy Policy', 'Advertise With Us', 'Add/Update Listing', 'Business Profile Login', 'F.A.Q.'] > > Then: > >>>> pattern = re.compile(r"^[A-Z\s&]+$") >>>> output = [x for x in list if pattern.match(x)] >>>> output > ['PHYSICAL FITNESS CONSULTANTS & TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', > 'HEALTH CLUBS & GYMNASIUMS', 'RACQUETBALL COURTS PRIVATE', 'GYMNASIUMS', > 'HEALTH & FITNESS CLUBS', 'HEALTH & FITNESS CLUBS', 'PERSONAL FITNESS > TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'EXERCISE & PHYSICAL FITNESS > PROGRAMS', 'FITNESS CENTERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS > & GYMNASIUMS', 'PERSONAL FITNESS TRAINERS'] Should've looked earlier. Their master list of categories http://www.usdirectory.com/cat/g0 shows a few commas, a bunch of dashes, and the ampersands we talked about. "OFFICE SERVICES, SUPPLIES & EQUIPMENT" gets removed because of the comma. "AUTOMOBILE - DEALERS" gets removed because of the dash. I updated your regex and it seems to have fixed it. orig: (r"^[A-Z\s&]+$") new : (r"^[A-Z\s&,-]+$") Thanks again. From eryksun at gmail.com Thu May 5 20:09:21 2016 From: eryksun at gmail.com (eryk sun) Date: Thu, 5 May 2016 19:09:21 -0500 Subject: python, ctypes and GetIconInfo issue In-Reply-To: References: Message-ID: On Thu, May 5, 2016 at 3:47 PM, wrote: > > I try to make the GetIconInfo function work, but I can't figure out > what I'm doing wrong. > > From the MSDN documentation the function is > > https://msdn.microsoft.com/en-us/library/windows/desktop/ms648070%28v=vs.85%29.aspx > > # BOOL WINAPI GetIconInfo( > # _In_ HICON hIcon, > # _Out_ PICONINFO piconinfo > # ); > > which I defined as > > GetIconInfo = windll.user32.GetIconInfo > GetIconInfo.argtypes = [HICON, POINTER(ICONINFO)] > GetIconInfo.restype = BOOL > GetIconInfo.errcheck = ErrorIfZero Please avoid windll. It caches the loaded library, which in turn caches function pointers. So all packages that use windll.user32 are potentially stepping on each others' toes with mutually incompatible function prototypes. It also doesn't allow configuring use_last_error=True to enable ctypes.get_last_error() for WinAPI function calls. > The structure piconinfo is described as > https://msdn.microsoft.com/en-us/library/windows/desktop/ms648052%28v=vs.85%29.aspx > > # typedef struct _ICONINFO { > # BOOL fIcon; > # DWORD xHotspot; > # DWORD yHotspot; > # HBITMAP hbmMask; > # HBITMAP hbmColor; > # } ICONINFO, *PICONINFO; > > my implementation is > > class ICONINFO(Structure): > __fields__ = [ > ('fIcon', BOOL), > ('xHotspot', DWORD), > ('yHotspot', DWORD), > ('hbmMask', HBITMAP), > ('hbmColor', HBITMAP), > ] The attribute name is "_fields_", not "__fields__", so you haven't actually defined any fields and sizeof(ICONINFO) is 0. When you pass this empty struct to GetIconInfo, it potentially overwrites and corrupts existing data on the heap that can lead to a crash later on. Here's the setup I created to test GetIconInfo and GetIconInfoEx. Maybe you can reuse some of this code, but if you're using XP this won't work as written because GetIconInfoEx was added in Vista. Note the use of a __del__ finalizer to call DeleteObject on the bitmaps. Otherwise, in a real application, calling GetIconInfo would leak memory. Using __del__ is convenient, but note that you can't reuse an instance without manually calling DeleteObject on the bitmaps. import ctypes from ctypes import wintypes kernel32 = ctypes.WinDLL('kernel32', use_last_error=True) user32 = ctypes.WinDLL('user32', use_last_error=True) gdi32 = ctypes.WinDLL('gdi32') MAX_PATH = 260 IMAGE_ICON = 1 class ICONINFO_BASE(ctypes.Structure): def __del__(self, gdi32=gdi32): if self.hbmMask: gdi32.DeleteObject(self.hbmMask) self.hbmMask = None if self.hbmColor: gdi32.DeleteObject(self.hbmColor) self.hbmColor = None class ICONINFO(ICONINFO_BASE): _fields_ = (('fIcon', wintypes.BOOL), ('xHotspot', wintypes.DWORD), ('yHotspot', wintypes.DWORD), ('hbmMask', wintypes.HBITMAP), ('hbmColor', wintypes.HBITMAP)) class ICONINFOEX(ICONINFO_BASE): _fields_ = (('cbSize', wintypes.DWORD), ('fIcon', wintypes.BOOL), ('xHotspot', wintypes.DWORD), ('yHotspot', wintypes.DWORD), ('hbmMask', wintypes.HBITMAP), ('hbmColor', wintypes.HBITMAP), ('wResID', wintypes.WORD), ('szModName', wintypes.WCHAR * MAX_PATH), ('szResName', wintypes.WCHAR * MAX_PATH)) def __init__(self, *args, **kwds): super(ICONINFOEX, self).__init__(*args, **kwds) self.cbSize = ctypes.sizeof(self) PICONINFO = ctypes.POINTER(ICONINFO) PICONINFOEX = ctypes.POINTER(ICONINFOEX) def check_bool(result, func, args): if not result: raise ctypes.WinError(ctypes.get_last_error()) return args kernel32.GetModuleHandleW.errcheck = check_bool kernel32.GetModuleHandleW.restype = wintypes.HMODULE kernel32.GetModuleHandleW.argtypes = ( wintypes.LPCWSTR,) # _In_opt_ lpModuleName # DeleteObject doesn't call SetLastError gdi32.DeleteObject.restype = wintypes.BOOL gdi32.DeleteObject.argtypes = ( wintypes.HGDIOBJ,) # _In_ hObject user32.LoadImageW.errcheck = check_bool user32.LoadImageW.restype = wintypes.HANDLE user32.LoadImageW.argtypes = ( wintypes.HINSTANCE, # _In_opt_ hinst wintypes.LPCWSTR, # _In_ lpszName wintypes.UINT, # _In_ uType ctypes.c_int, # _In_ cxDesired ctypes.c_int, # _In_ cyDesired wintypes.UINT,) # _In_ fuLoad user32.DestroyIcon.errcheck = check_bool user32.DestroyIcon.restype = wintypes.BOOL user32.DestroyIcon.argtypes = ( wintypes.HICON,) # _In_ hIcon user32.GetIconInfo.errcheck = check_bool user32.GetIconInfo.restype = wintypes.BOOL user32.GetIconInfo.argtypes = ( wintypes.HICON, # _In_ hIcon PICONINFO,) # _Out_ piconinfo # requires Vista+ user32.GetIconInfoExW.errcheck = check_bool user32.GetIconInfoExW.restype = wintypes.BOOL user32.GetIconInfoExW.argtypes = ( wintypes.HICON, # _In_ hIcon PICONINFOEX,) # _Out_ piconinfoex if __name__ == '__main__': hMain = kernel32.GetModuleHandleW(None) hIcon = user32.LoadImageW(hMain, wintypes.LPCWSTR(1), IMAGE_ICON, 0, 0, 0) try: info = ICONINFOEX() user32.GetIconInfoExW(hIcon, ctypes.byref(info)) print('fIcon : %d' % info.fIcon) print('wResID : %d' % info.wResID) print('szModName: %s' % info.szModName) finally: user32.DestroyIcon(hIcon) The __main__ test outputs the following for me in 3.5 and 2.7: fIcon : 1 wResID : 1 szModName: C:\Program Files\Python35\python.exe fIcon : 1 wResID : 1 szModName: C:\Program Files\Python27\python.exe From steve at pearwood.info Thu May 5 20:57:21 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 06 May 2016 10:57:21 +1000 Subject: Whittle it on down References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> <572b8d6f$0$1602$c3e8da3$5496439d@news.astraweb.com> Message-ID: <572bebf3$0$1585$c3e8da3$5496439d@news.astraweb.com> On Fri, 6 May 2016 04:27 am, Jussi Piitulainen wrote: > Random832's pattern is fine. You need to use re.fullmatch with it. py> re.fullmatch Traceback (most recent call last): File "", line 1, in AttributeError: 'module' object has no attribute 'fullmatch' -- Steven From christopher_reimer at icloud.com Thu May 5 21:26:23 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Thu, 05 May 2016 18:26:23 -0700 Subject: Pylint prefers list comprehension over filter... Message-ID: <572BF2BF.6000000@icloud.com> Greetings, Below is the code that I mentioned in an earlier thread. string = "Whiskey Tango Foxtrot" ''.join(list(filter(str.isupper, string))) 'WTF' That works fine and dandy. Except Pylint doesn't like it. According to this link, list comprehensions have replaced filters and the Pylint warning can be disabled. http://stackoverflow.com/questions/3569134/why-doesnt-pylint-like-built-in-functions Here's the replacement code using list comprehension: ''.join([x for x in string if x.isupper()]) Which is one is correct (Pythonic)? Or does it matter? Thank you, Chris R. From rosuav at gmail.com Thu May 5 21:33:39 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 6 May 2016 11:33:39 +1000 Subject: Pylint prefers list comprehension over filter... In-Reply-To: <572BF2BF.6000000@icloud.com> References: <572BF2BF.6000000@icloud.com> Message-ID: On Fri, May 6, 2016 at 11:26 AM, Christopher Reimer wrote: > Below is the code that I mentioned in an earlier thread. > > string = "Whiskey Tango Foxtrot" > ''.join(list(filter(str.isupper, string))) > > 'WTF' > > That works fine and dandy. Except Pylint doesn't like it. According to this > link, list comprehensions have replaced filters and the Pylint warning can > be disabled. > > http://stackoverflow.com/questions/3569134/why-doesnt-pylint-like-built-in-functions > > Here's the replacement code using list comprehension: > > ''.join([x for x in string if x.isupper()]) > > Which is one is correct (Pythonic)? Or does it matter? Nothing wrong with filter. Since join() is going to iterate over its argument anyway, you don't need the list() call, you can remove that, but you don't have to go for comprehensions: ''.join(filter(str.isupper, string)) Rule of thumb: If the function already exists, use filter or map. If you would be using filter/map with a lambda function, reach for a comprehension instead. In this case, str.isupper exists, so use it! ChrisA From me at ixokai.io Thu May 5 21:37:11 2016 From: me at ixokai.io (Stephen Hansen) Date: Thu, 05 May 2016 18:37:11 -0700 Subject: Pylint prefers list comprehension over filter... In-Reply-To: <572BF2BF.6000000@icloud.com> References: <572BF2BF.6000000@icloud.com> Message-ID: <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> On Thu, May 5, 2016, at 06:26 PM, Christopher Reimer wrote: > Which is one is correct (Pythonic)? Or does it matter? First, pylint is somewhat opinionated, and its default options shouldn't be taken as gospel. There's no correct: filter is fine. That said, the general consensus is, I believe, that list comprehensions are good, and using them is great. In your case, though, I would not use a list comprehension. I'd use a generator comprehension. It looks almost identical: ''.join(x for x in string if x.isupper()) The difference is, both filter and your list comprehension *build a list* which is not needed, and wasteful. The above skips building a list, instead returning a generator, and join pulls items out of it one at a time as it uses them. No needlessly creating a list only to use it and discard it. -- Stephen Hansen m e @ i x o k a i . i o From dan at tombstonezero.net Thu May 5 22:46:22 2016 From: dan at tombstonezero.net (Dan Sommers) Date: Fri, 6 May 2016 02:46:22 -0000 (UTC) Subject: Pylint prefers list comprehension over filter... References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> Message-ID: On Thu, 05 May 2016 18:37:11 -0700, Stephen Hansen wrote: > ''.join(x for x in string if x.isupper()) > The difference is, both filter and your list comprehension *build a > list* which is not needed, and wasteful. The above skips building a > list, instead returning a generator ... filter used to build a list, but now it doesn't (where "used to" means Python 2.7 and "now" means Python 3.5; I'm too lazy to track down the exact point(s) at which it changed): Python 2.7.11+ (default, Apr 17 2016, 14:00:29) [GCC 5.3.1 20160409] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> filter(lambda x:x+1, [1, 2, 3, 4]) [1, 2, 3, 4] Python 3.5.1+ (default, Apr 17 2016, 16:14:06) [GCC 5.3.1 20160409] on linux Type "help", "copyright", "credits" or "license" for more information. >>> filter(lambda x:x+1, [1, 2, 3, 4]) From torriem at gmail.com Thu May 5 22:49:13 2016 From: torriem at gmail.com (Michael Torrie) Date: Thu, 5 May 2016 20:49:13 -0600 Subject: After a year using Node.js, the prodigal son returns In-Reply-To: <5729b9d8$0$1612$c3e8da3$5496439d@news.astraweb.com> References: <5729b9d8$0$1612$c3e8da3$5496439d@news.astraweb.com> Message-ID: <572C0629.6030509@gmail.com> On 05/04/2016 02:59 AM, Steven D'Aprano wrote: > A year ago, Gavin Vickery decided to move away from Python and give > Javascript with Node.js a try. Twelve months later, he has written about his > experiences: > > > http://geekforbrains.com/post/after-a-year-of-nodejs-in-production Very interesting. Frankly Javascript sounds awful. Even on the front end. From rosuav at gmail.com Thu May 5 22:53:11 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 6 May 2016 12:53:11 +1000 Subject: After a year using Node.js, the prodigal son returns In-Reply-To: <572C0629.6030509@gmail.com> References: <5729b9d8$0$1612$c3e8da3$5496439d@news.astraweb.com> <572C0629.6030509@gmail.com> Message-ID: On Fri, May 6, 2016 at 12:49 PM, Michael Torrie wrote: > On 05/04/2016 02:59 AM, Steven D'Aprano wrote: >> A year ago, Gavin Vickery decided to move away from Python and give >> Javascript with Node.js a try. Twelve months later, he has written about his >> experiences: >> >> >> http://geekforbrains.com/post/after-a-year-of-nodejs-in-production > > Very interesting. Frankly Javascript sounds awful. Even on the front end. https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript JavaScript is terrible. Really, really bad. And because of that, it has the potential to sweep the world. ChrisA From rosuav at gmail.com Thu May 5 22:55:05 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 6 May 2016 12:55:05 +1000 Subject: Pylint prefers list comprehension over filter... In-Reply-To: References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> Message-ID: On Fri, May 6, 2016 at 12:46 PM, Dan Sommers wrote: > filter used to build a list, but now it doesn't (where "used to" means > Python 2.7 and "now" means Python 3.5; I'm too lazy to track down the > exact point(s) at which it changed): > > Python 2.7.11+ (default, Apr 17 2016, 14:00:29) > [GCC 5.3.1 20160409] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> filter(lambda x:x+1, [1, 2, 3, 4]) > [1, 2, 3, 4] > > Python 3.5.1+ (default, Apr 17 2016, 16:14:06) > [GCC 5.3.1 20160409] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> filter(lambda x:x+1, [1, 2, 3, 4]) > Most of these kinds of changes happened in 3.0, where backward-incompatible changes were accepted. A whole bunch of things stopped returning lists and started returning lazy iterables - range, filter/map, dict.keys(), etc - because most of the time, they're iterated over once and then dropped. ChrisA From jladasky at itu.edu Thu May 5 22:56:56 2016 From: jladasky at itu.edu (jladasky at itu.edu) Date: Thu, 5 May 2016 19:56:56 -0700 (PDT) Subject: How to become more motivated to learn Python In-Reply-To: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> References: <1780e532-e316-4d9f-951d-cc9147209e97@googlegroups.com> Message-ID: The best way to increase your motivation to learn Python is: 1. Select a non-trivial problem that you need to solve with programming. 2. Try to write the program you need in any other language (that you don't already know well). 3. Write the program you need in Python. 4. Gaze in astonishment at the time that you could have saved by skipping step 2. From me+python at ixokai.io Thu May 5 22:57:26 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Thu, 05 May 2016 19:57:26 -0700 Subject: Pylint prefers list comprehension over filter... In-Reply-To: References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> Message-ID: <1462503446.248479.599683745.79791DAC@webmail.messagingengine.com> On Thu, May 5, 2016, at 07:46 PM, Dan Sommers wrote: > On Thu, 05 May 2016 18:37:11 -0700, Stephen Hansen wrote: > > > ''.join(x for x in string if x.isupper()) > > > The difference is, both filter and your list comprehension *build a > > list* which is not needed, and wasteful. The above skips building a > > list, instead returning a generator ... > > filter used to build a list, but now it doesn't (where "used to" means > Python 2.7 and "now" means Python 3.5; I'm too lazy to track down the > exact point(s) at which it changed): Oh, didn't know that. Then again the OP was converting the output of filter *into* a list, which wasted a list either way. -- Stephen Hansen m e @ i x o k a i . i o From dan at tombstonezero.net Thu May 5 23:07:23 2016 From: dan at tombstonezero.net (Dan Sommers) Date: Fri, 6 May 2016 03:07:23 -0000 (UTC) Subject: Pylint prefers list comprehension over filter... References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> Message-ID: On Fri, 06 May 2016 02:46:22 +0000, Dan Sommers wrote: > Python 2.7.11+ (default, Apr 17 2016, 14:00:29) > [GCC 5.3.1 20160409] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> filter(lambda x:x+1, [1, 2, 3, 4]) > [1, 2, 3, 4] > > Python 3.5.1+ (default, Apr 17 2016, 16:14:06) > [GCC 5.3.1 20160409] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> filter(lambda x:x+1, [1, 2, 3, 4]) > Muphrey's Law strikes again. That lambda function is obviously a leftover from a call to *map* rather than a call to *filter*, but thanks everyone for not laughing and pointing. From rosuav at gmail.com Thu May 5 23:18:44 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 6 May 2016 13:18:44 +1000 Subject: Pylint prefers list comprehension over filter... In-Reply-To: References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> Message-ID: On Fri, May 6, 2016 at 1:07 PM, Dan Sommers wrote: > On Fri, 06 May 2016 02:46:22 +0000, Dan Sommers wrote: > >> Python 2.7.11+ (default, Apr 17 2016, 14:00:29) >> [GCC 5.3.1 20160409] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> filter(lambda x:x+1, [1, 2, 3, 4]) >> [1, 2, 3, 4] >> >> Python 3.5.1+ (default, Apr 17 2016, 16:14:06) >> [GCC 5.3.1 20160409] on linux >> Type "help", "copyright", "credits" or "license" for more information. >> >>> filter(lambda x:x+1, [1, 2, 3, 4]) >> > > Muphrey's Law strikes again. That lambda function is obviously a > leftover from a call to *map* rather than a call to *filter*, but thanks > everyone for not laughing and pointing. Hey, maybe you wanted to filter out all the -1 results. Maybe you have a search function that returns zero-based offsets, or -1 for "not found". Seems reasonable! And "x+1" is way shorter than "x!=-1", which means by definition that it's better. ChrisA From jussi.piitulainen at helsinki.fi Fri May 6 00:19:03 2016 From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen) Date: Fri, 06 May 2016 07:19:03 +0300 Subject: Whittle it on down References: <572ae25f$0$2821$c3e8da3$76491128@news.astraweb.com> <1462430766.25079.598726825.1B90C7A1@webmail.messagingengine.com> <572af811$0$1608$c3e8da3$5496439d@news.astraweb.com> <572b8d6f$0$1602$c3e8da3$5496439d@news.astraweb.com> <572bebf3$0$1585$c3e8da3$5496439d@news.astraweb.com> Message-ID: Steven D'Aprano writes: > On Fri, 6 May 2016 04:27 am, Jussi Piitulainen wrote: > >> Random832's pattern is fine. You need to use re.fullmatch with it. > > py> re.fullmatch > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'module' object has no attribute 'fullmatch' It's new in version 3.4 (of Python). From lordluke80 at gmail.com Fri May 6 02:56:13 2016 From: lordluke80 at gmail.com (lordluke80 at gmail.com) Date: Thu, 5 May 2016 23:56:13 -0700 (PDT) Subject: python - handling HTTP requests asynchronously Message-ID: Hi everyone, I need to generate a PDF report for each entry of a django queryset. There'll be between between 30k and 40k entries. The PDF is generated through an external API. Since currently is generated on demand, this is handled synchronously via an HTTP request/response. That will be different for this task, since I think I'll use a django management command to loop through the queryset and perform the PDF generation. Which approach should I follow for this task? I thought about 3 possibile solutions, although are technologies that I never used: 1) Celery: assign a task (http request with a different payload) to a worker, then retrieve it once it's done. 2) request-futures: using requests in a non-blocking way. 3) multiprocessing module, with e.g. 10 as workers limit. the goal is to use the API concurrently (e.g. send 10 or 100 http requests simultaneously, depending on how many concurrent requests the API can handle). Anybody here that handled a similar task and can give advices on how to proceed on this? From __peter__ at web.de Fri May 6 03:45:18 2016 From: __peter__ at web.de (Peter Otten) Date: Fri, 06 May 2016 09:45:18 +0200 Subject: Whittle it on down References: <1462426755.15465.598690257.42990546@webmail.messagingengine.com> Message-ID: DFS wrote: > On 5/5/2016 1:39 AM, Stephen Hansen wrote: > >> Given: >> >>>>> input = [u'Espa\xf1ol', 'Health & Fitness Clubs (36)', 'Health Clubs & >>>>> Gymnasiums (42)', 'Health Fitness Clubs', 'Name', 'Atlanta city >>>>> guide', 'edit address', 'Tweet', 'PHYSICAL FITNESS CONSULTANTS & >>>>> TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', >>>>> 'www.custombuiltpt.com/', 'RACQUETBALL COURTS PRIVATE', >>>>> 'www.lafitness.com', 'GYMNASIUMS', 'HEALTH & FITNESS CLUBS', >>>>> 'www.lafitness.com', 'HEALTH & FITNESS CLUBS', 'www.lafitness.com', >>>>> 'PERSONAL FITNESS TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'EXERCISE & >>>>> PHYSICAL FITNESS PROGRAMS', 'FITNESS CENTERS', 'HEALTH CLUBS & >>>>> GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', 'PERSONAL FITNESS TRAINERS', >>>>> '5', '4', '3', '2', '1', 'Yellow Pages', 'About Us', 'Contact Us', >>>>> 'Support', 'Terms of Use', 'Privacy Policy', 'Advertise With Us', 'Add > /Update Listing', 'Business Profile Login', 'F.A.Q.'] >> >> Then: >> >>>>> pattern = re.compile(r"^[A-Z\s&]+$") >>>>> output = [x for x in list if pattern.match(x)] >>>>> output > >> ['PHYSICAL FITNESS CONSULTANTS & TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', >> 'HEALTH CLUBS & GYMNASIUMS', 'RACQUETBALL COURTS PRIVATE', 'GYMNASIUMS', >> 'HEALTH & FITNESS CLUBS', 'HEALTH & FITNESS CLUBS', 'PERSONAL FITNESS >> TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'EXERCISE & PHYSICAL FITNESS >> PROGRAMS', 'FITNESS CENTERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS >> & GYMNASIUMS', 'PERSONAL FITNESS TRAINERS'] > > > Should've looked earlier. Their master list of categories > http://www.usdirectory.com/cat/g0 shows a few commas, a bunch of dashes, > and the ampersands we talked about. > > "OFFICE SERVICES, SUPPLIES & EQUIPMENT" gets removed because of the comma. > > "AUTOMOBILE - DEALERS" gets removed because of the dash. > > I updated your regex and it seems to have fixed it. > > orig: (r"^[A-Z\s&]+$") > new : (r"^[A-Z\s&,-]+$") > > > Thanks again. If there is a "master list" compare your candidates against it instead of using a heuristic, i. e. categories = set(master_list) output = [category for category in input if category in categories] You can find the categories with >>> import urllib.request >>> import bs4 >>> soup = bs4.BeautifulSoup(urllib.request.urlopen("http://www.usdirectory.com/cat/g0").read()) >>> categories = set() >>> for li in soup.find_all("li"): ... assert li.parent.parent["class"][0].startswith("category_items") ... categories.add(li.text) ... >>> print("\n".join(sorted(categories)[:10])) Accounting & Bookkeeping Services Adoption Services Adult Entertainment Advertising Agricultural Equipment & Supplies Agricultural Production Agricultural Services Aids Resources Aircraft Charters & Rentals Aircraft Dealers & Services From greg.ewing at canterbury.ac.nz Fri May 6 04:51:11 2016 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Fri, 06 May 2016 20:51:11 +1200 Subject: After a year using Node.js, the prodigal son returns In-Reply-To: References: <5729b9d8$0$1612$c3e8da3$5496439d@news.astraweb.com> <572C0629.6030509@gmail.com> Message-ID: Chris Angelico wrote: > https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript This video seems to be broken. It stops about 1/3 of the way through for me and says "No video with supported format and MIME type found". -- Greg From rosuav at gmail.com Fri May 6 05:09:49 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 6 May 2016 19:09:49 +1000 Subject: After a year using Node.js, the prodigal son returns In-Reply-To: References: <5729b9d8$0$1612$c3e8da3$5496439d@news.astraweb.com> <572C0629.6030509@gmail.com> Message-ID: On Fri, May 6, 2016 at 6:51 PM, Gregory Ewing wrote: > Chris Angelico wrote: > >> https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript > > > This video seems to be broken. It stops about 1/3 of the way > through for me and says "No video with supported format and > MIME type found". Ouch. I think the site's gone down. Maybe being posted on python-list slashdotted it? [1] ChrisA [1] Nah, I doubt it. From alister.ware at ntlworld.com Fri May 6 06:01:04 2016 From: alister.ware at ntlworld.com (alister) Date: Fri, 06 May 2016 10:01:04 GMT Subject: Whittle it on down References: <1462426755.15465.598690257.42990546@webmail.messagingengine.com> Message-ID: On Thu, 05 May 2016 19:31:33 -0400, DFS wrote: > On 5/5/2016 1:39 AM, Stephen Hansen wrote: > >> Given: >> >>>>> input = [u'Espa\xf1ol', 'Health & Fitness Clubs (36)', 'Health Clubs >>>>> & Gymnasiums (42)', 'Health Fitness Clubs', 'Name', 'Atlanta city >>>>> guide', 'edit address', 'Tweet', 'PHYSICAL FITNESS CONSULTANTS & >>>>> TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', >>>>> 'www.custombuiltpt.com/', 'RACQUETBALL COURTS PRIVATE', >>>>> 'www.lafitness.com', 'GYMNASIUMS', 'HEALTH & FITNESS CLUBS', >>>>> 'www.lafitness.com', 'HEALTH & FITNESS CLUBS', 'www.lafitness.com', >>>>> 'PERSONAL FITNESS TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'EXERCISE >>>>> & PHYSICAL FITNESS PROGRAMS', 'FITNESS CENTERS', 'HEALTH CLUBS & >>>>> GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', 'PERSONAL FITNESS >>>>> TRAINERS', '5', '4', '3', '2', '1', 'Yellow Pages', 'About Us', >>>>> 'Contact Us', 'Support', 'Terms of Use', 'Privacy Policy', >>>>> 'Advertise With Us', 'Add/Update Listing', 'Business Profile Login', >>>>> 'F.A.Q.'] >> >> Then: >> >>>>> pattern = re.compile(r"^[A-Z\s&]+$") >>>>> output = [x for x in list if pattern.match(x)] >>>>> output > >> ['PHYSICAL FITNESS CONSULTANTS & TRAINERS', 'HEALTH CLUBS & >> GYMNASIUMS', >> 'HEALTH CLUBS & GYMNASIUMS', 'RACQUETBALL COURTS PRIVATE', >> 'GYMNASIUMS', >> 'HEALTH & FITNESS CLUBS', 'HEALTH & FITNESS CLUBS', 'PERSONAL FITNESS >> TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'EXERCISE & PHYSICAL FITNESS >> PROGRAMS', 'FITNESS CENTERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH >> CLUBS & GYMNASIUMS', 'PERSONAL FITNESS TRAINERS'] > > > Should've looked earlier. Their master list of categories > http://www.usdirectory.com/cat/g0 shows a few commas, a bunch of dashes, > and the ampersands we talked about. > > "OFFICE SERVICES, SUPPLIES & EQUIPMENT" gets removed because of the > comma. > > "AUTOMOBILE - DEALERS" gets removed because of the dash. > > I updated your regex and it seems to have fixed it. > > orig: (r"^[A-Z\s&]+$") > new : (r"^[A-Z\s&,-]+$") > > > Thanks again. it looks to me like this system is trying to prevent SQL injection attacks by blacklisting certain characters. this is not the correct way to block such attacks & is probably not a good indicator to the quality of the rest of the application. -- When love is gone, there's always justice. And when justice is gone, there's always force. And when force is gone, there's always Mom. Hi, Mom! -- Laurie Anderson From python at ptoye.com Fri May 6 07:22:01 2016 From: python at ptoye.com (Peter Toye) Date: Fri, 6 May 2016 12:22:01 +0100 Subject: Slight problems with python in Windows Message-ID: <57495789.20160506122201@ptoye.com> I'm trying to install Python under Windows 7 so that I can use git-review and have found a few niggling issues. 1) Apparently (according to the git-review pages) pip has a problem with directories with spaces in their names. Python's default installation directory is under Program Files. I agree that this is a pip issue rather than a Python one, but maybe a warning message would help? 2) According to the Programs and Files section of the Windows Control Panel, installing Python also installs something called the Python Launcher. When I try to remove this (so I can reinstall Python in a better directory) is comes up with an error message: Error opening installation log file. Verify that the specified log file location exists and is writable. After reinstalling I now have 2 copies of the launcher.... I hope it doesn't give me any problems. 3) After uninstalling Python the installation directory is still there with a few files in it (possibly connected with the previous issue). Can I just delete it? Regards, Peter mailto:python at ptoye.com www.ptoye.com From steve at pearwood.info Fri May 6 08:42:59 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 06 May 2016 22:42:59 +1000 Subject: Pylint prefers list comprehension over filter... References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <1462503446.248479.599683745.79791DAC@webmail.messagingengine.com> Message-ID: <572c9155$0$1601$c3e8da3$5496439d@news.astraweb.com> On Fri, 6 May 2016 12:57 pm, Stephen Hansen wrote: > On Thu, May 5, 2016, at 07:46 PM, Dan Sommers wrote: >> On Thu, 05 May 2016 18:37:11 -0700, Stephen Hansen wrote: >> >> > ''.join(x for x in string if x.isupper()) >> >> > The difference is, both filter and your list comprehension *build a >> > list* which is not needed, and wasteful. The above skips building a >> > list, instead returning a generator ... >> >> filter used to build a list, but now it doesn't (where "used to" means >> Python 2.7 and "now" means Python 3.5; I'm too lazy to track down the >> exact point(s) at which it changed): > > Oh, didn't know that. Then again the OP was converting the output of > filter *into* a list, which wasted a list either way. In Python 2.7, run `from future_builtins import *` to get most of the Python 3 behaviour: py> filter(None, []) [] py> from future_builtins import * py> filter(None, []) -- Steven From mymyxin at gmail.com Fri May 6 09:36:03 2016 From: mymyxin at gmail.com (mymyxin at gmail.com) Date: Fri, 6 May 2016 06:36:03 -0700 (PDT) Subject: python, ctypes and GetIconInfo issue In-Reply-To: References: Message-ID: <1868a625-af1e-4937-a9ec-5ccf7710d48b@googlegroups.com> Hello eryk sun, first of all thank you very much, really appreciate your help. Please be informed that I'm a python beginner, so forgive me if my following questions sound stupid (also I'm not a native speaker). > Please avoid windll. It caches the loaded library, which in turn > caches function pointers. So all packages that use windll.user32 are > potentially stepping on each others' toes with mutually incompatible > function prototypes. It also doesn't allow configuring > use_last_error=True to enable ctypes.get_last_error() for WinAPI > function calls. I assume you are referring to this block of code GetIconInfo = windll.user32.GetIconInfo GetIconInfo.argtypes = [HICON, POINTER(ICONINFO)] GetIconInfo.restype = BOOL GetIconInfo.errcheck = ErrorIfZero where as you use user32 = ctypes.WinDLL('user32', use_last_error=True) user32.GetIconInfoExW.errcheck = check_bool user32.GetIconInfoExW.restype = wintypes.BOOL user32.GetIconInfoExW.argtypes = ( wintypes.HICON, # _In_ hIcon PICONINFOEX,) # _Out_ piconinfoex I've checked ctype docu included in python but don't find any hint about your concerns. May I ask you, do you know additional documents/sites which I can use to get a better understanding about caching issue? Or did I miss something from used documentation? > The attribute name is "_fields_", not "__fields__", so you haven't > actually defined any fields and sizeof(ICONINFO) is 0. When you pass > this empty struct to GetIconInfo, it potentially overwrites and > corrupts existing data on the heap that can lead to a crash later on. Ahhh, typically me. Thank you for pointing to it. > Here's the setup I created to test GetIconInfo and GetIconInfoEx. > Maybe you can reuse some of this code, but if you're using XP this > won't work as written because GetIconInfoEx was added in Vista. > Note the use of a __del__ finalizer to call DeleteObject on the > bitmaps. Otherwise, in a real application, calling GetIconInfo would > leak memory. Oh, you answered already an upcoming question I guess, thank you ;-) > Using __del__ is convenient, but note that you can't > reuse an instance without manually calling DeleteObject on the > bitmaps. Don't understand this. Isn't this covered by your example in base class? class ICONINFO_BASE(ctypes.Structure): def __del__(self, gdi32=gdi32): if self.hbmMask: gdi32.DeleteObject(self.hbmMask) self.hbmMask = None if self.hbmColor: gdi32.DeleteObject(self.hbmColor) self.hbmColor = None If I would do somthing like iconinfoex = ICONINFOEX() iconinfoex = None DeleteObject would be called once None gets assigned, wouldn't it? Again, thank you very much. Hubert From grant.b.edwards at gmail.com Fri May 6 09:45:13 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Fri, 6 May 2016 13:45:13 +0000 (UTC) Subject: After a year using Node.js, the prodigal son returns References: <5729b9d8$0$1612$c3e8da3$5496439d@news.astraweb.com> <572C0629.6030509@gmail.com> Message-ID: On 2016-05-06, Chris Angelico wrote: > On Fri, May 6, 2016 at 12:49 PM, Michael Torrie wrote: >> On 05/04/2016 02:59 AM, Steven D'Aprano wrote: >>> A year ago, Gavin Vickery decided to move away from Python and give >>> Javascript with Node.js a try. Twelve months later, he has written about his >>> experiences: >>> >>> >>> http://geekforbrains.com/post/after-a-year-of-nodejs-in-production >> >> Very interesting. Frankly Javascript sounds awful. Even on the front end. > > https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript > > JavaScript is terrible. Really, really bad. And because of that, it > has the potential to sweep the world. If your reasoning is correct, it'll never be able to overtake PHP. I've never written anything over a hundred or two lines in JavaScript, but for small stuff it seems OK -- though as others have noted there are some oddly missing batteries that result in use of a lot of small external libraries for things that any C, PHP, or Python user would have expected to be in the standard library. -- Grant Edwards grant.b.edwards Yow! I'm wearing PAMPERS!! at gmail.com From nospam at dfs.com Fri May 6 09:58:34 2016 From: nospam at dfs.com (DFS) Date: Fri, 6 May 2016 09:58:34 -0400 Subject: Whittle it on down In-Reply-To: References: <1462426755.15465.598690257.42990546@webmail.messagingengine.com> Message-ID: On 5/6/2016 3:45 AM, Peter Otten wrote: > DFS wrote: >> Should've looked earlier. Their master list of categories >> http://www.usdirectory.com/cat/g0 shows a few commas, a bunch of dashes, >> and the ampersands we talked about. >> >> "OFFICE SERVICES, SUPPLIES & EQUIPMENT" gets removed because of the comma. >> >> "AUTOMOBILE - DEALERS" gets removed because of the dash. >> >> I updated your regex and it seems to have fixed it. >> >> orig: (r"^[A-Z\s&]+$") >> new : (r"^[A-Z\s&,-]+$") >> >> >> Thanks again. > > If there is a "master list" compare your candidates against it instead of > using a heuristic, i. e. > > categories = set(master_list) > output = [category for category in input if category in categories] > > You can find the categories with > >>>> import urllib.request >>>> import bs4 >>>> soup = > bs4.BeautifulSoup(urllib.request.urlopen("http://www.usdirectory.com/cat/g0").read()) >>>> categories = set() >>>> for li in soup.find_all("li"): > ... assert li.parent.parent["class"][0].startswith("category_items") > ... categories.add(li.text) > ... >>>> print("\n".join(sorted(categories)[:10])) "import urllib.request ImportError: No module named request" I'm on python 2.7.11 > Accounting & Bookkeeping Services > Adoption Services > Adult Entertainment > Advertising > Agricultural Equipment & Supplies > Agricultural Production > Agricultural Services > Aids Resources > Aircraft Charters & Rentals > Aircraft Dealers & Services Yeah, I actually did something like that last night. Was trying to get their full tree structure, which goes 4 levels deep: ie Arts & Entertainment Newpapers News Dealers Prepess Services What I referred to as their 'master list' is actually just 2 levels deep. My bad. So far I haven't come across one that had anything in it but letters, dashes, commas or ampersands. Thanks From rosuav at gmail.com Fri May 6 10:04:17 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 7 May 2016 00:04:17 +1000 Subject: After a year using Node.js, the prodigal son returns In-Reply-To: References: <5729b9d8$0$1612$c3e8da3$5496439d@news.astraweb.com> <572C0629.6030509@gmail.com> Message-ID: On Fri, May 6, 2016 at 11:45 PM, Grant Edwards wrote: >> JavaScript is terrible. Really, really bad. And because of that, it >> has the potential to sweep the world. > > If your reasoning is correct, it'll never be able to overtake PHP. > > I've never written anything over a hundred or two lines in JavaScript, > but for small stuff it seems OK -- though as others have noted there > are some oddly missing batteries that result in use of a lot of small > external libraries for things that any C, PHP, or Python user would > have expected to be in the standard library. Except that it's pretty easy to switch out PHP for Python, or anything else. JavaScript is what it is because it's hard to just use a different language. ChrisA From nospam at dfs.com Fri May 6 10:41:30 2016 From: nospam at dfs.com (DFS) Date: Fri, 6 May 2016 10:41:30 -0400 Subject: Whittle it on down In-Reply-To: References: <1462426755.15465.598690257.42990546@webmail.messagingengine.com> Message-ID: On 5/6/2016 9:58 AM, DFS wrote: > On 5/6/2016 3:45 AM, Peter Otten wrote: >> DFS wrote: > >>> Should've looked earlier. Their master list of categories >>> http://www.usdirectory.com/cat/g0 shows a few commas, a bunch of dashes, >>> and the ampersands we talked about. >>> >>> "OFFICE SERVICES, SUPPLIES & EQUIPMENT" gets removed because of the >>> comma. >>> >>> "AUTOMOBILE - DEALERS" gets removed because of the dash. >>> >>> I updated your regex and it seems to have fixed it. >>> >>> orig: (r"^[A-Z\s&]+$") >>> new : (r"^[A-Z\s&,-]+$") >>> >>> >>> Thanks again. >> >> If there is a "master list" compare your candidates against it instead of >> using a heuristic, i. e. >> >> categories = set(master_list) >> output = [category for category in input if category in categories] >> >> You can find the categories with >> >>>>> import urllib.request >>>>> import bs4 >>>>> soup = >> bs4.BeautifulSoup(urllib.request.urlopen("http://www.usdirectory.com/cat/g0").read()) >> >>>>> categories = set() >>>>> for li in soup.find_all("li"): >> ... assert li.parent.parent["class"][0].startswith("category_items") >> ... categories.add(li.text) >> ... >>>>> print("\n".join(sorted(categories)[:10])) > > > > "import urllib.request > ImportError: No module named request" Figured it out using urllib2. Your code returns 411 categories from that first page. There are up to 4 levels of categorization: Level 1: Arts & Entertainment Level 2: Newspapers Level 3: Newspaper Brokers Level 3: Newspaper Dealers Back Number Level 3: Newspaper Delivery Level 3: Newspaper Distributors Level 3: Newsracks Level 3: Printers Newspapers Level 3: Newspaper Dealers Level 3: News Dealers Level 4: News Dealers Wholesale Level 4: Shoppers News Publications Level 3: News Service Level 4: Newspaper Feature Syndicates Level 4: Prepress Services http://www.usdirectory.com/cat/g0 shows 21 Level 1 categories, and 390 Level 2. To get the Level 3 and 4 you have to drill-down using the hyperlinks. How to do it in python code is beyond my skills at this point. Get the hrefs and load them and parse, then get the next level and load them and parse, etc.? From mymyxin at gmail.com Fri May 6 10:49:28 2016 From: mymyxin at gmail.com (mymyxin at gmail.com) Date: Fri, 6 May 2016 07:49:28 -0700 (PDT) Subject: python, ctypes and GetIconInfo issue In-Reply-To: References: Message-ID: <5dd5db3e-ffa9-4c7b-9385-dc20f85dd45b@googlegroups.com> A further question if you don't mind. In your example you used a base class and ICONINFO well as ICONINFOEX inherit it. As the members of ICONINFO are part of ICONINFOEX couldn't we do something like class ICONINFO_BASE(ctypes.Structure): def __del__(self, ): if self.hbmMask: gdi32.DeleteObject(self.hbmMask) self.hbmMask = None if self.hbmColor: gdi32.DeleteObject(self.hbmColor) self.hbmColor = None class ICONINFO(ICONINFO_BASE): _fields_ = (('fIcon', wintypes.BOOL), ('xHotspot', wintypes.DWORD), ('yHotspot', wintypes.DWORD), ('hbmMask', wintypes.HBITMAP), ('hbmColor', wintypes.HBITMAP)) class ICONINFOEX(ICONINFO): _fields_ = (('cbSize', wintypes.DWORD), # ('fIcon', wintypes.BOOL), # ('xHotspot', wintypes.DWORD), # ('yHotspot', wintypes.DWORD), # ('hbmMask', wintypes.HBITMAP), # ('hbmColor', wintypes.HBITMAP), ('wResID', wintypes.WORD), ('szModName', wintypes.WCHAR * MAX_PATH), ('szResName', wintypes.WCHAR * MAX_PATH)) def __init__(self, *args, **kwds): super(ICONINFOEX, self).__init__(*args, **kwds) self.cbSize = ctypes.sizeof(self) A dir on instance of that class indicates that it should be possible as it contains all needed members and size seems to be correct as well, but a call to GetIconInfoExW fails with: WindowsError: [Error 87] The parameter is incorrect. info = ICONINFOEX() info.cbSize = ctypes.sizeof(info) print dir(info) print info.cbSize user32.GetIconInfoExW(hIcon, ctypes.byref(info)) Thank you Hubert From walters.justin01 at gmail.com Fri May 6 11:23:01 2016 From: walters.justin01 at gmail.com (justin walters) Date: Fri, 6 May 2016 08:23:01 -0700 Subject: python - handling HTTP requests asynchronously In-Reply-To: References: Message-ID: On Thu, May 5, 2016 at 11:56 PM, wrote: > Hi everyone, > I need to generate a PDF report for each entry of a django queryset. > There'll be between between 30k and 40k entries. > > The PDF is generated through an external API. Since currently is generated > on demand, this is handled synchronously via an HTTP request/response. That > will be different for this task, since I think I'll use a django management > command to loop through the queryset and perform the PDF generation. > > Which approach should I follow for this task? I thought about 3 possibile > solutions, although are technologies that I never used: > > 1) Celery: assign a task (http request with a different payload) to a > worker, then retrieve it once it's done. > > 2) request-futures: using requests in a non-blocking way. > > 3) multiprocessing module, with e.g. 10 as workers limit. > > > the goal is to use the API concurrently (e.g. send 10 or 100 http requests > simultaneously, depending on how many concurrent requests the API can > handle). > > Anybody here that handled a similar task and can give advices on how to > proceed on this? > -- > https://mail.python.org/mailman/listinfo/python-list > Have you tried channels: https://github.com/andrewgodwin/channels ? If it's an asyncronous request/response cycle you're looking for, it should work well. Essentially, you have worker processes that receive a message over a websocket connection and then send the new message back to the "group". I would recommend using the redis in memory transaction layer if you go this route as it is the fastest and most efficient. The best part about channels is that you can still run a normal django app alongside it. You can have only one app use websockets while the rest use standard http.. From __peter__ at web.de Fri May 6 11:44:29 2016 From: __peter__ at web.de (Peter Otten) Date: Fri, 06 May 2016 17:44:29 +0200 Subject: Whittle it on down References: <1462426755.15465.598690257.42990546@webmail.messagingengine.com> Message-ID: DFS wrote: > There are up to 4 levels of categorization: > http://www.usdirectory.com/cat/g0 shows 21 Level 1 categories, and 390 > Level 2. To get the Level 3 and 4 you have to drill-down using the > hyperlinks. > > How to do it in python code is beyond my skills at this point. Get the > hrefs and load them and parse, then get the next level and load them and > parse, etc.? Yes, that should work ;) From pertti.kosunen at pp.nic.fi Fri May 6 12:03:14 2016 From: pertti.kosunen at pp.nic.fi (Pertti Kosunen) Date: Fri, 6 May 2016 19:03:14 +0300 Subject: Slight problems with python in Windows In-Reply-To: References: <57495789.20160506122201@ptoye.com> Message-ID: On 6.5.2016 14:22, Peter Toye wrote: > I'm trying to install Python under Windows 7 so that I can use git-review and have found a few niggling issues. > > 1) Apparently (according to the git-review pages) pip has a problem with directories with spaces in their names. Python's default installation directory is under Program Files. I agree that this is a pip issue rather than a Python one, but maybe a warning message would help? Select install to all users and it should install to root \PythonXY directory where XY is major version number. From zachary.ware+pylist at gmail.com Fri May 6 12:17:23 2016 From: zachary.ware+pylist at gmail.com (Zachary Ware) Date: Fri, 6 May 2016 11:17:23 -0500 Subject: Slight problems with python in Windows In-Reply-To: <57495789.20160506122201@ptoye.com> References: <57495789.20160506122201@ptoye.com> Message-ID: Hi Peter, On Fri, May 6, 2016 at 6:22 AM, Peter Toye wrote: > I'm trying to install Python under Windows 7 so that I can use git-review and have found a few niggling issues. > > 1) Apparently (according to the git-review pages) pip has a problem with directories with spaces in their names. Python's default installation directory is under Program Files. I agree that this is a pip issue rather than a Python one, but maybe a warning message would help? I don't believe this is true anymore, I've successfully used pip with 3.5 installed in Program Files, and also just now in a test venv named "test venv". Do note that with installation in Program Files, you get the benefits of the install directory being writable only to administrators, but also the drawbacks: only administrators can use pip to install to the global site-packages. You can use either 'pip --user', or create a venv in a directory writable to you and use it. Also note that you can't use "pip.exe" to upgrade pip itself since it can't overwrite "pip.exe" while it's in use; use 'python -m pip' instead. > 2) According to the Programs and Files section of the Windows Control Panel, installing Python also installs something called the Python Launcher. When I try to remove this (so I can reinstall Python in a better directory) is comes up with an error message: The Python Launcher is a very handy tool called 'py.exe' which makes it much easier to use more than one version of Python on a Windows machine. In an all users install, py.exe is installed to C:\Windows and is thus always available on PATH, so you can invoke Python 3.5 by calling 'py -3.5' without having to adjust your PATH. The error message is odd, though, would you mind trying to reproduce it and opening a bug at bugs.python.org? > Error opening installation log file. Verify that the specified log file location exists and is writable. > > After reinstalling I now have 2 copies of the launcher.... I hope it doesn't give me any problems. It shouldn't. The launcher only installs 2 files, py.exe and pyw.exe (counterpart to pythonw.exe), both in C:\Windows. > 3) After uninstalling Python the installation directory is still there with a few files in it (possibly connected with the previous issue). Can I just delete it? Yes, that should be fine. I would guess it's still there due to pip artifacts in Lib\ and Scripts\. Hope this helps, -- Zach From rustompmody at gmail.com Fri May 6 14:11:03 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Fri, 6 May 2016 11:11:03 -0700 (PDT) Subject: After a year using Node.js, the prodigal son returns In-Reply-To: References: <5729b9d8$0$1612$c3e8da3$5496439d@news.astraweb.com> <572C0629.6030509@gmail.com> Message-ID: <44d2cc37-b314-422f-b035-964c912c2466@googlegroups.com> On Friday, May 6, 2016 at 8:23:27 AM UTC+5:30, Chris Angelico wrote: > On Fri, May 6, 2016 at 12:49 PM, Michael Torrie wrote: > > On 05/04/2016 02:59 AM, Steven D'Aprano wrote > >> A year ago, Gavin Vickery decided to move away from Python and give > >> Javascript with Node.js a try. Twelve months later, he has written about his > >> experiences: > >> > >> > >> http://geekforbrains.com/post/after-a-year-of-nodejs-in-production > > > > Very interesting. Frankly Javascript sounds awful. Even on the front end. > > https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript > > JavaScript is terrible. Really, really bad. And because of that, it > has the potential to sweep the world. If python community is passionate in hating javascript it may wish to look at webassembly: https://en.wikipedia.org/wiki/WebAssembly https://medium.com/javascript-scene/what-is-webassembly-the-dawn-of-a-new-era-61256ec5a8f6 From nospam at dfs.com Fri May 6 15:10:37 2016 From: nospam at dfs.com (DFS) Date: Fri, 6 May 2016 15:10:37 -0400 Subject: A fun python CLI program for all to enjoy! Message-ID: getAddresses.py Scrapes addresses from www.usdirectory.com and stores them in a SQLite database, or writes them to text files for mailing labels, etc Now, just by typing 'fast food Taco Bell 10 db all' you can find out how many Taco Bells are within 10 miles of you, and store all the addresses in your own address database. No more convoluted Googling, or hitting the 'Next Page' button, or fumbling with the Yellow Pages... Note: the db structure is flat on purpose, and the .csv files aren't quote delimited. Put the program in its own directory. It creates the SQLite database there, and writes files there, too. Reviews of code, bug reports, criticisms, suggestions for improvement, etc are all welcome. Enjoy! ======================================================================== #getAddresses.py import os, sys, requests, time, datetime from lxml import html import pyodbc, sqlite3, re #show values of variables, HTML content, etc #set it to False for short/concise program output verbose = False if verbose == True: print "The verbose setting is turned On." print "" #check if address is unique addrCheck = [] def addrUnique(addr): if addr not in addrCheck: x = True addrCheck.append(addr) else: x = False return x #validate and parse command line def showHelp(): print "" print " Enter search word(s), city or zip, state, miles to search, txt or csv or db, # addresses to save (no commas)" print "" print " eg: restaurant Knoxville TN 10 txt 50" print " search for restaurants within 10 miles of Knoxville TN, and write" print " the first 50 address to a txt file" print "" print " eg: furniture 30303 GA 20 csv all" print " search for furniture within 20 miles of zip 30303 GA," print " and write all results to a csv file" print "" print " eg: boxing gyms Detroit MI 10 db 5" print " search for boxing gyms within 10 miles of Detroit MI, and store" print " the first 5 results in a database" print "" print " All entries are case-insensitive (ie TX or tx are acceptable)" exit(0) argCnt = len(sys.argv) if argCnt < 7: showHelp() if verbose == True: print "" print str(argCnt) + " arguments" keyw = "" #eg restaurant, boxing gym if argCnt == 7: keyw = sys.argv[1] #one search word if argCnt > 7: #multiple search words for i in range(1,argCnt-5): keyw = keyw + sys.argv[i] + "+" keyw = keyw[:-1] #drop trailing + sign cityzip = sys.argv[argCnt-5] #eg Atlanta or 30339 state = sys.argv[argCnt-4] #eg GA miles = sys.argv[argCnt-3] #eg 5,10,20,30,50 (website allows max 30) store = sys.argv[argCnt-2] #write address to file or database addrWant = sys.argv[argCnt-1] #eg save All or number >0 if addrWant.lower() != "all": #how many addresses to save if addrWant.isdigit() == False: showHelp() if addrWant == "0" : showHelp() addrWant = int(addrWant) elif addrWant.lower() == "all": addrWant = addrWant.lower() else: addrWant = int(addrWant) if store != "csv" and store != "txt" and store != "db": showHelp() #begin timing the code startTime = time.clock() #website, SQLite db, search string, current date/time for use with db datasrc = "www.usdirectory.com" dbName = "addresses.sqlite" search = keyw + " " + str(cityzip) + " " + state + " " + str(miles) + " " + str(addrWant) loaddt = datetime.datetime.now() #write addresses to file #each time the same search is done, the file is deleted and recreated if store == "csv" or store == "txt": #csv will write in .csv format - header and 1 line per address #txt will write out 3 lines per address, then blank before next address webfile = "usdirectory.com_"+keyw+"_"+cityzip+"_"+state+"."+store f = open(webfile,"w") if store == "csv": f.write("Name,Address,CityStateZip\n") f.close #store addresses in database cSQL = "" if store == "db": #creates a SQLite database that Access 2003 can't read #conn = sqlite3.connect(dbName) #also creates a SQLite database that Access 2003 can't read conn = pyodbc.connect('Driver={SQLite3 ODBC Driver};Database=' + dbName) db = conn.cursor() cSQL = "CREATE TABLE If Not Exists ADDRESSES " cSQL += "(datasrc, search, category, name, street, city, state, zip, loaddt, " cSQL += "PRIMARY KEY (datasrc, search, name, street));" db.execute(cSQL) # cSQL = "CREATE TABLE If Not Exists CATEGORIES " # cSQL += "(catID INTEGER PRIMARY KEY, catDesc);" # db.execute(cSQL) # db.execute("CREATE UNIQUE INDEX If Not Exists UIDX_CATDESC ON CATEGORIES (catDesc);") conn.commit() if verbose == True: print("connected to database: " + dbName) print cSQL print("created table: addresses") print("") if verbose == True: print "Search summary" print "------------------------------" print "Keywords: " + keyw print "City/Zip: " + cityzip print "State : " + state print "Radius : " + str(miles) + " miles" print "Save : " + str(addrWant) + " addresses to " + store print "------------------------------" print "" #build url wBase = "http://www.usdirectory.com" wForm = "/ypr.aspx?fromform=qsearch" wKeyw = "&qhqn=" + keyw wCityZip = "&qc=" + cityzip wState = "&qs=" + state wDist = "&rg=" + str(miles) wSort = "&sb=a2z" #sort alpha wPage = "&ap=" #used with the results page number webpage = wBase + wForm + wKeyw + wCityZip + wState + wDist if verbose == True: print "Search url: \n" + webpage print "" #delete previous results of identical search if store == "db": cSQL = "DELETE FROM addresses " cSQL += "WHERE datasrc = '" + datasrc + "' " cSQL += "AND search = '" + search + "';" if verbose == True: print cSQL db.execute(cSQL) conn.commit() #query web server, save results print "searching..." i = 0 dupes = 0 addrReturned = 0 addrSaved = 0 while 1: wPageNbr = wPage + str(i+1) webpage = wBase + wForm + wKeyw + wCityZip + wState + wDist + wSort + wPageNbr page = requests.get(webpage) tree = html.fromstring(page.content) #no matches matches = tree.xpath('//strong/text()') if i == 0 and "No results were found" in str(matches): print "No results found for that search" exit(0) os.remove(webfile) #parse number of addresses returned #some searches return 2 items: ['Found N results', 'junk'] #some searches return 3 items: ['Filter this search','Found N results','junk'] if i == 0: match = tree.xpath('//div[@class="header_text"]/text()') if len(match) > 2: match.pop(0) #remove first element if 2 match = [int(s) for s in match[0].split() if s.isdigit()] addrFound = match[0] if addrWant != "all": addrWant = min(addrWant,addrFound) print str(addrFound) + " matches found (" + str(addrWant) + " will be saved)" print "" #split names, addresses into lists nms = tree.xpath('//span[@class="header_text3"]/text()') if len(nms) == 0: break addr = tree.xpath('//span[@class="text3"]/text()') addr = [t.replace("\r\n", "") for t in addr] addr = filter(None, (t.strip() for t in addr)) street = [s.split(',')[0] for s in addr] city = [c.split(',')[1].strip() for c in addr] state = [s[-8:][:2] for s in addr] zip = [z[-5:] for z in addr] #get usdirectory.com categories category = tree.xpath('//a/text()') category = [c.strip() for c in category] category = filter(None, category) pattern = re.compile(r"^[A-Z\s&,-]+$") category = [x for x in category if pattern.match(x)] #screen feedback print "retrieving page " + str(i+1) + ": " + str(len(nms)) + " addresses" if verbose == True: print "" print "Names: \n" + str(nms) print "" print "Addresses: \n" + str(addr) print "" print "Categories: \n" + str(category) print "----------------------------------------------------------------------------------------" print "" #data integrity check - make sure all lists have same # of items lenData = [len(category),len(nms),len(addr),len(street),len(city),len(state),len(zip)] if len(set(lenData)) != 1: print "Data parsing issue. One or more lists has an incorrect number of items. Program will exit." exit(0) if verbose == True: if len(set(lenData)) == 1: print "Verified: each list has " + str(len(nms)) + " items in it." #write addresses to file if store == "txt" or store == "csv": addrList = [] for j in range(len(nms)): if addrUnique(nms[j]+' '+street[j]) == True: if store == "txt": addrList.append(nms[j]+'\n'+street[j]+'\n'+city[j]+', '+state[j]+' '+zip[j]+'\n\n') if store == "csv": addrList.append(nms[j]+',' +street[j]+',' +city[j]+' ' +state[j]+' '+zip[j]+'\n') addrSaved += 1 else: dupes += 1 print " * duplicate address found: " + nms[j] + ", " + street[j] addrReturned += 1 if addrWant != "all": if addrSaved >= addrWant: break f = open(webfile,"a") for address in addrList: f.write(address) f.close #write addresses to database if store == "db": for j in range(len(nms)): dupeRow = False cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?,?,?,?,?)" #(datasrc,search,category,name,street,city,state,zip,loaddt) " Vals = datasrc,search,category[j],nms[j],street[j],city[j],state[j],zip[j],str(loaddt) if verbose == True: print cSQL + ',' + str(Vals) try: db.execute(cSQL, Vals) except (pyodbc.Error) as programError: if str(programError).find("UNIQUE constraint failed") > 0: dupeRow = True dupes +=1 print " * duplicate address found: " + nms[j] + ", " + street[j] pass addrReturned += 1 if dupeRow == False: addrSaved += 1 if addrWant != "all": if addrSaved >= addrWant: break conn.commit() if addrSaved >= addrFound or addrSaved >= addrWant: break i += 1 time.sleep(2) #finish if (store == "csv" or store == "txt"): print "\nFinished\nWrote " + str(addrSaved) + " addresses to file " + webfile elif store == "db": db.close() conn.close() print "\nFinished\nStored " + str(addrSaved) + " addresses in database: " + dbName if dupes > 0: print "(" + str(dupes) + " duplicate addresses ignored)" #timer endTime = time.clock() print "processing time: %.2g seconds" %(endTime-startTime) #bug in www.usdirectory.com code: usually overreports matches by 1 if (addrWant == "all") and (addrReturned != addrFound): print "Note: " + datasrc + " reported " + str(addrFound) + " matches, but returned " + str(addrReturned) ======================================================================== From python at mrabarnett.plus.com Fri May 6 16:30:17 2016 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 6 May 2016 21:30:17 +0100 Subject: A fun python CLI program for all to enjoy! In-Reply-To: References: Message-ID: <7e52b918-2087-f93f-43cb-3411c1cdc881@mrabarnett.plus.com> On 2016-05-06 20:10, DFS wrote: > getAddresses.py > > Scrapes addresses from www.usdirectory.com and stores them in a SQLite > database, or writes them to text files for mailing labels, etc > > Now, just by typing 'fast food Taco Bell 10 db all' you can find > out how many Taco Bells are within 10 miles of you, and store all the > addresses in your own address database. > > No more convoluted Googling, or hitting the 'Next Page' button, or > fumbling with the Yellow Pages... > > Note: the db structure is flat on purpose, and the .csv files aren't > quote delimited. > > Put the program in its own directory. It creates the SQLite database > there, and writes files there, too. > > Reviews of code, bug reports, criticisms, suggestions for improvement, > etc are all welcome. > OK, you asked for it... :-) 1. It's shorter and clearer not to compare with True or False: if verbose: and: if not dupeRow: 2. You can print a blank line with an empty print statement: print 3. When looking for unique items, a set is a better choice than a list: addrCheck = set() def addrUnique(addr): if addr not in addrCheck: x = True addrCheck.add(addr) else: x = False return x 4. Try string formatting instead multiple concatenation: print "%s arguments" % argCnt 5. Strings have a .join method, and when you combine it with string slicing: keyw = "+".join(sys.argv[1 : argCnt - 5]) 6. Another example of string formatting: search = "%s %s %s %s %s" % (keyw, cityzip, state, miles, addrWant) 7. It's recommended to use the 'with' statement when handling files: with open(webfile, "w") as f: if store == "csv": f.write("Name,Address,CityStateZip\n") If you don't want to use the 'with' statement, note that closing the file is: f.close() It needs the "()"! 8. When using SQL, you shouldn't try to insert the values yourself; you should use parametrised queries: cSQL = "DELETE FROM addresses WHERE datasrc = ? AND search = ?;" if verbose: print cSQL db.execute(cSQL, (datasrc, search)) conn.commit() It'll insert the values where the "?" are and will do any necessary quoting itself. (Actually, some drivers use "?", others use "%s", so if it doesn't work with one, try the other.) The way you wrote it, it would fail if a value contained a "'". It's that kind of thing that leads to SQL injection attacks. From beliavsky at aol.com Fri May 6 16:35:20 2016 From: beliavsky at aol.com (beliavsky at aol.com) Date: Fri, 6 May 2016 13:35:20 -0700 (PDT) Subject: Python is an Equal Opportunity Programming Language In-Reply-To: References: Message-ID: On Thursday, May 5, 2016 at 3:00:01 PM UTC-4, Terry Reedy wrote: > https://motherboard.vice.com/blog/python-is-an-equal-opportunity-programming-language > > from an 'Intel(R) Software Evangelist' > -- > Terry Jan Reedy >From the link: MB: What is it about Python that makes it friendly to women? Is it something about the actual language itself? Or is it more of a subcultural thing within the community? DS: One thing that I think causes this is the founder of the Python project, a guy named Guido van Rossum. He's referred to as the "BDFL"--the Benevolent Dictator for Life. The way to think of him is like Linus Torvalds of Linux. Most of his keynote at that conference was answering questions from the people who had attended. And he actually said, "Let's alternate between men and women asking questions."On the second day of the conference, he was wearing a shirt from PyLadies, another nonprofit like Django Girls that helps women learn how to program on Python. ********************************************************* This not "equal opportunity". It is a quota system. It's my impression that in the U.S., Asians are over-represented among programmers relative to their share of the population and that whites and especially blacks are under-represented. Should we impose racial quotas on questions at conferences and call that "equal opportunity" as well? From ethan at stoneleaf.us Fri May 6 17:07:15 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 06 May 2016 14:07:15 -0700 Subject: Python is an Equal Opportunity Programming Language In-Reply-To: References: Message-ID: <572D0783.5050707@stoneleaf.us> On 05/06/2016 01:35 PM, beliavsky--- via Python-list wrote: > Most of [Guido's] keynote at that conference was answering questions from > the people who had attended. And he actually said, "Let's alternate between > men and women asking questions."On the second day of the conference, he was > wearing a shirt from PyLadies, another nonprofit like Django Girls that helps > women learn how to program on Python. > > ********************************************************* > > This not "equal opportunity". It is a quota system. It's a corrective action, a way of getting men accustomed to listening to women and hearing good ideas and questions from them, and a way to accustom women to speaking in (currently) male dominated groups. And it is far more equal opportunity than having 25 males ask questions and only one or two females. -- ~Ethan~ From beliavsky at aol.com Fri May 6 17:45:38 2016 From: beliavsky at aol.com (beliavsky at aol.com) Date: Fri, 6 May 2016 14:45:38 -0700 (PDT) Subject: Python is an Equal Opportunity Programming Language In-Reply-To: References: <572D0783.5050707@stoneleaf.us> Message-ID: On Friday, May 6, 2016 at 5:07:28 PM UTC-4, Ethan Furman wrote: > On 05/06/2016 01:35 PM, beliavsky--- via Python-list wrote: > > > Most of [Guido's] keynote at that conference was answering questions from > > the people who had attended. And he actually said, "Let's alternate > between > > men and women asking questions."On the second day of the conference, > he was > > wearing a shirt from PyLadies, another nonprofit like Django Girls > that helps > > women learn how to program on Python. > > > > ********************************************************* > > > > This not "equal opportunity". It is a quota system. > > It's a corrective action, a way of getting men accustomed to listening > to women and hearing good ideas and questions from them, and a way to > accustom women to speaking in (currently) male dominated groups. It's silly to say that just because a group is over-represented that it "dominates". If a conference has more Asians than whites does that necessarily make it Asian-dominated? > And it is far more equal opportunity than having 25 males ask questions > and only one or two females. Not if there are 25 males with questions and only one or two females with questions. Among the people who have questions, you could choose randomly. You and Terry Reedy misuse the term "equal opportunity". From wrightalexw at gmail.com Fri May 6 18:20:35 2016 From: wrightalexw at gmail.com (alex wright) Date: Fri, 6 May 2016 18:20:35 -0400 Subject: Python is an Equal Opportunity Programming Language In-Reply-To: References: <572D0783.5050707@stoneleaf.us> Message-ID: It seems like it would be equal opportunity between sexes. 1:1 opportunity to ask based on apparent sex. It is not equal representation necessarily. On May 6, 2016 5:53 PM, "beliavsky--- via Python-list" < python-list at python.org> wrote: > On Friday, May 6, 2016 at 5:07:28 PM UTC-4, Ethan Furman wrote: > > On 05/06/2016 01:35 PM, beliavsky--- via Python-list wrote: > > > > > Most of [Guido's] keynote at that conference was answering questions > from > > > the people who had attended. And he actually said, "Let's alternate > > between > > > men and women asking questions."On the second day of the conference, > > he was > > > wearing a shirt from PyLadies, another nonprofit like Django Girls > > that helps > > > women learn how to program on Python. > > > > > > ********************************************************* > > > > > > This not "equal opportunity". It is a quota system. > > > > It's a corrective action, a way of getting men accustomed to listening > > to women and hearing good ideas and questions from them, and a way to > > accustom women to speaking in (currently) male dominated groups. > > It's silly to say that just because a group is over-represented that it > "dominates". If a conference has more Asians than whites does that > necessarily make it Asian-dominated? > > > And it is far more equal opportunity than having 25 males ask questions > > and only one or two females. > > Not if there are 25 males with questions and only one or two females with > questions. Among the people who have questions, you could choose randomly. > You and Terry Reedy misuse the term "equal opportunity". > -- > https://mail.python.org/mailman/listinfo/python-list > From nospam at dfs.com Fri May 6 18:43:08 2016 From: nospam at dfs.com (DFS) Date: Fri, 6 May 2016 18:43:08 -0400 Subject: Whittle it on down In-Reply-To: References: <1462426755.15465.598690257.42990546@webmail.messagingengine.com> Message-ID: On 5/6/2016 11:44 AM, Peter Otten wrote: > DFS wrote: > >> There are up to 4 levels of categorization: > >> http://www.usdirectory.com/cat/g0 shows 21 Level 1 categories, and 390 >> Level 2. To get the Level 3 and 4 you have to drill-down using the >> hyperlinks. >> >> How to do it in python code is beyond my skills at this point. Get the >> hrefs and load them and parse, then get the next level and load them and >> parse, etc.? > > Yes, that should work ;) How about you do it, and I'll tell you if you did it right? ha! From nospam at dfs.com Fri May 6 19:12:41 2016 From: nospam at dfs.com (DFS) Date: Fri, 6 May 2016 19:12:41 -0400 Subject: A fun python CLI program for all to enjoy! In-Reply-To: References: <7e52b918-2087-f93f-43cb-3411c1cdc881@mrabarnett.plus.com> Message-ID: On 5/6/2016 4:30 PM, MRAB wrote: > On 2016-05-06 20:10, DFS wrote: >> getAddresses.py >> >> Scrapes addresses from www.usdirectory.com and stores them in a SQLite >> database, or writes them to text files for mailing labels, etc >> >> Now, just by typing 'fast food Taco Bell 10 db all' you can find >> out how many Taco Bells are within 10 miles of you, and store all the >> addresses in your own address database. >> >> No more convoluted Googling, or hitting the 'Next Page' button, or >> fumbling with the Yellow Pages... >> >> Note: the db structure is flat on purpose, and the .csv files aren't >> quote delimited. >> >> Put the program in its own directory. It creates the SQLite database >> there, and writes files there, too. >> >> Reviews of code, bug reports, criticisms, suggestions for improvement, >> etc are all welcome. >> > OK, you asked for it... :-) > > 1. It's shorter and clearer not to compare with True or False: > > if verbose: > > and: > > if not dupeRow: Done. It will take some getting used to, though. I like that it's shorter, but I could do the same in VBA and almost always chose not to. > 2. You can print a blank line with an empty print statement: > > print Done. I actually like the way print looks better than print "" > 3. When looking for unique items, a set is a better choice than a list: > > addrCheck = set() > > def addrUnique(addr): > if addr not in addrCheck: > x = True > addrCheck.add(addr) > else: > x = False > return x Done. I researched this just now on StackOverflow: "Sets are significantly faster when it comes to determining if an object is present in the set" and "lists are very nice to sort and have order while sets are nice to use when you don't want duplicates and don't care about order." The speed difference won't matter here in my little app, but it's better to use the right construct for the job. > 4. Try string formatting instead multiple concatenation: > > print "%s arguments" % argCnt You're referring to this line: print str(argCnt) + " arguments" Is there a real benefit of using string formatting here? (other than the required str() conversion) > 5. Strings have a .join method, and when you combine it with string > slicing: > > keyw = "+".join(sys.argv[1 : argCnt - 5]) Slick. Works a treat, and saved 2 lines of code. String handling is another area in which python shines compared to VB. > 6. Another example of string formatting: > > search = "%s %s %s %s %s" % (keyw, cityzip, state, miles, addrWant) Done. It's shorter, and doesn't require the str() conversion I had to do on several of the items. If I can remember to use it, it should eliminate these: "TypeError: cannot concatenate 'str' and 'int' objects" > 7. It's recommended to use the 'with' statement when handling files: > > with open(webfile, "w") as f: > if store == "csv": > f.write("Name,Address,CityStateZip\n") Done. I read that using 'with' means Python closes the file even if an exception occurs. So a definite benefit. > If you don't want to use the 'with' statement, note that closing the > file is: > > f.close() > > It needs the "()"! I used close() in 1 place, but close without parens in 2 other places. So it works either way. Good catch. (it's moot now: all 'f.open()/f.close()' replaced by 'with open()') > 8. When using SQL, you shouldn't try to insert the values yourself; you > should use parametrised queries: > > cSQL = "DELETE FROM addresses WHERE datasrc = ? AND search = ?;" > if verbose: > print cSQL > db.execute(cSQL, (datasrc, search)) > conn.commit() > > It'll insert the values where the "?" are and will do any necessary > quoting itself. (Actually, some drivers use "?", others use "%s", so if > it doesn't work with one, try the other.) > > The way you wrote it, it would fail if a value contained a "'". It's > that kind of thing that leads to SQL injection attacks. Fixed. You'll notice later on in the code I used the parameterized method for INSERTS. I hate the look of that method, but it does make dealing with apostrophes easier, and makes it safer as you say. Thanks for the code review, RMAB. Good improvements. From ethan at stoneleaf.us Fri May 6 19:29:19 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 06 May 2016 16:29:19 -0700 Subject: A fun python CLI program for all to enjoy! In-Reply-To: References: <7e52b918-2087-f93f-43cb-3411c1cdc881@mrabarnett.plus.com> Message-ID: <572D28CF.3080100@stoneleaf.us> On 05/06/2016 04:12 PM, DFS wrote: > On 5/6/2016 4:30 PM, MRAB wrote: >> If you don't want to use the 'with' statement, note that closing the >> file is: >> >> f.close() >> >> It needs the "()"! > > I used close() in 1 place, but close without parens in 2 other places. > So it works either way. Good catch. No, it doesn't. `f.close` simple returns the close function, it doesn't call it. The "it works" was simply because Python closed the files for you later. Not a big deal in a small program like this, but still a mistake. -- ~Ethan~ From eryksun at gmail.com Fri May 6 19:38:49 2016 From: eryksun at gmail.com (eryk sun) Date: Fri, 6 May 2016 18:38:49 -0500 Subject: python, ctypes and GetIconInfo issue In-Reply-To: <1868a625-af1e-4937-a9ec-5ccf7710d48b@googlegroups.com> References: <1868a625-af1e-4937-a9ec-5ccf7710d48b@googlegroups.com> Message-ID: On Fri, May 6, 2016 at 8:36 AM, wrote: > >> Please avoid windll. It caches the loaded library, which in turn >> caches function pointers. So all packages that use windll.user32 are >> potentially stepping on each others' toes with mutually incompatible >> function prototypes. It also doesn't allow configuring >> use_last_error=True to enable ctypes.get_last_error() for WinAPI >> function calls. > I assume you are referring to this block of code > > GetIconInfo = windll.user32.GetIconInfo > GetIconInfo.argtypes = [HICON, POINTER(ICONINFO)] > GetIconInfo.restype = BOOL > GetIconInfo.errcheck = ErrorIfZero > > where as you use > > user32 = ctypes.WinDLL('user32', use_last_error=True) > user32.GetIconInfoExW.errcheck = check_bool > user32.GetIconInfoExW.restype = wintypes.BOOL > user32.GetIconInfoExW.argtypes = ( > wintypes.HICON, # _In_ hIcon > PICONINFOEX,) # _Out_ piconinfoex > > I've checked ctype docu included in python but don't find any hint about your concerns. > May I ask you, do you know additional documents/sites which I can use to get a better > understanding about caching issue? Or did I miss something from used documentation? You haven't missed anything in the documentation. The ctypes docs need work, and some of the examples are bad, if not wrong. For example, the GetModuleHandleA examples incorrectly handle the pointer result because they were never updated for 64-bit Windows. One can't use a Python function as the restype with a C function that returns a pointer because it will be truncated to a C int. Whoever wrote the GetModuleHandleA examples either doesn't know how this feature is implemented in ctypes (probably not, since I think Thomas Heller wrote the example), or doesn't know that a Windows HMODULE is a pointer to the module's base address, or was just writing sloppy code in the era of 32-bit Windows. In this case, look at the CDLL [1] and LibraryLoader [2] classes. Note how CDLL.__getattr__ caches function pointers using setattr(self, name, func). Note how LibraryLoader.__getattr__ caches libraries using setattr(self, name, dll), and how it instantiates the library using self._dlltype(name), with no way to specify use_last_error=True. This has caused real problems for projects such as colorama (fixed) and pyreadline (still broken), and I've seen potential problems in several other projects that naively copy the cdll and windll examples from the docs. It's not their fault. The docs are just bad on this subject. [1]: https://hg.python.org/cpython/file/v3.5.1/Lib/ctypes/__init__.py#l314 [2]: https://hg.python.org/cpython/file/v3.5.1/Lib/ctypes/__init__.py#l410 >> Using __del__ is convenient, but note that you can't >> reuse an instance without manually calling DeleteObject on the >> bitmaps. > > Don't understand this. Isn't this covered by your example in base class? I'm talking about reusing an instance, to avoid the cost of repeated allocation and deallocation. For example: info = ICONINFOEX() for hIcon in hIcons: user32.GetIconInfoExW(hIcon, ctypes.byref(info)) print('fIcon : %d' % info.fIcon) print('wResID : %d' % info.wResID) print('szModName: %s' % info.szModName) gdi32.DeleteObject(info.hbmMask) gdi32.DeleteObject(info.hbmColor) From eryksun at gmail.com Fri May 6 19:39:04 2016 From: eryksun at gmail.com (eryk sun) Date: Fri, 6 May 2016 18:39:04 -0500 Subject: python, ctypes and GetIconInfo issue In-Reply-To: <5dd5db3e-ffa9-4c7b-9385-dc20f85dd45b@googlegroups.com> References: <5dd5db3e-ffa9-4c7b-9385-dc20f85dd45b@googlegroups.com> Message-ID: On Fri, May 6, 2016 at 9:49 AM, wrote: > > In your example you used a base class > and ICONINFO well as ICONINFOEX inherit it. > As the members of ICONINFO are part of ICONINFOEX > couldn't we do something like > > class ICONINFO_BASE(ctypes.Structure): > def __del__(self, ): > if self.hbmMask: > gdi32.DeleteObject(self.hbmMask) > self.hbmMask = None > if self.hbmColor: > gdi32.DeleteObject(self.hbmColor) > self.hbmColor = None > > class ICONINFO(ICONINFO_BASE): > _fields_ = (('fIcon', wintypes.BOOL), > ('xHotspot', wintypes.DWORD), > ('yHotspot', wintypes.DWORD), > ('hbmMask', wintypes.HBITMAP), > ('hbmColor', wintypes.HBITMAP)) > > class ICONINFOEX(ICONINFO): > _fields_ = (('cbSize', wintypes.DWORD), > # ('fIcon', wintypes.BOOL), > # ('xHotspot', wintypes.DWORD), > # ('yHotspot', wintypes.DWORD), > # ('hbmMask', wintypes.HBITMAP), > # ('hbmColor', wintypes.HBITMAP), > ('wResID', wintypes.WORD), > ('szModName', wintypes.WCHAR * MAX_PATH), > ('szResName', wintypes.WCHAR * MAX_PATH)) In this case, cbSize field will be offset after hbmColor: >>> ICONINFOEX.hbmColor.offset 24 >>> ICONINFOEX.cbSize.offset 32 A struct subclass appends its fields to the base class fields. In theory, you can do this in some cases, but in practice I don't recommend it (see below). For example, look at SHARE_INFO_0 [1], SHARE_INFO_1 [2], and SHARE_INFO_2 [3], which are used to query different levels of information about network shares. [1]: https://msdn.microsoft.com/en-us/library/bb525402 [2]: https://msdn.microsoft.com/en-us/library/bb525407 [3]: https://msdn.microsoft.com/en-us/library/bb525408 It can help to maintain a consistent type hierarchy, such as in the following answer that I wrote to list network shares on Windows: http://stackoverflow.com/a/36848031/205580 When ctypes checks the type of a pointer argument, it first checks whether its _type_ is a subclass of the _type_ of the corresponding pointer type in argtypes. If not, it falls back on checking whether the pointer argument itself is an instance of the argtypes pointer type. Similarly, for a byref() argument it checks whether the referent is an instance of the _type_ of the argtypes pointer type. Maintaining a consistent type hierarchy provides type safety without having to tediously cast pointers. However, I don't recommend subclassing to append _fields_ because it has a bug. The code that updates the StgDictObject (i.e. the subclass of dict used by ctypes types for FFI storgage info) for structs and union types doesn't doesn't properly initialize the ffi_type elements array. The length field of the stgdict needs to be the total number of fields, inclusive of all base classes, in order to copy the entire ffi_type elements array from the base class. However, storing the total length in the stgdict's length field would require rewriting the way an instance of a struct is recursively initialized over the base classes. This bug affects passing structs by value. This isn't common (passing unions by value isn't even supported), so I haven't bothered to submit a patch for this. Here's an example crash on 64-bit Linux: test.c: #include typedef struct _data_t { int x, y, z; } data_t; int test(data_t d) { printf("%d, %d, %d\n", d.x, d.y, d.z); return 0; } test.py: from ctypes import * lib = CDLL('./test.so') class A(Structure): _fields_ = (('a', c_int), ('b', c_int), ('c', c_int),) class B(Structure): _fields_ = (('a', c_int),) class C(B): _fields_ = (('b', c_int),) class D(C): _fields_ = (('c', c_int),) print('test A') lib.test(A(42, 84, 168)) print('test D') lib.test(D(42, 84, 168)) output: test A 42, 84, 168 test D Aborted From nospam at dfs.com Fri May 6 19:58:50 2016 From: nospam at dfs.com (DFS) Date: Fri, 6 May 2016 19:58:50 -0400 Subject: A fun python CLI program for all to enjoy! In-Reply-To: References: <7e52b918-2087-f93f-43cb-3411c1cdc881@mrabarnett.plus.com> <572D28CF.3080100@stoneleaf.us> Message-ID: On 5/6/2016 7:29 PM, Ethan Furman wrote: > On 05/06/2016 04:12 PM, DFS wrote: >> On 5/6/2016 4:30 PM, MRAB wrote: > >>> If you don't want to use the 'with' statement, note that closing the >>> file is: >>> >>> f.close() >>> >>> It needs the "()"! >> >> I used close() in 1 place, but close without parens in 2 other places. >> So it works either way. Good catch. > > No, it doesn't. `f.close` simple returns the close function, it doesn't > call it. The "it works" was simply because Python closed the files for > you later. > > Not a big deal in a small program like this, but still a mistake. Yes. Check out the answer by 'unutbu' here: http://stackoverflow.com/questions/1832528/is-close-necessary-when-using-iterator-on-a-python-file-object He says "I...checked /proc/PID/fd for when the file descriptor was closed. It appears that when you break out of the for loop, the file is closed for you." Improper f.close didn't seem to affect any of the files my program wrote - and I checked a lot of them when I was writing the code. Maybe it worked because the last time the file was written to was in a for loop, so I got lucky and the files weren't truncated? Don't know. Did you notice any other gotchas in the program? From python at mrabarnett.plus.com Fri May 6 20:38:44 2016 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 7 May 2016 01:38:44 +0100 Subject: A fun python CLI program for all to enjoy! In-Reply-To: References: <7e52b918-2087-f93f-43cb-3411c1cdc881@mrabarnett.plus.com> <572D28CF.3080100@stoneleaf.us> Message-ID: On 2016-05-07 00:58, DFS wrote: > On 5/6/2016 7:29 PM, Ethan Furman wrote: >> On 05/06/2016 04:12 PM, DFS wrote: >>> On 5/6/2016 4:30 PM, MRAB wrote: >> >>>> If you don't want to use the 'with' statement, note that closing the >>>> file is: >>>> >>>> f.close() >>>> >>>> It needs the "()"! >>> >>> I used close() in 1 place, but close without parens in 2 other places. >>> So it works either way. Good catch. >> >> No, it doesn't. `f.close` simple returns the close function, it doesn't >> call it. The "it works" was simply because Python closed the files for >> you later. >> >> Not a big deal in a small program like this, but still a mistake. > > > Yes. > > Check out the answer by 'unutbu' here: > > http://stackoverflow.com/questions/1832528/is-close-necessary-when-using-iterator-on-a-python-file-object > > He says "I...checked /proc/PID/fd for when the file descriptor was > closed. It appears that when you break out of the for loop, the file is > closed for you." > If you read the comments for that answer, you'll find the explanation. > Improper f.close didn't seem to affect any of the files my program wrote > - and I checked a lot of them when I was writing the code. > > Maybe it worked because the last time the file was written to was in a > for loop, so I got lucky and the files weren't truncated? Don't know. > > Did you notice any other gotchas in the program? > From steve at pearwood.info Fri May 6 23:33:28 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 07 May 2016 13:33:28 +1000 Subject: Python is an Equal Opportunity Programming Language References: Message-ID: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> On Sat, 7 May 2016 06:35 am, beliavsky at aol.com wrote: > This not "equal opportunity". It is a quota system. I must ask, what do you think the phrase "quota system" means? Who is setting and enforcing this quota, and given that only about 1 in 20 Python programmers is a woman, do you think men are seriously missing out on any opportunities? > It's my > impression that in the U.S., Asians are over-represented among programmers > relative to their share of the population and that whites and especially > blacks are under-represented. Should we impose racial quotas on questions > at conferences and call that "equal opportunity" as well? I don't know. Are there systematic social forces that discourage whites or blacks from taking up programming? With an AOL email address, you're probably in the USA, and with an email username like "beliavsky" I'm guessing you're probably Chinese. Nah just kidding, you're probably of Eastern European or Russian ancestry, and probably very white indeed. - Do you feel systematically excluded and biased against because of your skin colour? - Do white-fellas like yourself find yourself repeatedly missing out on opportunities because employers and managers bypass you as soon as they realise you are white? - When you do manage to find a job, do you feel that employers and managers consistently hold you to a higher standard than your Asian colleagues, expecting you to work twice as hard to get half the recognition? - Do you get patronised by your colleagues because you're just a whitey? - Do you find that there is a systematic and repeating assumption that white-fellas like you can't program? Do people review your code with "It's not bad, for a whitey"? - Do you find that even when you are on an hourly rate, not a salary, you consistently get offered lower pay for the same work as your Asian colleagues? - During staff meetings and conferences, do you find that your Asian colleagues form cliques that exclude you, preventing you from establishing the sort of networks that a professional needs? If you can answer "Yes" to four or more of those questions, then perhaps there is a case for something to combat the overwhelming anti-white racism that you're suffering from. -- Steven From rosuav at gmail.com Sat May 7 00:04:51 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 7 May 2016 14:04:51 +1000 Subject: Python is an Equal Opportunity Programming Language In-Reply-To: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sat, May 7, 2016 at 1:33 PM, Steven D'Aprano wrote: > On Sat, 7 May 2016 06:35 am, beliavsky at aol.com wrote: > >> This not "equal opportunity". It is a quota system. > > I must ask, what do you think the phrase "quota system" means? > > Who is setting and enforcing this quota, and given that only about 1 in 20 > Python programmers is a woman, do you think men are seriously missing out > on any opportunities? > The problem with quotas isn't "women don't deserve to be heard" (because they most assuredly do!), but that a restriction can sometimes force awkwardnesses that weren't there to start with. It's unlikely to be an issue at PyCon, but the same problem has come up in other contexts. A great summary comes from the TV show "Yes, Minister" [1], in which the eponymous Minster wishes to promote a woman, and aiming for 25% women in senior positions (a quota, exactly on par with "alternating questions from men and women"). In that case, the "quota-promoted" woman objected, specifically because she didn't want to be part of some 25%, she wanted to go somewhere that would respect her for her accomplishments. So it's possible to disagree with the quota system without disagreeing with the goal it's trying to accomplish (or, conversely, without agreeing with the imbalance that it's trying to address). It's a sensitive matter that has to be handled carefully. In the case of PyCon questions, I fully agree with it; there were enough women present that it wasn't a ridiculous suggestion, and it encourages people to speak up who might otherwise have kept quiet. But just because that worked well, it doesn't mean we should automatically enact quotas everywhere, as some sort of "gender/race/culture imbalance panacea", because it isn't. ChrisA [1] https://en.wikipedia.org/wiki/Equal_Opportunities_(Yes_Minister) From gokoproject at gmail.com Sat May 7 00:25:45 2016 From: gokoproject at gmail.com (John Wong) Date: Sat, 7 May 2016 00:25:45 -0400 Subject: Python is an Equal Opportunity Programming Language In-Reply-To: References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sat, May 7, 2016 at 12:04 AM, Chris Angelico wrote: > > > In the case of PyCon questions, I fully agree with it; there were > enough women present that it wasn't a ridiculous suggestion, and it > encourages people to speak up who might otherwise have kept quiet. But > just because that worked well, it doesn't mean we should automatically > enact quotas everywhere, as some sort of "gender/race/culture > imbalance panacea", because it isn't. > I think it is a good call to ask if non-male attendees would be interested in asking question. I didn't attend those PyCons so I don't know how many male and female attendees lined up awaiting to ask Guido questions. If there were 25 male and 1 female standing in the line, while I do admire Guido (or just about anybody) encouraging more non-male to speak, is it worth asking whether we place pressure on the females attendees if were to say "hey look, we got a lot of male attendees asking, please more female attendees." I totally understand there is a long history of females being treated as inferior (even in America here!), but too much encouragement or too eager to seek more females speaking is almost like saying females are shy and can't speak up without the presence of a heroic voice. I am a male and I am Asian so I am usually regarded as majority in the tech world so I don't always feel underrepresented and can be biased here. Recently I went to some company's website and on the career page I found a banner photo full of white males and maybe 3-4 females in the pictures, holding beers having a great smile posing for a group picture. It could be really genuine, but I felt so uncomfortable immediately because (1) the ratio of male:female is so out balanced, (2) I felt the company was selling the "equal opportunity" sloan too hard. What I am saying is don't try so hard, people will apply job if they want the job, regardless of gender and ethnicity. Similarly, if females attendees want to ask questions, they will. We shouldn't broadcast every single time "we gotta have more females speaking, or more underrepresented people speaking." When I am hanging out with my friends, whether they are male or female, I don't really think of he/she. I think of them as friends, as human being, no need to differentiate whether they are Mexican or Black or Asian. Just human being. Sexual assault laws in some countries are pretty stupid in the sense that female sexual assault offender would receive light punishment compared to female offender. While social and history would justify such law (because again, males historically dominated women), we still treat people inferior by gender and ethnicity. I don't know, this is a sensitive issue. People are either coerced to believe in one kind of response, or perceive as anti-X if given a different kind of response. Thanks. John From me+python at ixokai.io Sat May 7 02:03:41 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Fri, 06 May 2016 23:03:41 -0700 Subject: A fun python CLI program for all to enjoy! In-Reply-To: References: <7e52b918-2087-f93f-43cb-3411c1cdc881@mrabarnett.plus.com> <572D28CF.3080100@stoneleaf.us> Message-ID: <1462601021.570549.600707729.5AB38312@webmail.messagingengine.com> On Fri, May 6, 2016, at 04:58 PM, DFS wrote: > Improper f.close didn't seem to affect any of the files my program wrote > - and I checked a lot of them when I was writing the code. To be clear, its not an "improper" f.close. That command is simply not closing the file. Period. "f.close" is how you get the 'close' function from the 'f' object, and then... you do nothing with it. If you removed "f.close" entirely, you'd get the exact same behavior as you have now. The "f.close" does nothing. That said, in CPython semantics, closing a file explicitly is often not required. CPython is reference-counted. Once the references to an object reaches 0, CPython deletes the object. This is an implementation detail of the CPython and not a guarantee of the Python language itself, which is why explicit close calls are preferred. So while 'f.close' does nothing, CPython might be closing the file *anyways*, and it might work... but that 'might' is hard to reason about without a deeper understanding, so using explicit closing mechanics (either via f.close() or with or something else) is strongly recommended. For example, if you were to do: for item in sequence: f = open(item, 'wb') f.write("blah") It probably works fine. The first time through, 'f' is bound to a file object, and you write to it. The second time through, 'f' is bound to a *new file object*, and the original file object now has 0 references, so is automatically deleted. The last sequence through, f is not closed: the 'for loop' is not a scope which deletes its internal name bindings when its done. So that 'f' will likely remain open until the very end of the current function, which may be an issue for you. Implicit closing actually works in a large number of situations in CPython, but it isn't a good thing to rely on. It only works in simple operations where you aren't accidentally storing a reference somewhere else. You have to keep track of the references in your head to make sure things will get closed at proper times. The 'with' statement clearly defines when resources should be closed, so its preferred (As I see you've adopted from other responses). But its also needed in other Python implementations which might not follow CPython's reference counting scheme. I'm not giving further feedback because MRAB caught everything I thought was an issue. -- Stephen Hansen m e @ i x o k a i . i o From greg.ewing at canterbury.ac.nz Sat May 7 02:24:45 2016 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Sat, 07 May 2016 18:24:45 +1200 Subject: A fun python CLI program for all to enjoy! In-Reply-To: References: <7e52b918-2087-f93f-43cb-3411c1cdc881@mrabarnett.plus.com> <572D28CF.3080100@stoneleaf.us> Message-ID: DFS wrote: > Maybe it worked because the last time the file was written to was in a > for loop, so I got lucky and the files weren't truncated? Don't know. It "works" because CPython disposes of objects as soon as they are not referenced anywhere. Other implementations of Python (e.g. Jython, PyPy) might not do that. -- Greg From anthony at cajuntechie.org Sat May 7 02:36:48 2016 From: anthony at cajuntechie.org (Anthony Papillion) Date: Sat, 07 May 2016 01:36:48 -0500 Subject: Why do these statements evaluate the way they do? Message-ID: <9D4F2568-405C-419B-9B18-7376B34143CD@cajuntechie.org> I'm trying to figure out why the following statements evaluate the way they do and I'm not grasping it for some reason. I'm hoping someone can help me. 40+2 is 42 #evaluates to True But 2**32 is 2**32 #evaluates to False This is an example taken from a Microsoft blog on the topic. They say the reason is because the return is based on identity and not value but, to me, these statements are fairly equal. Can someone clue me in? Anthony -- Sent from my Android device with K-9 Mail. Please excuse my brevity. From greg.ewing at canterbury.ac.nz Sat May 7 02:43:45 2016 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Sat, 07 May 2016 18:43:45 +1200 Subject: Python is an Equal Opportunity Programming Language In-Reply-To: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> Message-ID: Steven D'Aprano wrote: > Who is setting and enforcing this quota, and given that only about 1 in 20 > Python programmers is a woman, do you think men are seriously missing out > on any opportunities? Suppose there are 100 people wanting to ask questions, and there is only time to answer 10 questions. If the 1 in 20 ratio holds, then 5 of those people are women and the other 95 are men. Alternating between men and women means that all of the women get their questions answered, and only 5/95 of the men. So in this example, if you're a woman you have a 100% chance of getting answered, and if you're a man you only have a 5.26% chance. Whether you think this is a good strategy or not, beliavsky is right that it's not "equal". -- Greg From rosuav at gmail.com Sat May 7 02:48:22 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 7 May 2016 16:48:22 +1000 Subject: Why do these statements evaluate the way they do? In-Reply-To: <9D4F2568-405C-419B-9B18-7376B34143CD@cajuntechie.org> References: <9D4F2568-405C-419B-9B18-7376B34143CD@cajuntechie.org> Message-ID: On Sat, May 7, 2016 at 4:36 PM, Anthony Papillion wrote: > I'm trying to figure out why the following statements evaluate the way they do and I'm not grasping it for some reason. I'm hoping someone can help me. > > 40+2 is 42 #evaluates to True > But > 2**32 is 2**32 #evaluates to False > > This is an example taken from a Microsoft blog on the topic. They say the reason is because the return is based on identity and not value but, to me, these statements are fairly equal. Frankly, you shouldn't care. Integers and strings should always be compared for equality ("=="), not identity ("is"). Everything else is an interpreter optimization; what you'll find is that some small integers are cached deep within CPython, so the integer 7 is always the same object. The exact set that's cached varies from version to version of CPython, and other interpreters mightn't do that at all - or might not even have actual in-memory objects for *any* integers in the machine word range, using their values directly. Or anything at all. ChrisA From kevin.p.dwyer at gmail.com Sat May 7 03:04:04 2016 From: kevin.p.dwyer at gmail.com (Kev Dwyer) Date: Sat, 07 May 2016 08:04:04 +0100 Subject: Why do these statements evaluate the way they do? References: <9D4F2568-405C-419B-9B18-7376B34143CD@cajuntechie.org> Message-ID: Anthony Papillion wrote: > I'm trying to figure out why the following statements evaluate the way > they do and I'm not grasping it for some reason. I'm hoping someone can > help me. > > 40+2 is 42 #evaluates to True > But > 2**32 is 2**32 #evaluates to False > > This is an example taken from a Microsoft blog on the topic. They say the > reason is because the return is based on identity and not value but, to > me, these statements are fairly equal. > > Can someone clue me in? > > Anthony The *is* operator tests for identity, that is whether the objects on either side of the operator are the same object. CPython caches ints in the range -5 to 256 as an optimisation, so ints in this range are always the same object, and so the is operator returns True. Outside this range, a new int is created as required, and comparisons using is return False. This can be seen by looking at the id of the ints: Python 3.5.1 (default, Dec 29 2015, 10:53:52) [GCC 4.8.3 20140627 [gcc-4_8-branch revision 212064]] on linux Type "help", "copyright", "credits" or "license" for more information. >>> a = 42 >>> b = 42 >>> a is b True >>> id(a) 9186720 >>> id(b) 9186720 >>> c = 2 ** 32 >>> d = 2 ** 32 >>> c is d False >>> id(c) 140483107705136 >>> id(d) 140483107705168 From rustompmody at gmail.com Sat May 7 03:07:53 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Sat, 7 May 2016 00:07:53 -0700 (PDT) Subject: Python is an Equal Opportunity Programming Language In-Reply-To: References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Saturday, May 7, 2016 at 12:13:59 PM UTC+5:30, Gregory Ewing wrote: > Steven D'Aprano wrote: > > Who is setting and enforcing this quota, and given that only about 1 in 20 > > Python programmers is a woman, do you think men are seriously missing out > > on any opportunities? > > Suppose there are 100 people wanting to ask questions, and > there is only time to answer 10 questions. If the 1 in 20 > ratio holds, then 5 of those people are women and the other > 95 are men. > > Alternating between men and women means that all of the > women get their questions answered, and only 5/95 of the > men. So in this example, if you're a woman you have a 100% > chance of getting answered, and if you're a man you only > have a 5.26% chance. > > Whether you think this is a good strategy or not, > beliavsky is right that it's not "equal". Usually I steer away from these (type of) discussions. However... Trump seems to be winning And with that there are these kind of discussions... Here's a short snippet of an exchange I had with an ex-student of mine: --------------------------------- Student: This may be a pretty controversial statement. But if I'd have voting rights in this country I'd have voted for Trump. You are at-least getting what you are seeing. Everyone else is trying to be politically correct where as this guy doesn't give rat's a** about it. My response: Political correctness is incorrect doesn't mean Political incorrectness is correct :-) ------------------------------- On the question of quotas: Any corrective system that seeks to redress an inquity must BY-DESIGN asymptotically self-destruct. Else we have a problem being 'cured' with a remedy-worse-than-the-evil. IOW: A world of A-oppresses-B is not improved by one of B-oppresses-A. As examples of asymptotic self-destruction: 1. Say there is a quota for education for some kind of minority. Should it be the same from kindergarten to MDs in neuro-surgery? If yes would you go to such a quota-MD when the need arises? 2. Likewise say a quota of say X% seats reserved is put in place. Should this X remain untouched for 5? 10? 20? 50? 100? years As for Guido's Q/A practices: I find it good that he does as he does... As long as it does not become a habit!! From me at ixokai.io Sat May 7 03:49:29 2016 From: me at ixokai.io (Stephen Hansen) Date: Sat, 07 May 2016 00:49:29 -0700 Subject: Why do these statements evaluate the way they do? In-Reply-To: <9D4F2568-405C-419B-9B18-7376B34143CD@cajuntechie.org> References: <9D4F2568-405C-419B-9B18-7376B34143CD@cajuntechie.org> Message-ID: <1462607369.586191.600741793.64CEDF09@webmail.messagingengine.com> On Fri, May 6, 2016, at 11:36 PM, Anthony Papillion wrote: > I'm trying to figure out why the following statements evaluate the way > they do and I'm not grasping it for some reason. I'm hoping someone can > help me. > > 40+2 is 42 #evaluates to True > But > 2**32 is 2**32 #evaluates to False > > This is an example taken from a Microsoft blog on the topic. They say the > reason is because the return is based on identity and not value but, to > me, these statements are fairly equal. Yes, those statements are fairly *equal*. But you are not testing *equality*. The "is" operator is testing *object identity*. >>> a = 123456 >>> b = 123456 >>> a is b False >>> a == b True Equality means, "do these objects equal the same", identity means, "are these specific objects the exact same objects". In the above, a and b are different objects. There's two objects that contain a value of 12345. They are equal (see the == test), but they are different objects. If you write on a piece of paper "12345", and I write on a piece of paper "12345", are those two pieces of paper identical? Of course not. Your paper is in your hand, mine is in mine. These are different pieces of paper. Just look at them: they're clearly not the same thing. That said, if you look at the number they contain, they're the same value. Our two pieces of paper are equivalent, but they're different objects. This might get confusing for you because: >>> a = 123 >>> b = 123 >>> a is b True >>> a == b True This happens because Python caches small integers, so everytime you write '123', it re-uses that same '123' object everyone else is using. It shares those objects as an optimization. The long and short of it is: you should almost never use 'is' for comparing integers (or strings). It doesn't mean what you think it does and isn't useful to you. Compare equality. In general, the only things you should use 'is' for is when comparing to singletons like None, True or False (And consider strongly not comparing against False/True with is, but instead just 'if thing' and if its True, it passes). Otherwise, 'is' should only be used when you're comparing *object identity*. You don't need to do that usually. Only do it when it matters to you that an object with value A might be equal to an object of value B, but you care that they're really different objects. -- Stephen Hansen m e @ i x o k a i . i o From me+python at ixokai.io Sat May 7 04:02:16 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sat, 07 May 2016 01:02:16 -0700 Subject: Python is an Equal Opportunity Programming Language In-Reply-To: References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1462608136.590130.600748113.38B9CEA2@webmail.messagingengine.com> On Fri, May 6, 2016, at 11:43 PM, Gregory Ewing wrote: > Steven D'Aprano wrote: > > Who is setting and enforcing this quota, and given that only about 1 in 20 > > Python programmers is a woman, do you think men are seriously missing out > > on any opportunities? > > Suppose there are 100 people wanting to ask questions, and > there is only time to answer 10 questions. If the 1 in 20 > ratio holds, then 5 of those people are women and the other > 95 are men. > > Alternating between men and women means that all of the > women get their questions answered, and only 5/95 of the > men. So in this example, if you're a woman you have a 100% > chance of getting answered, and if you're a man you only > have a 5.26% chance. > > Whether you think this is a good strategy or not, > beliavsky is right that it's not "equal". This is a pedantically and nonsensical definition of "equal", that ignores the many, many reasons why there are 1 in 20 women in that conference. Its looking at the end effect and ignoring everything that leads up to it, and deciding its instead special rights -- this is the great argument against minorities getting a voice, that their requests for equal *opportunity* are instead *special rights* that diminish the established majority's entrenched power. Those women are dealing with suppression, discrimination and dismissal on multiple levels that leave them in a disenfranchised position. Recognizing those faults and taking corrective action is fundamentally an act in the name of equality. Correcting for inequalities can not, itself, be a purely "equal" task done in pure blindness of the contextual reality of what is going on in the world. -- Stephen Hansen m e @ i x o k a i . i o From rosuav at gmail.com Sat May 7 04:18:07 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 7 May 2016 18:18:07 +1000 Subject: Why do these statements evaluate the way they do? In-Reply-To: <1462607369.586191.600741793.64CEDF09@webmail.messagingengine.com> References: <9D4F2568-405C-419B-9B18-7376B34143CD@cajuntechie.org> <1462607369.586191.600741793.64CEDF09@webmail.messagingengine.com> Message-ID: On Sat, May 7, 2016 at 5:49 PM, Stephen Hansen wrote: > The long and short of it is: you should almost never use 'is' for > comparing integers (or strings). It doesn't mean what you think it does > and isn't useful to you. Compare equality. > > In general, the only things you should use 'is' for is when comparing to > singletons like None, True or False (And consider strongly not comparing > against False/True with is, but instead just 'if thing' and if its True, > it passes). > > Otherwise, 'is' should only be used when you're comparing *object > identity*. You don't need to do that usually. Only do it when it matters > to you that an object with value A might be equal to an object of value > B, but you care that they're really different objects. Equality sometimes is immaterial; there are times when you want to know whether this object you got back is the same one that was sent in, and equality's out of the question. (It's funny how there's this parallel thread about "equal opportunity", in which gender equality is the focus. In this thread, equality really CAN be irrelevant.) For example, to adequately distinguish between "argument omitted" and "some argument was provided", this idiom is commonly used: _SENTINEL = object() def next(iterator, default=_SENTINEL): try: return iterator.__next__() except StopIteration: if default is _SENTINEL: raise return default It makes no difference whatsoever whether the default might compare equal to something; all that matters is: "Is this the sentinel?". And this check is faster and safer than equality, anyway. But with integers and strings, always do equality checks. (Interned strings can safely be identity-checked, which would be faster in the case where they're unequal, but that's a micro-optimization that's safe ONLY if both strings have definitely been interned, so it's better not to bother.) ChrisA From python at ptoye.com Sat May 7 04:49:48 2016 From: python at ptoye.com (Peter Toye) Date: Sat, 7 May 2016 09:49:48 +0100 Subject: Slight problems with python in Windows In-Reply-To: References: <57495789.20160506122201@ptoye.com> Message-ID: <1046065648.20160507094948@ptoye.com> Thanks Zachary. I should have put a newbie warning as I've never used or installed Python before, so some of what you've written goes over my head! Friday, May 6, 2016, 5:17:23 PM, you wrote: > Hi Peter, > On Fri, May 6, 2016 at 6:22 AM, Peter Toye wrote: >> I'm trying to install Python under Windows 7 so that I can use git-review and have found a few niggling issues. >> >> 1) Apparently (according to the git-review pages) pip has a problem with directories with spaces in their names. Python's default installation directory is under Program Files. I agree that this is a pip issue rather than a Python one, but maybe a warning message would help? > I don't believe this is true anymore, I've successfully used pip with > 3.5 installed in Program Files, and also just now in a test venv named > "test venv". Do note that with installation in Program Files, you get > the benefits of the install directory being writable only to > administrators, but also the drawbacks: only administrators can use > pip to install to the global site-packages. You can use either 'pip > --user', or create a venv in a directory writable to you and use it. > Also note that you can't use "pip.exe" to upgrade pip itself since it > can't overwrite "pip.exe" while it's in use; use 'python -m pip' > instead. My information is obviously out of date - I'll try to get it changed in the git-review wiki. >> 2) According to the Programs and Files section of the Windows Control Panel, installing Python also installs something called the Python Launcher. When I try to remove this (so I can reinstall Python in a better directory) is comes up with an error message: > The Python Launcher is a very handy tool called 'py.exe' which makes > it much easier to use more than one version of Python on a Windows > machine. In an all users install, py.exe is installed to C:\Windows > and is thus always available on PATH, so you can invoke Python 3.5 by > calling 'py -3.5' without having to adjust your PATH. The error > message is odd, though, would you mind trying to reproduce it and > opening a bug at bugs.python.org? When I have time.This has already taken longer than it should... >> Error opening installation log file. Verify that the specified log file location exists and is writable. >> >> After reinstalling I now have 2 copies of the launcher.... I hope it doesn't give me any problems. > It shouldn't. The launcher only installs 2 files, py.exe and pyw.exe > (counterpart to pythonw.exe), both in C:\Windows. >> 3) After uninstalling Python the installation directory is still there with a few files in it (possibly connected with the previous issue). Can I just delete it? > Yes, that should be fine. I would guess it's still there due to pip > artifacts in Lib\ and Scripts\. > Hope this helps, It does - thanks. From marko at pacujo.net Sat May 7 04:50:34 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Sat, 07 May 2016 11:50:34 +0300 Subject: Python is an Equal Opportunity Programming Language References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87zis2i751.fsf@elektro.pacujo.net> Gregory Ewing : > Suppose there are 100 people wanting to ask questions, and there is > only time to answer 10 questions. If the 1 in 20 ratio holds, then 5 > of those people are women and the other 95 are men. > > Alternating between men and women means that all of the women get > their questions answered, and only 5/95 of the men. So in this > example, if you're a woman you have a 100% chance of getting answered, > and if you're a man you only have a 5.26% chance. The United States has an "egalitarian" quota system that seeks to promote diversity. By law, at most 7% of green cards can be awarded to citizens of any individual country. So, by this fair principle, in any given year, at most 7% of the green cards can go to citizens of Finland (pop. 5 million) and at most 7% of the green cards can go to citizens of India (pop. 1 billion). Indian and Chinese H1B holders are getting screwed, which is of course the whole objective of the country limits. The US used to have more explicitly worded immigration laws: How this relates to Python? Well, I bet thousands of Asian Python coders in the United States are under the threat of deportation because of country limits. See also: Marko From alister.ware at ntlworld.com Sat May 7 04:51:50 2016 From: alister.ware at ntlworld.com (alister) Date: Sat, 07 May 2016 08:51:50 GMT Subject: A fun python CLI program for all to enjoy! References: <7e52b918-2087-f93f-43cb-3411c1cdc881@mrabarnett.plus.com> <572D28CF.3080100@stoneleaf.us> Message-ID: On Sat, 07 May 2016 18:24:45 +1200, Gregory Ewing wrote: > DFS wrote: >> Maybe it worked because the last time the file was written to was in a >> for loop, so I got lucky and the files weren't truncated? Don't know. > > It "works" because CPython disposes of objects as soon as they are not > referenced anywhere. Other implementations of Python (e.g. Jython, PyPy) > might not do that. to provide an example try the following code in the interactive interpreter >>>f=open('somefile','w') >>print f.write('line 1') None >>>print f.close built-in method close of file object at 0x7fb4c9580660> >>>print f.write('line 2') None >>>print f.close() None >>>print f.write('line 3') ValueError: I/O operation on closed file somefile will contain line 1 line 2 -- manager in the cable duct From python at ptoye.com Sat May 7 05:07:39 2016 From: python at ptoye.com (Peter Toye) Date: Sat, 7 May 2016 10:07:39 +0100 Subject: Slight problems with python in Windows In-Reply-To: References: <57495789.20160506122201@ptoye.com> Message-ID: <598003213.20160507100739@ptoye.com> Zachary, An update - see below. Best regards, Peter mailto:python at ptoye.com www.ptoye.com >> 2) According to the Programs and Files section of the Windows Control Panel, installing Python also installs something called the Python Launcher. When I try to remove this (so I can reinstall Python in a better directory) is comes up with an error message: > The Python Launcher is a very handy tool called 'py.exe' which makes > it much easier to use more than one version of Python on a Windows > machine. In an all users install, py.exe is installed to C:\Windows > and is thus always available on PATH, so you can invoke Python 3.5 by > calling 'py -3.5' without having to adjust your PATH. The error > message is odd, though, would you mind trying to reproduce it and > opening a bug at bugs.python.org? >> Error opening installation log file. Verify that the specified log file location exists and is writable. >> I tried uninstalling the launcher before uninstalling Python and it worked OK. Obviously there's a dependency here. Or the Python uninstaller should also uninstall the launcher first. But one other oddity. For reasons I don't quite follow, the page https://www.python.org/downloads/ downloads the 32-bit version of Python, which is what I first installed without knowing any better. Then I noticed that the installation had (32-bit) in it, poked around a bit and found the 64-bit version. When I installed that the launcher still said (32-bit). Possible bug here, and certainly a very misleading web page? From steve at pearwood.info Sat May 7 06:05:15 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 07 May 2016 20:05:15 +1000 Subject: Why do these statements evaluate the way they do? References: <9D4F2568-405C-419B-9B18-7376B34143CD@cajuntechie.org> Message-ID: <572dbddd$0$1620$c3e8da3$5496439d@news.astraweb.com> On Sat, 7 May 2016 04:36 pm, Anthony Papillion wrote: > I'm trying to figure out why the following statements evaluate the way > they do and I'm not grasping it for some reason. I'm hoping someone can > help me. > > 40+2 is 42 #evaluates to True > But > 2**32 is 2**32 #evaluates to False That is not a guarantee of the language, it is an implementation detail. The only language guarantee is that: 40 + 2 == 42 which MUST be true. But the Python interpreter has a choice here: - it can create *two* distinct int objects with value 42; - or it can cache the int object and re-use it. In the first case, `40 + 2 is 42` will return False. In the second case, it will return True. Remember that `is` does not test for equality, it checks for object identity, namely, "are the two objects one and the same object?" The standard Python interpreter caches "small integers". The definition of "small" depends on the version, but 42 is included, and 2**32 is not. So you may find that 40 + 2 is 42 re-uses the same cached object and returns True, but 2**32 is 2**32 creates a new int object each time and returns False. The lesson here is: (1) ** NEVER ** use `is` to test for equality. The `is` operator is not a funny well of spelling == it is a completely different operator that tests for object identity. (2) You cannot rely on Python object caches, as that is implementation and version dependent. It is an optimization, to speed up some arithmetic at the expense of using a little more memory. (3) Almost the only acceptable use for `is` is to test for the singleton object None. (There are other uses, but they are unusual and testing for None is common. Always write: if obj is None: ... rather than: if obj == None: ... But for (nearly) everything else, always use == equality. -- Steven From greg.ewing at canterbury.ac.nz Sat May 7 07:52:28 2016 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Sat, 07 May 2016 23:52:28 +1200 Subject: Python is an Equal Opportunity Programming Language In-Reply-To: References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <1462608136.590130.600748113.38B9CEA2@webmail.messagingengine.com> Message-ID: Stephen Hansen wrote: > On Fri, May 6, 2016, at 11:43 PM, Gregory Ewing wrote: > >>Whether you think this is a good strategy or not, >>beliavsky is right that it's not "equal". > > This is a pedantically and nonsensical definition of "equal", that > ignores the many, many reasons why there are 1 in 20 women in that > conference. You seem to be saying that if we did take all that into account, and did the arithmetic accordingly, we would conclude that Guido was, after all, treating the men and the women equally. But that doesn't follow. If it's really the case that for every woman at the conference there were another 19 that wanted to go but were prevented simply because they are women, then treating the women who did happen to make it to the conference preferentially does nothing to help the ones who didn't. I suppose on purely arithmetic grounds you could say that out of the total population of potential attendees, men and women ended up with an equal chance of getting a question answered at the conference. But that assumes the goal of getting a question answered is the only one that matters. Missing out on the conference altogether is surely a much bigger injustice! So Guido's affirmative action can at best redress only a small part of the balance. But depending on the circumstances, it could actually make it *worse*. Suppose for some bizarre reason the women who made it to the conference did so because they had red hair. (Maybe the guy taking the conference bookings had a thing for redheads, I don't know.) Now we have the situation where every red-haired female python enthusiast is guaranteed to get their question answered, simply because of the colour of their hair. All the non-red-haired female python enthusiasts might not be very happy about that. Now admittedly that's a pretty far-fetched scenario, but without knowing all the reasons for those 19 out of 20 women being barred from the conference, we can't *know* that there isn't something equally spurious happening. > Correcting for inequalities can not, itself, be a purely "equal" task > done in pure blindness of the contextual reality of what is going on in > the world. I don't think I disagree with that. I tend toward the view that it's not possible to fix those kinds of inequalities by concatenating them with further inequalities. They can only truly be addressed by removing whatever barriers are responsible in the first place. Don't get me wrong, I'm not saying Guido shouldn't have done what he did. But I don't think it makes sense to talk about it in terms of equality, except in a very narrow mathematical way, and then only by making some very handwavey assumptions about the numbers involved. -- Greg From __peter__ at web.de Sat May 7 09:59:00 2016 From: __peter__ at web.de (Peter Otten) Date: Sat, 07 May 2016 15:59 +0200 Subject: A fun python CLI program for all to enjoy! References: Message-ID: DFS wrote: > getAddresses.py > > Scrapes addresses from www.usdirectory.com and stores them in a SQLite > database, or writes them to text files for mailing labels, etc > > Now, just by typing 'fast food Taco Bell 10 db all' you can find > out how many Taco Bells are within 10 miles of you, and store all the > addresses in your own address database. > > No more convoluted Googling, or hitting the 'Next Page' button, or > fumbling with the Yellow Pages... > > Note: the db structure is flat on purpose, and the .csv files aren't > quote delimited. > > Put the program in its own directory. It creates the SQLite database > there, and writes files there, too. > > Reviews of code, bug reports, criticisms, suggestions for improvement, > etc are all welcome. - Avoid module-level code and global variables - Use functions that do one thing and operate on explicitly passed arguments - You have if store == ...: ... sprinkled across your module. You will have to change your code in many places if you want to add another output format. With a linear structure like STORE_FUNCS = { "db": store_in_db, "txt": store_as_text, "csv": store_as_csv, } def main(): args = read_arguments() records = read_records(args) records = unique(records) if args.limit: records = itertools.islice(records, args.limit) STORE_FUNCS[args.storage_format](args, records) if __name__ == "__main__": main() further enhancements will be a lot easier to implement. The main() function avoids accidental uncontrolled globals. If you want one you have to declare it: def main(): global verbose args = read_arguments() verbose = args.verbose ... From zljubisic at gmail.com Sat May 7 10:27:53 2016 From: zljubisic at gmail.com (zljubisic at gmail.com) Date: Sat, 7 May 2016 07:27:53 -0700 (PDT) Subject: How to see html code under the particular web page element? Message-ID: <5eaf19c1-b57d-4771-859b-ae55972234e8@googlegroups.com> Hi, on page: https://hrti.hrt.hr/#/video/show/2203605/trebizat-prica-o-jednoj-vodi-i-jednom-narodu-dokumentarni-film there is a picture and in the middle of the picture there is a text "Prijavite se za gledanje!" If I open the page in firefox and then use menu Tools > Web developer > Page source, there is no such text, but if i right click on the "Prijavite se za gledanje!" and than choose "Inspect element" I get the following html code: What is the difference between firfox Page source and inspect element? How go get html with "Prijavite se za gledanje!" element by using python 3? Regerds. From steve at pearwood.info Sat May 7 11:16:49 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 08 May 2016 01:16:49 +1000 Subject: Python is an Equal Opportunity Programming Language References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> Message-ID: <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> On Sat, 7 May 2016 06:50 pm, Marko Rauhamaa wrote: > The United States has an "egalitarian" quota system that seeks to > promote diversity. By law, at most 7% of green cards can be awarded to > citizens of any individual country. So, by this fair principle, in any > given year, at most 7% of the green cards can go to citizens of Finland > (pop. 5 million) and at most 7% of the green cards can go to citizens of > India (pop. 1 billion). > > Obviously this system is a conspiracy to benefit citizens of Andorra (population 85 thousand), Marshall Islands (pop. 70 thousand), Liechtenstein (pop. 37 thousand), Nauru (pop. 9 thousand) and the Vatican City (pop. 842). > Indian and Chinese H1B holders are getting screwed, which is of course > the whole objective of the country limits. The *whole* objective? You don't think that *part* of the objective may be to ensure that citizens of countries other than India and China can get green cards too? Given that there are only a limited number of green cards available overall, without per country limits it is conceivable that they would all go to people from one or two countries. Perhaps the country limits are also in place, at least in part, to manage the rate at which new immigrants arrive in the country? It's not that country limits act as a permanent barrier to getting a green card. It's a per year limit, and there is a first-come, first-served queue system in place. If an applicant is otherwise eligible for a green card, the country limit will only delay, not prevent, them from getting a green card. E.g. if there are (let's say) a maximum of 1000 green cards available for people from Nauru, and the entire population applies in 2016. Let's assume that they are all eligible under one clause or another (e.g. family ties, employment, refugee status, national interest, etc.). Then the first 1000 applicants will be granted a green card in the first year, followed by the next 1000 the following year, and so on. New applicants go to the back of the queue. Meanwhile, this unexpected flood of immigrants from Nauru have no effect on the chances of Pope Francis being granted a green card, what with the Vatican having its own country limit of 1000 as well. > The US used to have more explicitly worded immigration laws: > > > Yes, many countries had, and still have, overtly racist immigration laws. You should try immigrating into Japan, or Saudi Arabia, and getting citizenship. > How this relates to Python? Well, I bet thousands of Asian Python coders > in the United States are under the threat of deportation because of > country limits. I'm not sure why you think that "H1B holders" are at threat of deportation. So long as they meet the conditions of the work visa, they are entirely entitled to stay and work in the country. There are good arguments for removing the H1B programme. It's used to flood the market with relatively cheap labour made up of people who are less likely to unionise and more likely to put up with bad treatment, and drive wages down for others in the same field. But the inequities of the H1B programme are not caused by the existence of country limits. -- Steven From rosuav at gmail.com Sat May 7 11:38:16 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2016 01:38:16 +1000 Subject: Python is an Equal Opportunity Programming Language In-Reply-To: <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sun, May 8, 2016 at 1:16 AM, Steven D'Aprano wrote: > > Obviously this system is a conspiracy to benefit citizens of Andorra > (population 85 thousand), Marshall Islands (pop. 70 thousand), > Liechtenstein (pop. 37 thousand), Nauru (pop. 9 thousand) and the Vatican > City (pop. 842). The Vatican City is always standing by to mess with your statistics. Never mind about its pop. density - it has the highest pope density in the world, peaking at roughly 2 per square kilometer. The way I'd read the 7% maximum is a requirement on immigration to accept people from a variety of origin countries - at least fifteen unique countries per year, assuming the maximum number of green cards is issued. But immigration laws are a pretty terrible mess the world over, from what I've seen, and I wish countries could drop the whole "but we have to protect ourselves from foreigners" thing. At some point, those "foreigners" become "citizens", and just as worthy of your protection as those who were born here - why fight them off for a while before you welcome them? ChrisA From marko at pacujo.net Sat May 7 11:48:28 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Sat, 07 May 2016 18:48:28 +0300 Subject: Python is an Equal Opportunity Programming Language References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87posxj2cz.fsf@elektro.pacujo.net> Steven D'Aprano : > On Sat, 7 May 2016 06:50 pm, Marko Rauhamaa wrote: >> Indian and Chinese H1B holders are getting screwed, which is of course >> the whole objective of the country limits. > > The *whole* objective? You don't think that *part* of the objective > may be to ensure that citizens of countries other than India and China > can get green cards too? Given that there are only a limited number of > green cards available overall, without per country limits it is > conceivable that they would all go to people from one or two > countries. And? If 40% of the humanity happens to live in China and India, they should have a 40% chance at getting in if you were targeting some kind of ethnic fairness. However, the whole ethnic consideration is silly at best, cruel at worst. You should only look at human beings as individuals. > Perhaps the country limits are also in place, at least in part, to > manage the rate at which new immigrants arrive in the country? No, there's a separate annual total maximum. > It's not that country limits act as a permanent barrier to getting a green > card. It's a per year limit, and there is a first-come, first-served queue > system in place. If an applicant is otherwise eligible for a green card, > the country limit will only delay, not prevent, them from getting a green > card. If you do the math, the wait times grow without bounds. In the end, they may grow longer than your lifetime. >> How this relates to Python? Well, I bet thousands of Asian Python coders >> in the United States are under the threat of deportation because of >> country limits. > > I'm not sure why you think that "H1B holders" are at threat of > deportation. So long as they meet the conditions of the work visa, > they are entirely entitled to stay and work in the country. An H1B is only good for 6 years max. From then on, you'll be out of status. You might still get some kind of provision to stay while you wait for your paperwork to be processed, but as far as I know, you are not allowed to exit the United States while you wait. I was an H1B holder. Because of an amusing bureaucratic adventure (the INS lost my paperwork and would only lift a finger after a court found the United States in contempt), I ran out of status and was subject to deportation for a couple of years. Luckily, I was part of a special legal provision that allowed you to get a green card as long as you had started the process before a particular date. One of the final steps in the process was to file the actual green card application. One of the questions on the application form was: "Are you in the country legally?" I answered, "No," and the form instructed me to add $1,000 to the application fee. My process took five years. > There are good arguments for removing the H1B programme. It's used to > flood the market with relatively cheap labour made up of people who > are less likely to unionise and more likely to put up with bad > treatment, and drive wages down for others in the same field. But the > inequities of the H1B programme are not caused by the existence of > country limits. I don't think I ever ran into that phenomenon. It's of course difficult to measure based on individual experiences alone. What I can tell from the current labor market in Finland is that it is very difficult to find any*body*, let alone any native, who'd *apply* for decent, run-of-the-mill software development jobs. And Finland is in a slump, with constant news of IT layoffs, especially considering the ongoing Nokia/Microsoft implosion. It's really weird. I have never run into a situation where the employer has decided to hire a software developer based on a salary bidding competition. What companies are doing is they are outsourcing whole projects or products to, say, Latvian or Russian companies, but that has little to do with guest worker policies. Marko From marko at pacujo.net Sat May 7 11:57:13 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Sat, 07 May 2016 18:57:13 +0300 Subject: Python is an Equal Opportunity Programming Language References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87lh3lj1ye.fsf@elektro.pacujo.net> Chris Angelico : > But immigration laws are a pretty terrible mess the world over, from > what I've seen, and I wish countries could drop the whole "but we have > to protect ourselves from foreigners" thing. At some point, those > "foreigners" become "citizens", and just as worthy of your protection > as those who were born here - why fight them off for a while before > you welcome them? There's a strong racist undercurrent there for sure, but it's not the whole story. You need to put some impedance to uncontrolled immigration. A functional, enlightened, prosperous democracy is a very recent historical anomaly. You don't want to jeopardize it na?vely. Marko From larry.martell at gmail.com Sat May 7 12:37:43 2016 From: larry.martell at gmail.com (Larry Martell) Date: Sat, 7 May 2016 12:37:43 -0400 Subject: How to see html code under the particular web page element? In-Reply-To: <5eaf19c1-b57d-4771-859b-ae55972234e8@googlegroups.com> References: <5eaf19c1-b57d-4771-859b-ae55972234e8@googlegroups.com> Message-ID: On Sat, May 7, 2016 at 10:27 AM, wrote: > Hi, > > on page: > https://hrti.hrt.hr/#/video/show/2203605/trebizat-prica-o-jednoj-vodi-i-jednom-narodu-dokumentarni-film > > there is a picture and in the middle of the picture there is a text "Prijavite se za gledanje!" > > If I open the page in firefox and then use menu Tools > Web developer > Page source, there is no such text, but if i right click on the "Prijavite se za gledanje!" and than choose "Inspect element" I get the following html code: > > > > What is the difference between firfox Page source and inspect element? > How go get html with "Prijavite se za gledanje!" element by using python 3? There's probably some Angular JS code involved with this. Look at the source (not the page source, the source code in the developer's tools). From nospam at dfs.com Sat May 7 12:51:00 2016 From: nospam at dfs.com (DFS) Date: Sat, 7 May 2016 12:51:00 -0400 Subject: pylint woes Message-ID: This more-anal-than-me program generated almost 2 warnings for every line of code in my program. w t hey? DFS comments +-------------------------+------------+ ------------------------------- |message id |occurrences | +=========================+============+ |mixed-indentation |186 | I always use tab +-------------------------+------------+ |invalid-name |82 | every single variable name?! +-------------------------+------------+ |bad-whitespace |65 | mostly because I line up = signs: var1 = value var10 = value +-------------------------+------------+ |trailing-whitespace |59 | heh! +-------------------------+------------+ |multiple-statements |23 | do this to save lines. Will continue doing it. +-------------------------+------------+ |no-member |5 | "Module 'pyodbc' has no 'connect' member" Yes it does. "Module 'pyodbc' has no 'Error' member" Yes it does. Issue with pylint, or pyodbc? +-------------------------+------------+ |line-too-long |5 | meh +-------------------------+------------+ |wrong-import-order |4 | does it matter? +-------------------------+------------+ |missing-docstring |4 | what's the difference between a docstring and a # comment? +-------------------------+------------+ |superfluous-parens |3 | I like to surround 'or' statments with parens +-------------------------+------------+ |redefined-outer-name |3 | fixed. changed local var names. +-------------------------+------------+ |redefined-builtin |2 | fixed. Was using 'zip' and 'id' +-------------------------+------------+ |multiple-imports |2 | doesn't everyone? +-------------------------+------------+ |consider-using-enumerate |2 | see below [1] +-------------------------+------------+ |bad-builtin |2 | warning because I used filter? +-------------------------+------------+ |unused-import |1 | fixed +-------------------------+------------+ |unnecessary-pass |1 | fixed. left over from Try..Except +-------------------------+------------+ |missing-final-newline |1 | I'm using Notepad++, with EOL Conversion set to 'Windows Format'. How or should I fix this? +-------------------------+------------+ |fixme |1 | a TODO statement +-------------------------+------------+ Global evaluation ----------------- Your code has been rated at -7.64/10 I assume -7.64 is really bad? Has anyone ever in history gotten 10/10 from pylint for a non-trivial program? After fixes and disabling various warnings: "Your code has been rated at 8.37/10" That's about as good as it's gonna get! [1] pylint says "Consider using enumerate instead of iterating with range and len" the offending code is: for j in range(len(list1)): do something with list1[j], list2[j], list3[j], etc. enumeration would be: for j,item in enumerate(list1): do something with list1[j], list2[j], list3[j], etc. Is there an advantage to using enumerate() here? From laurent.pointal at free.fr Sat May 7 12:54:37 2016 From: laurent.pointal at free.fr (Laurent Pointal) Date: Sat, 07 May 2016 18:54:37 +0200 Subject: After a year using Node.js, the prodigal son returns References: <5729b9d8$0$1612$c3e8da3$5496439d@news.astraweb.com> <572C0629.6030509@gmail.com> Message-ID: <572e1dce$0$19764$426a34cc@news.free.fr> Chris Angelico wrote: > On Fri, May 6, 2016 at 11:45 PM, Grant Edwards > wrote: >>> JavaScript is terrible. Really, really bad. And because of that, it >>> has the potential to sweep the world. >> >> If your reasoning is correct, it'll never be able to overtake PHP. >> >> I've never written anything over a hundred or two lines in JavaScript, >> but for small stuff it seems OK -- though as others have noted there >> are some oddly missing batteries that result in use of a lot of small >> external libraries for things that any C, PHP, or Python user would >> have expected to be in the standard library. > > Except that it's pretty easy to switch out PHP for Python, or anything > else. JavaScript is what it is because it's hard to just use a > different language. Maybe Pyhton, using Brython (http://www.brython.info/) (ok, its translated into JavaScript for execution in the web browser? maybe somedays it will be asmsjs) A+ Laurent. From rosuav at gmail.com Sat May 7 13:01:17 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2016 03:01:17 +1000 Subject: pylint woes In-Reply-To: References: Message-ID: On Sun, May 8, 2016 at 2:51 AM, DFS wrote: > [1] > pylint says "Consider using enumerate instead of iterating with range and > len" > > the offending code is: > for j in range(len(list1)): > do something with list1[j], list2[j], list3[j], etc. > > enumeration would be: > for j,item in enumerate(list1): > do something with list1[j], list2[j], list3[j], etc. > > Is there an advantage to using enumerate() here? The suggestion from a human would be to use zip(), or possibly to change your data structures. for item1, item2, item3 in zip(list1, list2, list3): do something with the items ChrisA From random832 at fastmail.com Sat May 7 14:40:56 2016 From: random832 at fastmail.com (Random832) Date: Sat, 07 May 2016 14:40:56 -0400 Subject: Python is an Equal Opportunity Programming Language In-Reply-To: <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1462646456.2677797.601019937.6FC9DC8E@webmail.messagingengine.com> On Sat, May 7, 2016, at 11:16, Steven D'Aprano wrote: > > Indian and Chinese H1B holders are getting screwed, which is of course > > the whole objective of the country limits. > > The *whole* objective? You don't think that *part* of the objective may > be > to ensure that citizens of countries other than India and China can get > green cards too? If not for the quotas, a citizen of some other country would have an equal chance to get a green card as a citizen of India or China. This is a much simpler question than Python convention attendees, since there are *in fact* more Indian or Chinese people in actual existence than citizens of any given other country, which can't be blamed on any form of discrimination. > Perhaps the country limits are also in place, at least in part, to manage > the rate at which new immigrants arrive in the country? That's immaterial, we're not talking about a limit on the total number of visas. From jim at dodgen.us Sat May 7 14:40:56 2016 From: jim at dodgen.us (Jim Dodgen) Date: Sat, 7 May 2016 11:40:56 -0700 Subject: redirecting stdout and stderr to /dev/null Message-ID: I'm new to python but well versed on other languages such as C and Perl I'm have problems redirecting stdout and stderr to /dev/null in a program that does a fork and exec. T found this method googling around and it is quite elegant compared to to the Perl version. So to isolate things I made a much shorter test program and it still is not redirecting. What am I doing wrong? test program test.py ----------------- cut here ------------------- import sys import os f = open(os.devnull, 'w') sys.stdout = f sys.stderr = f os.execl("/bin/ping", "", "-w", "20", "192.168.1.1"); ------------------ cut here ------------------- results when run root at dev:/home/jim# python test.py PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data. 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=1.36 ms 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=1.00 ms 64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=1.01 ms 64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=1.16 ms 64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=1.02 ms ^C --- 192.168.1.1 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4004ms rtt min/avg/max/mdev = 1.002/1.114/1.361/0.136 ms root at dev:/home/jim# *Jim Dodgen* From michael.selik at gmail.com Sat May 7 14:42:33 2016 From: michael.selik at gmail.com (Michael Selik) Date: Sat, 07 May 2016 18:42:33 +0000 Subject: pylint woes In-Reply-To: References: Message-ID: On Sat, May 7, 2016 at 12:56 PM DFS wrote: > |mixed-indentation |186 | I always use tab > Don't mix tabs and spaces. I suggest selecting all lines and using your editor to convert spaces to tabs. Usually there's a feature to "tabify". > +-------------------------+------------+ > |invalid-name |82 | every single variable name?! > Class names should be CamelCase Everything else should be lowercase_with_underscores > +-------------------------+------------+ > |bad-whitespace |65 | mostly because I line up = > signs: > var1 = value > var10 = value > Sure, that's your style. But pylint likes a different style. It's good to use a standard. If it's just you, I suggest conforming to pylint. If you're already on a team, use your team's standard. +-------------------------+------------+ > |trailing-whitespace |59 | heh! > Get rid of it. Save some bytes. > +-------------------------+------------+ > |multiple-statements |23 | do this to save lines. > Will continue doing it. > If you want to share your code with others, you should conform to community standards to make things easier for others to read. Further, if you think the core contributors are expert programmers, you should probably take their advice: "sparse is better than dense". Do your future-self a favor and write one statement per line. Today you find it easy to read. Six months from now you won't. > +-------------------------+------------+ > |no-member |5 | > > "Module 'pyodbc' has no 'connect' member" Yes it does. > "Module 'pyodbc' has no 'Error' member" Yes it does. > > Issue with pylint, or pyodbc? > Not sure. Maybe pyodbc is written in a way that pylint can't see it's connect or Error method/attribute. > +-------------------------+------------+ > |line-too-long |5 | meh > Yeah, I think 80 characters can be somewhat tight. Still, 5 long lines in 200ish lines of code? Sounds like you might be doing too much in those lines or have too many levels of indentation. "Sparse is better than dense" "Flat is better than nested" > +-------------------------+------------+ > |wrong-import-order |4 | does it matter? > No. I think pylint likes to alphabetize. With only 4 imports, it doesn't matter. Still, why not alphabetize? > +-------------------------+------------+ > |missing-docstring |4 | what's the difference between > a docstring and a # comment? > Docstrings are tools for introspection. Many things in Python access the __doc__ attribute to help you. Comments are never seen by module users. > +-------------------------+------------+ > |superfluous-parens |3 | I like to surround 'or' > statments with parens > Ok. But over time you'll get used to not needing them. Edward Tufte says you should have a high "information-to-ink" ratio. > +-------------------------+------------+ > |redefined-outer-name |3 | fixed. changed local var names. > +-------------------------+------------+ > |redefined-builtin |2 | fixed. Was using 'zip' and 'id' > +-------------------------+------------+ > |multiple-imports |2 | doesn't everyone? > Yeah, I do that as well. > +-------------------------+------------+ > |consider-using-enumerate |2 | see below [1] > As Chris explained. > +-------------------------+------------+ > |bad-builtin |2 | warning because I used filter? > I think pylint likes comprehensions better. IMHO filter is OK. If you're using a lambda, change to a comprehension. > +-------------------------+------------+ > |unused-import |1 | fixed > +-------------------------+------------+ > |unnecessary-pass |1 | fixed. left over from > Try..Except > +-------------------------+------------+ > |missing-final-newline |1 | I'm using Notepad++, with > EOL Conversion set to > 'Windows Format'. How > or should I fix this? > Add a few blank lines to the end of your file. > +-------------------------+------------+ > |fixme |1 | a TODO statement > +-------------------------+------------+ > > Global evaluation > ----------------- > Your code has been rated at -7.64/10 > > > > I assume -7.64 is really bad? > > Has anyone ever in history gotten 10/10 from pylint for a non-trivial > program? > I'm certain of it. From pkpearson at nowhere.invalid Sat May 7 14:43:44 2016 From: pkpearson at nowhere.invalid (Peter Pearson) Date: 7 May 2016 18:43:44 GMT Subject: pylint woes References: Message-ID: On Sat, 7 May 2016 12:51:00 -0400, DFS wrote: > This more-anal-than-me program generated almost 2 warnings for every > line of code in my program. w t hey? Thank you for putting a sample of pylint output in front of my eyes; you inspired me to install pylint and try it out. If it teaches me even half as much as it's teaching you, I'll consider it a great blessing. -- To email me, substitute nowhere->runbox, invalid->com. From christopher_reimer at icloud.com Sat May 7 14:52:23 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Sat, 07 May 2016 11:52:23 -0700 Subject: pylint woes In-Reply-To: References: Message-ID: <572E3967.9060206@icloud.com> On 5/7/2016 9:51 AM, DFS wrote: > Has anyone ever in history gotten 10/10 from pylint for a non-trivial > program? I routinely get 10/10 for my code. While pylint isn't perfect and idiosyncratic at times, it's a useful tool to help break bad programming habits. Since I came from a Java background, I had to unlearn everything from Java before I could write Pythonic code. It might help to use an IDE that offers PEP8-compliant code suggestions (I use PyCharm IDE). > That's about as good as it's gonna get! You can do better. You should strive for 10/10 whenever possible, figure out why you fall short and ask for help on the parts that don't make sense. > pylint says "Consider using enumerate instead of iterating with range > and len" > > the offending code is: > for j in range(len(list1)): > do something with list1[j], list2[j], list3[j], etc. This code is reeking with bad habits to be broken. Assigning a throwaway variable to walk the index is unnecessary when Python can do it for you behind the scenes. As Chris A. pointed out in his post, you should use zip() to walk through the values of each list at the same time. Thank you, Chris R. From michael.selik at gmail.com Sat May 7 15:04:21 2016 From: michael.selik at gmail.com (Michael Selik) Date: Sat, 07 May 2016 19:04:21 +0000 Subject: python - handling HTTP requests asynchronously In-Reply-To: References: Message-ID: On Fri, May 6, 2016 at 3:01 AM wrote: > The PDF is generated through an external API. Since currently is generated > on demand, this is handled synchronously via an HTTP request/response. Are you sending the request or are you receiving the request? If you are sending, you can just use threads as you are only doing IO. If you are receiving the requests and generating PDFs, you may want to use subprocesses if the PDF-generation is compute-intensive. > 3) multiprocessing module, with e.g. 10 as workers limit. > multiprocessing.Pool is an easy way to use subprocesses multiprocessing.pool.ThreadPool is an easy way to use threads. It's not well documented, but has the exact same interface as Pool. the goal is to use the API concurrently (e.g. send 10 or 100 http requests > simultaneously, depending on how many concurrent requests the API can > handle). > Sounds like you want to use threads. How does the API let you know you're hitting it too frequently? Perhaps you want to code an exponential backoff and retry wrapper for your API requests. From martin at linux-ip.net Sat May 7 15:16:00 2016 From: martin at linux-ip.net (Martin A. Brown) Date: Sat, 7 May 2016 12:16:00 -0700 Subject: redirecting stdout and stderr to /dev/null In-Reply-To: References: Message-ID: Hello there, >I'm new to python but well versed on other languages such as C and >Perl > >I'm have problems redirecting stdout and stderr to /dev/null in a >program that does a fork and exec. T found this method googling >around and it is quite elegant compared to to the Perl version. > >So to isolate things I made a much shorter test program and it >still is not redirecting. What am I doing wrong? > >test program test.py >----------------- cut here ------------------- >import sys >import os > >f = open(os.devnull, 'w') >sys.stdout = f >sys.stderr = f >os.execl("/bin/ping", "", "-w", "20", "192.168.1.1"); >------------------ cut here ------------------- Think about the file descriptors. Unix doesn't care what the name is, rather that the process inherits the FDs from the parent. So, your solution might need to be a bit more complicated to achieve what you desire. Run the following to see what I mean. realstdout = sys.stdout realstderr = sys.stderr f = open(os.devnull, 'w') sys.stdout = f sys.stderr = f print("realstdout FD: %d" % (realstdout.fileno(),), file=realstdout) print("realstderr FD: %d" % (realstderr.fileno(),), file=realstdout) print("sys.stdout FD: %d" % (sys.stdout.fileno(),), file=realstdout) print("sys.stderr FD: %d" % (sys.stderr.fileno(),), file=realstdout) That should produce output that looks like this: realstdout FD: 1 realstderr FD: 2 sys.stdout FD: 3 sys.stderr FD: 3 I hope that's a good hint... I like the idea of simply calling the next program using one of the exec() variants, but you'll have to adjust the file descriptors, rather than just the names used by Python. If you don't need to exec(), but just run a child, then here's the next hint (this is for Python 3.5): import subprocess cmd = ["ping", "-w", "20", "192.168.1.1"] devnull = subprocess.DEVNULL proc = subprocess.run(cmd, stdout=devnull, stderr=devnull) proc.check_returncode() (By the way, your "ping" command looked like it had an empty token in the second arg position. Looked weird to me, so I removed it in my examples.) For subprocess.run, see: https://docs.python.org/3/library/subprocess.html#subprocess.run For earlier Python versions without run(), you can use Popen(): import subprocess cmd = ["/bin/ping", "-w", "20", "192.168.1.1"] devnull = subprocess.DEVNULL proc = subprocess.Popen(cmd, stdout=devnull, stderr=devnull) retcode = proc.wait() if retcode != 0: raise FlamingHorribleDeath You will have to define FlamingHorribleDeath or figure out what you want to do in the event of the various different types of failure....if you don't then, you'll just see this: NameError: name 'FlamingHorribleDeath' is not defined Good luck, -Martin -- Martin A. Brown http://linux-ip.net/ From christopher_reimer at icloud.com Sat May 7 15:17:39 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Sat, 07 May 2016 12:17:39 -0700 Subject: Pylint prefers list comprehension over filter... In-Reply-To: <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> Message-ID: <572E3F53.5010703@icloud.com> On 5/5/2016 6:37 PM, Stephen Hansen wrote: > On Thu, May 5, 2016, at 06:26 PM, Christopher Reimer wrote: >> Which is one is correct (Pythonic)? Or does it matter? > First, pylint is somewhat opinionated, and its default options shouldn't > be taken as gospel. There's no correct: filter is fine. Since the code I'm working on is resume fodder (i.e., "Yes, I code in Python! Check out my chess engine code on GitHub!"), I want it to be as Pythonic and PEP8-compliant as possible. That includes scoring 10/10 with pylint. Never know when an asshat hiring manager would reject my resume out of hand because my code fell short with pylint. For my purposes, I'm using the list comprehension over filter to keep pylint happy. Thank you, Chris R. From me+python at ixokai.io Sat May 7 15:21:19 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sat, 07 May 2016 12:21:19 -0700 Subject: pylint woes In-Reply-To: References: Message-ID: <1462648879.705656.601038593.103230FB@webmail.messagingengine.com> Pylint is very opinionated. Feel free to adjust its configuration to suit your opinions of style. In particular, several of these might be related to PEP8 style issues. On Sat, May 7, 2016, at 09:51 AM, DFS wrote: > DFS comments > +-------------------------+------------+ ------------------------------- > |message id |occurrences | > +=========================+============+ > |mixed-indentation |186 | I always use tab And yet, it appears there's some space indentation in there. In Notepad++ enable View->Show Symbol->Show White Space and Tab and Show Indent Guide. > +-------------------------+------------+ > |invalid-name |82 | every single variable name?! It probably defaults to PEP8 names, which are variables_like_this, not variablesLikeThis. > +-------------------------+------------+ > |bad-whitespace |65 | mostly because I line up = > signs: > var1 = value > var10 = value Yeah and PEP8 says don't do that. Adjust the configuration of pylint if you want. > +-------------------------+------------+ > |multiple-statements |23 | do this to save lines. > Will continue doing it. This you really shouldn't do, imho. Saving lines is not a virtue, readability is -- dense code is by definition less readable. > +-------------------------+------------+ > |no-member |5 | > > "Module 'pyodbc' has no 'connect' member" Yes it does. > "Module 'pyodbc' has no 'Error' member" Yes it does. > > Issue with pylint, or pyodbc? Pylint. > +-------------------------+------------+ > |line-too-long |5 | meh I'm largely meh on this too. But again its a PEP8 thing. > +-------------------------+------------+ > |wrong-import-order |4 | does it matter? Its useful to have a standard so you can glance and tell what's what and from where, but what that standard is, is debatable. > +-------------------------+------------+ > |missing-docstring |4 | what's the difference between > a docstring and a # comment? A docstring is a docstring, a comment is a comment. Google python docstrings :) Python prefers files to have a docstring on top, and functions beneath their definition. Comments should be used as little as possible, as they must be maintained: an incorrect comment is worse then no comment. Go for clear code that doesn't *need* commenting. > +-------------------------+------------+ > |superfluous-parens |3 | I like to surround 'or' > statments with parens Why? > +-------------------------+------------+ > |multiple-imports |2 | doesn't everyone? I don't actually know what its complaining at. > +-------------------------+------------+ > |bad-builtin |2 | warning because I used filter? Don't know what its complaining at about here either. > +-------------------------+------------+ > |missing-final-newline |1 | I'm using Notepad++, with > EOL Conversion set to > 'Windows Format'. How > or should I fix this? Doesn't have anything to do with it. Just scroll to the bottom and press enter. It wants to end on a newline, not code. > Global evaluation > ----------------- > Your code has been rated at -7.64/10 > > I assume -7.64 is really bad? > > Has anyone ever in history gotten 10/10 from pylint for a non-trivial > program? No clue, I don't use pylint at all. > [1] > pylint says "Consider using enumerate instead of iterating with range > and len" > > the offending code is: > for j in range(len(list1)): > do something with list1[j], list2[j], list3[j], etc. > > enumeration would be: > for j,item in enumerate(list1): > do something with list1[j], list2[j], list3[j], etc. > > Is there an advantage to using enumerate() here? Its cleaner, easier to read. In Python 2 where range() returns a list, its faster. (In python2, xrange returns a lazy evaluating range) Use the tools Python gives you. Why reinvent enumerate when its built in? -- Stephen Hansen m e @ i x o k a i . i o From me at ixokai.io Sat May 7 15:23:19 2016 From: me at ixokai.io (Stephen Hansen) Date: Sat, 07 May 2016 12:23:19 -0700 Subject: pylint woes In-Reply-To: <572E3967.9060206@icloud.com> References: <572E3967.9060206@icloud.com> Message-ID: <1462648999.705771.601040345.069F0886@webmail.messagingengine.com> On Sat, May 7, 2016, at 11:52 AM, Christopher Reimer wrote: > You can do better. You should strive for 10/10 whenever possible, > figure out why you fall short and ask for help on the parts that don't > make sense. I think this is giving far too much weight to pylint's opinion on what is "good" or "bad" programming habits. -- Stephen Hansen m e @ i x o k a i . i o From zljubisic at gmail.com Sat May 7 15:27:21 2016 From: zljubisic at gmail.com (zljubisic at gmail.com) Date: Sat, 7 May 2016 12:27:21 -0700 (PDT) Subject: How to see html code under the particular web page element? In-Reply-To: References: <5eaf19c1-b57d-4771-859b-ae55972234e8@googlegroups.com> Message-ID: <11a08d8e-9960-4c4f-a95c-254f5c0abd33@googlegroups.com> > There's probably some Angular JS code involved with this. Look at the > source (not the page source, the source code in the developer's > tools). Is it possible to get that code using python? From christopher_reimer at icloud.com Sat May 7 15:37:20 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Sat, 07 May 2016 12:37:20 -0700 Subject: Pylint prefers list comprehension over filter... In-Reply-To: <1462503446.248479.599683745.79791DAC@webmail.messagingengine.com> References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <1462503446.248479.599683745.79791DAC@webmail.messagingengine.com> Message-ID: <572E43F0.6060501@icloud.com> On 5/5/2016 7:57 PM, Stephen Hansen wrote: > On Thu, May 5, 2016, at 07:46 PM, Dan Sommers wrote: >> On Thu, 05 May 2016 18:37:11 -0700, Stephen Hansen wrote: >> >>> ''.join(x for x in string if x.isupper()) >>> The difference is, both filter and your list comprehension *build a >>> list* which is not needed, and wasteful. The above skips building a >>> list, instead returning a generator ... >> filter used to build a list, but now it doesn't (where "used to" means >> Python 2.7 and "now" means Python 3.5; I'm too lazy to track down the >> exact point(s) at which it changed): > Oh, didn't know that. Then again the OP was converting the output of > filter *into* a list, which wasted a list either way. My line of code was something I copied off the Internet and modified it until it did I exactly what I wanted it to do. That means that the many Python 2 code examples available on the Internet are using a redundant list operation with Python 3. Isn't that a "smell" that pylint should pick up on? Thank you, Chris R. From tjreedy at udel.edu Sat May 7 15:40:23 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 7 May 2016 15:40:23 -0400 Subject: pylint woes In-Reply-To: References: Message-ID: On 5/7/2016 12:51 PM, DFS wrote: > This more-anal-than-me program generated almost 2 warnings for every > line of code in my program. w t hey? If you don't like it, why do you use it? I suppose the answer is that it did find a few things to check. You might be happier with pychecker, which is much less aggressive. I believe will find the things you did fix. > DFS comments > +-------------------------+------------+ ------------------------------- > |message id |occurrences | > +=========================+============+ > |mixed-indentation |186 | I always use tab > +-------------------------+------------+ > |invalid-name |82 | every single variable name?! I would need examples to comment. > +-------------------------+------------+ > |trailing-whitespace |59 | heh! Any code editor should have a command to fix this. IDLE: Format => strip trailing whitespace Notepad++: Macro => trim trailing and save, Alt-Shift-S others ... > +-------------------------+------------+ > |no-member |5 | > > "Module 'pyodbc' has no 'connect' member" Yes it does. > "Module 'pyodbc' has no 'Error' member" Yes it does. > > Issue with pylint, or pyodbc? Worth looking into. Could be a bug somewhere. But I don't have pyodbc installed. > +-------------------------+------------+ > |line-too-long |5 | meh For following the PEP guideline when patching CPython, this is helpful. > +-------------------------+------------+ > |wrong-import-order |4 | does it matter? Consistency in imports ultimately makes easier reading. Many idlelib files use this order: stdlib modules other than tkinter and idlelib (alphabetically); tkinter (tkinter first, then submodules); idlelib (alphabetically). When I edit files, I sometimes reorder imports to conform. > +-------------------------+------------+ > |missing-docstring |4 | what's the difference between > a docstring and a # comment? # Comments only appear in the source '''Docstrings are copied to the compiled code object, are interactively accessible, and are used for help(ojb) output.''' > +-------------------------+------------+ > |superfluous-parens |3 | I like to surround 'or' > statments with parens I would need examples to comment > +-------------------------+------------+ > |bad-builtin |2 | warning because I used filter? If they are still doing this in the latest release, it is an arrogance and inconsistency bug on their part. Disable this check. > +-------------------------+------------+ > |missing-final-newline |1 | I'm using Notepad++, with > EOL Conversion set to > 'Windows Format'. That says to replace final '\n' with '\r\n'. It does not affect a missing final newline ;-) How or should I fix this? Fix by hitting 'Enter' at the end of the last line. Should you? I think it a good habit. > After fixes and disabling various warnings: > "Your code has been rated at 8.37/10" Being able to customize pylint by turning off warnings is its saving feature. -- Terry Jan Reedy From christopher_reimer at icloud.com Sat May 7 15:43:36 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Sat, 07 May 2016 12:43:36 -0700 Subject: pylint woes In-Reply-To: <1462648999.705771.601040345.069F0886@webmail.messagingengine.com> References: <572E3967.9060206@icloud.com> <1462648999.705771.601040345.069F0886@webmail.messagingengine.com> Message-ID: <572E4568.4080009@icloud.com> On 5/7/2016 12:23 PM, Stephen Hansen wrote: > On Sat, May 7, 2016, at 11:52 AM, Christopher Reimer wrote: >> You can do better. You should strive for 10/10 whenever possible, >> figure out why you fall short and ask for help on the parts that don't >> make sense. > I think this is giving far too much weight to pylint's opinion on what > is "good" or "bad" programming habits. I forgot to add the warning, "Use pylint with a dash of salt on a lemon slice and a shot of tequila." :) Thank you, Chris R. From rgacote at appropriatesolutions.com Sat May 7 15:52:51 2016 From: rgacote at appropriatesolutions.com (Ray Cote) Date: Sat, 7 May 2016 15:52:51 -0400 Subject: pylint woes In-Reply-To: <572E3967.9060206@icloud.com> References: <572E3967.9060206@icloud.com> Message-ID: On Sat, May 7, 2016 at 2:52 PM, Christopher Reimer < christopher_reimer at icloud.com> wrote: > On 5/7/2016 9:51 AM, DFS wrote: > >> Has anyone ever in history gotten 10/10 from pylint for a non-trivial >> program? >> > > I routinely get 10/10 for my code. While pylint isn't perfect and > idiosyncratic at times, it's a useful tool to help break bad programming > habits. > I?m impressed with 10/10. My approach is to ensure flake8 (a combination of pyflakes and pep8 checking) does not report any warnings and then run pyLint as a final check. Code usually ends up in the 9.0 to 9.5 range, sometimes a bit higher. Also find it useful to add some additional short names we use to the allowed names list. Biggest issue I have with pyLint is that it complains when function parameters are indented twice vs. once. pyFlakes likes the twice. Example: def function_name( parm_1, long_parm_name, ?. end_of_long_list_of params) parm_1 = long_parm_name ?Ray -- Raymond Cote, President voice: +1.603.924.6079 email: rgacote at AppropriateSolutions.com skype: ray.cote From christopher_reimer at icloud.com Sat May 7 16:20:01 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Sat, 07 May 2016 13:20:01 -0700 Subject: pylint woes In-Reply-To: References: <572E3967.9060206@icloud.com> Message-ID: <572E4DF1.8000504@icloud.com> On 5/7/2016 12:52 PM, Ray Cote wrote: > I?m impressed with 10/10. > My approach is to ensure flake8 (a combination of pyflakes and pep8 > checking) does not report any warnings and then run pyLint as a final > check. I just installed pyflakes and ran it against my 10/10 files. It's not complaining about anything. So I ran it against my unit tests that I'm still writing, haven't cleaned up and checked against pylint. I got dinged for using a star import on a file with a common variables, which was an easy fix. Thank you, Chris R. From marko at pacujo.net Sat May 7 16:31:19 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Sat, 07 May 2016 23:31:19 +0300 Subject: Pylint prefers list comprehension over filter... References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> Message-ID: <87vb2phap4.fsf@elektro.pacujo.net> Christopher Reimer : > Never know when an asshat hiring manager would reject my resume out of > hand because my code fell short with pylint. Remember that it's not only the company checking you out but also you checking the company out. Would you want to work for an asshat hiring manager? Of course, you might not have the luxury of being picky. Marko From christopher_reimer at icloud.com Sat May 7 16:40:33 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Sat, 07 May 2016 13:40:33 -0700 Subject: python chess engines In-Reply-To: References: Message-ID: <572E52C1.7010303@icloud.com> On 5/3/2016 10:13 PM, DFS wrote: > Wanted to start a new thread, rather than use the 'motivated' thread. > > Can you play your game at the console? Nope. Only displays the board on the console. An early version had the forward movement for pawns implemented. > The way I think about a chess engine is it doesn't even display a > board. It accepts a move as input, records the move, analyzes the > positions after the move, and returns the next move. My code has display and engine as separate classes with a main loop coordinating things between the two. The main loop requests the board state (dict) from the engine class and passes that to show board on the display class. > Here's the UCI protocol. > http://download.shredderchess.com/div/uci.zip Very interesting. Something to add to my research notes. I'll muddle through with my code and let it grow organically for now. If I decide to write a chess engine that implements the UCI protocol, I'll start over with a clean slate. Thank you, Chris R. From rosuav at gmail.com Sat May 7 17:22:31 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2016 07:22:31 +1000 Subject: Pylint prefers list comprehension over filter... In-Reply-To: <572E3F53.5010703@icloud.com> References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> Message-ID: On Sun, May 8, 2016 at 5:17 AM, Christopher Reimer wrote: > On 5/5/2016 6:37 PM, Stephen Hansen wrote: >> >> On Thu, May 5, 2016, at 06:26 PM, Christopher Reimer wrote: >>> >>> Which is one is correct (Pythonic)? Or does it matter? >> >> First, pylint is somewhat opinionated, and its default options shouldn't >> be taken as gospel. There's no correct: filter is fine. > > > Since the code I'm working on is resume fodder (i.e., "Yes, I code in > Python! Check out my chess engine code on GitHub!"), I want it to be as > Pythonic and PEP8-compliant as possible. That includes scoring 10/10 with > pylint. Never know when an asshat hiring manager would reject my resume out > of hand because my code fell short with pylint. > > For my purposes, I'm using the list comprehension over filter to keep pylint > happy. Wrong thinking. Make it Pythonic - but don't concern yourself with pylint's final score. Read pylint's output and learn from it, but don't treat a 10/10 score as the ultimate in ratings, because it just isn't. Also, be sure you read this part of PEP 8: https://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds ChrisA From nhilterbrand at gmail.com Sat May 7 17:34:53 2016 From: nhilterbrand at gmail.com (Nathan Hilterbrand) Date: Sat, 7 May 2016 17:34:53 -0400 Subject: Pylint prefers list comprehension over filter... In-Reply-To: References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> Message-ID: <572E5F7D.8050604@gmail.com> On 05/07/2016 05:22 PM, Chris Angelico wrote: > On Sun, May 8, 2016 at 5:17 AM, Christopher Reimer > wrote: >> >> pylint. Never know when an asshat hiring manager would reject my resume out >> of hand because my code fell short with pylint. >> I see that as a good motivation to make sure your code was good, clean, and definitely did fall a bit short with pylint. I have worked for asshat managers before. It is something to be avoided. Nathan From me at ixokai.io Sat May 7 17:35:50 2016 From: me at ixokai.io (Stephen Hansen) Date: Sat, 07 May 2016 14:35:50 -0700 Subject: Pylint prefers list comprehension over filter... In-Reply-To: <572E3F53.5010703@icloud.com> References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> Message-ID: <1462656950.727725.601099753.30D90B95@webmail.messagingengine.com> On Sat, May 7, 2016, at 12:17 PM, Christopher Reimer wrote: > On 5/5/2016 6:37 PM, Stephen Hansen wrote: > > On Thu, May 5, 2016, at 06:26 PM, Christopher Reimer wrote: > >> Which is one is correct (Pythonic)? Or does it matter? > > First, pylint is somewhat opinionated, and its default options shouldn't > > be taken as gospel. There's no correct: filter is fine. > > Since the code I'm working on is resume fodder (i.e., "Yes, I code in > Python! Check out my chess engine code on GitHub!"), I want it to be as > Pythonic and PEP8-compliant as possible. That includes scoring 10/10 > with pylint. Never know when an asshat hiring manager would reject my > resume out of hand because my code fell short with pylint. > > For my purposes, I'm using the list comprehension over filter to keep > pylint happy. Bear in mind, when I say, "Pylint is opinionated", I mean the tool -- especially in its default configuration -- has its own opinion of what is good style, and *I think its wrong on a number of points*. Its fine to use, but I'd read over PEP8 (the document, not the tool) and apply style guide recommendations thoughtfully, not mechanically. -- Stephen Hansen m e @ i x o k a i . i o From christopher_reimer at icloud.com Sat May 7 17:36:59 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Sat, 07 May 2016 14:36:59 -0700 Subject: Pylint prefers list comprehension over filter... In-Reply-To: <87vb2phap4.fsf@elektro.pacujo.net> References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> <87vb2phap4.fsf@elektro.pacujo.net> Message-ID: <572E5FFB.5050306@icloud.com> On 5/7/2016 1:31 PM, Marko Rauhamaa wrote: > Christopher Reimer : >> Never know when an asshat hiring manager would reject my resume out of >> hand because my code fell short with pylint. > Remember that it's not only the company checking you out but also you > checking the company out. > > Would you want to work for an asshat hiring manager? > > Of course, you might not have the luxury of being picky. Most asshat managers I've dealt with often say to me, "Here's a mole hole, go fix it." I look at the mole hole (little problem), look at the mountain (big problem) in the distant, and jump over the mole hole to start climbing the mountain. After I reduce the mountain to a mole hole, the asshat manager gets promoted and I'm looking for a job again. Thank you, Chris R. From lordluke80 at gmail.com Sat May 7 17:42:03 2016 From: lordluke80 at gmail.com (lordluke80 at gmail.com) Date: Sat, 7 May 2016 14:42:03 -0700 (PDT) Subject: python - handling HTTP requests asynchronously In-Reply-To: References: Message-ID: <59a499d5-13e1-4092-895f-34aca0827525@googlegroups.com> Il giorno sabato 7 maggio 2016 21:04:47 UTC+2, Michael Selik ha scritto: > On Fri, May 6, 2016 at 3:01 AM wrote: > > > The PDF is generated through an external API. Since currently is generated > > on demand, this is handled synchronously via an HTTP request/response. > > > Are you sending the request or are you receiving the request? > If you are sending, you can just use threads as you are only doing IO. > If you are receiving the requests and generating PDFs, you may want to use > subprocesses if the PDF-generation is compute-intensive. > > > > 3) multiprocessing module, with e.g. 10 as workers limit. > > > > multiprocessing.Pool is an easy way to use subprocesses > multiprocessing.pool.ThreadPool is an easy way to use threads. It's not > well documented, but has the exact same interface as Pool. > > the goal is to use the API concurrently (e.g. send 10 or 100 http requests > > simultaneously, depending on how many concurrent requests the API can > > handle). > > > > Sounds like you want to use threads. How does the API let you know you're > hitting it too frequently? Perhaps you want to code an exponential backoff > and retry wrapper for your API requests. Thanks for the reply. Currently the django view that does the job does three http request: - the first does a POST and send the payload used during the PDF rendering, the response contains a url to check the pdf generation progress; - the second loop over that url, checking the progress of pdf generation. Once the response contains the keyword 'status': 'complete', then it give also a url for the file retrieval; - the third one is a GET to retrieve the file, the reponse contains the binary content of the file, then this content is read and wrapped as attachment of a django http response, and then returned to the user. the goal is to reuse the existing code as much as possible, possibly doing concurrently, and saving the file instead on a folder. From rosuav at gmail.com Sat May 7 17:46:17 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2016 07:46:17 +1000 Subject: Pylint prefers list comprehension over filter... In-Reply-To: <1462656950.727725.601099753.30D90B95@webmail.messagingengine.com> References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> <1462656950.727725.601099753.30D90B95@webmail.messagingengine.com> Message-ID: On Sun, May 8, 2016 at 7:35 AM, Stephen Hansen wrote: > On Sat, May 7, 2016, at 12:17 PM, Christopher Reimer wrote: >> On 5/5/2016 6:37 PM, Stephen Hansen wrote: >> > On Thu, May 5, 2016, at 06:26 PM, Christopher Reimer wrote: >> >> Which is one is correct (Pythonic)? Or does it matter? >> > First, pylint is somewhat opinionated, and its default options shouldn't >> > be taken as gospel. There's no correct: filter is fine. >> >> Since the code I'm working on is resume fodder (i.e., "Yes, I code in >> Python! Check out my chess engine code on GitHub!"), I want it to be as >> Pythonic and PEP8-compliant as possible. That includes scoring 10/10 >> with pylint. Never know when an asshat hiring manager would reject my >> resume out of hand because my code fell short with pylint. >> >> For my purposes, I'm using the list comprehension over filter to keep >> pylint happy. > > Bear in mind, when I say, "Pylint is opinionated", I mean the tool -- > especially in its default configuration -- has its own opinion of what > is good style, and *I think its wrong on a number of points*. Agreed. The OP has already run into this, and disabled some checks; the logical next step is to stop treating a 10/10 score as mandatory. > Its fine to use, but I'd read over PEP8 (the document, not the tool) and > apply style guide recommendations thoughtfully, not mechanically. Definitely. It's times like this that I'm really annoyed by the abuse of the name "pep8" to mean the tool, especially since it's not perfectly matched to the document (notably, it can't encode the flexibilities, so it comes out as rigid). Tools like this can be incredibly useful. But they are TOOLS, and the code belongs to you. They are not your masters. ChrisA From rosuav at gmail.com Sat May 7 17:56:37 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2016 07:56:37 +1000 Subject: pylint woes In-Reply-To: References: Message-ID: On Sun, May 8, 2016 at 4:42 AM, Michael Selik wrote: > >> +-------------------------+------------+ >> |line-too-long |5 | meh >> > > Yeah, I think 80 characters can be somewhat tight. Still, 5 long lines in > 200ish lines of code? Sounds like you might be doing too much in those > lines or have too many levels of indentation. > "Sparse is better than dense" > "Flat is better than nested" Others have commented on this, but I'll weigh in with one point that hasn't been mentioned yet. A lot of tools will complain when you exceed 80 (or 79) characters per line; but it depends somewhat on *how far* you exceeded it. Some people opt instead for a 100-character limit, or even 120, but most programmers agree that a 200-character line (or more!) is too long. So if this is complaining about five lines out of your entire program that just snuck over the 80-character limit (eg 86 characters long), it's not a concern, and my recommendation would be to relax the restriction. And if those few lines are ginormous hunks of data (static list initialization, or something), you might consider dumping them out to external files rather than wrapping them into big code blocks. But if they're truly long lines of code, wrap or split them. ChrisA From mymyxin at gmail.com Sat May 7 17:58:23 2016 From: mymyxin at gmail.com (mymyxin at gmail.com) Date: Sat, 7 May 2016 14:58:23 -0700 (PDT) Subject: python, ctypes and GetIconInfo issue In-Reply-To: References: <5dd5db3e-ffa9-4c7b-9385-dc20f85dd45b@googlegroups.com> Message-ID: Hello eryk sun, thank you very much for your help and detailed answers. With the provided links and useful information I should be able to get a better understanding how ctypes works internally. Thank you Hubert From christopher_reimer at icloud.com Sat May 7 18:01:22 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Sat, 07 May 2016 15:01:22 -0700 Subject: Pylint prefers list comprehension over filter... In-Reply-To: References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> Message-ID: <572E65B2.4010305@icloud.com> On 5/7/2016 2:22 PM, Chris Angelico wrote: > On Sun, May 8, 2016 at 5:17 AM, Christopher Reimer > wrote: >> Since the code I'm working on is resume fodder (i.e., "Yes, I code in >> Python! Check out my chess engine code on GitHub!"), I want it to be >> as Pythonic and PEP8-compliant as possible. That includes scoring >> 10/10 with pylint. Never know when an asshat hiring manager would >> reject my resume out of hand because my code fell short with pylint. >> For my purposes, I'm using the list comprehension over filter to keep >> pylint happy. > Wrong thinking. Make it Pythonic - but don't concern yourself with > pylint's final score. Read pylint's output and learn from it, but > don't treat a 10/10 score as the ultimate in ratings, because it just > isn't. I agree with that in principle. But... > Also, be sure you read this part of PEP 8: > > https://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds Recruiters and hiring managers *are* hobgoblins with with little minds. And definitely not PEP8-complaint. :) When I was out of work for two years (2009-10), underemployed for six months (working 20 hours per month), and filed for Chapter Seven bankruptcy in 2011, I only had 20 job interviews during that time. That was a learning experience. I did everything possible to present myself and my resume as perfectly as possible. When I had another bout of unemployment that lasted eight months (2013-14), I had 60 job interviews and three job offers to pick from at the end. Of course, that was for IT support contracts. Maybe programming jobs will have fewer hobgoblins. Thank you, Chris R. From Radek1 at holych.org Sat May 7 18:04:36 2016 From: Radek1 at holych.org (=?UTF-8?Q?Radek_Hol=C3=BD?=) Date: Sun, 8 May 2016 00:04:36 +0200 Subject: Pylint prefers list comprehension over filter... In-Reply-To: <572E3F53.5010703@icloud.com> References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> Message-ID: 2016-05-07 21:17 GMT+02:00 Christopher Reimer : > On 5/5/2016 6:37 PM, Stephen Hansen wrote: > >> On Thu, May 5, 2016, at 06:26 PM, Christopher Reimer wrote: >> >>> Which is one is correct (Pythonic)? Or does it matter? >>> >> First, pylint is somewhat opinionated, and its default options shouldn't >> be taken as gospel. There's no correct: filter is fine. >> > > Since the code I'm working on is resume fodder (i.e., "Yes, I code in > Python! Check out my chess engine code on GitHub!"), I want it to be as > Pythonic and PEP8-compliant as possible. That includes scoring 10/10 with > pylint. Never know when an asshat hiring manager would reject my resume out > of hand because my code fell short with pylint. > > For my purposes, I'm using the list comprehension over filter to keep > pylint happy. > > > Thank you, > > Chris R. > -- > https://mail.python.org/mailman/listinfo/python-list > Don't forget that you can also place a pylintrc file (with some reasonable comments) into the repository or use the other means to disable selected rules.... From jim at dodgen.us Sat May 7 19:54:01 2016 From: jim at dodgen.us (Jim Dodgen) Date: Sat, 7 May 2016 16:54:01 -0700 Subject: redirecting stdout and stderr to /dev/null In-Reply-To: References: Message-ID: *Thanks for the help* On Sat, May 7, 2016 at 12:16 PM, Martin A. Brown wrote: > > Hello there, > > >I'm new to python but well versed on other languages such as C and > >Perl > > > >I'm have problems redirecting stdout and stderr to /dev/null in a > >program that does a fork and exec. T found this method googling > >around and it is quite elegant compared to to the Perl version. > > > >So to isolate things I made a much shorter test program and it > >still is not redirecting. What am I doing wrong? > > > >test program test.py > >----------------- cut here ------------------- > >import sys > >import os > > > >f = open(os.devnull, 'w') > >sys.stdout = f > >sys.stderr = f > >os.execl("/bin/ping", "", "-w", "20", "192.168.1.1"); > >------------------ cut here ------------------- > > Think about the file descriptors. > > Unix doesn't care what the name is, rather that the process inherits > the FDs from the parent. So, your solution might need to be a bit > more complicated to achieve what you desire. Run the following to > see what I mean. > > realstdout = sys.stdout > realstderr = sys.stderr > f = open(os.devnull, 'w') > sys.stdout = f > sys.stderr = f > > print("realstdout FD: %d" % (realstdout.fileno(),), file=realstdout) > print("realstderr FD: %d" % (realstderr.fileno(),), file=realstdout) > print("sys.stdout FD: %d" % (sys.stdout.fileno(),), file=realstdout) > print("sys.stderr FD: %d" % (sys.stderr.fileno(),), file=realstdout) > > That should produce output that looks like this: > > realstdout FD: 1 > realstderr FD: 2 > sys.stdout FD: 3 > sys.stderr FD: 3 > > I hope that's a good hint... > > I like the idea of simply calling the next program using one of the > exec() variants, but you'll have to adjust the file descriptors, > rather than just the names used by Python. > > If you don't need to exec(), but just run a child, then here's the > next hint (this is for Python 3.5): > > import subprocess > cmd = ["ping", "-w", "20", "192.168.1.1"] > devnull = subprocess.DEVNULL > proc = subprocess.run(cmd, stdout=devnull, stderr=devnull) > proc.check_returncode() > > (By the way, your "ping" command looked like it had an empty token > in the second arg position. Looked weird to me, so I removed it > in my examples.) > The empty token is needed but useless, it is arg[0] most people just repeat the program name > > For subprocess.run, see: > > https://docs.python.org/3/library/subprocess.html#subprocess.run > > For earlier Python versions without run(), you can use Popen(): > > import subprocess > cmd = ["/bin/ping", "-w", "20", "192.168.1.1"] > devnull = subprocess.DEVNULL > proc = subprocess.Popen(cmd, stdout=devnull, stderr=devnull) > retcode = proc.wait() > if retcode != 0: > raise FlamingHorribleDeath > > You will have to define FlamingHorribleDeath or figure out what you > want to do in the event of the various different types of > failure....if you don't then, you'll just see this: > > NameError: name 'FlamingHorribleDeath' is not defined > > Good luck, > > -Martin > > -- > Martin A. Brown > http://linux-ip.net/ seems not to find subprocess.DEVNULL I'm on 2.7.6 Ubuntu, and 2.7.9 on Raspbian I expect I need 3.5 here are some Tracebacks I get, the second one is when I tried os.devnull in place of subprocess.DEVNULL Traceback (most recent call last): File "test.py", line 5, in devnull = subprocess.DEVNULL AttributeError: 'module' object has no attribute 'DEVNULL' Traceback (most recent call last): File "test.py", line 6, in proc = subprocess.Popen(cmd, stdout=os.devnull, stderr=os.devnull) File "/usr/lib/python2.7/subprocess.py", line 702, in __init__ errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr) File "/usr/lib/python2.7/subprocess.py", line 1128, in _get_handles c2pwrite = stdout.fileno() AttributeError: 'str' object has no attribute 'fileno' One other observation it looks as Popen behaves the same way as my fork exec would From rosuav at gmail.com Sat May 7 20:10:07 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2016 10:10:07 +1000 Subject: redirecting stdout and stderr to /dev/null In-Reply-To: References: Message-ID: On Sun, May 8, 2016 at 9:54 AM, Jim Dodgen wrote: > The empty token is needed but useless, it is arg[0] most people just repeat > the program name Far from useless. It's how a process learns its own name, and yes, repeating the image name is the most common way to provide that. > One other observation it looks as Popen behaves the same way as my fork > exec would Indeed. In fact, I would strongly recommend never using an explicit fork/exec from Python - always use subprocess or equivalent. On non-Unix platforms, fork/exec may not be available, but subprocess can use other methods of invoking programs. ChrisA From jim at dodgen.us Sat May 7 20:18:09 2016 From: jim at dodgen.us (Jim Dodgen) Date: Sat, 7 May 2016 17:18:09 -0700 Subject: redirecting stdout and stderr to /dev/null In-Reply-To: References: Message-ID: Thanks Chris I now have things working using a version 3.4.3 it finds subprocess.DEVNULL just fine *Jim Dodgen* On Sat, May 7, 2016 at 5:10 PM, Chris Angelico wrote: > On Sun, May 8, 2016 at 9:54 AM, Jim Dodgen wrote: > > The empty token is needed but useless, it is arg[0] most people just > repeat > > the program name > > Far from useless. It's how a process learns its own name, and yes, > repeating the image name is the most common way to provide that. > > > One other observation it looks as Popen behaves the same way as my fork > > exec would > > Indeed. In fact, I would strongly recommend never using an explicit > fork/exec from Python - always use subprocess or equivalent. On > non-Unix platforms, fork/exec may not be available, but subprocess can > use other methods of invoking programs. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > From nospam at dfs.com Sat May 7 21:16:25 2016 From: nospam at dfs.com (DFS) Date: Sat, 7 May 2016 21:16:25 -0400 Subject: pylint woes In-Reply-To: References: Message-ID: On 5/7/2016 1:01 PM, Chris Angelico wrote: > On Sun, May 8, 2016 at 2:51 AM, DFS wrote: >> [1] >> pylint says "Consider using enumerate instead of iterating with range and >> len" >> >> the offending code is: >> for j in range(len(list1)): >> do something with list1[j], list2[j], list3[j], etc. >> >> enumeration would be: >> for j,item in enumerate(list1): >> do something with list1[j], list2[j], list3[j], etc. >> >> Is there an advantage to using enumerate() here? > > The suggestion from a human would be to use zip(), or possibly to > change your data structures. Happens like this: address data is scraped from a website: names = tree.xpath() addr = tree.xpath() I want to store the data atomically, so I parse street, city, state, and zip into their own lists. "1250 Peachtree Rd, Atlanta, GA 30303 street = [s.split(',')[0] for s in addr] city = [c.split(',')[1].strip() for c in addr] state = [s[-8:][:2] for s in addr] zipcd = [z[-5:] for z in addr] names = ["Taco Bell", "Wendy's"] addr = ['928 Buford Dr, Tucker, GA 30043', '4880 Ptree Pkwy, Atlanta, GA 30303'] street = ['928 Buford Dr', '4880 Sugarloaf Pkwy'] city = ['Tucker','Atlanta'] state = ['GA','GA'] zipcd = ['30043','30303'] When you say 'possibly change data structures'... to what? > for item1, item2, item3 in zip(list1, list2, list3): > do something with the items ziplists = zip(names,street,city,state,zipcd) print ziplists [('Taco Bell', '928 Buford Dr', 'Tucker', 'GA', '30043'), ("Wendy's", '4880 Sugarloaf Pkwy', 'Atlanta', 'GA', '30303')] Why is it better to zip() them up and use: for item1, item2, item3 in zip(list1, list2, list3): do something with the items than for j in range(len(list1)): do something with list1[j], list2[j], list3[j], etc. From ben+python at benfinney.id.au Sat May 7 21:21:10 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 08 May 2016 11:21:10 +1000 Subject: redirecting stdout and stderr to /dev/null References: Message-ID: <85r3ddfipl.fsf@benfinney.id.au> Jim Dodgen writes: > I'm have problems redirecting stdout and stderr to /dev/null in a > program that does a fork and exec. You may be interested in the ?python-daemon? library . It takes care of all the fiddly bits to turn your program into a well-behaved Unix daemon. -- \ ?I wish there was a knob on the TV to turn up the intelligence. | `\ There's a knob called ?brightness? but it doesn't work.? | _o__) ?Eugene P. Gallagher | Ben Finney From rosuav at gmail.com Sat May 7 21:36:26 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2016 11:36:26 +1000 Subject: pylint woes In-Reply-To: References: Message-ID: On Sun, May 8, 2016 at 11:16 AM, DFS wrote: > On 5/7/2016 1:01 PM, Chris Angelico wrote: >> The suggestion from a human would be to use zip(), or possibly to >> change your data structures. > > > Happens like this: > > address data is scraped from a website: > > names = tree.xpath() > addr = tree.xpath() > > I want to store the data atomically, so I parse street, city, state, and zip > into their own lists. > > "1250 Peachtree Rd, Atlanta, GA 30303 > > street = [s.split(',')[0] for s in addr] > city = [c.split(',')[1].strip() for c in addr] > state = [s[-8:][:2] for s in addr] > zipcd = [z[-5:] for z in addr] So you're iterating over addr lots of times, and building separate lists. As an alternative, you could iterate over it *once*, and have a single object representing an address. > Why is it better to zip() them up and use: > > for item1, item2, item3 in zip(list1, list2, list3): > do something with the items > > than > > > for j in range(len(list1)): > do something with list1[j], list2[j], list3[j], etc. Because 'j' is insignificant here, as is the length of the list. What you're doing is iterating over three parallel lists - not counting numbers. Imagine that, instead of lists, you just have *sequences* - ordered collections of things. You can follow a recipe without knowing the numbers of the individual lines; you just need to know the sequence. Here, iterate over this collection: * Collect ingredients. * Cream the butter and the sugar. * Sift the salt into the flour. * Fold the mixture into an origami crane. These instructions work whether they're numbered or not. ChrisA From ben+python at benfinney.id.au Sat May 7 21:38:33 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 08 May 2016 11:38:33 +1000 Subject: redirecting stdout and stderr to /dev/null References: Message-ID: <85inypfhwm.fsf@benfinney.id.au> Chris Angelico writes: > On Sun, May 8, 2016 at 9:54 AM, Jim Dodgen wrote: > > The empty token is needed but useless, it is arg[0] most people just > > repeat the program name > > Far from useless. It's how a process learns its own name, and yes, > repeating the image name is the most common way to provide that. In particular, a program's name may not be its file name; it can be called by one of several different names dependeing on how it is installed on the system. Certainly the programmer writing the code cannot hard-code what the command name will be that invokes the program. Only ?sys.argv[0]?, read at run time, can tell. > Indeed. In fact, I would strongly recommend never using an explicit > fork/exec from Python - always use subprocess or equivalent. On > non-Unix platforms, fork/exec may not be available, but subprocess can > use other methods of invoking programs. I've already mentioned earlier, but to be sure: the ?python-daemon? library takes care of the details of becoming a Unix daemon process. -- \ ?? a Microsoft Certified System Engineer is to information | `\ technology as a McDonalds Certified Food Specialist is to the | _o__) culinary arts.? ?Michael Bacarella | Ben Finney From tjreedy at udel.edu Sat May 7 21:40:36 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 7 May 2016 21:40:36 -0400 Subject: Pylint prefers list comprehension over filter... In-Reply-To: <572E3F53.5010703@icloud.com> References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> Message-ID: On 5/7/2016 3:17 PM, Christopher Reimer wrote: > For my purposes, I'm using the list comprehension over filter to keep > pylint happy. How sad. The pylint developers arrogantly take it on themselves to revise Python, against the wishes of Guido and the other core developers, and you and feel obligated to follow them. They should at least add a disclaimer "Using the default options, pylint checks that your code complies with the pylint-approved subset of Python." -- Terry Jan Reedy From tjreedy at udel.edu Sat May 7 21:44:46 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 7 May 2016 21:44:46 -0400 Subject: pylint woes In-Reply-To: References: <572E3967.9060206@icloud.com> Message-ID: On 5/7/2016 3:52 PM, Ray Cote wrote: > Biggest issue I have with pyLint is that it complains when function > parameters are indented twice vs. once. pyFlakes likes the twice. > Example: > def function_name( > parm_1, > long_parm_name, > ?. > end_of_long_list_of params) > parm_1 = long_parm_name This is the recommendation in PEP 8. I would otherwise insert a blank line before the body. tjr From me+python at ixokai.io Sat May 7 22:14:25 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sat, 07 May 2016 19:14:25 -0700 Subject: pylint woes In-Reply-To: References: Message-ID: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> On Sat, May 7, 2016, at 06:16 PM, DFS wrote: > Why is it better to zip() them up and use: > > for item1, item2, item3 in zip(list1, list2, list3): > do something with the items > > than > > for j in range(len(list1)): > do something with list1[j], list2[j], list3[j], etc. Although Chris has a perfectly good and valid answer why conceptually the zip is better, let me put forth: the zip is simply clearer, more readable and more maintainable. This is a question of style and to a certain degree aesthetics, so is somewhat subjective, but range(len(list1)) and list1[j] are all indirection, when item1 is clearly (if given a better name then 'item1') something distinct you're working on. -- Stephen Hansen m e @ i x o k a i . i o From nospam at dfs.com Sat May 7 22:15:38 2016 From: nospam at dfs.com (DFS) Date: Sat, 7 May 2016 22:15:38 -0400 Subject: pylint woes In-Reply-To: References: Message-ID: On 5/7/2016 9:36 PM, Chris Angelico wrote: > On Sun, May 8, 2016 at 11:16 AM, DFS wrote: >> On 5/7/2016 1:01 PM, Chris Angelico wrote: >>> The suggestion from a human would be to use zip(), or possibly to >>> change your data structures. >> >> >> Happens like this: >> >> address data is scraped from a website: >> >> names = tree.xpath() >> addr = tree.xpath() >> >> I want to store the data atomically, so I parse street, city, state, and zip >> into their own lists. >> >> "1250 Peachtree Rd, Atlanta, GA 30303 >> >> street = [s.split(',')[0] for s in addr] >> city = [c.split(',')[1].strip() for c in addr] >> state = [s[-8:][:2] for s in addr] >> zipcd = [z[-5:] for z in addr] > > So you're iterating over addr lots of times, and building separate > lists. As an alternative, you could iterate over it *once*, and have a > single object representing an address. I like the idea of one iteration, but how? (I'll be trying myself before I check back in) Remember, it's required to split the data up, to give flexibility in sorting, searching, output, etc. I saw a cool example where someone built a list and used it to do a bulk INSERT. That probably won't work well here, because one of the options I give the user is # of addresses to store. So I do invididual INSERTs using the 'for j in range()' method, which makes it easier to track how many addresses have been stored. >> Why is it better to zip() them up and use: >> >> for item1, item2, item3 in zip(list1, list2, list3): >> do something with the items >> >> than >> >> >> for j in range(len(list1)): >> do something with list1[j], list2[j], list3[j], etc. > > Because 'j' is insignificant here, as is the length of the list. Sorry, but I don't understand what you mean by insignificant. j keeps track of the position in the list - regardless of the length of the list. > What > you're doing is iterating over three parallel lists - not counting > numbers. Imagine that, instead of lists, you just have *sequences* - > ordered collections of things. You can follow a recipe without knowing > the numbers of the individual lines; you just need to know the > sequence. Here, iterate over this collection: > > * Collect ingredients. > * Cream the butter and the sugar. > * Sift the salt into the flour. > * Fold the mixture into an origami crane. > > These instructions work whether they're numbered or not. Again, not following you. The only reason for j in range(len(list1)): do something with list1[j], list2[j], list3[j], etc. or for item1, item2, item3 in zip(list1, list2, list3): do something with the items works is because each list has the same number of items. From python at mrabarnett.plus.com Sat May 7 22:21:33 2016 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 8 May 2016 03:21:33 +0100 Subject: pylint woes In-Reply-To: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> Message-ID: <65e07b36-fceb-f03c-d122-b8340f872b1c@mrabarnett.plus.com> On 2016-05-08 03:14, Stephen Hansen wrote: > On Sat, May 7, 2016, at 06:16 PM, DFS wrote: > >> Why is it better to zip() them up and use: >> >> for item1, item2, item3 in zip(list1, list2, list3): >> do something with the items >> >> than >> >> for j in range(len(list1)): >> do something with list1[j], list2[j], list3[j], etc. > > Although Chris has a perfectly good and valid answer why conceptually > the zip is better, let me put forth: the zip is simply clearer, more > readable and more maintainable. > > This is a question of style and to a certain degree aesthetics, so is > somewhat subjective, but range(len(list1)) and list1[j] are all > indirection, when item1 is clearly (if given a better name then 'item1') > something distinct you're working on. > +1 If you're iterating through multiple sequences in parallel, zip is the way to go. From steve at pearwood.info Sat May 7 22:43:46 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 08 May 2016 12:43:46 +1000 Subject: Python is an Equal Opportunity Programming Language References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> <1462646456.2677797.601019937.6FC9DC8E@webmail.messagingengine.com> Message-ID: <572ea7e4$0$1587$c3e8da3$5496439d@news.astraweb.com> On Sun, 8 May 2016 04:40 am, Random832 wrote: > On Sat, May 7, 2016, at 11:16, Steven D'Aprano wrote: >> > Indian and Chinese H1B holders are getting screwed, which is of course >> > the whole objective of the country limits. >> >> The *whole* objective? You don't think that *part* of the objective may >> be >> to ensure that citizens of countries other than India and China can get >> green cards too? > > If not for the quotas, a citizen of some other country would have an > equal chance to get a green card as a citizen of India or China. If you have a big hat with 5,000,000 tickets marked "Indian", and 500 tickets marked "Finish", and you stick your hand in the hat and rummage around and pick a random ticket, do you really think that you have an equal chance of selecting an Indian ticket and a Finish ticket? > This is > a much simpler question than Python convention attendees, since there > are *in fact* more Indian or Chinese people in actual existence than > citizens of any given other country, which can't be blamed on any form > of discrimination. > >> Perhaps the country limits are also in place, at least in part, to manage >> the rate at which new immigrants arrive in the country? > > That's immaterial, we're not talking about a limit on the total number > of visas. How is it immaterial? Regardless of the limit on the total number of visas, per country limits put an upper limit how quickly new immigrants from any one specific country can arrive. -- Steven From rosuav at gmail.com Sat May 7 22:50:46 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2016 12:50:46 +1000 Subject: pylint woes In-Reply-To: References: Message-ID: On Sun, May 8, 2016 at 12:15 PM, DFS wrote: > On 5/7/2016 9:36 PM, Chris Angelico wrote: >> >> On Sun, May 8, 2016 at 11:16 AM, DFS wrote: >>> >>> street = [s.split(',')[0] for s in addr] >>> city = [c.split(',')[1].strip() for c in addr] >>> state = [s[-8:][:2] for s in addr] >>> zipcd = [z[-5:] for z in addr] >> >> >> So you're iterating over addr lots of times, and building separate >> lists. As an alternative, you could iterate over it *once*, and have a >> single object representing an address. > > I like the idea of one iteration, but how? (I'll be trying myself before I > check back in) > > Remember, it's required to split the data up, to give flexibility in > sorting, searching, output, etc. Start by unpacking the comprehensions into statement form. street = [] for s in addr: street.append(s.split(',')[0]) city = [] for c in addr: city.append(c.split(',')[1].strip()) state = [] for s in addr: state.append(s[-8:][:2]) zipcd = [] for z in addr: zipcd.append(z[-5:]) Now see how you're doing the same thing four times? Let's start by keeping it the way it is, but combine the loops. street, city, state, zipcd = [], [], [], [] for a in addr: street.append(a.split(',')[0]) city.append(a.split(',')[1].strip()) state.append(a[-8:][:2]) zipcd.append(a[-5:]) Side point: I prefer collections to be named in the plural, so these would be "streets", and "addrs". This lets you follow a very simple rule of iteration: "for item in collection" or "for singular in plural". In this case, "for address in addresses" is classic iteration. So, now that you have a single loop picking up the different pieces, it's easy to build up a simple object that represents an address. # Either this from collections import namedtuple Address = namedtuple("Address", ["street", "city", "state", "zipcd"]) # or this from types import SimpleNamespace class Address(SimpleNamespace): pass addresses = [] for a in addr: addresses.append(Address( street=a.split(',')[0], city=a.split(',')[1].strip(), state=a[-8:][:2], zipcd=a[-5:], ) Voila! One iteration, and a single object representing an address. > I saw a cool example where someone built a list and used it to do a bulk > INSERT. That probably won't work well here, because one of the options I > give the user is # of addresses to store. So I do invididual INSERTs using > the 'for j in range()' method, which makes it easier to track how many > addresses have been stored. You could slice it if you actually want that. >>> Why is it better to zip() them up and use: >>> >>> for item1, item2, item3 in zip(list1, list2, list3): >>> do something with the items >>> >>> than >>> >>> >>> for j in range(len(list1)): >>> do something with list1[j], list2[j], list3[j], etc. >> >> >> Because 'j' is insignificant here, as is the length of the list. > > Sorry, but I don't understand what you mean by insignificant. j keeps track > of the position in the list - regardless of the length of the list. Right, but *who cares* what the position is? All you want to do is the "do something" bit. Don't think in terms of concrete and discrete operations in a computer; think in the abstract (what are you trying to accomplish?), and then represent that in code. >> What >> you're doing is iterating over three parallel lists - not counting >> numbers. Imagine that, instead of lists, you just have *sequences* - >> ordered collections of things. You can follow a recipe without knowing >> the numbers of the individual lines; you just need to know the >> sequence. Here, iterate over this collection: >> >> * Collect ingredients. >> * Cream the butter and the sugar. >> * Sift the salt into the flour. >> * Fold the mixture into an origami crane. >> >> These instructions work whether they're numbered or not. > > Again, not following you. > > > The only reason > > for j in range(len(list1)): > do something with list1[j], list2[j], list3[j], etc. > > or > > for item1, item2, item3 in zip(list1, list2, list3): > do something with the items > > works is because each list has the same number of items. Sure, but who cares what each item's position is? All that matters is that they have corresponding positions, which is what zip() does. Imagine you don't even have the whole lists yet. Imagine someone's still writing stuff to them as you work. Maybe they're infinite in length. You can't iterate up to the length of list1, because it doesn't HAVE a length. But you can still zip it up with other parallel collections, and iterate over them all. ChrisA From steve at pearwood.info Sat May 7 22:53:48 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 08 May 2016 12:53:48 +1000 Subject: Pylint prefers list comprehension over filter... References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> <1462656950.727725.601099753.30D90B95@webmail.messagingengine.com> Message-ID: <572eaa3d$0$1586$c3e8da3$5496439d@news.astraweb.com> On Sun, 8 May 2016 07:35 am, Stephen Hansen wrote: > I'd read over PEP8 (the document, not the tool) and > apply style guide recommendations thoughtfully, not mechanically. Guido is not a fan of automated PEP8 checkers. He agrees entirely with your comment: apply style guides thoughtfully, not mechanically. (Cannot be repeated often enough.) -- Steven From nospam at dfs.com Sat May 7 23:04:12 2016 From: nospam at dfs.com (DFS) Date: Sat, 7 May 2016 23:04:12 -0400 Subject: pylint woes In-Reply-To: References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> Message-ID: On 5/7/2016 10:14 PM, Stephen Hansen wrote: > On Sat, May 7, 2016, at 06:16 PM, DFS wrote: > >> Why is it better to zip() them up and use: >> >> for item1, item2, item3 in zip(list1, list2, list3): >> do something with the items >> >> than >> >> for j in range(len(list1)): >> do something with list1[j], list2[j], list3[j], etc. > > Although Chris has a perfectly good and valid answer why conceptually > the zip is better, let me put forth: the zip is simply clearer, more > readable and more maintainable. > > This is a question of style and to a certain degree aesthetics, so is > somewhat subjective, but range(len(list1)) and list1[j] are all > indirection, when item1 is clearly (if given a better name then 'item1') > something distinct you're working on. The lists I actually use are: for j in range(len(nms)): cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" vals = nms[j],street[j],city[j],state[j],zipcd[j] The enumerated version would be: ziplists = zip(nms,street,city,state,zipcd) for nm,street,city,state,zipcd in ziplists: cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" vals = nm,street,city,state,zipcd I guess the enumeration() is a little nicer to look at. Why do you think it's more maintainable? Aside: I haven't tried, but is 'names' a bad idea or illegal for the name of a python list or variable? Thanks From rosuav at gmail.com Sat May 7 23:15:57 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2016 13:15:57 +1000 Subject: Python is an Equal Opportunity Programming Language In-Reply-To: <572ea7e4$0$1587$c3e8da3$5496439d@news.astraweb.com> References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> <1462646456.2677797.601019937.6FC9DC8E@webmail.messagingengine.com> <572ea7e4$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sun, May 8, 2016 at 12:43 PM, Steven D'Aprano wrote: > On Sun, 8 May 2016 04:40 am, Random832 wrote: > >> On Sat, May 7, 2016, at 11:16, Steven D'Aprano wrote: >>> > Indian and Chinese H1B holders are getting screwed, which is of course >>> > the whole objective of the country limits. >>> >>> The *whole* objective? You don't think that *part* of the objective may >>> be >>> to ensure that citizens of countries other than India and China can get >>> green cards too? >> >> If not for the quotas, a citizen of some other country would have an >> equal chance to get a green card as a citizen of India or China. > > If you have a big hat with 5,000,000 tickets marked "Indian", and 500 > tickets marked "Finish", and you stick your hand in the hat and rummage > around and pick a random ticket, do you really think that you have an equal > chance of selecting an Indian ticket and a Finish ticket? So the question is: Do we care about country equality or individual equality? You can't have both. As soon as you divide a population up into unequal parts, you have to figure out what you actually mean by "equality". For instance, should the states in a country have equal representation in federal government, or should their sway be affected by the number of people in each state? In a Wikipedia article, how much space should be given to a narrowly-held view compared to a widely-held one? In a summary of a thousand people's comments, how many favorable ones and how many unfavorable ones should be quoted? In an extreme case of the latter, suppose you have room to post ten comments, and of the thousand, only four were against. Do you post all four, and restrict the other side to four to be "fair", or do you post one of them against nine "yea" comments, or do you pick randomly from the entire pool of questions (which would give you a 4% chance of drawing even a single negative comment)? Which is "fairest"? If two people both want to enter the country, it's *obviously* right that they should have equal chance as individuals. And it's equally obvious that it's not fair to let one highly populous country push out every other country. Somewhere in there you need math and decisions. ChrisA From christopher_reimer at icloud.com Sat May 7 23:22:58 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Sat, 07 May 2016 20:22:58 -0700 Subject: Pylint prefers list comprehension over filter... In-Reply-To: References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> Message-ID: <572EB112.1000606@icloud.com> On 5/7/2016 6:40 PM, Terry Reedy wrote: > On 5/7/2016 3:17 PM, Christopher Reimer wrote: > >> For my purposes, I'm using the list comprehension over filter to keep >> pylint happy. > > How sad. The pylint developers arrogantly take it on themselves to > revise Python, against the wishes of Guido and the other core > developers, and you and feel obligated to follow them. > > They should at least add a disclaimer "Using the default options, pylint > checks that your code complies with the pylint-approved subset of Python." I wasn't aware that I was picking sides in a religious war by altering one line of code. O_o Thank you, Chris R. From steve at pearwood.info Sat May 7 23:25:54 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 08 May 2016 13:25:54 +1000 Subject: pylint woes References: Message-ID: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> On Sun, 8 May 2016 02:51 am, DFS wrote: > This more-anal-than-me program generated almost 2 warnings for every > line of code in my program. w t hey? > > > DFS comments > +-------------------------+------------+ ------------------------------- > |message id |occurrences | > +=========================+============+ > |mixed-indentation |186 | I always use tab Obviously not. There are 186 occurrences where you use mixed tabs and spaces. Try running Tabnanny on you file: python -m tabnanny > +-------------------------+------------+ > |invalid-name |82 | every single variable name?! Maybe. What are they called? > +-------------------------+------------+ > |bad-whitespace |65 | mostly because I line up = > signs: > var1 = value > var10 = value Yuck. How much time do you waste aligning assignments whenever you add or delete or edit a variable? > +-------------------------+------------+ > |trailing-whitespace |59 | heh! > +-------------------------+------------+ > |multiple-statements |23 | do this to save lines. > Will continue doing it. Why? Do you think that there's a world shortage of newline characters? Is the Enter key on your keyboard broken? > +-------------------------+------------+ > |no-member |5 | > > "Module 'pyodbc' has no 'connect' member" Yes it does. > "Module 'pyodbc' has no 'Error' member" Yes it does. > > Issue with pylint, or pyodbc? *shrug* More likely with Pylint. > +-------------------------+------------+ > |line-too-long |5 | meh > +-------------------------+------------+ > |wrong-import-order |4 | does it matter? Probably not. I'm curious what it thinks is the right import order. > +-------------------------+------------+ > |missing-docstring |4 | what's the difference between > a docstring and a # comment? Comments exist only in the source code. Docstrings are available for interactive use with help(), for runtime introspection, and for doctests. https://docs.python.org/2/library/doctest.html > +-------------------------+------------+ > |multiple-imports |2 | doesn't everyone? You mean something like this? import spam, ham, eggs, cheese *shrug* It's a style thing. > +-------------------------+------------+ > |consider-using-enumerate |2 | see below [1] Absolutely use enumerate. > +-------------------------+------------+ > |bad-builtin |2 | warning because I used filter? Well that's just stupid. Bad PyLint. This should absolutely not be turned on by default. -- Steven From nospam at dfs.com Sat May 7 23:28:54 2016 From: nospam at dfs.com (DFS) Date: Sat, 7 May 2016 23:28:54 -0400 Subject: pylint woes In-Reply-To: References: Message-ID: On 5/7/2016 3:40 PM, Terry Reedy wrote: > On 5/7/2016 12:51 PM, DFS wrote: >> This more-anal-than-me program generated almost 2 warnings for every >> line of code in my program. w t hey? > > If you don't like it, why do you use it? I've never used it before last night. I was shocked at what it spewed back at me. > I suppose the answer is that it did find a few things to check. You > might be happier with pychecker, which is much less aggressive. I'll give it a shot. > I believe will find the things you did fix. I'm not parsing this statement. You mean pychecker will find the same things pylint found, and that I fixed? If it finds them after I fixed them... it's a magical program :) DFS comments >> +-------------------------+------------+ ------------------------------- >> |message id |occurrences | >> +=========================+============+ >> |mixed-indentation |186 | I always use tab >> +-------------------------+------------+ >> |invalid-name |82 | every single variable name?! > > I would need examples to comment. Invalid constant name "cityzip" (invalid-name) Invalid constant name "state" (invalid-name) Invalid constant name "miles" (invalid-name) Invalid constant name "store" (invalid-name) Invalid variable name "rs" (invalid-name) >> +-------------------------+------------+ >> |trailing-whitespace |59 | heh! > > Any code editor should have a command to fix this. > IDLE: Format => strip trailing whitespace > Notepad++: Macro => trim trailing and save, Alt-Shift-S > others ... That did it. >> +-------------------------+------------+ >> |no-member |5 | >> >> "Module 'pyodbc' has no 'connect' member" Yes it does. >> "Module 'pyodbc' has no 'Error' member" Yes it does. >> >> Issue with pylint, or pyodbc? > > Worth looking into. Could be a bug somewhere. But I don't have pyodbc > installed. > >> +-------------------------+------------+ >> |line-too-long |5 | meh > > For following the PEP guideline when patching CPython, this is helpful. > >> +-------------------------+------------+ >> |wrong-import-order |4 | does it matter? > > Consistency in imports ultimately makes easier reading. > Many idlelib files use this order: stdlib modules other than tkinter and > idlelib (alphabetically); tkinter (tkinter first, then submodules); > idlelib (alphabetically). When I edit files, I sometimes reorder > imports to conform. It complains 2x about this: import os, sys, time, datetime import pyodbc, sqlite3 import re, requests from lxml import html But I think there are some pylint bugs here: ------------------------------------------------------------------------- standard import "import pyodbc, sqlite3" comes before "import pyodbc, sqlite3" (wrong-import-order) * complains that the line comes before itself? ------------------------------------------------------------------------- standard import "import re, requests" comes before "import pyodbc, sqlite3" (wrong-import-order) * So I switched them, and then it complained about that: standard import "import pyodbc, sqlite3" comes before "import re, requests" (wrong-import-order) ------------------------------------------------------------------------- You can't win with pylint... And, the author probably isn't a native English-speaker, since when he says 'comes before' I think he means 'should come before'. >> +-------------------------+------------+ >> |missing-docstring |4 | what's the difference between >> a docstring and a # comment? > > # Comments only appear in the source > '''Docstrings are copied to the compiled code object, are interactively > accessible, and are used for help(ojb) output.''' > > >> +-------------------------+------------+ >> |superfluous-parens |3 | I like to surround 'or' >> statments with parens > > I would need examples to comment if ("Please choose a state" in str(matches)): if (var == "val" or var2 == "val2"): >> +-------------------------+------------+ >> |bad-builtin |2 | warning because I used filter? > > If they are still doing this in the latest release, it is an arrogance > and inconsistency bug on their part. Disable this check. $ pylint --version No config file found, using default configuration pylint 1.5.5, astroid 1.4.5 Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec 5 2015, 20:32:19) [MSC v.1500 32 bit (Intel)] It says "Used builtin function 'filter'. Using a list comprehension can be clearer. (bad-builtin)" >> +-------------------------+------------+ >> |missing-final-newline |1 | I'm using Notepad++, with >> EOL Conversion set to >> 'Windows Format'. > > That says to replace final '\n' with '\r\n'. It does not affect a > missing final newline ;-) > > How or should I fix this? > > Fix by hitting 'Enter' at the end of the last line. > Should you? I think it a good habit. Done >> After fixes and disabling various warnings: >> "Your code has been rated at 8.37/10" > > Being able to customize pylint by turning off warnings is its saving > feature. Yes. If I had to see 300-350 lines of output every time I wouldn't ever use it again. Overall, I do like a majority of the things it suggested. From nospam at dfs.com Sat May 7 23:38:24 2016 From: nospam at dfs.com (DFS) Date: Sat, 7 May 2016 23:38:24 -0400 Subject: pylint woes In-Reply-To: References: <572E3967.9060206@icloud.com> Message-ID: On 5/7/2016 2:52 PM, Christopher Reimer wrote: > On 5/7/2016 9:51 AM, DFS wrote: >> Has anyone ever in history gotten 10/10 from pylint for a non-trivial >> program? > > I routinely get 10/10 for my code. While pylint isn't perfect and > idiosyncratic at times, it's a useful tool to help break bad programming > habits. Since I came from a Java background, I had to unlearn everything > from Java before I could write Pythonic code. It might help to use an > IDE that offers PEP8-compliant code suggestions (I use PyCharm IDE). > >> That's about as good as it's gonna get! > > You can do better. 10/10 on pylint isn't better. It's being robotic and conforming to the opinions of the author of that app. In fact, I think: import os, sys, time, socket is much more readable than, and preferable to, import os import sys import time import socket but pylint complains about the former. > You should strive for 10/10 whenever possible, nah > figure out why you fall short and ask for help on the parts that don't > make sense. I actually agree with ~3/4 of the suggestions it makes. My code ran fine before pylint tore it a new one, and it doesn't appear to run any better after making various fixes. But between you clp guys and pylint, the code is definitely improving. >> pylint says "Consider using enumerate instead of iterating with range >> and len" >> >> the offending code is: >> for j in range(len(list1)): >> do something with list1[j], list2[j], list3[j], etc. > > This code is reeking with bad habits to be broken. Assigning a throwaway > variable to walk the index is unnecessary when Python can do it for you > behind the scenes. Don't you think python also allocates a throwaway variable for use with zip and enumerate()? > As Chris A. pointed out in his post, you should use > zip() to walk through the values of each list at the same time. Yeah, zip looks interesting. I just started using python a month ago, and didn't know about zip until pylint pointed it out (it said I redefined a builtin by using 'zip' as a list name). Edit: I already put zip() it in place. Only improvement I think is it looks cleaner - got rid of a bunch of [j]s. > Thank you, > > Chris R. No, thank /you/, DFS From random832 at fastmail.com Sat May 7 23:40:58 2016 From: random832 at fastmail.com (Random832) Date: Sat, 07 May 2016 23:40:58 -0400 Subject: Python is an Equal Opportunity Programming Language In-Reply-To: <572ea7e4$0$1587$c3e8da3$5496439d@news.astraweb.com> References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> <1462646456.2677797.601019937.6FC9DC8E@webmail.messagingengine.com> <572ea7e4$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1462678858.3638806.601230905.2CF826C5@webmail.messagingengine.com> On Sat, May 7, 2016, at 22:43, Steven D'Aprano wrote: > > If not for the quotas, a citizen of some other country would have an > > equal chance to get a green card as a citizen of India or China. > > If you have a big hat with 5,000,000 tickets marked "Indian", and 500 > tickets marked "Finish", and you stick your hand in the hat and rummage > around and pick a random ticket, do you really think that you have an > equal > chance of selecting an Indian ticket and a Finish ticket? But that's not what it is. You would have, say, 1,000 tickets labeled "green card" and 100,000 tickets labeled "no green card", and (say) 12,000 Indian people and 50 Finnish people each get their turn drawing from that same bucket. In your version, the Finnish people draw from a bucket with 500 green card tickets and no "no green card" cards, and the Indian people draw from a bucket with 500 green card tickets and 11,500 "no green card" tickets. From me+python at ixokai.io Sat May 7 23:46:58 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sat, 07 May 2016 20:46:58 -0700 Subject: pylint woes In-Reply-To: References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> Message-ID: <1462679218.789727.601235065.67122D8D@webmail.messagingengine.com> On Sat, May 7, 2016, at 08:04 PM, DFS wrote: > The lists I actually use are: > > for j in range(len(nms)): > cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" > vals = nms[j],street[j],city[j],state[j],zipcd[j] > > > The enumerated version would be: > > ziplists = zip(nms,street,city,state,zipcd) > for nm,street,city,state,zipcd in ziplists: > cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" > vals = nm,street,city,state,zipcd > > > I guess the enumeration() is a little nicer to look at. Why do you > think it's more maintainable? Code is read more then its written. That which is nicer to look at, therefore, is easier to read. That which is easier to read is easier to maintain. Beyond that, its simpler, and more clearly articulates in the local space what's going on. > Aside: I haven't tried, but is 'names' a bad idea or illegal for the > name of a python list or variable? Nothing wrong with names. Or 'name', for that matter. Try to avoid abbreviations. -- Stephen Hansen m e @ i x o k a i . i o From jim at dodgen.us Sat May 7 23:49:16 2016 From: jim at dodgen.us (Jim Dodgen) Date: Sat, 7 May 2016 20:49:16 -0700 Subject: redirecting stdout and stderr to /dev/null In-Reply-To: <85inypfhwm.fsf@benfinney.id.au> References: <85inypfhwm.fsf@benfinney.id.au> Message-ID: Great help. My Python program is a rewrite of a Perl program I wrote. An interesting exercise. The reason being it is targeted for a Raspberry Pi and for the Pi Python has the most support. *Jim Dodgen* On Sat, May 7, 2016 at 6:38 PM, Ben Finney wrote: > Chris Angelico writes: > > > On Sun, May 8, 2016 at 9:54 AM, Jim Dodgen wrote: > > > The empty token is needed but useless, it is arg[0] most people just > > > repeat the program name > > > > Far from useless. It's how a process learns its own name, and yes, > > repeating the image name is the most common way to provide that. > > In particular, a program's name may not be its file name; it can be > called by one of several different names dependeing on how it is > installed on the system. > > Certainly the programmer writing the code cannot hard-code what the > command name will be that invokes the program. Only ?sys.argv[0]?, read > at run time, can tell. > > > Indeed. In fact, I would strongly recommend never using an explicit > > fork/exec from Python - always use subprocess or equivalent. On > > non-Unix platforms, fork/exec may not be available, but subprocess can > > use other methods of invoking programs. > > I've already mentioned earlier, but to be sure: the ?python-daemon? > library takes care of > the details of becoming a Unix daemon process. > > -- > \ ?? a Microsoft Certified System Engineer is to information | > `\ technology as a McDonalds Certified Food Specialist is to the | > _o__) culinary arts.? ?Michael Bacarella | > Ben Finney > > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Sat May 7 23:51:03 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2016 13:51:03 +1000 Subject: pylint woes In-Reply-To: References: Message-ID: On Sun, May 8, 2016 at 1:28 PM, DFS wrote: > Invalid constant name "cityzip" (invalid-name) > Invalid constant name "state" (invalid-name) > Invalid constant name "miles" (invalid-name) > Invalid constant name "store" (invalid-name) > Invalid variable name "rs" (invalid-name) ... huh?? The first four seem to have been incorrectly detected as constants. How are they used? The last one is probably "too short". Or something. > standard import "import re, requests" comes before "import pyodbc, sqlite3" > (wrong-import-order) > > * So I switched them, and then it complained about that: > > standard import "import pyodbc, sqlite3" comes before "import re, requests" > (wrong-import-order) > > ------------------------------------------------------------------------- > > You can't win with pylint... Probably that means it got confused by the alphabetization - "pyodbc" should come before "re" and "requests", but "sqlite3" should come after. Either fix the first problem by splitting them onto separate lines, or ignore this as a cascaded error. My general principle is that things on one line should *belong* on one line. So having "import re, requests" makes no sense, but I might have something like "import os, sys" when the two modules are both used in one single line of code and never again. Otherwise, splitting them out is the easiest. >>> +-------------------------+------------+ >>> |superfluous-parens |3 | I like to surround 'or' >>> statments with parens >> >> >> I would need examples to comment > > > > if ("Please choose a state" in str(matches)): > if (var == "val" or var2 == "val2"): Cut the parens. Easy! > It says "Used builtin function 'filter'. Using a list comprehension can be > clearer. (bad-builtin)" Kill that message and keep using filter. ChrisA From me+python at ixokai.io Sat May 7 23:55:22 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sat, 07 May 2016 20:55:22 -0700 Subject: pylint woes In-Reply-To: References: Message-ID: <1462679722.792380.601237649.3175E2FD@webmail.messagingengine.com> On Sat, May 7, 2016, at 08:28 PM, DFS wrote: > >> +-------------------------+------------+ > >> |superfluous-parens |3 | I like to surround 'or' > >> statments with parens > > > > I would need examples to comment > > > if ("Please choose a state" in str(matches)): > if (var == "val" or var2 == "val2"): Gah, don't do that. You're adding meaningless noise. Especially in the first case. -- Stephen Hansen m e @ i x o k a i . i o From rosuav at gmail.com Sat May 7 23:56:59 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2016 13:56:59 +1000 Subject: pylint woes In-Reply-To: References: <572E3967.9060206@icloud.com> Message-ID: On Sun, May 8, 2016 at 1:38 PM, DFS wrote: >> This code is reeking with bad habits to be broken. Assigning a throwaway >> variable to walk the index is unnecessary when Python can do it for you >> behind the scenes. > > > Don't you think python also allocates a throwaway variable for use with zip > and enumerate()? Nope. But even if it did, it wouldn't matter. Concern yourself with your code, and let the implementation take care of itself. ChrisA From nospam at dfs.com Sun May 8 00:10:32 2016 From: nospam at dfs.com (DFS) Date: Sun, 8 May 2016 00:10:32 -0400 Subject: pylint woes In-Reply-To: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> References: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5/7/2016 11:25 PM, Steven D'Aprano wrote: > On Sun, 8 May 2016 02:51 am, DFS wrote: > >> This more-anal-than-me program generated almost 2 warnings for every >> line of code in my program. w t hey? >> >> >> DFS comments >> +-------------------------+------------+ ------------------------------- >> |message id |occurrences | >> +=========================+============+ >> |mixed-indentation |186 | I always use tab > > Obviously not. There are 186 occurrences where you use mixed tabs and > spaces. I mean I always use tab after : The program won't run otherwise. If I use spaces, 100% of the time it throws: IndentationError: unindent does not match any outer indentation level > Try running Tabnanny on you file: > > python -m tabnanny Didn't seem to do anything. >> +-------------------------+------------+ >> |invalid-name |82 | every single variable name?! > > Maybe. What are they called? > > >> +-------------------------+------------+ >> |bad-whitespace |65 | mostly because I line up = >> signs: >> var1 = value >> var10 = value > > Yuck. How much time do you waste aligning assignments whenever you add or > delete or edit a variable? Lots. It takes hours to add or delete 3 whitespaces. >> +-------------------------+------------+ >> |trailing-whitespace |59 | heh! >> +-------------------------+------------+ >> |multiple-statements |23 | do this to save lines. >> Will continue doing it. > > Why? Do you think that there's a world shortage of newline characters? Is > the Enter key on your keyboard broken? I do it because I like it. if verbose: print var python doesn't complain. >> +-------------------------+------------+ >> |no-member |5 | >> >> "Module 'pyodbc' has no 'connect' member" Yes it does. >> "Module 'pyodbc' has no 'Error' member" Yes it does. >> >> Issue with pylint, or pyodbc? > > *shrug* More likely with Pylint. >> +-------------------------+------------+ >> |line-too-long |5 | meh >> +-------------------------+------------+ >> |wrong-import-order |4 | does it matter? > > Probably not. I'm curious what it thinks is the right import order. "wrong-import-order (C0411): %s comes before %s Used when PEP8 import order is not respected (standard imports first, then third-party libraries, then local imports)" https://docs.pylint.org/features.html I think there are some pylint bugs here: ------------------------------------------------------------------------- standard import "import pyodbc, sqlite3" comes before "import pyodbc, sqlite3" (wrong-import-order) * complains that the line comes before itself? ------------------------------------------------------------------------- standard import "import re, requests" comes before "import pyodbc, sqlite3" (wrong-import-order) * So I switched them, and then it complained about that: standard import "import pyodbc, sqlite3" comes before "import re, requests" (wrong-import-order) ------------------------------------------------------------------------- >> +-------------------------+------------+ >> |missing-docstring |4 | what's the difference between >> a docstring and a # comment? > > Comments exist only in the source code. > > Docstrings are available for interactive use with help(), for runtime > introspection, and for doctests. > > https://docs.python.org/2/library/doctest.html Thanks >> +-------------------------+------------+ >> |multiple-imports |2 | doesn't everyone? > > You mean something like this? > > import spam, ham, eggs, cheese > > *shrug* It's a style thing. pylint gives you demerits for that. >> +-------------------------+------------+ >> |consider-using-enumerate |2 | see below [1] > > Absolutely use enumerate. Everyone else says so, too. But other than cleaner-looking code, I'm not understanding how it's a real advantage over: for j in range(len(list)): >> +-------------------------+------------+ >> |bad-builtin |2 | warning because I used filter? > > Well that's just stupid. Bad PyLint. This should absolutely not be turned on > by default. It says "Used builtin function 'filter'. Using a list comprehension can be clearer. (bad-builtin)" From rosuav at gmail.com Sun May 8 00:21:49 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2016 14:21:49 +1000 Subject: pylint woes In-Reply-To: References: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sun, May 8, 2016 at 2:10 PM, DFS wrote: >>> +-------------------------+------------+ >>> |trailing-whitespace |59 | heh! >>> +-------------------------+------------+ >>> |multiple-statements |23 | do this to save lines. >>> Will continue doing it. >> >> >> Why? Do you think that there's a world shortage of newline characters? Is >> the Enter key on your keyboard broken? > > > I do it because I like it. > > if verbose: print var > > python doesn't complain. That's a massively-debated point. In that specific example, I'd support the one-liner; however, I'd also recommend this technique: # for Python 2, you need to start with this line from __future__ import print_function if verbose: verbiage = print else: def verbiage(*args): pass Then, instead of "if verbose: print(var)", you would use "verbiage(var)". Of course, you want something better than "verbiage" as your name; the nature of your verbose output might give a clue as to what name would work. ChrisA From nospam at dfs.com Sun May 8 00:40:55 2016 From: nospam at dfs.com (DFS) Date: Sun, 8 May 2016 00:40:55 -0400 Subject: pylint woes In-Reply-To: References: Message-ID: On 5/7/2016 11:51 PM, Chris Angelico wrote: > On Sun, May 8, 2016 at 1:28 PM, DFS wrote: >> Invalid constant name "cityzip" (invalid-name) >> Invalid constant name "state" (invalid-name) >> Invalid constant name "miles" (invalid-name) >> Invalid constant name "store" (invalid-name) >> Invalid variable name "rs" (invalid-name) > > ... huh?? The first four seem to have been incorrectly detected as > constants. How are they used? The first four are set once and not changed. Probably that's why it calls it a constant. > The last one is probably "too short". Or something. In this case, rs is a pyodbc row object. rs = cursor.fetchone() >> standard import "import re, requests" comes before "import pyodbc, sqlite3" >> (wrong-import-order) >> >> * So I switched them, and then it complained about that: >> >> standard import "import pyodbc, sqlite3" comes before "import re, requests" >> (wrong-import-order) >> >> ------------------------------------------------------------------------- >> >> You can't win with pylint... > > Probably that means it got confused by the alphabetization - "pyodbc" > should come before "re" and "requests", but "sqlite3" should come > after. Either fix the first problem by splitting them onto separate > lines, or ignore this as a cascaded error. > > My general principle is that things on one line should *belong* on one > line. So having "import re, requests" makes no sense, but I might have > something like "import os, sys" when the two modules are both used in > one single line of code and never again. Otherwise, splitting them out > is the easiest. I like to put them on a related line. Didn't know where re belonged, and I don't like putting them on single line each. >>>> +-------------------------+------------+ >>>> |superfluous-parens |3 | I like to surround 'or' >>>> statments with parens >>> >>> >>> I would need examples to comment >> >> >> >> if ("Please choose a state" in str(matches)): >> if (var == "val" or var2 == "val2"): > > Cut the parens. Easy! Maybe. I actually like my 'or' parens. Habit maybe, because of this situation: if (var == "val" or var2 == "val2") and (var3 == val3 or var4 == val4): >> It says "Used builtin function 'filter'. Using a list comprehension can be >> clearer. (bad-builtin)" > > Kill that message and keep using filter. Unfortunately, 'bad-builtin' caught 2 truly bad uses of built-ins (zip() and id()), so I'll leave that warning in. 2.7.11 built-ins: abs() divmod() input() open() staticmethod() all() enumerate() int() ord() str() any() eval() isinstance() pow() sum() basestring() execfile() issubclass() print() super() bin() file() iter() property() tuple() bool() filter() len() range() type() bytearray() float() list() raw_input() unichr() callable() format() locals() reduce() unicode() chr() frozenset() long() reload() vars() classmethod() getattr() map() repr() xrange() cmp() globals() max() reversed() zip() compile() hasattr() memoryview() round() __import__() complex() hash() min() set() delattr() help() next() setattr() dict() hex() object() slice() dir() id() oct() sorted() I probably would've used dict as an object name at some point, too. From rosuav at gmail.com Sun May 8 00:55:54 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2016 14:55:54 +1000 Subject: pylint woes In-Reply-To: References: Message-ID: On Sun, May 8, 2016 at 2:40 PM, DFS wrote: >>> It says "Used builtin function 'filter'. Using a list comprehension can >>> be >>> clearer. (bad-builtin)" >> >> >> Kill that message and keep using filter. > > > > Unfortunately, 'bad-builtin' caught 2 truly bad uses of built-ins (zip() and > id()), so I'll leave that warning in. > Hrm, that would be called "shadowing" built-ins, not bad use of them. Shadowing isn't usually a problem - unless you actually need id(), there's nothing wrong with using the name id for a database key. Very different from this message though. ChrisA From ian.g.kelly at gmail.com Sun May 8 01:09:18 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Sat, 7 May 2016 23:09:18 -0600 Subject: pylint woes In-Reply-To: References: Message-ID: On Sat, May 7, 2016 at 9:28 PM, DFS wrote: > But I think there are some pylint bugs here: > ------------------------------------------------------------------------- > > standard import "import pyodbc, sqlite3" comes before "import pyodbc, > sqlite3" (wrong-import-order) > > * complains that the line comes before itself? I think that it actually wants you to import sqlite3 (a standard library module) before pyodbc (a third-party module). The message is confusing because they happen to be on the same line. PEP8 has some advice on import ordering, which this probably follows. > > ------------------------------------------------------------------------- > > standard import "import re, requests" comes before "import pyodbc, sqlite3" > (wrong-import-order) > > * So I switched them, and then it complained about that: > > standard import "import pyodbc, sqlite3" comes before "import re, requests" > (wrong-import-order) Same thing. It wants the re import to come before pyodbc, and it wants sqlite3 to come before requests. From jussi.piitulainen at helsinki.fi Sun May 8 01:50:12 2016 From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen) Date: Sun, 08 May 2016 08:50:12 +0300 Subject: pylint woes References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> Message-ID: DFS writes: > The lists I actually use are: > > for j in range(len(nms)): > cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" > vals = nms[j],street[j],city[j],state[j],zipcd[j] > > > The enumerated version would be: > > ziplists = zip(nms,street,city,state,zipcd) > for nm,street,city,state,zipcd in ziplists: > cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" > vals = nm,street,city,state,zipcd > > > I guess the enumeration() is a little nicer to look at. Why do you > think it's more maintainable? The following variations avoid the naming of the result of zip at all, and also save a line or two, depending on what you actually do in the loop, without introducing overly long lines. Judge for yourself. You don't need to name the individual components, if you only actually use vals: for vals in zip(nms,street,city,state,zipcd): cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" Or you can opt to name the tuple and its components the other way around: for vals in zip(nms,street,city,state,zipcd): nm,street,city,state,zipcd = vals cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" From marko at pacujo.net Sun May 8 02:58:42 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Sun, 08 May 2016 09:58:42 +0300 Subject: Python is an Equal Opportunity Programming Language References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> <1462646456.2677797.601019937.6FC9DC8E@webmail.messagingengine.com> <572ea7e4$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87eg9dghnh.fsf@elektro.pacujo.net> Chris Angelico : > So the question is: Do we care about country equality or individual > equality? You can't have both. That's why there's been a long-standing initiative to split California into multiple states: Each state gets two senate seats, and California, being the most populous state, suffers. The biggest analogous outrage in the world, though, is that of the International Football Association Board (IFAB), where the [English] Football Association, the Scottish Football Association, the Football Association of Wales and the [Northern] Irish Football Association each have a vote while the ROW (aka FIFA) has four votes. Marko From marko at pacujo.net Sun May 8 03:06:10 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Sun, 08 May 2016 10:06:10 +0300 Subject: Python is an Equal Opportunity Programming Language References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> <1462646456.2677797.601019937.6FC9DC8E@webmail.messagingengine.com> <572ea7e4$0$1587$c3e8da3$5496439d@news.astraweb.com> <1462678858.3638806.601230905.2CF826C5@webmail.messagingengine.com> Message-ID: <87a8k1ghb1.fsf@elektro.pacujo.net> Random832 : > But that's not what it is. You would have, say, 1,000 tickets labeled > "green card" and 100,000 tickets labeled "no green card", and (say) > 12,000 Indian people and 50 Finnish people each get their turn drawing > from that same bucket. In your version, the Finnish people draw from a > bucket with 500 green card tickets and no "no green card" cards, and > the Indian people draw from a bucket with 500 green card tickets and > 11,500 "no green card" tickets. Or: In its majestic equality, the law forbids rich and poor alike to sleep under bridges, beg in the streets and steal loaves of bread. Marko From phd at phdru.name Sun May 8 03:32:52 2016 From: phd at phdru.name (Oleg Broytman) Date: Sun, 8 May 2016 09:32:52 +0200 Subject: SQLObject 2.2 Message-ID: <20160508073251.GB6648@phdru.name> Hello! I'm pleased to announce version 2.2.0, the first stable release of branch 2.2 of SQLObject. What's new in SQLObject ======================= Features & Interface -------------------- * Add function col.use_microseconds(True/False). Default is to use microseconds (True). * For MSSQL use datetime2(6) and time(6) columns. * Columns for ForeignKey are created using idType of the referenced table. Minor features -------------- * Add sqlbuilder.CONCAT to generate concatenation command (either using function CONCAT() or '||' operator). * Minor refactoring to pave the way to Python 3. * Document MSSQL server versions. Bugfixes -------- * Fix a bug: mxDateTime doesn't support microseconds; %s in mxDateTime format means ticks. Tests ----- * Speedup SQLite connections in tests. * Added new test help setupCyclicClasses to setup classes with mutual references. Contributors for this release are Andrew Ziem and Nathan Edwards. For a more complete list, please see the news: http://sqlobject.org/News.html What is SQLObject ================= SQLObject is an object-relational mapper. Your database tables are described as classes, and rows are instances of those classes. SQLObject is meant to be easy to use and quick to get started with. SQLObject supports a number of backends: MySQL, PostgreSQL, SQLite, Firebird, Sybase, MSSQL and MaxDB (also known as SAPDB). Python 2.6 or 2.7 is required. Where is SQLObject ================== Site: http://sqlobject.org Development: http://sqlobject.org/devel/ Mailing list: https://lists.sourceforge.net/mailman/listinfo/sqlobject-discuss Archives: http://news.gmane.org/gmane.comp.python.sqlobject Download: https://pypi.python.org/pypi/SQLObject/2.2.0 News and changes: http://sqlobject.org/News.html Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From steve+comp.lang.python at pearwood.info Sun May 8 05:07:06 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 08 May 2016 19:07:06 +1000 Subject: Python is an Equal Opportunity Programming Language References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> <1462646456.2677797.601019937.6FC9DC8E@webmail.messagingengine.com> <572ea7e4$0$1587$c3e8da3$5496439d@news.astraweb.com> <1462678858.3638806.601230905.2CF826C5@webmail.messagingengine.com> Message-ID: <572f01bc$0$2741$c3e8da3$76491128@news.astraweb.com> On Sunday 08 May 2016 13:40, Random832 wrote: > On Sat, May 7, 2016, at 22:43, Steven D'Aprano wrote: >> > If not for the quotas, a citizen of some other country would have an >> > equal chance to get a green card as a citizen of India or China. >> >> If you have a big hat with 5,000,000 tickets marked "Indian", and 500 >> tickets marked "Finish", and you stick your hand in the hat and rummage >> around and pick a random ticket, do you really think that you have an >> equal >> chance of selecting an Indian ticket and a Finish ticket? > > But that's not what it is. You would have, say, 1,000 tickets labeled > "green card" and 100,000 tickets labeled "no green card", and (say) > 12,000 Indian people and 50 Finnish people each get their turn drawing > from that same bucket. In your version, the Finnish people draw from a > bucket with 500 green card tickets and no "no green card" cards, and the > Indian people draw from a bucket with 500 green card tickets and 11,500 > "no green card" tickets. That's not how the green card works. The US immigration system is pretty messed up in many ways, being the overly complex and confusing product of many competing and contradictory requirements[1], but there's nothing even vaguely analogous to being randomly denied entry by pure chance. There's no such thing as "No Green Card" tickets that you can draw -- in principle at least, if you are refused entry, it is because you do not meet the requirements for a visa. (E.g. you have AIDS, are a known trafficker or smuggler, a member of the Communist Party in certain countries, have lied to the immigration officer, have publicly called for the violent overthrow of the US government, or are unable to provide sufficient evidence that you will go back home when your visa expires.) Aside: immigration officials have great power in deciding who meets the requirement for a visa, and the US government and courts are unable to over- rule them except in a matter of the interpretation of the law. With one exception: the rules for banning terrorists can be over-ruled by the Attorney-General and the Secretary of State. In other words, of all the things which a person might do to render themselves ineligible for a visa into the USA, there is no higher power able to over-ride the immigration official, *except* that if you fail the "No Terrorists Allowed" rule, the A- G and Sec of State, acting together, can choose to grant you an exemption. Go figure. (The intent, I presume, is that if somebody like General Pinochet of Chile is found guilty of crimes against humanity by the European courts, which would count as terrorism, the US could still grant him a visa.) [1] Some businesses want easy immigration, so they can drive down the cost of skilled or unskilled labour; some people want to prohibit work visas, to keep wages high; some want to allow people of diverse nationalities into the melting pot, while others want to keep the national character as it is without being overrun by large numbers of foreigners with strange or terrible customs and practices like sati, female genital mutilation[2], or effective social safety nets. [2] Apparently *male* genital mutilation is perfectly acceptable. -- Steve From gengyangcai at gmail.com Sun May 8 06:21:55 2016 From: gengyangcai at gmail.com (Cai Gengyang) Date: Sun, 8 May 2016 03:21:55 -0700 (PDT) Subject: Python PygLatin Message-ID: <31c06c3b-b622-4791-a607-d0ed2a741da1@googlegroups.com> I just "clicked" through the lesson on Conditionals and Control Flows and am on the lesson "PygLatin" . This will hopefully be a more interesting and interactive lesson because I will be building a PygLatin Translator ... It seems to me like it will take a long time before I can reach the point where I can become a master of programming languages and even create new programming languages and technologies. This is a long road ahead, but I believe it is worth it because the power of a new technology does eventually translate into money. If one looks at the Forbes List, you will see that there are 4 programmers amongst the top ten richest people in the world (Bill Gates, Mark Zuckerberg, Larry Ellison and Jeff Bezos) , a very large percentage. Science and Technology is in a sense the most egalitarian field in the world, because it involves using your brains and creativity. You don't need to have a father who is a director at Goldman Sachs or a mother who is the admissions officer at Harvard to succeed in this line. All you need is have the ability to create something that many users love to use. From beliavsky at aol.com Sun May 8 06:22:30 2016 From: beliavsky at aol.com (beliavsky at aol.com) Date: Sun, 8 May 2016 03:22:30 -0700 (PDT) Subject: Python is an Equal Opportunity Programming Language In-Reply-To: References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <1462608136.590130.600748113.38B9CEA2@webmail.messagingengine.com> Message-ID: On Saturday, May 7, 2016 at 4:02:32 AM UTC-4, Stephen Hansen wrote: > On Fri, May 6, 2016, at 11:43 PM, Gregory Ewing wrote: > > Steven D'Aprano wrote: > > > Who is setting and enforcing this quota, and given that only about 1 in 20 > > > Python programmers is a woman, do you think men are seriously missing out > > > on any opportunities? > > > > Suppose there are 100 people wanting to ask questions, and > > there is only time to answer 10 questions. If the 1 in 20 > > ratio holds, then 5 of those people are women and the other > > 95 are men. > > > > Alternating between men and women means that all of the > > women get their questions answered, and only 5/95 of the > > men. So in this example, if you're a woman you have a 100% > > chance of getting answered, and if you're a man you only > > have a 5.26% chance. > > > > Whether you think this is a good strategy or not, > > beliavsky is right that it's not "equal". > > This is a pedantically and nonsensical definition of "equal", that > ignores the many, many reasons why there are 1 in 20 women in that > conference. Its looking at the end effect and ignoring everything that > leads up to it, and deciding its instead special rights -- this is the > great argument against minorities getting a voice, that their requests > for equal *opportunity* are instead *special rights* that diminish the > established majority's entrenched power. > > Those women are dealing with suppression, discrimination and dismissal > on multiple levels that leave them in a disenfranchised position. The sex disparity in Python and in tech in general could be due in part to discrimination, but it could also be due to different male and female interests and (gasp) aptitudes on average. Are Asian-Americans over-represented in tech because whites have been suppressed? There are far more female than male teachers. I don't attribute it to anti-male suppression but to greater female interest in working with children. In our public middle school (grades 6-8, ages 11-13) there is a programming club that is open to girls. My son is shut out because of his sex. That is just as wrong as excluding him because of his skin color. I oppose such discrimination. From steve at pearwood.info Sun May 8 07:36:25 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 08 May 2016 21:36:25 +1000 Subject: pylint woes References: Message-ID: <572f24bc$0$1618$c3e8da3$5496439d@news.astraweb.com> On Sun, 8 May 2016 11:16 am, DFS wrote: > address data is scraped from a website: > > names = tree.xpath() > addr = tree.xpath() Why are you scraping the data twice? names = addr = tree.xpath() or if you prefer the old-fashioned: names = tree.xpath() addr = names but that raises the question, how can you describe the same set of data as both "names" and "addr[esses]" and have them both be accurate? > I want to store the data atomically, I'm not really sure what you mean by "atomically" here. I know what *I* mean by "atomically", which is to describe an operation which either succeeds entirely or fails. But I don't know what you mean by it. > so I parse street, city, state, and > zip into their own lists. None of which is atomic. > "1250 Peachtree Rd, Atlanta, GA 30303 > > street = [s.split(',')[0] for s in addr] > city = [c.split(',')[1].strip() for c in addr] > state = [s[-8:][:2] for s in addr] > zipcd = [z[-5:] for z in addr] At this point, instead of iterating over the same list four times, doing the same thing over and over again, you should do things the old-fashioned way: streets, cities, states, zipcodes = [], [], [], [] for word in addr: items = word.split(',') streets.append(items[0]) cities.append(items[1].strip()) states.append(word[-8:-2]) zipcodes.append(word[-5:]) Oh, and use better names. "street" is a single street, not a list of streets, note plural. -- Steven From steve at pearwood.info Sun May 8 08:02:50 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 08 May 2016 22:02:50 +1000 Subject: Pylint prefers list comprehension over filter... References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> <572E65B2.4010305@icloud.com> Message-ID: <572f2aec$0$1583$c3e8da3$5496439d@news.astraweb.com> On Sun, 8 May 2016 08:01 am, Christopher Reimer wrote: > On 5/7/2016 2:22 PM, Chris Angelico wrote: >> Also, be sure you read this part of PEP 8: >> >> https://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds > Recruiters and hiring managers *are* hobgoblins with with little minds. > And definitely not PEP8-complaint. :) Do you think that recruiters and hiring managers read your code at all, let alone that they read it will an eye to PEP 8 compliance? No recruiter[1] will do this. No recruiter will even know what PEP 8 is. They're looking for technical buzzwords ("ten years experience with Django"), they aren't qualified to judge whether your Django code is good, bad or indifferent. That's up to the client. A hiring manager with a technical background might, once you are in consideration for the job. More likely they will delegate any judgement to a technical manager, or programmer, who may or may not be a hobgoblin with a little mind. There are plenty of programmers who are obsessive about following PEP 8 even when it makes the code worse: mywidget.component['key'] = (mywidget.grippley.count_item(spam or eggs) + 1) (I've met plenty of technical people who are opinionated and badly informed, not just managers. Just last week, I was told by one programmer that Mersenne Twister, the default RNG used by Python, is well-known to be biased, and that "everybody knows" not to use it to choose an item from a list because it is documented "everywhere" as being more likely to choose the first or the last item than any of the others.) [1] Sweeping generalisation. In a world of 7 billion people, there's probably one or two exceptions somewhere. -- Steven From steve at pearwood.info Sun May 8 08:08:03 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 08 May 2016 22:08:03 +1000 Subject: Python is an Equal Opportunity Programming Language References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> <87lh3lj1ye.fsf@elektro.pacujo.net> Message-ID: <572f2c25$0$1589$c3e8da3$5496439d@news.astraweb.com> On Sun, 8 May 2016 01:57 am, Marko Rauhamaa wrote: > A functional, enlightened, prosperous democracy is a very recent > historical anomaly. You don't want to jeopardize it na?vely. Perhaps by implementing per-country limits on immigration? *wink* -- Steven From darcy at VybeNetworks.com Sun May 8 08:50:06 2016 From: darcy at VybeNetworks.com (D'Arcy J.M. Cain) Date: Sun, 8 May 2016 08:50:06 -0400 Subject: pylint woes In-Reply-To: References: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> Message-ID: <20160508085006.020cef65@imp> On Sun, 8 May 2016 14:21:49 +1000 Chris Angelico wrote: > if verbose: > verbiage = print > else: > def verbiage(*args): pass I have never understood why the def couldn't start on the same line as the else: if verbose: verbiage = print else: def verbiage(*args): pass The colon effectively starts a block so why not allow it? By the way, I think you meant "def verbiage(*args, **kws): pass" > Then, instead of "if verbose: print(var)", you would use > "verbiage(var)". Of course, you want something better than "verbiage" > as your name; the nature of your verbose output might give a clue as > to what name would work. How about "print"? if not verbose: def print(*args, **kws): pass -- D'Arcy J.M. Cain Vybe Networks Inc. http://www.VybeNetworks.com/ IM:darcy at Vex.Net VoIP: sip:darcy at VybeNetworks.com From rosuav at gmail.com Sun May 8 09:01:55 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2016 23:01:55 +1000 Subject: pylint woes In-Reply-To: <20160508085006.020cef65@imp> References: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> <20160508085006.020cef65@imp> Message-ID: On Sun, May 8, 2016 at 10:50 PM, D'Arcy J.M. Cain wrote: > On Sun, 8 May 2016 14:21:49 +1000 > Chris Angelico wrote: >> if verbose: >> verbiage = print >> else: >> def verbiage(*args): pass > > I have never understood why the def couldn't start on the same line as > the else: > > if verbose: verbiage = print > else: def verbiage(*args): pass > > The colon effectively starts a block so why not allow it? Having two colons makes it a bit messy, so I can fully accept that this *shouldn't* be done. Whether or not it's reasonable that it *can't* be done is a question for the parser; but even if the parser permitted it, I would expect style guides to advise against it. > By the way, I think you meant "def verbiage(*args, **kws): pass" In the general case, yes. But in this specific case, I actually prefer not to accept keyword args in a null function; maybe permit sep=" " and end="\n", but if someone sets file or flush, it's probably a mistake (you most likely don't want verbiage("message", file=logfile) to silently not do it). YMMV; maybe you want that, so yeah, toss in the kwargs absorber. >> Then, instead of "if verbose: print(var)", you would use >> "verbiage(var)". Of course, you want something better than "verbiage" >> as your name; the nature of your verbose output might give a clue as >> to what name would work. > > How about "print"? > > if not verbose: > def print(*args, **kws): pass The danger of that is that it's too general. I like to recommend a little thing called "IIDPIO debugging" - If In Doubt, Print It Out. That means: If you have no idea what a piece of code is doing, slap in a print() call somewhere. It'll tell you that (a) the code is actually being executed, and (b) whatever info you put between the parens (ideally, some key variable or parameter). Part A is often the important bit :) The trouble with a verbose flag controlling all print() calls is that IIDPIO debugging suddenly doesn't work; plus, it's easy to copy and paste code to some other module and not notice that you don't have a verbosity check at the top, and then wonder why disabling verbose doesn't fully work. Both problems are solved by having a dedicated spam function, which will simply error out if you didn't set it up properly. But again, if you know what you're doing, go for it! This is exactly why print became a function in Py3 - so that you *can* override it. ChrisA From __peter__ at web.de Sun May 8 10:12:20 2016 From: __peter__ at web.de (Peter Otten) Date: Sun, 08 May 2016 16:12:20 +0200 Subject: pylint woes References: Message-ID: Chris Angelico wrote: > On Sun, May 8, 2016 at 1:28 PM, DFS wrote: >> Invalid constant name "cityzip" (invalid-name) >> Invalid constant name "state" (invalid-name) >> Invalid constant name "miles" (invalid-name) >> Invalid constant name "store" (invalid-name) >> Invalid variable name "rs" (invalid-name) > > ... huh?? The first four seem to have been incorrectly detected as > constants. How are they used? As globals. pylint doesn't like it when you put your normal code outside a function, and I agree with the general idea. The problem is that it's not smart enough to recognize the exceptions like plus_one = make_adder(1) where plus_one obeys the naming convention for a function, but the linter asks you to change the line to PLUS_ONE = make_adder(1) In Row = collections.namedtuple("Row", "alpha beta") though pylint does recognize that collections.namedtuple() produces a class, so there might also be a way to teach it how to handle the custom factory. The OP should of course put the whole shebang into a main() function. That also simplifies it to determine which values should be passed as arguments when he breaks his blob into smaller parts. From __peter__ at web.de Sun May 8 10:19:56 2016 From: __peter__ at web.de (Peter Otten) Date: Sun, 08 May 2016 16:19:56 +0200 Subject: pylint woes References: <572E3967.9060206@icloud.com> Message-ID: DFS wrote: > On 5/7/2016 2:52 PM, Christopher Reimer wrote: >> On 5/7/2016 9:51 AM, DFS wrote: >>> Has anyone ever in history gotten 10/10 from pylint for a non-trivial >>> program? >> >> I routinely get 10/10 for my code. While pylint isn't perfect and >> idiosyncratic at times, it's a useful tool to help break bad programming >> habits. Since I came from a Java background, I had to unlearn everything >> from Java before I could write Pythonic code. It might help to use an >> IDE that offers PEP8-compliant code suggestions (I use PyCharm IDE). >> >>> That's about as good as it's gonna get! >> >> You can do better. > > 10/10 on pylint isn't better. Not always, but where you and pylint disagree I'm more likely to side with the tool ;) > It's being robotic and conforming to the > opinions of the author of that app. The problem are the tool's limitations, the "being robotic" rather than following someone else's opinions. > In fact, I think: > > import os, sys, time, socket > > is much more readable than, and preferable to, > > import os > import sys > import time > import socket > > but pylint complains about the former. Do you use version control? >> You should strive for 10/10 whenever possible, > > nah > > >> figure out why you fall short and ask for help on the parts that don't >> make sense. > > I actually agree with ~3/4 of the suggestions it makes. My code ran > fine before pylint tore it a new one, and it doesn't appear to run any > better after making various fixes. Do you write unit tests? From ian.g.kelly at gmail.com Sun May 8 10:24:33 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Sun, 8 May 2016 08:24:33 -0600 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: <20160508064113.GU12028@ando.pearwood.info> References: <20160508015932.GS12028@ando.pearwood.info> <20160508064113.GU12028@ando.pearwood.info> Message-ID: On May 8, 2016 12:42 AM, "Steven D'Aprano" wrote: > > def pvariance(data, mu=None): > if iter(data) is data: > data = list(data) > n = len(data) > if n < 1: > raise StatisticsError('pvariance requires at least one data point') > ss = _ss(data, mu) > T, ss = _ss(data, mu) Copy-paste error? > return _convert(ss/n, T) From nospam at dfs.com Sun May 8 10:25:59 2016 From: nospam at dfs.com (DFS) Date: Sun, 8 May 2016 10:25:59 -0400 Subject: pylint woes In-Reply-To: References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> Message-ID: On 5/8/2016 1:50 AM, Jussi Piitulainen wrote: > DFS writes: > >> The lists I actually use are: >> >> for j in range(len(nms)): >> cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" >> vals = nms[j],street[j],city[j],state[j],zipcd[j] >> >> >> The enumerated version would be: >> >> ziplists = zip(nms,street,city,state,zipcd) >> for nm,street,city,state,zipcd in ziplists: >> cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" >> vals = nm,street,city,state,zipcd >> >> >> I guess the enumeration() is a little nicer to look at. Why do you >> think it's more maintainable? > > The following variations avoid the naming of the result of zip at all, > and also save a line or two, depending on what you actually do in the > loop, without introducing overly long lines. Judge for yourself. I tried: for nm,street,city,state,zipcd in zip(nms,street,city,state,zipcd): but felt it was too long and wordy. > You don't need to name the individual components, if you only actually > use vals: > > for vals in zip(nms,street,city,state,zipcd): > cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" I like that one. But I do one more thing (get a category ID) than just use the vals. -------------------------------------------------------------------- ziplists = zip(categories,names,streets,cities,states,zipcodes) for category,name,street,city,state,zipcode in ziplists: dupeRow, pyodbcErr = False, False catID = getDataID("catID","CATEGORIES","catDesc",category) cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?,?,?,?,?)" vals = datasrcID,searchID,catID,name,street,city,state,zipcode,str(loaddt) try: db.execute(cSQL, vals) except (pyodbc.Error) as programError: if str(programError).find("UNIQUE constraint failed") > 0: dupeRow = True dupes +=1 print " * duplicate address found: "+name+", "+street else: pyodbcErr = True print "ODBC error: %s " % programError addrReturned += 1 if not dupeRow and not pyodbcErr: addrSaved += 1 if addrWant != "all": if addrSaved >= addrWant: break conn.commit() -------------------------------------------------------------------- That's the 'post to db' routine > Or you can opt to name the tuple and its components the other way > around: > > for vals in zip(nms,street,city,state,zipcd): > nm,street,city,state,zipcd = vals > cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" I like the first one better. python is awesome, but too many options for doing the same thing also makes it difficult. For me, anyway. Thanks From nospam at dfs.com Sun May 8 10:26:26 2016 From: nospam at dfs.com (DFS) Date: Sun, 8 May 2016 10:26:26 -0400 Subject: pylint woes In-Reply-To: References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> <1462679218.789727.601235065.67122D8D@webmail.messagingengine.com> Message-ID: On 5/7/2016 11:46 PM, Stephen Hansen wrote: > On Sat, May 7, 2016, at 08:04 PM, DFS wrote: >> The lists I actually use are: >> >> for j in range(len(nms)): >> cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" >> vals = nms[j],street[j],city[j],state[j],zipcd[j] >> >> >> The enumerated version would be: >> >> ziplists = zip(nms,street,city,state,zipcd) >> for nm,street,city,state,zipcd in ziplists: >> cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" >> vals = nm,street,city,state,zipcd >> >> >> I guess the enumeration() is a little nicer to look at. Why do you >> think it's more maintainable? > > Code is read more then its written. > > That which is nicer to look at, therefore, is easier to read. > > That which is easier to read is easier to maintain. > > Beyond that, its simpler, and more clearly articulates in the local > space what's going on. That last one sounds like an art critic trying to exlain why Jackson Pollock's work doesn't suck. >> Aside: I haven't tried, but is 'names' a bad idea or illegal for the >> name of a python list or variable? > > Nothing wrong with names. Or 'name', for that matter. Try to avoid > abbreviations. np From rosuav at gmail.com Sun May 8 10:36:04 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 9 May 2016 00:36:04 +1000 Subject: pylint woes In-Reply-To: References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> Message-ID: On Mon, May 9, 2016 at 12:25 AM, DFS wrote: > for category,name,street,city,state,zipcode in ziplists: > try: db.execute(cSQL, vals) > except (pyodbc.Error) as programError: > if str(programError).find("UNIQUE constraint failed") > 0: > dupeRow = True > dupes +=1 > print " * duplicate address found: "+name+", "+street > else: > pyodbcErr = True > print "ODBC error: %s " % programError > conn.commit() > -------------------------------------------------------------------- > ... and then you just commit???!? ChrisA From nospam at dfs.com Sun May 8 11:06:15 2016 From: nospam at dfs.com (DFS) Date: Sun, 8 May 2016 11:06:15 -0400 Subject: pylint woes In-Reply-To: References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> Message-ID: On 5/8/2016 10:36 AM, Chris Angelico wrote: > On Mon, May 9, 2016 at 12:25 AM, DFS wrote: >> for category,name,street,city,state,zipcode in ziplists: >> try: db.execute(cSQL, vals) >> except (pyodbc.Error) as programError: >> if str(programError).find("UNIQUE constraint failed") > 0: >> dupeRow = True >> dupes +=1 >> print " * duplicate address found: "+name+", "+street >> else: >> pyodbcErr = True >> print "ODBC error: %s " % programError >> conn.commit() >> -------------------------------------------------------------------- >> > > ... and then you just commit???!? > > ChrisA That's what commit() does. From rustompmody at gmail.com Sun May 8 11:09:11 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Sun, 8 May 2016 08:09:11 -0700 (PDT) Subject: Python is an Equal Opportunity Programming Language In-Reply-To: <572f2c25$0$1589$c3e8da3$5496439d@news.astraweb.com> References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> <87lh3lj1ye.fsf@elektro.pacujo.net> <572f2c25$0$1589$c3e8da3$5496439d@news.astraweb.com> Message-ID: <96444428-5553-4471-929e-ec800e09d428@googlegroups.com> On Sunday, May 8, 2016 at 5:38:18 PM UTC+5:30, Steven D'Aprano wrote: > On Sun, 8 May 2016 01:57 am, Marko Rauhamaa wrote: > > > A functional, enlightened, prosperous democracy is a very recent > > historical anomaly. You don't want to jeopardize it na?vely. > > Perhaps by implementing per-country limits on immigration? > > *wink* See: https://www.washingtonpost.com/news/rampage/wp/2016/05/07/ivy-league-economist-interrogated-for-doing-math-on-american-airlines-flight/ Closing line: "In America today, the only thing more terrifying than foreigners is...math." Wonder how close to terrorists pythonists are From me+python at ixokai.io Sun May 8 11:11:41 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sun, 08 May 2016 08:11:41 -0700 Subject: pylint woes In-Reply-To: References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> Message-ID: <1462720301.895956.601491969.5975510C@webmail.messagingengine.com> On Sun, May 8, 2016, at 07:25 AM, DFS wrote: > for nm,street,city,state,zipcd in zip(nms,street,city,state,zipcd): > > for vals in zip(nms,street,city,state,zipcd): > > nm,street,city,state,zipcd = vals > > cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" > > > I like the first one better. python is awesome, but too many options > for doing the same thing also makes it difficult. For me, anyway. Eeh, Now you're just making trouble for yourself. for name, street, city, state, zipcd in zip(names, streets, cities, states, zipcds): .... may be sorta vaguely long, but its not that long. Just do it and move on. Get over whatever makes you not like it. -- Stephen Hansen m e @ i x o k a i . i o From me+python at ixokai.io Sun May 8 11:15:36 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sun, 08 May 2016 08:15:36 -0700 Subject: pylint woes In-Reply-To: References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> Message-ID: <1462720536.896455.601493505.776FAC7D@webmail.messagingengine.com> On Sun, May 8, 2016, at 08:06 AM, DFS wrote: > On 5/8/2016 10:36 AM, Chris Angelico wrote: > > ... and then you just commit???!? > > > > That's what commit() does. > I assure you, he knows what commit does :) The point is, you don't usually commit after an error happens. You rollback. Or correct the data. Since the data didn't go in, there should (in theory) be nothing TO commit if an error happens. Or, there should be partial data in that needs a rollback before you decide to do something else. -- Stephen Hansen m e @ i x o k a i . i o From rosuav at gmail.com Sun May 8 11:15:38 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 9 May 2016 01:15:38 +1000 Subject: pylint woes In-Reply-To: References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> Message-ID: On Mon, May 9, 2016 at 1:06 AM, DFS wrote: > On 5/8/2016 10:36 AM, Chris Angelico wrote: >> >> On Mon, May 9, 2016 at 12:25 AM, DFS wrote: >>> >>> for category,name,street,city,state,zipcode in ziplists: >>> try: db.execute(cSQL, vals) >>> except (pyodbc.Error) as programError: >>> if str(programError).find("UNIQUE constraint failed") > 0: >>> dupeRow = True >>> dupes +=1 >>> print " * duplicate address found: "+name+", "+street >>> else: >>> pyodbcErr = True >>> print "ODBC error: %s " % programError >>> conn.commit() >>> -------------------------------------------------------------------- >>> >> >> ... and then you just commit???!? >> >> ChrisA > > > > That's what commit() does. Yes. Even if you got an error part way through, you just blithely commit. What?! And yes, I am flat-out boggling at this. ChrisA From sergio.am.spina at gmail.com Sun May 8 11:18:39 2016 From: sergio.am.spina at gmail.com (Sergio Spina) Date: Sun, 8 May 2016 08:18:39 -0700 (PDT) Subject: Help for a complex RE Message-ID: <2aa55bd8-2ea4-41f7-b188-d45dff7d3bb7@googlegroups.com> In the following ipython session: > Python 3.5.1+ (default, Feb 24 2016, 11:28:57) > Type "copyright", "credits" or "license" for more information. > > IPython 2.3.0 -- An enhanced Interactive Python. > > In [1]: import re > > In [2]: patt = r""" # the match pattern is: > ...: .+ # one or more characters > ...: [ ] # followed by a space > ...: (?=[@#D]:) # that is followed by one of the > ...: # chars "@#D" and a colon ":" > ...: """ > > In [3]: pattern = re.compile(patt, re.VERBOSE) > > In [4]: m = pattern.match("Jun at i Bun#i @:Janji") > > In [5]: m.group() > Out[5]: 'Jun at i Bun#i ' > > In [6]: m = pattern.match("Jun at i Bun#i @:Janji D:Banji") > > In [7]: m.group() > Out[7]: 'Jun at i Bun#i @:Janji ' > > In [8]: m = pattern.match("Jun at i Bun#i @:Janji D:Banji #:Junji") > > In [9]: m.group() > Out[9]: 'Jun at i Bun#i @:Janji D:Banji ' Why the regex engine stops the search at last piece of string? Why not at the first match of the group "@:"? What can it be a regex pattern with the following result? > In [1]: m = pattern.match("Jun at i Bun#i @:Janji D:Banji #:Junji") > > In [2]: m.group() > Out[2]: 'Jun at i Bun#i ' From steve at pearwood.info Sun May 8 11:34:08 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 09 May 2016 01:34:08 +1000 Subject: [Python-ideas] Boolean parameters guidelines References: <20160508015932.GS12028@ando.pearwood.info> <20160508064113.GU12028@ando.pearwood.info> Message-ID: <572f5c71$0$1597$c3e8da3$5496439d@news.astraweb.com> Not sure why this has migrated to this list instead of Python-Ideas. Possibly a copy-paste error? *wink* On Mon, 9 May 2016 12:24 am, Ian Kelly wrote: > On May 8, 2016 12:42 AM, "Steven D'Aprano" wrote: >> >> def pvariance(data, mu=None): >> if iter(data) is data: >> data = list(data) >> n = len(data) >> if n < 1: >> raise StatisticsError('pvariance requires at least one data > point') >> ss = _ss(data, mu) >> T, ss = _ss(data, mu) > > Copy-paste error? I have literally no idea how that happened. Anyway, it's fixed now. -- Steven From steve at pearwood.info Sun May 8 11:51:25 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 09 May 2016 01:51:25 +1000 Subject: pylint woes References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> Message-ID: <572f607f$0$1588$c3e8da3$5496439d@news.astraweb.com> On Mon, 9 May 2016 12:25 am, DFS wrote: >>> for j in range(len(nms)): >>> cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" >>> vals = nms[j],street[j],city[j],state[j],zipcd[j] Why are you assigning cSQL to the same string over and over again? Sure, assignments are cheap, but they're not infinitely cheap. They still have a cost. Instead of paying that cost once, you pay it over and over again, which adds up. Worse, it is misleading. I had to read that code snippet three or four times before I realised that cSQL was exactly the same each time. > I tried: > > for nm,street,city,state,zipcd in zip(nms,street,city,state,zipcd): > > but felt it was too long and wordy. It's long and wordy because you're doing something long and wordy. It is *inherently* long and wordy to process five things, whether you write it as: for i in range(len(names)): name = names[i] street = streets[i] city = cities[i] state = states[i] zipcode = zipcodes[i] process(...) or as: for name, street, city, state, zipcode in zip( names, streets, cities, states, zipcodes ): process(...) > I like the first one better. python is awesome, but too many options > for doing the same thing also makes it difficult. For me, anyway. That's the difference between a master and an apprentice. The apprentice likes to follow fixed steps the same way each time. The master craftsman knows her tools backwards, and can choose the right tool for the job, and when the choice of tool really doesn't matter and you can use whatever happens to be the closest to hand. -- Steven From ian.g.kelly at gmail.com Sun May 8 12:08:46 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Sun, 8 May 2016 10:08:46 -0600 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: <572f5c71$0$1597$c3e8da3$5496439d@news.astraweb.com> References: <20160508015932.GS12028@ando.pearwood.info> <20160508064113.GU12028@ando.pearwood.info> <572f5c71$0$1597$c3e8da3$5496439d@news.astraweb.com> Message-ID: On May 8, 2016 9:37 AM, "Steven D'Aprano" wrote: > > Not sure why this has migrated to this list instead of Python-Ideas. Because Gmail has somehow never gotten around to implementing reply-to-list and I'm terrible at choosing the right one. From __peter__ at web.de Sun May 8 12:15:25 2016 From: __peter__ at web.de (Peter Otten) Date: Sun, 08 May 2016 18:15:25 +0200 Subject: Help for a complex RE References: <2aa55bd8-2ea4-41f7-b188-d45dff7d3bb7@googlegroups.com> Message-ID: Sergio Spina wrote: > In the following ipython session: > >> Python 3.5.1+ (default, Feb 24 2016, 11:28:57) >> Type "copyright", "credits" or "license" for more information. >> >> IPython 2.3.0 -- An enhanced Interactive Python. >> >> In [1]: import re >> >> In [2]: patt = r""" # the match pattern is: >> ...: .+ # one or more characters >> ...: [ ] # followed by a space >> ...: (?=[@#D]:) # that is followed by one of the >> ...: # chars "@#D" and a colon ":" >> ...: """ >> >> In [3]: pattern = re.compile(patt, re.VERBOSE) >> >> In [4]: m = pattern.match("Jun at i Bun#i @:Janji") >> >> In [5]: m.group() >> Out[5]: 'Jun at i Bun#i ' >> >> In [6]: m = pattern.match("Jun at i Bun#i @:Janji D:Banji") >> >> In [7]: m.group() >> Out[7]: 'Jun at i Bun#i @:Janji ' >> >> In [8]: m = pattern.match("Jun at i Bun#i @:Janji D:Banji #:Junji") >> >> In [9]: m.group() >> Out[9]: 'Jun at i Bun#i @:Janji D:Banji ' > > Why the regex engine stops the search at last piece of string? > Why not at the first match of the group "@:"? > What can it be a regex pattern with the following result? > >> In [1]: m = pattern.match("Jun at i Bun#i @:Janji D:Banji #:Junji") >> >> In [2]: m.group() >> Out[2]: 'Jun at i Bun#i ' Compare: >>> re.compile("a+").match("aaaa").group() 'aaaa' >>> re.compile("a+?").match("aaaa").group() 'a' By default pattern matching is "greedy" -- the ".+" part of your regex matches as many characters as possible. Adding a ? like in ".+?" triggers non-greedy matching. From larry.martell at gmail.com Sun May 8 12:25:02 2016 From: larry.martell at gmail.com (Larry Martell) Date: Sun, 8 May 2016 12:25:02 -0400 Subject: starting docker container messes up terminal settings In-Reply-To: References: Message-ID: On Mon, May 2, 2016 at 10:28 AM, Larry Martell wrote: > On Mon, May 2, 2016 at 10:08 AM, Joaquin Alzola > wrote: >>>I am starting a docker container from a subprocess.Popen and it works, but when the script returns, the terminal settings of my shell are messed up. Nothing is echoed and return doesn't cause a >newline. I can fix this with 'tset' in the terminal, but I don't want to require that. Has anyone here worked with docker and had seen and solved this issue? >> >> It is good to put part of the code you think is causing the error (Popen subprocess) > > cmd = ['sudo', > 'docker', > 'run', > '-t', > '-i', > 'elucidbio/capdata:v2', > 'bash' > ] > p = subprocess.Popen(cmd, stdout=subprocess.PIPE, > stderr=subprocess.STDOUT) If anyone cares (which is doubtful), I fixed this by removing -t and passing in stdin=None to Popen. From steve at pearwood.info Sun May 8 12:27:14 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 09 May 2016 02:27:14 +1000 Subject: Python is an Equal Opportunity Programming Language References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <1462608136.590130.600748113.38B9CEA2@webmail.messagingengine.com> Message-ID: <572f68e4$0$1620$c3e8da3$5496439d@news.astraweb.com> On Sun, 8 May 2016 08:22 pm, beliavsky at aol.com wrote: > There are > far more female than male teachers. I don't attribute it to anti-male > suppression but to greater female interest in working with children. Of course there is suppression of male teachers, particularly but not only for very young children. http://www.cea-ace.ca/education-canada/article/false-accusations-growing-fear-classroom http://www.adelaidenow.com.au/news/south-australia/men-too-scared-to-teach-for-fear-of-being-falsely-accused-of-childsex-offences/story-fni6uo1m-1226913910688 http://abcnews.go.com/GMA/Parenting/Story?id=6070282&page=1 Quote: "I know they're thinking, 'He must be a predator or something. He must be some type of pedophile. Why is he in here? He should be working for the city, dumping trash, a janitor or something of that nature,'" Maiden said. Not only do parents' gender bias drive men out of teaching, but that same gender bias influences the choices men make themselves: - fear of false accusations of being a sexual predator; - fear of having your sexuality questioned ("looking after kids is women's work"); - low status and pay. Most men are extremely status-conscious (if often unconsciously) and then recognise the status (and pay!) of teachers is low: University lecturers have medium status; University tutors have less; High school teachers less again; Primary school teachers even less; And pre-school teachers have practical no status. Basically, the younger the child, the lower the status and the pay. So most men simply don't even consider it as a job. Funny the lies we, as a society, tell ourselves. As they say, don't listen to what people *say* they value, look at what they spend their money on. Our society says that we value our young kids beyond all price, but we entrust them into the hands of underpaid, overworked pre-school teachers who are practically considered drudges. Meanwhile we pay millions of dollars to over-muscled and under-socialised man-children to chase after a ball for a few minutes a week. > In our public middle school (grades 6-8, ages 11-13) there is a > programming club that is open to girls. My son is shut out because of his > sex. That is just as wrong as excluding him because of his skin color. I > oppose such discrimination. Unless there is a separate programming club for boys, or mixed boys and girls, so would I. But the mere existence of a girls-only programming club is not in and of itself discriminatory. I don't know about programming, but in terms of general schooling: - on average, boys do better in mixed sex classes than in same sex classes; - but for girls it is the other way around. Since boys don't suffer any loss from mixed sex classes, but girls do, it is common sense to offer girls a same sex option to let them catch up. Relevant: http://www.robeastaway.com/blog/boys-versus-girls -- Steven From sergio.am.spina at gmail.com Sun May 8 12:32:02 2016 From: sergio.am.spina at gmail.com (Sergio Spina) Date: Sun, 8 May 2016 09:32:02 -0700 (PDT) Subject: Help for a complex RE In-Reply-To: References: <2aa55bd8-2ea4-41f7-b188-d45dff7d3bb7@googlegroups.com> Message-ID: <6a3fe5cd-0ba9-4017-a763-76c896b8c843@googlegroups.com> Il giorno domenica 8 maggio 2016 18:16:56 UTC+2, Peter Otten ha scritto: > Sergio Spina wrote: > > > In the following ipython session: > > > >> Python 3.5.1+ (default, Feb 24 2016, 11:28:57) > >> Type "copyright", "credits" or "license" for more information. > >> > >> IPython 2.3.0 -- An enhanced Interactive Python. > >> > >> In [1]: import re > >> > >> In [2]: patt = r""" # the match pattern is: > >> ...: .+ # one or more characters > >> ...: [ ] # followed by a space > >> ...: (?=[@#D]:) # that is followed by one of the > >> ...: # chars "@#D" and a colon ":" > >> ...: """ > >> > >> In [3]: pattern = re.compile(patt, re.VERBOSE) > >> > >> In [4]: m = pattern.match("Jun at i Bun#i @:Janji") > >> > >> In [5]: m.group() > >> Out[5]: 'Jun at i Bun#i ' > >> > >> In [6]: m = pattern.match("Jun at i Bun#i @:Janji D:Banji") > >> > >> In [7]: m.group() > >> Out[7]: 'Jun at i Bun#i @:Janji ' > >> > >> In [8]: m = pattern.match("Jun at i Bun#i @:Janji D:Banji #:Junji") > >> > >> In [9]: m.group() > >> Out[9]: 'Jun at i Bun#i @:Janji D:Banji ' > > > > Why the regex engine stops the search at last piece of string? > > Why not at the first match of the group "@:"? > > What can it be a regex pattern with the following result? > > > >> In [1]: m = pattern.match("Jun at i Bun#i @:Janji D:Banji #:Junji") > >> > >> In [2]: m.group() > >> Out[2]: 'Jun at i Bun#i ' > > Compare: > > >>> re.compile("a+").match("aaaa").group() > 'aaaa' > >>> re.compile("a+?").match("aaaa").group() > 'a' > > By default pattern matching is "greedy" -- the ".+" part of your regex > matches as many characters as possible. Adding a ? like in ".+?" triggers > non-greedy matching. > In [2]: patt = r""" # the match pattern is: > ...: .+ # one or more characters > ...: [ ] # followed by a space > ...: (?=[@#D]:) # ONLY IF is followed by one of the <<< please note > ...: # chars "@#D" and a colon ":" > ...: """ >From the python documentation > (?=...) > Matches if ... matches next, but doesn't consume any of the string. > This is called a lookahead assertion. For example, > Isaac (?=Asimov) will match 'Isaac ' only if it's followed by 'Asimov'. I know about greedy and not-greedy, but the problem remains. From christopher_reimer at icloud.com Sun May 8 12:57:31 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Sun, 08 May 2016 09:57:31 -0700 Subject: Pylint prefers list comprehension over filter... In-Reply-To: <572f2aec$0$1583$c3e8da3$5496439d@news.astraweb.com> References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> <572E65B2.4010305@icloud.com> <572f2aec$0$1583$c3e8da3$5496439d@news.astraweb.com> Message-ID: <572F6FFB.4060900@icloud.com> On 5/8/2016 5:02 AM, Steven D'Aprano wrote: > On Sun, 8 May 2016 08:01 am, Christopher Reimer wrote: > >> On 5/7/2016 2:22 PM, Chris Angelico wrote: > >>> Also, be sure you read this part of PEP 8: >>> >>> > https://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds >> Recruiters and hiring managers *are* hobgoblins with with little minds. >> And definitely not PEP8-complaint. :) > > Do you think that recruiters and hiring managers read your code at all, let > alone that they read it will an eye to PEP 8 compliance? I meant more than a few technical professionals who got pushed out of the job to become recruiters. They tend to ask about everything on the resume to poke holes where they can. Each time I tightened the wording of my resume whenever they exposed something. If I posted a GitHub link on my resume, they *would* check it out. If they know Python, they *would* run it. As for PEP 8 compliance, it was a joke (see the smiley). Laugh, it was funny. > No recruiter[1] will do this. No recruiter will even know what PEP 8 is. > They're looking for technical buzzwords ("ten years experience with > Django"), they aren't qualified to judge whether your Django code is good, > bad or indifferent. That's up to the client. When I used to apply to Linux system admin jobs, the recruiters would have a checklist box for the "Red Hat GUI Thing" as an absolute requirement. My Linux work experience was exclusively remote command line. On the rare occasions that I have used the Linux GUI, I always used what got installed as the default GUI for Fedora or Mint. The first time I said I didn't know what the "Red Hat GUI Thing" was and explained that I was fast learner, the recruiter hung up on me. I tried to argue with other recruiters that I knew the command line equivalent for the GUI. No dice. Some recruiters even accused me of making up techno-babble. They all hung up on me. After a dozen phone calls like that (all for positions at different companies), I generally stopped applying to Linux jobs. This year I built an inexpensive PC to run Linux and installed the current Red Hat Linux. Guess what? The "Red Hat GUI Thing" wasn't installed. In fact, the GUI was Gnome by default. According to my coworkers, Red Hat started phasing out their own branded GUI several years ago. I guess the recruiters haven't gotten the memo yet, as I had a phone call about the "Red Hat GUI Thing" last year. > A hiring manager with a technical background might, once you are in > consideration for the job. More likely they will delegate any judgement to > a technical manager, or programmer, who may or may not be a hobgoblin with > a little mind. Every hiring manager I've ever interviewed with had a technical background. The only exception was when the final interview was with the marketing director at hardware company. I declined to take the job. If you know your Dilbert, it's bad luck when a hardware company is run by the marketing department. I wasn't surprised that the company filed bankruptcy a few years later. Thank you, Chris R. From steve at pearwood.info Sun May 8 13:12:14 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 09 May 2016 03:12:14 +1000 Subject: Python PygLatin References: <31c06c3b-b622-4791-a607-d0ed2a741da1@googlegroups.com> Message-ID: <572f7370$0$1605$c3e8da3$5496439d@news.astraweb.com> On Sun, 8 May 2016 08:21 pm, Cai Gengyang wrote: > If one looks at the Forbes List, you will > see that there are 4 programmers amongst the top ten richest people in the > world (Bill Gates, Mark Zuckerberg, Larry Ellison and Jeff Bezos) , a very > large percentage. Science and Technology is in a sense the most > egalitarian field in the world, because it involves using your brains and > creativity. You don't need to have a father who is a director at Goldman > Sachs or a mother who is the admissions officer at Harvard to succeed in > this line. Bill Gates III's father was a prominent lawyer, his mother was on the board of directors for First Interstate BancSystem and United Way, and one of his grandfathers was a national bank president. Gates himself went to Harvard. Zuckerberg's paternal grandparents were successful middle class, described as being the first on the block to own a colour TV. (This was back in the days when colour TVs were an expensive toy that few could afford.) His parents were also very successful professionals: a dentist and a psychiatrist. And he too went to Harvard. Despite the jeans and tee-shirts Zuckerberg is known for wearing, he's firmly from the professional/upper class. Bezos comes from a family of land-holders from Texas. His grandfather was regional director of the U.S. Atomic Energy Commission, and was financially successful enough to retire at an early age. He didn't go to Harvard, but he did go to Princeton. Ellison is the son of an unwed mother who gave him up for adoption by her aunt and uncle, comfortably middle-class. That makes him the closest out of the group as a "regular guy". -- Steven From christopher_reimer at icloud.com Sun May 8 13:14:03 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Sun, 08 May 2016 10:14:03 -0700 Subject: Python is an Equal Opportunity Programming Language In-Reply-To: <87eg9dghnh.fsf@elektro.pacujo.net> References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> <1462646456.2677797.601019937.6FC9DC8E@webmail.messagingengine.com> <572ea7e4$0$1587$c3e8da3$5496439d@news.astraweb.com> <87eg9dghnh.fsf@elektro.pacujo.net> Message-ID: <572F73DB.4060904@icloud.com> On 5/7/2016 11:58 PM, Marko Rauhamaa wrote: > Chris Angelico : > >> So the question is: Do we care about country equality or individual >> equality? You can't have both. > That's why there's been a long-standing initiative to split California > into multiple states: > > > > Each state gets two senate seats, and California, being the most > populous state, suffers. The Six Californias is a proposal to divide up the 54 electoral votes that California has in presidential elections, which is a solidly blue state for the Democrats. Half the population lives in the Los Angeles, San Francisco and Sacramento regions. Put these three regions into separate states, the other three regions will become solidly red states for the Republicans and tilt the presidential elections in their favor. The proposal is a solution for the underlying problem that the California Republican Party has more in common with the endangered spotted owl than one-tenth of the US population. It's easier to redraw the lines than compete for votes. Thank you, Chris R. From tjreedy at udel.edu Sun May 8 13:17:50 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 8 May 2016 13:17:50 -0400 Subject: Help for a complex RE In-Reply-To: <6a3fe5cd-0ba9-4017-a763-76c896b8c843@googlegroups.com> References: <2aa55bd8-2ea4-41f7-b188-d45dff7d3bb7@googlegroups.com> <6a3fe5cd-0ba9-4017-a763-76c896b8c843@googlegroups.com> Message-ID: On 5/8/2016 12:32 PM, Sergio Spina wrote: > Il giorno domenica 8 maggio 2016 18:16:56 UTC+2, Peter Otten ha scritto: >> Sergio Spina wrote: >> >>> In the following ipython session: >>> >>>> Python 3.5.1+ (default, Feb 24 2016, 11:28:57) >>>> Type "copyright", "credits" or "license" for more information. >>>> >>>> IPython 2.3.0 -- An enhanced Interactive Python. >>>> >>>> In [1]: import re >>>> >>>> In [2]: patt = r""" # the match pattern is: >>>> ...: .+ # one or more characters >>>> ...: [ ] # followed by a space >>>> ...: (?=[@#D]:) # that is followed by one of the >>>> ...: # chars "@#D" and a colon ":" >>>> ...: """ >>>> >>>> In [3]: pattern = re.compile(patt, re.VERBOSE) >>>> >>>> In [4]: m = pattern.match("Jun at i Bun#i @:Janji") >>>> >>>> In [5]: m.group() >>>> Out[5]: 'Jun at i Bun#i ' >>>> >>>> In [6]: m = pattern.match("Jun at i Bun#i @:Janji D:Banji") >>>> >>>> In [7]: m.group() >>>> Out[7]: 'Jun at i Bun#i @:Janji ' >>>> >>>> In [8]: m = pattern.match("Jun at i Bun#i @:Janji D:Banji #:Junji") >>>> >>>> In [9]: m.group() >>>> Out[9]: 'Jun at i Bun#i @:Janji D:Banji ' >>> >>> Why the regex engine stops the search at last piece of string? >>> Why not at the first match of the group "@:"? >>> What can it be a regex pattern with the following result? >>> >>>> In [1]: m = pattern.match("Jun at i Bun#i @:Janji D:Banji #:Junji") >>>> >>>> In [2]: m.group() >>>> Out[2]: 'Jun at i Bun#i ' >> >> Compare: >> >>>>> re.compile("a+").match("aaaa").group() >> 'aaaa' >>>>> re.compile("a+?").match("aaaa").group() >> 'a' >> >> By default pattern matching is "greedy" -- the ".+" part of your regex >> matches as many characters as possible. Adding a ? like in ".+?" triggers >> non-greedy matching. > >> In [2]: patt = r""" # the match pattern is: >> ...: .+ # one or more characters Peter meant that you should replace '.+' with '.+?' to get the non-greedy match. >> ...: [ ] # followed by a space >> ...: (?=[@#D]:) # ONLY IF is followed by one of the <<< please note >> ...: # chars "@#D" and a colon ":" >> ...: """ > > From the python documentation > >> (?=...) >> Matches if ... matches next, but doesn't consume any of the string. >> This is called a lookahead assertion. For example, >> Isaac (?=Asimov) will match 'Isaac ' only if it's followed by 'Asimov'. > > I know about greedy and not-greedy, but the problem remains. Greedy '.+' matches the whole string. The matcher then back up to find a space -- initially the last space. It then, and only then, checks the lookahead assertion. If that failed, it would back up again. In your examples, it succeeds, and the matcher stops. -- Terry Jan Reedy From christopher_reimer at icloud.com Sun May 8 13:25:11 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Sun, 08 May 2016 10:25:11 -0700 Subject: Python is an Equal Opportunity Programming Language In-Reply-To: <96444428-5553-4471-929e-ec800e09d428@googlegroups.com> References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> <87lh3lj1ye.fsf@elektro.pacujo.net> <572f2c25$0$1589$c3e8da3$5496439d@news.astraweb.com> <96444428-5553-4471-929e-ec800e09d428@googlegroups.com> Message-ID: <572F7677.2080006@icloud.com> On 5/8/2016 8:09 AM, Rustom Mody wrote: > See: > https://www.washingtonpost.com/news/rampage/wp/2016/05/07/ivy-league-economist-interrogated-for-doing-math-on-american-airlines-flight/ > > Closing line: "In America today, the only thing more terrifying than foreigners is...math." > > > Wonder how close to terrorists pythonists are I wonder how many Americans are aware that they use Hindu-Arabic numerals in daily transactions? https://en.wikipedia.org/wiki/Hindu%E2%80%93Arabic_numeral_system Thank you, Chris R. From steve at pearwood.info Sun May 8 13:25:15 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 09 May 2016 03:25:15 +1000 Subject: pylint woes References: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> Message-ID: <572f767d$0$1619$c3e8da3$5496439d@news.astraweb.com> On Sun, 8 May 2016 02:10 pm, DFS wrote: > I mean I always use tab after : > > The program won't run otherwise. If I use spaces, 100% of the time it > throws: > > IndentationError: unindent does not match any outer indentation level Then you should be more careful about your spaces. If you indent by four spaces, you have to outdent by four -- not three, not five, but four. The best way to do this is to use an editor that will count the spaces for you. Any decent programmer's editor will allow you to set the TAB key to indent by X spaces, and the Shift-TAB key to dedent by the same amount. If you're counting spaces yourself, you're just making more work for yourself. Or use tabs -- that's acceptable as well. Just don't mix tabs and spaces in the same file. >>> +-------------------------+------------+ >>> |bad-whitespace |65 | mostly because I line up = >>> signs: >>> var1 = value >>> var10 = value >> >> Yuck. How much time do you waste aligning assignments whenever you add or >> delete or edit a variable? > > Lots. It takes hours to add or delete 3 whitespaces. Yes, you're right. It takes you five minutes to line everything up the first time. Then you change the name of a variable, and now you have to realign everything -- that's an extra minute gone. Then you add another line, and have to realign again, another couple of minutes. Over the lifespan of the program, you'll probably have spent multiple hours wasting time realigning blocks of assignments. >>> +-------------------------+------------+ >>> |trailing-whitespace |59 | heh! >>> +-------------------------+------------+ >>> |multiple-statements |23 | do this to save lines. >>> Will continue doing it. >> >> Why? Do you think that there's a world shortage of newline characters? Is >> the Enter key on your keyboard broken? > > I do it because I like it. > > if verbose: print var > > python doesn't complain. Hmmm. Well, that's not too bad. I thought you mean something like: addr = getaddress(key); addr[2] = addr.upper(); print addr which is just horrible. -- Steven From christopher_reimer at icloud.com Sun May 8 13:34:00 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Sun, 08 May 2016 10:34:00 -0700 Subject: Python is an Equal Opportunity Programming Language In-Reply-To: <572f68e4$0$1620$c3e8da3$5496439d@news.astraweb.com> References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <1462608136.590130.600748113.38B9CEA2@webmail.messagingengine.com> <572f68e4$0$1620$c3e8da3$5496439d@news.astraweb.com> Message-ID: <572F7888.1040505@icloud.com> On 5/8/2016 9:27 AM, Steven D'Aprano wrote: > On Sun, 8 May 2016 08:22 pm, beliavsky at aol.com wrote: > >> There are >> far more female than male teachers. I don't attribute it to anti-male >> suppression but to greater female interest in working with children. > Of course there is suppression of male teachers, particularly but not only > for very young children. A college instructor encouraged me to become a teacher, especially as boys from single mom families needed a daily role model. I looked into it and took some preparatory childhood classes. When the local university had a presentation for their teacher program, I went, saw how sausage got made, and ran like hell. Thank you, Chris R. From python.list at tim.thechases.com Sun May 8 13:48:12 2016 From: python.list at tim.thechases.com (Tim Chase) Date: Sun, 8 May 2016 12:48:12 -0500 Subject: Python is an Equal Opportunity Programming Language In-Reply-To: <572F7677.2080006@icloud.com> References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> <87lh3lj1ye.fsf@elektro.pacujo.net> <572f2c25$0$1589$c3e8da3$5496439d@news.astraweb.com> <96444428-5553-4471-929e-ec800e09d428@googlegroups.com> <572F7677.2080006@icloud.com> Message-ID: <20160508124812.45873aa4@bigbox.christie.dr> On ????-?-? ?:???, Christopher Reimer wrote: >> Closing line: "In America today, the only thing more terrifying >> than foreigners is...math." >> Wonder how close to terrorists pythonists are > > I wonder how many Americans are aware that they use Hindu-Arabic > numerals in daily transactions? There must be ? good reasons not to succumb to those "terrorist" numerals. I can name ? or ? other numeric systems that are just as useful. ;-) -tkc From alister.ware at ntlworld.com Sun May 8 13:53:16 2016 From: alister.ware at ntlworld.com (alister) Date: Sun, 08 May 2016 17:53:16 GMT Subject: Python PygLatin References: <31c06c3b-b622-4791-a607-d0ed2a741da1@googlegroups.com> <572f7370$0$1605$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Mon, 09 May 2016 03:12:14 +1000, Steven D'Aprano wrote: > On Sun, 8 May 2016 08:21 pm, Cai Gengyang wrote: > >> If one looks at the Forbes List, you will see that there are 4 >> programmers amongst the top ten richest people in the world (Bill >> Gates, Mark Zuckerberg, Larry Ellison and Jeff Bezos) , a very large >> percentage. Science and Technology is in a sense the most egalitarian >> field in the world, because it involves using your brains and >> creativity. You don't need to have a father who is a director at >> Goldman Sachs or a mother who is the admissions officer at Harvard to >> succeed in this line. > > Bill Gates III's father was a prominent lawyer, his mother was on the > board of directors for First Interstate BancSystem and United Way, and > one of his grandfathers was a national bank president. Gates himself > went to Harvard. > > Zuckerberg's paternal grandparents were successful middle class, > described as being the first on the block to own a colour TV. (This was > back in the days when colour TVs were an expensive toy that few could > afford.) His parents were also very successful professionals: a dentist > and a psychiatrist. And he too went to Harvard. Despite the jeans and > tee-shirts Zuckerberg is known for wearing, he's firmly from the > professional/upper class. > > Bezos comes from a family of land-holders from Texas. His grandfather > was regional director of the U.S. Atomic Energy Commission, and was > financially successful enough to retire at an early age. He didn't go to > Harvard, but he did go to Princeton. > > Ellison is the son of an unwed mother who gave him up for adoption by > her aunt and uncle, comfortably middle-class. That makes him the closest > out of the group as a "regular guy". And at least 2 of the above reached their position using business practices that could be described as less than 100% honorable & above board. -- Wait ... is this a FUN THING or the END of LIFE in Petticoat Junction?? From christopher_reimer at icloud.com Sun May 8 14:01:58 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Sun, 08 May 2016 11:01:58 -0700 Subject: Python PygLatin In-Reply-To: References: <31c06c3b-b622-4791-a607-d0ed2a741da1@googlegroups.com> <572f7370$0$1605$c3e8da3$5496439d@news.astraweb.com> Message-ID: <572F7F16.8040809@icloud.com> On 5/8/2016 10:53 AM, alister wrote: > On Mon, 09 May 2016 03:12:14 +1000, Steven D'Aprano wrote: > >> On Sun, 8 May 2016 08:21 pm, Cai Gengyang wrote: >> >>> If one looks at the Forbes List, you will see that there are 4 >>> programmers amongst the top ten richest people in the world (Bill >>> Gates, Mark Zuckerberg, Larry Ellison and Jeff Bezos) , a very large >>> percentage. Science and Technology is in a sense the most egalitarian >>> field in the world, because it involves using your brains and >>> creativity. You don't need to have a father who is a director at >>> Goldman Sachs or a mother who is the admissions officer at Harvard to >>> succeed in this line. >> Bill Gates III's father was a prominent lawyer, his mother was on the >> board of directors for First Interstate BancSystem and United Way, and >> one of his grandfathers was a national bank president. Gates himself >> went to Harvard. >> >> Zuckerberg's paternal grandparents were successful middle class, >> described as being the first on the block to own a colour TV. (This was >> back in the days when colour TVs were an expensive toy that few could >> afford.) His parents were also very successful professionals: a dentist >> and a psychiatrist. And he too went to Harvard. Despite the jeans and >> tee-shirts Zuckerberg is known for wearing, he's firmly from the >> professional/upper class. >> >> Bezos comes from a family of land-holders from Texas. His grandfather >> was regional director of the U.S. Atomic Energy Commission, and was >> financially successful enough to retire at an early age. He didn't go to >> Harvard, but he did go to Princeton. >> >> Ellison is the son of an unwed mother who gave him up for adoption by >> her aunt and uncle, comfortably middle-class. That makes him the closest >> out of the group as a "regular guy". > And at least 2 of the above reached their position using business > practices that could be described as less than 100% honorable & above > board. What do you expect from people who haven't graduated from Harvard? :P Thank you, Chris R. From __peter__ at web.de Sun May 8 14:19:31 2016 From: __peter__ at web.de (Peter Otten) Date: Sun, 08 May 2016 20:19:31 +0200 Subject: Help for a complex RE References: <2aa55bd8-2ea4-41f7-b188-d45dff7d3bb7@googlegroups.com> <6a3fe5cd-0ba9-4017-a763-76c896b8c843@googlegroups.com> Message-ID: Sergio Spina wrote: > I know about greedy and not-greedy, but the problem remains. This makes me wonder why you had to ask >>> Why the regex engine stops the search at last piece of string? >>> Why not at the first match of the group "@:"? To make it crystal clear this time: >>> import re >>> >>> patt = r""" # the match pattern is: ... .+? # one or more characters ... [ ] # followed by a space ... (?=[@#D]:) # that is followed by one of the ... # chars "@#D" and a colon ":" ... """ >>> pattern = re.compile(patt, re.VERBOSE) >>> m = pattern.match("Jun at i Bun#i @:Janji D:Banji #:Junji") >>> m.group() 'Jun at i Bun#i ' That's exactly what you asked for in >>> What can it be a regex pattern with the following result? >>> >>>> In [1]: m = pattern.match("Jun at i Bun#i @:Janji D:Banji #:Junji") >>>> >>>> In [2]: m.group() >>>> Out[2]: 'Jun at i Bun#i ' From orgnut at yahoo.com Sun May 8 16:45:02 2016 From: orgnut at yahoo.com (Larry Hudson) Date: Sun, 8 May 2016 13:45:02 -0700 Subject: pylint woes In-Reply-To: References: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> <20160508085006.020cef65@imp> Message-ID: <7aadnaamRdfTOLLKnZ2dnUU7-eXNnZ2d@giganews.com> On 05/08/2016 06:01 AM, Chris Angelico wrote: [snip...] > ... I like to recommend a > little thing called "IIDPIO debugging" - If In Doubt, Print It Out. > That means: If you have no idea what a piece of code is doing, slap in > a print() call somewhere. It'll tell you that (a) the code is actually > being executed, and (b) whatever info you put between the parens > (ideally, some key variable or parameter)... My personal variation of IIPPID debugging is to use input() instead of print(). For example: input('x = {}, y = {} --> '.format(x, y)) Then the program stops at this point so you can examine the values. will continue the program or ^C will abort (if you see what the problem is now). Of course this can't be used in all situations, but it's handy where it can. Note that my personal preference is to stick that "-->" as a prompt at the end, but obviously this (or a similar marker) is optional. From dan at tombstonezero.net Sun May 8 16:49:51 2016 From: dan at tombstonezero.net (Dan Sommers) Date: Sun, 8 May 2016 20:49:51 -0000 (UTC) Subject: pylint woes References: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> <20160508085006.020cef65@imp> Message-ID: On Sun, 08 May 2016 23:01:55 +1000, Chris Angelico wrote: > ... I like to recommend a little thing called "IIDPIO debugging" - If > In Doubt, Print It Out. That means: If you have no idea what a piece > of code is doing, slap in a print() call somewhere. It'll tell you > that (a) the code is actually being executed, and (b) whatever info > you put between the parens (ideally, some key variable or > parameter). Part A is often the important bit :) ... Having spent a long time developing embedded systems, I wholeheartedly agree. In spirit. Isn't that what the logging module is for? Fine grained control, as centralized or distributed as is warranted, over program output? > ... The trouble with a verbose flag controlling all print() calls is > that IIDPIO debugging suddenly doesn't work; plus, it's easy to copy > and paste code to some other module and not notice that you don't have > a verbosity check at the top, and then wonder why disabling verbose > doesn't fully work. Both problems are solved by having a dedicated > spam function, which will simply error out if you didn't set it up > properly. Hey! That sounds just like the logging module.... ;-) Dan From nospam at dfs.com Sun May 8 17:04:02 2016 From: nospam at dfs.com (DFS) Date: Sun, 8 May 2016 17:04:02 -0400 Subject: pylint woes In-Reply-To: <572f607f$0$1588$c3e8da3$5496439d@news.astraweb.com> References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> <572f607f$0$1588$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5/8/2016 11:51 AM, Steven D'Aprano wrote: > On Mon, 9 May 2016 12:25 am, DFS wrote: > >>>> for j in range(len(nms)): >>>> cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" >>>> vals = nms[j],street[j],city[j],state[j],zipcd[j] > > Why are you assigning cSQL to the same string over and over again? I like it in cloxe proximity to the vals statement. > Sure, assignments are cheap, but they're not infinitely cheap. They still > have a cost. Instead of paying that cost once, you pay it over and over > again, which adds up. Adds up to what? > Worse, it is misleading. I had to read that code snippet three or four times > before I realised that cSQL was exactly the same each time. You had to read 5 words three or four times? Seriously? >> I tried: >> >> for nm,street,city,state,zipcd in zip(nms,street,city,state,zipcd): >> >> but felt it was too long and wordy. > > It's long and wordy because you're doing something long and wordy. It is > *inherently* long and wordy to process five things, whether you write it > as: > > for i in range(len(names)): > name = names[i] > street = streets[i] > city = cities[i] > state = states[i] > zipcode = zipcodes[i] > process(...) > > or as: > > for name, street, city, state, zipcode in zip( > names, streets, cities, states, zipcodes > ): > process(...) I like mine best of all: ziplists = zip(names,streets,cities,states,zipcodes) for name,street,city,state,zipcode in ziplists: >> I like the first one better. python is awesome, but too many options >> for doing the same thing also makes it difficult. For me, anyway. > > > That's the difference between a master and an apprentice. The apprentice > likes to follow fixed steps the same way each time. The master craftsman > knows her tools backwards, and can choose the right tool for the job, and > when the choice of tool really doesn't matter and you can use whatever > happens to be the closest to hand. "her tools"... you're a woman? From nospam at dfs.com Sun May 8 17:05:24 2016 From: nospam at dfs.com (DFS) Date: Sun, 8 May 2016 17:05:24 -0400 Subject: pylint woes In-Reply-To: References: Message-ID: On 5/7/2016 2:43 PM, Peter Pearson wrote: > On Sat, 7 May 2016 12:51:00 -0400, DFS wrote: >> This more-anal-than-me program generated almost 2 warnings for every >> line of code in my program. w t hey? > > Thank you for putting a sample of pylint output in front of my eyes; > you inspired me to install pylint and try it out. If it teaches me even > half as much as it's teaching you, I'll consider it a great blessing. Cool. I don't agree with some of them, but there's no doubt adhering to them will result in more well-formed code. From nospam at dfs.com Sun May 8 17:06:34 2016 From: nospam at dfs.com (DFS) Date: Sun, 8 May 2016 17:06:34 -0400 Subject: pylint woes In-Reply-To: References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> Message-ID: On 5/8/2016 11:15 AM, Chris Angelico wrote: > On Mon, May 9, 2016 at 1:06 AM, DFS wrote: >> On 5/8/2016 10:36 AM, Chris Angelico wrote: >>> >>> On Mon, May 9, 2016 at 12:25 AM, DFS wrote: >>>> >>>> for category,name,street,city,state,zipcode in ziplists: >>>> try: db.execute(cSQL, vals) >>>> except (pyodbc.Error) as programError: >>>> if str(programError).find("UNIQUE constraint failed") > 0: >>>> dupeRow = True >>>> dupes +=1 >>>> print " * duplicate address found: "+name+", "+street >>>> else: >>>> pyodbcErr = True >>>> print "ODBC error: %s " % programError >>>> conn.commit() >>>> -------------------------------------------------------------------- >>>> >>> >>> ... and then you just commit???!? >>> >>> ChrisA >> >> >> >> That's what commit() does. > > Yes. Even if you got an error part way through, you just blithely commit. What?! > > And yes, I am flat-out boggling at this. > > ChrisA I'm boggling that you're boggling. From nospam at dfs.com Sun May 8 17:16:56 2016 From: nospam at dfs.com (DFS) Date: Sun, 8 May 2016 17:16:56 -0400 Subject: pylint woes In-Reply-To: <572f767d$0$1619$c3e8da3$5496439d@news.astraweb.com> References: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> <572f767d$0$1619$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5/8/2016 1:25 PM, Steven D'Aprano wrote: > On Sun, 8 May 2016 02:10 pm, DFS wrote: >>>> +-------------------------+------------+ >>>> |bad-whitespace |65 | mostly because I line up = >>>> signs: >>>> var1 = value >>>> var10 = value >>> >>> Yuck. How much time do you waste aligning assignments whenever you add or >>> delete or edit a variable? >> >> Lots. It takes hours to add or delete 3 whitespaces. > > Yes, you're right. It takes you five minutes to line everything up the first > time. Then you change the name of a variable, and now you have to realign > everything -- that's an extra minute gone. Then you add another line, and > have to realign again, another couple of minutes. Over the lifespan of the > program, you'll probably have spent multiple hours wasting time realigning > blocks of assignments. Do you actually believe what you just wrote? If yes, you should quit programming. If not, why did you say it? >>>> +-------------------------+------------+ >>>> |trailing-whitespace |59 | heh! >>>> +-------------------------+------------+ >>>> |multiple-statements |23 | do this to save lines. >>>> Will continue doing it. >>> >>> Why? Do you think that there's a world shortage of newline characters? Is >>> the Enter key on your keyboard broken? >> >> I do it because I like it. >> >> if verbose: print var >> >> python doesn't complain. > > Hmmm. Well, that's not too bad. I thought you mean something like: > > addr = getaddress(key); addr[2] = addr.upper(); print addr > > which is just horrible. I was surprised to see the PEP8 guide approve of: "Yes: if x == 4: print x, y; x, y = y, x" https://www.python.org/dev/peps/pep-0008/#pet-peeves From nospam at dfs.com Sun May 8 17:24:09 2016 From: nospam at dfs.com (DFS) Date: Sun, 8 May 2016 17:24:09 -0400 Subject: pylint woes In-Reply-To: <572f24bc$0$1618$c3e8da3$5496439d@news.astraweb.com> References: <572f24bc$0$1618$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5/8/2016 7:36 AM, Steven D'Aprano wrote: > On Sun, 8 May 2016 11:16 am, DFS wrote: > >> address data is scraped from a website: >> >> names = tree.xpath() >> addr = tree.xpath() > > Why are you scraping the data twice? Because it exists in 2 different sections of the document. names = tree.xpath('//span[@class="header_text3"]/text()') addresses = tree.xpath('//span[@class="text3"]/text()') I thought you were a "master who knew her tools", and I was the apprentice? So why did "the master" think xpath() was magic? > names = addr = tree.xpath() > > or if you prefer the old-fashioned: > > names = tree.xpath() > addr = names > > but that raises the question, how can you describe the same set of data as > both "names" and "addr[esses]" and have them both be accurate? > > >> I want to store the data atomically, > > I'm not really sure what you mean by "atomically" here. I know what *I* mean > by "atomically", which is to describe an operation which either succeeds > entirely or fails. That's atomicity. > But I don't know what you mean by it. http://www.databasedesign-resource.com/atomic-database-values.html >> so I parse street, city, state, and >> zip into their own lists. > > None of which is atomic. All of which are atomic. >> "1250 Peachtree Rd, Atlanta, GA 30303 >> >> street = [s.split(',')[0] for s in addr] >> city = [c.split(',')[1].strip() for c in addr] >> state = [s[-8:][:2] for s in addr] >> zipcd = [z[-5:] for z in addr] > > At this point, instead of iterating over the same list four times, doing the > same thing over and over again, you should do things the old-fashioned way: > > streets, cities, states, zipcodes = [], [], [], [] > for word in addr: > items = word.split(',') > streets.append(items[0]) > cities.append(items[1].strip()) > states.append(word[-8:-2]) > zipcodes.append(word[-5:]) That's a good one. Chris Angelico mentioned something like that, too, and I already put it place. > Oh, and use better names. "street" is a single street, not a list of > streets, note plural. I'll use whatever names I like. From me+python at ixokai.io Sun May 8 17:38:24 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sun, 08 May 2016 14:38:24 -0700 Subject: pylint woes In-Reply-To: References: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> <572f767d$0$1619$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1462743504.956579.601680113.41764235@webmail.messagingengine.com> On Sun, May 8, 2016, at 02:16 PM, DFS wrote: > I was surprised to see the PEP8 guide approve of: > > "Yes: if x == 4: print x, y; x, y = y, x" > > https://www.python.org/dev/peps/pep-0008/#pet-peeves That is not approving of that line of code as something to mimic, its speaking *only* about *whitespace*. ALL its saying is, "don't put spaces before commas, colons or semicolons". You can infer nothing else about it. -- Stephen Hansen m e @ i x o k a i . i o From joel.goldstick at gmail.com Sun May 8 17:39:44 2016 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Sun, 8 May 2016 17:39:44 -0400 Subject: pylint woes In-Reply-To: References: <572f24bc$0$1618$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sun, May 8, 2016 at 5:24 PM, DFS wrote: > On 5/8/2016 7:36 AM, Steven D'Aprano wrote: >> >> On Sun, 8 May 2016 11:16 am, DFS wrote: >> >>> address data is scraped from a website: >>> >>> names = tree.xpath() >>> addr = tree.xpath() >> >> >> Why are you scraping the data twice? > > > > Because it exists in 2 different sections of the document. > > names = tree.xpath('//span[@class="header_text3"]/text()') > addresses = tree.xpath('//span[@class="text3"]/text()') > > > I thought you were a "master who knew her tools", and I was the apprentice? > > So why did "the master" think xpath() was magic? > > > > > > >> names = addr = tree.xpath() >> >> or if you prefer the old-fashioned: >> >> names = tree.xpath() >> addr = names >> >> but that raises the question, how can you describe the same set of data as >> both "names" and "addr[esses]" and have them both be accurate? >> >> >>> I want to store the data atomically, >> >> >> I'm not really sure what you mean by "atomically" here. I know what *I* >> mean >> by "atomically", which is to describe an operation which either succeeds >> entirely or fails. > > > That's atomicity. > > > >> But I don't know what you mean by it. > > http://www.databasedesign-resource.com/atomic-database-values.html > > > >>> so I parse street, city, state, and >>> zip into their own lists. >> >> >> None of which is atomic. > > > All of which are atomic. > > > >>> "1250 Peachtree Rd, Atlanta, GA 30303 >>> >>> street = [s.split(',')[0] for s in addr] >>> city = [c.split(',')[1].strip() for c in addr] >>> state = [s[-8:][:2] for s in addr] >>> zipcd = [z[-5:] for z in addr] >> >> >> At this point, instead of iterating over the same list four times, doing >> the >> same thing over and over again, you should do things the old-fashioned >> way: >> >> streets, cities, states, zipcodes = [], [], [], [] >> for word in addr: >> items = word.split(',') >> streets.append(items[0]) >> cities.append(items[1].strip()) >> states.append(word[-8:-2]) >> zipcodes.append(word[-5:]) > > > > > That's a good one. > > Chris Angelico mentioned something like that, too, and I already put it > place. > > > >> Oh, and use better names. "street" is a single street, not a list of >> streets, note plural. > > > > I'll use whatever names I like. > > > > > > -- > https://mail.python.org/mailman/listinfo/python-list Starting to look like trolling. Lots of good advice here. If you ask, and don't like the advice, don't use it. -- Joel Goldstick http://joelgoldstick.com/blog http://cc-baseballstats.info/stats/birthdays From nospam at dfs.com Sun May 8 17:46:28 2016 From: nospam at dfs.com (DFS) Date: Sun, 8 May 2016 17:46:28 -0400 Subject: pylint woes In-Reply-To: References: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> <572f767d$0$1619$c3e8da3$5496439d@news.astraweb.com> <1462743504.956579.601680113.41764235@webmail.messagingengine.com> Message-ID: On 5/8/2016 5:38 PM, Stephen Hansen wrote: > On Sun, May 8, 2016, at 02:16 PM, DFS wrote: >> I was surprised to see the PEP8 guide approve of: >> >> "Yes: if x == 4: print x, y; x, y = y, x" >> >> https://www.python.org/dev/peps/pep-0008/#pet-peeves > > That is not approving of that line of code as something to mimic, its > speaking *only* about *whitespace*. > > ALL its saying is, "don't put spaces before commas, colons or > semicolons". You can infer nothing else about it. I can infer that it's 100% approved of, since he used it as an example. From me+python at ixokai.io Sun May 8 18:05:23 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sun, 08 May 2016 15:05:23 -0700 Subject: pylint woes In-Reply-To: References: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> <572f767d$0$1619$c3e8da3$5496439d@news.astraweb.com> <1462743504.956579.601680113.41764235@webmail.messagingengine.com> Message-ID: <1462745123.961210.601695129.615D10A0@webmail.messagingengine.com> On Sun, May 8, 2016, at 02:46 PM, DFS wrote: > On 5/8/2016 5:38 PM, Stephen Hansen wrote: > > On Sun, May 8, 2016, at 02:16 PM, DFS wrote: > >> I was surprised to see the PEP8 guide approve of: > >> > >> "Yes: if x == 4: print x, y; x, y = y, x" > >> > >> https://www.python.org/dev/peps/pep-0008/#pet-peeves > > > > That is not approving of that line of code as something to mimic, its > > speaking *only* about *whitespace*. > > > > ALL its saying is, "don't put spaces before commas, colons or > > semicolons". You can infer nothing else about it. > > I can infer that it's 100% approved of, since he used it as an example. Not if you don't want to be a fool. And at this point I'm signing out from helping you. -- Stephen Hansen m e @ i x o k a i . i o From rosuav at gmail.com Sun May 8 18:07:25 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 9 May 2016 08:07:25 +1000 Subject: pylint woes In-Reply-To: <7aadnaamRdfTOLLKnZ2dnUU7-eXNnZ2d@giganews.com> References: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> <20160508085006.020cef65@imp> <7aadnaamRdfTOLLKnZ2dnUU7-eXNnZ2d@giganews.com> Message-ID: On Mon, May 9, 2016 at 6:45 AM, Larry Hudson via Python-list wrote: > On 05/08/2016 06:01 AM, Chris Angelico wrote: > [snip...] >> >> ... I like to recommend a >> little thing called "IIDPIO debugging" - If In Doubt, Print It Out. >> That means: If you have no idea what a piece of code is doing, slap in >> a print() call somewhere. It'll tell you that (a) the code is actually >> being executed, and (b) whatever info you put between the parens >> (ideally, some key variable or parameter)... > > > My personal variation of IIPPID debugging is to use input() instead of > print(). For example: > > input('x = {}, y = {} --> '.format(x, y)) > > Then the program stops at this point so you can examine the values. > will continue the program or ^C will abort (if you see what the problem is > now). Of course this can't be used in all situations, but it's handy where > it can. > > Note that my personal preference is to stick that "-->" as a prompt at the > end, but obviously this (or a similar marker) is optional. Neat technique. Not something to use *every* time (and not always sensible - eg you don't normally want to stall out a GUI thread), but definitely worth keeping in the arsenal. ChrisA From rosuav at gmail.com Sun May 8 18:10:44 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 9 May 2016 08:10:44 +1000 Subject: pylint woes In-Reply-To: References: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> <20160508085006.020cef65@imp> Message-ID: On Mon, May 9, 2016 at 6:49 AM, Dan Sommers wrote: > On Sun, 08 May 2016 23:01:55 +1000, Chris Angelico wrote: > >> ... I like to recommend a little thing called "IIDPIO debugging" - If >> In Doubt, Print It Out. That means: If you have no idea what a piece >> of code is doing, slap in a print() call somewhere. It'll tell you >> that (a) the code is actually being executed, and (b) whatever info >> you put between the parens (ideally, some key variable or >> parameter). Part A is often the important bit :) ... > > Having spent a long time developing embedded systems, I wholeheartedly > agree. In spirit. Isn't that what the logging module is for? Fine > grained control, as centralized or distributed as is warranted, over > program output? > >> ... The trouble with a verbose flag controlling all print() calls is >> that IIDPIO debugging suddenly doesn't work; plus, it's easy to copy >> and paste code to some other module and not notice that you don't have >> a verbosity check at the top, and then wonder why disabling verbose >> doesn't fully work. Both problems are solved by having a dedicated >> spam function, which will simply error out if you didn't set it up >> properly. > > Hey! That sounds just like the logging module.... ;-) Absolutely. I say "print" in IIDPIO because it's a word that people understand across languages, across frameworks, etc, etc, but when you start doing more of it, the logging module is definitely superior - if you need just one reason to use it, it would be to *leave those prints in place* so the next person doesn't need to reach for IIDPIO at all. (Also, I teach print() because it's one less module to explain. But experienced programmers should get some familiarity with it.) ChrisA From rosuav at gmail.com Sun May 8 18:17:36 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 9 May 2016 08:17:36 +1000 Subject: Python is an Equal Opportunity Programming Language In-Reply-To: <20160508124812.45873aa4@bigbox.christie.dr> References: <572d620a$0$1617$c3e8da3$5496439d@news.astraweb.com> <87zis2i751.fsf@elektro.pacujo.net> <572e06e3$0$1605$c3e8da3$5496439d@news.astraweb.com> <87lh3lj1ye.fsf@elektro.pacujo.net> <572f2c25$0$1589$c3e8da3$5496439d@news.astraweb.com> <96444428-5553-4471-929e-ec800e09d428@googlegroups.com> <572F7677.2080006@icloud.com> <20160508124812.45873aa4@bigbox.christie.dr> Message-ID: On Mon, May 9, 2016 at 3:48 AM, Tim Chase wrote: > On ????-?-? ?:???, Christopher Reimer wrote: >>> Closing line: "In America today, the only thing more terrifying >>> than foreigners is...math." >>> Wonder how close to terrorists pythonists are >> >> I wonder how many Americans are aware that they use Hindu-Arabic >> numerals in daily transactions? > > There must be ? good reasons not to succumb to those "terrorist" > numerals. I can name ? or ? other numeric systems that are just as > useful. ;-) Hands up those of you who are fluent in Latin. *looks around* Hmm, a couple! More than I thought. Hands up those of you who've written Python scripts. Yep, good, good, now I know you're all awake, at least. Now, hands up those who've used Latin script. Hmm, curious. ChrisA From nospam at dfs.com Sun May 8 18:24:39 2016 From: nospam at dfs.com (DFS) Date: Sun, 8 May 2016 18:24:39 -0400 Subject: pylint woes In-Reply-To: References: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> <572f767d$0$1619$c3e8da3$5496439d@news.astraweb.com> <1462743504.956579.601680113.41764235@webmail.messagingengine.com> <1462745123.961210.601695129.615D10A0@webmail.messagingengine.com> Message-ID: On 5/8/2016 6:05 PM, Stephen Hansen wrote: > On Sun, May 8, 2016, at 02:46 PM, DFS wrote: >> On 5/8/2016 5:38 PM, Stephen Hansen wrote: >>> On Sun, May 8, 2016, at 02:16 PM, DFS wrote: >>>> I was surprised to see the PEP8 guide approve of: >>>> >>>> "Yes: if x == 4: print x, y; x, y = y, x" >>>> >>>> https://www.python.org/dev/peps/pep-0008/#pet-peeves >>> >>> That is not approving of that line of code as something to mimic, its >>> speaking *only* about *whitespace*. >>> >>> ALL its saying is, "don't put spaces before commas, colons or >>> semicolons". You can infer nothing else about it. >> >> I can infer that it's 100% approved of, since he used it as an example. > > Not if you don't want to be a fool. Why would you label the author of that style guide - Guido van Rossum - a fool? > And at this point I'm signing out from helping you. A fool and a fool are soon parted. From nospam at dfs.com Sun May 8 19:10:24 2016 From: nospam at dfs.com (DFS) Date: Sun, 8 May 2016 19:10:24 -0400 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? Message-ID: sSQL = "line 1\n" sSQL += "line 2\n" sSQL += "line 3" From python.list at tim.thechases.com Sun May 8 19:37:38 2016 From: python.list at tim.thechases.com (Tim Chase) Date: Sun, 8 May 2016 18:37:38 -0500 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: References: Message-ID: <20160508183738.2f551043@bigbox.christie.dr> While I'm not Steven... On 2016-05-08 19:10, DFS wrote: > sSQL = "line 1\n" > sSQL += "line 2\n" > sSQL += "line 3" If you're only doing it once, it's adequate. If you're doing it within a loop for thing in some_iter(): s = "line1\n" s += "line2\n" s += "line3" use(s, thing) it's suboptimal. Solutions include hoisting it out of the loop: s = "line1\n" s += "line2\n" s += "line3" for thing in some_iter(): use(s, thing) and just specifying it with a multi-line string (still better off hoisted out of the loop): s = """line1 line2 line3""" for thing in some_iter(): use(s, thing) If you're accruing in a loop s = "" for thing in some_iter(): s += mung(thing) then that's a bad code-smell (you get quadratic behavior as the strings are constantly resized), usually better replaced with s = "".join(mung(thing) for thing in some_iter()) or build them up as a list, then join the results: lst = [] for thing in some_iter(): if should_use(thing): lst.append(thing) s = "\n".join(lst) -tkc From joel.goldstick at gmail.com Sun May 8 20:02:19 2016 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Sun, 8 May 2016 20:02:19 -0400 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: References: Message-ID: On Sun, May 8, 2016 at 7:10 PM, DFS wrote: > sSQL = "line 1\n" > sSQL += "line 2\n" > sSQL += "line 3" > -- > https://mail.python.org/mailman/listinfo/python-list What is your point DFS? You found pylint, and you don't like what it tells you. Its a tool, and it can tell you some things. Your code seems to be a mix of tabs and spaces, and you don't like standard python practices. That's fine, coding is (at least alone) an individual pursuit. You can code anyway you like. But, learning from the work of others, can be awfully useful. Most people I know use spaces, they set their text editor up to make tabs turn into 4 spaces. Problem solved. As far as wanting to put several statements on a line, you can do that if you like, but others have concluded that the code is easier to read if you don't. If that isn't your view, do what you like. It depends upon what problems you want to fight. If you adhere to standards, your code is more easily reviewed and easier to get help from others, if that is what you ask. -- Joel Goldstick http://joelgoldstick.com/blog http://cc-baseballstats.info/stats/birthdays From PointedEars at web.de Sun May 8 20:44:11 2016 From: PointedEars at web.de (Thomas 'PointedEars' Lahn) Date: Mon, 09 May 2016 02:44:11 +0200 Subject: String concatenation (was: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement?) References: Message-ID: <3414323.Mt0gs2Yp5W@PointedEars.de> DFS wrote: > sSQL = "line 1\n" > sSQL += "line 2\n" > sSQL += "line 3" What is wrong with it in Python is that it is unnecessarily inefficient. Python has implicit string concatenation if all operands are string literals: #----------------------------------------------------------------------- sSQL = ("line 1\n" "line 2\n" "line 3") #----------------------------------------------------------------------- Alternatively, you can join a list of strings: #----------------------------------------------------------------------- sSQL = "\n".join([ "line 1", "line 2", "line 3", ]) #----------------------------------------------------------------------- Python also has a way that avoids using several string literals: #----------------------------------------------------------------------- sSQL = """line 1 line 2 line 3""" #----------------------------------------------------------------------- or (to have the text aligned) #----------------------------------------------------------------------- sSQL = """\ line 1 line 2 line 3""" #----------------------------------------------------------------------- With the ?%? string operator (deprecated), str.format(), and str.Template, you can use other values in string values even without concatenation. Next time, RTFM first: p. Finally, with SQL you should prefer Prepared Statements and Stored Procedures, not bare strings, to avoid SQL injection: And if you want to communicate only with a specific person, I strongly suggest you send *them* an *e-mail* instead of to the newsgroup/mailing list. Also, it would be a good idea if you posted under your real name. Internet is the thing with cables; Usenet is the thing with people. I for one tend to avoid communicating with few-letter entities; exceptions to that would probably include only E.T., M.J., ALF, and K.I.T.T. -- PointedEars Twitter: @PointedEars2 Please do not cc me. / Bitte keine Kopien per E-Mail. From greg.ewing at canterbury.ac.nz Sun May 8 21:17:44 2016 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Mon, 09 May 2016 13:17:44 +1200 Subject: pylint woes In-Reply-To: References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> <1462720536.896455.601493505.776FAC7D@webmail.messagingengine.com> Message-ID: Stephen Hansen wrote: > The point is, you don't usually commit after an error happens. You > rollback. He might want to commit the ones that *did* go in. That's not necessarily wrong. It all depends on the surrounding requirements and workflow. -- Greg From orgnut at yahoo.com Sun May 8 21:28:46 2016 From: orgnut at yahoo.com (Larry Hudson) Date: Sun, 8 May 2016 18:28:46 -0700 Subject: pylint woes In-Reply-To: References: <572eb1c3$0$1616$c3e8da3$5496439d@news.astraweb.com> <20160508085006.020cef65@imp> <7aadnaamRdfTOLLKnZ2dnUU7-eXNnZ2d@giganews.com> Message-ID: On 05/08/2016 03:07 PM, Chris Angelico wrote: > On Mon, May 9, 2016 at 6:45 AM, Larry Hudson via Python-list > wrote: >> On 05/08/2016 06:01 AM, Chris Angelico wrote: >> [snip...] >>> >>> ... I like to recommend a >>> little thing called "IIDPIO debugging" - If In Doubt, Print It Out. >>> That means: If you have no idea what a piece of code is doing, slap in >>> a print() call somewhere. It'll tell you that (a) the code is actually >>> being executed, and (b) whatever info you put between the parens >>> (ideally, some key variable or parameter)... >> >> >> My personal variation of IIPPID debugging is to use input() instead of >> print(). For example: >> >> input('x = {}, y = {} --> '.format(x, y)) >> >> Then the program stops at this point so you can examine the values. >> will continue the program or ^C will abort (if you see what the problem is >> now). Of course this can't be used in all situations, but it's handy where >> it can. >> >> Note that my personal preference is to stick that "-->" as a prompt at the >> end, but obviously this (or a similar marker) is optional. > > Neat technique. Not something to use *every* time (and not always > sensible - eg you don't normally want to stall out a GUI thread), but > definitely worth keeping in the arsenal. > > ChrisA > Agreed. As I said in my post, it is certainly not a universally valid approach, but I do find it useful in many cases. -=- Larry -=- From rosuav at gmail.com Sun May 8 22:16:40 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 9 May 2016 12:16:40 +1000 Subject: String concatenation (was: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement?) In-Reply-To: <3414323.Mt0gs2Yp5W@PointedEars.de> References: <3414323.Mt0gs2Yp5W@PointedEars.de> Message-ID: On Mon, May 9, 2016 at 10:44 AM, Thomas 'PointedEars' Lahn wrote: > With the ?%? string operator (deprecated), str.format(), and str.Template, > you can use other values in string values even without concatenation. Not deprecated. Don't spread FUD. > Finally, with SQL you should prefer Prepared Statements and Stored > Procedures, not bare strings, to avoid SQL injection: > > He is safe. He's using parameterized queries. > Also, it would be a good idea if you posted under your real name. Internet > is the thing with cables; Usenet is the thing with people. I for one tend > to avoid communicating with few-letter entities; exceptions to that would > probably include only E.T., M.J., ALF, and K.I.T.T. I'm not using Usenet, Mr PointedEars. ChrisA From rosuav at gmail.com Sun May 8 22:18:19 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 9 May 2016 12:18:19 +1000 Subject: pylint woes In-Reply-To: References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> <1462720536.896455.601493505.776FAC7D@webmail.messagingengine.com> Message-ID: On Mon, May 9, 2016 at 11:17 AM, Gregory Ewing wrote: > Stephen Hansen wrote: >> >> The point is, you don't usually commit after an error happens. You >> rollback. > > > He might want to commit the ones that *did* go in. > That's not necessarily wrong. It all depends on the > surrounding requirements and workflow. If that's the case, they should probably be being committed part way. Generally, once the transaction state is in error, further operations can't be done. (At least, that's how it is with *good* database backends. I don't know what MySQL does.) ChrisA From nospam at dfs.com Sun May 8 22:58:45 2016 From: nospam at dfs.com (DFS) Date: Sun, 8 May 2016 22:58:45 -0400 Subject: pylint woes In-Reply-To: References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> <1462720536.896455.601493505.776FAC7D@webmail.messagingengine.com> Message-ID: On 5/8/2016 9:17 PM, Gregory Ewing wrote: > Stephen Hansen wrote: >> The point is, you don't usually commit after an error happens. You >> rollback. > > He might want to commit the ones that *did* go in. > That's not necessarily wrong. It all depends on the > surrounding requirements and workflow. Bingo. From steve at pearwood.info Sun May 8 23:09:03 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 09 May 2016 13:09:03 +1000 Subject: pylint woes References: <1462673665.770286.601202553.071593AE@webmail.messagingengine.com> <572f607f$0$1588$c3e8da3$5496439d@news.astraweb.com> Message-ID: <572fff51$0$1586$c3e8da3$5496439d@news.astraweb.com> On Mon, 9 May 2016 07:04 am, DFS wrote: > On 5/8/2016 11:51 AM, Steven D'Aprano wrote: >> On Mon, 9 May 2016 12:25 am, DFS wrote: >> >>>>> for j in range(len(nms)): >>>>> cSQL = "INSERT INTO ADDRESSES VALUES (?,?,?,?,?)" >>>>> vals = nms[j],street[j],city[j],state[j],zipcd[j] >> >> Why are you assigning cSQL to the same string over and over again? > > I like it in cloxe proximity to the vals statement. The line immediately above the loop is in close proximity. Is that not close enough? >> Sure, assignments are cheap, but they're not infinitely cheap. They still >> have a cost. Instead of paying that cost once, you pay it over and over >> again, which adds up. > > Adds up to what? Potentially a significant waste of time. You know what they say about financial waste: "a million here, a million there, and soon we're talking about real money". The first point is that this is a micro-pessimisation: even though it doesn't cost much individually, its still a needless expense that your program keeps paying. Its like friction on your code. Python is neither the fastest nor the slowest language available. It is often "fast enough", but it is also an easy language to write slow code in. If you were programming in C, the compiler would almost surely see that the assignment was to a constant, and automatically and silently hoist it outside the loop. That's one reason why C is so fast: it aggressively optimizes your code. (Too aggressively, in my opinion, but that's another story.) But the Python compiler isn't that sophisticated. It's up to us, the programmers, to be mindful of the friction we add to our code, because if we aren't mindful of it, we can easily end up with needlessly slow code. But as I said, that's not the major problem with doing the assignment in the loop. It's more about readability and your reader's expectations than the extra time it costs. >> Worse, it is misleading. I had to read that code snippet three or four >> times before I realised that cSQL was exactly the same each time. > > You had to read 5 words three or four times? Seriously? Yes, seriously, because when most people read code they skim it, speed reading, looking for salient points of interest. They don't point their finger under each word and read it aloud syllable by syllable like a pre-schooler with reading difficulties. (At least I don't, I can't speak for others.) So the first couple of times I glanced at it, it just looked like any other assignment without the details registering. Then the next couple of times I thought that it must be *me* making the mistake, I must be reading it wrong. Maybe there's something I missed? I read code with the default assumption that it is more or less sensible. Over the various posts and replies to posts, I had probably glanced at that line a dozen times, and then read it more carefully three or four times, before I was sure I had read what I thought I had read. [...] > I like mine best of all: > > ziplists = zip(names,streets,cities,states,zipcodes) > for name,street,city,state,zipcode in ziplists: Using a temporary, single-use variable is okay, but it's often unnecessary. But be aware that there is a particular risk with loops. You may be tempted to think you can re-use ziplists to iterate over it twice: data = zip(names, streets, cities, states, zipcodes) for a, b, c in data: do_stuff() ... # later for x, y, z in data: do_something_else() and that's perfectly fine, *but* there is a risk that if the for loop data being iterated over is an iterator, it will have been exhausted by the first loop and the second loop won't run at all. In Python 2, zip() returns a list, but in Python 3, it returns an iterator. So beware of using such temp variables unless you know what you're doing. >>> I like the first one better. python is awesome, but too many options >>> for doing the same thing also makes it difficult. For me, anyway. >> >> >> That's the difference between a master and an apprentice. The apprentice >> likes to follow fixed steps the same way each time. The master craftsman >> knows her tools backwards, and can choose the right tool for the job, and >> when the choice of tool really doesn't matter and you can use whatever >> happens to be the closest to hand. > > "her tools"... you're a woman? What makes you think I was talking about myself? I was talking about people in general -- when we are beginners, its only natural that (like most beginners) we're more comfortable with a limited amount of choice. Its hard to remember what option to use when there's only one, let alone when there's ten. But as we progress to mastery of the language, we'll come to understand the subtle differences between options, when they matter, and when they don't. If I had said "his tools", would you have thought I was talking specifically about a man? Does it matter if I'm a woman? Would that make my advice better or worse? -- Steven From mr.eightnoteight at gmail.com Sun May 8 23:31:16 2016 From: mr.eightnoteight at gmail.com (srinivas devaki) Date: Mon, 9 May 2016 09:01:16 +0530 Subject: String concatenation (was: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement?) In-Reply-To: References: <3414323.Mt0gs2Yp5W@PointedEars.de> Message-ID: f be gfdnbh be b GB GB BH GB vbjfhjb GB bffbbubffffbv GB hbu hbu fjbjfbbbufhbvh VB have fbbbbqbgvfb NB bb GB GB GB GB bbu GB vu GB vu GB GB b GB fbufjnb BH GB GB bvvfbubffffbjubuv GB b fbufbbby GB bfffffff GB f GB bbbu GB GB ffinj GB vh vh fjb GB fj GB h h GB gjfthey're the b GB gjf GBG GBG q GB fbb b bh VB ffbff GBG fbfvrgv On May 9, 2016 7:49 AM, "Chris Angelico" wrote: On Mon, May 9, 2016 at 10:44 AM, Thomas 'PointedEars' Lahn wrote: > With the ?%? string operator (deprecated), str.format(), and str.Template, > you can use other values in string values even without concatenation. Not deprecated. Don't spread FUD. > Finally, with SQL you should prefer Prepared Statements and Stored > Procedures, not bare strings, to avoid SQL injection: > > He is safe. He's using parameterized queries. > Also, it would be a good idea if you posted under your real name. Internet > is the thing with cables; Usenet is the thing with people. I for one tend > to avoid communicating with few-letter entities; exceptions to that would > probably include only E.T., M.J., ALF, and K.I.T.T. I'm not using Usenet, Mr PointedEars. ChrisA -- https://mail.python.org/mailman/listinfo/python-list From mr.eightnoteight at gmail.com Sun May 8 23:43:26 2016 From: mr.eightnoteight at gmail.com (srinivas devaki) Date: Mon, 9 May 2016 09:13:26 +0530 Subject: String concatenation (was: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement?) In-Reply-To: References: <3414323.Mt0gs2Yp5W@PointedEars.de> Message-ID: I'm so sorry, forgot to lock my phone. On May 9, 2016 9:01 AM, "srinivas devaki" wrote: > f be gfdnbh be b GB GB BH GB vbjfhjb GB bffbbubffffbv GB hbu hbu > fjbjfbbbufhbvh VB have fbbbbqbgvfb NB bb GB GB GB GB bbu GB vu GB vu GB GB > b GB fbufjnb BH GB GB bvvfbubffffbjubuv GB b fbufbbby GB bfffffff GB f GB > bbbu GB GB ffinj GB vh vh fjb GB fj GB h h GB gjfthey're the b GB gjf GBG > GBG q GB fbb b bh VB ffbff GBG fbfvrgv > On May 9, 2016 7:49 AM, "Chris Angelico" wrote: > > On Mon, May 9, 2016 at 10:44 AM, Thomas 'PointedEars' Lahn > wrote: > > With the ?%? string operator (deprecated), str.format(), and > str.Template, > > you can use other values in string values even without concatenation. > > Not deprecated. Don't spread FUD. > > > Finally, with SQL you should prefer Prepared Statements and Stored > > Procedures, not bare strings, to avoid SQL injection: > > > > > > He is safe. He's using parameterized queries. > > > Also, it would be a good idea if you posted under your real name. > Internet > > is the thing with cables; Usenet is the thing with people. I for one > tend > > to avoid communicating with few-letter entities; exceptions to that would > > probably include only E.T., M.J., ALF, and K.I.T.T. > > I'm not using Usenet, Mr PointedEars. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > > From steve at pearwood.info Sun May 8 23:46:25 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 09 May 2016 13:46:25 +1000 Subject: pylint woes References: <572f24bc$0$1618$c3e8da3$5496439d@news.astraweb.com> Message-ID: <57300813$0$1620$c3e8da3$5496439d@news.astraweb.com> On Mon, 9 May 2016 07:24 am, DFS wrote: > On 5/8/2016 7:36 AM, Steven D'Aprano wrote: >> On Sun, 8 May 2016 11:16 am, DFS wrote: >> >>> address data is scraped from a website: >>> >>> names = tree.xpath() >>> addr = tree.xpath() >> >> Why are you scraping the data twice? > > > Because it exists in 2 different sections of the document. > > names = tree.xpath('//span[@class="header_text3"]/text()') > addresses = tree.xpath('//span[@class="text3"]/text()') How was I supposed to know that you were providing two different arguments to the method? It looked like a pure-function call, which should always return the same thing. Communication errors are on the sender, not the receiver. You didn't say what tree was, so I judged that xpath was some argument-less method of tree that returned some attribute, sufficiently cleaned up. It would be more obvious to pass a placeholder argument: names = tree.xpath(this) addr = tree.xpath(that) >>> I want to store the data atomically, >> >> I'm not really sure what you mean by "atomically" here. I know what *I* >> mean by "atomically", which is to describe an operation which either >> succeeds entirely or fails. > > That's atomicity. Right. And that doesn't apply to the portion of your code we're discussing. There's no storage involved. The list comps do either succeed entirely or fail, until you actually get to writing to the database, there's nothing that "store the data atomically" would apply to that I saw. [...] >>> so I parse street, city, state, and >>> zip into their own lists. >> >> None of which is atomic. > > All of which are atomic. Sorry, my poor choice of words. None of which are atomic *storage*. >> Oh, and use better names. "street" is a single street, not a list of >> streets, note plural. > > I'll use whatever names I like. *shrug* Its your code, you can name all your variables after "Mr. Meeseeks" if you want. for meeseeks, Meeseeks, meeSeeks, MEEseeks in zip( MESEEKS, meeeseeks, MeesEeks, MEEsEEkS): mEEsEeKSS.append(meeSeeks) ... Just don't ask others to read it. -- Steven From p_s_d_a_s_i_l_v_a_ns at netcabo.pt Mon May 9 00:20:50 2016 From: p_s_d_a_s_i_l_v_a_ns at netcabo.pt (Paulo da Silva) Date: Mon, 9 May 2016 05:20:50 +0100 Subject: A problem with classes - derived type Message-ID: Hi! Suppose I have a class A whose implementation I don't know about. That class A has a method f that returns a A object. class A: ... def f(self, <...>): ... Now I want to write B derived from A with method f1. I want f1 to return a B object: class B(A): ... def f1(self, <...>): ... res=f(<...>) How do I return res as a B object? Thanks. From nospam at dfs.com Mon May 9 00:22:46 2016 From: nospam at dfs.com (DFS) Date: Mon, 9 May 2016 00:22:46 -0400 Subject: Is there a reason zip() wipes out data? Message-ID: python 2.7.11 docs: "The returned list is truncated in length to the length of the shortest argument sequence." a = ['who','let','the'] b = ['dogs','out?'] c = zip(a,b) print c [('who', 'dogs'), ('let', 'out?')] Wouldn't it be better to return an empty element than silently kill your data? [('who', 'dogs'), ('let', 'out?'), ('the', '')] Edit: I see they addressed this in 3.5 (maybe earlier), with an option: "itertools.zip_longest(*iterables, fillvalue=None) Make an iterator that aggregates elements from each of the iterables. If the iterables are of uneven length, missing values are filled-in with fillvalue. Iteration continues until the longest iterable is exhausted." From dan at tombstonezero.net Mon May 9 00:29:54 2016 From: dan at tombstonezero.net (Dan Sommers) Date: Mon, 9 May 2016 04:29:54 -0000 (UTC) Subject: Is there a reason zip() wipes out data? References: Message-ID: On Mon, 09 May 2016 00:22:46 -0400, DFS wrote: > python 2.7.11 docs: "The returned list is truncated in length to the > length of the shortest argument sequence." > > > a = ['who','let','the'] > b = ['dogs','out?'] > c = zip(a,b) > > print c > [('who', 'dogs'), ('let', 'out?')] > > > Wouldn't it be better to return an empty element than silently kill your > data? > > [('who', 'dogs'), ('let', 'out?'), ('the', '')] >>> dfs_zip([1, 2, 3], [4, 5]) [(1, 4), (2, 5), (3, '')] No, that's not much better, thanks. In the face of ambiguity, refuse the temptation to guess. From gengyangcai at gmail.com Mon May 9 00:51:18 2016 From: gengyangcai at gmail.com (Cai Gengyang) Date: Sun, 8 May 2016 21:51:18 -0700 (PDT) Subject: Python PygLatin In-Reply-To: References: <31c06c3b-b622-4791-a607-d0ed2a741da1@googlegroups.com> <572f7370$0$1605$c3e8da3$5496439d@news.astraweb.com> Message-ID: <0f407efd-0e74-4b6b-9f4c-f7645cc7be96@googlegroups.com> I am guessing that the 2 you mentioned are Bill Gates and Larry Ellison ? I heard that they have tons of lawsuits against them in their career (anti-monopoly, anti-competitive laws filed against them both from the government and from individuals) ? Paul Graham has this very interesting related essay on meanness and success ----------------------http://paulgraham.com/mean.html On Monday, May 9, 2016 at 1:53:31 AM UTC+8, alister wrote: > On Mon, 09 May 2016 03:12:14 +1000, Steven D'Aprano wrote: > > > On Sun, 8 May 2016 08:21 pm, Cai Gengyang wrote: > > > >> If one looks at the Forbes List, you will see that there are 4 > >> programmers amongst the top ten richest people in the world (Bill > >> Gates, Mark Zuckerberg, Larry Ellison and Jeff Bezos) , a very large > >> percentage. Science and Technology is in a sense the most egalitarian > >> field in the world, because it involves using your brains and > >> creativity. You don't need to have a father who is a director at > >> Goldman Sachs or a mother who is the admissions officer at Harvard to > >> succeed in this line. > > > > Bill Gates III's father was a prominent lawyer, his mother was on the > > board of directors for First Interstate BancSystem and United Way, and > > one of his grandfathers was a national bank president. Gates himself > > went to Harvard. > > > > Zuckerberg's paternal grandparents were successful middle class, > > described as being the first on the block to own a colour TV. (This was > > back in the days when colour TVs were an expensive toy that few could > > afford.) His parents were also very successful professionals: a dentist > > and a psychiatrist. And he too went to Harvard. Despite the jeans and > > tee-shirts Zuckerberg is known for wearing, he's firmly from the > > professional/upper class. > > > > Bezos comes from a family of land-holders from Texas. His grandfather > > was regional director of the U.S. Atomic Energy Commission, and was > > financially successful enough to retire at an early age. He didn't go to > > Harvard, but he did go to Princeton. > > > > Ellison is the son of an unwed mother who gave him up for adoption by > > her aunt and uncle, comfortably middle-class. That makes him the closest > > out of the group as a "regular guy". > > And at least 2 of the above reached their position using business > practices that could be described as less than 100% honorable & above > board. > > > > > -- > Wait ... is this a FUN THING or the END of LIFE in Petticoat Junction?? From harirammanohar at gmail.com Mon May 9 01:20:31 2016 From: harirammanohar at gmail.com (harirammanohar at gmail.com) Date: Sun, 8 May 2016 22:20:31 -0700 (PDT) Subject: pygame easy create Message-ID: <1f66805c-3a0d-41be-ad35-1377ef25edfc@googlegroups.com> is there anyway (IDE/package) that allows me to create graphics/game just like that (by instructing..., if i say create hills on the screen, it should generate pygame code....)....Anyway :) :) From no.email at nospam.invalid Mon May 9 02:06:06 2016 From: no.email at nospam.invalid (Paul Rubin) Date: Sun, 08 May 2016 23:06:06 -0700 Subject: Is there a reason zip() wipes out data? References: Message-ID: <87twi76a0h.fsf@jester.gateway.pace.com> DFS writes: > Edit: I see they addressed this in 3.5 (maybe earlier), with an option: > "itertools.zip_longest(*iterables, fillvalue=None) This is available in 2.7 as itertools.izip_longest From kaiser.yann at gmail.com Mon May 9 02:51:53 2016 From: kaiser.yann at gmail.com (Yann Kaiser) Date: Mon, 09 May 2016 06:51:53 +0000 Subject: A problem with classes - derived type In-Reply-To: References: Message-ID: If you can't change A to use something like "type(self)(...)" to create its return value, you could use the dark side and swap res's __class__: res.__class__ = B Or: res.__class__ = type(self) Do note that B.__init__ will not be run when you do this, so it is up to you to execute any additional initialization B might require. Alternatively if it makes sense for you you can call B's methods on an A object like so: B.b_method(a_object, ...) On Mon, 9 May 2016 at 06:26 Paulo da Silva wrote: > Hi! > > Suppose I have a class A whose implementation I don't know about. > That class A has a method f that returns a A object. > > class A: > ... > def f(self, <...>): > ... > > Now I want to write B derived from A with method f1. I want f1 to return > a B object: > > class B(A): > ... > def f1(self, <...>): > ... > res=f(<...>) > > How do I return res as a B object? > > Thanks. > -- > https://mail.python.org/mailman/listinfo/python-list > -- Yann Kaiser kaiser.yann at gmail.com yann.kaiser at efrei.net +33 6 51 64 01 89 https://github.com/epsy From __peter__ at web.de Mon May 9 02:59:20 2016 From: __peter__ at web.de (Peter Otten) Date: Mon, 09 May 2016 08:59:20 +0200 Subject: A problem with classes - derived type References: Message-ID: Paulo da Silva wrote: > Hi! > > Suppose I have a class A whose implementation I don't know about. > That class A has a method f that returns a A object. > > class A: > ... > def f(self, <...>): > ... > > Now I want to write B derived from A with method f1. I want f1 to return > a B object: > > class B(A): > ... > def f1(self, <...>): > ... > res=f(<...>) > > How do I return res as a B object? In the general case you need enough knowledge about A to create a B instance from an A instance: class B(A): @classmethod def from_A(cls, a): b = cls(...) # or B(...) return b def f1(self, ...): return self.from_A(self.f(...)) If the internal state doesn't change between A and B, and A is written in Python changing the class of the A instance to B class B(A): def f1(...): a = self.f(...) a.__class__ = B return a may also work. From steve+comp.lang.python at pearwood.info Mon May 9 03:53:45 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 09 May 2016 17:53:45 +1000 Subject: What's wrong with this concatenation statement? References: Message-ID: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> On Monday 09 May 2016 09:10, DFS wrote: > sSQL = "line 1\n" > sSQL += "line 2\n" > sSQL += "line 3" Pointlessly provocative subject line edited. Since all three lines are constants know by the programmer at the time the source code is written, it should be written as: sSQL = """line 1 line 2 line 3""" Or if you prefer: sSQL = "line 1\nline 2\nline 3" Or even: sSQL = ("line 1\n" "line 2\n" "line 3") taking advantage of Python's compile-time implicit concatenation of string constants. -- Steve From alister.ware at ntlworld.com Mon May 9 04:52:52 2016 From: alister.ware at ntlworld.com (alister) Date: Mon, 09 May 2016 08:52:52 GMT Subject: Python PygLatin References: <31c06c3b-b622-4791-a607-d0ed2a741da1@googlegroups.com> <572f7370$0$1605$c3e8da3$5496439d@news.astraweb.com> <572F7F16.8040809@icloud.com> Message-ID: On Sun, 08 May 2016 11:01:58 -0700, Christopher Reimer wrote: > On 5/8/2016 10:53 AM, alister wrote: >> On Mon, 09 May 2016 03:12:14 +1000, Steven D'Aprano wrote: >> >>> On Sun, 8 May 2016 08:21 pm, Cai Gengyang wrote: >>> >>>> If one looks at the Forbes List, you will see that there are 4 >>>> programmers amongst the top ten richest people in the world (Bill >>>> Gates, Mark Zuckerberg, Larry Ellison and Jeff Bezos) , a very large >>>> percentage. Science and Technology is in a sense the most egalitarian >>>> field in the world, because it involves using your brains and >>>> creativity. You don't need to have a father who is a director at >>>> Goldman Sachs or a mother who is the admissions officer at Harvard to >>>> succeed in this line. >>> Bill Gates III's father was a prominent lawyer, his mother was on the >>> board of directors for First Interstate BancSystem and United Way, and >>> one of his grandfathers was a national bank president. Gates himself >>> went to Harvard. >>> >>> Zuckerberg's paternal grandparents were successful middle class, >>> described as being the first on the block to own a colour TV. (This >>> was back in the days when colour TVs were an expensive toy that few >>> could afford.) His parents were also very successful professionals: a >>> dentist and a psychiatrist. And he too went to Harvard. Despite the >>> jeans and tee-shirts Zuckerberg is known for wearing, he's firmly from >>> the professional/upper class. >>> >>> Bezos comes from a family of land-holders from Texas. His grandfather >>> was regional director of the U.S. Atomic Energy Commission, and was >>> financially successful enough to retire at an early age. He didn't go >>> to Harvard, but he did go to Princeton. >>> >>> Ellison is the son of an unwed mother who gave him up for adoption by >>> her aunt and uncle, comfortably middle-class. That makes him the >>> closest out of the group as a "regular guy". >> And at least 2 of the above reached their position using business >> practices that could be described as less than 100% honorable & above >> board. > > What do you expect from people who haven't graduated from Harvard? :P > > Thank you, > > Chris R. i think the two people i am thinking of both did go to Harvard, i don't know if they graduated or not -- Old MacDonald had an agricultural real estate tax abatement. From alister.ware at ntlworld.com Mon May 9 05:12:32 2016 From: alister.ware at ntlworld.com (alister) Date: Mon, 09 May 2016 09:12:32 GMT Subject: Python PygLatin References: <31c06c3b-b622-4791-a607-d0ed2a741da1@googlegroups.com> <572f7370$0$1605$c3e8da3$5496439d@news.astraweb.com> <0f407efd-0e74-4b6b-9f4c-f7645cc7be96@googlegroups.com> Message-ID: <4wYXy.434610$7l1.329049@fx39.am4> On Sun, 08 May 2016 21:51:18 -0700, Cai Gengyang wrote: > I am guessing that the 2 you mentioned are Bill Gates and Larry Ellison > ? I heard that they have tons of lawsuits against them in their career > (anti-monopoly, anti-competitive laws filed against them both from the > government and from individuals) ? > > Paul Graham has this very interesting related essay on meanness and > success ----------------------http://paulgraham.com/mean.html > 50% correct, although I was thinking more of "borrowed code/ip", doublespace is one term that springs to mind. Also it is the accepted practice to use interleaved posting on this mailing list/newsgroup please do not top post -- Goes (Went) over like a lead balloon. From michael at stroeder.com Mon May 9 06:00:17 2016 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Mon, 9 May 2016 12:00:17 +0200 Subject: pypi download links (e.g. for ansible) Message-ID: HI! Deep-links for downloading a specific version from PyPI seemed to work like this: $ wget https://pypi.python.org/packages/source/a/ansible/ansible-2.0.1.0.tar.gz [..] Saving to: ?ansible-2.0.1.0.tar.gz? But this recent version does not work: $ wget https://pypi.python.org/packages/source/a/ansible/ansible-2.0.2.0.tar.gz [..] HTTP request sent, awaiting response... 404 Not Found Ciao, Michael. From harirammanohar at gmail.com Mon May 9 06:15:26 2016 From: harirammanohar at gmail.com (harirammanohar at gmail.com) Date: Mon, 9 May 2016 03:15:26 -0700 (PDT) Subject: pygame easy create In-Reply-To: <1f66805c-3a0d-41be-ad35-1377ef25edfc@googlegroups.com> References: <1f66805c-3a0d-41be-ad35-1377ef25edfc@googlegroups.com> Message-ID: <2a7bf274-f1f8-4ab1-ad13-ed2dedaaad8d@googlegroups.com> On Monday, May 9, 2016 at 10:50:47 AM UTC+5:30, hariram... at gmail.com wrote: > is there anyway (IDE/package) that allows me to create graphics/game just like that (by instructing..., if i say create hills on the screen, it should generate pygame code....)....Anyway :) :) Atleast i tried with pyglet,felt that it will be easier than pygame..... still again throwing me error some GL is missed, why they wont be straight forward in working if we select the env we have... some sites says 2.7 and >=3(3.4+) will have pip module by default, but i am using 3.4.3 where i dont have pip and also tried python get-pip.py which also not worked...... From harirammanohar at gmail.com Mon May 9 06:16:42 2016 From: harirammanohar at gmail.com (harirammanohar at gmail.com) Date: Mon, 9 May 2016 03:16:42 -0700 (PDT) Subject: pypi download links (e.g. for ansible) In-Reply-To: References: Message-ID: <21a4b51c-f1c1-42b5-8810-91ac4012d279@googlegroups.com> On Monday, May 9, 2016 at 3:30:31 PM UTC+5:30, Michael Str?der wrote: > HI! > > Deep-links for downloading a specific version from PyPI seemed to work like this: > > $ wget https://pypi.python.org/packages/source/a/ansible/ansible-2.0.1.0.tar.gz > [..] > Saving to: 'ansible-2.0.1.0.tar.gz' > > But this recent version does not work: > > $ wget https://pypi.python.org/packages/source/a/ansible/ansible-2.0.2.0.tar.gz > [..] > HTTP request sent, awaiting response... 404 Not Found > > Ciao, Michael. how its working for you, for me it says ssl connection error,if i do wget from linux as root user. From harirammanohar at gmail.com Mon May 9 06:17:58 2016 From: harirammanohar at gmail.com (harirammanohar at gmail.com) Date: Mon, 9 May 2016 03:17:58 -0700 (PDT) Subject: pygame easy create In-Reply-To: <1f66805c-3a0d-41be-ad35-1377ef25edfc@googlegroups.com> References: <1f66805c-3a0d-41be-ad35-1377ef25edfc@googlegroups.com> Message-ID: <8c7799cb-e83d-45e5-bffd-55ffc6f86ba9@googlegroups.com> On Monday, May 9, 2016 at 10:50:47 AM UTC+5:30, hariram... at gmail.com wrote: > is there anyway (IDE/package) that allows me to create graphics/game just like that (by instructing..., if i say create hills on the screen, it should generate pygame code....)....Anyway :) :) is there any way to create high graphics game using python, else its waste to learn pygame..... is there any module is there to create high grapihc game modules... From robin.koch at t-online.de Mon May 9 06:21:56 2016 From: robin.koch at t-online.de (Robin Koch) Date: Mon, 9 May 2016 12:21:56 +0200 Subject: Python PygLatin In-Reply-To: <31c06c3b-b622-4791-a607-d0ed2a741da1@googlegroups.com> References: <31c06c3b-b622-4791-a607-d0ed2a741da1@googlegroups.com> Message-ID: Am 08.05.2016 um 12:21 schrieb Cai Gengyang: > If one looks at the Forbes List, you will see that there are 4 > programmers amongst the top ten richest people in the world (Bill > Gates, Mark Zuckerberg, Larry Ellison and Jeff Bezos) , a very large > percentage. You might elaborate your knowledge about conditional probability as well. P("X is programmer" | "X is in Forbes Top 10") != P("X is in Forbes Top 10" | "X is programmer") -- Robin Koch From michael at stroeder.com Mon May 9 06:27:44 2016 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Mon, 9 May 2016 12:27:44 +0200 Subject: pypi download links (e.g. for ansible) In-Reply-To: <21a4b51c-f1c1-42b5-8810-91ac4012d279@googlegroups.com> References: <21a4b51c-f1c1-42b5-8810-91ac4012d279@googlegroups.com> Message-ID: harirammanohar at gmail.com wrote: > On Monday, May 9, 2016 at 3:30:31 PM UTC+5:30, Michael Str?der wrote: >> HI! >> >> Deep-links for downloading a specific version from PyPI seemed to work like this: >> >> $ wget https://pypi.python.org/packages/source/a/ansible/ansible-2.0.1.0.tar.gz >> [..] >> Saving to: 'ansible-2.0.1.0.tar.gz' >> >> But this recent version does not work: >> >> $ wget https://pypi.python.org/packages/source/a/ansible/ansible-2.0.2.0.tar.gz >> [..] >> HTTP request sent, awaiting response... 404 Not Found > > how its working for you, for me it says ssl connection error,if i do wget from linux as root user. On my Linux installation the usually Mozilla web browser CA cert bundle is installed. Ciao, Michael. From steve at pearwood.info Mon May 9 06:44:31 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 09 May 2016 20:44:31 +1000 Subject: pypi download links (e.g. for ansible) References: Message-ID: <57306a11$0$1616$c3e8da3$5496439d@news.astraweb.com> On Mon, 9 May 2016 08:00 pm, Michael Strc3b6der wrote: > HI! > > Deep-links for downloading a specific version from PyPI seemed to work > like this: > > $ wget > https://pypi.python.org/packages/source/a/ansible/ansible-2.0.1.0.tar.gz > [..] > Saving to: ?ansible-2.0.1.0.tar.gz? > > But this recent version does not work: > > $ wget > https://pypi.python.org/packages/source/a/ansible/ansible-2.0.2.0.tar.gz > [..] > HTTP request sent, awaiting response... 404 Not Found Do you have a question, or are you just sharing? The Download button for 2.0.2.0 links to: https://pypi.python.org/packages/b3/0e/5f3ee8884866a3d5e3b8ba86e9caa85ecdec75adabac8924b1c122339e7f/ansible-2.0.2.0.tar.gz if that helps. -- Steven From michael at stroeder.com Mon May 9 07:35:47 2016 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Mon, 9 May 2016 13:35:47 +0200 Subject: pypi download links (e.g. for ansible) In-Reply-To: <57306a11$0$1616$c3e8da3$5496439d@news.astraweb.com> References: <57306a11$0$1616$c3e8da3$5496439d@news.astraweb.com> Message-ID: Steven D'Aprano wrote: > On Mon, 9 May 2016 08:00 pm, Michael Strc3b6der wrote: > >> HI! >> >> Deep-links for downloading a specific version from PyPI seemed to work >> like this: >> >> $ wget >> https://pypi.python.org/packages/source/a/ansible/ansible-2.0.1.0.tar.gz >> [..] >> Saving to: ?ansible-2.0.1.0.tar.gz? >> >> But this recent version does not work: >> >> $ wget >> https://pypi.python.org/packages/source/a/ansible/ansible-2.0.2.0.tar.gz >> [..] >> HTTP request sent, awaiting response... 404 Not Found > > > Do you have a question, or are you just sharing? > > The Download button for 2.0.2.0 links to: > > https://pypi.python.org/packages/b3/0e/5f3ee8884866a3d5e3b8ba86e9caa85ecdec75adabac8924b1c122339e7f/ansible-2.0.2.0.tar.gz Yes, but in .spec files of openSUSE RPMs the more readable links above are used and I'd like to keep it that way. And openSUSE build service checks whether it's downloadable. It works for ansible-2.0.1.0 but not for 2.0.2.0. Ciao, Michael. From rustompmody at gmail.com Mon May 9 10:54:41 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Mon, 9 May 2016 07:54:41 -0700 (PDT) Subject: Python PygLatin In-Reply-To: <31c06c3b-b622-4791-a607-d0ed2a741da1@googlegroups.com> References: <31c06c3b-b622-4791-a607-d0ed2a741da1@googlegroups.com> Message-ID: <3e01cea3-38b5-4de7-a35c-c07c1f80fdaf@googlegroups.com> On Sunday, May 8, 2016 at 3:52:12 PM UTC+5:30, Cai Gengyang wrote: > I just "clicked" through the lesson on Conditionals and Control Flows and am on the lesson "PygLatin" . > > This will hopefully be a more interesting and interactive lesson because I will be building a PygLatin Translator ... > > It seems to me like it will take a long time before I can reach the point where I can become a master of programming languages and even create new programming languages and technologies. This is a long road ahead, but I believe it is worth it because the power of a new technology does eventually translate into money. If one looks at the Forbes List, you will see that there are 4 programmers amongst the top ten richest people in the world (Bill Gates, Mark Zuckerberg, Larry Ellison and Jeff Bezos) , a very large percentage. Science and Technology is in a sense the most egalitarian field in the world, because it involves using your brains and creativity. You don't need to have a father who is a director at Goldman Sachs or a mother who is the admissions officer at Harvard to succeed in this line. All you need is have the ability to create something that many users love to use. Here are some physical creations which humans have made: 1. Petronas [1] 2. Chartres [2] 3. Egypt pyramids [3] 4. Arunachala temple [4] What do you think is the ratio of those that designed/conceptualized these to those who laid stone/bricks/steel? My estimate 1:100 to 1:10000. Software is no different. If you train as a pilot you have a better chance of landing the lead position on the Mars-mission than if you train as a programmer and hope to become Bill Gates II. Helpful to read "It doesnt matter" [5] in this respect. Dont get fooled by the title. What it really says is that as IT transitions from innovation to utility, it matters more and more but only in a negative sense. eg if the power or the water or the roads are bad we get very indignant. When they are fine the so-called 'blue-collar-workers' who provide them are faceless, invisible. Cost gravity [6] explains the mechanics of how that happens In other words we IT-professionals are the blue-collar workers of the 21st century. Why is this not more widely undersood? Because CS-education is grossly out of sync with IT-practice. See [7] -- CS education is habitually taught as though only only the first column exists. In the IT world its mostly the second that counts. Steven reminds of the caste-system that the super-rich are usually sons of the rich in our field like anywhere else. This is true but misunderstood. It is not that money begets more money like some kind of magnet. Its rather that the rich have a certain freedom to their thinking/outlook. An outlook that others too can with some effort emulate. Triune brain [8] explains that we have 3 brains corresponding to our evolutionary stages: - reptilian - mammalian - specifically human (cortex) When we are in survive/sex mode we are essentially reptiles. When we are in emotional mode we are a bit better -- mammals. If we can get beyond both we can be truly human. This is hard for anyone, but for the poor the survival needs predominate so its harder. For some thoughts on switching from the more animal to the more human aspects of our functioning see [9] [I guess the advantage the rich have with survival is lost to sex] 2 Refs ====== 1. https://upload.wikimedia.org/wikipedia/commons/6/6d/Petronas_Twin_Towers_2010_April.jpg 2. http://whc.unesco.org/uploads/thumbs/site_0081_0001-750-0-20151104132433.jpg 3. https://upload.wikimedia.org/wikipedia/commons/a/af/All_Gizah_Pyramids.jpg 4. https://arunachaleshwarar.files.wordpress.com/2012/09/tem.jpg 5. https://hbr.org/2003/05/it-doesnt-matter 6. http://content.cultureandempire.com/preface.html 7. http://blog.languager.org/2010/02/service-and-product-mindsets.html 8. http://www.kheper.net/topics/intelligence/MacLean.htm 9. http://blog.languager.org/2010/05/declaration-imperation-and-language.html From torriem at gmail.com Mon May 9 11:01:25 2016 From: torriem at gmail.com (Michael Torrie) Date: Mon, 9 May 2016 09:01:25 -0600 Subject: Python PygLatin In-Reply-To: <31c06c3b-b622-4791-a607-d0ed2a741da1@googlegroups.com> References: <31c06c3b-b622-4791-a607-d0ed2a741da1@googlegroups.com> Message-ID: <5730A645.4070204@gmail.com> On 05/08/2016 04:21 AM, Cai Gengyang wrote: > This is a long > road ahead, but I believe it is worth it because the power of a new > technology does eventually translate into money. If this is your prime motivation, I think you'll be very disappointed. A good programmer certainly can make a good salary. But the road to riches? Well that's actually relatively rare. Just look at the the Apple Store or Google Play Store. Thousands of games and apps and very few small developers making the big dollars through it. Some manage to get lucky but most do not. Really good programmers do it because they enjoy doing it, whether it's the problem solving aspects, or maybe because they enjoy creating a product that is useful to others. While computer programming is a learned skill, some have more aptitude and intuition for it than others. If you honestly enjoy it then you'll do fine. If you don't find it intellectually stimulating then you'll definitely struggle as far as a career path goes. From kwpolska at gmail.com Mon May 9 11:26:15 2016 From: kwpolska at gmail.com (Chris Warrick) Date: Mon, 9 May 2016 17:26:15 +0200 Subject: pypi download links (e.g. for ansible) In-Reply-To: References: <57306a11$0$1616$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 9 May 2016 at 13:35, Michael Str?der wrote: > Steven D'Aprano wrote: >> On Mon, 9 May 2016 08:00 pm, Michael Strc3b6der wrote: >> >>> HI! >>> >>> Deep-links for downloading a specific version from PyPI seemed to work >>> like this: >>> >>> $ wget >>> https://pypi.python.org/packages/source/a/ansible/ansible-2.0.1.0.tar.gz >>> [..] >>> Saving to: ?ansible-2.0.1.0.tar.gz? >>> >>> But this recent version does not work: >>> >>> $ wget >>> https://pypi.python.org/packages/source/a/ansible/ansible-2.0.2.0.tar.gz >>> [..] >>> HTTP request sent, awaiting response... 404 Not Found >> >> >> Do you have a question, or are you just sharing? >> >> The Download button for 2.0.2.0 links to: >> >> https://pypi.python.org/packages/b3/0e/5f3ee8884866a3d5e3b8ba86e9caa85ecdec75adabac8924b1c122339e7f/ansible-2.0.2.0.tar.gz > > Yes, but in .spec files of openSUSE RPMs the more readable links above are used > and I'd like to keep it that way. And openSUSE build service checks whether it's > downloadable. It works for ansible-2.0.1.0 but not for 2.0.2.0. > > Ciao, Michael. > > -- > https://mail.python.org/mailman/listinfo/python-list PyPI URLs were changed recently. There is, however, a new supported way to get dependable URLs: https://bitbucket.org/pypa/pypi/issues/438/backwards-compatible-un-hashed-package The current link: https://files.pythonhosted.org/packages/source/a/ansible/ansible-2.0.1.0.tar.gz (URLs in the pypi.io domain were used briefly and will still work) -- Chris Warrick PGP: 5EAAEA16 From sohcahtoa82 at gmail.com Mon May 9 13:42:30 2016 From: sohcahtoa82 at gmail.com (sohcahtoa82 at gmail.com) Date: Mon, 9 May 2016 10:42:30 -0700 (PDT) Subject: pygame easy create In-Reply-To: <2a7bf274-f1f8-4ab1-ad13-ed2dedaaad8d@googlegroups.com> References: <1f66805c-3a0d-41be-ad35-1377ef25edfc@googlegroups.com> <2a7bf274-f1f8-4ab1-ad13-ed2dedaaad8d@googlegroups.com> Message-ID: <61a15673-a29e-4152-8bf5-ac1e82c68d3c@googlegroups.com> On Monday, May 9, 2016 at 3:15:45 AM UTC-7, hariram... at gmail.com wrote: > On Monday, May 9, 2016 at 10:50:47 AM UTC+5:30, hariram... at gmail.com wrote: > > is there anyway (IDE/package) that allows me to create graphics/game just like that (by instructing..., if i say create hills on the screen, it should generate pygame code....)....Anyway :) :) > > Atleast i tried with pyglet,felt that it will be easier than pygame..... still again throwing me error some GL is missed, why they wont be straight forward in working if we select the env we have... > > some sites says 2.7 and >=3(3.4+) will have pip module by default, but i am using 3.4.3 where i dont have pip and also tried python get-pip.py which also not worked...... "still again throwing me error some GL is missed" Can you say *EXACTLY* what the error is? And can you copy/paste the relevant lines of code that lead to that error? Also, judging from your other messages, it looks like you might be needing to read the Python or PyGame tutorials. Programming is problem solving. You can't just saying "Draw a hill" and PyGame (or whatever module you're using) will draw a hill. You need to write code that describes how to draw a hill. From michael.selik at gmail.com Mon May 9 14:29:05 2016 From: michael.selik at gmail.com (Michael Selik) Date: Mon, 09 May 2016 18:29:05 +0000 Subject: String concatenation (was: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement?) In-Reply-To: References: <3414323.Mt0gs2Yp5W@PointedEars.de> Message-ID: You're saying that wasn't a coded message? On Sun, May 8, 2016, 10:44 PM srinivas devaki wrote: > I'm so sorry, forgot to lock my phone. > On May 9, 2016 9:01 AM, "srinivas devaki" > wrote: > > > f be gfdnbh be b GB GB BH GB vbjfhjb GB bffbbubffffbv GB hbu hbu > > fjbjfbbbufhbvh VB have fbbbbqbgvfb NB bb GB GB GB GB bbu GB vu GB vu GB > GB > > b GB fbufjnb BH GB GB bvvfbubffffbjubuv GB b fbufbbby GB bfffffff GB f GB > > bbbu GB GB ffinj GB vh vh fjb GB fj GB h h GB gjfthey're the b GB gjf GBG > > GBG q GB fbb b bh VB ffbff GBG fbfvrgv > > On May 9, 2016 7:49 AM, "Chris Angelico" wrote: > > > > On Mon, May 9, 2016 at 10:44 AM, Thomas 'PointedEars' Lahn > > wrote: > > > With the ?%? string operator (deprecated), str.format(), and > > str.Template, > > > you can use other values in string values even without concatenation. > > > > Not deprecated. Don't spread FUD. > > > > > Finally, with SQL you should prefer Prepared Statements and Stored > > > Procedures, not bare strings, to avoid SQL injection: > > > > > > > > > > He is safe. He's using parameterized queries. > > > > > Also, it would be a good idea if you posted under your real name. > > Internet > > > is the thing with cables; Usenet is the thing with people. I for one > > tend > > > to avoid communicating with few-letter entities; exceptions to that > would > > > probably include only E.T., M.J., ALF, and K.I.T.T. > > > > I'm not using Usenet, Mr PointedEars. > > > > ChrisA > > -- > > https://mail.python.org/mailman/listinfo/python-list > > > > > -- > https://mail.python.org/mailman/listinfo/python-list > From p_s_d_a_s_i_l_v_a_ns at netcabo.pt Mon May 9 14:59:46 2016 From: p_s_d_a_s_i_l_v_a_ns at netcabo.pt (Paulo da Silva) Date: Mon, 9 May 2016 19:59:46 +0100 Subject: A problem with classes - derived type References: Message-ID: ?s 05:20 de 09-05-2016, Paulo da Silva escreveu: Thank you Yann and Peter. I really didn't know anything about those "things". So far I have worked a lot with classes but they are written by me. Now I needed to derive pandas.Series (for example) and it has some methods that return pandas.Series objects. And I needed to return the derived object, not the pandas.Series one. The problem is now fixed. I'll find some time to read a little more about this to improve my pyhon knowledge. Thank you very much. From zljubisic at gmail.com Mon May 9 15:56:02 2016 From: zljubisic at gmail.com (zljubisic at gmail.com) Date: Mon, 9 May 2016 12:56:02 -0700 (PDT) Subject: json.loads(...) ValueError: Expecting value: line 1 column 1 (char 0) Message-ID: <137a2593-c307-405f-8f23-3959099ff16f@googlegroups.com> Hi, in python3 my variable looks like this: a = b'{"uuid":"5730e8666ffa02.34177329","error":""}' str(a) = 'b\'{"uuid":"5730e8666ffa02.34177329","error":""}\'' If I execute the following command I get the error: >>> json.loads(str(a)) Traceback (most recent call last): File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.1.2\helpers\pydev\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec exec(exp, global_vars, local_vars) File "", line 1, in File "C:\Program Files\Python34\lib\json\__init__.py", line 318, in loads return _default_decoder.decode(s) File "C:\Program Files\Python34\lib\json\decoder.py", line 343, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Program Files\Python34\lib\json\decoder.py", line 361, in raw_decode raise ValueError(errmsg("Expecting value", s, err.value)) from None ValueError: Expecting value: line 1 column 1 (char 0) Why I am getting this error? If I set variable a to the '{"uuid":"5730e8666ffa02.34177329","error":""}' everything works as expected. Regards. From __peter__ at web.de Mon May 9 16:17:30 2016 From: __peter__ at web.de (Peter Otten) Date: Mon, 09 May 2016 22:17:30 +0200 Subject: json.loads(...) ValueError: Expecting value: line 1 column 1 (char 0) References: <137a2593-c307-405f-8f23-3959099ff16f@googlegroups.com> Message-ID: zljubisic at gmail.com wrote: > Hi, > > in python3 my variable looks like this: > > a = b'{"uuid":"5730e8666ffa02.34177329","error":""}' > str(a) = 'b\'{"uuid":"5730e8666ffa02.34177329","error":""}\'' > > If I execute the following command I get the error: > >>>> json.loads(str(a)) > Traceback (most recent call last): > File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition > 2016.1.2\helpers\pydev\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec > exec(exp, global_vars, local_vars) > File "", line 1, in > File "C:\Program Files\Python34\lib\json\__init__.py", line 318, in > loads > return _default_decoder.decode(s) > File "C:\Program Files\Python34\lib\json\decoder.py", line 343, in > decode > obj, end = self.raw_decode(s, idx=_w(s, 0).end()) > File "C:\Program Files\Python34\lib\json\decoder.py", line 361, in > raw_decode > raise ValueError(errmsg("Expecting value", s, err.value)) from None > ValueError: Expecting value: line 1 column 1 (char 0) > > Why I am getting this error? > If I set variable a to the '{"uuid":"5730e8666ffa02.34177329","error":""}' > everything works as expected. Look at the traceback: "line 1 column 1 (char 0)" mentioned in the error message is the leading "b". When you convert a byte string to unicode with str(bytestr) the "b" prefix and the quotation marks are part of the resulting string, but not valid JSON. Try a.decode() instead of str(a): >>> a = b'{"uuid":"5730e8666ffa02.34177329","error":""}' >>> json.loads(a.decode()) {'error': '', 'uuid': '5730e8666ffa02.34177329'} From python at mrabarnett.plus.com Mon May 9 16:21:05 2016 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 9 May 2016 21:21:05 +0100 Subject: json.loads(...) ValueError: Expecting value: line 1 column 1 (char 0) In-Reply-To: <137a2593-c307-405f-8f23-3959099ff16f@googlegroups.com> References: <137a2593-c307-405f-8f23-3959099ff16f@googlegroups.com> Message-ID: <0bc0f2ad-3190-4ba0-534b-ab11015eeb54@mrabarnett.plus.com> On 2016-05-09 20:56, zljubisic at gmail.com wrote: > Hi, > > in python3 my variable looks like this: > > a = b'{"uuid":"5730e8666ffa02.34177329","error":""}' > str(a) = 'b\'{"uuid":"5730e8666ffa02.34177329","error":""}\'' > > If I execute the following command I get the error: > >>>> json.loads(str(a)) > Traceback (most recent call last): > File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.1.2\helpers\pydev\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec > exec(exp, global_vars, local_vars) > File "", line 1, in > File "C:\Program Files\Python34\lib\json\__init__.py", line 318, in loads > return _default_decoder.decode(s) > File "C:\Program Files\Python34\lib\json\decoder.py", line 343, in decode > obj, end = self.raw_decode(s, idx=_w(s, 0).end()) > File "C:\Program Files\Python34\lib\json\decoder.py", line 361, in raw_decode > raise ValueError(errmsg("Expecting value", s, err.value)) from None > ValueError: Expecting value: line 1 column 1 (char 0) > > Why I am getting this error? > If I set variable a to the '{"uuid":"5730e8666ffa02.34177329","error":""}' everything works as expected. > The b-prefix is Python-specific. It's not valid JSON syntax. The JSON format is defined here: http://www.json.org/ From ben.usenet at bsb.me.uk Mon May 9 16:41:48 2016 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Mon, 09 May 2016 21:41:48 +0100 Subject: json.loads(...) ValueError: Expecting value: line 1 column 1 (char 0) References: <137a2593-c307-405f-8f23-3959099ff16f@googlegroups.com> Message-ID: <87shxrc6b7.fsf@bsb.me.uk> zljubisic at gmail.com writes: > in python3 my variable looks like this: > > a = b'{"uuid":"5730e8666ffa02.34177329","error":""}' > str(a) = 'b\'{"uuid":"5730e8666ffa02.34177329","error":""}\'' > > If I execute the following command I get the error: > >>>> json.loads(str(a)) > Traceback (most recent call last): > File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.1.2\helpers\pydev\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec > exec(exp, global_vars, local_vars) > File "", line 1, in > File "C:\Program Files\Python34\lib\json\__init__.py", line 318, in loads > return _default_decoder.decode(s) > File "C:\Program Files\Python34\lib\json\decoder.py", line 343, in decode > obj, end = self.raw_decode(s, idx=_w(s, 0).end()) > File "C:\Program Files\Python34\lib\json\decoder.py", line 361, in raw_decode > raise ValueError(errmsg("Expecting value", s, err.value)) from None > ValueError: Expecting value: line 1 column 1 (char 0) > > Why I am getting this error? The result of str(a) is not a valid JSON string. It starts with a b but a JSON string must start with a digit of one of -, ", {, [, t, f, n (the letters being legal only if they are the start of true, false or null. In fact (in Python 3), passing bytes to str() without an encoding is a special case -- you get an informal string representation. I'm not sure you can be sure what you get though I imagine it's designed to be the same as Python 2 gave. > If I set variable a to the > '{"uuid":"5730e8666ffa02.34177329","error":""}' everything works as > expected. That string starts with { so it's OK. You probably what something like json.loads(str(a, 'ascii')) or maybe 'utf-8' or 'windows-1285' or... well you get the idea. You need to say how the bytes should be turned into string characters. -- Ben. From tjreedy at udel.edu Mon May 9 19:07:38 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 9 May 2016 19:07:38 -0400 Subject: json.loads(...) ValueError: Expecting value: line 1 column 1 (char 0) In-Reply-To: <137a2593-c307-405f-8f23-3959099ff16f@googlegroups.com> References: <137a2593-c307-405f-8f23-3959099ff16f@googlegroups.com> Message-ID: On 5/9/2016 3:56 PM, zljubisic at gmail.com wrote: > Hi, > > in python3 my variable looks like this: > > a = b'{"uuid":"5730e8666ffa02.34177329","error":""}' > str(a) = 'b\'{"uuid":"5730e8666ffa02.34177329","error":""}\'' > > If I execute the following command I get the error: > >>>> json.loads(str(a)) > Traceback (most recent call last): > File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.1.2\helpers\pydev\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec > exec(exp, global_vars, local_vars) > File "", line 1, in > File "C:\Program Files\Python34\lib\json\__init__.py", line 318, in loads > return _default_decoder.decode(s) > File "C:\Program Files\Python34\lib\json\decoder.py", line 343, in decode > obj, end = self.raw_decode(s, idx=_w(s, 0).end()) > File "C:\Program Files\Python34\lib\json\decoder.py", line 361, in raw_decode > raise ValueError(errmsg("Expecting value", s, err.value)) from None > ValueError: Expecting value: line 1 column 1 (char 0) > > Why I am getting this error? > If I set variable a to the '{"uuid":"5730e8666ffa02.34177329","error":""}' everything works as expected. This means that the two versions of 'a' are not the same. So what you should have done to debug is print the second to see what you actually passed to json. Editorial: Programming classes should teach basic debugging better. I have seen numerous newbie Stackoverflow questions where the person should have started with adding a print statement before posting a question. -- Terry Jan Reedy From 007brendan at gmail.com Mon May 9 19:54:00 2016 From: 007brendan at gmail.com (Brendan Abel) Date: Mon, 9 May 2016 16:54:00 -0700 Subject: Are imports supposed to be like this? Message-ID: Consider the following example python package where `a.py` and `b.py` depend on each other: /package __init__.py a.py b.py There are several ways I could import the "a.py" module in "b.py" import package.a # Absolute import import package.a as a_mod # Absolute import bound to different name from package import a # Alternate absolute import import a # Implicit relative import (deprecated, py2 only) from . import a # Explicit relative import Unfortunately, only the 1st and 4th syntax actually work when you have circular dependencies (the rest all raise `ImportError` or `AttributeError`), and the 4th syntax only works in python 2 and is generally discouraged because of the possibility of name conflicts. I'd much rather use relative imports, or the "from x import y" syntax, or at least be able to use the "as" import syntax. If I have a deeply nested package, the imports become unruly rather quickly, and I have to use that long, ugly name throughout the entire module! import package.subpackage.submodule.module # fugly! Are imports designed to work this way, or is this a bug in the import machinery? What reasoning is there for the first syntax to work, but all the others should fail? Admittedly, I haven't used Python3 yet, does it fix this? It seems odd to me that the documentation seems to encourage relative imports or at least the "from x.y.z import a" forms of imports, yet they don't work the same as "import x.y.z.a". //Brendan From rosuav at gmail.com Mon May 9 20:20:33 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 10 May 2016 10:20:33 +1000 Subject: Are imports supposed to be like this? In-Reply-To: References: Message-ID: On Tue, May 10, 2016 at 9:54 AM, Brendan Abel <007brendan at gmail.com> wrote: > Consider the following example python package where `a.py` and `b.py` > depend on each other: > > /package > __init__.py > a.py > b.py > > > There are several ways I could import the "a.py" module in "b.py" > > import package.a # Absolute import > import package.a as a_mod # Absolute import bound to different name > from package import a # Alternate absolute import > import a # Implicit relative import (deprecated, py2 > only) > from . import a # Explicit relative import > > Unfortunately, only the 1st and 4th syntax actually work when you have > circular dependencies (the rest all raise `ImportError` or > `AttributeError`), and the 4th syntax only works in python 2 and is > generally discouraged because of the possibility of name conflicts. The fifth is the one that I would recommend. Can you give an example of a circular dependency that makes it fail? Here's the trivial case that I tried: rosuav at sikorsky:~/tmp$ mkdir package rosuav at sikorsky:~/tmp$ touch package/__init__.py rosuav at sikorsky:~/tmp$ cat >package/a.py from . import b def func_a(x): print("func_a: %d" % x) if x % 2: b.func_b(x-1) rosuav at sikorsky:~/tmp$ cat >package/b.py from . import a def func_b(x): print("func_b: %d" % x) if x % 2: a.func_a(x-1) rosuav at sikorsky:~/tmp$ python3 Python 3.6.0a0 (default:98678738b7e9, May 2 2016, 13:37:04) [GCC 5.3.1 20160409] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import package.a >>> package.a.func_a(5) func_a: 5 func_b: 4 >>> import package.b >>> package.b.func_b(5) func_b: 5 func_a: 4 >>> The imports work fine as long as you use strictly "from . import modulename", and not "from .modulename import objectname". Circular imports of the latter form will indeed fail, but if you have non-circular imports, you can stop them from failing by forcing module load order: rosuav at sikorsky:~/tmp$ cat package/a.py from .b import func_b def func_a(x): print("func_a: %d" % x) if x % 2: func_b(x-1) rosuav at sikorsky:~/tmp$ cat >package/__init__.py from . import a from . import b This does prevent lazy loading, though, so if your circular imports are in an optional part of the package (and one that takes a long time to load), you might want to consider setting it up some other way. ChrisA From random832 at fastmail.com Mon May 9 20:26:05 2016 From: random832 at fastmail.com (Random832) Date: Mon, 09 May 2016 20:26:05 -0400 Subject: Are imports supposed to be like this? In-Reply-To: References: Message-ID: <1462839965.2919682.602944817.37131FE3@webmail.messagingengine.com> On Mon, May 9, 2016, at 19:54, Brendan Abel wrote: > Consider the following example python package where `a.py` and `b.py` > depend on each other: > > /package > __init__.py > a.py > b.py > > > There are several ways I could import the "a.py" module in "b.py" > > import package.a # Absolute import > import package.a as a_mod # Absolute import bound to different name > from package import a # Alternate absolute import > import a # Implicit relative import (deprecated, > py2 > only) > from . import a # Explicit relative import Can you show a complete example of what doesn't work with one or more of these? Because if I have: - empty package/__init__.py - package/a.py consisting of only "from . import b" - package/b.py consisting of only "from . import a" - in the directory containing 'package', running python 3.5 interactively, executing: >>> import package.a everything seems to work. From steve at pearwood.info Mon May 9 22:13:03 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 10 May 2016 12:13:03 +1000 Subject: json.loads(...) ValueError: Expecting value: line 1 column 1 (char 0) References: <137a2593-c307-405f-8f23-3959099ff16f@googlegroups.com> Message-ID: <573143b0$0$1593$c3e8da3$5496439d@news.astraweb.com> On Tue, 10 May 2016 09:07 am, Terry Reedy wrote: > Editorial: Programming classes should teach basic debugging better. I > have seen numerous newbie Stackoverflow questions where the person > should have started with adding a print statement before posting a > question. +10000 -- Steven From steve at pearwood.info Mon May 9 22:32:22 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 10 May 2016 12:32:22 +1000 Subject: String concatenation References: <3414323.Mt0gs2Yp5W@PointedEars.de> <2279805.ZaV9TF1YuJ@PointedEars.de> Message-ID: <57314838$0$1586$c3e8da3$5496439d@news.astraweb.com> On Tue, 10 May 2016 07:21 am, Thomas 'PointedEars' Lahn wrote: > Chris Angelico wrote: > >> On Mon, May 9, 2016 at 10:44 AM, Thomas 'PointedEars' Lahn >> wrote: >>> With the ?%? string operator (deprecated), str.format(), and >>> str.Template, you can use other values in string values even without >>> concatenation. >> >> Not deprecated. Don't spread FUD. > > If only you cared to read what I referred to: > > ,- That is excellent advice Thomas! If only you had followed it yourself, instead of blindly and mechanically copying and pasting. If you had bothered to actually read it, you would have seen that *nowhere* does it say that the % string operator is deprecated. Deprecation has a specific meaning involving a formal process of removing a feature from the language. It doesn't merely mean "this old feature has quirks and we think you should use this new feature instead". Some of the core developers like `format` better and think that people should use it in preference to the % string operator. That much is true. But that is far from being deprecated. > Note the key words: ?old?, ?quirks?, ?errors?. Irrelevant to the question of deprecation. Floats are old (they go back to the first release of Python), they have many quirks (x + y - x is not necessarily equal to y), and people make many errors with floats. Does this mean they are deprecated? Of course not. -- Steven From rosuav at gmail.com Mon May 9 22:42:36 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 10 May 2016 12:42:36 +1000 Subject: String concatenation In-Reply-To: <57314838$0$1586$c3e8da3$5496439d@news.astraweb.com> References: <3414323.Mt0gs2Yp5W@PointedEars.de> <2279805.ZaV9TF1YuJ@PointedEars.de> <57314838$0$1586$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, May 10, 2016 at 12:32 PM, Steven D'Aprano wrote: > Floats are old (they go back to the first release of Python), they have many > quirks (x + y - x is not necessarily equal to y), and people make many > errors with floats. Does this mean they are deprecated? Of course not. Careful there Steven - now that cdecimal is in core, people might start saying that. Particularly if (as is periodically requested, and which I think would be a good idea) Decimal literals become a thing. And the removal of core types HAS happened. Correct me if I'm wrong, but didn't the first release of Python have only the short integer type, and then a completely new 'long' type was added? Automatic promotion blurred the distinction, and then Python 3.0 removed the 'int' type and renamed 'long'. So it's theoretically possible for Decimal to replace float... ... except that that would actually be a bad idea, which a lot of people don't realize. ChrisA From rustompmody at gmail.com Mon May 9 23:45:56 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Mon, 9 May 2016 20:45:56 -0700 (PDT) Subject: String concatenation In-Reply-To: <2279805.ZaV9TF1YuJ@PointedEars.de> References: <3414323.Mt0gs2Yp5W@PointedEars.de> <2279805.ZaV9TF1YuJ@PointedEars.de> Message-ID: On Tuesday, May 10, 2016 at 2:52:13 AM UTC+5:30, Thomas 'PointedEars' Lahn wrote: > Chris Angelico wrote: > > > On Mon, May 9, 2016 at 10:44 AM, Thomas 'PointedEars' Lahn wrote: > >> Also, it would be a good idea if you posted under your real name. > >> Internet is the thing with cables; Usenet is the thing with people. > >> I for one tend to avoid communicating with few-letter entities; > >> exceptions to that would probably include only E.T., M.J., ALF, and > >> K.I.T.T. > > > > I'm not using Usenet, Mr PointedEars. > > I don't care. This basic courtesy extended to strangers whose help you seek > originates in real life, not Usenet. And it is objectionable to discriminate on people's names; the same way it is objectionable to discriminate on people's skin-color, religion, country, sex etc See: http://www.alphr.com/computing/1000378/facebook-rejects-native-american-names-as-fake-again From rustompmody at gmail.com Tue May 10 00:22:17 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Mon, 9 May 2016 21:22:17 -0700 (PDT) Subject: json.loads(...) ValueError: Expecting value: line 1 column 1 (char 0) In-Reply-To: <573143b0$0$1593$c3e8da3$5496439d@news.astraweb.com> References: <137a2593-c307-405f-8f23-3959099ff16f@googlegroups.com> <573143b0$0$1593$c3e8da3$5496439d@news.astraweb.com> Message-ID: <37add758-036c-4316-98a1-efb2f514dad1@googlegroups.com> On Tuesday, May 10, 2016 at 7:43:17 AM UTC+5:30, Steven D'Aprano wrote: > On Tue, 10 May 2016 09:07 am, Terry Reedy wrote: > > > Editorial: Programming classes should teach basic debugging better. I > > have seen numerous newbie Stackoverflow questions where the person > > should have started with adding a print statement before posting a > > question. > > +10000 Maybe the tutorial should have a small chapter on debugging (introspecting) showing how to use - print - type - dir And then iterating on these From harirammanohar at gmail.com Tue May 10 01:17:08 2016 From: harirammanohar at gmail.com (harirammanohar at gmail.com) Date: Mon, 9 May 2016 22:17:08 -0700 (PDT) Subject: pygame easy create In-Reply-To: <61a15673-a29e-4152-8bf5-ac1e82c68d3c@googlegroups.com> References: <1f66805c-3a0d-41be-ad35-1377ef25edfc@googlegroups.com> <2a7bf274-f1f8-4ab1-ad13-ed2dedaaad8d@googlegroups.com> <61a15673-a29e-4152-8bf5-ac1e82c68d3c@googlegroups.com> Message-ID: <4261534c-aab7-4a4e-8c7c-d693ae55d0be@googlegroups.com> On Monday, May 9, 2016 at 11:12:47 PM UTC+5:30, sohca... at gmail.com wrote: > On Monday, May 9, 2016 at 3:15:45 AM UTC-7, hariram... at gmail.com wrote: > > On Monday, May 9, 2016 at 10:50:47 AM UTC+5:30, hariram... at gmail.com wrote: > > > is there anyway (IDE/package) that allows me to create graphics/game just like that (by instructing..., if i say create hills on the screen, it should generate pygame code....)....Anyway :) :) > > > > Atleast i tried with pyglet,felt that it will be easier than pygame..... still again throwing me error some GL is missed, why they wont be straight forward in working if we select the env we have... > > > > some sites says 2.7 and >=3(3.4+) will have pip module by default, but i am using 3.4.3 where i dont have pip and also tried python get-pip.py which also not worked...... > > "still again throwing me error some GL is missed" > > Can you say *EXACTLY* what the error is? And can you copy/paste the relevant lines of code that lead to that error? > > Also, judging from your other messages, it looks like you might be needing to read the Python or PyGame tutorials. Programming is problem solving. You can't just saying "Draw a hill" and PyGame (or whatever module you're using) will draw a hill. You need to write code that describes how to draw a hill. code: #!/home/python/3.4.3/python import os,sys sys.path.append('/home/python/3.4.3/custommodules/lib/python3.4/site-packages') import pyglet window = pyglet.window.Window() label = pyglet.text.Label('Hello, world', font_name='Times New Roman', font_size=36, x=window.width//2, y=window.height//2, anchor_x='center', anchor_y='center') @window.event def on_draw(): window.clear() label.draw() pyglet.app.run() op: -bash-4.1$ ./pygletpgm.py Traceback (most recent call last): File "./pygletpgm.py", line 5, in import pyglet ImportError: No module named 'pyglet' I am able to successfully install pyglet module in custommodules... another issue: -bash-4.1$ python get-pip.py /tmp/tmpUdukeA/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. Collecting pip /tmp/tmpUdukeA/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. /tmp/tmpUdukeA/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. Could not find a version that satisfies the requirement pip (from versions: ) No matching distribution found for pip From harirammanohar at gmail.com Tue May 10 01:21:43 2016 From: harirammanohar at gmail.com (harirammanohar at gmail.com) Date: Mon, 9 May 2016 22:21:43 -0700 (PDT) Subject: pygame easy create In-Reply-To: <61a15673-a29e-4152-8bf5-ac1e82c68d3c@googlegroups.com> References: <1f66805c-3a0d-41be-ad35-1377ef25edfc@googlegroups.com> <2a7bf274-f1f8-4ab1-ad13-ed2dedaaad8d@googlegroups.com> <61a15673-a29e-4152-8bf5-ac1e82c68d3c@googlegroups.com> Message-ID: <479b6deb-1c52-4fd8-a214-edc3d6b2696e@googlegroups.com> On Monday, May 9, 2016 at 11:12:47 PM UTC+5:30, sohca... at gmail.com wrote: > On Monday, May 9, 2016 at 3:15:45 AM UTC-7, hariram... at gmail.com wrote: > > On Monday, May 9, 2016 at 10:50:47 AM UTC+5:30, hariram... at gmail.com wrote: > > > is there anyway (IDE/package) that allows me to create graphics/game just like that (by instructing..., if i say create hills on the screen, it should generate pygame code....)....Anyway :) :) > > > > Atleast i tried with pyglet,felt that it will be easier than pygame..... still again throwing me error some GL is missed, why they wont be straight forward in working if we select the env we have... > > > > some sites says 2.7 and >=3(3.4+) will have pip module by default, but i am using 3.4.3 where i dont have pip and also tried python get-pip.py which also not worked...... > > "still again throwing me error some GL is missed" > > Can you say *EXACTLY* what the error is? And can you copy/paste the relevant lines of code that lead to that error? > > Also, judging from your other messages, it looks like you might be needing to read the Python or PyGame tutorials. Programming is problem solving. You can't just saying "Draw a hill" and PyGame (or whatever module you're using) will draw a hill. You need to write code that describes how to draw a hill. yeah coding is a problem solving as you said, its right but i have asked any utility is there for easy design in the same fashion.... okay can you answser my question, using python modules (pygame/pyglet whatever it may be) can we design games with higher graphics (like world of tanks, freedom fighter, mission impossible:rouge nation, contract killer etc....) From gengyangcai at gmail.com Tue May 10 02:13:30 2016 From: gengyangcai at gmail.com (Cai Gengyang) Date: Mon, 9 May 2016 23:13:30 -0700 (PDT) Subject: An educational site written in Python (from YCombinator's RFS) Message-ID: Ok, so after reading YCombinator's RFS, I have decided that I want to work on this : ------------------------------------------------------------------------------------------------------------------- EDUCATION If we can fix education, we can eventually do everything else on this list. The first attempts to use technology to fix education have focused on using the Internet to distribute traditional content to a wider audience. This is good, but the Internet is a fundamentally different medium and capable of much more. Solutions that combine the mass scale of technology with one-on-one in-person interaction are particularly interesting to us. This may not require a "breakthrough" technology in the classical sense, but at a minimum it will require very new ways of doing things. ------------------------------------------------------------------------------------------------------------------- I want to create such a site using Python. What are the various steps I need to take to create such a site ? This is a big project, but one that is worth doing ... Any suggestions / help appreciated ? Thanks alot Gengyang From steve+comp.lang.python at pearwood.info Tue May 10 02:46:07 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 10 May 2016 16:46:07 +1000 Subject: String concatenation References: <3414323.Mt0gs2Yp5W@PointedEars.de> <2279805.ZaV9TF1YuJ@PointedEars.de> Message-ID: <573183b1$0$2741$c3e8da3$76491128@news.astraweb.com> On Tuesday 10 May 2016 13:45, Rustom Mody wrote: > See: > http://www.alphr.com/computing/1000378/facebook-rejects-native-american- names-as-fake-again Somebody should set up a kick-starter to pay someone to change their legal name to "Facebook-Are-Arseholes", then open a Facebook account with it. I'd contribute a couple of bucks. ("That's MISTER Facebook-Are-Arseholes to you!") Unfortunately, Australia's naming laws almost certainly wouldn't allow such a name. But I think the US would. -- Steve From joel.goldstick at gmail.com Tue May 10 03:00:18 2016 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Tue, 10 May 2016 03:00:18 -0400 Subject: An educational site written in Python (from YCombinator's RFS) In-Reply-To: References: Message-ID: On Tue, May 10, 2016 at 2:13 AM, Cai Gengyang wrote: > Ok, so after reading YCombinator's RFS, I have decided that I want to work on this : > > > ------------------------------------------------------------------------------------------------------------------- > > EDUCATION > > If we can fix education, we can eventually do everything else on this list. > The first attempts to use technology to fix education have focused on using the Internet to distribute traditional content to a wider audience. This is good, but the Internet is a fundamentally different medium and capable of much more. > > Solutions that combine the mass scale of technology with one-on-one in-person interaction are particularly interesting to us. > > This may not require a "breakthrough" technology in the classical sense, but at a minimum it will require very new ways of doing things. > > ------------------------------------------------------------------------------------------------------------------- > > I want to create such a site using Python. What are the various steps I need to take to create such a site ? This is a big project, but one that is worth doing ... Any suggestions / help appreciated ? Thanks alot > > > Gengyang > -- > https://mail.python.org/mailman/listinfo/python-list I'm not sure there is a question here. Good luck with your quest -- Joel Goldstick http://joelgoldstick.com/blog http://cc-baseballstats.info/stats/birthdays From no.email at nospam.invalid Tue May 10 03:13:51 2016 From: no.email at nospam.invalid (Paul Rubin) Date: Tue, 10 May 2016 00:13:51 -0700 Subject: String concatenation References: <3414323.Mt0gs2Yp5W@PointedEars.de> <2279805.ZaV9TF1YuJ@PointedEars.de> <573183b1$0$2741$c3e8da3$76491128@news.astraweb.com> Message-ID: <87twi675cg.fsf@jester.gateway.pace.com> Steven D'Aprano writes: > Australia's naming laws almost certainly wouldn't allow such a name. https://en.wikipedia.org/wiki/Facebook_real-name_policy_controversy#Vietnamese From steve+comp.lang.python at pearwood.info Tue May 10 03:54:07 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 10 May 2016 17:54:07 +1000 Subject: String concatenation References: <3414323.Mt0gs2Yp5W@PointedEars.de> <2279805.ZaV9TF1YuJ@PointedEars.de> <573183b1$0$2741$c3e8da3$76491128@news.astraweb.com> <87twi675cg.fsf@jester.gateway.pace.com> Message-ID: <573193a0$0$11094$c3e8da3@news.astraweb.com> On Tuesday 10 May 2016 17:13, Paul Rubin wrote: > Steven D'Aprano writes: >> Australia's naming laws almost certainly wouldn't allow such a name. > > https://en.wikipedia.org/wiki/Facebook_real- name_policy_controversy#Vietnamese "Phuc Dat Bich" was a hoax, but it probably would be allowed in Australia. I'm surprised that Spanish names are not affected. Consider a woman who goes by the personal name of Maria Teresa, whose father's first surname was Garc?a and mother's first surname was Ram?rez. Her name would therefore be Maria Teresa Garc?a Ram?rez. If she marries El? Arroyo L?pez, then she might change her name to Maria Teresa Garc?a Ram?rez de Arroyo. With six words, that would fall foul of Facebook's foul naming policy. http://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/ -- Steve From dpalao.python at gmail.com Tue May 10 04:15:51 2016 From: dpalao.python at gmail.com (David Palao) Date: Tue, 10 May 2016 10:15:51 +0200 Subject: String concatenation In-Reply-To: <573193a0$0$11094$c3e8da3@news.astraweb.com> References: <3414323.Mt0gs2Yp5W@PointedEars.de> <2279805.ZaV9TF1YuJ@PointedEars.de> <573183b1$0$2741$c3e8da3$76491128@news.astraweb.com> <87twi675cg.fsf@jester.gateway.pace.com> <573193a0$0$11094$c3e8da3@news.astraweb.com> Message-ID: 2016-05-10 9:54 GMT+02:00 Steven D'Aprano : > On Tuesday 10 May 2016 17:13, Paul Rubin wrote: > >> Steven D'Aprano writes: >>> Australia's naming laws almost certainly wouldn't allow such a name. >> >> https://en.wikipedia.org/wiki/Facebook_real- > name_policy_controversy#Vietnamese > > "Phuc Dat Bich" was a hoax, but it probably would be allowed in Australia. > > I'm surprised that Spanish names are not affected. Consider a woman who goes by > the personal name of Maria Teresa, whose father's first surname was Garc?a and > mother's first surname was Ram?rez. Her name would therefore be Maria Teresa > Garc?a Ram?rez. If she marries El? Arroyo L?pez, then she might change her name > to Maria Teresa Garc?a Ram?rez de Arroyo. With six words, that would fall foul > of Facebook's foul naming policy. > > > http://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/ > > > -- > Steve > > -- > https://mail.python.org/mailman/listinfo/python-list In Spain "de Arroyo" officially does not become part of the name. The same applies to other countries as well. Not 100% sure that it is true in every Spanish speaking country though. Best From georgieelize00 at gmail.com Tue May 10 05:01:45 2016 From: georgieelize00 at gmail.com (George Molsom) Date: Tue, 10 May 2016 02:01:45 -0700 (PDT) Subject: TypeError: unorderable types: function() < int() Message-ID: <1fc32599-0264-460c-8178-057558d19be5@googlegroups.com> I have created a program in class 'make a game that tests how good people are at guessing when 10 seconds has elapsed.' The following are the code I currently have and the error produced when I attempt to run it. I have tried everything I can think of to resolve the issue, and I have also run the code through a checker, which has said that there are no mistakes. I have also shown a friend who is a programmer, and he cannot find a problem with it. The teacher doesn't actually know the solution to the problem so I was wondering if someone could point me in the right direction to get this working please? import time def second(int): time.strftime("%S") start = input('Press enter when you are ready to start') time1 = time.strftime("%S") then = time.time() end = input('Press enter when you think 10 seconds has passed') time2 = time.strftime("%S") def totaltime(int): (time2-time1) if totaltime == '10': print ('YOU ACTUALLY DID IT') if totaltime < 10: print ('Haha you took too long! Your result was:', totaltime,'seconds') if totaltime > 10: print('Too early TRY AGAIN! Your result was:', totaltime, 'seconds') Press enter when you are ready to start Press enter when you think 10 seconds has passed Traceback (most recent call last): File "E:/Computing/Python/Little book of challenges/Challenge 6 10 seconds experiment.py", line 20, in if totaltime < 10: TypeError: unorderable types: function() < int() From ben+python at benfinney.id.au Tue May 10 05:16:48 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 10 May 2016 19:16:48 +1000 Subject: TypeError: unorderable types: function() < int() References: <1fc32599-0264-460c-8178-057558d19be5@googlegroups.com> Message-ID: <858tziff27.fsf@benfinney.id.au> George Molsom writes: > I have created a program in class 'make a game that tests how good > people are at guessing when 10 seconds has elapsed.' Welcome! You may want to join the dedicated beginners forum where we collaboratively teach foundational Python concepts. > Traceback (most recent call last): > File "E:/Computing/Python/Little book of challenges/Challenge 6 10 seconds experiment.py", line 20, in > if totaltime < 10: > TypeError: unorderable types: function() < int() That's right. The ?totaltime? name refers to a function. To ask whether an integer object is less than a function object is not a meaningful comparison. If you want to *call* the function, use the ?call this function? syntax:: totaltime() That will evaluate to the return value when you call it, so use the return value:: if totaltime() < 10: # ? -- \ ?There's a certain part of the contented majority who love | `\ anybody who is worth a billion dollars.? ?John Kenneth | _o__) Galbraith, 1992-05-23 | Ben Finney From info at egenix.com Tue May 10 05:38:07 2016 From: info at egenix.com (eGenix Team: M.-A. Lemburg) Date: Tue, 10 May 2016 11:38:07 +0200 Subject: ANN: eGenix pyOpenSSL Distribution 0.13.15 Message-ID: <5731ABFF.2080406@egenix.com> ________________________________________________________________________ ANNOUNCING eGenix.com pyOpenSSL Distribution Version 0.13.15 An easy-to-install and easy-to-use distribution of the pyOpenSSL Python interface for OpenSSL - available for Windows, Mac OS X and Unix platforms This announcement is also available on our web-site for online reading: http://www.egenix.com/company/news/eGenix-pyOpenSSL-Distribution-0.13.15.html ________________________________________________________________________ INTRODUCTION The eGenix.com pyOpenSSL Distribution includes everything you need to get started with SSL in Python. It comes with an easy-to-use installer that includes the most recent OpenSSL library versions in pre-compiled form, making your application independent of OS provided OpenSSL libraries: http://www.egenix.com/products/python/pyOpenSSL/ pyOpenSSL is an open-source Python add-on that allows writing SSL/TLS- aware network applications as well as certificate management tools: https://launchpad.net/pyopenssl/ OpenSSL is an open-source implementation of the SSL/TLS protocol: http://www.openssl.org/ ________________________________________________________________________ NEWS This new release of the eGenix.com pyOpenSSL Distribution includes the following updates: New in OpenSSL -------------- * Switched the included OpenSSL libraries to 1.0.2h. The OpenSSL 1.0.2 branch will receive long term support (LTS), so is an ideal basis for development. See https://openssl.org/news/secadv/20160503.txt ?for a complete list of security fixes in 1.0.2h. The following fixes are relevant for pyOpenSSL applications: - CVE-2016-2108 Two bugs in the ASN.1 parser were found which could cause memory corruption issues. - CVE-2016-2107 A MITM attacker can use a padding oracle attack to decrypt traffic when the connection uses an AES CBC cipher and the server support AES-NI. This issue was introduced as part of the fix for Lucky 13 padding attack (CVE-2013-0169). - Several low priority issues were fixed as well. * Updated the Mozilla CA root bundle to version 2016-04-16. Please see the product changelog for the full set of changes. http://www.egenix.com/products/python/pyOpenSSL/changelog.html pyOpenSSL / OpenSSL Binaries Included ------------------------------------- In addition to providing sources, we make binaries available that include both pyOpenSSL and the necessary OpenSSL libraries for all supported platforms: Windows, Linux, Mac OS X and FreeBSD, for x86 and x64. To simplify installation, we have uploaded a web installer to PyPI which will automatically choose the right binary for your platform, so a simple pip install egenix-pyopenssl will get you the package with OpenSSL libraries installed. Please see our installation instructions for details: http://www.egenix.com/products/python/pyOpenSSL/#Installation We have also added .egg-file distribution versions of our eGenix.com pyOpenSSL Distribution for Windows, Linux and Mac OS X to the available download options. These make setups using e.g. zc.buildout and other egg-file based installers a lot easier. ________________________________________________________________________ DOWNLOADS The download archives and instructions for installing the package can be found at: http://www.egenix.com/products/python/pyOpenSSL/ ________________________________________________________________________ UPGRADING Before installing this version of pyOpenSSL, please make sure that you uninstall any previously installed pyOpenSSL version. Otherwise, you could end up not using the included OpenSSL libs. _______________________________________________________________________ SUPPORT Commercial support for these packages is available from eGenix.com. Please see http://www.egenix.com/services/support/ for details about our support offerings. ________________________________________________________________________ MORE INFORMATION For more information about the eGenix pyOpenSSL Distribution, licensing and download instructions, please visit our web-site or write to sales at egenix.com. About eGenix (http://www.egenix.com/): eGenix is a Python software project, consulting and product company delivering expert services and professional quality products for companies, Python users and developers. We specialize in database driven applications, large scale software designs and integration. Enjoy, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 10 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From harirammanohar at gmail.com Tue May 10 07:37:35 2016 From: harirammanohar at gmail.com (harirammanohar at gmail.com) Date: Tue, 10 May 2016 04:37:35 -0700 (PDT) Subject: pygame easy create In-Reply-To: <479b6deb-1c52-4fd8-a214-edc3d6b2696e@googlegroups.com> References: <1f66805c-3a0d-41be-ad35-1377ef25edfc@googlegroups.com> <2a7bf274-f1f8-4ab1-ad13-ed2dedaaad8d@googlegroups.com> <61a15673-a29e-4152-8bf5-ac1e82c68d3c@googlegroups.com> <479b6deb-1c52-4fd8-a214-edc3d6b2696e@googlegroups.com> Message-ID: <5fc27e74-bfdb-4dfe-9690-da7672841ff7@googlegroups.com> On Tuesday, May 10, 2016 at 10:51:58 AM UTC+5:30, hariram... at gmail.com wrote: > On Monday, May 9, 2016 at 11:12:47 PM UTC+5:30, sohca... at gmail.com wrote: > > On Monday, May 9, 2016 at 3:15:45 AM UTC-7, hariram... at gmail.com wrote: > > > On Monday, May 9, 2016 at 10:50:47 AM UTC+5:30, hariram... at gmail.com wrote: > > > > is there anyway (IDE/package) that allows me to create graphics/game just like that (by instructing..., if i say create hills on the screen, it should generate pygame code....)....Anyway :) :) > > > > > > Atleast i tried with pyglet,felt that it will be easier than pygame..... still again throwing me error some GL is missed, why they wont be straight forward in working if we select the env we have... > > > > > > some sites says 2.7 and >=3(3.4+) will have pip module by default, but i am using 3.4.3 where i dont have pip and also tried python get-pip.py which also not worked...... > > > > "still again throwing me error some GL is missed" > > > > Can you say *EXACTLY* what the error is? And can you copy/paste the relevant lines of code that lead to that error? > > > > Also, judging from your other messages, it looks like you might be needing to read the Python or PyGame tutorials. Programming is problem solving. You can't just saying "Draw a hill" and PyGame (or whatever module you're using) will draw a hill. You need to write code that describes how to draw a hill. > > yeah coding is a problem solving as you said, its right but i have asked any utility is there for easy design in the same fashion.... > > okay can you answser my question, using python modules (pygame/pyglet whatever it may be) can we design games with higher graphics (like world of tanks, freedom fighter, mission impossible:rouge nation, contract killer etc....) Traceback (most recent call last): File "./pyg.py", line 6, in window = pyglet.window.Window() File "_build/bdist.linux-x86_64/egg/pyglet/__init__.py", line 357, in __getattr__ File "_build/bdist.linux-x86_64/egg/pyglet/window/__init__.py", line 133, in File "_build/bdist.linux-x86_64/egg/pyglet/gl/__init__.py", line 97, in File "_build/bdist.linux-x86_64/egg/pyglet/gl/lib.py", line 142, in File "_build/bdist.linux-x86_64/egg/pyglet/gl/lib_glx.py", line 50, in File "_build/bdist.linux-x86_64/egg/pyglet/lib.py", line 160, in load_library ImportError: Library "GL" not found. From rosuav at gmail.com Tue May 10 08:31:00 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 10 May 2016 22:31:00 +1000 Subject: TypeError: unorderable types: function() < int() In-Reply-To: References: <1fc32599-0264-460c-8178-057558d19be5@googlegroups.com> Message-ID: On Tue, May 10, 2016 at 10:19 PM, Dennis Lee Bieber wrote: > You invoke strftime() [in default: use current time mode], but pass it > a format that is not defined in the documentation (or wasn't in Python 2.7 > which I'm still running). #2 > %s (lowercase) formats the seconds field of the time of day. And worst, > since you don't return the result, it just gets thrown away. #3 > Sorry to nitpick, but this isn't the case. time.strftime("%S") is what you're talking about (getting the "seconds since beginning of current minute"). There is a "%s" format string, but it's not supported on all platforms; it actually would be more useful here, as it represents the time as seconds since 1970 (aka "Unix time"). Of course, time.time() would still be far more useful here, as you subsequently recommended. ChrisA From steve at pearwood.info Tue May 10 09:40:25 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 10 May 2016 23:40:25 +1000 Subject: TypeError: unorderable types: function() < int() References: <1fc32599-0264-460c-8178-057558d19be5@googlegroups.com> Message-ID: <5731e4ca$0$1603$c3e8da3$5496439d@news.astraweb.com> Hello George, and welcome! On Tue, 10 May 2016 07:01 pm, George Molsom wrote: > I have created a program in class 'make a game that tests how good people > are at guessing when 10 seconds has elapsed.' > > The following are the code I currently have and the error produced when I > attempt to run it. I have tried everything I can think of to resolve the > issue, and I have also run the code through a checker, which has said that > there are no mistakes. I have also shown a friend who is a programmer, and > he cannot find a problem with it. The teacher doesn't actually know the > solution to the problem so I was wondering if someone could point me in > the right direction to get this working please? Neither your friend who is a programmer nor the teacher can read an error message? > TypeError: unorderable types: function() < int() In fairness, it is a bit of a rubbish error message. But what is it saying it that you are trying to check whether a function is less than a number. You are not comparing the *result* of calling the function with the number, but the function itself. For example, suppose we have a function that (to keep it simple) always returns 3. def func(): return 3 Now you go to check whether it is less than some other number: if func() < 9: print("smaller") That's what you *meant* to do, but unfortunately you left off the round brackets (parentheses), which means that instead of calling the function and then comparing the result of that with 9, you compare the FUNCTION itself with 9. if func < 9: ... Obviously this is nonsense. You can't compare functions with integers, the question is ludicrous. So Python rightly complains with an error message. The immediate fix is to add the brackets in so that you call the function first to get a result, and then compare the result with the number. Another comment about your game: You are asking the player to hit enter at 10 seconds. *Exactly* ten seconds. The computer can measure time to well under a millionth of a second, I'm pretty sure that nobody, no matter how skillful, can be expected to hit enter after exactly 10.000000 seconds, not 10.000001 or 9.999999 seconds. I think it is reasonable to give them a little bit of slack to be above or under 10 seconds by a fraction of a second. So you should round the time to one decimal place: totaltime = round(totaltime, 1) before checking whether it is equal to 10. -- Steven From rosuav at gmail.com Tue May 10 09:47:46 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 10 May 2016 23:47:46 +1000 Subject: TypeError: unorderable types: function() < int() In-Reply-To: <5731e4ca$0$1603$c3e8da3$5496439d@news.astraweb.com> References: <1fc32599-0264-460c-8178-057558d19be5@googlegroups.com> <5731e4ca$0$1603$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, May 10, 2016 at 11:40 PM, Steven D'Aprano wrote: > You are asking the player to hit enter at 10 seconds. *Exactly* ten seconds. > The computer can measure time to well under a millionth of a second, I'm > pretty sure that nobody, no matter how skillful, can be expected to hit > enter after exactly 10.000000 seconds, not 10.000001 or 9.999999 seconds. > But the times were being calculated at one-second resolution, which solves that but raises its own issues. ChrisA From pcmanticore at gmail.com Tue May 10 09:51:28 2016 From: pcmanticore at gmail.com (Claudiu Popa) Date: Tue, 10 May 2016 06:51:28 -0700 (PDT) Subject: Pylint prefers list comprehension over filter... In-Reply-To: References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> Message-ID: On Sunday, May 8, 2016 at 4:41:02 AM UTC+3, Terry Reedy wrote: > On 5/7/2016 3:17 PM, Christopher Reimer wrote: > > > For my purposes, I'm using the list comprehension over filter to keep > > pylint happy. > > How sad. The pylint developers arrogantly take it on themselves to > revise Python, against the wishes of Guido and the other core > developers, and you and feel obligated to follow them. > > They should at least add a disclaimer "Using the default options, pylint > checks that your code complies with the pylint-approved subset of Python." > > -- > Terry Jan Reedy Hi, Thank you for letting us know. While pylint is indeed opinionated in some cases, we're not trying to be "arrogant", as you put it, towards Guido or the other core developers. What's sad in this particular case is that the feedback had to come in rather a harsh manner, on this group, instead of being reported as a bug or an enhancement on pylint's bug tracker. Anyway, I wanted to tell you that I agree with your opinion regarding that message and as such, it is removed and won't be emitted anymore in the next release (1.6) Thank you. From steve at pearwood.info Tue May 10 09:55:24 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 10 May 2016 23:55:24 +1000 Subject: TypeError: unorderable types: function() < int() References: <1fc32599-0264-460c-8178-057558d19be5@googlegroups.com> <5731e4ca$0$1603$c3e8da3$5496439d@news.astraweb.com> Message-ID: <5731e84d$0$1615$c3e8da3$5496439d@news.astraweb.com> On Tue, 10 May 2016 11:47 pm, Chris Angelico wrote: > On Tue, May 10, 2016 at 11:40 PM, Steven D'Aprano > wrote: >> You are asking the player to hit enter at 10 seconds. *Exactly* ten >> seconds. The computer can measure time to well under a millionth of a >> second, I'm pretty sure that nobody, no matter how skillful, can be >> expected to hit enter after exactly 10.000000 seconds, not 10.000001 or >> 9.999999 seconds. >> > > But the times were being calculated at one-second resolution, which > solves that but raises its own issues. Ah, I didn't realise that time.strftime("%S") was one-second resolution (obvious in hindsight, duh!) and the call to time.time() is apparently unused. -- Steven From steve at pearwood.info Tue May 10 10:58:22 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 11 May 2016 00:58:22 +1000 Subject: Pylint prefers list comprehension over filter... References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> Message-ID: <5731f70f$0$1590$c3e8da3$5496439d@news.astraweb.com> Hi Claudiu, On Tue, 10 May 2016 11:51 pm, Claudiu Popa wrote: > Thank you for letting us know. While pylint is indeed > opinionated in some cases, we're not trying to be > "arrogant", as you put it, towards Guido or the other core > developers. What's sad in this particular case is that the > feedback had to come in rather a harsh manner, on this group, > instead of being reported as a bug or an enhancement on pylint's > bug tracker. > > Anyway, I wanted to tell you that I agree with your opinion > regarding that message and as such, it is removed and won't be > emitted anymore in the next release (1.6) Thanks for being so understanding! Speaking for myself, I think that the test for filter versus list comprehensions is a reasonable test to include, so long as it is disabled by default. Does PyLint support opt-in checks? -- Steven From steve at pearwood.info Tue May 10 11:20:03 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 11 May 2016 01:20:03 +1000 Subject: Pylint prefers list comprehension over filter... References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> Message-ID: <5731fc24$0$1599$c3e8da3$5496439d@news.astraweb.com> A further comment: On Tue, 10 May 2016 11:51 pm, Claudiu Popa wrote: > Thank you for letting us know. While pylint is indeed > opinionated in some cases, we're not trying to be > "arrogant", And from the docs: "What Pylint says is not to be taken as gospel and Pylint isn?t smarter than you are: it may warn you about things that you have conscientiously done." https://docs.pylint.org/intro.html#what-pylint-is-not which has some nice advice for how to deal with pylint so the warnings are not overwhelming. I think it is time for me to check it out... -- Steven From pcmanticore at gmail.com Tue May 10 11:35:47 2016 From: pcmanticore at gmail.com (pcmanticore at gmail.com) Date: Tue, 10 May 2016 08:35:47 -0700 (PDT) Subject: Pylint prefers list comprehension over filter... In-Reply-To: <5731f70f$0$1590$c3e8da3$5496439d@news.astraweb.com> References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> <5731f70f$0$1590$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tuesday, May 10, 2016 at 5:58:37 PM UTC+3, Steven D'Aprano wrote: > Hi Claudiu, > > > On Tue, 10 May 2016 11:51 pm, Claudiu Popa wrote: > > > Thank you for letting us know. While pylint is indeed > > opinionated in some cases, we're not trying to be > > "arrogant", as you put it, towards Guido or the other core > > developers. What's sad in this particular case is that the > > feedback had to come in rather a harsh manner, on this group, > > instead of being reported as a bug or an enhancement on pylint's > > bug tracker. > > > > Anyway, I wanted to tell you that I agree with your opinion > > regarding that message and as such, it is removed and won't be > > emitted anymore in the next release (1.6) > > Thanks for being so understanding! > > Speaking for myself, I think that the test for filter versus list > comprehensions is a reasonable test to include, so long as it is disabled > by default. Does PyLint support opt-in checks? > > > > -- > Steven Hi Steven, Yes, pylint is pretty configurable regarding its checks. First of all, we have extensions, which are checks which were found unsuitable to be part of the core, thus they need to be manually activated, as in: $ pylint --load-plugins=pylint.extensions.name_of_the_extension We also have a group of checks that are disabled by default, as is the case of the Python 3 porting checker, recommended as well by the Python 3 porting guide (https://docs.python.org/3/howto/pyporting.html?highlight=pylint) These needs to be enabled manually, as in: $ pylint --enable=python3 $ pylint --enable=other_disabled_group Last, but not least, each check can be (de)activated manually: $ pylint --disable=no-member # don't care about conventions and refactoring warnings $ pylint --disable=C,R $ pylint --disable=all --enable=no-member,other_check_I_want The bad-builtin check is now an extension, so using the first case would enable it. Another thing that is going to change with the next release is the introduction of tiers. Basically, pylint overwhelms the user right now with its enabled checks and we're trying to split these into tiers, as seen in the following: $ pylint myproject # core checkers enabled 10/10 - Congrats, you're clean on a core. You might try with "--pedantic" now. $ pylint myproject --pedantic # pedantic checkers enabled. 10/10 - Congrats, you're clean on pedantic. You might try with --refactoring now $ pylint myproject --refactoring # refactoring checkers enabled 10/10 - Congrats, you're clean on refactoring. Last up, try with --style now. $ pylint myproject --style 10/10 - Now you're pylint clean. Or running them all: $ pylint myproject --all (The discussion and whatnot can be found here https://github.com/PyCQA/pylint/issues/746) From steve at pearwood.info Tue May 10 11:45:29 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 11 May 2016 01:45:29 +1000 Subject: What should a decorator do if an attribute already exists? Message-ID: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> I have a decorator that adds an attribute to the decorated function: def decorate(func): instrument = make_instrument() @functools.wraps(func) def inner(*args): instrument.start() result = func(*args) instrument.finish() return result inner.instrument = instrument return inner The actual nature of the instrumentation isn't important: depending on the decorator, it might count the number of function calls made, how long it takes, count cache hits, or something else. My question is, what should I do if the decorated function already has an instrument attribute? 1. raise an exception? 2. raise a warning, and over-write the attribute? 3. raise a warning, and skip adding the attribute? 4. raise a warning, and rename the existing instrument to something else before writing my own instrument? 5. silently over-write the attribute? I think 5 is clearly wrong, 4 is too difficult, and 3 seems pointless. So I think either 1 or 2 is the right thing to do. Thoughts? -- Steven From me+python at ixokai.io Tue May 10 12:06:17 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Tue, 10 May 2016 09:06:17 -0700 Subject: What should a decorator do if an attribute already exists? In-Reply-To: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> References: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1462896377.1463523.603681369.3BBAEA61@webmail.messagingengine.com> On Tue, May 10, 2016, at 08:45 AM, Steven D'Aprano wrote: > I have a decorator that adds an attribute to the decorated function: > [...] > My question is, what should I do if the decorated function already has an > instrument attribute? > > 1. raise an exception? This. Your decorator should, IMHO, treat the attribute as private data, and if something else is using the same thing, something has clearly gone wrong and raising the error early and clearly is right. -- Stephen Hansen m e @ i x o k a i . i o From nospam at dfs.com Tue May 10 12:16:19 2016 From: nospam at dfs.com (DFS) Date: Tue, 10 May 2016 12:16:19 -0400 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5/9/2016 3:53 AM, Steven D'Aprano wrote: > On Monday 09 May 2016 09:10, DFS wrote: > >> sSQL = "line 1\n" >> sSQL += "line 2\n" >> sSQL += "line 3" > > Pointlessly provocative subject line edited. huh? You called yourself a "master crafts/wo/man". ---------------------------------------------------------------------- DFS: "python is awesome, but too many options for doing the same thing also makes it difficult. For me, anyway." DuhPricko: "That's the difference between a master and an apprentice. The apprentice likes to follow fixed steps the same way each time. The master craftsman knows her tools backwards, and can choose the right tool for the job, and when the choice of tool really doesn't matter and you can use whatever happens to be the closest to hand." ---------------------------------------------------------------------- > Since all three lines are constants know by the programmer at the time the > source code is written, it should be written as: > > > sSQL = """line 1 > line 2 > line 3""" > > Or if you prefer: > > sSQL = "line 1\nline 2\nline 3" > > > Or even: > > sSQL = ("line 1\n" > "line 2\n" > "line 3") > > taking advantage of Python's compile-time implicit concatenation of string > constants. or sSQL = "line 1\n" \ "line 2\n" \ "line 3" But no, "master", your answers are incorrect. What's wrong with that concat statement is that += concatenation is frowned upon by python's creator, and is not recommended (in PEP8): Code should be written in a way that does not disadvantage other implementations of Python (PyPy, Jython, IronPython, Cython, Psyco, and such). For example, do not rely on CPython's efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b . This optimization is fragile even in CPython (it only works for some types) and isn't present at all in implementations that don't use refcounting. In performance sensitive parts of the library, the ''.join() form should be used instead. This will ensure that concatenation occurs in linear time across various implementations. https://www.python.org/dev/peps/pep-0008/#programming-recommendations 'master craftswoman' my ass... From rustompmody at gmail.com Tue May 10 12:41:18 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Tue, 10 May 2016 09:41:18 -0700 (PDT) Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87f10da7-11a9-49fa-8466-6a7a97651130@googlegroups.com> On Tuesday, May 10, 2016 at 9:46:45 PM UTC+5:30, DFS wrote: > On 5/9/2016 3:53 AM, Steven D'Aprano wrote: > > On Monday 09 May 2016 09:10, DFS wrote: > > > >> sSQL = "line 1\n" > >> sSQL += "line 2\n" > >> sSQL += "line 3" > > > > Pointlessly provocative subject line edited. > > > huh? You called yourself a "master crafts/wo/man". > > > DuhPricko: > > > But no, "master", your answers are incorrect. What's wrong with that > 'master craftswoman' my ass... You do a good job This your teacher? https://www.youtube.com/watch?v=FMEe7JqBgvg From me+python at ixokai.io Tue May 10 12:42:58 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Tue, 10 May 2016 09:42:58 -0700 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1462898578.1471033.603720241.686524F7@webmail.messagingengine.com> On Tue, May 10, 2016, at 09:16 AM, DFS wrote: > But no, "master", your answers are incorrect. What's wrong with that > concat statement is that += concatenation is frowned upon by python's > creator, and is not recommended (in PEP8): > > > Code should be written in a way that does not disadvantage other > implementations of Python (PyPy, Jython, IronPython, Cython, Psyco, and > such). > > For example, do not rely on CPython's efficient implementation of > in-place string concatenation for statements in the form a += b or a = a > + b . This optimization is fragile even in CPython (it only works for > some types) and isn't present at all in implementations that don't use > refcounting. In performance sensitive parts of the library, the > ''.join() form should be used instead. This will ensure that > concatenation occurs in linear time across various implementations. > You once again misread PEP8. Not one of Steven's answers used string concatenation, except for the implicit literal concatenation, which all other implementations support. Note: > sSQL = """line 1 > line 2 > line 3""" No concatenation. One string literal. Works in all implementations. > sSQL = "line 1\nline 2\nline 3" No concatenation. One string literal. Works in all implementations. > > sSQL = ("line 1\n" > "line 2\n" > "line 3") Concatenation occurs at compile time, implicitly. Works in all implementations. The PEP says when building a string dynamically (that is, adding together two strings that exist and are separate), use "".join() in performance sensitive places. It says don't combine two strings with the addition operator -- because while that's sometimes efficient in CPython, the language doesn't guarantee it. You'll notice that the one time Steven combined two strings, it was implicitly at compile time. In every Python implementation, it is guaranteed that: >>> a = "a" "a" >>> b = "aa" Are the same. Two+ string literals are implicitly combined into one at compile time. > 'master craftswoman' my ass... Yes, you're being that. Please stop. -- Stephen Hansen m e @ i x o k a i . i o From steve at pearwood.info Tue May 10 12:44:04 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 11 May 2016 02:44:04 +1000 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> Message-ID: <57320fd5$0$1610$c3e8da3$5496439d@news.astraweb.com> On Wed, 11 May 2016 02:16 am, DFS wrote: > huh? You called yourself a "master crafts/wo/man". [...] > DuhPricko *plonk* -- Steven From __peter__ at web.de Tue May 10 12:46:02 2016 From: __peter__ at web.de (Peter Otten) Date: Tue, 10 May 2016 18:46:02 +0200 Subject: What should a decorator do if an attribute already exists? References: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: Steven D'Aprano wrote: > I have a decorator that adds an attribute to the decorated function: > > > def decorate(func): > instrument = make_instrument() > > @functools.wraps(func) > def inner(*args): > instrument.start() > result = func(*args) > instrument.finish() > return result > > inner.instrument = instrument > return inner > > > The actual nature of the instrumentation isn't important: depending on the > decorator, it might count the number of function calls made, how long it > takes, count cache hits, or something else. > > My question is, what should I do if the decorated function already has an > instrument attribute? > > 1. raise an exception? > > 2. raise a warning, and over-write the attribute? > 3. raise a warning, and skip adding the attribute? > 4. raise a warning, and rename the existing instrument to > something else before writing my own instrument? > > 5. silently over-write the attribute? > > > I think 5 is clearly wrong, 4 is too difficult, and 3 seems pointless. So > I think either 1 or 2 is the right thing to do. > > Thoughts? Silently overwrite it. For @decorate @decorate def f(): pass the original instrument is still accessible as f.__wrapped__.instrument, so 5 is the easy way to get 4. From python at mrabarnett.plus.com Tue May 10 13:42:45 2016 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 10 May 2016 18:42:45 +0100 Subject: What should a decorator do if an attribute already exists? In-Reply-To: <1462896377.1463523.603681369.3BBAEA61@webmail.messagingengine.com> References: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> <1462896377.1463523.603681369.3BBAEA61@webmail.messagingengine.com> Message-ID: <1292a35a-2ed4-23cb-cf6b-fe918cd24099@mrabarnett.plus.com> On 2016-05-10 17:06, Stephen Hansen wrote: > On Tue, May 10, 2016, at 08:45 AM, Steven D'Aprano wrote: >> I have a decorator that adds an attribute to the decorated function: >> [...] >> My question is, what should I do if the decorated function already has an >> instrument attribute? >> >> 1. raise an exception? > > This. Your decorator should, IMHO, treat the attribute as private data, > and if something else is using the same thing, something has clearly > gone wrong and raising the error early and clearly is right. > If it's not clear what you should do, you could look at the Zen. It says: "In the face of ambiguity, refuse the temptation to guess." To me, that suggests raising an exception. From ethan at stoneleaf.us Tue May 10 13:52:36 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 10 May 2016 10:52:36 -0700 Subject: What should a decorator do if an attribute already exists? In-Reply-To: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> References: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: <57321FE4.30502@stoneleaf.us> On 05/10/2016 08:45 AM, Steven D'Aprano wrote: > I have a decorator that adds an attribute to the decorated function: > My question is, what should I do if the decorated function already has an > instrument attribute? If the decorator is adding an attribute for the decorated thing to use, and that thing may already have the correct attribute, do nothing; otherwise, raise an exception. -- ~Ethan~ From ned at nedbatchelder.com Tue May 10 13:57:37 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Tue, 10 May 2016 10:57:37 -0700 (PDT) Subject: An educational site written in Python (from YCombinator's RFS) In-Reply-To: References: Message-ID: On Tuesday, May 10, 2016 at 2:13:42 AM UTC-4, Cai Gengyang wrote: > > EDUCATION > > If we can fix education, we can eventually do everything else on this list. > The first attempts to use technology to fix education have focused on using the Internet to distribute traditional content to a wider audience. This is good, but the Internet is a fundamentally different medium and capable of much more. > > Solutions that combine the mass scale of technology with one-on-one in-person interaction are particularly interesting to us. > > This may not require a "breakthrough" technology in the classical sense, but at a minimum it will require very new ways of doing things. > > ------------------------------------------------------------------------------------------------------------------- > > I want to create such a site using Python. What are the various steps I need to take to create such a site ? This is a big project, but one that is worth doing ... Any suggestions / help appreciated ? Thanks alot If you want to join an existing project, Open edX is open-source Python/Django, and powers edx.org and 250 other sites around the world. I'm guessing it isn't quite what this proposal is getting at, but the description here is very vague, so I'm not sure what they are getting at. http://open.edx.org if you want to take a look. --Ned. From nospam at dfs.com Tue May 10 14:03:22 2016 From: nospam at dfs.com (DFS) Date: Tue, 10 May 2016 14:03:22 -0400 Subject: The irony Message-ID: "There should be one-- and preferably only one --obvious way to do it." https://www.python.org/dev/peps/pep-0020/ ----------------------------------- sSQL = "line 1\n" sSQL += "line 2\n" sSQL += "line 3" ----------------------------------- sSQL = ("line 1\n" "line 2\n" "line 3") ----------------------------------- sSQL = "\n".join([ "line 1", "line 2", "line 3", ]) ----------------------------------- sSQL = """line 1 line 2 line 3""" ----------------------------------- sSQL = """\ line 1 line 2 line 3""" ----------------------------------- sSQL = "line 1\n" \ "line 2\n" \ "line 3" ----------------------------------- Which is the "one obvious way" to do it? I liked: sSQL = "line 1\n" sSQL += "line 2\n" sSQL += "line 3" but it's frowned upon in PEP8. From ian.g.kelly at gmail.com Tue May 10 14:15:24 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 10 May 2016 12:15:24 -0600 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, May 10, 2016 at 10:16 AM, DFS wrote: > On 5/9/2016 3:53 AM, Steven D'Aprano wrote: >> >> On Monday 09 May 2016 09:10, DFS wrote: >> >>> sSQL = "line 1\n" >>> sSQL += "line 2\n" >>> sSQL += "line 3" >> >> >> Pointlessly provocative subject line edited. > > > > huh? You called yourself a "master crafts/wo/man". Stop trolling. > ---------------------------------------------------------------------- > DFS: "python is awesome, but too many options for doing the same thing also > makes it difficult. For me, anyway." > > DuhPricko: "That's the difference between a master and an apprentice. The > apprentice likes to follow fixed steps the same way each time. The master > craftsman knows her tools backwards, and can choose the right tool for the > job, and when the choice of tool really doesn't matter and you can use > whatever happens to be the closest to hand." > ---------------------------------------------------------------------- So what, you're upset by the implication that you haven't already mastered Python when you apparently just started a couple months ago? Temper your expectations. > 'master craftswoman' my ass... Take your gender insults somewhere else. They're not welcome here. From python.list at tim.thechases.com Tue May 10 14:27:41 2016 From: python.list at tim.thechases.com (Tim Chase) Date: Tue, 10 May 2016 13:27:41 -0500 Subject: The irony In-Reply-To: References: Message-ID: <20160510132741.650fff13@bigbox.christie.dr> (interspersing letters to name your examples) On 2016-05-10 14:03, DFS wrote: A (nope) > ----------------------------------- > sSQL = "line 1\n" > sSQL += "line 2\n" > sSQL += "line 3" B (see below) > ----------------------------------- > sSQL = ("line 1\n" > "line 2\n" > "line 3") C (see below) > ----------------------------------- > sSQL = "\n".join([ > "line 1", > "line 2", > "line 3", > ]) D (almost) > ----------------------------------- > sSQL = """line 1 > line 2 > line 3""" E (almost) > ----------------------------------- > sSQL = """\ > line 1 > line 2 > line 3""" F (not bad) > ----------------------------------- > sSQL = "line 1\n" \ > "line 2\n" \ > "line 3" > Which is the "one obvious way" to do it? Readability is one of Python's key strengths. I find my choice(s) influenced by ease-of-reading and ease-of-editing (along with its friend, ease-of-reading-diffs). As leading indentation doesn't usually matter in SQL, I tend to just use triple-quoted strings for all my SQL, formatting so it looks like good SQL: sql = """ SELECT a.foo, b.bar FROM tblA a INNER JOIN tblB b ON a.id = b.a_id WHERE a.osteopathy > b.saturation ORDER BY b.monkey_balloon """ This does mean that there's a superfluous newline at the beginning & end of the string and unneeded indentation for each line. But SQL doesn't care, and if it had any noticeable performance issue (ooh, a couple dozen extra bytes getting sent over the wire), it could be post-processed to a clean one-liner. That said, there are cases where the leading indentation does matter to you, then I tend to use your "C" example for code executed once, or a modification of B for code in loops: sql = ( "line1\n" "line2\n" "line3" ) (the difference being the newline after the open-paren to align all the strings' starting offset) If the redundancy of the "\n" characters became sufficiently annoying, I'd define a constant (evaluated once, not in a loop) using the "\n".join(...) method. Seek the code that reads best for you and meets your needs. -tkc From ned at nedbatchelder.com Tue May 10 14:33:41 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Tue, 10 May 2016 11:33:41 -0700 (PDT) Subject: The irony In-Reply-To: References: Message-ID: <2b238291-22ce-4999-84a2-e1cfc45c193d@googlegroups.com> On Tuesday, May 10, 2016 at 2:03:47 PM UTC-4, DFS wrote: > Which is the "one obvious way" to do it? > > I liked: > > sSQL = "line 1\n" > sSQL += "line 2\n" > sSQL += "line 3" > > > but it's frowned upon in PEP8. I would use a way you didn't show: sSQL = """ line1 line2 line3 """ SQL doesn't mind the extra whitespace, so don't worry about the leading spaces. If you really wanted to control it tightly, I would use (and have used): sSQL = textwrap.dedent("""\ line1 line2 line3 """) --Ned. From nospam at dfs.com Tue May 10 15:06:53 2016 From: nospam at dfs.com (DFS) Date: Tue, 10 May 2016 15:06:53 -0400 Subject: An educational site written in Python (from YCombinator's RFS) In-Reply-To: References: Message-ID: On 5/10/2016 2:13 AM, Cai Gengyang wrote: > Ok, so after reading YCombinator's RFS, I have decided that I want to > work on this : > > > ------------------------------------------------------------------------------------------------------------------- > > EDUCATION > > If we can fix education, we can eventually do everything else on this > list. The first attempts to use technology to fix education have > focused on using the Internet to distribute traditional content to a > wider audience. This is good, but the Internet is a fundamentally > different medium and capable of much more. > > Solutions that combine the mass scale of technology with one-on-one > in-person interaction are particularly interesting to us. one room, one teacher, one student, and one web browser? > This may not require a "breakthrough" technology in the classical > sense, but at a minimum it will require very new ways of doing > things. > > ------------------------------------------------------------------------------------------------------------------- > > I want to create such a site using Python. What are the various > steps I need to take to create such a site ? This is a big project, > but one that is worth doing ... Any suggestions / help appreciated ? > Thanks alot When you say 'such a site' what do you envision? From zljubisic at gmail.com Tue May 10 15:27:50 2016 From: zljubisic at gmail.com (zljubisic at gmail.com) Date: Tue, 10 May 2016 12:27:50 -0700 (PDT) Subject: json.loads(...) ValueError: Expecting value: line 1 column 1 (char 0) In-Reply-To: <37add758-036c-4316-98a1-efb2f514dad1@googlegroups.com> References: <137a2593-c307-405f-8f23-3959099ff16f@googlegroups.com> <573143b0$0$1593$c3e8da3$5496439d@news.astraweb.com> <37add758-036c-4316-98a1-efb2f514dad1@googlegroups.com> Message-ID: That was it. Thanks guys for your help. Best regards. From tjreedy at udel.edu Tue May 10 15:34:06 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 10 May 2016 15:34:06 -0400 Subject: Pylint prefers list comprehension over filter... In-Reply-To: References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> Message-ID: On 5/10/2016 9:51 AM, Claudiu Popa wrote: > Thank you for letting us know. While pylint is indeed > opinionated in some cases, we're not trying to be > "arrogant", as you put it, towards Guido or the other core > developers. What's sad in this particular case is that the > feedback had to come in rather a harsh manner, on this group, > instead of being reported as a bug or an enhancement on pylint's > bug tracker. My impression is that the objection to 'bad builtin' was communicated somehow to someone involved with PyLint at least a year ago and ignored or outright rejected. But I don't remember details. I might be mistaken or have been given false information. > Anyway, I wanted to tell you that I agree with your opinion > regarding that message and as such, it is removed and won't be > emitted anymore in the next release (1.6) Then my harsh comment re pylint is not currently valid. Thank you. Hiring managers misusing the score to evaluate applicants is a different issue. -- Terry Jan Reedy From tjreedy at udel.edu Tue May 10 15:47:07 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 10 May 2016 15:47:07 -0400 Subject: Pylint prefers list comprehension over filter... In-Reply-To: References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> <5731f70f$0$1590$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5/10/2016 11:35 AM, pcmanticore at gmail.com wrote: > The bad-builtin check is now an extension, so using the first case > would enable it. The 'old' (not 'bad') builtin check should include using map instead of a comprehension. The check should also pay attention to whether the function argument is an existing function or a one newly created with a lambda expression. "'lambda' in " I think filter(None, iterable) (== filter(bool, iterable)) should be separately flagged as a style point. It is a hack that became unneeded when bool was added. > Another thing that is going to change with the next release is > the introduction of tiers. Basically, pylint overwhelms the user > right now with its enabled checks and we're trying to split these > into tiers, as seen in the following: This looks good. -- Terry Jan Reedy From rosuav at gmail.com Tue May 10 16:22:46 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 11 May 2016 06:22:46 +1000 Subject: Pylint prefers list comprehension over filter... In-Reply-To: References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> <5731f70f$0$1590$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Wed, May 11, 2016 at 1:35 AM, wrote: > Basically, pylint overwhelms the user > right now with its enabled checks and we're trying to split these > into tiers, as seen in the following: > > $ pylint myproject > # core checkers enabled > 10/10 - Congrats, you're clean on a core. You might try with "--pedantic" now. > > $ pylint myproject --pedantic > # pedantic checkers enabled. > 10/10 - Congrats, you're clean on pedantic. You might try with --refactoring now > > $ pylint myproject --refactoring > # refactoring checkers enabled > 10/10 - Congrats, you're clean on refactoring. Last up, try with --style now. > > $ pylint myproject --style > 10/10 - Now you're pylint clean. > > Or running them all: > > $ pylint myproject --all Definitely support this notion. Not sure the "on a core" part makes sense, but that's a minor triviality. ChrisA From nospam at dfs.com Tue May 10 16:42:10 2016 From: nospam at dfs.com (DFS) Date: Tue, 10 May 2016 16:42:10 -0400 Subject: Pylint prefers list comprehension over filter... In-Reply-To: References: <572BF2BF.6000000@icloud.com> <1462498631.232041.599637409.25D91C08@webmail.messagingengine.com> <572E3F53.5010703@icloud.com> Message-ID: On 5/10/2016 3:34 PM, Terry Reedy wrote: > On 5/10/2016 9:51 AM, Claudiu Popa wrote: > >> Thank you for letting us know. While pylint is indeed >> opinionated in some cases, we're not trying to be >> "arrogant", as you put it, towards Guido or the other core >> developers. What's sad in this particular case is that the >> feedback had to come in rather a harsh manner, on this group, >> instead of being reported as a bug or an enhancement on pylint's >> bug tracker. > > My impression is that the objection to 'bad builtin' was communicated > somehow to someone involved with PyLint at least a year ago and ignored > or outright rejected. But I don't remember details. I might be > mistaken or have been given false information. > >> Anyway, I wanted to tell you that I agree with your opinion >> regarding that message and as such, it is removed and won't be >> emitted anymore in the next release (1.6) > > Then my harsh comment re pylint is not currently valid. Thank you. > > Hiring managers misusing the score to evaluate applicants is a different > issue. Interesting. Got any horror stories to share? From nospam at dfs.com Tue May 10 17:21:11 2016 From: nospam at dfs.com (DFS) Date: Tue, 10 May 2016 17:21:11 -0400 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5/10/2016 2:15 PM, Ian Kelly wrote: > On Tue, May 10, 2016 at 10:16 AM, DFS wrote: >> On 5/9/2016 3:53 AM, Steven D'Aprano wrote: >>> >>> On Monday 09 May 2016 09:10, DFS wrote: >>> >>>> sSQL = "line 1\n" >>>> sSQL += "line 2\n" >>>> sSQL += "line 3" >>> >>> >>> Pointlessly provocative subject line edited. >> >> >> >> huh? You called yourself a "master crafts/wo/man". > > Stop trolling. Challenging someone's boastful claim isn't trolling. >> ---------------------------------------------------------------------- >> DFS: "python is awesome, but too many options for doing the same thing also >> makes it difficult. For me, anyway." >> >> DuhPricko: "That's the difference between a master and an apprentice. The >> apprentice likes to follow fixed steps the same way each time. The master >> craftsman knows her tools backwards, and can choose the right tool for the >> job, and when the choice of tool really doesn't matter and you can use >> whatever happens to be the closest to hand." >> ---------------------------------------------------------------------- > > So what, you're upset by the implication that you haven't already > mastered Python when you apparently just started a couple months ago? > Temper your expectations. Of course that wasn't my expectation. That would be crazy... My expectation was to not be met by smug, sanctimonious and condescending replies from the likes of DuhPricko. >> 'master craftswoman' my ass... > > Take your gender insults somewhere else. They're not welcome here. Take your net-nannying somewhere else. It's not welcome /anywhere/. From rosuav at gmail.com Tue May 10 17:27:29 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 11 May 2016 07:27:29 +1000 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Wed, May 11, 2016 at 7:21 AM, DFS wrote: > Take your net-nannying somewhere else. It's not welcome /anywhere/. DFS, you are out of line. Please reconsider the *way* you are saying things, and especially, don't call people names. ChrisA From ethan at stoneleaf.us Tue May 10 18:12:30 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 10 May 2016 15:12:30 -0700 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> Message-ID: <57325CCE.9020206@stoneleaf.us> On 05/10/2016 02:21 PM, DFS wrote: > On 5/9/2016 3:53 AM, Steven D'Aprano wrote: >> Pointlessly provocative subject line edited. > huh? You called yourself a "master crafts/wo/man". > Challenging someone's boastful claim isn't trolling. > My expectation was to not be met by smug, sanctimonious and > condescending replies from the likes of DuhPricko. > 'master craftswoman' my ass... > Take your net-nannying somewhere else. It's not welcome /anywhere/. Having found and read D'Aprano's post I can assure you that: - he wasn't claiming to be a master craftsman - he wasn't being arrogant I can also assure you that *you* are being hostile, rude, and discriminatory, and such behaviour is not welcome here. Any further posts of this nature and you will be placed on moderation. -- ~Ethan~ From nospam at dfs.com Tue May 10 18:36:34 2016 From: nospam at dfs.com (DFS) Date: Tue, 10 May 2016 18:36:34 -0400 Subject: pylint woes In-Reply-To: References: Message-ID: On 5/7/2016 10:50 PM, Chris Angelico wrote: > On Sun, May 8, 2016 at 12:15 PM, DFS wrote: >> The only reason >> >> for j in range(len(list1)): >> do something with list1[j], list2[j], list3[j], etc. >> >> or >> >> for item1, item2, item3 in zip(list1, list2, list3): >> do something with the items >> >> works is because each list has the same number of items. > > Sure, but who cares what each item's position is? All that matters is > that they have corresponding positions, which is what zip() does. They have corresponding positions because zip() possibly truncates data! > Imagine you don't even have the whole lists yet. Imagine someone's > still writing stuff to them as you work. Maybe they're infinite in > length. You can't iterate up to the length of list1, because it > doesn't HAVE a length. But you can still zip it up with other parallel > collections, and iterate over them all. Disregarding a list of infinite length. If lists are still being created: * at every moment in time, len(list1) returns a length that doesn't change even if data is added to the list after the call to len(). Example: If the list has 100 items in it at the point len(list) is called: for i in range(len(list1)) will never iterate more than 100x, no matter how large list1 grows to. Caveat: since list1 may be bigger or smaller than the other lists at that moment in time, an error may occur when using list2[i], list3[i]. Is that all correct as you understand it? * at every moment in time, zip(list1, list2, etc) will return a fixed, same-length lists of tuples, which doesn't change even if data is added to any of the lists after the call to zip(). Example: if the lists have 100, 97 and 102 items in them at the point zip(lists) is called: for item1, item2, item3 in zip(list1, list2, list3) will never iterate beyond 97x, even if the lists grow while the enumeration is occurring. Caveat: since zip() possibly truncates lists, the results - the usage of the data - could be completely invalid. Is that all correct as you understand it? So, if that's all correct, it doesn't matter whether you use 'for i in range(len(lists))' or 'for item in zip(lists)': neither will guarantee data integrity. I haven't timed them, but as I see it, neither has a definite advantage over the other. So what I decided to do was build the lists, check that the lengths are all the same (exit the program if not), zip() them, and use enumeration because it looks less a little less clunky. I see no advantage beyond appearance. From sohcahtoa82 at gmail.com Tue May 10 18:44:08 2016 From: sohcahtoa82 at gmail.com (sohcahtoa82 at gmail.com) Date: Tue, 10 May 2016 15:44:08 -0700 (PDT) Subject: The irony In-Reply-To: References: Message-ID: <0aaa1f48-3b73-40ce-a215-2619a7a51923@googlegroups.com> On Tuesday, May 10, 2016 at 11:03:47 AM UTC-7, DFS wrote: > "There should be one-- and preferably only one --obvious way to do it." > > https://www.python.org/dev/peps/pep-0020/ > > Each method of string concatenation has different uses. > ----------------------------------- > sSQL = "line 1\n" > sSQL += "line 2\n" > sSQL += "line 3" This method is simple, easy to read, and is fine for cases where you're not going to be concatenating more than a couple strings *AT RUN-TIME*. > ----------------------------------- > sSQL = ("line 1\n" > "line 2\n" > "line 3") This performs implicit string concatenation *AT COMPILE TIME*. It doesn't work if the strings you are putting together are variables. > ----------------------------------- > sSQL = "\n".join([ > "line 1", > "line 2", > "line 3", > ]) This joins several strings (at run time) with a string separating them. Though an empty string, comma, or a new line are the most common strings to use, you can use any string. > ----------------------------------- > sSQL = """line 1 > line 2 > line 3""" > ----------------------------------- > sSQL = """\ > line 1 > line 2 > line 3""" These two are effectively the same, sure. I'll give you that. > ----------------------------------- > sSQL = "line 1\n" \ > "line 2\n" \ > "line 3" This is just implicit string concatenation and no real different than your second option except you used the explicit line continuation character instead of using the implicit line continuation created by using parentheses. > ----------------------------------- > > > Which is the "one obvious way" to do it? > I liked: > > sSQL = "line 1\n" > sSQL += "line 2\n" > sSQL += "line 3" > > > but it's frowned upon in PEP8. It's frowned upon because it is incredibly slow when used a lot. Every time you do a += on a string, it has to re-allocate memory, whereas other methods either figure it out at compile time or figure out how much memory will be needed and allocate it once. From ethan at stoneleaf.us Tue May 10 19:16:20 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 10 May 2016 16:16:20 -0700 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> Message-ID: <57326BC4.4080804@stoneleaf.us> On 05/10/2016 02:21 PM, DFS wrote: [some inflammatory nonsense] and is now being moderated. If you see flame-bait in one of the unmoderated venues please ignore it. Thanks. -- ~Ethan~ From michael.selik at gmail.com Tue May 10 19:43:00 2016 From: michael.selik at gmail.com (Michael Selik) Date: Tue, 10 May 2016 23:43:00 +0000 Subject: What should a decorator do if an attribute already exists? In-Reply-To: References: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, May 10, 2016 at 11:48 AM Peter Otten <__peter__ at web.de> wrote: > Steven D'Aprano wrote: > > I have a decorator that adds an attribute to the decorated function: > > inner.instrument = instrument > > return inner > > the original instrument is still accessible as f.__wrapped__.instrument I'd say Peter's example is Option 6: Wrap the instrumented function with a second instrumented wrapper. Since you have various kinds of instruments, silently adding another layer makes sense to me. Is it OK if the order matters? Will some of the instruments get confused because they're working on a wrapper instead of the original function? From michael.selik at gmail.com Tue May 10 19:54:47 2016 From: michael.selik at gmail.com (Michael Selik) Date: Tue, 10 May 2016 23:54:47 +0000 Subject: python - handling HTTP requests asynchronously In-Reply-To: <59a499d5-13e1-4092-895f-34aca0827525@googlegroups.com> References: <59a499d5-13e1-4092-895f-34aca0827525@googlegroups.com> Message-ID: On Sat, May 7, 2016 at 4:46 PM wrote: > Il giorno sabato 7 maggio 2016 21:04:47 UTC+2, Michael Selik ha scritto: > > On Fri, May 6, 2016 at 3:01 AM wrote: > > > > > The PDF is generated through an external API. Since currently is > generated > > > on demand, this is handled synchronously via an HTTP request/response. > > > > > > Are you sending the request or are you receiving the request? > > If you are sending, you can just use threads as you are only doing IO. > > If you are receiving the requests and generating PDFs, you may want to > use > > subprocesses if the PDF-generation is compute-intensive. > > > > > > > 3) multiprocessing module, with e.g. 10 as workers limit. > > > > > > > multiprocessing.Pool is an easy way to use subprocesses > > multiprocessing.pool.ThreadPool is an easy way to use threads. It's not > > well documented, but has the exact same interface as Pool. > > > > the goal is to use the API concurrently (e.g. send 10 or 100 http > requests > > > simultaneously, depending on how many concurrent requests the API can > > > handle). > > > > > > > Sounds like you want to use threads. How does the API let you know you're > > hitting it too frequently? Perhaps you want to code an exponential > backoff > > and retry wrapper for your API requests. > > Thanks for the reply. > Currently the django view that does the job does three http request: > - the first does a POST and send the payload used during the PDF > rendering, the response contains a url to check the pdf generation progress; > - the second loop over that url, checking the progress of pdf generation. > Once the response contains the keyword 'status': 'complete', then it give > also a url for the file retrieval; > - the third one is a GET to retrieve the file, the reponse contains the > binary content of the file, then this content is read and wrapped as > attachment of a django http response, and then returned to the user. > > the goal is to reuse the existing code as much as possible, possibly doing > concurrently, and saving the file instead on a folder. > I'm not a Django expert. Does the Django View require all that stuff to happen before Django can send an HTTP Response back to the user? If so, I suggest you respond to the user immediately: "now generating a bunch of PDFs..." and take care of the rest in the background. Perhaps just write to a file, database, or message queue the info for the PDFs to generate. Have a separate program periodically read the file, database, or message queue to do that work and then maybe email the user when it's completed. From PointedEars at web.de Tue May 10 20:16:30 2016 From: PointedEars at web.de (Thomas 'PointedEars' Lahn) Date: Wed, 11 May 2016 02:16:30 +0200 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? References: <3414323.Mt0gs2Yp5W@PointedEars.de> Message-ID: <21232585.YymZx1Mvhl@PointedEars.de> DFS wrote: > On 5/8/2016 8:44 PM, Thomas 'PointedEars' Lahn wrote: >> DFS wrote: >>> sSQL = "line 1\n" >>> sSQL += "line 2\n" >>> sSQL += "line 3" >> >> [?] >> #----------------------------------------------------------------------- >> sSQL = ("line 1\n" >> "line 2\n" >> "line 3") >> #----------------------------------------------------------------------- >> >> [?] > > or > > sSQL = "line 1\n" \ > "line 2\n" \ > "line 3" > Parentheses are less error-prone. >> With the ?%? string operator (deprecated), > > according to who? TFM. >> Next time, RTFM first: > > > Ironically, PointyHead, [?] > >> Internet is the thing with cables; Usenet is the thing with people. I >> for one tend to avoid communicating with few-letter entities; exceptions >> to that would probably include only E.T., M.J., ALF, and K.I.T.T. > > How about FU? *PLONK* (Note to moderators: But you have let *this* through, huh?) . . . . . . . . . . . . . -- PointedEars Twitter: @PointedEars2 Please do not cc me. / Bitte keine Kopien per E-Mail. From sohcahtoa82 at gmail.com Tue May 10 20:38:15 2016 From: sohcahtoa82 at gmail.com (sohcahtoa82 at gmail.com) Date: Tue, 10 May 2016 17:38:15 -0700 (PDT) Subject: String concatenation (was: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement?) In-Reply-To: <3414323.Mt0gs2Yp5W@PointedEars.de> References: <3414323.Mt0gs2Yp5W@PointedEars.de> Message-ID: <208df469-b1e2-4183-a117-fa700d680235@googlegroups.com> On Sunday, May 8, 2016 at 5:44:25 PM UTC-7, Thomas 'PointedEars' Lahn wrote: > Also, it would be a good idea if you posted under your real name. Internet > is the thing with cables; Usenet is the thing with people. I for one tend > to avoid communicating with few-letter entities; exceptions to that would > probably include only E.T., M.J., ALF, and K.I.T.T. > > -- > PointedEars > > Twitter: @PointedEars2 > Please do not cc me. / Bitte keine Kopien per E-Mail. I don't blame people for not wanting to use their real name on the Internet, especially if you're a woman. There are a lot of crazy people out there that will find out where you live and send death threats just because you disagree with them or you happen to be a woman that enjoys video games or exists in the tech world. From python at mrabarnett.plus.com Tue May 10 21:02:01 2016 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 11 May 2016 02:02:01 +0100 Subject: pylint woes In-Reply-To: References: Message-ID: <00fa454d-3a62-6282-89dd-d84005db9891@mrabarnett.plus.com> On 2016-05-10 23:36, DFS wrote: [snip] > > If lists are still being created: > > * at every moment in time, len(list1) returns a length that doesn't > change even if data is added to the list after the call to len(). > > Example: If the list has 100 items in it at the point len(list) is called: > > for i in range(len(list1)) > > will never iterate more than 100x, no matter how large list1 grows to. > > Caveat: since list1 may be bigger or smaller than the other lists at > that moment in time, an error may occur when using list2[i], list3[i]. > > > Is that all correct as you understand it? > Yes. > > > * at every moment in time, zip(list1, list2, etc) will return a fixed, > same-length lists of tuples, which doesn't change even if data is added > to any of the lists after the call to zip(). > > Example: if the lists have 100, 97 and 102 items in them at the point > zip(lists) is called: > > for item1, item2, item3 in zip(list1, list2, list3) > > will never iterate beyond 97x, even if the lists grow while the > enumeration is occurring. > > Caveat: since zip() possibly truncates lists, the results - the usage of > the data - could be completely invalid. > > > Is that all correct as you understand it? > In Python 2, zip iterates through the arguments immediately and returns a list of tuples, so the answer is yes. In Python 3, zip returns a lazy iterator (like itertools.izip in Python 2) that gets the values from the arguments _on demand_, so the answer is no. [snip] From steve at pearwood.info Tue May 10 21:17:42 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 11 May 2016 11:17:42 +1000 Subject: Moderation [was Re: ...What's wrong with this concatenation statement?] References: <3414323.Mt0gs2Yp5W@PointedEars.de> <21232585.YymZx1Mvhl@PointedEars.de> Message-ID: <57328837$0$22142$c3e8da3$5496439d@news.astraweb.com> On Wed, 11 May 2016 10:16 am, Thomas 'PointedEars' Lahn wrote: > (Note to moderators: But you have let *this* through, huh?) Moderation only affects the mailing list python-list at python.org, the Usenet mirror remains unmoderated. If you are reading this via Usenet, you may see DFS's insults, but it doesn't appear here: https://mail.python.org/pipermail/python-list/2016-May/thread.html -- Steven From rosuav at gmail.com Tue May 10 21:43:50 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 11 May 2016 11:43:50 +1000 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: <21232585.YymZx1Mvhl@PointedEars.de> References: <3414323.Mt0gs2Yp5W@PointedEars.de> <21232585.YymZx1Mvhl@PointedEars.de> Message-ID: On Wed, May 11, 2016 at 10:16 AM, Thomas 'PointedEars' Lahn wrote: >>> With the ?%? string operator (deprecated), >> >> according to who? > > TFM. [citation needed] ChrisA From beliavsky at aol.com Tue May 10 22:00:41 2016 From: beliavsky at aol.com (beliavsky at aol.com) Date: Tue, 10 May 2016 19:00:41 -0700 (PDT) Subject: Intel Distribution for Python Message-ID: <31116921-144f-498e-873f-63510c6b8c63@googlegroups.com> The Intel Distribution for Python 2017 Beta https://software.intel.com/en-us/python-distribution is available for Windows, Linux, and Mac OS for Python 2.7 and 3.5. "The Beta product adds new Python packages like scikit-learn, mpi4py, numba, conda, tbb (Python interfaces to Intel? Threading Building Blocks) and pyDAAL (Python interfaces to Intel? Data Analytics Acceleration Library). The Beta also delivers performance improvements for NumPy/SciPy through linking with performance libraries like Intel? MKL, Intel? Message Passing Interface (Intel? MPI), Intel? TBB and Intel? DAAL." I just installed Intel Python today. Has anyone tried it? Does it run your programs faster than the usual CPython? From no.email at nospam.invalid Tue May 10 22:14:55 2016 From: no.email at nospam.invalid (Paul Rubin) Date: Tue, 10 May 2016 19:14:55 -0700 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87mvnx7334.fsf@jester.gateway.pace.com> DFS writes: > But, I am dead serious about becoming a good Python developer, and I > truly appreciate all clp replies. People are more likely to reply to you if your posting style makes you enjoyable instead of annoying to engage with. That's community spirit. Friendly participation is always welcomed even when the person is new or otherwise not highly knowledgeable. Failing that, if you're a knowledgeable enough developer that people learn things from your posts even at moments when your style is annoying, they might interact with you for the self-interested sake of benefiting from access to your knowledge. There are some regulars here whose stuff is always worth reading even though they can be prickly at times. If you don't bring worthwhile amounts of knowledge AND your posting style is annoying, there's a considerable incentive to not bother reading your stuff at all. That will tend to decrease the amount of replies you receive. Hope this helps! From rosuav at gmail.com Tue May 10 23:12:16 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 11 May 2016 13:12:16 +1000 Subject: Intel Distribution for Python In-Reply-To: <31116921-144f-498e-873f-63510c6b8c63@googlegroups.com> References: <31116921-144f-498e-873f-63510c6b8c63@googlegroups.com> Message-ID: On Wed, May 11, 2016 at 12:00 PM, beliavsky--- via Python-list wrote: > The Intel Distribution for Python 2017 Beta https://software.intel.com/en-us/python-distribution is available for Windows, Linux, and Mac OS for Python 2.7 and 3.5. > > "The Beta product adds new Python packages like scikit-learn, mpi4py, numba, conda, tbb (Python interfaces to Intel? Threading Building Blocks) and pyDAAL (Python interfaces to Intel? Data Analytics Acceleration Library). The Beta also delivers performance improvements for NumPy/SciPy through linking with performance libraries like Intel? MKL, Intel? Message Passing Interface (Intel? MPI), Intel? TBB and Intel? DAAL." > > I just installed Intel Python today. Has anyone tried it? Does it run your programs faster than the usual CPython? > I haven't used it, but based on a reading of their blurbs, I suspect you won't see any significant improvement in base Python code - the advantage is the numeric computation work. For general Python performance, check out PyPy, although it tends to lag behind CPython in versions somewhat. However, PyPy doesn't do anything for your numpy performance, and doesn't even guarantee that everything works: http://pypy.org/download.html#installing-numpy So if the Intel Math Kernel Library lives up to the descriptions, it might be the thing to fill in this gap - "use PyPy to speed up your Python code, or Intel Python to speed up your numpy code". (Most programs won't have performance issues on both of those at once.) It's worth noting that Intel engineers have been proposing a number of performance improvements for backporting into the CPython core, and some of them even made it into 2.7. I've no idea whether the page you linked to is part of the same project or not; it might be completely separate, or it might be the source of all those improvements. ChrisA From arsh840 at gmail.com Tue May 10 23:17:20 2016 From: arsh840 at gmail.com (Arshpreet Singh) Date: Tue, 10 May 2016 20:17:20 -0700 (PDT) Subject: Intel Distribution for Python In-Reply-To: <31116921-144f-498e-873f-63510c6b8c63@googlegroups.com> References: <31116921-144f-498e-873f-63510c6b8c63@googlegroups.com> Message-ID: Thanks for the information, I just applied for program but I got one mail about license and expiration. This software license expires on October 29, 2016. I am not able to understand that can anyone put some light on that how license can be expired? From tjreedy at udel.edu Tue May 10 23:47:49 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 10 May 2016 23:47:49 -0400 Subject: Intel Distribution for Python In-Reply-To: References: <31116921-144f-498e-873f-63510c6b8c63@googlegroups.com> Message-ID: On 5/10/2016 11:12 PM, Chris Angelico wrote: > On Wed, May 11, 2016 at 12:00 PM, beliavsky--- via Python-list > wrote: >> The Intel Distribution for Python 2017 Beta https://software.intel.com/en-us/python-distribution is available for Windows, Linux, and Mac OS for Python 2.7 and 3.5. >> >> "The Beta product adds new Python packages like scikit-learn, mpi4py, numba, conda, tbb (Python interfaces to Intel? Threading Building Blocks) and pyDAAL (Python interfaces to Intel? Data Analytics Acceleration Library). The Beta also delivers performance improvements for NumPy/SciPy through linking with performance libraries like Intel? MKL, Intel? Message Passing Interface (Intel? MPI), Intel? TBB and Intel? DAAL." >> >> I just installed Intel Python today. Has anyone tried it? Does it run your programs faster than the usual CPython? >> > > I haven't used it, but based on a reading of their blurbs, I suspect > you won't see any significant improvement in base Python code - the > advantage is the numeric computation work. All their benchmarks are heavy numeric computation. > For general Python performance, check out PyPy, although it tends to > lag behind CPython in versions somewhat. However, PyPy doesn't do > anything for your numpy performance, and doesn't even guarantee that > everything works: > > http://pypy.org/download.html#installing-numpy > > So if the Intel Math Kernel Library lives up to the descriptions, it > might be the thing to fill in this gap - "use PyPy to speed up your > Python code, or Intel Python to speed up your numpy code". (Most > programs won't have performance issues on both of those at once.) > > It's worth noting that Intel engineers have been proposing a number of > performance improvements for backporting into the CPython core, and > some of them even made it into 2.7. I've no idea whether the page you > linked to is part of the same project or not; it might be completely > separate, or it might be the source of all those improvements. I presume it is all related. What I find interesting is that Intel thinks it will be more profitable to be involved in numerical computation driven by python than not to. -- Terry Jan Reedy From rosuav at gmail.com Wed May 11 00:02:29 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 11 May 2016 14:02:29 +1000 Subject: Intel Distribution for Python In-Reply-To: References: <31116921-144f-498e-873f-63510c6b8c63@googlegroups.com> Message-ID: On Wed, May 11, 2016 at 1:17 PM, Arshpreet Singh wrote: > Thanks for the information, I just applied for program but I got one mail about license and expiration. > > > This software license expires on October 29, 2016. > > > I am not able to understand that can anyone put some light on that how license can be expired? If I own something (say, a skillet that can fry your breakfast and then clonk an intruder on the head), I can permit you to use it under any terms I like. That includes putting a time limit on it ("you can use it now, but I'll need it when I go to look at the floating lights"). It's the same with software. The license terms are "Until 20161029, you may use this as long as blah blah blah, but after 20161029, you may not use this under any circumstances". A copyright owner is free to put any restrictions on the use of the program; if that prevents it from being useful to you, you're free to ignore the program altogether. ChrisA From priisdk at gmail.com Wed May 11 02:33:17 2016 From: priisdk at gmail.com (Poul Riis) Date: Tue, 10 May 2016 23:33:17 -0700 (PDT) Subject: Animations with mayavi and moviepy Message-ID: <79ea0a45-84a9-4789-abe9-466e0e43b297@googlegroups.com> The animation example below (taken from http://zulko.github.io/blog/2014/11/29/data-animations-with-python-and-moviepy/) stops after producing and displaying the 41 frames. In stead, after producing the 41 frames I want to make it loop continuously. I have googled and googled and googled and still not found out how to do it. I guess that it is very simple but how? The program prints out some rather uninteresting information - how can I suppress that? Poul Riis import numpy as np import mayavi.mlab as mlab import moviepy.editor as mpy duration= 2 # duration of the animation in seconds (it will loop) # MAKE A FIGURE WITH MAYAVI fig_myv = mlab.figure(size=(220,220), bgcolor=(1,1,1)) X, Y = np.linspace(-2,2,200), np.linspace(-2,2,200) XX, YY = np.meshgrid(X,Y) ZZ = lambda d: np.sinc(XX**2+YY**2)+np.sin(XX+d) # ANIMATE THE FIGURE WITH MOVIEPY, WRITE AN ANIMATED GIF def make_frame(t): mlab.clf() # clear the figure (to reset the colors) mlab.mesh(YY,XX,ZZ(2*np.pi*t/duration), figure=fig_myv) return mlab.screenshot(antialiased=True) animation = mpy.VideoClip(make_frame, duration=duration) animation.write_gif("sinc.gif", fps=20) From martin846 at gmail.com Wed May 11 05:52:13 2016 From: martin846 at gmail.com (martin846 at gmail.com) Date: Wed, 11 May 2016 02:52:13 -0700 (PDT) Subject: Importing from __future__ doesn't work under PDB Message-ID: <42b10847-958a-4845-a08a-0bab359846a9@googlegroups.com> Hi folks, I've been trying to use pdb to debug a Python 2.7 program and this has really stumped me..... When debugging a program that uses from __future__ import division in Python 2.7, it would be useful to be able to have the same behaviour at the pdb prompt. However, it looks like pdb doesn't honour either the import statement in the original program, or one run explicitly at the pdb prompt. Given the program test.py: from __future__ import division print(2/3) I get the following interactive pdb session: martin at martin-H97-D3H:~$ python -m pdb test.py > /home/martin/test.py(1)() -> from __future__ import division (Pdb) n > /home/martin/test.py(2)() -> print(2/3) (Pdb) n 0.666666666667 --Return-- > /home/martin/test.py(2)()->None -> print(2/3) (Pdb) 2/3 0 (Pdb) from __future__ import division (Pdb) 2/3 0 (Pdb) In other words, I get different results for the same expression (2/3) in the program and on the pdb prompt, which makes debugging tricky. I cannot figure out how to persuade pdb to actually load the division module. Is there an approved way? Normal modules load as expected in pdb: (Pdb) import os (Pdb) os.getcwd() '/home/martin' so this is obviously some special magic that applies only to the __future__ module. Any ideas? From lorenzofsutton at gmail.com Wed May 11 06:17:57 2016 From: lorenzofsutton at gmail.com (Lorenzo Sutton) Date: Wed, 11 May 2016 12:17:57 +0200 Subject: The irony In-Reply-To: References: Message-ID: <4b7636d9-3a3f-1015-0749-7f34e08dca95@gmail.com> On 10/05/2016 20:03, DFS wrote: > "There should be one-- and preferably only one --obvious way to do it." > > https://www.python.org/dev/peps/pep-0020/ "Explicit is better than implicit." What is your use case and scenario? :-) Maybe it's better to write a function to automatise this so that if instead of "line 1\n ..." you want "banana 1~banana 2~ " etc. you can simply change parameters? That said join and list comprehensions would also come to mind, but not sure how "obvious" that is... Lorenzo. > > > ----------------------------------- > sSQL = "line 1\n" > sSQL += "line 2\n" > sSQL += "line 3" > ----------------------------------- > sSQL = ("line 1\n" > "line 2\n" > "line 3") > ----------------------------------- > sSQL = "\n".join([ > "line 1", > "line 2", > "line 3", > ]) > ----------------------------------- > sSQL = """line 1 > line 2 > line 3""" > ----------------------------------- > sSQL = """\ > line 1 > line 2 > line 3""" > ----------------------------------- > sSQL = "line 1\n" \ > "line 2\n" \ > "line 3" > ----------------------------------- > > > Which is the "one obvious way" to do it? > > I liked: > > sSQL = "line 1\n" > sSQL += "line 2\n" > sSQL += "line 3" > > > but it's frowned upon in PEP8. > > From rosuav at gmail.com Wed May 11 06:21:38 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 11 May 2016 20:21:38 +1000 Subject: Importing from __future__ doesn't work under PDB In-Reply-To: <42b10847-958a-4845-a08a-0bab359846a9@googlegroups.com> References: <42b10847-958a-4845-a08a-0bab359846a9@googlegroups.com> Message-ID: On Wed, May 11, 2016 at 7:52 PM, wrote: > In other words, I get different results for the same expression (2/3) in the program and on the pdb prompt, which makes debugging tricky. I cannot figure out how to persuade pdb to actually load the division module. Is there an approved way? Normal modules load as expected in pdb: > > (Pdb) import os > (Pdb) os.getcwd() > '/home/martin' > > so this is obviously some special magic that applies only to the __future__ module. Any ideas? You're right that future directives have special magic. I don't know how to trigger that magic with pdb, but it doesn't surprise me that it doesn't acknowledge future directives in the file you're debugging (there could be multiple files - future directives apply only to that one module). Worst case, you should be able to hack it in like this: # pdb_future.py def futurize(*directives): import __future__ flags = 0 for directive in directives: flags |= getattr(__future__, directive).compiler_flag orig_compile = compile # Snapshot for closure def compile_future(source, filename, mode): return orig_compile(source, filename, mode, flags) return compile_future import pdb pdb.compile = futurize('print_function', 'division', 'absolute_import') pdb.main() I doubt that Python 2.7's pdb will grow a feature like this, but you could propose it for 3.6 and see if anyone supports backporting it. Otherwise, injecting a wrapper will work, but it ain't pretty. ChrisA From martin846 at gmail.com Wed May 11 06:41:32 2016 From: martin846 at gmail.com (martin846 at gmail.com) Date: Wed, 11 May 2016 03:41:32 -0700 (PDT) Subject: Importing from __future__ doesn't work under PDB In-Reply-To: References: <42b10847-958a-4845-a08a-0bab359846a9@googlegroups.com> Message-ID: <993ea9f5-adf0-4adc-84ac-f39056c39c78@googlegroups.com> Many thanks for the quick reply. I must admit I was surprised not to find any note about this anywhere in the docs (or indeed anywhere online); it seems like something that people would have run into before. I've added a request to the documentation tracker to have a note added to help people who end up in the same position. From info at egenix.com Wed May 11 07:28:24 2016 From: info at egenix.com (eGenix Team: M.-A. Lemburg) Date: Wed, 11 May 2016 13:28:24 +0200 Subject: ANN: eGenix PyRun - One file Python Runtime 2.2.0 Message-ID: <57331758.50306@egenix.com> ________________________________________________________________________ ANNOUNCING eGenix PyRun - One file Python Runtime Version 2.2.0 An easy-to-use single file relocatable Python run-time - available for Linux, Mac OS X and Unix platforms, with support for Python 2.6, 2.7, 3.4 and * now also for Python 3.5 * This announcement is also available on our web-site for online reading: http://www.egenix.com/company/news/eGenix-PyRun-2.2.0-GA.html ________________________________________________________________________ INTRODUCTION eGenix PyRun is our open source, one file, no installation version of Python, making the distribution of a Python interpreter to run based scripts and applications to Unix based systems as simple as copying a single file. eGenix PyRun's executable only needs 11MB for Python 2 and 13MB for Python 3, but still supports most Python application and scripts - and it can be compressed to just 3-4MB using upx, if needed. Compared to a regular Python installation of typically 100MB on disk, eGenix PyRun is ideal for applications and scripts that need to be distributed to several target machines, client installations or customers. It makes "installing" Python on a Unix based system as simple as copying a single file. eGenix has been using eGenix PyRun internally in the mxODBC Connect Server product since 2008 with great success and decided to make it available as a stand-alone open-source product. We provide both the source archive to build your own eGenix PyRun, as well as pre-compiled binaries for Linux, FreeBSD and Mac OS X, as 32- and 64-bit versions. The binaries can be downloaded manually, or you can let our automatic install script install-pyrun take care of the installation: ./install-pyrun dir and you're done. Please see the product page for more details: http://www.egenix.com/products/python/PyRun/ ________________________________________________________________________ NEWS This minor level release of eGenix PyRun comes with the following enhancements: Enhancements / Changes ---------------------- * Ported eGenix PyRun to Python 3.5.1. install-pyrun Quick Install Enhancements --------------------------------------------- eGenix PyRun includes a shell script called install-pyrun, which greatly simplifies installation of PyRun. It works much like the virtualenv shell script used for creating new virtual environments (except that there's nothing virtual about PyRun environments). https://downloads.egenix.com/python/install-pyrun With the script, an eGenix PyRun installation is as simple as running: ./install-pyrun targetdir This will automatically detect the platform, download and install the right pyrun version into targetdir. We have updated this script since the last release: * Updated install-pyrun to default to eGenix PyRun 2.2.0 and its feature set. For a complete list of changes, please see the eGenix PyRun Changelog: http://www.egenix.com/products/python/PyRun/changelog.html ________________________________________________________________________ LICENSE eGenix PyRun is distributed under the eGenix.com Public License 1.1.0 which is an Open Source license similar to the Python license. You can use eGenix PyRun in both commercial and non-commercial settings without fee or charge. Please see our license page for more details: http://www.egenix.com/products/python/PyRun/license.html The package comes with full source code. ________________________________________________________________________ DOWNLOADS The download archives and instructions for installing eGenix PyRun can be found at: http://www.egenix.com/products/python/PyRun/ As always, we are providing pre-built binaries for all common platforms: Windows 32/64-bit, Linux 32/64-bit, FreeBSD 32/64-bit, Mac OS X 32/64-bit. Source code archives are available for installation on other platforms, such as Solaris, AIX, HP-UX, etc. _______________________________________________________________________ SUPPORT Commercial support for this product is available from eGenix.com. Please see http://www.egenix.com/services/support/ for details about our support offerings. ________________________________________________________________________ MORE INFORMATION For more information about eGenix PyRun, licensing and download instructions, please visit our web-site: http://www.egenix.com/products/python/PyRun/ About eGenix (http://www.egenix.com/): eGenix is a Python software project, consulting and product company delivering expert services and professional quality products for companies, Python users and developers. We specialize in database driven applications, large scale software designs and integration. Enjoy, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 11 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From beliavsky at aol.com Wed May 11 08:20:52 2016 From: beliavsky at aol.com (beliavsky at aol.com) Date: Wed, 11 May 2016 05:20:52 -0700 (PDT) Subject: Intel Distribution for Python In-Reply-To: References: <31116921-144f-498e-873f-63510c6b8c63@googlegroups.com> Message-ID: On Tuesday, May 10, 2016 at 11:17:33 PM UTC-4, Arshpreet Singh wrote: > Thanks for the information, I just applied for program but I got one mail about license and expiration. > > > This software license expires on October 29, 2016. > > > I am not able to understand that can anyone put some light on that how license can be expired? I assume that the Intel python.exe executable will stop running on that date. When I had a trial license for the Intel Fortran compiler, ifort.exe stopped working after one month. From ned at nedbatchelder.com Wed May 11 15:39:48 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 11 May 2016 12:39:48 -0700 (PDT) Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: <21232585.YymZx1Mvhl@PointedEars.de> References: <3414323.Mt0gs2Yp5W@PointedEars.de> <21232585.YymZx1Mvhl@PointedEars.de> Message-ID: On Tuesday, May 10, 2016 at 8:16:43 PM UTC-4, Thomas 'PointedEars' Lahn wrote: > DFS wrote: > > > On 5/8/2016 8:44 PM, Thomas 'PointedEars' Lahn wrote: > >> With the ?%? string operator (deprecated), > > > > according to who? > > TFM. > It's easy to be confused on this point. Early on in the 3.x planning, there was talk of deprecating %-formatting. It didn't happen, though. The docs have discussed it in a few different ways over the years. Python 3.0 and 3.1 said: https://docs.python.org/3.1/library/stdtypes.html#old-string-formatting-operations Note: The formatting operations described here are obsolete and may go away in future versions of Python. Use the new String Formatting in new code. 3.2 said: https://docs.python.org/3.2/library/stdtypes.html#old-string-formatting-operations As the new String Formatting syntax is more flexible and handles tuples and dictionaries naturally, it is recommended for new code. However, there are no current plans to deprecate printf-style formatting. 3.3 and up say: https://docs.python.org/3.3/library/stdtypes.html#printf-style-string-formatting Note: The formatting operations described here exhibit a variety of quirks that lead to a number of common errors (such as failing to display tuples and dictionaries correctly). Using the newer str.format() interface helps avoid these errors, and also provides a generally more powerful, flexible and extensible approach to formatting text. So yes, .format is newer and more powerful, and there are good reasons to recommend it. But %-formatting is not deprecated. --NMB. From sohcahtoa82 at gmail.com Wed May 11 17:30:31 2016 From: sohcahtoa82 at gmail.com (sohcahtoa82 at gmail.com) Date: Wed, 11 May 2016 14:30:31 -0700 (PDT) Subject: String concatenation In-Reply-To: <2322246.dinlaEg1Qz@PointedEars.de> References: <3414323.Mt0gs2Yp5W@PointedEars.de> <208df469-b1e2-4183-a117-fa700d680235@googlegroups.com> <2322246.dinlaEg1Qz@PointedEars.de> Message-ID: <2bb362b3-0763-422b-ac05-32b9f3a09b09@googlegroups.com> On Wednesday, May 11, 2016 at 12:14:43 PM UTC-7, Thomas 'PointedEars' Lahn wrote: > sohcahtoa82 at gmail.com wrote: > > > I don't blame people for not wanting to use their real name on the > > Internet, especially if you're a woman. There are a lot of crazy people > > out there that will find out where you live and send death threats just > > because you disagree with them or you happen to be a woman that enjoys > > video games or exists in the tech world. > > FUD and paranoia. And this is _not_ the Internet. > > -- > PointedEars > > Twitter: @PointedEars2 > Please do not cc me. / Bitte keine Kopien per E-Mail. You call it FUD and paranoia, I call it reality. Are you familiar with the term "doxxing"? Are you familiar with the shenanigans of GamerGate and Anonymous/4chan? Or even some people on reddit. It doesn't just happen to celebrities, either. Is it likely to happen to me? No, not really, but that doesn't mean I shouldn't be concerned at the possibility. And how is this mailing list NOT part of the Internet? From paul at mad-scientist.net Wed May 11 17:39:24 2016 From: paul at mad-scientist.net (Paul Smith) Date: Wed, 11 May 2016 17:39:24 -0400 Subject: Finding .so files without setting LD_LIBRARY_PATH Message-ID: <1463002764.2579.26.camel@mad-scientist.net> Hi all. I have a locally-built version of Python (2.7.11) that I'm copying around to different systems, running all different versions of GNU/Linux. Because I need this to work across systems I'm bundling important .so's with my Python installation (libcrypto, libssl, libreadline, libgmp) which are dynamically loaded when I do things like "import ssl" etc. I have a wrapper script around this Python that everyone runs, rather than invoking it directly, and it is able to set some environment variables etc. So, I had been setting LD_LIBRARY_PATH before I invoke Python so it can find the "right" versions of libcrypto.so etc. That works fine, but here's the problem: because LD_LIBRARY_PATH is in Python's environment it is also passed down to programs invoked by Python. That means if I (for example) invoke subprocess.call(['ssh', ...]) then it fails because the system ssh is looking for the system libcrypto.so, and when it finds the Python libcrypto.so instead (because of LD_LIBRARY_PATH) it fails. What I'd like to do is have a way of setting the library path that Python uses when it tries to load .so files (for example in the ssl module which loads lib-dynload/_ssl.so which links to libssl.so and libcrypto.so), WITHOUT setting LD_LIBRARY_PATH in the environment that Python passes to its children. Is there any way to do this? It seems like there must be some way to do it, because if after I start python with LD_LIBRARY_PATH set, then I delete it from os.environ, then I import ssl it works, so it's still set in Python's environment. But I don't want to have to modify all my scripts to unset LD_LIBRARY_PATH, plus that's not the right thing to do in case someone needs to set LD_LIBRARY_PATH for themselves. What I've tried so far is writing a little Python script that unsets LD_LIBRARY_PATH and setting PYTHONSETUP to the name of that script in my wrapper, which does seem to work, but that takes away the ability for users to use PYTHONSETUP themselves which is sub-optimal. Does anyone have any other ideas? From ned at nedbatchelder.com Wed May 11 17:50:05 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 11 May 2016 14:50:05 -0700 (PDT) Subject: String concatenation In-Reply-To: <2bb362b3-0763-422b-ac05-32b9f3a09b09@googlegroups.com> References: <3414323.Mt0gs2Yp5W@PointedEars.de> <208df469-b1e2-4183-a117-fa700d680235@googlegroups.com> <2322246.dinlaEg1Qz@PointedEars.de> <2bb362b3-0763-422b-ac05-32b9f3a09b09@googlegroups.com> Message-ID: <1bb847c1-7ac2-4c5e-a1ed-11ded09367dd@googlegroups.com> On Wednesday, May 11, 2016 at 5:30:48 PM UTC-4, sohca... at gmail.com wrote: > On Wednesday, May 11, 2016 at 12:14:43 PM UTC-7, Thomas 'PointedEars' Lahn wrote: > > sohcahtoa82 at gmail.com wrote: > > > > > I don't blame people for not wanting to use their real name on the > > > Internet, especially if you're a woman. There are a lot of crazy people > > > out there that will find out where you live and send death threats just > > > because you disagree with them or you happen to be a woman that enjoys > > > video games or exists in the tech world. > > > > FUD and paranoia. And this is _not_ the Internet. > > > > -- > > PointedEars > > > > Twitter: @PointedEars2 > > Please do not cc me. / Bitte keine Kopien per E-Mail. > > You call it FUD and paranoia, I call it reality. Are you familiar with the term "doxxing"? Are you familiar with the shenanigans of GamerGate and Anonymous/4chan? Or even some people on reddit. It doesn't just happen to celebrities, either. Is it likely to happen to me? No, not really, but that doesn't mean I shouldn't be concerned at the possibility. > > And how is this mailing list NOT part of the Internet? It's really not worth debating this point. Thomas has a fixation on people needing to supply their real names, but is the only one who feels this way. --NMB. From rosuav at gmail.com Wed May 11 19:07:33 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 12 May 2016 09:07:33 +1000 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: References: <3414323.Mt0gs2Yp5W@PointedEars.de> <21232585.YymZx1Mvhl@PointedEars.de> Message-ID: On Thu, May 12, 2016 at 5:39 AM, Ned Batchelder wrote: > On Tuesday, May 10, 2016 at 8:16:43 PM UTC-4, Thomas 'PointedEars' Lahn wrote: >> DFS wrote: >> >> > On 5/8/2016 8:44 PM, Thomas 'PointedEars' Lahn wrote: >> >> With the ?%? string operator (deprecated), >> > >> > according to who? >> >> TFM. >> > > It's easy to be confused on this point. Early on in the 3.x planning, > there was talk of deprecating %-formatting. It didn't happen, though. > > The docs have discussed it in a few different ways over the years. > > Python 3.0 and 3.1 said: > > https://docs.python.org/3.1/library/stdtypes.html#old-string-formatting-operations > > Note: The formatting operations described here are obsolete and > may go away in future versions of Python. Use the new String > Formatting in new code. > > 3.3 and up say: > > https://docs.python.org/3.3/library/stdtypes.html#printf-style-string-formatting Note also the change in the link: the current docs don't even call it "old string formatting" any more; the two are perfectly parallel. But at no point was percent formatting EVER deprecated, which is a very specific technical term. As you say, there was talk of MAYBE deprecating it. ChrisA From python.list at tim.thechases.com Wed May 11 19:31:36 2016 From: python.list at tim.thechases.com (Tim Chase) Date: Wed, 11 May 2016 18:31:36 -0500 Subject: % formatting vs .format() (was: What's wrong with this concatenation statement?) In-Reply-To: References: <3414323.Mt0gs2Yp5W@PointedEars.de> <21232585.YymZx1Mvhl@PointedEars.de> Message-ID: <20160511183136.3c5420f8@bigbox.christie.dr> On 2016-05-11 12:39, Ned Batchelder wrote: > The docs have discussed it in a few different ways over the years. > > Python 3.0 and 3.1 said: > > 3.2 said: > > 3.3 and up say: I've picked up on the "% formatting not going away, .format() more flexible" vibe but had missed the evolution in the docs to clarify the intent. Thanks for the succinct summary of how that's changed. -tkc From eryksun at gmail.com Wed May 11 22:08:43 2016 From: eryksun at gmail.com (eryk sun) Date: Thu, 12 May 2016 03:08:43 +0100 Subject: Finding .so files without setting LD_LIBRARY_PATH In-Reply-To: <1463002764.2579.26.camel@mad-scientist.net> References: <1463002764.2579.26.camel@mad-scientist.net> Message-ID: On Wed, May 11, 2016 at 10:39 PM, Paul Smith wrote: > Hi all. I have a locally-built version of Python (2.7.11) that I'm > copying around to different systems, running all different versions of > GNU/Linux. ... > What I'd like to do is have a way of setting the library path that > Python uses when it tries to load .so files (for example in the ssl > module which loads lib-dynload/_ssl.so which links to libssl.so and > libcrypto.so), WITHOUT setting LD_LIBRARY_PATH in the environment that > Python passes to its children. An ELF header can contain either an RPATH or a RUNPATH to extend the library search path at load time. RPATH is searched before LD_LIBRARY_PATH, while RUNPATH is searched after LD_LIBRARY_PATH. Note that the Linux loader only uses an embedded RUNPATH when searching for the immediate dependencies of an object. Use an RPATH instead if you need to extend the path that dlopen searches at runtime (e.g. for loading shared libraries via ctypes). Paths can be defined relative to the object using ${ORIGIN}. For example, the following configures a Python build to add a relative RPATH: $ ./configure LDFLAGS='-Wl,-rpath,\$${ORIGIN}/lib' $ make Or a RUNPATH: $ ./configure LDFLAGS='-Wl,-rpath,\$${ORIGIN}/lib,--enable-new-dtags' $ make Here's the resulting RPATH in the ELF header: $ readelf -d python | grep RPATH 0x000000000000000f (RPATH) Library rpath: [${ORIGIN}/lib] From louis.a.russ at gmail.com Wed May 11 22:51:59 2016 From: louis.a.russ at gmail.com (louis.a.russ at gmail.com) Date: Wed, 11 May 2016 19:51:59 -0700 (PDT) Subject: More list building In-Reply-To: References: Message-ID: <9983c36a-7c06-45b6-9ab8-845eee5915d9@googlegroups.com> On Wednesday, May 11, 2016 at 1:22:09 PM UTC-4, DFS wrote: > Have: > p1 = ['Now', 'the', 'for', 'good'] > p2 = ['is', 'time', 'all', 'men'] > > want > [('Now','is','the','time'), ('for','all','good','men')] > > This works: > > p = [] > for i in xrange(0,len(p1),2): > p.insert(i,(p1[i],p2[i],p1[i+1],p2[i+1])) > > > But it seems clunky. > > Better way(s)? > > Thanks Would this work for you? Or do you need something more general? t = list(zip(p1,p2)) p = [t[0]+t[1],t[2]+t[3]] From jussi.piitulainen at helsinki.fi Thu May 12 00:55:14 2016 From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen) Date: Thu, 12 May 2016 07:55:14 +0300 Subject: Finding .so files without setting LD_LIBRARY_PATH References: <1463002764.2579.26.camel@mad-scientist.net> Message-ID: eryk sun writes: > On Wed, May 11, 2016 at 10:39 PM, Paul Smith wrote: >> Hi all. I have a locally-built version of Python (2.7.11) that I'm >> copying around to different systems, running all different versions of >> GNU/Linux. > ... >> What I'd like to do is have a way of setting the library path that >> Python uses when it tries to load .so files (for example in the ssl >> module which loads lib-dynload/_ssl.so which links to libssl.so and >> libcrypto.so), WITHOUT setting LD_LIBRARY_PATH in the environment that >> Python passes to its children. > > An ELF header can contain either an RPATH or a RUNPATH to extend the > library search path at load time. RPATH is searched before > LD_LIBRARY_PATH, while RUNPATH is searched after LD_LIBRARY_PATH. Note > that the Linux loader only uses an embedded RUNPATH when searching for > the immediate dependencies of an object. Use an RPATH instead if you > need to extend the path that dlopen searches at runtime (e.g. for > loading shared libraries via ctypes). > > Paths can be defined relative to the object using ${ORIGIN}. For > example, the following configures a Python build to add a relative > RPATH: > > $ ./configure LDFLAGS='-Wl,-rpath,\$${ORIGIN}/lib' > $ make > > Or a RUNPATH: > > $ ./configure LDFLAGS='-Wl,-rpath,\$${ORIGIN}/lib,--enable-new-dtags' > $ make > > Here's the resulting RPATH in the ELF header: > > $ readelf -d python | grep RPATH > 0x000000000000000f (RPATH) Library rpath: [${ORIGIN}/lib] There's a tool (GPLv3+) called patchelf that can set these in an ELF binary: https://nixos.org/patchelf.html https://github.com/NixOS/patchelf From dieter at handshake.de Thu May 12 03:08:35 2016 From: dieter at handshake.de (dieter) Date: Thu, 12 May 2016 09:08:35 +0200 Subject: Finding .so files without setting LD_LIBRARY_PATH References: <1463002764.2579.26.camel@mad-scientist.net> Message-ID: <874ma3u51o.fsf@handshake.de> Paul Smith writes: > ... > That works fine, but here's the problem: because LD_LIBRARY_PATH is in > Python's environment it is also passed down to programs invoked by > Python. That means if I (for example) invoke subprocess.call(['ssh', > ...]) then it fails because the system ssh is looking for the system > libcrypto.so, and when it finds the Python libcrypto.so instead > (because of LD_LIBRARY_PATH) it fails. I have had a similar problem - however with "PYTHONPATH" instead of "LD_LIBRARY_PATH". I solved it by removing "PYTHONPATH" from "os.environ" during Python startup. This way, the envvar "PYTHONPATH" was effective to set up "sys.path" but processes started from this python would not see it. From steve+comp.lang.python at pearwood.info Thu May 12 03:27:52 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Thu, 12 May 2016 17:27:52 +1000 Subject: String concatenation References: <3414323.Mt0gs2Yp5W@PointedEars.de> <2279805.ZaV9TF1YuJ@PointedEars.de> <573183b1$0$2741$c3e8da3$76491128@news.astraweb.com> <87twi675cg.fsf@jester.gateway.pace.com> <573193a0$0$11094$c3e8da3@news.astraweb.com> Message-ID: <57343078$0$1509$c3e8da3$5496439d@news.astraweb.com> On Tuesday 10 May 2016 18:15, David Palao wrote: > 2016-05-10 9:54 GMT+02:00 Steven D'Aprano > : >> I'm surprised that Spanish names are not affected. Consider a woman who goes >> by the personal name of Maria Teresa, whose father's first surname was >> Garc?a and mother's first surname was Ram?rez. Her name would therefore be >> Maria Teresa Garc?a Ram?rez. If she marries El? Arroyo L?pez, then she might >> change her name to Maria Teresa Garc?a Ram?rez de Arroyo. With six words, >> that would fall foul of Facebook's foul naming policy. > In Spain "de Arroyo" officially does not become part of the name. The > same applies to other countries as well. Not 100% sure that it is true > in every Spanish speaking country though. Thanks for the clarification David. Nevertheless, whether it is part of her *legal* name or not, some Spanish women prefer to use her husband's name as part of her preferred real name, that is, the name she answers to and the name she prefers to sign on correspondence. Maybe it's a generation thing? Perhaps in Spain the women using Facebook and the women taking their husband's name don't intersect? -- Steve From steve+comp.lang.python at pearwood.info Thu May 12 03:31:34 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Thu, 12 May 2016 17:31:34 +1000 Subject: Analytical Geometry in Python with GeoMath References: <154790603d7.1040ee3bd58689.8427362743863775803@vmesel.com> <5729bc1a$0$2927$c3e8da3$76491128@news.astraweb.com> <8F08F6A8-09F9-46BE-9D88-900490515868@vmesel.com> Message-ID: <57343157$0$1509$c3e8da3$5496439d@news.astraweb.com> Hi Vinicius, On Thursday 05 May 2016 04:16, Vinicius wrote: > To add a point, you do: > From geomath import point > A = point.Point(x,y) > A.distance(PointB) > A.mispoint(PointB) > A.quadrant() How does your library compare with Eukleides? http://www.eukleides.org/quickstart.html -- Steve From steve+comp.lang.python at pearwood.info Thu May 12 03:43:08 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Thu, 12 May 2016 17:43:08 +1000 Subject: String concatenation References: <3414323.Mt0gs2Yp5W@PointedEars.de> <2279805.ZaV9TF1YuJ@PointedEars.de> <57314838$0$1586$c3e8da3$5496439d@news.astraweb.com> Message-ID: <5734340d$0$2804$c3e8da3$76491128@news.astraweb.com> On Tuesday 10 May 2016 12:42, Chris Angelico wrote: > On Tue, May 10, 2016 at 12:32 PM, Steven D'Aprano > wrote: >> Floats are old (they go back to the first release of Python), they have >> many quirks (x + y - x is not necessarily equal to y), and people make >> many errors with floats. Does this mean they are deprecated? Of course >> not. > > Careful there Steven - now that cdecimal is in core, people might > start saying that. They *might*, but they haven't *yet*. And people can say all sorts of things, doesn't make them true. > Particularly if (as is periodically requested, and > which I think would be a good idea) Decimal literals become a thing. > And the removal of core types HAS happened. Correct me if I'm wrong, > but didn't the first release of Python have only the short integer > type, and then a completely new 'long' type was added? The oldest Python I have installed is 0.9.1, which not only lacks the L suffix, but it also lacks ** for exponentiation. There's also no "long" type: >>> n = 2 >>> for i in range(100): ... n = n * 2 ... Unhandled exception: run-time error: integer overflow Stack backtrace (innermost last): File "", line 2 > Automatic > promotion blurred the distinction, and then Python 3.0 removed the > 'int' type and renamed 'long'. So it's theoretically possible for > Decimal to replace float... > > ... except that that would actually be a bad idea, which a lot of > people don't realize. Indeed. Decimal might not be old, but it does have quirks and it can certainly lead to errors (both misuse and rounding errors, among others). It is subject to the same sorts of floating point issues as binary floats. -- Steve From marco.nawijn at colosso.nl Thu May 12 04:21:29 2016 From: marco.nawijn at colosso.nl (marco.nawijn at colosso.nl) Date: Thu, 12 May 2016 01:21:29 -0700 (PDT) Subject: More list building In-Reply-To: References: Message-ID: <9f2bb67f-7812-41c9-a31c-aeb7ce2788ad@googlegroups.com> On Wednesday, May 11, 2016 at 7:22:09 PM UTC+2, DFS wrote: > Have: > p1 = ['Now', 'the', 'for', 'good'] > p2 = ['is', 'time', 'all', 'men'] > > want > [('Now','is','the','time'), ('for','all','good','men')] > > This works: > > p = [] > for i in xrange(0,len(p1),2): > p.insert(i,(p1[i],p2[i],p1[i+1],p2[i+1])) > > > But it seems clunky. > > Better way(s)? > > Thanks Another way using some array manipulation: (using Python 2.7 and numpy) # coding: utf-8 import numpy as np p1 = np.array(('Now', 'the', 'for', 'good')) p2 = np.array(('is', 'time', 'all', 'men')) p1s = np.split(p1, 2) p2s = np.split(p2, 2) a1 = np.vstack((p1, p2)) r1 = np.hstack((a1[:, 0], a1[:, 1])) r2 = np.hstack((a1[:, 2], a1[:, 3])) print r1 print r2 Produces the following output: ['Now' 'is' 'the' 'time'] ['for' 'all' 'good' 'men'] From mr.eightnoteight at gmail.com Thu May 12 04:23:55 2016 From: mr.eightnoteight at gmail.com (srinivas devaki) Date: Thu, 12 May 2016 13:53:55 +0530 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: <20160508183738.2f551043@bigbox.christie.dr> References: <20160508183738.2f551043@bigbox.christie.dr> Message-ID: On May 9, 2016 5:31 AM, "Tim Chase" wrote: > > then that's a bad code-smell (you get quadratic behavior as the > strings are constantly resized), usually better replaced with > I just want to point out that in Python s += str in loop is not giving quadratic behavior. I don't know why but it runs fast. I'm very much interested to know why it is so? In [3]: %%timeit ...: s = '' ...: for x in xrange(10**6): ...: s += str(x) ...: 1 loop, best of 3: 383 ms per loop In [4]: %%timeit s = '' for x in xrange(10**6): s = s + str(x) ...: 1 loop, best of 3: 383 ms per loop From rosuav at gmail.com Thu May 12 04:40:22 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 12 May 2016 18:40:22 +1000 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: References: <20160508183738.2f551043@bigbox.christie.dr> Message-ID: On Thu, May 12, 2016 at 6:23 PM, srinivas devaki wrote: > On May 9, 2016 5:31 AM, "Tim Chase" wrote: >> >> then that's a bad code-smell (you get quadratic behavior as the >> strings are constantly resized), usually better replaced with >> > > I just want to point out that in Python s += str in loop is not giving > quadratic behavior. I don't know why but it runs fast. I'm very much > interested to know why it is so? > > In [3]: %%timeit > ...: s = '' > ...: for x in xrange(10**6): > ...: s += str(x) > ...: > 1 loop, best of 3: 383 ms per loop > > In [4]: %%timeit > s = '' > for x in xrange(10**6): > s = s + str(x) > ...: > 1 loop, best of 3: 383 ms per loop Some versions of CPython do include an optimization for this. However, it's not guaranteed, and it's easy to disrupt. For starters, it works only if you're appending to a string, not prepending; this will be much slower: s = '' for x in range(10**6): s = str(x) + s And other Pythons may not optimize this. So don't depend on it. ChrisA From __peter__ at web.de Thu May 12 05:08:04 2016 From: __peter__ at web.de (Peter Otten) Date: Thu, 12 May 2016 11:08:04 +0200 Subject: More list building References: <9983c36a-7c06-45b6-9ab8-845eee5915d9@googlegroups.com> Message-ID: louis.a.russ at gmail.com wrote: > On Wednesday, May 11, 2016 at 1:22:09 PM UTC-4, DFS wrote: >> Have: >> p1 = ['Now', 'the', 'for', 'good'] >> p2 = ['is', 'time', 'all', 'men'] >> >> want >> [('Now','is','the','time'), ('for','all','good','men')] >> >> This works: >> >> p = [] >> for i in xrange(0,len(p1),2): >> p.insert(i,(p1[i],p2[i],p1[i+1],p2[i+1])) >> >> >> But it seems clunky. >> >> Better way(s)? >> >> Thanks > > Would this work for you? Or do you need something more general? > t = list(zip(p1,p2)) > p = [t[0]+t[1],t[2]+t[3]] This can be generalized as >>> t = iter(zip(p1, p2)) >>> [a + b for a, b in zip(t, t)] [('Now', 'is', 'the', 'time'), ('for', 'all', 'good', 'men')] (The iter() call can be omitted in Python 3.) From alister.ware at ntlworld.com Thu May 12 05:12:00 2016 From: alister.ware at ntlworld.com (alister) Date: Thu, 12 May 2016 09:12:00 GMT Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, 10 May 2016 19:40:02 -0400, DFS wrote: > Sometimes I try to be a funny smart-aleck and it doesn't work. this is the problem everyone is having with your post, you acknowledge that it doesn't work so why keep trying. I too can fall guilty of this behavior (I can be a bit condescending of one of our engineers calls for help & asks a question he should already know) but have learnt that when I am asking for help it is probably not a good idea to upset the person I am asking. > > But, I am dead serious about becoming a good Python developer, and I > truly appreciate all clp replies. > > >> and especially, don't call people names. > > Maybe. I'll always call a spade a spade. > > > > Note: Angelico is about the coolest last name I've heard in a long time. > Is it real? -- I can mend the break of day, heal a broken heart, and provide temporary relief to nymphomaniacs. -- Larry Lee From rosuav at gmail.com Thu May 12 05:23:14 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 12 May 2016 19:23:14 +1000 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, May 12, 2016 at 7:12 PM, alister wrote: > [presumably DFS wrote, but I didn't see] >> Note: Angelico is about the coolest last name I've heard in a long time. >> Is it real? It is, actually! It's a precious heirloom. It belonged to my father, the great Talldad; and it belonged to his father (my grandfather), and his father. It's been in the family for quite a while, and I'm very careful not to let it get damaged. ChrisA From ned at nedbatchelder.com Thu May 12 05:36:24 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 12 May 2016 02:36:24 -0700 (PDT) Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: References: <20160508183738.2f551043@bigbox.christie.dr> Message-ID: <27532248-2e60-4b3f-98b5-436d357a3338@googlegroups.com> On Thursday, May 12, 2016 at 4:24:55 AM UTC-4, srinivas devaki wrote: > On May 9, 2016 5:31 AM, "Tim Chase" wrote: > > > > then that's a bad code-smell (you get quadratic behavior as the > > strings are constantly resized), usually better replaced with > > > > I just want to point out that in Python s += str in loop is not giving > quadratic behavior. I don't know why but it runs fast. I'm very much > interested to know why it is so? > > In [3]: %%timeit > ...: s = '' > ...: for x in xrange(10**6): > ...: s += str(x) > ...: > 1 loop, best of 3: 383 ms per loop > > In [4]: %%timeit > s = '' > for x in xrange(10**6): > s = s + str(x) > ...: > 1 loop, best of 3: 383 ms per loop The CPython optimization depends on the string having only a single reference. A seemingly unrelated change to the code can change the performance significantly: In [1]: %%timeit ...: s = "" ...: for x in xrange(100000): ...: s = s + str(x) ...: 10 loops, best of 3: 33.5 ms per loop In [2]: %%timeit ...: s = t = "" ...: for x in xrange(100000): ...: s = t = s + str(x) ...: 1 loop, best of 3: 1.57 s per loop Be careful out there... --Ned. From jon+usenet at unequivocal.co.uk Thu May 12 09:39:13 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Thu, 12 May 2016 13:39:13 -0000 (UTC) Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> <57325CCE.9020206@stoneleaf.us> Message-ID: On 2016-05-10, DFS wrote: > This isn't a moderated group. That doesn't represent the whole truth. The Usenet group is unmoderated but is gatewayed to a mailing list, which of course can be moderated. So while you can't be moderated on the Usenet group, much of the potential audience for posts here are actually reading via the list, and hence won't see your posts if they are moderated out. (Personally I think this situation is more than a little silly and the group should be changed to moderated.) From rosuav at gmail.com Thu May 12 09:50:11 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 12 May 2016 23:50:11 +1000 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> <57325CCE.9020206@stoneleaf.us> Message-ID: On Thu, May 12, 2016 at 11:39 PM, Jon Ribbens wrote: > On 2016-05-10, DFS wrote: >> This isn't a moderated group. > > That doesn't represent the whole truth. The Usenet group is > unmoderated but is gatewayed to a mailing list, which of course can be > moderated. So while you can't be moderated on the Usenet group, much > of the potential audience for posts here are actually reading via the > list, and hence won't see your posts if they are moderated out. > > (Personally I think this situation is more than a little silly and > the group should be changed to moderated.) For example, I'm seeing this, but you're replying to something I didn't see. It's one of the many advantages of reading the mailing list rather than the newsgroup. ChrisA From torriem at gmail.com Thu May 12 10:39:17 2016 From: torriem at gmail.com (Michael Torrie) Date: Thu, 12 May 2016 08:39:17 -0600 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? In-Reply-To: References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> Message-ID: <57349595.6000702@gmail.com> On 05/12/2016 03:12 AM, alister wrote: > On Tue, 10 May 2016 19:40:02 -0400, DFS wrote: > >> Sometimes I try to be a funny smart-aleck and it doesn't work. > > this is the problem everyone is having with your post, you acknowledge > that it doesn't work so why keep trying. > > I too can fall guilty of this behavior (I can be a bit condescending of > one of our engineers calls for help & asks a question he should already > know) but have learnt that when I am asking for help it is probably not > a good idea to upset the person I am asking. >> Maybe. I'll always call a spade a spade. It's really sad to see folks like DFS hop on the list with apparent enthusiasm for Python and an excitement to learn, only to resort to name calling and walk away in a huff when folks ask them not to speak that way around here. I'm not sure why this is. I recall that the same thing happened not so long ago with another poster I recall. Overall I think list members are pretty patient with newbies. From steve at pearwood.info Thu May 12 11:06:18 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 13 May 2016 01:06:18 +1000 Subject: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement? References: <20160508183738.2f551043@bigbox.christie.dr> <27532248-2e60-4b3f-98b5-436d357a3338@googlegroups.com> Message-ID: <57349bec$0$1601$c3e8da3$5496439d@news.astraweb.com> On Thu, 12 May 2016 07:36 pm, Ned Batchelder wrote: > The CPython optimization depends on the string having only a single > reference. A seemingly unrelated change to the code can change the > performance significantly: > > In [1]: %%timeit > ...: s = "" > ...: for x in xrange(100000): > ...: s = s + str(x) > ...: > 10 loops, best of 3: 33.5 ms per loop > > In [2]: %%timeit > ...: s = t = "" > ...: for x in xrange(100000): > ...: s = t = s + str(x) > ...: > 1 loop, best of 3: 1.57 s per loop Nice demonstration! But it is actually even worse than that. The optimization depends on memory allocation details which means that some CPython interpreters cannot use it, depending on the operating system and version. Consequently, reliance on it can and has lead to embarrassments like this performance bug which only affected *some* Windows users. In 2009, Chris Withers asked for help debugging a problem where Python httplib was hundreds of times slower than other tools, like wget and Internet Explorer: https://mail.python.org/pipermail/python-dev/2009-August/091125.html A few weeks later, Simon Cross realised the problem was probably the quadratic behaviour of repeated string addition: https://mail.python.org/pipermail/python-dev/2009-September/091582.html leading to this quote from Antoine Pitrou: "Given differences between platforms in realloc() performance, it might be the reason why it goes unnoticed under Linux but degenerates under Windows." https://mail.python.org/pipermail/python-dev/2009-September/091583.html and Guido's comment: "Also agreed that this is an embarrassment." https://mail.python.org/pipermail/python-dev/2009-September/091592.html So beware of relying on the CPython string concatenation optimization in production code! Here's the tracker issue that added the optimization in the first place: http://bugs.python.org/issue980695 The feature was controversial at the time (and remains slightly so): https://mail.python.org/pipermail/python-dev/2004-August/046686.html My opinion is that it is great for interactive use at the Python prompt, but I would never use it in code I cared about. -- Steven From paul at mad-scientist.net Thu May 12 11:34:27 2016 From: paul at mad-scientist.net (Paul Smith) Date: Thu, 12 May 2016 11:34:27 -0400 Subject: Finding .so files without setting LD_LIBRARY_PATH In-Reply-To: References: <1463002764.2579.26.camel@mad-scientist.net> Message-ID: <1463067267.2579.47.camel@mad-scientist.net> On Thu, 2016-05-12 at 07:55 +0300, Jussi Piitulainen wrote: > eryk sun writes: > > > On Wed, May 11, 2016 at 10:39 PM, Paul Smith wrote: > > > Hi all. I have a locally-built version of Python (2.7.11) that I'm > > > copying around to different systems, running all different versions of > > > GNU/Linux. > > ... > > > What I'd like to do is have a way of setting the library path that > > > Python uses when it tries to load .so files (for example in the ssl > > > module which loads lib-dynload/_ssl.so which links to libssl.so and > > > libcrypto.so), WITHOUT setting LD_LIBRARY_PATH in the environment that > > > Python passes to its children. > > > > An ELF header can contain either an RPATH or a RUNPATH to extend the > > library search path at load time. Yes, I'm familiar with rpath. However I don't know how to set it for Python .so's that appear in the lib-dynload directory, including site -packages, which is where it needs to be ... I tried to go through the Modules/Setup etc. but to no avail :(. > There's a tool (GPLv3+) called patchelf that can set these in an ELF > binary: Aha! That's excellent! The only one I was aware of was chrpath which doesn't allow you to add an rpath if one doesn't exist, or add an rpath which is longer than the one that already exists. This will make things much simpler. Cheers! From kobsx4 at gmail.com Thu May 12 11:47:54 2016 From: kobsx4 at gmail.com (kobsx4 at gmail.com) Date: Thu, 12 May 2016 08:47:54 -0700 (PDT) Subject: Average calculation Program *need help* Message-ID: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> Hello all, I have been struggling with this code for 3 hours now and I'm still stumped. My problem is that when I run the following code: ------------------------------------------------------------------------------ #this function will get the total scores def getScores(totalScores, number): for counter in range(0, number): score = input('Enter their score: ') totalScores = totalScores + score while not (score >= 0 and score <= 100): print "Your score must be between 0 and 100." score = input('Enter their score: ') return totalScores ------------------------------------------------------------------------------ the program is supposed to find the average of two test scores and if one of the scores is out of the score range (0-100), an error message is displayed. The main problem with this is that when someone types in a number outside of the range, it'll ask them to enter two scores again, but ends up adding all of the scores together (including the invalid ones) and dividing by how many there are. Please help. From rustompmody at gmail.com Thu May 12 12:01:27 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Thu, 12 May 2016 09:01:27 -0700 (PDT) Subject: Average calculation Program *need help* In-Reply-To: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> References: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> Message-ID: <42f1887b-f846-4255-a73b-00593339222c@googlegroups.com> On Thursday, May 12, 2016 at 9:18:08 PM UTC+5:30, Jake Kobs wrote: > Hello all, I have been struggling with this code for 3 hours now and I'm still stumped. My problem is that when I run the following code: > ------------------------------------------------------------------------------ > #this function will get the total scores > def getScores(totalScores, number): > for counter in range(0, number): > score = input('Enter their score: ') > totalScores = totalScores + score > > while not (score >= 0 and score <= 100): > > print "Your score must be between 0 and 100." > score = input('Enter their score: ') > > > > return totalScores > ------------------------------------------------------------------------------ > the program is supposed to find the average of two test scores and if one of the scores is out of the score range (0-100), an error message is displayed. The main problem with this is that when someone types in a number outside of the range, it'll ask them to enter two scores again, but ends up adding all of the scores together (including the invalid ones) and dividing by how many there are. Please help. I suggest you conceptualize the problem into 3 sub-problems: - IO - average computation - validation (between 0 and 100) For starters ignore validation. So we only have IO + computation ie Input then computation then Output Now if your input needs to feed into a computation you need a *data-structure* I suggest you consider lists eg the 5 scores 4 2 1 3 5 would be represented as the list [4,2,1,3,5] Their average -- 3 -- is to be *computed* and then to be given back to the user -- ie *output* Does this help start you off? From ikorot01 at gmail.com Thu May 12 12:06:24 2016 From: ikorot01 at gmail.com (Igor Korot) Date: Thu, 12 May 2016 12:06:24 -0400 Subject: Average calculation Program *need help* In-Reply-To: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> References: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> Message-ID: Hi, On Thu, May 12, 2016 at 11:47 AM, wrote: > Hello all, I have been struggling with this code for 3 hours now and I'm still stumped. My problem is that when I run the following code: > ------------------------------------------------------------------------------ > #this function will get the total scores > def getScores(totalScores, number): > for counter in range(0, number): > score = input('Enter their score: ') > totalScores = totalScores + score > > while not (score >= 0 and score <= 100): > > print "Your score must be between 0 and 100." > score = input('Enter their score: ') > > > > return totalScores > ------------------------------------------------------------------------------ > the program is supposed to find the average of two test scores and if one of the scores is out of the score range (0-100), an error message is displayed. The main problem with this is that when someone types in a number outside of the range, it'll ask them to enter two scores again, but ends up adding all of the scores together (including the invalid ones) and dividing by how many there are. Please help. Can you solve this problem with just pen and paper? Do that and then translate the algorithm to python. Thank you. > -- > https://mail.python.org/mailman/listinfo/python-list From martin at linux-ip.net Thu May 12 12:13:58 2016 From: martin at linux-ip.net (Martin A. Brown) Date: Thu, 12 May 2016 09:13:58 -0700 Subject: Average calculation Program *need help* In-Reply-To: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> References: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> Message-ID: Greetings kobsx4, >Hello all, I have been struggling with this code for 3 hours now >and I'm still stumped. My problem is that when I run the following >code: >------------------------------------------------------------------------------ >#this function will get the total scores >def getScores(totalScores, number): > for counter in range(0, number): > score = input('Enter their score: ') > totalScores = totalScores + score > > while not (score >= 0 and score <= 100): > > print "Your score must be between 0 and 100." > score = input('Enter their score: ') > > > > return totalScores >------------------------------------------------------------------------------ >the program is supposed to find the average of two test scores and >if one of the scores is out of the score range (0-100), an error >message is displayed. The main problem with this is that when >someone types in a number outside of the range, it'll ask them to >enter two scores again, but ends up adding all of the scores >together (including the invalid ones) and dividing by how many >there are. Please help. Suggestion #1: -------------- When you are stuck on a small piece of code, set it aside (stop looking at it) and start over again; sometimes rewriting with different variable names and a clean slate helps to highlight the problem. Professional programmers will tell you that they are quite accustomed to 'throwing away' code. Don't be afraid to do it. (While you are still learning, you might want to keep the old chunk of code around to examine so that you can maybe figure out what you did wrong.) Suggestion #2: -------------- Put a print statement or two in the loop, so that you see how your variables are changing. For example, just before your 'while' line, maybe something like: print "score=%d totalScores=%d" % (score, totalScores,) Suggestion #3: -------------- Break the problem down even smaller (Rustom Mody appears to have beat me to the punch on that suggestion, so I'll just point to his email.) Hint #1: -------- What is the value of your variable totalScores each time through the loop? Does it ever get reset? Good luck with your degubbing! -Martin -- Martin A. Brown http://linux-ip.net/ From python at mrabarnett.plus.com Thu May 12 12:14:52 2016 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 12 May 2016 17:14:52 +0100 Subject: Average calculation Program *need help* In-Reply-To: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> References: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> Message-ID: <5c7d5f21-62ad-bb66-fbd2-4f4c50069881@mrabarnett.plus.com> On 2016-05-12 16:47, kobsx4 at gmail.com wrote: > Hello all, I have been struggling with this code for 3 hours now and I'm still stumped. My problem is that when I run the following code: > ------------------------------------------------------------------------------ > #this function will get the total scores > def getScores(totalScores, number): > for counter in range(0, number): > score = input('Enter their score: ') > totalScores = totalScores + score > > while not (score >= 0 and score <= 100): > > print "Your score must be between 0 and 100." > score = input('Enter their score: ') > > > > return totalScores > ------------------------------------------------------------------------------ > the program is supposed to find the average of two test scores and if one of the scores is out of the score range (0-100), an error message is displayed. The main problem with this is that when someone types in a number outside of the range, it'll ask them to enter two scores again, but ends up adding all of the scores together (including the invalid ones) and dividing by how many there are. Please help. > Basically what you want is: repeat as many times as there are scores you want: ask for the score while it's invalid: ask for it again add it to the total What you're actually doing is asking for the scores and adding them _before_ checking. Don't include something before you've checked that it's OK. From oysteijo at gmail.com Thu May 12 15:28:40 2016 From: oysteijo at gmail.com (oysteijo at gmail.com) Date: Thu, 12 May 2016 12:28:40 -0700 (PDT) Subject: Calling python from C with OpenMP Message-ID: <8224bdd2-9afe-487b-804b-f3b88dee2028@googlegroups.com> Hi, I have a framework written in C and I need to call Python from that framework. I have written the code, and it runs fine, however when I recompile with OpenMP enabled, I get segmentation faults and some times an error message: Fatal Python error: GC object already tracked I'm able to reconstruct the bug with this simple code: /* main.c */ #include #include int main() { wchar_t *program = Py_DecodeLocale( "My_problem", NULL ); if( !program ){ fprintf(stderr, "cannot decode python program name\n"); return -1; } Py_SetProgramName( program ); Py_Initialize(); PyObject *sys = PyImport_ImportModule("sys"); PyObject *path = PyObject_GetAttrString(sys, "path"); PyList_Append(path, PyUnicode_FromString(".")); PyObject *module_filename = PyUnicode_FromString( "multiplier" ); if(!module_filename){ printf("Cannot create python module multiplier.\n"); return -1; } PyObject *module = PyImport_Import( module_filename ); if(!module){ printf("Cannot create python.\n"); return -1; } PyObject *mult_obj = PyObject_CallMethod( module ,"multiplier", "i", 7); if(!mult_obj){ printf("Cannot create python multiplier class instance\n"); return -1; } Py_DECREF( module ); Py_DECREF( module_filename ); Py_DECREF( path ); Py_DECREF( sys ); /* Up to now we have actually done: * >>> import multiplier * >>> mult_obj = multipier.multiplier(7) */ /* lets try something like: * >>> for x in range(10): * ... printf(mult_obj.do_multiply(x)) */ #pragma omp parallel for for( int i = 0; i < 10; i++ ){ PyObject *ret = PyObject_CallMethod( mult_obj, "do_multiply", "i", i ); if( !ret ){ printf("Cannot call 'do_multiply'\n"); continue; } printf("The value calculated in Python was: %3d\n", (int) PyLong_AsLong(ret)); Py_DECREF(ret); } Py_DECREF(mult_obj); Py_Finalize(); return 0; } Compile with: gcc -std=gnu99 -O3 -Wall -Wextra -fopenmp `pkg-config --cflags --libs python3` -lgomp main.c -o main Then you need the python code: # multiplier.py class multiplier(object): def __init__(self, factor): self.factor = factor def do_multiply(self, x): return self.factor * x First question: Does my C code leak memory? Valgrind says it does, but the memory footprint of the executable is stable while looping? Second and most important question: When I run this code it sometimes segementation faults, and sometimes some threads run normal and some other threads says "Cannot call 'do_multiply'". Sometimes I get the message: Fatal Python error: GC object already tracked. And some times it even runs normally... I understand there is some kind of race condition here, where python tries to refer to some memory that has been already released. But how can I avoid this? What am I doing wrong? (or, less likely, is this a bug?) Maybe needless to say, but the code works when compiled w/o OpenMP. Using Python 3.5.1 Thanks, -?ystein From ben+python at benfinney.id.au Thu May 12 15:36:27 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 13 May 2016 05:36:27 +1000 Subject: Why online forums have bad behaviour (was: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement?) References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> <57349595.6000702@gmail.com> Message-ID: <85vb2jdq6c.fsf_-_@benfinney.id.au> Michael Torrie writes: > It's really sad to see folks like DFS hop on the list with apparent > enthusiasm for Python and an excitement to learn, only to resort to > name calling and walk away in a huff when folks ask them not to speak > that way around here. I'm not sure why this is. TL;DR: because we're all human, and human behaviour needs either immediate face-to-face feedback or social enforcement to correct selfishness and abrasiveness. Where face-to-face feedback is lacking, social enforcement needs to take more of the load. Many people have a false sense of entitlement to be caustic in dealing with others, and have no better response to a request that they tone it down than to escalate their bad behaviour. This behaviour is usually counteracted in face-to-face interaction, by being confronted with the immediate result on the other person: most people don't enjoy *seeing* other people become upset, so most people tend to work harder to be more polite in face-to-face discussion. On an internet forum, especially one with such low bandwidth as text, these feedback mechanisms are not sufficient (not immediate enough, and not informative enough) for the person to experience a link from their bad behaviour to the unpleasant consequences. This isn't a new problem. It's not new to the internet, and it certainly isn't new to humans. What is new, though, is that many online communities ? the Python community specifically ? have decided we are not going to tolerate anti-social behaviour, and we have also enacted policies to enforce that decision. We'll always have some anti-social actors, and bad actions by otherwise good actors. Many of them when confronted will respond with petulance and name-calling and bullying and other schoolyard reflexes. We have to be consistent in rejecting such behaviour from our community. -- \ ?Writing a book is like washing an elephant: there no good | `\ place to begin or end, and it's hard to keep track of what | _o__) you've already covered.? ?anonymous | Ben Finney From tshortik at gmx.de Thu May 12 16:51:43 2016 From: tshortik at gmx.de (=?UTF-8?Q?Dirk_B=c3=a4chle?=) Date: Thu, 12 May 2016 22:51:43 +0200 Subject: Design: Idiom for classes and methods that are customizable by the user? Message-ID: <5734ECDF.7070006@gmx.de> Hi there, I'm one of the SCons (http://www.scons.org) developers and am working on a partial redesign of our architecture. I'd like to come up with a decent proposal about how to extend and rewrite our code base, such that it's easier for a user to overload and replace some parts of the functionality. Concrete example ================ We're a build system and all the code is written in Python. Even more so, our build scripts (build description files) are Python scripts...allowing the user to have the full power of this beautiful language at his/her fingertips whenever it's needed. The single build steps are file-oriented, meaning that our "targets" are always either files or directories (simplified). Each of these files and directories is tracked by an instance of the "Node" class, which is able to provide infos like: - Did this (source) file change? - Which are my children (meaning, on which other Nodes do I depend)? - Which commands, or "Actions", are required to build this Node (could be a Program or a Library, for example)? We then have a "Taskmaster" which operates on the set of known "Node"s (think: "Strategy"). It loops over the Nodes and tries to find one that isn't up-to-date. If all its dependencies are met (=all its children are up-to-date), the Node is ready to get built (again, largely simplified). What happens now and then is, that users are unhappy with the way this Taskmaster proceeds. One peculiar detail is, that our "default" Taskmaster always deletes the old target file before re-building it...and in special situations this may be seen as unwanted. So it would be good to offer a set of Taskmasters to the user, where he can choose from. Even better would be, if the user could add a Taskmaster of his own (probably derived from the "original") and activate it...from a build description file (=Python script), so without touching the core sources. To put it shortly, the user should be able to slip a new Taskmaster under the covers of SCons...and it should behave as if it would've been built-in. My current approach =================== I'm currently following the "Factory" pattern (more or less) as I know it from C++ and similar languages. In the Taskmaster module I have a dictionary: # The built-in Taskmasters types = {'default' : DefaultTaskmaster, 'noclean' : NocleanTaskmaster} def create(key, targets, top, node): """ Simple factory for creating an actual Taskmaster, based on the given key. """ if key in types: return types[key](targets, top, node) return DefaultTaskmaster(targets, top, node) def add(key, taskm_class): """ Register the given Taskmaster class, if its key doesn't exist yet. """ if not key in types: types[key] = taskm_class with two supporting functions. I'm leaving out all the boilerplate stuff here, like parsing command-line options for the to-be-instantiated Taskmaster type and so on. But this is the core idea so far, simple and certainly "pythonic" to some degree... My questions ============ - Is this a good approach, that I could use for other parts of the architecture as well, e.g. the Node class mentioned above? - Are there other options that have stood the test of time under operational conditions? If yes, I'd be interested to get links and pointers to the corresponding projects (I'm prepared to read stuff and do further investigations on my own). When talking about "other options" I'm mainly thinking in the direction of "plugins" (Yapsy?)...not sure whether this idea could really fly. Some criteria ============= - The approach should work under 2.7.x (and higher) and Python 3.x as well. We're currently rewriting our code (runs only under Python 2.7 so far) to a common codebase, using "futurize". - It should be applicable to classes and simple methods within arbitrary modules. One specialized scheme for classes and methods each would be okay though. - Heavy-weight mechanism are a no-go...in large build projects we have to initialize 800k instances of the Node class, and more. - Some of our classes, especially Node, use the "slots" mechanism in order to save memory (and yes, it's necessary! ;) ). So, certain techniques (=metaclasses?) are probably not compatible with that... Your advice and input on these thoughts are welcome. If required, I can provide more details in a certain area...but I didn't want to make this initial email too long. Thanks a lot in advance for your answers and best regards, Dirk Baechle From marko at pacujo.net Thu May 12 18:01:53 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Fri, 13 May 2016 01:01:53 +0300 Subject: Design: Idiom for classes and methods that are customizable by the user? References: <5734ECDF.7070006@gmx.de> Message-ID: <87futnaqb2.fsf@elektro.pacujo.net> Dirk B?chle : > I'm one of the SCons (http://www.scons.org) developers I take the opportunity to thank you and your colleagues for the wonderful building tool that I have used professionally to great success since 2003. > We're a build system and all the code is written in Python. Even more > so, our build scripts (build description files) are Python > scripts...allowing the user to have the full power of this beautiful > language at his/her fingertips whenever it's needed. It is indeed great to have Python available when needed. However, I have striven not to need it. SConscript files should be understood by casual maintainers who aren't necessarily fluent with Python. > To put it shortly, the user should be able to slip a new Taskmaster > under the covers of SCons...and it should behave as if it would've > been built-in. I have tried to resist the temptation, successfully so far. > # The built-in Taskmasters > types = {'default' : DefaultTaskmaster, > 'noclean' : NocleanTaskmaster} > > def create(key, targets, top, node): > """ Simple factory for creating an actual Taskmaster, > based on the given key. > """ > if key in types: > return types[key](targets, top, node) > > return DefaultTaskmaster(targets, top, node) > > def add(key, taskm_class): > """ Register the given Taskmaster class, if its key doesn't > exist yet. > """ > if not key in types: > types[key] = taskm_class > > [...] > > - Is this a good approach As an approach, that looks ok. The code looks a bit too lenient, though, which can easily surprise and baffle the user. I think it would be better to fail: ======================================================================== class AlreadyExistsError(Exception): pass types = {} def create(key, targets, top, node): """ Simple factory for creating an actual Taskmaster, based on the given key. """ return types[key](targets, top, node) def add(key, taskm_class): """ Register the given Taskmaster class, if its key doesn't exist yet. """ if key in types: raise AlreadyExistsError("Task manager '%s' already exists" % key) types[key] = taskm_class # The built-in Taskmasters add('default', DefaultTaskmaster) add('noclean', NocleanTaskmaster) ======================================================================== > - Are there other options that have stood the test of time under > operational conditions? If yes, I'd be interested to get links and > pointers to the corresponding projects (I'm prepared to read stuff and > do further investigations on my own). Based on the given information, I can't give recommendations. However, in general, I'm suspicious of the factory pattern. Whenever possible, I prefer explicit construction. For example, why do you need a key? Couldn't you simply pass the task master class as an argument? Marko From tshortik at gmx.de Thu May 12 18:42:17 2016 From: tshortik at gmx.de (=?UTF-8?Q?Dirk_B=c3=a4chle?=) Date: Fri, 13 May 2016 00:42:17 +0200 Subject: Design: Idiom for classes and methods that are customizable by the user? In-Reply-To: <87futnaqb2.fsf@elektro.pacujo.net> References: <5734ECDF.7070006@gmx.de> <87futnaqb2.fsf@elektro.pacujo.net> Message-ID: <573506C9.3070206@gmx.de> Hi Marko, thank you very much for your quick reply. On 13.05.2016 00:01, Marko Rauhamaa wrote: >> >> [...] >> >> - Is this a good approach > > As an approach, that looks ok. The code looks a bit too lenient, though, > which can easily surprise and baffle the user. I think it would be > better to fail: > > ======================================================================== > class AlreadyExistsError(Exception): > pass > > [...] > Sure, I'll have exceptions in mind for the final implementation. > > [...] > > For example, why do you need a key? Couldn't you simply pass the task > master class as an argument? > The idea behind this is, to be able to select classes by giving a parameter on the command-line. So at some point a translation from a given "key" to its actual class has to happen, I guess. Regards, Dirk From sturla.molden at gmail.com Thu May 12 20:04:23 2016 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 13 May 2016 00:04:23 +0000 (UTC) Subject: Calling python from C with OpenMP References: <8224bdd2-9afe-487b-804b-f3b88dee2028@googlegroups.com> Message-ID: <1288606789484790413.405054sturla.molden-gmail.com@news.gmane.org> wrote: > Second and most important question: When I run this code it sometimes > segementation faults, and sometimes some threads run normal and some > other threads says "Cannot call 'do_multiply'". Sometimes I get the > message: Fatal Python error: GC object already tracked. And some times it > even runs normally... > I understand there is some kind of race condition here, where python > tries to refer to some memory that has been already released. But how can > I avoid this? What am I doing wrong? (or, less likely, is this a bug?) You must own the GIL before you can safely use the Python C API, object creation and refcounting in particular. Use the "Simplified GIL API" to grab the GIL and release it when you are done. From jsf80238 at gmail.com Thu May 12 23:05:57 2016 From: jsf80238 at gmail.com (Jason Friedman) Date: Thu, 12 May 2016 21:05:57 -0600 Subject: Why online forums have bad behaviour (was: Steve D'Aprano, you're the "master". What's wrong with this concatenation statement?) In-Reply-To: <85vb2jdq6c.fsf_-_@benfinney.id.au> References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> <57349595.6000702@gmail.com> <85vb2jdq6c.fsf_-_@benfinney.id.au> Message-ID: > TL;DR: because we're all human, and human behaviour needs either > immediate face-to-face feedback or social enforcement to correct > selfishness and abrasiveness. Where face-to-face feedback is lacking, > social enforcement needs to take more of the load. > > > Many people have a false sense of entitlement to be caustic in dealing > with others, and have no better response to a request that they tone it > down than to escalate their bad behaviour. > > This behaviour is usually counteracted in face-to-face interaction, by > being confronted with the immediate result on the other person: most > people don't enjoy *seeing* other people become upset, so most people > tend to work harder to be more polite in face-to-face discussion. > > On an internet forum, especially one with such low bandwidth as text, > these feedback mechanisms are not sufficient (not immediate enough, and > not informative enough) for the person to experience a link from their > bad behaviour to the unpleasant consequences. > > > This isn't a new problem. It's not new to the internet, and it certainly > isn't new to humans. > > What is new, though, is that many online communities ? the Python > community specifically ? have decided we are not going to tolerate > anti-social behaviour, and we have also enacted policies to enforce that > decision. > > We'll always have some anti-social actors, and bad actions by otherwise > good actors. Many of them when confronted will respond with petulance > and name-calling and bullying and other schoolyard reflexes. We have to > be consistent in rejecting such behaviour from our community. Well said. From kobsx4 at gmail.com Fri May 13 00:22:00 2016 From: kobsx4 at gmail.com (Jake Kobs) Date: Thu, 12 May 2016 21:22:00 -0700 (PDT) Subject: Average calculation Program *need help* In-Reply-To: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> References: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> Message-ID: On Thursday, May 12, 2016 at 10:48:08 AM UTC-5, Jake Kobs wrote: > Hello all, I have been struggling with this code for 3 hours now and I'm still stumped. My problem is that when I run the following code: > ------------------------------------------------------------------------------ > #this function will get the total scores > def getScores(totalScores, number): > for counter in range(0, number): > score = input('Enter their score: ') > totalScores = totalScores + score > > while not (score >= 0 and score <= 100): > > print "Your score must be between 0 and 100." > score = input('Enter their score: ') > > > > return totalScores > ------------------------------------------------------------------------------ > the program is supposed to find the average of two test scores and if one of the scores is out of the score range (0-100), an error message is displayed. The main problem with this is that when someone types in a number outside of the range, it'll ask them to enter two scores again, but ends up adding all of the scores together (including the invalid ones) and dividing by how many there are. Please help. I still can't get it. Someone please tell me lol. I have done everything I can and still I get bad answers. From ben+python at benfinney.id.au Fri May 13 00:55:13 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 13 May 2016 14:55:13 +1000 Subject: Average calculation Program *need help* References: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> Message-ID: <85inyieevi.fsf@benfinney.id.au> Jake Kobs writes: > I still can't get it. Someone please tell me lol. I have done > everything I can and still I get bad answers. You may want to join our dedicated beginner tutoring forum , which specialises in collaborative teaching of the fundamentals of Python. -- \ ?If we don't believe in freedom of expression for people we | `\ despise, we don't believe in it at all.? ?Noam Chomsky, | _o__) 1992-11-25 | Ben Finney From torriem at gmail.com Fri May 13 00:57:10 2016 From: torriem at gmail.com (Michael Torrie) Date: Thu, 12 May 2016 22:57:10 -0600 Subject: Average calculation Program *need help* In-Reply-To: References: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> Message-ID: <57355EA6.50905@gmail.com> On 05/12/2016 10:22 PM, Jake Kobs wrote: > On Thursday, May 12, 2016 at 10:48:08 AM UTC-5, Jake Kobs wrote: >> Hello all, I have been struggling with this code for 3 hours now and I'm still stumped. My problem is that when I run the following code: >> ------------------------------------------------------------------------------ >> #this function will get the total scores >> def getScores(totalScores, number): >> for counter in range(0, number): >> score = input('Enter their score: ') >> totalScores = totalScores + score >> >> while not (score >= 0 and score <= 100): >> >> print "Your score must be between 0 and 100." >> score = input('Enter their score: ') >> >> >> >> return totalScores >> ------------------------------------------------------------------------------ >> the program is supposed to find the average of two test scores and if one of the scores is out of the score range (0-100), an error message is displayed. The main problem with this is that when someone types in a number outside of the range, it'll ask them to enter two scores again, but ends up adding all of the scores together (including the invalid ones) and dividing by how many there are. Please help. > > I still can't get it. Someone please tell me lol. I have done everything I can and still I get bad answers. Tell us what you've done, precisely. We're here to help you learn, not give you the answers. One thing that strikes me is that your while loop doesn't appear to be indented properly. You have it running after all the scores have been inputted in the for loop, but I suspect this isn't what you want. You want the while loop to occur each time through the for loop. Do you know how to move this while loop to be inside the for loop? From kobsx4 at gmail.com Fri May 13 01:03:40 2016 From: kobsx4 at gmail.com (Jake Kobs) Date: Thu, 12 May 2016 22:03:40 -0700 (PDT) Subject: Average calculation Program *need help* In-Reply-To: References: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> <57355EA6.50905@gmail.com> Message-ID: <26884493-24d4-4d3d-b7d8-816f043b28b7@googlegroups.com> On Thursday, May 12, 2016 at 11:57:28 PM UTC-5, Michael Torrie wrote: > On 05/12/2016 10:22 PM, Jake Kobs wrote: > > On Thursday, May 12, 2016 at 10:48:08 AM UTC-5, Jake Kobs wrote: > >> Hello all, I have been struggling with this code for 3 hours now and I'm still stumped. My problem is that when I run the following code: > >> ------------------------------------------------------------------------------ > >> #this function will get the total scores > >> def getScores(totalScores, number): > >> for counter in range(0, number): > >> score = input('Enter their score: ') > >> totalScores = totalScores + score > >> > >> while not (score >= 0 and score <= 100): > >> > >> print "Your score must be between 0 and 100." > >> score = input('Enter their score: ') > >> > >> > >> > >> return totalScores > >> ------------------------------------------------------------------------------ > >> the program is supposed to find the average of two test scores and if one of the scores is out of the score range (0-100), an error message is displayed. The main problem with this is that when someone types in a number outside of the range, it'll ask them to enter two scores again, but ends up adding all of the scores together (including the invalid ones) and dividing by how many there are. Please help. > > > > I still can't get it. Someone please tell me lol. I have done everything I can and still I get bad answers. > > Tell us what you've done, precisely. We're here to help you learn, not > give you the answers. > > One thing that strikes me is that your while loop doesn't appear to be > indented properly. You have it running after all the scores have been > inputted in the for loop, but I suspect this isn't what you want. You > want the while loop to occur each time through the for loop. Do you > know how to move this while loop to be inside the for loop? Im not sure how to move it inside the for loop. I've been working on this small problem for like 4 hours lol. From ben+python at benfinney.id.au Fri May 13 01:07:04 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 13 May 2016 15:07:04 +1000 Subject: Distinction between =?utf-8?B?4oCcY2xhc3PigJ0=?= and =?utf-8?B?4oCcdHlwZeKAnQ==?= Message-ID: <85eg96eebr.fsf@benfinney.id.au> Howdy all, Ever since Python's much-celebrated Grand Unification of classes and types, I have used those terms interchangeably: every class is a type, and every type is a class. That may be an unwise conflation. With the recent rise of optional type annotation in Python 3, more people are speaking about the important distinction between a class and a type. This recent message from GvR, discussing a relevant PEP, advocates keeping them separate: PEP 484 [?] tries to make a clear terminological between classes (the things you have at runtime) and types (the things that type checkers care about). There's a big overlap because most classes are also types -- but not the other way around! E.g. Any is a type but not a class (you can neither inherit from Any nor instantiate it), and the same is true for unions and type variables. [?] As a Bear of Little Brain, this leaves me clueless. What is the distinction Guido alludes to, and how are Python classes not also types? And why is this distinction important, and who moved my cheesecake? -- \ ?It is the responsibility of intellectuals to tell the truth | `\ and to expose lies.? ?Noam Chomsky, 1967-02-23 | _o__) | Ben Finney From rustompmody at gmail.com Fri May 13 01:21:31 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Thu, 12 May 2016 22:21:31 -0700 (PDT) Subject: =?UTF-8?B?UmU6IERpc3RpbmN0aW9uIGJldHdlZW4g4oCcY2xhc3PigJ0gYW5kIOKAnHR5cGXigJ0=?= In-Reply-To: References: <85eg96eebr.fsf@benfinney.id.au> Message-ID: <96254278-f3b9-4e12-a5b6-dbf29e20d386@googlegroups.com> On Friday, May 13, 2016 at 10:37:34 AM UTC+5:30, Ben Finney wrote: > Howdy all, > > Ever since Python's much-celebrated Grand Unification of classes and > types, I have used those terms interchangeably: every class is a type, > and every type is a class. > > That may be an unwise conflation. With the recent rise of optional type > annotation in Python 3, more people are speaking about the important > distinction between a class and a type. > > This recent message from GvR, discussing a relevant PEP, advocates > keeping them separate: > > PEP 484 [?] tries to make a clear terminological between classes > (the things you have at runtime) and types (the things that type > checkers care about). > > There's a big overlap because most classes are also types -- but not > the other way around! E.g. Any is a type but not a class (you can > neither inherit from Any nor instantiate it), and the same is true > for unions and type variables. [?] > > > > As a Bear of Little Brain, this leaves me clueless. What is the > distinction Guido alludes to, and how are Python classes not also types? > > And why is this distinction important, and who moved my cheesecake? I cannot speak for Guido. However there is a well-known confusion here, most recently spelt out in Bob Harper's epochal book: http://www.cs.cmu.edu/~rwh/pfpl.html If types are compile time things then they are part of the *syntax* of the language If types are runtime things they are some kind of tags attached to objects *at runtime* When a language like python with a clear historical orientation towards the second starts embracing the first, it is natural that people start getting confused -- which I suspect includes Guido. From rosuav at gmail.com Fri May 13 01:23:28 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 13 May 2016 15:23:28 +1000 Subject: =?UTF-8?B?UmU6IERpc3RpbmN0aW9uIGJldHdlZW4g4oCcY2xhc3PigJ0gYW5kIOKAnHR5cGXigJ0=?= In-Reply-To: <85eg96eebr.fsf@benfinney.id.au> References: <85eg96eebr.fsf@benfinney.id.au> Message-ID: On Fri, May 13, 2016 at 3:07 PM, Ben Finney wrote: > This recent message from GvR, discussing a relevant PEP, advocates > keeping them separate: > > PEP 484 [?] tries to make a clear terminological between classes > (the things you have at runtime) and types (the things that type > checkers care about). > > There's a big overlap because most classes are also types -- but not > the other way around! E.g. Any is a type but not a class (you can > neither inherit from Any nor instantiate it), and the same is true > for unions and type variables. [?] > > > > As a Bear of Little Brain, this leaves me clueless. What is the > distinction Guido alludes to, and how are Python classes not also types? The difference, as I understand it, is that types are abstract and classes are concrete. The problem is that 'type' is an actual concrete thing, so terminology is messy; but the examples are pretty clear - you can't instantiate Any the way you can instantiate str or int. There's a similar difference with ABCs like Sequence - you can't construct a Sequence, but you can document that you expect to be passed one. Maybe the solution is to have a new word for "things a type checker looks for"... except that that really does want to be, uhh, "types". Or maybe the solution is to have a new word for "things you can construct with the class keyword"... except that that that's "types" too, because they're all subclasses of 'type'. > And why is this distinction important, and who moved my cheesecake? Because it wouldn't make sense if you asked who moved your bacon. Mmm. Bacon. With cheesecake for afters. ChrisA From torriem at gmail.com Fri May 13 01:29:47 2016 From: torriem at gmail.com (Michael Torrie) Date: Thu, 12 May 2016 23:29:47 -0600 Subject: Average calculation Program *need help* In-Reply-To: <26884493-24d4-4d3d-b7d8-816f043b28b7@googlegroups.com> References: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> <57355EA6.50905@gmail.com> <26884493-24d4-4d3d-b7d8-816f043b28b7@googlegroups.com> Message-ID: <5735664B.8000506@gmail.com> On 05/12/2016 11:03 PM, Jake Kobs wrote: > Im not sure how to move it inside the for loop. I've been working on > this small problem for like 4 hours lol. I'm sorry it's so frustrating. Sounds like you haven't got down some of the most basic fundamentals yet. In Python, things that should happen one after another are placed at the same indent level on the screen. For example: if something: do_this() then_this() then_that() Everything indented in from the start of the if statement is in a block and only happens if the if statement condition is true. If there're statements at the same level as the if, then they happen *after* the if and it's block. In other words, indentation is what tells Python where you want things to run. If you want to move the while loop inside the for loop, you have to adjust it's indentation accordingly (and the indentation in it's own block). This is a big gotcha for people unfamiliar with programming in Python, and unfamiliar with programming in general. In other languages, indents don't have to be a certain way, as long as you have the write statements to close off the loop. However, Python's method forces you to think like a programmer and to lay things out on the screen in a logical fashion, like a writer's outline. If you're still stuck, you will probably want to sit down with your teacher and have him or her go over this with you. This is important basic stuff you need to have clear in your mind to program computers. From kobsx4 at gmail.com Fri May 13 01:46:51 2016 From: kobsx4 at gmail.com (Jake Kobs) Date: Thu, 12 May 2016 22:46:51 -0700 (PDT) Subject: Average calculation Program *need help* In-Reply-To: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> References: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> Message-ID: <25510084-74ff-4f7f-b99f-bb29bc675899@googlegroups.com> Thank you for the help..I think I'm getting closer, but I feel like after they enter an invalid number, it should reset the invalid number(s) somehow. Here's my updated code: ------------------------------------------------------------------------------ #this function will get the total scores def getScores(totalScores, number): for counter in range(0, number): score = input('Enter their score: ') if (score < 100 and score > 0): totalScores = totalScores + score while (score > 100 or score < 0): print "Your scores must be between 0 and 100." score = input('Enter their score: ') score = input('Enter their score: ') totalScores = totalScores + score return totalScores ---------------------------------------------------------------------------- From marko at pacujo.net Fri May 13 01:53:56 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Fri, 13 May 2016 08:53:56 +0300 Subject: Design: Idiom for classes and methods that are customizable by the user? References: <5734ECDF.7070006@gmx.de> <87futnaqb2.fsf@elektro.pacujo.net> <573506C9.3070206@gmx.de> Message-ID: <87shxmh5aj.fsf@elektro.pacujo.net> Dirk B?chle : >> For example, why do you need a key? Couldn't you simply pass the task >> master class as an argument? > > The idea behind this is, to be able to select classes by giving a > parameter on the command-line. So at some point a translation from a > given "key" to its actual class has to happen, I guess. I see. So are the task masters interchangeable? I would have imagined they would modify the default operational semantics. If so, the task manager should be fixed programmatically in the SConstruct file. Marko From greg.ewing at canterbury.ac.nz Fri May 13 02:35:06 2016 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Fri, 13 May 2016 18:35:06 +1200 Subject: Design: Idiom for classes and methods that are customizable by the user? In-Reply-To: References: <5734ECDF.7070006@gmx.de> Message-ID: Dirk B?chle wrote: > What happens now and then is, that users are unhappy with the way this > Taskmaster proceeds. One peculiar detail is, that our "default" > Taskmaster always deletes the old target file before re-building > it...and in special situations this may be seen as unwanted. I'm not convinced that replacing the Taskmaster for the entire build is the right way to address this kind of thing. What if you want to override that behaviour for some targets but not others? A single object that controls behaviour globally makes things inflexible. (Distutils suffers from this problem with its "command classes" that can only be overridden in their entirety or not at all.) It seems to me that it would be better for the Taskmaster to delegate as much behaviour to the Nodes as it can, and provide a way of designating which Node classes to use for particular targets. I'd even suggest that *all* of the build logic should be in the Nodes, and the Taskmaster class shouldn't exist at all. The top level logic should just tell the final Nodes to bring themselves up to date, and they recursively do likewise for their dependent nodes. > I'm currently following the "Factory" pattern (more or less) as I know > it from C++ and similar languages. This statement sets off alarm bells for me. If you're using some design pattern in Python just because you learned to do it that way in C++/Java/whatever, you're probably making it more complicated than it needs to be. -- Greg From no.email at nospam.invalid Fri May 13 02:42:34 2016 From: no.email at nospam.invalid (Paul Rubin) Date: Thu, 12 May 2016 23:42:34 -0700 Subject: Distinction between =?utf-8?B?4oCcY2xhc3PigJ0=?= and =?utf-8?B?4oCcdHlwZeKAnQ==?= References: <85eg96eebr.fsf@benfinney.id.au> Message-ID: <87shxm4fxh.fsf@jester.gateway.pace.com> Ben Finney writes: > There's a big overlap because most classes are also types -- but not > the other way around! E.g. Any is a type but not a class (you can > neither inherit from Any nor instantiate it), and the same is true > for unions and type variables. [?] > As a Bear of Little Brain, this leaves me clueless. What is the > distinction Guido alludes to, and how are Python classes not also types? I thought I understood Guido's explanation but maybe I missed something. Let C be a class, maybe defined by a class statement or maybe a builtin like "int". You can make an instance of C the usual way: x = C() And you can have a type annotation that says function f expects an arg that is an instance of C: def f(x : C) -> int: ... You might alternatively write a function whose arg must be either an int or a string: def f(s : Union[int, str]) -> int : ... or (I think, I haven't tried it) you can equivalently bind that type to a variable: T = Union[int, str] def f(s : T) -> int : ... The point here is that T is a type but it is not a class. You can't instantiate T by saying x = T() and expecting to get back some value that is (indeterminately) an int or a string. That is, there's stuff (like instantiation) that you can do with types that happen to be classes, but there are also types that aren't classes. From nick.a.sarbicki at gmail.com Fri May 13 03:10:17 2016 From: nick.a.sarbicki at gmail.com (Nick Sarbicki) Date: Fri, 13 May 2016 07:10:17 +0000 Subject: Average calculation Program *need help* In-Reply-To: <25510084-74ff-4f7f-b99f-bb29bc675899@googlegroups.com> References: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> <25510084-74ff-4f7f-b99f-bb29bc675899@googlegroups.com> Message-ID: On Fri, 13 May 2016, 06:52 Jake Kobs, wrote: > Thank you for the help..I think I'm getting closer, but I feel like after > they enter an invalid number, it should reset the invalid number(s) > somehow. Here's my updated code: > > ------------------------------------------------------------------------------ > #this function will get the total scores > def getScores(totalScores, number): > for counter in range(0, number): > score = input('Enter their score: ') > if (score < 100 and score > 0): > totalScores = totalScores + score > while (score > 100 or score < 0): > > print "Your scores must be between 0 and 100." > score = input('Enter their score: ') > score = input('Enter their score: ') > totalScores = totalScores + score > > > return totalScores > > ---------------------------------------------------------------------------- > -- > You're starting to get it which is good. The if is a good idea to begin with, but your while loop still executes after the for loop finishes as it is not properly indented. As Michael said, indentation indicates what statement has ownership of a block. Currently your while loop has it's own block outside the for loop. This means it gets executed after your for loop has finished looping through your range. You want the while to be held within the for loop so it gets executed immediately after each input statement. Think. Your if is inside the for loop and gets executed after every input. Your input is inside the for loop. Your while and it's contents are not in the for loop and get executed after the for loop finishes. The only difference that matters here is indentation. Try solve that bit first. > From dieter at handshake.de Fri May 13 03:21:56 2016 From: dieter at handshake.de (dieter) Date: Fri, 13 May 2016 09:21:56 +0200 Subject: Design: Idiom for classes and methods that are customizable by the user? References: <5734ECDF.7070006@gmx.de> Message-ID: <87lh3eifsb.fsf@handshake.de> Dirk B?chle writes: > ... > My questions > ============ > > - Is this a good approach, that I could use for other parts of the architecture as well, e.g. the Node class mentioned above? You might look at the "adpater" pattern. It is heavily used in Zope - and there looks something like: * components are abstracted by interfaces (--> "zope.interface") * where flexibility is needed, the code looks like component = queryAdapter(, interface, default=...) or component = queryUtility(interface, default=...) to get a component implementing "interface". (utilities are in some way adapters for "None", i.e. those independent of a context) * there is an adapter registry, its content is specified either via an XML specification file or via Python code From greg.ewing at canterbury.ac.nz Fri May 13 04:24:44 2016 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Fri, 13 May 2016 20:24:44 +1200 Subject: Distinction between =?windows-1252?Q?=93class=94_and__?= =?windows-1252?Q?_=93type=94?= In-Reply-To: <87shxm4fxh.fsf@jester.gateway.pace.com> References: <85eg96eebr.fsf@benfinney.id.au> <87shxm4fxh.fsf@jester.gateway.pace.com> Message-ID: Paul Rubin wrote: > You can't instantiate T by saying > > x = T() > > and expecting to get back some value that is (indeterminately) an int or > a string. Unless it's Python 6000 running on a quantum computer... -- Greg From Christopher.Amoroso at mail.citytech.cuny.edu Fri May 13 09:05:00 2016 From: Christopher.Amoroso at mail.citytech.cuny.edu (Christopher.Amoroso at mail.citytech.cuny.edu) Date: Fri, 13 May 2016 13:05:00 +0000 Subject: Fw: Question about issue with opening Python In-Reply-To: References: , Message-ID: ________________________________ From: Christopher.Amoroso at mail.citytech.cuny.edu Sent: Thursday, May 12, 2016 7:35 PM To: python-list at python.org Subject: Fw: Question about issue with opening Python I downloaded an older version of Python and for about an hour it was working, but started to get the same error message I received when using the latest version of Python. ________________________________ From: Christopher.Amoroso at mail.citytech.cuny.edu Sent: Thursday, May 12, 2016 10:36 AM To: python-list at python.org Subject: Question about issue with opening Python Hello, I downloaded Python a few days ago and was using without any issues until yesterday. Every time I try to open the IDLE I get an error message that you can see the attachment. I tried to reinstall it but still get the same message. If there's any way you could help it would be greatly appreciated. Thank you. -Christopher From steve at pearwood.info Fri May 13 09:37:16 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 13 May 2016 23:37:16 +1000 Subject: Fw: Question about issue with opening Python References: Message-ID: <5735d88d$0$1620$c3e8da3$5496439d@news.astraweb.com> On Fri, 13 May 2016 11:05 pm, Christopher.Amoroso at mail.citytech.cuny.edu wrote: > I downloaded an older version of Python and for about an hour it was > working, but started to get the same error message I received when using > the latest version of Python. Do you have a question, or are you just sharing? Is the error message a secret, or should we guess what it is? My guess is: +++ OUT OF CUCUMBERS ERROR +++ REBOOT UNIVERSE +++ Am I close? If not, then you should copy and paste the error message. Please don't send a screen shot: this is a text only newsgroup, and attachments will be automatically deleted. Also, some people reading these messages may be fully or partially blind, and while their screen reader software will work fine with text, a screen shot is useless for them. If you absolutely must use a screen shot, because the error cannot be copied and is too long to retype, then please post it as a link to imgurl: https://imgur.com/ -- Steven From bgailer at gmail.com Fri May 13 09:39:04 2016 From: bgailer at gmail.com (Bob Gailer) Date: Fri, 13 May 2016 09:39:04 -0400 Subject: Fw: Question about issue with opening Python In-Reply-To: References: Message-ID: On May 13, 2016 9:22 AM, "Christopher.Amoroso at mail.citytech.cuny.edu" < Christopher.Amoroso at mail.citytech.cuny.edu> wrote: > > > > > > ________________________________ > From: Christopher.Amoroso at mail.citytech.cuny.edu > Sent: Thursday, May 12, 2016 7:35 PM > To: python-list at python.org > Subject: Fw: Question about issue with opening Python > > > > I downloaded an older version of Python and for about an hour it was working, but started to get the same error message I received when using the latest version of Python. This mailing list does not take attachments. Please copy and paste the error into a reply. From torriem at gmail.com Fri May 13 09:43:29 2016 From: torriem at gmail.com (Michael Torrie) Date: Fri, 13 May 2016 07:43:29 -0600 Subject: Fw: Question about issue with opening Python In-Reply-To: References: Message-ID: <5735DA01.9080707@gmail.com> On 05/13/2016 07:05 AM, Christopher.Amoroso at mail.citytech.cuny.edu wrote: > I downloaded an older version of Python and for about an hour it was > working, but started to get the same error message I received when > using the latest version of Python. You'll have to tells us what the error was that you are seeing. Attachments are not allowed on this list, so you'll have to type it, or if you can, copy and paste the exact error. From torriem at gmail.com Fri May 13 09:52:38 2016 From: torriem at gmail.com (Michael Torrie) Date: Fri, 13 May 2016 07:52:38 -0600 Subject: Fw: Question about issue with opening Python In-Reply-To: References: <5735DA01.9080707@gmail.com> Message-ID: <5735DC26.4040806@gmail.com> On 05/13/2016 07:47 AM, Christopher.Amoroso at mail.citytech.cuny.edu wrote: > Hello, > > Thank you for your reply. It says "IDLE's subprocess didn't make > connection. Either IDLE can't start a subprocess or personal firewall > software is blocking the connection." It worked after the first hour > or so after downloading the program but now when I open it I keep > getting this message. I've tried to turn off the firewall as well but > still get this error message. Have you tried googling for this error message? I did just now and one of the top search results including several suggestions, including making sure you haven't created python scripts and saved them in the same directory as the python executable or IDLE itself. If you do that, and you happen to name your python program the same name as a built-in module, it will prevent IDLE from loading. If you have been saving any python programs in the same directory as Python or IDLE, move them somewhere else. Just a shot in the dark. could be way off base. From kobsx4 at gmail.com Fri May 13 11:38:39 2016 From: kobsx4 at gmail.com (Jake Kobs) Date: Fri, 13 May 2016 08:38:39 -0700 (PDT) Subject: Average calculation Program *need help* In-Reply-To: References: <011a2f60-aafd-484f-8cb2-6017aaf1bb29@googlegroups.com> <25510084-74ff-4f7f-b99f-bb29bc675899@googlegroups.com> Message-ID: <13a4ef84-6cc8-4f7d-a615-75bd183e3d93@googlegroups.com> Thank you so much! I finally got it. :) From gengyangcai at gmail.com Fri May 13 12:05:19 2016 From: gengyangcai at gmail.com (Cai Gengyang) Date: Fri, 13 May 2016 09:05:19 -0700 (PDT) Subject: An educational site written in Python (from YCombinator's RFS) In-Reply-To: References: Message-ID: My "vision" if you will for this site is one which should have these basic functionalities : 1) A forum for students , educators, parents and lawyers to post and discuss issues about education, fees, admissions process , visa issues. 2) A search engine for users to well (search for stuff lol) 2) Puzzles , Quizzes, Games and Tests in various topics. (Make it dynamic, animated and really interesting , I mean, I really love math, physics and engineering but a lot of lecturers are just fucking boring to listen to if you know what I mean. (makes you feel like you want to punch something. It would be great if every lecturer and professor in the world was as eloquent as Christopher Hitchens or as funny as Bill Maher, but especially in the hard sciences and math, alot of the lecturers are extremely smart but also very boring to listen to. 3) A live online discussion forum where users can discuss and chat see each other in real-time with a live video thingy (i don't know what its called) with some translator to help translate foreign accents and help users from different countries communicate properly. Imagine you are a poor genius amateur mathematician from Sudan who has just found a partial proof to the Riemann Hypothesis and you want to discuss it live in person over the internet with other math enthusiasts but can't speak English. A live translator that translates what you are speaking in real-time (if such a thing were possible, I have no idea) would be a good idea to create 4) A system to help parents connect with tutors whom they want to hire for their kids. I have no idea what the tutoring industry is like in other countries , but its a huge industry in Singapore with its super "kiasu" parents). There are already several such sites available, but I guess I could clone them and build them with different design (there can be infinite types of design, so its not like a zero-sum game). Especially as the population grows , there's likely to be more demand for these kind of sites and services. Alot of websites have very poor generic design and are unappealing to look at. edx.org is a great example , perhaps a competitor / clone with different functionalities and better design , more videos, graphics , more interactive On Wednesday, May 11, 2016 at 3:07:17 AM UTC+8, DFS wrote: > On 5/10/2016 2:13 AM, Cai Gengyang wrote: > > Ok, so after reading YCombinator's RFS, I have decided that I want to > > work on this : > > > > > > ------------------------------------------------------------------------------------------------------------------- > > > > EDUCATION > > > > If we can fix education, we can eventually do everything else on this > > list. The first attempts to use technology to fix education have > > focused on using the Internet to distribute traditional content to a > > wider audience. This is good, but the Internet is a fundamentally > > different medium and capable of much more. > > > > Solutions that combine the mass scale of technology with one-on-one > > in-person interaction are particularly interesting to us. > > one room, one teacher, one student, and one web browser? > > > > > This may not require a "breakthrough" technology in the classical > > sense, but at a minimum it will require very new ways of doing > > things. > > > > ------------------------------------------------------------------------------------------------------------------- > > > > I want to create such a site using Python. What are the various > > steps I need to take to create such a site ? This is a big project, > > but one that is worth doing ... Any suggestions / help appreciated ? > > Thanks alot > > > When you say 'such a site' what do you envision? From aidan_silcock at yahoo.co.uk Fri May 13 12:18:59 2016 From: aidan_silcock at yahoo.co.uk (Aidan Silcock) Date: Fri, 13 May 2016 16:18:59 +0000 (UTC) Subject: Python 3.5.1 Not Working In-Reply-To: <1863074209.3155141.1463155144093.JavaMail.yahoo@mail.yahoo.com> References: <1863074209.3155141.1463155144093.JavaMail.yahoo.ref@mail.yahoo.com> <1863074209.3155141.1463155144093.JavaMail.yahoo@mail.yahoo.com> Message-ID: <698881343.3134101.1463156339584.JavaMail.yahoo@mail.yahoo.com> Sent from Yahoo Mail on Android On Fri, 13 May, 2016 at 16:59, Aidan Silcock wrote: HelloI have tried to download python 3.5.1 today and it has downloaded but each time I try to open it it says I need to Modify, Repair or Uninstall the program.I have tried repairing it neumerous times and it will say it was successful but then when I go to open it again it comes up with the same message.Can you help?Aidan From davidgshi at yahoo.co.uk Fri May 13 12:19:34 2016 From: davidgshi at yahoo.co.uk (David Shi) Date: Fri, 13 May 2016 16:19:34 +0000 (UTC) Subject: How to put back a number-based index In-Reply-To: <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> I lost my indexes after grouping in Pandas. I managed to rest_index and got back the index column. But How can I get back a index row? Regards. David From oysteijo at gmail.com Fri May 13 12:22:36 2016 From: oysteijo at gmail.com (=?UTF-8?Q?=C3=98ystein_Sch=C3=B8nning=2DJohansen?=) Date: Fri, 13 May 2016 09:22:36 -0700 (PDT) Subject: Calling python from C with OpenMP In-Reply-To: References: <8224bdd2-9afe-487b-804b-f3b88dee2028@googlegroups.com> <1288606789484790413.405054sturla.molden-gmail.com@news.gmane.org> Message-ID: <91ea4bb0-2d34-4a27-8b18-640c79712f64@googlegroups.com> On Friday, May 13, 2016 at 2:04:53 AM UTC+2, Sturla Molden wrote: > You must own the GIL before you can safely use the Python C API, object > creation and refcounting in particular. Use the "Simplified GIL API" to > grab the GIL and release it when you are done. I've now read about the GIL and it looks like I am in deep problems. I've added the GILState lock to the threaded loop like this: #pragma omp parallel for for( int i = 0; i < 10; i++ ){ PyGILState_STATE gstate; gstate = PyGILState_Ensure(); PyObject *ret = PyObject_CallMethod( mult_obj, "do_multiply", "i", i ); if( !ret ){ printf("Cannot call 'do_multiply'\n"); continue; } printf("The value calculated in Python was: %3d\n", (int) PyLong_AsLong(ret)); Py_DECREF(ret); PyGILState_Release(gstate); } .... but still no success. Have I done it right? regs, -?ystein From ganesh1pal at gmail.com Fri May 13 12:23:47 2016 From: ganesh1pal at gmail.com (Ganesh Pal) Date: Fri, 13 May 2016 21:53:47 +0530 Subject: Skipping test using unittest SkipTest and exit status Message-ID: Hi Team, Iam on python 2.7 and Linux . I need inputs on the below program , Iam skipping the unittest from setUpClass in following way # raise unittest.SkipTest(message) The test are getting skipped but I have two problem . (1) This script is in turn read by other scripts which considers the test have passed based on the scripts return code , but the test have actually been skipped , How do include an exit status to indicates that the test have failed (2) Why is the message in the raise statement i.e raise unittest.SkipTest("Class setup failed skipping test") not getting displayed . Also thinking if we could replace raise unittest.SkipTest with assert statement ? Sample code: #!/usr/bin/env python import unittest import logging class ScanTest(unittest.TestCase): @classmethod def setUpClass(self): """ Initial setup before unittest run """ pdb.set_trace() self.scan = False if not self.scan: logging.error("Failed scanning ") raise unittest.SkipTest("Class setup failed skipping test") self.data = True if not self.data: logging.error("Failed getting data ") raise unittest.SkipTest("Class setup failed skipping test") logging.info("SETUP.....Done") def test_01_inode_scanion(self): """ test01: inode scanion """ logging.info("### Executing test01: ###") @classmethod def tearDownClass(self): """ Cleanup all the data & logs """ logging.info("Cleaning all data") def main(): """ ---MAIN--- """ try: unittest.main() except Exception as e: logging.exception(e) sys.exit(1) if __name__ == '__main__': main() Sample output gpal-ae9703e-1# python unitest1.py ERROR:root:Failed scanning s ---------------------------------------------------------------------- Ran 0 tests in 0.000s OK (skipped=1) Regards, Ganesh From steve at pearwood.info Fri May 13 12:51:38 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 14 May 2016 02:51:38 +1000 Subject: Skipping test using unittest SkipTest and exit status References: Message-ID: <5736061c$0$1604$c3e8da3$5496439d@news.astraweb.com> On Sat, 14 May 2016 02:23 am, Ganesh Pal wrote: > Hi Team, > > Iam on python 2.7 and Linux . I need inputs on the below program , "I am" is two words, not one. I hope you wouldn't write "Youare" or "Heis" :-) Whenever you write "Iam", I read it as the name "Ian", which is very distracting. > Iam skipping the unittest from setUpClass in following way # raise > unittest.SkipTest(message) > > The test are getting skipped but I have two problem . > > (1) This script is in turn read by other scripts which considers the > test have passed based on the scripts return code , but the test have > actually been skipped , How do include an exit status to indicates that > the test have failed But the test *hasn't* failed. A skipped test is not a failed test. If you want the test to count as failed, you must let it fail. You can use the fail() method for that. https://docs.python.org/2/library/unittest.html#unittest.TestCase.fail > (2) Why is the message in the raise statement i.e raise > unittest.SkipTest("Class setup failed skipping test") not getting > displayed . Raising a SkipTest exception is equivalent to calling the skipTest method, which marks the test as an expected skipped test, not a failure. Since it is not a failure, it doesn't display the exception. If you run unittest in verbose mode, the skip message will be displayed. See the documentation here: https://docs.python.org/2/library/unittest.html#skipping-tests-and-expected-failures -- Steven From ned at nedbatchelder.com Fri May 13 12:52:44 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Fri, 13 May 2016 09:52:44 -0700 (PDT) Subject: An educational site written in Python (from YCombinator's RFS) In-Reply-To: References: Message-ID: <3e1860ba-703e-4070-80c7-01f673e1232d@googlegroups.com> On Friday, May 13, 2016 at 12:05:33 PM UTC-4, Cai Gengyang wrote: > edx.org is a great example , perhaps a competitor / clone with different functionalities and better design , more videos, graphics , more interactive As I mentioned in a previous reply, edx.org runs on Open edX, which is open source, written in Python. There's no need for a competitor/clone, you can use it as a starting point, and improve it how you like. The content is completely up to the course designer, you can include whatever videos or graphics you like. For interactives, we have have the XBlock API which lets you create new courseware components that work however you like. --Ned. From gordon at panix.com Fri May 13 12:52:52 2016 From: gordon at panix.com (John Gordon) Date: Fri, 13 May 2016 16:52:52 +0000 (UTC) Subject: Python 3.5.1 Not Working References: <1863074209.3155141.1463155144093.JavaMail.yahoo.ref@mail.yahoo.com> <1863074209.3155141.1463155144093.JavaMail.yahoo@mail.yahoo.com> <698881343.3134101.1463156339584.JavaMail.yahoo@mail.yahoo.com> Message-ID: In Aidan Silcock writes: > On Fri, 13 May, 2016 at 16:59, Aidan Silcock wrote: > Hello I have tried to download python 3.5.1 today and it has downloaded > but each time I try to open it it says I need to Modify, Repair or > Uninstall the program.I have tried repairing it neumerous times and it > will say it was successful but then when I go to open it again it comes > up with the same message.Can you help?Aidan Hi Aidan, Are you using Windows XP? Unfortunately, versions of Python above 3.4.3 do not support Windows XP. You'll have to download version 3.4.3 if you are using Windows XP. -- John Gordon A is for Amy, who fell down the stairs gordon at panix.com B is for Basil, assaulted by bears -- Edward Gorey, "The Gashlycrumb Tinies" From michael.selik at gmail.com Fri May 13 12:56:56 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 13 May 2016 16:56:56 +0000 Subject: How to put back a number-based index In-Reply-To: <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list < python-list at python.org> wrote: > I lost my indexes after grouping in Pandas. > I managed to rest_index and got back the index column. > But How can I get back a index row? > Was the grouping an aggregation? If so, the original indexes are meaningless. What you could do is reset_index before the grouping and when you aggregate decide how to handle the formerly-known-as-index column (min, max, mean, ?). From random832 at fastmail.com Fri May 13 13:09:56 2016 From: random832 at fastmail.com (Random832) Date: Fri, 13 May 2016 13:09:56 -0400 Subject: Python 3.5.1 Not Working In-Reply-To: <698881343.3134101.1463156339584.JavaMail.yahoo@mail.yahoo.com> References: <1863074209.3155141.1463155144093.JavaMail.yahoo.ref@mail.yahoo.com> <1863074209.3155141.1463155144093.JavaMail.yahoo@mail.yahoo.com> <698881343.3134101.1463156339584.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1463159396.2181388.607124969.51527457@webmail.messagingengine.com> On Fri, May 13, 2016, at 12:18, Aidan Silcock via Python-list wrote: > > > Sent from Yahoo Mail on Android > > On Fri, 13 May, 2016 at 16:59, Aidan Silcock > wrote: HelloI have tried to download python 3.5.1 today and it has > downloaded but each time I try to open it it says I need to Modify, > Repair or Uninstall the program.I have tried repairing it neumerous > times and it will say it was successful but then when I go to open it > again it comes up with the same message.Can you help? The file you downloaded is the installer, which installs (or modifies, repairs, or uninstalls) Python. To actually *run* Python you need to find it in the start menu. Don't keep reopening the downloaded file. From python at mrabarnett.plus.com Fri May 13 13:12:08 2016 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 13 May 2016 18:12:08 +0100 Subject: Calling python from C with OpenMP In-Reply-To: <91ea4bb0-2d34-4a27-8b18-640c79712f64@googlegroups.com> References: <8224bdd2-9afe-487b-804b-f3b88dee2028@googlegroups.com> <1288606789484790413.405054sturla.molden-gmail.com@news.gmane.org> <91ea4bb0-2d34-4a27-8b18-640c79712f64@googlegroups.com> Message-ID: <4cb4d566-709d-1139-b515-2eb3cdbbbc9a@mrabarnett.plus.com> On 2016-05-13 17:22, ?ystein Sch?nning-Johansen wrote: > On Friday, May 13, 2016 at 2:04:53 AM UTC+2, Sturla Molden wrote: >> You must own the GIL before you can safely use the Python C API, object >> creation and refcounting in particular. Use the "Simplified GIL API" to >> grab the GIL and release it when you are done. > > I've now read about the GIL and it looks like I am in deep problems. > > I've added the GILState lock to the threaded loop like this: > > #pragma omp parallel for > for( int i = 0; i < 10; i++ ){ > PyGILState_STATE gstate; > gstate = PyGILState_Ensure(); > PyObject *ret = PyObject_CallMethod( mult_obj, "do_multiply", "i", i ); > if( !ret ){ > printf("Cannot call 'do_multiply'\n"); > continue; > } > printf("The value calculated in Python was: %3d\n", (int) PyLong_AsLong(ret)); > Py_DECREF(ret); > PyGILState_Release(gstate); > } > > .... but still no success. Have I done it right? > > regs, > -?ystein > Every PyGILState_Ensure call must be matched with a PyGILState_Release call. The way it's currently written, it won't call PyGILState_Release if ret is NULL. However, I don't think you'll gain much here because you can gain from multi-threading only if the threads can run in parallel. You need to hold the GIL while making Python calls, and only 1 thread can hold the GIL at any time. From gengyangcai at gmail.com Fri May 13 14:16:22 2016 From: gengyangcai at gmail.com (Cai Gengyang) Date: Fri, 13 May 2016 11:16:22 -0700 (PDT) Subject: An educational site written in Python (from YCombinator's RFS) In-Reply-To: <3e1860ba-703e-4070-80c7-01f673e1232d@googlegroups.com> References: <3e1860ba-703e-4070-80c7-01f673e1232d@googlegroups.com> Message-ID: <6ae26ba8-35a7-4f48-afb5-3bb90c695f9d@googlegroups.com> Ned, At the risk of sounding like a naggy grandmother, I am not trying to argue with you (since it is pointless arguing over the internet with a stranger whom i don't know and I hate arguing) but It is important to note that whether there is a need/demand for a competitor or clone is not up to you or me to decide, it is up to the market and users to decide. If following your logic that there is no need for competition, then we would only have 1 search engine in the world. It is competition that creates real wealth in the world and improvement in people's lives. Before Google, there were already tons of search engines in existence like Yahoo and Altavista that had hundreds of millions if not billions of users. I am not sure exactly what special "sauce" that Google had that made it scale so fast so successfully and become the dominant search engine over others , but the point I was simply trying to make is that competition is a great thing in the free market, and that users are always willing to switch or use a new alternative if it is significantly better / different or novel (different design). Very few things are totally novel ... most technologies are copycats of one another. Even Steve Jobs, whom many would describe as the most creative tech innovator for the past 20 or 30 years probably stole / copied / cloned his ideas for his many inventions from various other sources (while constantly whining and accusing Microsoft and Bill Gates of ripping off his ideas lol). Amongst the "educational startup" sector, there are also several up and coming sites : http://www.inc.com/ilan-mochari/16-startups-that-will-disrupt-the-education-market.html and I predict that there will be demand for more similar sites in the future by users As Paul Graham mentioned : "Startups are often ruthless competitors, but they're competing in a game won by making what people want." On Saturday, May 14, 2016 at 12:52:58 AM UTC+8, Ned Batchelder wrote: > On Friday, May 13, 2016 at 12:05:33 PM UTC-4, Cai Gengyang wrote: > > edx.org is a great example , perhaps a competitor / clone with different functionalities and better design , more videos, graphics , more interactive > > As I mentioned in a previous reply, edx.org runs on Open edX, which is > open source, written in Python. There's no need for a competitor/clone, > you can use it as a starting point, and improve it how you like. > > The content is completely up to the course designer, you can include > whatever videos or graphics you like. For interactives, we have have > the XBlock API which lets you create new courseware components that > work however you like. > > --Ned. From michael.selik at gmail.com Fri May 13 14:28:38 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 13 May 2016 18:28:38 +0000 Subject: =?UTF-8?B?UmU6IERpc3RpbmN0aW9uIGJldHdlZW4g4oCcY2xhc3PigJ0gYW5kIOKAnHR5cGXigJ0=?= In-Reply-To: <85eg96eebr.fsf@benfinney.id.au> References: <85eg96eebr.fsf@benfinney.id.au> Message-ID: On Fri, May 13, 2016 at 1:10 AM Ben Finney wrote: > Howdy all, > > Ever since Python's much-celebrated Grand Unification of classes and > types, I have used those terms interchangeably: every class is a type, > and every type is a class. > > That may be an unwise conflation. With the recent rise of optional type > annotation in Python 3, more people are speaking about the important > distinction between a class and a type. > The concept of a type existed before the concept of object-oriented programming. A type is more of a mathematical concept. Do some reading on "type theory" if you'd like to know more. A type system ideally allows logical deductions about the validity of your expressions/statements. Stuff a static analyzer is good at. With the advent of object-orientation and the concept of inheritance, things have gotten a little confused. You could make a distinction between "type inheritance" and "implementation/code inheritance" but many languages do not. When you make a subclass in Python, it inherits both the type and the code of the superclass. I suppose you could say the composition vs inheritance argument stems from some people advocating code inheritance without type inheritance. From ned at nedbatchelder.com Fri May 13 14:42:37 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Fri, 13 May 2016 11:42:37 -0700 (PDT) Subject: An educational site written in Python (from YCombinator's RFS) In-Reply-To: <6ae26ba8-35a7-4f48-afb5-3bb90c695f9d@googlegroups.com> References: <3e1860ba-703e-4070-80c7-01f673e1232d@googlegroups.com> <6ae26ba8-35a7-4f48-afb5-3bb90c695f9d@googlegroups.com> Message-ID: <687b491e-822c-4635-8145-ca2e45027d5c@googlegroups.com> On Friday, May 13, 2016 at 2:16:37 PM UTC-4, Cai Gengyang wrote: > Ned, > > At the risk of sounding like a naggy grandmother, I am not trying to argue with you (since it is pointless arguing over the internet with a stranger whom i don't know and I hate arguing) but It is important to note that whether there is a need/demand for a competitor or clone is not up to you or me to decide, it is up to the market and users to decide. If following your logic that there is no need for competition, then we would only have 1 search engine in the world. It is competition that creates real wealth in the world and improvement in people's lives. Before Google, there were already tons of search engines in existence like Yahoo and Altavista that had hundreds of millions if not billions of users. I am not sure exactly what special "sauce" that Google had that made it scale so fast so successfully and become the dominant search engine over others , but the point I was simply trying to make is that competition is a great thing in the free market, and that users are always willing to switch or use a new alternative if it is significantly better / different or novel (different design). Very few things are totally novel ... most technologies are copycats of one another. Even Steve Jobs, whom many would describe as the most creative tech innovator for the past 20 or 30 years probably stole / copied / cloned his ideas for his many inventions from various other sources (while constantly whining and accusing Microsoft and Bill Gates of ripping off his ideas lol). Amongst the "educational startup" sector, there are also several up and coming sites : http://www.inc.com/ilan-mochari/16-startups-that-will-disrupt-the-education-market.html and I predict that there will be demand for more similar sites in the future by users Sorry I wasn't clear. I certainly didn't mean to squelch competition, and I'm not looking to argue. I just wanted to make sure you understood the Open edX option. I wish you all the best. More and better educational sites will be good for the world. --Ned. > > > As Paul Graham mentioned : "Startups are often ruthless competitors, but they're competing in a game won by making what people want." > > > On Saturday, May 14, 2016 at 12:52:58 AM UTC+8, Ned Batchelder wrote: > > On Friday, May 13, 2016 at 12:05:33 PM UTC-4, Cai Gengyang wrote: > > > edx.org is a great example , perhaps a competitor / clone with different functionalities and better design , more videos, graphics , more interactive > > > > As I mentioned in a previous reply, edx.org runs on Open edX, which is > > open source, written in Python. There's no need for a competitor/clone, > > you can use it as a starting point, and improve it how you like. > > > > The content is completely up to the course designer, you can include > > whatever videos or graphics you like. For interactives, we have have > > the XBlock API which lets you create new courseware components that > > work however you like. > > > > --Ned. From davidgshi at yahoo.co.uk Fri May 13 15:22:14 2016 From: davidgshi at yahoo.co.uk (David Shi) Date: Fri, 13 May 2016 19:22:14 +0000 (UTC) Subject: How to put back a number-based index In-Reply-To: References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> Message-ID: <783608464.3224005.1463167334618.JavaMail.yahoo@mail.yahoo.com> Hello, Michael,Thank you. ?Yes, aster grouping I lost my indexing in both x, y directions. How to convert a row, and a column into indexes or labels? On Friday, 13 May 2016, 17:57, Michael Selik wrote: On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list wrote: I lost my indexes after grouping in Pandas. I managed to rest_index and got back the index column. But How can I get back a index row? Was the grouping an aggregation? If so, the original indexes are meaningless. What you could do is reset_index before the grouping and when you aggregate decide how to handle the formerly-known-as-index column (min, max, mean, ?). From davidgshi at yahoo.co.uk Fri May 13 15:27:14 2016 From: davidgshi at yahoo.co.uk (David Shi) Date: Fri, 13 May 2016 19:27:14 +0000 (UTC) Subject: How to put back a number-based index In-Reply-To: References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> Message-ID: <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> Hello, Michael, Why reset_index before grouping? Regards. David On Friday, 13 May 2016, 17:57, Michael Selik wrote: On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list wrote: I lost my indexes after grouping in Pandas. I managed to rest_index and got back the index column. But How can I get back a index row? Was the grouping an aggregation? If so, the original indexes are meaningless. What you could do is reset_index before the grouping and when you aggregate decide how to handle the formerly-known-as-index column (min, max, mean, ?). From michael.selik at gmail.com Fri May 13 15:56:17 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 13 May 2016 19:56:17 +0000 Subject: How to put back a number-based index In-Reply-To: <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> Message-ID: In order to preserve your index after the aggregation, you need to make sure it is considered a data column (via reset_index) and then choose how your aggregation will operate on that column. On Fri, May 13, 2016 at 3:29 PM David Shi wrote: > Hello, Michael, > > Why reset_index before grouping? > > Regards. > > David > > > On Friday, 13 May 2016, 17:57, Michael Selik > wrote: > > > > > On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list < > python-list at python.org> wrote: > > I lost my indexes after grouping in Pandas. > I managed to rest_index and got back the index column. > But How can I get back a index row? > > > Was the grouping an aggregation? If so, the original indexes are > meaningless. What you could do is reset_index before the grouping and when > you aggregate decide how to handle the formerly-known-as-index column (min, > max, mean, ?). > > > From michael.selik at gmail.com Fri May 13 15:58:42 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 13 May 2016 19:58:42 +0000 Subject: How to put back a number-based index In-Reply-To: References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> Message-ID: Just in case I misunderstood, why don't you make a little example of before and after the grouping? This mailing list does not accept attachments, so you'll have to make do with pasting a few rows of comma-separated or tab-separated values. On Fri, May 13, 2016 at 3:56 PM Michael Selik wrote: > In order to preserve your index after the aggregation, you need to make > sure it is considered a data column (via reset_index) and then choose how > your aggregation will operate on that column. > > On Fri, May 13, 2016 at 3:29 PM David Shi wrote: > >> Hello, Michael, >> >> Why reset_index before grouping? >> >> Regards. >> >> David >> >> >> On Friday, 13 May 2016, 17:57, Michael Selik >> wrote: >> >> >> >> >> On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list < >> python-list at python.org> wrote: >> >> I lost my indexes after grouping in Pandas. >> I managed to rest_index and got back the index column. >> But How can I get back a index row? >> >> >> Was the grouping an aggregation? If so, the original indexes are >> meaningless. What you could do is reset_index before the grouping and when >> you aggregate decide how to handle the formerly-known-as-index column (min, >> max, mean, ?). >> >> >> From davidgshi at yahoo.co.uk Fri May 13 16:00:20 2016 From: davidgshi at yahoo.co.uk (David Shi) Date: Fri, 13 May 2016 20:00:20 +0000 (UTC) Subject: How to put back a number-based index In-Reply-To: References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> Message-ID: <201835036.3207767.1463169620641.JavaMail.yahoo@mail.yahoo.com> Dear MIchael, I am very confused. Can you send me a link to a working example? Regards. David On Friday, 13 May 2016, 20:56, Michael Selik wrote: In order to preserve your index after the aggregation, you need to make sure it is considered a data column (via reset_index) and then choose how your aggregation will operate on that column. On Fri, May 13, 2016 at 3:29 PM David Shi wrote: Hello, Michael, Why reset_index before grouping? Regards. David On Friday, 13 May 2016, 17:57, Michael Selik wrote: On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list wrote: I lost my indexes after grouping in Pandas. I managed to rest_index and got back the index column. But How can I get back a index row? Was the grouping an aggregation? If so, the original indexes are meaningless. What you could do is reset_index before the grouping and when you aggregate decide how to handle the formerly-known-as-index column (min, max, mean, ?). From davidgshi at yahoo.co.uk Fri May 13 16:11:35 2016 From: davidgshi at yahoo.co.uk (David Shi) Date: Fri, 13 May 2016 20:11:35 +0000 (UTC) Subject: How to put back a number-based index In-Reply-To: References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> Message-ID: <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> Dear Michael, I have done a number of operation in between. Providing that information does not help you How to reset index after grouping and various operations is of interest. How to type in a command to find out its current dataframe? Regards. David On Friday, 13 May 2016, 20:58, Michael Selik wrote: Just in case I misunderstood, why don't you make a little example of before and after the grouping? This mailing list does not accept attachments, so you'll have to make do with pasting a few rows of comma-separated or tab-separated values. On Fri, May 13, 2016 at 3:56 PM Michael Selik wrote: In order to preserve your index after the aggregation, you need to make sure it is considered a data column (via reset_index) and then choose how your aggregation will operate on that column. On Fri, May 13, 2016 at 3:29 PM David Shi wrote: Hello, Michael, Why reset_index before grouping? Regards. David On Friday, 13 May 2016, 17:57, Michael Selik wrote: On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list wrote: I lost my indexes after grouping in Pandas. I managed to rest_index and got back the index column. But How can I get back a index row? Was the grouping an aggregation? If so, the original indexes are meaningless. What you could do is reset_index before the grouping and when you aggregate decide how to handle the formerly-known-as-index column (min, max, mean, ?). From oysteijo at gmail.com Fri May 13 16:16:58 2016 From: oysteijo at gmail.com (=?UTF-8?Q?=C3=98ystein_Sch=C3=B8nning=2DJohansen?=) Date: Fri, 13 May 2016 13:16:58 -0700 (PDT) Subject: Calling python from C with OpenMP In-Reply-To: References: <8224bdd2-9afe-487b-804b-f3b88dee2028@googlegroups.com> <1288606789484790413.405054sturla.molden-gmail.com@news.gmane.org> <91ea4bb0-2d34-4a27-8b18-640c79712f64@googlegroups.com> <4cb4d566-709d-1139-b515-2eb3cdbbbc9a@mrabarnett.plus.com> Message-ID: <2a0461ad-fd4c-4ffa-9a4a-b1bf2a21b6cf@googlegroups.com> On Friday, May 13, 2016 at 7:12:33 PM UTC+2, MRAB wrote: > Every PyGILState_Ensure call must be matched with a PyGILState_Release > call. The way it's currently written, it won't call PyGILState_Release > if ret is NULL. Yeah, that's tiny bug, however it is not the main problem... > However, I don't think you'll gain much here because you can gain from > multi-threading only if the threads can run in parallel. You need to > hold the GIL while making Python calls, and only 1 thread can hold the > GIL at any time. In addition to the fact that it still does not work, what you here point out, is the main problem. I actually think I'm giving up for now. Maybe I'm able to solve this later.... (when python does not use a GIL anymore...?) Thanks to Sturla and MRAB anyway! -?ystein From davidgshi at yahoo.co.uk Fri May 13 16:19:12 2016 From: davidgshi at yahoo.co.uk (David Shi) Date: Fri, 13 May 2016 20:19:12 +0000 (UTC) Subject: How to put back a number-based index In-Reply-To: <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> Message-ID: <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> Hello, Michael, I typed in df.index I got the followingMultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY']], labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 45, 44, 46, 48, 47, 49]], names=[u'StateFIPS', 0])Regards. David On Friday, 13 May 2016, 21:11, David Shi wrote: Dear Michael, I have done a number of operation in between. Providing that information does not help you How to reset index after grouping and various operations is of interest. How to type in a command to find out its current dataframe? Regards. David On Friday, 13 May 2016, 20:58, Michael Selik wrote: Just in case I misunderstood, why don't you make a little example of before and after the grouping? This mailing list does not accept attachments, so you'll have to make do with pasting a few rows of comma-separated or tab-separated values. On Fri, May 13, 2016 at 3:56 PM Michael Selik wrote: In order to preserve your index after the aggregation, you need to make sure it is considered a data column (via reset_index) and then choose how your aggregation will operate on that column. On Fri, May 13, 2016 at 3:29 PM David Shi wrote: Hello, Michael, Why reset_index before grouping? Regards. David On Friday, 13 May 2016, 17:57, Michael Selik wrote: On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list wrote: I lost my indexes after grouping in Pandas. I managed to rest_index and got back the index column. But How can I get back a index row? Was the grouping an aggregation? If so, the original indexes are meaningless. What you could do is reset_index before the grouping and when you aggregate decide how to handle the formerly-known-as-index column (min, max, mean, ?). From michael.selik at gmail.com Fri May 13 16:33:11 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 13 May 2016 20:33:11 +0000 Subject: Design: Idiom for classes and methods that are customizable by the user? In-Reply-To: References: <5734ECDF.7070006@gmx.de> Message-ID: On Fri, May 13, 2016 at 2:41 AM Gregory Ewing wrote: > Dirk B?chle wrote: > > I'm currently following the "Factory" pattern (more or less) as I know > > it from C++ and similar languages. > > This statement sets off alarm bells for me. If you're using some > design pattern in Python just because you learned to do it that > way in C++/Java/whatever, you're probably making it more > complicated than it needs to be. > I share Greg's trepidation when I hear a phrase like that, but the general idea of a registry of classes or functions and then picking the right one based on string input is fine. How would the API work for custom Taskmasters? It's not so great to require that the user must explicitly ``add`` their derived class after defining it. Perhaps that add function could be a decorator? def register(key): def decorator(cls): if not issubclass(cls, Taskmaster): raise TypeError('You fool!') registry[key] = cls return decorator @register('noclean') class NoCleanTaskMaster(TaskMaster): "Looks clean enough, why worry?" From davidgshi at yahoo.co.uk Fri May 13 16:33:54 2016 From: davidgshi at yahoo.co.uk (David Shi) Date: Fri, 13 May 2016 20:33:54 +0000 (UTC) Subject: How to put back a number-based index In-Reply-To: <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> Hello, Michael, I changed groupby with one column. The index is different. Index([ u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY'], dtype='object', name=0) How to use this index? Regards. David On Friday, 13 May 2016, 21:19, David Shi wrote: Hello, Michael, I typed in df.index I got the followingMultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY']], labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 45, 44, 46, 48, 47, 49]], names=[u'StateFIPS', 0])Regards. David On Friday, 13 May 2016, 21:11, David Shi wrote: Dear Michael, I have done a number of operation in between. Providing that information does not help you How to reset index after grouping and various operations is of interest. How to type in a command to find out its current dataframe? Regards. David On Friday, 13 May 2016, 20:58, Michael Selik wrote: Just in case I misunderstood, why don't you make a little example of before and after the grouping? This mailing list does not accept attachments, so you'll have to make do with pasting a few rows of comma-separated or tab-separated values. On Fri, May 13, 2016 at 3:56 PM Michael Selik wrote: In order to preserve your index after the aggregation, you need to make sure it is considered a data column (via reset_index) and then choose how your aggregation will operate on that column. On Fri, May 13, 2016 at 3:29 PM David Shi wrote: Hello, Michael, Why reset_index before grouping? Regards. David On Friday, 13 May 2016, 17:57, Michael Selik wrote: On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list wrote: I lost my indexes after grouping in Pandas. I managed to rest_index and got back the index column. But How can I get back a index row? Was the grouping an aggregation? If so, the original indexes are meaningless. What you could do is reset_index before the grouping and when you aggregate decide how to handle the formerly-known-as-index column (min, max, mean, ?). From michael.selik at gmail.com Fri May 13 16:43:32 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 13 May 2016 20:43:32 +0000 Subject: How to put back a number-based index In-Reply-To: <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> Message-ID: Here's an example. >>> import pandas as pd >>> df = pd.DataFrame({'group': list('AB') * 2, 'data': range(4)}, index=list('wxyz')) >>> df data group w 0 A x 1 B y 2 A z 3 B >>> df = df.reset_index() >>> df index data group 0 w 0 A 1 x 1 B 2 y 2 A 3 z 3 B >>> df.groupby('group').max() index data group A y 2 B z 3 If that doesn't help, you'll need to explain what you're trying to accomplish in detail -- what variables you started with, what transformations you want to do, and what variables you hope to have when finished. On Fri, May 13, 2016 at 4:36 PM David Shi wrote: > Hello, Michael, > > I changed groupby with one column. > > The index is different. > > Index([ u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', > u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', > u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', > u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', > u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', > u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', > u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', > u'WY'], > dtype='object', name=0) > > > How to use this index? > > > Regards. > > > David > > > > On Friday, 13 May 2016, 21:19, David Shi wrote: > > > Hello, Michael, > > I typed in df.index > > I got the following > > MultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY']], > labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 45, 44, 46, 48, 47, 49]], > names=[u'StateFIPS', 0]) > > Regards. > > > David > > > > On Friday, 13 May 2016, 21:11, David Shi wrote: > > > Dear Michael, > > I have done a number of operation in between. > > Providing that information does not help you > > How to reset index after grouping and various operations is of interest. > > How to type in a command to find out its current dataframe? > > Regards. > > David > > > On Friday, 13 May 2016, 20:58, Michael Selik > wrote: > > > Just in case I misunderstood, why don't you make a little example of > before and after the grouping? This mailing list does not accept > attachments, so you'll have to make do with pasting a few rows of > comma-separated or tab-separated values. > > On Fri, May 13, 2016 at 3:56 PM Michael Selik > wrote: > > In order to preserve your index after the aggregation, you need to make > sure it is considered a data column (via reset_index) and then choose how > your aggregation will operate on that column. > > On Fri, May 13, 2016 at 3:29 PM David Shi wrote: > > Hello, Michael, > > Why reset_index before grouping? > > Regards. > > David > > > On Friday, 13 May 2016, 17:57, Michael Selik > wrote: > > > > > On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list < > python-list at python.org> wrote: > > I lost my indexes after grouping in Pandas. > I managed to rest_index and got back the index column. > But How can I get back a index row? > > > Was the grouping an aggregation? If so, the original indexes are > meaningless. What you could do is reset_index before the grouping and when > you aggregate decide how to handle the formerly-known-as-index column (min, > max, mean, ?). > > > > > > > > > From davidgshi at yahoo.co.uk Fri May 13 16:51:41 2016 From: davidgshi at yahoo.co.uk (David Shi) Date: Fri, 13 May 2016 20:51:41 +0000 (UTC) Subject: How to put back a number-based index In-Reply-To: References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> Message-ID: <2075459214.3318248.1463172701996.JavaMail.yahoo@mail.yahoo.com> Dear Michael, To avoid complication, I only groupby using one column. It is OK now. ?But, how to refer to new row index? ?How do I use floating index? Float64Index([ 1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], dtype='float64', name=u'StateFIPS') Regards. David On Friday, 13 May 2016, 21:43, Michael Selik wrote: Here's an example. ? ? >>> import pandas as pd? ? >>> df = pd.DataFrame({'group': list('AB') * 2, 'data': range(4)}, index=list('wxyz'))? ? >>> df? ? ? ?data group? ? w ? ? 0 ? ? A? ? x ? ? 1 ? ? B? ? y ? ? 2 ? ? A? ? z ? ? 3 ? ? B? ? >>> df = df.reset_index()? ? >>> df? ? ? index ?data group? ? 0 ? ? w ? ? 0 ? ? A? ? 1 ? ? x ? ? 1 ? ? B? ? 2 ? ? y ? ? 2 ? ? A? ? 3 ? ? z ? ? 3 ? ? B? ? >>> df.groupby('group').max()? ? ? ? ? index ?data? ? group? ? A ? ? ? ? y ? ? 2? ? B ? ? ? ? z ? ? 3 If that doesn't help, you'll need to explain what you're trying to accomplish in detail -- what variables you started with, what transformations you want to do, and what variables you hope to have when finished. On Fri, May 13, 2016 at 4:36 PM David Shi wrote: Hello, Michael, I changed groupby with one column. The index is different. Index([ u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY'], dtype='object', name=0) How to use this index? Regards. David On Friday, 13 May 2016, 21:19, David Shi wrote: Hello, Michael, I typed in df.index I got the followingMultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY']], labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 45, 44, 46, 48, 47, 49]], names=[u'StateFIPS', 0])Regards. David On Friday, 13 May 2016, 21:11, David Shi wrote: Dear Michael, I have done a number of operation in between. Providing that information does not help you How to reset index after grouping and various operations is of interest. How to type in a command to find out its current dataframe? Regards. David On Friday, 13 May 2016, 20:58, Michael Selik wrote: Just in case I misunderstood, why don't you make a little example of before and after the grouping? This mailing list does not accept attachments, so you'll have to make do with pasting a few rows of comma-separated or tab-separated values. On Fri, May 13, 2016 at 3:56 PM Michael Selik wrote: In order to preserve your index after the aggregation, you need to make sure it is considered a data column (via reset_index) and then choose how your aggregation will operate on that column. On Fri, May 13, 2016 at 3:29 PM David Shi wrote: Hello, Michael, Why reset_index before grouping? Regards. David On Friday, 13 May 2016, 17:57, Michael Selik wrote: On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list wrote: I lost my indexes after grouping in Pandas. I managed to rest_index and got back the index column. But How can I get back a index row? Was the grouping an aggregation? If so, the original indexes are meaningless. What you could do is reset_index before the grouping and when you aggregate decide how to handle the formerly-known-as-index column (min, max, mean, ?). From michael.selik at gmail.com Fri May 13 17:02:21 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 13 May 2016 21:02:21 +0000 Subject: How to put back a number-based index In-Reply-To: <2075459214.3318248.1463172701996.JavaMail.yahoo@mail.yahoo.com> References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> <2075459214.3318248.1463172701996.JavaMail.yahoo@mail.yahoo.com> Message-ID: To clarify that you're specifying the index as a label, use df.iloc >>> df = pd.DataFrame({'X': range(4)}, index=list('abcd')) >>> df X a 0 b 1 c 2 d 3 >>> df.loc['a'] X 0 Name: a, dtype: int64 >>> df.iloc[0] X 0 Name: a, dtype: int64 On Fri, May 13, 2016 at 4:54 PM David Shi wrote: > Dear Michael, > > To avoid complication, I only groupby using one column. > > It is OK now. But, how to refer to new row index? How do I use floating > index? > > Float64Index([ 1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, > 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, > 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, > 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, > 51.0, 53.0, 54.0, 55.0, 56.0], > dtype='float64', name=u'StateFIPS') > > > Regards. > > > David > > > > On Friday, 13 May 2016, 21:43, Michael Selik > wrote: > > > Here's an example. > > >>> import pandas as pd > >>> df = pd.DataFrame({'group': list('AB') * 2, 'data': range(4)}, > index=list('wxyz')) > >>> df > data group > w 0 A > x 1 B > y 2 A > z 3 B > >>> df = df.reset_index() > >>> df > index data group > 0 w 0 A > 1 x 1 B > 2 y 2 A > 3 z 3 B > >>> df.groupby('group').max() > index data > group > A y 2 > B z 3 > > If that doesn't help, you'll need to explain what you're trying to > accomplish in detail -- what variables you started with, what > transformations you want to do, and what variables you hope to have when > finished. > > On Fri, May 13, 2016 at 4:36 PM David Shi wrote: > > Hello, Michael, > > I changed groupby with one column. > > The index is different. > > Index([ u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', > u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', > u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', > u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', > u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', > u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', > u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', > u'WY'], > dtype='object', name=0) > > > How to use this index? > > > Regards. > > > David > > > > On Friday, 13 May 2016, 21:19, David Shi wrote: > > > Hello, Michael, > > I typed in df.index > > I got the following > > MultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY']], > labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 45, 44, 46, 48, 47, 49]], > names=[u'StateFIPS', 0]) > > Regards. > > > David > > > > On Friday, 13 May 2016, 21:11, David Shi wrote: > > > Dear Michael, > > I have done a number of operation in between. > > Providing that information does not help you > > How to reset index after grouping and various operations is of interest. > > How to type in a command to find out its current dataframe? > > Regards. > > David > > > On Friday, 13 May 2016, 20:58, Michael Selik > wrote: > > > Just in case I misunderstood, why don't you make a little example of > before and after the grouping? This mailing list does not accept > attachments, so you'll have to make do with pasting a few rows of > comma-separated or tab-separated values. > > On Fri, May 13, 2016 at 3:56 PM Michael Selik > wrote: > > In order to preserve your index after the aggregation, you need to make > sure it is considered a data column (via reset_index) and then choose how > your aggregation will operate on that column. > > On Fri, May 13, 2016 at 3:29 PM David Shi wrote: > > Hello, Michael, > > Why reset_index before grouping? > > Regards. > > David > > > On Friday, 13 May 2016, 17:57, Michael Selik > wrote: > > > > > On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list < > python-list at python.org> wrote: > > I lost my indexes after grouping in Pandas. > I managed to rest_index and got back the index column. > But How can I get back a index row? > > > Was the grouping an aggregation? If so, the original indexes are > meaningless. What you could do is reset_index before the grouping and when > you aggregate decide how to handle the formerly-known-as-index column (min, > max, mean, ?). > > > > > > > > > > > From hajramezanali at gmail.com Fri May 13 17:36:48 2016 From: hajramezanali at gmail.com (Ehsan Hajiramezanali) Date: Fri, 13 May 2016 16:36:48 -0500 Subject: How to use pip to install dtrx? Message-ID: Hi, I want to use pip to install dtrx. However, I got the following error. ~~~ $ pip install --allow-external dtrx dtrx DEPRECATION: --allow-external has been deprecated and will be removed in the future. Due to changes in the repository protocol, it no longer has any effect. Collecting dtrx Could not find a version that satisfies the requirement dtrx (from versions: ) No matching distribution found for dtrx ~~~ Is there any way to solve this problem? Thanks in advance. Best regards, Ehsan From davidgshi at yahoo.co.uk Fri May 13 17:51:58 2016 From: davidgshi at yahoo.co.uk (David Shi) Date: Fri, 13 May 2016 21:51:58 +0000 (UTC) Subject: How to put back a number-based index In-Reply-To: References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> <2075459214.3318248.1463172701996.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1584945027.3340053.1463176318146.JavaMail.yahoo@mail.yahoo.com> Hello, Michael, How to convert a float type column into an integer or label or string type? On Friday, 13 May 2016, 22:02, Michael Selik wrote: To clarify that you're specifying the index as a label, use df.iloc ? ? >>> df = pd.DataFrame({'X': range(4)}, index=list('abcd'))? ? >>> df? ? ? ?X? ? a ?0? ? b ?1? ? c ?2? ? d ?3? ? >>> df.loc['a']? ? X ? ?0? ? Name: a, dtype: int64? ? >>> df.iloc[0]? ? X ? ?0? ? Name: a, dtype: int64 On Fri, May 13, 2016 at 4:54 PM David Shi wrote: Dear Michael, To avoid complication, I only groupby using one column. It is OK now.? But, how to refer to new row index?? How do I use floating index? Float64Index([ 1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], dtype='float64', name=u'StateFIPS') Regards. David On Friday, 13 May 2016, 21:43, Michael Selik wrote: Here's an example. ? ? >>> import pandas as pd? ? >>> df = pd.DataFrame({'group': list('AB') * 2, 'data': range(4)}, index=list('wxyz'))? ? >>> df? ? ? ?data group? ? w ? ? 0 ? ? A? ? x ? ? 1 ? ? B? ? y ? ? 2 ? ? A? ? z ? ? 3 ? ? B? ? >>> df = df.reset_index()? ? >>> df? ? ? index ?data group? ? 0 ? ? w ? ? 0 ? ? A? ? 1 ? ? x ? ? 1 ? ? B? ? 2 ? ? y ? ? 2 ? ? A? ? 3 ? ? z ? ? 3 ? ? B? ? >>> df.groupby('group').max()? ? ? ? ? index ?data? ? group? ? A ? ? ? ? y ? ? 2? ? B ? ? ? ? z ? ? 3 If that doesn't help, you'll need to explain what you're trying to accomplish in detail -- what variables you started with, what transformations you want to do, and what variables you hope to have when finished. On Fri, May 13, 2016 at 4:36 PM David Shi wrote: Hello, Michael, I changed groupby with one column. The index is different. Index([ u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY'], dtype='object', name=0) How to use this index? Regards. David On Friday, 13 May 2016, 21:19, David Shi wrote: Hello, Michael, I typed in df.index I got the followingMultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY']], labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 45, 44, 46, 48, 47, 49]], names=[u'StateFIPS', 0])Regards. David On Friday, 13 May 2016, 21:11, David Shi wrote: Dear Michael, I have done a number of operation in between. Providing that information does not help you How to reset index after grouping and various operations is of interest. How to type in a command to find out its current dataframe? Regards. David On Friday, 13 May 2016, 20:58, Michael Selik wrote: Just in case I misunderstood, why don't you make a little example of before and after the grouping? This mailing list does not accept attachments, so you'll have to make do with pasting a few rows of comma-separated or tab-separated values. On Fri, May 13, 2016 at 3:56 PM Michael Selik wrote: In order to preserve your index after the aggregation, you need to make sure it is considered a data column (via reset_index) and then choose how your aggregation will operate on that column. On Fri, May 13, 2016 at 3:29 PM David Shi wrote: Hello, Michael, Why reset_index before grouping? Regards. David On Friday, 13 May 2016, 17:57, Michael Selik wrote: On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list wrote: I lost my indexes after grouping in Pandas. I managed to rest_index and got back the index column. But How can I get back a index row? Was the grouping an aggregation? If so, the original indexes are meaningless. What you could do is reset_index before the grouping and when you aggregate decide how to handle the formerly-known-as-index column (min, max, mean, ?). From michael.selik at gmail.com Fri May 13 17:58:37 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 13 May 2016 21:58:37 +0000 Subject: How to put back a number-based index In-Reply-To: <1584945027.3340053.1463176318146.JavaMail.yahoo@mail.yahoo.com> References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> <2075459214.3318248.1463172701996.JavaMail.yahoo@mail.yahoo.com> <1584945027.3340053.1463176318146.JavaMail.yahoo@mail.yahoo.com> Message-ID: What have code you tried? What error message are you receiving? On Fri, May 13, 2016, 5:54 PM David Shi wrote: > Hello, Michael, > > How to convert a float type column into an integer or label or string type? > > > On Friday, 13 May 2016, 22:02, Michael Selik > wrote: > > > To clarify that you're specifying the index as a label, use df.iloc > > >>> df = pd.DataFrame({'X': range(4)}, index=list('abcd')) > >>> df > X > a 0 > b 1 > c 2 > d 3 > >>> df.loc['a'] > X 0 > Name: a, dtype: int64 > >>> df.iloc[0] > X 0 > Name: a, dtype: int64 > > On Fri, May 13, 2016 at 4:54 PM David Shi wrote: > > Dear Michael, > > To avoid complication, I only groupby using one column. > > It is OK now. But, how to refer to new row index? How do I use floating > index? > > Float64Index([ 1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, > 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, > 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, > 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, > 51.0, 53.0, 54.0, 55.0, 56.0], > dtype='float64', name=u'StateFIPS') > > > Regards. > > > David > > > > On Friday, 13 May 2016, 21:43, Michael Selik > wrote: > > > Here's an example. > > >>> import pandas as pd > >>> df = pd.DataFrame({'group': list('AB') * 2, 'data': range(4)}, > index=list('wxyz')) > >>> df > data group > w 0 A > x 1 B > y 2 A > z 3 B > >>> df = df.reset_index() > >>> df > index data group > 0 w 0 A > 1 x 1 B > 2 y 2 A > 3 z 3 B > >>> df.groupby('group').max() > index data > group > A y 2 > B z 3 > > If that doesn't help, you'll need to explain what you're trying to > accomplish in detail -- what variables you started with, what > transformations you want to do, and what variables you hope to have when > finished. > > On Fri, May 13, 2016 at 4:36 PM David Shi wrote: > > Hello, Michael, > > I changed groupby with one column. > > The index is different. > > Index([ u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', > u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', > u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', > u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', > u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', > u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', > u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', > u'WY'], > dtype='object', name=0) > > > How to use this index? > > > Regards. > > > David > > > > On Friday, 13 May 2016, 21:19, David Shi wrote: > > > Hello, Michael, > > I typed in df.index > > I got the following > > MultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY']], > labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 45, 44, 46, 48, 47, 49]], > names=[u'StateFIPS', 0]) > > Regards. > > > David > > > > On Friday, 13 May 2016, 21:11, David Shi wrote: > > > Dear Michael, > > I have done a number of operation in between. > > Providing that information does not help you > > How to reset index after grouping and various operations is of interest. > > How to type in a command to find out its current dataframe? > > Regards. > > David > > > On Friday, 13 May 2016, 20:58, Michael Selik > wrote: > > > Just in case I misunderstood, why don't you make a little example of > before and after the grouping? This mailing list does not accept > attachments, so you'll have to make do with pasting a few rows of > comma-separated or tab-separated values. > > On Fri, May 13, 2016 at 3:56 PM Michael Selik > wrote: > > In order to preserve your index after the aggregation, you need to make > sure it is considered a data column (via reset_index) and then choose how > your aggregation will operate on that column. > > On Fri, May 13, 2016 at 3:29 PM David Shi wrote: > > Hello, Michael, > > Why reset_index before grouping? > > Regards. > > David > > > On Friday, 13 May 2016, 17:57, Michael Selik > wrote: > > > > > On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list < > python-list at python.org> wrote: > > I lost my indexes after grouping in Pandas. > I managed to rest_index and got back the index column. > But How can I get back a index row? > > > Was the grouping an aggregation? If so, the original indexes are > meaningless. What you could do is reset_index before the grouping and when > you aggregate decide how to handle the formerly-known-as-index column (min, > max, mean, ?). > > > > > > > > > > > > > From tjreedy at udel.edu Fri May 13 18:28:05 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 13 May 2016 18:28:05 -0400 Subject: =?UTF-8?B?UmU6IERpc3RpbmN0aW9uIGJldHdlZW4g4oCcY2xhc3PigJ0gYW5kIA==?= =?UTF-8?B?4oCcdHlwZeKAnQ==?= In-Reply-To: <85eg96eebr.fsf@benfinney.id.au> References: <85eg96eebr.fsf@benfinney.id.au> Message-ID: On 5/13/2016 1:07 AM, Ben Finney wrote: > Howdy all, > > Ever since Python's much-celebrated Grand Unification of classes and > types, I have used those terms interchangeably: every class is a type, > and every type is a class. > > That may be an unwise conflation. With the recent rise of optional type > annotation in Python 3, more people are speaking about the important > distinction between a class and a type. > > This recent message from GvR, discussing a relevant PEP, advocates > keeping them separate: > > PEP 484 [?] tries to make a clear terminological between classes > (the things you have at runtime) and types (the things that type > checkers care about). > > There's a big overlap because most classes are also types -- but not > the other way around! E.g. Any is a type but not a class (you can > neither inherit from Any nor instantiate it), and the same is true > for unions and type variables. [?] > > > > As a Bear of Little Brain, this leaves me clueless. What is the > distinction Guido alludes to, and how are Python classes not also types? I suspect that one could produce a class that is not a type, in Guido's meaning, with a metaclass that is not a subclass of the type class. I don't otherwise know what Guido might have meant. -- Terry Jan Reedy From no.email at nospam.invalid Fri May 13 19:06:35 2016 From: no.email at nospam.invalid (Paul Rubin) Date: Fri, 13 May 2016 16:06:35 -0700 Subject: Distinction between =?utf-8?B?4oCcY2xhc3PigJ0=?= and =?utf-8?B?4oCcdHlwZeKAnQ==?= References: <85eg96eebr.fsf@benfinney.id.au> Message-ID: <87r3d536dg.fsf@jester.gateway.pace.com> Terry Reedy writes: > I suspect that one could produce a class that is not a type, in > Guido's meaning, with a metaclass that is not a subclass of the type > class. I don't otherwise know what Guido might have meant. I think meant that if X is a class, then X is (usually) also a type; but the reverse is not true. We used to think of type and class as the same thing in practice. We didn't have to concern ourselves about too much about theoretical or pedantic differences that might exist. Now with PEP 484, the situation where X is a type but not a class is significant enough in practice that we have to be more careful about the distinction than we were in the Python 2 era. There may(?) also be situations where X is a class but not a type, but I don't think that's being considered as important as the other direction. From no.email at nospam.invalid Fri May 13 22:09:48 2016 From: no.email at nospam.invalid (Paul Rubin) Date: Fri, 13 May 2016 19:09:48 -0700 Subject: Calling python from C with OpenMP References: <8224bdd2-9afe-487b-804b-f3b88dee2028@googlegroups.com> <1288606789484790413.405054sturla.molden-gmail.com@news.gmane.org> <91ea4bb0-2d34-4a27-8b18-640c79712f64@googlegroups.com> <4cb4d566-709d-1139-b515-2eb3cdbbbc9a@mrabarnett.plus.com> <2a0461ad-fd4c-4ffa-9a4a-b1bf2a21b6cf@googlegroups.com> <0nqcjbh34gb68vpp9nchi9vl4mqq6qgn2g@4ax.com> Message-ID: <87inyh2xw3.fsf@jester.gateway.pace.com> Dennis Lee Bieber writes: > It's been tried -- but the non-GIL implementations tend to be > slower at everything else. Has Micropython been compared? CPython needs the GIL because of its frequent twiddling of reference counts. Without the GIL, multi-threaded CPython would have to acquire and release a lock whenever it touched a refcount, which slows things down badly. MicroPython uses a tracing garbage collector instead of refcounts, so there's no issue of having to lock refcounts all the time. It's fairly common in such systems to stop all the user threads during GC, but they can happily run in parallel the rest of the time. Come to think of it, I don't know if MicroPython currently supports threads at all! But its implementation style (i.e. no refcounts) is more parallelism-friendly than CPython's. From davidgshi at yahoo.co.uk Fri May 13 23:15:44 2016 From: davidgshi at yahoo.co.uk (David Shi) Date: Sat, 14 May 2016 03:15:44 +0000 (UTC) Subject: How to put back a number-based index In-Reply-To: References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> <2075459214.3318248.1463172701996.JavaMail.yahoo@mail.yahoo.com> <1584945027.3340053.1463176318146.JavaMail.yahoo@mail.yahoo.com> Message-ID: <363052112.3322670.1463195744354.JavaMail.yahoo@mail.yahoo.com> Hello, Michael, I tried to discover the problem. df[0] ? yields nothingdf[1] ?yields nothingdf[2] yields nothing However, df[3] gives the following:sid -9223372036854775808 NaN 1 133738.70 4 295256.11 5 137733.09 6 409413.58 8 269600.97 9 12852.94 Can we split this back to normal? or turn it into a dictionary, so that I can put values back properly. I like to use sid as index, some way. Regards. David On Friday, 13 May 2016, 22:58, Michael Selik wrote: What have code you tried? What error message are you receiving? On Fri, May 13, 2016, 5:54 PM David Shi wrote: Hello, Michael, How to convert a float type column into an integer or label or string type? On Friday, 13 May 2016, 22:02, Michael Selik wrote: To clarify that you're specifying the index as a label, use df.iloc ? ? >>> df = pd.DataFrame({'X': range(4)}, index=list('abcd'))? ? >>> df? ? ? ?X? ? a ?0? ? b ?1? ? c ?2? ? d ?3? ? >>> df.loc['a']? ? X ? ?0? ? Name: a, dtype: int64? ? >>> df.iloc[0]? ? X ? ?0? ? Name: a, dtype: int64 On Fri, May 13, 2016 at 4:54 PM David Shi wrote: Dear Michael, To avoid complication, I only groupby using one column. It is OK now.? But, how to refer to new row index?? How do I use floating index? Float64Index([ 1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], dtype='float64', name=u'StateFIPS') Regards. David On Friday, 13 May 2016, 21:43, Michael Selik wrote: Here's an example. ? ? >>> import pandas as pd? ? >>> df = pd.DataFrame({'group': list('AB') * 2, 'data': range(4)}, index=list('wxyz'))? ? >>> df? ? ? ?data group? ? w ? ? 0 ? ? A? ? x ? ? 1 ? ? B? ? y ? ? 2 ? ? A? ? z ? ? 3 ? ? B? ? >>> df = df.reset_index()? ? >>> df? ? ? index ?data group? ? 0 ? ? w ? ? 0 ? ? A? ? 1 ? ? x ? ? 1 ? ? B? ? 2 ? ? y ? ? 2 ? ? A? ? 3 ? ? z ? ? 3 ? ? B? ? >>> df.groupby('group').max()? ? ? ? ? index ?data? ? group? ? A ? ? ? ? y ? ? 2? ? B ? ? ? ? z ? ? 3 If that doesn't help, you'll need to explain what you're trying to accomplish in detail -- what variables you started with, what transformations you want to do, and what variables you hope to have when finished. On Fri, May 13, 2016 at 4:36 PM David Shi wrote: Hello, Michael, I changed groupby with one column. The index is different. Index([ u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY'], dtype='object', name=0) How to use this index? Regards. David On Friday, 13 May 2016, 21:19, David Shi wrote: Hello, Michael, I typed in df.index I got the followingMultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY']], labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 45, 44, 46, 48, 47, 49]], names=[u'StateFIPS', 0])Regards. David On Friday, 13 May 2016, 21:11, David Shi wrote: Dear Michael, I have done a number of operation in between. Providing that information does not help you How to reset index after grouping and various operations is of interest. How to type in a command to find out its current dataframe? Regards. David On Friday, 13 May 2016, 20:58, Michael Selik wrote: Just in case I misunderstood, why don't you make a little example of before and after the grouping? This mailing list does not accept attachments, so you'll have to make do with pasting a few rows of comma-separated or tab-separated values. On Fri, May 13, 2016 at 3:56 PM Michael Selik wrote: In order to preserve your index after the aggregation, you need to make sure it is considered a data column (via reset_index) and then choose how your aggregation will operate on that column. On Fri, May 13, 2016 at 3:29 PM David Shi wrote: Hello, Michael, Why reset_index before grouping? Regards. David On Friday, 13 May 2016, 17:57, Michael Selik wrote: On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list wrote: I lost my indexes after grouping in Pandas. I managed to rest_index and got back the index column. But How can I get back a index row? Was the grouping an aggregation? If so, the original indexes are meaningless. What you could do is reset_index before the grouping and when you aggregate decide how to handle the formerly-known-as-index column (min, max, mean, ?). From davidgshi at yahoo.co.uk Fri May 13 23:27:13 2016 From: davidgshi at yahoo.co.uk (David Shi) Date: Sat, 14 May 2016 03:27:13 +0000 (UTC) Subject: How to put back a number-based index In-Reply-To: <363052112.3322670.1463195744354.JavaMail.yahoo@mail.yahoo.com> References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> <2075459214.3318248.1463172701996.JavaMail.yahoo@mail.yahoo.com> <1584945027.3340053.1463176318146.JavaMail.yahoo@mail.yahoo.com> <363052112.3322670.1463195744354.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1963342009.3344305.1463196433057.JavaMail.yahoo@mail.yahoo.com> Hello, Michael, This is very weird. 55 145340.20 56 253333.43 Name: 3, dtype: float64 It looks like two columns, but it shows one single object. Any clue? On Saturday, 14 May 2016, 4:15, David Shi wrote: Hello, Michael, I tried to discover the problem. df[0] ? yields nothingdf[1] ?yields nothingdf[2] yields nothing However, df[3] gives the following:sid -9223372036854775808 NaN 1 133738.70 4 295256.11 5 137733.09 6 409413.58 8 269600.97 9 12852.94 Can we split this back to normal? or turn it into a dictionary, so that I can put values back properly. I like to use sid as index, some way. Regards. David On Friday, 13 May 2016, 22:58, Michael Selik wrote: What have code you tried? What error message are you receiving? On Fri, May 13, 2016, 5:54 PM David Shi wrote: Hello, Michael, How to convert a float type column into an integer or label or string type? On Friday, 13 May 2016, 22:02, Michael Selik wrote: To clarify that you're specifying the index as a label, use df.iloc ? ? >>> df = pd.DataFrame({'X': range(4)}, index=list('abcd'))? ? >>> df? ? ? ?X? ? a ?0? ? b ?1? ? c ?2? ? d ?3? ? >>> df.loc['a']? ? X ? ?0? ? Name: a, dtype: int64? ? >>> df.iloc[0]? ? X ? ?0? ? Name: a, dtype: int64 On Fri, May 13, 2016 at 4:54 PM David Shi wrote: Dear Michael, To avoid complication, I only groupby using one column. It is OK now.? But, how to refer to new row index?? How do I use floating index? Float64Index([ 1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], dtype='float64', name=u'StateFIPS') Regards. David On Friday, 13 May 2016, 21:43, Michael Selik wrote: Here's an example. ? ? >>> import pandas as pd? ? >>> df = pd.DataFrame({'group': list('AB') * 2, 'data': range(4)}, index=list('wxyz'))? ? >>> df? ? ? ?data group? ? w ? ? 0 ? ? A? ? x ? ? 1 ? ? B? ? y ? ? 2 ? ? A? ? z ? ? 3 ? ? B? ? >>> df = df.reset_index()? ? >>> df? ? ? index ?data group? ? 0 ? ? w ? ? 0 ? ? A? ? 1 ? ? x ? ? 1 ? ? B? ? 2 ? ? y ? ? 2 ? ? A? ? 3 ? ? z ? ? 3 ? ? B? ? >>> df.groupby('group').max()? ? ? ? ? index ?data? ? group? ? A ? ? ? ? y ? ? 2? ? B ? ? ? ? z ? ? 3 If that doesn't help, you'll need to explain what you're trying to accomplish in detail -- what variables you started with, what transformations you want to do, and what variables you hope to have when finished. On Fri, May 13, 2016 at 4:36 PM David Shi wrote: Hello, Michael, I changed groupby with one column. The index is different. Index([ u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY'], dtype='object', name=0) How to use this index? Regards. David On Friday, 13 May 2016, 21:19, David Shi wrote: Hello, Michael, I typed in df.index I got the followingMultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY']], labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 45, 44, 46, 48, 47, 49]], names=[u'StateFIPS', 0])Regards. David On Friday, 13 May 2016, 21:11, David Shi wrote: Dear Michael, I have done a number of operation in between. Providing that information does not help you How to reset index after grouping and various operations is of interest. How to type in a command to find out its current dataframe? Regards. David On Friday, 13 May 2016, 20:58, Michael Selik wrote: Just in case I misunderstood, why don't you make a little example of before and after the grouping? This mailing list does not accept attachments, so you'll have to make do with pasting a few rows of comma-separated or tab-separated values. On Fri, May 13, 2016 at 3:56 PM Michael Selik wrote: In order to preserve your index after the aggregation, you need to make sure it is considered a data column (via reset_index) and then choose how your aggregation will operate on that column. On Fri, May 13, 2016 at 3:29 PM David Shi wrote: Hello, Michael, Why reset_index before grouping? Regards. David On Friday, 13 May 2016, 17:57, Michael Selik wrote: On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list wrote: I lost my indexes after grouping in Pandas. I managed to rest_index and got back the index column. But How can I get back a index row? Was the grouping an aggregation? If so, the original indexes are meaningless. What you could do is reset_index before the grouping and when you aggregate decide how to handle the formerly-known-as-index column (min, max, mean, ?). From michael.selik at gmail.com Fri May 13 23:30:38 2016 From: michael.selik at gmail.com (Michael Selik) Date: Sat, 14 May 2016 03:30:38 +0000 Subject: How to put back a number-based index In-Reply-To: <363052112.3322670.1463195744354.JavaMail.yahoo@mail.yahoo.com> References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> <2075459214.3318248.1463172701996.JavaMail.yahoo@mail.yahoo.com> <1584945027.3340053.1463176318146.JavaMail.yahoo@mail.yahoo.com> <363052112.3322670.1463195744354.JavaMail.yahoo@mail.yahoo.com> Message-ID: What were you hoping to get from ``df[0]``? When you say it "yields nothing" do you mean it raised an error? What was the error message? Have you tried a Google search for "pandas set index"? http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.set_index.html On Fri, May 13, 2016 at 11:18 PM David Shi wrote: > Hello, Michael, > > I tried to discover the problem. > > df[0] yields nothing > df[1] yields nothing > df[2] yields nothing > > However, df[3] gives the following: > > sid > -9223372036854775808 NaN > 1 133738.70 > 4 295256.11 > 5 137733.09 > 6 409413.58 > 8 269600.97 > 9 12852.94 > > > Can we split this back to normal? or turn it into a dictionary, so that I can put values back properly. > > > I like to use sid as index, some way. > > > Regards. > > > David > > > > On Friday, 13 May 2016, 22:58, Michael Selik > wrote: > > > What have code you tried? What error message are you receiving? > > On Fri, May 13, 2016, 5:54 PM David Shi wrote: > > Hello, Michael, > > How to convert a float type column into an integer or label or string type? > > > On Friday, 13 May 2016, 22:02, Michael Selik > wrote: > > > To clarify that you're specifying the index as a label, use df.iloc > > >>> df = pd.DataFrame({'X': range(4)}, index=list('abcd')) > >>> df > X > a 0 > b 1 > c 2 > d 3 > >>> df.loc['a'] > X 0 > Name: a, dtype: int64 > >>> df.iloc[0] > X 0 > Name: a, dtype: int64 > > On Fri, May 13, 2016 at 4:54 PM David Shi wrote: > > Dear Michael, > > To avoid complication, I only groupby using one column. > > It is OK now. But, how to refer to new row index? How do I use floating > index? > > Float64Index([ 1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, > 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, > 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, > 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, > 51.0, 53.0, 54.0, 55.0, 56.0], > dtype='float64', name=u'StateFIPS') > > > Regards. > > > David > > > > On Friday, 13 May 2016, 21:43, Michael Selik > wrote: > > > Here's an example. > > >>> import pandas as pd > >>> df = pd.DataFrame({'group': list('AB') * 2, 'data': range(4)}, > index=list('wxyz')) > >>> df > data group > w 0 A > x 1 B > y 2 A > z 3 B > >>> df = df.reset_index() > >>> df > index data group > 0 w 0 A > 1 x 1 B > 2 y 2 A > 3 z 3 B > >>> df.groupby('group').max() > index data > group > A y 2 > B z 3 > > If that doesn't help, you'll need to explain what you're trying to > accomplish in detail -- what variables you started with, what > transformations you want to do, and what variables you hope to have when > finished. > > On Fri, May 13, 2016 at 4:36 PM David Shi wrote: > > Hello, Michael, > > I changed groupby with one column. > > The index is different. > > Index([ u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', > u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', > u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', > u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', > u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', > u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', > u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', > u'WY'], > dtype='object', name=0) > > > How to use this index? > > > Regards. > > > David > > > > On Friday, 13 May 2016, 21:19, David Shi wrote: > > > Hello, Michael, > > I typed in df.index > > I got the following > > MultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY']], > labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 45, 44, 46, 48, 47, 49]], > names=[u'StateFIPS', 0]) > > Regards. > > > David > > > > On Friday, 13 May 2016, 21:11, David Shi wrote: > > > Dear Michael, > > I have done a number of operation in between. > > Providing that information does not help you > > How to reset index after grouping and various operations is of interest. > > How to type in a command to find out its current dataframe? > > Regards. > > David > > > On Friday, 13 May 2016, 20:58, Michael Selik > wrote: > > > Just in case I misunderstood, why don't you make a little example of > before and after the grouping? This mailing list does not accept > attachments, so you'll have to make do with pasting a few rows of > comma-separated or tab-separated values. > > On Fri, May 13, 2016 at 3:56 PM Michael Selik > wrote: > > In order to preserve your index after the aggregation, you need to make > sure it is considered a data column (via reset_index) and then choose how > your aggregation will operate on that column. > > On Fri, May 13, 2016 at 3:29 PM David Shi wrote: > > Hello, Michael, > > Why reset_index before grouping? > > Regards. > > David > > > On Friday, 13 May 2016, 17:57, Michael Selik > wrote: > > > > > On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list < > python-list at python.org> wrote: > > I lost my indexes after grouping in Pandas. > I managed to rest_index and got back the index column. > But How can I get back a index row? > > > Was the grouping an aggregation? If so, the original indexes are > meaningless. What you could do is reset_index before the grouping and when > you aggregate decide how to handle the formerly-known-as-index column (min, > max, mean, ?). > > > > > > > > > > > > > > > From michael.selik at gmail.com Fri May 13 23:31:15 2016 From: michael.selik at gmail.com (Michael Selik) Date: Sat, 14 May 2016 03:31:15 +0000 Subject: How to put back a number-based index In-Reply-To: References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> <2075459214.3318248.1463172701996.JavaMail.yahoo@mail.yahoo.com> <1584945027.3340053.1463176318146.JavaMail.yahoo@mail.yahoo.com> <363052112.3322670.1463195744354.JavaMail.yahoo@mail.yahoo.com> Message-ID: It looks like you're getting a Series. Apparently more that one row has the same index. On Fri, May 13, 2016 at 11:30 PM Michael Selik wrote: > What were you hoping to get from ``df[0]``? > When you say it "yields nothing" do you mean it raised an error? What was > the error message? > > Have you tried a Google search for "pandas set index"? > > http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.set_index.html > > On Fri, May 13, 2016 at 11:18 PM David Shi wrote: > >> Hello, Michael, >> >> I tried to discover the problem. >> >> df[0] yields nothing >> df[1] yields nothing >> df[2] yields nothing >> >> However, df[3] gives the following: >> >> sid >> -9223372036854775808 NaN >> 1 133738.70 >> 4 295256.11 >> 5 137733.09 >> 6 409413.58 >> 8 269600.97 >> 9 12852.94 >> >> >> Can we split this back to normal? or turn it into a dictionary, so that I can put values back properly. >> >> >> I like to use sid as index, some way. >> >> >> Regards. >> >> >> David >> >> >> >> On Friday, 13 May 2016, 22:58, Michael Selik >> wrote: >> >> >> What have code you tried? What error message are you receiving? >> >> On Fri, May 13, 2016, 5:54 PM David Shi wrote: >> >> Hello, Michael, >> >> How to convert a float type column into an integer or label or string >> type? >> >> >> On Friday, 13 May 2016, 22:02, Michael Selik >> wrote: >> >> >> To clarify that you're specifying the index as a label, use df.iloc >> >> >>> df = pd.DataFrame({'X': range(4)}, index=list('abcd')) >> >>> df >> X >> a 0 >> b 1 >> c 2 >> d 3 >> >>> df.loc['a'] >> X 0 >> Name: a, dtype: int64 >> >>> df.iloc[0] >> X 0 >> Name: a, dtype: int64 >> >> On Fri, May 13, 2016 at 4:54 PM David Shi wrote: >> >> Dear Michael, >> >> To avoid complication, I only groupby using one column. >> >> It is OK now. But, how to refer to new row index? How do I use floating >> index? >> >> Float64Index([ 1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, >> 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, >> 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, >> 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, >> 51.0, 53.0, 54.0, 55.0, 56.0], >> dtype='float64', name=u'StateFIPS') >> >> >> Regards. >> >> >> David >> >> >> >> On Friday, 13 May 2016, 21:43, Michael Selik >> wrote: >> >> >> Here's an example. >> >> >>> import pandas as pd >> >>> df = pd.DataFrame({'group': list('AB') * 2, 'data': range(4)}, >> index=list('wxyz')) >> >>> df >> data group >> w 0 A >> x 1 B >> y 2 A >> z 3 B >> >>> df = df.reset_index() >> >>> df >> index data group >> 0 w 0 A >> 1 x 1 B >> 2 y 2 A >> 3 z 3 B >> >>> df.groupby('group').max() >> index data >> group >> A y 2 >> B z 3 >> >> If that doesn't help, you'll need to explain what you're trying to >> accomplish in detail -- what variables you started with, what >> transformations you want to do, and what variables you hope to have when >> finished. >> >> On Fri, May 13, 2016 at 4:36 PM David Shi wrote: >> >> Hello, Michael, >> >> I changed groupby with one column. >> >> The index is different. >> >> Index([ u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', >> u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', >> u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', >> u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', >> u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', >> u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', >> u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', >> u'WY'], >> dtype='object', name=0) >> >> >> How to use this index? >> >> >> Regards. >> >> >> David >> >> >> >> On Friday, 13 May 2016, 21:19, David Shi wrote: >> >> >> Hello, Michael, >> >> I typed in df.index >> >> I got the following >> >> MultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY']], >> labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 45, 44, 46, 48, 47, 49]], >> names=[u'StateFIPS', 0]) >> >> Regards. >> >> >> David >> >> >> >> On Friday, 13 May 2016, 21:11, David Shi wrote: >> >> >> Dear Michael, >> >> I have done a number of operation in between. >> >> Providing that information does not help you >> >> How to reset index after grouping and various operations is of interest. >> >> How to type in a command to find out its current dataframe? >> >> Regards. >> >> David >> >> >> On Friday, 13 May 2016, 20:58, Michael Selik >> wrote: >> >> >> Just in case I misunderstood, why don't you make a little example of >> before and after the grouping? This mailing list does not accept >> attachments, so you'll have to make do with pasting a few rows of >> comma-separated or tab-separated values. >> >> On Fri, May 13, 2016 at 3:56 PM Michael Selik >> wrote: >> >> In order to preserve your index after the aggregation, you need to make >> sure it is considered a data column (via reset_index) and then choose how >> your aggregation will operate on that column. >> >> On Fri, May 13, 2016 at 3:29 PM David Shi wrote: >> >> Hello, Michael, >> >> Why reset_index before grouping? >> >> Regards. >> >> David >> >> >> On Friday, 13 May 2016, 17:57, Michael Selik >> wrote: >> >> >> >> >> On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list < >> python-list at python.org> wrote: >> >> I lost my indexes after grouping in Pandas. >> I managed to rest_index and got back the index column. >> But How can I get back a index row? >> >> >> Was the grouping an aggregation? If so, the original indexes are >> meaningless. What you could do is reset_index before the grouping and when >> you aggregate decide how to handle the formerly-known-as-index column (min, >> max, mean, ?). >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> From davidgshi at yahoo.co.uk Fri May 13 23:46:51 2016 From: davidgshi at yahoo.co.uk (David Shi) Date: Sat, 14 May 2016 03:46:51 +0000 (UTC) Subject: How to put back a number-based index In-Reply-To: <1963342009.3344305.1463196433057.JavaMail.yahoo@mail.yahoo.com> References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> <2075459214.3318248.1463172701996.JavaMail.yahoo@mail.yahoo.com> <1584945027.3340053.1463176318146.JavaMail.yahoo@mail.yahoo.com> <363052112.3322670.1463195744354.JavaMail.yahoo@mail.yahoo.com> <1963342009.3344305.1463196433057.JavaMail.yahoo@mail.yahoo.com> Message-ID: <621836310.3434017.1463197611396.JavaMail.yahoo@mail.yahoo.com> Hello, Michael, I do not understand this. I tried list =df[3] it worked. ?But, it does not behave like a list. list[0] nothinglist[1] a valuelist[2] nothing list[4] a value It behaves like a dictionary. On Saturday, 14 May 2016, 4:27, David Shi wrote: Hello, Michael, This is very weird. 55 145340.20 56 253333.43 Name: 3, dtype: float64 It looks like two columns, but it shows one single object. Any clue? On Saturday, 14 May 2016, 4:15, David Shi wrote: Hello, Michael, I tried to discover the problem. df[0] ? yields nothingdf[1] ?yields nothingdf[2] yields nothing However, df[3] gives the following:sid -9223372036854775808 NaN 1 133738.70 4 295256.11 5 137733.09 6 409413.58 8 269600.97 9 12852.94 Can we split this back to normal? or turn it into a dictionary, so that I can put values back properly. I like to use sid as index, some way. Regards. David On Friday, 13 May 2016, 22:58, Michael Selik wrote: What have code you tried? What error message are you receiving? On Fri, May 13, 2016, 5:54 PM David Shi wrote: Hello, Michael, How to convert a float type column into an integer or label or string type? On Friday, 13 May 2016, 22:02, Michael Selik wrote: To clarify that you're specifying the index as a label, use df.iloc ? ? >>> df = pd.DataFrame({'X': range(4)}, index=list('abcd'))? ? >>> df? ? ? ?X? ? a ?0? ? b ?1? ? c ?2? ? d ?3? ? >>> df.loc['a']? ? X ? ?0? ? Name: a, dtype: int64? ? >>> df.iloc[0]? ? X ? ?0? ? Name: a, dtype: int64 On Fri, May 13, 2016 at 4:54 PM David Shi wrote: Dear Michael, To avoid complication, I only groupby using one column. It is OK now.? But, how to refer to new row index?? How do I use floating index? Float64Index([ 1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], dtype='float64', name=u'StateFIPS') Regards. David On Friday, 13 May 2016, 21:43, Michael Selik wrote: Here's an example. ? ? >>> import pandas as pd? ? >>> df = pd.DataFrame({'group': list('AB') * 2, 'data': range(4)}, index=list('wxyz'))? ? >>> df? ? ? ?data group? ? w ? ? 0 ? ? A? ? x ? ? 1 ? ? B? ? y ? ? 2 ? ? A? ? z ? ? 3 ? ? B? ? >>> df = df.reset_index()? ? >>> df? ? ? index ?data group? ? 0 ? ? w ? ? 0 ? ? A? ? 1 ? ? x ? ? 1 ? ? B? ? 2 ? ? y ? ? 2 ? ? A? ? 3 ? ? z ? ? 3 ? ? B? ? >>> df.groupby('group').max()? ? ? ? ? index ?data? ? group? ? A ? ? ? ? y ? ? 2? ? B ? ? ? ? z ? ? 3 If that doesn't help, you'll need to explain what you're trying to accomplish in detail -- what variables you started with, what transformations you want to do, and what variables you hope to have when finished. On Fri, May 13, 2016 at 4:36 PM David Shi wrote: Hello, Michael, I changed groupby with one column. The index is different. Index([ u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY'], dtype='object', name=0) How to use this index? Regards. David On Friday, 13 May 2016, 21:19, David Shi wrote: Hello, Michael, I typed in df.index I got the followingMultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY']], labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 45, 44, 46, 48, 47, 49]], names=[u'StateFIPS', 0])Regards. David On Friday, 13 May 2016, 21:11, David Shi wrote: Dear Michael, I have done a number of operation in between. Providing that information does not help you How to reset index after grouping and various operations is of interest. How to type in a command to find out its current dataframe? Regards. David On Friday, 13 May 2016, 20:58, Michael Selik wrote: Just in case I misunderstood, why don't you make a little example of before and after the grouping? This mailing list does not accept attachments, so you'll have to make do with pasting a few rows of comma-separated or tab-separated values. On Fri, May 13, 2016 at 3:56 PM Michael Selik wrote: In order to preserve your index after the aggregation, you need to make sure it is considered a data column (via reset_index) and then choose how your aggregation will operate on that column. On Fri, May 13, 2016 at 3:29 PM David Shi wrote: Hello, Michael, Why reset_index before grouping? Regards. David On Friday, 13 May 2016, 17:57, Michael Selik wrote: On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list wrote: I lost my indexes after grouping in Pandas. I managed to rest_index and got back the index column. But How can I get back a index row? Was the grouping an aggregation? If so, the original indexes are meaningless. What you could do is reset_index before the grouping and when you aggregate decide how to handle the formerly-known-as-index column (min, max, mean, ?). From steve at pearwood.info Sat May 14 01:05:20 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 14 May 2016 15:05:20 +1000 Subject: Distinction between =?UTF-8?B?4oCcY2xhc3PigJ0gYW5kIOKAnHR5cGXigJ0=?= References: <85eg96eebr.fsf@benfinney.id.au> Message-ID: <5736b212$0$1587$c3e8da3$5496439d@news.astraweb.com> On Fri, 13 May 2016 03:07 pm, Ben Finney wrote: > Howdy all, > > Ever since Python's much-celebrated Grand Unification of classes and > types, I have used those terms interchangeably: every class is a type, > and every type is a class. > > That may be an unwise conflation. With the recent rise of optional type > annotation in Python 3, more people are speaking about the important > distinction between a class and a type. [...] > As a Bear of Little Brain, this leaves me clueless. What is the > distinction Guido alludes to, and how are Python classes not also types? > > And why is this distinction important, and who moved my cheesecake? You're not alone. Unfortunately, this confusion is endemic in IT and on the internet. People use types all the time, but without understanding that there are (at least) two related but distinct meanings for the word. Even though it isn't specifically about this distinction, you should start by reading this essay about the distinction between static and dynamic typing here: https://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ Then you should read Guido's PEP 483 on "gradual typing": https://www.python.org/dev/peps/pep-0483/ And this answer on Stackoverflow: http://stackoverflow.com/a/25114770 (Note that despite the question specifying language agnostic answers, the second most highly-rated answer is language specific.) The Wikipedia article is heavy going, but worth reading: https://en.wikipedia.org/wiki/Type_theory As Python programmers, the simplest way to look at this question is: (1) Class (also known as a "type", although for the avoidance of confusion I shall avoid that terminology here) in the Python sense is an entity which tells the interpreter what methods and attributes a value has. In other words, classes define behaviour. In Python classes are "first-class values" and can be created on the fly, bound to names, put inside lists, etc. This is not essential to the concept though, many languages, such as Java, do not treat classes as first-class. Unfortunately, for historical reasons, the function[1] which returns the class of a value is called "type()". And the class of all classes (the metaclass) is also called "type". Which made type and class synonyms from Python 2.2 onwards. (2) Type, in the PEP 483/484 sense, is a label which tells the static type-checker what kind of values are acceptable in a particular place. This is the "type theory" sense as in the Wikipedia article. In principle these PEP 484 types could be completely abstract, but Python's typing module will generally use classes (sense 1 above) to implement these label. (3) In Python especially, the differences between (1) and (2) are more theoretical than practical, but there are a couple of obvious exceptions: - Union types should be considered a purely abstract label. (In practice, they'll actually be implemented as a class, but that's just an implementation detail.) Unions cannot be instantiated, nor subclassed. For example, take this example: py> import typing py> Foo = typing.Union[str, int] Foo is now a label to tell the type-checker you will accept "a str, or an int, but nothing else". It is *not* a subclass of str and int. There is no such thing as an instance of Foo, nor can you subclass it. If you try, you get a TypeError: py> Foo() [...] TypeError: Cannot instantiate class Bar(Foo): pass [...] TypeError: Cannot subclass typing.Union[str, int] This is from Python 3.6 pre-alpha. The messages may change in the future. Foo is only used as a purely abstract label to tell the type-checker to accept either a str, or an int. So in that sense, although the Python implementation of Foo is a class object, that's just a detail of the implementation. - The other obvious example is Any. Like Unions, Any cannot be instantiated or subclassed and is intended as a purely abstract label for "anything at all". One can consider Any to be the equivalent of a Union of "all possible types", and it basically exists for the type-checker to record a state of maximum ignorance. ("I don't know what this will be, it could be anything.") So types and classes essentially have different purposes, even though they overlap. Types (in this "type theory", PEP 484 sense) are for proving compile-time facts about code. Classes (in Python) are for specifying the behaviour and implementation of objects. [1] Actually, the function "type()" and the class "type" are implemented as the same object, which does two different things depending on how many arguments it gets. With one argument, type(x) returns the class of x. With three arguments, type(name, bases, namespace) creates and returns a new class. -- Steven From andreas.roehler at online.de Sat May 14 03:25:57 2016 From: andreas.roehler at online.de (=?UTF-8?Q?Andreas_R=c3=b6hler?=) Date: Sat, 14 May 2016 09:25:57 +0200 Subject: =?UTF-8?B?UmU6IERpc3RpbmN0aW9uIGJldHdlZW4g4oCcY2xhc3PigJ0gYW5kIA==?= =?UTF-8?B?4oCcdHlwZeKAnQ==?= In-Reply-To: References: <85eg96eebr.fsf@benfinney.id.au> Message-ID: <5736D305.9030606@online.de> > I suspect that one could produce a class that is not a type, Say: has not a complete type definition. Think of type for example with the distinction of strings and numbers. Types start from low level units. A class definition must know about strings and numbers, it inherits this knowledge from the syntax of language. Even an empty class must be constructed. Types may be composed resp derived from other types. Any class finely will constitute a type composed by its body. In case an definition may used for example equally for addition or string-concatenation, ambiguity constitutes a composed resp. derived type by themselves. In general terms: the type tells the structure of arguments it receives alongside with the way, it deals with them. From davidgshi at yahoo.co.uk Sat May 14 06:16:30 2016 From: davidgshi at yahoo.co.uk (David Shi) Date: Sat, 14 May 2016 10:16:30 +0000 (UTC) Subject: How to put back a number-based index In-Reply-To: References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> <2075459214.3318248.1463172701996.JavaMail.yahoo@mail.yahoo.com> <1584945027.3340053.1463176318146.JavaMail.yahoo@mail.yahoo.com> <363052112.3322670.1463195744354.JavaMail.yahoo@mail.yahoo.com> Message-ID: <477298994.3432356.1463220990479.JavaMail.yahoo@mail.yahoo.com> Hello, Michael, I discovered that the problem is "two columns of data are put together" and "are recognised as one column". This is very strange. ?I would like to understand the subject well. And, how many ways are there to investigate into the nature of objects dynamically? Some object types only get shown as an object. ?Are there anything to be typed in Python, to reveal objects. Regards. David On Saturday, 14 May 2016, 4:30, Michael Selik wrote: What were you hoping to get from ``df[0]``?When you say it "yields nothing" do you mean it raised an error? What was the error message? Have you tried a Google search for "pandas set index"?http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.set_index.html On Fri, May 13, 2016 at 11:18 PM David Shi wrote: Hello, Michael, I tried to discover the problem. df[0] ? yields nothingdf[1] ?yields nothingdf[2] yields nothing However, df[3] gives the following:sid -9223372036854775808 NaN 1 133738.70 4 295256.11 5 137733.09 6 409413.58 8 269600.97 9 12852.94 Can we split this back to normal? or turn it into a dictionary, so that I can put values back properly. I like to use sid as index, some way. Regards. David On Friday, 13 May 2016, 22:58, Michael Selik wrote: What have code you tried? What error message are you receiving? On Fri, May 13, 2016, 5:54 PM David Shi wrote: Hello, Michael, How to convert a float type column into an integer or label or string type? On Friday, 13 May 2016, 22:02, Michael Selik wrote: To clarify that you're specifying the index as a label, use df.iloc ? ? >>> df = pd.DataFrame({'X': range(4)}, index=list('abcd'))? ? >>> df? ? ? ?X? ? a ?0? ? b ?1? ? c ?2? ? d ?3? ? >>> df.loc['a']? ? X ? ?0? ? Name: a, dtype: int64? ? >>> df.iloc[0]? ? X ? ?0? ? Name: a, dtype: int64 On Fri, May 13, 2016 at 4:54 PM David Shi wrote: Dear Michael, To avoid complication, I only groupby using one column. It is OK now.? But, how to refer to new row index?? How do I use floating index? Float64Index([ 1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], dtype='float64', name=u'StateFIPS') Regards. David On Friday, 13 May 2016, 21:43, Michael Selik wrote: Here's an example. ? ? >>> import pandas as pd? ? >>> df = pd.DataFrame({'group': list('AB') * 2, 'data': range(4)}, index=list('wxyz'))? ? >>> df? ? ? ?data group? ? w ? ? 0 ? ? A? ? x ? ? 1 ? ? B? ? y ? ? 2 ? ? A? ? z ? ? 3 ? ? B? ? >>> df = df.reset_index()? ? >>> df? ? ? index ?data group? ? 0 ? ? w ? ? 0 ? ? A? ? 1 ? ? x ? ? 1 ? ? B? ? 2 ? ? y ? ? 2 ? ? A? ? 3 ? ? z ? ? 3 ? ? B? ? >>> df.groupby('group').max()? ? ? ? ? index ?data? ? group? ? A ? ? ? ? y ? ? 2? ? B ? ? ? ? z ? ? 3 If that doesn't help, you'll need to explain what you're trying to accomplish in detail -- what variables you started with, what transformations you want to do, and what variables you hope to have when finished. On Fri, May 13, 2016 at 4:36 PM David Shi wrote: Hello, Michael, I changed groupby with one column. The index is different. Index([ u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY'], dtype='object', name=0) How to use this index? Regards. David On Friday, 13 May 2016, 21:19, David Shi wrote: Hello, Michael, I typed in df.index I got the followingMultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY']], labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 45, 44, 46, 48, 47, 49]], names=[u'StateFIPS', 0])Regards. David On Friday, 13 May 2016, 21:11, David Shi wrote: Dear Michael, I have done a number of operation in between. Providing that information does not help you How to reset index after grouping and various operations is of interest. How to type in a command to find out its current dataframe? Regards. David On Friday, 13 May 2016, 20:58, Michael Selik wrote: Just in case I misunderstood, why don't you make a little example of before and after the grouping? This mailing list does not accept attachments, so you'll have to make do with pasting a few rows of comma-separated or tab-separated values. On Fri, May 13, 2016 at 3:56 PM Michael Selik wrote: In order to preserve your index after the aggregation, you need to make sure it is considered a data column (via reset_index) and then choose how your aggregation will operate on that column. On Fri, May 13, 2016 at 3:29 PM David Shi wrote: Hello, Michael, Why reset_index before grouping? Regards. David On Friday, 13 May 2016, 17:57, Michael Selik wrote: On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list wrote: I lost my indexes after grouping in Pandas. I managed to rest_index and got back the index column. But How can I get back a index row? Was the grouping an aggregation? If so, the original indexes are meaningless. What you could do is reset_index before the grouping and when you aggregate decide how to handle the formerly-known-as-index column (min, max, mean, ?). From ben+python at benfinney.id.au Sat May 14 08:55:35 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 14 May 2016 22:55:35 +1000 Subject: Why online forums have bad behaviour References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> <57349595.6000702@gmail.com> <85vb2jdq6c.fsf_-_@benfinney.id.au> Message-ID: <85vb2gdcjc.fsf@benfinney.id.au> Ben Finney writes: > TL;DR: because we're all human, and human behaviour needs either > immediate face-to-face feedback or social enforcement to correct > selfishness and abrasiveness. Some people rightly regret this universal human tendency. I have been contacted privately and asked (my paraphrase): My temper often gets the better of me, and I then engage in behaviour that escalates the hostility of the exchange. I don't want that to happen; how can I improve? One one level, I can't help with managing anger. Advice that appeals to rational motives and level-headed appraisal don't help; the very problem expressed is that the undesirable behaviour in this case *isn't* rational or level-headed. So merely saying how one *should* act is not enough. On another level, I can point out an omission in my summary above. Having been corrected by face-to-face feedback and/or social enforcement, we can *change* our tendencies as individuals: over time, with much effort, we form new habits that are more important to us than our old behaviour. These habits, well chosen and consistently followed, can help our irrational, habit-driven self act more in line with what our cool-headed, reflective self would prefer us to do. So as a partial answer the above question: * Recognise that *most* human activity, most of the time, is not decided rationally in that moment, but is instead driven by emotion and habit. If you dislike someone's behaviour, consider that they may not have a well-thought-out or coherent rason for it; and, if pressed to come up with a reason, we will employ all our faculties to *make up* a reason (typically without being aware that's what we're doing!) that somehow depicts us in a better light than others. This is true for clever people even *more* so than less-clever people. Being clever doesn't necessarily make one's decisions better; but it does make one better at *making up* justifications for why one's behaviour is good and doesn't need changing. So, don't be so attached to asking a person *why* they did something you don't like. Simply because of how human decisions work, the reasoning they give is likely to be a post-hoc rationalisation even if they wanted to answer truthfully. And, similarly, don't put much trust in the reasons you later express for your own actions. Instead, you can try to focus on what each of you *wants* to happen; if you disagree about what you want, that's a good place to focus. You may simply have to leave someone be if your values are incompatible with their expressed goals. If they express goals you can agree with, you can try to focus on whether the likely consequences of actions support those goals. If it seems likely to hinder those goals, that may be a more dispassionate discussion that can actually progress what you each want. * Recognise that the impulse to respond quickly is *by definition* an impulse that defeats wise reflective judgement. Acting quickly is necessary most of the time: our waking life is a continual stream of micro-decisions to be made, so of course we make most of those decisions on emotion in the moment, informed by habits established over time. You will only ever engage the broader-picture, dispassionate, reflective aspect of yourself ? the one which is capable of abstract thought, the one which can expressly consider the consequences of what you're about to do ? by *slowing down*. If you feel strongly about a topic, respond less often. Craft each message, not to make it longer, but to make it *more effective*; simplify sentences, remove needlessly accusatory or abrasive phrasing, etc. Often taking more time on a message makes it significantly shorter! In the extreme: realise that though something may be worth expressing, it may not need to be expressed *right now*, and may not need to be expressed *by you*. Often the best decision your reflective, considerate mind will come to is: don't send the message at all. But you'll only get any of that if you choose to spend the time, and reject the impulse to respond immediately to messages. * Recognise that you are one person composing the message and sending it once, but it gets *read* many more times, perhaps by many more people. You already know what it feels like to write the message. Put effort in, every message you write, to stop and consider how it likely feels to *read* this message. If read by someone who already disagrees with your position, is this message likely to communicate well? To express your position better? To make the reader feel respected? To give them space to disagree without personal insult? To be concise enough, well-phrased enough, to be read by that person at all? You care enough about your message to write it. Other people don't necessarily have that advantage. You need to make your message worth their effort to read and comprehend. All of which boils down to: Recognise how *badly designed* our brains are for communicating in this medium, and give yourself ? and your readers ? the best chance of spending our attention wisely and well. -- \ ?Putting the word ?faith? in front of something is no excuse | `\ for barbarism and cruelty and ignorance and stupidity.? | _o__) ?Christopher Hitchens | Ben Finney From steve at pearwood.info Sat May 14 10:12:32 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 15 May 2016 00:12:32 +1000 Subject: Performance with and without the garbage collector Message-ID: <5737325a$0$1610$c3e8da3$5496439d@news.astraweb.com> Just for kicks, I've been playing around with running code snippets with and without the garbage collector enabled, looking to see if it will make any obvious difference to performance. So far, I haven't found any. For instance, I tried: a = [i**3 for i in range(2000000)] del a[:] thinking that garbage collecting almost two million ints would surely show some performance difference, but it doesn't. Is anyone able to demonstrate a replicable performance impact due to garbage collection? -- Steven From ganesh1pal at gmail.com Sat May 14 10:28:24 2016 From: ganesh1pal at gmail.com (Ganesh Pal) Date: Sat, 14 May 2016 19:58:24 +0530 Subject: Skipping test using unittest SkipTest and exit status In-Reply-To: <5736061c$0$1604$c3e8da3$5496439d@news.astraweb.com> References: <5736061c$0$1604$c3e8da3$5496439d@news.astraweb.com> Message-ID: > > > Hi Team, > > > > Iam on python 2.7 and Linux . I need inputs on the below program , > > "I am" is two words, not one. I hope you wouldn't write "Youare" > or "Heis" :-) Whenever you write "Iam", I read it as the name "Ian", which > is very distracting. > > I am lazy fellow and you are smart guy. just a sentence with few words . Take care :) > > Iam skipping the unittest from setUpClass in following way # raise > > unittest.SkipTest(message) > > > > The test are getting skipped but I have two problem . > > > > (1) This script is in turn read by other scripts which considers the > > test have passed based on the scripts return code , but the test have > > actually been skipped , How do include an exit status to indicates > that > > the test have failed > > But the test *hasn't* failed. A skipped test is not a failed test. > > If you want the test to count as failed, you must let it fail. You can use > the fail() method for that. > > https://docs.python.org/2/library/unittest.html#unittest.TestCase.fail > > 1. How about raising failureException : I was thinking of using failureException instead of fail() method , If I replace my code with raise unittest.TestCase.failureException("class setup failed") The script show the below output , this looks fine for me. Do you see any problems with this ? gpal-ae9703e-1# python unitest1.py ERROR:root:Failed scanning E ====================================================================== ERROR: setUpClass (__main__.ScanTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "unitest1.py", line 20, in setUpClass raise unittest.TestCase.failureException("class setup failed") AssertionError: class setup failed ---------------------------------------------------------------------- Ran 0 tests in 0.000s FAILED (errors=1) 2. I find assert and raise RunTimeError also fitting my program ,please suggest whats best form unittest fixture point of view. if not self.scan: logging.error("Failed scanning ") assert False, "Class setup failed skipping test" if not self.scan: logging.error("Failed scanning ") raise RuntimeError. My overall ,idea is Setup class fails then don't run any of the next statements and exit the tests. Regards, Ganesh From michael.selik at gmail.com Sat May 14 10:29:53 2016 From: michael.selik at gmail.com (Michael Selik) Date: Sat, 14 May 2016 14:29:53 +0000 Subject: How to put back a number-based index In-Reply-To: <477298994.3432356.1463220990479.JavaMail.yahoo@mail.yahoo.com> References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> <2075459214.3318248.1463172701996.JavaMail.yahoo@mail.yahoo.com> <1584945027.3340053.1463176318146.JavaMail.yahoo@mail.yahoo.com> <363052112.3322670.1463195744354.JavaMail.yahoo@mail.yahoo.com> <477298994.3432356.1463220990479.JavaMail.yahoo@mail.yahoo.com> Message-ID: David, it sounds like you'll need a thorough introduction to the basics of Python. Check out the tutorial: https://docs.python.org/3/tutorial/ On Sat, May 14, 2016 at 6:19 AM David Shi wrote: > Hello, Michael, > > I discovered that the problem is "two columns of data are put together" > and "are recognised as one column". > > This is very strange. I would like to understand the subject well. > > And, how many ways are there to investigate into the nature of objects > dynamically? > > Some object types only get shown as an object. Are there anything to be > typed in Python, to reveal objects. > > Regards. > > David > > > On Saturday, 14 May 2016, 4:30, Michael Selik > wrote: > > > What were you hoping to get from ``df[0]``? > When you say it "yields nothing" do you mean it raised an error? What was > the error message? > > Have you tried a Google search for "pandas set index"? > > http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.set_index.html > > On Fri, May 13, 2016 at 11:18 PM David Shi wrote: > > Hello, Michael, > > I tried to discover the problem. > > df[0] yields nothing > df[1] yields nothing > df[2] yields nothing > > However, df[3] gives the following: > > sid > -9223372036854775808 NaN > 1 133738.70 > 4 295256.11 > 5 137733.09 > 6 409413.58 > 8 269600.97 > 9 12852.94 > > > Can we split this back to normal? or turn it into a dictionary, so that I can put values back properly. > > > I like to use sid as index, some way. > > > Regards. > > > David > > > > On Friday, 13 May 2016, 22:58, Michael Selik > wrote: > > > What have code you tried? What error message are you receiving? > > On Fri, May 13, 2016, 5:54 PM David Shi wrote: > > Hello, Michael, > > How to convert a float type column into an integer or label or string type? > > > On Friday, 13 May 2016, 22:02, Michael Selik > wrote: > > > To clarify that you're specifying the index as a label, use df.iloc > > >>> df = pd.DataFrame({'X': range(4)}, index=list('abcd')) > >>> df > X > a 0 > b 1 > c 2 > d 3 > >>> df.loc['a'] > X 0 > Name: a, dtype: int64 > >>> df.iloc[0] > X 0 > Name: a, dtype: int64 > > On Fri, May 13, 2016 at 4:54 PM David Shi wrote: > > Dear Michael, > > To avoid complication, I only groupby using one column. > > It is OK now. But, how to refer to new row index? How do I use floating > index? > > Float64Index([ 1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, > 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, > 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, > 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, > 51.0, 53.0, 54.0, 55.0, 56.0], > dtype='float64', name=u'StateFIPS') > > > Regards. > > > David > > > > On Friday, 13 May 2016, 21:43, Michael Selik > wrote: > > > Here's an example. > > >>> import pandas as pd > >>> df = pd.DataFrame({'group': list('AB') * 2, 'data': range(4)}, > index=list('wxyz')) > >>> df > data group > w 0 A > x 1 B > y 2 A > z 3 B > >>> df = df.reset_index() > >>> df > index data group > 0 w 0 A > 1 x 1 B > 2 y 2 A > 3 z 3 B > >>> df.groupby('group').max() > index data > group > A y 2 > B z 3 > > If that doesn't help, you'll need to explain what you're trying to > accomplish in detail -- what variables you started with, what > transformations you want to do, and what variables you hope to have when > finished. > > On Fri, May 13, 2016 at 4:36 PM David Shi wrote: > > Hello, Michael, > > I changed groupby with one column. > > The index is different. > > Index([ u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', > u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', > u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', > u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', > u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', > u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', > u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', > u'WY'], > dtype='object', name=0) > > > How to use this index? > > > Regards. > > > David > > > > On Friday, 13 May 2016, 21:19, David Shi wrote: > > > Hello, Michael, > > I typed in df.index > > I got the following > > MultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY']], > labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 45, 44, 46, 48, 47, 49]], > names=[u'StateFIPS', 0]) > > Regards. > > > David > > > > On Friday, 13 May 2016, 21:11, David Shi wrote: > > > Dear Michael, > > I have done a number of operation in between. > > Providing that information does not help you > > How to reset index after grouping and various operations is of interest. > > How to type in a command to find out its current dataframe? > > Regards. > > David > > > On Friday, 13 May 2016, 20:58, Michael Selik > wrote: > > > Just in case I misunderstood, why don't you make a little example of > before and after the grouping? This mailing list does not accept > attachments, so you'll have to make do with pasting a few rows of > comma-separated or tab-separated values. > > On Fri, May 13, 2016 at 3:56 PM Michael Selik > wrote: > > In order to preserve your index after the aggregation, you need to make > sure it is considered a data column (via reset_index) and then choose how > your aggregation will operate on that column. > > On Fri, May 13, 2016 at 3:29 PM David Shi wrote: > > Hello, Michael, > > Why reset_index before grouping? > > Regards. > > David > > > On Friday, 13 May 2016, 17:57, Michael Selik > wrote: > > > > > On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list < > python-list at python.org> wrote: > > I lost my indexes after grouping in Pandas. > I managed to rest_index and got back the index column. > But How can I get back a index row? > > > Was the grouping an aggregation? If so, the original indexes are > meaningless. What you could do is reset_index before the grouping and when > you aggregate decide how to handle the formerly-known-as-index column (min, > max, mean, ?). > > > > > > > > > > > > > > > > > From songofacandy at gmail.com Sat May 14 10:32:12 2016 From: songofacandy at gmail.com (INADA Naoki) Date: Sat, 14 May 2016 23:32:12 +0900 Subject: Performance with and without the garbage collector In-Reply-To: <5737325a$0$1610$c3e8da3$5496439d@news.astraweb.com> References: <5737325a$0$1610$c3e8da3$5496439d@news.astraweb.com> Message-ID: Mark and Sweep GC is triggered when many new **tracked** objects are created and not destroyed. Since integer doesn't have reference to another objects, it's not tracked object. So your sample code doesn't show difference of GC. You can see gc stats via gc.set_debug(gc.DEBUG_STATS). Play this script with time command. import gc class Tree: def __init__(self, n): if n > 0: self.a = Tree(n-1) self.b = Tree(n-1) self.n = n def main(): #gc.set_debug(gc.DEBUG_STATS) #gc.disable() #gc.set_threshold(150000, 10, 10) for _ in range(10): Tree(16) main() On Sat, May 14, 2016 at 11:12 PM, Steven D'Aprano wrote: > Just for kicks, I've been playing around with running code snippets with > and > without the garbage collector enabled, looking to see if it will make any > obvious difference to performance. > > So far, I haven't found any. > > For instance, I tried: > > a = [i**3 for i in range(2000000)] > del a[:] > > thinking that garbage collecting almost two million ints would surely show > some performance difference, but it doesn't. > > Is anyone able to demonstrate a replicable performance impact due to > garbage > collection? > > > > -- > Steven > > -- > https://mail.python.org/mailman/listinfo/python-list > -- INADA Naoki From michael.selik at gmail.com Sat May 14 10:33:49 2016 From: michael.selik at gmail.com (Michael Selik) Date: Sat, 14 May 2016 14:33:49 +0000 Subject: How to put back a number-based index In-Reply-To: References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> <2075459214.3318248.1463172701996.JavaMail.yahoo@mail.yahoo.com> <1584945027.3340053.1463176318146.JavaMail.yahoo@mail.yahoo.com> <363052112.3322670.1463195744354.JavaMail.yahoo@mail.yahoo.com> <477298994.3432356.1463220990479.JavaMail.yahoo@mail.yahoo.com> Message-ID: You might also be interested in "Python for Data Analysis" for a thorough discussion of Pandas. http://shop.oreilly.com/product/0636920023784.do On Sat, May 14, 2016 at 10:29 AM Michael Selik wrote: > David, it sounds like you'll need a thorough introduction to the basics of > Python. > Check out the tutorial: https://docs.python.org/3/tutorial/ > > On Sat, May 14, 2016 at 6:19 AM David Shi wrote: > >> Hello, Michael, >> >> I discovered that the problem is "two columns of data are put together" >> and "are recognised as one column". >> >> This is very strange. I would like to understand the subject well. >> >> And, how many ways are there to investigate into the nature of objects >> dynamically? >> >> Some object types only get shown as an object. Are there anything to be >> typed in Python, to reveal objects. >> >> Regards. >> >> David >> >> >> On Saturday, 14 May 2016, 4:30, Michael Selik >> wrote: >> >> >> What were you hoping to get from ``df[0]``? >> When you say it "yields nothing" do you mean it raised an error? What was >> the error message? >> >> Have you tried a Google search for "pandas set index"? >> >> http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.set_index.html >> >> On Fri, May 13, 2016 at 11:18 PM David Shi wrote: >> >> Hello, Michael, >> >> I tried to discover the problem. >> >> df[0] yields nothing >> df[1] yields nothing >> df[2] yields nothing >> >> However, df[3] gives the following: >> >> sid >> -9223372036854775808 NaN >> 1 133738.70 >> 4 295256.11 >> 5 137733.09 >> 6 409413.58 >> 8 269600.97 >> 9 12852.94 >> >> >> Can we split this back to normal? or turn it into a dictionary, so that I can put values back properly. >> >> >> I like to use sid as index, some way. >> >> >> Regards. >> >> >> David >> >> >> >> On Friday, 13 May 2016, 22:58, Michael Selik >> wrote: >> >> >> What have code you tried? What error message are you receiving? >> >> On Fri, May 13, 2016, 5:54 PM David Shi wrote: >> >> Hello, Michael, >> >> How to convert a float type column into an integer or label or string >> type? >> >> >> On Friday, 13 May 2016, 22:02, Michael Selik >> wrote: >> >> >> To clarify that you're specifying the index as a label, use df.iloc >> >> >>> df = pd.DataFrame({'X': range(4)}, index=list('abcd')) >> >>> df >> X >> a 0 >> b 1 >> c 2 >> d 3 >> >>> df.loc['a'] >> X 0 >> Name: a, dtype: int64 >> >>> df.iloc[0] >> X 0 >> Name: a, dtype: int64 >> >> On Fri, May 13, 2016 at 4:54 PM David Shi wrote: >> >> Dear Michael, >> >> To avoid complication, I only groupby using one column. >> >> It is OK now. But, how to refer to new row index? How do I use floating >> index? >> >> Float64Index([ 1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, >> 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, >> 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, >> 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, >> 51.0, 53.0, 54.0, 55.0, 56.0], >> dtype='float64', name=u'StateFIPS') >> >> >> Regards. >> >> >> David >> >> >> >> On Friday, 13 May 2016, 21:43, Michael Selik >> wrote: >> >> >> Here's an example. >> >> >>> import pandas as pd >> >>> df = pd.DataFrame({'group': list('AB') * 2, 'data': range(4)}, >> index=list('wxyz')) >> >>> df >> data group >> w 0 A >> x 1 B >> y 2 A >> z 3 B >> >>> df = df.reset_index() >> >>> df >> index data group >> 0 w 0 A >> 1 x 1 B >> 2 y 2 A >> 3 z 3 B >> >>> df.groupby('group').max() >> index data >> group >> A y 2 >> B z 3 >> >> If that doesn't help, you'll need to explain what you're trying to >> accomplish in detail -- what variables you started with, what >> transformations you want to do, and what variables you hope to have when >> finished. >> >> On Fri, May 13, 2016 at 4:36 PM David Shi wrote: >> >> Hello, Michael, >> >> I changed groupby with one column. >> >> The index is different. >> >> Index([ u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', >> u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', >> u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', >> u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', >> u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', >> u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', >> u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', >> u'WY'], >> dtype='object', name=0) >> >> >> How to use this index? >> >> >> Regards. >> >> >> David >> >> >> >> On Friday, 13 May 2016, 21:19, David Shi wrote: >> >> >> Hello, Michael, >> >> I typed in df.index >> >> I got the following >> >> MultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY']], >> labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 45, 44, 46, 48, 47, 49]], >> names=[u'StateFIPS', 0]) >> >> Regards. >> >> >> David >> >> >> >> On Friday, 13 May 2016, 21:11, David Shi wrote: >> >> >> Dear Michael, >> >> I have done a number of operation in between. >> >> Providing that information does not help you >> >> How to reset index after grouping and various operations is of interest. >> >> How to type in a command to find out its current dataframe? >> >> Regards. >> >> David >> >> >> On Friday, 13 May 2016, 20:58, Michael Selik >> wrote: >> >> >> Just in case I misunderstood, why don't you make a little example of >> before and after the grouping? This mailing list does not accept >> attachments, so you'll have to make do with pasting a few rows of >> comma-separated or tab-separated values. >> >> On Fri, May 13, 2016 at 3:56 PM Michael Selik >> wrote: >> >> In order to preserve your index after the aggregation, you need to make >> sure it is considered a data column (via reset_index) and then choose how >> your aggregation will operate on that column. >> >> On Fri, May 13, 2016 at 3:29 PM David Shi wrote: >> >> Hello, Michael, >> >> Why reset_index before grouping? >> >> Regards. >> >> David >> >> >> On Friday, 13 May 2016, 17:57, Michael Selik >> wrote: >> >> >> >> >> On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list < >> python-list at python.org> wrote: >> >> I lost my indexes after grouping in Pandas. >> I managed to rest_index and got back the index column. >> But How can I get back a index row? >> >> >> Was the grouping an aggregation? If so, the original indexes are >> meaningless. What you could do is reset_index before the grouping and when >> you aggregate decide how to handle the formerly-known-as-index column (min, >> max, mean, ?). >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> From wrightalexw at gmail.com Sat May 14 10:43:52 2016 From: wrightalexw at gmail.com (alex wright) Date: Sat, 14 May 2016 10:43:52 -0400 Subject: How to put back a number-based index In-Reply-To: References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> <2075459214.3318248.1463172701996.JavaMail.yahoo@mail.yahoo.com> <1584945027.3340053.1463176318146.JavaMail.yahoo@mail.yahoo.com> <363052112.3322670.1463195744354.JavaMail.yahoo@mail.yahoo.com> <477298994.3432356.1463220990479.JavaMail.yahoo@mail.yahoo.com> Message-ID: I have recently been going through "Data Science From Scratch" which may be interesting. There is a podcast with the author on talk python to me. https://talkpython.fm/episodes/show/56/data-science-from-scratch On Sat, May 14, 2016 at 10:33 AM, Michael Selik wrote: > You might also be interested in "Python for Data Analysis" for a thorough > discussion of Pandas. > http://shop.oreilly.com/product/0636920023784.do > > On Sat, May 14, 2016 at 10:29 AM Michael Selik > wrote: > > > David, it sounds like you'll need a thorough introduction to the basics > of > > Python. > > Check out the tutorial: https://docs.python.org/3/tutorial/ > > > > On Sat, May 14, 2016 at 6:19 AM David Shi wrote: > > > >> Hello, Michael, > >> > >> I discovered that the problem is "two columns of data are put together" > >> and "are recognised as one column". > >> > >> This is very strange. I would like to understand the subject well. > >> > >> And, how many ways are there to investigate into the nature of objects > >> dynamically? > >> > >> Some object types only get shown as an object. Are there anything to be > >> typed in Python, to reveal objects. > >> > >> Regards. > >> > >> David > >> > >> > >> On Saturday, 14 May 2016, 4:30, Michael Selik > >> wrote: > >> > >> > >> What were you hoping to get from ``df[0]``? > >> When you say it "yields nothing" do you mean it raised an error? What > was > >> the error message? > >> > >> Have you tried a Google search for "pandas set index"? > >> > >> > http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.set_index.html > >> > >> On Fri, May 13, 2016 at 11:18 PM David Shi > wrote: > >> > >> Hello, Michael, > >> > >> I tried to discover the problem. > >> > >> df[0] yields nothing > >> df[1] yields nothing > >> df[2] yields nothing > >> > >> However, df[3] gives the following: > >> > >> sid > >> -9223372036854775808 NaN > >> 1 133738.70 > >> 4 295256.11 > >> 5 137733.09 > >> 6 409413.58 > >> 8 269600.97 > >> 9 12852.94 > >> > >> > >> Can we split this back to normal? or turn it into a dictionary, so > that I can put values back properly. > >> > >> > >> I like to use sid as index, some way. > >> > >> > >> Regards. > >> > >> > >> David > >> > >> > >> > >> On Friday, 13 May 2016, 22:58, Michael Selik > >> wrote: > >> > >> > >> What have code you tried? What error message are you receiving? > >> > >> On Fri, May 13, 2016, 5:54 PM David Shi wrote: > >> > >> Hello, Michael, > >> > >> How to convert a float type column into an integer or label or string > >> type? > >> > >> > >> On Friday, 13 May 2016, 22:02, Michael Selik > >> wrote: > >> > >> > >> To clarify that you're specifying the index as a label, use df.iloc > >> > >> >>> df = pd.DataFrame({'X': range(4)}, index=list('abcd')) > >> >>> df > >> X > >> a 0 > >> b 1 > >> c 2 > >> d 3 > >> >>> df.loc['a'] > >> X 0 > >> Name: a, dtype: int64 > >> >>> df.iloc[0] > >> X 0 > >> Name: a, dtype: int64 > >> > >> On Fri, May 13, 2016 at 4:54 PM David Shi > wrote: > >> > >> Dear Michael, > >> > >> To avoid complication, I only groupby using one column. > >> > >> It is OK now. But, how to refer to new row index? How do I use > floating > >> index? > >> > >> Float64Index([ 1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, > 13.0, 16.0, > >> 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, > 26.0, 27.0, > >> 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, > 37.0, 38.0, > >> 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, > 49.0, 50.0, > >> 51.0, 53.0, 54.0, 55.0, 56.0], > >> dtype='float64', name=u'StateFIPS') > >> > >> > >> Regards. > >> > >> > >> David > >> > >> > >> > >> On Friday, 13 May 2016, 21:43, Michael Selik > >> wrote: > >> > >> > >> Here's an example. > >> > >> >>> import pandas as pd > >> >>> df = pd.DataFrame({'group': list('AB') * 2, 'data': range(4)}, > >> index=list('wxyz')) > >> >>> df > >> data group > >> w 0 A > >> x 1 B > >> y 2 A > >> z 3 B > >> >>> df = df.reset_index() > >> >>> df > >> index data group > >> 0 w 0 A > >> 1 x 1 B > >> 2 y 2 A > >> 3 z 3 B > >> >>> df.groupby('group').max() > >> index data > >> group > >> A y 2 > >> B z 3 > >> > >> If that doesn't help, you'll need to explain what you're trying to > >> accomplish in detail -- what variables you started with, what > >> transformations you want to do, and what variables you hope to have when > >> finished. > >> > >> On Fri, May 13, 2016 at 4:36 PM David Shi > wrote: > >> > >> Hello, Michael, > >> > >> I changed groupby with one column. > >> > >> The index is different. > >> > >> Index([ u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', > u'DC', > >> u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', > u'IN', > >> u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', > u'MI', > >> u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', > u'NE', > >> u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', > u'OK', > >> u'OR', u'PA', u'RI', u'SC', u'SD', u'State', > u'TN', > >> u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', > u'WV', > >> u'WY'], > >> dtype='object', name=0) > >> > >> > >> How to use this index? > >> > >> > >> Regards. > >> > >> > >> David > >> > >> > >> > >> On Friday, 13 May 2016, 21:19, David Shi wrote: > >> > >> > >> Hello, Michael, > >> > >> I typed in df.index > >> > >> I got the following > >> > >> MultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, > 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, > 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, > 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, > 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', > u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', > u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', > u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', > u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', > u'WI', u'WV', u'WY']], > >> labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, > 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, > 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, > 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, > 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, > 45, 44, 46, 48, 47, 49]], > >> names=[u'StateFIPS', 0]) > >> > >> Regards. > >> > >> > >> David > >> > >> > >> > >> On Friday, 13 May 2016, 21:11, David Shi wrote: > >> > >> > >> Dear Michael, > >> > >> I have done a number of operation in between. > >> > >> Providing that information does not help you > >> > >> How to reset index after grouping and various operations is of interest. > >> > >> How to type in a command to find out its current dataframe? > >> > >> Regards. > >> > >> David > >> > >> > >> On Friday, 13 May 2016, 20:58, Michael Selik > >> wrote: > >> > >> > >> Just in case I misunderstood, why don't you make a little example of > >> before and after the grouping? This mailing list does not accept > >> attachments, so you'll have to make do with pasting a few rows of > >> comma-separated or tab-separated values. > >> > >> On Fri, May 13, 2016 at 3:56 PM Michael Selik > >> wrote: > >> > >> In order to preserve your index after the aggregation, you need to make > >> sure it is considered a data column (via reset_index) and then choose > how > >> your aggregation will operate on that column. > >> > >> On Fri, May 13, 2016 at 3:29 PM David Shi > wrote: > >> > >> Hello, Michael, > >> > >> Why reset_index before grouping? > >> > >> Regards. > >> > >> David > >> > >> > >> On Friday, 13 May 2016, 17:57, Michael Selik > >> wrote: > >> > >> > >> > >> > >> On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list < > >> python-list at python.org> wrote: > >> > >> I lost my indexes after grouping in Pandas. > >> I managed to rest_index and got back the index column. > >> But How can I get back a index row? > >> > >> > >> Was the grouping an aggregation? If so, the original indexes are > >> meaningless. What you could do is reset_index before the grouping and > when > >> you aggregate decide how to handle the formerly-known-as-index column > (min, > >> max, mean, ?). > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > -- > https://mail.python.org/mailman/listinfo/python-list > -- "On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question." -Charles Babbage, 19th century English mathematician, philosopher, inventor and mechanical engineer who originated the concept of a programmable computer. From michael.selik at gmail.com Sat May 14 10:44:10 2016 From: michael.selik at gmail.com (Michael Selik) Date: Sat, 14 May 2016 14:44:10 +0000 Subject: Why online forums have bad behaviour In-Reply-To: <85vb2gdcjc.fsf@benfinney.id.au> References: <5730420b$0$1509$c3e8da3$5496439d@news.astraweb.com> <57349595.6000702@gmail.com> <85vb2jdq6c.fsf_-_@benfinney.id.au> <85vb2gdcjc.fsf@benfinney.id.au> Message-ID: On Sat, May 14, 2016 at 8:57 AM Ben Finney wrote: > If you dislike someone's behaviour, consider that they may not have a > well-thought-out or coherent rason for it; and, if pressed to come up > with a reason, we will employ all our faculties to *make up* a reason > (typically without being aware that's what we're doing!) that somehow > depicts us in a better light than others. > Since this was an assertion, I thought it'd be helpful to cite some evidence. There's some very interesting studies of split-brain folks. If the corpus callosum is severed, the right and left hemispheres cannot communicate, allowing a researcher to talk to each hemisphere separately. An action determined by the right hemisphere will be observed by the left. If then asked why the action was taken, the left hemisphere will make something up, completely unaware that it was not the one who thought of the action. I couldn't find a link to the research I'm referencing, but this Wikipedia article is getting at a similar topic ( https://en.wikipedia.org/wiki/Left_brain_interpreter) and one could probably chase down the original research via its citations. From rosuav at gmail.com Sat May 14 10:58:22 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 15 May 2016 00:58:22 +1000 Subject: Skipping test using unittest SkipTest and exit status In-Reply-To: References: <5736061c$0$1604$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sun, May 15, 2016 at 12:28 AM, Ganesh Pal wrote: > The script show the below output , this looks fine for me. Do you see any > problems with this ? > > gpal-ae9703e-1# python unitest1.py > ERROR:root:Failed scanning > E > ====================================================================== > ERROR: setUpClass (__main__.ScanTest) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "unitest1.py", line 20, in setUpClass > raise unittest.TestCase.failureException("class setup failed") > AssertionError: class setup failed > > ---------------------------------------------------------------------- > Ran 0 tests in 0.000s > > FAILED (errors=1) > > 2. I find assert and raise RunTimeError also fitting my program ,please > suggest whats best form unittest fixture point of view. > > > if not self.scan: > logging.error("Failed scanning ") > assert False, "Class setup failed skipping test" > > if not self.scan: > logging.error("Failed scanning ") > raise RuntimeError. > > My overall ,idea is Setup class fails then don't run any of the next > statements and exit the tests. There are three quite different things happening in your three examples. 1) When a test *fails*, it means that the test ran successfully, but didn't give the expected result. (This is sometimes a good thing - for example, in test-driven development, you first create a test, then run the test suite and see that it correctly fails, and then implement the feature or fix, at which point the test suite will pass.) A test failure happens when you call assertEqual with two unequal values, for instance. 2) A test *exception* is generally a failure of the test suite itself. It means that your code ran, but something went wrong, and an exception was raised. That's what happens when you raise arbitrary exceptions, including RuntimeError and AssertionError. 3) An *assertion failure* is (conceptually) an error in some function's preconditions/postconditions, or a complete and utter muck-up in implementation. It's something that should never actually happen. It's something where you're legitimately allowed to skip over every assert statement - in fact, that's exactly what happens when you run Python in optimized mode. Every exception has its purpose. With a lot of them, you can find out that purpose by looking at its docstring: >>> import builtins >>> for name in dir(builtins): ... obj = getattr(builtins, name) ... if isinstance(obj, type) and issubclass(obj, BaseException): ... print("%s: %s" % (obj.__name__, obj.__doc__.split("\n")[0])) ... ArithmeticError: Base class for arithmetic errors. AssertionError: Assertion failed. AttributeError: Attribute not found. BaseException: Common base class for all exceptions BlockingIOError: I/O operation would block. BrokenPipeError: Broken pipe. BufferError: Buffer error. BytesWarning: Base class for warnings about bytes and buffer related problems, mostly ChildProcessError: Child process error. ConnectionAbortedError: Connection aborted. ConnectionError: Connection error. ConnectionRefusedError: Connection refused. ConnectionResetError: Connection reset. DeprecationWarning: Base class for warnings about deprecated features. EOFError: Read beyond end of file. OSError: Base class for I/O related errors. Exception: Common base class for all non-exit exceptions. FileExistsError: File already exists. FileNotFoundError: File not found. FloatingPointError: Floating point operation failed. FutureWarning: Base class for warnings about constructs that will change semantically GeneratorExit: Request that a generator exit. OSError: Base class for I/O related errors. ImportError: Import can't find module, or can't find name in module. ImportWarning: Base class for warnings about probable mistakes in module imports IndentationError: Improper indentation. IndexError: Sequence index out of range. InterruptedError: Interrupted by signal. IsADirectoryError: Operation doesn't work on directories. KeyError: Mapping key not found. KeyboardInterrupt: Program interrupted by user. LookupError: Base class for lookup errors. MemoryError: Out of memory. NameError: Name not found globally. NotADirectoryError: Operation only works on directories. NotImplementedError: Method or function hasn't been implemented yet. OSError: Base class for I/O related errors. OverflowError: Result too large to be represented. PendingDeprecationWarning: Base class for warnings about features which will be deprecated PermissionError: Not enough permissions. ProcessLookupError: Process not found. RecursionError: Recursion limit exceeded. ReferenceError: Weak ref proxy used after referent went away. ResourceWarning: Base class for warnings about resource usage. RuntimeError: Unspecified run-time error. RuntimeWarning: Base class for warnings about dubious runtime behavior. StopAsyncIteration: Signal the end from iterator.__anext__(). StopIteration: Signal the end from iterator.__next__(). SyntaxError: Invalid syntax. SyntaxWarning: Base class for warnings about dubious syntax. SystemError: Internal error in the Python interpreter. SystemExit: Request to exit from the interpreter. TabError: Improper mixture of spaces and tabs. TimeoutError: Timeout expired. TypeError: Inappropriate argument type. UnboundLocalError: Local name referenced but not bound to a value. UnicodeDecodeError: Unicode decoding error. UnicodeEncodeError: Unicode encoding error. UnicodeError: Unicode related error. UnicodeTranslateError: Unicode translation error. UnicodeWarning: Base class for warnings about Unicode related problems, mostly UserWarning: Base class for warnings generated by user code. ValueError: Inappropriate argument value (of correct type). Warning: Base class for warning categories. ZeroDivisionError: Second argument to a division or modulo operation was zero. Granted, not all of those docstrings tell us much (BufferError - "Buffer error."), but it's a start. When you're picking something to raise, try to be as appropriate as possible (don't signal a database connection problem with TabError, for instance), and if nothing seems particularly right, subclass the nearest match and make your own. Neither RuntimeError nor AssertionError is the right way to say "my test shouldn't be run"; you should decide whether you want the test to fail or report an exception, and if you want the latter, check the list above for something more appropriate. Hope that helps! ChrisA From laurent.pointal at free.fr Sat May 14 13:37:11 2016 From: laurent.pointal at free.fr (Laurent Pointal) Date: Sat, 14 May 2016 19:37:11 +0200 Subject: Performance with and without the garbage collector References: <5737325a$0$1610$c3e8da3$5496439d@news.astraweb.com> Message-ID: <57376247$0$26276$426a74cc@news.free.fr> Steven D'Aprano wrote: > Just for kicks, I've been playing around with running code snippets with > and without the garbage collector enabled, looking to see if it will make > any obvious difference to performance. > > So far, I haven't found any. > > For instance, I tried: > > a = [i**3 for i in range(2000000)] > del a[:] > > thinking that garbage collecting almost two million ints would surely show > some performance difference, but it doesn't. > > Is anyone able to demonstrate a replicable performance impact due to > garbage collection? As CPython objects are reference-counted, you may see the GC in action if you setup some cycles in your script (here, your two million ints are removed ? except for some small ints which remain alive for internal optimization). >>> a = [i**3 for i in range(2000000)] >>> b = [a] >>> a.append(b) >>> del a # a remain referenced in b list >>> del b # b remain referenced in a list Now, how to test for performance impact? A+ Laurent. From no.email at nospam.invalid Sat May 14 14:41:55 2016 From: no.email at nospam.invalid (Paul Rubin) Date: Sat, 14 May 2016 11:41:55 -0700 Subject: Performance with and without the garbage collector References: <5737325a$0$1610$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87d1oopjm4.fsf@jester.gateway.pace.com> Steven D'Aprano writes: > Is anyone able to demonstrate a replicable performance impact due to > garbage collection? As Laurent describes, Python uses refcounting, and then does some gc in case there was dead circular structure that the refcounting missed. Your example had no circular structure so there was nothing to gc, and after refcounting freed the contents of your big array, there was hardly anything for the gc to even inspect. A standard example where you want gc is something like an XML DOM: a big tree of nodes where the parents point to the children and the children point back to the parents, so there are cyclic references all over the place. This can also be handled semi-manually with weakrefs and it's sometimes done that way in gc'd systems to improve performance, but gc is the fully automatic way to deal with it. From pyhack at outlook.com Sat May 14 17:41:10 2016 From: pyhack at outlook.com (Pyhack Blog) Date: Sat, 14 May 2016 21:41:10 +0000 Subject: Help AES Implemetation Message-ID: Hi, I have two AES implementation programs: AES-1: http://pastebin.com/TrQ5iaxc AES-2: http://pastebin.com/mXRyprKL I have one binary file which is encrypted with some other AES program and it is getting decrypted with AES-1 but not with AES-2. I have shared both the programs link with you can someone please help me in identifying where exactly AES-2 is wrong in implementation. Regards PyHack From steve at pearwood.info Sat May 14 23:20:24 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 15 May 2016 13:20:24 +1000 Subject: Performance with and without the garbage collector References: <5737325a$0$1610$c3e8da3$5496439d@news.astraweb.com> Message-ID: <5737eafa$0$1614$c3e8da3$5496439d@news.astraweb.com> For some reason, this post from INADA Naoki shows up on the python-list email archives, but not comp.lang.python, which is what I use. INADA Naoki wrote at Sat May 14 10:32:12 EDT 2016: > Mark and Sweep GC is triggered when many new **tracked** objects are > created and not destroyed. > > Since integer doesn't have reference to another objects, it's not tracked > object. So your sample code doesn't show difference of GC. > You can see gc stats via gc.set_debug(gc.DEBUG_STATS). Thanks for the explanation! So lists, tuples, sets etc would all be tracked? Following the suggestion to use this code: class Tree: def __init__(self, n): if n > 0: self.a = Tree(n-1) self.b = Tree(n-1) self.n = n def main(): for _ in range(10): Tree(16) I can see a significant difference in performance. With the gc on, running main() on my computer takes 11.2 seconds, with the gc off, it takes 7.5 seconds. Nice, and thank you! -- Steven From g.rodola at gmail.com Sun May 15 04:13:09 2016 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sun, 15 May 2016 04:13:09 -0400 Subject: ANN: psutil 4.2.0 with Windows service support is out Message-ID: Full blog post: http://grodola.blogspot.com/2016/05/psutil-420-windows-services-and-python.html -- Giampaolo - http://grodola.blogspot.com From g.rodola at gmail.com Sun May 15 04:13:10 2016 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sun, 15 May 2016 04:13:10 -0400 Subject: ANN: psutil 4.2.0 with Windows service support is out Message-ID: Full blog post: http://grodola.blogspot.com/2016/05/psutil-420-windows-services-and-python.html -- Giampaolo - http://grodola.blogspot.com From cl at isbd.net Sun May 15 04:47:59 2016 From: cl at isbd.net (cl at isbd.net) Date: Sun, 15 May 2016 09:47:59 +0100 Subject: How to get a directory list sorted by date? Message-ID: I have a little Python program I wrote myself which copies images from a camera (well, any mounted directory) to my picture archive. The picture archive is simply a directory hierarchy of dates with years at the top, then months, then days. My Python program simply extracts the date from the image (put there by the camera) and copies the image to the appropriate place in the picture archive. There is one small bit of speeding up done by my program, it checks if the file is already in the archive and doesn't copy if it's already there. I don't generally clear pictures off the camera memory cards so it's often the case that most of the pcitures are already in the archive and all I actually want to do is copy the last couple of weeks of new pictures. As camera memory card sizes get bigger (and images get more detailed) this is beginning to take rather a long time. So, to the question, how can I get a list of the files on the memory card (i.e. files in a directory) in date/time order, latest first. I can then copy them until the first one I find which is already in the archive. The file date may not *exactly* match the date/time in the image file but it will produce the right order which is what I need. What I want is a list in the order produced by:- ls --sort=time I suppose I could use a system call but that seems a little inelegant. -- Chris Green ? From __peter__ at web.de Sun May 15 05:46:13 2016 From: __peter__ at web.de (Peter Otten) Date: Sun, 15 May 2016 11:46:13 +0200 Subject: How to get a directory list sorted by date? References: Message-ID: cl at isbd.net wrote: > I have a little Python program I wrote myself which copies images from > a camera (well, any mounted directory) to my picture archive. The > picture archive is simply a directory hierarchy of dates with years at > the top, then months, then days. > > My Python program simply extracts the date from the image (put there > by the camera) and copies the image to the appropriate place in the > picture archive. > > There is one small bit of speeding up done by my program, it checks if > the file is already in the archive and doesn't copy if it's already > there. I don't generally clear pictures off the camera memory cards > so it's often the case that most of the pcitures are already in the > archive and all I actually want to do is copy the last couple of weeks > of new pictures. > > As camera memory card sizes get bigger (and images get more detailed) > this is beginning to take rather a long time. > > So, to the question, how can I get a list of the files on the memory > card (i.e. files in a directory) in date/time order, latest first. I > can then copy them until the first one I find which is already in the > archive. The file date may not *exactly* match the date/time in the > image file but it will produce the right order which is what I need. > > What I want is a list in the order produced by:- > ls --sort=time > > I suppose I could use a system call but that seems a little inelegant. Personally I'd be worried that something goes wrong with this approach (one changed file time leading you to discard many pictures), but to answer the question in the subject: you can build the path from name and folder, then find the time with os.path.getmtime(), then sort: def sorted_dir(folder): def getmtime(name): path = os.path.join(folder, name) return os.path.getmtime(path) return sorted(os.listdir(folder), key=getmtime, reverse=True) The same idea will work with pathlib and os.scandir(): def _getmtime(entry): return entry.stat().st_mtime def sd_sorted_dir(folder): return sorted(os.scandir(folder), key=_getmtime, reverse=True) def pl_sorted_dir(folder): """folder: a pathlib.Path() object""" return sorted(folder.iterdir(), key=_getmtime, reverse=True) From tshortik at gmx.de Sun May 15 06:01:19 2016 From: tshortik at gmx.de (=?UTF-8?Q?Dirk_B=c3=a4chle?=) Date: Sun, 15 May 2016 12:01:19 +0200 Subject: Design: Idiom for classes and methods that are customizable by the user? In-Reply-To: <87shxmh5aj.fsf@elektro.pacujo.net> References: <5734ECDF.7070006@gmx.de> <87futnaqb2.fsf@elektro.pacujo.net> <573506C9.3070206@gmx.de> <87shxmh5aj.fsf@elektro.pacujo.net> Message-ID: <573848EF.6010700@gmx.de> Hi Marko, Am 13.05.2016 um 07:53 schrieb Marko Rauhamaa: > Dirk B?chle : > >>> For example, why do you need a key? Couldn't you simply pass the task >>> master class as an argument? >> >> The idea behind this is, to be able to select classes by giving a >> parameter on the command-line. So at some point a translation from a >> given "key" to its actual class has to happen, I guess. > > I see. So are the task masters interchangeable? I would have imagined > they would modify the default operational semantics. If so, the task > manager should be fixed programmatically in the SConstruct file. > yes, I'd like to be able to replace Taskmasters as well. For example, switching from the "default" file-oriented taskmaster, to one that also supports pseudo targets (always run) like in "make" or "aqualid". After all, we're mainly a "build framework"...so supporting different approaches to a given "build process" makes sense to me. Apart from that, it's basically just an example of what I'm trying to achieve. To which classes in our architecture the idiom is actually applied, is left as a decision that is made sometime later. Dirk From tshortik at gmx.de Sun May 15 06:21:14 2016 From: tshortik at gmx.de (=?UTF-8?Q?Dirk_B=c3=a4chle?=) Date: Sun, 15 May 2016 12:21:14 +0200 Subject: Design: Idiom for classes and methods that are customizable by the user? In-Reply-To: References: <5734ECDF.7070006@gmx.de> Message-ID: <57384D9A.2010500@gmx.de> Hi Gregory, thanks a lot for your answer and comments. Am 13.05.2016 um 08:35 schrieb Gregory Ewing: > Dirk B?chle wrote: > > [...] > > I'd even suggest that *all* of the build logic should be in > the Nodes, and the Taskmaster class shouldn't exist at all. > The top level logic should just tell the final Nodes to bring > themselves up to date, and they recursively do likewise for > their dependent nodes. > This is pretty much what we have now, and the starting point for the redesign. Our current Node class already has too much knowledge about the build logic, and users frequently complain that it's too hard to derive from our base "Node" to build their own upon. Our aim is to make the Node class in particular "slimmer", instead of "fatter". ;) >> I'm currently following the "Factory" pattern (more or less) as I know it from C++ and similar languages. > > This statement sets off alarm bells for me. If you're using some > design pattern in Python just because you learned to do it that > way in C++/Java/whatever, you're probably making it more > complicated than it needs to be. > Yes, I know what you mean. I used the term "factory" for the idiom of addressing a function or class by a key (=string), and the "more or less" was supposed to clarify that I'm not planning to implement a dedicated "TaskmasterFactory" as class...as you'd find in a design book when looking up the "Factory" pattern. It may well be that some of my wordings and expressions are a bit misleading...sorry, I'm not a native speaker. Best regards, Dirk From tshortik at gmx.de Sun May 15 06:23:36 2016 From: tshortik at gmx.de (=?UTF-8?Q?Dirk_B=c3=a4chle?=) Date: Sun, 15 May 2016 12:23:36 +0200 Subject: Design: Idiom for classes and methods that are customizable by the user? In-Reply-To: <87lh3eifsb.fsf@handshake.de> References: <5734ECDF.7070006@gmx.de> <87lh3eifsb.fsf@handshake.de> Message-ID: <57384E28.70305@gmx.de> Hello Dieter, Am 13.05.2016 um 09:21 schrieb dieter: > Dirk B?chle writes: >> ... >> My questions >> ============ >> >> - Is this a good approach, that I could use for other parts of the architecture as well, e.g. the Node class mentioned above? > > You might look at the "adpater" pattern. > > It is heavily used in Zope - and there looks something like: > thanks a lot for bringing up Zope, I'll definitely have a look at it. Best regards, Dirk From steve+comp.lang.python at pearwood.info Sun May 15 06:48:09 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 15 May 2016 20:48:09 +1000 Subject: How to get a directory list sorted by date? References: Message-ID: <573853ec$0$1603$c3e8da3$5496439d@news.astraweb.com> On Sunday 15 May 2016 18:47, cl at isbd.net wrote: > What I want is a list in the order produced by:- > ls --sort=time I'm not sure if this is the best way to do it, but what I would do is sort the directory list by the file's time metadata, which you can access using os.stat. To sort the file names, I use a key function which takes the filename, joins it to the parent directory, grabs its stat data, and extracts the time field. This is done only once per file name. import os import functools def by_date(where, fname): return os.stat(os.path.join(where, fname)).st_mtime location = '/tmp/' files = sorted(os.listdir(location), key=functools.partial(by_date, location)) The mysterious call to functools.partial creates a new function which is exactly the same as by_date except the "where" argument is already filled in. If functools.partial scares you, you can replace it with: key=lambda fname: by_date(location, fname) and if lambda scares you, you can create a top-level function: def keyfunc(fname): return by_date(location, fname) Also, remember that most operating systems provide (at least) three different times. I'm not sure which one you want, but if I had to guess, I would probably guess mtime. If I remember correctly: atime is usually the last access time; mtime is usually the last modification time; ctime is the creation time, but only on OS-X; ctime on Linux and Windows is, um, something else... To use one of the other two, change "st_mtime" to "st_ctime" or "st_atime". -- Steve From davidgshi at yahoo.co.uk Sun May 15 07:05:00 2016 From: davidgshi at yahoo.co.uk (David Shi) Date: Sun, 15 May 2016 11:05:00 +0000 (UTC) Subject: Pandas GroupBy does not behave consistently In-Reply-To: References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> <2075459214.3318248.1463172701996.JavaMail.yahoo@mail.yahoo.com> <1584945027.3340053.1463176318146.JavaMail.yahoo@mail.yahoo.com> <363052112.3322670.1463195744354.JavaMail.yahoo@mail.yahoo.com> <477298994.3432356.1463220990479.JavaMail.yahoo@mail.yahoo.com> <885651822.3710117.1463240251579.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1889403023.4028249.1463310300618.JavaMail.yahoo@mail.yahoo.com> Hello, Michael, Pandas GroupBy does not behave consistently. Last time, when we had conversation, I used grouby. ?It works well. Now, I thought to re-write the program, so that I can end up with a clean script. But, the problem is that a lot of columns are missing after groupby application. Any idea? Regards. David On Saturday, 14 May 2016, 17:00, Michael Selik wrote: This StackOverflow question was the first search result when I Googled for "Python why is there a little u"http://stackoverflow.com/questions/11279331/what-does-the-u-symbol-mean-in-front-of-string-values On Sat, May 14, 2016, 11:40 AM David Shi wrote: Hello, Michael, Why there is a little u ? ?u'ID',? Why can be done to it?? How to handle such objects? Can it be turn into list easily? Regards. David On Saturday, 14 May 2016, 15:34, Michael Selik wrote: You might also be interested in "Python for Data Analysis" for a thorough discussion of Pandas.http://shop.oreilly.com/product/0636920023784.do On Sat, May 14, 2016 at 10:29 AM Michael Selik wrote: David, it sounds like you'll need a thorough introduction to the basics of Python.Check out the tutorial:?https://docs.python.org/3/tutorial/ On Sat, May 14, 2016 at 6:19 AM David Shi wrote: Hello, Michael, I discovered that the problem is "two columns of data are put together" and "are recognised as one column". This is very strange.? I would like to understand the subject well. And, how many ways are there to investigate into the nature of objects dynamically? Some object types only get shown as an object.? Are there anything to be typed in Python, to reveal objects. Regards. David On Saturday, 14 May 2016, 4:30, Michael Selik wrote: What were you hoping to get from ``df[0]``?When you say it "yields nothing" do you mean it raised an error? What was the error message? Have you tried a Google search for "pandas set index"?http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.set_index.html On Fri, May 13, 2016 at 11:18 PM David Shi wrote: Hello, Michael, I tried to discover the problem. df[0] ? yields nothingdf[1] ?yields nothingdf[2] yields nothing However, df[3] gives the following:sid -9223372036854775808 NaN 1 133738.70 4 295256.11 5 137733.09 6 409413.58 8 269600.97 9 12852.94 Can we split this back to normal? or turn it into a dictionary, so that I can put values back properly. I like to use sid as index, some way. Regards. David On Friday, 13 May 2016, 22:58, Michael Selik wrote: What have code you tried? What error message are you receiving? On Fri, May 13, 2016, 5:54 PM David Shi wrote: Hello, Michael, How to convert a float type column into an integer or label or string type? On Friday, 13 May 2016, 22:02, Michael Selik wrote: To clarify that you're specifying the index as a label, use df.iloc ? ? >>> df = pd.DataFrame({'X': range(4)}, index=list('abcd'))? ? >>> df? ? ? ?X? ? a ?0? ? b ?1? ? c ?2? ? d ?3? ? >>> df.loc['a']? ? X ? ?0? ? Name: a, dtype: int64? ? >>> df.iloc[0]? ? X ? ?0? ? Name: a, dtype: int64 On Fri, May 13, 2016 at 4:54 PM David Shi wrote: Dear Michael, To avoid complication, I only groupby using one column. It is OK now.? But, how to refer to new row index?? How do I use floating index? Float64Index([ 1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], dtype='float64', name=u'StateFIPS') Regards. David On Friday, 13 May 2016, 21:43, Michael Selik wrote: Here's an example. ? ? >>> import pandas as pd? ? >>> df = pd.DataFrame({'group': list('AB') * 2, 'data': range(4)}, index=list('wxyz'))? ? >>> df? ? ? ?data group? ? w ? ? 0 ? ? A? ? x ? ? 1 ? ? B? ? y ? ? 2 ? ? A? ? z ? ? 3 ? ? B? ? >>> df = df.reset_index()? ? >>> df? ? ? index ?data group? ? 0 ? ? w ? ? 0 ? ? A? ? 1 ? ? x ? ? 1 ? ? B? ? 2 ? ? y ? ? 2 ? ? A? ? 3 ? ? z ? ? 3 ? ? B? ? >>> df.groupby('group').max()? ? ? ? ? index ?data? ? group? ? A ? ? ? ? y ? ? 2? ? B ? ? ? ? z ? ? 3 If that doesn't help, you'll need to explain what you're trying to accomplish in detail -- what variables you started with, what transformations you want to do, and what variables you hope to have when finished. On Fri, May 13, 2016 at 4:36 PM David Shi wrote: Hello, Michael, I changed groupby with one column. The index is different. Index([ u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY'], dtype='object', name=0) How to use this index? Regards. David On Friday, 13 May 2016, 21:19, David Shi wrote: Hello, Michael, I typed in df.index I got the followingMultiIndex(levels=[[1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 53.0, 54.0, 55.0, 56.0], [u'AL', u'AR', u'AZ', u'CA', u'CO', u'CT', u'DC', u'DE', u'FL', u'GA', u'IA', u'ID', u'IL', u'IN', u'KS', u'KY', u'LA', u'MA', u'MD', u'ME', u'MI', u'MN', u'MO', u'MS', u'MT', u'NC', u'ND', u'NE', u'NH', u'NJ', u'NM', u'NV', u'NY', u'OH', u'OK', u'OR', u'PA', u'RI', u'SC', u'SD', u'State', u'TN', u'TX', u'UT', u'VA', u'VT', u'WA', u'WI', u'WV', u'WY']], labels=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], [0, 2, 1, 3, 4, 5, 7, 6, 8, 9, 11, 12, 13, 10, 14, 15, 16, 19, 18, 17, 20, 21, 23, 22, 24, 27, 31, 28, 29, 30, 32, 25, 26, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 45, 44, 46, 48, 47, 49]], names=[u'StateFIPS', 0])Regards. David On Friday, 13 May 2016, 21:11, David Shi wrote: Dear Michael, I have done a number of operation in between. Providing that information does not help you How to reset index after grouping and various operations is of interest. How to type in a command to find out its current dataframe? Regards. David On Friday, 13 May 2016, 20:58, Michael Selik wrote: Just in case I misunderstood, why don't you make a little example of before and after the grouping? This mailing list does not accept attachments, so you'll have to make do with pasting a few rows of comma-separated or tab-separated values. On Fri, May 13, 2016 at 3:56 PM Michael Selik wrote: In order to preserve your index after the aggregation, you need to make sure it is considered a data column (via reset_index) and then choose how your aggregation will operate on that column. On Fri, May 13, 2016 at 3:29 PM David Shi wrote: Hello, Michael, Why reset_index before grouping? Regards. David On Friday, 13 May 2016, 17:57, Michael Selik wrote: On Fri, May 13, 2016 at 12:27 PM David Shi via Python-list wrote: I lost my indexes after grouping in Pandas. I managed to rest_index and got back the index column. But How can I get back a index row? Was the grouping an aggregation? If so, the original indexes are meaningless. What you could do is reset_index before the grouping and when you aggregate decide how to handle the formerly-known-as-index column (min, max, mean, ?). From python.list at tim.thechases.com Sun May 15 07:15:11 2016 From: python.list at tim.thechases.com (Tim Chase) Date: Sun, 15 May 2016 06:15:11 -0500 Subject: How to get a directory list sorted by date? In-Reply-To: References: Message-ID: <20160515061511.7c62b0e1@bigbox.christie.dr> On 2016-05-15 11:46, Peter Otten wrote: > def sorted_dir(folder): > def getmtime(name): > path = os.path.join(folder, name) > return os.path.getmtime(path) > > return sorted(os.listdir(folder), key=getmtime, reverse=True) > > The same idea will work with pathlib and os.scandir(): > > def _getmtime(entry): > return entry.stat().st_mtime > > def sd_sorted_dir(folder): > return sorted(os.scandir(folder), key=_getmtime, reverse=True) unless sorted() returns a lazy sorter, you lose most of the advantages of scandir() being lazy since you have to read the entire directory list into memory to sort it. -tkc From python.list at tim.thechases.com Sun May 15 07:19:17 2016 From: python.list at tim.thechases.com (Tim Chase) Date: Sun, 15 May 2016 06:19:17 -0500 Subject: How to get a directory list sorted by date? In-Reply-To: <573853ec$0$1603$c3e8da3$5496439d@news.astraweb.com> References: <573853ec$0$1603$c3e8da3$5496439d@news.astraweb.com> Message-ID: <20160515061917.10d6f398@bigbox.christie.dr> On 2016-05-15 20:48, Steven D'Aprano wrote: > Also, remember that most operating systems provide (at least) three > different times. I'm not sure which one you want, but if I had to > guess, I would probably guess mtime. If I remember correctly: > > atime is usually the last access time; > mtime is usually the last modification time; > ctime is the creation time, but only on OS-X; > ctime on Linux and Windows is, um, something else... > > To use one of the other two, change "st_mtime" to "st_ctime" or > "st_atime". As an added wrinkle, some of us turn off "atime" in our /etc/ftab because it tends to slow things down with little benefit in return. So Steven was right to recommend "mtime" as it's usually what people want. -tkc From bluebox03 at gmail.com Sun May 15 07:19:22 2016 From: bluebox03 at gmail.com (tommy yama) Date: Sun, 15 May 2016 20:19:22 +0900 Subject: Developers using Python QT framework ?? Message-ID: Hi folks, Are there people using QT cross-platform framework on the list? I appreciate for your advices and references regarding this. I am a novice python programmer who started using this framework. Thanks a lot in advance! Tomo From rosuav at gmail.com Sun May 15 07:59:53 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 15 May 2016 21:59:53 +1000 Subject: How to get a directory list sorted by date? In-Reply-To: <20160515061511.7c62b0e1@bigbox.christie.dr> References: <20160515061511.7c62b0e1@bigbox.christie.dr> Message-ID: On Sun, May 15, 2016 at 9:15 PM, Tim Chase wrote: > On 2016-05-15 11:46, Peter Otten wrote: >> def sorted_dir(folder): >> def getmtime(name): >> path = os.path.join(folder, name) >> return os.path.getmtime(path) >> >> return sorted(os.listdir(folder), key=getmtime, reverse=True) >> >> The same idea will work with pathlib and os.scandir(): >> >> def _getmtime(entry): >> return entry.stat().st_mtime >> >> def sd_sorted_dir(folder): >> return sorted(os.scandir(folder), key=_getmtime, reverse=True) > > unless sorted() returns a lazy sorter, you lose most of the advantages > of scandir() being lazy since you have to read the entire directory > list into memory to sort it. I'm not sure a lazy sorter exists. You can't know which is the oldest/newest file in a directory without knowing all their mtimes. ChrisA From __peter__ at web.de Sun May 15 09:03:03 2016 From: __peter__ at web.de (Peter Otten) Date: Sun, 15 May 2016 15:03:03 +0200 Subject: How to get a directory list sorted by date? References: <20160515061511.7c62b0e1@bigbox.christie.dr> Message-ID: Tim Chase wrote: > On 2016-05-15 11:46, Peter Otten wrote: >> def sorted_dir(folder): >> def getmtime(name): >> path = os.path.join(folder, name) >> return os.path.getmtime(path) >> >> return sorted(os.listdir(folder), key=getmtime, reverse=True) >> >> The same idea will work with pathlib and os.scandir(): >> >> def _getmtime(entry): >> return entry.stat().st_mtime >> >> def sd_sorted_dir(folder): >> return sorted(os.scandir(folder), key=_getmtime, reverse=True) > > unless sorted() returns a lazy sorter, you lose most of the advantages > of scandir() being lazy since you have to read the entire directory > list into memory to sort it. I mentioned it because it avoids the duplicate stat call of the older approach which is rather inelegant. Taking another look at the OP's problem I think sorting could be entirely avoided. Instead have the memory card remember when its data was last archived. Pseudo-code: try: cut_off = load_cut_off(memory_card) pics = (pic for pic in memory_card if gettime(pic) > cut_off) except: pics = memory_card for pic in pics: copy_to_archive(pic) save_cut_off(now(), memory_card) As long as computer and camera time are in sync... From vincent.vande.vyvre at telenet.be Sun May 15 09:46:17 2016 From: vincent.vande.vyvre at telenet.be (Vincent Vande Vyvre) Date: Sun, 15 May 2016 15:46:17 +0200 Subject: How to get a directory list sorted by date? In-Reply-To: References: Message-ID: <57387DA9.6030500@telenet.be> Le 15/05/2016 10:47, cl at isbd.net a ?crit : > I have a little Python program I wrote myself which copies images from > a camera (well, any mounted directory) to my picture archive. The > picture archive is simply a directory hierarchy of dates with years at > the top, then months, then days. > > My Python program simply extracts the date from the image (put there > by the camera) and copies the image to the appropriate place in the > picture archive. > > There is one small bit of speeding up done by my program, it checks if > the file is already in the archive and doesn't copy if it's already > there. I don't generally clear pictures off the camera memory cards > so it's often the case that most of the pcitures are already in the > archive and all I actually want to do is copy the last couple of weeks > of new pictures. > > As camera memory card sizes get bigger (and images get more detailed) > this is beginning to take rather a long time. > > So, to the question, how can I get a list of the files on the memory > card (i.e. files in a directory) in date/time order, latest first. I > can then copy them until the first one I find which is already in the > archive. The file date may not *exactly* match the date/time in the > image file but it will produce the right order which is what I need. > > What I want is a list in the order produced by:- > ls --sort=time > > I suppose I could use a system call but that seems a little inelegant. > Hi, Seeing all the answers, I think there's a misunderstood here. The date used to sort the picture is not the file date provided by os.stat but the shooting date written into the metadata of each picture. Can the OP confirm that ? So, if the name of the picture are unchanged why not just compare the file name ? Vincent From vincent.vande.vyvre at telenet.be Sun May 15 10:15:28 2016 From: vincent.vande.vyvre at telenet.be (Vincent Vande Vyvre) Date: Sun, 15 May 2016 16:15:28 +0200 Subject: Developers using Python QT framework ?? In-Reply-To: References: Message-ID: <57388480.5060904@telenet.be> Le 15/05/2016 13:19, tommy yama a ?crit : > Hi folks, > > Are there people using QT cross-platform framework on the list? > I appreciate for your advices and references regarding this. > > I am a novice python programmer who started using this framework. > > > Thanks a lot in advance! > > Tomo Yes. Note, it is also a mailing list PyQt: https://www.riverbankcomputing.com/mailman/listinfo/pyqt Vincent From grant.b.edwards at gmail.com Sun May 15 10:36:18 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Sun, 15 May 2016 14:36:18 +0000 (UTC) Subject: How to get a directory list sorted by date? References: <20160515061511.7c62b0e1@bigbox.christie.dr> Message-ID: On 2016-05-15, Tim Chase wrote: > On 2016-05-15 11:46, Peter Otten wrote: >> def sorted_dir(folder): >> def getmtime(name): >> path = os.path.join(folder, name) >> return os.path.getmtime(path) >> >> return sorted(os.listdir(folder), key=getmtime, reverse=True) >> >> The same idea will work with pathlib and os.scandir(): >> >> def _getmtime(entry): >> return entry.stat().st_mtime >> >> def sd_sorted_dir(folder): >> return sorted(os.scandir(folder), key=_getmtime, reverse=True) > > unless sorted() returns a lazy sorter, What's a lazy sorter? -- Grant From bluebox03 at gmail.com Sun May 15 10:38:03 2016 From: bluebox03 at gmail.com (tommy yama) Date: Sun, 15 May 2016 23:38:03 +0900 Subject: Developers using Python QT framework ?? In-Reply-To: <57388480.5060904@telenet.be> References: <57388480.5060904@telenet.be> Message-ID: Hey there, Many thanks, Vincent, On Sun, May 15, 2016 at 11:15 PM, Vincent Vande Vyvre < vincent.vande.vyvre at telenet.be> wrote: > Le 15/05/2016 13:19, tommy yama a ?crit : > >> Hi folks, >> >> Are there people using QT cross-platform framework on the list? >> I appreciate for your advices and references regarding this. >> >> I am a novice python programmer who started using this framework. >> >> >> Thanks a lot in advance! >> >> Tomo >> > Yes. > > Note, it is also a mailing list PyQt: > https://www.riverbankcomputing.com/mailman/listinfo/pyqt > > Vincent > -- > https://mail.python.org/mailman/listinfo/python-list > From joel.goldstick at gmail.com Sun May 15 10:45:00 2016 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Sun, 15 May 2016 10:45:00 -0400 Subject: How to get a directory list sorted by date? In-Reply-To: References: <20160515061511.7c62b0e1@bigbox.christie.dr> Message-ID: On Sun, May 15, 2016 at 10:36 AM, Grant Edwards wrote: > On 2016-05-15, Tim Chase wrote: >> On 2016-05-15 11:46, Peter Otten wrote: >>> def sorted_dir(folder): >>> def getmtime(name): >>> path = os.path.join(folder, name) >>> return os.path.getmtime(path) >>> >>> return sorted(os.listdir(folder), key=getmtime, reverse=True) >>> >>> The same idea will work with pathlib and os.scandir(): >>> >>> def _getmtime(entry): >>> return entry.stat().st_mtime >>> >>> def sd_sorted_dir(folder): >>> return sorted(os.scandir(folder), key=_getmtime, reverse=True) >> >> unless sorted() returns a lazy sorter, > > What's a lazy sorter? Some postal workers? just kidding > > -- > Grant > > -- > https://mail.python.org/mailman/listinfo/python-list -- Joel Goldstick http://joelgoldstick.com/blog http://cc-baseballstats.info/stats/birthdays From pavlos.parissis at gmail.com Sun May 15 11:49:49 2016 From: pavlos.parissis at gmail.com (Pavlos Parissis) Date: Sun, 15 May 2016 17:49:49 +0200 Subject: intermittent ValueErrors from subprocess Message-ID: <57389A9D.1010205@gmail.com> Hi, I get intermittent ValueErrors[1] from subprocess when I check if an IP is assigned to loopback interface by running: /sbin/ip address show dev lo to 10.52.12.2/32 I use subprocess.check_output like this: cmd = [ '/sbin/ip', 'address', 'show', 'dev', "{}".format(self.config['interface']), 'to', "{}".format(self.config['ip_prefix']), ] try: out = subprocess.check_output( cmd, universal_newlines=True, timeout=1) except subprocess.CalledProcessError as error: return True except subprocess.TimeoutExpired: return True else: if self.config['ip_prefix'] in out: return True else: return False and I get the following exception: ValueError: Invalid file object: <_io.TextIOWrapper name=11 encoding='UTF-8'> As a consequence of the raised exception thread dies and I now catch ValueError in the same try block in order to avoid the crash. It has happened ~5 times and this code is executed from multiple threads and every ~10secs on several(~40) systems for more than 18months. So, it is quite impossible for me to reproduce it. It could be that the system returns corrupted data or python fails for some unknown reason to parse the output. The program uses Python 3.4.4 and runs on Debian stable and CentOS 6/7. The full code path in question can be found here: https://github.com/unixsurfer/anycast_healthchecker/blob/master/anycast_healthchecker/servicecheck.py#L90 I did a search on bugs.python.org about it but I didn't find anything. Has anyone seen this behavior before? Cheers, Pavlos [1] https://gist.github.com/unixsurfer/67db620d87f667423f6f6e3a04e0bff5 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From michael.selik at gmail.com Sun May 15 11:50:38 2016 From: michael.selik at gmail.com (Michael Selik) Date: Sun, 15 May 2016 15:50:38 +0000 Subject: How to get a directory list sorted by date? In-Reply-To: References: <20160515061511.7c62b0e1@bigbox.christie.dr> Message-ID: On Sun, May 15, 2016, 10:37 AM Grant Edwards wrote: > On 2016-05-15, Tim Chase wrote: > > On 2016-05-15 11:46, Peter Otten wrote: > >> def sorted_dir(folder): > >> def getmtime(name): > >> path = os.path.join(folder, name) > >> return os.path.getmtime(path) > >> > >> return sorted(os.listdir(folder), key=getmtime, reverse=True) > >> > >> The same idea will work with pathlib and os.scandir(): > >> > >> def _getmtime(entry): > >> return entry.stat().st_mtime > >> > >> def sd_sorted_dir(folder): > >> return sorted(os.scandir(folder), key=_getmtime, reverse=True) > > > > unless sorted() returns a lazy sorter, > > What's a lazy sorter? > One that doesn't calculate the next item in the sequence until you ask for it. It's impossible unless you don't mind an approximation rather than correct sort. > From grant.b.edwards at gmail.com Sun May 15 12:00:03 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Sun, 15 May 2016 16:00:03 +0000 (UTC) Subject: How to get a directory list sorted by date? References: <20160515061511.7c62b0e1@bigbox.christie.dr> Message-ID: On 2016-05-15, Michael Selik wrote: > On Sun, May 15, 2016, 10:37 AM Grant Edwards wrote: >> On 2016-05-15, Tim Chase wrote: >>> >>> unless sorted() returns a lazy sorter, >> >> What's a lazy sorter? > > One that doesn't calculate the next item in the sequence until you > ask for it. It's impossible Why? As long as the function has access to the entire sequence, it should be trivial. Just find the min (or max) item in the sequence, remove it, then return it. It's horribly inefficient, but... > unless you don't mind an approximation rather than correct sort. I have a feeling that I've missed the joke somewhere. -- Grant From python.list at tim.thechases.com Sun May 15 14:12:32 2016 From: python.list at tim.thechases.com (Tim Chase) Date: Sun, 15 May 2016 13:12:32 -0500 Subject: How to get a directory list sorted by date? In-Reply-To: References: <20160515061511.7c62b0e1@bigbox.christie.dr> Message-ID: <20160515131232.200268cf@bigbox.christie.dr> On 2016-05-15 14:36, Grant Edwards wrote: > On 2016-05-15, Tim Chase wrote: > > unless sorted() returns a lazy sorter, > > What's a lazy sorter? A hypothetical algorithm that can spool out a sorted sequence without holding the entire sequence in memory at the same time. Though I typed that initial reply a little quickly, as I hadn't known at the time that scandir()'s results also provide stat() results with reduced overhead (whereas listdir() just returns the filenames, so you have to stat() each one). Thanks, Peter, for highlighting this feature. So in light of my newfound knowledge, I humbly retract my snark and agree that it's better to exploit scandir()'s inclusion of stat() details without additional calls. -tkc From michael.selik at gmail.com Sun May 15 15:03:28 2016 From: michael.selik at gmail.com (Michael Selik) Date: Sun, 15 May 2016 19:03:28 +0000 Subject: Pandas GroupBy does not behave consistently In-Reply-To: <1889403023.4028249.1463310300618.JavaMail.yahoo@mail.yahoo.com> References: <1878444350.3064027.1463156091513.JavaMail.yahoo.ref@mail.yahoo.com> <1878444350.3064027.1463156091513.JavaMail.yahoo@mail.yahoo.com> <1974768074.3081564.1463156374539.JavaMail.yahoo@mail.yahoo.com> <877815788.3201804.1463167634101.JavaMail.yahoo@mail.yahoo.com> <434110787.3205994.1463170295636.JavaMail.yahoo@mail.yahoo.com> <2098601859.3273341.1463170752937.JavaMail.yahoo@mail.yahoo.com> <1215747866.3212467.1463171634713.JavaMail.yahoo@mail.yahoo.com> <2075459214.3318248.1463172701996.JavaMail.yahoo@mail.yahoo.com> <1584945027.3340053.1463176318146.JavaMail.yahoo@mail.yahoo.com> <363052112.3322670.1463195744354.JavaMail.yahoo@mail.yahoo.com> <477298994.3432356.1463220990479.JavaMail.yahoo@mail.yahoo.com> <885651822.3710117.1463240251579.JavaMail.yahoo@mail.yahoo.com> <1889403023.4028249.1463310300618.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Sun, May 15, 2016 at 7:07 AM David Shi wrote: > Hello, Michael, > > Pandas GroupBy does not behave consistently. > > Last time, when we had conversation, I used grouby. It works well. > > Now, I thought to re-write the program, so that I can end up with a clean > script. > > But, the problem is that a lot of columns are missing after groupby > application. > > Any idea? > I'd guess that the columns lost after a groupby had an inappropriate datatype for the operation you were doing. If you'd like a more thorough response, you'll need to post the code you tried and the results you got. From cl at isbd.net Sun May 15 16:23:57 2016 From: cl at isbd.net (cl at isbd.net) Date: Sun, 15 May 2016 21:23:57 +0100 Subject: How to get a directory list sorted by date? References: <20160515061511.7c62b0e1@bigbox.christie.dr> <20160515131232.200268cf@bigbox.christie.dr> Message-ID: Tim Chase wrote: > On 2016-05-15 14:36, Grant Edwards wrote: > > On 2016-05-15, Tim Chase wrote: > > > unless sorted() returns a lazy sorter, > > > > What's a lazy sorter? > > A hypothetical algorithm that can spool out a sorted sequence without > holding the entire sequence in memory at the same time. > > Though I typed that initial reply a little quickly, as I hadn't known > at the time that scandir()'s results also provide stat() results with > reduced overhead (whereas listdir() just returns the filenames, so > you have to stat() each one). Thanks, Peter, for highlighting this > feature. > > So in light of my newfound knowledge, I humbly retract my snark and > agree that it's better to exploit scandir()'s inclusion of stat() > details without additional calls. > Thanks everyone for the ideas etc. -- Chris Green ? From rosuav at gmail.com Sun May 15 17:52:19 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 16 May 2016 07:52:19 +1000 Subject: How to get a directory list sorted by date? In-Reply-To: References: <20160515061511.7c62b0e1@bigbox.christie.dr> Message-ID: On Mon, May 16, 2016 at 2:00 AM, Grant Edwards wrote: > On 2016-05-15, Michael Selik wrote: >> On Sun, May 15, 2016, 10:37 AM Grant Edwards wrote: >>> On 2016-05-15, Tim Chase wrote: >>>> >>>> unless sorted() returns a lazy sorter, >>> >>> What's a lazy sorter? >> >> One that doesn't calculate the next item in the sequence until you >> ask for it. It's impossible > > Why? As long as the function has access to the entire sequence, it > should be trivial. Just find the min (or max) item in the sequence, > remove it, then return it. It's horribly inefficient, but... > >> unless you don't mind an approximation rather than correct sort. > > I have a feeling that I've missed the joke somewhere. Sure, it's not impossible to implement, but generally the point of lazy generation is that it's more efficient. Going from O(N log N) with O(N) memory to O(N*N) with O(1) memory is not usually what you want! ChrisA From rosuav at gmail.com Sun May 15 18:08:09 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 16 May 2016 08:08:09 +1000 Subject: intermittent ValueErrors from subprocess In-Reply-To: <57389A9D.1010205@gmail.com> References: <57389A9D.1010205@gmail.com> Message-ID: On Mon, May 16, 2016 at 1:49 AM, Pavlos Parissis wrote: > I use subprocess.check_output like this: > > cmd = [ > '/sbin/ip', > 'address', > 'show', > 'dev', > "{}".format(self.config['interface']), > 'to', > "{}".format(self.config['ip_prefix']), > ] Unrelated to your problem: "{}".format(x) is the same as str(x), and if x is already a string, it does exactly nothing. Most likely you can just put your configs straight in there. > try: > out = subprocess.check_output( > cmd, > universal_newlines=True, > timeout=1) > except subprocess.CalledProcessError as error: > return True > except subprocess.TimeoutExpired: > return True > else: > if self.config['ip_prefix'] in out: > return True > else: > return False > > > and I get the following exception: > > ValueError: Invalid file object: <_io.TextIOWrapper name=11 > encoding='UTF-8'> Searching the CPython sources for that exception shows one hit: selectors.py, where it converts a file object to an integer file descriptor. (You could have helped out by showing us the full traceback.) Is it possible you were running out of file descriptors, or in some other way unable to create the pipe? ChrisA From pavlos.parissis at gmail.com Sun May 15 18:32:02 2016 From: pavlos.parissis at gmail.com (Pavlos Parissis) Date: Mon, 16 May 2016 00:32:02 +0200 Subject: intermittent ValueErrors from subprocess In-Reply-To: References: <57389A9D.1010205@gmail.com> Message-ID: <5738F8E2.1030403@gmail.com> On 16/05/2016 12:08 ??, Chris Angelico wrote: > On Mon, May 16, 2016 at 1:49 AM, Pavlos Parissis > wrote: >> I use subprocess.check_output like this: >> >> cmd = [ >> '/sbin/ip', >> 'address', >> 'show', >> 'dev', >> "{}".format(self.config['interface']), >> 'to', >> "{}".format(self.config['ip_prefix']), >> ] > > Unrelated to your problem: "{}".format(x) is the same as str(x), and > if x is already a string, it does exactly nothing. Most likely you can > just put your configs straight in there. > Thanks for the tip, I will change it as you suggested. >> try: >> out = subprocess.check_output( >> cmd, >> universal_newlines=True, >> timeout=1) >> except subprocess.CalledProcessError as error: >> return True >> except subprocess.TimeoutExpired: >> return True >> else: >> if self.config['ip_prefix'] in out: >> return True >> else: >> return False >> >> >> and I get the following exception: >> >> ValueError: Invalid file object: <_io.TextIOWrapper name=11 >> encoding='UTF-8'> > > Searching the CPython sources for that exception shows one hit: > selectors.py, where it converts a file object to an integer file > descriptor. (You could have helped out by showing us the full > traceback.) I did, https://gist.github.com/unixsurfer/67db620d87f667423f6f6e3a04e0bff5 > Is it possible you were running out of file descriptors, > or in some other way unable to create the pipe? I don't think as I see right now only 8 FDs: sudo ls -1 /proc/22706/fd|wc 8 8 16 Thanks, Pavlos -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From rosuav at gmail.com Sun May 15 18:59:19 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 16 May 2016 08:59:19 +1000 Subject: intermittent ValueErrors from subprocess In-Reply-To: <5738F8E2.1030403@gmail.com> References: <57389A9D.1010205@gmail.com> <5738F8E2.1030403@gmail.com> Message-ID: On Mon, May 16, 2016 at 8:32 AM, Pavlos Parissis wrote: >> Searching the CPython sources for that exception shows one hit: >> selectors.py, where it converts a file object to an integer file >> descriptor. (You could have helped out by showing us the full >> traceback.) > > I did, https://gist.github.com/unixsurfer/67db620d87f667423f6f6e3a04e0bff5 Ah. I didn't click that link in your original post - didn't know it was the traceback. Better would have been to at least say so; best would have been to include it inline. >> Is it possible you were running out of file descriptors, >> or in some other way unable to create the pipe? > > I don't think as I see right now only 8 FDs: > > sudo ls -1 /proc/22706/fd|wc > 8 8 16 > If you can recreate the problem consistently, it would be worth messing around with slightly lower level APIs - using subprocess.Popen rather than check_output, for instance - and see what you can do without the pipes. Somewhere, something's failing, and it's not easy to see what. ChrisA From pavlos.parissis at gmail.com Sun May 15 19:17:41 2016 From: pavlos.parissis at gmail.com (Pavlos Parissis) Date: Mon, 16 May 2016 01:17:41 +0200 Subject: intermittent ValueErrors from subprocess In-Reply-To: References: <57389A9D.1010205@gmail.com> <5738F8E2.1030403@gmail.com> Message-ID: <57390395.8070005@gmail.com> On 16/05/2016 12:59 ??, Chris Angelico wrote: > On Mon, May 16, 2016 at 8:32 AM, Pavlos Parissis > wrote: >>> Searching the CPython sources for that exception shows one hit: >>> selectors.py, where it converts a file object to an integer file >>> descriptor. (You could have helped out by showing us the full >>> traceback.) >> >> I did, https://gist.github.com/unixsurfer/67db620d87f667423f6f6e3a04e0bff5 > > Ah. I didn't click that link in your original post - didn't know it > was the traceback. Better would have been to at least say so; best > would have been to include it inline. > I don't usually include traces as they have long lines and several e-mail clients mess with them in a way that makes the trace unreadable. >>> Is it possible you were running out of file descriptors, >>> or in some other way unable to create the pipe? >> >> I don't think as I see right now only 8 FDs: >> >> sudo ls -1 /proc/22706/fd|wc >> 8 8 16 >> > > If you can recreate the problem consistently, I can't. This is my main problem. This code has been executed ~100K and that exception has occurred only ~5 times. > it would be worth > messing around with slightly lower level APIs - using subprocess.Popen > rather than check_output, for instance - and see what you can do > without the pipes. What do you mean by that? > Somewhere, something's failing, and it's not easy > to see what. > > ChrisA > Thanks once again, Pavlos -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From rosuav at gmail.com Sun May 15 19:32:37 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 16 May 2016 09:32:37 +1000 Subject: intermittent ValueErrors from subprocess In-Reply-To: <57390395.8070005@gmail.com> References: <57389A9D.1010205@gmail.com> <5738F8E2.1030403@gmail.com> <57390395.8070005@gmail.com> Message-ID: On Mon, May 16, 2016 at 9:17 AM, Pavlos Parissis wrote: > On 16/05/2016 12:59 ??, Chris Angelico wrote: >> Ah. I didn't click that link in your original post - didn't know it >> was the traceback. Better would have been to at least say so; best >> would have been to include it inline. >> > > I don't usually include traces as they have long lines and several > e-mail clients mess with them in a way that makes the trace unreadable. If your client can't send them, fix your client. If someone else's client can't read them, that's not your problem. >> If you can recreate the problem consistently, > > I can't. This is my main problem. This code has been executed ~100K and > that exception has occurred only ~5 times. Yeah, I figured. Makes it tough. >> it would be worth >> messing around with slightly lower level APIs - using subprocess.Popen >> rather than check_output, for instance - and see what you can do >> without the pipes. > > What do you mean by that? You're currently using a very high level API that says "run this program, make sure it exits zero, and give me its output". Somewhere inside there, something's going wrong. So the first thing I'd do would be to tease apart the job into smaller and simpler parts; that means, in this case, going to something like subprocess.Popen, with explicit use of pipes and such. You could look at the source code for check_output for a start. But being unable to recreate the problem on demand makes it hard. ChrisA From flebber.crue at gmail.com Sun May 15 22:21:23 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Sun, 15 May 2016 19:21:23 -0700 (PDT) Subject: sqlite 3 attribute error __exit__ Message-ID: Hi I have a file and want to create the sqlite3 db. Using with however i receive an attribute error and it causes an exit. The relevant section of the file is: import sqlite3 conn = sqlite3.connect("trial.db") with conn, conn.cursor() as cur: # First, create tables. cur.execute("drop table if exists meetings, races, horses") cur.execute("create table meetings (" + ", ".join("%s varchar" % fld for fld in meetattrs) + ")") cur.execute("create table races (" + ", ".join("%s varchar" % fld for fld in raceattrs) + ")") cur.execute("create table horses (" + ", ".join("%s varchar" % fld for fld in horseattrs) + ")") However it runs producing this error for line 30 which is with conn, conn.cursor() as cur: (pyXML) [sayth at manjaro pyXML]$ python racemeeting.py data/*xml Traceback (most recent call last): File "racemeeting.py", line 31, in with conn, conn.cursor() as cur: AttributeError: __exit__ Why would i get this error? Sayth From ben+python at benfinney.id.au Sun May 15 22:52:01 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 16 May 2016 12:52:01 +1000 Subject: sqlite 3 attribute error __exit__ References: Message-ID: <85mvnqd8a6.fsf@benfinney.id.au> Sayth Renshaw writes: > with conn, conn.cursor() as cur: What are you expecting this ?with? statement to do? As you've written it, the statement declares your intent to enter both ?conn? and ?conn.cursor()? as context managers. To ?enter a context manager? entails calling the ?__enter__? method on the context manager object. So that will happen to both ?conn? and ?conn.cursor()?. Are both of those objects context managers? If not, you will get the error that you reported. -- \ ?Pinky, are you pondering what I'm pondering?? ?I think so, | `\ Brain, but if we get Sam Spade, we'll never have any puppies.? | _o__) ?_Pinky and The Brain_ | Ben Finney From flebber.crue at gmail.com Sun May 15 22:56:33 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Sun, 15 May 2016 19:56:33 -0700 (PDT) Subject: sqlite 3 attribute error __exit__ In-Reply-To: References: Message-ID: <2840096d-68ed-4cf6-a82a-b8f17d4cedd2@googlegroups.com> On Monday, 16 May 2016 12:45:26 UTC+10, DFS wrote: > On 5/15/2016 10:21 PM, Sayth Renshaw wrote: > > Hi > > > > I have a file and want to create the sqlite3 db. Using with however i receive an attribute error and it causes an exit. > > > > The relevant section of the file is: > > > > import sqlite3 > > > > conn = sqlite3.connect("trial.db") > > with conn, conn.cursor() as cur: > > # First, create tables. > > cur.execute("drop table if exists meetings, races, horses") > > cur.execute("create table meetings (" + > > ", ".join("%s varchar" % fld for fld in meetattrs) > > + ")") > > cur.execute("create table races (" + > > ", ".join("%s varchar" % fld for fld in raceattrs) > > + ")") > > cur.execute("create table horses (" + > > ", ".join("%s varchar" % fld for fld in horseattrs) > > + ")") > > > > > > However it runs producing this error for line 30 which is > > with conn, conn.cursor() as cur: > > > > (pyXML) [sayth at manjaro pyXML]$ python racemeeting.py data/*xml > > Traceback (most recent call last): > > File "racemeeting.py", line 31, in > > with conn, conn.cursor() as cur: > > AttributeError: __exit__ > > > > Why would i get this error? > > > > Sayth > > > Answer #66 looks like the ticket: > > http://stackoverflow.com/questions/7447284/how-to-troubleshoot-an-attributeerror-exit-in-multiproccesing-in-python > > > Have you tried: > > cur = conn.cursor() cur = conn.cursor() does work thought I should have been able to use with to handle it for me. Thanks Sayth From flebber.crue at gmail.com Sun May 15 23:21:24 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Sun, 15 May 2016 20:21:24 -0700 (PDT) Subject: Trouble porting glob bash behavior with argparse to windows shell In-Reply-To: <125a6208-8752-482e-ac8f-8ef2be012dae@googlegroups.com> References: <80f6672a-2b69-4749-821d-a92be107862a@googlegroups.com> <664703cb-ac1f-411d-a0d7-0a50a683d2fb@googlegroups.com> <125a6208-8752-482e-ac8f-8ef2be012dae@googlegroups.com> Message-ID: <8d064bd8-5c4e-4693-97d0-2bff5a102ee3@googlegroups.com> On Wednesday, 4 May 2016 17:57:32 UTC+10, Sayth Renshaw wrote: > Oops sorry noticed you did in the glob. Sorry squinting at phone. > > Sayth Hi this seems to be causing me an error in my thinking as well as the program. I am creating a function GetArgs to take a path and file extension from the command line. However I cannot call it effectively. I will clrify this is my function import argparse import glob import os import sqlite3 def GetArgs(parser): '''parse XML from command line''' parser.add_argument("path", nargs="+") parser.add_argument('-e', '--extension', default='', help='File extension to filter by.') args = parser.parse_args() files = set() name_pattern = "*" + args.extension for path in args.path: files.update(glob.glob(os.path.join(path, name_pattern))) return files Then later in program I am attempting to call it an a for statement. filesToProcess = GetArgs() for meeting in filesToProcess: meetdata = [meeting.get(attr) for attr in meetattrs] cur.execute("insert into meetings values (" + ",".join(["%s"] * len(meetattrs)) + ")", meetdata) this fails as i would expect, however if I declare a list as the GetArgs() argument it fails as well. Where my confusion is that I created the function to take arguments from the command line, so I don't have that variable to supply until executed. Have i overbaked the cake? Sayth From flebber.crue at gmail.com Sun May 15 23:27:43 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Sun, 15 May 2016 20:27:43 -0700 (PDT) Subject: How to use pip to install dtrx? In-Reply-To: References: Message-ID: On Saturday, 14 May 2016 07:39:23 UTC+10, Ehsan Hajiramezanali wrote: > Hi, > > I want to use pip to install dtrx. However, I got the following error. > > ~~~ > $ pip install --allow-external dtrx dtrx > DEPRECATION: --allow-external has been deprecated and will be removed > in the future. Due to changes in the repository protocol, it no longer > has any effect. > Collecting dtrx > Could not find a version that satisfies the requirement dtrx (from versions: ) > No matching distribution found for dtrx > ~~~ > > Is there any way to solve this problem? > > Thanks in advance. > > Best regards, > Ehsan You can sownload it and run this on it. python setup.py install --prefix=/usr/local Of course it appears to not be maintained in anyway for 9 years. I found pyunpack https://github.com/ponty/pyunpack that page has a link to similar projects to which you may find useful if pyunpack isn't. https://github.com/ponty/pyunpack#similar-projects Sayth From flebber.crue at gmail.com Sun May 15 23:40:59 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Sun, 15 May 2016 20:40:59 -0700 (PDT) Subject: sqlite 3 attribute error __exit__ In-Reply-To: References: <85mvnqd8a6.fsf@benfinney.id.au> Message-ID: <0d834bc2-22fd-4d53-92b1-6fafa6b25a4f@googlegroups.com> > > As you've written it, the statement declares your intent to enter both > ?conn? and ?conn.cursor()? as context managers. > > > > To ?enter a context manager? entails calling the ?__enter__? method on > the context manager object. So that will happen to both ?conn? and > ?conn.cursor()?. > > Are both of those objects context managers? If not, you will get the > error that you reported. > I actually am unsure about context managers. I was talking to ChrisA the other day and using psycog2 this code worked. conn = psycopg2.connect("") with conn, conn.cursor() as cur: # First, create tables. cur.execute("drop table if exists meetings, races, horses") cur.execute("create table meetings (" + ", ".join("%s varchar" % fld for fld in meetattrs) + ")") cur.execute("create table races (" + ", ".join("%s varchar" % fld for fld in raceattrs) + ")") cur.execute("create table horses (" + ", ".join("%s varchar" % fld for fld in horseattrs) + ")") I tried to change it to sqlite3 and it fails. I thought it may have been sqlite specific. Found this http://initd.org/psycopg/articles/2013/04/07/psycopg-25-released/ I found using a context managers search and psycopg2 provides this as a feature. from the site. Connections and cursors as context managers A recent DBAPI extension has standardized the use of connections and cursors as context managers: it is now possible to use an idiom such as: with psycopg2.connect(DSN) as conn: with conn.cursor() as curs: curs.execute(SQL) with the intuitive behaviour: when the cursor block exits the cursor is closed; when the connection block exits normally the current transaction is committed, if it exits with an exception instead the transaction is rolled back, in either case the connection is ready to be used again (FIXED: the connection is NOT closed as originally stated). thanks Sayth From __peter__ at web.de Mon May 16 02:58:50 2016 From: __peter__ at web.de (Peter Otten) Date: Mon, 16 May 2016 08:58:50 +0200 Subject: Trouble porting glob bash behavior with argparse to windows shell References: <80f6672a-2b69-4749-821d-a92be107862a@googlegroups.com> <664703cb-ac1f-411d-a0d7-0a50a683d2fb@googlegroups.com> <125a6208-8752-482e-ac8f-8ef2be012dae@googlegroups.com> <8d064bd8-5c4e-4693-97d0-2bff5a102ee3@googlegroups.com> Message-ID: Sayth Renshaw wrote: > On Wednesday, 4 May 2016 17:57:32 UTC+10, Sayth Renshaw wrote: >> Oops sorry noticed you did in the glob. Sorry squinting at phone. >> >> Sayth > > Hi > > this seems to be causing me an error in my thinking as well as the > program. I am creating a function GetArgs to take a path and file > extension from the command line. > > However I cannot call it effectively. I will clrify this is my function > > import argparse > import glob > import os > import sqlite3 > > > def GetArgs(parser): > '''parse XML from command line''' > parser.add_argument("path", nargs="+") > parser.add_argument('-e', '--extension', default='', > help='File extension to filter by.') > args = parser.parse_args() > > files = set() > name_pattern = "*" + args.extension > for path in args.path: > files.update(glob.glob(os.path.join(path, name_pattern))) > return files > > Then later in program I am attempting to call it an a for statement. > > filesToProcess = GetArgs() > for meeting in filesToProcess: > meetdata = [meeting.get(attr) for attr in meetattrs] > cur.execute("insert into meetings values (" + > ",".join(["%s"] * len(meetattrs)) + ")", meetdata) > > > this fails as i would expect, however if I declare a list as the GetArgs() > argument it fails as well. > > Where my confusion is that I created the function to take arguments from > the command line, so I don't have that variable to supply until executed. > > Have i overbaked the cake? The actual arguments are in sys.argv and will be implicitly accessed by the parser.parse_args() method invocation. The problem is simply that you don't create an argparse.ArgumentParser() instance. I suggest that you do that inside the GetArgs() function: def GetArgs(): # no arguments parser = argparse.ArgumentParser() # your current code below parser.add_argument("path", nargs="+") ... From fal at fastwebnet.it Mon May 16 04:14:26 2016 From: fal at fastwebnet.it (Francesco Loffredo) Date: Mon, 16 May 2016 10:14:26 +0200 Subject: What should a decorator do if an attribute already exists? In-Reply-To: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> References: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1fb91cda-5583-1ab1-a14c-7bf3110daabc@fastwebnet.it> On 10/05/2016 17:45, Steven D'Aprano wrote: > I have a decorator that adds an attribute to the decorated function: > > > def decorate(func): > instrument = make_instrument() > > @functools.wraps(func) > def inner(*args): > instrument.start() > result = func(*args) > instrument.finish() > return result > > inner.instrument = instrument > return inner > > > The actual nature of the instrumentation isn't important: depending on the > decorator, it might count the number of function calls made, how long it > takes, count cache hits, or something else. > > My question is, what should I do if the decorated function already has an > instrument attribute? > > 1. raise an exception? > > 2. raise a warning, and over-write the attribute? > 3. raise a warning, and skip adding the attribute? > 4. raise a warning, and rename the existing instrument to > something else before writing my own instrument? > > 5. silently over-write the attribute? > > > I think 5 is clearly wrong, 4 is too difficult, and 3 seems pointless. So I > think either 1 or 2 is the right thing to do. > > Thoughts? > CAVEAT: I speak out of utter ignorance, please don't slap me if i'm saying something blatantly stupid... From your example, it seems that you use your instrument only inside your decorator. So I think "instrument" could be a "private variable". What if you called your instrument "__instrument", taking advantage of name mangling? This would, IMHO, solve entirely the name clash problem, and you could even access your instrument from outside, using its "mangled" name. This, of course, leads to another question: what happens to name mangling in a decorator? What will be actually called the variable "__instrument"? And what happens if you want to add another instrument, decorating the target twice? Francesco From airween at gmail.com Mon May 16 05:05:39 2016 From: airween at gmail.com (Ervin =?utf-8?Q?Heged=C3=BCs?=) Date: Mon, 16 May 2016 11:05:39 +0200 Subject: Autotool - compile module for both Python 2 _and_ 3 Message-ID: <20160516090539.GA8327@arxnet.hu> Hi All, there is a library, which written in C. I'ld like to use it from Python - from Python 2 _and_ 3. I can make the autotools* files for Python 2 and Python 3, but only exclusively. I can't make it for both in same time. There is a macro for autotool, called ax_python_devel.m4. This macro uses same names, so when I use it, and pass the arguments (eg. --with-python --with-python3), then only the latest name will be used (eg. PYTHON_CPPFLAGS = -I/usr/include/python3.4m), and compile system uses only that for both versions. Is there any "best practice" to solve this problem? Practical advice or links are welcome! Thanks, Ervin -- I ? UTF-8 From redirect.null at gmail.com Mon May 16 07:22:01 2016 From: redirect.null at gmail.com (redirect.null at gmail.com) Date: Mon, 16 May 2016 04:22:01 -0700 (PDT) Subject: How to create development Python environment on Linux. Message-ID: <815aa265-45c4-40a4-860d-beb89cd9a78e@googlegroups.com> I have a Linux system (Mint 17.3 based in Ubuntu 14.04) on which I wish to do some Python development. The system has Python 2.7.6 installed already (there is a Python 3 installation too but I won't be needing to use that to start with). I need to install various Python modules for the work I'm going to do and I'm trying to work out how best to do this in a way that does not risk corrupting the system's Python installation. I've learned that there are (at least) two mechanisms for installing Python packages in a way that does not impact the system installation, virtualenv and user scheme installations. I have several questions/issues. 1. I am unsure which of these technologies is most suitable for my needs. Virtualenv sounds like what I need, but two things give me pause for thought. First I came across this github issue https://github.com/pypa/virtualenv/pull/697 ...which highlights what seem like severe issues in the vitualenv implementation and proposes to rewrite the whole library. This raises two questions. i) Is this rewrite complete? ii) Is 'legacy' virtualenv or virtualenv-rewrite now the recommended library? The second thing that gives me pause for thought is the remark in this article http://python-packaging-user-guide.readthedocs.io/en/latest/install_requirements_linux/#installing-pip-setuptools-wheel-with-linux-package-managers ...where it says 'Recent Debian/Ubuntu versions have modified pip to use the ?User Scheme? by default, which is a significant behavior change that can be surprising to some users.' These concerns caused me to try and research more in the area of user installs and package installation, which leads me to my next two issues. 2. User install site.py behaviour confusing. The first thing I wanted to work out was whether 'user install' was indeed enabled by default on my system as described in the article above. Here is the documentation for site.py https://docs.python.org/2/library/site.html This is the documenation for 2.7.11 while I only have 2.7.6. I don't see a link to documentation for 2.7.6 and I dont know whether this version disparity is significant or not. However, the documentation starts by saying that this module is loaded by default. That being so I would expect sys.prefix to be /usr/local. If I fire up Python import sys and dump sys.prefix it outputs /usr. If I dump site.USER_BASE I get 'name 'site' is not defined'. If I import site and dump site.USER_BASE I get '/home//.local'. This confuses the hell out of me. Two documentary sources suggest this behaviour should be enabled by default, yet it doesn't seem to be, I don't know why, and I don't know how to enable it, or even whether I should. Can anyone either explain why I'm seeing what I'm seeing here, or point the in the direction to some documentation that can. 3. Installers My system Python installation doesn't appear to have either easy_install nor pip nor virtualenv installed. This leaves me with what seems like a circular problem. I'm pretty sure I need pip, but there are (at least) two ways to get it. i) apt-get pip ii) or download and run get-pip.py. I've come across various credible warnings of the dangers of using apt-get and pip to manage the same packages, but once pip is installed I was under the impression you need to use pip to update itself before it will install any packages as pip must be up to date in order to access PyPI. This page http://python-packaging-user-guide.readthedocs.io/en/latest/install_requirements_linux/#installing-pip-setuptools-wheel-with-linux-package-managers ...suggests I should use apt-get to install pip, but that gives rise to the following sequence of operations apt-get pip pip install -U pip ..and isn't that exactly what I should be trying to avoid, installing a package using apt-get and then managing that package using pip? I'd have though that using get-pip.py would be safer, but even then that might only be with the provision that I execute get-pip.py in a Python environment separate from the system core Python installation, otherwise I'm in the same boat of using get-pip.py to upgrade my apt-get managed system Python installation. What to do? 4. Python version. In addition to all this I'm wondering whether I wouldn't be a good idea to have the Python installation I use for development up to 2.7.11 before I start as my system installation of 2.7.6 is 3 years out of date. ---- In short I'm hopelessly confused and could really use some guidance. P.S. What happened to 'batteries included'? Thanks. Neut. From redirect.null at gmail.com Mon May 16 07:29:25 2016 From: redirect.null at gmail.com (redirect.null at gmail.com) Date: Mon, 16 May 2016 04:29:25 -0700 (PDT) Subject: How to create development Python environment on Linux. In-Reply-To: <815aa265-45c4-40a4-860d-beb89cd9a78e@googlegroups.com> References: <815aa265-45c4-40a4-860d-beb89cd9a78e@googlegroups.com> Message-ID: Rereading this through I notice I'm a bit vague on one point. In Point 2. User install site.py behaviour confusing, one of the things that confuses me is that the documentation for site.py here https://docs.python.org/2/library/site.html ...states (in bold) that 'This module is automatically imported during initialization', so I'm confused as to why I have to manaully import it before I can access it? Is the documentation wrong, or did the behaviour change between versions 2.7.6 and 2.7.11 or does the documentation mean something other that what it seems to mean? From jeanmichel at sequans.com Mon May 16 08:20:35 2016 From: jeanmichel at sequans.com (jmp) Date: Mon, 16 May 2016 14:20:35 +0200 Subject: What should a decorator do if an attribute already exists? In-Reply-To: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> References: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 05/10/2016 05:45 PM, Steven D'Aprano wrote: > I have a decorator that adds an attribute to the decorated function: [snip] > I think 5 is clearly wrong, 4 is too difficult, and 3 seems pointless. So I > think either 1 or 2 is the right thing to do. > > Thoughts? It depends if the attribute "instrument" is part of the public interface. If not, just find a different name unlikely to clash with an existing one *and* raise an exception if it does ever happen anyway. If the attribute is part of the public interface but you re using the code only internally then 1/ raise an exception, otherwise I don't know :o) It seems to me that over-writing silently (or not) may lead to bugs difficult to spot. jmp From kevinjacobconway at gmail.com Mon May 16 09:06:51 2016 From: kevinjacobconway at gmail.com (Kevin Conway) Date: Mon, 16 May 2016 13:06:51 +0000 Subject: What should a decorator do if an attribute already exists? In-Reply-To: References: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: > I have a decorator that adds an attribute to the decorated function I might try to argue that this is not actually a decorator or, at least, this is not a great decorator pattern for Python. Adding the attribute to the function object implies you need to access it at some later point. If so then your decorator has changed the public interface of the wrapped item. Downstream code will depend on the new interface which is only provided by the decorator so it becomes a hard requirement. IMO, decorators are supposed to be transparent. Removing a decorator should not invalidate other code. With your particular example, I believe you are actually implementing a context manager and the more appropriate code would look like: with Instrument() as instrument: # other code On Mon, May 16, 2016, 07:23 jmp wrote: > On 05/10/2016 05:45 PM, Steven D'Aprano wrote: > > I have a decorator that adds an attribute to the decorated function: > [snip] > > I think 5 is clearly wrong, 4 is too difficult, and 3 seems pointless. > So I > > think either 1 or 2 is the right thing to do. > > > > Thoughts? > > It depends if the attribute "instrument" is part of the public interface. > > If not, just find a different name unlikely to clash with an existing > one *and* raise an exception if it does ever happen anyway. > > If the attribute is part of the public interface but you re using the > code only internally then 1/ raise an exception, otherwise I don't know :o) > > It seems to me that over-writing silently (or not) may lead to bugs > difficult to spot. > > jmp > > -- > https://mail.python.org/mailman/listinfo/python-list > From grant.b.edwards at gmail.com Mon May 16 12:06:09 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 16 May 2016 16:06:09 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? Message-ID: This is not Python specific, though I'm turning to Python to do some experimentation and to try to prototype a solution. Is there any way to limit the number of connections a browser uses to download a web page? Browser writers seems to assume that all https servers are massively parallel server farms with hardware crypto support. So, when a browser wants to load a page that has the main html file, a css file, a javascript library or two, and a few icons and background bitmaps, they browser opens up a half-dozen SSL connections in parallel. That's fine when the server is Facebook's server farm. But when it's a small embedded device running at 40MHz with a single-threaded web server and software crypto, it turns a 2-second page load time into a 15-second page load time. When we first added https support years ago, this wasn't a problem. A browser would open _an_ SSL connection (handshake time around 2 seconds), and then send mutliple HTTP requests over that connection to grab a half-dozen files. Each HTTP request would take a few tens of milliseconds, and life was good. Now that 2-second page load takes up to 10-15 seconds because of all the SSL connection setup overhead involved in handling a half-dozen "parallel" connections. I was _hoping_ there was an HTTP header or HTML meta tag that could be used to smack the browser with a clue bat, but there doesn't seem to be. [Please tell me I'm wrong...] Some browsers used to have a global "max parallel connections" setting that the user could control, but a) that seems to be gone from recent versions of browsers I've looked at, and b) we can't ask customers to change that setting just for the benefit of our devices. So now I'm going to set up a simple Python HTTP server to try some other approaches: 1) Only allow the listening socket to accept 1 connection at a time. 2) Accept the TCP connection, but don't allow the SSL handshaking to start on the "extra" connections. 3) ??? 4) Profits! Any ideas? -- Grant Edwards grant.b.edwards Yow! What's the MATTER at Sid? ... Is your BEVERAGE gmail.com unsatisfactory? From rosuav at gmail.com Mon May 16 12:15:50 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 17 May 2016 02:15:50 +1000 Subject: OT: limit number of connections from browser to my server? In-Reply-To: References: Message-ID: On Tue, May 17, 2016 at 2:06 AM, Grant Edwards wrote: > So, when a browser wants to load a page that has the main html file, a > css file, a javascript library or two, and a few icons and background > bitmaps, they browser opens up a half-dozen SSL connections in > parallel. > > That's fine when the server is Facebook's server farm. > > But when it's a small embedded device running at 40MHz with a > single-threaded web server and software crypto, it turns a 2-second > page load time into a 15-second page load time. > ... > So now I'm going to set up a simple Python HTTP server to try some > other approaches: > > 1) Only allow the listening socket to accept 1 connection at a time. > > 2) Accept the TCP connection, but don't allow the SSL handshaking to > start on the "extra" connections. > > 3) ??? > > 4) Profits! > > Any ideas? If your server is single-threaded, it ought to be processing only one connection at a time anyway. Are you sure parallel connections are the problem here? The solution might actually be to move all your static files elsewhere. Slap 'em up onto github.io or something, and then the browser is free to make all the parallel connections it likes; your embedded device can just serve the stuff that actually varies (presumably the main HTML file). I know that isn't what you asked for, but it's something to consider :) ChrisA From zachary.ware+pylist at gmail.com Mon May 16 12:25:40 2016 From: zachary.ware+pylist at gmail.com (Zachary Ware) Date: Mon, 16 May 2016 11:25:40 -0500 Subject: How to create development Python environment on Linux. In-Reply-To: <815aa265-45c4-40a4-860d-beb89cd9a78e@googlegroups.com> References: <815aa265-45c4-40a4-860d-beb89cd9a78e@googlegroups.com> Message-ID: On Mon, May 16, 2016 at 6:22 AM, wrote: > I have a Linux system (Mint 17.3 based in Ubuntu 14.04) on which I wish to do some Python development. The system has Python 2.7.6 installed already (there is a Python 3 installation too but I won't be needing to use that to start with). Not what you asked for, but I would encourage you to look into whether it's possible for you to use Python 3 instead of Python 2 for what you're doing. If it's possible, starting with Python 3 will save you several headaches in the future. > [...] > 1. I am unsure which of these technologies is most suitable for my needs. Virtualenv sounds like what I need, but two things give me pause for thought. First I came across this github issue > > https://github.com/pypa/virtualenv/pull/697 > > ...which highlights what seem like severe issues in the vitualenv implementation and proposes to rewrite the whole library. This raises two questions. > > i) Is this rewrite complete? > ii) Is 'legacy' virtualenv or virtualenv-rewrite now the recommended library? virtualenv is a bit of a hack, by necessity. Python 2 doesn't truly support virtualenv, so virtualenv has to make do with what it has available. That said, it has worked well for my needs and for many others. (The real solution is Python 3.3+'s venv module :)) > The second thing that gives me pause for thought is the remark in this article > > http://python-packaging-user-guide.readthedocs.io/en/latest/install_requirements_linux/#installing-pip-setuptools-wheel-with-linux-package-managers > > ...where it says > > 'Recent Debian/Ubuntu versions have modified pip to use the ?User Scheme? by default, which is a significant behavior change that can be surprising to some users.' > > These concerns caused me to try and research more in the area of user installs and package installation, which leads me to my next two issues. The real issue here is mixing the use of the OS package manager with the use of pip. If you only install python packages using the OS package manager (e.g., `apt-get install python-requests`) or only with pip (e.g., `pip install requests`), you'll probably be fine. If you try to use both, though, you may wind up in a weird situation. > 2. User install site.py behaviour confusing. > > The first thing I wanted to work out was whether 'user install' was indeed enabled by default on my system as described in the article above. Here is the documentation for site.py > > https://docs.python.org/2/library/site.html > > This is the documenation for 2.7.11 while I only have 2.7.6. I don't see a link to documentation for 2.7.6 and I dont know whether this version disparity is significant or not. It shouldn't be, but here's the 2.7.6 documentation anyway: https://docs.python.org/release/2.7.6/library/site.html > However, the documentation starts by saying that this module is loaded by default. That being so I would expect sys.prefix to be /usr/local. Why do you expect that? sys.prefix is baked in at compile time of the python interpreter, and refers to where Python is installed on the system. `/usr/local` is the default prefix for custom builds, but most Linux distributions install Python under `/usr`. That the site module is loaded by default just means that particular locations outside of the standard library path are automatically added to sys.path (the list of directories where Python will look when you try to import something). Compare the output of `python -c "import pprint,sys;pprint.pprint(sys.path)"` with and without the -S and -s options. > If I fire up Python import sys and dump sys.prefix it outputs /usr. > If I dump site.USER_BASE I get 'name 'site' is not defined'. Even though site is imported by default by the interpreter, it doesn't bind the name 'site' anywhere in the default namespace. If you want to use the contents of the site module, you still need to import site yourself (this second import simply looks up 'site' in the sys.modules dict and returns the module it finds there). > If I import site and dump site.USER_BASE I get '/home//.local'. > > This confuses the hell out of me. Two documentary sources suggest this behaviour should be enabled by default, yet it doesn't seem to be, I don't know why, and I don't know how to enable it, or even whether I should. > > Can anyone either explain why I'm seeing what I'm seeing here, or point the in the direction to some documentation that can. Hopefully I've cleared things up a bit above. Please ask again if it's still murky! > 3. Installers > > My system Python installation doesn't appear to have either easy_install nor pip nor virtualenv installed. This leaves me with what seems like a circular problem. > > I'm pretty sure I need pip, but there are (at least) two ways to get it. > i) apt-get pip > ii) or download and run get-pip.py. > > I've come across various credible warnings of the dangers of using apt-get and pip to manage the same packages, but once pip is installed I was under the impression you need to use pip to update itself before it will install any packages as pip must be up to date in order to access PyPI. > > This page > > http://python-packaging-user-guide.readthedocs.io/en/latest/install_requirements_linux/#installing-pip-setuptools-wheel-with-linux-package-managers > > ...suggests I should use apt-get to install pip, but that gives rise to the following sequence of operations > > apt-get pip > pip install -U pip > > ..and isn't that exactly what I should be trying to avoid, installing a package using apt-get and then managing that package using pip? > > I'd have though that using get-pip.py would be safer, but even then that might only be with the provision that I execute get-pip.py in a Python environment separate from the system core Python installation, otherwise I'm in the same boat of using get-pip.py to upgrade my apt-get managed system Python installation. What to do? This is where virtualenv helps: create your venv, then use the venv's pip to update pip and then install the packages you want. > 4. Python version. > > In addition to all this I'm wondering whether I wouldn't be a good idea to have the Python installation I use for development up to 2.7.11 before I start as my system installation of 2.7.6 is 3 years out of date. This depends somewhat on where your code will be running in production: if you develop with 2.7.11 but deploy to 2.7.6, you may run into bugs that have been fixed in the interim, or you may try to use some of the security-sensitive features that have been added in more recent 2.7 releases (particularly in the ssl module). Ideally, you should probably be using the latest version for both development and deployment. The simplest way to do that is likely to compile and install your own Python. Here's a quick-start guide for that: - `sudo apt-get build-dep python` to install dependencies (like a C compiler and a few external library headers, like openssl-dev) - Either download and unzip the source from https://python.org/downloads, or clone the repository from hg.python.org/cpython or github.com/python/cpython. If you go the repository route, you'll want to update to the correct revision for the release you want; for the hg repo it'll be tagged "vM.m.p" where M is the major version (2), m is the minor version (7), and p is the patch version (11) (the github repo has no tags yet, it's currently an unofficial mirror). - From the root of the source tree, run `./configure && make profile-opt && make test && sudo make install`. This will build and install a production-ready Python under `/usr/local`. Note that depending on how your PATH is set up, `python` may refer to either the system Python at `/usr/bin/python` or your own Python at `/usr/local/bin/python`. You can avoid a bit of ambiguity by doing `make altinstall` instead of `make install`, which will not create `/usr/local/bin/python`; you'd have to invoke your Python using `python2.7` (again subject to your PATH setup), but `python` would always be the system Python. > In short I'm hopelessly confused and could really use some guidance. > P.S. What happened to 'batteries included'? The standard library is a large complement of high-capacity batteries; PyPI is the Three Gorges Dam. -- Zach From rgaddi at highlandtechnology.invalid Mon May 16 12:34:04 2016 From: rgaddi at highlandtechnology.invalid (Rob Gaddi) Date: Mon, 16 May 2016 16:34:04 -0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: Message-ID: Chris Angelico wrote: > On Tue, May 17, 2016 at 2:06 AM, Grant Edwards > wrote: >> So, when a browser wants to load a page that has the main html file, a >> css file, a javascript library or two, and a few icons and background >> bitmaps, they browser opens up a half-dozen SSL connections in >> parallel. >> >> That's fine when the server is Facebook's server farm. >> >> But when it's a small embedded device running at 40MHz with a >> single-threaded web server and software crypto, it turns a 2-second >> page load time into a 15-second page load time. >> ... >> So now I'm going to set up a simple Python HTTP server to try some >> other approaches: >> >> 1) Only allow the listening socket to accept 1 connection at a time. >> >> 2) Accept the TCP connection, but don't allow the SSL handshaking to >> start on the "extra" connections. >> >> 3) ??? >> >> 4) Profits! >> >> Any ideas? > > If your server is single-threaded, it ought to be processing only one > connection at a time anyway. Are you sure parallel connections are the > problem here? > > The solution might actually be to move all your static files > elsewhere. Slap 'em up onto github.io or something, and then the > browser is free to make all the parallel connections it likes; your > embedded device can just serve the stuff that actually varies > (presumably the main HTML file). I know that isn't what you asked for, > but it's something to consider :) > > ChrisA Oooof. Not to be rude, Chris, but your "software guy" is showing. Grant's got the right of it; if you're shipping a box with an RJ-45 and a webpage, and you want the customer to be able to always make it work, then it needs to be a self-contained entity. The belief that your external dependancies will always be there is why leftpad was able to break everything, and why Google just bricked a bunch of people's expensive Revolv Hubs. The problem with processing one connection at a time is that TCP doesn't transmit your data when you ask it to, it holds onto it for a couple hundred ms to make sure you didn't have anything else to say on that socket. Those build up, and you get horrific page load times because the system is having to single track all the files. Grant, the bad news is that I know this because our firware guy had _exactly_ this problem, with exactly your scenario, about a month ago. http, not https, but the problem remains the same but for some heavy math. After a lot of door knocking, poking, prodding, and hoping, the conclusion he reached was that what you want can't be done, and he had to gut and redesign the web server to support parallel connections. Turned a 45 second page load into south of one, but it wasn't pretty and chewed up a bunch of RAM. We had 256K to play in; I'm assuming you've got closer to 32K. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. From random832 at fastmail.com Mon May 16 12:40:07 2016 From: random832 at fastmail.com (Random832) Date: Mon, 16 May 2016 12:40:07 -0400 Subject: How to get a directory list sorted by date? In-Reply-To: References: <20160515061511.7c62b0e1@bigbox.christie.dr> Message-ID: <1463416807.3921159.609333921.50A0A282@webmail.messagingengine.com> On Sun, May 15, 2016, at 17:52, Chris Angelico wrote: > On Mon, May 16, 2016 at 2:00 AM, Grant Edwards > wrote: > > On 2016-05-15, Michael Selik wrote: > >> On Sun, May 15, 2016, 10:37 AM Grant Edwards wrote: > >>> On 2016-05-15, Tim Chase wrote: > >>>> > >>>> unless sorted() returns a lazy sorter, > >>> > >>> What's a lazy sorter? > >> > >> One that doesn't calculate the next item in the sequence until you > >> ask for it. It's impossible > > > > Why? As long as the function has access to the entire sequence, it > > should be trivial. Just find the min (or max) item in the sequence, > > remove it, then return it. It's horribly inefficient, but... > > > >> unless you don't mind an approximation rather than correct sort. > > > > I have a feeling that I've missed the joke somewhere. > > Sure, it's not impossible to implement, but generally the point of > lazy generation is that it's more efficient. Going from O(N log N) > with O(N) memory to O(N*N) with O(1) memory is not usually what you > want! It could be "lazy" in the sense that it doesn't consume the source iterator or do any sorting at all until you start iterating it. And if you perform some operation (such as, for example, sorting again by a different key) that is destructive of the original order it could dispense with the sorting process (or compose the two key functions, if the sorting algorithm is meant to be stable) Or, for example, if you filter the sorted collection, a "lazy" filter/sorter could reorder the operations so that the unsorted list is filtered before being sorted, reducing N for the sort operation. From grant.b.edwards at gmail.com Mon May 16 12:40:53 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 16 May 2016 16:40:53 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: Message-ID: On 2016-05-16, Chris Angelico wrote: > On Tue, May 17, 2016 at 2:06 AM, Grant Edwards> wrote: >> So, when a browser wants to load a page that has the main html file, a >> css file, a javascript library or two, and a few icons and background >> bitmaps, they browser opens up a half-dozen SSL connections in >> parallel. >> >> That's fine when the server is Facebook's server farm. >> >> But when it's a small embedded device running at 40MHz with a >> single-threaded web server and software crypto, it turns a 2-second >> page load time into a 15-second page load time. [...] > If your server is single-threaded, it ought to be processing only one > connection at a time anyway. Are you sure parallel connections are the > problem here? It handles one HTTP request at a time, but it allows multiple connections to be open simultaneously. It's got a sort of "event loop" where each time through the loop it accepts all new connections, shuts down any closed connections, and then handles requests (one at a time) on any open connections that are readable. I can see from the devices logs and from Chrome's network monitor that it has opened 5-6 parallel SSL connections by the time the page is loaded. I can also see that pending HTTP requests on already-opened connections are being delayed by the SSL connection setup times of the newer connections. > The solution might actually be to move all your static files > elsewhere. Slap 'em up onto github.io or something, and then the > browser is free to make all the parallel connections it likes; your > embedded device can just serve the stuff that actually varies > (presumably the main HTML file). I know that isn't what you asked > for, but it's something to consider :) Our devices are often used on air-gapped industrial networks with no connection to the interwebs. We can't assume access to any other web servers. I've thought about trying to do as much as possible with server-side includes for .js and .css files and embedding graphics elements in the HTML pages for icons, background, etc. But, that makes development more complicated and you give up the advanges of browser caching of css, js, png, and gif files. -- Grant Edwards grant.b.edwards Yow! Is it clean in other at dimensions? gmail.com From rosuav at gmail.com Mon May 16 12:52:08 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 17 May 2016 02:52:08 +1000 Subject: OT: limit number of connections from browser to my server? In-Reply-To: References: Message-ID: On Tue, May 17, 2016 at 2:34 AM, Rob Gaddi wrote: >> The solution might actually be to move all your static files >> elsewhere. Slap 'em up onto github.io or something, and then the >> browser is free to make all the parallel connections it likes; your >> embedded device can just serve the stuff that actually varies >> (presumably the main HTML file). I know that isn't what you asked for, >> but it's something to consider :) >> >> ChrisA > > Oooof. Not to be rude, Chris, but your "software guy" is showing. > Grant's got the right of it; if you're shipping a box with an RJ-45 and > a webpage, and you want the customer to be able to always make it > work, then it needs to be a self-contained entity. The belief that your > external dependancies will always be there is why leftpad was able to > break everything, and why Google just bricked a bunch of people's > expensive Revolv Hubs. I agree, but I also make no apology for suggesting the option of getting someone else to do some of the work. In this case, it can be rejected for the exact reason you cite (dependencies are a cost, and in this case way too high a cost), and that's fine and correct. Ultimately, if the job gets done, everything else is implementation detail, with consequences - and I know a lot of people who'll willingly sacrifice "reliability in the face of an internet connection outage" in favour of "less than fifteen second response time". Hence my suggestion, albeit couched in "might be" and "something to consider". :) ChrisA From steve at pearwood.info Mon May 16 13:11:17 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 17 May 2016 03:11:17 +1000 Subject: OT: limit number of connections from browser to my server? References: Message-ID: <5739ff37$0$1605$c3e8da3$5496439d@news.astraweb.com> On Tue, 17 May 2016 02:06 am, Grant Edwards wrote: > This is not Python specific, though I'm turning to Python to do some > experimentation and to try to prototype a solution. > > Is there any way to limit the number of connections a browser uses to > download a web page? Well, you control the server. It's your own server, right? Can you not tell the server to limit how many connections it accepts from any one client? That's not a rhetorical question -- I don't know the answer, and I know nothing about HTTPS server. But I would imagine that they don't allow unlimited connections from each client. Can you tell your server to only allow N connections from each client, where N is small enough? I suppose that's your question, right? Of you were using Apache, this might point you in the right direction... http://serverfault.com/questions/252555/limit-simultaneous-connections-per-ip-with-apache2 -- Steven From laurent.pointal at free.fr Mon May 16 13:13:35 2016 From: laurent.pointal at free.fr (Laurent Pointal) Date: Mon, 16 May 2016 19:13:35 +0200 Subject: A tough one: split on word length? References: Message-ID: <5739ffbf$0$26281$426a74cc@news.free.fr> DFS wrote: > Have: > '584323 Fri 13 May 2016 17:37:01 -0000 (UTC) 584324 Fri 13 May 2016 > 13:44:40 -0400 584325 13 May 2016 17:45:25 GMT 584326 Fri 13 May 2016 > 13:47:28 -0400' > > Want: > [('584323', 'Fri 13 May 2016 17:37:01 -0000 (UTC)'), > ('584324', 'Fri 13 May 2016 13:44:40 -0400'), > ('584325', '13 May 2016 17:45:25 GMT'), > ('584326', 'Fri 13 May 2016 13:47:28 -0400')] > > > Or maybe split() on space, then run through and add words of 6+ numbers > to the list, then recombine everything until you hit the next group of > 6+ numbers, and so on? > > The data is guaranteed to contain those 6+ groups of numbers. Test with regexp under Python3 >>> import re >>> s = '584323 Fri 13 May 2016 17:37:01 -0000 (UTC) 584324 Fri 13 May 2016 13:44:40 -0400 584325 13 May 2016 17:45:25 GMT 584326 Fri 13 May 2016 13:47:28 -0400' >>> re.split("(\d{6})(.*?)", s) ['', '584323', '', ' Fri 13 May 2016 17:37:01 -0000 (UTC) ', '584324', '', ' Fri 13 May 2016 13:44:40 -0400 ', '584325', '', ' 13 May 2016 17:45:25 GMT ', '584326', '', ' Fri 13 May 2016 13:47:28 -0400'] Dismiss empty items and strip whitespaces at begin or end of string, and that's done. A+ Laurent. Note: re experts will provide a cleaner solution. From jeanmichel at sequans.com Mon May 16 13:26:36 2016 From: jeanmichel at sequans.com (jmp) Date: Mon, 16 May 2016 19:26:36 +0200 Subject: OT: limit number of connections from browser to my server? In-Reply-To: References: Message-ID: On 05/16/2016 06:06 PM, Grant Edwards wrote: > This is not Python specific, though I'm turning to Python to do some > experimentation and to try to prototype a solution. > > Is there any way to limit the number of connections a browser uses to > download a web page? Browser writers seems to assume that all https > servers are massively parallel server farms with hardware crypto > support. > > So, when a browser wants to load a page that has the main html file, a > css file, a javascript library or two, and a few icons and background > bitmaps, they browser opens up a half-dozen SSL connections in > parallel. > > That's fine when the server is Facebook's server farm. > > But when it's a small embedded device running at 40MHz with a > single-threaded web server and software crypto, it turns a 2-second > page load time into a 15-second page load time. > > When we first added https support years ago, this wasn't a problem. A > browser would open _an_ SSL connection (handshake time around 2 > seconds), and then send mutliple HTTP requests over that connection to > grab a half-dozen files. Each HTTP request would take a few tens of > milliseconds, and life was good. > > Now that 2-second page load takes up to 10-15 seconds because of all > the SSL connection setup overhead involved in handling a half-dozen > "parallel" connections. > > I was _hoping_ there was an HTTP header or HTML meta tag that could be > used to smack the browser with a clue bat, but there doesn't seem to > be. [Please tell me I'm wrong...] > > Some browsers used to have a global "max parallel connections" setting > that the user could control, but a) that seems to be gone from recent > versions of browsers I've looked at, and b) we can't ask customers to > change that setting just for the benefit of our devices. > > So now I'm going to set up a simple Python HTTP server to try some > other approaches: > > 1) Only allow the listening socket to accept 1 connection at a time. > > 2) Accept the TCP connection, but don't allow the SSL handshaking to > start on the "extra" connections. > > 3) ??? > > 4) Profits! > > Any ideas? > Have you considered upgrading the device with a recent CPU ? Or is it completely out of the picture ? Depending on what you are selling, it may be actually cheaper than spending time trying to make it work. You could also "externalize" the web service, a 35$ raspberry pi would do it. Of course I do realize that everything I said may not make any sense, we'd need to know a little bit more about the "device". If 35$ double the price, that may not be a good idea. That being said, your first idea seems also a good lead, have your server refuse more than one connection. jm From __peter__ at web.de Mon May 16 13:34:45 2016 From: __peter__ at web.de (Peter Otten) Date: Mon, 16 May 2016 19:34:45 +0200 Subject: OT: limit number of connections from browser to my server? References: Message-ID: Grant Edwards wrote: > This is not Python specific, though I'm turning to Python to do some > experimentation and to try to prototype a solution. > > Is there any way to limit the number of connections a browser uses to > download a web page? Browser writers seems to assume that all https > servers are massively parallel server farms with hardware crypto > support. > > So, when a browser wants to load a page that has the main html file, a > css file, a javascript library or two, and a few icons and background > bitmaps, they browser opens up a half-dozen SSL connections in > parallel. [brainstorm-mode on] I think HTTP/2 allows multiple requests over a single TCP connection. From larry.martell at gmail.com Mon May 16 13:36:15 2016 From: larry.martell at gmail.com (Larry Martell) Date: Mon, 16 May 2016 13:36:15 -0400 Subject: gevent Message-ID: Does anyone here use gevent? I posted a question on the gevent mailing list, here, and on SO and did not get any replies on any of them. I have a client that is using it - I had never heard of it before, and now I am wondering how big the user base and community is. From sohcahtoa82 at gmail.com Mon May 16 13:57:06 2016 From: sohcahtoa82 at gmail.com (sohcahtoa82 at gmail.com) Date: Mon, 16 May 2016 10:57:06 -0700 (PDT) Subject: OT: limit number of connections from browser to my server? In-Reply-To: References: Message-ID: On Monday, May 16, 2016 at 10:35:28 AM UTC-7, Peter Otten wrote: > Grant Edwards wrote: > > > This is not Python specific, though I'm turning to Python to do some > > experimentation and to try to prototype a solution. > > > > Is there any way to limit the number of connections a browser uses to > > download a web page? Browser writers seems to assume that all https > > servers are massively parallel server farms with hardware crypto > > support. > > > > So, when a browser wants to load a page that has the main html file, a > > css file, a javascript library or two, and a few icons and background > > bitmaps, they browser opens up a half-dozen SSL connections in > > parallel. > > [brainstorm-mode on] > > I think HTTP/2 allows multiple requests over a single TCP connection. HTTP/1.1 already supports it, but most browsers have it disabled by default. https://en.wikipedia.org/wiki/HTTP_pipelining From random832 at fastmail.com Mon May 16 14:02:08 2016 From: random832 at fastmail.com (Random832) Date: Mon, 16 May 2016 14:02:08 -0400 Subject: OT: limit number of connections from browser to my server? In-Reply-To: References: Message-ID: <1463421728.3941354.609420521.3E2992BE@webmail.messagingengine.com> On Mon, May 16, 2016, at 13:34, Peter Otten wrote: > I think HTTP/2 allows multiple requests over a single TCP connection. HTTP/1.1 allows the same just fine, the problem here is convincing the browser to actually do it. From sohcahtoa82 at gmail.com Mon May 16 14:04:33 2016 From: sohcahtoa82 at gmail.com (sohcahtoa82 at gmail.com) Date: Mon, 16 May 2016 11:04:33 -0700 (PDT) Subject: A strange one: a commented line throws an error In-Reply-To: References: Message-ID: <7e109b65-d6f5-4de9-8d84-167776eb922e@googlegroups.com> On Monday, May 16, 2016 at 10:25:54 AM UTC-7, DFS wrote: > print "test" > # stz source pytz.timezone() instance (for na?ve local datetimes) > > $ python temp.py > File "temp.py", line 2 > SyntaxError: Non-ASCII character '\xc3' in file temp.py on line 2, but > no encoding declared; see http://python.org/dev/peps/pep-0263/ for details > > > > > python 2.7.11 on Windows Yup. That's the error I would expect when you use non-ASCII characters in your code without declaring the character encoding as described in PEP 263, which is conveniently linked in the exception details. From random832 at fastmail.com Mon May 16 14:04:52 2016 From: random832 at fastmail.com (Random832) Date: Mon, 16 May 2016 14:04:52 -0400 Subject: OT: limit number of connections from browser to my server? In-Reply-To: References: Message-ID: <1463421892.3941681.609425321.08BA9E0D@webmail.messagingengine.com> On Mon, May 16, 2016, at 13:57, sohcahtoa82 at gmail.com wrote: > On Monday, May 16, 2016 at 10:35:28 AM UTC-7, Peter Otten wrote: > > I think HTTP/2 allows multiple requests over a single TCP connection. > > HTTP/1.1 already supports it, but most browsers have it disabled by > default. > > https://en.wikipedia.org/wiki/HTTP_pipelining Technically, what he's asking for isn't even pipelining. https://en.wikipedia.org/wiki/HTTP_persistent_connection (The new feature in HTTP/2, incidentally, is support for multiple *simultaneous* responses) From moagstar at gmail.com Mon May 16 14:49:42 2016 From: moagstar at gmail.com (Daniel Bradburn) Date: Mon, 16 May 2016 18:49:42 +0000 Subject: gevent In-Reply-To: References: Message-ID: Hi Larry, I'm afraid I don't have any experience using gevent with wsgi, perhaps you could try posting directly on the Google group https://groups.google.com/forum/#!forum/gevent On Mon, 16 May 2016, 20:28 Larry Martell, wrote: > I don't see it either. I posted it twice - May 3 and May 14. I did it > from email - does that group not accept email postings? I didn't get a > bounce though. > > Here is my post: > > I have a python server that has this in the main: > > from gevent import pywsgi > > try: > httpd = pywsgi.WSGIServer(('0.0.0.0', 8000), app) > httpd.serve_forever() > except KeyboardInterrupt: > pass > > Recently we began getting HTTPError: 504 Server Error: Gateway > Time-out on requests to the server. Looking in the logs I saw this: > > Traceback (most recent call last): > File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line > 508, in handle_one_response > self.run_application() > File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line > 495, in run_application > self.process_result() > File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line > 486, in process_result > self.write(data) > File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line > 380, in write > self._write_with_headers(data) > File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line > 400, in _write_with_headers > self._sendall(towrite) > File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line > 355, in _sendall > self.socket.sendall(data) > File "/usr/local/lib/python2.7/dist-packages/gevent/socket.py", line > 460, in sendall > data_sent += self.send(_get_memory(data, data_sent), flags) > File "/usr/local/lib/python2.7/dist-packages/gevent/socket.py", line > 437, in send > return sock.send(data, flags) > error: [Errno 32] Broken pipe > {'GATEWAY_INTERFACE': 'CGI/1.1', > 'HTTP_ACCEPT': '*/*', > 'HTTP_ACCEPT_ENCODING': 'gzip, deflate', > 'HTTP_CONNECTION': 'close', > 'HTTP_HOST': 'foo.bar.com', > 'HTTP_USER_AGENT': 'python-requests/2.9.1', > 'HTTP_X_FORWARDED_FOR': '192.168.10.5', > 'HTTP_X_REAL_IP': '192.168.10.5', > 'PATH_INFO': '/readings', > 'QUERY_STRING': 'target=2225217193085335062&tool=HM', > 'REMOTE_ADDR': '172.17.42.1', > 'REMOTE_PORT': '41708', > 'REQUEST_METHOD': 'GET', > 'SCRIPT_NAME': '', > 'SERVER_NAME': 'fdffcd119506', > 'SERVER_PORT': '8000', > 'SERVER_PROTOCOL': 'HTTP/1.0', > 'SERVER_SOFTWARE': 'gevent/1.0 Python/2.7', > 'wsgi.errors': ', mode 'w' at 0x7f4c299bc1e0>, > 'wsgi.input': , > 'wsgi.multiprocess': False, > 'wsgi.multithread': False, > 'wsgi.run_once': False, > 'wsgi.url_scheme': 'http', > 'wsgi.version': (1, 0)} failed with error > > Once this occurs it seems that no further requests are served. I > googled this and found many places where people said this was > happening, and many places where people explained why it was > happening, but no place did I find a way to fix or workaround this. > > Does anyone know how to get it to continue processing after this occurs? > > On Mon, May 16, 2016 at 2:15 PM, Daniel Bradburn > wrote: > > Hi Larry > > > > When did you post the query to the gevent list? I don't see it on > > gevent at googlegroups.com > > > > My perception is that gevent is fairly widely used although I don't have > any > > stats, see for example http://www.gevent.org/success.html. Personally > it is > > my go to library whenever I need async networking in python. > > > > What was the question you wanted answering? > > > > On 16 May 2016 7:39 p.m., "Larry Martell" > wrote: > >> > >> Does anyone here use gevent? I posted a question on the gevent mailing > >> list, here, and on SO and did not get any replies on any of them. I > >> have a client that is using it - I had never heard of it before, and > >> now I am wondering how big the user base and community is. > >> -- > >> https://mail.python.org/mailman/listinfo/python-list > From grant.b.edwards at gmail.com Mon May 16 15:19:34 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 16 May 2016 19:19:34 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: Message-ID: On 2016-05-16, Rob Gaddi wrote: > Grant, the bad news is that I know this because our firware guy had > _exactly_ this problem, with exactly your scenario, about a month ago. > http, not https, but the problem remains the same but for some heavy > math. After a lot of door knocking, poking, prodding, and hoping, the > conclusion he reached was that what you want can't be done, Bummer. > and he had to gut and redesign the web server to support parallel > connections. Mine does (in effect) support parallel connections. With HTTP it all works fine: browser opens 4-5 connections in parallel; sends requests on all of them; responses come back on all of them. A few of the connections then get reused for additional requests, and so on. > Turned a 45 second page load into south of one, but it wasn't pretty and > chewed up a bunch of RAM. We had 256K to play in; I'm assuming you've > got closer to 32K. I've actually got plenty of RAM. I just can't afford the CPU time it takes to do the public-key crypto stuff that happens each time an SSL connection starts up. I'm hoping that if I refuse additional connections (or stall them before the SSL handshake), I can convince the browser to make use of extant connections. However, in my case, there are some cases where a connection can't be re-used because the server doesn't know how big the reply is going to be and the only way to identy "end of reply" is for the server to close the connection. In theory I can fix that by adding support to the server for chunked encoding in the cases where the reply size is unknown. But that won't do me any good if the browser opens 4 more connections before it's even seen the response from the first one. -- Grant Edwards grant.b.edwards Yow! I'm having a RELIGIOUS at EXPERIENCE ... and I don't gmail.com take any DRUGS From ben.usenet at bsb.me.uk Mon May 16 15:27:10 2016 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Mon, 16 May 2016 20:27:10 +0100 Subject: A tough one: split on word length? References: Message-ID: <87bn45ztv5.fsf@bsb.me.uk> DFS writes: > Have: > '584323 Fri 13 May 2016 17:37:01 -0000 (UTC) 584324 Fri 13 May 2016 > 13:44:40 -0400 584325 13 May 2016 17:45:25 GMT 584326 Fri 13 May 2016 > 13:47:28 -0400' > > Want: > [('584323', 'Fri 13 May 2016 17:37:01 -0000 (UTC)'), > ('584324', 'Fri 13 May 2016 13:44:40 -0400'), > ('584325', '13 May 2016 17:45:25 GMT'), > ('584326', 'Fri 13 May 2016 13:47:28 -0400')] [m for m in re.findall(r'(\d{6}) (.*?) ?(?:(?=\d{6})|$)', s)] -- Ben. From grant.b.edwards at gmail.com Mon May 16 15:28:43 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 16 May 2016 19:28:43 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: <5739ff37$0$1605$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-16, Steven D'Aprano wrote: > On Tue, 17 May 2016 02:06 am, Grant Edwards wrote: > >> This is not Python specific, though I'm turning to Python to do some >> experimentation and to try to prototype a solution. >> >> Is there any way to limit the number of connections a browser uses to >> download a web page? > > Well, you control the server. It's your own server, right? Can you > not tell the server to limit how many connections it accepts from > any one client? Definitely. > That's not a rhetorical question -- I don't know the answer, and I > know nothing about HTTPS server. But I would imagine that they don't > allow unlimited connections from each client. Can you tell your > server to only allow N connections from each client, where N is > small enough? Yep, I should be able to do that. I can even reduce it to a simpler case of only allowing N connections total. Saying we only support one browser session at a time is probably acceptable if it solves the long https page load times. Before I go to the effort of hacking up the server to change how/when it accepts connections[1], I want to try prototyping something like that Python to make sure that refusing or stalling a "parallel" connection won't cause the browser to throw up its hands and tell the user "this page can't be loaded." [1] We didn't write the server. We licensed source from somebody yonks ago. It was then abaondoned and maintined by us and by applying patches we got from other users in our situation. -- Grant Edwards grant.b.edwards Yow! I'm RELIGIOUS!! at I love a man with gmail.com a HAIRPIECE!! Equip me with MISSILES!! From grant.b.edwards at gmail.com Mon May 16 15:31:04 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 16 May 2016 19:31:04 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: Message-ID: On 2016-05-16, jmp wrote: > Have you considered upgrading the device with a recent CPU ? Or is it > completely out of the picture ? Not an option. We have to continue to support devices that are in the field. The newer models that are coming out now run at 133MHz instead of 44MHz, and page load times for https still aren't much better. > That being said, your first idea seems also a good lead, have your > server refuse more than one connection. Just got to figure out how to try it out without wasting a lot of time reverse-engineering the web server. Hence a Python prototype. :) -- Grant Edwards grant.b.edwards Yow! Gee, I feel kind of at LIGHT in the head now, gmail.com knowing I can't make my satellite dish PAYMENTS! From grant.b.edwards at gmail.com Mon May 16 15:33:24 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 16 May 2016 19:33:24 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: Message-ID: On 2016-05-16, Peter Otten <__peter__ at web.de> wrote: > Grant Edwards wrote: > >> This is not Python specific, though I'm turning to Python to do some >> experimentation and to try to prototype a solution. >> >> Is there any way to limit the number of connections a browser uses to >> download a web page? Browser writers seems to assume that all https >> servers are massively parallel server farms with hardware crypto >> support. >> >> So, when a browser wants to load a page that has the main html file, a >> css file, a javascript library or two, and a few icons and background >> bitmaps, they browser opens up a half-dozen SSL connections in >> parallel. > > [brainstorm-mode on] > > I think HTTP/2 allows multiple requests over a single TCP connection. HTTP 1.1 did also. And our server supports it. The problem is that modern browsers won't wait and send requests serially over a single connection. They try to "optimize" page load times by opening as many connections as they cat right away and requesting everything in parallel. When the server has a single CPU, that just wastes a lot of time -- particulary when connections have a high per-connection overhead like SSL does. -- Grant Edwards grant.b.edwards Yow! Is this TERMINAL fun? at gmail.com From grant.b.edwards at gmail.com Mon May 16 15:35:25 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 16 May 2016 19:35:25 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: <1463421892.3941681.609425321.08BA9E0D@webmail.messagingengine.com> Message-ID: On 2016-05-16, Random832 wrote: > On Mon, May 16, 2016, at 13:57, sohcahtoa82 at gmail.com wrote: >> On Monday, May 16, 2016 at 10:35:28 AM UTC-7, Peter Otten wrote: >> >>> I think HTTP/2 allows multiple requests over a single TCP connection. >> >> HTTP/1.1 already supports it, but most browsers have it disabled by >> default. >> >> https://en.wikipedia.org/wiki/HTTP_pipelining Oops, sorry, I didn't see that you were referring to something other than persistent connections. > Technically, what he's asking for isn't even pipelining. > > https://en.wikipedia.org/wiki/HTTP_persistent_connection Right. I just want convince browsers to use persistent connections and as few of them as possible. -- Grant Edwards grant.b.edwards Yow! One FISHWICH coming at up!! gmail.com From netcrime4 at gmail.com Mon May 16 15:46:13 2016 From: netcrime4 at gmail.com (netcrime4 at gmail.com) Date: Mon, 16 May 2016 12:46:13 -0700 (PDT) Subject: Wanted Python programmer to join team Message-ID: My team is getting more projects that it can handle so we are looking for Python programers to join. You will be given tasks to complete full or part of the project. Majority of projects consist of data mining(scraping) so experence in this field is advantage. But we also have various tasks so we are looking for Python programmers in general. Job: Remote Requirements: Responsability Good Python knownladge Clean code(PEP, ZEN) Advantage: Django Flask Scrapy BeautifulSoup PhantomJS Selenium Regular Expresion Css Javascript Html Populiari? API's i?manymas Unix administravimas GIT Salary: We will agree on each task Skype: piefektas Contact me now with short description about yourself, your skills and projects you have worked on. From best_lay at yahoo.com Mon May 16 16:01:38 2016 From: best_lay at yahoo.com (Wildman) Date: Mon, 16 May 2016 15:01:38 -0500 Subject: Wanted Python programmer to join team References: Message-ID: On Mon, 16 May 2016 12:46:13 -0700, netcrime4 wrote: > My team is getting more projects that it can handle so we are looking for Python programers to join. You will be given tasks to complete full or part of the project. > > Majority of projects consist of data mining(scraping) so experence in this field is advantage. No thanks. -- GNU/Linux user #557453 The cow died so I don't need your bull! From rgaddi at highlandtechnology.invalid Mon May 16 16:09:05 2016 From: rgaddi at highlandtechnology.invalid (Rob Gaddi) Date: Mon, 16 May 2016 20:09:05 -0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: Message-ID: Grant Edwards wrote: > On 2016-05-16, Rob Gaddi wrote: > >> Grant, the bad news is that I know this because our firware guy had >> _exactly_ this problem, with exactly your scenario, about a month ago. >> http, not https, but the problem remains the same but for some heavy >> math. After a lot of door knocking, poking, prodding, and hoping, the >> conclusion he reached was that what you want can't be done, > > Bummer. > >> and he had to gut and redesign the web server to support parallel >> connections. > > Mine does (in effect) support parallel connections. With HTTP it all > works fine: browser opens 4-5 connections in parallel; sends requests > on all of them; responses come back on all of them. A few of the > connections then get reused for additional requests, and so on. > >> Turned a 45 second page load into south of one, but it wasn't pretty and >> chewed up a bunch of RAM. We had 256K to play in; I'm assuming you've >> got closer to 32K. > > I've actually got plenty of RAM. I just can't afford the CPU time it > takes to do the public-key crypto stuff that happens each time an SSL > connection starts up. > > I'm hoping that if I refuse additional connections (or stall them > before the SSL handshake), I can convince the browser to make use of > extant connections. > > However, in my case, there are some cases where a connection can't be > re-used because the server doesn't know how big the reply is going to > be and the only way to identy "end of reply" is for the server to > close the connection. In theory I can fix that by adding support to > the server for chunked encoding in the cases where the reply size is > unknown. But that won't do me any good if the browser opens 4 more > connections before it's even seen the response from the first one. > That's easy then. Redesign the product to outsource the crypto to a custom-designed FPGA which pretends to be DDR on the memory bus to maximize data throughput. By the time you're done with all of that, the HTTP standards should have evolved to obviate the original problem. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. From D.Strohl at F5.com Mon May 16 16:34:39 2016 From: D.Strohl at F5.com (Dan Strohl) Date: Mon, 16 May 2016 20:34:39 +0000 Subject: Wanted Python programmer to join team In-Reply-To: References: Message-ID: <7c1497d29c7146328545aa918522f986@seaexchmbx03.olympus.F5Net.com> > My team is getting more projects that it can handle so we are looking for > Python programers to join. You will be given tasks to complete full or part of > the project. > > Skype: piefektas > > Contact me now with short description about yourself, your skills and > projects you have worked on. > Sorry... nope. You lost me with using an email address like "netcrime4" and using "gmail" as your address. As a suggestion next time, try using a more professional address, such as one for a company, and one that does not suggest that the work might not be totally legit. (NOTE: I am NOT saying you aren?t legit, nor that the work is criminal, just that the impression you gave was less than professional, and even if I was looking, I would not consider contacting someone who presented that way.) From no.email at nospam.invalid Mon May 16 16:41:38 2016 From: no.email at nospam.invalid (Paul Rubin) Date: Mon, 16 May 2016 13:41:38 -0700 Subject: OT: limit number of connections from browser to my server? References: Message-ID: <87a8jpohvh.fsf@jester.gateway.pace.com> Grant Edwards writes: > I've actually got plenty of RAM. I just can't afford the CPU time it > takes to do the public-key crypto stuff that happens each time an SSL > connection starts up. I think you should only have to do that once, then use TLS session resumption for additional connections. There is also something called TLS-PSK in TLS 1.3. Do you mind saying the application, and what clients you have to support? What TLS stack are you using? There is generally also a way to configure browsers to limit the number of outgoing connections. I'll probably be meeting with some TLS experts tomorrow night for unrelated reasons, so I can ask them about this if you want. From redirect.null at gmail.com Mon May 16 17:28:08 2016 From: redirect.null at gmail.com (redirect.null at gmail.com) Date: Mon, 16 May 2016 14:28:08 -0700 (PDT) Subject: How to create development Python environment on Linux. In-Reply-To: <815aa265-45c4-40a4-860d-beb89cd9a78e@googlegroups.com> References: <815aa265-45c4-40a4-860d-beb89cd9a78e@googlegroups.com> Message-ID: <29b59dce-2036-43f6-8166-7e3ad15e8e4c@googlegroups.com> Thanks Zach, that's a big help. The only reason I want to get a Python 2.7 environment working first is because I'll be working on third party code and that's the platform it uses. For any new projects I would use Python 3. After considering your guidance I think what I will do is install virtualenv using apt-get and then use that to create a dev environment. Is it ok to run get-pip.py in a virtual environment? I won't worry about using the latest version of 2.7 for now, since it's only one or two third party open source projects I'll use 2.7 for and they don't need a more recent version. There are a couple of other things I'm not quite clear on, such as where it would be best to create my new virtual environment (I'm tempted to put it in /usr/local if that means it can be used by all user accounts on my machine), and how I can can control which Python environment is used by the various system and user programs that depend on them, but I expect I can find that information on the web, though I'll make another post here if I do get stuck. From grant.b.edwards at gmail.com Mon May 16 17:32:40 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 16 May 2016 21:32:40 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: <87a8jpohvh.fsf@jester.gateway.pace.com> Message-ID: On 2016-05-16, Paul Rubin wrote: > Grant Edwards writes: >> I've actually got plenty of RAM. I just can't afford the CPU time >> it takes to do the public-key crypto stuff that happens each time >> an SSL connection starts up. > > I think you should only have to do that once, then use TLS session > resumption for additional connections. Thanks, I'll look into that -- I've seen the term before, but that's about it. Is it something the server tells the client to do? And more to the point, will all popular browsers do it? > There is also something called TLS-PSK in TLS 1.3. Do you mind > saying the application, and what clients you have to support? The application is something proprietary running on proprietary hardware (32-bit ARM processor running a typical RTOS and a BSD-derived network stack). The web server is a heavily modified version of GoAhead 2.something. I have to support the usual suspect list of browsers: IE, Firefox, Chrome, Safari. > What TLS stack are you using? It's not an open-source one. Beyond that, I can't really say. > There is generally also a way to configure browsers to limit the > number of outgoing connections. I can't ask the browser user to change settings. > I'll probably be meeting with some TLS experts tomorrow night for > unrelated reasons, so I can ask them about this if you want. -- Grant Edwards grant.b.edwards Yow! I want you to MEMORIZE at the collected poems of gmail.com EDNA ST VINCENT MILLAY ... BACKWARDS!! From michael.selik at gmail.com Mon May 16 17:56:22 2016 From: michael.selik at gmail.com (Michael Selik) Date: Mon, 16 May 2016 21:56:22 +0000 Subject: How to create development Python environment on Linux. In-Reply-To: <29b59dce-2036-43f6-8166-7e3ad15e8e4c@googlegroups.com> References: <815aa265-45c4-40a4-860d-beb89cd9a78e@googlegroups.com> <29b59dce-2036-43f6-8166-7e3ad15e8e4c@googlegroups.com> Message-ID: On Mon, May 16, 2016 at 5:31 PM wrote: > After considering your guidance I think what I will do is install > virtualenv using apt-get and then use that to create a dev environment. Is > it ok to run get-pip.py in a virtual environment? > Recent versions of the virtualenv application create virtual environments that already have pip. It's the default setting last time I checked. From grant.b.edwards at gmail.com Mon May 16 18:09:26 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 16 May 2016 22:09:26 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: <5739ff37$0$1605$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-16, Steven D'Aprano wrote: > On Tue, 17 May 2016 02:06 am, Grant Edwards wrote: > >> This is not Python specific, though I'm turning to Python to do some >> experimentation and to try to prototype a solution. >> >> Is there any way to limit the number of connections a browser uses to >> download a web page? > > Well, you control the server. It's your own server, right? Can you not tell > the server to limit how many connections it accepts from any one client? I tried two approaches on Linux using a Python server prototype that has some code I wrote that accepts connections and then calls SimpleHTTPRequestHandler. For starters I'm sticking with HTTP (no SSL) and just stuck a 2-second (delay to simulate SSL startup time) between the accept() returning and the call to the request handler . 1) Refuse to accept more than one connection at a time by closing the listening socket while there is an active connection. On both Firefox and Chrome, this gets you the initial HTML page and perhaps one addition file (css, js, png, whatever). The files that the browser normally requests using the addition 4-5 requests simply fail (broken img, missing css and js files). The Chrome network monitor thingy shows those as "failed". In my test case, only allowing a single connection resulted in 7 failed files out of the 9 which are loaded for the page. 2) Limit the listen() backlog to 0. On Linux, a 0 backlog doesn't prevent the stack from sending back a SYN/ACK in response to a SYN. It just prevents the connection from ACKing anything else that's sent by the client. This makes the client think the connection is open. In my tests, the browser then sits there waiting for a response on what it _thinks_ is an open TCP connection. It eventually times out and fails. It looks to me like the browsers decide how many connections to use (in my test case it's usually around 5), assigns individual files to each connection, and then presses "go". If any connection fails, stalls, or falls over, the files assigned to that connection are failed rather than fetched from an extant working connection. Next I'll try adding the SSL layer -- perhaps stalling the connection after the accept() and before the SSL handshake will have a different affect on the browser that stalling a plaintext TCP connection before responding to the request. -- Grant Edwards grant.b.edwards Yow! PARDON me, am I at speaking ENGLISH? gmail.com From rosuav at gmail.com Mon May 16 19:07:02 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 17 May 2016 09:07:02 +1000 Subject: Wanted Python programmer to join team In-Reply-To: <7c1497d29c7146328545aa918522f986@seaexchmbx03.olympus.F5Net.com> References: <7c1497d29c7146328545aa918522f986@seaexchmbx03.olympus.F5Net.com> Message-ID: On Tue, May 17, 2016 at 6:34 AM, Dan Strohl via Python-list wrote: >> My team is getting more projects that it can handle so we are looking for >> Python programers to join. You will be given tasks to complete full or part of >> the project. >> >> Skype: piefektas >> >> Contact me now with short description about yourself, your skills and >> projects you have worked on. >> > > Sorry... nope. You lost me with using an email address like "netcrime4" and using "gmail" as your address. As a suggestion next time, try using a more professional address, such as one for a company, and one that does not suggest that the work might not be totally legit. > > (NOTE: I am NOT saying you aren?t legit, nor that the work is criminal, just that the impression you gave was less than professional, and even if I was looking, I would not consider contacting someone who presented that way.) > I'm not overly bothered by the use of GMail for a business address, although you're right that a registered domain looks better. (And "netcrime4" definitely doesn't fill me with confidence.) But more bothersome is the sending of job postings to a list/newsgroup that isn't meant to be used for that. Instead, try posting here: https://www.python.org/jobs/ If you can't convince the Job Board folks that you're legit, you probably won't be able to post. (Which I think is a good thing.) ChrisA From steve at pearwood.info Mon May 16 22:30:55 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 17 May 2016 12:30:55 +1000 Subject: OT: limit number of connections from browser to my server? References: Message-ID: <573a8261$0$1584$c3e8da3$5496439d@news.astraweb.com> On Tue, 17 May 2016 05:33 am, Grant Edwards wrote: > The problem is that > modern browsers won't wait and send requests serially over a single > connection. They try to "optimize" page load times by opening as many > connections as they cat right away and requesting everything in > parallel. (Even more OT for the OT discussion.) And the funny[1] thing about this is, for all these optimizations, web browsers in 2016 browsing the web in 2016 are generally *slower* than 1996 browsers browsing the web in 1996. And it has been like this for coming close to a decade now. At least two major reasons: websites are even more full of cruft than ever before, and the mobile web. http://www.antipope.org/charlie/blog-static/2008/05/why_your_internet_experience_i.html http://www.wired.com/insights/2014/11/the-web-is-getting-slower/ [1] In the sense of "I must laugh or else I will cry". -- Steven From steve at pearwood.info Mon May 16 22:37:39 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 17 May 2016 12:37:39 +1000 Subject: Wanted Python programmer to join team References: <7c1497d29c7146328545aa918522f986@seaexchmbx03.olympus.F5Net.com> Message-ID: <573a83f4$0$22140$c3e8da3$5496439d@news.astraweb.com> On Tue, 17 May 2016 09:07 am, Chris Angelico wrote: > I'm not overly bothered by the use of GMail for a business address, It's 2016. Using a gmail address for your business (unless you're a really small business, like a sole trader or something) is equivalent to a postal address of "Leave mail with the lady in the milk bar on the corner". It's not hard to run your own mail server. *I* can do it. At the very least, register a domain and tell Gmail to use that, and *pretend* you're running your own mail server. -- Steven From steve at pearwood.info Mon May 16 22:50:04 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 17 May 2016 12:50:04 +1000 Subject: OT: limit number of connections from browser to my server? References: Message-ID: <573a86de$0$1587$c3e8da3$5496439d@news.astraweb.com> On Tue, 17 May 2016 02:52 am, Chris Angelico wrote: > On Tue, May 17, 2016 at 2:34 AM, Rob Gaddi > wrote: >>> The solution might actually be to move all your static files >>> elsewhere. Slap 'em up onto github.io or something, and then the >>> browser is free to make all the parallel connections it likes; your >>> embedded device can just serve the stuff that actually varies >>> (presumably the main HTML file). I know that isn't what you asked for, >>> but it's something to consider :) >>> >>> ChrisA >> >> Oooof. Not to be rude, Chris, but your "software guy" is showing. >> Grant's got the right of it; if you're shipping a box with an RJ-45 and >> a webpage, and you want the customer to be able to always make it >> work, then it needs to be a self-contained entity. The belief that your >> external dependancies will always be there is why leftpad was able to >> break everything, and why Google just bricked a bunch of people's >> expensive Revolv Hubs. Schadenfreude is a beautiful emotion :-) "Yes, let's put a critical requirement of our business in the hands of a third party with absolutely *no* obligations to us, and no government oversight. What could *possibly* go wrong???" > I agree, but I also make no apology for suggesting the option of > getting someone else to do some of the work. In this case, it can be > rejected for the exact reason you cite (dependencies are a cost, and > in this case way too high a cost), and that's fine and correct. > Ultimately, if the job gets done, everything else is implementation > detail, with consequences - and I know a lot of people who'll > willingly sacrifice "reliability in the face of an internet connection > outage" in favour of "less than fifteen second response time". How can you not serve a web page over your LAN in 15s? I mean, you could *almost* do it by hand, copying the files onto a USB stick and walking them across the room in 15 seconds. Maybe 30. -- Steven From rosuav at gmail.com Mon May 16 22:56:07 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 17 May 2016 12:56:07 +1000 Subject: Wanted Python programmer to join team In-Reply-To: <573a83f4$0$22140$c3e8da3$5496439d@news.astraweb.com> References: <7c1497d29c7146328545aa918522f986@seaexchmbx03.olympus.F5Net.com> <573a83f4$0$22140$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, May 17, 2016 at 12:37 PM, Steven D'Aprano wrote: > On Tue, 17 May 2016 09:07 am, Chris Angelico wrote: > >> I'm not overly bothered by the use of GMail for a business address, > > It's 2016. Using a gmail address for your business (unless you're a really > small business, like a sole trader or something) is equivalent to a postal > address of "Leave mail with the lady in the milk bar on the corner". > > It's not hard to run your own mail server. *I* can do it. At the very least, > register a domain and tell Gmail to use that, and *pretend* you're running > your own mail server. And a lot of job postings do come from that sort of really small business, trying to expand a bit. Plus, some of them want some anonymity (why, I don't know, but there are plenty of jobs posted without too much in the way of company details), so maybe they do have a domain as well. Anyway, it's a small downside, but nothing compared to the other issues with the post. ChrisA From rosuav at gmail.com Mon May 16 23:02:07 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 17 May 2016 13:02:07 +1000 Subject: OT: limit number of connections from browser to my server? In-Reply-To: <573a86de$0$1587$c3e8da3$5496439d@news.astraweb.com> References: <573a86de$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, May 17, 2016 at 12:50 PM, Steven D'Aprano wrote: > How can you not serve a web page over your LAN in 15s? > > I mean, you could *almost* do it by hand, copying the files onto a USB stick > and walking them across the room in 15 seconds. Maybe 30. Don't forget to encrypt them by hand. ChrisA From steve+comp.lang.python at pearwood.info Tue May 17 01:05:06 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 17 May 2016 15:05:06 +1000 Subject: What should a decorator do if an attribute already exists? References: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> <1fb91cda-5583-1ab1-a14c-7bf3110daabc@fastwebnet.it> Message-ID: <573aa683$0$11102$c3e8da3@news.astraweb.com> On Monday 16 May 2016 18:14, Francesco Loffredo wrote: > On 10/05/2016 17:45, Steven D'Aprano wrote: >> I have a decorator that adds an attribute to the decorated function: [...] >> My question is, what should I do if the decorated function already has an >> instrument attribute? > From your example, it seems that you use your instrument only inside > your decorator. So I think "instrument" could be a "private variable". My example was misleading, sorry about that. Although the instrumentation is used inside the decorator, it is actually part of the public API for the function. So the user can do this: @decorate def myfunction(): ... # later myfunction.instrument.query() and see what data has been collected. So it is important that the name of the attribute be a public name. > What if you called your instrument "__instrument", taking advantage of > name mangling? Double underscore name-mangling only works inside classes. [...] > And what happens if you want to add another instrument, decorating the > target twice? If the names are the same, the second decorator will over-write the first decorator's instrumentation with its own. -- Steve From steve+comp.lang.python at pearwood.info Tue May 17 01:07:03 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 17 May 2016 15:07:03 +1000 Subject: What should a decorator do if an attribute already exists? References: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: <573aa6f8$0$11102$c3e8da3@news.astraweb.com> On Monday 16 May 2016 22:20, jmp wrote: > On 05/10/2016 05:45 PM, Steven D'Aprano wrote: >> I have a decorator that adds an attribute to the decorated function: > [snip] >> I think 5 is clearly wrong, 4 is too difficult, and 3 seems pointless. So I >> think either 1 or 2 is the right thing to do. >> >> Thoughts? > > It depends if the attribute "instrument" is part of the public interface. Yes it is. > If not, just find a different name unlikely to clash with an existing > one *and* raise an exception if it does ever happen anyway. > > If the attribute is part of the public interface but you re using the > code only internally then 1/ raise an exception, otherwise I don't know :o) > > It seems to me that over-writing silently (or not) may lead to bugs > difficult to spot. At this point, I think I will raise a warning. The user can use the warnings module to turn that warning into an error, if they want to, otherwise it's not necessarily an error. -- Steve From steve+comp.lang.python at pearwood.info Tue May 17 01:12:23 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 17 May 2016 15:12:23 +1000 Subject: What should a decorator do if an attribute already exists? References: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: <573aa837$0$11102$c3e8da3@news.astraweb.com> On Monday 16 May 2016 23:06, Kevin Conway wrote: >> I have a decorator that adds an attribute to the decorated function > > I might try to argue that this is not actually a decorator or, at least, > this is not a great decorator pattern for Python. Adding the attribute to > the function object implies you need to access it at some later point. If > so then your decorator has changed the public interface of the wrapped > item. Downstream code will depend on the new interface which is only > provided by the decorator so it becomes a hard requirement. This analysis is correct, apart from the value judgement "this is not a great decorator pattern". > IMO, decorators are supposed to be transparent. Removing a decorator should > not invalidate other code. Consider the three most obvious decorators in the language: property, classmethod and staticmethod. What happens if you remove them? > With your particular example, I believe you are actually implementing a > context manager and the more appropriate code would look like: > > with Instrument() as instrument: > # other code It is funny you say that, because my project offers a context manager interface as well. The context manager interface is based on this recipe: http://code.activestate.com/recipes/577896 but it adds a decorator interface so you can decorate a function, run the function as often as you want, then look at the timing statistics. -- Steve From rosuav at gmail.com Tue May 17 01:19:38 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 17 May 2016 15:19:38 +1000 Subject: What should a decorator do if an attribute already exists? In-Reply-To: <573aa683$0$11102$c3e8da3@news.astraweb.com> References: <5732021b$0$1587$c3e8da3$5496439d@news.astraweb.com> <1fb91cda-5583-1ab1-a14c-7bf3110daabc@fastwebnet.it> <573aa683$0$11102$c3e8da3@news.astraweb.com> Message-ID: On Tue, May 17, 2016 at 3:05 PM, Steven D'Aprano wrote: > Although the instrumentation is used inside the decorator, it is actually part > of the public API for the function. So the user can do this: > > > @decorate > def myfunction(): > ... > > > # later > myfunction.instrument.query() > > and see what data has been collected. > > So it is important that the name of the attribute be a public name. Sounds to me like a collision is largely not your problem, so I'd go with just tossing in a simple warning as a courtesy. There are plenty of situations where that courtesy wouldn't be given, and the caller would have to cope with the clash, so this would already be one up on that. Not worth going to a heap of hassle to deal with it, as most functions don't grow new attributes at a great rate. A quick check and a warning, and that's plenty. ChrisA From steve+comp.lang.python at pearwood.info Tue May 17 01:20:38 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 17 May 2016 15:20:38 +1000 Subject: Wanted Python programmer to join team References: <7c1497d29c7146328545aa918522f986@seaexchmbx03.olympus.F5Net.com> <573a83f4$0$22140$c3e8da3$5496439d@news.astraweb.com> Message-ID: <573aaa28$0$1512$c3e8da3$5496439d@news.astraweb.com> On Tuesday 17 May 2016 12:56, Chris Angelico wrote: > On Tue, May 17, 2016 at 12:37 PM, Steven D'Aprano > wrote: >> On Tue, 17 May 2016 09:07 am, Chris Angelico wrote: >> >>> I'm not overly bothered by the use of GMail for a business address, >> >> It's 2016. Using a gmail address for your business (unless you're a really >> small business, like a sole trader or something) is equivalent to a postal >> address of "Leave mail with the lady in the milk bar on the corner". >> >> It's not hard to run your own mail server. *I* can do it. At the very least, >> register a domain and tell Gmail to use that, and *pretend* you're running >> your own mail server. > > And a lot of job postings do come from that sort of really small > business, trying to expand a bit. Plus, some of them want some > anonymity (why, I don't know, but there are plenty of jobs posted > without too much in the way of company details) That probably means the job advert is coming from a recruiter. They don't want people to contact the company directly, and they want to hide the fact that they are a recruiter. Personally, I think that advertising a job position without saying who you are, what you do, and offering at least an indicative salary range, are *astonishingly* rude (to say nothing of counter-productive). If I see a job for (let's say) Blackwater[1], paying $900,000 a year, then I know that (1) I don't want to work for them, and (2) even if I did, I wouldn't be qualified; so I don't waste either my time or theirs applying. But when I see a job for some unnamed company with an unknown salary doing something often couched in the vaguest possible terms, I end up wasting everyone's time. This is part of the reason why it's not unusual for people end up applying for one or two hundred jobs before even getting a response, let alone an offer. [1] Or whatever they call themselves these days. -- Steve From steve+comp.lang.python at pearwood.info Tue May 17 01:22:13 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 17 May 2016 15:22:13 +1000 Subject: OT: limit number of connections from browser to my server? References: <573a86de$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: <573aaa87$0$1512$c3e8da3$5496439d@news.astraweb.com> On Tuesday 17 May 2016 13:02, Chris Angelico wrote: > On Tue, May 17, 2016 at 12:50 PM, Steven D'Aprano > wrote: >> How can you not serve a web page over your LAN in 15s? >> >> I mean, you could *almost* do it by hand, copying the files onto a USB stick >> and walking them across the room in 15 seconds. Maybe 30. > > Don't forget to encrypt them by hand. For that I'll use rot13, twice for extra security. -- Steve From gheskett at wdtv.com Tue May 17 01:26:22 2016 From: gheskett at wdtv.com (Gene Heskett) Date: Tue, 17 May 2016 01:26:22 -0400 Subject: OT: limit number of connections from browser to my server? In-Reply-To: References: <573a86de$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: <201605170126.22319.gheskett@wdtv.com> On Monday 16 May 2016 23:02:07 Chris Angelico wrote: > On Tue, May 17, 2016 at 12:50 PM, Steven D'Aprano wrote: > > How can you not serve a web page over your LAN in 15s? > > > > I mean, you could *almost* do it by hand, copying the files onto a > > USB stick and walking them across the room in 15 seconds. Maybe 30. > > Don't forget to encrypt them by hand. > > ChrisA Yes, rot13, twice for good measure... Cheers, Gene Heskett -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) Genes Web page From rosuav at gmail.com Tue May 17 01:29:39 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 17 May 2016 15:29:39 +1000 Subject: Wanted Python programmer to join team In-Reply-To: <573aaa28$0$1512$c3e8da3$5496439d@news.astraweb.com> References: <7c1497d29c7146328545aa918522f986@seaexchmbx03.olympus.F5Net.com> <573a83f4$0$22140$c3e8da3$5496439d@news.astraweb.com> <573aaa28$0$1512$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, May 17, 2016 at 3:20 PM, Steven D'Aprano wrote: >> And a lot of job postings do come from that sort of really small >> business, trying to expand a bit. Plus, some of them want some >> anonymity (why, I don't know, but there are plenty of jobs posted >> without too much in the way of company details) > > That probably means the job advert is coming from a recruiter. They don't want > people to contact the company directly, and they want to hide the fact that > they are a recruiter. Ehh, that's possible. Given that I seldom get any responses back from them, I can't tell whether they (a) are using a recruiter who hates my guts, (b) are using a recruiter who is utterly incompetent, (c) are doing the recruitment themselves, and are so utterly flat-out busy that they can't fire off a simple email to their applicants, or (d) are doing it themselves, and don't consider failed applicants worthy of an email. Could be any of the above, for all I can tell. Or (e), the job was posted from just outside the Bermuda Triangle. ChrisA From greg.ewing at canterbury.ac.nz Tue May 17 02:06:37 2016 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Tue, 17 May 2016 18:06:37 +1200 Subject: OT: limit number of connections from browser to my server? In-Reply-To: References: <5739ff37$0$1605$c3e8da3$5496439d@news.astraweb.com> Message-ID: Is there some way you can get more stuff into a single html page? For example, use inline css and image data instead of delivering them as separate files. -- Greg From marko at pacujo.net Tue May 17 02:18:17 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Tue, 17 May 2016 09:18:17 +0300 Subject: Wanted Python programmer to join team References: <7c1497d29c7146328545aa918522f986@seaexchmbx03.olympus.F5Net.com> <573a83f4$0$22140$c3e8da3$5496439d@news.astraweb.com> <573aaa28$0$1512$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87a8jpgqc6.fsf@elektro.pacujo.net> Steven D'Aprano : > Personally, I think that advertising a job position without saying who > you are, what you do, and offering at least an indicative salary > range, are *astonishingly* rude I don't believe they care. > (to say nothing of counter-productive). Maybe, maybe not. I bet the zebras on the savannah consider the lions astonishingly rude and their strategy counter-productive. The savannah would be a nicer place if the lions ate grass like everybody else. Marko From marko at pacujo.net Tue May 17 02:23:58 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Tue, 17 May 2016 09:23:58 +0300 Subject: OT: limit number of connections from browser to my server? References: <5739ff37$0$1605$c3e8da3$5496439d@news.astraweb.com> Message-ID: <874m9xgq2p.fsf@elektro.pacujo.net> Gregory Ewing : > Is there some way you can get more stuff into a single > html page? For example, use inline css and image data > instead of delivering them as separate files. Better yet, is there some way you could send less stuff? There are two points of note here. 1. The top ten sites are significantly lighter than the rest (worth noting if you want to be a top website). 2. While the web in general is steadily getting heavier, the top sites have turned the corner. Marko From tshortik at gmx.de Tue May 17 03:13:02 2016 From: tshortik at gmx.de (=?UTF-8?Q?Dirk_B=c3=a4chle?=) Date: Tue, 17 May 2016 09:13:02 +0200 Subject: Autotool - compile module for both Python 2 _and_ 3 In-Reply-To: <20160516090539.GA8327@arxnet.hu> References: <20160516090539.GA8327@arxnet.hu> Message-ID: <573AC47E.8060006@gmx.de> Hi Ervin, On 16.05.2016 11:05, Ervin Heged?s wrote: > Hi All, > > > there is a library, which written in C. I'ld like to use it from > Python - from Python 2 _and_ 3. > > I can make the autotools* files for Python 2 and Python 3, but > only exclusively. I can't make it for both in same time. > > [...] > > Is there any "best practice" to solve this problem? > > Practical advice or links are welcome! > here's my practical advice: have a look at SCons (http://www.scons.org). It handles variant builds (multiple packages/target) from the same source very well, even in parallel. Best regards, Dirk From dieter at handshake.de Tue May 17 03:26:11 2016 From: dieter at handshake.de (dieter) Date: Tue, 17 May 2016 09:26:11 +0200 Subject: Performance with and without the garbage collector References: <5737325a$0$1610$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87r3d1uovg.fsf@handshake.de> Steven D'Aprano writes: > ... > Is anyone able to demonstrate a replicable performance impact due to garbage > collection? I have had some experience with the performance impacts of garbage collection -- not completely replicable but rather frequently visible. Huge Zope instance (with around 2 GB heap usage; millions of objects); a badly designed (external) component that created thousands of temporary objects during searches. What we could observe: occasionally, our Zope got unresponsive for about 1 minute; a monitoring software typically told us that at least on of Zope's worker threads was in the external component mentioned above. The analysis finally revealed: creating and releasing thousands of short lived temporary objects frequently called for a garbage collection. The garbage collection prevented all other Python work while running -- and this caused outages in the order of a minute. From chris at simplistix.co.uk Tue May 17 03:31:44 2016 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 17 May 2016 08:31:44 +0100 Subject: testfixtures 4.10.0 Released! Message-ID: <573AC8E0.5000005@simplistix.co.uk> Hi All, I'm pleased to announce the release of testfixtures 4.10.0 featuring the following: - Fixed examples in documentation broken in 4.5.1. - Add RangeComparison for comparing against values that fall in a range. - Add MockPopen.set_default(). Thanks to Asaf Peleg for the RangeComparison implementation. The package is on PyPI and a full list of all the links to docs, issue trackers and the like can be found here: https://github.com/Simplistix/testfixtures Any questions, please do ask on the Testing in Python list or on the Simplistix open source mailing list... cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From steve+comp.lang.python at pearwood.info Tue May 17 03:39:17 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 17 May 2016 17:39:17 +1000 Subject: Wanted Python programmer to join team References: <7c1497d29c7146328545aa918522f986@seaexchmbx03.olympus.F5Net.com> <573a83f4$0$22140$c3e8da3$5496439d@news.astraweb.com> <573aaa28$0$1512$c3e8da3$5496439d@news.astraweb.com> <87a8jpgqc6.fsf@elektro.pacujo.net> Message-ID: <573acaa7$0$1607$c3e8da3$5496439d@news.astraweb.com> On Tuesday 17 May 2016 16:18, Marko Rauhamaa wrote: > Steven D'Aprano : >> Personally, I think that advertising a job position without saying who >> you are, what you do, and offering at least an indicative salary >> range, are *astonishingly* rude > > I don't believe they care. > >> (to say nothing of counter-productive). > > Maybe, maybe not. > > > I bet the zebras on the savannah consider the lions astonishingly rude > and their strategy counter-productive. The savannah would be a nicer > place if the lions ate grass like everybody else. A strange analogy. Employers and potential employees are not really in a predator/prey relationship. (Employers and *actual* employees sometimes are, but that's a sign of a really dysfunctional business culture.) The problem is that recruiter's best interests do not align neatly with either potential employees *or* employers. They're like real estate agents. The incentives for a recruiter is to find a barely acceptable hire as quickly as possible for the least amount of effort possible. There's no point in doing extra work to find the best new hire, if the employer is willing to take a so- so hire. Since the employer is only seeing potentials that the recruiter passes on, the employer has no way of telling what the pool of would-be employees is really like. I'm not saying that all recruiters are unscrupulous or are intentionally deceiving the other parties, but the incentives are such that: - recruiters will take a bit less care to choose the right employee for the job; - they'll take a bit less care worrying about attracting the right people, because their relationship with the parties is (on average) quite short; - they're more likely The worst part of this is the vicious circle aspect. The less care recruiters put into targeting their positions, the more they get inundated with poor quality applicants. This trains applicants to carpet bomb recruiters and employers (since they're trained to expect that all job ads are misleading, and also because the unemployment office requires them to apply to X positions a week, whether X suitable positions exist or not), which gives employers an incentive to use recruiters (rather than deal with the carpet bombing of applicants). -- Steve From steve+comp.lang.python at pearwood.info Tue May 17 03:42:04 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 17 May 2016 17:42:04 +1000 Subject: Wanted Python programmer to join team References: <7c1497d29c7146328545aa918522f986@seaexchmbx03.olympus.F5Net.com> <573a83f4$0$22140$c3e8da3$5496439d@news.astraweb.com> <573aaa28$0$1512$c3e8da3$5496439d@news.astraweb.com> <87a8jpgqc6.fsf@elektro.pacujo.net> <573acaa7$0$1607$c3e8da3$5496439d@news.astraweb.com> Message-ID: <573acb4c$0$1607$c3e8da3$5496439d@news.astraweb.com> On Tuesday 17 May 2016 17:39, Steven D'Aprano wrote: > - they're more likely Er, apparently they're not more likely to do anything specific, just more likely. -- Steve From steve+comp.lang.python at pearwood.info Tue May 17 03:50:53 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 17 May 2016 17:50:53 +1000 Subject: Quote of the day Message-ID: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> Overhead in the office today: "I don't have time to learn an existing library - much faster to make my own mistakes!" -- Steve From Radek1 at holych.org Tue May 17 04:12:32 2016 From: Radek1 at holych.org (=?UTF-8?Q?Radek_Hol=C3=BD?=) Date: Tue, 17 May 2016 10:12:32 +0200 Subject: Quote of the day In-Reply-To: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> Message-ID: 2016-05-17 9:50 GMT+02:00 Steven D'Aprano < steve+comp.lang.python at pearwood.info>: > Overhead in the office today: > > > "I don't have time to learn an existing library - much faster to make my > own > mistakes!" > > > > -- > Steve > > -- > https://mail.python.org/mailman/listinfo/python-list > *THUMBS UP* At least they are aware of that "own mistakes" part... Not like my employer... From marko at pacujo.net Tue May 17 04:27:32 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Tue, 17 May 2016 11:27:32 +0300 Subject: Wanted Python programmer to join team References: <7c1497d29c7146328545aa918522f986@seaexchmbx03.olympus.F5Net.com> <573a83f4$0$22140$c3e8da3$5496439d@news.astraweb.com> <573aaa28$0$1512$c3e8da3$5496439d@news.astraweb.com> <87a8jpgqc6.fsf@elektro.pacujo.net> <573acaa7$0$1607$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87y479f5sb.fsf@elektro.pacujo.net> Steven D'Aprano : > On Tuesday 17 May 2016 16:18, Marko Rauhamaa wrote: > >> Steven D'Aprano : >>> Personally, I think that advertising a job position without saying who >>> you are, what you do, and offering at least an indicative salary >>> range, are *astonishingly* rude >> I don't believe they care. >> >>> (to say nothing of counter-productive). >> Maybe, maybe not. >> >> I bet the zebras on the savannah consider the lions astonishingly rude >> and their strategy counter-productive. The savannah would be a nicer >> place if the lions ate grass like everybody else. > > A strange analogy. Employers and potential employees are not really in a > predator/prey relationship. The unsavory recruiters are the predators. > The problem is that recruiter's best interests do not align neatly > with either potential employees *or* employers. They're like real > estate agents. The incentives for a recruiter is to find a barely > acceptable hire as quickly as possible for the least amount of effort > possible. There's no point in doing extra work to find the best new > hire, if the employer is willing to take a so- so hire. Since the > employer is only seeing potentials that the recruiter passes on, the > employer has no way of telling what the pool of would-be employees is > really like. Correct. Also, they may genuinely not care about ethics of any sort. > I'm not saying that all recruiters are unscrupulous or are intentionally > deceiving the other parties, Nor am I. > but the incentives are such that: Different recruiters follow different strategies, methods and practices. Psychopathy stays in our gene pool over generations because it sometimes *is* a winning strategy. Marko From marko at pacujo.net Tue May 17 04:30:22 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Tue, 17 May 2016 11:30:22 +0300 Subject: Quote of the day References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87twhxf5nl.fsf@elektro.pacujo.net> Radek Hol? : > 2016-05-17 9:50 GMT+02:00 Steven D'Aprano < > steve+comp.lang.python at pearwood.info>: > >> Overhead in the office today: >> >> "I don't have time to learn an existing library - much faster to make >> my own mistakes!" > > *THUMBS UP* At least they are aware of that "own mistakes" part... Not > like my employer... Also: With a third party solution I don't need to fix the bugs. But with an in-house solution I at least *can* fix the bugs. The feeling of powerlessness can be crushing when you depend on a third-party component that is broken with no fix in sight. Marko From sivan at vitakka.co Tue May 17 04:53:16 2016 From: sivan at vitakka.co (Sivan Greenberg) Date: Tue, 17 May 2016 11:53:16 +0300 Subject: Quote of the day In-Reply-To: <87twhxf5nl.fsf@elektro.pacujo.net> References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> Message-ID: But isn't that counter wise to batteries included? :) On Tue, May 17, 2016 at 11:30 AM, Marko Rauhamaa wrote: > Radek Hol? : > > > 2016-05-17 9:50 GMT+02:00 Steven D'Aprano < > > steve+comp.lang.python at pearwood.info>: > > > >> Overhead in the office today: > >> > >> "I don't have time to learn an existing library - much faster to make > >> my own mistakes!" > > > > *THUMBS UP* At least they are aware of that "own mistakes" part... Not > > like my employer... > > Also: > > With a third party solution I don't need to fix the bugs. > > But with an in-house solution I at least *can* fix the bugs. > > The feeling of powerlessness can be crushing when you depend on a > third-party component that is broken with no fix in sight. > > > Marko > -- > https://mail.python.org/mailman/listinfo/python-list > -- Sivan Greenberg Co founder & CTO Vitakka Consulting From jeanmichel at sequans.com Tue May 17 04:58:36 2016 From: jeanmichel at sequans.com (jmp) Date: Tue, 17 May 2016 10:58:36 +0200 Subject: OT: limit number of connections from browser to my server? In-Reply-To: References: Message-ID: On 05/16/2016 09:31 PM, Grant Edwards wrote: > On 2016-05-16, jmp wrote: > >> Have you considered upgrading the device with a recent CPU ? Or is it >> completely out of the picture ? > > Not an option. We have to continue to support devices that are in the > field. The newer models that are coming out now run at 133MHz instead > of 44MHz, and page load times for https still aren't much better. > >> That being said, your first idea seems also a good lead, have your >> server refuse more than one connection. > > Just got to figure out how to try it out without wasting a lot of time > reverse-engineering the web server. Hence a Python prototype. :) > I don't have time to read the whole thread but if I got it right, the main CPU consuming part is the crypto. Why not drop the https part an support only http ? Is is a device that needs to be accessed in untrusted networks ? Sorry for asking the obvious :o) jm From paul.nospam at rudin.co.uk Tue May 17 05:54:31 2016 From: paul.nospam at rudin.co.uk (Paul Rudin) Date: Tue, 17 May 2016 10:54:31 +0100 Subject: Quote of the day References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> Message-ID: <87mvnp57s8.fsf@rudin.co.uk> Marko Rauhamaa writes: > Radek Hol? : > >> 2016-05-17 9:50 GMT+02:00 Steven D'Aprano < >> steve+comp.lang.python at pearwood.info>: >> >>> Overhead in the office today: >>> >>> "I don't have time to learn an existing library - much faster to make >>> my own mistakes!" >> >> *THUMBS UP* At least they are aware of that "own mistakes" part... Not >> like my employer... > > Also: > > With a third party solution I don't need to fix the bugs. > > But with an in-house solution I at least *can* fix the bugs. > > The feeling of powerlessness can be crushing when you depend on a > third-party component that is broken with no fix in sight. > > Presumably it depends on whether you have the source for the third party component... From cfkaran2 at gmail.com Tue May 17 06:39:16 2016 From: cfkaran2 at gmail.com (Cem Karan) Date: Tue, 17 May 2016 06:39:16 -0400 Subject: Quote of the day In-Reply-To: <87twhxf5nl.fsf@elektro.pacujo.net> References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> Message-ID: On May 17, 2016, at 4:30 AM, Marko Rauhamaa wrote: > Radek Hol? : > >> 2016-05-17 9:50 GMT+02:00 Steven D'Aprano < >> steve+comp.lang.python at pearwood.info>: >> >>> Overhead in the office today: >>> >>> "I don't have time to learn an existing library - much faster to make >>> my own mistakes!" >> >> *THUMBS UP* At least they are aware of that "own mistakes" part... Not >> like my employer... > > Also: > > With a third party solution I don't need to fix the bugs. > > But with an in-house solution I at least *can* fix the bugs. > > The feeling of powerlessness can be crushing when you depend on a > third-party component that is broken with no fix in sight. +1000 on this one. Just downloaded and used a library that came with unit tests, which all passed. When I started using it, I kept getting odd errors. Digging into it, I discovered they had commented out the bodies of some of the unit tests... glad it was open source, at least I *could* dig into the code and figure out what was going on :/ Thanks, Cem Karan From larry.martell at gmail.com Tue May 17 06:41:55 2016 From: larry.martell at gmail.com (Larry Martell) Date: Tue, 17 May 2016 06:41:55 -0400 Subject: Wanted Python programmer to join team In-Reply-To: References: <7c1497d29c7146328545aa918522f986@seaexchmbx03.olympus.F5Net.com> <573a83f4$0$22140$c3e8da3$5496439d@news.astraweb.com> <573aaa28$0$1512$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, May 17, 2016 at 1:29 AM, Chris Angelico wrote: > On Tue, May 17, 2016 at 3:20 PM, Steven D'Aprano > wrote: >>> And a lot of job postings do come from that sort of really small >>> business, trying to expand a bit. Plus, some of them want some >>> anonymity (why, I don't know, but there are plenty of jobs posted >>> without too much in the way of company details) >> >> That probably means the job advert is coming from a recruiter. They don't want >> people to contact the company directly, and they want to hide the fact that >> they are a recruiter. > > Ehh, that's possible. Given that I seldom get any responses back from > them, I can't tell whether they (a) are using a recruiter who hates my > guts, (b) are using a recruiter who is utterly incompetent, (c) are > doing the recruitment themselves, and are so utterly flat-out busy > that they can't fire off a simple email to their applicants, or (d) > are doing it themselves, and don't consider failed applicants worthy > of an email. Could be any of the above, for all I can tell. Or (e), > the job was posted from just outside the Bermuda Triangle. Recruiters have changed so much in my career. Back in the early 80's you would have to have an interview with the recruiter before they would even consider submitting you for a job. They would know your skill set and they would not present a job you were not qualified for. They returned phone calls and emails. Nowadays I get emails and calls for jobs with requirements that are nowhere on my resume (e.g. .NET or sales) and when I am qualified and interested and I reply, in most cases I get no response at all. From rosuav at gmail.com Tue May 17 06:48:29 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 17 May 2016 20:48:29 +1000 Subject: Quote of the day In-Reply-To: <87mvnp57s8.fsf@rudin.co.uk> References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> <87mvnp57s8.fsf@rudin.co.uk> Message-ID: On Tue, May 17, 2016 at 7:54 PM, Paul Rudin wrote: >> Also: >> >> With a third party solution I don't need to fix the bugs. >> >> But with an in-house solution I at least *can* fix the bugs. >> >> The feeling of powerlessness can be crushing when you depend on a >> third-party component that is broken with no fix in sight. >> >> > > Presumably it depends on whether you have the source for the third party > component... Yes and no. A student of mine asked me how hard it would be to use in Python a service that provided Java, .NET, and a couple of other SDKs, but not Python. Source is available for them, sure, but they're so massive and complicated that it's utterly impractical. (I had been hoping the SDKs were basically just offering a friendly API to an underlying HTTP-based service, but no.) So the options were (1) use Jython so the Java SDK became usable, or (2) fire off a subprocess that does the work and pipes it back to your app, or (3) spend about fifty years porting a gigantic lot of code to a new language. And if there'd been bugs in any of the code, well, options 1 and 2 mean my student (who knows Python but none of the SDK languages on offer) would be completely unable to fix it, source or no source - and with option 3, it'd make the port virtually impossible. Having the source available is great. It tells you which projects you DON'T want to touch. ChrisA fully aware that some of his projects will be in other people's "DON'T want to touch" boxes From airween at gmail.com Tue May 17 07:14:32 2016 From: airween at gmail.com (Ervin =?utf-8?Q?Heged=C3=BCs?=) Date: Tue, 17 May 2016 13:14:32 +0200 Subject: Autotool - compile module for both Python 2 _and_ 3 In-Reply-To: <573AC47E.8060006@gmx.de> References: <20160516090539.GA8327@arxnet.hu> <573AC47E.8060006@gmx.de> Message-ID: <20160517111432.GA4523@arxnet.hu> Hi Dirk, On Tue, May 17, 2016 at 09:13:02AM +0200, Dirk B?chle wrote: > Hi Ervin, > > On 16.05.2016 11:05, Ervin Heged?s wrote: > >Hi All, > > > > > >there is a library, which written in C. I'ld like to use it from > >Python - from Python 2 _and_ 3. > > > >I can make the autotools* files for Python 2 and Python 3, but > >only exclusively. I can't make it for both in same time. > > > >[...] > > > >Is there any "best practice" to solve this problem? > > > >Practical advice or links are welcome! > > > > here's my practical advice: have a look at SCons > (http://www.scons.org). It handles variant builds (multiple > packages/target) from the same source very well, even in parallel. many thanks - I'll check it out soon. Thanks, a. From rbistolfi at gmail.com Tue May 17 07:57:59 2016 From: rbistolfi at gmail.com (Rodrigo Bistolfi) Date: Tue, 17 May 2016 08:57:59 -0300 Subject: OT: limit number of connections from browser to my server? In-Reply-To: References: Message-ID: Long shot here: Create a JS framework for loading resources in a better way: 1. Load HTTP and your JS core. 2. Load the rest of the resources via JS (maybe using promises for chaining the requests one after the other) From marko at pacujo.net Tue May 17 08:21:34 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Tue, 17 May 2016 15:21:34 +0300 Subject: Quote of the day References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> <87mvnp57s8.fsf@rudin.co.uk> Message-ID: <87poskg9ip.fsf@elektro.pacujo.net> Paul Rudin : > Marko Rauhamaa writes: >> The feeling of powerlessness can be crushing when you depend on a >> third-party component that is broken with no fix in sight. > > Presumably it depends on whether you have the source for the third > party component... Just having such an experience. The linux kernel has a critical bug in a major distribution (who shall be left unnamed here) that has been fixed in a later kernel version. Thanks to linux being free software, I managed to pin down the root cause after more than a month of debugging. I sent a bug report to the linux vendor and attached a tiny patch. The vendor has graciously agreed to consider releasing an update in the summer (we are in the process of verifying the fix). The problem was first detected in December. A semi-reliable reproduction was discovered in early February. The root cause and proposed fix was identified mid-March. A vendor fix will likely come out by the end of June. That's a long time to be without a product to sell. Marko From mal at europython.eu Tue May 17 08:23:45 2016 From: mal at europython.eu (M.-A. Lemburg) Date: Tue, 17 May 2016 14:23:45 +0200 Subject: EuroPython 2016 Keynote: Nicholas Tollervey Message-ID: <573B0D51.8000502@europython.eu> We are pleased to introduce our first keynote speaker for EuroPython 2016: *** Nicholas Tollervey *** About Nicholas Tollervey ------------------------ Nicholas is a classically trained musician, philosophy graduate, teacher, author (for O'Reilly) and freelance programmer. He was a founding member of the London Python Code Dojo, created and organized PyCon UK?s education track and is a fellow of the Python Software Foundation. He also proposed, coordinated and contributed to the PSF?s partnership with the BBC in the creation of the micro:bit. He?s just like this biography: concise, honest and full of useful information. The Keynote: A Million Children (and MicroPython) ------------------------------------------------- The BBC micro:bit is a small programmable device for children. A million of them have been handed out to the UK?s 11 and 12 years olds. The Python Software Foundation was a partner in this moon-shot scheme and, thanks to the efforts of Damien George, MicroPython (a full reimplementation of Python 3) runs on the device: https://en.wikipedia.org/wiki/Micro_Bit http://micropython.org/ In his keynote, Nicholas will tell the story of the project, describe Python?s role in it and explain how the wider Python community can become involved. It may involve demonstrations, live coding and audience participation. With gravitational regards, -- EuroPython 2016 Team http://ep2016.europython.eu/ http://www.europython-society.org/ From grant.b.edwards at gmail.com Tue May 17 10:22:30 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 17 May 2016 14:22:30 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: <5739ff37$0$1605$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-17, Gregory Ewing wrote: > Is there some way you can get more stuff into a single > html page? For example, use inline css and image data > instead of delivering them as separate files. Yes. That's one option that's still on the table, and that's probably what the smart money is betting on: use server-side includes for css and javascript files. For image data, I'd need to add a server-side-include-and-base64-encode capability, but that shouldn't take more than a couple hours. Then the HTML editing starts... I've seen examples of including base64 image data in HTML tags, but don't know how to do it in CSS for things like backgrounds. That said, I've got two more things to try to see if I can convince browsers to behave: 1) Stall the connection after the TCP connection opens but before the SSL handshake. 2) Try listen(0) on my BSD-derived network stack to limit the number of TCP connections. IIRC, BSD behaves differently than Linux when the listen queue overflows. I doubt either one will help -- and even if they do, it'll probably all fall apart again when new browser versions come out. I'm also looking into SSL session resumption. My SSL stack appears to support it, but I can't tell if it's being used or not by browsers. It may be a moot point. All of the descriptions I've read about SSL session resumption say it's for low-overhead reopening of a previously closed connection. That's not what I need: I need to speed up opening of a new connection while the first connection is still open. -- Grant Edwards grant.b.edwards Yow! Mary Tyler Moore's at SEVENTH HUSBAND is wearing gmail.com my DACRON TANK TOP in a cheap hotel in HONOLULU! From grant.b.edwards at gmail.com Tue May 17 10:23:57 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 17 May 2016 14:23:57 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: <5739ff37$0$1605$c3e8da3$5496439d@news.astraweb.com> <874m9xgq2p.fsf@elektro.pacujo.net> Message-ID: On 2016-05-17, Marko Rauhamaa wrote: > Gregory Ewing : > >> Is there some way you can get more stuff into a single >> html page? For example, use inline css and image data >> instead of delivering them as separate files. > > Better yet, is there some way you could send less stuff? No. Somebody else decides what the pages look like and behaves. I don't get to toss out logos, background images, customized checkbox glyphs, etc. I get to make it work. :) -- Grant Edwards grant.b.edwards Yow! I have a very good at DENTAL PLAN. Thank you. gmail.com From paul.nospam at rudin.co.uk Tue May 17 10:27:08 2016 From: paul.nospam at rudin.co.uk (Paul Rudin) Date: Tue, 17 May 2016 15:27:08 +0100 Subject: Quote of the day References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> <87mvnp57s8.fsf@rudin.co.uk> <87poskg9ip.fsf@elektro.pacujo.net> Message-ID: <87inyc69qb.fsf@rudin.co.uk> Marko Rauhamaa writes: > Paul Rudin : > >> Marko Rauhamaa writes: >>> The feeling of powerlessness can be crushing when you depend on a >>> third-party component that is broken with no fix in sight. >> >> Presumably it depends on whether you have the source for the third >> party component... > > Just having such an experience. The linux kernel has a critical bug in a > major distribution (who shall be left unnamed here) that has been fixed > in a later kernel version. > > Thanks to linux being free software, I managed to pin down the root > cause after more than a month of debugging. I sent a bug report to the > linux vendor and attached a tiny patch. The vendor has graciously agreed > to consider releasing an update in the summer (we are in the process of > verifying the fix). > > The problem was first detected in December. A semi-reliable reproduction > was discovered in early February. The root cause and proposed fix was > identified mid-March. A vendor fix will likely come out by the end of > June. > > That's a long time to be without a product to sell. > But you do have the option of building a kernel incorporating your fix and using that. From grant.b.edwards at gmail.com Tue May 17 10:27:38 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 17 May 2016 14:27:38 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: Message-ID: On 2016-05-17, jmp wrote: > I don't have time to read the whole thread but if I got it right, > the main CPU consuming part is the crypto. Yep. > Why not drop the https part an support only http ? Product spec explicitly states HTTPS only. I'm told that is not open for discussion. The customer is a large, somewhat bureaucratic German corporation, and they generally mean it when they say something is non-negotiable. > Is is a device that needs to be accessed in untrusted networks? > Sorry for asking the obvious :o) -- Grant Edwards grant.b.edwards Yow! I'm EMOTIONAL at now because I have gmail.com MERCHANDISING CLOUT!! From grant.b.edwards at gmail.com Tue May 17 10:29:08 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 17 May 2016 14:29:08 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: Message-ID: On 2016-05-17, Rodrigo Bistolfi wrote: > Long shot here: Create a JS framework for loading resources in a better way: > > 1. Load HTTP and your JS core. > > 2. Load the rest of the resources via JS (maybe using promises for > chaining the requests one after the other) I thought about that. It's probably more work than going the "single-fetch" route via server-side includes, and probably more fragile. -- Grant Edwards grant.b.edwards Yow! An air of FRENCH FRIES at permeates my nostrils!! gmail.com From tshortik at gmx.de Tue May 17 10:51:54 2016 From: tshortik at gmx.de (=?UTF-8?Q?Dirk_B=c3=a4chle?=) Date: Tue, 17 May 2016 16:51:54 +0200 Subject: Design: Idiom for classes and methods that are customizable by the user? In-Reply-To: References: <5734ECDF.7070006@gmx.de> Message-ID: <573B300A.5080502@gmx.de> Hi Michael, and thanks a lot for chiming in on this topic. On 13.05.2016 22:33, Michael Selik wrote: > >> [...] >> > I share Greg's trepidation when I hear a phrase like that, but the general > idea of a registry of classes or functions and then picking the right one > based on string input is fine. > See my answer to Gregory, addressing his concerns. ;) > How would the API work for custom Taskmasters? It's not so great to require > that the user must explicitly ``add`` their derived class after defining > it. Perhaps that add function could be a decorator? > Our current API doesn't use decorators at all, since it's also aimed at people with no (or only some) knowledge of Python. Until a year or so ago, we have been backwards compatible down to Python 1.5.2...and then to Python 2.4 for some time. Only during the last two months we made a clear cut and are now seeing Python 2.7 as our floor to further develop from. So to some degree, decorators were technically not possible to support...and would have looked strange to the average user perhaps? We're now free to add them into the mix for our new Python 2.7.x/3.y codebase, but this would probably require an API discussion first. However, I'll keep the "decorator" idea in mind when I get back to putting "all the dots on the i" for the interface. Best regards, Dirk From zachary.ware+pylist at gmail.com Tue May 17 11:32:35 2016 From: zachary.ware+pylist at gmail.com (Zachary Ware) Date: Tue, 17 May 2016 10:32:35 -0500 Subject: How to create development Python environment on Linux. In-Reply-To: <29b59dce-2036-43f6-8166-7e3ad15e8e4c@googlegroups.com> References: <815aa265-45c4-40a4-860d-beb89cd9a78e@googlegroups.com> <29b59dce-2036-43f6-8166-7e3ad15e8e4c@googlegroups.com> Message-ID: On Mon, May 16, 2016 at 4:28 PM, wrote: > Thanks Zach, that's a big help. The only reason I want to get a Python 2.7 environment working first is because I'll be working on third party code and that's the platform it uses. For any new projects I would use Python 3. Fair enough :) > After considering your guidance I think what I will do is install virtualenv using apt-get and then use that to create a dev environment. Is it ok to run get-pip.py in a virtual environment? Sounds like a plan. get-pip.py should work fine in a venv, but first check to see if pip is already there, as Michael mentioned. > I won't worry about using the latest version of 2.7 for now, since it's only one or two third party open source projects I'll use 2.7 for and they don't need a more recent version. > > There are a couple of other things I'm not quite clear on, such as where it would be best to create my new virtual environment (I'm tempted to put it in /usr/local if that means it can be used by all user accounts on my machine), and how I can can control which Python environment is used by the various system and user programs that depend on them, but I expect I can find that information on the web, though I'll make another post here if I do get stuck. The thing about virtual environments is that you can create as many as you want wherever you want, and they're all independent of each other. All you need to do is have a way to consistently recreate the same environment, which is usually easiest to do with a 'requirements.txt' file with all of your dependencies' versions pinned. Then recreating the venv is as simple as `virtualenv --python=some_python /path/to/venv && /path/to/venv/bin/pip install -r /path/to/requirements.txt`. Then you can do your development wherever you want and install into somewhere in `/usr/local` when you deploy. For example, one project I work on creates a venv in `/usr/local/lib/project_name/`, then creates links to the entry point scripts (that actually live in `/usr/local/lib/project_name/bin/`) in `/usr/local/bin/`. As for controlling which environment is used: all system scripts/apps should specify some version-specific flavor of /usr/bin/python in their shebang--if they don't it's an OS bug and should be filed as such. For your stuff, the path of least headache is probably to create a venv for each one, and specify the path to the venv's python in shebang lines. Hope this helps, -- Zach From rgaddi at highlandtechnology.invalid Tue May 17 12:59:26 2016 From: rgaddi at highlandtechnology.invalid (Rob Gaddi) Date: Tue, 17 May 2016 16:59:26 -0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: <573a86de$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: Steven D'Aprano wrote: > On Tue, 17 May 2016 02:52 am, Chris Angelico wrote: > >> On Tue, May 17, 2016 at 2:34 AM, Rob Gaddi >> wrote: >>>> The solution might actually be to move all your static files >>>> elsewhere. Slap 'em up onto github.io or something, and then the >>>> browser is free to make all the parallel connections it likes; your >>>> embedded device can just serve the stuff that actually varies >>>> (presumably the main HTML file). I know that isn't what you asked for, >>>> but it's something to consider :) >>>> >>>> ChrisA >>> >>> Oooof. Not to be rude, Chris, but your "software guy" is showing. >>> Grant's got the right of it; if you're shipping a box with an RJ-45 and >>> a webpage, and you want the customer to be able to always make it >>> work, then it needs to be a self-contained entity. The belief that your >>> external dependancies will always be there is why leftpad was able to >>> break everything, and why Google just bricked a bunch of people's >>> expensive Revolv Hubs. > > Schadenfreude is a beautiful emotion :-) > > "Yes, let's put a critical requirement of our business in the hands of a > third party with absolutely *no* obligations to us, and no government > oversight. What could *possibly* go wrong???" > > >> I agree, but I also make no apology for suggesting the option of >> getting someone else to do some of the work. In this case, it can be >> rejected for the exact reason you cite (dependencies are a cost, and >> in this case way too high a cost), and that's fine and correct. >> Ultimately, if the job gets done, everything else is implementation >> detail, with consequences - and I know a lot of people who'll >> willingly sacrifice "reliability in the face of an internet connection >> outage" in favour of "less than fifteen second response time". > > How can you not serve a web page over your LAN in 15s? > > I mean, you could *almost* do it by hand, copying the files onto a USB stick > and walking them across the room in 15 seconds. Maybe 30. > Simple, because embedded web servers running on toy microprocessors are HARD. When you're trying to work with whatever browser the customer may have, you have no control over the number of simultaneous connections it will try to make to your box. If you don't allow those connections it can cause huge stalls by forcing the TCP layer to time out. If you do, in Grant's case, it forces you to perform tons of expensive public-key crypto on a 40 MHz processor (which, hmmm, external memory bus, ~40 MHz... Coldfire?). A lot of things you can take for granted on a compute monster (like a Chromebook or Atom based laptop) get much more complicated when you're resource constrained. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. From michael.selik at gmail.com Tue May 17 14:13:06 2016 From: michael.selik at gmail.com (Michael Selik) Date: Tue, 17 May 2016 18:13:06 +0000 Subject: Design: Idiom for classes and methods that are customizable by the user? In-Reply-To: <573B300A.5080502@gmx.de> References: <5734ECDF.7070006@gmx.de> <573B300A.5080502@gmx.de> Message-ID: On Tue, May 17, 2016 at 11:18 AM Dirk B?chle wrote: > > > It's not so great to require > > that the user must explicitly ``add`` their derived class after defining > > it. Perhaps that add function could be a decorator? > > Our current API doesn't use decorators at all, since it's also aimed at > people with no (or only some) knowledge of Python. I think you'll find that users of decorators do not need any special knowledge. The Flask framework (http://flask.pocoo.org/) is proof of that. > decorators...would have looked strange to the average user perhaps? > Many beginners are willing to mimic your example code without questioning what that "@" thingy is doing above the function or class. Compare that with trying to explain to a C++ user that in Python classes are actually objects and you can pass them to the ``add`` function to register them... that's going to take a while. We're now free to add them into the mix for our new Python 2.7.x/3.y > codebase, but this would probably require an API discussion > first. > An alternative would be creating a TaskmasterMeta metaclass that registers the Taskmaster subclass when it's defined. I'm a little reluctant to recommend that one. I find decorators to be more elegant. From grant.b.edwards at gmail.com Tue May 17 14:42:12 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 17 May 2016 18:42:12 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: <573a86de$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-17, Rob Gaddi wrote: > >> How can you not serve a web page over your LAN in 15s? >> >> I mean, you could *almost* do it by hand, copying the files onto a >> USB stick and walking them across the room in 15 seconds. Maybe 30. > > Simple, because embedded web servers running on toy microprocessors are > HARD. 40MHz with multiple MB of RAM is pretty high-end in my book. I've worked on projects where the clock speed was in KHz, the supply current was measrued in micro-Amps, and there were a few KB of RAM. Of course you don't try to do TCP and SSL on something like that. > When you're trying to work with whatever browser the customer may > have, you have no control over the number of simultaneous connections > it will try to make to your box. If you don't allow those connections > it can cause huge stalls by forcing the TCP layer to time out. If you > do, in Grant's case, it forces you to perform tons of expensive > public-key crypto on a 40 MHz processor (which, hmmm, external memory > bus, ~40 MHz... Coldfire?). The 40MHz one is a Samsung ARM7TDMI. There's a newer model with a 133MHz Cortex-M3. For most things it's 2-3 times faster than the ARM7, but the ARM7 has an I/D cache and the M3 doesn't. So there are few highly localized tasks where there's not a lot of difference. > A lot of things you can take for granted on a compute monster (like > a Chromebook or Atom based laptop) get much more complicated when > you're resource constrained. And that's what keeps me paid. :) -- Grant Edwards grant.b.edwards Yow! I know th'MAMBO!! at I have a TWO-TONE CHEMISTRY gmail.com SET!! From torriem at gmail.com Tue May 17 15:42:24 2016 From: torriem at gmail.com (Michael Torrie) Date: Tue, 17 May 2016 13:42:24 -0600 Subject: Quote of the day In-Reply-To: <87inyc69qb.fsf@rudin.co.uk> References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> <87mvnp57s8.fsf@rudin.co.uk> <87poskg9ip.fsf@elektro.pacujo.net> <87inyc69qb.fsf@rudin.co.uk> Message-ID: <7bc71dff-5240-d9f2-c1f8-8863afc24c89@gmail.com> On 05/17/2016 08:27 AM, Paul Rudin wrote: > Marko Rauhamaa writes: >> That's a long time to be without a product to sell. > > But you do have the option of building a kernel incorporating your fix > and using that. Sure as an individual end user that may be the best option. But not necessarily for a business. The cost of doing that could be prohibitive. Sometimes we forget just how costly open source software can be (really *all* software). They can either deal with lost revenue waiting, or they can budget a tremendous amount of money, time, and effort to support their own kernel which would entail doing updates, QA testing, etc. Letting the upstream vendor do all that (their core business after all) is often the least costly option. Though it sounds like they've already spent a lot of money doing QA to identify this bug. When I did IT professionally, our policy with regards to Linux was to stick with existing packages from a known set of (mostly) official channels and to discourage any installing of libraries and frameworks from source. Allowing packages to be installed from source was just a maintenance nightmare. RPM (or deb or whatever) brings at least a tiny bit of stability and consistency. From marko at pacujo.net Tue May 17 16:21:40 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Tue, 17 May 2016 23:21:40 +0300 Subject: Quote of the day References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> <87mvnp57s8.fsf@rudin.co.uk> <87poskg9ip.fsf@elektro.pacujo.net> <87inyc69qb.fsf@rudin.co.uk> <7bc71dff-5240-d9f2-c1f8-8863afc24c89@gmail.com> Message-ID: <871t5077vv.fsf@elektro.pacujo.net> Michael Torrie : > On 05/17/2016 08:27 AM, Paul Rudin wrote: >> Marko Rauhamaa writes: >>> That's a long time to be without a product to sell. >> >> But you do have the option of building a kernel incorporating your fix >> and using that. > > Sure as an individual end user that may be the best option. But not > necessarily for a business. Correct, the answer would be no. > The cost of doing that could be prohibitive. Really, the customer would simply refuse to do it. They are not in the business of building kernels. Also, they would immediately fall out of any kind of distro support if they improvised with their own kernel. > When I did IT professionally, our policy with regards to Linux was to > stick with existing packages from a known set of (mostly) official > channels and to discourage any installing of libraries and frameworks > from source. Allowing packages to be installed from source was just a > maintenance nightmare. RPM (or deb or whatever) brings at least a tiny > bit of stability and consistency. Exactly. Marko From nad at python.org Tue May 17 17:15:19 2016 From: nad at python.org (Ned Deily) Date: Tue, 17 May 2016 17:15:19 -0400 Subject: Python 3.6.0a1 is now available Message-ID: <28E1448B-F604-48EB-B4E4-0AE2528C8383@python.org> On behalf of the Python development community and the Python 3.6 release team, I'm happy to announce the availability of Python 3.6.0a1. 3.6.0a1 is the first of four planned alpha releases of Python 3.6, the next major release of Python. During the alpha phase, Python 3.6 remains under heavy development: additional features will be added and existing features may be modified or deleted. Please keep in mind that this is a preview release and its use is not recommended for production environments. You can find Python 3.6.0a1 here: https://www.python.org/downloads/release/python-360a1/ The next release of Python 3.6 will be 3.6.0a2, currently scheduled for 2016-06-13. Enjoy! --Ned -- Ned Deily nad at python.org -- [] From steve+comp.lang.python at pearwood.info Wed May 18 01:26:02 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Wed, 18 May 2016 15:26:02 +1000 Subject: Backwards-incompatible changes Message-ID: <573bfceb$0$11126$c3e8da3@news.astraweb.com> Every change breaks somebody's workflow. http://xkcd.com/1172/ -- Steve From silver0346 at gmail.com Wed May 18 04:32:08 2016 From: silver0346 at gmail.com (silver0346 at gmail.com) Date: Wed, 18 May 2016 01:32:08 -0700 (PDT) Subject: OrderedDict Message-ID: <7522e947-03d5-46e4-a74a-e5b312da47ad@googlegroups.com> Hi all, I have a understanding problem with return values from xmltodict. I have a xml file. Content: With code __f_name = '' with open(__f_name) as __fd: __doc = xmltodict.parse(__fd.read()) __doc I get OrderedDict([(u'profiles', OrderedDict([(u'profile', OrderedDict([(u'@id', u'visio02'), (u'@revision', u'2015051501'), (u'package', OrderedDict([(u'@package-id', u'0964-gpg4win')]))]))]))]) If I use __doc['profiles']['profile']['package'][0]['@package-id'] I get Traceback (most recent call last): File "", line 1, in KeyError: 0 If I change xml file like this: and run code from above the result is: OrderedDict([(u'profiles', OrderedDict([(u'profile', OrderedDict([(u'@id', u'visio02'), (u'@revision', u'2015051501'), (u'package', [OrderedDict([(u'@package-id', u'0964-gpg4win')]), OrderedDict([(u'@package-id', u'0965-gpg4win')])])]))]))]) No prints __doc['profiles']['profile']['package'][0]['@package-id']: u'0964-gpg4win' Can everybody explain this? Many thanks in advance From rosuav at gmail.com Wed May 18 04:54:04 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 18 May 2016 18:54:04 +1000 Subject: OrderedDict In-Reply-To: <7522e947-03d5-46e4-a74a-e5b312da47ad@googlegroups.com> References: <7522e947-03d5-46e4-a74a-e5b312da47ad@googlegroups.com> Message-ID: On Wed, May 18, 2016 at 6:32 PM, wrote: > Hi all, > > I have a understanding problem with return values from xmltodict. > > I have a xml file. Content: > > > > > > > > > > > > > > > > > No prints __doc['profiles']['profile']['package'][0]['@package-id']: > > u'0964-gpg4win' > > Can everybody explain this? This is one of the inherent problems of trying to convert XML (a structured document format) into a nested dictionary (a structured tree structure). In a dict, you can't have more than one value associated with a given key; in XML, and similar container-tag-based structures, you can - as in this example, where you have two package tags. When that gets converted into a dictionary, they get joined up into a list, which is why you can subscript it with the integer 0 to get the first one. If it's not made into a list, subscriping gives you the next level of dict straight away. If you're dealing with some files that have one package and some that have multiple, the easiest way would be something like this: packages = __doc['profiles']['profile']['package'] if isinstance(packages, dict): packages = [packages] for package in packages: # Do whatever you need with the package print(package['@package-id']) Incidentally, the double-leading-underscore names are not what I'd recommend; use simple names, and if you need them to be private, put this into a function (which will make them all function-local by default). The double leading underscore has special meaning inside a class, which you most likely don't intend. ChrisA From __peter__ at web.de Wed May 18 05:28:10 2016 From: __peter__ at web.de (Peter Otten) Date: Wed, 18 May 2016 11:28:10 +0200 Subject: OrderedDict References: <7522e947-03d5-46e4-a74a-e5b312da47ad@googlegroups.com> Message-ID: silver0346 at gmail.com wrote: > Hi all, > > I have a understanding problem with return values from xmltodict. > > I have a xml file. Content: > > > > > > > > > > With code > > __f_name = '' > with open(__f_name) as __fd: > __doc = xmltodict.parse(__fd.read()) > > __doc > > I get > > OrderedDict([(u'profiles', OrderedDict([(u'profile', OrderedDict([(u'@id', > u'visio02'), (u'@revision', u'2015051501'), (u'package', > OrderedDict([(u'@package-id', u'0964-gpg4win')]))]))]))]) > > If I use > > __doc['profiles']['profile']['package'][0]['@package-id'] > > I get > > Traceback (most recent call last): > File "", line 1, in > KeyError: 0 > > If I change xml file like this: > > > > > > > > > > and run code from above the result is: > > OrderedDict([(u'profiles', OrderedDict([(u'profile', OrderedDict([(u'@id', > u'visio02'), (u'@revision', u'2015051501'), (u'package', > [OrderedDict([(u'@package-id', u'0964-gpg4win')]), > OrderedDict([(u'@package-id', u'0965-gpg4win')])])]))]))]) > > No prints __doc['profiles']['profile']['package'][0]['@package-id']: > > u'0964-gpg4win' > > Can everybody explain this? Not everybody, but Chris and a few others can ;) > Many thanks in advance I don't see an official way to pass a custom dict type to the library, but if you are not afraid to change its source code the following patch will allow you to access the value of dictionaries with a single entry as d[0]: $ diff -u py2b_xmltodict/local/lib/python2.7/site-packages/xmltodict.py py2_xmltodict/local/lib/python2.7/site-packages/xmltodict.py --- py2b_xmltodict/local/lib/python2.7/site-packages/xmltodict.py 2016-05-18 11:18:44.000000000 +0200 +++ py2_xmltodict/local/lib/python2.7/site-packages/xmltodict.py 2016-05-18 11:11:13.417665697 +0200 @@ -35,6 +35,13 @@ __version__ = '0.10.1' __license__ = 'MIT' +_OrderedDict = OrderedDict +class OrderedDict(_OrderedDict): + def __getitem__(self, key): + if key == 0: + [result] = self.values() + return result + return _OrderedDict.__getitem__(self, key) class ParsingInterrupted(Exception): pass From rosuav at gmail.com Wed May 18 06:47:01 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 18 May 2016 20:47:01 +1000 Subject: OrderedDict In-Reply-To: References: <7522e947-03d5-46e4-a74a-e5b312da47ad@googlegroups.com> Message-ID: On Wed, May 18, 2016 at 7:28 PM, Peter Otten <__peter__ at web.de> wrote: > I don't see an official way to pass a custom dict type to the library, > but if you are not afraid to change its source code the following patch > will allow you to access the value of dictionaries with a single entry as d[0]: > > $ diff -u py2b_xmltodict/local/lib/python2.7/site-packages/xmltodict.py py2_xmltodict/local/lib/python2.7/site-packages/xmltodict.py > --- py2b_xmltodict/local/lib/python2.7/site-packages/xmltodict.py 2016-05-18 11:18:44.000000000 +0200 > +++ py2_xmltodict/local/lib/python2.7/site-packages/xmltodict.py 2016-05-18 11:11:13.417665697 +0200 > @@ -35,6 +35,13 @@ > __version__ = '0.10.1' > __license__ = 'MIT' > > +_OrderedDict = OrderedDict > +class OrderedDict(_OrderedDict): > + def __getitem__(self, key): > + if key == 0: > + [result] = self.values() > + return result > + return _OrderedDict.__getitem__(self, key) > > class ParsingInterrupted(Exception): > pass Easier than patching might be monkeypatching. class OrderedDict(OrderedDict): ... getitem code as above ... xmltodict.OrderedDict = OrderedDict Try it, see if it works. ChrisA From thomas at mlynarczyk-webdesign.de Wed May 18 08:05:56 2016 From: thomas at mlynarczyk-webdesign.de (Thomas Mlynarczyk) Date: Wed, 18 May 2016 14:05:56 +0200 Subject: Quote of the day In-Reply-To: References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> Message-ID: On 17/05/16 12:39, Cem Karan wrote: > Just downloaded and used a library that came with unit tests, which all passed. > [...] > I discovered they had commented out the bodies of some of the unit tests... Shouldn't the unit test framework have those "empty" tests reported as "todo"/"incomplete" or whatever? (I know that PHPUnit reports such tests as "passed" which I find utterly wrong.) Thomas -- Ce n'est pas parce qu'ils sont nombreux ? avoir tort qu'ils ont raison! (Coluche) From __peter__ at web.de Wed May 18 08:24:39 2016 From: __peter__ at web.de (Peter Otten) Date: Wed, 18 May 2016 14:24:39 +0200 Subject: OrderedDict References: <7522e947-03d5-46e4-a74a-e5b312da47ad@googlegroups.com> Message-ID: Chris Angelico wrote: > On Wed, May 18, 2016 at 7:28 PM, Peter Otten <__peter__ at web.de> wrote: >> I don't see an official way to pass a custom dict type to the library, >> but if you are not afraid to change its source code the following patch >> will allow you to access the value of dictionaries with a single entry as >> d[0]: >> >> $ diff -u py2b_xmltodict/local/lib/python2.7/site-packages/xmltodict.py >> py2_xmltodict/local/lib/python2.7/site-packages/xmltodict.py >> --- py2b_xmltodict/local/lib/python2.7/site-packages/xmltodict.py >> 2016-05-18 11:18:44.000000000 +0200 >> +++ py2_xmltodict/local/lib/python2.7/site-packages/xmltodict.py >> 2016-05-18 11:11:13.417665697 +0200 @@ -35,6 +35,13 @@ >> __version__ = '0.10.1' >> __license__ = 'MIT' >> >> +_OrderedDict = OrderedDict >> +class OrderedDict(_OrderedDict): >> + def __getitem__(self, key): >> + if key == 0: >> + [result] = self.values() >> + return result >> + return _OrderedDict.__getitem__(self, key) >> >> class ParsingInterrupted(Exception): >> pass > > Easier than patching might be monkeypatching. > > class OrderedDict(OrderedDict): > ... getitem code as above ... > xmltodict.OrderedDict = OrderedDict > > Try it, see if it works. It turns out I was wrong on (at least) two accounts: - xmltodict does offer a way to specify the dict type - the proposed dict implementation will not solve the OP's problem Here is an improved fix which should work: $ cat sample.xml $ cat sample2.xml $ cat demo.py import collections import sys import xmltodict class MyOrderedDict(collections.OrderedDict): def __getitem__(self, key): if key == 0 and len(self) == 1: return self return super(MyOrderedDict, self).__getitem__(key) def main(): filename = sys.argv[1] with open(filename) as f: doc = xmltodict.parse(f.read(), dict_constructor=MyOrderedDict) print "doc:\n{}\n".format(doc) print "package-id: {}".format( doc['profiles']['profile']['package'][0]['@package-id']) if __name__ == "__main__": main() $ python demo.py sample.xml doc: MyOrderedDict([(u'profiles', MyOrderedDict([(u'profile', MyOrderedDict([(u'@id', u'visio02'), (u'@revision', u'2015051501'), (u'package', MyOrderedDict([(u'@package-id', u'0964-gpg4win')]))]))]))]) package-id: 0964-gpg4win $ python demo.py sample2.xml doc: MyOrderedDict([(u'profiles', MyOrderedDict([(u'profile', MyOrderedDict([(u'@id', u'visio02'), (u'@revision', u'2015051501'), (u'package', [MyOrderedDict([(u'@package-id', u'0964-gpg4win')]), MyOrderedDict([(u'@package-id', u'0965-gpg4win')])])]))]))]) package-id: 0964-gpg4win From rosuav at gmail.com Wed May 18 08:32:21 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 18 May 2016 22:32:21 +1000 Subject: Quote of the day In-Reply-To: References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> Message-ID: On Wed, May 18, 2016 at 10:05 PM, Thomas Mlynarczyk wrote: > On 17/05/16 12:39, Cem Karan wrote: >> Just downloaded and used a library that came with unit tests, which all passed. >> [...] >> I discovered they had commented out the bodies of some of the unit > tests... > > Shouldn't the unit test framework have those "empty" tests reported as > "todo"/"incomplete" or whatever? (I know that PHPUnit reports such tests > as "passed" which I find utterly wrong.) In Python, the unittest framework allows you to 'skip' tests for any reason. That would be the best. ChrisA From grant.b.edwards at gmail.com Wed May 18 09:58:06 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Wed, 18 May 2016 13:58:06 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: <573a86de$0$1587$c3e8da3$5496439d@news.astraweb.com> <9e6njbpf40s1hi41pgje70gl0m2bk5j12n@4ax.com> Message-ID: On 2016-05-17, Dennis Lee Bieber wrote: > On Tue, 17 May 2016 18:42:12 +0000 (UTC), Grant Edwards > declaimed the following: > > >>40MHz with multiple MB of RAM is pretty high-end in my book. I've > > How about a 68040 running at 30MHz (when not running in a dual system > with a 20MHz board) The '040 even supports hardware floating point and virtual memory (and it has I/D cache). That's livin' large! We've got brand-new products coming out later this year without all that fancy stuff. -- Grant Edwards grant.b.edwards Yow! I haven't been married at in over six years, but we gmail.com had sexual counseling every day from Oral Roberts!! From ned at nedbatchelder.com Wed May 18 11:21:04 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 18 May 2016 08:21:04 -0700 (PDT) Subject: Quote of the day In-Reply-To: References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> Message-ID: <8e92e249-c46e-4b92-a93b-778879461273@googlegroups.com> On Wednesday, May 18, 2016 at 8:06:11 AM UTC-4, Thomas Mlynarczyk wrote: > On 17/05/16 12:39, Cem Karan wrote: > > Just downloaded and used a library that came with unit tests, which all passed. > > [...] > > I discovered they had commented out the bodies of some of the unit > tests... > > Shouldn't the unit test framework have those "empty" tests reported as > "todo"/"incomplete" or whatever? (I know that PHPUnit reports such tests > as "passed" which I find utterly wrong.) The xUnit pattern is that if a test runs without an error or a failed assertion, then the test passes. An empty test function that does nothing will meet this criterion, so it passes. Ideally, an empty test wouldn't be a success, but I'm not sure how the test runner could determine that it was empty. I guess it could introspect the test function to see if it had any real code in it, but I don't know of a test runner that does that. --Ned. From thomas at mlynarczyk-webdesign.de Wed May 18 11:35:49 2016 From: thomas at mlynarczyk-webdesign.de (Thomas Mlynarczyk) Date: Wed, 18 May 2016 17:35:49 +0200 Subject: Quote of the day In-Reply-To: <8e92e249-c46e-4b92-a93b-778879461273@googlegroups.com> References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> <8e92e249-c46e-4b92-a93b-778879461273@googlegroups.com> Message-ID: On 18/05/16 17:21, Ned Batchelder wrote: > Ideally, an empty test wouldn't be a success, but I'm not sure how > the test runner could determine that it was empty. I guess it could > introspect the test function to see if it had any real code in it, > but I don't know of a test runner that does that. Simple: a function which does not produce at least one "failure" or "pass" does not test anything. No need to introspect the code. Just check if the total score of failures and passes has changed after the function was run. Thomas -- Ce n'est pas parce qu'ils sont nombreux ? avoir tort qu'ils ont raison! (Coluche) From ned at nedbatchelder.com Wed May 18 11:47:06 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 18 May 2016 08:47:06 -0700 (PDT) Subject: Quote of the day In-Reply-To: References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> <8e92e249-c46e-4b92-a93b-778879461273@googlegroups.com> Message-ID: On Wednesday, May 18, 2016 at 11:36:03 AM UTC-4, Thomas Mlynarczyk wrote: > On 18/05/16 17:21, Ned Batchelder wrote: > > Ideally, an empty test wouldn't be a success, but I'm not sure how > > the test runner could determine that it was empty. I guess it could > > introspect the test function to see if it had any real code in it, > > but I don't know of a test runner that does that. > > Simple: a function which does not produce at least one "failure" or > "pass" does not test anything. No need to introspect the code. Just > check if the total score of failures and passes has changed after the > function was run. For test frameworks that use explicit assertion methods (unittest has self.assertEqual, for example), that could work. I'm not sure whether py.test and the other "bare assert" frameworks have the instrumentation to make that possible. --Ned. From steve at pearwood.info Wed May 18 11:47:33 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 19 May 2016 01:47:33 +1000 Subject: Extract the middle N chars of a string Message-ID: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> Extracting the first N or last N characters of a string is easy with slicing: s[:N] # first N s[-N:] # last N Getting the middle N seems like it ought to be easy: s[N//2:-N//2] but that is wrong. It's not even the right length! py> s = 'aardvark' py> s[5//2:-5//2] 'rdv' So after spending a ridiculous amount of time on what seemed like it ought to be a trivial function, and an embarrassingly large number of off-by-one and off-by-I-don't-even errors, I eventually came up with this: def mid(string, n): """Return middle n chars of string.""" L = len(string) if n <= 0: return '' elif n < L: Lr = L % 2 a, ar = divmod(L-n, 2) b, br = divmod(L+n, 2) a += Lr*ar b += Lr*br string = string[a:b] return string which works for me: # string with odd number of characters py> for i in range(1, 8): ... print mid('abcdefg', i) ... d de cde cdef bcdef bcdefg abcdefg # string with even number of characters py> for i in range(1, 7): ... print mid('abcdef', i) ... c cd bcd bcde abcde abcdef Is this the simplest way to get the middle N characters? -- Steven From ethan at stoneleaf.us Wed May 18 12:05:31 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 18 May 2016 09:05:31 -0700 Subject: Quote of the day In-Reply-To: References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> <8e92e249-c46e-4b92-a93b-778879461273@googlegroups.com> Message-ID: <573C92CB.9070502@stoneleaf.us> On 05/18/2016 08:35 AM, Thomas Mlynarczyk wrote: > On 18/05/16 17:21, Ned Batchelder wrote: >> Ideally, an empty test wouldn't be a success, but I'm not sure how >> the test runner could determine that it was empty. I guess it could >> introspect the test function to see if it had any real code in it, >> but I don't know of a test runner that does that. > > Simple: a function which does not produce at least one "failure" or > "pass" does not test anything. No need to introspect the code. Just > check if the total score of failures and passes has changed after the > function was run. Not so simple: I have tests that do nothing besides build objects. If building the objects raises no errors the test passed. Although, for the benefit of empty tests not passing I could add a do-nothing assert: self.assertTrue(created_obj) (it's a do-nothing because if the object wasn't created the test would have already failed). -- ~Ethan~ From steve at pearwood.info Wed May 18 12:11:09 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 19 May 2016 02:11:09 +1000 Subject: setrecursionlimit Message-ID: <573c941f$0$22141$c3e8da3$5496439d@news.astraweb.com> The documentation for setrecursion limit warns against setting the limit too high: [quote] The highest possible limit is platform-dependent. A user may need to set the limit higher when they have a program that requires deep recursion and a platform that supports a higher limit. This should be done with care, because a too-high limit can lead to a crash. [end quote] https://docs.python.org/3/library/sys.html#sys.setrecursionlimit Indeed, if you set the recursion limit too high, you can smash the memory heap and get a segfault. How exactly does that work? Why doesn't setrecursionlimit() raise an exception when you try to set it too high? For example: sys.setrecursionlimit(2000000000) succeeds on my system, even though that's a ludicrously high number. (It is more than half the number of bytes of memory my computer has.) So why can't Python tell if I'm setting the limit too high? (I'm assuming that if it could, it would.) -- Steven From ned at nedbatchelder.com Wed May 18 12:19:25 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 18 May 2016 09:19:25 -0700 (PDT) Subject: setrecursionlimit In-Reply-To: <573c941f$0$22141$c3e8da3$5496439d@news.astraweb.com> References: <573c941f$0$22141$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Wednesday, May 18, 2016 at 12:11:25 PM UTC-4, Steven D'Aprano wrote: > The documentation for setrecursion limit warns against setting the limit too > high: > > [quote] > The highest possible limit is platform-dependent. A user may need to > set the limit higher when they have a program that requires deep > recursion and a platform that supports a higher limit. This should > be done with care, because a too-high limit can lead to a crash. > [end quote] > > https://docs.python.org/3/library/sys.html#sys.setrecursionlimit > > Indeed, if you set the recursion limit too high, you can smash the memory > heap and get a segfault. How exactly does that work? > > Why doesn't setrecursionlimit() raise an exception when you try to set it > too high? For example: > > sys.setrecursionlimit(2000000000) > > succeeds on my system, even though that's a ludicrously high number. (It is > more than half the number of bytes of memory my computer has.) > > > So why can't Python tell if I'm setting the limit too high? > > (I'm assuming that if it could, it would.) I believe the issue here is that Python recursion results in the C stack growing. Each Python function call creates a number of C function calls, which grows the C stack. The crash you see is the C stack overflowing. Is there a way to know how large the C stack can grow, and how much it will grow for each Python function call? That sounds complicated to get right. --Ned. From rgaddi at highlandtechnology.invalid Wed May 18 12:29:56 2016 From: rgaddi at highlandtechnology.invalid (Rob Gaddi) Date: Wed, 18 May 2016 16:29:56 -0000 (UTC) Subject: setrecursionlimit References: <573c941f$0$22141$c3e8da3$5496439d@news.astraweb.com> Message-ID: Ned Batchelder wrote: > On Wednesday, May 18, 2016 at 12:11:25 PM UTC-4, Steven D'Aprano wrote: >> The documentation for setrecursion limit warns against setting the limit too >> high: >> >> [quote] >> The highest possible limit is platform-dependent. A user may need to >> set the limit higher when they have a program that requires deep >> recursion and a platform that supports a higher limit. This should >> be done with care, because a too-high limit can lead to a crash. >> [end quote] >> >> https://docs.python.org/3/library/sys.html#sys.setrecursionlimit >> >> Indeed, if you set the recursion limit too high, you can smash the memory >> heap and get a segfault. How exactly does that work? >> >> Why doesn't setrecursionlimit() raise an exception when you try to set it >> too high? For example: >> >> sys.setrecursionlimit(2000000000) >> >> succeeds on my system, even though that's a ludicrously high number. (It is >> more than half the number of bytes of memory my computer has.) >> >> >> So why can't Python tell if I'm setting the limit too high? >> >> (I'm assuming that if it could, it would.) > > I believe the issue here is that Python recursion results in the C stack > growing. Each Python function call creates a number of C function calls, > which grows the C stack. The crash you see is the C stack overflowing. > > Is there a way to know how large the C stack can grow, and how much it > will grow for each Python function call? That sounds complicated to get > right. > > --Ned. It's probably trivial to look at a number and say "Yeah, no, that's CLEARLY too high." based on the minimum number of bytes a stack frame can require. Guaranteeing that some number lower than that is safe is almost certainly impossible. So you'd get an exception for truly stupid numbers, but a lack of exception is no guarantee of safety. Which is worth what it's worth, I guess. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. From ian.g.kelly at gmail.com Wed May 18 12:34:12 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 18 May 2016 10:34:12 -0600 Subject: Extract the middle N chars of a string In-Reply-To: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> References: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Wed, May 18, 2016 at 9:47 AM, Steven D'Aprano wrote: > Extracting the first N or last N characters of a string is easy with > slicing: > > s[:N] # first N > s[-N:] # last N > > Getting the middle N seems like it ought to be easy: > > s[N//2:-N//2] > > but that is wrong. It's not even the right length! > > py> s = 'aardvark' > py> s[5//2:-5//2] > 'rdv' > > > So after spending a ridiculous amount of time on what seemed like it ought > to be a trivial function, and an embarrassingly large number of off-by-one > and off-by-I-don't-even errors, I eventually came up with this: > > def mid(string, n): > """Return middle n chars of string.""" > L = len(string) > if n <= 0: > return '' > elif n < L: > Lr = L % 2 > a, ar = divmod(L-n, 2) > b, br = divmod(L+n, 2) > a += Lr*ar > b += Lr*br > string = string[a:b] > return string > > > which works for me: > > > # string with odd number of characters > py> for i in range(1, 8): > ... print mid('abcdefg', i) > ... > d > de > cde > cdef > bcdef > bcdefg > abcdefg > # string with even number of characters > py> for i in range(1, 7): > ... print mid('abcdef', i) > ... > c > cd > bcd > bcde > abcde > abcdef > > > > Is this the simplest way to get the middle N characters? There's an ambiguity in the problem description when N and the length of the string have different parity, but how about: def mid(string, n): L = len(string) a = (L - n) // 2 return string[a:a+n] py> for i in range(1, 8): ... print(mid('abcdefg', i)) ... d cd cde bcde bcdef abcdef abcdefg From __peter__ at web.de Wed May 18 12:41:21 2016 From: __peter__ at web.de (Peter Otten) Date: Wed, 18 May 2016 18:41:21 +0200 Subject: Extract the middle N chars of a string References: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: Steven D'Aprano wrote: > Extracting the first N or last N characters of a string is easy with > slicing: > > s[:N] # first N > s[-N:] # last N > > Getting the middle N seems like it ought to be easy: > > s[N//2:-N//2] > > but that is wrong. It's not even the right length! > > py> s = 'aardvark' > py> s[5//2:-5//2] > 'rdv' > > > So after spending a ridiculous amount of time on what seemed like it ought > to be a trivial function, and an embarrassingly large number of off-by-one > and off-by-I-don't-even errors, I eventually came up with this: > > def mid(string, n): > """Return middle n chars of string.""" > L = len(string) > if n <= 0: > return '' > elif n < L: > Lr = L % 2 > a, ar = divmod(L-n, 2) > b, br = divmod(L+n, 2) > a += Lr*ar > b += Lr*br > string = string[a:b] > return string > > > which works for me: > > > # string with odd number of characters > py> for i in range(1, 8): > ... print mid('abcdefg', i) > ... > d > de > cde > cdef > bcdef > bcdefg > abcdefg > # string with even number of characters > py> for i in range(1, 7): > ... print mid('abcdef', i) > ... > c > cd > bcd > bcde > abcde > abcdef > > > > Is this the simplest way to get the middle N characters? >>> def mid(s, n): ... shave = len(s) - n ... if shave > 0: ... shave //= 2 ... s = s[shave:shave+n] ... return s ... >>> def show(s): ... for i in range(len(s)+1): ... print(i, repr(s), "-->", repr(mid(s, i))) ... >>> show("abcdefg") 0 'abcdefg' --> '' 1 'abcdefg' --> 'd' 2 'abcdefg' --> 'cd' 3 'abcdefg' --> 'cde' 4 'abcdefg' --> 'bcde' 5 'abcdefg' --> 'bcdef' 6 'abcdefg' --> 'abcdef' 7 'abcdefg' --> 'abcdefg' >>> show("abcdef") 0 'abcdef' --> '' 1 'abcdef' --> 'c' 2 'abcdef' --> 'cd' 3 'abcdef' --> 'bcd' 4 'abcdef' --> 'bcde' 5 'abcdef' --> 'abcde' 6 'abcdef' --> 'abcdef' Not exactly the same results as your implementation though. From steve at pearwood.info Wed May 18 12:50:39 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 19 May 2016 02:50:39 +1000 Subject: Quote of the day References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> <8e92e249-c46e-4b92-a93b-778879461273@googlegroups.com> <573C92CB.9070502@stoneleaf.us> Message-ID: <573c9d61$0$1603$c3e8da3$5496439d@news.astraweb.com> On Thu, 19 May 2016 02:05 am, Ethan Furman wrote: > On 05/18/2016 08:35 AM, Thomas Mlynarczyk wrote: >> On 18/05/16 17:21, Ned Batchelder wrote: > >>> Ideally, an empty test wouldn't be a success, but I'm not sure how >>> the test runner could determine that it was empty. I guess it could >>> introspect the test function to see if it had any real code in it, >>> but I don't know of a test runner that does that. >> >> Simple: a function which does not produce at least one "failure" or >> "pass" does not test anything. No need to introspect the code. Just >> check if the total score of failures and passes has changed after the >> function was run. I think you have misunderstood how unittest currently works. A do-nothing test already counts as a pass. Here's a dumb test which is pointless, followed by an even dumber one that does literally nothing: [steve at ando ~]$ cat dumbtest.py import unittest class MyTest(unittest.TestCase): def test_something(self): self.assertEqual(100, 100.0) def test_nothing(self): pass And now watch as both the pointless and the do-nothing tests count as passed: [steve at ando ~]$ python -m unittest --verbose dumbtest test_nothing (dumbtest.MyTest) ... ok test_something (dumbtest.MyTest) ... ok ---------------------------------------------------------------------- Ran 2 tests in 0.001s OK So to start with, for your solution to be workable, we'd have to change the way unittest decides what is a success and what isn't. > Not so simple: I have tests that do nothing besides build objects. If > building the objects raises no errors the test passed. It wouldn't be hard to add a "success" method to unittest, so that after building the object you just call self.success() to flag it as passing. But now the obvious way to have fake unittests is: def test_nothing(self): self.success() The problem here is not a technical problem. It is a cultural or human problem: somebody, due to malice, incompetence, overwork, ignorance or stupidity, wrote a fake test that didn't actually test anything. Maybe their intentions were good, and they meant for it to do something and it just got forgotten... or maybe they deliberately thought that they could satisfy the letter of the requirement "must have unit tests" without putting in the hard work to satisfy the spirit of it. Either way, until the testing framework contains enough artificial intelligence to actually reason about whether the test is *useful* or not, there's no technological way to solve this problem. You need a person[1] intelligent enough to make a judgement "wait a minute, this code doesn't test anything useful". And that's a hard problem. Even human beings do poorly at that. The idea that the test framework could solve it is naive. > Although, for the benefit of empty tests not passing I could add a > do-nothing assert: > > self.assertTrue(created_obj) > > (it's a do-nothing because if the object wasn't created the test would > have already failed). It wouldn't have failed. It would have raised an exception, which is different. (Curse you English, we need more words for describing kinds of failure!!!) unittest supports four different test results: - pass - fail - error (raise an exception) - skip which print as . F E S respectively. The tests you're describing will print as E rather than F, which is better than a failure. A test failure is code that silently does the wrong thing: "I find it amusing when novice programmers believe their main job is preventing programs from crashing. ... More experienced programmers realize that correct code is great, code that crashes could use improvement, but incorrect code that doesn?t crash is a horrible nightmare." -- Chris Smith while an E means that your code will cleverly raise an exception instead of doing the wrong thing :-) [1] Human or machine person, I don't care. -- Steven From __peter__ at web.de Wed May 18 12:54:56 2016 From: __peter__ at web.de (Peter Otten) Date: Wed, 18 May 2016 18:54:56 +0200 Subject: Extract the middle N chars of a string References: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: Peter Otten wrote: >>>> def mid(s, n): > ... shave = len(s) - n > ... if shave > 0: shave += len(s) % 2 > ... shave //= 2 > ... s = s[shave:shave+n] > ... return s > Not exactly the same results as your implementation though. The extra line should fix that, but it looks, err -- odd. From ckaynor at zindagigames.com Wed May 18 12:56:48 2016 From: ckaynor at zindagigames.com (Chris Kaynor) Date: Wed, 18 May 2016 09:56:48 -0700 Subject: setrecursionlimit In-Reply-To: References: <573c941f$0$22141$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Wed, May 18, 2016 at 9:19 AM, Ned Batchelder wrote: > I believe the issue here is that Python recursion results in the C stack > growing. Each Python function call creates a number of C function calls, > which grows the C stack. The crash you see is the C stack overflowing. > > Is there a way to know how large the C stack can grow, and how much it > will grow for each Python function call? That sounds complicated to get > right. > I'm fairly sure that it is, in fact, basically impossible to get right. Some Python calls will use more memory on the C stack than others. They may call more C functions internally, or C functions that require more stack space. In the most extreme example, you could have a single Python call crash (for example, having a super deeply recursive call as a single Python call), or you could theoretically have extremely deep Python recursion without a problem (presuming optimizations that do not exist in CPython for a number of, generally good, reasons). Even in more typical cases, I believe a Python call with keyword arguments requires more stack than one with only positional arguments, which may require more than one with no arguments (depending on which CPython APIs were used). Additionally, even within the standard library, some of the native calls will require more stack (think OS calls) than most of the basic math functions and simple operators (like int add). The root of the issue is that, much of the time, C function arguments are allocated on the stack, if they exceed certain limits based on the calling convention used. 32-bit Windows only allocates the "this" pointer as a register, all other arguments are stack [1]. 64-bit Windows allocates up-to 4 word arguments as registers, and the rest on the stack [2]. I do not know what the Linux conventions are. Naturally, these rules may vary based on the compiler, for any functions the compiler knows is being compiled by the compiler - the rules listed for the OS are only required for OS calls, however most compilers will follow them for ease. Additionally, any local variables that do not fit in registers will be offloaded to the stack, and sometimes they will be offloaded even if they do fit, at the compiler's decision, especially if function calls are made. All of this means that, as Ned mentioned, it is very complicated to figure out a *recursion* depth that will cause a stack overflow. Generally, there will be an OS method to determine the stack size in *bytes*, however (and often, the application can control this when creating threads and in executable meta data for the main thread). There is basically no way to convert that bytes to a recursion level, however - the best you can do is as Rob said, see that a depth is obviously higher than valid, as it is a reasonable (but not guaranteed) guess that each recusion will use some number of bytes of stack. [1] https://msdn.microsoft.com/en-us/library/984x0h58.aspx [2] https://msdn.microsoft.com/en-us/library/zthk2dkh.aspx Chris From python at mrabarnett.plus.com Wed May 18 13:00:10 2016 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 18 May 2016 18:00:10 +0100 Subject: Extract the middle N chars of a string In-Reply-To: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> References: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: <7c327e41-62ef-4b1f-5914-5e939b377bef@mrabarnett.plus.com> On 2016-05-18 16:47, Steven D'Aprano wrote: > Extracting the first N or last N characters of a string is easy with > slicing: > > s[:N] # first N > s[-N:] # last N > > Getting the middle N seems like it ought to be easy: > > s[N//2:-N//2] > > but that is wrong. It's not even the right length! > > py> s = 'aardvark' > py> s[5//2:-5//2] > 'rdv' > > > So after spending a ridiculous amount of time on what seemed like it ought > to be a trivial function, and an embarrassingly large number of off-by-one > and off-by-I-don't-even errors, I eventually came up with this: > > def mid(string, n): > """Return middle n chars of string.""" > L = len(string) > if n <= 0: > return '' > elif n < L: > Lr = L % 2 > a, ar = divmod(L-n, 2) > b, br = divmod(L+n, 2) > a += Lr*ar > b += Lr*br > string = string[a:b] > return string > > > which works for me: > > > # string with odd number of characters > py> for i in range(1, 8): > ... print mid('abcdefg', i) > ... > d > de > cde > cdef > bcdef > bcdefg > abcdefg > # string with even number of characters > py> for i in range(1, 7): > ... print mid('abcdef', i) > ... > c > cd > bcd > bcde > abcde > abcdef > > > > Is this the simplest way to get the middle N characters? > I think your results are inconsistent. For an odd number of characters you have "abc" + "de" + "fg", i.e. more on the left, but for an even number of characters you have "a" + "bcd" + "ef", i.e. more on the right. My own solution is: def mid(string, n): """Return middle n chars of string.""" if n <= 0: return '' if n > len(string): return string ofs = (len(string) - n) // 2 return string[ofs : ofs + n] If there's an odd number of characters remaining, it always has more on the right. From steve at pearwood.info Wed May 18 13:15:14 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 19 May 2016 03:15:14 +1000 Subject: setrecursionlimit References: <573c941f$0$22141$c3e8da3$5496439d@news.astraweb.com> Message-ID: <573ca324$0$1604$c3e8da3$5496439d@news.astraweb.com> On Thu, 19 May 2016 02:29 am, Rob Gaddi wrote: > Ned Batchelder wrote: >> Is there a way to know how large the C stack can grow, and how much it >> will grow for each Python function call? That sounds complicated to get >> right. >> >> --Ned. > > It's probably trivial to look at a number and say "Yeah, no, that's > CLEARLY too high." based on the minimum number of bytes a stack frame > can require. Guaranteeing that some number lower than that is safe is > almost certainly impossible. So you'd get an exception for truly > stupid numbers, but a lack of exception is no guarantee of safety. > Which is worth what it's worth, I guess. Not being a C programmer, I don't really understand this. The idea I have in mind is a model of program memory I learned[1] way back in the 80s. I don't know if it's still valid. Your application has a bunch of memory available, which broadly speaking can be divided into three chunks: globals, the stack, and the heap. The application knows what globals exist, and can allocate a fixed block of memory big enough for them, so that's not very interesting. It just sits there, holding space for the globals, and doesn't grow or shrink. Then there's the stack, which holds local variables whenever a function is called, the return address of the caller, and other stuff. There's a fixed bottom to the stack, but the top can grown and shrink depending on how many functions you call and how many local variables they use. Then there's everything else, which is the heap, and it can grown and shrink in both directions (up to some maximum, of course): bottom [ globals | stack -----> <----- heap -----> ] top If the stack grows into the heap, or vice versa, Bad Things happen. At best you get a crash. At worst you get arbitrary code execution. I don't really understand why the system can't track the current top of the stack and bottom of the heap, and if they're going to collide, halt the process. That would still be kinda awful, in a sudden "your application just died" kind of way, but it would be better than "your computer is now owned by some hacker in Hong Kong, who is now renting it by the hour to some spammer in Texas". [1] I say "learned", but in reality I more sort of absorbed it by osmosis. Nobody ever actually sat down and told me how the stack and heap work, so the model I have might be completely wrong, even for 1980s tech. -- Steven From ian.g.kelly at gmail.com Wed May 18 13:38:36 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 18 May 2016 11:38:36 -0600 Subject: setrecursionlimit In-Reply-To: <573ca324$0$1604$c3e8da3$5496439d@news.astraweb.com> References: <573c941f$0$22141$c3e8da3$5496439d@news.astraweb.com> <573ca324$0$1604$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Wed, May 18, 2016 at 11:15 AM, Steven D'Aprano wrote: > I don't really understand why the system can't track the current top of the > stack and bottom of the heap, and if they're going to collide, halt the > process. That would still be kinda awful, in a sudden "your application > just died" kind of way, but it would be better than "your computer is now > owned by some hacker in Hong Kong, who is now renting it by the hour to > some spammer in Texas". Seems kind of expensive for little benefit to have to make that check on every function call. From ckaynor at zindagigames.com Wed May 18 13:47:05 2016 From: ckaynor at zindagigames.com (Chris Kaynor) Date: Wed, 18 May 2016 10:47:05 -0700 Subject: setrecursionlimit In-Reply-To: <573ca324$0$1604$c3e8da3$5496439d@news.astraweb.com> References: <573c941f$0$22141$c3e8da3$5496439d@news.astraweb.com> <573ca324$0$1604$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Wed, May 18, 2016 at 10:15 AM, Steven D'Aprano wrote: > I don't really understand why the system can't track the current top of the > stack and bottom of the heap, and if they're going to collide, halt the > process. That would still be kinda awful, in a sudden "your application > just died" kind of way, but it would be better than "your computer is now > owned by some hacker in Hong Kong, who is now renting it by the hour to > some spammer in Texas". > Most modern OSs will track it, and kill the app (hence the exception/crash that occurs), rather than allow access outside the memory. What generally happens is that, when a thread is created (including the main thread during startup), the OS will allocate enough pages to hold the requested stack, plus one as a buffer. Most of these are virtual pages, with no backing memory allocated (either in RAM or the page file). When the next page is first requested, the OS will actually allocate the RAM needed for that page of the stack. If the final guard page is hit, the OS will throw an exception, which generally kills the app. This means there is a overhead when a previously unused stack page is hit, however, as this generally does not happen often, it is generally acceptable. I cannot quickly find the reference material I learned this from, and naturally it will vary based on the OS, however this is pretty standard for general purpose, modern OSes. Chris From auriocus at gmx.de Wed May 18 14:11:37 2016 From: auriocus at gmx.de (Christian Gollwitzer) Date: Wed, 18 May 2016 20:11:37 +0200 Subject: setrecursionlimit In-Reply-To: <573ca324$0$1604$c3e8da3$5496439d@news.astraweb.com> References: <573c941f$0$22141$c3e8da3$5496439d@news.astraweb.com> <573ca324$0$1604$c3e8da3$5496439d@news.astraweb.com> Message-ID: Am 18.05.16 um 19:15 schrieb Steven D'Aprano: > Not being a C programmer, I don't really understand this. > > The idea I have in mind is a model of program memory I learned[1] way back > in the 80s. I don't know if it's still valid. Your application has a bunch > of memory available, which broadly speaking can be divided into three > chunks: globals, the stack, and the heap. > > The application knows what globals exist, and can allocate a fixed block of > memory big enough for them, so that's not very interesting. It just sits > there, holding space for the globals, and doesn't grow or shrink. > > Then there's the stack, which holds local variables whenever a function is > called, the return address of the caller, and other stuff. There's a fixed > bottom to the stack, but the top can grown and shrink depending on how many > functions you call and how many local variables they use. Until here, it is reasonably accurate. On some machines, the stack does grow in the other direction, but that does not matter either. ON x86, it grows from top to bottom > Then there's everything else, which is the heap, and it can grown and shrink > in both directions (up to some maximum, of course): > > bottom [ globals | stack -----> <----- heap -----> ] top > > If the stack grows into the heap, or vice versa, Bad Things happen. At best > you get a crash. At worst you get arbitrary code execution. No, you have virtual memory management in effect in the OS which maps the real memory addresses into your address space. On 64 bit, a collision between stack and heap is practically impossible. > I don't really understand why the system can't track the current top of the > stack and bottom of the heap, and if they're going to collide, halt the > process. It does. But in a different way. For the heap, you need to call a function which asks for more memory. It returns an error code, if the memory can't be supplied. The problem is that often in this case, the program needs more memory to handle that, e.g. to format an error message. If you allocate memory in small pieces until it is exhausted, the program will die in unforeseen ways. If you try to alloc 1TB on the heap and it fails, there is enough room for a clean shutdown. Unless the C program is buggy and does not check the error. On the stack, you don't allocate by telling the OS. You simply increase the stack pointer register. This is a single machine instruction, very fast, and unfeasible to trap by the OS and intercept. Instead, the stack is framed by pages which are non-writeable. As soon as the program tries to write there, it segfaults (SIGSEGV or SIGBUS). At this point there is no way to cleanly exit the program, therefore you see the crash/segfault. It might happen that you overrun the stack so much as to reach writeable memory again. But not under normal circumstances, where only a few bytes are pushed/popped. > That would still be kinda awful, in a sudden "your application > just died" kind of way, but it would be better than "your computer is now > owned by some hacker in Hong Kong, who is now renting it by the hour to > some spammer in Texas". Stack overflow does not usually lead to security risks. A buffer overflow is different: It means that the program allocates a fixed-size buffer on the stack, which overflows and writes into the return addresses / local variables of functions higher up the callchain. The basic problem here is, that the C programmer was too lazy to get the memory from the heap. Christian From grant.b.edwards at gmail.com Wed May 18 16:45:19 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Wed, 18 May 2016 20:45:19 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: Message-ID: On 2016-05-16, Grant Edwards wrote: > Is there any way to limit the number of connections a browser uses to > download a web page? [Long-winded tail of woe...] > So now I'm going to set up a simple Python HTTP server to try some > other approaches: > > 1) Only allow the listening socket to accept 1 connection at a time. That doesn't work. If you refuse connections, at least one browser (Chrome) fails some of the fetches and you end up sans css, js, png, whatever. > 2) Accept the TCP connection, but don't allow the SSL handshaking to > start on the "extra" connections. That seems to work for Chrome (which seems to be the worst of the lot). If on the 2nd, 3rd, 4th, 5th connections you accept the TCP connection and then stall before doing the SSL hadshake, Chrome stays happy _and_ will shift all fetches to the one connection who's handshake has finished. You probably can't depend on this behavior, but OTOH it can't hurt. However, since the web server is single-threaded, doing this in practice ends up be rather complicated. > 3) ??? I did find a bug in the glue between the SSL stack and the web server that was causing (under certain conditions) an ssl-read operation to block when it shouldn't. Fixing that brought my cold page load time down from 15 to 7-9 seconds. The most promising approach is probably to minimize the number of files by doing server-side includes for css, js, and image data. If I also add support for chunked transport, that should make a big difference. There are currently certain cases where the reply size isn't known, so the connection can't be left open and re-used. Each time a connection has to be closed by ther server to indicate endof-response, it's a big performance hit. If the browser supported chunked transport, it should allow all connections to be left open and reused. That would then makes a huge difference on subsequent page load times as people work with the thing... -- Grant Edwards grant.b.edwards Yow! I'm having BEAUTIFUL at THOUGHTS about the INSIPID gmail.com WIVES of smug and wealthy CORPORATE LAWYERS ... From jacob.scott at gmail.com Wed May 18 17:01:32 2016 From: jacob.scott at gmail.com (Jacob Scott) Date: Wed, 18 May 2016 14:01:32 -0700 Subject: Resources/pointers for writing maintable, testable Python Message-ID: Many years ago, when I was primarily writing Java, I found Misko Hevery's Guide: Writing Testable Code to be incredibly helpful in guiding the design and structure of my codebase, and as reference for checking if my code was smelly. Today, I'm happily writing primarily Python (unfortunately, 2.7 -- but I'm not sure it makes that much of a difference), but I haven't found anything that speaks to me in the same way. Some of the best resources I've found, but which don't quite cover all of what I'm looking for, include - PEP-8 and PEP-20 - The Hitchhiker's Guide to Python - Effective Python I'd appreciate any pointers to resources I might have missed, general thoughts on the topic, etc. Thanks, Jacob From armorsmith42 at gmail.com Wed May 18 17:23:49 2016 From: armorsmith42 at gmail.com (Andrew Farrell) Date: Wed, 18 May 2016 17:23:49 -0400 Subject: Resources/pointers for writing maintable, testable Python In-Reply-To: References: Message-ID: Hi Jacob, You are probably looking for the book Test-Driven Development with Python . You'll also want to look at py.test Cheers! Andrew Farrell On Wed, May 18, 2016 at 5:01 PM, Jacob Scott wrote: > Many years ago, when I was primarily writing Java, I found Misko > Hevery's Guide: > Writing Testable Code > > to > be incredibly helpful in guiding the design and structure of my codebase, > and as reference for checking if my code was smelly. > > Today, I'm happily writing primarily Python (unfortunately, 2.7 -- but I'm > not sure it makes that much of a difference), but I haven't found anything > that speaks to me in the same way. Some of the best resources I've found, > but which don't quite cover all of what I'm looking for, include > > - PEP-8 and PEP-20 > - The Hitchhiker's Guide to Python > > - Effective Python > > I'd appreciate any pointers to resources I might have missed, general > thoughts on the topic, etc. > > Thanks, > > Jacob > -- > https://mail.python.org/mailman/listinfo/python-list > From random832 at fastmail.com Wed May 18 17:28:43 2016 From: random832 at fastmail.com (Random832) Date: Wed, 18 May 2016 17:28:43 -0400 Subject: Extract the middle N chars of a string In-Reply-To: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> References: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1463606923.1353025.611955401.37F5D0F6@webmail.messagingengine.com> On Wed, May 18, 2016, at 11:47, Steven D'Aprano wrote: > So after spending a ridiculous amount of time on what seemed like it > ought > to be a trivial function, and an embarrassingly large number of > off-by-one > and off-by-I-don't-even errors, I eventually came up with this: > > def mid(string, n): > """Return middle n chars of string.""" > L = len(string) > if n <= 0: > return '' > elif n < L: > Lr = L % 2 > a, ar = divmod(L-n, 2) > b, br = divmod(L+n, 2) > a += Lr*ar > b += Lr*br > string = string[a:b] > return string My take: def mid(string, n): if n > len(string): n = len(string) if n <= 0: n = 0 offset = int(len(string)/2+n/2) return string[offset:offset+n] It doesn't get the same result as yours when the length is odd and N is even, but the problem is ambiguous there and I feel like my algorithm is more clear. I'm funneling the special cases through the slice statement at the end rather than simply returning string and '' because it's conceptually nicer and because it could be used for other purposes than slicing strings. From grant.b.edwards at gmail.com Wed May 18 17:35:31 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Wed, 18 May 2016 21:35:31 +0000 (UTC) Subject: Extract the middle N chars of a string References: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-18, Steven D'Aprano wrote: > Getting the middle N seems like it ought to be easy: I'm still trying to figure out when one would want to do that... -- Grant Edwards grant.b.edwards Yow! My CODE of ETHICS at is vacationing at famed gmail.com SCHROON LAKE in upstate New York!! From greg.ewing at canterbury.ac.nz Wed May 18 18:52:09 2016 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Thu, 19 May 2016 10:52:09 +1200 Subject: Quote of the day In-Reply-To: <8e92e249-c46e-4b92-a93b-778879461273@googlegroups.com> References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> <8e92e249-c46e-4b92-a93b-778879461273@googlegroups.com> Message-ID: Ned Batchelder wrote: > I'm not sure how > the test runner could determine that it was empty. I guess it could > introspect the test function to see if it had any real code in it, Then people would just get clever at putting dummy code in the test that fools the test runner but doesn't really test anything... -- Greg From greg.ewing at canterbury.ac.nz Wed May 18 18:58:50 2016 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Thu, 19 May 2016 10:58:50 +1200 Subject: OT: limit number of connections from browser to my server? In-Reply-To: References: Message-ID: Grant Edwards wrote: > Product spec explicitly states HTTPS only. I'm told that is not open > for discussion. The customer is a large, somewhat bureaucratic German > corporation, and they generally mean it when they say something is > non-negotiable. They're probably being sensible. The way the Internet of Things is shaping up, it's far better to have too much security than too little. -- Greg From ethan at stoneleaf.us Wed May 18 19:30:54 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 18 May 2016 16:30:54 -0700 Subject: Quote of the day In-Reply-To: References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> <8e92e249-c46e-4b92-a93b-778879461273@googlegroups.com> Message-ID: <573CFB2E.8000703@stoneleaf.us> On 05/18/2016 03:52 PM, Gregory Ewing wrote: > Ned Batchelder wrote: >> I'm not sure how >> the test runner could determine that it was empty. I guess it could >> introspect the test function to see if it had any real code in it, > > Then people would just get clever at putting dummy code > in the test that fools the test runner but doesn't really > test anything... Some would have, sure. Either way, it's a solved issue now because we (finally ;) have the @skip decorator. -- ~Ethan~ From marcusjmurphy at gmail.com Wed May 18 20:02:50 2016 From: marcusjmurphy at gmail.com (marcusjmurphy at gmail.com) Date: Wed, 18 May 2016 17:02:50 -0700 (PDT) Subject: Basic Concepts In-Reply-To: References: Message-ID: <5156d16c-eddb-4302-b9da-db0ff2768f6e@googlegroups.com> On Wednesday, April 27, 2016 at 2:29:25 AM UTC-7, Smith wrote: > Fill in the blanks to declare a variable, add 5 to it and print its value: > > >>> x = 4 > >>> x_ = 5 > >>> print_ > > > Any suggestion ? > > Thanks >>> x = 4 >>> x += 5 >>> print(x) 9 From steve at pearwood.info Wed May 18 20:43:50 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 19 May 2016 10:43:50 +1000 Subject: Quote of the day References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> <8e92e249-c46e-4b92-a93b-778879461273@googlegroups.com> <573CFB2E.8000703@stoneleaf.us> Message-ID: <573d0c48$0$1595$c3e8da3$5496439d@news.astraweb.com> On Thu, 19 May 2016 09:30 am, Ethan Furman wrote: > On 05/18/2016 03:52 PM, Gregory Ewing wrote: >> Ned Batchelder wrote: > >>> I'm not sure how >>> the test runner could determine that it was empty. I guess it could >>> introspect the test function to see if it had any real code in it, >> >> Then people would just get clever at putting dummy code >> in the test that fools the test runner but doesn't really >> test anything... > > Some would have, sure. > > Either way, it's a solved issue now because we (finally ;) have the > @skip decorator. That only solves the problem for responsible, decent developers. But I guarantee you that, right now, as we speak, there is some poor schmuck out there whose Pointy Haired Boss has given him a Key Performance Indicator of X tests passing (not failing or skipped) per week, and he's responding by writing tests which pass by not testing anything. -- Steven From steve at pearwood.info Wed May 18 20:48:16 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 19 May 2016 10:48:16 +1000 Subject: Extract the middle N chars of a string References: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: <573d0d53$0$1598$c3e8da3$5496439d@news.astraweb.com> On Thu, 19 May 2016 07:35 am, Grant Edwards wrote: > On 2016-05-18, Steven D'Aprano wrote: > >> Getting the middle N seems like it ought to be easy: > > I'm still trying to figure out when one would want to do that... I wanted to centre some text and truncate it to a fixed width. So I needed the middle N characters. -- Steven From ben+python at benfinney.id.au Wed May 18 20:55:57 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 19 May 2016 10:55:57 +1000 Subject: Resources/pointers for writing maintable, testable Python References: Message-ID: <85posic1cy.fsf@benfinney.id.au> Jacob Scott writes: > Today, I'm happily writing primarily Python (unfortunately, 2.7 -- but I'm > not sure it makes that much of a difference) Python 2.7 is still viable, but is certainly a dead end. The difference increases month by month, and the advantage is only going to increase to Python 3. Any new code base should not be written in Python 2. Any libraries you need which don't work yet on Python 3 should be seriously reconsidered. > I'd appreciate any pointers to resources I might have missed, general > thoughts on the topic, etc. Code Like A Pythonista was written in the Python 2 era but is still excellent advice today. -- \ ?I have the simplest tastes. I am always satisfied with the | `\ best.? ?Oscar Wilde, quoted in _Chicago Brothers of the Book_, | _o__) 1917 | Ben Finney From ethan at stoneleaf.us Wed May 18 20:59:50 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 18 May 2016 17:59:50 -0700 Subject: Quote of the day In-Reply-To: <573d0c48$0$1595$c3e8da3$5496439d@news.astraweb.com> References: <573acd5f$0$1603$c3e8da3$5496439d@news.astraweb.com> <87twhxf5nl.fsf@elektro.pacujo.net> <8e92e249-c46e-4b92-a93b-778879461273@googlegroups.com> <573CFB2E.8000703@stoneleaf.us> <573d0c48$0$1595$c3e8da3$5496439d@news.astraweb.com> Message-ID: <573D1006.4030200@stoneleaf.us> On 05/18/2016 05:43 PM, Steven D'Aprano wrote: > On Thu, 19 May 2016 09:30 am, Ethan Furman wrote: > >> On 05/18/2016 03:52 PM, Gregory Ewing wrote: >>> Ned Batchelder wrote: >> >>>> I'm not sure how >>>> the test runner could determine that it was empty. I guess it could >>>> introspect the test function to see if it had any real code in it, >>> >>> Then people would just get clever at putting dummy code >>> in the test that fools the test runner but doesn't really >>> test anything... >> >> Some would have, sure. >> >> Either way, it's a solved issue now because we (finally ;) have the >> @skip decorator. > > That only solves the problem for responsible, decent developers. Since I'm not a manager (and when I was, I wasn't the PHB type), responsible, decent developers are where I focus my attention. -- ~Ethan~ From steve at pearwood.info Wed May 18 21:03:27 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 19 May 2016 11:03:27 +1000 Subject: Extract the middle N chars of a string References: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> <1463606923.1353025.611955401.37F5D0F6@webmail.messagingengine.com> Message-ID: <573d10e0$0$1612$c3e8da3$5496439d@news.astraweb.com> On Thu, 19 May 2016 07:28 am, Random832 wrote: > My take: > > def mid(string, n): > if n > len(string): n = len(string) > if n <= 0: n = 0 > offset = int(len(string)/2+n/2) > return string[offset:offset+n] > > It doesn't get the same result as yours when the length is odd and N is > even, but the problem is ambiguous there and I feel like my algorithm is > more clear. I'm not sure that "the algorithm is more clear" is the right way to judge this. Especially when your function doesn't even come *close* to meeting the requirements. It doesn't even return substrings of the right length! py> for i in range(8): ... print mid('abcdefg', i) ... e ef fg fg g g py> for i in range(7): ... print mid('abcdef', i) ... d ef ef f f I sympathise, I really do, because as my first post says, this really does seem like it ought to be a no-brainer trivial exercise in slicing. Instead, it is remarkably subtle and tricky to get right. -- Steven From kobsx4 at gmail.com Wed May 18 21:04:27 2016 From: kobsx4 at gmail.com (Jake Kobs) Date: Wed, 18 May 2016 18:04:27 -0700 (PDT) Subject: Program prints questions for user input, but won't show the answer output Message-ID: <1cc14787-7061-45c9-a70e-1b16e3f5ef3a@googlegroups.com> Here is the code: #Lab 9-4 Blood Drive #the main function def main(): endProgram = 'no' while endProgram == 'no': print # declare variables pints = [0] * 7 totalPints = 0 averagePints = 0 highPints = 0 lowPints = 0 # function calls pints = getPints(pints) totalPints = getTotal(pints, totalPints) averagePints = getAverage(totalPints, averagePints) highPints = getHigh(pints, highPints) lowPints = getLow(pints, lowPints) displayInfo(averagePints, highPints, lowPints) endProgram = raw_input('Do you want to end program? (Enter no or yes): ') while not (endProgram == 'yes' or endProgram == 'no'): print 'Please enter a yes or no' endProgram = raw_input('Do you want to end program? (Enter no or yes): ') #the getPints function def getPints(pints): counter = 0 while counter < 7: pints[counter] = input("Enter pints collected: ") counter = counter + 1 return pints #the getTotal function def getTotal(pints, totalPints): counter = 0 while counter < 7: totalPints = totalPints + pints[counter] counter = counter + 1 return totalPints #the getAverage function def getAverage(totalPints, averagePints): averagePints = totalPints / 7 return averagePints #the getHigh function def getHigh(pints, highPints): highPints = pints[0] counter = 1 while counter < 7: if (pints[counter] > highPints): highPints = pints[counter] counter = counter + 1 return highPints #the getLow function def getLow(pints, lowPints): lowPints = pints[0] counter = 1 while counter < 7: if (pints[counter] < lowPints): lowPints = pints[counter] counter = counter + 1 return lowPints #the displayInfo function def displayInfo(averagePints, highPints, lowPints): print "The average pints donated was: ", averagePints print "The highest amount of pints donated was: ", highPints print "The lowest amount of pints donated was: ", lowPints return displayInfo(averagePints, highPints, lowPints) main() The problem is that the display info isn't shown after the user types in their 7 numerical values. Please help. From jacob.scott at gmail.com Wed May 18 21:15:35 2016 From: jacob.scott at gmail.com (Jacob Scott) Date: Wed, 18 May 2016 18:15:35 -0700 Subject: Resources/pointers for writing maintable, testable Python In-Reply-To: <85posic1cy.fsf@benfinney.id.au> References: <85posic1cy.fsf@benfinney.id.au> Message-ID: Ah, what I should have done is note that I am writing Python 2.7 (and this is at work, with all that entails...), but am happy to take advice that applies only to Python 3 (even 3.5 or 3.6.0a1!) and work backwards to apply it to Python 2.7. I think I would be (perhaps pleasantly) surprised if there was a wide gulf between Python 2.7 and Python 3 *in terms of advice/resources applicable to my original question*. Based on my (admittedly shallow) understanding of overall Python 2.7/3 differences, the biggest changes (from e.g. http://sebastianraschka.com/Articles/2014_python_2_3_key_diff.html) tend to be a bit lower level (utf-8 str) than what I'm focused on (maintainable and testable classes, functions, modules, etc). Thanks for the pointer to Code Like A Pythonista and the feedback on 2.7 vs 3! Jacob On Wed, May 18, 2016 at 5:55 PM, Ben Finney wrote: > Jacob Scott writes: > > > Today, I'm happily writing primarily Python (unfortunately, 2.7 -- but > I'm > > not sure it makes that much of a difference) > > Python 2.7 is still viable, but is certainly a dead end. The difference > increases month by month, and the advantage is only going to increase to > Python 3. > > Any new code base should not be written in Python 2. Any libraries you > need which don't work yet on Python 3 should be seriously reconsidered. > > > I'd appreciate any pointers to resources I might have missed, general > > thoughts on the topic, etc. > > Code Like A Pythonista was written in the Python 2 era > > but is still excellent advice today. > > -- > \ ?I have the simplest tastes. I am always satisfied with the | > `\ best.? ?Oscar Wilde, quoted in _Chicago Brothers of the Book_, | > _o__) 1917 | > Ben Finney > > -- > https://mail.python.org/mailman/listinfo/python-list > From gordon at panix.com Wed May 18 21:16:49 2016 From: gordon at panix.com (John Gordon) Date: Thu, 19 May 2016 01:16:49 +0000 (UTC) Subject: Program prints questions for user input, but won't show the answer output References: <1cc14787-7061-45c9-a70e-1b16e3f5ef3a@googlegroups.com> Message-ID: In <1cc14787-7061-45c9-a70e-1b16e3f5ef3a at googlegroups.com> Jake Kobs writes: > Here is the code: > def getHigh(pints, highPints): > highPints = pints[0] > counter = 1 > while counter < 7: > if (pints[counter] > highPints): > highPints = pints[counter] > counter = counter + 1 > return highPints getHigh() goes into an infinite loop if pints[counter] is less than or equal to highPints. > def getLow(pints, lowPints): > lowPints = pints[0] > counter = 1 > while counter < 7: > if (pints[counter] < lowPints): > lowPints = pints[counter] > counter = counter + 1 > return lowPints And getLow() has a very similar problem. I suspect you want to unindent the 'counter = counter + 1' statement so that it is NOT inside the 'if' statement. -- John Gordon A is for Amy, who fell down the stairs gordon at panix.com B is for Basil, assaulted by bears -- Edward Gorey, "The Gashlycrumb Tinies" From python at mrabarnett.plus.com Wed May 18 21:31:43 2016 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 19 May 2016 02:31:43 +0100 Subject: Program prints questions for user input, but won't show the answer output In-Reply-To: <1cc14787-7061-45c9-a70e-1b16e3f5ef3a@googlegroups.com> References: <1cc14787-7061-45c9-a70e-1b16e3f5ef3a@googlegroups.com> Message-ID: On 2016-05-19 02:04, Jake Kobs wrote: > Here is the code: > > #Lab 9-4 Blood Drive > > #the main function > def main(): > endProgram = 'no' > while endProgram == 'no': > print > # declare variables > pints = [0] * 7 > totalPints = 0 > averagePints = 0 > highPints = 0 > lowPints = 0 > > > > > > > > > > > > > # function calls > pints = getPints(pints) > totalPints = getTotal(pints, totalPints) > averagePints = getAverage(totalPints, averagePints) > highPints = getHigh(pints, highPints) > lowPints = getLow(pints, lowPints) > displayInfo(averagePints, highPints, lowPints) > > endProgram = raw_input('Do you want to end program? (Enter no or yes): ') > while not (endProgram == 'yes' or endProgram == 'no'): > print 'Please enter a yes or no' > endProgram = raw_input('Do you want to end program? (Enter no or yes): ') > > #the getPints function > def getPints(pints): > counter = 0 > while counter < 7: > pints[counter] = input("Enter pints collected: ") > counter = counter + 1 > return pints > #the getTotal function > def getTotal(pints, totalPints): > > counter = 0 > while counter < 7: > totalPints = totalPints + pints[counter] > counter = counter + 1 > return totalPints > #the getAverage function > def getAverage(totalPints, averagePints): > averagePints = totalPints / 7 > return averagePints > #the getHigh function > def getHigh(pints, highPints): > highPints = pints[0] > counter = 1 > while counter < 7: > if (pints[counter] > highPints): > highPints = pints[counter] The indentation here is wrong: > counter = counter + 1 It will add 1 _only_ if pints[counter] > highPints. > return highPints > #the getLow function > def getLow(pints, lowPints): > lowPints = pints[0] > counter = 1 > while counter < 7: > if (pints[counter] < lowPints): > lowPints = pints[counter] The indentation here is wrong: > counter = counter + 1 It will add 1 _only_ if pints[counter] < highPints. > return lowPints > #the displayInfo function > def displayInfo(averagePints, highPints, lowPints): > print "The average pints donated was: ", averagePints > print "The highest amount of pints donated was: ", highPints > print "The lowest amount of pints donated was: ", lowPints Why is 'displayInfo' calling itself here? > return displayInfo(averagePints, highPints, lowPints) > > main() > > The problem is that the display info isn't shown after the user types in their 7 numerical values. Please help. > From steve at pearwood.info Wed May 18 21:34:59 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 19 May 2016 11:34:59 +1000 Subject: Extract the middle N chars of a string References: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> <7c327e41-62ef-4b1f-5914-5e939b377bef@mrabarnett.plus.com> Message-ID: <573d1844$0$1593$c3e8da3$5496439d@news.astraweb.com> On Thu, 19 May 2016 03:00 am, MRAB wrote: > I think your results are inconsistent. > > For an odd number of characters you have "abc" + "de" + "fg", i.e. more > on the left, but for an even number of characters you have "a" + "bcd" + > "ef", i.e. more on the right. Correct. That's intentional. I didn't start with an algorithm. I started by manually extracting the N middle characters, for various values of N, then wrote code to get the same result. For example, if I start with an odd-length string, like "inquisition", then the "middle N" cases for odd-N are no-brainers, because they have to be centered on the middle character: N=1 's' N=3 'isi' N=5 'uisit' For even-N, I had a choice: N=2 'is' or 'si' and to be perfectly frank, I didn't really care much either way and just arbitrarily picked the second, based on the fact that string.center() ends up with a slight bias to the right: py> 'ab'.center(5, '*') '**ab*' For even-length string, like "aardvark", the even-N case is the no-brainer: N=2 "dv" N=4 "rdva" N=6 "ardvar" but with odd-N I have a choice: N=3 "rdv" or "dva" In this case, I *intentionally* biased it the other way, so that (in some sense) overall the mid() function would be unbiased: - half the cases, there's no bias at all; - a quarter of the time, there's a bias to the right; - a quarter of the time, there's a bias to the left; - so on average, the bias is zero. > My own solution is: > > > def mid(string, n): > """Return middle n chars of string.""" > if n <= 0: > return '' > if n > len(string): > return string > ofs = (len(string) - n) // 2 > return string[ofs : ofs + n] > > > If there's an odd number of characters remaining, it always has more on > the right. Thanks to you and Ian (who independently posted a similar solution), that's quite good too if you don't care about the bias. -- Steven From steve at pearwood.info Wed May 18 21:37:57 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 19 May 2016 11:37:57 +1000 Subject: Extract the middle N chars of a string References: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: <573d18f5$0$1593$c3e8da3$5496439d@news.astraweb.com> On Thu, 19 May 2016 02:54 am, Peter Otten wrote: > Peter Otten wrote: > >>>>> def mid(s, n): >> ... shave = len(s) - n >> ... if shave > 0: > shave += len(s) % 2 >> ... shave //= 2 >> ... s = s[shave:shave+n] >> ... return s > >> Not exactly the same results as your implementation though. > > The extra line should fix that, but it looks, err -- odd. Nice! Thank you. I like this solution. -- Steven From kobsx4 at gmail.com Wed May 18 21:50:46 2016 From: kobsx4 at gmail.com (Jake Kobs) Date: Wed, 18 May 2016 18:50:46 -0700 (PDT) Subject: Program prints questions for user input, but won't show the answer output In-Reply-To: <1cc14787-7061-45c9-a70e-1b16e3f5ef3a@googlegroups.com> References: <1cc14787-7061-45c9-a70e-1b16e3f5ef3a@googlegroups.com> Message-ID: <276ca193-338e-4cea-9dce-1af54f900da9@googlegroups.com> MRAB, I am not quite sure how to return the print statements so I thought that returning the displayInfo def would help.. Im so lost. From steve at pearwood.info Wed May 18 22:14:50 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 19 May 2016 12:14:50 +1000 Subject: Program prints questions for user input, but won't show the answer output References: <1cc14787-7061-45c9-a70e-1b16e3f5ef3a@googlegroups.com> <276ca193-338e-4cea-9dce-1af54f900da9@googlegroups.com> Message-ID: <573d219b$0$1598$c3e8da3$5496439d@news.astraweb.com> On Thu, 19 May 2016 11:50 am, Jake Kobs wrote: > MRAB, > > I am not quite sure how to return the print statements so I thought that > returning the displayInfo def would help.. Im so lost. There's no need to return the text that you printed. You printed it, the job is done. Here is some general advice that may help you be less lost. (1) Each function should do "one thing". For example, it either collects information, or it displays it. It shouldn't do both. Your code seems pretty good at following that guideline already, well done. (2) Use existing functions as much as possible. The Python docs has a list of the built-in functions here: https://docs.python.org/2/library/functions.html Refer to that as often as possible. For example, your functions getHigh, getLow and getTotal functions just re-invent the wheel. Python already has functions to do that: max, min, sum In Python 2, there's no built-in function for average, but for your purposes you can just use sum(pints)/len(pints). (3) Whoever taught you to write while loops should be taken out and horse-whipped. While loops are for looping when you don't know in advance how many times you need to repeat. When you do know how many times to repeat, use a for-loop: # I want to repeat seven times for counter in range(7): print "Loop", counter Isn't that simpler than a while loop? # I want to repeat seven times counter = 7 while counter > 0: # or should that be >= 0? >= 1? I forget! print "Loop", counter counter -= 1 And its faster too. (4) Don't use "input". (The reasons are technical, if you really want to know, please ask.) Instead of writing: input("Enter pints collected: ") write this: int(raw_input("Enter pints collected: ")) -- Steven From python at mrabarnett.plus.com Wed May 18 22:16:56 2016 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 19 May 2016 03:16:56 +0100 Subject: Program prints questions for user input, but won't show the answer output In-Reply-To: <276ca193-338e-4cea-9dce-1af54f900da9@googlegroups.com> References: <1cc14787-7061-45c9-a70e-1b16e3f5ef3a@googlegroups.com> <276ca193-338e-4cea-9dce-1af54f900da9@googlegroups.com> Message-ID: <88f4a956-c8c5-4e94-c52b-dd48073900fc@mrabarnett.plus.com> On 2016-05-19 02:50, Jake Kobs wrote: > MRAB, > > I am not quite sure how to return the print statements so I thought that returning the displayInfo def would help.. Im so lost. > "return the print statements"? The print statements ... print! Have a search for Python tutorials online and pick one that suits you. I see that you're using Python 2. I'd suggest trying Python 3 unless you have a good reason for using Python 2. From orgnut at yahoo.com Wed May 18 22:58:32 2016 From: orgnut at yahoo.com (Larry Hudson) Date: Wed, 18 May 2016 19:58:32 -0700 Subject: Summing/combining tuples In-Reply-To: References: Message-ID: On 05/18/2016 05:59 PM, DFS wrote: > Have aList = [ > ('x','Name1', 1, 85), > ('x','Name2', 3, 219), > ('x','Name2', 1, 21), > ('x','Name3', 6, 169) > ] > > want > > aList = [ > ('Name1', 1, 85), > ('Name2', 4, 240), > ('Name3', 6, 169) > ] > > > This drops the first element in each tuple: > alist = [(b,c,d) for a,b,c,d in alist] Slicing may be more efficient (perhaps, perhaps not -- don't know). And probably effectively no difference, but slightly shorter to write. alist = [t[1:] for t in alist] > > And this summation creates 2 lists that then have to be combined: > > groups1 = defaultdict(int) > for nm, matches, words in alist: > groups1[nm] += matches > > groups2 = defaultdict(int) > for nm, matches, words in alist: > groups2[nm] += words > > > > Is there something shorter and sweeter for the summation? > > Thanks > Why two loops? Put both summations in a single loop. Then you're only scanning the alist once instead of twice. groups1 = defaultdict(int) groups2 = defaultdict(int) for nm, matches, words in alist: groups1[nm] += matches groups2[nm] += words -=- Larry -=- From rosuav at gmail.com Wed May 18 23:09:23 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 19 May 2016 13:09:23 +1000 Subject: Resources/pointers for writing maintable, testable Python In-Reply-To: References: <85posic1cy.fsf@benfinney.id.au> Message-ID: On Thu, May 19, 2016 at 11:15 AM, Jacob Scott wrote: > I think I would be (perhaps pleasantly) surprised if there was a wide gulf > between Python 2.7 and Python 3 *in terms of advice/resources applicable to > my original question*. Based on my (admittedly shallow) understanding of > overall Python 2.7/3 differences, the biggest changes (from e.g. > http://sebastianraschka.com/Articles/2014_python_2_3_key_diff.html) tend to > be a bit lower level (utf-8 str) than what I'm focused on (maintainable and > testable classes, functions, modules, etc). It's not really about UTF-8 (and the article you link to is flat wrong in describing Py3's str type as UTF-8); it's about the recognition of a fundamental difference between text and bytes. Looking at it another way: Python 2 lets you pretend it's all easy, so long as you're working with the simple Latin letters A-Z and a-z, but when you need anything else (other alphabets, non-ASCII symbols like dashes and quotes and emoji, diacritical marks, etc, etc), Python 2 doesn't help you at all, and suddenly you're on your own. Python 3 forces you to think up-front about things (a little bit of extra work/hassle), but then gives you all the world's languages for free. That's a very high level feature: you ask a user to enter his/her name, and instead of requiring that it be English without diacritical marks, you can accept (almost[1]) anything. Sometimes, UTF-8 lets you *pretend* that Python 2 (or C or PHP or whatever other language you pick) can handle all those other characters too, but it's a massively leaky abstraction. The other broad change in Python 3 is a shift in focus from concrete lists to lazy objects. In Py2, range() returns a list of numbers; in Py3, it returns a "range object", which functions just like that list in many ways (subscripting, iterating, etc work the same way), but is far more memory-efficient on huge ranges. In Py2, a dictionary's keys() method returns a list; in Py3, it returns a special view object. That kind of thing. Quite a few of the other differences mentioned in that article are actually the removal of already-deprecated syntax. For instance, the 'except NameError as err:' syntax works in 2.6 and 2.7 as well as 3.x, and is the recommended way to spell it (unless you need to support ancient Pythons - 2.5 is pretty old now, but some people do support it). For backward compatibility, all of the 2.x line still supports the old syntax, but new code should be using the reliable syntax - the comma is ambiguous (what if you wanted to accept more than one exception type?). Similarly, you shouldn't be calling an iterator's .next() method directly; always call the built-in next() function. Python 3 made this a lot clearer by renaming the method to __next__(), which clarifies that this is a "magic method" or "special method. Dunder methods are for defining, not calling. (Rule of thumb, not entirely true, but close on.) For the rest, Python 3 basically has the advantage of six additional years of development time. That's given us features like exception chaining, core support for virtual environments, a variety of new stdlib modules, support for asynchronous I/O, a new multiplication operator, and soon (3.6), interpolated string 'literals' (more like a list comprehension than a string literal). And that gap is ever widening. Using 2.7 means abandoning all those features. It's still in support, but it's bug fixes and security patches only - which, as the Red Queen explained to Alice, is where you have to run as fast as you can just to stay in one place. ChrisA [1] Some people's names can't be represented in Unicode. But excluding those names is a lot less significant than excluding the huge proportion of people whose names aren't representable in ASCII. From rustompmody at gmail.com Wed May 18 23:19:08 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Wed, 18 May 2016 20:19:08 -0700 (PDT) Subject: setrecursionlimit In-Reply-To: <1116f4d7-5533-4171-90d2-69411133aa28@googlegroups.com> References: <573c941f$0$22141$c3e8da3$5496439d@news.astraweb.com> <573ca324$0$1604$c3e8da3$5496439d@news.astraweb.com> <1116f4d7-5533-4171-90d2-69411133aa28@googlegroups.com> Message-ID: <337f233a-32c4-4944-96bb-731c3f56463d@googlegroups.com> On Thursday, May 19, 2016 at 3:13:44 AM UTC+5:30, bream wrote: > On Wednesday, May 18, 2016 at 6:47:42 PM UTC+1, Chris Kaynor wrote: > > On Wed, May 18, 2016 at 10:15 AM, Steven D'Aprano wrote: > > > > > I don't really understand why the system can't track the current top of the > > > stack and bottom of the heap, and if they're going to collide, halt the > > > process. That would still be kinda awful, in a sudden "your application > > > just died" kind of way, but it would be better than "your computer is now > > > owned by some hacker in Hong Kong, who is now renting it by the hour to > > > some spammer in Texas". > > > > > > > Most modern OSs will track it, and kill the app (hence the exception/crash > > that occurs), rather than allow access outside the memory. > > Chris > > Oh so funny, I just had to roar with laughter, unless you (plural) classify VMS as a "modern OS". Then of course some youngsters think the OS is simply software, whereas us old gits know that a combination of hardware and software makes for something that is rather more solid. Hi Mark -- howdy! Much truth in your comment. Dijkstra pointed out (in mid-70s!) the essential tradeoff between hardware and software. He did it for numeric oveflow but it applies equally for stack overflow. In short software costs are variable costs: An instruction that is executed a million times costs million-fold one that is executed only once. For hardware its a fixed cost: Once the investment in silicon is done, once or a billion times is the same. Therefore -- and this is Dijkstra's key insight -- if a test is very skew, ie if the if-part is 99.9% likely to be taken and the else part only 0.1% the programmer will be increasingly tempted to drop the test and hope/pray that the 0.1% does not happen. With a hardware interrupt this temptation would be obviated. Both the mess in catching numeric overflow as well as stackoverflow looks like its C's fault. I consider it as the fault of currently fashionable stock hardware From rustompmody at gmail.com Wed May 18 23:31:31 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Wed, 18 May 2016 20:31:31 -0700 (PDT) Subject: Resources/pointers for writing maintable, testable Python In-Reply-To: References: <85posic1cy.fsf@benfinney.id.au> Message-ID: On Thursday, May 19, 2016 at 6:26:26 AM UTC+5:30, Ben Finney wrote: > Jacob Scott writes: > > > Today, I'm happily writing primarily Python (unfortunately, 2.7 -- but I'm > > not sure it makes that much of a difference) > > Python 2.7 is still viable, but is certainly a dead end. The difference > increases month by month, and the advantage is only going to increase to > Python 3. > > Any new code base should not be written in Python 2. Any libraries you > need which don't work yet on Python 3 should be seriously reconsidered. > > > I'd appreciate any pointers to resources I might have missed, general > > thoughts on the topic, etc. > > Code Like A Pythonista was written in the Python 2 era > > but is still excellent advice today. If is python-2 and still the best excellent advice today doesn't it go somewhat counter to pyhthon-2 is a dead-end ?? :-) Need to point this out since the opposite case to Chris' "switch to 3 or else suffer in hell..." needs to be articulated: Python-3 is nice but 2 is ok. The diffs are not such a big deal From rosuav at gmail.com Wed May 18 23:47:15 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 19 May 2016 13:47:15 +1000 Subject: Resources/pointers for writing maintable, testable Python In-Reply-To: References: <85posic1cy.fsf@benfinney.id.au> Message-ID: On Thu, May 19, 2016 at 1:31 PM, Rustom Mody wrote: > On Thursday, May 19, 2016 at 6:26:26 AM UTC+5:30, Ben Finney wrote: >> Code Like A Pythonista was written in the Python 2 era >> >> but is still excellent advice today. > > > If is python-2 and still the best excellent advice today > doesn't it go somewhat counter to pyhthon-2 is a dead-end ?? :-) Nope. Py2 is a dead end because it isn't moving forward. It's staying right where it is. There can certainly be advice written about Python 2 that is worth reading, though. In fact, I have some books written about REXX on OS/2 which I would recommend to someone learning Python on Linux. I probably have some books from the 1980s that are still worth reading. (Software books older than that won't be on my shelf, but quite likely do exist.) > Need to point this out since the opposite case to Chris' > "switch to 3 or else suffer in hell..." needs to be articulated: That is not my stance. Python 2 is still usable - it just isn't moving forward. > Python-3 is nice but 2 is ok. The diffs are not such a big deal The differences are getting to be a bigger and bigger deal. I wouldn't advise anyone to use Python 2.4 unless compatibility with Red Hat Enterprise Linux 5, because 2.4 misses out on heaps of stuff that newer versions have. It's the same with 2.7 - use it if compatibility with systems without 3.x is important, otherwise use the latest. (Of course, that's new projects. Existing code implies a porting cost, which has to be factored in; but making the jump to 3.5 or 3.6 is well worth it IMO.) ChrisA From steve+comp.lang.python at pearwood.info Wed May 18 23:56:21 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Thu, 19 May 2016 13:56:21 +1000 Subject: Resources/pointers for writing maintable, testable Python References: <85posic1cy.fsf@benfinney.id.au> Message-ID: <573d3967$0$2797$c3e8da3$76491128@news.astraweb.com> On Thursday 19 May 2016 13:31, Rustom Mody wrote: > On Thursday, May 19, 2016 at 6:26:26 AM UTC+5:30, Ben Finney wrote: >> Code Like A Pythonista was written in the Python 2 era >> >> but is still excellent advice today. > > > If is python-2 and still the best excellent advice today > doesn't it go somewhat counter to pyhthon-2 is a dead-end ?? :-) Ben didn't say that it is the *best* excellent advice. But even if he did, your conclusion is invalid. I still have books written for Python 1.4 and 1.5 that I turn to when I can't remember % targets and regular expression codes, and my copy of the Python Cookbook is still full of excellent code even though it is written for Python 2. > Need to point this out since the opposite case to Chris' > "switch to 3 or else suffer in hell..." needs to be articulated: > > Python-3 is nice but 2 is ok. The diffs are not such a big deal Python 2 is still an excellent language. It's just not going forward -- no new features will be added to it. -- Steve From rustompmody at gmail.com Thu May 19 00:03:58 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Wed, 18 May 2016 21:03:58 -0700 (PDT) Subject: Resources/pointers for writing maintable, testable Python In-Reply-To: <573d3967$0$2797$c3e8da3$76491128@news.astraweb.com> References: <85posic1cy.fsf@benfinney.id.au> <573d3967$0$2797$c3e8da3$76491128@news.astraweb.com> Message-ID: <33610353-cfde-4098-977b-af429f4ba514@googlegroups.com> On Thursday, May 19, 2016 at 9:26:39 AM UTC+5:30, Steven D'Aprano wrote: > On Thursday 19 May 2016 13:31, Rustom Mody wrote: > > > On Thursday, May 19, 2016 at 6:26:26 AM UTC+5:30, Ben Finney wrote: > > >> Code Like A Pythonista was written in the Python 2 era > >> > >> but is still excellent advice today. > > > > > > If is python-2 and still the best excellent advice today > > doesn't it go somewhat counter to pyhthon-2 is a dead-end ?? :-) > > Ben didn't say that it is the *best* excellent advice. > > But even if he did, your conclusion is invalid. I still have books written for > Python 1.4 and 1.5 that I turn to when I can't remember % targets and regular > expression codes, and my copy of the Python Cookbook is still full of excellent > code even though it is written for Python 2. And you have the knowledge and experience to know that when you refer to a python 2 (or 1) reference and apply it in 3 and there are minor glitches, you know where to go for clarifications. Maybe the OP also knows... maybe not... No case against python3, just that the pro-3 rhetoric need not be so shrill From best_lay at yahoo.com Thu May 19 00:17:04 2016 From: best_lay at yahoo.com (Wildman) Date: Wed, 18 May 2016 23:17:04 -0500 Subject: Extract the middle N chars of a string References: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, 19 May 2016 01:47:33 +1000, Steven D'Aprano wrote: > Is this the simplest way to get the middle N characters? This will return a sub-string of any length starting at any point. This is the way the old VB mid$ function worked. def mid(string, start, length): # start begins at 0 if string = "": return None if start > len(string) - 1: start = len(string) - 1 if length > len(string): length = len(string) if length < 1: length = 1 return string[start : start + length] -- GNU/Linux user #557453 "The Constitution only gives people the right to pursue happiness. You have to catch it yourself." -Benjamin Franklin From orgnut at yahoo.com Thu May 19 00:49:12 2016 From: orgnut at yahoo.com (Larry Hudson) Date: Wed, 18 May 2016 21:49:12 -0700 Subject: Program prints questions for user input, but won't show the answer output In-Reply-To: <276ca193-338e-4cea-9dce-1af54f900da9@googlegroups.com> References: <1cc14787-7061-45c9-a70e-1b16e3f5ef3a@googlegroups.com> <276ca193-338e-4cea-9dce-1af54f900da9@googlegroups.com> Message-ID: On 05/18/2016 06:50 PM, Jake Kobs wrote: > MRAB, > > I am not quite sure how to return the print statements so I thought that returning the displayInfo def would help.. Im so lost. > Why do you think you want to _return_ a print statement? The print statement _DOES_ the printing, there is nothing that needs to be returned. Sometimes you might want to return a _string_ to be printed where it was called from, but that's not what you want here. You say that the display info isn't shown... It looks to me like it it isn't shown because your program will crash. What you have here is called "infinite recursion": displayInfo() calls displayInfo() which calls displayInfo() which calls displayInfo() which calls ... and so on forever. Another comment: Your getHigh() and getLow() functions are not necessary. Python already has max() and min() functions built in to do this exact same thing. Instead you can use: highPints = max(pints) lowPints = min(pints) Of course, writing your own versions as a leaning experience is good too. But I would suggest that a for loop instead of a while in your version would be better. For one thing, it eliminates the counter. My suggested version... def getHigh(pints): high = 0 for pint in pints: if pint > high: high = pint return high And getLow() is similar (but set the initial value to a high number). The following example may be more advanced than you're ready for now, but you _might_ find it worth studying. It's possible to do both in a single function: def getMaxMin(pints): high = low = pints[0] # Initialize high and low to the first value of the array for pint in pints[1:]: # Scan through the remainder of the pints array if pint > high: high = pint if pint < low: low = pint return high, low You would use it as: highPints, lowPints = getMaxMin(pints) I'm NOT necessarily recommending this, but just suggesting it as an example to study. Good luck with your Python studies. It's a great language to learn. -=- Larry -=- PS. A final thought... I don't know your situation so this may not be reasonable but... I HIGHLY recommend switching to Python 3 instead of 2. Version 2 is at "end-of-life" and won't be updated any further. Version 3 is where all future development will occur. It is definitely the better version, and while there are certainly differences, to a beginning programmer the learning curve will be the same. From greg.ewing at canterbury.ac.nz Thu May 19 05:00:02 2016 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Thu, 19 May 2016 21:00:02 +1200 Subject: setrecursionlimit In-Reply-To: <573ca324$0$1604$c3e8da3$5496439d@news.astraweb.com> References: <573c941f$0$22141$c3e8da3$5496439d@news.astraweb.com> <573ca324$0$1604$c3e8da3$5496439d@news.astraweb.com> Message-ID: Steven D'Aprano wrote: > I don't really understand why the system can't track the current top of the > stack and bottom of the heap, and if they're going to collide, halt the > process. That effectively *is* what it does. The reason it manifests as a segfault is because of the way it goes about detecting the heap/stack collision. It would be very expensive to explicitly check for this every time something is pushed or popped on the stack, so what OSes typically do instead is reserve a buffer zone of unmapped memory between the stack and the heap. If the stack overflows, you end up trying to reference memory in the unmapped area, and a segfault results. This is not foolproof -- if you allocate a *really* big stack frame, you could leap right over the buffer zone and clobber the heap. But it works well enough most of the time and succeeds in stopping the program before it accidentally launches the nuclear missiles. Hardware support for stack bounds checkinbg would of course make all this easier and more reliable, but the x86 architecture doesn't provide anything like that, unfortunately. -- Greg From greg.ewing at canterbury.ac.nz Thu May 19 05:04:45 2016 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Thu, 19 May 2016 21:04:45 +1200 Subject: setrecursionlimit In-Reply-To: <337f233a-32c4-4944-96bb-731c3f56463d@googlegroups.com> References: <573c941f$0$22141$c3e8da3$5496439d@news.astraweb.com> <573ca324$0$1604$c3e8da3$5496439d@news.astraweb.com> <1116f4d7-5533-4171-90d2-69411133aa28@googlegroups.com> <337f233a-32c4-4944-96bb-731c3f56463d@googlegroups.com> Message-ID: Rustom Mody wrote: > Both the mess in catching numeric overflow as well as stackoverflow looks like > its C's fault. > I consider it as the fault of currently fashionable stock hardware The sad thing about C is that it doesn't even help you detect integer overflow in *software*. Every machine I've ever seen has a flag that gets set if an integer addition overflows. But C doesn't provide any portable way of testing that flag, even if you wanted to. -- Greg From greg.ewing at canterbury.ac.nz Thu May 19 05:13:28 2016 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Thu, 19 May 2016 21:13:28 +1200 Subject: Resources/pointers for writing maintable, testable Python In-Reply-To: References: <85posic1cy.fsf@benfinney.id.au> Message-ID: Chris Angelico wrote: > [1] Some people's names can't be represented in Unicode. Like this fellow's, for instance: https://www.youtube.com/watch?v=hNoS2BU6bbQ -- Greg From rosuav at gmail.com Thu May 19 05:26:11 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 19 May 2016 19:26:11 +1000 Subject: Resources/pointers for writing maintable, testable Python In-Reply-To: References: <85posic1cy.fsf@benfinney.id.au> Message-ID: On Thu, May 19, 2016 at 7:13 PM, Gregory Ewing wrote: > Chris Angelico wrote: >> >> [1] Some people's names can't be represented in Unicode. > > > Like this fellow's, for instance: > > https://www.youtube.com/watch?v=hNoS2BU6bbQ His name actually CAN be represented in Unicode. It's his address that I'd have trouble with. (Smart copper, by the way.) ChrisA From oscar.j.benjamin at gmail.com Thu May 19 05:27:45 2016 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 19 May 2016 10:27:45 +0100 Subject: setrecursionlimit In-Reply-To: <573c941f$0$22141$c3e8da3$5496439d@news.astraweb.com> References: <573c941f$0$22141$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 18 May 2016 at 17:11, Steven D'Aprano wrote: > The documentation for setrecursion limit warns against setting the limit too > high: > > [quote] > The highest possible limit is platform-dependent. A user may need to > set the limit higher when they have a program that requires deep > recursion and a platform that supports a higher limit. This should > be done with care, because a too-high limit can lead to a crash. > [end quote] > > https://docs.python.org/3/library/sys.html#sys.setrecursionlimit > > Indeed, if you set the recursion limit too high, you can smash the memory > heap and get a segfault. How exactly does that work? > > Why doesn't setrecursionlimit() raise an exception when you try to set it > too high? For example: > > sys.setrecursionlimit(2000000000) > > succeeds on my system, even though that's a ludicrously high number. (It is > more than half the number of bytes of memory my computer has.) > > > So why can't Python tell if I'm setting the limit too high? > > (I'm assuming that if it could, it would.) It's not altogether impossible to do this but it requires ducking underneath the C level to the machine code level. The C standard doesn't define the exact number of bytes that should be used by a stack frame. The exact number depends on the calling convention used for your OS/hardware/compiler combination and also on the optimisation/debug settings used by the compiler. A good optimiser might be able to completely eliminate your function so that it doesn't touch the stack for example. Also it depends ultimately on the code path that leads to PyEval_EvalFrameEx calling itself recursively. The recursion here is indirect and there are multiple paths for it, depending on whether the function is called with no arguments or whether it is a generator etc. Then after that you need to consider extension modules. For example a numpy array can store Python objects and perform operations on them. If we have an ndarray of Fractions then there's no way at compile time to know how much stack space ndarray.__add__ (implemented with a load of complex C code) will need before calling e.g. Fraction.__add__. Given the extension module case it's clearly impossible to compute the hardware-stack memory requirements of a Python-level frame at compile time. Christian has already explained why this wouldn't really work at runtime either. There isn't even a portable way at the C level to query the current value of the stack pointer. And even if you could you'd have to make hardware-specific assumptions to be able to use that information. My understanding (although I have no direct experience here) is that Stackless Python is an alternative implementation that gets around all of these problems by avoiding the recursion altogether. -- Oscar From nobody at nowhere.invalid Thu May 19 06:28:32 2016 From: nobody at nowhere.invalid (Nobody) Date: Thu, 19 May 2016 11:28:32 +0100 Subject: setrecursionlimit References: <573c941f$0$22141$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Wed, 18 May 2016 09:19:25 -0700, Ned Batchelder wrote: > Is there a way to know how large the C stack can grow, Yes. For the main thread, getrlimit(RLIMIT_STACK). For other threads, pthread_attr_getstacksize(). > and how much it will grow for each Python function call? No. Depending upon the interpreter implementation, the stack might not grow at all in the case where a function written in Python calls another function written in Python. Calling out into native code (which may then call back into Python code) is bound to grow the thread by some amount. While the interpreter could keep track of how much space is left on the stack, there's no way of knowing in advance how much any given C function will need. If there isn't enough, there is no robust way to recover from the resulting segfault. From kyosohma at gmail.com Thu May 19 11:33:03 2016 From: kyosohma at gmail.com (Mike Driscoll) Date: Thu, 19 May 2016 08:33:03 -0700 (PDT) Subject: Resources/pointers for writing maintable, testable Python In-Reply-To: References: Message-ID: On Wednesday, May 18, 2016 at 4:48:28 PM UTC-5, Andrew Farrell wrote: > Hi Jacob, > > You are probably looking for the book Test-Driven Development with Python > . > You'll also want to look at py.test > > Cheers! > Andrew Farrell I was under the impression that this book is primarily aimed at Python/Django web testing. I saw "Testing Python: Applying Unit Testing, TDD, BDD and Acceptance Testing" is getting good reviews too though. Mike From jacob.scott at gmail.com Thu May 19 12:10:21 2016 From: jacob.scott at gmail.com (Jacob Scott) Date: Thu, 19 May 2016 09:10:21 -0700 Subject: Resources/pointers for writing maintable, testable Python In-Reply-To: References: Message-ID: Indeed, I skimmed the TOC for Test-Driven Development with Python and it does look to be rather Django-centric (which makes it a bit less helpful to me). I will take a look at "Testing Python: Applying Unit Testing, TDD, BDD and Acceptance Testing"! Thanks, Jacob On Thu, May 19, 2016 at 8:33 AM, Mike Driscoll wrote: > On Wednesday, May 18, 2016 at 4:48:28 PM UTC-5, Andrew Farrell wrote: > > Hi Jacob, > > > > You are probably looking for the book Test-Driven Development with Python > > . > > You'll also want to look at py.test > > > > Cheers! > > Andrew Farrell > > I was under the impression that this book is primarily aimed at > Python/Django web testing. I saw "Testing Python: Applying Unit Testing, > TDD, BDD and Acceptance Testing" is getting good reviews too though. > > Mike > -- > https://mail.python.org/mailman/listinfo/python-list > From tjreedy at udel.edu Thu May 19 12:23:27 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 19 May 2016 12:23:27 -0400 Subject: Resources/pointers for writing maintable, testable Python In-Reply-To: References: Message-ID: On 5/19/2016 11:33 AM, Mike Driscoll wrote: > On Wednesday, May 18, 2016 at 4:48:28 PM UTC-5, Andrew Farrell wrote: >> Hi Jacob, >> >> You are probably looking for the book Test-Driven Development with Python >> . Electronic version is free online. > I was under the impression that this book is primarily aimed at Python/Django web testing. I saw It is. However, the first four chapters cover the general principles of TDD, so one can read them while thinking of web development as just an illustrative example. In my case, I learned better how to test IDLE from a user perspective. For tkinter apps, an external program such as Selenium is not needed. Tk/tkinter have the simulated event generation and introspection needed to simulate a user hitting keys, clicking mouse buttons, and reading the screen. -- Terry Jan Reedy From theherk at gmail.com Thu May 19 12:31:39 2016 From: theherk at gmail.com (Herkermer Sherwood) Date: Thu, 19 May 2016 09:31:39 -0700 Subject: for / while else doesn't make sense Message-ID: Most keywords in Python make linguistic sense, but using "else" in for and while structures is kludgy and misleading. I am under the assumption that this was just utilizing an already existing keyword. Adding another like "andthen" would not be good. But there is already a reserved keyword that would work great here. "finally". It is already a known keyword used in try blocks, but would work perfectly here. Best of all, it would actually make sense. Unfortunately, it wouldn't follow the semantics of try/except/else/finally. Is it better to follow the semantics used elsewhere in the language, or have the language itself make sense semantically? I think perhaps "finally" should be added to for and while to do the same thing as "else". What do you think? From kbellamary at aol.com Thu May 19 12:47:15 2016 From: kbellamary at aol.com (kbellamary at aol.com) Date: Thu, 19 May 2016 12:47:15 -0400 Subject: Python 3.5.1 Message-ID: <154c9e9fc6e-24e-1151@webprd-m75.mail.aol.com> Thisis my first encountering with Python. I have successfully downloaded Python3.5.1 for Windows but see only a black window with command prompt. I do not see IDLE under PYthon on Windows Start Menu. Downloaded version of Python is based on 32-bit and my PC is 64-bit. Couldyo please help and provide guidance? Thank you very much. Bella From ian.g.kelly at gmail.com Thu May 19 13:02:26 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 19 May 2016 11:02:26 -0600 Subject: for / while else doesn't make sense In-Reply-To: References: Message-ID: On Thu, May 19, 2016 at 10:31 AM, Herkermer Sherwood wrote: > Most keywords in Python make linguistic sense, but using "else" in for and > while structures is kludgy and misleading. I am under the assumption that > this was just utilizing an already existing keyword. Adding another like > "andthen" would not be good. "else" makes sense from a certain point of view, but I think that logic may not be communicated well. At the start of each loop iteration, the loop construct makes a test for whether the loop should continue or not. If that test ever fails (i.e. if the condition of the while loop is false), the else block is executed instead. So you can think of it as a repeated if-else where the else block has the additional effect of exiting the loop. > But there is already a reserved keyword that would work great here. > "finally". It is already a known keyword used in try blocks, but would work > perfectly here. Best of all, it would actually make sense. > > Unfortunately, it wouldn't follow the semantics of try/except/else/finally. "finally" in exception handling denotes a block that is *always* executed. Using it for a block that is only sometimes executed would dilute that meaning. From ned at nedbatchelder.com Thu May 19 13:22:16 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 19 May 2016 10:22:16 -0700 (PDT) Subject: for / while else doesn't make sense In-Reply-To: References: Message-ID: <90ac7bc8-d24c-443e-aead-9759a180ce5c@googlegroups.com> On Thursday, May 19, 2016 at 12:43:56 PM UTC-4, Herkermer Sherwood wrote: > Most keywords in Python make linguistic sense, but using "else" in for and > while structures is kludgy and misleading. I am under the assumption that > this was just utilizing an already existing keyword. Adding another like > "andthen" would not be good. > > But there is already a reserved keyword that would work great here. > "finally". It is already a known keyword used in try blocks, but would work > perfectly here. Best of all, it would actually make sense. > > Unfortunately, it wouldn't follow the semantics of try/except/else/finally. > > Is it better to follow the semantics used elsewhere in the language, or > have the language itself make sense semantically? > > I think perhaps "finally" should be added to for and while to do the same > thing as "else". What do you think? For/else has always caused people consternation. My best stab at explaining it is this: the else clause is executed if no break was encountered in the body of the for loop. A simple structure would look like this: for thing in container: if something_about(thing): # Found it! do_something(thing) break else: # Didn't find it.. no_such_thing() I think of the "else" as being paired with the "if" inside the loop. At run time, you execute a number of "if"s, one for each iteration around the loop. The "else" is what gets executed if none of the "if"s was true. In that sense, it's exactly the right keyword to use. --Ned. From steve at pearwood.info Thu May 19 13:46:53 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 20 May 2016 03:46:53 +1000 Subject: for / while else doesn't make sense References: Message-ID: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> On Fri, 20 May 2016 02:31 am, Herkermer Sherwood wrote: > Most keywords in Python make linguistic sense, but using "else" in for and > while structures is kludgy and misleading. I am under the assumption that > this was just utilizing an already existing keyword. Adding another like > "andthen" would not be good. If I could steal the keys to Guido's time machine, I would go back in time and change the for...else and while...else keywords to for...then and while...then. Alas, we missed the opportunity for a major backwards-incompatible change. Python 3.0 is long past, we're up to 3.5 now, 3.6 is in alpha, there's no way the keyword is going to be changed :-( > But there is already a reserved keyword that would work great here. > "finally". It is already a known keyword used in try blocks, but would > work perfectly here. Best of all, it would actually make sense. No. for...else doesn't operate like a finally block. The idea of finally is that it executes no matter what happens[1]. That's completely the opposite of for...else: the whole point of for...else is that "break" will jump out of the block without executing the else part. > Unfortunately, it wouldn't follow the semantics of > try/except/else/finally. Exactly. > Is it better to follow the semantics used elsewhere in the language, or > have the language itself make sense semantically? If Python was a younger language with fewer users, less existing code, and no backwards-compatibility guarantees, I would argue for changing for...else to for...then. But Python is over 20 years old, has tens or hundreds of thousands of users, tens of millions of lines of code, and quite strict backwards-compatibility guarantees. > I think perhaps "finally" should be added to for and while to do the same > thing as "else". What do you think? I think adding "finally" as an alias is just needlessly confusing. Think of the millions of people who aren't English speakers who nevertheless had to memorise weird and unintuitive words like "for", "while", "if" etc. The least we English speakers can do is suck it up and memorise *one* weird case, "else". [1] Well, *almost* no matter what. If you pull the power from the computer, the finally block never gets a chance to run. -- Steven From jon+usenet at unequivocal.co.uk Thu May 19 13:55:00 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Thu, 19 May 2016 17:55:00 -0000 (UTC) Subject: for / while else doesn't make sense References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-19, Steven D'Aprano wrote: > On Fri, 20 May 2016 02:31 am, Herkermer Sherwood wrote: >> Most keywords in Python make linguistic sense, but using "else" in for and >> while structures is kludgy and misleading. I am under the assumption that >> this was just utilizing an already existing keyword. Adding another like >> "andthen" would not be good. > > If I could steal the keys to Guido's time machine, I would go back in time > and change the for...else and while...else keywords to for...then and > while...then. I guess we should thank our lucky stars that you don't have a time machine then, since that change would very much be one for the worse in my opinion. for...else is perfectly straightforward and clearly the right keywords to use. for...then would be entirely wrong. From steve at pearwood.info Thu May 19 14:02:24 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 20 May 2016 04:02:24 +1000 Subject: for / while else doesn't make sense References: <90ac7bc8-d24c-443e-aead-9759a180ce5c@googlegroups.com> Message-ID: <573dffb3$0$1611$c3e8da3$5496439d@news.astraweb.com> On Fri, 20 May 2016 03:22 am, Ned Batchelder wrote: > On Thursday, May 19, 2016 at 12:43:56 PM UTC-4, Herkermer Sherwood wrote: >> Most keywords in Python make linguistic sense, but using "else" in for >> and while structures is kludgy and misleading. I am under the assumption >> that this was just utilizing an already existing keyword. Adding another >> like "andthen" would not be good. >> >> But there is already a reserved keyword that would work great here. >> "finally". It is already a known keyword used in try blocks, but would >> work perfectly here. Best of all, it would actually make sense. >> >> Unfortunately, it wouldn't follow the semantics of >> try/except/else/finally. >> >> Is it better to follow the semantics used elsewhere in the language, or >> have the language itself make sense semantically? >> >> I think perhaps "finally" should be added to for and while to do the same >> thing as "else". What do you think? > > For/else has always caused people consternation. > > My best stab at explaining it is this: the else clause is executed if no > break was encountered in the body of the for loop. You're describing the consequences of the break: it breaks out of the entire for statement, which includes the (badly named) "else" clause. It's also misleading: break is not the only way to skip the else clause. You can also return, or raise. As far as I can determine, if you consider all the corner cases, the best description of the behaviour is: - the "else" clause is UNCONDITIONALLY executed after the for-loop has completed, but before any code past the for...else blocks gets to run. - anything which breaks outside of the for...else will prevent execution of the else clause. returns jumps all the way out of the function, break doesn't jump out of the function but it jumps out of the combined for...else block. > A simple structure would look like this: > > for thing in container: > if something_about(thing): > # Found it! > do_something(thing) > break > else: > # Didn't find it.. > no_such_thing() > > I think of the "else" as being paired with the "if" inside the loop. The fatal flaw with that is that there is no requirement that there be any such "if" inside the loop. In fact, there's no need to even have a break in the loop at all! > At run time, you execute a number of "if"s, one for each iteration "One"? for x in sequence: if condition: break if x < 0: break if today is Tuesday: break if i.feel_like_it(): break else: print("which `if` matches this `else`?") > around the loop. The "else" is what gets executed if none of the > "if"s was true. In that sense, it's exactly the right keyword to > use. So long as you don't think too hard about what's actually going on, and only consider the most obvious case, it almost makes a bit of sense :-) But really, the else clause *actually is* an unconditional block which executes after the loop. That's how it is implemented, and that's how it is best understood. -- Steven From theherk at gmail.com Thu May 19 14:47:28 2016 From: theherk at gmail.com (theherk at gmail.com) Date: Thu, 19 May 2016 11:47:28 -0700 (PDT) Subject: for / while else doesn't make sense In-Reply-To: <90ac7bc8-d24c-443e-aead-9759a180ce5c@googlegroups.com> References: <90ac7bc8-d24c-443e-aead-9759a180ce5c@googlegroups.com> Message-ID: This is exactly what I'm referencing. We can do mental gymnastics for it to make sense, but remove the `break` in your code and what happens? The logic goes away. The code ends up executing anyway, which is what makes it more like "finally" to me. Although, as Ian pointed out that would cause breakages for the "finally" meaning, because with the break you wouldn't expect it to execute, but we expect "finally" always to execute. It is a great example for sure, but it only makes sense in specific constructs. I think this is a really interesting topic to which there may not be a better answer. On Thursday, May 19, 2016 at 10:22:31 AM UTC-7, Ned Batchelder wrote: > On Thursday, May 19, 2016 at 12:43:56 PM UTC-4, Herkermer Sherwood wrote: > > Most keywords in Python make linguistic sense, but using "else" in for and > > while structures is kludgy and misleading. I am under the assumption that > > this was just utilizing an already existing keyword. Adding another like > > "andthen" would not be good. > > > > But there is already a reserved keyword that would work great here. > > "finally". It is already a known keyword used in try blocks, but would work > > perfectly here. Best of all, it would actually make sense. > > > > Unfortunately, it wouldn't follow the semantics of try/except/else/finally. > > > > Is it better to follow the semantics used elsewhere in the language, or > > have the language itself make sense semantically? > > > > I think perhaps "finally" should be added to for and while to do the same > > thing as "else". What do you think? > > For/else has always caused people consternation. > > My best stab at explaining it is this: the else clause is executed if no > break was encountered in the body of the for loop. A simple structure > would look like this: > > for thing in container: > if something_about(thing): > # Found it! > do_something(thing) > break > else: > # Didn't find it.. > no_such_thing() > > I think of the "else" as being paired with the "if" inside the loop. > At run time, you execute a number of "if"s, one for each iteration > around the loop. The "else" is what gets executed if none of the > "if"s was true. In that sense, it's exactly the right keyword to > use. > > --Ned. From david at jardine.de Thu May 19 15:49:31 2016 From: david at jardine.de (David Jardine) Date: Thu, 19 May 2016 21:49:31 +0200 Subject: for / while else doesn't make sense In-Reply-To: References: <90ac7bc8-d24c-443e-aead-9759a180ce5c@googlegroups.com> Message-ID: <20160519194931.GA1313@gennes> On Thu, May 19, 2016 at 11:47:28AM -0700, theherk at gmail.com wrote: > This is exactly what I'm referencing. We can do mental gymnastics for it > to make sense, but remove the `break` in your code and what happens? The > logic goes away. Quite. What would the code mean? Why would you use "else" if you were always going to drop out of the end of the loop anyway? You need it in a situation where you're "hoping" to find something: meet the 6.30 train at the station, look at each passenger that gets off and if one is wearing a pink carnation in his/her buttonhole go off together and live happily ever after. But when the last passenger has got off and you haven't seen a pink carnation? That's where the "else" clause comes in. You hope you won't need it. > [...] > On Thursday, May 19, 2016 at 10:22:31 AM UTC-7, Ned Batchelder wrote: > > > > [...] > > > > For/else has always caused people consternation. > > > > My best stab at explaining it is this: the else clause is executed if no > > break was encountered in the body of the for loop. A simple structure > > would look like this: > > > > for thing in container: > > if something_about(thing): > > # Found it! > > do_something(thing) > > break > > else: > > # Didn't find it.. > > no_such_thing() > > > > I think of the "else" as being paired with the "if" inside the loop. > > At run time, you execute a number of "if"s, one for each iteration > > around the loop. The "else" is what gets executed if none of the > > "if"s was true. In that sense, it's exactly the right keyword to > > use. > > Cheers, David From rosuav at gmail.com Thu May 19 16:01:23 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 20 May 2016 06:01:23 +1000 Subject: for / while else doesn't make sense In-Reply-To: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, May 20, 2016 at 3:46 AM, Steven D'Aprano wrote: > The idea of finally is > that it executes no matter what happens[1]. > > [1] Well, *almost* no matter what. If you pull the power from the computer, > the finally block never gets a chance to run. Nor if you kill -9 the process, or get into an infinite loop, or any number of other things. Specifically, what the finally block guarantees is that it will be executed *before any code following the try block*. In this example: try: code1 except Exception: code2 else: code3 finally: code4 code5 Once you hit code1, you are absolutely guaranteed that code5 *will not* be run prior to code4. ChrisA From kyosohma at gmail.com Thu May 19 16:08:07 2016 From: kyosohma at gmail.com (Mike Driscoll) Date: Thu, 19 May 2016 13:08:07 -0700 (PDT) Subject: Python 3.5.1 In-Reply-To: References: <154c9e9fc6e-24e-1151@webprd-m75.mail.aol.com> Message-ID: On Thursday, May 19, 2016 at 12:21:52 PM UTC-5, kbell... at aol.com wrote: > Thisis my first encountering with Python. I have successfully downloaded Python3.5.1 for Windows but see only a black window with command prompt. I do not see IDLE under PYthon on Windows Start Menu. > > > Downloaded version of Python is based on 32-bit and my PC is 64-bit. > > Couldyo please help and provide guidance? > > > Thank you very much. > > > Bella You should have a folder in your Start Menu for Python. Inside of that, there should be a shortcut for IDLE. If there's not, then I'm guessing it didn't install correctly. I haven't had any problems installing Python 3.5 on my Windows PCs, although I have had issues getting it installed in certain locked down virtual environments. Try uninstalling and then reinstalling Python 3.5 Mike From kyosohma at gmail.com Thu May 19 16:10:51 2016 From: kyosohma at gmail.com (Mike Driscoll) Date: Thu, 19 May 2016 13:10:51 -0700 (PDT) Subject: Resources/pointers for writing maintable, testable Python In-Reply-To: References: Message-ID: <909ee5e9-ae16-4ebe-a6c1-c838ff3bb8fd@googlegroups.com> On Thursday, May 19, 2016 at 11:23:53 AM UTC-5, Terry Reedy wrote: > On 5/19/2016 11:33 AM, Mike Driscoll wrote: > > On Wednesday, May 18, 2016 at 4:48:28 PM UTC-5, Andrew Farrell wrote: > >> Hi Jacob, > >> > >> You are probably looking for the book Test-Driven Development with Python > >> . > > Electronic version is free online. > > > I was under the impression that this book is primarily aimed at Python/Django web testing. I saw > > It is. However, the first four chapters cover the general principles of > TDD, so one can read them while thinking of web development as just an > illustrative example. > > In my case, I learned better how to test IDLE from a user perspective. > For tkinter apps, an external program such as Selenium is not needed. > Tk/tkinter have the simulated event generation and introspection needed > to simulate a user hitting keys, clicking mouse buttons, and reading the > screen. > > -- > Terry Jan Reedy I am curious. Where is this documented? Are you referring to calling the invoke() method on each widget? Mike From ian.g.kelly at gmail.com Thu May 19 16:11:42 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 19 May 2016 14:11:42 -0600 Subject: for / while else doesn't make sense In-Reply-To: References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, May 19, 2016 at 2:01 PM, Chris Angelico wrote: > On Fri, May 20, 2016 at 3:46 AM, Steven D'Aprano wrote: >> The idea of finally is >> that it executes no matter what happens[1]. >> >> [1] Well, *almost* no matter what. If you pull the power from the computer, >> the finally block never gets a chance to run. > > Nor if you kill -9 the process, or get into an infinite loop, or any > number of other things. Specifically, what the finally block > guarantees is that it will be executed *before any code following the > try block*. In this example: > > try: > code1 > except Exception: > code2 > else: > code3 > finally: > code4 > code5 > > Once you hit code1, you are absolutely guaranteed that code5 *will > not* be run prior to code4. The guarantee is stronger than that. It's possible to exit the try block without passing execution to code5 at all. The finally block is still guaranteed to be executed in this case. From rosuav at gmail.com Thu May 19 16:27:47 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 20 May 2016 06:27:47 +1000 Subject: for / while else doesn't make sense In-Reply-To: References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, May 20, 2016 at 6:11 AM, Ian Kelly wrote: > On Thu, May 19, 2016 at 2:01 PM, Chris Angelico wrote: >> On Fri, May 20, 2016 at 3:46 AM, Steven D'Aprano wrote: >>> The idea of finally is >>> that it executes no matter what happens[1]. >>> >>> [1] Well, *almost* no matter what. If you pull the power from the computer, >>> the finally block never gets a chance to run. >> >> Nor if you kill -9 the process, or get into an infinite loop, or any >> number of other things. Specifically, what the finally block >> guarantees is that it will be executed *before any code following the >> try block*. In this example: >> >> try: >> code1 >> except Exception: >> code2 >> else: >> code3 >> finally: >> code4 >> code5 >> >> Once you hit code1, you are absolutely guaranteed that code5 *will >> not* be run prior to code4. > > The guarantee is stronger than that. It's possible to exit the try > block without passing execution to code5 at all. The finally block is > still guaranteed to be executed in this case. Yes, but I don't know how to depict "other code outside of the try block" in a way that doesn't fall foul of other nitpicks like "well, you could call that function from within the try block" :) ChrisA From marko at pacujo.net Thu May 19 16:28:26 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Thu, 19 May 2016 23:28:26 +0300 Subject: for / while else doesn't make sense References: <90ac7bc8-d24c-443e-aead-9759a180ce5c@googlegroups.com> Message-ID: <87r3cx4wt1.fsf@elektro.pacujo.net> theherk at gmail.com: > This is exactly what I'm referencing. We can do mental gymnastics for > it to make sense, Programming languages are not English. Any resemblance is purely coincidental. Marko From tjreedy at udel.edu Thu May 19 16:52:24 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 19 May 2016 16:52:24 -0400 Subject: Python 3.5.1 In-Reply-To: <154c9e9fc6e-24e-1151@webprd-m75.mail.aol.com> References: <154c9e9fc6e-24e-1151@webprd-m75.mail.aol.com> Message-ID: On 5/19/2016 12:47 PM, Bella via Python-list wrote: > > Thisis my first encountering with Python. I have successfully > downloaded Python3.5.1 for Windows but see only a black window with > command prompt. If you start Python from the Python directory on the Start menu, you should see a black window with a '>>> ' prompt. This is the interactive Python prompt. The title bar should also say 'Python'. If you open a Windows Command Prompt window, it will have a different title and a '/User/login_name>' prompt (on Win 10). > I do not see IDLE under PYthon on Windows Start Menu. Perhaps you unclicked the option to install tkinter, IDLE, and turtle. In the Python window, enter 'import tkinter' and report the result. > Downloaded version of Python is based on 32-bit and my PC is 64-bit. That is not a problem. What Windows version? If XP, 3.5 will not work. -- Terry Jan Reedy From fillmore_remove at hotmail.com Thu May 19 17:09:52 2016 From: fillmore_remove at hotmail.com (Fillmore) Date: Thu, 19 May 2016 17:09:52 -0400 Subject: Python script reading from sys.stdin and debugger Message-ID: Hello PyMasters! Long story short: cat myfile.txt | python -m pdb myscript.py doens't work (pdb hijacking stdin?). Google indicates that someone has fixed this with named pipes, but, call me stupid, I don't understand how I need to set up those pipes, how I need to modify my script and, above all, how I now need to invoke the program. Help and suggestions appreciated. I am using Python 3.4 on Cygwin and Ubuntu. Thanks From tjreedy at udel.edu Thu May 19 18:45:30 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 19 May 2016 18:45:30 -0400 Subject: Resources/pointers for writing maintable, testable Python In-Reply-To: <909ee5e9-ae16-4ebe-a6c1-c838ff3bb8fd@googlegroups.com> References: <909ee5e9-ae16-4ebe-a6c1-c838ff3bb8fd@googlegroups.com> Message-ID: On 5/19/2016 4:10 PM, Mike Driscoll wrote: > On Thursday, May 19, 2016 at 11:23:53 AM UTC-5, Terry Reedy wrote: >> In my case, I learned better how to test IDLE from a user perspective. >> For tkinter apps, an external program such as Selenium is not needed. >> Tk/tkinter have the simulated event generation and introspection needed >> to simulate a user hitting keys, clicking mouse buttons, and reading the >> screen. > I am curious. Where is this documented? Are you referring to calling > the invoke() method on each widget? For widget commands, yes. (And thanks for reminding of the method.) For events: http://infohost.nmt.edu/tcc/help/pubs/tkinter/web/universal.html (An indispensible tkinter reference) says: ''' w.event_generate(sequence, **kw) This method causes an event to trigger without any external stimulus. The handling of the event is the same as if it had been triggered by an external stimulus. The sequence argument describes the event to be triggered. You can set values for selected fields in the Event object by providing keyword=value arguments, where the keyword specifies the name of a field in the Event object. See Section 54, ?Events? for a full discussion of events. ''' This omits some essentials. tcl.tk/man/tcl8.6/TkCmd/event.htm has much more, including the need to put focus on Text and Entry. From Stackoverflow, I learned that .update() is needed *before* .event_generate. The following works. import tkinter as tk root = tk.Tk() def prt(): print('Handler called') button = tk.Button(root, text='Click', command=prt) button.place(x=20, y=20) def ev(e): print(e.x, e.y) button.bind('', ev) button.update() button.event_generate('', x=0, y=0) button.event_generate('') button.invoke() entry=tk.Entry(root) entry.place(x=20, y=50) entry.focus_force() entry.update() entry.event_generate('') The event is reported, the handler is called, and 'a' is inserted. (Inserting text could be done more easily, but some key events such as do have text equivalents.) -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Thu May 19 19:51:55 2016 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Fri, 20 May 2016 11:51:55 +1200 Subject: for / while else doesn't make sense In-Reply-To: References: Message-ID: Herkermer Sherwood wrote: > But there is already a reserved keyword that would work great here. > "finally". > > Unfortunately, it wouldn't follow the semantics of try/except/else/finally. > > Is it better to follow the semantics used elsewhere in the language, or > have the language itself make sense semantically? I think using "finally" this way would be more confusing than the status quo. Currently, if you see a "finally" somewhere, it means the block following is always executed come what may. But with this change, you would need to look up some arbitrary distance to see whether it belonged to a "try" or a "for". It's not so bad with "else" because you need to look back to find out what condition the "else" refers to anyway. -- Greg \ From steve at pearwood.info Thu May 19 20:06:12 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 20 May 2016 10:06:12 +1000 Subject: for / while else doesn't make sense References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> Message-ID: <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> On Fri, 20 May 2016 03:55 am, Jon Ribbens wrote: > I guess we should thank our lucky stars that you don't have a time > machine then, since that change would very much be one for the worse > in my opinion. for...else is perfectly straightforward and clearly > the right keywords to use. for...then would be entirely wrong. "Entirely" wrong? "Clearly" the right keyword? "Perfectly" straightforward? They are extremely strong words given the posts where *even the defenders* of "else" have admitted that it is a hard keyword to understand. But that's okay. Maybe you've thought of something the rest of us haven't, and have an entire consistent mental model of for...else that is easy to understand and makes it "perfectly straightforward and clearly the right keyword". Can you explain your model which makes "else" appropriate? In my experience, some people (including me) misunderstand "for...else" to mean that the else block runs if the for block *doesn't*. It took me the longest time to understand why this didn't work as I expected: for x in seq: pass else: print("seq is empty") because like many people, my mental model was "the for block runs, OR ELSE the else block runs". This *incorrect* model seems like it works: if you set seq=[], say, it prints "seq is empty" as expected. But its wrong: set seq=[1, 2, 3], and it *still* prints "seq is empty". My model of what was going on was faulty. I never would have thought of that model if it had been called "then": for x in seq: pass then: print("executes after the for block completes") which describes what actually happens: regardless of whether seq is empty or not, the loop runs, THEN the "else" block unconditionally executes. That has the advantage of also matching the implementation, both as byte-code, and in C. I've seen people assume that the for loop actually sets a hidden flag to record whether or not a break was executed. There is no flag. It simply doesn't get executed because code execution jumps past it. But please do explain your execution model of "for...else". If it is better than mine, I'll be happy to use it instead. -- Steven From g.starck at gmail.com Thu May 19 22:02:06 2016 From: g.starck at gmail.com (gst) Date: Thu, 19 May 2016 19:02:06 -0700 (PDT) Subject: for / while else doesn't make sense In-Reply-To: <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> Message-ID: <46a9d199-1c7c-47f7-935e-60a1eef6d378@googlegroups.com> I know this does not bring anything valuable but: Definitively agree with your mental model !! 'then' and only "then" is the best keyword in the situation which is on the table right now. it totally fix the confusing "else" actual mess (for at least 2 reasons). Python 4.0 ? My son will thank us ! NB: - I'm not a native English speaker - but I'm using the "(for,while)/else" way sometimes, but damn the "then" keyword is at least better here ! From silver0346 at gmail.com Fri May 20 01:15:19 2016 From: silver0346 at gmail.com (silver0346 at gmail.com) Date: Thu, 19 May 2016 22:15:19 -0700 (PDT) Subject: OrderedDict In-Reply-To: References: <7522e947-03d5-46e4-a74a-e5b312da47ad@googlegroups.com> Message-ID: <32f4717a-2f7a-4b31-9933-6907716e190c@googlegroups.com> On Wednesday, May 18, 2016 at 2:25:16 PM UTC+2, Peter Otten wrote: > Chris Angelico wrote: > > > On Wed, May 18, 2016 at 7:28 PM, Peter Otten <__peter__ at web.de> wrote: > >> I don't see an official way to pass a custom dict type to the library, > >> but if you are not afraid to change its source code the following patch > >> will allow you to access the value of dictionaries with a single entry as > >> d[0]: > >> > >> $ diff -u py2b_xmltodict/local/lib/python2.7/site-packages/xmltodict.py > >> py2_xmltodict/local/lib/python2.7/site-packages/xmltodict.py > >> --- py2b_xmltodict/local/lib/python2.7/site-packages/xmltodict.py > >> 2016-05-18 11:18:44.000000000 +0200 > >> +++ py2_xmltodict/local/lib/python2.7/site-packages/xmltodict.py > >> 2016-05-18 11:11:13.417665697 +0200 @@ -35,6 +35,13 @@ > >> __version__ = '0.10.1' > >> __license__ = 'MIT' > >> > >> +_OrderedDict = OrderedDict > >> +class OrderedDict(_OrderedDict): > >> + def __getitem__(self, key): > >> + if key == 0: > >> + [result] = self.values() > >> + return result > >> + return _OrderedDict.__getitem__(self, key) > >> > >> class ParsingInterrupted(Exception): > >> pass > > > > Easier than patching might be monkeypatching. > > > > class OrderedDict(OrderedDict): > > ... getitem code as above ... > > xmltodict.OrderedDict = OrderedDict > > > > Try it, see if it works. > > It turns out I was wrong on (at least) two accounts: > > - xmltodict does offer a way to specify the dict type > - the proposed dict implementation will not solve the OP's problem > > Here is an improved fix which should work: > > > $ cat sample.xml > > > > > > > $ cat sample2.xml > > > > > > > > $ cat demo.py > import collections > import sys > import xmltodict > > > class MyOrderedDict(collections.OrderedDict): > def __getitem__(self, key): > if key == 0 and len(self) == 1: > return self > return super(MyOrderedDict, self).__getitem__(key) > > > def main(): > filename = sys.argv[1] > with open(filename) as f: > doc = xmltodict.parse(f.read(), dict_constructor=MyOrderedDict) > > print "doc:\n{}\n".format(doc) > print "package-id: {}".format( > doc['profiles']['profile']['package'][0]['@package-id']) > > > if __name__ == "__main__": > main() > $ python demo.py sample.xml > doc: > MyOrderedDict([(u'profiles', MyOrderedDict([(u'profile', > MyOrderedDict([(u'@id', u'visio02'), (u'@revision', u'2015051501'), > (u'package', MyOrderedDict([(u'@package-id', u'0964-gpg4win')]))]))]))]) > > package-id: 0964-gpg4win > $ python demo.py sample2.xml > doc: > MyOrderedDict([(u'profiles', MyOrderedDict([(u'profile', > MyOrderedDict([(u'@id', u'visio02'), (u'@revision', u'2015051501'), > (u'package', [MyOrderedDict([(u'@package-id', u'0964-gpg4win')]), > MyOrderedDict([(u'@package-id', u'0965-gpg4win')])])]))]))]) > > package-id: 0964-gpg4win I have tested the first solution. Works nice. Before I used xml.etree to parse 2000 xml files. Execution time decrease from more then 5 min to 20 sec. Great. On weekend I will test the solution with the own class. Many thanks. From orgnut at yahoo.com Fri May 20 02:36:18 2016 From: orgnut at yahoo.com (Larry Hudson) Date: Thu, 19 May 2016 23:36:18 -0700 Subject: Summing/combining tuples In-Reply-To: References: Message-ID: <-vKdnYwTYL7-LaPKnZ2dnUU7-L3NnZ2d@giganews.com> On 05/18/2016 09:53 PM, DFS wrote: > On 5/18/2016 10:58 PM, Larry Hudson wrote: [snip...] >> Why two loops? Put both summations in a single loop. Then you're only >> scanning the alist once instead of twice. >> >> groups1 = defaultdict(int) >> groups2 = defaultdict(int) >> for nm, matches, words in alist: >> groups1[nm] += matches >> groups2[nm] += words >> >> -=- Larry -=- > > > That gives me two lists - how do I combine them? > > In the end that'll be at least 6 lines of code. Do you have a one-liner to summarize the data? > > Thanks > One-liner? No way. I did manage to get the whole conversion down to 6 lines... (And I am far from an expert pythonista, a more experienced/expert programmer can probably do better.) A two-line helper function A two-line for loop A one-line list comprehension One line to declare a dictionary as an intermediate variable # A helper function -- adds two tuples # Could add square brackets to return a 2-element list instead of # a tuple, but the final result is the same either way. def addtpl(t1, t2): return t1[0]+t2[0], t1[1]+t2[1] # Accumulate alist data into an intermediate temporary dictionary tdic = {} for dat in alist: tdic[dat[1]] = addtpl(tdic.get(dat[1], (0,0)), dat[2:]) # Convert the dictionary back to a list of tuples result = [(str(n), tdic[n][0], tdic[n][1]) for n in tdic] Of course, this doesn't retain the original order, but you can use the sorted() function to sort the results by the names. Here is the same thing as a single function that returns the sorted list... (It also automatically throws away the temporary dictionary. But it's now 7 lines because of the added def line.) def combine(lst): def addtpl(t1, t2): return [t1[0]+t2[0], t1[1]+t2[1] tdic = {} for dat in lst: tdic[dat[1]] = addtpl(tdic.get(dat[1], (0,0)), dat[2:]) return sorted([(str(n), tdic[n][0], tdic[n][1]) for n in tdic]) -=-Larry -=- From bob.martin at excite.com Fri May 20 07:45:31 2016 From: bob.martin at excite.com (Bob Martin) Date: Fri, 20 May 2016 07:45:31 BST Subject: for / while else doesn't make sense References: Message-ID: in 759855 20160519 185500 Jon Ribbens wrote: >On 2016-05-19, Steven D'Aprano wrote: >> On Fri, 20 May 2016 02:31 am, Herkermer Sherwood wrote: >>> Most keywords in Python make linguistic sense, but using "else" in for and >>> while structures is kludgy and misleading. I am under the assumption that >>> this was just utilizing an already existing keyword. Adding another like >>> "andthen" would not be good. >> >> If I could steal the keys to Guido's time machine, I would go back in time >> and change the for...else and while...else keywords to for...then and >> while...then. > >I guess we should thank our lucky stars that you don't have a time >machine then, since that change would very much be one for the worse >in my opinion. for...else is perfectly straightforward and clearly >the right keywords to use. for...then would be entirely wrong. Yes. "else" and "then" have opposite meanings. From me+python at ixokai.io Fri May 20 02:53:57 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Thu, 19 May 2016 23:53:57 -0700 Subject: for / while else doesn't make sense In-Reply-To: <46a9d199-1c7c-47f7-935e-60a1eef6d378@googlegroups.com> References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> <46a9d199-1c7c-47f7-935e-60a1eef6d378@googlegroups.com> Message-ID: <1463727237.686008.613428553.209A17B8@webmail.messagingengine.com> On Thu, May 19, 2016, at 07:02 PM, gst wrote: > Python 4.0 ? My son will thank us ! No, he won't, because while Python 4.0 will happen, likely after Python 3.9, it will not be a major backwards compatible breaking point. Some people infer that because 3.0 was, 4.0, 5.0, 6.0 are open to it. They aren't. -- Stephen Hansen m e @ i x o k a i . i o From cloverobert at gmail.com Fri May 20 03:30:39 2016 From: cloverobert at gmail.com (Robert Clove) Date: Fri, 20 May 2016 13:00:39 +0530 Subject: JNLP File download and run Message-ID: Hi, Can someone give me pseudo code to download and JNLP file from a URL and run it? Looks like a advance concept in python Regards From python at lucidity.plus.com Fri May 20 04:09:19 2016 From: python at lucidity.plus.com (Erik) Date: Fri, 20 May 2016 09:09:19 +0100 Subject: for / while else doesn't make sense In-Reply-To: References: Message-ID: <573EC62F.4090401@lucidity.plus.com> On 20/05/16 00:51, Gregory Ewing wrote: > It's not so bad with "else" because you need to look back > to find out what condition the "else" refers to anyway. With my tongue only slightly in my cheek, if it was desirable to "fix"/clarify this syntax then I would suggest adding some optional (existing) trailing keywords to 'else' in this context that spells it out: for item in seq: if foo(item): break else if not break: nomatch() I would rule out "elif not break" - this is just about adding additional trailing words to existing syntax to spell it out (which might be a good teaching aid as well as turn into personal preferred style). I guess that it _could_ be a syntax error to reference "break" if the for/while loop does not contain the keyword in its body - this could actually address one of the "confusing" things mentioned in this thread, like: seq = [] for item in seq: pass else if not break: pass # Syntax error - "break" referenced in "for/else" clause but not present in loop body. Someone also pointed out that there are other flow control mechanisms that could prevent the 'else' from being executed: "return" and "raise". One could extend the above to allow one or more of those to be specified: else if not break or return or raise: That looks like a lot of noise, but again it could help make code more explicit (both to the human reader, and to a compile-time check): for item in seq: if foo(item): break if bar(item): return else if not return: pass # Syntax error - extended "for/else" clause does not reference "break", which exists in loop body. I would _not_ suggest that the above should ever mean "if it doesn't return, this code is executed regardless of whether 'break' happened" - one would remove the 'else' clause altogether for that. "raise" is more problematic as an exception can always be raised by pretty much _anything_ in the body without the keyword being present. Perhaps "or raise" is just a completely benign, optional keyword for the completists. Or perhaps it's simply not mentioned at all and is always implied (as is true of exceptions in other constructs such as plain "if/else", and just generally). The additional keywords would effectively just be a static checked filter - the original bare "else" would effectively mean "else if not any_flow_control_out_of_the_loop" (which is what it means today). In summary: Adding the "if not" extension is the suggestion, with "break" as the filter. "return" as an additional filter is a possibility which may add something, "raise" as an additional filter is a possibility which probably doesn't add anything (and may actually be distracting). I'm not entirely sure if _I_ like this yet (there are some things I've suggested that I like and some I'm not sure about and some I don't like but I've mentioned them anyway) - I'm just throwing it out there ;) E. From rocky at gnu.org Fri May 20 04:31:01 2016 From: rocky at gnu.org (rocky) Date: Fri, 20 May 2016 01:31:01 -0700 (PDT) Subject: Python script reading from sys.stdin and debugger In-Reply-To: References: Message-ID: On Thursday, May 19, 2016 at 5:10:08 PM UTC-4, Fillmore wrote: > Hello PyMasters! > > Long story short: > > cat myfile.txt | python -m pdb myscript.py > > doens't work (pdb hijacking stdin?). > > Google indicates that someone has fixed this with named pipes, but, call > me stupid, I don't understand how I need to set up those pipes, how I > need to modify my script and, above all, how I now need to invoke the > program. > > Help and suggestions appreciated. I am using Python 3.4 on Cygwin and > Ubuntu. > > Thanks The debugger trepan3k follows the gdb command set so it gdb's "source" command. See http://python2-trepan.readthedocs.io/en/latest/commands/support/source.html or type "help source" inside the debugger. To install trepan3k "easy_install trepan3k" (I'm not totally sure pip install works yet). See https://pypi.python.org/pypi/trepan3k/ for mor details. From rocky at gnu.org Fri May 20 04:39:25 2016 From: rocky at gnu.org (rocky) Date: Fri, 20 May 2016 01:39:25 -0700 (PDT) Subject: Call to Python backtrace listing packages - show the exact location Message-ID: <6f36619d-5900-4309-8d86-dda583d96cae@googlegroups.com> A little while ago I wrote uncompyle6 which can deparse Python C bytecode. Currently it runs on 2.6-2.7 and 3.2 and up. I think an underused part of that is that you can at runtime give it a bytecode offset and it will show you where inside a line you are at. It also can show the surrounding expression/statement of the parent context. I think it would be neat if those packages which show backtraces offered the ability to also show the expression at the point of error as well. My trepan debuggers in fact use this in their "deparse" statement. http://python2-trepan.readthedocs.io/en/latest/commands/data/deparse.html Pypy package https://pypi.python.org/pypi?name=uncompyle6 From jon+usenet at unequivocal.co.uk Fri May 20 07:55:34 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Fri, 20 May 2016 11:55:34 -0000 (UTC) Subject: for / while else doesn't make sense References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-20, Steven D'Aprano wrote: > On Fri, 20 May 2016 03:55 am, Jon Ribbens wrote: >> I guess we should thank our lucky stars that you don't have a time >> machine then, since that change would very much be one for the worse >> in my opinion. for...else is perfectly straightforward and clearly >> the right keywords to use. for...then would be entirely wrong. > > "Entirely" wrong? "Clearly" the right keyword? "Perfectly" straightforward? > > They are extremely strong words given the posts where *even the defenders* > of "else" have admitted that it is a hard keyword to understand. But that's > okay. Maybe you've thought of something the rest of us haven't, and have an > entire consistent mental model of for...else that is easy to understand and > makes it "perfectly straightforward and clearly the right keyword". > > Can you explain your model which makes "else" appropriate? Certainly. "else:" is (almost?) invariably used in the situation where you are iterating through something in order to find a value which matches a certain condition. So the "for:" block means "find this value" and the "else:" means "else do this". > In my experience, some people (including me) misunderstand "for...else" to > mean that the else block runs if the for block *doesn't*. It took me the > longest time to understand why this didn't work as I expected: > > for x in seq: > pass > else: > print("seq is empty") > > because like many people, my mental model was "the for block runs, OR ELSE > the else block runs". This *incorrect* model seems like it works: if you > set seq=[], say, it prints "seq is empty" as expected. > > But its wrong: set seq=[1, 2, 3], and it *still* prints "seq is empty". My > model of what was going on was faulty. The problem there is that the mental model is *completely* wrong. "else:" has nothing at all to do with the number of values in the iterated sequence. > I never would have thought of that model if it had been called "then": > > for x in seq: > pass > then: > print("executes after the for block completes") I would find that very confusing. "then:" makes it sound like executing that block is the usual case, when in practice it is usually the exception - the fallback code if the expected value was not found. From mal at europython.eu Fri May 20 08:18:57 2016 From: mal at europython.eu (M.-A. Lemburg) Date: Fri, 20 May 2016 14:18:57 +0200 Subject: EuroPython 2016 Keynote: Jameson Rollins Message-ID: <573F00B1.1000207@europython.eu> We are pleased to introduce our second keynote speaker for EuroPython 2016: *** Jameson Rollins *** About Jameson Rollins --------------------- Jameson is a staff scientist in the LIGO project, based at the California Institute of Technology: "I have worked on many aspects of gravitational wave detection over the years, from laser light sources, to algorithms for low-latency data analysis. I?m currently interested in problems of detector control, and am the developer of the LIGO automation system. I have a B.S. in physics from the University of Michigan, and a Ph.D. in physics from Columbia University in the City of New York." The Keynote: LIGO - The Dawn of Gravitational Wave Astronomy ------------------------------------------------------------ Scientists have been searching for the elusive gravitational wave for more than half a century. On September 14, 2015, the Laser Interferometer Gravitational-wave Observatory (LIGO) finally observed the gravitational wave signature from the merger of two black holes. * https://www.ligo.caltech.edu/detection This detection marks the dawn of a new age of gravitational wave astronomy, where we routinely hear the sounds emanating from deep within the most energetic events in the Universe. * https://en.wikipedia.org/wiki/Gravitational-wave_astronomy This talk will cover the events leading up to one of the most important discoveries of the last century, and the myriad of ways in which Python enabled the effort. With gravitational regards, -- EuroPython 2016 Team http://ep2016.europython.eu/ http://www.europython-society.org/ From kyosohma at gmail.com Fri May 20 10:06:02 2016 From: kyosohma at gmail.com (Mike Driscoll) Date: Fri, 20 May 2016 07:06:02 -0700 (PDT) Subject: Wanted Python programmer to join team In-Reply-To: <573aaa28$0$1512$c3e8da3$5496439d@news.astraweb.com> References: <7c1497d29c7146328545aa918522f986@seaexchmbx03.olympus.F5Net.com> <573a83f4$0$22140$c3e8da3$5496439d@news.astraweb.com> <573aaa28$0$1512$c3e8da3$5496439d@news.astraweb.com> Message-ID: <2ba81c4b-4134-4f26-8f66-749efda19675@googlegroups.com> On Tuesday, May 17, 2016 at 12:20:53 AM UTC-5, Steven D'Aprano wrote: > On Tuesday 17 May 2016 12:56, Chris Angelico wrote: > > > On Tue, May 17, 2016 at 12:37 PM, Steven D'Aprano <> > > wrote: > >> On Tue, 17 May 2016 09:07 am, Chris Angelico wrote: > >> > >>> I'm not overly bothered by the use of GMail for a business address, > >> > >> It's 2016. Using a gmail address for your business (unless you're a really > >> small business, like a sole trader or something) is equivalent to a postal > >> address of "Leave mail with the lady in the milk bar on the corner". > >> > >> It's not hard to run your own mail server. *I* can do it. At the very least, > >> register a domain and tell Gmail to use that, and *pretend* you're running > >> your own mail server. > > > > And a lot of job postings do come from that sort of really small > > business, trying to expand a bit. Plus, some of them want some > > anonymity (why, I don't know, but there are plenty of jobs posted > > without too much in the way of company details) > > That probably means the job advert is coming from a recruiter. They don't want > people to contact the company directly, and they want to hide the fact that > they are a recruiter. > > Personally, I think that advertising a job position without saying who you are, > what you do, and offering at least an indicative salary range, are > *astonishingly* rude (to say nothing of counter-productive). If I see a job for > (let's say) Blackwater[1], paying $900,000 a year, then I know that (1) I don't > want to work for them, and (2) even if I did, I wouldn't be qualified; so I > don't waste either my time or theirs applying. But when I see a job for some > unnamed company with an unknown salary doing something often couched in the > vaguest possible terms, I end up wasting everyone's time. +1 I have had recruiters from within Company A bug me about their company, but when I asked about a salary range, they said that they wouldn't discuss that until a later stage but that I would be happy with it. How would they know? They don't know what I make or what would make me happy! I ignored them after that even though I was interested in working for Company A. They do a lot of weird things. Frankly the ones that look like they spam everyone provide more information than the more professional recruiters. - Mike From michael.selik at gmail.com Fri May 20 11:26:59 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 20 May 2016 15:26:59 +0000 Subject: for / while else doesn't make sense In-Reply-To: References: Message-ID: On Thu, May 19, 2016 at 1:04 PM Ian Kelly wrote: > On Thu, May 19, 2016 at 10:31 AM, Herkermer Sherwood > wrote: > > Most keywords in Python make linguistic sense, but using "else" in for > and > > while structures is kludgy and misleading. I am under the assumption that > > this was just utilizing an already existing keyword. Adding another like > > "andthen" would not be good. > > "else" makes sense from a certain point of view, but I think that > logic may not be communicated well. At the start of each loop > iteration, the loop construct makes a test for whether the loop should > continue or not. If that test ever fails (i.e. if the condition of the > while loop is false), the else block is executed instead. So you can > think of it as a repeated if-else where the else block has the > additional effect of exiting the loop. > > > But there is already a reserved keyword that would work great here. > > "finally". It is already a known keyword used in try blocks, but would > work > > perfectly here. Best of all, it would actually make sense. > > > > Unfortunately, it wouldn't follow the semantics of > try/except/else/finally. > > "finally" in exception handling denotes a block that is *always* > executed. Using it for a block that is only sometimes executed would > dilute that meaning. > It's unfortunate that so many people responded so quickly, since Ian's explanation was so clear (I thought). For further clarity, I'll write out the implicit if-statement that Ian described, though in my case it'd be at the end of the block, somewhat like a do-while: IF keep_looping() THEN GOTO LOOP_START ELSE GOTO LOOP_COMPLETED Also, Nick Coghlan has a good post about this ( http://python-notes.curiousefficiency.org/en/latest/python_concepts/break_else.html ) From torriem at gmail.com Fri May 20 11:36:53 2016 From: torriem at gmail.com (Michael Torrie) Date: Fri, 20 May 2016 09:36:53 -0600 Subject: JNLP File download and run In-Reply-To: References: Message-ID: On 05/20/2016 01:30 AM, Robert Clove wrote: > Hi, > > Can someone give me pseudo code to download and JNLP file from a URL and > run it? > > Looks like a advance concept in python You could use the urllib module to download the file, then use the subprocess module to spawn the javaws executable and pass it the location of the jnlp file as a parameter. https://docs.python.org/3.6/howto/urllib2.html https://docs.python.org/3.6/library/subprocess.html There are other ways besides launching javaws directly, such as asking cmd.exe to invoke "start" and the jnlp file so that the default javaws executable will run. On Mac there's the "open" binary that can do the same thing, and on Linux, xdg-open. From zachary.ware+pylist at gmail.com Fri May 20 11:59:59 2016 From: zachary.ware+pylist at gmail.com (Zachary Ware) Date: Fri, 20 May 2016 10:59:59 -0500 Subject: for / while else doesn't make sense In-Reply-To: <573EC62F.4090401@lucidity.plus.com> References: <573EC62F.4090401@lucidity.plus.com> Message-ID: On Fri, May 20, 2016 at 3:09 AM, Erik wrote: > On 20/05/16 00:51, Gregory Ewing wrote: >> >> It's not so bad with "else" because you need to look back >> to find out what condition the "else" refers to anyway. > > > With my tongue only slightly in my cheek, if it was desirable to > "fix"/clarify this syntax then I would suggest adding some optional > (existing) trailing keywords to 'else' in this context that spells it out: > > for item in seq: > if foo(item): > break > else if not break: > nomatch() With tongue firmly cheeked, you can always use the special `:#` operator: for item in seq: if foo(item): break else:# if no break: nomatch() This has the benefit that you can use whatever syntax you like after the `:#`, and use it in any version of Python you want. -- Zach From christopher_reimer at icloud.com Fri May 20 14:50:37 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Fri, 20 May 2016 11:50:37 -0700 Subject: How do I subclass the @property setter method? Message-ID: <2200dfd0-7470-0c86-c015-a6a14a2f51f2@icloud.com> Greetings, My chess engine has a Piece class with the following methods that use the @property decorator to read and write the position value. @property def position(self): return self._position @position.setter def position(self, position): if self._first_move: self._first_move = False self._position = position Blank is a subclass of the Piece class that represents an empty space on the board and is a placeholder in the Board class _state dict. Since Blank is a placeholder and not a playable game piece, I want to make @position.setter non-operational (i.e, make no changes to the position value). @Piece.position.setter def position(self, position): pass This code works and the corresponding unit test blows up because I haven't changed it yet, but the PyCharm IDE complains that the Blank.position signature doesn't match the Piece.position signature. Never mind that I copy and paste the identical declaration from the Piece class. This code does work, blows up the unit test, and keeps PyCharm happy. @property def position(self): return super().position @position.setter def position(self, position): pass Re-declaring @property and calling super seems redundant. Not sure if I found a bug with the PyCharm hint feature or I'm not subclassing the @property setter correctly. Which is it? Thank you, Chris R. From christopher_reimer at icloud.com Fri May 20 15:20:51 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Fri, 20 May 2016 12:20:51 -0700 Subject: for / while else doesn't make sense In-Reply-To: References: <573EC62F.4090401@lucidity.plus.com> Message-ID: On 5/20/2016 8:59 AM, Zachary Ware wrote: > On Fri, May 20, 2016 at 3:09 AM, Erik wrote: >> On 20/05/16 00:51, Gregory Ewing wrote: >>> It's not so bad with "else" because you need to look back >>> to find out what condition the "else" refers to anyway. >> >> With my tongue only slightly in my cheek, if it was desirable to >> "fix"/clarify this syntax then I would suggest adding some optional >> (existing) trailing keywords to 'else' in this context that spells it out: >> >> for item in seq: >> if foo(item): >> break >> else if not break: >> nomatch() > With tongue firmly cheeked, you can always use the special `:#` operator: > > for item in seq: > if foo(item): > break > else:# if no break: > nomatch() > > This has the benefit that you can use whatever syntax you like after > the `:#`, and use it in any version of Python you want. According to "Effective Python: 59 Specific Ways to Write Better Python" by Brett Slatkin, Item 12 recommends against using the else block after for and while loops (see page 25): "Avoid using else blocks after loops because their behavior isn't intuitive and can be confusing." Until I read the book, I wasn't aware of this feature (or bug). Doesn't seem like a feature I would use since it's not commonly found in other programming languages. As the author demonstrates in his book, I would probably write a helper function instead. Item 13 does recommend using the else block for try/except/else/finally in exception handling. :) Thank you, Chris R. From christopher_reimer at icloud.com Fri May 20 16:09:36 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Fri, 20 May 2016 13:09:36 -0700 Subject: How do I subclass the @property setter method? In-Reply-To: <2200dfd0-7470-0c86-c015-a6a14a2f51f2@icloud.com> References: <2200dfd0-7470-0c86-c015-a6a14a2f51f2@icloud.com> Message-ID: <4fb93ec4-e988-cf36-1d53-9948cce0f176@icloud.com> On 5/20/2016 11:50 AM, Christopher Reimer wrote: > This code does work, blows up the unit test, and keeps PyCharm happy. > > @property > def position(self): > return super().position > > @position.setter > def position(self, position): > pass > > Re-declaring @property and calling super seems redundant. Not sure if > I found a bug with the PyCharm hint feature or I'm not subclassing the > @property setter correctly. Which is it? Never mind. This is a known bug for PyCharm IDE. https://youtrack.jetbrains.com/issue/PY-12803 I sent a separate email to technical support to inquire if this bug and similar bugs will ever get fixed. This issue was initially reported three years ago. Not sure if I should post my own bug report. Thank you, Chris R. From saxri89 at gmail.com Fri May 20 16:24:56 2016 From: saxri89 at gmail.com (Xristos Xristoou) Date: Fri, 20 May 2016 13:24:56 -0700 (PDT) Subject: errors with QGIS API USING PYTHON Message-ID: i want to use python API from the QGIS in my python idle out from QGIS program , but i have some errors if i try to import qgis.core in my idle. i have python 2.7 64 bit (my version) and i install QGIS via OSGeo4W64. first i try to set two PATHS PYTHONPATH = C:\OSGeo4W64\apps\qgis\pythonPYTHONPATH(because there is the python python) and set PATH=C:\OSGeo4W64\apps\qgis\binPATH(because there is the python modules modules).now if i try to import qgis.core then show me error message no module name siperror in idle after this error i try to import paths inside the idle via sys.path.extend import sys sys.path.extend([r"C:\OSGeo4W64\apps\qgis\python", r"C:\OSGeo4W64\apps\qgis\bin", r"C:\OSGeo4W64\bin", r"C:\OSGeo4W64\apps\qgis\bin", r"C:\OSGeo4W64\apps\qgis\python", r"C:\OSGeo4W64\apps\Python27", r"C:\OSGeo4W64\apps\Python27\Scripts", r"C:\OSGeo4W64\apps\Python27\sip", r"C:\OSGeo4W64\apps\Python27\sip\Qt", r"C:\OSGeo4W64\apps\Python27\sip\QtCore", r"C:\OSGeo4W64\apps\Python27\sip\Qwt5", r"C:\OSGeo4W64\apps\Python27\sip\QtScriptTools", r"C:\OSGeo4W64\apps\Python27\DLLs", r"C:\OSGeo4W64\apps\Python27\Lib", r"C:\OSGeo4W64\apps\Python27\Lib\site-packages", r"C:\OSGeo4W64\apps\Python27\Tools\Scripts", r"C:\OSGeo4W64\apps\qgis\plugins", r"C:\OSGeo4W64\apps\qgis\python\qsci_apis", r"C:\OSGeo4W64\apps\qgis\python\qgis", r"C:\OSGeo4W64\apps\qgis\python\qgis\core", r"C:\OSGeo4W64\apps\qgis\python\qgis\analysis", r"C:\OSGeo4W64\apps\qgis\python\PyQt", r"C:\OSGeo4W64\apps\qgis\python\PyQt4", r"C:\OSGeo4W64\apps\qgis\python\PyQt4\uic\widget-plugins", r"C:\OSGeo4W64\apps\qgis\bin", r"C:\Python27", r"C:\Python27\Scripts", r"C:\OSGeo4W64\apps\qgis\python"]) i use more paths for sure but now show me DLL error load failed error 2 finaly i try other method from this bat file(question). my bat file. set PYTHONPATH=C:\OSGeo4W64\apps\qgis\python set PATH=C:\OSGeo4W64\apps\qgis\bin;C:\OSGeo4W64\bin;C:\OSGeo4W64\apps\qgis\bin;C:\OSGeo4W64\apps\qgis\python;C:\OSGeo4W64\apps\Python27;C:\OSGeo4W64\apps\Python27\Scripts;C:\OSGeo4W64\apps\Python27\sip;C:\OSGeo4W64\apps\Python27\sip\Qt;C:\OSGeo4W64\apps\Python27\sip\QtCore;C:\OSGeo4W64\apps\Python27\sip\Qwt5;C:\OSGeo4W64\apps\Python27\sip\QtScriptTools;C:\OSGeo4W64\apps\Python27\DLLs;C:\OSGeo4W64\apps\Python27\Lib;C:\OSGeo4W64\apps\Python27\Lib\site-packages;C:\OSGeo4W64\apps\Python27\Tools\Scripts;C:\OSGeo4W64\apps\qgis\plugins;C:\OSGeo4W64\apps\qgis\python\qsci_apis;C:\OSGeo4W64\apps\qgis\python\qgis;C:\OSGeo4W64\apps\qgis\python\qgis\core;C:\OSGeo4W64\apps\qgis\python\qgis\analysis;C:\OSGeo4W64\apps\qgis\python\PyQt;C:\OSGeo4W64\apps\qgis\python\PyQt4;C:\OSGeo4W64\apps\qgis\python\PyQt4\uic\widget-plugins;C:\OSGeo4W64\apps\qgis\bin;C:\Python27\;C:\Python27\Scripts if i run this bat file in the OSGeo4w shell work but only in the shell shell image. but not work again in my idle again show me error no module name SIPerror sip why?any idea how to fix that? the problem is the paths sure but i dont know how to fix that.err From saxri89 at gmail.com Fri May 20 17:39:15 2016 From: saxri89 at gmail.com (Xristos Xristoou) Date: Fri, 20 May 2016 14:39:15 -0700 (PDT) Subject: python for complte program ? Message-ID: hello i want to ask if a python is a good for to do it a complete program(.exe)? with user interface,module interface,background scripts where compile if the users calls some scripts,windows with interaction with the users? how can i do that with the python? how can libs need for this ? some tutorial for complete programs ? From joel.goldstick at gmail.com Fri May 20 18:03:54 2016 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Fri, 20 May 2016 18:03:54 -0400 Subject: python for complte program ? In-Reply-To: References: Message-ID: On Fri, May 20, 2016 at 5:39 PM, Xristos Xristoou wrote: > hello i want to ask if a python is a good for to do it a complete program(.exe)? > with user interface,module interface,background scripts where compile if > the users calls some scripts,windows with interaction with the users? > how can i do that with the python? > how can libs need for this ? > some tutorial for complete programs ? > -- > https://mail.python.org/mailman/listinfo/python-list There are some tools that will let you package your program with python, but if your user has python on his system, you may not need to do this -- Joel Goldstick http://joelgoldstick.com/blog http://cc-baseballstats.info/stats/birthdays From steve at pearwood.info Fri May 20 18:43:44 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 21 May 2016 08:43:44 +1000 Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> Message-ID: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> On Sat, 21 May 2016 05:20 am, Christopher Reimer wrote: > According to "Effective Python: 59 Specific Ways to Write Better Python" > by Brett Slatkin, Item 12 recommends against using the else block after > for and while loops (see page 25): "Avoid using else blocks after loops > because their behavior isn't intuitive and can be confusing." By that logic, we ought to: - avoid using floats because their behaviour isn't intuitive and can be confusing; - avoid using lists because their behaviour isn't intuitive and can be confusing; - avoid using classes because their behaviour isn't intuitive and can be confusing; - avoid any form of asynchronous functions because their behaviour isn't intuitive and can be confusing; and so on. I can give examples of unintuitive and confusing behaviour for all of those things, and more, except the last. And the reason I can't give examples for async programming is because it confuses me and I don't understand it well enough to give even a minimal example. Just about the only things in Python which are intuitive and not confusing to somebody are None and ints. Or, we can *learn how to use the features* and stop thinking that programming is a matter of intuition. And most importantly, stop thinking that features need to be judged entirely by the least knowledgeable programmers. > Until I read the book, I wasn't aware of this feature (or bug). Doesn't > seem like a feature I would use since it's not commonly found in other > programming languages. As the author demonstrates in his book, I would > probably write a helper function instead. Sorry, was that called "Ineffective Python"? There is absolutely nothing effective about re-inventing the wheel badly or writing unnecessary code. Are you programming in those other languages or in Python? If you're programming in, say, Javascript, I can completely understand you deciding to limit yourself to features available in Javascript. Indeed to try to use Python language features in Javascript would be an exercise in frustration. Likewise using Ruby language features in Python is nothing but SyntaxError after SyntaxError, it makes it hard to get work done. But the idea that you should avoid a Python feature while programming in Python because Javascript doesn't have it, or Ruby, or C, is surely the height of muddleheaded thinking. You're not programming Javascript, Ruby or C, you're programming in Python. The whole point of picking one language over another is to get access to the tools and features that language offers. Otherwise you're just wasting your time. -- Steven From theherk at gmail.com Fri May 20 19:24:10 2016 From: theherk at gmail.com (theherk at gmail.com) Date: Fri, 20 May 2016 16:24:10 -0700 (PDT) Subject: for / while else doesn't make sense In-Reply-To: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> Message-ID: You seem to have missed the point. Nobody is suggesting, I don't believe, that all of a language should be intuitive. Rather that if any part of it is unnecessarily counter-intuitive, it may be worth looking for a better solution. Python is a very well designed language when it comes to in linguistic presentation. In this case however, it is not only unintuitive but counter-intuitive. The original question was simply, "Is it better to follow the semantics used elsewhere in the language, or have the language itself make sense semantically?" So, it is better to leave counter-intuitive constructs so they are consistent across the language or try to make each keyword make semantic sense where it is used? On Friday, May 20, 2016 at 3:43:58 PM UTC-7, Steven D'Aprano wrote: > On Sat, 21 May 2016 05:20 am, Christopher Reimer wrote: > > > According to "Effective Python: 59 Specific Ways to Write Better Python" > > by Brett Slatkin, Item 12 recommends against using the else block after > > for and while loops (see page 25): "Avoid using else blocks after loops > > because their behavior isn't intuitive and can be confusing." > > By that logic, we ought to: > > - avoid using floats because their behaviour isn't intuitive and > can be confusing; > - avoid using lists because their behaviour isn't intuitive and > can be confusing; > - avoid using classes because their behaviour isn't intuitive and > can be confusing; > - avoid any form of asynchronous functions because their behaviour > isn't intuitive and can be confusing; > > and so on. I can give examples of unintuitive and confusing behaviour for > all of those things, and more, except the last. And the reason I can't give > examples for async programming is because it confuses me and I don't > understand it well enough to give even a minimal example. > > Just about the only things in Python which are intuitive and not confusing > to somebody are None and ints. > > Or, we can *learn how to use the features* and stop thinking that > programming is a matter of intuition. And most importantly, stop thinking > that features need to be judged entirely by the least knowledgeable > programmers. > > > > Until I read the book, I wasn't aware of this feature (or bug). Doesn't > > seem like a feature I would use since it's not commonly found in other > > programming languages. As the author demonstrates in his book, I would > > probably write a helper function instead. > > Sorry, was that called "Ineffective Python"? > > There is absolutely nothing effective about re-inventing the wheel badly or > writing unnecessary code. > > Are you programming in those other languages or in Python? If you're > programming in, say, Javascript, I can completely understand you deciding > to limit yourself to features available in Javascript. Indeed to try to use > Python language features in Javascript would be an exercise in frustration. > Likewise using Ruby language features in Python is nothing but SyntaxError > after SyntaxError, it makes it hard to get work done. > > But the idea that you should avoid a Python feature while programming in > Python because Javascript doesn't have it, or Ruby, or C, is surely the > height of muddleheaded thinking. You're not programming Javascript, Ruby or > C, you're programming in Python. The whole point of picking one language > over another is to get access to the tools and features that language > offers. Otherwise you're just wasting your time. > > > > -- > Steven From ethan at stoneleaf.us Fri May 20 19:58:14 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 20 May 2016 16:58:14 -0700 Subject: for / while else doesn't make sense In-Reply-To: References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> Message-ID: <573FA496.5050708@stoneleaf.us> On 05/20/2016 04:55 AM, Jon Ribbens wrote: > On 2016-05-20, Steven D'Aprano wrote: >> On Fri, 20 May 2016 03:55 am, Jon Ribbens wrote: >>> I guess we should thank our lucky stars that you don't have a time >>> machine then, since that change would very much be one for the worse >>> in my opinion. for...else is perfectly straightforward and clearly >>> the right keywords to use. for...then would be entirely wrong. >> >> "Entirely" wrong? "Clearly" the right keyword? "Perfectly" straightforward? >> >> They are extremely strong words given the posts where *even the defenders* >> of "else" have admitted that it is a hard keyword to understand. But that's >> okay. Maybe you've thought of something the rest of us haven't, and have an >> entire consistent mental model of for...else that is easy to understand and >> makes it "perfectly straightforward and clearly the right keyword". >> >> Can you explain your model which makes "else" appropriate? > > Certainly. "else:" is (almost?) invariably used in the situation where > you are iterating through something in order to find a value which > matches a certain condition. So the "for:" block means "find this > value" and the "else:" means "else do this". I'm happy that you have a working mental model for for/else (seriously, I am) but please don't discount the confusion and consternation for the many folks who don't start with that mental model. The number of times I have /wanted/ to use the for/else structure for searching is small (and I remember them both ;) -- every other time what I wanted was an _else_ that ran iff the iterable was already empty when the _for_ encountered it. >> because like many people, my mental model was "the for block runs, OR ELSE >> the else block runs". This *incorrect* model seems like it works: if you >> set seq=[], say, it prints "seq is empty" as expected. > The problem there is that the mental model is *completely* wrong. D'oh. Completely wrong, but easy to guess because of the similarity with if/else. >> I never would have thought of that model if it had been called "then": >> >> for x in seq: >> pass >> then: >> print("executes after the for block completes") > > I would find that very confusing. "then:" makes it sound like > executing that block is the usual case, when in practice it is > usually the exception - the fallback code if the expected value > was not found. If you don't take the extra step of _break_ it is the usual case. Most of my for loops always run all the way through, so why have an else? Two possible reasons: - the loop didn't run at all - the loop didn't run all the way Guido picked the one that was obvious to him (him being Dutch and all ;) . I just read Nick's post about it, and while it helps, I think the syntactic similarity between try/else and for/else (did the block run without error) is dwarfed by the semantic difference: a try/else else block runs if nothing /bad/ happened whereas a for/else else block runs if something bad /did/ happen; to wit, the thing you were looking for was not found (or your loop didn't run at all ;) . But as others have said, this isn't going to change now, and I'm okay with that. But, please, be a bit more understanding of those who don't immediately grok the for/else and while/else loops. -- ~Ethan~ From ben+python at benfinney.id.au Fri May 20 20:21:34 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 21 May 2016 10:21:34 +1000 Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> Message-ID: <85d1ogb6r5.fsf@benfinney.id.au> Steven D'Aprano writes: > Just about the only things in Python which are intuitive and not > confusing to somebody are None and ints. I'll go even further: * The behaviour of ?int? is confusing to some. For example, to those who expect integers to produce fractions when divided. * The behaviour of ?None? is confusing to some. For example, to those who expect it to be a non-value, with no type. Examples of both those have appeared in this very forum in recent years. So, I think we can assert with confidence: if ?avoid bevause it is unintuitive and confusing to some newcomers? were a good reason to avoid a Python feature, *all* Python features would be subject to oblivion. > Or, we can *learn how to use the features* and stop thinking that > programming is a matter of intuition. And most importantly, stop > thinking that features need to be judged entirely by the least > knowledgeable programmers. Yes. Appeals to intuition are irrelevant in talking about language features, IMO. The intuitions of newcomers should weigh heavily in language design. That is *not* the same as appealing to intuition: intuition gives you none of the essential and difficult concepts necessary to programming. > But the idea that you should avoid a Python feature while programming > in Python because Javascript doesn't have it, or Ruby, or C, is surely > the height of muddleheaded thinking. You're not programming > Javascript, Ruby or C, you're programming in Python. This is not to say that every Python feature can be used without concern. Steven is not arguing that avoidance of a feature is never justified. Rather, he's demonstrating that *that particular justification* is void. There may be a good reason to avoid Python behaviour Foo, but ?because JavaScript/Ruby/Lisp/BASIC/INTERCAL doesn't have behaviour Foo? is not a valid justification. The case must be argued on other merits, if any. -- \ ?Special today: no ice cream.? ?mountain inn, Switzerland | `\ | _o__) | Ben Finney From jon+usenet at unequivocal.co.uk Fri May 20 20:24:50 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Sat, 21 May 2016 00:24:50 -0000 (UTC) Subject: for / while else doesn't make sense References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> <573FA496.5050708@stoneleaf.us> Message-ID: On 2016-05-20, Ethan Furman wrote: > On 05/20/2016 04:55 AM, Jon Ribbens wrote: >> Certainly. "else:" is (almost?) invariably used in the situation where >> you are iterating through something in order to find a value which >> matches a certain condition. So the "for:" block means "find this >> value" and the "else:" means "else do this". > > I'm happy that you have a working mental model for for/else (seriously, > I am) but please don't discount the confusion and consternation for the > many folks who don't start with that mental model. > > The number of times I have /wanted/ to use the for/else structure for > searching is small (and I remember them both ;) -- every other time what > I wanted was an _else_ that ran iff the iterable was already empty when > the _for_ encountered it. Well that's not a circumstance that's appropriate for the construct. Saying you want a different construct entirely is an entirely different argument. >> I would find that very confusing. "then:" makes it sound like >> executing that block is the usual case, when in practice it is >> usually the exception - the fallback code if the expected value >> was not found. > > If you don't take the extra step of _break_ it is the usual case. Having an "for: else:" clause without a "break" would be so unusual that it's literally nonexistent, because it would always be a bug. So no, it isn't the usual case for "for: else:". > But as others have said, this isn't going to change now, and I'm okay > with that. But, please, be a bit more understanding of those who don't > immediately grok the for/else and while/else loops. You're misunderstanding me. I'm not saying that the meaning of "for: else:" is 100% intuitively obvious. I'm saying that it's *more* obvious than it would be if it used any of the other existing keywords. I suppose I'm also saying that there isn't any other obvious word that could be made into a keyword that would be better than "else" (even if we assumed that adding a new keyword was a cost-free exercise). From jon+usenet at unequivocal.co.uk Fri May 20 20:35:48 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Sat, 21 May 2016 00:35:48 -0000 (UTC) Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-20, Steven D'Aprano wrote: > By that logic, we ought to: > > - avoid using floats because their behaviour isn't intuitive and > can be confusing; To be fair, I'm very sympathetic to that argument. I think programming languages should never magically produce floats out of nowhere unless the programmer has explicitly done "import float" or "float('3.23')" or somesuch. They're misunderstood so often that any convenience they provide is outweighed by the danger they bring. "(1/10) * (1/10) * 10 != (1/10)" anyone? I was distinctly unhappy with the Python 3 "2/3 ~= 0.6666" thing and regard it as a very retrograde change. From jfong at ms4.hinet.net Fri May 20 21:05:38 2016 From: jfong at ms4.hinet.net (jfong at ms4.hinet.net) Date: Fri, 20 May 2016 18:05:38 -0700 (PDT) Subject: How to memory dump an object? Message-ID: <6c556e26-1350-4fdc-ae9f-7ef198a3ad6b@googlegroups.com> Is there any tools which can do the memory dump of an object so I can view their content or implementation? For example, >>> s1 = '\x80abc' >>> b1 = b'\x80abc' What are exactly stored in memory for each of them? Is their content really the same? This kind of tool should be helpful "for me" to learn the inner detail of Python. --Jach From christopher_reimer at icloud.com Fri May 20 21:23:57 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Fri, 20 May 2016 18:23:57 -0700 Subject: for / while else doesn't make sense In-Reply-To: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> Message-ID: <44ec67e8-0c6e-54d9-c2fc-a4d9b78efb3d@icloud.com> On 5/20/2016 3:43 PM, Steven D'Aprano wrote: > But the idea that you should avoid a Python feature while programming in > Python because Javascript doesn't have it, or Ruby, or C, is surely the > height of muddleheaded thinking. You're not programming Javascript, Ruby or > C, you're programming in Python. The whole point of picking one language > over another is to get access to the tools and features that language > offers. Otherwise you're just wasting your time. For many years I have resisted specializing in a programming language, as I can easily write any program in pseudo code and figure out the syntax for a particular language. Now it does help that most languages have derived from C and share a common feature set (i.e., string, integer, float, if/else, while, for, etc.). From my perspective, tacking on an else block to the end of a for or while loop looks like a bug or a not very well thought out feature. If I was translating a Python program with for/else or while/else statements into a different language, those statements will have to be rewritten anyway. Thank you, Chris R. From deva.seetharam at gmail.com Fri May 20 21:32:18 2016 From: deva.seetharam at gmail.com (deva.seetharam at gmail.com) Date: Fri, 20 May 2016 18:32:18 -0700 (PDT) Subject: Graceful exit from Python + multiprocessing daemons Message-ID: <2663b7ab-8501-4c4b-8dcf-9d18c2a48273@googlegroups.com> Hello, Greetings! I would like to get your advice wrt following situation: I have a Linux daemon written in python (version 2.7) using the python-daemon (https://pypi.python.org/pypi/python-daemon) module. The objective of using python daemon is to run as an init.d script in Linux. This gets instantiated and started using a main function. The main function also creates a Driver object. That object in turn creates two daemon processes (multiprocessing.Process with daemon flag set to True) a producer and a consumer. The producer puts data in a multiprocessing.Queue and the consumer reads those data items and processes them. I have two challenges. Please advise how these can be handled: 1. When the program is stopped, I would like to preserve all the elements in the Queue so it can be processed later. (So that data in the Queue is not lost.) However, when I call the finalize function, it says the Queue is empty even though elements are in the Queue. One thing I noticed is when the finalize function is called, the parent ID changes to the calling process from 1. 2. When the daemon (python App stop) is stopped, I get an assertion error as follows: Terminating on signal 15 Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/usr/lib/python2.7/multiprocessing/util.py", line 325, in _exit_function p.join() File "/usr/lib/python2.7/multiprocessing/process.py", line 143, in join assert self._parent_pid == os.getpid(), 'can only join a child process' AssertionError: can only join a child process I am including the skeletal structure of the code for your review and advice. --------------------------------------------------------------- class Producer(object): def run(): self.data_queue.put(new_item) def finalize(self): dump_file = open('/tmp/dumper', 'wb') while not self.data_queue.empty(): record = self.data_queue.get() pickle.dump(record, dump_file) dump_file.close() class Consumer(object): def run(): next_item = self.data_queue.get() process_item(next_item) def finalize(self): dump_file = open('/tmp/dumper', 'wb') while not self.data_queue.empty(): record = self.data_queue.get() pickle.dump(record, dump_file) dump_file.close() class Driver(object) def run_task(self, task, dummy): while not self.stop_event.is_set(): task.run() task.finalize() self.logger.debug('Exiting task') def __create_workers__(self): task1 = Producer(data_queue) self.task_list.append(task1) p1 = multiprocessing.Process(target=self.run_task, name='T1', args=(task1, 1)) self.worker_list.append(p1) task2 = Consumer(data_queue) self.task_list.append(task2) p2 = multiprocessing.Process(target=self.run_task, name='T2', args=(task2, 2)) self.worker_list.append(p2) def run(self): self.__create_workers__() for worker in self.worker_list: worker.daemon = True worker.start() def finalize(self): for task in self.task_list: self.logger.debug('finalizing ') task.finalize() if __name__ == "__main__": the_driver = Driver(app_logger=logger) the_driver.run() # the following is a python-daemon object the_daemon_object = Damien(logger) daemon_runner.do_action() --------------------------------------------------------------- From ned at nedbatchelder.com Fri May 20 21:59:55 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Fri, 20 May 2016 18:59:55 -0700 (PDT) Subject: How to memory dump an object? In-Reply-To: <6c556e26-1350-4fdc-ae9f-7ef198a3ad6b@googlegroups.com> References: <6c556e26-1350-4fdc-ae9f-7ef198a3ad6b@googlegroups.com> Message-ID: On Friday, May 20, 2016 at 9:05:51 PM UTC-4, jf... at ms4.hinet.net wrote: > Is there any tools which can do the memory dump of an object so I can view their content or implementation? For example, > > >>> s1 = '\x80abc' > >>> b1 = b'\x80abc' > > What are exactly stored in memory for each of them? Is their content really the same? This kind of tool should be helpful "for me" to learn the inner detail of Python. I don't know of a tool that will do that, other than running CPython under the gdb debugger, and examining memory that way. In Python 2, those two objects are the same, because '...' is a byte string, and b'...' is a byte string. I should say, those objects' memory starts out exactly the same. Objects have reference counts which change as names come and go: >>> s1 = '\x80abc' >>> b1 = b'\x80abc' >>> b2 = b1 Now the first string has a reference count of 1, and b1 has a reference count of 2. Those counts are in the objects' memory, so now their memory contents are different. --Ned. From rosuav at gmail.com Fri May 20 22:05:12 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 May 2016 12:05:12 +1000 Subject: for / while else doesn't make sense In-Reply-To: References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sat, May 21, 2016 at 10:35 AM, Jon Ribbens wrote: > On 2016-05-20, Steven D'Aprano wrote: >> By that logic, we ought to: >> >> - avoid using floats because their behaviour isn't intuitive and >> can be confusing; > > To be fair, I'm very sympathetic to that argument. I think programming > languages should never magically produce floats out of nowhere unless > the programmer has explicitly done "import float" or "float('3.23')" > or somesuch. They're misunderstood so often that any convenience > they provide is outweighed by the danger they bring. > > "(1/10) * (1/10) * 10 != (1/10)" anyone? I was distinctly unhappy with > the Python 3 "2/3 ~= 0.6666" thing and regard it as a very retrograde > change. The trouble is, what SHOULD 2/3 return? * An integer? Makes a lot of sense to a C programmer. Not so much to someone who is expecting a nonzero value. This isn't terrible (hey, Python 2 managed with it no problem), but will definitely confuse a number of people. * A float? That's what we currently have. Not perfect, but it's going to confuse less people than 0 will. * A decimal.Decimal? That has its own issues (for instance, (x+y)/2 can return a value that isn't between x and y), and it still can't represent two thirds properly. * A fractions.Fraction? Well, at least that can perfectly represent a ratio of integers. But it plays REALLY badly with other non-integer types, and other operations than basic arithmetic (ever tried to take the square root of a fraction?), so it's really only suited to situations where you're working exclusively with fractions. * Something else? You say that Py3 making 1/10 => 0.1 was a "very retrograde change". Why? Yes, now you're seeing floats; but would you be less surprised by the Py2 version? Sure, Py2 has your little toy example working: >>> (1/10) * (1/10) * 10 == (1/10) True but that's because 1/10 is zero, not because it's been represented accurately! The biggest advantage of having integer division yield an integer is that it forces people to think about their data types; but since Python's float has a standing (core language support) that Decimal and Fraction don't have, I don't think it's a problem. It's the same with the complex type: >>> 2 ** 0.5 1.4142135623730951 >>> (-2) ** 0.5 (8.659560562354934e-17+1.4142135623730951j) Core data types will migrate between themselves as needed. (And this proves some of the inherent inaccuracies in floating point arithmetic; real number arithmetic says that the square root of -2 has a zero real part.) Interestingly, fractions.Fraction doesn't handle non-integer exponentiation, and punts to float: >>> fractions.Fraction(4) ** fractions.Fraction(2) Fraction(16, 1) >>> fractions.Fraction(4) ** fractions.Fraction(2, 3) 2.5198420997897464 >>> fractions.Fraction(4) ** fractions.Fraction(3, 2) 8.0 >>> fractions.Fraction(-4) ** fractions.Fraction(1, 2) (1.2246467991473532e-16+2j) Some data types just aren't well suited to certain operations. ChrisA From grant.b.edwards at gmail.com Fri May 20 22:17:28 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Sat, 21 May 2016 02:17:28 +0000 (UTC) Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-20, Steven D'Aprano wrote: > On Sat, 21 May 2016 05:20 am, Christopher Reimer wrote: > >> According to "Effective Python: 59 Specific Ways to Write Better Python" >> by Brett Slatkin, Item 12 recommends against using the else block after >> for and while loops (see page 25): "Avoid using else blocks after loops >> because their behavior isn't intuitive and can be confusing." > > By that logic, we ought to: > > - avoid using floats because their behaviour isn't intuitive and > can be confusing; Well, a lot of people probably should avoid floats. I've often said that anybody who hasn't taken a numerical methods class shouldn't be allowed to use floating point. -- Grant From rosuav at gmail.com Fri May 20 22:28:40 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 May 2016 12:28:40 +1000 Subject: How to memory dump an object? In-Reply-To: <6c556e26-1350-4fdc-ae9f-7ef198a3ad6b@googlegroups.com> References: <6c556e26-1350-4fdc-ae9f-7ef198a3ad6b@googlegroups.com> Message-ID: On Sat, May 21, 2016 at 11:05 AM, wrote: > Is there any tools which can do the memory dump of an object so I can view their content or implementation? For example, > >>>> s1 = '\x80abc' >>>> b1 = b'\x80abc' > > What are exactly stored in memory for each of them? Is their content really the same? This kind of tool should be helpful "for me" to learn the inner detail of Python. > MASSIVE CAVEAT: This is *not* anything that the Python language specifies. You can mess around with this in one interpreter (say, CPython 3.5) and get one result, but in a different interpreter (Jython, IronPython, MicroPython, Brython), or a different version of the same interpreter (CPython 3.2), or even the exact same interpreter on different platforms (32-bit vs 64-bit, Linux vs Windows vs Mac OS vs other, or different CPUs), you may get completely different results. And some language implementations (eg PyPy) don't necessarily even keep the object in memory at all times. But if you're curious about how things are stored, there are ways you can mess around and find stuff out. The easiest way to explore CPython's internals would probably be ctypes. In CPython, an object's identity is based on its address in memory, so you can cast that to a pointer. >>> import ctypes >>> s1 = "\x80abc" >>> ctypes.cast(id(s1), ctypes.c_voidp) c_void_p(140180250548144) >>> import sys >>> sys.getsizeof(s1) # See how much space it takes up in RAM 77 You can mess around with pointers to your heart's content: >>> ptr = ctypes.cast(id(s1), ctypes.POINTER(ctypes.c_uint8)) >>> bytes([ptr[i] for i in range(77)]) b'\x01\x00\x00\x00\x00\x00\x00\x00\x00\x04\x88\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x15!\xbc\x99y\xc4A]\xa43XB~\x7f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80abc\x00' Hey look, there it is! For more info on how CPython 3.3+ stores Unicode text, check out this document: https://www.python.org/dev/peps/pep-0393/ It's pretty smart :) ChrisA From rosuav at gmail.com Fri May 20 22:31:36 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 May 2016 12:31:36 +1000 Subject: for / while else doesn't make sense In-Reply-To: <44ec67e8-0c6e-54d9-c2fc-a4d9b78efb3d@icloud.com> References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <44ec67e8-0c6e-54d9-c2fc-a4d9b78efb3d@icloud.com> Message-ID: On Sat, May 21, 2016 at 11:23 AM, Christopher Reimer wrote: > On 5/20/2016 3:43 PM, Steven D'Aprano wrote: > >> But the idea that you should avoid a Python feature while programming in >> Python because Javascript doesn't have it, or Ruby, or C, is surely the >> height of muddleheaded thinking. You're not programming Javascript, Ruby >> or >> C, you're programming in Python. The whole point of picking one language >> over another is to get access to the tools and features that language >> offers. Otherwise you're just wasting your time. > > > For many years I have resisted specializing in a programming language, as I > can easily write any program in pseudo code and figure out the syntax for a > particular language. Now it does help that most languages have derived from > C and share a common feature set (i.e., string, integer, float, if/else, > while, for, etc.). From my perspective, tacking on an else block to the end > of a for or while loop looks like a bug or a not very well thought out > feature. If I was translating a Python program with for/else or while/else > statements into a different language, those statements will have to be > rewritten anyway. That's fine, as long as you (a) restrict your programming languages to those derived from C, and (b) restrict your programming style to the common subset of them all. Trouble is, that "common subset" is actually pretty small. Strings behave very differently in C and high level languages, and for loops are *very* different in different languages. So you'd be throwing out a large amount of expressiveness, plus you're completely unable to use languages built on some other model (eg LISP, or DeScribe Macro Language, or APL). ChrisA From dan at tombstonezero.net Fri May 20 23:19:49 2016 From: dan at tombstonezero.net (Dan Sommers) Date: Sat, 21 May 2016 03:19:49 -0000 (UTC) Subject: Summing/combining tuples References: Message-ID: On Wed, 18 May 2016 20:59:55 -0400, DFS wrote: > Have aList = [ > ('x','Name1', 1, 85), > ('x','Name2', 3, 219), > ('x','Name2', 1, 21), > ('x','Name3', 6, 169) > ] > > want > > aList = [ > ('Name1', 1, 85), > ('Name2', 4, 240), > ('Name3', 6, 169) > ] [snip] > Is there something shorter and sweeter for the summation? from itertools import groupby from operator import itemgetter result = [(k, sum(map(itemgetter(2), v)), sum(map(itemgetter(3), v))) for (k, v) in [(k, list(v)) for (k, v) in groupby(aList, itemgetter(1))]]) Shorter? In terms of number of expressions, yes. In terms of characters typed, no. Sweeter? YMMV. HTH, Dan From christopher_reimer at icloud.com Fri May 20 23:47:27 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Fri, 20 May 2016 20:47:27 -0700 Subject: for / while else doesn't make sense In-Reply-To: References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <44ec67e8-0c6e-54d9-c2fc-a4d9b78efb3d@icloud.com> Message-ID: <1198e259-0bbb-f18a-e8d1-114bc84bf24c@icloud.com> On 5/20/2016 7:31 PM, Chris Angelico wrote: > On Sat, May 21, 2016 at 11:23 AM, Christopher Reimer > wrote: >> On 5/20/2016 3:43 PM, Steven D'Aprano wrote: >> >>> But the idea that you should avoid a Python feature while programming in >>> Python because Javascript doesn't have it, or Ruby, or C, is surely the >>> height of muddleheaded thinking. You're not programming Javascript, Ruby >>> or >>> C, you're programming in Python. The whole point of picking one language >>> over another is to get access to the tools and features that language >>> offers. Otherwise you're just wasting your time. >> >> For many years I have resisted specializing in a programming language, as I >> can easily write any program in pseudo code and figure out the syntax for a >> particular language. Now it does help that most languages have derived from >> C and share a common feature set (i.e., string, integer, float, if/else, >> while, for, etc.). From my perspective, tacking on an else block to the end >> of a for or while loop looks like a bug or a not very well thought out >> feature. If I was translating a Python program with for/else or while/else >> statements into a different language, those statements will have to be >> rewritten anyway. > That's fine, as long as you (a) restrict your programming languages to > those derived from C, and (b) restrict your programming style to the > common subset of them all. Trouble is, that "common subset" is > actually pretty small. Strings behave very differently in C and high > level languages, and for loops are *very* different in different > languages. So you'd be throwing out a large amount of expressiveness, > plus you're completely unable to use languages built on some other > model (eg LISP, or DeScribe Macro Language, or APL). I don't have a problem with (a) because the majority of the programming languages I've been exposed to have derived from the C language. No offense to the LISPers, but LISP is a historical curiosity that I might blow the dust off and take a look at someday. I'll probably learn assembly language before I ever look at LISP. :) But I disagree with (b) on restricting myself to a common subset of ALL the programming languages. Pseudo code allows me to describe a program in very general details. Implementing a program in a programming language requires getting into very specific details. Of course, there are major and minor differences from language to language. If an oddball feature gets the job done, I'll use that. Or maybe not. If I'm uncertain about something, I'll keep going back and forth until I'm satisfied one way or another. The else block tacked on to for and while loops in Python seems very oddball-ish to me. I've always strive to follow best practice whenever possible. The one book I've read -- and so far, the only book on that feature -- recommends not using it. Based on my previous experience, I don't disagree with that author's opinion. If I have a compelling reason to use it, I'll use it. Or I'll simplify it to use helper functions. If I wanted to write portable code, I would have stayed with... Java. O_o Thank you, Chris R. From steve at pearwood.info Fri May 20 23:50:15 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 21 May 2016 13:50:15 +1000 Subject: for / while else doesn't make sense References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> <573FA496.5050708@stoneleaf.us> Message-ID: <573fdaf9$0$1584$c3e8da3$5496439d@news.astraweb.com> On Sat, 21 May 2016 10:24 am, Jon Ribbens wrote: > On 2016-05-20, Ethan Furman wrote: >> If you don't take the extra step of _break_ it is the usual case. > > Having an "for: else:" clause without a "break" would be so unusual > that it's literally nonexistent, because it would always be a bug. > So no, it isn't the usual case for "for: else:". What do you mean? A for...else without a break is perfectly legal code, and does *EXACTLY* what it is documented as doing: - first the "for" block runs, looping as appropriate; - THEN the "else" block runs, *once*. How is this "always a bug"? Would you classify the second line here: print("Hello World!") pass as a bug? What exactly would your bug report be? "pass statement does nothing, as expected. It should do nothing. Please fix." >> But as others have said, this isn't going to change now, and I'm okay >> with that. But, please, be a bit more understanding of those who don't >> immediately grok the for/else and while/else loops. > > You're misunderstanding me. I'm not saying that the meaning of > "for: else:" is 100% intuitively obvious. I should hope not, because as it stands with the horribly misleading keyword "else" it is counter-intuitive and confusing. Which is a crying shame, because it is a useful feature that actually does make a lot of sense, if only the keyword were better! > I'm saying that it's > *more* obvious than it would be if it used any of the other existing > keywords. I suppose I'm also saying that there isn't any other > obvious word that could be made into a keyword that would be better > than "else" (even if we assumed that adding a new keyword was a > cost-free exercise). Well, that's a matter of opinion. And you know what they same about opinions... there are always two, the foolish, pig-ignorant one, and mine. :-) -- Steven From robertvstepp at gmail.com Sat May 21 00:00:39 2016 From: robertvstepp at gmail.com (boB Stepp) Date: Fri, 20 May 2016 23:00:39 -0500 Subject: Extract the middle N chars of a string In-Reply-To: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> References: <573c8e97$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Wed, May 18, 2016 at 10:47 AM, Steven D'Aprano wrote: > Getting the middle N seems like it ought to be easy: > > s[N//2:-N//2] > > but that is wrong. It's not even the right length! > > py> s = 'aardvark' > py> s[5//2:-5//2] > 'rdv' > > > So after spending a ridiculous amount of time on what seemed like it ought > to be a trivial function, and an embarrassingly large number of off-by-one > and off-by-I-don't-even errors, I eventually came up with this: > > def mid(string, n): > """Return middle n chars of string.""" > L = len(string) > if n <= 0: > return '' > elif n < L: > Lr = L % 2 > a, ar = divmod(L-n, 2) > b, br = divmod(L+n, 2) > a += Lr*ar > b += Lr*br > string = string[a:b] > return string As some of you know, I usually post on the Tutor list while attempting to learn Python as time permits. I had to try my hand at this problem as a learning opportunity. I hope you don't mind if I explain how I got to my solution and welcome your critiques, so I may improve. I chose to cheat my answers to the right; I did not think about the possibility of alternating the sides to allot the extra character (when needed) to average things out until I read everyone's answers after getting my own. I started considering two strings, s_even = '0123456789' and s_odd = '123456789', with trial values of n = 4 and n = 5 for how many characters to extract. This gave me the following four desired outputs to replicate: 1) s_even with n = 5. Desired output: '34567' (Cheating right.) => Slice s_even[3:8] 2) s_even with n = 4. Desired output: '3456' (Exact.) => Slice s_even[3:7] 3) s_odd with n = 5. Desired output: '34567' (Exact.) => Slice s_odd[2:7] 4) s_odd with n = 4. Desired output: '4567' (Cheating right.) => Slice s_odd[3:7] Starting to generalize to get the desired indices for each case: 1) (len(s_even)//2 - n//2):(len(s_even)//2 + n//2 + 1) 2) (len(s_even)//2 - n//2):(len(s_even)//2 + n//2) 3) (len(s_odd)//2 - n//2):(len(s_odd)//2 + n//2 + 1) 4) (len(s_odd)//2 + 1 - n//2):(len(s_odd)//2 + n//2 + 1) Looking at the starting index for each case, I had an extra 1 for case (4), which, in table form: n even n odd s_even 0 0 s_odd 1 0 To duplicate this I came up with the expression: (len(s)%2) * (1 - n%2) Similarly, for the ending slice index, all cases have an extra "+ 1" except for case (2), with the following table: n even n odd s_even 0 1 s_odd 1 1 And the expression: 1 - ((len(s) + 1)%2 * (n +1)%2) All this was scribbled onto scratch paper, so I hope I did not make any typos! This led me to the following code: py3: def mid(s, n): ... index0_offset = (len(s)%2) * (1 - n%2) ... index1_offset = 1 - ((len(s) + 1)%2) * ((n + 1)%2) ... index0 = len(s)//2 - n//2 + index0_offset ... index1 = len(s)//2 + n//2 + index1_offset ... return s[index0:index1] ... py3: s = '0123456789' py3: n = 5 py3: mid(s, n) '34567' py3: n = 4 py3: mid(s, n) '3456' py3: s = '123456789' py3: n = 5 py3: mid(s, n) '34567' py3: n = 4 py3: mid(s, n) '4567' py3: s = 'aardvark' py3: n = 5 py3: mid(s, n) 'rdvar' This also returns an empty string for values of n <= 0. As far as I can tell, my solution works (Given cheating right.). I ran it on all of Steve's examples, and I got what I expected given that I am consistently cheating right. But I am not sure my code adequately conveys an understanding of what I am doing to the casual reader. Thoughts? TIA! boB From rosuav at gmail.com Sat May 21 00:01:52 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 May 2016 14:01:52 +1000 Subject: for / while else doesn't make sense In-Reply-To: <573fdaf9$0$1584$c3e8da3$5496439d@news.astraweb.com> References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> <573FA496.5050708@stoneleaf.us> <573fdaf9$0$1584$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sat, May 21, 2016 at 1:50 PM, Steven D'Aprano wrote: > On Sat, 21 May 2016 10:24 am, Jon Ribbens wrote: > >> On 2016-05-20, Ethan Furman wrote: > >>> If you don't take the extra step of _break_ it is the usual case. >> >> Having an "for: else:" clause without a "break" would be so unusual >> that it's literally nonexistent, because it would always be a bug. >> So no, it isn't the usual case for "for: else:". > > > What do you mean? A for...else without a break is perfectly legal code, and > does *EXACTLY* what it is documented as doing: > > - first the "for" block runs, looping as appropriate; > - THEN the "else" block runs, *once*. > > How is this "always a bug"? > > Would you classify the second line here: > > print("Hello World!") > pass > > > as a bug? What exactly would your bug report be? "pass statement does > nothing, as expected. It should do nothing. Please fix." > Yes, I would. It's not a bug in Python or CPython - it's a bug in the second line of code there. It implies something that isn't the case. It's like having this code: if True: pass elif False: pass else: assert True It's well-defined code. You know exactly what Python should do. But is it good code, or is it the sort of thing that gets posted here: http://thedailywtf.com/articles/a-spiritual-journey ChrisA From jfong at ms4.hinet.net Sat May 21 00:47:16 2016 From: jfong at ms4.hinet.net (jfong at ms4.hinet.net) Date: Fri, 20 May 2016 21:47:16 -0700 (PDT) Subject: How to memory dump an object? In-Reply-To: References: <6c556e26-1350-4fdc-ae9f-7ef198a3ad6b@googlegroups.com> Message-ID: Sorry, forget to mention that I am working on version 3.4 Following the steps given in Chris's reply, I get the result from bytes string: >>> b1 = b'\x80abc' >>> ctypes.cast(id(b1), ctypes.c_voidp) c_void_p(35495992) >>> sys.getsizeof(b1) 21 >>> b1ptr = ctypes.cast(id(b1), ctypes.POINTER(ctypes.c_uint8)) >>> bytes([b1ptr[i] for i in range(21)]) b'\x01\x00\x00\x00X\xa1e^\x04\x00\x00\x00\x04\xff\x04&\x80abc\x00' >>> It obviously has the same "content" as the s1 string has and the object's structure are much different as expected (77 vs 21 bytes:-) If I do this as Ned suggested: >>> b2 = b1 I get: >>> bytes([b1ptr[i] for i in range(21)]) b'\x02\x00\x00\x00X\xa1e^\x04\x00\x00\x00\x04\xff\x04&\x80abc\x00' >>> So, at least, we know where the reference count was stored:-) It's amazing the "tool" is so simple. Actually I feel a little embarrassed that I do know every statements in it but just no idea of combining them together. Thanks a lot, Chris. --Jach From steve at pearwood.info Sat May 21 00:56:10 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 21 May 2016 14:56:10 +1000 Subject: How to memory dump an object? References: <6c556e26-1350-4fdc-ae9f-7ef198a3ad6b@googlegroups.com> Message-ID: <573fea6c$0$1587$c3e8da3$5496439d@news.astraweb.com> On Sat, 21 May 2016 11:05 am, jfong at ms4.hinet.net wrote: > Is there any tools which can do the memory dump of an object so I can view > their content or implementation? No standard tool. There may be third-party tools, but they would be implementation-specific. > For example, > >>>> s1 = '\x80abc' >>>> b1 = b'\x80abc' > > What are exactly stored in memory for each of them? That will depend on the version and implementation of the Python interpreter you are looking at. Strings especially are implemented in many different ways. Before Python 3.3, CPython used two different implementations, which you choose when compiling the interpreter from source: - "narrow builds" use two-bytes per character, and have difficulty with Unicode codepoints after U+FFFF; - "wide builds" use four-bytes per character, and can cope with the entire Unicode range up to U+10FFFF; - I think that Jython uses the Java string implementation; - and that IronPython uses the .Net string implementation. Starting at Python 3.3, the CPython implementation will dynamically choose a one-byte, two-byte or four-byte per character implementation, as needed. > Is their content really the same? The first is the Unicode string with four code points: LATIN SMALL LETTER A LATIN SMALL LETTER B LATIN SMALL LETTER C the other contains a byte-string with four bytes: \x80 \x61 (ASCII 'a') \x62 (ASCII 'b') \x63 (ASCII 'c') So in *some* sense they are the same content, but only in the sense that this list: [128, 97, 98, 99] *also* has the same content. Instead of a low-level memory dump, which isn't actually terribly useful (for many objects, such as lists, all it would show is a bunch of pointers), there are other high-level introspection tools available in the `inspect` module. You can also use: sys.getsizeof to see how much memory is used by an object. -- Steven From rosuav at gmail.com Sat May 21 01:01:30 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 May 2016 15:01:30 +1000 Subject: How to memory dump an object? In-Reply-To: References: <6c556e26-1350-4fdc-ae9f-7ef198a3ad6b@googlegroups.com> Message-ID: On Sat, May 21, 2016 at 2:47 PM, wrote: > It's amazing the "tool" is so simple. Actually I feel a little embarrassed that I do know every statements in it but just no idea of combining them together. Thanks a lot, Chris. > It's a pretty complicated tool, actually - but you're using it in a simple way here. If you're curious, you can poke around in the CPython sources to track down the data structures involved - as you noted, the refcount is at the beginning of every object, and after that you'll find some neat info like a pointer to its type. You can even change some of this stuff, but be aware that you can EASILY segfault Python like this :D ChrisA From dan at tombstonezero.net Sat May 21 01:14:02 2016 From: dan at tombstonezero.net (Dan Sommers) Date: Sat, 21 May 2016 05:14:02 -0000 (UTC) Subject: Summing/combining tuples References: Message-ID: On Sat, 21 May 2016 03:19:49 +0000, Dan Sommers wrote: >> Is there something shorter and sweeter for the summation? > > from itertools import groupby > from operator import itemgetter > > result = [(k, > sum(map(itemgetter(2), v)), > sum(map(itemgetter(3), v))) > for (k, v) in [(k, list(v)) > for (k, v) > in groupby(aList, itemgetter(1))]]) Or even: from itertools import groupby from operator import itemgetter from functools import reduce result = [(k, *reduce(lambda acc, item: (acc[0] + item[2], acc[1] + item[3]), v, (0, 0))) for (k, v) in groupby(aList, itemgetter(1))] From rustompmody at gmail.com Sat May 21 01:18:02 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Fri, 20 May 2016 22:18:02 -0700 (PDT) Subject: for / while else doesn't make sense In-Reply-To: References: <573EC62F.4090401@lucidity.plus.com> Message-ID: <75115580-7a63-4cfb-9689-28eb84b580a0@googlegroups.com> On Saturday, May 21, 2016 at 1:51:19 AM UTC+5:30, Christopher Reimer wrote: > On 5/20/2016 8:59 AM, Zachary Ware wrote: > > > On Fri, May 20, 2016 at 3:09 AM, Erik wrote: > >> On 20/05/16 00:51, Gregory Ewing wrote: > >>> It's not so bad with "else" because you need to look back > >>> to find out what condition the "else" refers to anyway. > >> > >> With my tongue only slightly in my cheek, if it was desirable to > >> "fix"/clarify this syntax then I would suggest adding some optional > >> (existing) trailing keywords to 'else' in this context that spells it out: > >> > >> for item in seq: > >> if foo(item): > >> break > >> else if not break: > >> nomatch() > > With tongue firmly cheeked, you can always use the special `:#` operator: > > > > for item in seq: > > if foo(item): > > break > > else:# if no break: > > nomatch() > > > > This has the benefit that you can use whatever syntax you like after > > the `:#`, and use it in any version of Python you want. > > According to "Effective Python: 59 Specific Ways to Write Better Python" > by Brett Slatkin, Item 12 recommends against using the else block after > for and while loops (see page 25): "Avoid using else blocks after loops > because their behavior isn't intuitive and can be confusing." > > Until I read the book, I wasn't aware of this feature (or bug). Doesn't > seem like a feature I would use since it's not commonly found in other > programming languages. As the author demonstrates in his book, I would > probably write a helper function instead. > > Item 13 does recommend using the else block for try/except/else/finally > in exception handling. :) Firstly: let me say that for the specific case of loop-else I am in violent agreement: I find it so confusing that I dont even know what/how it works Coming to the more general attitude expressed above, this view can be eminently sensible or dangerously retrogressive depending... It is a sound and sane view because the field of computer science exists and legitimately so. Which means that, even though when we look around at the field of programming languages what we are struck by is a bedlam of - advertising - fanboyism - latest and bestest koolaid - holy (cow) wars And a corresponding deficit of anything really conceptual, real advances, real understanding; in short truckloads of BS. OTOH the fact that the field of CS exists and is not (only) BS is a good thing. IOW sticking to the well-established canonical core is sound policy compared to jumping onto the latest loud-rattling bandwagon. And yet... Human beings have the propensity of sticking to the norm rather than deviating even when the norm is grievously in error. The following lists some amazingly long lasting errors: http://blog.languager.org/2016/01/how-long.html This is related to the pedagogic principle called "Law of Primacy": | Things learned first create a strong impression in the mind that is difficult | to erase. For the instructor, this means that what is taught must be right | the first time. Given that for the most part, most of us are horribly uneducated [ http://www.creativitypost.com/education/9_elephants_in_the_classroom_that_should_unsettle_us ] how do we go about correcting our wrong primacies? Given that you imagine Lisp is a historical curiosity (have you heard of clojure?) And java (and presumably OOP) is relevant [see Stepanov on OOP: https://en.wikipedia.org/wiki/Object-oriented_programming#Criticism ] I suggest you start with: http://blog.languager.org/2016/01/primacy.html From marko at pacujo.net Sat May 21 02:03:28 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Sat, 21 May 2016 09:03:28 +0300 Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87d1og3q33.fsf@elektro.pacujo.net> theherk at gmail.com: > You seem to have missed the point. Nobody is suggesting, I don't > believe, that all of a language should be intuitive. Rather that if > any part of it is unnecessarily counter-intuitive, it may be worth > looking for a better solution. Python is a very well designed language > when it comes to in linguistic presentation. In this case however, it > is not only unintuitive but counter-intuitive. The for-else construct is a highly practical feature. It is in no way inherently counter-intuitive. As any new thing, it needs to be learned and used to be appreciated. Marko From python at lucidity.plus.com Sat May 21 03:20:02 2016 From: python at lucidity.plus.com (Erik) Date: Sat, 21 May 2016 08:20:02 +0100 Subject: for / while else doesn't make sense In-Reply-To: <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> Message-ID: <57400C22.7050706@lucidity.plus.com> On 20/05/16 01:06, Steven D'Aprano wrote: > In my experience, some people (including me) misunderstand "for...else" to > mean that the else block runs if the for block *doesn't*. It took me the > longest time to understand why this didn't work as I expected: > > for x in seq: > pass > else: > print("seq is empty") So why don't we consider if that should be a syntax error - "else" clause on "for" loop with no "break" in its body? I know that doesn't change your fundamental mental model, but it's a hint that it's wrong :) As it's backwards-incompatible, it could be introduced using a __future__ import (a precedent is 'generators' and the "yield" keyword back in the day) which those who would like the new check could add to the top of their sources. But also, as you say, any instances of that construct in the wild is almost certainly a bug, so it would be good to be able to test code using a command-line or similar switch to turn on the behaviour by default for testing existing bodies of code. I notice that pylint complains about this (as a warning). Is there any reason why this should _not_ just be considered an error and be done with it? E. From rosuav at gmail.com Sat May 21 03:29:20 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 May 2016 17:29:20 +1000 Subject: for / while else doesn't make sense In-Reply-To: <57400C22.7050706@lucidity.plus.com> References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> <57400C22.7050706@lucidity.plus.com> Message-ID: On Sat, May 21, 2016 at 5:20 PM, Erik wrote: > On 20/05/16 01:06, Steven D'Aprano wrote: > >> In my experience, some people (including me) misunderstand "for...else" to >> mean that the else block runs if the for block *doesn't*. It took me the >> longest time to understand why this didn't work as I expected: >> >> for x in seq: >> pass >> else: >> print("seq is empty") > > > So why don't we consider if that should be a syntax error - "else" clause on > "for" loop with no "break" in its body? I know that doesn't change your > fundamental mental model, but it's a hint that it's wrong :) > > As it's backwards-incompatible, it could be introduced using a __future__ > import (a precedent is 'generators' and the "yield" keyword back in the day) > which those who would like the new check could add to the top of their > sources. > > But also, as you say, any instances of that construct in the wild is almost > certainly a bug, so it would be good to be able to test code using a > command-line or similar switch to turn on the behaviour by default for > testing existing bodies of code. > > I notice that pylint complains about this (as a warning). Is there any > reason why this should _not_ just be considered an error and be done with > it? Because it's not the language parser's job to decide what's "sensible" and what's "not sensible". That's a linter's job. Some things fundamentally can't work, but others work fine and might just be not-very-useful - for example: def f(x): x if x <= 2 else x * f(x-1) This is syntactically-legal Python code, but it probably isn't doing what you want it to (in Python, a bare final expression does NOT become the function's return value, although that does happen in other languages). A linter should look at this and give a warning (even if the else part has side effects, the first part can't, and the condition shouldn't), but the language will happily evaluate it (this example from CPython 3.6): >>> dis.dis(f) 2 0 LOAD_FAST 0 (x) 3 LOAD_CONST 1 (2) 6 COMPARE_OP 1 (<=) 9 POP_JUMP_IF_FALSE 18 12 LOAD_FAST 0 (x) 15 JUMP_FORWARD 17 (to 35) >> 18 LOAD_FAST 0 (x) 21 LOAD_GLOBAL 0 (f) 24 LOAD_FAST 0 (x) 27 LOAD_CONST 2 (1) 30 BINARY_SUBTRACT 31 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 34 BINARY_MULTIPLY >> 35 POP_TOP 36 LOAD_CONST 0 (None) 39 RETURN_VALUE So, yes, this function might load up the value of x (position 12), then jump to 35 and discard that value, before unconditionally returning None. Not a problem. It's up to the linter, and ONLY the linter, to tell you about this. ChrisA From sahoo.bhagyadhar45 at gmail.com Sat May 21 03:55:29 2016 From: sahoo.bhagyadhar45 at gmail.com (bhagyadhar sahoo) Date: Sat, 21 May 2016 13:25:29 +0530 Subject: problem regarding installation Message-ID: i have downloaded the software from your site.. but while installing it shows some problem .. plz help me out..or let me know how can i get the solution,.. i am waiting for ur replay thanks bhagya From jfong at ms4.hinet.net Sat May 21 04:08:06 2016 From: jfong at ms4.hinet.net (jfong at ms4.hinet.net) Date: Sat, 21 May 2016 01:08:06 -0700 (PDT) Subject: How to memory dump an object? In-Reply-To: <6c556e26-1350-4fdc-ae9f-7ef198a3ad6b@googlegroups.com> References: <6c556e26-1350-4fdc-ae9f-7ef198a3ad6b@googlegroups.com> Message-ID: <9ee3a397-46fb-4deb-8997-e55e36df4398@googlegroups.com> Make a quick experiment under version 3.4.4 through this simple "tool" Chris had provided, now I know how the unicode string was stored in memory:-) >>> s1 = '\x80abc' >>> s1 '\x80abc' >>> len(s1) 4 >>> sys.getsizeof(s1) 41 >>> s1ptr = ctypes.cast(id(s1), ctypes.POINTER(ctypes.c_uint8)) >>> bytes([s1ptr[i] for i in range(41)]) b'\x01\x00\x00\x00\xe8\xa5g^\x04\x00\x00\x00\x04\xff\x04&\xa4\x00\x00d\x00\x00\x 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80abc\x00' >>> # a simple unicode string, each code point can be stored in one byte >>> s2 = '\u0080abc' >>> s2 '\x80abc' >>> len(s2) 4 >>> sys.getsizeof(s2) 41 >>> s2ptr = ctypes.cast(id(s2), ctypes.POINTER(ctypes.c_uint8)) >>> bytes([s2ptr[i] for i in range(41)]) b'\x01\x00\x00\x00\xe8\xa5g^\x04\x00\x00\x00\x04\xff\x04&\xa4\x00\x00|\x00\x00\x 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80abc\x00' >>> # Python change it automatically to store each code point into one byte >>> s3 = '\u07b4abc' >>> s3 '\u07b4abc' >>> len(s3) 4 >>> sys.getsizeof(s3) 46 >>> s3ptr = ctypes.cast(id(s3), ctypes.POINTER(ctypes.c_uint8)) >>> bytes([s3ptr[i] for i in range(46)]) b'\x01\x00\x00\x00\xe8\xa5g^\x04\x00\x00\x00\x19n\x93]\xa8\x00\x00dd\r\x1d\x02\x 00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\xb4\x07a\x00b\x00c\x00\x00\x00' >>> # use two bytes to store each code point, even the end-of-string mark >>> s4 = '\U00025049abc' >>> s4 '\U00025049abc' >>> len(s4) 4 >>> sys.getsizeof(s4) 56 >>> s4ptr = ctypes.cast(id(s4), ctypes.POINTER(ctypes.c_uint8)) >>> bytes([s4ptr[i] for i in range(56)]) b'\x01\x00\x00\x00\xe8\xa5g^\x04\x00\x00\x003\xdd\xa7"\xb0\x00\x00\x00\x00\x00\x 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00IP\x02\x00a\x00\x00\x00b\x 00\x00\x00c\x00\x00\x00\x00\x00\x00\x00' >>> # use four bytes to store each code point --Jach From marko at pacujo.net Sat May 21 04:37:43 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Sat, 21 May 2016 11:37:43 +0300 Subject: for / while else doesn't make sense References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> <57400C22.7050706@lucidity.plus.com> Message-ID: <8760u74xig.fsf@elektro.pacujo.net> Erik : > On 20/05/16 01:06, Steven D'Aprano wrote: >> In my experience, some people (including me) misunderstand >> "for...else" to mean that the else block runs if the for block >> *doesn't*. It took me the longest time to understand why this didn't >> work as I expected: >> >> for x in seq: >> pass >> else: >> print("seq is empty") > > So why don't we consider if that should be a syntax error - "else" > clause on "for" loop with no "break" in its body? I know that doesn't > change your fundamental mental model, but it's a hint that it's wrong > :) While we are at it, we should trigger a syntax error in these cases: a = 0 b += a # pointless to add a 0 c = 1 d *= c # pointless to multiply by 1 if False: # pointless to test for truth in falsity ... However, all of these "pointless" constructs crop up in real situations, and artifically flagging them as errors would lead to unnecessary frustration. For example, if False: yield None is the way to create a generator that doesn't generate any output. Marko From tshortik at gmx.de Sat May 21 04:52:05 2016 From: tshortik at gmx.de (=?UTF-8?Q?Dirk_B=c3=a4chle?=) Date: Sat, 21 May 2016 10:52:05 +0200 Subject: How do I subclass the @property setter method? In-Reply-To: <2200dfd0-7470-0c86-c015-a6a14a2f51f2@icloud.com> References: <2200dfd0-7470-0c86-c015-a6a14a2f51f2@icloud.com> Message-ID: <574021B5.1010408@gmx.de> Hi Christopher, On 20.05.2016 20:50, Christopher Reimer wrote: > Greetings, > > My chess engine has a Piece class with the following methods that use > the @property decorator to read and write the position value. > slightly off topic: is your chess engine available in a public repo somewhere? I think I've started to implement something similar (display of chess boards and wrapping different chess engines), so I'd like to have a look. ;) Best regards, Dirk From none at invalid.com Sat May 21 04:52:27 2016 From: none at invalid.com (mm0fmf) Date: Sat, 21 May 2016 09:52:27 +0100 Subject: problem regarding installation In-Reply-To: References: Message-ID: On 21/05/2016 08:55, bhagyadhar sahoo wrote: > i have downloaded the software from your site.. > but while installing it shows some problem .. > plz help me out..or let me know how can i get the solution,.. i am waiting > for ur replay > > thanks > bhagya > We don't know which web site you are talking about. We don't know which software you are talking about. We don't know what operating system you are using. We don't know what the problem was. How can we help you if you don't explain the problem in some detail? From __peter__ at web.de Sat May 21 05:14:44 2016 From: __peter__ at web.de (Peter Otten) Date: Sat, 21 May 2016 11:14:44 +0200 Subject: problem regarding installation References: Message-ID: bhagyadhar sahoo wrote: > i have downloaded the software from your site.. > but while installing it shows some problem .. > plz help me out..or let me know how can i get the solution,.. i am waiting > for ur replay We are sorry, your operating system is no longer supported by current versions of Python. Try Python 3.4.3. From rosuav at gmail.com Sat May 21 05:17:57 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 May 2016 19:17:57 +1000 Subject: problem regarding installation In-Reply-To: References: Message-ID: On Sat, May 21, 2016 at 7:14 PM, Peter Otten <__peter__ at web.de> wrote: > bhagyadhar sahoo wrote: > >> i have downloaded the software from your site.. >> but while installing it shows some problem .. >> plz help me out..or let me know how can i get the solution,.. i am waiting >> for ur replay > > We are sorry, your operating system is no longer supported by current > versions of Python. > > Try Python 3.4.3. Or try a better operating system. ChrisA From steve at pearwood.info Sat May 21 05:56:33 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 21 May 2016 19:56:33 +1000 Subject: for / while else doesn't make sense References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> <573FA496.5050708@stoneleaf.us> <573fdaf9$0$1584$c3e8da3$5496439d@news.astraweb.com> Message-ID: <574030d3$0$1598$c3e8da3$5496439d@news.astraweb.com> On Sat, 21 May 2016 02:01 pm, Chris Angelico wrote: > On Sat, May 21, 2016 at 1:50 PM, Steven D'Aprano > wrote: >> Would you classify the second line here: >> >> print("Hello World!") >> pass >> >> >> as a bug? What exactly would your bug report be? "pass statement does >> nothing, as expected. It should do nothing. Please fix." >> > > Yes, I would. It's not a bug in Python or CPython - it's a bug in the > second line of code there. It implies something that isn't the case. What do you think it implies? What part of the docs for "pass" implies this thing? help("pass"): ``pass`` is a null operation --- when it is executed, nothing happens. It is useful as a placeholder when a statement is required syntactically, but no code needs to be executed, for example: [examples snipped] > It's like having this code: > > if True: > pass > elif False: > pass > else: > assert True Hmmm. Well, let see: py> if True: ... pass ... elif False: ... pass ... else: ... assert True ... Traceback (most recent call last): File "", line 6, in AssertionError What witchcraft is this? S P O I L E R S P A C E I'm running Python 2, and have redefined True = False. But even in Python 3, what exactly is the bug supposed to be? The Python 3 compiler, as naive and simple as it is, is perfectly capable of compiling out the dead code: py> dis.dis("""if True: pass ... elif False: pass ... else: assert True ... """) 1 0 LOAD_CONST 0 (None) 3 RETURN_VALUE So the worst we can say about this is that it is pointless dead code. > It's well-defined code. You know exactly what Python should do. But is > it good code, Well, it's not *great* code, that's for sure. Is it good code? That depends. There are certainly semantic differences between Python 2 and 3, and without knowing the intention of the author (that would be you) its hard to say exactly what the code should be. But it's *legal* code that does exactly what the language documentation says it ought to do. > or is it the sort of thing that gets posted here: > > http://thedailywtf.com/articles/a-spiritual-journey I'm not sure how that specific Daily-WTF article is relevant. But in general, is it worthy of Daily-WTF? No, I don't think so. I think that your "True, False" example could do with some refactoring and cleanup: perhaps it is dead code that should be completely removed, but that's not necessarily Daily-WTF worthy. Or it's some part of some sort of Python 2+3 compatibility layer intending to detect rebindings to True or False, in which case it is certainly ugly (and probably buggy) but may be needed. As far as my example, using "pass" in the code... no, it's not WTF-worthy either. It's at worst worth a raised eyebrow. Without knowing the context of where it came from, it isn't even clear that it is pointless code. Perhaps it is from a test suite checking that `pass` is correctly parsed, compiled and executed as a do-nothing statement. -- Steven From steve at pearwood.info Sat May 21 06:05:13 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 21 May 2016 20:05:13 +1000 Subject: Education [was Re: for / while else doesn't make sense] References: <573EC62F.4090401@lucidity.plus.com> <75115580-7a63-4cfb-9689-28eb84b580a0@googlegroups.com> Message-ID: <574032db$0$22142$c3e8da3$5496439d@news.astraweb.com> On Sat, 21 May 2016 03:18 pm, Rustom Mody wrote: > Given that for the most part, most of us are horribly uneducated [ > http://www.creativitypost.com/education/9_elephants_in_the_classroom_that_should_unsettle_us > ] > how do we go about correcting our wrong primacies? An interesting article, but very US-centric. Nevertheless, let's not forget that the general school system is designed to churn out more-or-less identical workers capable of pulling the levers on late 19th and early 20th century machines for their bosses. The fact that it does more than that, even if badly, is a bonus. -- Steven From mal at europython.eu Sat May 21 06:05:41 2016 From: mal at europython.eu (M.-A. Lemburg) Date: Sat, 21 May 2016 12:05:41 +0200 Subject: PyData EuroPython 2016 Message-ID: <574032F5.3090500@europython.eu> We are excited to announce a complete PyData (http://pydata.org/) track at EuroPython 2016 in Bilbao, Basque Country, Spain, from July 14-24. *** PyData EuroPython 2016 *** https://ep2016.europython.eu/en/events/pydata/ The PyData track will be part of the EuroPython 2016 conference, so you won?t need to buy extra tickets to join. We will have more than 30 talks, 5 training and 2 poster sessions dedicated to PyData on Thursday 21st and Friday 22nd of July. If you?d like to attend the PyData EuroPython 2016 track, please register for EuroPython 2016 soon. https://ep2016.europython.eu/en/registration/ With gravitational regards, -- EuroPython 2016 Team http://ep2016.europython.eu/ http://www.europython-society.org/ From rosuav at gmail.com Sat May 21 06:08:21 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 May 2016 20:08:21 +1000 Subject: for / while else doesn't make sense In-Reply-To: <574030d3$0$1598$c3e8da3$5496439d@news.astraweb.com> References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> <573FA496.5050708@stoneleaf.us> <573fdaf9$0$1584$c3e8da3$5496439d@news.astraweb.com> <574030d3$0$1598$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sat, May 21, 2016 at 7:56 PM, Steven D'Aprano wrote: > On Sat, 21 May 2016 02:01 pm, Chris Angelico wrote: > >> On Sat, May 21, 2016 at 1:50 PM, Steven D'Aprano >> wrote: > >>> Would you classify the second line here: >>> >>> print("Hello World!") >>> pass >>> >>> >>> as a bug? What exactly would your bug report be? "pass statement does >>> nothing, as expected. It should do nothing. Please fix." >>> >> >> Yes, I would. It's not a bug in Python or CPython - it's a bug in the >> second line of code there. It implies something that isn't the case. > > What do you think it implies? > > What part of the docs for "pass" implies this thing? > > help("pass"): > > ``pass`` is a null operation --- when it is executed, nothing > happens. It is useful as a placeholder when a statement is > required syntactically, but no code needs to be executed, for > example: [examples snipped] So why is a statement required syntactically after that print call? Surely that implies something about the programmer's intent? It certainly isn't required according to the code you've shown me; and if someone submitted this code to me, I'd query it ("was there something else meant to be here?"). ChrisA From steve at pearwood.info Sat May 21 06:39:24 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 21 May 2016 20:39:24 +1000 Subject: for / while else doesn't make sense References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> <57400C22.7050706@lucidity.plus.com> Message-ID: <57403ade$0$1593$c3e8da3$5496439d@news.astraweb.com> On Sat, 21 May 2016 05:20 pm, Erik wrote: > On 20/05/16 01:06, Steven D'Aprano wrote: > >> In my experience, some people (including me) misunderstand "for...else" >> to mean that the else block runs if the for block *doesn't*. It took me >> the longest time to understand why this didn't work as I expected: >> >> for x in seq: >> pass >> else: >> print("seq is empty") > > So why don't we consider if that should be a syntax error - "else" > clause on "for" loop with no "break" in its body? I know that doesn't > change your fundamental mental model, but it's a hint that it's wrong :) Just for the record, that's not my mental model *now*. It took me a long time to work out what for...else was actually doing, but some years ago I finally managed to do so. It's my argument that had the keyword been "then" (implying that the `then` block is unconditionally executed after the `for` block, which is the actual behaviour of the interpreter) rather than "else" implying that it is an alternative to the `for` block) I wouldn't have come up with the wrong mental model in the first place. And I'm not the only one who has come up with the same, wrong, model. But you do ask a good question. Why isn't for...else with no break a syntax error? I suppose it could be. But that's a *stylistic* question, and Python generally treats that as "none of the compiler's business". It's not the business of the compiler to enforce good code, only legal code. We can legally write lots of code of questionable value: while 0: print("surprise!") n = 1 + int(x) - 1 x = x try: something() finally: pass without the compiler judging us and making it a syntax error. Why should for...else be any different? The behaviour is perfectly well defined: - first the `for` block runs, as many times as needed; - then the `else` block runs, once. Of course, the usual four keywords (continue, break, return, raise) will change the order of execution by jumping to: - the top of the for loop; - past the for...else blocks; - out of the current function; - the nearest exception handler respectively. But in the absence of a jump, for...else works as specified, regardless of whether there is a break in it or not. > As it's backwards-incompatible, it could be introduced using a > __future__ import (a precedent is 'generators' and the "yield" keyword > back in the day) which those who would like the new check could add to > the top of their sources. Sure, we *could* do this, but as I said, it does go against the usual philosophy that the compiler shouldn't make judgements on what is good code and what isn't. > But also, as you say, any instances of that construct in the wild is > almost certainly a bug, I don't think it was me calling it a bug. > so it would be good to be able to test code > using a command-line or similar switch to turn on the behaviour by > default for testing existing bodies of code. I think that switch is called "use Pylint, PyChecker, Jedi or some other opinionated linter or style-checker". > I notice that pylint complains about this (as a warning). Is there any > reason why this should _not_ just be considered an error and be done > with it? Because there's nothing actually broken with it. There's lots of code which a human programmer would recognise as "silly", or "wrong", but is well-defined and legal. Just because you can't think of a reason to do something doesn't mean it should be prohibited. -- Steven From steve at pearwood.info Sat May 21 06:55:17 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 21 May 2016 20:55:17 +1000 Subject: for / while else doesn't make sense References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> <573FA496.5050708@stoneleaf.us> <573fdaf9$0$1584$c3e8da3$5496439d@news.astraweb.com> <574030d3$0$1598$c3e8da3$5496439d@news.astraweb.com> Message-ID: <57403e97$0$1608$c3e8da3$5496439d@news.astraweb.com> On Sat, 21 May 2016 08:08 pm, Chris Angelico wrote: > On Sat, May 21, 2016 at 7:56 PM, Steven D'Aprano > wrote: >> On Sat, 21 May 2016 02:01 pm, Chris Angelico wrote: >> >>> On Sat, May 21, 2016 at 1:50 PM, Steven D'Aprano >>> wrote: >> >>>> Would you classify the second line here: >>>> >>>> print("Hello World!") >>>> pass >>>> >>>> >>>> as a bug? What exactly would your bug report be? "pass statement does >>>> nothing, as expected. It should do nothing. Please fix." >>>> >>> >>> Yes, I would. It's not a bug in Python or CPython - it's a bug in the >>> second line of code there. It implies something that isn't the case. >> >> What do you think it implies? >> >> What part of the docs for "pass" implies this thing? >> >> help("pass"): >> >> ``pass`` is a null operation --- when it is executed, nothing >> happens. It is useful as a placeholder when a statement is >> required syntactically, but no code needs to be executed, for >> example: [examples snipped] > > So why is a statement required syntactically after that print call? Who said it was required? The docs say that `pass` is useful when a statement is required, not the other way: you can put `pass` anywhere a statement can go, without it being required. > Surely that implies something about the programmer's intent? It > certainly isn't required according to the code you've shown me; and if > someone submitted this code to me, I'd query it ("was there something > else meant to be here?"). Sure. Does that make it a bug? I don't think so. I think that a linter should flag it as unnecessary code, but the compiler certainly shouldn't prohibit it. And a human reviewer might be able to judge whether it belongs or not. Some people might not like "FIXME" or "XXX" comments to flag missing code, and use `pass` statements as placeholders. Who are we to tell them they shouldn't? I think we agree that it's not the compiler's job to enforce good coding standards. -- Steven From rosuav at gmail.com Sat May 21 07:10:31 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 May 2016 21:10:31 +1000 Subject: for / while else doesn't make sense In-Reply-To: <57403e97$0$1608$c3e8da3$5496439d@news.astraweb.com> References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> <573FA496.5050708@stoneleaf.us> <573fdaf9$0$1584$c3e8da3$5496439d@news.astraweb.com> <574030d3$0$1598$c3e8da3$5496439d@news.astraweb.com> <57403e97$0$1608$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sat, May 21, 2016 at 8:55 PM, Steven D'Aprano wrote: > I think we agree that it's not the compiler's job to enforce good coding > standards. Yep. I believe we are in (possibly belligerent) agreement. ChrisA From steve at pearwood.info Sat May 21 07:26:22 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 21 May 2016 21:26:22 +1000 Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <87d1og3q33.fsf@elektro.pacujo.net> Message-ID: <574045e1$0$22142$c3e8da3$5496439d@news.astraweb.com> On Sat, 21 May 2016 04:03 pm, Marko Rauhamaa wrote: > theherk at gmail.com: > >> You seem to have missed the point. Nobody is suggesting, I don't >> believe, that all of a language should be intuitive. Rather that if >> any part of it is unnecessarily counter-intuitive, it may be worth >> looking for a better solution. Python is a very well designed language >> when it comes to in linguistic presentation. In this case however, it >> is not only unintuitive but counter-intuitive. > > The for-else construct is a highly practical feature. It is in no way > inherently counter-intuitive. As any new thing, it needs to be learned > and used to be appreciated. Yes, this! I might have issues with the keyword chosen, but the feature itself is very clever. Does anyone know of other languages that include the same feature? It's very rare for Python to innovate in language features. (I wonder if it came from ABC?) -- Steven From steve at pearwood.info Sat May 21 07:26:49 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 21 May 2016 21:26:49 +1000 Subject: for / while else doesn't make sense References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> <5o8vjb1rlgj46i5jumk7p80nej3cq4jgn4@4ax.com> Message-ID: <574045fa$0$22142$c3e8da3$5496439d@news.astraweb.com> On Sat, 21 May 2016 09:57 am, Dennis Lee Bieber wrote: > On Fri, 20 May 2016 11:55:34 -0000 (UTC), Jon Ribbens > declaimed the following: > >> >>I would find that very confusing. "then:" makes it sound like >>executing that block is the usual case, when in practice it is >>usually the exception - the fallback code if the expected value >>was not found. > > And I'll second that... > > In those languages that use "then" as a keyword, it separates the "if" > conditional from the "true" block of code. Or -- visualize replacing the > ":" on the "for" with the word "then" > > for x in sequence then > do stuff with x > else > do something with no x > > > If a different keyword is to be introduced, I nominate "otherwise" "otherwise" fails for the same reason that "else" fails: it suggests that the else block is an alternative to the for block, which is exactly what it is NOT. -- Steven From ian.g.kelly at gmail.com Sat May 21 09:51:34 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Sat, 21 May 2016 07:51:34 -0600 Subject: for / while else doesn't make sense In-Reply-To: <574045e1$0$22142$c3e8da3$5496439d@news.astraweb.com> References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <87d1og3q33.fsf@elektro.pacujo.net> <574045e1$0$22142$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sat, May 21, 2016 at 5:26 AM, Steven D'Aprano wrote: > Does anyone know of other languages that include the same feature? It's very > rare for Python to innovate in language features. > > (I wonder if it came from ABC?) According to Raymond Hettinger starting at about 15:50 in this video: http://pyvideo.org/video/1780/transforming-code-into-beautiful-idiomatic-pytho It was invented by Donald Knuth (including the choice of keyword). From grant.b.edwards at gmail.com Sat May 21 11:20:17 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Sat, 21 May 2016 15:20:17 +0000 (UTC) Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <87d1og3q33.fsf@elektro.pacujo.net> <574045e1$0$22142$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-21, Ian Kelly wrote: > On Sat, May 21, 2016 at 5:26 AM, Steven D'Aprano wrote: > >> Does anyone know of other languages that include the same feature? >> It's very rare for Python to innovate in language features. >> >> (I wonder if it came from ABC?) > > According to Raymond Hettinger starting at about 15:50 in this > video: > > http://pyvideo.org/video/1780/transforming-code-into-beautiful-idiomatic-pytho > > It was invented by Donald Knuth (including the choice of keyword). If true, that alone dismisses with prejudice any question of changing it. As if backwards compatibility weren't a compelling enough argument. Besides which, I happen to think it makes sense the way it is. [I haven't been paying very close attention to this thread for a while, so I assume that "this feature" and "it" still refer to having an else clause in for/while loops?] I can't count the number of times I've wished C had such a feature, and have sometimes resorted to using a "goto" to get the same semantics in an easy to read/understand way. -- Grant From huey.y.jiang at gmail.com Sat May 21 11:22:41 2016 From: huey.y.jiang at gmail.com (sweating_ant@yahoo.com) Date: Sat, 21 May 2016 08:22:41 -0700 (PDT) Subject: Image loading problem Message-ID: <743e5fbe-3a6c-49ea-ac5e-9bc055f652b5@googlegroups.com> Hi All, I am working on an image project, and I can display my image in main(). I mean, I can load my image in my main(). Needless, it is awkward. I am trying to load my image with a function, but got an empty image window popped up, no image content loaded. Please take a look at code: rom Tkinter import * def load_img(win): img = PhotoImage(file="xxx.gif") Label(win, image=img).pack() win = Tk() load_img(win) win.mainloop() Somebody can help me out? Thanks! From christopher_reimer at icloud.com Sat May 21 11:51:05 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Sat, 21 May 2016 08:51:05 -0700 Subject: Education [was Re: for / while else doesn't make sense] In-Reply-To: <574032db$0$22142$c3e8da3$5496439d@news.astraweb.com> References: <573EC62F.4090401@lucidity.plus.com> <75115580-7a63-4cfb-9689-28eb84b580a0@googlegroups.com> <574032db$0$22142$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5/21/2016 3:05 AM, Steven D'Aprano wrote: > On Sat, 21 May 2016 03:18 pm, Rustom Mody wrote: > >> Given that for the most part, most of us are horribly uneducated [ >> > http://www.creativitypost.com/education/9_elephants_in_the_classroom_that_should_unsettle_us >> ] >> how do we go about correcting our wrong primacies? > > > An interesting article, but very US-centric. > > Nevertheless, let's not forget that the general school system is designed to > churn out more-or-less identical workers capable of pulling the levers on > late 19th and early 20th century machines for their bosses. The fact that > it does more than that, even if badly, is a bonus. Under various proposals in the U.S., everyone will soon learn how to program and/or become a computer scientist. Won't be long before some snotty-nosed brat graduates from preschool, takes a look at your code, and poops in his diapers. He will then dips his finger into his diaper, write on the whiteboard how your code can be written in a single line, and summary dismiss you with security escorting you off the premises. Gotta love the future. :) Thank you, Chris R. From christopher_reimer at icloud.com Sat May 21 11:59:46 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Sat, 21 May 2016 08:59:46 -0700 Subject: How do I subclass the @property setter method? In-Reply-To: <574021B5.1010408@gmx.de> References: <2200dfd0-7470-0c86-c015-a6a14a2f51f2@icloud.com> <574021B5.1010408@gmx.de> Message-ID: <859deecb-8630-a2b2-4976-bde4d4788910@icloud.com> On 5/21/2016 1:52 AM, Dirk B?chle wrote: > Hi Christopher, > > On 20.05.2016 20:50, Christopher Reimer wrote: >> Greetings, >> >> My chess engine has a Piece class with the following methods that use >> the @property decorator to read and write the position value. >> > > slightly off topic: is your chess engine available in a public repo > somewhere? I think I've started to implement something similar (display > of chess boards and wrapping different chess engines), so I'd like to > have a look. ;) Not at this time. I'll send a post to the list when I make the code available. My chess engine doesn't do much beyond displaying a text-only board. Since a chess engine presents an unlimited number of programming challenges, I'm using it to learn the finer points of Python. Thank you, Chris R. From __peter__ at web.de Sat May 21 12:54:44 2016 From: __peter__ at web.de (Peter Otten) Date: Sat, 21 May 2016 18:54:44 +0200 Subject: Image loading problem References: <743e5fbe-3a6c-49ea-ac5e-9bc055f652b5@googlegroups.com> Message-ID: sweating_ant at yahoo.com wrote: > I am working on an image project, and I can display my image in main(). I mean, I can load my image in my main(). Needless, it is awkward. I am trying to load my image with a function, but got an empty image window popped up, no image content loaded. Please take a look at code: > > > > rom Tkinter import * > > def load_img(win): > img = PhotoImage(file="xxx.gif") > Label(win, image=img).pack() > > win = Tk() > load_img(win) > win.mainloop() > > Somebody can help me out? Thanks! It's not your fault, there's an odd quirk in the library: you have to keep a reference of the PhotoImage instance around to prevent the image from being garbage-collected. For example from Tkinter import * def load_img(win): global img img = PhotoImage(file="xxx.gif") Label(win, image=img).pack() win = Tk() load_img(win) win.mainloop() will work because img is now a global that is not deleted until the script ends (but don't run the function twice -- then the first image will disappear). From marko at pacujo.net Sat May 21 13:08:32 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Sat, 21 May 2016 20:08:32 +0300 Subject: Education [was Re: for / while else doesn't make sense] References: <573EC62F.4090401@lucidity.plus.com> <75115580-7a63-4cfb-9689-28eb84b580a0@googlegroups.com> <574032db$0$22142$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87y4732van.fsf@elektro.pacujo.net> Christopher Reimer : > Under various proposals in the U.S., everyone will soon learn how to > program and/or become a computer scientist. Won't be long before some > snotty-nosed brat graduates from preschool, takes a look at your code, > and poops in his diapers. He will then dips his finger into his > diaper, write on the whiteboard how your code can be written in a > single line, and summary dismiss you with security escorting you off > the premises. > > Gotta love the future. :) Unfortunately, most CS graduates don't seem to know how to program. Yes, some highschoolers could excel in the post of a senior software engineer -- I've had the privilege of working alongside several specimens. However, it has been known for half a century that good developers are hard to come by. I think it is essential to learn the principles of programming just like it is essential to learn the overall principles of nuclear fission or be able to locate China on the map. However, a small minority of humanity will ever earn a living writing code. At the same time, it may be that in the not-too-distant future, the *only* jobs available will be coding jobs as we start to take the finishing steps of automating all manufacturing, transportation and services. Then, we will have a smallish class of overworked coders who have no use or time for money and vast masses of jobless party-goers who enjoy the fruits of the coders' labor. Marko From pkpearson at nowhere.invalid Sat May 21 13:10:03 2016 From: pkpearson at nowhere.invalid (Peter Pearson) Date: 21 May 2016 17:10:03 GMT Subject: Image loading problem References: <743e5fbe-3a6c-49ea-ac5e-9bc055f652b5@googlegroups.com> Message-ID: On Sat, 21 May 2016 08:22:41 -0700 (PDT), sweating_ant at yahoo.com wrote: > > I am working on an image project, and I can display my image in > main(). I mean, I can load my image in my main(). Needless, it is > awkward. I am trying to load my image with a function, but got an > empty image window popped up, no image content loaded. Please take a > look at code: > > rom Tkinter import * > > def load_img(win): > img = PhotoImage(file="xxx.gif") > Label(win, image=img).pack() > > win = Tk() > load_img(win) > win.mainloop() > This seems to work: ----------- from Tkinter import Tk, PhotoImage, Label def load_img(win): img = PhotoImage(file="/home/peter/dow.gif") Label(win, image=img).pack() return img win = Tk() img = load_img(win) win.mainloop() ----------- I guess the problem was that the "img" created in load_img went out of scope and was deleted when load_img returned. I just tried the "return img" as an experiment to keep it from going out of scope; from a software-engineering standpoint this might be a gross kluge. Baffling? Personally, I never got past feeling generally bewildered in Tkinter-land, and now I use matplotlib for all my graphical needs. I had great hopes for PyGUI at one point, but I hear very little about it now. -- To email me, substitute nowhere->runbox, invalid->com. From davros at bellaliant.net Sat May 21 13:13:29 2016 From: davros at bellaliant.net (John McKenzie) Date: Sat, 21 May 2016 17:13:29 GMT Subject: Python and GPSD Message-ID: Good day, all. I need help using the Python bindings for GPSD. Specifically, I would like to take a latitude reading, put it in a variable and use it later. The problem is that every example I see involves constantly taking changing readings. That part I have working for myself by following existing examples. If I put any code in any kind of loop my latitude variable will keep changing. If I do not put in a loop, it will appear to be zero. (I do not live on the equator.) The Python GPS module is installed, cgps works fine, other aspects of the script involving GPS work fine. I want to be able to turn on a single board computing device, have it get a latitude reading, put that coordinate into a variable, then go along its merry way using that variable for the rest of the script. (Running my script upon boot, running the GPSD service, etc, is all fine and working.) My script: http://hastebin.com/zuwamuqoxe.coffee On line 37 I try to define an initial latitude variable, intLatitude. It comes up as zero right now. I need it to be whatever the GPS says for latitude when the script starts and stay that way. Help is appreciated. Apologies to those who read my script for the poor code quality. Learned a little Python last year, then only started again with it recently. Still a Python newbie, and am rushed for this particular project. Thanks. From gherron at digipen.edu Sat May 21 13:16:48 2016 From: gherron at digipen.edu (Gary Herron) Date: Sat, 21 May 2016 10:16:48 -0700 Subject: Image loading problem In-Reply-To: <743e5fbe-3a6c-49ea-ac5e-9bc055f652b5@googlegroups.com> References: <743e5fbe-3a6c-49ea-ac5e-9bc055f652b5@googlegroups.com> Message-ID: <57409800.3090401@digipen.edu> On 05/21/2016 08:22 AM, sweating_ant at yahoo.com wrote: > Hi All, > > > I am working on an image project, and I can display my image in main(). I mean, I can load my image in my main(). Needless, it is awkward. I am trying to load my image with a function, but got an empty image window popped up, no image content loaded. Please take a look at code: > > > > rom Tkinter import * > > def load_img(win): > img = PhotoImage(file="xxx.gif") > Label(win, image=img).pack() > > win = Tk() > load_img(win) > win.mainloop() > > Somebody can help me out? Thanks! > I believe this the problem (However It's been long since I used Tkinter, so be warned ... ): The function load_img creates a local variable named img which goes out of scope and is deleted immediately when the function returns. However, Tkinter needs you to keep that image around as long as the Label uses it. So, some solutions are: keep_me = [] # Global for keeping references to images def load_img(win): img = PhotoImage(file="xxx.gif") keep_me.append(img) Label(win, image=img).pack() or def load_img(win): img = PhotoImage(file="xxx.gif") Label(win, image=img).pack() return img saved_img = load_img(win) ... Gary Herron -- Dr. Gary Herron Professor of Computer Science DigiPen Institute of Technology (425) 895-4418 From tjreedy at udel.edu Sat May 21 15:23:42 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 21 May 2016 15:23:42 -0400 Subject: Image loading problem In-Reply-To: References: <743e5fbe-3a6c-49ea-ac5e-9bc055f652b5@googlegroups.com> Message-ID: On 5/21/2016 12:54 PM, Peter Otten wrote: > sweating_ant at yahoo.com wrote: > >> I am working on an image project, and I can display my image in main(). I > mean, I can load my image in my main(). Needless, it is awkward. I am trying > to load my image with a function, but got an empty image window popped up, > no image content loaded. Please take a look at code: >> >> >> >> rom Tkinter import * >> >> def load_img(win): >> img = PhotoImage(file="xxx.gif") >> Label(win, image=img).pack() >> >> win = Tk() >> load_img(win) >> win.mainloop() >> >> Somebody can help me out? Thanks! > > It's not your fault, there's an odd quirk in the library: you have to keep a > reference of the PhotoImage instance around to prevent the image from being > garbage-collected. For example > > from Tkinter import * > > def load_img(win): > global img > > img = PhotoImage(file="xxx.gif") > Label(win, image=img).pack() > > win = Tk() > load_img(win) > win.mainloop() > > will work because img is now a global that is not deleted until the script > ends (but don't run the function twice -- then the first image will > disappear). The usual procedure is to create an App class and make images attributes of the App instance. Then load_img would be a method and the first line would be 'self.img = ...'. One can also have a list of images, to use in a slide show, or a dict of images, so users can select. -- Terry Jan Reedy From random832 at fastmail.com Sat May 21 15:55:05 2016 From: random832 at fastmail.com (Random832) Date: Sat, 21 May 2016 15:55:05 -0400 Subject: Image loading problem In-Reply-To: References: <743e5fbe-3a6c-49ea-ac5e-9bc055f652b5@googlegroups.com> Message-ID: <1463860505.689253.614738601.76B18B37@webmail.messagingengine.com> On Sat, May 21, 2016, at 12:54, Peter Otten wrote: > It's not your fault, there's an odd quirk in the library: you have to > keep a reference of the PhotoImage instance around to prevent the > image from being garbage-collected. Just out of curiosity, why is this a "quirk" and not a bug? Why isn't the reference held by the Label? From python at lucidity.plus.com Sat May 21 16:48:09 2016 From: python at lucidity.plus.com (Erik) Date: Sat, 21 May 2016 21:48:09 +0100 Subject: for / while else doesn't make sense In-Reply-To: <57403ade$0$1593$c3e8da3$5496439d@news.astraweb.com> References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> <57400C22.7050706@lucidity.plus.com> <57403ade$0$1593$c3e8da3$5496439d@news.astraweb.com> Message-ID: <5740C989.4070408@lucidity.plus.com> On 21/05/16 11:39, Steven D'Aprano wrote: > Just for the record, that's not my mental model *now*. Sure. And I should have written "one's mental model" - the model of anyone writing that code (not you personally) who thought the same at the time. > It took me a long time to work out what for...else was actually doing, but > some years ago I finally managed to do so. Coming from a partly assembler background, I happened to get it straight away just because the thought of separate jump targets for the loop condition being satisfied and breaking out of the loop being different was quite natural (and a very nice feature to have in a high level language!). > But you do ask a good question. Why isn't for...else with no break a syntax > error? I suppose it could be. But that's a *stylistic* question, and Python > generally treats that as "none of the compiler's business". It's not the > business of the compiler to enforce good code, only legal code. On 21/05/16 08:29, Chris Angelico wrote: > It's up to the linter, and ONLY the linter, to tell you about this. Apologies for joining two threads here, but my reply is the same to both, so it makes sense to me. Let me tell you a story ;) Back in the mid-to-late 1980s I worked with C compilers on hardware that could take several minutes to compile even a fairly trivial program. They errored on syntactically incorrect code and happily compiled syntactically correct code. Sometimes the output of the compiler wouldn't execute as expected because of "undefined behaviour" of some parts of the language (which the compilers could quite legally accept but would not produce valid code for - even illegal ops are fair game at that point). They would create valid code for the valid syntax of buggy code ("if (x = y) { foo(); }") without a whimper. At that time, we had a program called 'lint'. Every so often we might run it on our sources and find all sorts of questionable behaviour that our code might have that we should look harder at (such as that above). We didn't run it all the time because it took so much longer to run than the compiler itself. Over time, some compilers started adding the checks and advisories that "lint" gave to their messages. For some branded compilers ("Green Hills", "SAS/C"), this even became a selling point. They started to do this while still retaining the original compilation speed. Things got faster, more was added to the compiler, and once the compiler started to do everything it did and more, lint died(*). And now, today, the compilers all do far more than the original 'lint' program did in almost zero time every time some source is compiled. It is free; it is not something one has to remember to run every so often. So, back to Python ;) The responses of "we can all write suspect/bad/ineffectual code - so just run the linter" takes me back those 30 years to when we HAD to do that with our C code too ... There must be a better way. I realise that Python has the issue that sometimes the person doing the compilation is the end user (by virtue of them executing a .py file), so lint-style compiler warnings aren't really appropriate - and that's the reason why I suggested a syntax error: I understand that it's not really any such thing, but it's all I had to work with and it ensured such code could not get as far as the end user. So I guess my question is perhaps whether Python compilers should start to go down the same path that C compilers did 30 years ago (by starting to include some linter functionality) but in a way that only outputs the messages to developers and not end users. Also, the current "pylint" blurs the edges between style (identifier names) and questionable code ("for/else" with no "break"). E. <-- waiting to be shot down again. (*) Though to be fair, there are now even more deeply checking (commercial) static tools available, which effectively fill the gap that 'lint' used to. From michael.selik at gmail.com Sat May 21 22:55:40 2016 From: michael.selik at gmail.com (Michael Selik) Date: Sun, 22 May 2016 02:55:40 +0000 Subject: for / while else doesn't make sense In-Reply-To: <5740C989.4070408@lucidity.plus.com> References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> <57400C22.7050706@lucidity.plus.com> <57403ade$0$1593$c3e8da3$5496439d@news.astraweb.com> <5740C989.4070408@lucidity.plus.com> Message-ID: On Sat, May 21, 2016, 4:52 PM Erik wrote: > So I guess my question is perhaps whether Python compilers should start > to go down the same path that C compilers did 30 years ago (by starting > to include some linter functionality) > Well, there's that whole optional type hints thing. You should be asking how to preserve Python's simplicity while adding more static analysis features. > From steve at pearwood.info Sat May 21 22:57:15 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 22 May 2016 12:57:15 +1000 Subject: for / while else doesn't make sense References: <573dfc0f$0$1586$c3e8da3$5496439d@news.astraweb.com> <573e54f7$0$1615$c3e8da3$5496439d@news.astraweb.com> <57400C22.7050706@lucidity.plus.com> <57403ade$0$1593$c3e8da3$5496439d@news.astraweb.com> <5740C989.4070408@lucidity.plus.com> Message-ID: <5741200d$0$1588$c3e8da3$5496439d@news.astraweb.com> On Sun, 22 May 2016 06:48 am, Erik wrote: > Let me tell you a story ;) Back in the mid-to-late > 1980s I worked with C compilers on hardware that could take several > minutes to compile even a fairly trivial program. They errored on > syntactically incorrect code and happily compiled syntactically correct > code. Sometimes the output of the compiler wouldn't execute as expected > because of "undefined behaviour" of some parts of the language (which > the compilers could quite legally accept but would not produce valid > code for - even illegal ops are fair game at that point). They would > create valid code for the valid syntax of buggy code ("if (x = y) { > foo(); }") without a whimper. Don't get me started about C and undefined behaviour. Fortunately, Python has nothing even remotely like C undefined behaviour. > At that time, we had a program called 'lint'. [...] > And now, today, the compilers all do far more than the original 'lint' > program did in almost zero time every time some source is compiled. It > is free; it is not something one has to remember to run every so often. This is certainly not the case for Python. With C, you run your compiler+linter once, and it builds an executable which can then run without the compiler or linter. With Python, *every time you run* the code, the compiler runs. The compiler is the interpreter. It can, sometimes, skip some of the compilation steps (parsing of source code) by use of cached byte-code files, but not all of them. Including a linter will increase the size of the compiler significantly, which much be distributed or installed for even the smallest Python script, and it will have runtime implications re compilation time, execution time, and memory use. If you make the linter optional, say, a Python module that you install separately and run only if you choose, then you have the status quo. > So, back to Python ;) > > The responses of "we can all write suspect/bad/ineffectual code - so > just run the linter" takes me back those 30 years to when we HAD to do > that with our C code too ... > > There must be a better way. Yes. And the better way is... don't write a language where you NEED a linter because the language specification is so fecking *insane* that no human being can reliably write correct code without running into undefined behaviour which *can and will* have effects that propagate in both directions, contaminating code which is correct in unpredictible ways. https://blogs.msdn.microsoft.com/oldnewthing/20140627-00/?p=633/ One difference between C and Python is that most of the things which Python linters look at don't actually have implications for correctness. Here are a few of the features that Pylint looks at: * line length; * variable naming standards; * unused imports; * unnecessary semi-colons; * use of deprecated modules; and a complete list here: http://pylint-messages.wikidot.com/all-codes As you can see, most of the actual errors Pylint will pick up would result in a runtime error, e,g, opening a file with an invalid mode. Most codes are for code quality issues, related to maintenance issues, not bug detection. Coming back to for...else with no break, the behaviour is perfectly well-defined, and does exactly what it is documented as doing. It's hard to see that it is a "bug" for something to do what exactly what it is designed to do. If somebody, for their own idiosyncratic reasons, wants to write: for x in seq: spam() else: eggs() (Note: I've done this -- see below.) and the language insists on a break, they will just pointlessly defeat the compiler: shutup_stupid_compiler = False for x in seq: if shutup_stupid_compiler: break spam else: eggs Thus entering an arms race where the compiler is seen as something to be silenced rather than something that helps you. I said that I've written for...else with no break. Why would I do such a thing? Because I wanted to clearly mark that the code in the else was a unit of code that went with the for-loop. Code displays varying levels of cohesiveness. Python lets you group related code in four levels: - the function/method; - the class; - the module; - the package but sometimes you have code which is smaller than a function that needs to be considered as a unit, but is not enough to justify putting it into a function. When that happens, we usually delimit it with comments, or sometimes even just a blank line: code that goes together different bunch of code that goes together So, when I had a bunch of code that included a for-loop, and something immediately after the for-loop, I could have written: for x in seq: block code that goes with the loop different bunch of code but I thought it communicated the grouping better to write it as: for x in seq: block else: # runs after the for code that goes with the loop different bunch of code I've since changed my mind. That's too subtle and too idiosyncratic for my liking. It clashes with the keyword "else", which I maintain is badly named. If it was named "next", which I maintain describes what it does much better, then things might be different, but given the status quo, I've gone back to doing it the old-fashioned way, with a comment. But the point is, that's a matter of *taste*, not a matter for the compiler. If somebody else wanted to do it my way, well, that's between them and whoever else works on their code. You probably wouldn't want the compiler to raise a syntax error because you put a blank line or a comment somewhere the compiler writer disapproved off. For example, some people insist that the first line of code must follow immediately after the docstring, some prefer to leave a blank line: def spam(): """Docs""" code def eggs(): """Docs""" code Your suggestion to raise a syntax error in the case of for...else without break strikes me as no different from the idea that we should raise a syntax error if there is/isn't a blank line after the docstring. (Choose one.) -- Steven From huey.y.jiang at gmail.com Sun May 22 00:01:47 2016 From: huey.y.jiang at gmail.com (huey.y.jiang at gmail.com) Date: Sat, 21 May 2016 21:01:47 -0700 (PDT) Subject: Image loading problem In-Reply-To: <743e5fbe-3a6c-49ea-ac5e-9bc055f652b5@googlegroups.com> References: <743e5fbe-3a6c-49ea-ac5e-9bc055f652b5@googlegroups.com> Message-ID: <85ec4ea5-3e5d-481f-85f8-de3b9d372bb1@googlegroups.com> Thanks so much! All of methods works! From flebber.crue at gmail.com Sun May 22 03:26:39 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Sun, 22 May 2016 00:26:39 -0700 (PDT) Subject: what is new with int conversion in Python 3 Message-ID: <648b46dc-c962-4f95-83bc-faa65ae3c51a@googlegroups.com> I am doing a passage in a book that was written for python 2 i am writing everything in 3. This is the author Ivan Idris code to show time difference between python and numpy arrays. The only edit I made was to fix the print statements. #!/usr/bin/env/python import sys from datetime import datetime import numpy as np """ This program demonstrates vector addition the Python way. Run from the command line as follows python vectorsum.py n where n is an integer that specifies the size of the vectors. The first vector to be added contains the squares of 0 up to n. The second vector contains the cubes of 0 up to n. The program prints the last 2 elements of the sum and the elapsed time. """ def numpysum(n): a = np.arange(n) ** 2 b = np.arange(n) ** 3 c = a + b return c def pythonsum(n): a = range(n) b = range(n) c = [] for i in range(len(a)): a[i] = i ** 2 b[i] = i ** 3 c.append(a[i] + b[i]) return c size = int(sys.argv[1]) start = datetime.now() c = pythonsum(size) delta = datetime.now() - start print("The last 2 elements of the sum", c[-2:]) print("PythonSum elapsed time in microseconds", delta.microseconds) start = datetime.now() c = numpysum(size) delta = datetime.now() - start print("The last 2 elements of the sum", c[-2:]) print("NumPySum elapsed time in microseconds", delta.microseconds) However when I run this I get a valuerror. So either something has changed with int or datetime I cannot google a consistent answer. --------------------------------------------------------------------------- ValueError Traceback (most recent call last) in () 37 return c 38 ---> 39 size = int(sys.argv[1]) 40 41 start = datetime.now() ValueError: invalid literal for int() with base 10: '-f' Had this before? Sayth From rosuav at gmail.com Sun May 22 03:51:51 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 22 May 2016 17:51:51 +1000 Subject: what is new with int conversion in Python 3 In-Reply-To: <648b46dc-c962-4f95-83bc-faa65ae3c51a@googlegroups.com> References: <648b46dc-c962-4f95-83bc-faa65ae3c51a@googlegroups.com> Message-ID: On Sun, May 22, 2016 at 5:26 PM, Sayth Renshaw wrote: > However when I run this I get a valuerror. So either something has changed with int or datetime I cannot google a consistent answer. > > > --------------------------------------------------------------------------- > ValueError Traceback (most recent call last) > in () > 37 return c > 38 > ---> 39 size = int(sys.argv[1]) > 40 > 41 start = datetime.now() > > ValueError: invalid literal for int() with base 10: '-f' > > Had this before? Look carefully at the error, and the line that it's coming up on. Firstly, you can rule out datetime, as nothing has been done with datetime except import it. Secondly, the invalid literal doesn't look like a decimal number at all; in fact, it looks to me like a flag of some sort. Try adding this above the failing line: print(sys.argv) And also, try running this at the terminal: $ file /usr/bin/env/python How are you invoking Python? The shebang looks wrong; perhaps it should be "/usr/bin/env python" (note the space where you have an additional slash), but perhaps it's not even being significant here. ChrisA From steve at pearwood.info Sun May 22 03:55:27 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 22 May 2016 17:55:27 +1000 Subject: what is new with int conversion in Python 3 References: <648b46dc-c962-4f95-83bc-faa65ae3c51a@googlegroups.com> Message-ID: <574165f0$0$1609$c3e8da3$5496439d@news.astraweb.com> On Sun, 22 May 2016 05:26 pm, Sayth Renshaw wrote: > I am doing a passage in a book that was written for python 2 i am writing > everything in 3. [...] > However when I run this I get a valuerror. So either something has changed > with int or datetime I cannot google a consistent answer. Neither. It will help if you read the error message: > ValueError: invalid literal for int() with base 10: '-f' You're passing -f as the first argument to the script, instead of an integer. Look at the command you are typing. My guess is that you are typing something like python vectorsum.py -f 27 instead of python vectorsum.py 27 -- Steven From __peter__ at web.de Sun May 22 04:32:48 2016 From: __peter__ at web.de (Peter Otten) Date: Sun, 22 May 2016 10:32:48 +0200 Subject: what is new with int conversion in Python 3 References: <648b46dc-c962-4f95-83bc-faa65ae3c51a@googlegroups.com> Message-ID: Sayth Renshaw wrote: Read carefully: > Run from the command line as follows > > python vectorsum.py n > > where n is an integer that specifies the size of the vectors. So to run the script with Python 3 you could do $ python3 vectorsum.py 42 in the shell. This implicitly sets sys.argv[1] to "42" so that the conversion size = int(sys.argv[1]) can succeed. This conversion failed because as the traceback indicates > ValueError Traceback (most recent call last) > in () > 37 return c > 38 > ---> 39 size = int(sys.argv[1]) > 40 > 41 start = datetime.now() > > ValueError: invalid literal for int() with base 10: '-f' > sys.argv[1] is "-f" which is not a valid (base-10) integer. This value probably got into ipython3 because you invoked it with something like $ ipython3 console -f foo Python 3.4.3 (default, Oct 14 2015, 20:28:29) Type "copyright", "credits" or "license" for more information. IPython 1.2.1 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. In [1]: import vectorsum --------------------------------------------------------------------------- ValueError Traceback (most recent call last) in () ----> 1 import vectorsum /home/petto/vectorsum.py in () 37 return c 38 ---> 39 size = int(sys.argv[1]) 40 41 start = datetime.now() ValueError: invalid literal for int() with base 10: '-f' In [2]: ! python3 vectorsum.py 42 The last 2 elements of the sum [65600, 70602] PythonSum elapsed time in microseconds 106 The last 2 elements of the sum [65600 70602] NumPySum elapsed time in microseconds 121 By the way, I had to fix another problem to make the "In [2]:" invocation work. Let's see if you can do that yourself. From __peter__ at web.de Sun May 22 05:03:15 2016 From: __peter__ at web.de (Peter Otten) Date: Sun, 22 May 2016 11:03:15 +0200 Subject: Image loading problem References: <743e5fbe-3a6c-49ea-ac5e-9bc055f652b5@googlegroups.com> <1463860505.689253.614738601.76B18B37@webmail.messagingengine.com> Message-ID: Random832 wrote: > On Sat, May 21, 2016, at 12:54, Peter Otten wrote: >> It's not your fault, there's an odd quirk in the library: you have to >> keep a reference of the PhotoImage instance around to prevent the >> image from being garbage-collected. > > Just out of curiosity, why is this a "quirk" and not a bug? I agree that this is at least a usability bug, but I think someone with a clue (Fredrik Lundh?) wrote it that way intentionally. > Why isn't the reference held by the Label? I can only speculate: to avoid a tcl/python reference cycle? You might file a bug report and see what comes of it. From flebber.crue at gmail.com Sun May 22 06:59:45 2016 From: flebber.crue at gmail.com (Sayth Renshaw) Date: Sun, 22 May 2016 03:59:45 -0700 (PDT) Subject: what is new with int conversion in Python 3 In-Reply-To: <648b46dc-c962-4f95-83bc-faa65ae3c51a@googlegroups.com> References: <648b46dc-c962-4f95-83bc-faa65ae3c51a@googlegroups.com> Message-ID: On Sunday, 22 May 2016 17:26:51 UTC+10, Sayth Renshaw wrote: > I am doing a passage in a book that was written for python 2 i am writing everything in 3. > > This is the author Ivan Idris code to show time difference between python and numpy arrays. The only edit I made was to fix the print statements. > > #!/usr/bin/env/python > > import sys > from datetime import datetime > import numpy as np > > """ > This program demonstrates vector addition the Python way. > Run from the command line as follows > > python vectorsum.py n > > where n is an integer that specifies the size of the vectors. > > The first vector to be added contains the squares of 0 up to n. > The second vector contains the cubes of 0 up to n. > The program prints the last 2 elements of the sum and the elapsed time. > """ > > def numpysum(n): > a = np.arange(n) ** 2 > b = np.arange(n) ** 3 > c = a + b > > return c > > def pythonsum(n): > a = range(n) > b = range(n) > c = [] > > for i in range(len(a)): > a[i] = i ** 2 > b[i] = i ** 3 > c.append(a[i] + b[i]) > > return c > > size = int(sys.argv[1]) > > start = datetime.now() > c = pythonsum(size) > delta = datetime.now() - start > print("The last 2 elements of the sum", c[-2:]) > print("PythonSum elapsed time in microseconds", delta.microseconds) > > start = datetime.now() > c = numpysum(size) > delta = datetime.now() - start > print("The last 2 elements of the sum", c[-2:]) > print("NumPySum elapsed time in microseconds", delta.microseconds) > > > However when I run this I get a valuerror. So either something has changed with int or datetime I cannot google a consistent answer. > > > --------------------------------------------------------------------------- > ValueError Traceback (most recent call last) > in () > 37 return c > 38 > ---> 39 size = int(sys.argv[1]) > 40 > 41 start = datetime.now() > > ValueError: invalid literal for int() with base 10: '-f' > > Had this before? > > Sayth Thank you all, I was way wrong. I was invoking it from within ipython notebook which I don't usually use. Thanks' Sayth From jon+usenet at unequivocal.co.uk Sun May 22 10:15:35 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Sun, 22 May 2016 14:15:35 -0000 (UTC) Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-21, Chris Angelico wrote: > On Sat, May 21, 2016 at 10:35 AM, Jon Ribbens > wrote: >> To be fair, I'm very sympathetic to that argument. I think programming >> languages should never magically produce floats out of nowhere unless >> the programmer has explicitly done "import float" or "float('3.23')" >> or somesuch. They're misunderstood so often that any convenience >> they provide is outweighed by the danger they bring. >> >> "(1/10) * (1/10) * 10 != (1/10)" anyone? I was distinctly unhappy with >> the Python 3 "2/3 ~= 0.6666" thing and regard it as a very retrograde >> change. > > The trouble is, what SHOULD 2/3 return? > > * An integer? Makes a lot of sense to a C programmer. Not so much to > someone who is expecting a nonzero value. This isn't terrible (hey, > Python 2 managed with it no problem), but will definitely confuse a > number of people. Yes, it should return an integer - and not because I think Python should behave like C on principle, but because: Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. and floats are complicated. > * A float? That's what we currently have. Not perfect, but it's going > to confuse less people than 0 will. That's a trap for those people though - it lulls them into thinking that they understand what's going on, when in fact they don't, because they don't understand floats, because almost nobody understands floats. So they don't understand their program, and - even worse - they don't know that they don't understand it. Programming languages should do what they are told, and very little more. They should not wander off on surprising jaunts of their own invention out of the control of the programmer. It should be possible to know and understand the language, or at least the subset of it that you are likely to need for your everyday purposes. Floats are generally not understood, so they shouldn't be suddenly turning up un-called for. Python generally sticks to this idea very well, which is one of the things that I think make it an excellent programming language, so it is a shame that in the Python 2 to Python 3 change when mistakes were being rectified, a new one was introduced. From marko at pacujo.net Sun May 22 10:58:24 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Sun, 22 May 2016 17:58:24 +0300 Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87d1oe2l7z.fsf@elektro.pacujo.net> Jon Ribbens : > On 2016-05-21, Chris Angelico wrote: >> The trouble is, what SHOULD 2/3 return? > > [...] > > Yes, it should return an integer - and not because I think Python > should behave like C on principle, but because: > > Explicit is better than implicit. > Simple is better than complex. > Complex is better than complicated. > > and floats are complicated. Scheme has the best of both worlds: scheme@(guile-user)> 2/3 $1 = 2/3 scheme@(guile-user)> (exact->inexact $1) $2 = 0.6666666666666666 > That's a trap for those people though - it lulls them into thinking > that they understand what's going on, when in fact they don't, because > they don't understand floats, because almost nobody understands > floats. So they don't understand their program, and - even worse - > they don't know that they don't understand it. I don't understand this rant. Numeric programming is one of the oldest uses for computers. Rounding errors have been there since the beginning. If you think people have a hard time getting floats, integers are at least as hard to get. How about classes, closures, threads, asyncio...? Python ought to be the perfect language for seasoned experts. It doesn't need to be dumbed down for noobs. Marko From jon+usenet at unequivocal.co.uk Sun May 22 11:09:30 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Sun, 22 May 2016 15:09:30 -0000 (UTC) Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <87d1oe2l7z.fsf@elektro.pacujo.net> Message-ID: On 2016-05-22, Marko Rauhamaa wrote: > Jon Ribbens : >> That's a trap for those people though - it lulls them into thinking >> that they understand what's going on, when in fact they don't, because >> they don't understand floats, because almost nobody understands >> floats. So they don't understand their program, and - even worse - >> they don't know that they don't understand it. > > I don't understand this rant. And I don't understand your use of the word "rant". > Numeric programming is one of the oldest uses for computers. > Rounding errors have been there since the beginning. If you think > people have a hard time getting floats, integers are at least as > hard to get. That is clearly nonsense. > How about classes, closures, threads, asyncio...? People don't tend to think of those as simple things that they already understand when they don't. Also, those things don't tend to turn up to the party uninvited. From steve at pearwood.info Sun May 22 11:19:04 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 23 May 2016 01:19:04 +1000 Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> Message-ID: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> On Mon, 23 May 2016 12:15 am, Jon Ribbens wrote: > On 2016-05-21, Chris Angelico wrote: >> On Sat, May 21, 2016 at 10:35 AM, Jon Ribbens >> wrote: >>> To be fair, I'm very sympathetic to that argument. I think programming >>> languages should never magically produce floats out of nowhere unless >>> the programmer has explicitly done "import float" or "float('3.23')" >>> or somesuch. They're misunderstood so often that any convenience >>> they provide is outweighed by the danger they bring. >>> >>> "(1/10) * (1/10) * 10 != (1/10)" anyone? I was distinctly unhappy with >>> the Python 3 "2/3 ~= 0.6666" thing and regard it as a very retrograde >>> change. >> >> The trouble is, what SHOULD 2/3 return? >> >> * An integer? Makes a lot of sense to a C programmer. Not so much to >> someone who is expecting a nonzero value. This isn't terrible (hey, >> Python 2 managed with it no problem), but will definitely confuse a >> number of people. > > Yes, it should return an integer - and not because I think Python > should behave like C on principle, but because: > > Explicit is better than implicit. > Simple is better than complex. > Complex is better than complicated. > > and floats are complicated. How is this any better though? Complicated or not, people want to divide 1 by 2 and get 0.5. That is the functional requirement. Furthermore, they want to use the ordinary division symbol / rather than having to import some library or call a function. Having 1/2 return 0 (as Python 2 does by default) doesn't make the language any less complicated. It doesn't avoid the complexity of floats, it merely breaks the principle of least surprise, and forces the programmer to add what they consider to be an unnecessary ".0" to one or the other of the operands. Swapping to a base-10 float will be numerically even worse than the binary floats we use now. Swapping to rationals add complexity and performance issues. So whatever you do, there is complexity and annoyance. >> * A float? That's what we currently have. Not perfect, but it's going >> to confuse less people than 0 will. > > That's a trap for those people though - it lulls them into thinking > that they understand what's going on, when in fact they don't, > because they don't understand floats, because almost nobody > understands floats. So they don't understand their program, and > - even worse - they don't know that they don't understand it. And how does forcing them to write 1.0/2 solve that? Or (hypothetical) float.divide(1, 2) if you want to be even more explicit :-) > Programming languages should do what they are told, and very little > more. Okay, now I'm confused. How is 1/2 returning 0.5 the language not doing what you've told it to do? > They should not wander off on surprising jaunts of their own > invention out of the control of the programmer. It should be possible > to know and understand the language, or at least the subset of it > that you are likely to need for your everyday purposes. Floats are > generally not understood, so they shouldn't be suddenly turning up > un-called for. How are they uncalled for? > Python generally sticks to this idea very well, which is one of the > things that I think make it an excellent programming language, so it > is a shame that in the Python 2 to Python 3 change when mistakes were > being rectified, a new one was introduced. *shrug* I've programmed in Python using classic integer division and true division, and in my experience and opinion, classic division is a real pain to work with. You're forever having to cast things to float or write .0 literals just to convince the interpreter to do division the way you expect. I suppose some language some day might experiment with swapping the operators, so that a/b is integer division and a//b is true division. -- Steven From rustompmody at gmail.com Sun May 22 11:26:40 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Sun, 22 May 2016 08:26:40 -0700 (PDT) Subject: for / while else doesn't make sense In-Reply-To: <87d1oe2l7z.fsf@elektro.pacujo.net> References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <87d1oe2l7z.fsf@elektro.pacujo.net> Message-ID: On Sunday, May 22, 2016 at 8:28:39 PM UTC+5:30, Marko Rauhamaa wrote: > Python ought to be the perfect language for seasoned experts. It doesn't > need to be dumbed down for noobs. There's a language you may have heard of that you'll LOVE -- C++ Or maybe Haskell On a somewhat more serious note: Speaking of Haskell there has recently been a spate of dissent in the Haskell with people like Mark Lenctzer, Eric Meijer etc saying they are quitting Haskell because its too hard to teach. [These names in roughly python-equivalents are like say Raymond Hettinger and Nick Coghlan] I'd say python has done an eminently decent job so far in being approachable +powerful. But of late its losing the edge in the noob-side at the cost of catering to cognoscenti. IMHO Pascal got things right (wrt pedagogy) that are wronger and wronger in the last 30 years; see http://blog.languager.org/2015/06/functional-programming-moving-target.html On a different note: MIT has replaced scheme by python. Cause to celebrate?? Depends... If being able to do more with less understanding is good... well maybe From rosuav at gmail.com Sun May 22 11:32:07 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 23 May 2016 01:32:07 +1000 Subject: for / while else doesn't make sense In-Reply-To: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Mon, May 23, 2016 at 1:19 AM, Steven D'Aprano wrote: > Okay, now I'm confused. How is 1/2 returning 0.5 the language not doing what > you've told it to do? That isn't the problem. With binary floats, 1/2 can be perfectly represented, so you have no trouble anywhere. The problem comes when you then try 1/5. What do you get? 3602879701896397/18014398509481984. Python shows that as 0.2. Then you do some more arithmetic, and the veil is pierced, and you discover that 1/5 doesn't actually return 0.2, but just something really really close to it - which it tells you is 0.2. I'm not saying that having 1/5 return 0 is better, but I'd like a broad acceptance that 0.2 is imperfect - that, in fact, *every* option is imperfect. ChrisA From marko at pacujo.net Sun May 22 11:50:59 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Sun, 22 May 2016 18:50:59 +0300 Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: <8737pa2isc.fsf@elektro.pacujo.net> Chris Angelico : > On Mon, May 23, 2016 at 1:19 AM, Steven D'Aprano wrote: >> Okay, now I'm confused. How is 1/2 returning 0.5 the language not doing what >> you've told it to do? > > That isn't the problem. With binary floats, 1/2 can be perfectly > represented, so you have no trouble anywhere. The problem comes when > you then try 1/5. What do you get? 3602879701896397/18014398509481984. > Python shows that as 0.2. Then you do some more arithmetic, and the > veil is pierced, and you discover that 1/5 doesn't actually return > 0.2, but just something really really close to it - which it tells you > is 0.2. > > I'm not saying that having 1/5 return 0 is better, but I'd like a > broad acceptance that 0.2 is imperfect - that, in fact, *every* option > is imperfect. Ah, that reminds me of an ancient joke: Ask an engineer what is two times two. He'll take out his slide rule, quickly move the slider and reply: "Approximately four." I remember learning my first programming language, Basic, when I was 16. One of the very first things to notice was the way "the computer" worked with approximations. Marko From jon+usenet at unequivocal.co.uk Sun May 22 11:52:40 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Sun, 22 May 2016 15:52:40 -0000 (UTC) Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-22, Steven D'Aprano wrote: > On Mon, 23 May 2016 12:15 am, Jon Ribbens wrote: >> Yes, it should return an integer - and not because I think Python >> should behave like C on principle, but because: >> >> Explicit is better than implicit. >> Simple is better than complex. >> Complex is better than complicated. >> >> and floats are complicated. > > How is this any better though? Complicated or not, people want to divide 1 > by 2 and get 0.5. That is the functional requirement. Furthermore, they > want to use the ordinary division symbol / rather than having to import > some library or call a function. That's a circular argument. You're defining the result as the requirement and then saying that proves the result is necessary. Clearly, people managed when 1/2 returned 0, and continue to do so today in Python 2 and other languages. > Having 1/2 return 0 (as Python 2 does by default) doesn't make the > language any less complicated. It doesn't avoid the complexity of > floats, it merely breaks the principle of least surprise, No, it *adheres* to the principle of least surprise. Floats appearing out of nowhere is surprising. Python 2's behaviour adhered to the principle, and Python 3's breaks it. >> That's a trap for those people though - it lulls them into thinking >> that they understand what's going on, when in fact they don't, >> because they don't understand floats, because almost nobody >> understands floats. So they don't understand their program, and >> - even worse - they don't know that they don't understand it. > > And how does forcing them to write 1.0/2 solve that? Because it forces them to consciously address the fact that they are asking for, and getting, floats, and that floats are not something the language is willingly to silently foist upon them. >> Programming languages should do what they are told, and very little >> more. > > Okay, now I'm confused. How is 1/2 returning 0.5 the language not doing what > you've told it to do? I didn't ask for floats, I got floats. That's how. >> They should not wander off on surprising jaunts of their own >> invention out of the control of the programmer. It should be possible >> to know and understand the language, or at least the subset of it >> that you are likely to need for your everyday purposes. Floats are >> generally not understood, so they shouldn't be suddenly turning up >> un-called for. > > How are they uncalled for? By... not being called for? I must admit I don't entirely understand your question. From rosuav at gmail.com Sun May 22 12:35:24 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 23 May 2016 02:35:24 +1000 Subject: for / while else doesn't make sense In-Reply-To: References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Mon, May 23, 2016 at 1:52 AM, Jon Ribbens wrote: > That's a circular argument. You're defining the result as the > requirement and then saying that proves the result is necessary. > Clearly, people managed when 1/2 returned 0, and continue to do so > today in Python 2 and other languages. > Python's int and float types are both approximations to a non-representable type called a "real number". You learned about numbers in your early childhood - you learned about the basic concepts like addition (you have one apple here and one apple there, so you have two apples), and division (you take those two apples and split them between four people by cutting them both in half). If you ask someone how much apple everyone gets when you divide one apple between two people, the answer should be "half an apple". Not "no apples" - of course there are situations where things are indivisible, but numbers themselves aren't, because there are times when you can indeed halve those apples just fine. The problem is that computers can't actually represent real numbers. We have to content ourselves with a variety of approximations, which we call numeric data types. Python then treats those data types as being a minor detail, and the underlying real number as important: >>> 1 == 1.0 == (1+0j) True >>> {1.0: "foo"}[1] 'foo' Now, there is the small problem that the numeric types can't be arranged into a perfect tower. If Python's integer were restricted to 2**32, you could automatically upcast any integer to a float losslessly, and you can already losslessly upcast a float to a complex simply by adding 0j to it. But most people don't work with numbers big enough to be unrepresentable in 64-bit IEEE floating point: >>> (1<<53)+1 9007199254740993 >>> (1<<53)+1 == (1<<53) False >>> (1<<53)+1.0 == (1<<53) True So for *most people*, this treatment works perfectly. An int will upcast to a float when you apply the division operator to it. An int or float will upcast to complex when you apply the exponentiation operator: >>> (-4)**0.5 (1.2246467991473532e-16+2j) Nearly everything stored in a computer is an abstraction that can leak. In this case, we can't perfectly represent real numbers or calculate with them, so we do the best we can. Binary floating point is far from perfect in purity, but it's not bad in practicality. Remind me what PEP 20 says about that? Gotcha. ChrisA From jon+usenet at unequivocal.co.uk Sun May 22 12:46:21 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Sun, 22 May 2016 16:46:21 -0000 (UTC) Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-22, Chris Angelico wrote: > Python's int and float types are both approximations to a > non-representable type called a "real number". Sorry, I have to stop you there as the entire premise of your post is clearly wrong. "int" is not "an approximation of real numbers", it's a model of the mathematical concept "integers", and it's not an approximation, and since the long/int unification you can't even overflow it as I understand things (barring ridiculous situations like running out of memory). From rustompmody at gmail.com Sun May 22 13:22:42 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Sun, 22 May 2016 10:22:42 -0700 (PDT) Subject: for / while else doesn't make sense In-Reply-To: References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sunday, May 22, 2016 at 10:20:11 PM UTC+5:30, Jon Ribbens wrote: > On 2016-05-22, Chris Angelico wrote: > > Python's int and float types are both approximations to a > > non-representable type called a "real number". > > Sorry, I have to stop you there as the entire premise of your post is > clearly wrong. "int" is not "an approximation of real numbers", it's > a model of the mathematical concept "integers", and it's not an > approximation, and since the long/int unification you can't even > overflow it as I understand things (barring ridiculous situations like > running out of memory). Well maybe Chris should have said (or meant to say?) In math: ? ? ? whereas in programming int and float are disjoint types. So structurally the (int,float) type pair poorly approximates the (?, ?) pair of math sets Doesnt mean I agree with > we can't perfectly represent real numbers or calculate with them, so we do > the best we can Floats are a grotesque travesty of ? At the least, interval arithmetic can help automatically do the numerical analysis for you. Then there are all kinds of rational approximations like continued fractions which are better than ? All the way to "computable real numbers" We're stuck with them because that's the hardware we've got. Nothing intrinsic or necessary about it From random832 at fastmail.com Sun May 22 13:25:24 2016 From: random832 at fastmail.com (Random832) Date: Sun, 22 May 2016 13:25:24 -0400 Subject: for / while else doesn't make sense In-Reply-To: <87d1oe2l7z.fsf@elektro.pacujo.net> References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <87d1oe2l7z.fsf@elektro.pacujo.net> Message-ID: <1463937924.1827012.615232721.2982AD4D@webmail.messagingengine.com> On Sun, May 22, 2016, at 10:58, Marko Rauhamaa wrote: > Scheme has the best of both worlds: > > scheme@(guile-user)> 2/3 > $1 = 2/3 > scheme@(guile-user)> (exact->inexact $1) > $2 = 0.6666666666666666 Why shouldn't Python do this? Imagine some future version of Python: >>> x = 2/3 >>> x (2/3) >>> type(x) # if it's going to be so integrated into the language it's # hardly sensible to keep calling it 'fractions.Fraction' >>> float(x) 0.6666666666666666 On Sun, May 22, 2016, at 11:52, Jon Ribbens wrote: > No, it *adheres* to the principle of least surprise. Floats appearing > out of nowhere is surprising. Python 2's behaviour adhered to the > principle, and Python 3's breaks it. Disregarding for the moment the particular imperfections of the float representation (which would be gone if we used Fraction instead), this is only true if the concrete types of results are regarded as part of the result rather than as an implementation detail for how best to return the requested value. I think it would be entirely reasonable for Fractions to not only appear out of nowhere, but to *disappear* when an operation on them yields a value which is an integer. Values are more important than types. Types are less important than values. From random832 at fastmail.com Sun May 22 13:30:37 2016 From: random832 at fastmail.com (Random832) Date: Sun, 22 May 2016 13:30:37 -0400 Subject: for / while else doesn't make sense In-Reply-To: References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1463938237.1828035.615243249.4F306225@webmail.messagingengine.com> On Sun, May 22, 2016, at 12:46, Jon Ribbens wrote: > Sorry, I have to stop you there as the entire premise of your post is > clearly wrong. "int" is not "an approximation of real numbers", it's > a model of the mathematical concept "integers", It is a representation of Z, a subset of R (as is float, technically, though that particular subset has no nice name like Z and Q) The operators that apply to it are the operations on R, even operations under which Z (or even R) is not closed. From rustompmody at gmail.com Sun May 22 13:34:55 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Sun, 22 May 2016 10:34:55 -0700 (PDT) Subject: for / while else doesn't make sense In-Reply-To: References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <87d1oe2l7z.fsf@elektro.pacujo.net> <1463937924.1827012.615232721.2982AD4D@webmail.messagingengine.com> Message-ID: On Sunday, May 22, 2016 at 10:55:43 PM UTC+5:30, Random832 wrote: > Values are more important than types. Types are less important than > values. A stronger version that I occasionally tell my students: Values are in reality Types are in our heads Unfortunately we only know how to think thoughts inside our heads Which means we are stuck with the imperfections of our thinking apparatus From jon+usenet at unequivocal.co.uk Sun May 22 13:55:09 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Sun, 22 May 2016 17:55:09 -0000 (UTC) Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <1463938237.1828035.615243249.4F306225@webmail.messagingengine.com> Message-ID: On 2016-05-22, Random832 wrote: > On Sun, May 22, 2016, at 12:46, Jon Ribbens wrote: >> Sorry, I have to stop you there as the entire premise of your post is >> clearly wrong. "int" is not "an approximation of real numbers", it's >> a model of the mathematical concept "integers", > > It is a representation of Z, a subset of R Yes, that's what I just said. "Z" is just (an approximation of!) a symbol that means "the set of integers". > (as is float, technically, though that particular subset has no nice > name like Z and Q) The operators that apply to it are the operations > on R, even operations under which Z (or even R) is not closed. No, in Python integers are closed under the standard arithmetic operators (+ - * / % **) - except, since Python 3, for "/", which is now a special case. From jon+usenet at unequivocal.co.uk Sun May 22 14:06:17 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Sun, 22 May 2016 18:06:17 -0000 (UTC) Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <87d1oe2l7z.fsf@elektro.pacujo.net> <1463937924.1827012.615232721.2982AD4D@webmail.messagingengine.com> Message-ID: On 2016-05-22, Random832 wrote: > On Sun, May 22, 2016, at 11:52, Jon Ribbens wrote: >> No, it *adheres* to the principle of least surprise. Floats appearing >> out of nowhere is surprising. Python 2's behaviour adhered to the >> principle, and Python 3's breaks it. > > Disregarding for the moment the particular imperfections of the float > representation (which would be gone if we used Fraction instead), this > is only true if the concrete types of results are regarded as part of > the result rather than as an implementation detail for how best to > return the requested value. > > I think it would be entirely reasonable for Fractions to not only appear > out of nowhere, but to *disappear* when an operation on them yields a > value which is an integer. > > Values are more important than types. Types are less important than > values. This would be true if we had some Grand Unified Lossless Number Type. Unfortunately, we don't, and we're not likely to any time soon. From random832 at fastmail.com Sun May 22 14:14:47 2016 From: random832 at fastmail.com (Random832) Date: Sun, 22 May 2016 14:14:47 -0400 Subject: for / while else doesn't make sense In-Reply-To: References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <1463938237.1828035.615243249.4F306225@webmail.messagingengine.com> Message-ID: <1463940887.1835433.615265121.35FBEC6D@webmail.messagingengine.com> On Sun, May 22, 2016, at 13:55, Jon Ribbens wrote: > No, in Python integers are closed under the standard arithmetic > operators (+ - * / % **) Z is not closed under standard division, as surely as N isn't closed under subtraction and R isn't closed under exponentiation. That is a mathematical fact, not one about any particular language. What you are saying is that Python 2's "/" is _not_ standard division (you want to talk about the principle of least surprise...), and is therefore _not_ a standard arithmetic operation. It's not Euclidean division, either, since it gives a negative remainder for negative divisors. From random832 at fastmail.com Sun May 22 14:17:16 2016 From: random832 at fastmail.com (Random832) Date: Sun, 22 May 2016 14:17:16 -0400 Subject: for / while else doesn't make sense In-Reply-To: References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <87d1oe2l7z.fsf@elektro.pacujo.net> <1463937924.1827012.615232721.2982AD4D@webmail.messagingengine.com> Message-ID: <1463941036.1836057.615267705.392C51E6@webmail.messagingengine.com> On Sun, May 22, 2016, at 14:06, Jon Ribbens wrote: > This would be true if we had some Grand Unified Lossless Number Type. > Unfortunately, we don't, and we're not likely to any time soon. Scheme manages fine without one. It uses lossless types where it can, and lets you detect that an "inexact" number (which could be float, or could be any of the other types with an inexact flag set) was used where it can't. From random832 at fastmail.com Sun May 22 15:04:13 2016 From: random832 at fastmail.com (Random832) Date: Sun, 22 May 2016 15:04:13 -0400 Subject: OT: limit number of connections from browser to my server? In-Reply-To: References: Message-ID: <1463943853.1843750.615287929.64BA8A0B@webmail.messagingengine.com> On Wed, May 18, 2016, at 18:58, Gregory Ewing wrote: > Grant Edwards wrote: > > Product spec explicitly states HTTPS only. I'm told that is not open > > for discussion. The customer is a large, somewhat bureaucratic German > > corporation, and they generally mean it when they say something is > > non-negotiable. > > They're probably being sensible. The way the Internet of > Things is shaping up, it's far better to have too much > security than too little. HTTPS provides little to no security on a device which has no domain name, since we don't have any well-established way to manage self-signed certificates, or certificates signed on a basis other than the domain name. It'd be nice if there were a way for IOT devices to have a certificate signed *by the manufacturer*. The entire SSL browser UI paradigm is predicated on the fact that what is verified by a certificate is the domain name, which must match the CN field of the certificate, and provides no way to present a certificate issued on another basis to the user. From grant.b.edwards at gmail.com Sun May 22 15:29:46 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Sun, 22 May 2016 19:29:46 +0000 (UTC) Subject: OT: limit number of connections from browser to my server? References: <1463943853.1843750.615287929.64BA8A0B@webmail.messagingengine.com> Message-ID: On 2016-05-22, Random832 wrote: > On Wed, May 18, 2016, at 18:58, Gregory Ewing wrote: >> Grant Edwards wrote: >>> Product spec explicitly states HTTPS only. I'm told that is not open >>> for discussion. The customer is a large, somewhat bureaucratic German >>> corporation, and they generally mean it when they say something is >>> non-negotiable. >> >> They're probably being sensible. The way the Internet of Things is >> shaping up, it's far better to have too much security than too >> little. > > HTTPS provides little to no security on a device which has no domain > name, since we don't have any well-established way to manage > self-signed certificates, or certificates signed on a basis other > than the domain name. It'd be nice if there were a way for IOT > devices to have a certificate signed *by the manufacturer*. The customer can install their own certificate on the server and configure their browsers to require that certificate. They can also configure the server to require that the browser authenticate itself with a specific certificate (which they would have to install on the browser). So, in theory, HTTPS _could_ provide a decent level of security for products like these. Whether anybody actually goes to the trouble to do that, I don't know. I doubt they do, since it requires more than one mouse click, and reading more than 140 characters of text. And, it requires that you understand how SSL certificates work, how to generate them, and in some cases how to set up an internal domain name and DNS server for devices on an air-gapped LAN. -- Grant From torriem at gmail.com Sun May 22 15:37:55 2016 From: torriem at gmail.com (Michael Torrie) Date: Sun, 22 May 2016 13:37:55 -0600 Subject: Image loading problem In-Reply-To: <1463860505.689253.614738601.76B18B37@webmail.messagingengine.com> References: <743e5fbe-3a6c-49ea-ac5e-9bc055f652b5@googlegroups.com> <1463860505.689253.614738601.76B18B37@webmail.messagingengine.com> Message-ID: <9341584a-bbdb-c9c8-2cf2-d631671631d3@gmail.com> On 05/21/2016 01:55 PM, Random832 wrote: > On Sat, May 21, 2016, at 12:54, Peter Otten wrote: >> It's not your fault, there's an odd quirk in the library: you have to >> keep a reference of the PhotoImage instance around to prevent the >> image from being garbage-collected. > > Just out of curiosity, why is this a "quirk" and not a bug? Why isn't > the reference held by the Label? The reference is indeed held by the label but the problem is the label is a Tcl/Tk object, thinly wrapped in Python. It's essentially an impedance mismatch between two different languages and object models that each do their own reference holding and counting. I've run into this issue with PySide also as one is instantiating C++ objects that do their own internal reference counting through the Python wrapper. I'm sure Python wrappers could try to correct for this somewhat, but it's not a trivial thing to solve by any means. From lists at onemanifest.net Sun May 22 15:42:25 2016 From: lists at onemanifest.net (lists at onemanifest.net) Date: Sun, 22 May 2016 21:42:25 +0200 Subject: numpy problem Message-ID: <46F1751E-CA1E-4AB3-B6EB-589E1E2FD1D9@onemanifest.net> Hi, I've got a nympy problem I can't get my head around. (numpy is new to me). I've got a 2D array with values: values = np.array( [[ 20, 38, 4, 45, 65], [ 81, 44, 38, 57, 92], [ 92, 41, 16, 77, 44], [ 53, 62, 9, 75, 12], [ 58, 2, 60, 100, 29], [ 63, 15, 48, 43, 71], [ 80, 97, 87, 64, 60], [ 16, 16, 70, 88, 80], [ 19, 1, 73, 39, 97], [ 48, 3, 27, 81, 14]]) And an array of indexes that for shows which row to keep for each column of values: keep = np.array([2, 3, 1, 9, 2]) So, the result should be an array like array([ values[2,0], values[3,1], values[1,2], values[9,3], values[2,4] ]) == np.array([92, 62, 38, 81, 44]) Can this be accomplished in a vectorized manner? Thx, Peter -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ben.usenet at bsb.me.uk Sun May 22 15:51:19 2016 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Sun, 22 May 2016 20:51:19 +0100 Subject: for / while else doesn't make sense References: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <1463938237.1828035.615243249.4F306225@webmail.messagingengine.com> Message-ID: <877felub0o.fsf@bsb.me.uk> Jon Ribbens writes: > No, in Python integers are closed under the standard arithmetic > operators (+ - * / % **) - except, since Python 3, for "/", which > is now a special case. 2 ** -1 is 0.5 even in Python 2[*]. I agree with your general point (that floats should not pop up unbidden) but I don't think you need to exclude the possibly that an operator can do that. With perfect hindsight, I think I'd have had the integers closed under operators +, -, *, //, % and (say) ^, whilst making it clear that / and ** produce floats. There's no reason to see this as being any less explicit that writing 1.0 as a way to make your intent to use floats explicit. * Not a Python expert so all I means is that I get 0.5 on my machine and I'm assuming that's what Python 2 mandates as the result. -- Ben. From random832 at fastmail.com Sun May 22 16:19:45 2016 From: random832 at fastmail.com (Random832) Date: Sun, 22 May 2016 16:19:45 -0400 Subject: Image loading problem In-Reply-To: <9341584a-bbdb-c9c8-2cf2-d631671631d3@gmail.com> References: <743e5fbe-3a6c-49ea-ac5e-9bc055f652b5@googlegroups.com> <1463860505.689253.614738601.76B18B37@webmail.messagingengine.com> <9341584a-bbdb-c9c8-2cf2-d631671631d3@gmail.com> Message-ID: <1463948385.1856506.615331497.58809B55@webmail.messagingengine.com> On Sun, May 22, 2016, at 15:37, Michael Torrie wrote: > The reference is indeed held by the label but the problem is the label > is a Tcl/Tk object, thinly wrapped in Python. Okay but then in that case why doesn't the image get instantiated as a Tcl/Tk object which the label holds a reference to and then nobody cares if the python image object gets collected? (And anyway maybe the wrappers shouldn't be so thin if it causes problems like these.) From auriocus at gmx.de Sun May 22 16:57:46 2016 From: auriocus at gmx.de (Christian Gollwitzer) Date: Sun, 22 May 2016 22:57:46 +0200 Subject: Image loading problem In-Reply-To: References: <743e5fbe-3a6c-49ea-ac5e-9bc055f652b5@googlegroups.com> <1463860505.689253.614738601.76B18B37@webmail.messagingengine.com> <9341584a-bbdb-c9c8-2cf2-d631671631d3@gmail.com> <1463948385.1856506.615331497.58809B55@webmail.messagingengine.com> Message-ID: Am 22.05.16 um 22:19 schrieb Random832: > On Sun, May 22, 2016, at 15:37, Michael Torrie wrote: >> The reference is indeed held by the label but the problem is the label >> is a Tcl/Tk object, thinly wrapped in Python. > > Okay but then in that case why doesn't the image get instantiated as a > Tcl/Tk object which the label holds a reference to and then nobody cares > if the python image object gets collected? Actually, I think it could be solved, and not too complicated either. In Tk, objects are deleted explicitly. This is how you'd do it natively: image create photo bla -file myicon.png # create an image called "bla" pack [label .l -image bla] # display it in a label with name .l Of course, the label references the image object. For instance, if you read in new image data bla read newicon.png -shrink then the label will update. What happens, when you create all those things from Tkinter, the Label object deletes the image in it's destructor: image delete bla # this will turn the label blank This is IMHO correct, but when you add an image to the label through Tkinter, then it only does it through Tk (i.e. the second line in the above code). It should store a reference the Python image object inside the label object. This is akin to a "dangling pointer" in C. It would just be some work to replicate this for all widgets in Tk, there are buttons etc. but in essence I think it would work. Christian From steve at pearwood.info Sun May 22 18:09:06 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 23 May 2016 08:09:06 +1000 Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> On Mon, 23 May 2016 01:52 am, Jon Ribbens wrote: > On 2016-05-22, Steven D'Aprano wrote: >> How is this any better though? Complicated or not, people want to divide >> 1 by 2 and get 0.5. That is the functional requirement. Furthermore, they >> want to use the ordinary division symbol / rather than having to import >> some library or call a function. > > That's a circular argument. You're defining the result as the > requirement and then saying that proves the result is necessary. > Clearly, people managed when 1/2 returned 0, and continue to do so > today in Python 2 and other languages. I'm not defining the result. 4000+ years of mathematics defines the result. If you get up off your chair and wander around and ask people other than C programmers "What's one divide by two?", I am confident that virtually zero percent will answer "zero". People only managed when 1/2 returned 0 by *working around the problem*, and yes, it is a problem. They work around it by explicitly casting values to float (which, if carelessly done, just introduces new problems), or by using "from __future__ import division". >> Having 1/2 return 0 (as Python 2 does by default) doesn't make the >> language any less complicated. It doesn't avoid the complexity of >> floats, it merely breaks the principle of least surprise, > > No, it *adheres* to the principle of least surprise. Floats appearing > out of nowhere is surprising. Python 2's behaviour adhered to the > principle, and Python 3's breaks it. The float isn't appearing out of nowhere. It appears because you're performing a division. When you call `len("hello world")`, are you shocked that an int appears out of nowhere? Of course not. That's what len() does. Why should you be shocked that division returns a fractional quantity? That's what division does! Divide a cake into two pieces, and you have two half cakes, not no cake. >>> That's a trap for those people though - it lulls them into thinking >>> that they understand what's going on, when in fact they don't, >>> because they don't understand floats, because almost nobody >>> understands floats. So they don't understand their program, and >>> - even worse - they don't know that they don't understand it. >> >> And how does forcing them to write 1.0/2 solve that? > > Because it forces them to consciously address the fact that they are > asking for, and getting, floats, and that floats are not something > the language is willingly to silently foist upon them. It does no such thing. Have you met any programmers? It forces them to add an extraneous .0 to the end of their value, and give it no further thought until somebody reports a bug that Fraction calculations are silently coerced to floats, and that Decimal calculations raise an exception. And then they close the bug report "Will not fix" because it's too hard. >>> Programming languages should do what they are told, and very little >>> more. >> >> Okay, now I'm confused. How is 1/2 returning 0.5 the language not doing >> what you've told it to do? > > I didn't ask for floats, I got floats. That's how. You performed a division. What did you expect, a dict? >>> They should not wander off on surprising jaunts of their own >>> invention out of the control of the programmer. It should be possible >>> to know and understand the language, or at least the subset of it >>> that you are likely to need for your everyday purposes. Floats are >>> generally not understood, so they shouldn't be suddenly turning up >>> un-called for. >> >> How are they uncalled for? > > By... not being called for? I must admit I don't entirely understand > your question. You performed a division. By definition, this involves returning a fractional amount, or at least the possibility of returning a fractional amount. To say that it is "uncalled for" to receive a fractional amount is, frankly, bizarre. -- Steven From ian.g.kelly at gmail.com Sun May 22 19:04:11 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Sun, 22 May 2016 17:04:11 -0600 Subject: for / while else doesn't make sense In-Reply-To: References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <1463938237.1828035.615243249.4F306225@webmail.messagingengine.com> Message-ID: On Sun, May 22, 2016 at 11:55 AM, Jon Ribbens wrote: > On 2016-05-22, Random832 wrote: >> On Sun, May 22, 2016, at 12:46, Jon Ribbens wrote: >>> Sorry, I have to stop you there as the entire premise of your post is >>> clearly wrong. "int" is not "an approximation of real numbers", it's >>> a model of the mathematical concept "integers", >> >> It is a representation of Z, a subset of R > > Yes, that's what I just said. "Z" is just (an approximation of!) > a symbol that means "the set of integers". > >> (as is float, technically, though that particular subset has no nice >> name like Z and Q) The operators that apply to it are the operations >> on R, even operations under which Z (or even R) is not closed. > > No, in Python integers are closed under the standard arithmetic > operators (+ - * / % **) - except, since Python 3, for "/", which > is now a special case. If you want Python integers to be closed under division *and* be mathematically correct then the result of 1 / 2 should be the multiplicative inverse of 2, which is *undefined* in Z. While that might be an argument for raising an exception, it's not in any way a justification of returning 0. From jon+usenet at unequivocal.co.uk Sun May 22 20:34:54 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Mon, 23 May 2016 00:34:54 -0000 (UTC) Subject: for / while else doesn't make sense References: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <1463938237.1828035.615243249.4F306225@webmail.messagingengine.com> <877felub0o.fsf@bsb.me.uk> Message-ID: On 2016-05-22, Ben Bacarisse wrote: > Jon Ribbens writes: > >> No, in Python integers are closed under the standard arithmetic >> operators (+ - * / % **) - except, since Python 3, for "/", which >> is now a special case. > > 2 ** -1 is 0.5 even in Python 2[*]. Haha, excellent, well found. I was wondering if there were any edge cases I was wrong about. I suppose ideally I would make it so that 2 ** -1 throws an exception or something. But of course this particular train has left the station a long time ago. > I agree with your general point (that floats should not pop up unbidden) > but I don't think you need to exclude the possibly that an operator can > do that. With perfect hindsight, I think I'd have had the integers > closed under operators +, -, *, //, % and (say) ^, whilst making it > clear that / and ** produce floats. There's no reason to see this as > being any less explicit that writing 1.0 as a way to make your intent to > use floats explicit. My fundamental point is that floats are surprising, so people should not be surprised by them arriving unbidden - and most of the time, there is no need at all for them to turn up unannounced. Making that occurrence more likely rather than less was a mistake. From jon+usenet at unequivocal.co.uk Sun May 22 20:36:21 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Mon, 23 May 2016 00:36:21 -0000 (UTC) Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-22, Steven D'Aprano wrote: > On Mon, 23 May 2016 01:52 am, Jon Ribbens wrote: >> On 2016-05-22, Steven D'Aprano wrote: >>> How is this any better though? Complicated or not, people want to divide >>> 1 by 2 and get 0.5. That is the functional requirement. Furthermore, they >>> want to use the ordinary division symbol / rather than having to import >>> some library or call a function. >> >> That's a circular argument. You're defining the result as the >> requirement and then saying that proves the result is necessary. >> Clearly, people managed when 1/2 returned 0, and continue to do so >> today in Python 2 and other languages. > > I'm not defining the result. 4000+ years of mathematics defines the result. OK, I'm bored of you now. You clearly are not willing to imagine a world beyond your own preconceptions. I am not saying that my view is right, I'm just saying that yours is not automatically correct. If you won't even concede that much then this conversation is pointless. From jon+usenet at unequivocal.co.uk Sun May 22 21:00:14 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Mon, 23 May 2016 01:00:14 -0000 (UTC) Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-23, Chris Angelico wrote: > On Mon, May 23, 2016 at 10:36 AM, Jon Ribbens > wrote: >> On 2016-05-22, Steven D'Aprano wrote: >>> On Mon, 23 May 2016 01:52 am, Jon Ribbens wrote: >>>> On 2016-05-22, Steven D'Aprano wrote: >>>>> How is this any better though? Complicated or not, people want to divide >>>>> 1 by 2 and get 0.5. That is the functional requirement. Furthermore, they >>>>> want to use the ordinary division symbol / rather than having to import >>>>> some library or call a function. >>>> >>>> That's a circular argument. You're defining the result as the >>>> requirement and then saying that proves the result is necessary. >>>> Clearly, people managed when 1/2 returned 0, and continue to do so >>>> today in Python 2 and other languages. >>> >>> I'm not defining the result. 4000+ years of mathematics defines the result. >> >> OK, I'm bored of you now. You clearly are not willing to imagine >> a world beyond your own preconceptions. I am not saying that my view >> is right, I'm just saying that yours is not automatically correct. >> If you won't even concede that much then this conversation is pointless. > > The point of arithmetic in software is to do what mathematics defines. > Would you expect 1+2 to return 5? No. Why not? Where was the result > defined? Are you trying to compete with him for the Missing The Point Award? From rosuav at gmail.com Sun May 22 21:01:42 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 23 May 2016 11:01:42 +1000 Subject: for / while else doesn't make sense In-Reply-To: References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Mon, May 23, 2016 at 10:36 AM, Jon Ribbens wrote: > On 2016-05-22, Steven D'Aprano wrote: >> On Mon, 23 May 2016 01:52 am, Jon Ribbens wrote: >>> On 2016-05-22, Steven D'Aprano wrote: >>>> How is this any better though? Complicated or not, people want to divide >>>> 1 by 2 and get 0.5. That is the functional requirement. Furthermore, they >>>> want to use the ordinary division symbol / rather than having to import >>>> some library or call a function. >>> >>> That's a circular argument. You're defining the result as the >>> requirement and then saying that proves the result is necessary. >>> Clearly, people managed when 1/2 returned 0, and continue to do so >>> today in Python 2 and other languages. >> >> I'm not defining the result. 4000+ years of mathematics defines the result. > > OK, I'm bored of you now. You clearly are not willing to imagine > a world beyond your own preconceptions. I am not saying that my view > is right, I'm just saying that yours is not automatically correct. > If you won't even concede that much then this conversation is pointless. The point of arithmetic in software is to do what mathematics defines. Would you expect 1+2 to return 5? No. Why not? Where was the result defined? ChrisA From rocky at gnu.org Sun May 22 21:07:18 2016 From: rocky at gnu.org (rocky) Date: Sun, 22 May 2016 18:07:18 -0700 (PDT) Subject: RFC: name for project of a cross version disassembler, and unmarshal program Message-ID: <2e3189e0-c1e3-49fa-b6ce-53471c4374e7@googlegroups.com> I'm looking for a good name for a relatively new project I'll put on pypy. I've been working on a module to disassemble Python bytecode from many versions of Python. (Right now 2.3 .. 3.5 bytecode, largely works.) Of course, in order to do that you also need routines to unmarshal bytecode. So that's in there as well. In the future, I may could add a marshaler and an assembler to Python bytecode. I know, this is kind of perverse. At any rate the name I've been using is "pyxdis". See https://github.com/rocky/python-pyxdis. In the past I've been told by Polish-speaking people that my names are hard to pronounce. (If you've ever heard any Polish tongue twisters, you'll know that this really hurts.) Any suggestions for a better name? Thanks. From python at mrabarnett.plus.com Sun May 22 21:51:28 2016 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 23 May 2016 02:51:28 +0100 Subject: for / while else doesn't make sense In-Reply-To: References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: <403cf3d2-4fa3-bd90-ad7f-6e938d056cf3@mrabarnett.plus.com> On 2016-05-23 02:00, Jon Ribbens wrote: > On 2016-05-23, Chris Angelico wrote: >> On Mon, May 23, 2016 at 10:36 AM, Jon Ribbens >> wrote: >>> On 2016-05-22, Steven D'Aprano wrote: >>>> On Mon, 23 May 2016 01:52 am, Jon Ribbens wrote: >>>>> On 2016-05-22, Steven D'Aprano wrote: >>>>>> How is this any better though? Complicated or not, people want to divide >>>>>> 1 by 2 and get 0.5. That is the functional requirement. Furthermore, they >>>>>> want to use the ordinary division symbol / rather than having to import >>>>>> some library or call a function. >>>>> >>>>> That's a circular argument. You're defining the result as the >>>>> requirement and then saying that proves the result is necessary. >>>>> Clearly, people managed when 1/2 returned 0, and continue to do so >>>>> today in Python 2 and other languages. >>>> >>>> I'm not defining the result. 4000+ years of mathematics defines the result. >>> >>> OK, I'm bored of you now. You clearly are not willing to imagine >>> a world beyond your own preconceptions. I am not saying that my view >>> is right, I'm just saying that yours is not automatically correct. >>> If you won't even concede that much then this conversation is pointless. >> >> The point of arithmetic in software is to do what mathematics defines. >> Would you expect 1+2 to return 5? No. Why not? Where was the result >> defined? > > Are you trying to compete with him for the Missing The Point Award? > The relevant doc is PEP 238, dating to March 2001, when Python 2.2 was new. From mr.siyi.deng at gmail.com Sun May 22 23:15:30 2016 From: mr.siyi.deng at gmail.com (Siyi Deng) Date: Sun, 22 May 2016 20:15:30 -0700 (PDT) Subject: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 Message-ID: I have a dynamic library doing some numerical computations. I used ctypes to interact it by passing numpy arrays back and forth. Python 3.5 gives me the correct results. Python 2.7 gives me different, erroneous results, but it never crashes. How is this possible? There is no string operations involved whatsoever. From ben+python at benfinney.id.au Mon May 23 00:13:45 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 23 May 2016 14:13:45 +1000 Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: <85r3ct9zt2.fsf@benfinney.id.au> Jon Ribbens writes: > OK, I'm bored of you now. You clearly are not willing to imagine > a world beyond your own preconceptions. Steven has, in the message to which you responded, asked for you to *describe* this other world you assert exists. More concretely: Steven is not denying someone might have different expectations. On the contrary, you've said your expectations differ, and Steven is *explicitly asking* you to specify those expectations. And, instead of answering, you give this dismissal. Are your expectations so hard to describe? -- \ ?It is wrong to think that the task of physics is to find out | `\ how nature *is*. Physics concerns what we can *say* about | _o__) nature?? ?Niels Bohr | Ben Finney From rustompmody at gmail.com Mon May 23 02:09:32 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Sun, 22 May 2016 23:09:32 -0700 (PDT) Subject: for / while else doesn't make sense In-Reply-To: References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> Message-ID: On Monday, May 23, 2016 at 9:59:27 AM UTC+5:30, Ben Finney wrote: > Jon Ribbens writes: > > > OK, I'm bored of you now. You clearly are not willing to imagine > > a world beyond your own preconceptions. > > Steven has, in the message to which you responded, asked for you to > *describe* this other world you assert exists. > > More concretely: Steven is not denying someone might have different > expectations. On the contrary, you've said your expectations differ, and > Steven is *explicitly asking* you to specify those expectations. > > And, instead of answering, you give this dismissal. Are your > expectations so hard to describe? Steven is making wild and disingenuous statements; to wit: On Monday, May 23, 2016 at 3:39:19 AM UTC+5:30, Steven D'Aprano wrote: > I'm not defining the result. 4000+ years of mathematics defines the result. This is off by on order of magnitude. Decimal point started with Napier (improving on Stevin): 17th century OTOH it is plain numbers (?) that have been in use for some 4 millennia. > > If you get up off your chair and wander around and ask people other than C > programmers "What's one divide by two?", I am confident that virtually zero > percent will answer "zero". You forget that we (most of us?) went to school. My recollections of it -- ok maybe fogged by near 5 decades: I first learnt something called 'long-division' In that procedure you take 2 numbers called divisor and dividend And GET TWO NUMBERS a quotient and a remainder. [At that point only knew of the numbers we would later call ? (or was it ? -- not sure -- decimal point would come later] Later (again dont remember order) we were taught - short division - decimal numbers [I mention short division -- put numerator on top of denominator and cancel off factors -- because the symmetry of numerator:denominator and quotient:remainder is more apparent there than in long-division] In any case if learning primacy has any significance, pure integer division is more basic than decimal number division. To recapitulate the situation: Mathematics (mathematicians if you prefer) have a strong attachment to to two nice properties of operators: The first is obvious and unarguable -- totality The second does not have a standard term but is important enough -- I will call it 'homogeneity'. By this I mean a type of the form: t ? t ? t Its nice to have totality because one can avoid case-analysing: f(x) when x ? domain(f) Its nice to have homogeneity because homogeneous operators can be nested/unnested/played-with ie for ? : t ? t ? t x ? y ? z makes sense this way x ? (y ? z) or this way (x ? y) ? z With non-homogeneous ? these may not make sense. - Choosing ? to be total and homogeneous necessitates widening ? (or ?) to ? or ? (or something as messy) - Choosing ? to be non-homogeneous means needing to deal with quotients and remainders Cant write if (x/4 < 256)... have to write quot, rem = x/4 # throw away rem if quot < 256: ... Haskell has (almost) what I learnt at school: Prelude> let (q,r) = 7 `divMod` 3 Prelude> (q,r) (2,1) Replace the strange `divMod` with / and we are back to the behavior I first learnt at school So with some over-simplification: - the first choice leads to numerical analysis - the second leads to number theory To say that one is natural --especially the one that chooses something other than natural numbers! --and the other is surprising is nonsense. From petef4+usenet at gmail.com Mon May 23 02:16:54 2016 From: petef4+usenet at gmail.com (Pete Forman) Date: Mon, 23 May 2016 07:16:54 +0100 Subject: RFC: name for project of a cross version disassembler, and unmarshal program References: <2e3189e0-c1e3-49fa-b6ce-53471c4374e7@googlegroups.com> Message-ID: rocky writes: > I'm looking for a good name for a relatively new project I'll put on pypy. > > I've been working on a module to disassemble Python bytecode from many > versions of Python. (Right now 2.3 .. 3.5 bytecode, largely works.) > > Of course, in order to do that you also need routines to unmarshal > bytecode. So that's in there as well. > > In the future, I may could add a marshaler and an assembler to Python > bytecode. I know, this is kind of perverse. > > At any rate the name I've been using is "pyxdis". See > https://github.com/rocky/python-pyxdis. > > In the past I've been told by Polish-speaking people that my names are > hard to pronounce. (If you've ever heard any Polish tongue twisters, > you'll know that this really hurts.) > > Any suggestions for a better name? relipmoc -- Pete Forman From marko at pacujo.net Mon May 23 02:30:57 2016 From: marko at pacujo.net (Marko Rauhamaa) Date: Mon, 23 May 2016 09:30:57 +0300 Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> Message-ID: <87oa7x1e1q.fsf@elektro.pacujo.net> Rustom Mody : > Haskell has (almost) what I learnt at school: > > Prelude> let (q,r) = 7 `divMod` 3 > Prelude> (q,r) > (2,1) Python: >>> divmod(7, 3) (2, 1) > Replace the strange `divMod` with / and we are back to the behavior I > first learnt at school X We never used '/' in school for anything. We used 'X : Y' or '---'. Y Anyway, every calculator in the world produces: 1 ? 2 = ==> 0.5 Marko From rustompmody at gmail.com Mon May 23 02:46:24 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Sun, 22 May 2016 23:46:24 -0700 (PDT) Subject: for / while else doesn't make sense In-Reply-To: <87oa7x1e1q.fsf@elektro.pacujo.net> References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <87oa7x1e1q.fsf@elektro.pacujo.net> Message-ID: <9939905d-d482-4256-bcb2-14ad6b343129@googlegroups.com> On Monday, May 23, 2016 at 12:01:08 PM UTC+5:30, Marko Rauhamaa wrote: > Rustom Mody : > > > Haskell has (almost) what I learnt at school: > > > > Prelude> let (q,r) = 7 `divMod` 3 > > Prelude> (q,r) > > (2,1) > > Python: > > >>> divmod(7, 3) > (2, 1) > > > Replace the strange `divMod` with / and we are back to the behavior I > > first learnt at school > > X > We never used '/' in school for anything. We used 'X : Y' or '---'. > Y > > Anyway, every calculator in the world produces: > > 1 > ? > 2 > = > > ==> 0.5 Not true: https://www.youtube.com/watch?v=iynCW9O_x58 [And ive seen such 40 years ago] From steve+comp.lang.python at pearwood.info Mon May 23 03:09:56 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 23 May 2016 17:09:56 +1000 Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <87d1oe2l7z.fsf@elektro.pacujo.net> <1463937924.1827012.615232721.2982AD4D@webmail.messagingengine.com> Message-ID: <5742acc5$0$1590$c3e8da3$5496439d@news.astraweb.com> On Monday 23 May 2016 03:25, Random832 wrote: > Why shouldn't Python do this? > > Imagine some future version of Python: >>>> x = 2/3 >>>> x > (2/3) >>>> type(x) > You would have a lot of trouble convincing Guido that this was a good idea, because that's what ABC used to do, and it was a performance killer, as well as being horrible to work with. Rationals like 2/3 are fine. But the trouble is, by the time you've done a handful of calculations, you've probably got something like 5529748264768821/18014398509481984 and after a few dozen calculations you might have something like: 391303115027894573050315966944902650883330501592661514878103414119479331609846605406995557395414953/14109433351889544757158673468211253755455976544516272594484740551000078367597300074881469573563940864 and it just keeps getting worse and worse. For bonus points, without converting to floats, can you tell which of of the two numbers is bigger by sight? -- Steve From dieter at handshake.de Mon May 23 03:23:56 2016 From: dieter at handshake.de (dieter) Date: Mon, 23 May 2016 09:23:56 +0200 Subject: Python and GPSD References: Message-ID: <87oa7xdypf.fsf@handshake.de> John McKenzie writes: > I need help using the Python bindings for GPSD. Specifically, I would > like to take a latitude reading, put it in a variable and use it later. > The problem is that every example I see involves constantly taking > changing readings. That part I have working for myself by following > existing examples. I do not know "GPSD" - therefore, I have only very general remarks for you. A Python binding, any kind of, is just a binding: it allows you to interact with a third party entity (usually a library or a service) via Python. To use a binding successfully, you will need to understand this third party entity: how it expects to get initialized, what order of calls are supported, what input parameters are expected and what output the calls produce. Once, you have grapsed the essentials of the third party entity, you can have a close look at the Python binding to learn how to do all this from a Python script. I am speaking as the author of "dm.xmlsec.binding", a binding to the "XML Security Library". Surprises similar to those reported by you can occur when someone uses "dm.xmlsec.binding" without a thorough understanding of the "XML Security Library". From rocky at gnu.org Mon May 23 04:08:24 2016 From: rocky at gnu.org (rocky) Date: Mon, 23 May 2016 01:08:24 -0700 (PDT) Subject: RFC: name for project of a cross version disassembler, and unmarshal program In-Reply-To: References: <2e3189e0-c1e3-49fa-b6ce-53471c4374e7@googlegroups.com> Message-ID: On Monday, May 23, 2016 at 2:17:07 AM UTC-4, Pete Forman wrote: > rocky writes: > > > I'm looking for a good name for a relatively new project I'll put on pypy. > > > > I've been working on a module to disassemble Python bytecode from many > > versions of Python. (Right now 2.3 .. 3.5 bytecode, largely works.) > > > > Of course, in order to do that you also need routines to unmarshal > > bytecode. So that's in there as well. > > > > In the future, I may could add a marshaler and an assembler to Python > > bytecode. I know, this is kind of perverse. > > > > At any rate the name I've been using is "pyxdis". See > > https://github.com/rocky/python-pyxdis. > > > > In the past I've been told by Polish-speaking people that my names are > > hard to pronounce. (If you've ever heard any Polish tongue twisters, > > you'll know that this really hurts.) > > > > Any suggestions for a better name? > > relipmoc > > -- > Pete Forman Interesting. (For those who are slow like myself, it is "compile" backwards. From steve+comp.lang.python at pearwood.info Mon May 23 04:09:45 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 23 May 2016 18:09:45 +1000 Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> Message-ID: <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> On Monday 23 May 2016 16:09, Rustom Mody wrote: > Steven is making wild and disingenuous statements; to wit: > > On Monday, May 23, 2016 at 3:39:19 AM UTC+5:30, Steven D'Aprano wrote: >> I'm not defining the result. 4000+ years of mathematics defines the result. > > This is off by on order of magnitude. > Decimal point started with Napier (improving on Stevin): 17th century > OTOH it is plain numbers (?) that have been in use for some 4 millennia. Are you saying that the Egyptians, Babylonians and Greeks didn't know how to work with fractions? http://mathworld.wolfram.com/EgyptianFraction.html http://nrich.maths.org/2515 Okay, it's not quite 4000 years ago. Sometimes my historical sense of the distant past is a tad inaccurate. Shall we say 2000 years instead? >> If you get up off your chair and wander around and ask people other than C >> programmers "What's one divide by two?", I am confident that virtually zero >> percent will answer "zero". > > You forget that we (most of us?) went to school. Er, why would I forget that? That's the point -- people have learned about fractions. I didn't say "go off deep into the Amazonian rainforests, or into the New Guinea highlands, and ask innumerate hunter gatherers...". But even innumerate hunter gatherers will have an understanding that if you have one yam which you wish to share between two people, they will each get half. Not zero. -- Steve From steve+comp.lang.python at pearwood.info Mon May 23 04:33:57 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 23 May 2016 18:33:57 +1000 Subject: for / while else doesn't make sense References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: <5742c076$0$2880$c3e8da3$76491128@news.astraweb.com> On Monday 23 May 2016 10:36, Jon Ribbens wrote: > OK, I'm bored of you now. You clearly are not willing to imagine > a world beyond your own preconceptions. I am not saying that my view > is right, I'm just saying that yours is not automatically correct. > If you won't even concede that much then this conversation is pointless. I borrowed Guido's Time Machine, and wrote the following I suppose some language some day might experiment with swapping the operators, so that a/b is integer division and a//b is true division. *nine hours* before your response above. But perhaps that was explicit enough, so if it will satisfy you, I will state for the record that defining the / operator to do integer division is not necessarily bad. But it has been roundly and broadly rejected by the Python community. It was tried and rejected. One way or another, your opinion is a minority view. -- Steve From raxmic at gmail.com Mon May 23 04:39:35 2016 From: raxmic at gmail.com (ragav s) Date: Mon, 23 May 2016 01:39:35 -0700 (PDT) Subject: Setting Return-Path in email Message-ID: <69c455ff-0fbd-42a5-a3fb-4ad0b89ed48e@googlegroups.com> Hi all, How can i add different Return-path and fromid in python.i have pasted the below code for preview def sendMail(sub,fromid,to,cc,html): msg = MIMEMultipart('alternative') msg['Subject'] = sub msg['From'] = fromid msg['To'] = to toaddress = [to] if cc: msg['Cc'] = cc toaddress = to+","+ cc toaddress = toaddress.split(",") type = 'plain' part = MIMEText(html, type,'utf-8') msg.attach(part) s = smtplib.SMTP('localhost') s.sendmail(fromid,toaddress, msg.as_string()) s.quit() From vincent.vande.vyvre at telenet.be Mon May 23 05:11:00 2016 From: vincent.vande.vyvre at telenet.be (Vincent Vande Vyvre) Date: Mon, 23 May 2016 11:11:00 +0200 Subject: Setting Return-Path in email In-Reply-To: <69c455ff-0fbd-42a5-a3fb-4ad0b89ed48e@googlegroups.com> References: <69c455ff-0fbd-42a5-a3fb-4ad0b89ed48e@googlegroups.com> Message-ID: <5742C924.1000504@telenet.be> Le 23/05/2016 10:39, ragav s a ?crit : > Hi all, > > How can i add different Return-path and fromid in python.i have pasted the below code for preview > > > def sendMail(sub,fromid,to,cc,html): > msg = MIMEMultipart('alternative') > msg['Subject'] = sub > msg['From'] = fromid > msg['To'] = to > toaddress = [to] > if cc: > msg['Cc'] = cc > toaddress = to+","+ cc > toaddress = toaddress.split(",") > > type = 'plain' > part = MIMEText(html, type,'utf-8') > msg.attach(part) > > s = smtplib.SMTP('localhost') > s.sendmail(fromid,toaddress, msg.as_string()) > s.quit() Try with msg['Reply-To'] = the adress Vincent From fabien.maussion at gmail.com Mon May 23 05:27:59 2016 From: fabien.maussion at gmail.com (Fabien) Date: Mon, 23 May 2016 11:27:59 +0200 Subject: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 References: Message-ID: On 05/23/2016 05:15 AM, Siyi Deng wrote: > I have a dynamic library doing some numerical computations. > I used ctypes to interact it by passing numpy arrays back and forth. > Python 3.5 gives me the correct results. > Python 2.7 gives me different, erroneous results, but it never crashes. > How is this possible? There is no string operations involved whatsoever. You might be more successful if you describe your problem in more detail (maybe with a minimal working example?) and if you ask the scipy/numpy people: https://www.scipy.org/scipylib/mailing-lists.html Cheers From i.mehrzad at gmail.com Mon May 23 06:27:50 2016 From: i.mehrzad at gmail.com (Mehrzad Irani) Date: Mon, 23 May 2016 03:27:50 -0700 (PDT) Subject: variable argument unpacking Message-ID: <436682ac-ec56-4286-a36e-d5697a9fd83c@googlegroups.com> Hi All, Consider the situation [cti at iranim-rhel python_cti]$ cat a.py def a(a = 1, b = 2, c = 3, *d, **e): print(a, b, c) print(d) print(e) r = {'e': 7, 'f': 8, 'g': 9} a(**r) a(3, **r) r1 = (4,5,6) a(3,2,1,*r1, **r) a(*r1, **r) r1 = (4,5,6,7) a(*r1, **r) [cti at iranim-rhel python_cti]$ The output for this program is as follows: [cti at iranim-rhel python_cti]$ python a.py (1, 2, 3) () {'e': 7, 'g': 9, 'f': 8} ------------------ (3, 2, 3) () {'e': 7, 'g': 9, 'f': 8} ------------------ (3, 2, 1) (4, 5, 6) {'e': 7, 'g': 9, 'f': 8} ------------------ (4, 5, 6) () {'e': 7, 'g': 9, 'f': 8} ------------------ (4, 5, 6) (7,) {'e': 7, 'g': 9, 'f': 8} This program shows, that for d to get assigned, I would need to first assign a, b, c even though their default parameters have been set. Also, if I would like to unpack a, b, c using e; I would get a multiple assignment TypeError. Therefore, my question is - is there a way to assign d, without going through the assignments of a, b, c again, since they have already been assigned defaults? (I think I am missing something simple here) Thanks in advance. From steve+comp.lang.python at pearwood.info Mon May 23 06:29:16 2016 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 23 May 2016 20:29:16 +1000 Subject: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 References: Message-ID: <5742db7e$0$1597$c3e8da3$5496439d@news.astraweb.com> On Monday 23 May 2016 13:15, Siyi Deng wrote: > I have a dynamic library doing some numerical computations. > > I used ctypes to interact it by passing numpy arrays back and forth. > > Python 3.5 gives me the correct results. > > Python 2.7 gives me different, erroneous results, but it never crashes. > > How is this possible? There is no string operations involved whatsoever. This is only a guess, because you haven't shown your code or data, but I guess it could be related to the change from integer division to true division in Python 3. In Python 2.7, try this at the interactive interpreter. Do you get the same results? >>> import numpy >>> a =numpy.array([1, 2, 3]) >>> a/3 array([0, 0, 1]) That's because in Python 2, the / operator performs integer division, like C. In Python 3, it performs true division. Put "from __future__ import division" at the top of your script. In Python 2.7, it will give you the same behaviour as Python 3, and in Python 3, it will be harmless. Note that there are TWO underscores at the front and end of __future__: >>> from __future__ import division >>> a/3 array([ 0.33333333, 0.66666667, 1. ]) Please note that __future__ imports must be the FIRST line of code. They can follow comments, but must be before any other code. -- Steve From __peter__ at web.de Mon May 23 06:53:37 2016 From: __peter__ at web.de (Peter Otten) Date: Mon, 23 May 2016 12:53:37 +0200 Subject: variable argument unpacking References: <436682ac-ec56-4286-a36e-d5697a9fd83c@googlegroups.com> Message-ID: Mehrzad Irani wrote: > Hi All, > > Consider the situation > [cti at iranim-rhel python_cti]$ cat a.py > def a(a = 1, b = 2, c = 3, *d, **e): > print(a, b, c) > print(d) > print(e) > > r = {'e': 7, 'f': 8, 'g': 9} > > > a(**r) > a(3, **r) > > r1 = (4,5,6) > > a(3,2,1,*r1, **r) > a(*r1, **r) > > r1 = (4,5,6,7) > a(*r1, **r) > > [cti at iranim-rhel python_cti]$ > > The output for this program is as follows: > [cti at iranim-rhel python_cti]$ python a.py > (1, 2, 3) > () > {'e': 7, 'g': 9, 'f': 8} > ------------------ > (3, 2, 3) > () > {'e': 7, 'g': 9, 'f': 8} > ------------------ > (3, 2, 1) > (4, 5, 6) > {'e': 7, 'g': 9, 'f': 8} > ------------------ > (4, 5, 6) > () > {'e': 7, 'g': 9, 'f': 8} > ------------------ > (4, 5, 6) > (7,) > {'e': 7, 'g': 9, 'f': 8} > > This program shows, that for d to get assigned, I would need to first > assign a, b, c even though their default parameters have been set. > > Also, if I would like to unpack a, b, c using e; I would get a multiple > assignment TypeError. > > Therefore, my question is - is there a way to assign d, without going > through the assignments of a, b, c again, since they have already been > assigned defaults? (I think I am missing something simple here) > > Thanks in advance. Python 3 allows $ cat b.py def a(*d, a=1, b=2, c=3, **e): print(a, b, c) print(d) print(e) r = {'e': 7, 'f': 8, 'g': 9} a(**r) a(3, **r) r1 = (4,5,6) a(3,2,1,*r1, **r) a(*r1, **r) r1 = (4,5,6,7) a(*r1, **r) $ python3 b.py 1 2 3 () {'e': 7, 'f': 8, 'g': 9} 1 2 3 (3,) {'e': 7, 'f': 8, 'g': 9} 1 2 3 (3, 2, 1, 4, 5, 6) {'e': 7, 'f': 8, 'g': 9} 1 2 3 (4, 5, 6) {'e': 7, 'f': 8, 'g': 9} 1 2 3 (4, 5, 6, 7) {'e': 7, 'f': 8, 'g': 9} Perhaps that is more to your liking? I find the output as unreadable as of your a.py, so I won't bother to check... From __peter__ at web.de Mon May 23 08:19:28 2016 From: __peter__ at web.de (Peter Otten) Date: Mon, 23 May 2016 14:19:28 +0200 Subject: numpy problem References: <46F1751E-CA1E-4AB3-B6EB-589E1E2FD1D9@onemanifest.net> Message-ID: lists at onemanifest.net wrote: > I've got a 2D array with values: > > values = np.array( > [[ 20, 38, 4, 45, 65], > [ 81, 44, 38, 57, 92], > [ 92, 41, 16, 77, 44], > [ 53, 62, 9, 75, 12], > [ 58, 2, 60, 100, 29], > [ 63, 15, 48, 43, 71], > [ 80, 97, 87, 64, 60], > [ 16, 16, 70, 88, 80], > [ 19, 1, 73, 39, 97], > [ 48, 3, 27, 81, 14]]) > > And an array of indexes that for shows which row to keep for each column > of values: > > keep = np.array([2, 3, 1, 9, 2]) > > So, the result should be an array like array([ values[2,0], values[3,1], > values[1,2], values[9,3], values[2,4] ]) == np.array([92, 62, 38, 81, 44]) > > Can this be accomplished in a vectorized manner? How about values[keep].diagonal() From rustompmody at gmail.com Mon May 23 08:40:47 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Mon, 23 May 2016 05:40:47 -0700 (PDT) Subject: RFC: name for project of a cross version disassembler, and unmarshal program In-Reply-To: References: <2e3189e0-c1e3-49fa-b6ce-53471c4374e7@googlegroups.com> Message-ID: <3208147d-22bc-4873-b82c-85231effdec3@googlegroups.com> On Monday, May 23, 2016 at 1:38:41 PM UTC+5:30, rocky wrote: > On Monday, May 23, 2016 at 2:17:07 AM UTC-4, Pete Forman wrote: > > rocky writes: > > > > > I'm looking for a good name for a relatively new project I'll put on pypy. > > > > > > I've been working on a module to disassemble Python bytecode from many > > > versions of Python. (Right now 2.3 .. 3.5 bytecode, largely works.) > > > > > > Of course, in order to do that you also need routines to unmarshal > > > bytecode. So that's in there as well. > > > > > > In the future, I may could add a marshaler and an assembler to Python > > > bytecode. I know, this is kind of perverse. > > > > > > At any rate the name I've been using is "pyxdis". See > > > https://github.com/rocky/python-pyxdis. > > > > > > In the past I've been told by Polish-speaking people that my names are > > > hard to pronounce. (If you've ever heard any Polish tongue twisters, > > > you'll know that this really hurts.) > > > > > > Any suggestions for a better name? > > > > relipmoc > > > > -- > > Pete Forman > > Interesting. (For those who are slow like myself, it is "compile" backwards. Heh! I also wondered... And if you are into that kinda stuff, how about: ?o?d?l? ? (?) From lists at onemanifest.net Mon May 23 09:10:47 2016 From: lists at onemanifest.net (lists at onemanifest.net) Date: Mon, 23 May 2016 15:10:47 +0200 Subject: numpy problem In-Reply-To: References: <46F1751E-CA1E-4AB3-B6EB-589E1E2FD1D9@onemanifest.net> Message-ID: <17D9C96A-C4EB-426F-A4A7-AA6492D7DBAB@onemanifest.net> > > On 23 mei 2016, at 14:19, Peter Otten <__peter__ at web.de> wrote: > > lists at onemanifest.net wrote: > >> I've got a 2D array with values: >> >> values = np.array( >> [[ 20, 38, 4, 45, 65], >> [ 81, 44, 38, 57, 92], >> [ 92, 41, 16, 77, 44], >> [ 53, 62, 9, 75, 12], >> [ 58, 2, 60, 100, 29], >> [ 63, 15, 48, 43, 71], >> [ 80, 97, 87, 64, 60], >> [ 16, 16, 70, 88, 80], >> [ 19, 1, 73, 39, 97], >> [ 48, 3, 27, 81, 14]]) >> >> And an array of indexes that for shows which row to keep for each column >> of values: >> >> keep = np.array([2, 3, 1, 9, 2]) >> >> So, the result should be an array like array([ values[2,0], values[3,1], >> values[1,2], values[9,3], values[2,4] ]) == np.array([92, 62, 38, 81, 44]) >> >> Can this be accomplished in a vectorized manner? > > How about > > values[keep].diagonal() That seems to do the trick! Thx. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ian.g.kelly at gmail.com Mon May 23 10:14:18 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 23 May 2016 08:14:18 -0600 Subject: for / while else doesn't make sense In-Reply-To: <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> References: <573EC62F.4090401@lucidity.plus.com> <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Mon, May 23, 2016 at 2:09 AM, Steven D'Aprano wrote: > Are you saying that the Egyptians, Babylonians and Greeks didn't know how to > work with fractions? > > http://mathworld.wolfram.com/EgyptianFraction.html > > http://nrich.maths.org/2515 > > Okay, it's not quite 4000 years ago. Sometimes my historical sense of the > distant past is a tad inaccurate. Shall we say 2000 years instead? Those links give dates of 1650 BC and 1800 BC respectively, so I'd say your initial guess was closer. From ben.usenet at bsb.me.uk Mon May 23 10:29:30 2016 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Mon, 23 May 2016 15:29:30 +0100 Subject: for / while else doesn't make sense References: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87zirgrgol.fsf@bsb.me.uk> Ian Kelly writes: > On Mon, May 23, 2016 at 2:09 AM, Steven D'Aprano > wrote: >> Are you saying that the Egyptians, Babylonians and Greeks didn't know how to >> work with fractions? >> >> http://mathworld.wolfram.com/EgyptianFraction.html >> >> http://nrich.maths.org/2515 >> >> Okay, it's not quite 4000 years ago. Sometimes my historical sense of the >> distant past is a tad inaccurate. Shall we say 2000 years instead? > > Those links give dates of 1650 BC and 1800 BC respectively, so I'd say > your initial guess was closer. Right, but this is to miss the point. Let's say that 4000 years have defined 1/3 to be one third, but Python 3 (as do many programming languages) defines 1/3 to be something very very very very close to one third, and *that* idea is very very very very new! It's unfortunate that the example in this thread does not illustrate the main problem of shifting to binary floating point, because 1/2 happens to be exactly representable. -- Ben. From ian.g.kelly at gmail.com Mon May 23 10:49:12 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 23 May 2016 08:49:12 -0600 Subject: for / while else doesn't make sense In-Reply-To: <87zirgrgol.fsf@bsb.me.uk> References: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> Message-ID: On Mon, May 23, 2016 at 8:29 AM, Ben Bacarisse wrote: > Ian Kelly writes: > >> On Mon, May 23, 2016 at 2:09 AM, Steven D'Aprano >> wrote: >>> Are you saying that the Egyptians, Babylonians and Greeks didn't know how to >>> work with fractions? >>> >>> http://mathworld.wolfram.com/EgyptianFraction.html >>> >>> http://nrich.maths.org/2515 >>> >>> Okay, it's not quite 4000 years ago. Sometimes my historical sense of the >>> distant past is a tad inaccurate. Shall we say 2000 years instead? >> >> Those links give dates of 1650 BC and 1800 BC respectively, so I'd say >> your initial guess was closer. > > Right, but this is to miss the point. Let's say that 4000 years have > defined 1/3 to be one third, but Python 3 (as do many programming > languages) defines 1/3 to be something very very very very close to one > third, and *that* idea is very very very very new! It's unfortunate > that the example in this thread does not illustrate the main problem of > shifting to binary floating point, because 1/2 happens to be exactly > representable. I'm not going to dig back through the thread, but my recollection is that's exactly why that example was chosen. Since 1/2 can be represented exactly as a float, it *should* be represented as a float. Picking another value (0) that isn't even close to the exact value of 1/2 isn't helping anybody. From rosuav at gmail.com Mon May 23 10:57:07 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 May 2016 00:57:07 +1000 Subject: for / while else doesn't make sense In-Reply-To: <87zirgrgol.fsf@bsb.me.uk> References: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> Message-ID: On Tue, May 24, 2016 at 12:29 AM, Ben Bacarisse wrote: > Right, but this is to miss the point. Let's say that 4000 years have > defined 1/3 to be one third, but Python 3 (as do many programming > languages) defines 1/3 to be something very very very very close to one > third, and *that* idea is very very very very new! Have you ever written one third as 0.33333333 ? Because that's also something very very close to one third. ChrisA From billy.t.west at gmail.com Mon May 23 11:04:02 2016 From: billy.t.west at gmail.com (billy.t.west at gmail.com) Date: Mon, 23 May 2016 08:04:02 -0700 (PDT) Subject: Python 3.5 Cross Compile for ARM Message-ID: <2b02f870-d29f-4519-a5a6-00298aa900ca@googlegroups.com> Hi. Can anyone point me to documentation/instructions for cross compiling Python 3.5? I'm trying to compile Python for an ARM processor. It seems like something broke with cross compilation in 3.5 and a patch was created (https://bugs.python.org/issue22359), but I don't even know where to begin with this. If anyone could help me out, I would appreciate it. Thanks. From rustompmody at gmail.com Mon May 23 11:30:54 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Mon, 23 May 2016 08:30:54 -0700 (PDT) Subject: for / while else doesn't make sense In-Reply-To: <87zirgrgol.fsf@bsb.me.uk> References: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> Message-ID: On Monday, May 23, 2016 at 7:59:47 PM UTC+5:30, Ben Bacarisse wrote: > Ian Kelly writes: > > > On Mon, May 23, 2016 at 2:09 AM, Steven D'Aprano wrote: > >> Are you saying that the Egyptians, Babylonians and Greeks didn't know how to > >> work with fractions? > >> > >> http://mathworld.wolfram.com/EgyptianFraction.html > >> > >> http://nrich.maths.org/2515 > >> > >> Okay, it's not quite 4000 years ago. Sometimes my historical sense of the > >> distant past is a tad inaccurate. Shall we say 2000 years instead? > > > > Those links give dates of 1650 BC and 1800 BC respectively, so I'd say > > your initial guess was closer. > > Right, but this is to miss the point. Let's say that 4000 years have > defined 1/3 to be one third, but Python 3 (as do many programming > languages) defines 1/3 to be something very very very very close to one > third, and *that* idea is very very very very new! It's unfortunate > that the example in this thread does not illustrate the main problem of > shifting to binary floating point, because 1/2 happens to be exactly > representable. Yes the point is being missed but in a different direction: The SET (as a completed whole) of real numbers (?) is no more than a 100 years old. People may have used fractions earlier And even here the first line of Steven's http://nrich.maths.org/2515 says "Did you know that fractions as we use them today didn't exist in Europe until the 17th century?" Egypt and Babylon (and India for that matter) are really only of archaeological interest in the sense that there is almost complete loss of continuity from then to now That the set ? legitimately exists was a minority view -- Cantor,Dedekind, Weierstrass... On the other side Kronecker belligerently declared: "The good Lord made the natural numbers (Zahlen in German) All the rest is the work of man" This was the MAINSTREAM view in the 1880s. As late as 1918 Weyl and Polya took a bet that math concepts such as real numbers, sets, countability etc would be relegated to history as a bad dream and the pristine purity of constructive math would be firmly established -- where "constructive math" basically means ? is the only reasonable infinite set and that ? is anything but real! https://en.wikipedia.org/wiki/Hermann_Weyl#Foundations_of_mathematics For a conspectus showing that: - our current views are fairly recent (compared to Egypt, Babylon etc) - that far from being universally accepted they were hotly disputed - And thence gave rise to our field of CS see http://blog.languager.org/2015/03/cs-history-0.html From jon+usenet at unequivocal.co.uk Mon May 23 11:35:33 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Mon, 23 May 2016 15:35:33 -0000 (UTC) Subject: for / while else doesn't make sense References: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-23, breamoreboy at gmail.com wrote: > On Monday, May 23, 2016 at 2:04:01 AM UTC+1, Jon Ribbens wrote: >> On 2016-05-23, Chris Angelico wrote: >> > The point of arithmetic in software is to do what mathematics defines. >> > Would you expect 1+2 to return 5? No. Why not? Where was the result >> > defined? >> >> Are you trying to compete with him for the Missing The Point Award? > > We had the RUE, now we've got the Resident Arithmetic Expert or RAE. > Just what the doctor didn't order. Who's that then? Maybe they could chip in with their opinion. From steve at pearwood.info Mon May 23 11:47:22 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 24 May 2016 01:47:22 +1000 Subject: for / while else doesn't make sense References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> Message-ID: <5743260b$0$1601$c3e8da3$5496439d@news.astraweb.com> On Tue, 24 May 2016 12:57 am, Chris Angelico wrote: > On Tue, May 24, 2016 at 12:29 AM, Ben Bacarisse > wrote: >> Right, but this is to miss the point. Let's say that 4000 years have >> defined 1/3 to be one third, but Python 3 (as do many programming >> languages) defines 1/3 to be something very very very very close to one >> third, and *that* idea is very very very very new! > > Have you ever written one third as 0.33333333 ? Because that's also > something very very close to one third. For many purposes, 0.33 is close enough to one third. -- Steven From ian.g.kelly at gmail.com Mon May 23 11:53:12 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 23 May 2016 09:53:12 -0600 Subject: for / while else doesn't make sense In-Reply-To: References: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> Message-ID: On Mon, May 23, 2016 at 9:30 AM, Rustom Mody wrote: > Yes the point is being missed but in a different direction: > The SET (as a completed whole) of real numbers (?) is no more than a 100 years > old. > People may have used fractions earlier > > And even here the first line of Steven's http://nrich.maths.org/2515 says > "Did you know that fractions as we use them today didn't exist in Europe until the 17th century?" > > Egypt and Babylon (and India for that matter) are really only of archaeological > interest in the sense that there is almost complete loss of continuity > from then to now So 13th century European merchants would have been entirely incapable of cutting a cheese wheel in half in order to accommodate a customer who didn't the whole thing? > That the set ? legitimately exists was a minority view -- Cantor,Dedekind, > Weierstrass... I'm not sure where ? comes into this in the first place. Existing Python numeric types only represent various subsets of ? (in the case of fractions.Fraction, the entirety of ?). > On the other side Kronecker belligerently declared: > "The good Lord made the natural numbers (Zahlen in German) > All the rest is the work of man" > > This was the MAINSTREAM view in the 1880s. > > As late as 1918 Weyl and Polya took a bet that math concepts such as > real numbers, sets, countability etc would be relegated to history as a bad > dream and the pristine purity of constructive math would be firmly established > -- where "constructive math" basically means ? is the only reasonable infinite set and that ? is anything but real! > > https://en.wikipedia.org/wiki/Hermann_Weyl#Foundations_of_mathematics I'm rather skeptical that this bet would have extended to fractions. From rosuav at gmail.com Mon May 23 11:57:14 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 May 2016 01:57:14 +1000 Subject: for / while else doesn't make sense In-Reply-To: <5743260b$0$1601$c3e8da3$5496439d@news.astraweb.com> References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <5743260b$0$1601$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, May 24, 2016 at 1:47 AM, Steven D'Aprano wrote: > On Tue, 24 May 2016 12:57 am, Chris Angelico wrote: > >> On Tue, May 24, 2016 at 12:29 AM, Ben Bacarisse >> wrote: >>> Right, but this is to miss the point. Let's say that 4000 years have >>> defined 1/3 to be one third, but Python 3 (as do many programming >>> languages) defines 1/3 to be something very very very very close to one >>> third, and *that* idea is very very very very new! >> >> Have you ever written one third as 0.33333333 ? Because that's also >> something very very close to one third. > > For many purposes, 0.33 is close enough to one third. As is 3.14 close enough to ?. (Or 256/81 or any of the other historical values.) Approximations are nothing new. ChrisA From ian.g.kelly at gmail.com Mon May 23 12:02:41 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 23 May 2016 10:02:41 -0600 Subject: for / while else doesn't make sense In-Reply-To: References: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> Message-ID: On Mon, May 23, 2016 at 9:53 AM, Ian Kelly wrote: > I'm not sure where ? comes into this in the first place. Existing > Python numeric types only represent various subsets of ? (in the case > of fractions.Fraction, the entirety of ?). And of course I realized after sending that I forgot about complex numbers. But even there Python merely represents 2-tuples of ?. From robin at reportlab.com Mon May 23 12:39:28 2016 From: robin at reportlab.com (Robin Becker) Date: Mon, 23 May 2016 17:39:28 +0100 Subject: str(float) python 3 versus 2.7 Message-ID: <7c1af35c-1ac7-96c7-3cba-ab9c8e542d8e@chamonix.reportlab.co.uk> I had always imagined that the str founction did some kind of rounding on floats to prevent small numerical errors from showing up. The 2.7 documentation starts like this > class str(object='') > Return a string containing a nicely printable representation of an object. For s However, I see a difference in the behaviour of python3.3, 3.4 & 3.5 when compared to python 2.7. > C:\Users\rptlab>\python33\python.exe -c"print(str(3*0.2))" > 0.6000000000000001 > > C:\Users\rptlab>\python34\python.exe -c"print(str(3*0.2))" > 0.6000000000000001 > > C:\Users\rptlab>\python35\python.exe -c"print(str(3*0.2))" > 0.6000000000000001 > > C:\Users\rptlab>\python27\python.exe -c"print(str(3*0.2))" > 0.6 I suppose I am being naive and should use the round function when computing tick labels, but that leads to other issues. Is there a sensible way to take a set of floats and find a suitable format to show significant figures for all, but leave off the noise? -- Robin Becker From rgaddi at highlandtechnology.invalid Mon May 23 12:44:30 2016 From: rgaddi at highlandtechnology.invalid (Rob Gaddi) Date: Mon, 23 May 2016 16:44:30 -0000 (UTC) Subject: Education [was Re: for / while else doesn't make sense] References: <573EC62F.4090401@lucidity.plus.com> <75115580-7a63-4cfb-9689-28eb84b580a0@googlegroups.com> <574032db$0$22142$c3e8da3$5496439d@news.astraweb.com> <87y4732van.fsf@elektro.pacujo.net> Message-ID: Marko Rauhamaa wrote: > Christopher Reimer : > >> Under various proposals in the U.S., everyone will soon learn how to >> program and/or become a computer scientist. Won't be long before some >> snotty-nosed brat graduates from preschool, takes a look at your code, >> and poops in his diapers. He will then dips his finger into his >> diaper, write on the whiteboard how your code can be written in a >> single line, and summary dismiss you with security escorting you off >> the premises. >> >> Gotta love the future. :) > > Unfortunately, most CS graduates don't seem to know how to program. > > Yes, some highschoolers could excel in the post of a senior software > engineer -- I've had the privilege of working alongside several > specimens. However, it has been known for half a century that good > developers are hard to come by. > > I think it is essential to learn the principles of programming just like > it is essential to learn the overall principles of nuclear fission or be > able to locate China on the map. However, a small minority of humanity > will ever earn a living writing code. > > At the same time, it may be that in the not-too-distant future, the > *only* jobs available will be coding jobs as we start to take the > finishing steps of automating all manufacturing, transportation and > services. Then, we will have a smallish class of overworked coders who > have no use or time for money and vast masses of jobless party-goers who > enjoy the fruits of the coders' labor. > > > Marko Well, so long as we're going wildly OT... I think it's not a matter of who's going to earn a living by it. I think it's that, increasingly, programming is similar to carpentry. I can't reframe a house, and certainly can't build cabinetry, but I can do an adequate job putting up a simple wooden shelf. Looked at that way, it becomes a question of teaching people enough of the general principles to be able to muddle though and do a passable job on trivial "But all I want to do is" tasks. It's not a CS degree, it's shop class. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. From ben.usenet at bsb.me.uk Mon May 23 12:51:14 2016 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Mon, 23 May 2016 17:51:14 +0100 Subject: for / while else doesn't make sense References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> Message-ID: <87oa7wra4d.fsf@bsb.me.uk> Chris Angelico writes: > On Tue, May 24, 2016 at 12:29 AM, Ben Bacarisse wrote: >> Right, but this is to miss the point. Let's say that 4000 years have >> defined 1/3 to be one third, but Python 3 (as do many programming >> languages) defines 1/3 to be something very very very very close to one >> third, and *that* idea is very very very very new! > > Have you ever written one third as 0.33333333 ? Not that I recall, but, obviously, I can't be sure. I can't even tell without counting how many 3s there are there. Why do you ask? > Because that's also > something very very close to one third. Yes it is, but I don't get what point you are making. -- Ben. From steve at pearwood.info Mon May 23 12:59:31 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 24 May 2016 02:59:31 +1000 Subject: for / while else doesn't make sense References: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> Message-ID: <574336f5$0$1600$c3e8da3$5496439d@news.astraweb.com> On Tue, 24 May 2016 12:29 am, Ben Bacarisse wrote: > Ian Kelly writes: > >> On Mon, May 23, 2016 at 2:09 AM, Steven D'Aprano >> wrote: >>> Are you saying that the Egyptians, Babylonians and Greeks didn't know >>> how to work with fractions? >>> >>> http://mathworld.wolfram.com/EgyptianFraction.html >>> >>> http://nrich.maths.org/2515 >>> >>> Okay, it's not quite 4000 years ago. Sometimes my historical sense of >>> the distant past is a tad inaccurate. Shall we say 2000 years instead? >> >> Those links give dates of 1650 BC and 1800 BC respectively, so I'd say >> your initial guess was closer. > > Right, but this is to miss the point. Let's say that 4000 years have > defined 1/3 to be one third, but Python 3 (as do many programming > languages) defines 1/3 to be something very very very very close to one > third, and *that* idea is very very very very new! It's unfortunate > that the example in this thread does not illustrate the main problem of > shifting to binary floating point, because 1/2 happens to be exactly > representable. That's not really the point. I acknowledge that floats do not represent all rational numbers (a/b) exactly. Neither do decimal floats -- most school children will learn that 0.333333333333 is not 1/3 exactly, and anyone who has used a calculator will experience calculations that give (say) 0.999999999 or 1.0000000001 instead of 1. And you know what? *People cope.* For all the weirdness of floating point, for all the rounding errors and violations of mathematical properties, floating point maths is *still* the best way to perform numerical calculations for many purposes. In fact, even IEEE-754 arithmetic, which is correctly rounded and therefore introduces the least possible rounding error, is sometimes "too good" -- people often turn to GPUs instead of CPUs for less accurate but faster bulk calculations. The point is, most people wouldn't really care that much whether 1/3 returned a binary 0.3333333333, or decimal 0.3333333333, or an exact rational fraction 1/3. Most people wouldn't care too much if they got 32-bits of precision or 64, or 10 decimal places or 18. When cutting (say) a sheet of paper into three equal pieces, they are unlikely to be able to measure and cut with an accuracy better than 1/2 of a millimeter, so 15 decimal places is overkill. But one thing is certain: very few people, Jon Ribbens being one of them, expects 1/3 to return 0. And that is why Python changed the meaning of the / operator: because using it for integer division was deeply unpopular and a bug magnet. -- Steven From rosuav at gmail.com Mon May 23 12:59:53 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 May 2016 02:59:53 +1000 Subject: for / while else doesn't make sense In-Reply-To: <87oa7wra4d.fsf@bsb.me.uk> References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <87oa7wra4d.fsf@bsb.me.uk> Message-ID: On Tue, May 24, 2016 at 2:51 AM, Ben Bacarisse wrote: > Chris Angelico writes: > >> On Tue, May 24, 2016 at 12:29 AM, Ben Bacarisse wrote: >>> Right, but this is to miss the point. Let's say that 4000 years have >>> defined 1/3 to be one third, but Python 3 (as do many programming >>> languages) defines 1/3 to be something very very very very close to one >>> third, and *that* idea is very very very very new! >> >> Have you ever written one third as 0.33333333 ? > > Not that I recall, but, obviously, I can't be sure. I can't even tell > without counting how many 3s there are there. Why do you ask? > >> Because that's also >> something very very close to one third. > > Yes it is, but I don't get what point you are making. You asserted that representing one third as something almost, but not exactly, one third was a new idea. It is not. Ever since ancient times, approximations have been used. Python is no different from anything else; the only reason it _looks_ different is that it's an approximation in binary, converted to decimal for display. ChrisA From ian.g.kelly at gmail.com Mon May 23 13:05:00 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 23 May 2016 11:05:00 -0600 Subject: str(float) python 3 versus 2.7 In-Reply-To: <7c1af35c-1ac7-96c7-3cba-ab9c8e542d8e@chamonix.reportlab.co.uk> References: <7c1af35c-1ac7-96c7-3cba-ab9c8e542d8e@chamonix.reportlab.co.uk> Message-ID: On Mon, May 23, 2016 at 10:39 AM, Robin Becker wrote: > I had always imagined that the str founction did some kind of rounding on > floats to prevent small numerical errors from showing up. The 2.7 > documentation starts like this > >> class str(object='') >> Return a string containing a nicely printable representation of an object. >> For s > > > > > However, I see a difference in the behaviour of python3.3, 3.4 & 3.5 when > compared to python 2.7. > >> C:\Users\rptlab>\python33\python.exe -c"print(str(3*0.2))" >> 0.6000000000000001 >> >> C:\Users\rptlab>\python34\python.exe -c"print(str(3*0.2))" >> 0.6000000000000001 >> >> C:\Users\rptlab>\python35\python.exe -c"print(str(3*0.2))" >> 0.6000000000000001 >> >> C:\Users\rptlab>\python27\python.exe -c"print(str(3*0.2))" >> 0.6 > > > I suppose I am being naive and should use the round function when computing > tick labels, but that leads to other issues. > > Is there a sensible way to take a set of floats and find a suitable format > to show significant figures for all, but leave off the noise? This was changed in 3.2; see https://bugs.python.org/issue9337 If you want to show the float in a less noisy format, you can explicitly format it using the 'g' or 'n' presentation type, which essentially round to a given precision and strip off trailing zeros: >>> '{:g}'.format(3*0.2) '0.6' It may also be worthwhile to read the section at https://docs.python.org/3/library/string.html#format-specification-mini-language that covers the precise formatting rules used by this. From jon+usenet at unequivocal.co.uk Mon May 23 13:09:24 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Mon, 23 May 2016 17:09:24 -0000 (UTC) Subject: for / while else doesn't make sense References: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <574336f5$0$1600$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-23, Steven D'Aprano wrote: > But one thing is certain: very few people, Jon Ribbens being one of them, > expects 1/3 to return 0. And that is why Python changed the meaning of > the / operator: because using it for integer division was deeply unpopular > and a bug magnet. Making it return floats is also a bug magnet, just for more subtle bugs that are harder to diagnose. From rosuav at gmail.com Mon May 23 13:33:55 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 May 2016 03:33:55 +1000 Subject: for / while else doesn't make sense In-Reply-To: References: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <574336f5$0$1600$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, May 24, 2016 at 3:09 AM, Jon Ribbens wrote: > On 2016-05-23, Steven D'Aprano wrote: >> But one thing is certain: very few people, Jon Ribbens being one of them, >> expects 1/3 to return 0. And that is why Python changed the meaning of >> the / operator: because using it for integer division was deeply unpopular >> and a bug magnet. > > Making it return floats is also a bug magnet, just for more subtle > bugs that are harder to diagnose. See my earlier post. There are four broadly-viable options: int, float, Fraction, Decimal. *Every one of them* is a bug magnet in one way or another. Which kind of bug would you like? * 1235678678/3 == 411892892 * (1/3) + (1/3) + (1/3) != 1 * x != x + 0 * useless reprs after a few rounds of addition Take your pick. :) By the way, did you know that Python has support for fraction literals? Just put this at the top of your code: import fractions; f = fractions.Fraction(1) and then you can use the special "tagged literal" syntax, like with special forms of string literal: >>> f*22/7 + f*2/11 Fraction(256, 77) There's even a short-hand form for the common case where the numerator is 1: >>> f/2 + f/3 Fraction(5, 6) Very handy, particularly interactively. :) ChrisA From random832 at fastmail.com Mon May 23 13:44:20 2016 From: random832 at fastmail.com (Random832) Date: Mon, 23 May 2016 13:44:20 -0400 Subject: for / while else doesn't make sense In-Reply-To: References: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <574336f5$0$1600$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1464025460.2711536.616274849.5789A279@webmail.messagingengine.com> On Mon, May 23, 2016, at 13:33, Chris Angelico wrote: > and then you can use the special "tagged literal" syntax, like with > special forms of string literal: > > >>> f*22/7 + f*2/11 > Fraction(256, 77) I like the infix fraction literal syntax better: 22/f/7 + 2/f/11. From ian.g.kelly at gmail.com Mon May 23 13:52:53 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 23 May 2016 11:52:53 -0600 Subject: for / while else doesn't make sense In-Reply-To: <1464025460.2711536.616274849.5789A279@webmail.messagingengine.com> References: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <574336f5$0$1600$c3e8da3$5496439d@news.astraweb.com> <1464025460.2711536.616274849.5789A279@webmail.messagingengine.com> Message-ID: On Mon, May 23, 2016 at 11:44 AM, Random832 wrote: > On Mon, May 23, 2016, at 13:33, Chris Angelico wrote: >> and then you can use the special "tagged literal" syntax, like with >> special forms of string literal: >> >> >>> f*22/7 + f*2/11 >> Fraction(256, 77) > > I like the infix fraction literal syntax better: 22/f/7 + 2/f/11. Sadly, the postfix syntax 22/7**f is a total failure. From petef4+usenet at gmail.com Mon May 23 13:55:47 2016 From: petef4+usenet at gmail.com (Pete Forman) Date: Mon, 23 May 2016 18:55:47 +0100 Subject: RFC: name for project of a cross version disassembler, and unmarshal program References: <2e3189e0-c1e3-49fa-b6ce-53471c4374e7@googlegroups.com> <3208147d-22bc-4873-b82c-85231effdec3@googlegroups.com> Message-ID: Rustom Mody writes: > On Monday, May 23, 2016 at 1:38:41 PM UTC+5:30, rocky wrote: >> On Monday, May 23, 2016 at 2:17:07 AM UTC-4, Pete Forman wrote: >> > rocky writes: >> > >> > > I'm looking for a good name for a relatively new project I'll put >> > > on pypy. >> > > >> > > I've been working on a module to disassemble Python bytecode from >> > > many versions of Python. (Right now 2.3 .. 3.5 bytecode, largely >> > > works.) >> > > >> > > Of course, in order to do that you also need routines to >> > > unmarshal bytecode. So that's in there as well. >> > > >> > > In the future, I may could add a marshaler and an assembler to >> > > Python bytecode. I know, this is kind of perverse. >> > > >> > > At any rate the name I've been using is "pyxdis". See >> > > https://github.com/rocky/python-pyxdis. >> > > >> > > In the past I've been told by Polish-speaking people that my >> > > names are hard to pronounce. (If you've ever heard any Polish >> > > tongue twisters, you'll know that this really hurts.) >> > > >> > > Any suggestions for a better name? >> > >> > relipmoc >> > >> > -- >> > Pete Forman >> >> Interesting. (For those who are slow like myself, it is "compile" >> backwards. Well it's actually "compiler" but I'd not be offended if you want to develop the idea. > Heh! I also wondered... > And if you are into that kinda stuff, how about: ?o?d?l? ? (?) I did consider suggesting that but it falls foul of PEP 426. PyPI is not yet ready to dip its toes outside of the confines of ASCII. [wry smile emoji deleted] -- Pete Forman From jon+usenet at unequivocal.co.uk Mon May 23 13:57:30 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Mon, 23 May 2016 17:57:30 -0000 (UTC) Subject: for / while else doesn't make sense References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <574336f5$0$1600$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-23, Chris Angelico wrote: > On Tue, May 24, 2016 at 3:09 AM, Jon Ribbens > wrote: >> On 2016-05-23, Steven D'Aprano wrote: >>> But one thing is certain: very few people, Jon Ribbens being one of them, >>> expects 1/3 to return 0. And that is why Python changed the meaning of >>> the / operator: because using it for integer division was deeply unpopular >>> and a bug magnet. >> >> Making it return floats is also a bug magnet, just for more subtle >> bugs that are harder to diagnose. > > See my earlier post. There are four broadly-viable options: int, > float, Fraction, Decimal. *Every one of them* is a bug magnet in one > way or another. Which kind of bug would you like? The simplest one - isn't that obvious from what I've already said? From alanoe at linux.vnet.ibm.com Mon May 23 14:06:05 2016 From: alanoe at linux.vnet.ibm.com (Alan Evangelista) Date: Mon, 23 May 2016 15:06:05 -0300 Subject: for / while else doesn't make sense In-Reply-To: References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <574336f5$0$1600$c3e8da3$5496439d@news.astraweb.com> <1464025460.2711536.616274849.5789A279@webmail.messagingengine.com> Message-ID: <201605231806.u4NI3v2I043785@mx0a-001b2d01.pphosted.com> On 05/23/2016 02:52 PM, Ian Kelly wrote: > On Mon, May 23, 2016 at 11:44 AM, Random832 wrote: >> On Mon, May 23, 2016, at 13:33, Chris Angelico wrote: >>> and then you can use the special "tagged literal" syntax, like with special forms of string >>> literal: >>>>>> f*22/7 + f*2/11 >>> Fraction(256, 77) >> I like the infix fraction literal syntax better: 22/f/7 + 2/f/11. > Sadly, the postfix syntax 22/7**f is a total failure. It seems this discussion has nothing to do with for/while else anymore.... I suggest you rename the subject. From rosuav at gmail.com Mon May 23 14:14:37 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 May 2016 04:14:37 +1000 Subject: for / while else doesn't make sense In-Reply-To: References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <574336f5$0$1600$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, May 24, 2016 at 3:57 AM, Jon Ribbens wrote: > On 2016-05-23, Chris Angelico wrote: >> On Tue, May 24, 2016 at 3:09 AM, Jon Ribbens >> wrote: >>> On 2016-05-23, Steven D'Aprano wrote: >>>> But one thing is certain: very few people, Jon Ribbens being one of them, >>>> expects 1/3 to return 0. And that is why Python changed the meaning of >>>> the / operator: because using it for integer division was deeply unpopular >>>> and a bug magnet. >>> >>> Making it return floats is also a bug magnet, just for more subtle >>> bugs that are harder to diagnose. >> >> See my earlier post. There are four broadly-viable options: int, >> float, Fraction, Decimal. *Every one of them* is a bug magnet in one >> way or another. Which kind of bug would you like? > > The simplest one - isn't that obvious from what I've already said? Gotcha. You're advocating Fraction, which has no bugs whatsoever. Of course, it doesn't give you any more numbers than int gives you [1], but at least it doesn't restrict you to crazy things like only fifty-three significant digits [2] in its binary representation. And it's really easy to work with; performance is predictable (it's just two integers, and we know that the performance of integers is predictable), and they're easy to eyeball. ChrisA [1] Rationals and integers are both "countably infinite" - there are as many of each as there are counting numbers, odd numbers, prime numbers, squares, Fibonacci numbers, and dates in the Julian calendar. And dates that nerds didn't get because they were too busy proving which things are countably infinite and which aren't. [2] The number of the Love Bug. This is probably significant. From petef4+usenet at gmail.com Mon May 23 14:16:56 2016 From: petef4+usenet at gmail.com (Pete Forman) Date: Mon, 23 May 2016 19:16:56 +0100 Subject: for / while else doesn't make sense References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> Message-ID: Ian Kelly writes: > On Mon, May 23, 2016 at 8:29 AM, Ben Bacarisse wrote: >> Ian Kelly writes: >> >>> On Mon, May 23, 2016 at 2:09 AM, Steven D'Aprano >>> wrote: >>>> Are you saying that the Egyptians, Babylonians and Greeks didn't >>>> know how to work with fractions? >>>> >>>> http://mathworld.wolfram.com/EgyptianFraction.html >>>> >>>> http://nrich.maths.org/2515 >>>> >>>> Okay, it's not quite 4000 years ago. Sometimes my historical sense >>>> of the distant past is a tad inaccurate. Shall we say 2000 years >>>> instead? >>> >>> Those links give dates of 1650 BC and 1800 BC respectively, so I'd >>> say your initial guess was closer. >> >> Right, but this is to miss the point. Let's say that 4000 years have >> defined 1/3 to be one third, but Python 3 (as do many programming >> languages) defines 1/3 to be something very very very very close to >> one third, and *that* idea is very very very very new! It's >> unfortunate that the example in this thread does not illustrate the >> main problem of shifting to binary floating point, because 1/2 >> happens to be exactly representable. > > I'm not going to dig back through the thread, but my recollection is > that's exactly why that example was chosen. Since 1/2 can be > represented exactly as a float, it *should* be represented as a float. > Picking another value (0) that isn't even close to the exact value of > 1/2 isn't helping anybody. Something else which I do not think has been stated yet in this thread is that floating point is an inexact representation. Just because integers and binary fractions have an exact correspondence we ought not to be affording them special significance. Floating point 1 is not the integer 1, it stands for a range of numbers some fraction either side of 1. There are other ways of handling non-integral numbers, such as fixed point, rational and unum. However current computing hardware is very much oriented to floating point, IEEE in particular. -- Pete Forman From ian.g.kelly at gmail.com Mon May 23 15:24:30 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 23 May 2016 13:24:30 -0600 Subject: for / while else doesn't make sense In-Reply-To: References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> Message-ID: On Mon, May 23, 2016 at 12:16 PM, Pete Forman wrote: > Something else which I do not think has been stated yet in this thread > is that floating point is an inexact representation. Just because > integers and binary fractions have an exact correspondence we ought not > to be affording them special significance. Floating point 1 is not the > integer 1, it stands for a range of numbers some fraction either side of > 1. This is not the case. Floating point 1 means exactly 1, no more and no less. Results that aren't exactly representable get rounded to values that are, but this does not imply that the value once rounded is inexact. The value that 1/3 gets rounded to is exactly equal to 6004799503160661 / 18014398509481984, no more and no less. Treating floating point values as inexact would require an accumulation of the range of error. Otherwise, it would no longer be correct to say that 1.0 * 1.0 == 1.0. The result could with equal correctness be the next floating point number after 1.0, or the previous value before 1.0. Continue multiplying by 1.0 and the error creeps larger and larger. The defined result of the operation, however, is the exact value 1.0. From ben.usenet at bsb.me.uk Mon May 23 15:29:03 2016 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Mon, 23 May 2016 20:29:03 +0100 Subject: for / while else doesn't make sense References: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <574336f5$0$1600$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87iny4r2tc.fsf@bsb.me.uk> Steven D'Aprano writes: > On Tue, 24 May 2016 12:29 am, Ben Bacarisse wrote: > >> Ian Kelly writes: >> >>> On Mon, May 23, 2016 at 2:09 AM, Steven D'Aprano >>> wrote: >>>> Are you saying that the Egyptians, Babylonians and Greeks didn't know >>>> how to work with fractions? >>>> >>>> http://mathworld.wolfram.com/EgyptianFraction.html >>>> >>>> http://nrich.maths.org/2515 >>>> >>>> Okay, it's not quite 4000 years ago. Sometimes my historical sense of >>>> the distant past is a tad inaccurate. Shall we say 2000 years instead? >>> >>> Those links give dates of 1650 BC and 1800 BC respectively, so I'd say >>> your initial guess was closer. >> >> Right, but this is to miss the point. Let's say that 4000 years have >> defined 1/3 to be one third, but Python 3 (as do many programming >> languages) defines 1/3 to be something very very very very close to one >> third, and *that* idea is very very very very new! It's unfortunate >> that the example in this thread does not illustrate the main problem of >> shifting to binary floating point, because 1/2 happens to be exactly >> representable. > > That's not really the point. I acknowledge that floats do not represent all > rational numbers (a/b) exactly. Neither do decimal floats -- most school > children will learn that 0.333333333333 is not 1/3 exactly, and anyone who > has used a calculator will experience calculations that give (say) > 0.999999999 or 1.0000000001 instead of 1. Yes, I got that. I'm sure you're aware of the consequences of a floating point representation. > And you know what? *People cope.* Yes, I agree. > For all the weirdness of floating point, for all the rounding errors and > violations of mathematical properties, floating point maths is *still* the > best way to perform numerical calculations for many purposes. Indeed. > In fact, even IEEE-754 arithmetic, which is correctly rounded and therefore > introduces the least possible rounding error, is sometimes "too good" -- > people often turn to GPUs instead of CPUs for less accurate but faster bulk > calculations. Yes, they do. > The point is, most people wouldn't really care that much whether 1/3 > returned a binary 0.3333333333, or decimal 0.3333333333, or an exact > rational fraction 1/3. Most people wouldn't care too much if they got > 32-bits of precision or 64, or 10 decimal places or 18. When cutting (say) > a sheet of paper into three equal pieces, they are unlikely to be able to > measure and cut with an accuracy better than 1/2 of a millimeter, so 15 > decimal places is overkill. I agree here too. Most people cope just fine with floating point. Usenet magnifies the number of people reporting issues (due to testing for equality or using them for currency and so on), but there is almost certainly a silent majority who either just figure it out or understand how to use floats from the get-go. But there is an issue, however small, and it's due, largely, to the fact that people can't see the representation of floats very well. All sorts of things conspire (for the very best of engineering reasons) to make it hard to see what actual value you are holding. Yes, people have been doing arithmetic for thousands of years, but they've usually done it by manipulating the representation themselves. The "behind the scenes" effects of floating point are a relatively new phenomenon though. > But one thing is certain: very few people, Jon Ribbens being one of them, > expects 1/3 to return 0. And that is why Python changed the meaning of > the / operator: because using it for integer division was deeply unpopular > and a bug magnet. Yup. I agree with everything you've said. -- Ben. From ben.usenet at bsb.me.uk Mon May 23 15:55:52 2016 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Mon, 23 May 2016 20:55:52 +0100 Subject: for / while else doesn't make sense References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <87oa7wra4d.fsf@bsb.me.uk> Message-ID: <87d1ocr1kn.fsf@bsb.me.uk> Chris Angelico writes: > On Tue, May 24, 2016 at 2:51 AM, Ben Bacarisse wrote: >> Chris Angelico writes: >> >>> On Tue, May 24, 2016 at 12:29 AM, Ben Bacarisse wrote: >>>> Right, but this is to miss the point. Let's say that 4000 years have >>>> defined 1/3 to be one third, but Python 3 (as do many programming >>>> languages) defines 1/3 to be something very very very very close to one >>>> third, and *that* idea is very very very very new! >>> >>> Have you ever written one third as 0.33333333 ? >> >> Not that I recall, but, obviously, I can't be sure. I can't even tell >> without counting how many 3s there are there. Why do you ask? >> >>> Because that's also >>> something very very close to one third. >> >> Yes it is, but I don't get what point you are making. > > You asserted that representing one third as something almost, but not > exactly, one third was a new idea. Ah, I didn't mean to assert that at all. I meant to say that the new thing is that the approximation is tucked away in a representation largely hidden from the person doing the arithmetic. Its floats in programming languages that are new, not approximations. > It is not. Ever since ancient > times, approximations have been used. Python is no different from > anything else; the only reason it _looks_ different is that it's an > approximation in binary, converted to decimal for display. It's often hard for people to see that actual value[1] in a program, but not when they do arithmetic by hand. This is what confused me about your question. Were I to approximate 1/3 with 0.333 (on paper) there should be no surprises down the line. I know that 3 times it won't be 1, but 0.999. Confusion creeps in when the approximation is not explicit in the eye of the beginner. Obviously, to you and me, 1/3 is clearly an approximation in Python -- we know that / explicitly makes floats -- but not everyone knows the full consequences of that right away. I doubt anyone, beginner or not, would be surprised that 0.33333333 is not actually one third (though they *might* be surprised if it's not 33333333/100000000). [1] Not being a Python expert I don't know how you show that actual value of a float. What is the Pythonic way to do that? -- Ben. From mdickinson at enthought.com Mon May 23 16:17:44 2016 From: mdickinson at enthought.com (Mark Dickinson) Date: Mon, 23 May 2016 20:17:44 +0000 (UTC) Subject: for / while else doesn't make sense References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <87oa7wra4d.fsf@bsb.me.uk> <87d1ocr1kn.fsf@bsb.me.uk> Message-ID: Ben Bacarisse bsb.me.uk> writes: > [1] Not being a Python expert I don't know how you show that actual > value of a float. What is the Pythonic way to do that? I don't know about Pythonic, but here are some options. 1. Convert the float to Decimal, and print the result. This shows the exact binary value that's stored, but displays it in decimal. Be aware that the result will be hundreds of digits long for very large or very small floats. >>> print(Decimal(pi)) 3.141592653589793115997963468544185161590576171875 2. If you're happy with a hexadecimal representation, use the float.hex method. Again, this shows the exact value stored. >>> print(pi.hex()) 0x1.921fb54442d18p+1 3. To get an equivalent fraction, convert to the fractions.Fraction type or use the as_integer_ratio method. >>> from fractions import Fraction >>> print(Fraction(pi)) 884279719003555/281474976710656 >>> print(pi.as_integer_ratio()) (884279719003555, 281474976710656) -- Mark (who should probably take a numerical methods class someday) From l.mohanphy at gmail.com Mon May 23 16:20:07 2016 From: l.mohanphy at gmail.com (Mohan L) Date: Tue, 24 May 2016 01:50:07 +0530 Subject: [pyinotify] help required to send only one mail for chunk of events Message-ID: Hi All, I am using the bellow script to watch directories. Using 20 seconds to aggregate together a larger chunk of events and enabled coalescing of events. I wanted to send an email notification with content of logfile after 15 mins on any change. The idea is I want to send only one mail for chunk of events.Say for example, I want to send only one email when some one extract a tar.gz file in watched directory. I can write method to send email (say send_mail) which accept content of logfile. But I am not sure How to integrate it with below code so that my send_mail method get triggered only once for chunk of events. Any help will be really appreciated. Thanks for your time and efforts. Here is code: #!/usr/bin/env python import sys try: import pyinotify except: print "pyinotify not installed" sys.exit(1) import datetime from ConfigParser import SafeConfigParser class MyEventHandler(pyinotify.ProcessEvent): log = "" flog = None def process_IN_CREATE(self, event): """ This method processes a specific type of event: IN_CREATE. event is an instance of Event. """ self.write_msg("CREATING", event.pathname) def process_IN_DELETE(self, event): """ This method processes a specific type of event: IN_DELETE. event is an instance of Event. """ self.write_msg("DELETING", event.pathname) def process_IN_MODIFY(self, event): """ This method processes a specific type of event: IN_MODIFY. event is an instance of Event. """ self.write_msg("MODIFIED", event.pathname) def process_IN_OPEN(self, event): """ This method processes a specific type of event: IN_OPEN. event is an instance of Event. """ self.write_msg("OPENED", event.pathname) def process_IN_ACCESS(self, event): """ This method processes a specific type of event: IN_ACCESS. event is an instance of Event. """ self.write_msg("ACCESSED", event.pathname) def process_IN_ATTRIB(self, event): """ This method processes a specific type of event: IN_ATTRIB. event is an instance of Event. """ self.write_msg("ATTRIB", event.pathname) def process_IN_DELETE_SELF(self, event): """ This method processes a specific type of event: IN_DELETE_SELF. event is an instance of Event. """ self.write_msg("DELETE_SELF", event.pathname) def process_IN_MOVED_FROM(self, event): """ This method processes a specific type of event: IN_MOVED_FROM. event is an instance of Event. """ self.write_msg("FILE MOVED FROM", event.pathname) def process_IN_MOVED_TO(self, event): """ This method processes a specific type of event: IN_MOVED_TO. event is an instance of Event. """ try: msg = event.src_pathname + " -> " + event.pathname except: msg = event.pathname self.write_msg("FILE MOVED INTO", msg) def openlog(self, logfile): """ This method opens log file in append mode. """ self.log = logfile self.flog = file(self.log, 'a') def write_msg(self, event, msg): """ This method writes evens in logfile as well as stdout. """ out_msg = self.generate_timestamp() + "\t" + event + ": " + msg + "\n" if len(self.log) > 0: self.flog.write(out_msg) self.flog.flush() print "LOGGED", out_msg else: print "NOT LOGGED", out_msg def generate_timestamp(self): """ This method generate and return timestamp for log. Timestamp format is YYYY/MM/DD-HH:MM:SS. """ d = datetime.datetime.now() datestr = "%d/%.2d/%.2d-%.2d:%.2d:%.2d" % (d.year, d.month, d.day, d.hour, d.minute, d.second) return datestr def main(): parser = SafeConfigParser() parser.read('/etc/mywatcherscript.ini') ### The watch manager stores the watches and provides operations on watches ### wm = pyinotify.WatchManager() ### List of watched events. To specify two or more events just orize them ### mask = pyinotify.IN_CREATE | pyinotify.IN_DELETE | pyinotify.IN_MODIFY # List of watch path #watched_path_list=['/var/log','/tmp'] watched_path_list=parser.get('mywatcherscript', 'watched_path_list').split(',') pid_file_name=parser.get('mywatcherscript', 'pid_file_name') logfile=parser.get('mywatcherscript', 'logfile') # Add watch(s) on the provided path(s) with associated flag value. # Parameters: # path - Path to watch, the path can either be a file or a directory. # Also accepts a sequence (list) of paths. # mask (int) - Bitmask of events. # rec (bool) - Recursively add watches from path on all its subdirectories. # auto_add (bool) - Automatically add watches on newly created directories in watched parent directory. # # Returns: dict of {str: int} # dict of paths associated to watch descriptors. wm.add_watch(watched_path_list, mask, rec=True, auto_add=True) # event handler eh = MyEventHandler() eh.openlog(logfile) # notifier # Put an arbitrary large value (20 seconds) to aggregate together a larger # chunk of events. For instance if you repeat several times a given action # on the same file its events will be coalesced into a single event and only # one event of this type will be reported (for this period). notifier = pyinotify.Notifier(wm, eh, read_freq=20) #notifier = pyinotify.Notifier(wm, eh) # Enable coalescing of events. notifier.coalesce_events() #notifier.loop() notifier.loop(daemonize=True, pid_file=pid_file_name) if __name__ == '__main__': main() -- Thanks & Regards Mobile: 9444955058 E-Mail: l.mohanphy at gmail.com | thefossgeek at gmail.com IRC: neophy Blog : http://lmohanphy.livejournal.com/ From grant.b.edwards at gmail.com Mon May 23 16:22:54 2016 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 23 May 2016 20:22:54 +0000 (UTC) Subject: for / while else doesn't make sense References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> Message-ID: On 2016-05-23, Ian Kelly wrote: > On Mon, May 23, 2016 at 9:53 AM, Ian Kelly wrote: >> I'm not sure where ? comes into this in the first place. Existing >> Python numeric types only represent various subsets of ? (in the case >> of fractions.Fraction, the entirety of ?). > > And of course I realized after sending that I forgot about complex > numbers. But even there Python merely represents 2-tuples of ?. OK, admit it, now you're all just showing off the fact that you know how to type in those fancy symbols. -- Grant Edwards grant.b.edwards Yow! TAILFINS!! ... click at ... gmail.com From ben.usenet at bsb.me.uk Mon May 23 17:01:49 2016 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Mon, 23 May 2016 22:01:49 +0100 Subject: for / while else doesn't make sense References: <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <87oa7wra4d.fsf@bsb.me.uk> <87d1ocr1kn.fsf@bsb.me.uk> Message-ID: <877fekqyiq.fsf@bsb.me.uk> Mark Dickinson writes: > Ben Bacarisse bsb.me.uk> writes: >> [1] Not being a Python expert I don't know how you show that actual >> value of a float. What is the Pythonic way to do that? > > I don't know about Pythonic, but here are some options. > > 1. Convert the float to Decimal, and print the result. This shows > the exact binary value that's stored, but displays it in decimal. > Be aware that the result will be hundreds of digits long for > very large or very small floats. > > >>> print(Decimal(pi)) > 3.141592653589793115997963468544185161590576171875 > > 2. If you're happy with a hexadecimal representation, use the > float.hex method. Again, this shows the exact value stored. > > >>> print(pi.hex()) > 0x1.921fb54442d18p+1 > > 3. To get an equivalent fraction, convert to the fractions.Fraction > type or use the as_integer_ratio method. > > >>> from fractions import Fraction > >>> print(Fraction(pi)) > 884279719003555/281474976710656 > >>> print(pi.as_integer_ratio()) > (884279719003555, 281474976710656) Thanks. I worked out 1 and 3 after posting but I was not aware of how to get 2 in Python. It's the format I am most familiar with (and it's very compact for numbers that are anything but in decimal or as fractions) so that's handy. Once you told me that hex() does the trick I wondered if "%x" % 3.4 would give me 0x1.b333333333333p+1 but the float is converted to the expected integer. I doubt anyone relies on using %x for the integer part of a float so maybe %x could be provided for floats... -- Ben. From petef4+usenet at gmail.com Mon May 23 17:50:46 2016 From: petef4+usenet at gmail.com (Pete Forman) Date: Mon, 23 May 2016 22:50:46 +0100 Subject: for / while else doesn't make sense References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> Message-ID: Ian Kelly writes: > On Mon, May 23, 2016 at 12:16 PM, Pete Forman wrote: >> Something else which I do not think has been stated yet in this >> thread is that floating point is an inexact representation. Just >> because integers and binary fractions have an exact correspondence we >> ought not to be affording them special significance. Floating point 1 >> is not the integer 1, it stands for a range of numbers some fraction >> either side of 1. > > This is not the case. Floating point 1 means exactly 1, no more and no > less. Results that aren't exactly representable get rounded to values > that are, but this does not imply that the value once rounded is > inexact. The value that 1/3 gets rounded to is exactly equal to > 6004799503160661 / 18014398509481984, no more and no less. > > Treating floating point values as inexact would require an > accumulation of the range of error. Otherwise, it would no longer be > correct to say that 1.0 * 1.0 == 1.0. The result could with equal > correctness be the next floating point number after 1.0, or the > previous value before 1.0. Continue multiplying by 1.0 and the error > creeps larger and larger. The defined result of the operation, > however, is the exact value 1.0. Let us for the sake of argument consider a floating point representation that has one decimal point of precision. It is easier to talk about than the IEEE double precision used in Python. If you are talking about the multiplicative identity element then 1.0 is exactly unity and repeated multiplication will not change the result. However I am coming from scientific measurements where 1.0 is the stored value for observations between 0.95 and 1.05. If I wish to correlate two signals that were 1.03 and 1.04 at higher precision then 1.1 is actually a reasonable value for their "correct" product, even though 1.0 is what is prescribed by IEEE-style rules because they are both stored as 1.0. I agree that errors tend to creep ever larger. IEEE rounding rules seek to minimize the error of the result but management of the accumulated error is up to the scientist / programmer, it is not part of the IEEE arithmetic. -- Pete Forman From christopher_reimer at icloud.com Mon May 23 18:36:24 2016 From: christopher_reimer at icloud.com (Christopher Reimer) Date: Mon, 23 May 2016 15:36:24 -0700 Subject: for / while else doesn't make sense In-Reply-To: References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> Message-ID: <21745CE1-A5BA-4B24-86BF-0B64DA0C886E@icloud.com> > On May 23, 2016, at 1:22 PM, Grant Edwards wrote: > >> On 2016-05-23, Ian Kelly wrote: >>> On Mon, May 23, 2016 at 9:53 AM, Ian Kelly wrote: >>> I'm not sure where ? comes into this in the first place. Existing >>> Python numeric types only represent various subsets of ? (in the case >>> of fractions.Fraction, the entirety of ?). >> >> And of course I realized after sending that I forgot about complex >> numbers. But even there Python merely represents 2-tuples of ?. > > OK, admit it, now you're all just showing off the fact that you know > how to type in those fancy symbols. Those symbols are blowing my 8-bit ASCII brain. :) Chris R. From michael.selik at gmail.com Mon May 23 19:00:30 2016 From: michael.selik at gmail.com (Michael Selik) Date: Mon, 23 May 2016 23:00:30 +0000 Subject: numpy problem In-Reply-To: <17D9C96A-C4EB-426F-A4A7-AA6492D7DBAB@onemanifest.net> References: <46F1751E-CA1E-4AB3-B6EB-589E1E2FD1D9@onemanifest.net> <17D9C96A-C4EB-426F-A4A7-AA6492D7DBAB@onemanifest.net> Message-ID: On Mon, May 23, 2016 at 9:12 AM wrote: > > On 23 mei 2016, at 14:19, Peter Otten <__peter__ at web.de> wrote: > > lists at onemanifest.net wrote: > > > >> I've got a 2D array > >> And an array of indexes that for shows which row to keep for each column > >> of values: > >> > >> keep = np.array([2, 3, 1, 9, 2]) > >> > >> So, the result should be an array like array([ values[2,0], values[3,1], > >> values[1,2], values[9,3], values[2,4] ]) == np.array([92, 62, 38, 81, > 44]) > >> > > values[keep].diagonal() > > That seems to do the trick! > To clarify, the fancy index is selecting all rows you want to keep, which gives you a 5x5 array (repeating rows if necessary). Then you are getting the diagonal of that selection. If that was already clear, my apologies for the noise. From rosuav at gmail.com Mon May 23 20:09:36 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 May 2016 10:09:36 +1000 Subject: [pyinotify] help required to send only one mail for chunk of events In-Reply-To: References: Message-ID: On Tue, May 24, 2016 at 6:20 AM, Mohan L wrote: > I wanted to send an email notification with content of logfile after 15 > mins on any change. The idea is I want to send only one mail for chunk of > events.Say for example, I want to send only one email when some one extract > a tar.gz file in watched directory. I'm ignoring your actual code, because the formatting is all messed up - your mail/news client appears to have destroyed your indentation. But there are two straight-forward ways of doing what you want here: 1) Whenever there's a change, wait fifteen minutes, then send an email with all the recent changes. 2) Whenever there's a change, check to see when you last sent an email, and if it's more than fifteen minutes ago, send one immediately. I'm not entirely sure how to interpret your "after 15 mins", but my guess is that it's one of the above. Oh, I lied. I am looking at your code, just a bit. > try: > import pyinotify > except: > print "pyinotify not installed" > sys.exit(1) Don't do this. Ever. If you want to use pyinotify, there's one thing to do: import pyinotify If that doesn't work, Python will print an error message and then exit 1, so having code like this is at best just duplicating what already exists. The default message is far more helpful, though, so you're destroying information - a lot of it. You use a bare 'except:' further down, too; again, don't. Figure out _exactly_ what exception you're intending to catch, and catch that. Finally: Consider upgrading to Python 3. With a script this size, I doubt there's anything holding you on Python 2 (the two primary dependencies are pyinotify and the sending of emails, both of which are available on Py3), so this would be well worth porting. ChrisA From rosuav at gmail.com Mon May 23 20:11:24 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 May 2016 10:11:24 +1000 Subject: for / while else doesn't make sense In-Reply-To: <0m67kb5n173pvo4rtk5d47g7ds8gnnj0lh@4ax.com> References: <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <0m67kb5n173pvo4rtk5d47g7ds8gnnj0lh@4ax.com> Message-ID: On Tue, May 24, 2016 at 10:07 AM, Dennis Lee Bieber wrote: > Now... Figuring out how many pfennig to charge for that half wheel... > THAT requires number crunching Nah, you just take the number of pfennig you would have charged, and charge that many halbpfennig instead. At least, that's what you'd do in the Grand Duchy ruled over by the miserly Rudolph. ChrisA From steve at pearwood.info Mon May 23 20:38:58 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 24 May 2016 10:38:58 +1000 Subject: for / while else doesn't make sense References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> Message-ID: <5743a2a5$0$1620$c3e8da3$5496439d@news.astraweb.com> On Tue, 24 May 2016 04:16 am, Pete Forman wrote: > Something else which I do not think has been stated yet in this thread > is that floating point is an inexact representation. Just because > integers and binary fractions have an exact correspondence we ought not > to be affording them special significance. Floating point 1 is not the > integer 1, it stands for a range of numbers some fraction either side of > 1. That's a matter of interpretation. Sometimes that is the case: if the result of some calculation involving errors happens to result in 1.0 (rather than, say, 1.00270128) by the merest coincidence, then there is nothing special about 1.0 and it represents some interval, same as any other float in that situation. But if for some strange reason you decide to do integer maths using floats: x = 14.0/2.0 - 1.0 then the result, x == 1.0, is exact (assuming that the quantities 14.0 etc. represent exact constants rather than inexact measured quantities with associated measurement error). And: py> math.sqrt(1) 1.0 is not a range of numbers, it is an exact result. IEEE 754 floating point has the concept of exact and inexact calculations. When it comes to floats, Python unfortunately has no reliable way to tell whether a calculation was exact or inexact, but the decimal module does. py> from decimal import * py> with localcontext(Context(traps=[Inexact])): ... Decimal(25).sqrt() - Decimal(2)**Decimal(2) ... Decimal('1') That decimal 1 is *exact*. There's no rounding error. Provided that the constants themselves are free of any measurement error, there is no error at all in this calculation. Also exact: py> with localcontext(Context(traps=[Inexact])): ... Decimal(10).log10() ... Decimal('1') Whereas this Decimal 1 is inexact: py> d = Decimal("1.0000000000000000000000000001") py> assert d != Decimal(1) py> d.sqrt() == 1 True We can trap the inexact calculation: py> with localcontext(Context(traps=[Inexact])): ... d.sqrt() ... Traceback (most recent call last): File "", line 2, in decimal.Inexact: [] -- Steven From steve at pearwood.info Mon May 23 20:57:16 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 24 May 2016 10:57:16 +1000 Subject: Numerical methods [was Re: for / while else doesn't make sense] References: <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <87oa7wra4d.fsf@bsb.me.uk> <87d1ocr1kn.fsf@bsb.me.uk> Message-ID: <5743a6ee$0$1605$c3e8da3$5496439d@news.astraweb.com> On Tue, 24 May 2016 06:17 am, Mark Dickinson signed his post: > (who should probably take a numerical methods class someday) Really? I would have thought you could have taught a numerical methods class. You certainly know more about it than I do, and I have taken one! (Twice!) -- Steven From steve at pearwood.info Mon May 23 21:05:21 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 24 May 2016 11:05:21 +1000 Subject: for / while else doesn't make sense References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <21745CE1-A5BA-4B24-86BF-0B64DA0C886E@icloud.com> Message-ID: <5743a8d2$0$1587$c3e8da3$5496439d@news.astraweb.com> On Tue, 24 May 2016 08:36 am, Christopher Reimer wrote: > Those symbols are blowing my 8-bit ASCII brain. :) That's certainly true, because there is no such thing as 8-bit ASCII. ASCII is a 7-bit encoding. (Most implementations set the extra bit to zero, a few *very* old machines might have set it to one, and if I remember correctly, Wordstar used to use it for its own internal purposes.) What people usually mean by "extended 8-bit ASCII" is a large family of many different extensions to ASCII, many of which have multiple names for added confusion, such as Latin-1, ISO-8859-2, MacRoman, and many more. None of these are ASCII, although they contain ASCII as a subset of their characters (just as Unicode does). -- Steven From chris.barker at noaa.gov Mon May 23 21:36:06 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 23 May 2016 18:36:06 -0700 Subject: [Python-Dev] Python 3.6.0a1 is now available In-Reply-To: <28E1448B-F604-48EB-B4E4-0AE2528C8383@python.org> References: <28E1448B-F604-48EB-B4E4-0AE2528C8383@python.org> Message-ID: <-7150763876064620774@unknownmsgid> Ned, It looks like you're still building for OS-X v. 10.6. I can't find an official statement, but this: http://www.computerworld.com/article/2950580/operating-systems/the-end-is-near-for-os-x-mountain-lion-support.html Indicates that 10.8 is falling off Apple's support. Maybe it's time for a newer baseline. -Chris > On May 17, 2016, at 2:18 PM, Ned Deily wrote: > > On behalf of the Python development community and the Python 3.6 release > team, I'm happy to announce the availability of Python 3.6.0a1. > 3.6.0a1 is the first of four planned alpha releases of Python 3.6, > the next major release of Python. During the alpha phase, Python 3.6 > remains under heavy development: additional features will be added > and existing features may be modified or deleted. Please keep in mind > that this is a preview release and its use is not recommended for > production environments. > > You can find Python 3.6.0a1 here: > > https://www.python.org/downloads/release/python-360a1/ > > The next release of Python 3.6 will be 3.6.0a2, currently scheduled for > 2016-06-13. > > Enjoy! > > --Ned > > -- > Ned Deily > nad at python.org -- [] > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov From steve at pearwood.info Mon May 23 22:15:25 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 24 May 2016 12:15:25 +1000 Subject: for / while else doesn't make sense References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <574336f5$0$1600$c3e8da3$5496439d@news.astraweb.com> Message-ID: <5743b93e$0$1589$c3e8da3$5496439d@news.astraweb.com> On Tue, 24 May 2016 03:09 am, Jon Ribbens wrote: > On 2016-05-23, Steven D'Aprano wrote: >> But one thing is certain: very few people, Jon Ribbens being one of them, >> expects 1/3 to return 0. And that is why Python changed the meaning of >> the / operator: because using it for integer division was deeply >> unpopular and a bug magnet. > > Making it return floats is also a bug magnet, just for more subtle > bugs that are harder to diagnose. Floating point arithmetic does contain traps for the unwary, sometimes very subtle ones. But they aren't *bugs* -- they're inherent in the nature of floating point arithmetic. Using / for integer division is a bug magnet. See my response to the "Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5" thread: https://mail.python.org/pipermail/python-list/2016-May/709330.html The old way of using / was a bug magnet, because people would write code like this: def harmonic_mean(values): n = len(values) total = sum(1/x for x in values) return n/total and test it with floats: py> harmonic_mean([1.0, 2.0, 4.0]) 1.7142857142857142 which matches the exact result of 12/7. And then some day somebody sneaks in an integer by mistake, and the result goes completely to hell: py> harmonic_mean([1.0, 2, 4.0]) 2.4 Changing / to perform true division slays this particular bug magnet, without introducing any new ones. Floats themselves are an abstraction for real mathematical arithmetic. (Perhaps a better term is "reification".) And for most purposes, they work well-enough, especially with IEE-754 arithmetic. You can write a lot of naive maths code treating floats as if they were exact Real numbers, and with a few judicious tweaks to your printing routines, nobody will know the difference. But all abstractions (and reifications) leak: http://www.joelonsoftware.com/articles/LeakyAbstractions.html and floats are no exception. But what are you going to do? Use integer maths? You still have to deal with rounding error. In Australia, we have an 11% consumption tax, the GST. I cannot tell you how many times I've needed to add a 1 cent "Rounding" amount on invoices to get the results to work out correctly. E.g. if an item costs $17 including tax, then you have a choice in rounding the tax-free cost down to $15.31 or up to $15.32, depending on the rounding mode, which in turn gives you the tax-inclusive cost of either $16.99 or $17.01. Either way you need a one cent adjustment. If you use integer maths, it always rounds down: py> 1700*100//111 # $17.00/1.11 calculated in cents 1531 py> 1531*111//100 1699 (Reminds me of using Forth without a floating point stack.) The right solution is actually to use a Decimal format, or a correctly rounded IEEE-754 integer format. But even that will still leak somewhere. Using integers in this case is not only *harder* than using floats, but it's more likely to go wrong. Because integer division always rounds down, errors accumulate faster than with floats, where calculations will be correctly rounded to minimize the error. -- Steven From ned at nedbatchelder.com Mon May 23 22:19:45 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Mon, 23 May 2016 19:19:45 -0700 (PDT) Subject: for / while else doesn't make sense In-Reply-To: <5743a8d2$0$1587$c3e8da3$5496439d@news.astraweb.com> References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <21745CE1-A5BA-4B24-86BF-0B64DA0C886E@icloud.com> <5743a8d2$0$1587$c3e8da3$5496439d@news.astraweb.com> Message-ID: <71549549-f518-4c4c-818e-2751c09e2a32@googlegroups.com> On Monday, May 23, 2016 at 9:06:13 PM UTC-4, Steven D'Aprano wrote: > On Tue, 24 May 2016 08:36 am, Christopher Reimer wrote: > > > Those symbols are blowing my 8-bit ASCII brain. :) > > That's certainly true, because there is no such thing as 8-bit ASCII. ASCII > is a 7-bit encoding. (Most implementations set the extra bit to zero, a few > *very* old machines might have set it to one, and if I remember correctly, > Wordstar used to use it for its own internal purposes.) > > What people usually mean by "extended 8-bit ASCII" is a large family of many > different extensions to ASCII, many of which have multiple names for added > confusion, such as Latin-1, ISO-8859-2, MacRoman, and many more. None of > these are ASCII, although they contain ASCII as a subset of their > characters (just as Unicode does). Ugh, can we please stop with the "well, actually" pedantic tangents? I know we are fascinated by the details of computing, and yes, details matter, but this just gets exhausting. The good dynamic here of helping people with Python gets lost in the endless trivia threads, and bickering over details that are four or five times removed from the original question. Assume the best of others, and stay focused :) --Ned. From torriem at gmail.com Mon May 23 22:50:30 2016 From: torriem at gmail.com (Michael Torrie) Date: Mon, 23 May 2016 20:50:30 -0600 Subject: [Python-Dev] Python 3.6.0a1 is now available In-Reply-To: <-7150763876064620774@unknownmsgid> References: <28E1448B-F604-48EB-B4E4-0AE2528C8383@python.org> <-7150763876064620774@unknownmsgid> Message-ID: <6a1fba2d-0313-59ec-c4f9-62f4206f84d4@gmail.com> On 05/23/2016 07:36 PM, Chris Barker - NOAA Federal wrote: > Ned, > It looks like you're still building for OS-X v. 10.6. > > I can't find an official statement, but this: > http://www.computerworld.com/article/2950580/operating-systems/the-end-is-near-for-os-x-mountain-lion-support.html > > Indicates that 10.8 is falling off Apple's support. > > Maybe it's time for a newer baseline. Unless there's some technical reason that compiling for 10.6 and greater is a bad idea (breaks something on newer OS's or otherwise does something bad[1]), I think staying with 10.6 is preferrable. Obviously Python itself doesn't require anything that was introduced in OS X later versions, yet. I know of lots of users still hanging on to 10.6 because it just works for their needs. Until such time as Python requires a newer API, why not stick with what works. Does it not still run fine on 10.11 or whatever they are up to now? [1] Dropping support for Windows XP is as much to do with newer APIs that Python uses as anything else, if I'm not mistaken. From rustompmody at gmail.com Tue May 24 01:02:20 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Mon, 23 May 2016 22:02:20 -0700 (PDT) Subject: When were real numbers born? (was for / while else doesn't make sense) In-Reply-To: References: <573f9322$0$1616$c3e8da3$5496439d@news.astraweb.com> <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> Message-ID: On Tuesday, May 24, 2016 at 4:10:59 AM UTC+5:30, Ian wrote: > On Mon, May 23, 2016 at 9:30 AM, Rustom Mody wrote: > > Yes the point is being missed but in a different direction: > > The SET (as a completed whole) of real numbers (?) is no more than a 100 years > > old. > > People may have used fractions earlier > > > > And even here the first line of Steven's http://nrich.maths.org/2515 says > > "Did you know that fractions as we use them today didn't exist in Europe until the 17th century?" > > > > Egypt and Babylon (and India for that matter) are really only of archaeological > > interest in the sense that there is almost complete loss of continuity > > from then to now > > So 13th century European merchants would have been entirely incapable > of cutting a cheese wheel in half in order to accommodate a customer > who didn't the whole thing? That people could compute with fractions does not mean they had reified ? as a set. Cantor did something which was a complete No-No in math until that point -- assume that completed infinities are meaningful. A programming example would be the question: What does this program do after it finishes printing? i = 0 while True: print i i += 1 > > > That the set ? legitimately exists was a minority view -- Cantor,Dedekind, > > Weierstrass... > > I'm not sure where ? comes into this in the first place. Existing > Python numeric types only represent various subsets of ? (in the case > of fractions.Fraction, the entirety of ?). > > > On the other side Kronecker belligerently declared: > > "The good Lord made the natural numbers (Zahlen in German) > > All the rest is the work of man" > > > > This was the MAINSTREAM view in the 1880s. > > > > As late as 1918 Weyl and Polya took a bet that math concepts such as > > real numbers, sets, countability etc would be relegated to history as a bad > > dream and the pristine purity of constructive math would be firmly established > > -- where "constructive math" basically means ? is the only reasonable infinite set and that ? is anything but real! > > > > https://en.wikipedia.org/wiki/Hermann_Weyl#Foundations_of_mathematics > > I'm rather skeptical that this bet would have extended to fractions. Yes... Its hard to guess what Kronecker/Cantor/Hilbert/Brouwer etc believed other than the records we have. But we can make some guesses... Insofar as the fractions are enumerable and computable they would be said to exist (by everyone) Insofar as ? is non-denumerable its existence is suspect (by the constructivists) Now float is a ghastly approximation to ? doesnt preclude better computable approximations eg continued fractions (by the constructivists) From shagun.balla7 at gmail.com Tue May 24 01:03:25 2016 From: shagun.balla7 at gmail.com (shagun.balla7 at gmail.com) Date: Mon, 23 May 2016 22:03:25 -0700 (PDT) Subject: Business Apps Developement Message-ID: <53667497-f901-4c56-b8a0-325a667b9ddb@googlegroups.com> With th?u??nd? ?nd thousands ?f u??r? ?nd ?n?rm?u? amounts of d?wnl??d?, Andr?id A??li??ti?n? D?v?l??m?nt field i? humming with ??ti?n. More visit. business apps developement From cloverobert at gmail.com Tue May 24 02:01:37 2016 From: cloverobert at gmail.com (Robert Clove) Date: Tue, 24 May 2016 11:31:37 +0530 Subject: JNLP File download and run In-Reply-To: References: Message-ID: Can u provide the pseudo code for the same On Fri, May 20, 2016 at 9:06 PM, Michael Torrie wrote: > On 05/20/2016 01:30 AM, Robert Clove wrote: > > Hi, > > > > Can someone give me pseudo code to download and JNLP file from a URL and > > run it? > > > > Looks like a advance concept in python > > You could use the urllib module to download the file, then use the > subprocess module to spawn the javaws executable and pass it the > location of the jnlp file as a parameter. > > https://docs.python.org/3.6/howto/urllib2.html > https://docs.python.org/3.6/library/subprocess.html > > There are other ways besides launching javaws directly, such as asking > cmd.exe to invoke "start" and the jnlp file so that the default javaws > executable will run. On Mac there's the "open" binary that can do the > same thing, and on Linux, xdg-open. > > -- > https://mail.python.org/mailman/listinfo/python-list > From greg.ewing at canterbury.ac.nz Tue May 24 02:49:52 2016 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Tue, 24 May 2016 18:49:52 +1200 Subject: for / while else doesn't make sense In-Reply-To: References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> Message-ID: Pete Forman wrote: > However I am coming from scientific measurements where 1.0 is the stored > value for observations between 0.95 and 1.05. You only know that because you're keeping some extra information in your head about what the 1.0 stored in your computer represents. It's not inherent in the value itself. -- Greg From mr.siyi.deng at gmail.com Tue May 24 03:15:49 2016 From: mr.siyi.deng at gmail.com (Siyi Deng) Date: Tue, 24 May 2016 00:15:49 -0700 (PDT) Subject: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 In-Reply-To: References: Message-ID: Thanks for all the replies. It turned out that the Apple OS X stock python 2.7 gives the wrong results, but other distributions like 2.7 from miniconda gives the correct results. Facepalm. From dieter at handshake.de Tue May 24 03:23:27 2016 From: dieter at handshake.de (dieter) Date: Tue, 24 May 2016 09:23:27 +0200 Subject: Setting Return-Path in email References: <69c455ff-0fbd-42a5-a3fb-4ad0b89ed48e@googlegroups.com> Message-ID: <87d1ob2a34.fsf@handshake.de> ragav s writes: > How can i add different Return-path and fromid in python.i have pasted the below code for preview Note that there are two different return paths associated with an email exchange: one on the protocol level (SMPT - RFC821); the other at the message level. The one on the protocol level is used for reports about delivery problems; the one at the message level is used for replies. You specify the protocol level return path as "from_addr" in your call to "smpt.sendmail". The message level return path is usually defined by the "from:" message header and can be overridden by the "reply-to" message header. From rosuav at gmail.com Tue May 24 03:45:41 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 May 2016 17:45:41 +1000 Subject: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 In-Reply-To: References: Message-ID: On Tue, May 24, 2016 at 5:15 PM, Siyi Deng wrote: > Thanks for all the replies. > > It turned out that the Apple OS X stock python 2.7 gives the wrong results, but other distributions like 2.7 from miniconda gives the correct results. Facepalm. When you use a binary shared library, it has to be compiled against the correct Python. You're messing around with ctypes, so basically you've voided your warranty; *everything* you're doing is platform-specific. Have fun. :) ChrisA From jldunn2000 at gmail.com Tue May 24 03:49:40 2016 From: jldunn2000 at gmail.com (loial) Date: Tue, 24 May 2016 00:49:40 -0700 (PDT) Subject: ZipImportError: can't find module Message-ID: <1e511b73-e984-459c-9311-778888bcd6cc@googlegroups.com> I am suddenly having a problem with importing a module from a zip file in Python 2.4.1 What has been working for years suddenly has an error : zipimport.ZipImportError: can't find module 'mymodule' PYTHONPATH is correct, it points to the zip file containing mymodule N.B. the python script(unchanged) is called from a shell script using python myscript.py The python script also has a shebang line that references the 2.4.1 install of Python. Debugging the script shows that PYTHONPATH is set correctly, it points to the zip file containing mymodule Any ideas? Platform is Solaris 10 From cheapfiverrservices at gmail.com Tue May 24 04:05:44 2016 From: cheapfiverrservices at gmail.com (cheapfiverrservices at gmail.com) Date: Tue, 24 May 2016 01:05:44 -0700 (PDT) Subject: Need python script to scan website with safe browsing site status Message-ID: i need a python script which can scan provided list of websites and scan that list with https://www.google.com/transparencyreport/safebrowsing/diagnostic/index.html and save result of dangerous website or infected website in another text file. From tjreedy at udel.edu Tue May 24 04:10:17 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 24 May 2016 04:10:17 -0400 Subject: [Python-Dev] Python 3.6.0a1 is now available In-Reply-To: <6a1fba2d-0313-59ec-c4f9-62f4206f84d4@gmail.com> References: <28E1448B-F604-48EB-B4E4-0AE2528C8383@python.org> <-7150763876064620774@unknownmsgid> <6a1fba2d-0313-59ec-c4f9-62f4206f84d4@gmail.com> Message-ID: On 5/23/2016 10:50 PM, Michael Torrie wrote: > On 05/23/2016 07:36 PM, Chris Barker - NOAA Federal wrote: >> It looks like you're still building for OS-X v. 10.6. >> I can't find an official statement, but this: >> http://www.computerworld.com/article/2950580/operating-systems/the-end-is-near-for-os-x-mountain-lion-support.html >> Indicates that 10.8 is falling off Apple's support. > [1] Dropping support for Windows XP is as much to do with newer APIs > that Python uses as anything else, if I'm not mistaken. CPython developers wanted to use newer API years ago, and perhaps sometimes did in #ifdefs. Dropping of support for XP was specifically tied to Microsoft doing the same. The same will be true for Vista, Win7, and Win8. https://www.python.org/dev/peps/pep-0011/ I don't know what will happen if 'Windows 10' continues indefinitely, even as it mutates internally. We will manage somehow, along with other Windows software providers. -- Terry Jan Reedy From mal at europython.eu Tue May 24 04:38:55 2016 From: mal at europython.eu (M.-A. Lemburg) Date: Tue, 24 May 2016 10:38:55 +0200 Subject: EuroPython 2016 Keynote: Rachel Willmer Message-ID: <5744131F.5060306@europython.eu> We are pleased to announce our third keynote speaker for EuroPython 2016: *** Rachel Willmer *** About Rachel Willmer -------------------- Rachel has been working at the "bleeding edge" of technology for 30 years, as programmer, network engineer, manager, startup founder: "I remain insatiably curious about how today's new technology gives birth to tomorrow's new business opportunity. I am CEO/Founder of Luzme, the ebook search site, and a Google Developer Expert (Firebase)." The Keynote: 30 years of Fun & Profit Through Technology -------------------------------------------------------- Have you ever wondered how you could be your own boss? or how you could make money from your side project? or build the next Facebook or Uber. To be a coder in today's world of work is to have amazing opportunities to design the business life you want. "I've enjoyed the last 20 years without a 'real job', as company founder, freelancer and side-project-hacker. Now I am bootstrapping my current company to profitability. Listen to my stories and learn from my mistakes and successes." With gravitational regards, -- EuroPython 2016 Team http://ep2016.europython.eu/ http://www.europython-society.org/ PS: Please forward or retweet to help us reach all interested parties: https://twitter.com/europython/status/735024712061988864 Thanks. From rocky at gnu.org Tue May 24 06:16:56 2016 From: rocky at gnu.org (rocky) Date: Tue, 24 May 2016 03:16:56 -0700 (PDT) Subject: RFC: name for project of a cross version disassembler, and unmarshal program In-Reply-To: References: <2e3189e0-c1e3-49fa-b6ce-53471c4374e7@googlegroups.com> Message-ID: <0cf6d0e2-3925-4479-8344-c217a51e66d2@googlegroups.com> On Monday, May 23, 2016 at 2:17:07 AM UTC-4, Pete Forman wrote: > rocky writes: > > > I'm looking for a good name for a relatively new project I'll put on pypy. > > > > I've been working on a module to disassemble Python bytecode from many > > versions of Python. (Right now 2.3 .. 3.5 bytecode, largely works.) > > > > Of course, in order to do that you also need routines to unmarshal > > bytecode. So that's in there as well. > > > > In the future, I may could add a marshaler and an assembler to Python > > bytecode. I know, this is kind of perverse. > > > > At any rate the name I've been using is "pyxdis". See > > https://github.com/rocky/python-pyxdis. > > > > In the past I've been told by Polish-speaking people that my names are > > hard to pronounce. (If you've ever heard any Polish tongue twisters, > > you'll know that this really hurts.) > > > > Any suggestions for a better name? > > relipmoc > > -- > Pete Forman I do like the name relipmoc. I'll mention one little thing though. The names of the package this will be used in is uncompyle6 which is a variant of uncompyle2, uncompyle, decompyle. Those all had the "py" in there and reversing throws that off. Or I suppose relypmoc. Or if I want to get the cross-ness in there relipmocx or relypmocx. Again, I'd be interested in what others think. The only other thought I had so far was thinking about was dropping the "py" from pyxdis and calling it xdis since it is a cross version "dis" module. But I think it is as well a cross-version marshal program and may become a cross version assembler too. Does this change anything? From thomas at povtal.org Tue May 24 06:22:30 2016 From: thomas at povtal.org (thomas povtal.org) Date: Tue, 24 May 2016 12:22:30 +0200 (CEST) Subject: Spurious issue in CPython 2.7.5 Message-ID: <779717266.18172.1464085350716.JavaMail.open-xchange@ox.netsite.dk> Hi, Please excuse me if this is not the right place, but I have some issues with CPython on a NUMA machine. 1: I get "RuntimeWarning: tp_compare didn't return -1 or -2 for exception". It's a line like: "if Foo = False:" where Foo is a global variable (global Foo). Now, I've searched somewhat on google for indications on when this warning can be seen. However, I haven't really been able to understand why and even if it's significant or not. (At face value I'm nervous the Python runtime environment is corrupted for that process). 2: In my process later on I get: "OverflowError: long too big to convert". This happens in different places and seems to always relate to obtaining a length of something (dict or list created by list comprehension). Fx "for i in xrange(0, len_of_stuff, max_section_size):" en_of_stuff is always less than the max long (around 600). We're using gevent and I'm suspecting some "threading" could cause this, as I'm able to replicate it locally with the same data. Kind regards, Thomas From jon+usenet at unequivocal.co.uk Tue May 24 06:54:09 2016 From: jon+usenet at unequivocal.co.uk (Jon Ribbens) Date: Tue, 24 May 2016 10:54:09 -0000 (UTC) Subject: for / while else doesn't make sense References: <5741cde9$0$1587$c3e8da3$5496439d@news.astraweb.com> <57422e03$0$1596$c3e8da3$5496439d@news.astraweb.com> <85r3ct9zt2.fsf@benfinney.id.au> <5742bacb$0$1526$c3e8da3$5496439d@news.astraweb.com> <87zirgrgol.fsf@bsb.me.uk> <574336f5$0$1600$c3e8da3$5496439d@news.astraweb.com> <5743b93e$0$1589$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2016-05-24, Steven D'Aprano wrote: > On Tue, 24 May 2016 03:09 am, Jon Ribbens wrote: >> On 2016-05-23, Steven D'Aprano wrote: >>> But one thing is certain: very few people, Jon Ribbens being one of them, >>> expects 1/3 to return 0. And that is why Python changed the meaning of >>> the / operator: because using it for integer division was deeply >>> unpopular and a bug magnet. >> >> Making it return floats is also a bug magnet, just for more subtle >> bugs that are harder to diagnose. > > Floating point arithmetic does contain traps for the unwary, sometimes very > subtle ones. But they aren't *bugs* -- they're inherent in the nature of > floating point arithmetic. You seem to be repeatedly changing your mind as to which behaviours are "bugs" and which are not. I didn't say that floating point is buggy (whatever that would even mean), I said that using it attracts bugs due to people misunderstanding how it works. > In Australia, we have an 11% consumption tax, the GST. I cannot tell you how > many times I've needed to add a 1 cent "Rounding" amount on invoices to get > the results to work out correctly. Indeed. Using floats for currency calculations is one of the many traps they present, and an excellent example of why their use should not be encouraged. > E.g. if an item costs $17 including tax, then you have a choice in > rounding the tax-free cost down to $15.31 or up to $15.32, I very much doubt that you have any such choice - it will usually be specified by the tax regulations. The correct way to do currency stuff is either integers or Decimal. > Using integers in this case is not only *harder* than using floats, but it's > more likely to go wrong. Because integer division always rounds down, > errors accumulate faster than with floats, where calculations will be > correctly rounded to minimize the error. You are falling into the float trap again. This is not how you do accounting. From palpandi111 at gmail.com Tue May 24 08:50:36 2016 From: palpandi111 at gmail.com (Palpandi) Date: Tue, 24 May 2016 05:50:36 -0700 (PDT) Subject: Unit test a class with xml elements Message-ID: <4c25c98f-9cf0-4a70-b33b-1d2c982de646@googlegroups.com> Hi, How can I unit test a class which using xml elements? There is a posiibility of different combinations of xml. What is the better way to test this kind of class? XML binding is used here. Share if any examples available. Thanks. From rosuav at gmail.com Tue May 24 09:18:25 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 May 2016 23:18:25 +1000 Subject: Spurious issue in CPython 2.7.5 In-Reply-To: <779717266.18172.1464085350716.JavaMail.open-xchange@ox.netsite.dk> References: <779717266.18172.1464085350716.JavaMail.open-xchange@ox.netsite.dk> Message-ID: On Tue, May 24, 2016 at 8:22 PM, thomas povtal.org wrote: > Please excuse me if this is not the right place, but I have some issues > with CPython on a NUMA machine. > > We're using gevent and I'm suspecting some "threading" could cause this, > as I'm able to replicate it locally with the same data. Perfectly good place to ask. Since you can replicate the problem, can you post the code that will trigger this? There are a few possibilities. I've once (not "once in a